ISTQB, Fever Dreams and Testing

Someone over at the self-proclaimed united nations of testers, also known as the International Software Testing Qualifications Board (ISTQB) published recently a letter (a white paper really) of what I assume is the result of having a fever dream after having watched Blade Runner 2049. Something tells me they probably were left thinking that the Wallace Corporation is the hero of the story (For those who’ve never watched that movie, a similar analogy is believing the iceberg was the lead hero of the Titanic movie). Jokes aside, my belief is that what the white paper means for the industry and the testing professionals should not go unchecked. You can read the full thing here.

Here are a few excerpts on the white paper:

“Software testing is moving towards more automation”; (…) “In the future, there will be more testing technology solutions” (…) “The testing technology solutions will have ‘intelligence’ built into them.” (…) “These testing technology solutions will iteratively learn the testing discipline from testers and testing experts” (…) “The testing technology solutions will also learn from themselves and, therefore, teach themselves as part of the iterative learning and evolving testing discipline, especially emphasizing testing quality attributes.” (…) “The testing technology solutions will have the ability to explain to testers and testing experts the rationale supporting their recommendations and solutions.” (…) “As more software solutions are being made available for use and as these software solutions expand into more industries, there needs to be a greater definition and integration of the training solutions for testing professionals.” (…) “Testing professionals shall need to embrace the new methodologies and technologies around software development and testing, with intent to integrate these faster into their testing technology solutions.” (…)

If you know me personally or from my blog writings, you probably guess what I’m thinking: A lot of potential hogwash in so few sentences, to avoid calling it something else. But I for one am going to hold my feelings for a moment, let’s give the white-paper the benefit of the doubt and dissect a bit what this might mean, and I’ll be doing it keeping in mind:

• The people who have respect and love for the Testing craft, unlike the ISTQB as an abstract group (and I’m willing to grant, not necessarily all of its individuals since I know quite a few of them who love and respect Testing);
• The people that in one way or the other oppose the ISTQB angle in the craft;
• The people whose pockets and savings accounts are in the ISTQB crosshairs;
• The people whose pockets and savings accounts are in the crosshairs of groups or individuals that in one way or the other oppose the ISTQB angle in the craft for their own gain;
• Anyone else who may care. There are three points I think to make sense to explore: “automation”, “intelligence” and “embracing” the ideas behind the white-paper.

The “move towards automation”

The “saying” that the craft is pointed in the direction of automation holds some truth to it, as well as horse droppings. It’s undeniable that, from a “buzz-word” standpoint, indeed Software Testing seemingly has been moving more towards “Automation”. The common industry-accepted understanding is that there is such a thing as Manual Testing and the paired industrial understanding that where you see the word “Manual”, there’s bound to be a machine that can do it better, faster, cheaper. And in this sense, there’s no mute button for reality, and the reality is New tools. New software libraries. AI advertising of “it’ll do it for us”. New ways of representing a view in an object. New ways to call an API or publish a message in the context of a scenario that will assert something in the end. New ways of organizing “testing” code. etc. Even personally, it’s undeniable: I spend a portion of my working day coding, using tools, and maintaining scripts, and I am a Tester. The horse dropping part of it all comes when you confront the “buzz-word” with different abstractions of the whole actual Testing phenomenon, and in its truest raw form.

Testing is undeniably human.

There’s simply no way around it. Reducing Testing to logical assertions written in a programming language, or mixing that with some “intelligence” (aka. some “dumb” machine learning model), is indeed a faulty approximation of Testing as a whole, that comprises: investigating, exploring, purposeful play, dealing with confusion, risks, problem hunting, communicating, representing, and yes, also logical checking.

Excellent testing is a mix of all of these things, and all of them bear the undeniable mark of their creator: the human being.

The confrontation of the” buzz word is actually horse droppings” has been happening now for maybe almost two decades (maybe more), and folks left and right have come forth on plenty of occasions saying both “true-ish and horse dropping” words of disorder: “Test is Dead”, “Automation Testing is the future”, “Bots will replace human testers”, “Agile testing is a thing”, “The death of manual testing is NOW”, “Let’s all get along, testers are cheerleaders”, “The role of QA in Agile Teams” … The above ends up being paced alongside their counterpart: individuals and small groups of people pointing out that:

• No, testing isn’t dead, it’s more alive than ever. In fact, there’s an ever-growing hunger for testing that is not braindead but is rich with substance and purpose;
• No, you can’t automate testing, because testing is not only “logical checking”;
• No, bots can’t replace humans in Testing. They can aid in specific tasks;
• No, agile testing is not really a thing. You have testing and its surrounding context where it’s inserted;
• No, manual testing is not really a thing, the same way it doesn’t make sense to say manual thinking, manual questioning
• No, we can’t get along. We can still be friends outside of work, even lovers, but we need to have some level of critical distance to hunt down certain bugs and problems; • No, I can’t QA that for you. You want me to TEST it? Did anyone tell you yet the meaning of Quality Assurance?

And there are always these two sides, who both offend and feel offended, invested in a struggle between

• those who have over-simplifying models for something chaotic that is strongly dependent on its context,
• and those who oftentimes enclose themselves in tribal behavior, whose ideas suffer for being produced by faulty and imperfect humans.

Curiously enough, I think both sides would be able to come up with ways to tag any of the above descriptions to the opposing side. Every year there’s a whole living and breathing market for training, conferences, and posts built around promoting or deconstructing these words of disorder, propagated globally faster and faster on every iteration. Such is life.

The “intelligence” I admit there’s bound to be the software now and in the future that can mimic certain aspects of our natural intelligence. I also admit there is the possibility to build intelligent agents who can greatly surpass humans in certain specific capabilities or very specific activities.

To be honest, also I’m not prepared to go into all the ethics and existential issues that surround many of these topics. I have zero desire to trash the past and ongoing work of people who are dedicating their lives towards investigation in artificial intelligence, machine learning, or its several brain-children like deep learning, and networks, and specific algorithms.

But, I can talk about what I’ve experienced first-hand in one story to elaborate on my skepticism on any “intelligence” sales-pitches arguments.

My skepticism on implementation

There’s a couple of startups spewing around selling the “idea” (and implementation) that they’re developing “robots” with some capability of learning (in the shape of software as a service) that will go through your App or API and will signal inconsistencies, possible problems, and even predicts pain points, all in neat dashboards and well-integrated with any typical continuous-integration pipeline.

As Mr. MacMahon would say in character in a WWE show if he were a software person – “We’re developing this solution to finally replace those filthy and smelly humans and- *GASP!* (he gets surprised whipped by Steve Austin)”

If it’s intended to help/assist, then I don’t really have a problem with software that helps me or conveys me some bits of useful information. Anyone who knows me knows I’m somewhat of a freak adept at tooling like (hardcore) static analysis, proper unit tests (aka checks), and using code quality tools in hardcore and the extensive way if we’re serious about the whole deal, and especially in open-source projects.

The problem is this crap stuff is advertised as “the next step”, one to replace Testers in practice. Added to the mix, it might be the case the main “gurus” behind the foundation of said startups and software have very established credentials, and a subliminal, if not natural, disdain for the Testing craft (oversimplifying it to plain checking at different levels) and in terms of their usual lingo are the typical shallow “automators” – meaning they’ve done in part the same I also do every day – coding automated checks – but they elevated the checks to a replacement of deep and meaningful testing. Now, I won’t go ad hominem straight away. They don’t love or respect the craft with their time given on this earth, that’s fine by me. But using their established status or their history,

“I worked at EvilCorp for N years as Omega Director of Testing & Quality” (Yup, Mr. Robot reference) to sell “air in a jar” software, directly mocking Testers, that I can’t stand.

Opening the “air jar software” I invest some of my free time hacking away at any piece of software I can get my hands on. One of those sessions last year was spent hacking away at one of these fancy “Automated AI Testing” services. After some time tinkering around I was able to notice it didn’t actually do anything as advertised, and their entire Admin API was badly protected, so any attacker was also able to create accounts with admin permissions, spam the service with crap data and do a couple more mischievous stuff. Suffice it to say:

• No, it didn’t seem to work as advertised, it didn’t replace me as a Tester, and it didn’t even replace the most clumsy open-source static analysis or code quality tool out there.
• If any of my past, current, or future employers would ever upload a build artifact for that service, I would be the first one to warn that any script kiddies will be the first to get their hands at anything sitting in storage on that service.

As an extra point, and none of these companies will probably do this in the next 10 or more years if there’s a service that “tests” my build artifacts, the only feasible way I see of remotely adopting this as a helper stage in my CI pipelines is if I have full open access to what exactly that service does, and the “if-else” chains in the scripts it’s made of (advertised as AI). And by open access, I mean also opensource access to its code. Why? Two reasons:

1.  Evidence of testing & Test Reporting.

It’s as simple as this: as a Tester or a Test Manager, I expect myself or any of my Testers to produce and convey evidence of their Testing. I want to understand at any stage in a clear and human way:

• What was tested?
• Wow was it tested?
• What was left out?
• Which tooling and automated checks are on the scene?
• What was confusing?
• What wasn’t looked at?
• What are the main risks for different actors (example: what’s important for say, a regular coder, is not necessarily important for a sales rep, and vice versa…)?

And no, I’m not talking about some shallow test-case counts, percentages, graphs, push notifications, some “superficial” test reports, among many other things like this that are sold as “evidence of testing” or “quality reassuring”.

2. Power to the people.

Yeah, I’m going that way. Sure, GOOD Testing is expensive. But it is also my belief Testing is “free”, and I mean it as “free” as in “freedom” (not necessarily as in “free beer”). Knowledge surrounding testing and other topics should not be contained behind a corporate or academic veil. It should be free and available for the thirsty. 

Building a monopoly and then charging money for infrastructure plus some obfuscated service that has scripts that perform some automated checks on your build artifact is as sketchy (if not more) as it is selling licenses (plus painted-over infrastructure) for “load testing tools” when there’s plenty of open tools that work better in almost all cases, it’s just that there’s usually laziness involved in the load generation infrastructure setup part, and excuses start popping up.

Embrace or perish

If it’s by chance, it’s probably an unfortunate one that we read throughout the white-paper:

(…) there needs to be a greater definition and integration of the training solutions for testing professionals (…) (…) professionals shall need to embrace the new methodologies and technologies (…)

And let’s think for a moment… who are the self-proclaimed responsible entities that we know about that by mere causality… are the de-facto go-to “industry nice-guy/nice-gal” that sells, “validates” and “certifies” training solutions for testing professionals?

I’m not trying to imply anything, but let’s take this to an imaginary surreal fairy-tale scenario…

A tale of towel salesmanship, by Filipe F.:

I’m born Portuguese, so I’m by default a certified Towels salesperson, right?

Let’s say I’m also the founder of the International Jedi Council of Portuguese Towels (aka the IJCPT). We promote sell programs and certifications to validate other well respected and certified Towels salespeople like myself.

So tomorrow I go about my business and announce to the world that the future for towels-selling is Towels that have a special fabric called “smoke”. And where can people get training and certification? Well, I’m happy to say that the IJCPT is keeping up with these latest industry trends, and fear not, we can train sell any decent and concerned Towels salesperson preparation.

There’s a slight problem with the IJCPT… you see, we have a whole organizational structure, a bit like one of those genuine-honest-to-god-not-a-pyramid-scheme multi-level marketing institutions that are around? Do you know those? But hey, you can trust me, I’m trusted in all things Towel-selling by the whole industry by all my non-Towel-selling colleagues (like Senior Leaders, Recruiters, Managers, …). And for the record, I’m not saying “s-moke” Towels will 100% certain be the future… but I predict they will, and you should all pay attention to what I say.

So don’t forget: In practice, in this industry, no one else is acceptable in the eyes of God or Nature to “validate”, “certify” and “assure” your Towel selling skills. So you better take my expensive affordable certifications and training or you’ll be out of a job once the “s-moke” Towel technologies reign.

Fin.

Not implying anything, but you get what I mean, how this all might look weird for any “layman” in the end?

The last remark

Enough ranting.

I’d like to direct this part to the team behind the white paper if they ever read this post:

Yaron Tsubery, Dave Miller, Patricia A. McQuaid, Ceren Sahin Gebizli, Lucjan Stapp, Dani Almog, Filipe Carlos, Miroslav Bureš, Ritendra Banerjee, Adam Roman, Capers Jones, Alan Page, Bill Helfley, Colin Onita, Erik van Veenendaal , Harry Sneed, Jakub Rosinski, Klaudia Dussa-Zieger, Mark Gillenson, Olivier Denoo, Padmanabhan Balasubramanian, Paul Xzhang, Przemyslaw Niedzialkowski, Radoslaw Smilgin, Robert Binder, Robin Poston, Stephan Schulz, Steve Hutchison, Yao Shi, Alon Linetzki, Armin Metzger, Jasbir Dhaliwal, Kari Kakkonen, Leanne Howard, Michael Stahl, Tamas Stoeckert, Tilo Linz.

Fellas, I’m just a random dude. Not accomplished in much. Definitely not a bright mind. Ask some of my family members or old and current colleagues, they can tell me about all the times I was a jackass. It’s not necessarily brave from my side to also just take on some “group”, without considering there are real human beings behind it. You wanna make a buck selling (or imposing to the industry) training and “I’m certified” stickers, be my guest: I don’t like it. It’s no secret I’m against it, and my personal experience tells me time and time again, there is indeed a better way. But really, I’m not in an ideal position to judge any of you.

I would like for you, if any of you reads this, to keep this in your hearts: the future we will all get to live on the little time we have on this earth, in part is a future of our choosing, both individually and collectively. You’ve made your choice, and probably the near future will prove it the right choice for your pockets, personal well being and standing in the overall community. History is curious though, everyone says History doesn’t forget.

“People don’t forget, nothing gets forgiven” would a character say in the RDR game.

We unconsciously post our inner lives online, fly on planes, install security devices in our homes, drive to work, order food online, check our savings in a half-second, invest in Tesla stocks with a tap of our fingers, and later on, we will get automatically transferred our retirement money every month. Each of these things and so many more of them is built by imperfect humans, tested by imperfect humans (when and if they ever get meaningfully and decently tested), and sometimes, sure enough, kill imperfect humans. And I’m afraid because we don’t have to go very far off into the future, the present times prove to us that to make an “intelligent automated software testing” soup, or “half-assed certified” testing efforts and think it’ll remotely replace a craftsman and imperfect tester, is an entryway for more and more embarrassing bugs, which will potentially lead to the demise of many fellow humans. And the first responsible won’t be the tester or the developer who either missed finding the bug or inadvertently coded the bug. Neither will it be the greedy businessman standing behind them.

The first people responsible are the educators and in the limit, the “certifiers”. You may never be convinced that Testing is human to its core and instead paint it over with terrible bad jokes of science fiction (unworthy of the less good P. Dick or R. Heinlein novels), but keep in mind there are always consequences, positive and negative, in educating about Testing, whatever your angle is.

As educators, never forget that if we live, breathe and mostly make a profit of “defined and integrated training solutions and certifications”, for our fellow imperfect humans, nothing will ever clear any blood we may gather on our hands through time.

Take care.