Common Testing Mistakes – Are We Really Evolving

Last year when I was working on creating the #TestFlix e-book with Sandeep Garg, we used to have a lot of candid discussions around Quality, Testing & Life in general. Sandeep told me about Jerry Weinberg during one such discussion. It was the first time I got to know Jerry Weinberg. I started with reading his quotes, followed by some of his popular books like Secrets of Consulting & General System Design Thinking (still in progress :D).

A couple of months back I stumbled upon his book on Testing during a Happy Book Reading Club Session, i.e., Perfect Software & Other Illusions about Testing, which was released in 2008. At first, I thought that it might be outdated as we are in a technological era where things become irrelevant in months. Having an experience of Jerry Weinberg’s books, I knew that there might be some good stories and incidents in this book. I decided to give this a read to enjoy the old stories and get a feel of the old days’ testing issue(s).

I was enjoying the stories, incidents, and jerry’s style of writing in this ebook until I came across a section of the book where Jerry talked about some Common Testing Mistakes. When I read that list, I felt all these are still the day-to-day problems for testers and/or testing.

I shared this post on LinkedIn that day and went to sleep.

The next day when I woke up, my phone was filled with LinkedIn notifications (Thanks to Michael Bolton for sharing this post and helping boost my post reach), and I understood that what resonated with me last night also resonates with a lot of testers out there. The post got 100 re-shares and was read by a great number of Testers, Developers, Managers, Consultants, etc. The comments on the post affirmed that these Common Testing Mistakes are still very common and highly relevant.

Over the last 2-3 decades, we have all seen the evolution of software testing tools, resources, processes, testing methodologies, and a lot of things. However, after witnessing 30+ years of software testing journey as an industry, if the mistakes are still the same then it posts a very serious question in front of us, i.e. Is we really evolving? and if not, then why not? I feel it’s a lot because of getting caught up in the same old traps and ignorant behaviour towards reality.

Here, I would like to elaborate on each of these Common Testing Mistakes, Traps, and Ignored Reality:

1. Thinking that locating errors can be scheduled

The only way we could know how long it would take to locate bugs would be to know where they are. If we know that, we wouldn’t have to locate them at all.

Traps:

• Thinking of testing as a mechanical activity rather than a cognitive activity.
• Thinking that CI/CD/CT would eliminate the need for dedicated testing periods.
• Thinking that Green Automated Test Runs is equal to a quality product.
• Thinking that Automation will solve all our problems.

Reality:

• Testing is inherently exploratory.
• Scheduling locating errors is like Scheduling the arrest of criminals. NOT Possible for the majority of cases. Both need dedicated investigation, analysis, and evaluation.
• Testing is driven by Heuristics, which are fallible ways of solving a problem. They may fail too.

2. Not considering the time lost to task-switching

Task-switching can be beneficial, as we’ve seen, but like anything else, it has a cost. Each task switch loses a bit of time, so if you’re switching among about five tasks, you may be accomplishing nothing. Most people react to that situation by simply dropping some of the tasks altogether, which can be dangerous.

Traps:

• Thinking of Multitasking as a productivity skill.
• Thinking of testing as a passive activity that can be clubbed with another activity.
• Accepting and acting on all the tasks as and when they are around.
• Not planning your work.

Reality:

• If you multitask, you are bound to regret it, sooner or later.
• Responsible Testing requires a focused effort.
• Software engineering is a systematic application of engineering approaches to the development of software.
• If you fail to plan, then you plan to fail!

3. Treating testing as a low-priority task that can be interrupted for just about any reason.

Testing requires concentration if it’s to be done reliably.

Traps:

• Thinking that anyone can do testing.
• Thinking of testing as checking.
• Measuring testing progress with the number of bugs.
• Thinking of testing as ONLY Verification

Reality:

• Yes, just like cooking, anyone can do testing to some degree. But not everyone can do responsible testing.
• Quality is a multi-dimensional concept.
• The value of one bug could be sometimes greater than 50 other bugs. Bug count says NOTHING! COMMON TESTING
• Testing is NOT Verification, Testing includes Verification!

4. Demanding that testers pinpoint every failure.

Testers can help developers with this job if their time has been scheduled for it, but it is ultimately a developer’s responsibility. At least that’s what I’ve seen work best in the long run.

Traps:

• Thinking that testers should become developers and help pinpoint every failure.
• Thinking of problems as definite issues (Ex: True or False).
• Thinking that testers are mostly free and don’t do anything else apart from test execution.

Reality:

• Testing & Development mindset are totally different. Testers are good at thinking about hypothetical issues (what if?) instead of Imperative thinking (do this!).
• Most problems are beyond the definite boundaries of Yes and No. They require human judgment and deeper investigation. Single observation can have multiple interpretations.
• Testing is NOT EQUAL to Test Execution. It involves a lot of other work like designing tests, preparing test data, studying and understanding requirements, writing bugs, tracking bugs, evaluating customer issues, etc.

5. Demanding that testers locate every fault.

This is totally a developer’s job because developers have the needed skills. Testers generally don’t have these skills, though, at times, they may have useful hints.

Trap:

• Thinking that all faults are straightforward.
• Many organizations fake such claims to customers while billing testers under the name of Full Stack, T Shaped, or Star Testers.

Reality:

• Most faults hide behind the multiple layers of abstractions of the software. They need an understanding of the different layers and knowledge regarding the flow and design of the software.
• If it sounds too good to be true, it’s probably NOT true.

6. Repairing without retesting

Repairs done in a hurry are highly likely to make things worse. If you’re not in a hurry, you might be careful enough with the repairs not to need retesting, but if you’re not in a hurry, why not retest?

Trap:

• Thinking that repairs have been done diligently and the testing phase can be bypassed.
• Developer Optimism • Overconfidence on Unit Tests.
• Thinking that one more round of Testing might again destruct the product.

Reality:

• Most of the last moment fixes end up leaving side effects in the system.
• Over-optimism disillusions reality!
• Unit tests are just small fact checks and nothing more.
• Testing never destroys anything except the illusions of overconfidence.

7. Ignoring cross-connections

Commonly, the actions of programmers drive the need for testing, so that testing and programming are bound together. For instance, if programmers deliver code to testers late or in sub-par condition, then you’ll have to adjust test expectations.

Trap:

• Thinking of testing as an independent activity.
• Thinking of testing as fixed and defined.
• Ignoring the cost of bug analysis, investigation, advocacy, and retesting.

Reality:

• Testing is as much part of software development as development is. It’s NOT independent.
• Testing is inherently exploratory and continuously unfolds in newer dimensions over time.
• Side activities around testing often take more time than test execution.

8. Paying insufficient attention to testability

Code that is designed and built to be testable can greatly reduce the time and effort associated with all aspects of testing

Trap:

• Thinking that a magic framework would fix all testing challenges.
• Thinking that tools will amplify software testability.

Reality:

• The secret ingredient behind a good testing solution is a testable code.
• If testability is negative, tools would only amplify the negativity.

9. Insisting that all bugs be “reproducible”

Intermittent bugs need to be pursued with great vigor, not used as an excuse to delay testing or repairing. Use what information you have, and don’t waste testers’ time with unreasonable demands.

Trap:

• Thinking that each problem should be a result of some controlled parameters/configuration.
• Thinking that test system and reference system are equivalently configured.

Reality:

• During Testing, we only control some inputs and observe some outputs (results).
• Often the non-trivial seeming details like platform, runtime, supported libraries versions are not mentioned or ignored. Do you really know everything about your test system? Think again!

10. Confusing testing with “creating and executing test cases”.

Much of the testers’ work is not captured or encapsulated within the confines of an identifiable test case. Consider thinking instead in terms of test activities.

Trap:

• Thinking that Testing = Creating & executing test cases.
• Thinking of Testing to be revolving around test cases

Reality:

• Testing is much more than test cases. Testing ≠ Test Cases.
• Good Testing involves many tasks like research about products, bug hunting, analyzing specifications, creating data sets, creating reusable tests, creating checklists, research failures, writing persuasive reports, etc.

11. Demanding process overhaul in your company

If you work in an organization that thinks it’s the testers’ job alone to pinpoint and locate bugs or fetch coffee and donuts for the developers, you may now be tempted to confront your boss with a crass, revolutionary attitude. Even if you’re interested in increasing respect for testers in your organization, you’re not going to change the environment with rude demands. You might say to your boss, “I read that a tester is someone who tries different methods in order to make things work. I’ve come to suspect our testers could do better work if some things were different here. I’d love to have some time to discuss these things with you.”

Trap:

• Thinking that changes could be brought overnight.
• Thinking that new tools will solve all old problems.
• Trying to solve people’s problems with technical tools.
• Looking for rational logic over reasonable logic.

Reality:

• Marvin’s First Great Secret: Ninety percent of all illness cures itself – with absolutely no intervention from the doctor. Deal gently with systems that should be able to cure themselves.
• Most people can successfully absorb 10 per cent into their psychological category of “no problem”. Anything more would be embarrassing if you succeeded.
• No matter what or how they are saying, it’s always a people’s problem.
• If logic always worked, nobody would need consultants (or consultation). Consultants often come across contradictions.

After writing a lot about all the traps and reality, I would also like to share some references and must learn things based on my experience on how we can avoid these testing mistakes in our professional life:

• Learn and Master the Testing Foundations (Reference: BBST Foundations)
• Learn about Heuristics & Mental Models and why they matter (Reference: James Bach TTT Tribecast)
• Practice Time-Bound Charter Based Exploratory Testing (Reference: Explore It)
• Understand the importance of mindset for focused and good software testing (Reference: Blogpost on Mind, Matter, Testing and The Cargo Cult by Lalit Bhamare).
• Know that Testing is NOT Testcases (Reference: James Bach CAST 2014 Keynote)
• Learn and practice Bug Advocacy Skills (Reference: BBST Bug Advocacy)
• Learn about Consulting and some of its secrets (Reference: Secrets of Consulting)
• Learn Influential Writing (Reference: Blog post on Coaching testers how to email and influence stakeholders by Pradeep Soundrarajan) 

These topics and references are some of the most useful resources which have helped me to avoid many of the common testing mistakes and succeed as a professional tester. I hope that this would help you too in some way.

If you feel there are more resources, which should be added to this list, please share your thoughts and feedback with me @parwalrahul.

All the best for your quality journey!