Addressing The Risk Of Exploratory Test Part 1

In my experience, by far the biggest risk to the successful implementation of Exploratory Testing is the complete m also trying it for the first time.

Three years ago I was the first test manager in my company (to my knowledge) to formally introduce Exploratory Testing (ET) as a phase in a testing project. I easily articulated some of the benefits it would bring to the project. Earlier testing = earlier clarification of requirements, not to mention earlier, and therefore cheaper defect detection. I was able to deliver these benefits to the project, but my lack of true comprehension with regards to the definition of ET meant I did not fully understand how to support my team, who was also trying it for the first time.

In this series of articles, I will share the concerns that have arisen whilst implementing ET and some potential solutions to overcome them. This first article will address:

How to gain the confidence of the test team

After the first attempt at implementing ET, I attended James Bach’s Rapid Software Testing training course. I expected 5 days of training on Exploratory Testing. What I actually gained was so much more… 5 days of training on RST, of which ET is just a small, but essential part. James‘ course made me realise the wealth of underlying skills and techniques that anyone calling themselves a ‗tester‘ should strive to develop. It was these skills and understanding that my test team needed to develop, to be confident exploratory testers (not to mention better testers overall).

Isn’t Exploratory Testing just another name for Ad-hoc Testing?

This is often the first misconception that arises during the training courses I run. Of course, it could mean ad-hoc… to run ad-hoc tests, you would most likely be exploratory testing. However, we can still gain the benefits of ET (early testing, early effect detection, less script maintenance and rework), whilst placing sufficient structure around it to be confident of the coverage we will achieve.

More and more I use the term ‗Structured Exploratory Testing‘. ET can be structured because:

• We are not just banging a keyboard without thinking about our actions or the result.
• We do plan & communicate in advance what we are going to test.
• We do analyse and understand what we expect to see when we test.

But it is still exploratory testing because:

• We have not planned the exact steps we will take to perform a specific test.
• The tester has the freedom to choose the next best test to perform based on what they learned from their analysis, plus what they learn about the system whilst testing
• The test is repeatable but will not be performed in exactly the same way each time it is run.

How can a test be repeatable, if it cannot be run the same way each time?

This is another concern often raised by testers. Some in the testing industry have become so ingrained that a test must be performed exactly the same way each time (perhaps by companies selling very expensive automation tools, or very ‗robotic‘ testers?) that they miss the value in running the same test in a slightly different way. Think… when you use MS word, do you use it exactly the same way each time? Sometimes I use the mouse or sometimes a keyboard shortcut to achieve the same action. If one user performs actions differently, imagine the variance when a system has multiple users! Perhaps randomness in test data is important… or the number of times you click on a button, as a user may do in frustration when the application appears to be frozen (which often causes a complete system crash.. try it!). Any of these variances could give you important, new information about the product you are testing… but would it be possible, valuable or ethical to spend your company time writing a detailed test case for each of these scenarios?

When we repeatedly run the same test, we simply confirm what we already know. Would it add more value to learn something new about your application? As long as you test the core functionality, it is often cheap, yet informative, to change the specifics of how you test. And if it really is critical that the test is performed precisely… then congratulations, you have identified one of those rare occasions where Exploratory Testing would not be suitable… so write a script!

So, let me define Exploratory Testing! When using the term ‗ET‘ I mean ‗simultaneous (and cyclical) learning, test design and test execution’. With this definition in mind…

How can testers be sure not to miss anything important when testing, if no scripts are written beforehand?

Firstly, I want to clarify that ET does not mean ‗No documentation‘. We created documents known as ‗charters‘ or high-level test scenarios. We backed these up with lightweight but informative test matrices (See figure 1). As a manager these were sufficient assurance that my team was achieving the coverage required during testing. I had spoken with them and trusted they understood what and how to test, without writing it down first.

What surprised me was that some of my team struggled with the absence of the test script creation phase. Personally, I found writing test scripts a waste of time, as the steps are rarely followed to the letter during execution anyway. I have since realised that for a lot of testers, writing scripts is the method they use to develop their understanding of the system, and therefore design their tests and the expected results. These testers were not comfortable letting go of a process they‘d used for years, without having a new one to replace it. This removal of the structure is a big cause of worry in some testers. They fear they may miss testing something important if they haven‘t written down their test ideas or expected results in detail first.