Addressing The Risk Of Exploratory Testing Part 3

In my experience, by far the biggest risk to the successful implementation of Exploratory Testing (ET) is the complete misunderstanding of what is meant by the term. In this series of articles, I‘ve shared the concerns that have arisen whilst implementing ET and some potential solutions to overcome them.

In the first article, I addressed how to help the test team gain the confidence to Exploratory Test. In the 2nd, I shared my experience getting stakeholder buy-in. Once teams & stakeholders understand and agree that ET allows for better testing, it‘s up to the Test Manager to ensure that the team deliver. This last article will make some suggestions for:

How to manage a team of Exploratory Testers

There are several concerns a Test Manager may face when contemplating the implementation of Exploratory Testing. Not least of which is…

Where to start?

Imagine embarking on a brand new project. Your stakeholders have bought into the efficiency gains so you are free to use Structured ET throughout your testing…. so, ‗where to start‘ may not be an issue for you. That said, if you have never worked with a team of exploratory testers before, you might still be looking for a suitable start point from which to build your own confidence?

Have you considered when you may use ET already? Realising where we already use ET can help us gain confidence. What happens when the developers give you a bug fix release? Do you create new scripts for each bug? Or do you trust your testers know to retest the bug? In this situation, a tester not only tests the fix, but they also make informed decisions to perform regression testing. Understanding and testing whether the fix has broken something else is a perfect example of how teams use ET daily, yet rarely talk about it.

Perhaps you run your smoke/ sanity test on each new release from a checklist rather than a script? If so, your team are already using Structured ET (they have a documented mission to achieve, but some freedom in how to execute the test each time). When you understand how you already use ET, it becomes clearer where to start allocating official time on your testing project plan.

Consider trying ET at the beginning and again at the end of your testing cycles. Allow it at the beginning so your team can root out the more obvious showstoppers early on. Using ET at the end gives your testers a ‗last chance to break it‘ session, once all other tests are complete. Either of these will allow the safety net of your usual scripted tests, plus the chance to gain metrics on how many more defects/ issues were found when you used Exploratory testing methods.

So you are convinced that, with the right testers, ET will definitely add value. If you are still reticent, maybe you are really concerned about.

Are my team good enough to use ET?

Traditionally Test Managers see the number of new test scripts created each day. We may consider the output to see if our team are being productive. Numbers could make us challenge if John only wrote 5 tests today but 10 the day before. But can we realistically check the quality of every written test?

Structured ET actually allows a Test Manager to gain far greater insight into the quality of their team. By attending the Analysis sessions I talked about in article 1, the TM gains the first-hand experience of the quality of test ideas and the knowledge levels of the team members. It offers you the chance to question their test design, and even help to improve it (before the work has been done). Being in the analysis session adds major benefits to the test manager. Not only closer, verbal interaction with team members, which is shown to improve morale, but what better way to get quickly up to speed with the project. Not all TM‘s have the chance to fully analyse the specs. If you have more than 10 testers working for you, how could you individually check the quality of everyone‘s work? By taking part in these sessions you not only improve your project/system knowledge but if any of your team is struggling to contribute, you will learn very quickly and have chance to take action before testing starts. If your team use mind-maps to create a visual test model (see http://lnkd.in/hxhwpm), you can also make a better assessment of the quality of your team‘s work.

Regular Debriefs

Another aspect of structured ET, which aids the test manager, is the Debrief sessions. At a logical point each day, the whole test team meet. Each member has no more than a couple of minutes each to discuss:

1. What they tested today & deviations from the plan
2. Any challenges faced / risks identified
3. What they are testing tomorrow

You may feel too busy to attend this meeting regularly, but Test Managers who allow their testers to use ET are publicly stating, ―I appreciate that things (scope, system behaviour) will change during the project, and our plan will need to adapt accordingly‖. To successfully manage an ET team, the TM should attend this meeting daily, so they keep up with what has been learned by the team. This is very similar to a Daily Stand up in scrum/ agile and just as essential to the success of ET. It‘s another chance for the team to question/ challenge each other on how they tested features and to share knowledge about problems faced. Lastly, it‘s a great way to ensure nothing gets missed… and if it has been, at least you find out the same day.

Coaching Skills

We all hope that our manager will be a great coach. Someone, we can look up to who can help us improve our skills. Encouraging ET usage in the team not only improves morale but also gives fantastic opportunities to foster collaboration and coaching within the team. Pair testing is a great tool for this. Even 2 testers with no system knowledge will likely achieve more in an hour of pair testing than if they were left to work alone. If you have not been a hands-on tester for a while, why not get ‗back to the floor‘ and spend an hour with each of your testers to see how they test and offer some coaching. If that sounds daunting (especially if you can‘t remember the last time you tested something yourself), challenge yourself! Be open to learning something new. Set an example. Foster a team spirit that allows some mistakes. Appreciate that they can be learned from. Make sure the team know there is no failure, only feedback that can help us all improve. Get your team discussing testing and various techniques/approaches. Even set mini-challenges/ competitions as incentives to demonstrate an improvement in test skills. Building confidence and awareness of testing skills are essential in an exploratory testing team … and will create a team of testers that you can be proud to manage.

What about new joiners?

How can we train new joiners without scripts? This question comes up in every course I run. I counter it with a show of hands to the question ―Did you learn your current project by executing/reading other peoples‘ scripts?‖ Less than 10% learn by this method, yet us testers seem to think test scripts are the best source of knowledge. We have several teams who no longer use detailed test scripts. But, as per article 1, this does not mean we don‘t have any documentation. We build wiki‘s containing business/ technology information. We build Visual Test Models that are easy to learn from. We have checklists to guide testing. Combine this with pair testing sessions with an experienced tester and your new joiner will immediately feel part of the team. I‘ve seen the evidence that our new joiners are now delivering value to the team faster than before. We have even had feedback from developers that our new remote location testers are asking excellent questions! They are seeking information about the product, instead of asking dev to tell them how to test it. We are certain this is because knowledge was not handed to them. The expectation was set on day one that to contribute to this team, they must ask questions and develop their own test ideas. Additionally, they are immediately made responsible to ensure the wiki is up to date.

Obviously, you need to ensure when interviewing new joiners that they are up to the challenge. They do not need prior ET experience but they do have to show they are capable of original thought and demonstrate questioning skills. And why would you want anyone in your test team who could not do this?

Over a period of some months, the ideas above start taking shape. You and your team are now confident that structured ET can enable efficient testing and earlier defect detection. You‘re ready to use it more widely but want to know…

How can a Test Manager plan and measure test coverage without scripts?

Simply put, by using high-level test scenarios, checklists or charters. These act like test scripts, allowing the TM to estimate, plan coverage and allocate testing amongst a team. If you aren‘t comfortable to change everything on your project, you can still map requirements and defects to these scenarios. If necessary, they can be used as scripts for all other purposes, just that they have less written detail and therefore take less time to create and maintain (time that can be better spent on analysis and capturing important info on the wiki). Of course, this structure is needed in most companies, hence the term Structured Exploratory Testing. Another skill to develop is the ability to take good test evidence. My advice here is to take a look outside of the usual test management tools. Some excellent test evidence capture tools have emerged recently, which are either free or very affordable and feature-rich. If you are also concerned about how to report, read the section in my 2 nd article, that explains how to report coverage without needing scripts.

Like the rest of technology, new solutions are developing all the time. As testers, we also have to continually improve. We must develop skills that allow us to react to the changes around us. We must show our projects that we are a service to them, not a blocker. That we add value and are true system experts … else we may find ourselves sympathising with the dinosaurs!!

Structured Exploratory Testing (and the excellent analysis which must precede for it to be successful) has been the differentiator for the testers I work with. Every team who has taken on the challenge of learning and implementing SET, those who work hard to improve the way they communicate with the rest of the project team about testing, have achieved universally successful results, not just improved quality & efficiency in testing but in their reputation with the project team and the morale too.

You know you can achieve this in your team too, can‘t you?