What is Charter Driven Session-Based Exploratory Testing? Exploratory Testing

There have been a few attempts to define exploratory testing. To begin with, I suggest that all testing is exploratory and thus exploratory testing is just another word for testing.

I started using a definition from Cem Kaner in the mid-1990s to distinguish pre-scripted testing from exploratory testing.

Cem Kaner taught me that exploratory testing could be viewed as concurrent test design, test execution and test-related learning which included planning and refocusing based on what we learn as we test.

Cem Kaner eventually held a couple of small peer workshops about exploratory testing. Eventually, Cem Kaner posts the following description of exploratory testing on his blog entitled: “On the craft and community of software testing.”

“Exploratory software testing is a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the value of her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.”

When I implement an exploratory testing framework, I try to use the following process steps:

The first step is a kickoff meeting. The tester will be exploring. A collaborator will be commissioning the exploration. The collaborator can be a programmer as is more common when I implement CDSBET in an Agile team. The collaborator can be a test-lead as is more common when I implement CDSBET in a traditional or structured software development life cycle such as Waterfall or Rational Unified Process or V-Models.

Kick Off

The tester reviews the charter with the person commissioning the testing. We make sure that the scope and depth of the charter are discussed and agreed to. It is important at this time to also agree on the size of the upcoming session of testing in advance. I like to encourage the tester to discuss which variables, factors, conditions or data sources may be relevant.

Preparation

The tester gather whatever resource, data, tools or equipment are required for the upcoming session of testing. I never indicate a length for the preparation since it varies dramatically and it is really up to the team to manage and try to minimize this step. The tester will need healthy environments with appropriate data to full fill the charter.

Run

This is a time boxed session. I ask testers to implement this step with the cell phone, email, social networking and collaboration software turned off. I ask the tester to focus 100% of their attention of the session of testing. The tester will design and execute tests trying to learn about the charter. The tester will keep a record of decisions made, of tests attempted and of observations. The tester will use many diverse tools and technologies to complete this step. The step is always time-boxed. A typical session is about 90 minutes long. Some sessions are short between 15 and 30 minutes. Some sessions are long between ½ and 2 days. Long sessions may be used for non-functional test charters. I do not expect all testing to be completed in one session. After the session, the tester will review findings and decide whether it is necessary to invest in an additional session or perhaps to move onto a different charter instead.

Completion

The tester gathers their findings. The tester closed open files. The tester reports any bugs in bug tracking tools as required by the project team’s workflow. The tester relinquishes the environment. Note that the tester must be able to reset the environment to a predictable state so very often at the completion step the testing will create virtual images of the test environment and data.

Review

The tester reviews their findings with the person who commissioned the testing. IN agile teams this is often the developer. In the review meeting all findings are reviewed and decisions are made on how to act on the finding In essence the review step is culling test findings and turning them into action. Test results are fed back to the person who commissioned the testing and also to any stakeholder who would benefit from a knowledge of the findings. This is a call for action. I recommend the review meeting take place on the same day as the session of testing. Some of my customers to the review immediately after the session of testing. Some of my customers review multiple sessions from the same day by the same tester at the same time.

The key decision made it – should we do another session on the same charter or should we move onto something else.

Follow Up

The team acts based on the finding of the tester.

The tester acts based on the feedback from the person who commissioned the work and any other stakeholder involved. This list will vary from charter to charter.

Charters

Note that charters derive from test ideas. One test idea can map to many charters. Multiple test ideas can map to one charter or there can also be a one to one mapping between test ideas and test charters.

A test charter is a mission statement for testing. It is a goal.

Chartering in 1803

The Lewis and Clark Charter to Explore

Mandated by President Thomas Jefferson

 

“The object of your mission is to explore the Missouri River, and such principal streams of it, as, by its course and communication with the waters of the Pacific Ocean, whether the Columbia, Oregan, or any other river, may offer the most direct water-communication across the continent, for the purposes of commerce.

On an Agile team, a charter statement can be the “title” of a testing task. In my practical experience charters can be expressed in 160 or fewer characters. This number dates to my earlier days using CRT terminals. The CRT terminals had 80 columns and 24 rows. I could always describe a charter in two lines of less thus the rule of thumb a charter can be expressed in 160 or fewer characters of text.

Session Notes

There are many ways that exploratory testers can express their findings. A charter can be represented by a task in a workflow management system, for example a Jira ticket. Each session associated with that charter would need to be a sibling object. In a session object, the tester would include their findings, session notes, screenshots, screen videos with audio commentary, spreadsheets, data records, virtual images of a system under test, pointers to bug descriptions and commentary from developers, product owners, teammates, and other interested project stakeholders. Collecting and recording findings should be a natural part of the testing workflow.

Session notes are like medical notes on a patient chart or a professional engineer logbook. This is a record of the testing done describing decisions made and trials attempted. The session notes do not need to include analysis or assessment, generally, session notes focus on recording facts.

Text Files

In most session notes the tester puts a timestamp before each note and uses short bullet lists to describe test findings. Session notes are often recorded in simple text files but there are many variations.

Document Files

I have customers who use Microsoft Word and SnagIt (TechSmith) to keep session notes. The word document contains a chain of screenshots created by SnagIt. Microsoft Word macros ensure timestamps and document format is always consistent and reference-able.

Note-taking tools like Rapid Reporter.

Rapid Reporter is a free tool to help testers record notes during a test session. The notes recorded automatically include a timestamp. Text, screenshots, rtf files can be embedded in the notes which are very professionally rendered as clean html tables.

Mind Maps

Many testers choose mind mapping tools such as XMind or FreeMind to capture the visual representation of notes. Mind maps can include images, text, links to other objects and relationships between objects.

It is considered good practice to keep track of time testing, start time, end time, timestamps during testing.

Many testers choose to include information in their session notes about their use of time: set up, on charter testing, off the charter investigation, bug reporting.

When testing in a regulated environment consistent session notes can be used to demonstrate compliance to regulatory standards.

Some Other Test Estimation Techniques

There are many different test estimation techniques which may be relevant depending on your project context. As the EPA suggests – your mileage will vary. It is important to be aware of the many different approaches which can be applied.

 

Example Spreadsheet Available to Readers

Please contact the author, Robert Sabourin, via email at robsab@gmail.com to request a copy of an example spreadsheet illustrating how to use SEBTE.

References:

A guide to the project management body of knowledge (PMBOK guide). (2018). Exton: Project Management Institute. McConnell, S. (2006). Software Estimation: Demystifying the Black Art. Redmond, WA: Microsoft Press. Humphrey, W. S. (1997). Introduction to the personal software process. Reading, MA: Addison-Wesley. Sabourin, R. (2020). Charting the Course. Coming up with Great Test Ideas Just in Time. Montreal, QC: AmiBug.Com, Inc

Rob Sabourin

Rob has more than thirty-nine years of management experience leading teams of software development professionals. A highly-respected member of the software engineering community, Rob has managed, trained, mentored, and coached thousands of top professionals in the field. He frequently speaks at conferences and writes on software engineering, SQA, testing, management, and internationalization. Rob authored I am a Bug! the popular software testing children’s book. He works as an adjunct professor of software engineering at McGill University; and serves as the principal consultant (and president/janitor) of AmiBug.Com, Inc. Contact Rob atrsabourin@amibug.com