How does your team or organization measure quality? People often equate testing to good quality or ‘quality assurance’, but if you have good testing practices, does that mean you have a good quality product? Many teams measure process quality and don’t realize they forget about the product quality – which is what the customer cares about.

There are many things that go into a quality product, and testing is only one aspect. In this article, I explore the interaction between the development process (which includes testing) and the different types of quality measures that organizations use.

There are many dimensions to thinking about quality, but I’ll focus on product quality and process quality, and the correlation between the two.

How we develop our products, influences how we view our product quality. How we view our product, influences how we develop our product.

There are many contexts, and each may need a different way of looking at quality and in all cases, quality needs to be built into the product from the beginning and make the customers part of the process.

There are 3 sections to this article: 1. The Product Quality, 2. Process quality, and 3. how they might be measured.

Product Quality

Describing product quality is hard. There doesn’t seem to be any easy way to do it. Gerry Weinberg has used the definition “Quality is value to some person”. This seems to be the most popular definition because it’s true, and it’s easy. However, I think it might be too simplistic and understates some of the dimensions teams should be thinking about.

Consider your product. Is your product simple enough that you can say, I know what Sally likes in our product so if she says it’s a quality product, it must be so? Most of us do not have that luxury. We have many different types of customers, and end-users, and they all look for different aspects. They have different perspectives. There are also internal customers such as product management who want fitness for use and user experience for the customer, the finance group who cares about profit, or the regulatory governance group who cares about legality. Teams have to satisfy all those needs.

There are many conflicting interests in how we define quality, and we should be looking at them, having conversations about them, and making decisions based on those needs. There are different lenses in how we view quality, including – are we getting value for our money? There are also generational differences in how we view quality. For example, younger people seem to care more about the user experience more than some older folk. I recently had a conversation with a group of people (between 20 – 30) about coffee and coffee shops. They told me they never thought about the money if they liked the experience. It’s a different way of thinking about quality. There is no right or wrong – only different perspectives.

It’s these differences in expectations that make the quality discussion difficult.

Process Quality

Process quality is much easier to talk about so most organizations concentrate on that – how well do they build their products. Testing activities are one way to contribute to product quality. Many people think about testing only as testing the software after it is built, but testing activities also happen throughout the delivery cycle. Testing activities:

  • Provide feedback in many forms (defect reporting or code reviews are two examples).
  • Identify hidden assumptions – many are because of different perspectives.
  • Help identify and mitigate risks (product, business, technical are a few).
  • Give information about the state of the product.
  • Assess quality (assuming the team knows what that means)

Note: I do not believe that testing (investigating or evaluating) can assure (tell someone something positively or confidently to dispel any doubts) quality.

The agile testing quadrants (Figure 1), including all their variations, is a model to help teams talk about testing activities, and which ones are important to consider for their product. The testing activities in the diagram are examples of tests that a team could perform.

The left-hand side of the quadrants is about activities that happen before code is written – the focus is on preventing defects. The right-hand side is about finding defects in the code as quickly as possible. Both are needed but preventing defects in code is much more cost-effective. The top half is about tests written so that business can understand them, and the bottom half are tests that are written from a technical perspective. The business might care about the results but couldn’t read the tests. If you are interested, I suggest you read more in Chapter 8: Using Models to Help Plan, from More Agile Testing.

The testing activities shown in the lower right quadrant are quality attributes that address product constraints and risks. Many quality attributes are “expected” by our customers. For example, if your team is working on a medical device, safety is probably extremely important. Another example might be data integrity – each of us carries a lot of personal data in our phones. We expect the apps we use to treat our data with caution and not to share it.

When teams don’t think of different dimensions of product quality, they often overlook building it in. In every aspect of our delivery cycle, they need to consider the best way of building their product and how to build quality into their process.

As team members, we test ideas to make sure we are solving the right problem. We clarify our needs. Identify hidden assumptions and test our understanding of the problems by providing examples. We talk about the risks and which quality attributes are important and ask more questions to learn more. For example, do we need to think about the diversity of who is using our product? We may be building in limitations without even thinking about it. These types of testing are early in the cycle and are about preventing defects in code.

Testing activities during development include unit testing (TDD), code analysis, pair testing, exploratory testing (ET), test automation, even user acceptance testing (UAT). These activities help teams feel confident they are not introducing coding-level bugs. ET and UAT do address product quality by testing as different stakeholders and looking at the product holistically.

There are even testing activities that can happen after releasing to the customer – for example, testing in production (observability) and monitoring enables teams to get feedback from the customer’s actions and reactions. The learning from these activities feeds back into building new features or addressing something that wasn’t quite right.

So, with all these feedback loops, teams should have enough information to assess the quality, right? The question that still needs to be answered, is: “How do they know what to assess if they don’t know what to measure.”

Measuring Quality

Measuring quality is not easy, but we can agree that low quality is bad, right? But is that always the case? We sometimes accept lower quality because it comes at a cheaper price – that doesn’t make it bad.

Teams often measure things because they are easy. For example, the number of bugs or severity of bugs tells me how bad the quality is, not how good it is. Even if absolute numbers aren’t used, but watch the trend, it is not guaranteed that the product has good quality. It might be better than before, but still not great.

Accelerate has some good measures for process quality like cycle time, rework rates, etc. Other measures like test coverage tell us nothing about product quality but are also about process quality.

Organizations tend to use more qualitative measures for product quality like above based on surveys or feedback from customer support. Customer loyalty can be measured – how often do customers come back, or how long do they stay customers. For example, if a company offers a product for free, but gets income from add-on services, customer retention is extremely important.

As another tact is to ask questions about risk. Ask “What’s the worst thing that can happen?” – that would be the biggest risk. Ask, “What’s the best thing that could happen?”. That might be something you want to measure. A good quality strategy should be based on mitigating risks.

This might be a good time to consider those quality attributes from the lower right quadrant of the agile testing quadrants. Those tests are about mitigating risk, and a step towards measuring product quality.

Margaret Dineen did a talk and wrote a blog post about using sliders for quality attributes to start the conversation within a team. Identify quality attributes that are important to your product based on the identified risks and prioritize using priorities sliders as in Figure 2. Compare, discuss, and then take it to other stakeholders until a shared understanding is reached. It can be very revealing and may open up new conversations. Once you have that shared understanding, you can decide how to measure.

 

Conclusion

Everyone in an organization plays a part in delivering a high-quality product:

  • The organization and senior management provide a safe environment to enable questioning and learning.
  • Product management provides clear priorities to enable the teams to work effectively.
  • The business understands the ‘what’ and the ‘why’ and answers “Did we build the right thing”?
  • The delivery team strives to build it right.
  • The individual knows how they contribute to the quality of the product and works toward that end.

It is not enough to make sure the product works, it needs to satisfy all the needs of the customer. If your teams are not talking about quality first, you have the opportunity to start that conversation. If you don’t have that conversation, how will you know what testing needs to be done? How will you know what risks to consider?

Testing supports good quality but does not assure good quality. Process quality helps but doesn’t automatically make a product good.

The nature of quality is complex and diverse, so understand what you mean when you say ‘quality’!

References:

This is the heading

is an agile testing and process consultant with DragonFire Inc. She is the co-author with Lisa Crispin of Agile Testing Condensed: A Brief Introduction (LeanPub 2019), More Agile Testing: Learning Journeys for the Whole Team (Addison-Wesley 2014), and Agile Testing: A Practical Guide for Testers and Agile Teams (AddisonWesley, 2009), the Live Lessons Agile Testing Essentials video course, and “Agile Testing for the Whole Team” 3-day training course. Janet specializes in showing agile teams how testing activities are necessary to develop good quality products. She works with teams to transition to agile development and teaches agile testing courses worldwide. She contributes articles to publications and enjoys sharing her experiences at conferences and user group meetings around the world. For more about Janet’s work and her blog, visit https://janetgregory.ca or https://agiletester.ca You can also follow her on twitter @janetgregoryca or LinkedIn Together with Lisa Crispin, she has founded the Agile Testing Fellowship to grow a community of practitioners who care about quality. Check out https://agiletestingfellow.com to find out more about courses and membership.