Enlightening In Testing

Something about knowledge.

What you cannot know

E. W. Dijkstra expressed this principle of testing in 1969:

Testing shows the presence, not the absence of bugs.

Dijkstra was discussing testing with researchers focused on finding ways to prove that code was working. They were in a conference organized by NATO and although I was barely born when it took place, I would have loved to be there as according to the transcript Dijkstra terminated the discussion with this statement. It takes courage, but some of the wisest people around are those who tell others what they cannot know.

Computers are machines. We can assume that there is always a root cause for any undesired behaviour we may see. Even unpredictability would be a problem with root-causes.

Computers are also reliable: Unlike testing a human, testing a program running on a computer potentially, but objectively shows whether there is a bug in it. But the objectivity comes with a price: The test will never be able to say anything else about the program other than the fact that there is a bug.

Dijkstra’s statement relies on concepts of objectivity and reliability. It also relies on something else, however, namely an assumption about what a bug is.

Bugs are different

Bugs in computer programs in 1969 were quite simple: The program was loaded on the computer, it was fed some input which was then processed according to the program, and when the program had completed running, results were produced and stored on magnetic tape, punched cards, paper tape, or merely printed on the console or line printer. Defining a bug in that context follows simple reasoning: A bug equals an incorrect result.

For about 20 years, I have worked on enterprise systems used by users to create, access, and manage cases of varying kinds. Such systems are used to manage logistics, social services, customers, pensions, taxes, etc. These systems always have some automation running to process payments, messages, requests, claims and more. These automated processes handle interactions with other systems some of which may be internal to the organization running and developing the system, while some will be external. Compared to software in 1969, software systems today are more complex as for example, the relations between inputs and outputs in a modern software system is far from trivial.

The concept of bugs has therefore become complex.

Do bugs carry meaning?

I vividly remember some of the discussions we had around the dining table at home. My father was a computer engineer in the 1970’s and he read piles of code like it was poetry identifying bugs in it without even running it on the host computer. He worked on several pioneering computer projects. The highlight of his career was writing the library functions and microcode for a vector processing unit for a mainframe computer. It was custom designed for cartography.

Although I followed his path and became an engineer like him, I have always found the kind of logical reasoning he was so fluent at and which enabled him to read code the way he did difficult and unintuitive.

Fortunately, the industry had changed by the time I graduated from engineering university during the 90’s: My first job was programming a multimedia games title where the success of my work depended more on collaboration with the graphic designer and the composers for the music and sound effects we used than on my abilities to write perfect code. Later I changed to enterprise systems and became a tester. Where the software my father talked about at the dining table could be evaluated by its ability to produce consistent output, evaluating software in the 90’s was much more complicated. Performing these evaluations is still my profession.

Let me generalize a bit: 50 years ago, scientists discussed software as data processing. We still call what computers do “data processing”, but note that where the processes used to be linear and procedural, they are more complex today.

Dijkstra’s statements limit the scope of proof that testing can perform to falsification. In essence, that is still what a test can do when testing is evaluating outputs based on inputs. But is that what we do today?

No. We automate that 

So why is it that testers can still be great at finding bugs in software by running it and trying out different things it is supposed to do and should not do?

You know what a bug is, don’t you? You are a tester, aren’t you? Ok.

Then ask yourself the following questions: What is the meaning of the bugs you find? What do the bugs mean for users? What do they mean for stakeholders? What threats to the value of the system are you looking for?

Do you still know what a bug is? I am asking you those questions to make you doubt what you think you know.

Changing the meaning

Meaning is another complex concept.

It is intuitively easy, but when you start thinking about meaning, things are often messy. Let me be honest: I have found that when I thought I knew for sure what a bug meant, I was always wrong. At least to some degree.

So why am I still testing to find – prove! – those damned bugs? I do it too, so let us make it a “we” question: Why are we still testing to prove bugs?

Could we change our ways and avoid problems in testing?

My answer is that we can, and we do. I see testers changing testing every single day by asking themselves similar questions to the question I asked you above: What is the meaning of this?

The answer is often: “I cannot know for sure, but I think…”

Improving testing

A skype call with a remote tester

We had only briefly met each other before, the person in front of me, and myself. He was in fact not sitting in front of me, but in an office space 6300 km away. We were wearing headsets and staring at blurred images of each other on our screens. He was onboarded on the team before I started as the manager. I knew very little about him or his skills. I remember how I reminded myself of the cultural differences before greeting him welcome to this our first one-on-one meeting. I remember I was a bit anxious about the whole situation.

Working with people from different skill sets, backgrounds, and cultures is one of the most fascinating things about working in tech.

I have worked in tech for more than 25 years now, and that kind of experience is worth a lot. I am not thinking in terms of money, but I see that being where I am today is a good place: Where others might see chaos, I often see patterns. That has made me more robust and happier dealing with complexity and uncertainty.

When I was younger and less experienced, I looked to my senior colleagues to learn from them. I often looked at leaders with a combination of awe and frustration, especially when they organized meetings. I have always been impatient, and my impatience had given me some lessons in the past, so I started observing their behaviour and asked myself: “Why are they taking time out for these conversations that seem to go nowhere?”

It took a while before I understood.

Coming from different places

The conversation does not flow easily. I notice we are coming from quite different places and that getting a common understanding of his role is annoyingly difficult.

A: “The bug you reported earlier. Tell me about that.”

T: “I prepared the scripts according to the specification. I had to wait a long time until the feature was released from the developer. Executing the script, step 7 failed. I reported that.”

A: “Has the developer reached out?”

T: “No”

A: “I’d like you to reach out to him to ensure the bug is fixed.”

There is silence on the call. My team member at the other end of the network call clearly does not feel comfortable. My next sentence emerges with a slightly irritated tone:

A: “Testing is a critical process performed with the team, and not a matter of completing the test case and reporting whatever bugs you have found. If the developer does not understand that, we have to teach him.”

Making choices

A test manager early in my testing career once asked me about my gut feeling about the system I was testing. I still vividly remember the panic I felt when she asked as I had not imagined she would have been interested in my feelings: Software should work. Software does not involve feelings. That’s a different domain. That was my reasoning, the reasoning I had learnt at home by the dining table.

Her question, however, shaped a learning path for me as I worked on the project for more than 3 years. The learning path involved learning what it takes to work on a big project in a big organization. It is very much about the choices we make.

Choices and decisions are made all the time: A developer decides to solve a problem in a certain way. A tester chooses to perform a specific test. A manager reports a specific metric. A senior manager decides to allocate funds.

Choices made and decisions taken by developers and testers are usually very concrete. The effects are often seen immediately. This immediateness means that we often forget the choices we make and focus on the work carrying them out: Writing the code, performing the test, preparing the report.

Defining testing

I perform a test and find a bug. I perform another test, and things work as specified. I change an input value and repeat the test and the front-end crashes. Oops!

It is all happening right in front of me as a tester. It happens because I test. It happens because of the process of testing.

But let us talk about definitions: What defines the testing I? The defining thing about my testing is not the process I follow, but the choices I make in my testing.

Results matter, and the choices I make before getting these results to matter more as there is always direct causation between the choice and the outcome in testing. In the big project, for example, we were under pressure and I learnt I could sometimes make a difference by the way I was reporting my testing: Even though my testing could still only identify (prove!) bugs, there was a lot I could not report: Like the effects on users. So I focused on the people around me: and started reporting the absence of bugs as well because it helped me express appreciation of my developer colleagues’ work. The social contract we had on the team helping each out had to be remembered.

Taking time to socialize with colleagues was a choice I made. It was not in my job description, but it helped me establish something else. It helped me establish a sense of meaning with what I was doing.

People do not often see that.