Something about knowledge.
What you cannot know
E. W. Dijkstra expressed this principle of testing in 1969:
Testing shows the presence, not the absence of bugs.
Dijkstra was discussing testing with researchers focused on finding ways to prove that code was working. They were in a conference organized by NATO and although I was barely born when it took place, I would have loved to be there as according to the transcript Dijkstra terminated the discussion with this statement. It takes courage, but some of the wisest people around are those who tell others what they cannot know.
Computers are machines. We can assume that there is always a root cause for any undesired behaviour we may see. Even unpredictability would be a problem with root-causes.
Computers are also reliable: Unlike testing a human, testing a program running on a computer potentially, but objectively shows whether there is a bug in it. But the objectivity comes with a price: The test will never be able to say anything else about the program other than the fact that there is a bug.
Dijkstra’s statement relies on concepts of objectivity and reliability. It also relies on something else, however, namely an assumption about what a bug is.
Bugs are different
Bugs in computer programs in 1969 were quite simple: The program was loaded on the computer, it was fed some input which was then processed according to the program, and when the program had completed running, results were produced and stored on magnetic tape, punched cards, paper tape, or merely printed on the console or line printer. Defining a bug in that context follows simple reasoning: A bug equals an incorrect result.
For about 20 years, I have worked on enterprise systems used by users to create, access, and manage cases of varying kinds. Such systems are used to manage logistics, social services, customers, pensions, taxes, etc. These systems always have some automation running to process payments, messages, requests, claims and more. These automated processes handle interactions with other systems some of which may be internal to the organization running and developing the system, while some will be external. Compared to software in 1969, software systems today are more complex as for example, the relations between inputs and outputs in a modern software system is far from trivial.
The concept of bugs has therefore become complex.
Do bugs carry meaning?
I vividly remember some of the discussions we had around the dining table at home. My father was a computer engineer in the 1970’s and he read piles of code like it was poetry identifying bugs in it without even running it on the host computer. He worked on several pioneering computer projects. The highlight of his career was writing the library functions and microcode for a vector processing unit for a mainframe computer. It was custom designed for cartography.
Although I followed his path and became an engineer like him, I have always found the kind of logical reasoning he was so fluent at and which enabled him to read code the way he did difficult and unintuitive.
Fortunately, the industry had changed by the time I graduated from engineering university during the 90’s: My first job was programming a multimedia games title where the success of my work depended more on collaboration with the graphic designer and the composers for the music and sound effects we used than on my abilities to write perfect code. Later I changed to enterprise systems and became a tester. Where the software my father talked about at the dining table could be evaluated by its ability to produce consistent output, evaluating software in the 90’s was much more complicated. Performing these evaluations is still my profession.
Let me generalize a bit: 50 years ago, scientists discussed software as data processing. We still call what computers do “data processing”, but note that where the processes used to be linear and procedural, they are more complex today.
Dijkstra’s statements limit the scope of proof that testing can perform to falsification. In essence, that is still what a test can do when testing is evaluating outputs based on inputs. But is that what we do today?
No. We automate that
So why is it that testers can still be great at finding bugs in software by running it and trying out different things it is supposed to do and should not do?
You know what a bug is, don’t you? You are a tester, aren’t you? Ok.
Then ask yourself the following questions: What is the meaning of the bugs you find? What do the bugs mean for users? What do they mean for stakeholders? What threats to the value of the system are you looking for?
Do you still know what a bug is? I am asking you those questions to make you doubt what you think you know.
Changing the meaning
Meaning is another complex concept.
It is intuitively easy, but when you start thinking about meaning, things are often messy. Let me be honest: I have found that when I thought I knew for sure what a bug meant, I was always wrong. At least to some degree.
So why am I still testing to find – prove! – those damned bugs? I do it too, so let us make it a “we” question: Why are we still testing to prove bugs?
Could we change our ways and avoid problems in testing?
My answer is that we can, and we do. I see testers changing testing every single day by asking themselves similar questions to the question I asked you above: What is the meaning of this?
The answer is often: “I cannot know for sure, but I think…”
Improving testing
A skype call with a remote tester
We had only briefly met each other before, the person in front of me, and myself. He was in fact not sitting in front of me, but in an office space 6300 km away. We were wearing headsets and staring at blurred images of each other on our screens. He was onboarded on the team before I started as the manager. I knew very little about him or his skills. I remember how I reminded myself of the cultural differences before greeting him welcome to this our first one-on-one meeting. I remember I was a bit anxious about the whole situation.
Working with people from different skill sets, backgrounds, and cultures is one of the most fascinating things about working in tech.
I have worked in tech for more than 25 years now, and that kind of experience is worth a lot. I am not thinking in terms of money, but I see that being where I am today is a good place: Where others might see chaos, I often see patterns. That has made me more robust and happier dealing with complexity and uncertainty.
When I was younger and less experienced, I looked to my senior colleagues to learn from them. I often looked at leaders with a combination of awe and frustration, especially when they organized meetings. I have always been impatient, and my impatience had given me some lessons in the past, so I started observing their behaviour and asked myself: “Why are they taking time out for these conversations that seem to go nowhere?”
It took a while before I understood.
Coming from different places
The conversation does not flow easily. I notice we are coming from quite different places and that getting a common understanding of his role is annoyingly difficult.
A: “The bug you reported earlier. Tell me about that.”
T: “I prepared the scripts according to the specification. I had to wait a long time until the feature was released from the developer. Executing the script, step 7 failed. I reported that.”
A: “Has the developer reached out?”
T: “No”
A: “I’d like you to reach out to him to ensure the bug is fixed.”
There is silence on the call. My team member at the other end of the network call clearly does not feel comfortable. My next sentence emerges with a slightly irritated tone:
A: “Testing is a critical process performed with the team, and not a matter of completing the test case and reporting whatever bugs you have found. If the developer does not understand that, we have to teach him.”
Making choices
A test manager early in my testing career once asked me about my gut feeling about the system I was testing. I still vividly remember the panic I felt when she asked as I had not imagined she would have been interested in my feelings: Software should work. Software does not involve feelings. That’s a different domain. That was my reasoning, the reasoning I had learnt at home by the dining table.
Her question, however, shaped a learning path for me as I worked on the project for more than 3 years. The learning path involved learning what it takes to work on a big project in a big organization. It is very much about the choices we make.
Choices and decisions are made all the time: A developer decides to solve a problem in a certain way. A tester chooses to perform a specific test. A manager reports a specific metric. A senior manager decides to allocate funds.
Choices made and decisions taken by developers and testers are usually very concrete. The effects are often seen immediately. This immediateness means that we often forget the choices we make and focus on the work carrying them out: Writing the code, performing the test, preparing the report.
Defining testing
I perform a test and find a bug. I perform another test, and things work as specified. I change an input value and repeat the test and the front-end crashes. Oops!
It is all happening right in front of me as a tester. It happens because I test. It happens because of the process of testing.
But let us talk about definitions: What defines the testing I? The defining thing about my testing is not the process I follow, but the choices I make in my testing.
Results matter, and the choices I make before getting these results to matter more as there is always direct causation between the choice and the outcome in testing. In the big project, for example, we were under pressure and I learnt I could sometimes make a difference by the way I was reporting my testing: Even though my testing could still only identify (prove!) bugs, there was a lot I could not report: Like the effects on users. So I focused on the people around me: and started reporting the absence of bugs as well because it helped me express appreciation of my developer colleagues’ work. The social contract we had on the team helping each out had to be remembered.
Taking time to socialize with colleagues was a choice I made. It was not in my job description, but it helped me establish something else. It helped me establish a sense of meaning with what I was doing.
People do not often see that.
Should I let the tester go?
I felt I had to apologize to my tester to get the conversation back on track. I was a high-level project manager in a complex project, and I told my tester that I was ready to support him in his role. Testing was critical in this project.
“You need to approach the necessary people in the hierarchy when you find a bug. “
We agreed that I would e-mail the senior developer asking him to prioritize the bug reports.
At my next meeting with my manager, I raised the problem:
A: “It seems our testers are not allowed to address developers directly.”
I got his support to change that, but not the support I expected:
M: “Let me know if I need to make a statement. We have had problems with bad testers before.”
I felt worried. I did not want to let my tester go as he was good at his job. But he was not providing value to the project.
Enlightenment and gut feeling
Enlightenment is the concept of understanding things from a higher level. At almost 52 I feel more enlightened that I used to. A wise person once said that before enlightenment you chop wood and carry it home so you can burn it and keep your family warm. So is there less work now? No, he also said, that after enlightenment, you still chop wood and carry it home to keep warm.
Testing involves lots of tedious work of managing test data, preparing testing, performing testing, noting results, gathering, and communicating details in reports, collaborating with colleagues, taking time to socialize. I still do the labour.
But as I mentioned above, I have realized something about the labour, namely the importance of the choices I make while labouring.
This is where “gut feeling” comes into play. While we often think our choices have a reason, and most of us think about ourselves as rational beings who prepare and make plans, and who mostly follow the plans we make, we must admit that that cannot be the whole truth: In the moment of actually making a choice, we often end up improvising – instead of sticking to the plan we made.
There is always “gut feeling” involved in decisions. Gut feeling is there but can be difficult or outright impossible to explain in a situation. And if we try afterwards, the question is if we are not only making up a story about our reasoning.
Facing a dilemma
So, are we as humans fundamentally irrational? I do not think so. I just think our rationality is bounded: We can think rationally, but in the moment of making a choice, our judgment is tied to the event and we cannot say if we are. Because of that, improvising to me also means taking responsibility.
I was in a dilemma with my tester: My boss saw the problem I did not want to see that my tester was not ready to make bold decisions and take responsibility. He could test according to his books, but he did not have the courage to do more.
Should I let him go? I thought about it and made a choice. Some people need support in the form of clear directions, and training before they are ready to escape the plans and best practices. I decided to teach him that it’s a best practice to escape the best practices he had learned.
A new realization
Although the developer had assured me that he would attend to our bug reports, things were still not moving: Testing was slow, and bugs did not get fixed. Speed mattered: We had deadlines to meet and testing and debugging was becoming a bottleneck. I had to do something, but what? Everybody was working as fast as they could, it seemed.
One day I looked at one of the bug reports that my tester had raised. I could not make sense of it. No wonder the bugs did not get fixed! The bug reports were impossible to understand.
I felt I was at the root of the problem that had bugged me: The problem was about communication. It irritated me. I was especially irritated that I had not realized this before. I had pushed everyone to work faster and longer hours. Test more. I felt stupid.
I woke up early next morning frustrated as I realized I might have to let this guy go now. Had I wasted everyone’s time?
In office I called him up and explained the problem. I then made a choice: I took time to explain and ensure he understood that the current practice did not work, and that he had to change. He was not comfortable, and neither was I, but I kept us on the path.
Afterwards I felt better. Was it not my moral obligation to make this choice? Explaining the problem as well as I could giving my tester a chance to improve. He listened, but it was difficult for him. He did not know how to make the change. He was not ready.
But he had listened and seen the problem. He probably knew his role in the project was at stake.
I offered him to rewrite the open bug reports. That was easy for me. I would also make a template to guide him and his colleagues: A new best practice in the project. He thanked me. I felt we had a process and could make progress.
We discussed perspectives for a moment: His view on bugs, and the view developers would have on bugs.
We are all different.
Epilogue
The story ends here. It turned out the bug reports was not the only problem we had in the project. There was lots of friction in the project where people were sticking to whatever best practices they had been trained at and forgetting the best practice of doubting your best practices.
But over a year or two, we succeeded improving our processes and deliver a working product. At the end people were asking themselves questions about the meaning of what they were doing. They started doubting their proofs. They started saying: “I think this is because…”. And others chipped in in the discussions.
They started learning and because of that they started making more enlightened choices.
They still wrote code and tested it. Often the tests we did in my team still proved lots of bugs, but often they did not produce the obvious proof, so the bug reports had to become items of communication and discussion. To the outsider nothing changed: Development was still a job along the same lines as before as we went through ideas, concepts, designs, solutions, code, and back to testing ideas, scenarios, data, cases, scripts, and exploring. That was sad as we could not establish this way of working as a new model because what management saw was just a delivery and best practices followed.
But I knew what we did: We had more conversations about communication, collaboration, and making personal choices about things to do. And that helped us succeed.
The Germans have a beautiful word for enlightenment: Bildung. They use it in their word for education: Ausbildung. Ausbildung literally means taking your images and imaginations out of yourself, , and into the social contexts in which they can make a difference. Bildung is about having an image by your heart that you follow
Testing has potential to teach people things about the software they are working on. Done right, testing is an education for everyone taking part. But let’s use the German word: Ausbilding is about what you can do as a colleague: Help get the images they have out in the social context, make them items for exploration and discussion so they can make a difference to people in the real world.
Anders Dinsen
Anders thinks of himself as a driver of learning and development testing complex software. His title is test manager and he work for KMD, a NEC company in Copenhagen. He leads people in agile, waterfall, and hybrid contexts, is critically minded, and enjoys the short power-distances part of Danish organizational culture as it enables effective influencing and driving quality for those who matter. He has 25 years of experience as a developer, tester, manager, facilitator, and coach, and tested his first piece of code in 1982.