Over A Cup Of Tea With Dr Cem Kaner – Part 2

You have experience and interests in multiple fields like Economics, Law, Mathematics, Psychology, Professional Writing, Software, and Teaching. How do you find a balance between your interests being involved in them at the same time?

I don’t see these as multiple fields. The focus of my career has been the safety and satisfaction of software customers and software development workers. This is a complex topic. To have any hope of understanding it, I have to consider the same problems from different viewpoints.

What is your opinion makes a good / better / best tester?

I interpreted this question as focused on recruiting testers—how to choose a good/better/best tester.

From that viewpoint, I think there are many ways to be an excellent tester. The essence of the testing role is to investigate a software product or service in order to discover information about its quality. The tester typically conducts this investigation on behalf of other people. We usually think of testers involved in the process of developing a product. But other investigations use testers too. For example, maybe they’re trying to understand what risks come from adding this software to an existing service. Or maybe the testers are investigating the causes of an accident.

The ideal person for this investigation will be able to efficiently find new information that has value to the project, able to explain what was found in a way that works for the people who need the information, and is willing and able to work in a way creates trust, respect, and a pleasant workplace. Many different skills are in play here and they vary by project. For example,

• I’ve been working with a course management system that has a huge number of features. They put new builds into production every three weeks. Over time, the overall product is improving, but for months, it feels like they are changing an area by swapping new bugs for old ones. If I could hire only one tester for this project, I think the greatest value would come from someone who could improve the regression suite, perhaps with a focus on code-level regression tests to support refactoring.

• Contrast this with software designed to spot patterns in very large data sets and suggest new actions based on those patterns. If I could have only one tester on this project, I would most want someone who understands the underlying mathematics and the underlying domain (for example, someone who understands medical records) and who can apply this knowledge to build complex, realistic tests of the validity of the implementation. Most projects have several testers, not just one. The greatest value from an additional tester comes from someone who brings skills that aren’t yet well enough represented on the project. Similarly, the greatest value an additional tester brings to a test group comes from improving the group’s mix of skills so that it can be more effective with the types of projects that it normally tests. I wrote a very long paper (55 pages) on this in 2000. The paper tries to help you consider what skills you need in your group and how to decide whether the person you are interviewing has or can develop those skills. For that paper, please see http://kaner.com/pdfs/JobsRev6.pdf

Thanks, Cem. Now a question for individual tester’s career; how should a tester improve her prospects in software testing?

I hear this question most often from freshers, people who have recently gotten a job in testing or who are trying to get one. My impression of the typical person who asks this question is of someone who is intelligent, ambitious, willing to work hard, maybe university educated but without many skills that are well enough developed to be of practical value in the workplace. This person is at the start of a journey of self-improvement that will require many years of work.

I think the specific path will be unique for each person, but every successful path will involve both, deepening your knowledge and skills and diversifying knowledge, skills, and the ways you can apply them.

What do you see as the basic knowledge that freshers should have?

A smart fresher should be able to do basic bug-hunting in a few weeks. I think there are four central tasks:

1. Learn a variety of tricks for exposing the relatively easy to find bugs. I think Elisabeth Hendrickson’s book, Explore It! provides an excellent starting point. For testers who have a stronger knowledge of programming, I also highly recommend James Whittaker’s How to Break… series of books. Adding depth over time will involve learning more of the ways that programs fail and common ways to look for each type of failure. This is the essence of risk-based testing.

2. Learn the vocabulary of software testing and the basics of the main testing techniques. You’ll quickly discover that the vocabulary is inconsistent. Different people mean different, sometimes contradictory, things when they use the same words. Similarly, the basic descriptions of the techniques are sketchy and inconsistent. Adding depth over time with specific techniques will come from practical experience, including coaching by other testers, digging through books and searching the web for detailed examples (try them out yourself before relying on them—you will find specific, detailed suggestions that don’t actually work very well)

3. Learn good data handling skills and other good work habits. Learn how to organize your time, so you know what tasks you have to do and you keep track of what you have done and what is left. Learn how to track how you are spending your time and how long different tasks take you. Pay attention to your mistakes. What were they? Why did you make them? How can you improve? Learn how to describe your test results and your work status (your progress so far and the work you have left to do). Find a way to communicate professionally, giving accurate information without whining, pleading, blaming or making excuses.

4. Learn your subject area. For example, if you are testing software that relies heavily on statistics (all of the data mining relies heavily on statistics), you probably want to learn more about math and more about the management of complex collections of data. As you understand more about the subject area, you will be better able to imagine what can go wrong with it.

I think most commercial testing courses teach at the surface level, even some of the “advanced” ones. That is, I think they teach basic bug-hunting methods, testing vocabulary, basic descriptions of common test techniques, and the basics of good work habits.

The typical course also teaches a view of testing culture. They teach attitudes about software development and about the motivation and responsibilities of other people who develop software, manage development or market software. For example, they might include answers to questions like these: How much control should the testing group have over the quality of the software? Should the project manager be able to prioritize the tasks done by the testers? Who should decide what training the testers get? Who should decide how much time testing tasks should take and whether a tester is taking too long? Who should decide what reports the tester should prepare and what data to include or exclude? Under what circumstances should testers refuse to create a report or insist on modifying the assigned task? Who should decide what level of detail and accuracy belong in the specification? What should the testers do if they think this level is insufficient?

Some teachers are very passionate about their view. I am very passionate about my view. My passion does not make me right. The world has many cultures. Each culture offers insights and practices that you would be well served to learn. You can learn about them without expecting them to be the same. Cultural norms will vary from company to company and from industry to industry.

Be cautious about the attitudes that you adopt. As testers, we deliver news about the quality of the product and the progress of the project that is often difficult for someone to hear. That can lead to conflict. You cannot control what other people say to you or what their attitudes are about your type of work, but you can control your evaluation of what they say and how you respond to it. Make your own choices about the nature of conflicts that you will engage in. I think some of the certifications (and not just foundational ones) examine student knowledge at the basic level. They emphasize memorizable knowledge that can be easily graded as correct or incorrect. Questions that can be graded reasonably quickly, the same way by everyone (or by a computer), have no room for matters of judgment. Experts can differ in judgment. As you develop your expertise, you will have to develop your judgment. In doing so, you will have to develop far beyond what is tested by the certification exams, even beyond what I think is being marketed as “expert-level” certification.

Some exam questions also demand cultural/ethical answers that the examiners consider correct. A right-or-wrong-answer exam asks you whether you know what the examiner wants you to answer. It does not ask you whether you have accepted that person’s or organization’s values as your own. You can adapt your own values and be true to them while demonstrating your knowledge of someone else’s culture when you write their exam. Some testers stay at the basic level all of their careers. I think these types of jobs are less secure today than they were fifteen years ago.

Several courses seem designed to make you feel good about yourself and your skills even though your level of knowledge and skill is modest. Learning a few testing tricks or some basic programming or psychology can be very enlightening. Making you feel good is good marketing—for the course providers. Never forget, though, that if what you know can be packed into a few days or weeks of commercial courses, you can be easily replaced by someone who earns much less than you.

How can you go beyond the basics, to add depth?

Going deeper requires specialization. The starting question is what do you want to be good at? For example, imagine becoming a successful test tool creator. I think you have to learn more about bugs, about the kinds of bugs that you could build a tool to help people find. I think you have to learn more about creating usable, reliable programs that can help other testers (whose skills are weaker than yours) do their work. If your tools support workflow rather than bug hunting, I think you have to learn how to study workflows, how to learn from successful and unsuccessful examples in current use, how to design processes that people will find efficient, effective for their work, and pleasant to use, and how to write code to support those processes.

But imagine instead that you want to test financial applications. You can stay at the surface and hunt GUI bugs and find obvious usability bugs. Below that surface, though, are bugs that will cause people to lose money or to get into finance-related trouble. To go deeper, you have to learn about the type of financial application you are testing. What’s difficult in this field? What’s confusing? What kinds of things go wrong? If you want to assess the value of a program, you need to know enough about value, in that field, to talk intelligently, and think competently, about it.

You might not want to specialize by application type. You might specialize by problem type. Perhaps you will develop expertise in testing for security flaws or performance problems or video card incompatibility.

Or you might specialize by technique. For example, the master of scenario testing might become an expert in the qualitative research methods used by requirements analysts or human factors analysts. The master risk-based tester might become an expert in digging through the history of this product and related ones, finding ways that products of this kind have gone wrong before, or that the technology used in this product has led to failures in other products.

Each of these illustrates a different way to add depth to your work. This doesn’t come just from years of experience. It comes from a deliberate effort to gain knowledge and skill, from intentional practice and intentional searches for relevant information.

People with depth are hard to replace. They know more. They are better at what they do.

Of course, if you specialize in something that becomes irrelevant to the work of your company, you will either (a) have to find a new company or (b) find a new area of expertise.

The nice thing about active, intentional learning is that you don’t just learn content. You learn how to learn. Picking up a fourth area of depth will probably be a lot easier and faster than picking up the first.

Over your career, you might develop many areas of depth.

You said that testers should deepen their knowledge and diversify their skills. What does diversifying mean and how will it help them?

Almost all of the best testers I know have worked in several fields, not just testing. They bring many different types of knowledge and skills to their work as testers. For example, along with knowing a lot about testing, they might be skilled in programming, market research, software design, technical writing, etc.

In contrast, when I see a tester or test manager who has been in the field for many years but is not very open to new ideas, or who is constantly fighting with programmers and project managers, I often find out that person has worked in testing all their life. Their perspective is narrow, they don’t understand the viewpoints of the people in the other groups that they work with, and their career is often at risk because they haven’t kept up with technology.

After you’ve done testing for a while, I think you should take a break from it. Become a programmer. Or a technical writer. Or a technical support specialist. Do some marketing. Learn how to sell software.

Pay attention to your attitudes and experiences in your new role:

• How do you and your peers look at testers? Why?

• Are you doing the type of work that you used to evaluate when you were a tester? When you were a tester, you had ideas about how this work was done and what made it good or bad. Which of those ideas were completely wrong?

• When testers communicate with you, what works and what really doesn’t? Why? How would you change what these testers do, if you could?

Later, when you come back to testing, these experiences will be a foundation for communicating much more effectively.

Over a career, you can do this several times.

Testing is a service. We do tasks for other people. We find information that other people will (we hope) find useful. Every time you strike out in a new direction, you come back to testing with new insight about the ways testing can serve the rest of the organization.

Is better communication the main reason to diversify your skills?

Better communication is one reason.

Improving your technical abilities is another reason—an extremely important reason. Many people who start their careers in testing are weak programmers. This is OK, for a start, but if you want to progress in the field, you should develop strong programming skills. There are some very successful people in our field who are nonprogrammers, but I think it’s important to be realistic about what is more likely and what is less likely. I think that people who are good testers and who also have strong programming skills are more likely to earn more money, to find more interesting jobs, to be more secure in their jobs, and to find new jobs more easily. They are more likely to be able to imagine how to use technology to make their work more efficient. They’re more likely to be able to work effectively with programmers on agile projects. And there are testing methods, like high volume automated testing, that you simply can’t do if you can’t program.

Is programming knowledge essential for testing?

Not essential. Some very successful testers are not programmers.

However, when I look at the opportunities that open up for my university students, some require testers to use programming knowledge and others don’t. The ones that require programming knowledge are almost always more interesting, at better companies, and they often pay twice as much money.

Million-dollar advice, wasn’t it? Don’t forget to check out part 3 to know what Dr Cem advises around teaching and coaching testing. Stay tuned!