Over A Cup Of Tea With Dr Cem Kaner – Part 1

Evolution of the Art and Craft of Testing cannot be discussed without citing credits to the work done by Dr.Cem Kaner in the last few decades. Dr.Cem’s contribution to this field has had a significant influence on the way testing is being done by thinking and adaptive testers today.

Conversing about testing with Dr. Kaner, knowing more about his journey, and knowing more about his opinions on various subjects in the field was on our wish list for a long time. I got to speak with Dr. Cem on various aspects of his life, expertise, opinion, and suggestions recently. And this ‘series’ is a result of what we have been discussing for quite some time.

In part 1, we’ve covered Dr. Cem‘s career journey and his thoughts around testing education. Read more in part-2 and 3 of this interview series. Enjoy the ever-informative read from our conversation snippets!

The world of software testing needs no new introduction of Dr.Cem Kaner but we are curious to know about your journey in software testing.

When I was young, I worked in my father’s clothing stores. Many of my earliest memories are set in the stores. I was the son who was supposed to take over the business. One of my tasks as a teenager was to systematize our inventory control system, to settle our processes in order to make them suitable for computerization. We took the big step in 1970, contracting with a company that installed fancy computerized cash registers that sent data to be processed by sophisticated custom software. The result was a disaster. Their system was very expensive but compared to our manual
system, it was less accurate, provided less information, and took twice as long (two weeks). Even worse, the cash registers were too hard to use. They interfered with the work of our salespeople. Eventually, we pulled the cash registers out of the stores and made do with a manual system for another decade.

Any lessons that you learned from this experience?

I learned three important lessons from this:

(a) This system was worse than worthless but the central problems were design errors, not coding errors. Since then, “works as designed” has never meant “acceptable” to me.

(b) The most expensive problems involved un-usability. The system was so badly designed that our cashiers and salespeople kept making “user errors” that would take long times to work around. These people had been very effective in their jobs before the computerized cash registers and they became effective again once we threw out the technology. The “user error” problems were no problems with the users. They were problems with a system design that set these people up to fail. This was one of many occasions in our business when I would learn that recurring problems indicated failures of our systems, not of our staff. Years later, I would study Deming and realize that many of the lessons, he was teaching were generalizations of the same lessons my family had been learning through experience as we rebuilt a business from nothing (one bankrupt store) to a national chain.

(c) We could not get the system improved. We could not even cancel the contract. The best we could do was to stop using this terrible system (and keep paying for it). We learned that our traditional expectations about how contract law governs service contracts didn’t apply to software contracts. Over several years, as I saw many more examples, I would eventually realize that when customers have no power to hold their suppliers accountable, many suppliers will cheat their customers. And when the legal system protects dishonest suppliers, the most dishonest suppliers will be able to drive the honest ones out of business.

As our family’s friends computerized their businesses, I paid attention to their stories. The details were different but the same core problems appeared again and again.

Please tell us more about your teaching experience. Are there any special memories from university?

At university, I studied mainly mathematics and philosophy, especially Indian philosophy. I’m Jewish, as is most of my family, but my grandmother was a Buddhist. I was to spend about a decade coming to grips with my own belief system. A few parts of that seem relevant to my attitudes toward testing and the testing community:

(a) In the West, we take the “law of the excluded middle” as self-evident and as fundamental to logic. Under this rule, a proposition can have two states: something “is X” or “is not X”. There is no middle ground. In contrast, several Indian logicians assumed four states, “is X”, “is not X”, “is both X and not X” and “is neither X nor not X”. I puzzled over this for many years. I am not confident that my understanding of how this could be meaningful is the same as their understanding. But in human affairs, in interpreting people, how they interact, what situations they are in, and how to negotiate with them, I came to view the exclusion of the middle as a heuristic rather than a law or a rule. It is often useful to assume that “X or not X” are the only two possibilities, but when that is not productive, it is time to look for a third way.

(b) The texts that I read constantly challenged my perception of the world. Many appearances are just appearances. You see a snake, but when you touch it, it’s a rope. I would eventually adopt a pragmatic realism (I interpret the physical world around me as real because that interpretation is very useful, and it is at least as likely to be correct as less useful-for-me alternatives).
However, I also believe that much of what we perceive about the world is our own construction, our own model of reality. I see a specification, for example, as an incomplete and inaccurate description of someone’s fantasy of a system. Such a description can be very useful, but anyone who takes a specification (or any model) too seriously is asking to be bitten by a rope.

(c) Some people present differences of religion as an excuse for intolerance, including the treatment of others with rudeness, dishonesty, or violence. I view that attitude as a weakness. Unfortunately, some people are adept at gaining fame, power, or money from pushing that attitude. That, I view as a disease. Some people in the testing community sometimes characterize
differences in attitudes toward testing as comparable to hostile differences between religions. This can be an effective component of a marketing plan that portrays “us” as victims of the evil “them”. I see such characterizations as destructive. In graduate school, I studied psychology, especially how people perceive (for example, how we see, hear, feel, and experience time), how we interpret, organize and communicate information, and how we can design systems that make their users happier and more effective.

Almost all of my research was computer-controlled. To do this, I had to learn to program. Most of my programming was in Assembler. I reverse-engineered the Apple II operating system and rewrote parts of it and of its embedded version of BASIC to control equipment, collect data in real-time (with precise timing) and do complex statistical analyses of large sets of data. I often had to defend my work, so I had to learn how to test and use test results to persuasively argue that my code was trustworthy. In seven years of graduate study, I spent more than half writing code, writing automated tests, writing articles for programming magazines, and trying to understand what made programs “good”.

So, what was the start point for you to focus on software testing, and then how was the professional journey traversed for you?

I finished my studies in 1983, submitted my dissertation one morning, and flew to San Francisco that afternoon. I had no idea who would hire me so I applied for every job that looked interesting. People offered jobs as programmers, project managers, marketing managers, and software tester. The testing offer was at a company (named MicroPro then) that made the world’s best-selling word
processor (WordStar). They explained to me that WordStar had a difficult start on its new platform (the IBM PC) because it had not adapted its user interface to feel like a PC application. The program worked reliably as designed, but the customers didn’t like it. Therefore, they said, sales suffered badly.

The company was expanding the scope of the testing organization to prevent this problem from recurring. The job was as a supervisor in this expanding test group, heading their testing technology team. I would use my knowledge of human factors to evaluate software usability. And I would use my programming and testing skills to evaluate new test tools and processes, introducing the worthwhile ones into the department.

This was the most difficult-sounding job anyone told me about in an interview that year. It also sounded like the one I would learn the most from and the one where I could do the best right away. I jumped at the opportunity.

I had no experience managing testers at this point. I came to Silicon Valley a believer informal processes, traditional documentation, and the value of designing tasks in a way that maximized delegation of work. Scripted testing (manual or automated) seemed self-evidently good. Those ideas mapped well to the software engineering textbook approaches, to recent books on software factories and talks on testing factories, and to the new IEEE 829 standard on test documentation. So I tried this out.

For some projects, such as testing an adaptation of a well-established program to a new platform, we benefitted from the time spent planning and documenting our testing approach and specific tests.

For many other projects, exploratory tests found most of the bugs. As my ideas about testing project management were demolished, I spent more time seeking out other test managers to learn from their experience. For example, I joined the Bay Area Quality Assurance Club, went to their meetings, and hung out with some of the people I met there. They had similar problems, though everyone reacted to them differently.

What I learned generally was that effective work groups paid much more attention to the quality of their products and much less to the niceties of their processes. The workgroups that seemed most effective to me created very little formal documentation. The companies and workgroups that seemed to produce the shoddiest software also seemed to rely on the most heavily on formal documentation. Similarly, heavily documented testing (scripted tests) provided weaker support for delegation of work than I had expected, at a much higher cost.

I evolved. Instead of advocating The Right Way, I trained my staff that it was their responsibility to do effective testing within whatever software process or corporate culture they found themselves.

The other surprise for me in California was the level of stress. I had expected a fast-paced environment but, coming from a somewhat more polite culture (Canada), I was unprepared for the harsh corporate politics that were common in the hard-driving culture of 1980’s Silicon Valley. My older brother (also in San Francisco, soon to have his own Ph.D. in psychology) had formed a company that interpreted organizational dynamics as family dynamics. I joined them (part-time) to learn how to interpret and navigate the interpersonal dynamics in this new-to-me corporate culture.

In 1984, as my way of making sense of the technical, professional, and social dynamics of software testing, I started writing Testing Computer Software.

What difference do you find between formal testing education given at universities and commercial training?

The university setting provides a different pace for studying testing. We can assign homework to students. We can require students to think about what we’ve taught, to try our ideas on realistically complex software, to try hard things and to question us when their efforts fail. It is very difficult to find time for this in traditional commercial classes, and even when there is time; it is common for students in commercial classes to expect the course to be easy. University students usually expect to learn new things when they take a course. There is no shame in the fact that they don’t have that knowledge yet. Some commercial students, especially some managers, become angry when a course challenges their ideas or reveals their lack of knowledge or skill to other students. Course designers have to work around these differences.

Finding a way to bring the strengths of university education to commercial training has been my objective with BBST.

What is your opinion about creating a degree in testing?

There is a different type of interest that some people have in taking formal education in testing: they are hoping to get a degree in testing. I studied that option several years ago, as the chair of the curriculum committee in my department at Florida Tech. The school encouraged me to create a degree program in software testing. I designed a degree program that met all the requirements for a university degree in computer science AND all the requirements for a university degree in software engineering. Every graduate from this program would have taken the same required courses in math, computer science theory, software design, and programming as the CS graduates who got jobs as programmers.

As I spoke with hiring managers and human resource managers, what I learned was that if we gave students such a degree:

They would almost certainly find it easy to get a job in testing, but they would probably find it difficult to transfer out of testing into any other job in their company. With a stamp of “TESTER” on their forehead, these students would have to get another degree (probably a Master’s in CS) if they wanted to try their hand as programmers or designers.

A university degree should never lock someone into one type of job. It should help the graduate open many types of doors, not just one. And it should never be a barrier in front of one of those doors. Because there seems to be a commonplace industrial prejudice that people who really want to do testing shouldn’t be seen as suitable for other positions, we decided instead to create a “software engineering” degree that included lots of training in testing. Same courses, but the stamp on their foreheads is different. For now, this seems to be a better way.

We’d conclude part-1 here with Dr. Kaner on his career journey with some interesting stories and lessons he learned; along with his views on testing education. Part-2 of this series will feature what Dr. Kaner advises on becoming a good and efficient tester and shaping up to one’s testing career…