Interview With Griffin Jones

Hi Griffin, it has been an honour to interview for this special issue of Tea-time with Testers. Please tell us how have you been and what are you up to?

It is nice to talk with you again – I’ve been very busy.

Since we last spoke, I have continued as a fly-to-the-client, consultant – working in my Venn diagram combination of testing, agile, and regulated.

Six years ago, as an agile coach began working thru BigVisible Solutions, SolutionsIQ, and eventually Accenture on business agility transformations.

Currently, my typical client work is eighteen months, individuals and teams, and up to the C-level in their agile journey. Testing is a part of that ecosystem.

My speciality is helping individuals and teams unfold and move toward becoming the best versions of themselves in their situations and role. I look for the people that have that glow about them – that are ready and willing to level up. I find what they often need is a person to give them some time, attention, and a bit of direction.

It is a bit like being the Wizard of Oz, I help people rediscover the things that already have hidden inside themselves.

When I think of Weinbergians and regulated software, Griffin Jones is one name that my brain never skips. Would you like to tell us more about Griffin the Weinbergian and Griffin the expert on regulated software?

After college in 1987, I started working at Eastman Kodak as a tester – on a giant project that would digitize, index, store, and retrieve all the historical physical paper records of gigantic organizations. I stayed at Kodak till 2007, focusing on testing the initial imaging digitization of different business verticals. Computed radiography is an example of my regulated work, while movie special effects software is an example of ‘cool’, but unregulated work. The big point is that I sought out broad technical and line-of-business experiences while developing a deep capability in medical devices.

That broad experience led to my ideas about testing evolving (described with some exaggeration) from a Boris Beizer function point proof, with elaborated and traceable requirements, a Capers Jones metrics program, and a Quality Assurance gatekeeper’s veto mindset – toward an Exploratory Testing mindset as described by Cem Kaner and others. Bret Petticord’s Four Schools of Software Testing is a good take.

In the early 1990s, I discovered Jerry’s books, and his ideas fit for me. But I could never fully implement my insights at Kodak.

In 2007 I quit Kodak to work at the startup iCardiac Technologies (which designed and delivered cardiac safety analysis for pharmaceutical clinical trials) where I led their software testing effort and eventually became the Director of Quality and Regulatory Compliance. During my time at iCardiac, Jerry and Cem Kaner were both presenting at CAST 2008 in Toronto – and I decided to attend the conference and Jerry’s workshop.

Jerry and the many people from our community attending the conference – made the event magical for me. I was hooked. I left iCardiac and became an independent consultant, binging on Jerry and our community – trying to make up for a lost time.

Jerry and the many people from our community attending the conference – made the event magical for me. I was hooked. I left iCardiac and became an independent consultant, binging on Jerry and our community – trying to make up for a lost time.

I directly interacted with Jerry of the following years by attending multiple AYE conferences [https://www.ayeconference.com/] , attending the Problem-Solving Leadership workshop (PSL) [https:// www.estherderby.com/workshops/psl/] , and the Change Artistry workshop [associated book: https://leanpub.com/changeartistry]. My recent work with Jean McLendon and the work of Virginia Satir, and the Organization and Relationship Systems Coaching (ORSC) is just me following Jerry’s footsteps.

Jerry has been the most influential person in my life that I did not share a home with.

On the day I learned of Jerry’s death, I was working with a new client in Chicago. Throughout the day, I noticed each time I did a Jerry-like thing – and would remember with sadness-of-loss the past interaction where I learned that specific consulting move from him. I was struck by how often that happened. By the end of the day, those memories were still tainted with loss. But, there was also deep soulful gratitude to him for what he gave me: for what I had learned, and for the gift of his presence with me during that time in my life.

He profoundly changed me.

My regulated industry experiences are more straightforward.

First, I think I inherited some of it from my father who was deeply involved in the nuclear power industry at the local power plant as a trainer, safety officer, problem investigator, and designated company interface to the regulator.

Half of my career at Kodak was in Health Imaging. Plus I was often given a side-mission to create the Software Development LifeCycle for the program. I happened to be in Health Imaging in the middle 1990s and helped adapt the FDA Quality System Regulations (QRS) when they became effective.

I left Kodak to become employee #11 at iCardiac, where I was part of the team that created and implemented the entire FDA-compliant QRS. Multiple employees of the company were also Special Government Employees of the FDA which was our first customer. We built the QRS to allow us to implement agile practices in a compliant way. During this time I joined the Regulatory Affairs Professional Society (RAPS) and became the Director of Quality and Regulatory Compliance. iCardiac was incredibly successful, and I was hosting pharmaceutical onsite audits monthly. I started speaking at conferences and became a co-host of Workshop on Regulated Software Testing (WREST). My story of blending agile with regulatory compliance attracted people’s attention, so I left iCardiac and became an independent consultant.

I worked as a consultant in multiple roles with medical device, pharmaceutical, and financial services companies in a blended testing and regulatory affairs role.

In my agile coach role at SolutionsIQ / Accenture, I am often the liaison between the regulatory and compliance organization and the larger business agile transformation. I am fluent in “regulatory”.

How does testing differ when it comes to regulated software? Do testers assure quality in the regulated software domain?

Well, when it goes well, testing in that context has a four-part mission:

a. Gather evidence to support a “you built what you said you built”.

b. Gather evidence to support that “it produces the desired outcome”.

c. Actively gather evidence about risks to the testing effort and the business.

d. Weave all that evidence into a coherent story that your management and regulators will understand.

Elaborating on each of those:

a. Gather evidence to support a “you built what you said you built”

a.1 Which is like ‘verified’ to use the FDA legal meaning.

a.2 Which is like an attempt to understand in a very Cynefin ordered domain way.

a.3 Which is like to behave very ISO 9001-like.

a.4 Which is like to adopt a Phillip Crosy ‘conforms to requirements’ definition of quality.

[Cynefin: https://www.youtube.com/watch?v=N7oz366X0-8 ]

b. Gather evidence to support that “it produces the desired outcome”.

b.1 Which is like ‘validate’ to use the FDA legal meaning.

b.2 This is like an attempt to understand using a very Cynefin complex domain meaning.

b.3 Which is like to adopt a Joseph Juran ‘fitness for use’ definition of quality.

c. Actively gather evidence about risks to the testing effort and the business.

c.1 Gathering evidence of the active exploration of risks to the testing effort.

c.2 Gathering evidence of the active exploration of producer’s risks to their business (using ever harsher Karl Popperian assumptions).

c.3 Gathering evidence to help answer the question ‘can we operationally execute all of this in an acceptable way as an ongoing business?”.

d. Weave all that evidence into a coherent story that your management and regulators will understand.

d.1 Construct a coherent story of what it all means, in a very Michael Bolton Braiding the Stories way. https://www.developsense.com/blog/2012/02/braiding-thestories/

These missions work for non-regulated just as well. They are just good engineering. But when it goes bad, one (or more) of those four missions has failed:

a. We didn’t build what we say we built.

b. It doesn’t work.

c. We can’t function as a business doing this.

d. We can’t explain the story of what happened in a way that our regulators will let us stay in business.

Regarding “assuring quality”, well it depends. Some individuals will take on that mantle, but it is an overreach – as Brian Marik explained in The Testing Team’s Motto.

http:// www.exampler.com/testing-com/writings/ purpose-of-testing.htm

But, sometimes the organization is constructed around that belief, and the QA organization acts like an uber-program manager. Brett Petticord described it in The Four Schools of Testing as The Quality Assurance School. That pattern is tending to be becoming rarer, except in the case where the organization had a near-death encounter with their regulator.

Would you say that in a regulated software context, there is very little room for creativity and free-thinking for testers?

[Laughs out loud] Yes and no.

Magic happens when you can be creative in a compliant way. The important problems will require creative compliance to solve.

Yes, because there is the desire to gain more understanding and reduce risk. It often takes creativity and free-thinking to make that happen.

No, because working in the “compliant” way tends to be slower and more expensive. And if you are going to share this work as evidence, you are not prepared to share the unedited versions of your attempts with all the mistakes and failures transparently included.

And frankly, some of the work is grunt work that you just have to grind through. It is part of the Cynefin Simple domain.

Part of the problem is the language and zerosum thinking around control. There is a wonderful TED-talk, Lead Like a Great Conductor, by Italy Talgam – that illustrates control and creativity operating at different levels. See 14:50 to 16:47 of the video. [https:// w w w . t e d . c o m / t a l k s / itay_talgam_lead_like_the_great_conductors]

Look for opportunities to put down the baton, and give others the freedom to contribute in their own way.

It is also expressed in Turn the Ship Around, when the Captain “vowed to never give another order ever again”. Clarity of Intent, Competence – and then you can give people Freedom. [https://www.youtube.com/ watch?v=psAXMqxwol8]

How does the strategy for automation in testing get affected in the case of testing software in a regulated environment?

There are a few key points about using automation in this context

The speed, precision, and repetition that tools can offer if used well can be an enormous benefit.

But before they can be used, every tool in the entire software development life cycle and development stack needs to go through basic processes of: vendor qualification and product selection, installation qualification, operational qualification, and production qualification. These answer the questions:

a. Who and what are you making/buying, and why are they a good choice to choose them?

b. Can you install and configure it in your larger system?

c. Does it operate (for the aspects we care about) as they described it in their documentation?

d. If you run it in your actual production environment, with production data, and production standard operating procedures – do you get acceptable results?

e. Keep documentation describing what you did for all the previous steps and what decisions were made, and by whom; have it at hand and reviewable by a third party.

All that work is done before you use the tool for its intended purpose. Using the automation tools in these ways and preserving an authoritative and complete record of results is part of the significant work of being compliant.

The best use of automated tools that I have seen are hybrid systems that combine the strengths of tools with human judgment, where the result is better than either alone. The tools don’t fully replace the humans, rather they augment the judgment and evaluation capabilities of the people.

[This is elaborated in my talk What is Good Evidence?]

You have great experience in the testing field as well as in coaching teams for Agile. How do you think Agile has affected the testing profession and the industry perception of it?

Testing won but didn’t realize it.

Left-shift is a win for testing. Continuous integration and continuous delivery are a win for testing. TDD and BDD are a win for testing. People with T-shaped skills on teams is a win. Story acceptance criteria is a win. Frequent demonstration of valuable working software (to your customers) is a win. The whole team-ness is a win. Development and management have taken two big steps in the direction of “getting better incremental information via experimentation about what is really happening”, aka testing, is a win. But management will frame and explain it from their own points of view, so Testing doesn’t notice the win.

The best people and teams I have worked with were T-shaped with one or two deep specialities, multiple secondary skills, and always a desire to learn, teach, share, and pair. They tend to identify themselves to outsiders as “team members”. The formerly-known-as-testers on these teams all bringing something extra to the party: facilitation skills, technical skills, UX-design, analysis skills, the ability to influence management, understanding of the business, relationships to others in the broad organization, new-eyes — but they are all not one-trick testing-only-ponies.

The best situations seem to disperse the testers into delivery teams, while also gathering them back together in testing communities of practice – where they can hone their craft. It is described as Business Agility in Scaled Agile Framework (SAFe) – a responsive value delivery structure along with an organizational hierarchy to give some permanence. [https://www.scaledagileframework.com/businessagility/]

I remember having a discussion with you around feedback loops and the system collapse (How Software Is Built – Jerry Weinberg reference). In the regulated software industry, have you seen the system collapsing? If yes, what were the reasons, and if no what protected them?

Run-away systems either collapse or explode. In Managing the Risks of Organizational Accidents, the author James Reasons cites multiple examples of critical systems failing – planes crash, banks fail, chemical plants explode, etc. So catastrophe still happens, just infrequently or the consequences are contained (and the public don’t notice the near-miss).

Risk equals probability times impact. Regulated industries are regulated because by the nature of the work, they generate high enough potential risk of unacceptably high costs (especially to innocent and uninvolved bystanders), that society has imposed special conditions on the producers of those risks.

The people responsible for designing and running these systems have done a good job preventing single-points-of-failure. They recognize the critical nature of human communication and have instituted Crew Resource Management procedures and training. Since failures of the system are so infrequent, they actively search for and investigate “near misses”. Often, they discover that the action by the human in the loop was what prevented a catastrophe. Or not. [https://www.youtube.com/ watch?v=kjLrZ2SDDaU]

[Beyond The Black Box: The Forensics of Airplane Crashes by George Bibel; Organizational Accidents Revisited, by James Reasons; Normal Accidents: Living with High-Risk Technologies, by Perrow; Aircraft Safety: Accident Investigations, Analysis & Applications, by Krause; Crew Resource Management: Principles and Practices, by LeSage, Dyar, and Evans.]

I assert the recent Boeing 787 Max catastrophe was just the most visible outcome of a series of little failures with a root cause of the collapse of Boeing’s engineering culture. [See the Swiss Cheese Model: http://blog.enterprisetraining.com/swiss-cheese-accident-causationmodel/ ]

As our system grows bigger and older, a growing risk is the combination of: “your system and all its’ emergent properties in the present is infinitely larger than your head was in the past, or is now – so you can’t be absolutely sure what is going to happen”; and we are reluctant to retire anything – maintenance on and extension of existing systems can be very risky. Especially the low-risk changes.

A tiny little close-to-zero technical enhancement to a text editor on the Therac-25 was a technical cause of the device unexpectedly entering calibration mode, massively irradiating six people and killing them.

Reading the IEEE account of what happened and why it happened should make you weep. https://ieeexplore.ieee.org/document/274940

Do you think software testers by nature of their work and skills are best equipped to help prevent systems from running into collapse mode or not?

Yes, in the sense that they are at the intersection of where the questions are first asked, and where the resources and inclination are to investigate.

Tester’s super power is a belief that “things can be different” than they were described, and a strong inclination to run an empirical experiment to support or refute a theoretical analysis. Testers need to be able to observe without preconceptions the produced system, the production system, and the human system doing the production.

Management giving license to testing to comment on all of this is a political problem for the organization. Often the testing group’s mission is shrivelled because the testers lack the technical skills to investigate and the political skills to present the results. Or, management fears that they lack the skills to effect a change in the organization in response to what testing would find. So the questions go unasked.

Jim Bullock’s Big Book of Testing elaborates on this idea. [https:// www.linkedin.com/in/rarebirdenterprises/]

As someone who has closely worked in an industry where a small failure in software can risk the patient’s life, what is your opinion about the “Good enough quality” notion with which tech companies are trying to operate with, more and more (apparently)?

I have a few takes about “Good enough quality”:

First, context matters. Second, search for the palmed cards. Good enough for whom? For what values of ‘good’? For what purpose? For what costs? Compared to what? Who gets to decide? Who is impacted?

I think Jerry’s definition of Quality helps untangle the question: “Quality is value to some person that matters, (at some point in time)”. Who matters, or doesn’t matter? When do they matter? What does each of them value? Why do they matter?

Using that analysis, it seems to me that the general use of “Good enough quality” is about producers deploying what they think is a minimal viable product in a marketplace sooner versus later, and getting faster feedback. They have a gold rush model of how the marketplace works. They believe that consumers will tolerate a minimum viable product.

The extreme version of “good enough quality” is not really a viable commercial option in the regulated spaces because the regulator has the capability to block sales, confiscate your goods, freeze your business with investigations, fine you or just take your money, and put you in jail.

Regulators are incepted to search for bad actors and expand their regulatory power and scope.

Do you think our industry is suffering from a Cargo Cult problem especially when it comes to deciding about testing culture? (E.g. Google, Microsoft, Facebook test their product so and so way, let’s do that because it must be the best thing to do.)

Success begets imitation, and imitation is the sincerest form of flattery.

The fast follower is a strategy that can work, but to work, it has to be more than buying a template, or adopting the processes of others. They are not magical items that sit on our shelves.

In agile coaching, this is the “they are doing agile, rather than being agile” problem. It starts with a mindset, accepting “this is my problem to solve”, and engaging with the problem, including emotionally. It is like the scene from Bruce Lee’s entering the Dragon, the actions need emotional content. https://www.youtube.com/watch?v=sDW6vkuqGLg

If they are Cargo Culting their testing culture, they are focusing all their attention on the tangible artefacts, aka, the finger – versus the moon.

Choosing a high stakes strategy of half-forgotten, poorly understood, and badly executed rituals for channelling dangerous forces rarely leads to successful long term outcomes. But maybe we’ll get lucky this time.

I get a lot of ‘you must do test cases in a medical / life-critical environment because of auditors/FDA etc’ – is that true? Were you able to use other forms of testing documentation other than test cases which the auditors were satisfied with?

The short answer is, “No, the regulation does not explicitly call out for a thing called ‘test cases’”.But the longer answer is, “maybe”.

Maybe your company has painted their policies and procedures into a corner, and “test cases” is the only black-letter-of-the-law way to comply with your organization’s policies and procedures. Maybe people can’t conceive of a different way, or can’t convince an authority in the company to document a limited waiver to allow a different way.

Maybe your regulatory posture and relationship with customers and the regulator demands that you use test cases.

Maybe you can’t explain why a different way could be acceptable and address the concerns of those that matter that has concerns.

The most elegant solution I have implemented that survived audit was to embed inline a short test charter, description, and acceptance criteria with each requirement/story so they were tightly coupled. All testing was video recorded, both the tester and the system. All electronic test data, output, and logs were saved, analyzed, and preserved. Ditto with physical results.

The key in my opinion was having the tester confidently explaining real time what they were doing, why they were doing it, what system results they saw including anomalies, and what the meaning of the results was.

Pre-written text cases are wishes for what I hope will happen. What actually happened is bigger than a confirmation-bias “passed” test case checkbox. I want to see and hear a recording of what really happened. I want to watch the tester synthesizing meaning during the performance from tacit knowledge, observations, and pre-analysis and preparation. And express it so I can record it and share it with a regulator.

I have also seen traditional agile scrum stories with acceptance criteria pass audit if underneath the Jira ticket was enough evidence. Again, did your performance generate enough evidence, can you explain the evidence, does the story make sense?

Is it permissible to attempt, do you understand the different stakeholders and their concerns, can you design a test and make preparations, can you honestly and skillfully execute it, do you address the concerns raised in question #3, do you collect enough evidence of what was done during the execution, and can you construct a credible story with all of that information that a third party could examine and arrive at a similar conclusion? If all the answers are yes, then consider it. Like all things, consider asking for help.

You are an inspiration, Griffin. We would like to know more about things that made you who you are today.

I remember a summer weekend in 1975 when I was about ten. My father left me in the new truck alone while he ran into the store – the truck was off, and dad had the keys. I decided to investigate all the buttons and controls. I remember puzzling out that if I turned on the hazard lights, and turned on the turn signal – I could briefly energize the circuit to the fan or the radio. I always wondered if I had found a secret behaviour that even the designers were unaware existed.

Activities: Science fiction and military history stories, chess, coin collecting, dogs, skiing, and the stock market, hopefully running again soon.

People-wise: Mom and Dad. Going to a Jesuit high school. Jerry and his work with Virginia Satir. My wife.

And Cindy Halstand, who delivered a very personal message to me, “Don’t be afraid.” I have pondered and tried to heed that message since.

It is the interesting people I get to interact with that is both thrilling and exhausting.

What would be your advice for an experienced tester today and also for someone who is willing to make a career in testing?

Spend the time to work on yourself. Search out and make time to spend time with the best people you can. Partner with others, find a coach. Keep a journal, exercise, practice mediation, get enough sleep, and eat correctly. Read Becoming a Technical Leader, by Jerry. Read and explore other fields – how might that be similar to testing? Be kind to yourself – life is about making choices. Practice and get really good at something other than testing. Consider reading the best and most useful book I have ever read: More Secrets of Consulting, by Jerry.

Work on your Emotional Quotient, and the associated soft-but-difficult-skills. Work on being Congruent, in a Virginia Satir way.

May you grow into the best version of yourself possible, and may you help someone else you care about to do the same.