Testing testing: a response to http://stream.red56.co.uk/post/24561863889/just-too-much-quality

Hey Tim - this is a very interesting post.  I'm tempted to try to write a structured response, but I'm worried in the process I'm going to miss crucial things that I want to say.  So I'll just jot down some headings.

Quality is a trade-off.

This is what tends to get forgotten when people talk about quality, as you say.  I think it would be hugely beneficial if we could some how make these "trade-off"s more explicit.  As I was reading through this it seemed to me like it would be a good idea to draw some diagrams.  Draw some graphs and ask each team to "pin the tail on the donkey."  But how do you decided where along a continuum to decide on quality?  Maybe you can't/don't.  Maybe you use some outside signal.

What's the signal?

It strikes me that in something like the Toyota production system they have a few ways of signalling whether quality is hitting the sweet spot or not.  The big one is "Is the production line moving."  If it isn't, well then there's a problem that needs fixing.  Another is that they build deep, close, long-term relationships with their customers so that they have a good idea what they actually think of the product.  I'm not sure how exactly that gets fed back, but I remember reading from the Womack and Jones books that that aspect of the Toyota approach is one of the hardest for Western Companies to accept, let alone mimic.

I talked to some guys once from one of the price comparison sites and they told me that they'd got rid of their QA guys and replaced them with a focus on the conversion rate for the site (the overall number of people who switch suppliers as a result of visiting the site).  They added a new piece of functionality, if the conversion rate went up, they either left it alone or maybe looked at providing more improvements in that area.  If the conversion rate went down, they either pulled it or changed it.  Of course, I'm guessing, getting rid of the QA guys probably didn't make the developers more callous about testing for obvious defects, quite the reverse.

What you need to be asking with quality - maybe, is what's the signal that tells us we've got enough?  Is it really when there are no defects? 

Standards

In some recent Agile transformations I've repeatedly heard "what's the standard." Because software isn't in the quadrant of the Cynefin framework  where standards are useful, because Alastair Cockburn is exactly right that we need to have a methodology for every project  standards are very often a waste of time. Or possibly they are just at the wrong level of focus.  What if your standard said something like "1) Identify as a project what the signal is that tells you whether quality is 'good enough' 2) develop in adherence to this signal 3) inspect and adapt - change the signal, improve the clarity and communication of that signal.  Would that be OK?

Poke Yoke

"Special" characters. How many defects do you see that are related to special characters?  Some of them no so special, like apostrophes.  Would it be worth just writing a special characters test for every text field, or having access to some kind of library that tried to do put mad, malicious text in a field, as a starting point?  This would be a Poke Yoke approach to testing and chimes in with the W Edwards Deming concept of the red bean game.  Every defect tells us that our approach is wrong in some way, that our system for generating code needs improving - how can we improve it.

Money and Sense

You said this to me, and I'm not quite sure that you knew what you were saying, but I've  come up with an interpretation of it that I find hugely satisfying. I was reading "Grunch of Giants" and telling you about what Buckminster Fuller says, right at the beginning of that book - "You can either make money or sense."  And you said something like "Yes, but being a really skilled craftsman, that's a kind of nonsense."

And I don't know whether you meant this or not, but I took that to mean "maybe you can choose what kind of nonsense you use to make that money." Maybe.  

Rephrasing this - one approach to success is to be "insane" about something, to pay insane attention to something.  I think this is what the electricity bill switching quys were doing.  They were becoming insanely interested and sensitive to the conversation rate. Maybe one way of thinking about what Toyota are doing is "become insanely interested in making cars that their customers want at the rate of demand." What sorts of things could you be insane about? Agile process. Customer satisfaction? Visual design, code quality, defect rates. In a way, appeals to "good enough" are appeals to money rather than sense and so to non-sense rather than sense. They're saying "look, you will never be able to show logically that the code does exactly what it's supposed to do logically.  And anyway, actually, what's supposed to do is make money." My feeling is that you probably can't be insane about more than one thing at once.  But if you are insane about something, you might make even more money.

Posted via email from The Ginger Mumbly