The QA Team Should NEVER Find a Bug!

Many companies use metrics to measure the performance of the Sales Team, the Developers, the QA Team, etc. For QA the metrics frequently include:

• The number of bugs that escape into Production.
• The number of bugs caught in QA.

Many people believe that these two numbers are in opposition to each other – in order to have fewer bugs escape into Production, we must catch more bugs in QA. If we catch fewer bugs in QA then more bugs will escape into Production. The fundamental assumption here is that all bugs must either be caught in QA or escape into Production. Everyone agrees that they would like the number of bugs that escape into Production to be zero; I will argue that we should also want the number of bugs caught in QA to be zero. These two numbers are not in opposition; there doesn’t have to be an either/or tradeoff.

If we don’t catch the bugs in QA, how can we prevent them from escaping into Production? The answer is ‘Shift Left’ – catch them in Development, before they ever get to QA. That means (of course) better unit testing and (hopefully) TDD, both are great tools, but are not very effective for catching integration-layer bugs. The only way I know of to:

• Shift-left with the integration testing.
• Deliver automated tests in-sprint (before the developers finish writing the code).
• Catch all of the bugs in Development.
• Reduce to zero the number of bugs caught in QA.
• Reduce to zero the number of bugs that escape to Production is to change the methodology.

Introduce Specification by Example/BDD and automate the requirements rather than writing test cases. If the QA team has to write test cases before automating then it will be impossible to deliver test automation in-sprint. Using SBE and directly automating the requirements we can deliver automated tests to the Developers before they finish writing the code; the Developers can run the tests before they open a pull request, and find and fix the bugs before any bug reaches QA. Once that change is baked into the system, QA doesn’t have to run the tests – running them twice won’t find any bugs. Then QA will find zero bugs, and very few bugs (zero if we wrote complete requirements) will reach Production. It is a major paradigm shift, it isn’t magic, it isn’t easy, but it does work; I have helped teams do it. Have fun driving to zero!