Software Testing Economics

Exhaustive software testing is impossible. In this article series we will examine why this is so.

Why this topic

I have been thinking about this subject for almost a year, and I don’t know the answers to many questions yet.

A year ago, I was surprised to hear that the statement ‘Exhaustive testing is impossible’ was not obvious to everyone. I happened to speak with a person who asked ‘Why?’ and I referred him to the book by Glenford Myers, Software Reliability: Principles and Practices, where an example was given. The answer was: “Well, but that is not mathematically proven there!”. I recalled my mathematical background and said that there is Cantor’s theorem which states that a set of natural numbers is countable and if it is countable, then it is not finite. And if we get a calculator which adds natural numbers, then it will be IMPOSSIBLE to test it exhaustively, and so on and so forth.

Another source for arguments is the ISTQB syllabus which unambiguously states that exhaustive testing is impossible. Yet another source is a consequence from the definition of a Turing machine – that its holding problem is unsolvable, as stated in a standard course of automata theory. And finally, common sense: we are all software testers (those who are not yet should want to become testers, as there is no other way for human evolution). And we have some testing experience and understanding that it's impossible to write ALL test cases and select ALL test data. That we cannot travel ALL the paths in code, and perform ALL actions that users can do, and son and so forth.

It seems to be an absolute truth, but questions on this theme arise again and again at various levels, and especially at the top management level:

  • “When are you going to complete the software tests?” (“To complete” means “to have everything done”).
  • “When will ALL the defects be found?” (Of course, nobody will be satisfied with a half or even 99% of found bugs)
  • “And when will we deliver bug-free software to the customer?”
  • If there is a defect in operation - “Why was this defect not found?”
  • “And why were not ALL the defects in operation found before commissioning?”

How should you answer these questions? They all imply exhaustive testing, which we cannot conduct.

Software Testing Economics.jpg

Test completion criteria

Here another question arises: What are test completion criteria? Almost all test completion criteria should include something like this:

  • If the damage from an unclosed defect does not exceed the costs of its correction, you don’t need to correct this defect
  • If the damage from a potential defect does not exceed the costs of its searching and correction, you don’t need to search this defect
  • If the overall damage from all known unclosed defects does not exceed the benefits of implementing the existing system version, it should be implemented (even with such known unclosed defects)

Pay attention to the words “costs” and “benefits” in this list. Ultimately, if you have to spend more money (a universal measure of work and result) for defect correction than the damage that can be caused by such a defect, then you should not correct that defect. It’s hard to put up with this fact because it has far-reaching effects. More in the second part of this article.

Check out our software testing trainings and start working on developing or expanding your software testing skills.

Come learn with us!

Alexandr Alexandrov
Software Testing Consultant

Share the knowledge

Still have questions?
Connect with us
Thank you.
Your request has been received.
Thank you!
The form has been submitted successfully.