Over the course of the last year, I decided to dive deeper towards the source of traditional testing wisdom. That said, I read a book on the foundation knowledge behind one of the testing methodology (I won’t name anyone to protect the guilty). In the end, I was both surprised, and shocked. Here are some of the things that stood out to me. I will not name the book or the authors, as I was able to receive them as sort of a gift. If you want to know more details, feel free to contact me privately, and I might share more on the book.

What surprised me

There were a lot of things that surprised. Here are some of those pieces that I will paraphrase.

Even if you execute all tests, and those did not reveal any error, that does not mean that there are no additional test cases that would trigger errors. Yeah, I can agree to that.

Testing cannot proof absence of errors. Complete testing in this sense is impossible since there are too many possible combinations. I certainly can agree to that.

Testing effort at times is expressed as ratio of testers to developers. The authors claim they have seen ratios between one tester for 10 developers up to three testers per developer. The authors also conclude that testing effort practically varies a lot.

Testing consists of various short sections of of work activities, including test planning, test analysis, test design, test execution, evaluation of results, and conclusions of testing related work. The book claims that some of these activities may be carried out in parallel. The authors fail to mention exploratory testing at this point, but that’s what I was thinking, surely.

“Is it possible for a developer to test his own code?” The authors cannot give a commonly valid answer to this question. Yeah, I would also say, that depends.

Developers and testers can collaborate better with each other if they know each others’ work. Developers should know some basics about testing, as well as testers should know some basics about programming. I certainly agree to that.

Overall, I was wondering where some of the rumors like “developers cannot test their own code”, or “tester do not need to program” came up. It seems not from this particular testing methodology.

What shocked me

Most of me reading a traditional testing methodology book was triggered by James Bach. He claimed in April that he read some of the traditional testing literature, and was wondering what people were actually doing, and where the mention of people in that literature was. Since January 2013, I already was aware about Roberts’ Elements of Organizational Design, in short, PArchitecture, Routines, and Culture. So, if you address only the hierarchy in the organization, and change the routines, your are likely to miss out on an important factor, the humans that actually do the work, their culture. I found the same holds for that particular book that I read. The language that was used in the book was quite passive. If humans, testers, appeared, they were supposed to do something. The writing style bored me. How can a book about such a widespread testing methodology miss out on such an important audience as the reader? I was shocked.

What shocked me as well is the bogusness of test coverage metrics. It really hit me in the face. Here is an example. The technique used is boundary analysis. Test coverage can be derived by

boundary value coverage = amount of tested boundary values divided by the total amount of boundary values times 100%

The math is pretty straight forward, so what is my problem with something like that? Here is a another example from equivalence class partitioning:

equivalence class coverage = (amount of tested equivalence classes divided by total amount of equivalence classes) times 100%

Can you spot it now? Here are three problems I have with such a statement. First, it includes an ambiguous term. “total amount” in the equations above should be translated to “total amount of stuff we identified”. That would make it clearer that the applied technique needs a skilled tester. Take for example the total amount of equivalence classes. How were these derived? Did the analysis include analysis of solely input variables? Input and output variables? What about secondary dimensions? Proficient testers might realize that the world is a bit more complicated than the picture I read about coverage metrics in the book. To say the least, I have never found a single technique alone is applicable to derive good tests in my work. I also needed some combination of different techniques – and that is probably why it’s hard up to impossible to say anything about the coverage of the tests that I have ran.

My second problem with such statements is, that the unfamiliar test or project manager will be faced with some number, say “we have reached 75% test coverage yesterday”, and will not understand what that really means. Does that mean that we finished 75% of the tests we thought of when we knew little to nothing about the implemented product? Did we execute the most critical 75% tests? Will all areas be covered, even those areas of the product that existed before we began further development? Or does that number just reflect the percentage of test cases we came up with for the areas that we thought were changed during this development cycle? As you can see, there might be a lot of hidden assumptions in a statement like that. Most times in the past I have found a surprised project manager that later found out we had a different understanding about the underlying assumptions – in other words we finished with a shallow agreement about the statement.

Third problem, consider a workplace where a certain amount of test coverage is demanded by KPIs, and employees will receive a higher or lower pay due to that. Say, a manager asks the testers to deliver always 90% test coverage before the quality gate is passed. Test coverage will go up, for certain. The question is how. You can either deliver a higher amount of ran test cases, putting more effort in there. You can also identify fewer test cases, and still hit the asked for test coverage metric while delivering the basic same amount of work that you always delivered. Clearly, you can look good in face of your manager by coming up with fewer and poorer test cases, and still make a whole bunch of money out of it. I think at times that I am too honest to move away from such kind of positions. I don’t think it serves the manager, the product, and the customer well to do such a lousy and lazy job. I am sorry, my dear tester colleagues. Sure, you can still do a great job of reducing the amount of total test cases to a small amount, and still do very skillful job at test design. And I hope you work in one of the latter environments.

A last thing that shocked me was the amount of test techniques delivered in the book without stating when each technique is best applied. I think this is most crucial piece of a software testing training. Knowing your tools, but more important than knowing how to apply them is knowing when to apply them, and under which circumstances. The examples provided, being textbook examples, were of course simplified so that you understand the techniques in use without reading 300 pages of context first. Given the potential audience of the book, the authors didn’t happen to describe situations in which this particular technique would be worthwhile to follow. So, in the end, the reader wonders when to apply equivalence class partitioning over boundary value analysis, and which of the techniques is applicable for the requirements in that 300 pages document he has back at work. I can only hope that this particular skill is delivered during the training classes that go with the certification – but I doubt it.

What my journey has taught me

Overall, this journey has taught me that there is a lot of misguidance going one with some of the traditional text books. I am not surprised about that, but I really needed to find out what I would be objecting against. I mentioned the two outstanding arguments that I kept up to half a year later.

I recommend you to go to the source as well, and see what it teaches you. The authors usually are well-taught persons with good intentions, I am sure about that. So, there will be something that you can still learn from them. And I urge you to keep your critical mindset while reading on.