This year I took all three courses on Black-box Software Testing. Each of them means an investment of four weeks of my time, usually up to 2-4 hours per day. This was quite a blast, and I am happy that I made it through the courses.
One thing that stroke me in the first course was the different uses and misuses of code coverage discussed in the first part, the Foundations course. Here is a short description of things I have seen working, and not working so much.
The main point in the paper by Brian Marick on How to misuse code coverage is that you may use code coverage to measure things that your tests cover in your code, or you can use it as a hint for where you haven’t tested anything – and decide in brain-on mode whether to cover that particular area or not.
The former point is a measurement, and might become a management metric. Especially in Theory X thinking code coverage becomes a metric for the performance of individuals or teams. This has some drawbacks, as Kaner and Bond point out in their classic paper on Software Engineering Metrics – What do they measure and how do we know?. The main distinction is that measuring code coverage is a second-order measurement for something else. It’s a substitution, as Kahneman describes in Thinking Fast and Slow in the sense that we are answering a different question. The question we are trying to answer is whether this code is sufficiently tested, and how maintainable it might be in the long-run.
When putting in place a code coverage metric I have seen dysfunctions as Kaner and Bond describe them. Recently I heard from a company that demanded 100% code coverage. Though Robert C. Martin demands that this should be a goal for any great development team, when put into a measurement metric or in a contract, it more often than not becomes a problem. I also heard about a company where the development team was given the goal to reach at least 50% code coverage. This setting led to generated tests that aim for code coverage, but not for useful tests that help you drive and maintain the application in the long run.
On the other hand, when using code coverage to check what has not been covered with automated checks, we get a clearer picture about our situation. With our brains switched on we can distinguish between code that should better get some more tests to cover that particular complicated flow through the application, or whether to leave that empty try-catch-block as it is right now.
In the past I have applied code coverage more often successful from inside the team as a measurement to measure pieces of code that we did not (yet) cover. Think about it as an information radiator. Code coverage makes then clear which pieces of your code are not covered, yet, and you can think through whether you need more tests. Code coverage then becomes a first-order measurement, which the team uses to bring forward their development efforts.
Code coverage can be used in useful and misuseful ways. How are you using code coverage in your company?
Fell in the trap of total coverage – http://jlottosen.wordpress.com/2012/11/05/fell-in-the-trap-of-total-coverage/
Nice post Markus. I agree with what you say in your post (except one very minor point about using code coverage to see the coverage of automated checks – I suggest it also helps within the team for seeing the manual test coverage).
I presented a very similar opinion within my keynote address at the KWSQA conference earlier this year (which I will also present at Let’s Test and STARCanada next year).
I agree that it is a very useful to use within the team and dangerous when used outside the team. The same goes for Requirements coverage.
So as I said at the beginning, nice post.
Code Coverage (like any other metric) in isolation will likely yield bad results. It is important when designing metrics for a team to have about 3 complementary metrics. For example @nohz recently suggested Code Coverage, Cyclomatic Complexity and Code Churn as three metrics that, when in balance, help keep things in check. In this example, you want these three in balance; if you have high complexity you want to have high code coverage, but if complexity is low, code coverage is less important.
Hi Mike,
the project I was referring to had KPIs for cyclomatic complexity, code coverage, and architectural constraints in place (among others), yet showed bad unit tests were generated and merely introduced to meet the coverage criterium.
This once again convinced me that there is not an alternative to talking with the people involved, and coach them on their job.
Indeed. Nothing really beats understanding the values and principles underlying a practice.
In reality, code coverage is just the beginning!
Hi Markus,
I have worked in environments where both code coverage and Requirements coverage were in place. As Paul Holland had mentioned it is good to use with in the Team, we had used it both for developers & testers alike as part of Unit, Integration and System testing and worked together(some times) to improve. How ever now it is getting followed organization wide and it is one of the Primary KPIs. All the KPIs give a false sense of security till it reaches production and a slew of leak defects destroys careers as Testing teams were considered to be the Gate Keepers of Quality though testers are not responsible for any of the quality of the product life cycle phases except Testing.
Bottom line is we need to think more about improving the quality of Testing instead of worrying about meeting the metrics.
Thanks,
Jagan.
p.s. I agree the BBST test design course is a blast and it is one of the very active learning process i had faced recently. For me, still Bug advocacy is pending.