Yesterday James Bach rumored around a link to a test case execution time calculator. Besides the fact that it’s complete non-sense to use such a thing for professional testing, I started to play around a bit. I ended up with a ridiculous result.
I think there is more to it. So, here is your mission, if you dare to accept it.
Product:
Regression Test Calculator
Mission:
Test the regression test calculator for any flaws you can find. You might gain bonus points, if you can find out how the calculation is done. Another set of bonus points if you can come up with a better approach.
Not really a challenge to find the flaws, took me 2 minutes to get a divide by zero error
Warning: Division by zero in /srv/disk3/862004/www/testercrew.com/regression-calculator-exec.php on line 88
I wondered if I should pick up the glove on this one. As Phil points out it seems not to be much of challenge. Still I cannot resist to write down my results while testing the application for 15 minutes. (And I am ignoring the riduculity of the application.)
I start out by trying to see how it calculates.
It seems to calculate the number of regression test cases by dividing the number of hours with the minutes per test case times the number of testers. (There is an assumption in there that the hours are a measure of time and not of cost as it is so often.) This opens the posibility of checking for divisions. First try a zero division triggers Warning: Division by zero in /srv/disk3/862004/www/testercrew.com/regression-calculator-exec.php on line 71
Second try a division say 0.0083 for Hours Available (slightly less then half a minute) and the number drops to 0. Use 0.0084 and it rises to 1. So the calculation has typical rounding factor.
Lets try a negative for average time, ah a negative outcome, and a double negative, as expected a positive outcome.
So move on to Number or test cases in test suite and put a zero there. A presto: Warning: Division by zero in /srv/disk3/862004/www/testercrew.com/regression-calculator-exec.php on line 74
And what if we put a zero in High risk test cases. Again: Warning: Division by zero in /srv/disk3/862004/www/testercrew.com/regression-calculator-exec.php on line 88. So clearly zeros, divisions and negatives are not taken into account. What about very large numbers. Say 1.0E+309 (okay I tried a few times) which returns a -INF number of regression test cases and a message “Value greater than number of high risk test cases” which seems to be a kind of “warning” that you are able to perform more test cases than there are high risk cases.
Hmm, the 15 minutes are over, and I would some have like to trigger a few more warnings…
Hi Jean-Paul,
can you come up with a condensed report on your findings?
With smaller numbers it’s easy to figure that number of regression testcases is numbers of testers * hours * 60 / avarage time. So just dividing all available minutes to available resources.
Test Suite Regression Coverage is calculated by dividing Number of Regression Test Cases through Number of Test Cases in Test Suite. A similar calculation for High risk cases.
There is no error for having more high risk cases than total testcases, there should be one.
The obvious diviside by zero errors have been reported allready…
There is the notification “Value greater than number of high risk test cases” which will override the notification “Value greater than number of high risk test cases”. I think the second one is far more important and should override the first. Wonder if the first is interesting at all. Since the notification are in red they feel like errors, green would be better (I f you have time to get 100.00% coverage it’s OK).
Negative inputs are allowed, an error would be nice.
Inputs with comma’s are not accepted, in many countries 0,1 IS a number.
For improvements I think I would like to have a seperate input for “Average Test Time per Test Case” for the high risk cases.
So far the report of my 15 minute testsession….. enjoyed it….