At EuroSTAR 2012 I took the opportunity to learn something about the new international software testing standard since I wouldn’t take the time to read through all this stuff, anyways.
Stuart Reid explained that he’s convinced that we need testing standards. Reid explained that there are few standards for software testing, and they merely cover parts. For example, take unit testing: there are two standards: british and IEEE, they conflict with each other. Reid claimed there is no standard for integration testing, acceptance testing, and so on.
The ISO 29119 standard consists of the processes module, the documentation module, and the testing techniques module. They are based upon the concepts and vocabulary part. Reid claimed that each individual part can stand on its own. The new ISO standard will also send the already existing IEEE standards to its retirement.
In the first part, concepts and vocabulary are covered. This includes scope, conformance, normative references, and definitions. There are also different software testing concepts, and testing in different lifecycle models. FInally, it covers roles and responsibilities.
Part 2 consists of the testing process. These are organizational test process, test management test processes, and dynamic test processes. Part 3 is on test documentation. Reid didn’t break it down to a clearer level for me, other than saying “Test Documentation”. I think it’s mostly consisting of IEEE 839 stuff. In part 4 the techniques are covered. Black box vs. white box testing, test coverage measurements, and some examples for applying the techniques, and testing of quality characteristics (or non-functional requirements). The annex part also gives hints on the election of particular techniques and how to assess the effectiveness of different techniques.
What stroke me most about the things that Reid described was the level of treatment of the standards group. Getting such a standard seems to be a touch job. More over, things become political. I couldn’t count the number of times Reid mentioned blacklisting other people in the process of coming up with this particular standard. Maybe it’s more my interpretation, but I noticed a lot of people interested more in getting the name and influence into this piece, rather than trying to help coming up with a working testing model.
One example that I would like to repeat to give a hint. In the second part, Reid mentioned that static testing is missing because one country’s votes voted for getting this thing out. Once they had taken out the static testing techniques, that particular country voted to get them back again. Puzzled, I wonder why anyone would ever make the effort to work in such a work(-avoidance?-)group.
Reid explained that the standard is meant to be tailored to the needs of the company. They cover Agile and traditional, Reid claimed. I wonder where the explanation for Agile testing projects came from, and what sort of evidence the working group took to put the Agile pieces in place. Overall the description seemed to be too generous to be useful for me. Considering the world of ScrumButs out there, I think the ISO 29119 assessment will end up in “whatever we would like to do”.
I was thinking about helping in one of these groups in the future. Stuart cured me of that wish.
But he did trigger me on the Agile part, I can’t believe such a rigid organisation could have come up with all the answers.
Did he explain *why* we need testing standards ? Will everyone have to follow them ? Will the small company I work for have to ? What happens if we dont ? Sounds like a colossal waste of time and effort :(
Stuart explained his motivation behind it, but it didn’t resonate with me. Reid also explains that the ISO standard is a sort of “fits all projects” pill, and that you can tailor it. I doubt this tailoring will happen, or will happen in a good and meaningful way. People will find ways to trick this system for bad.
The people who devise such standards are overwhelmingly from companies who will benefit from its introduction. That’s inevitable. People like Markus, Phil and me have no incentive to get involved. It would be a massive waste of time and money. The companies that want to see a new standard are those who will sell consultancy services to implement processes to fit the standard. They will then write contracts to test according to the standard, and will be able to charge more because the standard will require more effort, even if it is misplaced effort.
Big suppliers will claim that regulators and auditors will require compliance as a demonstration that testing is being professionally carried out. That is a highly contentious claim, but it will have some success. Lawyers will insist on compliance being written into contracts, in the naive belief that this will protect customers. Compliance with standards will then become a mandatory goal in its own right. For large numbers of testers the question of whether or not they should comply will never arise. It is something that they will simply have to do if they want to keep their jobs.
Michael Bolton provided an excellent critique in a comment on my blog a few months ago. I am reluctant to plug my blog on Markus’s site, but Michael’s comments deserve a far wider audience than my blog can offer it. http://clarotesting.wordpress.com/2012/06/06/iso-29119-the-new-software-testing-standard-what-about-it/
My interest has always been in working in small, agile and startup environments, hard enough to get any decent budget for testing having this kind of standard would just make it worse, I don’t see how this could fit in.
Nice forum thread on STC here too – in response to James Claro’s blog post – http://www.softwaretestingclub.com/forum/topics/iso-29119-the-new-international-software-testing-standard-what
If I understood Stuart Reid’s answers correctly yesterday, all the parts are optional, and don’t necessarily introduce more overhead otherwise you choose to do so.
I am not so sure that this will be what people actually do.
Testers who create and promote testing standards usually say that they are optional. I don’t think that that is an acceptable defence. They want the prestige and status of standards, but claim they want the flexibility of optional guidelines.
What do they really want? Well, they want the best of both worlds. I don’t think that’s a credible position.
Stuart Reid may say that the individual elements of ISO 29119 are optional. In that case it should not be created as an ISO standard. Calling it a standard means lawyers, naive customers and cynical suppliers will insist on full compliance with the standard, “because that’s the standard”. The outside world sees the option in ISO standards as being the option between professionalism and irresponsibility, the option between producing a shoddy product and one that is fit for purpose. I’m afraid that the creators of ISO 29119 are promoting such false choices.