All posts by Markus Gärtner

Example-driven School of Testing – Updates

After having a discussion on the Example-driven School of Testing with Michael Bolton, I realised that I missed some points. Being a human I truly believe that I am allowed to miss a point once in a while, the critical point is to realise this for yourself.

The first point Michael mentioned was the idea that whenever the acceptance tests pass, then we – the project team – are done. Indeed acceptance tests tell the project team that they’re not done when they don’t pass. This is a something different – check the Wason selection task for a explanation. (Just now I realise that I have given false kudos to James Bach for this quote. Sorry, Michael.) My reply was to view acceptance tests as a goal in terms of S.M.A.R.T.: Specific, Measurable, Attainable, Relevant and Time-bound. You can measure when you might have something to deliver, i.e. when you’re agreed examples from your specification workshops pass. You can measure when you might be ready for bringing the software to the customer and it’s specific and should be – of course – business-relevant. A friend of Michael Bolton put this this way:

When the acceptance tests pass, then you’re ready to give it to a real tester to kick the snot out of it.

This led my thought to a post from February this year from me: Testing Symbiosis. The main motivation behind this post was a discussion on the Software Testing mailing list about Agile and Context-driven testing. The plot outline of the story was Cem Kaner’s reply on that list on a article from Lisa Crispin and Janet Gregory leading to Janet leaving the list. Enough history.

The outline of Testing Symbiosis is that Exploratory Testing and Test Automation combine each other and rely on each other. Just automated testing can lead to problems on your user interface or on the user experiences of your product while just testing exploratory may lead to serious regression bugs. The wise combination of the two can lead to a well tested software product whose quality level is perceived high when shipped to your customer. Here is Michael Bolton’s summary after reading Testing symbiosis:

That said, I think that the role of the tester as an automator of high-level regression/acceptance tests has been oversold. I’m hearing more and more people agree with that. I think your approach is most sane: “On the other hand you can more easily automate software testing, if the software is built for automation support. This means low coupling of classes, high cohesion, easy to realize dependency injections and an entry point behind the GUI. Of course by then you will have to test the GUI manually, but you won’t need to exercise every test through the slow GUI with the complete database as backend. There are ways to speed yourself up.”

1) Automation support.
2) Entry points behind the GUI.
3) Human interaction with the GUI, more automation below it.

These two points seemed to need some clarifications. Feel free to remind me on the left-out corners that are still missing.

Example-driven School of Testing

Some years ago Bret Pettichord defined four schools of software testing: The Analytic School, the Standard School, the Quality School and the Context-Driven School. These ideas were incorporated into the book Lessons Learned in Software Testing: A Context-Driven Approach by James Bach, Cem Kaner and Bret Pettichord. Later on Bret included the Agile School of testing. Some days ago I realized that the name for the Agile School of testing is rather poor. This is the hypothesis and I would like to propose a new name based on the insights of the last few years for the thing Bret called the Agile school of testing. Read up on this in the extended body. Bret’s initial hypothesis was based on Agile Software development being mostly about test-driven development with massive usage of automated regression tests while the code is written. That’s why he included the following core beliefs of the Agile School of Testing:

  • Software is an ongoing conversation
  • Testing tells us that a development story is complete
  • Tests must be automated
  • Key Question: Is the story done?

Some time later Brian Marick wrote down what already was in the heads of many people: Tests in the Agile world are based on examples. Additionally Brian raises the point to renamed test-driven development to example-driven development, since this reflects testing in the agile context more appropriately.

A bunch of techniques ending in ‘dd’ for ‘-driven developenment’ appeared – mainly inspired by the Agile School that started with test-driven development. Among these are Acceptance Test-driven developement, behaviour driven-development or Domain Driven Design (yeah, right, this one does not end in ‘development’).

Back in February I was introduced to the Specification By Example by Gojko Adzic, who transferred the idea of Agile Acceptance Testing based on examples to a process of specification. As pointed out in one of his presentations on FIT and Agile Acceptance Testing, examples elaborate requirements or specifications. On the other hand examples can also become tests – and this is basically what the Agile School of Testing teaches us. Testing is based on noting down examples in order to test that you’re done with developing the feature in your current iteration.

Based on this I propose to rename the Agile School of Software Testing to the Example-driven School of Software Testing. On the other hand this would also make the statement clear that it’s not just about Agile, but rather about examples as Brian Marick initially pointed out. Another benefit of this term would be the distinction towards the Context-Driven School. I raise the point that we must truely understand that there is no either Example-based or Context-Driven, these two can be adapted together or one of them alone of neither of them. From my point of view these two school are able to co-exist and complete each other when applied together.

Practicing relevance

On the Software Craftsmanship conference in London End of February Micah Martin presented a session on where he presented an implementation of Langston’s Ant in Ruby. As an introduction he raised the point how important practice for a software craftsman is. He used his background in Martial Arts as a motivator where Katas are used to fit into uncomfortable positions in order to gain flexibility for the fight with an opponent. By over and over practicing these fight-unrelated moves, the aspirant learns for the fight. (Actually I remember the first Karate Kid film, where something alike happens, when the kid needs to paint a fence etc.)

By that time I did not fully agree to Micah’s point on practicing. However I made a note on my personal digest by then to research something suitable for me. Since I don’t have a background in Martial Arts, but in Swimming competition and as a swimming trainer, I was looking for analogies like how I teach kids to learn swimming with some learning chains getting from the easy to the complex and from the well-known to the unknown. These ideas did not quite match the Kata analogy from Micah, since they were related to how people learn in their minds and the bridge towards software development seems to be too far away here.

During the last week I found a better suitable analogy however, which is partially motivated from my experiences in swimming. As a trainer I need to be able to save anyone from drowning. While this sounds rather reasonable since I get to train kids that are entrusted to me by their parents, during the latest courses to fresh-up my knowledge on water-rescue I realzed, that the training for the german DLRG groups – the german baywatch – does over and over practice how to rescue someone from drowning. While the one being rescued might attempt to reach your neck in panic to just save herself, the life-saver might end up needing to save herself as well. While there are some grasps to free oneself out of such situations, the DLRG practices these over and over in order to be able to use them naturally when getting in a panic situation – since every second counts be then.

Similarly during first-aid courses which I needed to take several times as well I was told to practice reanimation process and how to get an injured person into recovery position. When people get under stress they forget what they have learned, if they did not practice over and over. Similarly when a software project gets under stress, people may forget to apply test-driven development, refactoring and design patterns and introduce technical debt. On the other hand by practicing seemingly unrelated software Katas in order to improve your tdd skills might enable you to excel your work even in serious situations. As for first-aid and water-rescue practicing, knowing and maintaining software development practices, but not having to use them every day, makes myself feel safer and relaxed.

Apprenticeship

Obtiva and 8th Light are currently stressing out Software Craftsmanship in an apprenticeship program between the two. Jim Suchy currently writes down experiences from each day on their blog. The entries for the first three days can be found here, here and here. While taking out the links for this entry, I also ran over Colin’s Apprenticeship.

As it turns out I ran into the idea two times during the last few weeks. One of our business analyst from our product development department stated some weeks ago that he has been asking for taking some kind of apprenticeship in the System Integration department in order to get to know the product he is defining. While the technical view of the product is rather clear to him, he stated that he is unsure of the actual use cases during System Integration work, where we configure most of the system for the end customer.

Some weeks earlier I ran over the idea as a tester to work on-site for your end customer in Lisa Crispin’s and Janet Gregory’s book. From the viewpoint of a tester I would be able to follow the mission to deliver working software to the customer in a better way when I had some experience on how the product will be used in the day-to-day work. By attending call-center calls and participating while our product is used to sell product to the end-consumers of our customers, I would get a very good picture of the business flow and the motivation behind it. Exactly what Gojko Adzic addresses in Bridging the Communication Gap.

Today I sat down with a former group lead in order to identify what I shall present on FIT end of next week. While we were discussing things I stated that two of my colleagues will be on vacational leave for four weeks during summer this year, since they will become fathers. While discussing the bottleneck in which we will get by then, he made the suggestion to exchange one of his testers with my group in order to get deeper knowledge on how we use FIT as a testing framework. Personally I think of it as an apprenticeship program, that could lead in improved communications between our two departments. Hopefully something great is going to be built out of this idea – but I’ll try to motivate and follow-up on this idea.

Mapping practices for Personnel development

Influenced by Adewale Oshineyes session on Mapping Personal Practices, I lead a meeting on personnel development with one of my testers two weeks ago. Earlier this year I started to do some work on personnel development for my workers and we already had come up with four to six major categories worthwhile to consider as evolution points: Testing, Programming, Product Knowledge, Teamwork, Technical Leadership and Leaderhship with personal responsibility. Incluenced by this previous session, we came up with a mind map similarly structured.


The colleague we drew the map for is rather junior. In the map you can see a division to the major categories mentioned. Let me try to explain on how we came to the final map.

Housekeeping

First of all we build up the map with the personal practices of my colleague. Motivated by the flipchart page of our first session in January and Adewale’s session on mapping personal practices, we started off with the items my colleague already does to a reasonable degree and has already reached a good level in doing so. Our initial session was based on the Shu-Ha-Ri principle of Skillset development and we picked those practices where we agreed he had reached Ha-level already. For the Testing category this includes writing tests, bugtracking and bug verification and execution of regression tests on a regular basis. Related to our company’s products we identified knowledge for one of the major Sub-Products and one Component related to it that was built over the last year and my worker had tested heavily during this time period. Related to Teamwork we agreed on a great skill to support our developers and enabling team communications through direct contact and collaboration. I consider him to have huge strengths in these fields and respect him for this. He mastered a computer science degree with a minor subject in psychology, which of course helps in this field. Since he is rather on a junior level, there seemed to be no relevant practices for the programming and technical leadership category, so we left them empty.

Development

We then started to discuss areas of improvement. Reconsidering each category, we were able to identify one point for improvements. For the testing portion we agreed on getting more knowledge related to User Acceptance Tests. Currently our teams have big problems with this item and on the last few Lessons Learned workshops during this year we realized that our company as a whole needs to improve in this field. Relying on his communication skills this is a good point to combine two areas, basically. Related to programming we agreed on the possibility to go out and ask for help when dealing with problems while doing some Fixture programming for our test automation. Here again I was happy to include his communication skills into this area of improvement by identifying this. Related to the knowledge of our product, we choose to get more insights into the second major component our Sub-Product has to deal with. This will not only give him a better overview of the value we deliver to our customer, but also enable him to find out hidden assumptions in the plug-ins we test for the Sub-Product 1 and that indirectly talk to Sub-Product 2. (I know, it’s hard to explain, if you cannot mention more context.) In the teamwork area we agreed on two major points for improvements in the whole organization and I think that he can help us work on these items a lot. We called this one Synchronisation. This is related to getting developers and testers to a common understanding of the solution, to include testers from the very beginning of the project and to include them in discussions about the solution. On the other hand we have co-workers spread all over the planet: Germany, Poland, Romenia, Spain, Italy, Brazil, Turkey, Ukraine, Malaysia. A usual project consists of people working in at least two countries. Therefore distributed collaboration is a major topic in our company. Related to his very good communication skills he can help us improve here. For technical leadership we both agreed that there needs to be improvement in the technical knowledge before starting a discussion here.

Conclusion

After finishing the mindmap, I was very proud of the result. During the next few months I will have to track the achievements in these fields, so we have again a discussion basis for the review we agreed to take place in early June. There is no impact on salary of the like based on the outcome. I just realized during the last year that I need to take care of my peers for personnel development since no one else did. Using mind maps on this improved the discussion a lot, I think. And I was able to incorporate his best skill – communication – into the remaining areas to help the organization as a whole. One side note I would like to add: Most of the practices and feedback came from my colleague. I tried to facilitate and help him find out ways to improve himself without demanding them. All of the discussion was of course done in collaboration, where the biggest contribution came from my worker. I also asked him afterwards if I will be allowed to publish it online here after anonymizing it. (I had to translate the german terms to english and get rid of and company’s product names.)

The Telephone Translation Game

Elisabeth Hendrickson brought up the point, that BDUF (Big Design Up-front) is not comparable to the Telephone Game played by children. The more appropriate analogy is language translation, where you get Lost in translation. She describes the usual requirements gathering, writing a system concept, writing a design document, writing code phase to a translation process several languages:

The business person tells a story in BizSpeak to the Business Analyst who interprets it and retells it in BA-Speak (perhaps in terms of “Must,” “Should,” and “May” functional requirements) to the Systems Analyst who retells it in SysSpeak (perhaps UML Use Cases and/or Sequence Diagrams) to the Programmer who must translate it into TechSpeak (maybe State Models, Class Diagrams, and ERDs) and then code. The languages may have some commonalities, but they have a lot of differences too.

Though I was right getting off to bed, I needed to share this. Elisabeth enlightens the reader with a serious of sentence translations into different languages and then back to english ending up with a screwed up sentence. The key point is the fact to get everyone together in a project to share a common language on the software product your trying to build. Eric Evans called this the ubiquitous language and it is a key concept in Gojko Adzic’s book on bridging the communication gap.

What I’d go back and tell myself

Lisa Crispin motivated this morning an entry on What I’d go back and tell myself. Compared to Lisa I would like to get back a few years more towards my time in university that started in 1999 (yes, I’m not that old zealot). At the time around 2001 I took a course called Software practical that was divided in a theoretical leacture and a practical group work to build a system. During the lecture there was eXtreme Programming mentioned as a new technique with a delightning practice called Test-driven development. By that time I did not get the point and it took until 2008 to realize the benefits of test-driven development. Going back to 2001 or even 1999 I would tell myself that I really should read about test-driven development, refactoring, design patterns and unit tests and dive into more topics. Nearly two years from now I started to read on these things and get the ideas from the great books out there. Currently I feel the need to get even with the books, but am really overwhelmed so far from the pile of books that lies in front of me.

Another thing I would like to tell myself going back into the past, is that I will truly make a good start in software testing and development and see the opportunities to improve my work. Going back to 2005 I would tell myself, that I should not waste the time to investigate on PhD positions at the universities, but rather get some real hands-on in the business. Most of practices and techniques I use today I learned when I was out from university and I’m glad to have learned all this stuff.

Roofer Craftsmanship

Today a friend of mine returned his computer that I repaired. While we stood outside and talked a bit about this and that, he told me about his job and his employee. He is a roofer and working in a craft. The interesting thing started when he told me, that he does not expect his employee to be hired for building new houses. This made me a bit sceptic, so I asked for the reason. He told me, that architects from new house projects do tenders for all kind of crafts, likewise for roofers. His boss does not participate in these tenders, because he knows that his estimates will be too expensive for new houses so he will not get involved in the project. Those companies that get involved on the other hand are lead by bosses that do price dumping in order to get the contract. Often these companies have roofers that need work on four new house projects during one week he said. It was really clear to him, that the quality of these roofers’ work suffers from these unrealistic assignments. He used the german word “Pfusch” to describe the level of quality of this work. In one or two years this roof will lead to mildew in the walls and all the like. Since new house projects don’t save money on the roofers but also on the other craftspersons, the result is likely to turn out awkward after some years. For a comparison he said that his company is happy with one roof over a week. He also mentioned, that his boss every now and then tries to do the same – push the people to make their job faster. They then ask the boss what does he expect from their work? Should they use just one instead of the usual five nails for each root tile knowing that their work will be of less quality? This is the point where the boss gets the insight, that it does not make sense to push for work being finished faster.

How does this relate to Software Craftsmanship? Take a step back and read the Manifesto for Software Craftsmanship now. There are four statements in it, similarly to the Agile Manifesto. Let me re-state these and raise my points here.

Not only working software, but also well-crafted software.

My friend, the roofer, knows that it does not make sense to deliver a roof that is working today, but is going to fail in one or two year. That is why he chooses to do his craft – roofing – properly and deliver his best result to the happy customer. Sure, it will be more expensive, but for good. Likewise delivering software that works today, may result in software that does not fulfill the quality needs of tomorrow. James Bach currently has a series on his blog, where he describes the quality creation myth in perfection. (Make sure to read the comments as well.) Quality is not built into our software, it is a time-effective reception of the product we built. This means it may be of good quality today, but might be of bad tomorrow if it fails to realize the new quality standards by then.

Not only responding to change, but also steadily adding value

My friend, the roofer, knows that it does not make sense to deliver a roof that is poured in iron and costs a lot to change its color. Who is going to buy these kind of roofs anyways? Supermarkets with a flat root maybe, but not a usual housekeeper. Likewise in software we must not think of requirements, design, code, tests as being carved in stone as well. We need to steadily deliver something that makes our customers happy. If I want to change my roof today, I can call a roofer and he does just this for me. Surely, this might take some time, according to his open contracts, the progress he is able to make, the wheather, but that roofer should know what I would like to get and that I might change my mind in two or three years. In most software projects I was involved in so far most people didn’t know what they did two hours from now. Without having an impressive set of unit tests they will even not get back into that knowledge quickly anymore.

Not only individuals and interactions, but also a community of professionals

My friend, the roofer, knows that he needs to collaborate with the community of roofers. He pretty much knows that it will pay off to visit a course in Berlin starting from summer to make his journeyman degree together with other community professionals. He knows that there is something he can learn from these other girls and guys, too. Likewise Coding Dojos, conferences and exchange in person or over the internet helps to build that community and exchange thoughts on different techniques. My friend the roofer knows that he has the opportunity to learn something for his whole life. I know this as well.

Not only customer collaboration, but also productive partnerships

My frined, the roofer, knows that by delivering a worthwhile roof today will get his boss into the position that today’s customer recommends his work to other stakeholders. That is why he chooses to push for the best job he can do today, even if this means to stretch the projects deadline. Maybe the recommendations from today’s customer will lead to new contracts for the next ten years. Likewise by producing the best software you can today (for some hints watch Robert Martin’s talk on Craftsmanship and Ethics), you may bring in contracts of tomorrow to your company. This may as well mean that you have to make a more expensive offering in terms of time or effort. But this time and effort pays off.

Skill

There is one more thing the roofer knows, but it is not mentioned in the manifesto explicitly. The roofer knows several techniques how to build a roof. There is one way or the other. He has a skillset of techniques that he can bring into each new project in order to help the customer get the roof they need. That is craft. Likewise a software craftsperson should have a skillset handy for each different project she might be involved in – and even those skills needed in other projects as well. I can’t tell you that you will never need Mock Objects, maybe you will, maybe you won’t. But knowing Mock Objects if you need them, is better than the other way round. This is the condome principle: Better to have one and don’t need it as to need one and don’t have it.

Trade-Offs

While reading the excellent xUnit Test Patterns book I was introduced to the idea that making something less private on a class in order to get a test for it in place may break encapsulation for good. While this pattern ensures that you just have a single instance of a configuration or memory consuming object in your production system, you will have a problem with unit testing it. The xUnit book provides a pattern to make this easy. However, the Substituted Singleton pattern relies on the fact that you don’t define the singleton instance on your original class as private, but rather as protected, so you can override the instance by a subclass where you have influencing methods so you can influence your test behaviour directly. This breaks the encapsulation of the class you wrote for production use, but for good, since you can now test all your users of that singleton class rather than needing to do some special treatments of your environment in order to achieve the same. A similar reasoning can be applied for not using the final modifier on classes. These ones you will not be able to test by then, too. When deciding to do just this, you’re trading off the ability to unit test your code and show that you produced a good quality while doing so and communicating your intents for the next programmer having to deal with your class by breaking encapsulation in your class hierarchy. The first overweighs the second here.

Currently I am faced with a similar situation at work in a different context. We use Perforce for version control at work. This tool is able to indirectly track renamed files not directly – you have to do an integrate to produce a copy with a new name and delete the old file afterwards for that. The current eclipse plugin for Perforce does not cope with this. It just deletes the old file after having created the new file and opened that one for add. By submitting this change you loose the version history and cannot see anymore that the previous name of a class was Foo and is now Bar or that initially you had put up the class in the package foo.bar and now is placed in foo.baz. One of my colleagues realized this and now mandates that we should do a semi-manual refactoring for these kind of operations (rename class, move class). Personally for me the same trade-off as for encapsulation of fields in singletons exists here. Having tool support for doing these refactorings completely automatically is a big win and I can live with the fact that I will not obviously see where the file was coming from in the version history. The class was renamed to hopefully some good reason – i.e. to communicate intent in a better way or to have it placed under a package, where it is reasonable. On a side note all the history I would need should be placed in the unit tests for the classes. This trade-off is fine for me and I don’t see good reasons to get into the semi-automated process of manually integrating a file in the version control system, just to see the complete history of the file. It’s fine if I just see, “Oh I added that file and deleted that one in the first change to that file, seems like a renaming operation.” and follow-up on that file if needed (I don’t read the revision history often – most of the time I don’t have to.)