Matt Heusser blogged today on generating new ideas. In his entry, he cites from Weinberg’s Becoming a Technical Leader. Weinberg lists up the three obstacles to innovation – one of the three parts of his leadership model. The three obstacles to innovation are self-blindness, the No-Problem syndrome, and the single-solution belief. Continuing he explains the three keys to innovation based on this: Learn from your errors, learn from others, and copulate together two great ideas to form an even greater idea. Matt also mentions my struggle to find an article where a great software test automation framework became shelfware just because it couldn’t stand the technical challenge – or maybe just the human aspect of it? So far, we just found opinions on software test automation projects having failed, but no real hard data.

So, to overcome this lack of available material, let me write about my experience with software test automation, the problems we had, and how we overcame them. Thereby, I will not only enable myself to learn from my own errors, but also provide a system you may copy yourself, or which you might combine with another great idea that you have established in your company. In fact, most of this shouldn’t surprise you. During the last year, I also presented our approach on a test conference – here‘s the paper I wrote on this.

The Second System Effect

Frederick Brooks defines the Second System Effect in his popular book The Mythical Man-Month. Let me quote from page 55 of my copy:

An architect’s first work is apt to be spare and clean. He knows he doesn’t know what he’s doing, so he does it carefully and with great restraint.
[…]
This second is the most dangerous system a man ever designs.
[…]
The general tendency is to over-design the second system, using all the ideas and frills that were cautiously sidetracked on the first one.

I think Brooks’ words are very clear. But what has all of this to do with software test automation? Well, on my first project, we were using heavily shell scripts in order to execute functions, that would execute the system under test, and verify outputs and the like. Just after two years and a major struggle with this approach, I had found out about our problem: The Second System Effect in Software Test Automation.

Elegantly our system was started by our test scripts, whenever an account was put into the system, it was noticed down in a flat text file, so that later test scripts could read the information about the available account. We also noted down anything about the account in regards to different subsystems, which we simulated in our test automation harness. Do you see the second system? Yes, it’s just right there. We had build the application using shell scripts. This was our second system. Of course, the point where we couldn’t keep up with our developers, where the ones where they had clear technological advantages. So, this set us out to the major struggle. After two years, we were faced with the first customer, and at that time we were asked to test composed services. While this is an easy thing to do in most programming languages, it became a mess for our shell script system.

So, to define the Second System Effect in Software Test Automation, it’s the effect you get, when you build a second system with your test automation mechanisms. You may not immediately notice that you did, so you should better keep your light on.

Smells

Here are some smells you should keep an eye on. Does a change in the development code cost you a lot in your automation code? Well, maybe then you could have a second system. Does a change in the business logic affect several places of your automation code? Doesn’t smell too good. Do changes in unrelated parts of your application affect the outcome of your tests? Do you have high test maintenance costs? Do you have fragile and brittle tests? All these symptoms might point you to the second system effect.

The cure

But what can you do to avoid the Second System Effect? First of all, treat software test automation as a software development activity – and an important one. You will likely need software developers for this work. Bring testers and test automation developers closely together, so they agree easily on possible future changes. But it’s also vital to bring the application developers, the testers and the automation code developers close together. On an Agile team you will most likely find developers working out the automation code themselves. The closer you can get all these parties to collaborate, the less-likely it becomes that your automation code will suffer from the Second System Effect. Focus on the business intents of the users and customers, and avoid building the same system twice. Otherwise you will get twice the maintenance work.