Over the course of the past year, I had the opportunity to work with some great trainers. I learned a lot from them, and by delivering co-trainings together. Today, I decided it was time to reflect on some of that stuff. Blog entries work great for me to do so. Here is a first blog entry in a series of entries to come.

Stick long enough into testing, and you will face an argument pro or contra traditional test cases. Most of us have been there. Most of us know what worked for them in the past. Most of us won’t agree with each other. During a particular co-training, I became aware and reminded again about process control, and realized why I think exploratory testing is better suited in most software development shops around. Let’s see what process control consists of, and check in which of the models testing falls, and where exploratory testing can help you.

What is process control?

Process control comes in two approaches: defined, or empirical. What’s the difference? A process can be thought of as three variables that are relevant for controlling it:

1. Input
2. Process (as a black-box)
3. Output

This should sound familiar to you if you have been involved in software testing. This probably also rings a bell if you have been developing software for some time.

The defined approach to process control is suitable if we can control the input to our process, and the process is reliable, i.e. producing the same outputs if you happen to feed in the same inputs. In that approach we are able to determine which inputs to deliver to our process, and which outputs we can expect from it.

The empirical approach to process control is suitable if the outputs we observe are unreliable, and seem to be rather random. I would also claim that we don’t know enough the process and how it produces outputs so that we can come up with a reliable model about the process. In that case, we need (very short) feedback loops, and either vary the input to our black-box, or change the process a tiny bit, and see what happens with the outputs then. This principle is also spread as “Inspect & Adapt”. We adapt something, one thing, we inspect the results, and learn from it.

Test cases vs. exploratory testing

In order to compare the two process control approaches with testing, let’s take a look into what traditional test cases are, and what exploratory testing is.

To paraphrase Wikipedia, a formal test cases should consist of pre-conditions, and post-conditions after triggering a particular thing in the system under test. For example, if I enter 2, plus, and 2 into a calculator, and hit the equal sign, the post-condition should be to display 4. The pre-conditions in the formal definition relate to the inputs in process control, the post-conditions to the outputs, and the operation that we trigger relates to the black-box, the process in process control terms.

On to exploratory testing:

Exploratory is an approach to software testing that is concisely described as simultaneous learning, test design and test execution.

Let’s inspect the elements. During test execution, we feed the inputs to our system under test, and execute whatever operation necessary. We then observe the outputs of the system, and match them with the expected outputs. During test design we come up with questions to ask the system that will guide our learning about the system, and derive inputs to the system based upon our current understanding, and come up with expected outcomes. Finally, after observing the system, and its outputs we learn more about the system, our internal model of the system, and other questions we might want to ask the system. In short, we vary either the inputs to the system, or our internal understanding of the system, observe the outputs after feeding the system with the input, and then adapt accordingly.

Now what?

At this point, I realized that scripted test cases relate to the defined approach of process control, while exploratory testing relates to the empirical approach. But what does that mean in my daily work? When should I go with either of these?

What I learned from process control is the following: the defined approach is suitable for stable, reliable processes; the empirical approach is suitable for “out of control” processes. Where does software testing fall into? Oh, well, that depends on a lot of factors.

To make a long story short, if your programmers are producing reliable results, and you know where to look, and you have the influence about the project’s constraints, like time, budget, scope, and the competition, then software testing becomes a pretty reliable process, and a defined approach is probably more efficient than an empirical approach with all its overhead of non-necessary feedback loops. On the other hand, if your programmers produce unreliable, or “out of control” results, you work towards arbitrary deadlines, overtime, with unskilled testers, then an empirical process with feedback loops probably is better suited for you. (I am exaggerating a bit.)

Consider this scenario: You receive a new version to test of the software. You fire up your testing machine, run the first 20 tests, and find 100 bugs. In a defined approach you would continue with the remaining 480 test cases, and might find another 2400 bugs by doing so. When reporting that to whoever you are testing for, I can understand that stakeholder of your testing to be disappointed that it took you three days to deliver the message that the software is quite unusable. In an empirical approach you would look for a feedback loop after a reasonable time, i.e. 2 hours, that tells the disappointing news to the stakeholder of your testing, so that both of you can adapt to that.

In another scenario, if you fire up that new software version, and don’t find any problems while running 500 test cases, and that’s how you have worked for years, the burden of the extra feedback loops to the stakeholder of your testing every two hours will become a drag for you.

In other news, I think the empirical approach to process control is the reason why session-based test management works so well in many contexts. It helps you ensure the necessary feedback loops to adapt the process of that seems to be “out of control”. The debriefing activity together with short timeframes of uninterrupted testing, and test-related learning going on helps you to inspect and adapt the progress early, and often, and notice when things go awry. Using a defined approach in such circumstances does not ensure that you receive the necessary feedback, i.e. a junior tester that does not (and can not) realize that he has become stuck at whatever he is doing will be asking for help too late.

So, think about the context in which you are working, and look for opportunities where the defined or the empirical approach to controlling your software with tests will help you.