Exploratory Testing is a useful approach. Personally I found a model from my past, which explains Exploratory Testing based on face-detection systems. There are three different posts in this series: Learning, Scanning and the Differences. This third entry in the series copes with the differences between face-detection and testing.

Introduction

Thus far I have shown the parallels I see between face-detection algorithms and Exploratory Testing. One could conclude that it is possible to automate Exploratory Testing approaches, since face-detection can also be learned by a computer. This would be a wrong conclusion. First of all in face-detection the pattern space is already known. Second, face-detection is a single detection process of one focused portion of images. Third, face-detection needs supervision, too. I will elaborate these points a bit and give a conclusion in the end.

Known pattern space

Indeed, for the face-detection algorithm we had annotated data as input. We knew the pattern space to elaborate when hitting the start button of our training process. When the detector was trained, we needed to include several heuristics in order to make the face-detection more robust. All this together yields a rather weak situation for face-detectors.

For Exploratory Testing we don’t know the pattern space most of the time. We know some variables like the programming language used, the code complexity maybe. But most of the complex pattern space is unknown. Therefore a human is needed all the time to adapt to the situation at hand. In fact recently I learned that this is The Edge of Chaos like in The Wave Rider. Humans can adapt to overly complex situations well enough to go on with it. They apply some of their heuristics and each failure is a learning opportunity for them.

While the face-detection was based on offline learning – that is the detector was trained without real data flowing in from some wired camera – humans learn online most of the time. They have models based on their past. Human extend their available mental models and heuristics based on failure and adapt therefore to the situation at hand. I have not seen a system of hardware and some software doing this that well, yet. Maybe this will be possible in the future, but our approach used four to five years ago in university did not do that.

Focused application base

Face-detection is a subset of detection a variety of things in images. In fact the complexity of human faces with different colors and shapes was handled rather well. During my diploma thesis I tried to port the behavior to dynamic human hand gestures. While detecting was possible, we needed more context information about possible hand positions before trying to detect anything. We used information from a human body-model tracker in order to reduce the search space in the images for that. The approach was rather slow by that time, but it was possible. Similarly we had some research on detection of office materials like keyboards, a telephone, and sticky tape based on a head mounted camera. While the approach worked for these simpler items, it showed us some of the limitations also. The heuristics for face-detection did not apply to the changed scenario and we had to adapt the detection phase.

During Exploratory Testing we don’t know which type of software bugs will hit us. We start to explore interesting dimensions of the overall bug space, adapting as we go to more interesting fields for the users of the product. In Exploratory Testing we follow our mission – that might be finding critical bugs quickly. We try to elaborate the usefulness of our product for our customer. We solve the problem that our customer needs information about the product in order to decide whether or not to ship it. But the dynamic of the applicability might change over the course of our project. Still, we are able to deal with the product and test it thoroughly. Therefore the skill of the individual tester in the team is crucial for testing the application. Face-detection requires some previous learned techniques elaborated over and over again on some incoming images; Exploratory Testing requires adaptation to the circumstances as they appear and reveal themselves.

Supervision

For the training of our face-detection algorithm we had annotated data. That meant we needed to mark faces in several thousands of images and choosing other images which clearly contained no faces at all. That is a daunting task to do. Basically you need to bootstrap your face-detection, maybe overtime you can use some detected faces to make it more thorough, but you may also over-train it, thus resulting in a useless detector.

For Exploratory Testing we don’t need supervision. We need a skilled tester with a good model of the product and some heuristics she can apply to the situation at hand in the testing sessions. If you were able to tell where the bugs lurk in your software, you would not need to test it at all. You could simply remove the bugs where they are. Why evaluate something that you already know? That’s useless, that’s waste.

For the face-detection we had another dynamic applied. Sometimes we trained the detector too good, so that just exactly the same shapes were recognized, but no more. That meant that the detector was useless, since it could detect my face, but not yours. Or my face in some particular lightning conditions, but not outside in natural daylight. The detector became by that time inattentionally blind. When dealing with software you still want to recognize that performance bug lurking in it when going live with the first million real data to hit it.

Conclusion

In the end the differences boil down to applying the right heuristics at the right time – that is sapient learning ability – and skill – that is systems thinking and mental models. Humans have some unique ways to learn things, to associate the known with the unknown and to continuously adapt and adopt new things in their actions. These are abilities you can learn or teach yourself, but thus far I haven’t seen a machine doing this. Some may claim that this will never be the case. Since I’m not from Nostradamus’ family, I will leave this up for time to tell me.