James Bach encouraged me to think more thoroughly about the ideas that I came up with. In general for most ideas I try to give credits to the people who made me come up with it. While thinking over it, I didn’t know whom to contribute the idea of configuration unit tests, and I think I took more than idea to generate this one. So, let me introduce you to the context in which unit tests for product configuration makes sense, and the approach we took on it.

The Context

My company builds a highly-configurable system for mobile-phone providers. Due to the tariff jungle in this business domain, there is a clear need for this. Most of the tariff configuration is put together in a larger xml-file, called the ProductCatalog. At a recent customer this one currently expands to 18 megabytes of xml data in a single file. This products catalog consists of information about mobile phone usages, how much a certain call may cost at the also included timebands, and what the price sms and mms notifications to other mobiles may cost for a given tariff. Additionally there are bonus rules, discounts and one-time charges, i.e. for losing your card.

Internally we got a product development department and a system integration department – in which I’m working. The product development teams put together the overall system, which then is utilized from the system integration to put in the particular business needs from the customer for which we prepare the system. There is a development and a test team busy with the product releases, and there are development and test teams busy with the system integration part of it. The work around the system integration mainly focuses on generating the mentioned product catalog for the system delivered by the product development department.

The Problem

Initially we were mostly dealing with tests in the full system environment. Starting up an application that shall serve a mobile phone provider with millions of transactions per day, costs some resources in regard to min memory and CPU consumption. Additionally building up a system and distributing the 18 megabytes product catalog to it takes roughly 15 to 30 minutes in our test environments.

Execution of fully automated end-to-end tests in that system is fragile. Since we deal with many asynchronous calls between subsystem, there are active waiting periods, and timeouts, which sometimes does not work that well. Considering our most recent customer, we also faced the problem, that a full regression test suite with all tariffs and all combinations of possible call origins and destinations took roughly much more than 24 hours, so it was practically never run. Doing the same manually would take even more time.

The Solution

Considering our department layout, early on I had the idea to test just the product catalog. Since there are tests on smaller elements (which are of the size of most applications on my desktop), as well as a full system test in the product development department, I felt that we should start to rely on them to the degree that we’re just checking the product catalog based on an analysis of the xml-file. Mainly the produced output from the system integration development team was just a large xml-file which contained mostly the tariff prices and discounts, we could take the file and check the prices of the tariff combinations.

So, what we started to build is a unit test framework for these type of test. I blogged about the initial approach in August 2009. You can find the first steps of a unit test framework for xml here. In short the approach uses stylesheets which are applied on the product catalog xml-file. In the current implementation we use a second xml-file which contains the test data, thus resulting in a data-driven test. The underlying structures include xslt-functions like assertEquals and fail, which reflect what the xUnit equivalents are doing. Applying the stylesheet with the test data xml-file to the product catalog xml-file results in a xUnit-like test result, which is included in our continuous integration reports. That’s it.

So, why do I call this configuration unit tests? Well, the full system tests are obviously not unit tests, they’re integration, end-to-end or system tests – depending on the particular terms used in your environment. Since the major output from the development is the product catalog, testing just tiny units inside the product catalog is a unit tests, just like testing a small function in your application would be test of the smaller unit of the overall application.

The Gaps

Sure, there are gaps. Since we rely to a large degree on the abilities of our product development team, we may face the situation that this reliance is wrong. Therefore we still need some end-to-end tests. But since we tested the majority of the configuration using the unit tests, we just need some to see the whole chain functioning properly. Since the unit tests run much faster on the xml configuration than the end-to-end tests would, we have an overall execution time benefit. I don’t have numbers on this, since we didn’t exercise everything in the end-to-end tests (since it’s not possible most of the time).

So, the gap is basically what Jerry Weinberg calls “The Decomposition Fallacy” in Perfect Software.. and other illusions about testing. Testing everything in decompositions may result in a problem when putting everything together. So, in order to cope with it, we still need to start-up our system and exercise the end-to-end tests, but we may skip a big deal of them, since we already checked the prices on a finer level. The focus for the tests simply switched.

The Future

We are currently dealing with just a few of these configuration unit tests, but we’re preparing to expand this. There are some more configuration files despite the product catalog file, which may be unit tests for themselves. For one part we used a small simulator written in some programming language that is also feed with test data, loads the production configuration, and delivers some output, which is then compared. It’s a straight-forward combination of a simulator with a data-driven xUnit approach. If there’s some interest to it, I would be glad to deal with it in a follow-up blog entry.

Thanks to James Bach for making me realize to write something about this.