There is a good way to design unit tests

Software Design Tip: Properties, then Tests

In the 7 Mistakes that Cause Fragile Code, I wrote "Test what the code is supposed to do, not what it happens to do."

Today, I'd like to elaborate on this with an answer: how do you decide what unit tests to write?

Consider writing unit tests for a shopping cart with a lot of complicated pricing rules. Perhaps you write a test that, if the customer adds two items, enters a code for a special two-for-one deal, and then removes one item, the discount goes away. Perhaps you write a test that, if the customer enters a discount code that gives him 20% off on all electronics, and then adds several laptops, the discount applies to the new items. Perhaps you test that, if the customer adds some items, and then changes shipping address...

At one point do you stop?

The answer can only come from stepping away from the code and scenarios, and asking "What does it actually mean for this shopping cart to be correct?"

Correctness can be broken down into several properties. For instance, all these scenarios tested actually fall under one property: arbitrary sequences of add/remove/apply code actions should all produce the same result as simply adding all the items then applying the discount codes. Another property might be that, if an item is added to the cart, then that item is in the cart. (Sounds trivial? It may actually be too strong — what if the cart already has 2³¹ items?)

Once you have these properties, you have several ways of checking that they're true. The classic way, shown above, is to come up with a set of scenarios that exercise all cases, reducing this to the problem of how to determine which sequences of actions may produce "distinct" behavior (this is another topic). Another is testing it on random sequences of actions, as the RNG may be more creative than you. Or you may choose a third way of testing it, one familiar but often underappreciated: just manually inspecting the code.

Of equal importance, doing this also tells you what not to test. Does your property say that, if the customer orders more than 10 of something, the volume discount should be applied? Then you shouldn't be testing that the discount is 10%.

When you've done this for all properties, you are done. You now know exactly what every unit test is doing, and how they combine into an assurance that the shopping cart is correct.

I used this approach to guide my testing at Apptimize. Every new feature assigned became a game of trying to figure out its correctness properties. This list became my guide for writing tests. And everything I didn't automate became a handy reference for QA. Code robustness and test stability both improved. I convinced the rest of my team that this was the way to do testing, and it became enshrined as process.

The new process then failed, because breaking down correctness into properties is a new skill for most programmers. I had convinced them, but I hadn't trained them.

And that's why today I'm teaching you.