You are measuring traffic, right? Number of hits. Unique users. Good, just checking.
Do you measure the impact of each feature? Metro don’t (watch this space). Here’s an example.
We recently redesigned our masthead – the header, logo, background, navigation, search tool and banner space. It took us quite a while.
Some people thought this was a big step forward. Some people thought it was a big mistake. You can compare the before and after here.
So who was right?
We don’t know. Seriously, we are completely in the dark.
Overall, traffic has gone up. Maybe this is due to the masthead. Maybe traffic would have gone up more without the new masthead. There are many factors: The News of the World shutting down, other changes to our site, what popular shows are on TV and even the weather.
We are in the dark, but thankfully there is a light switch. It’s called AB testing. Google have a free, easy-to-use service for it. Thanks Google!
AB testing splits users into two random groups. One group gets the new feature, one group doesn’t. You can compare the behavior of the two groups to see what impact the feature has. It’s a controlled experiment.
AB testing can give some surprising results. Here is an example from the killer book Super Crunchers (chapter 2). The credit card company Capital One used AB testing on their mailouts for loans. They found that putting a picture of a smiling woman on the mailout had as big an impact as reducing the interest rate on the loan by 4.5%.
Discovering AB testing is like when the cops in The Wire discover phone taps.
AB testing has some caveats. Firstly, you need enough traffic for the randomness to even out (no problem for Metro). Secondly, the approach isn’t very mature for mobile apps.
And it’s actually called multivariant testing but AB sounds cool so i’ll stick with that for now.
AB testing gives product owners the empirical data they need to make decisions. Empirical data based on controlled experiments. Do not underestimate the power of this. It is the foundation of science. It has made possible our technological wonderland.
The alternative is to base product decisions on assumptions, and you know what they say about assumptions 🙂