Omnichannel commerce | Marketing
The ABCs of A/B Testing
November 10, 2015
A/B testing is elegant in its simplicity. Even a smaller organisation can conduct—and just as important, analyse the results of—A/B tests of web pages, emails, and direct mail pieces.
But even simple tactics and techniques have rules and best practises to ensure that you achieve optimal results. For A/B testing, these are:
Start with a hypothesis.
In other words, what specific aspect do you want to test? Specific is key. You want to home in on one element. Don't test two completely radical landing page designs; instead test one aspect of the landing page, such as the font or type size of your copy, the color of your "shop" button, the addition of a second photo.
What you're testing also needs to be replicable. Say you want to test a "more creative" subject line for your weekly enewsletter. Calling out an especially enticing offer for your test won't help you down the line if you don't plan on offering other equally sexy offers.
You should be able to sum up your hypothesis in one simple sentence: "We hypothesise that changing the color of our 'shop' button will lift response"; "We contend that reducing the character count of our subject lines to fewer than 40 characters will increase open rates"; "We think changing the copy of our direct mailer from third person to second will boost response."
Test only one variable.
We mentioned this above, but it bears emphasising: Other than the one aspect you're testing, your test version and your control should be exactly the same. If you're testing a subject line, for instance, both email sends need to be launched at the same time, with the exact same content in the body of the message.
Test both versions simultaneously.
Serving up a test page to all visitors to your website one week and measuring its performance against that of the control the following week will not provide accurate results. There may have been outside factors beyond your control that affected response: the weather in one part of the country, a major event or news story in another part.
Make sure your results are statistically significant.
Statistical significance ensures that your results are reliable and worth basing decisions on, rather than a result of coincidence or chance. Much of it depends on your response rate compared with your sample size and the size of your overall universe. EasyCalculation.com offers a formula, but if you're not a math whiz you might not find the calculation easy at all. Fortunately it also offers a free online calculator, as do other websites.
To achieve statistical confidence in your results, you may need to run your test longer than you'd anticipated. If on day one results are overwhelmingly in favor of one version, don't shut the test down early unless you have obtained sufficient response to make the results statistically significant. Free online calculators can also suggest how long you need to run the test.
Select your test and control groups randomly.
Sort your audiences randomly—for instance, on an nth-name basis. You want to rule out the possibility of results being skewed because one group was from the north while the other was from the south, or because one group was primarily long-time customers and the other newbies, or because one found its way to your business via social media and the other via paid advertising.
Don't limit yourself to a straight 50/50 test.
In other words, you can send your test email to only 20% of your audience, rather than 50%, if you're so inclined. It may mean you have to run the test longer than if you performed a 50/50 split, but it also reduces risk if the test turns out to be a flop.