A/B Testing: Statistical Significance or Revenue for Christmas!

by Hannah Johnston RedEye

For many retailers Christmas will make or break their year. Andrew Furlong, Sales Director at RedEye, advises adjusting your strategy by maximising optimisation with a revenue focus

So last night my wife and I sat down to plan our Christmas shopping, we have two kids and the Christmas list basically looks something like this:

As much as I can see the virtues of having an odd Excel obsession (it would make my life easier in many ways) I tend to be more spontaneous. So it’s quite surprising that here I am treating Christmas like a budget planning session.

Whether you realise it or not we all go through a significant shift in behaviour around the Christmas period. This is simply because it’s a lot of work and requires intricate planning to make sure that as parents, uncles, aunties, brothers, sisters, friends, colleagues, neighbours… we fulfil our Christmas duty.

From what I have observed I am far from alone in approaching Christmas in this way. In short, most of us are preparing to launch an attack on retailers which presents both a challenge and opportunity for them. It even appears that retailers encourage this type of frantic behaviour: Black Friday, Cyber Monday and Boxing Day sales are all locked into the Christmas shopping diary – if we want that 40 per cent discount we need a plan!

As the attack on retailers is launched they find themselves fighting for a share of the Christmas wallet. At this time of year ‘cash is king’ (sorry content!) and soft KPIs have to make way for hard revenue figures. If you haven’t already, it is also time to adjust your conversion rate optimisation process, because there are so many opportunities, particularly in terms of merchandising.

Unlimited opportunities

With all of us preparing to launch an attack on retailers it is reasonable to assume that:

• Traffic is going to go through the roof during the Christmas period
• Sample sizes mean we can get answers quickly
• By focusing on merchandising and core CTAs we can learn a lot and increase revenue

But it is also true that Christmas is probably the biggest planned and best-known external factor to impact the validity of active optimisation and A/B testing. So should I really test?

Yes of course you should test and revenue is the reason

I make no bones about the fact that while I love CRO and am passionate about A/B testing, I fundamentally believe that a robust CRO strategy is always about compromise. Christmas offers optimisation teams the opportunity to potentially generate more incremental revenue in a few weeks than throughout the rest of the year. Grab this opportunity with both hands!

However, things do need to be done slightly differently as Christmas is about taking an ultra-pragmatic approach to optimisation and testing. Outside of decorating your website for Christmas, here are a few things to get you started:

Value proposition

• Communicate and articulate your Christmas value proposition clearly: “All your Christmas gifts in one place”
• Delivery options/timeframes – an obvious one, but so important at this time of year – need to be clear and visible


• Christmas creates a natural deadline, use that to your advantage and build a sense of urgency: “5 days left to get it delivered in time for Christmas!”
• Adapt your offers, this is the perfect time to test ‘half price’ and ‘mix and match’ offers
• Make it easy for the shopper by guiding them to products you most want to sell:
• “10 bestselling/most popular gifts for teachers, colleagues…” “Ideas for him/ her”

The Experience

• Christmas is all about ‘giving’ so consider your CTAs by making them more in-keeping with gift buying e.g. “stocking up for Christmas” is a nice play on festive words and an effective CTA
• If you have a customer data platform and are able to identify your ‘true’ customers then isolate them and provide a distinct experience for them

CRO at Christmas

Over the festive period you need to adjust your CRO process because you are cramming into a short timeframe, a process that is typically geared to work over longer cycles. There are three key things to remember in particular:


• Speed and agility rule during this period
• Learn and apply from one week to the next maybe even from Monday to Thursday
• Testing for four weeks to get to a robust statistical significance will mean you are leaving money on the table


• You are likely to have to make some compromises to maximise the revenue opportunity, some of them you may even fundamentally disagree with – don’t get too hung up on it!
• Be realistic, you will probably not win a ‘Which Test Won’ Award for your tests in these few weeks so don’t be too purist
• Expect the unexpected and compromise: random traffic sources, bizarre behaviours, high bounce rates – don’t let these things distract you from the task at hand


• It is about taking as much of the wallet share as is physically possible – print this in big red bold letters and put it up on the wall!
• Obvious one, but please check you are tracking revenue as your goal for A/B testing and are confident in the accuracy of that tracking
• Keep it simple! Bottom line figures are those that matter most – micro goals: registrations, click through etc. are all important, but not right now
• Revenue per product, category, brand – all helpful but likely to require slightly deeper analysis and may prove to be a distraction and slow things down. If you can maintain the speed and momentum, track and use these
• Measure the revenue from each variation and throttle accordingly with a constant control

Statistical Significance

Statistical significance is important of course, but at this time of year you need to focus on revenue… so while it may seem a little controversial to downgrade statistical significance for the benefit of revenue, it isn’t that crazy when we look at what statistical significance actually tells us, namely:

• That there is a significant difference between 2 variations

During the Christmas period it doesn’t tell us:

• Which variant has the best chance to beat
• How impactful the test will be
• How much revenue you will make

When all things are said and done, it is simply about gaining a numerical confidence that there is a significant difference between A & B – that is pretty much it.

To conclude

Business is about compromise and no more so than during the Christmas period. For many retailers it will make or break their year. The rule book is different at this time of year because customers are in a zombie-like shopping state, so my key piece of advice is to adjust your strategy by maximising optimisation with a revenue focus.

The article was originally published on the RedEye blog