Abstract illustration of a ribbon of coral splitting into two on a violet field

← ResearchResearch note11 min read

Forking a bundle

A test should be something you can watch: two versions, one difference, both live at the same time.

Abstract

How does a merchant find out which version of a bundle earns more? The literature on online experiments is clear that the only reliable way is to run both versions at once on randomly divided shoppers, and that the common substitute, changing the bundle and comparing with last month, cannot separate the change from the calendar. The same literature shows how easily a real test is misread by looking too early or expecting too much from too little traffic. For a bundle, that means forking on differences large enough to detect, such as how a saving is framed, and deciding how long the test runs before it starts.

Key findings

  1. 01Changing a bundle and comparing with last month is not a test, because the month changed too.
  2. 02Only two versions live at the same time, with shoppers split at random, can show that the change caused the difference.
  3. 03Checking a test repeatedly and stopping when it looks good can turn a 5% false alarm rate into more than 25%.
  4. 04The traffic a test needs grows with the square of the precision you ask for, so small stores should test big differences or wait.
  5. 05Published pricing research suggests that how a bundle saving is framed is worth a fork, and that the famous decoy effect may not survive real product pages.

Every decision in a bundle is a guess until it is tested. The price is a guess. The order is a guess. Whether the mug belongs is a guess. Good merchants guess well, and they would still rather know.

The usual way to find out is to change something and watch. Drop the price in March, compare the month with February, and call the difference the result. It feels like a test. It is not one, and the reasons are some of the best understood in applied statistics.

We think of a real test as a fork. The bundle splits into two versions with one difference between them, and both go live. Shoppers are divided between the two at random, and every order is counted against the version it came from. This note sets out what the published work says about why the fork is necessary, how it still goes wrong, and which differences on a bundle are worth the wait.

Fig. 1One bundle, two versions, one differenceIllustrative

Version A

$99

100

Version B

$89

123Wins
Revenue per visitor, version A = 100OriginalVariant

Why comparing with last month is not a test

A before and after comparison rests on one assumption: that the only thing that changed between the two periods was the bundle. That is never true. Payday moved. The weather turned. An email went out, a competitor ran a sale, a holiday fell in one month and not the other. Each of these moves sales on its own, and a before and after comparison adds all of them to the effect of your change and hands you the total.

The standard reference on online experiments, a survey by Ron Kohavi and colleagues at Microsoft, makes the point at a much finer grain. Even inside a single test they advise running for at least a week or two and then in whole weeks, because the people who visit at the weekend are often a different group from the people who visit on a Tuesday, and they extend the warning to holidays and seasons.1 If Saturday cannot stand in for Tuesday, March cannot stand in for February.

There is a second, quieter problem. Merchants rarely change a bundle at random. They change it after a bad month. A bad month is partly a real problem and partly bad luck, and luck does not repeat. Statisticians call what follows regression to the mean: unusually high or low measurements tend to be followed by ones closer to the average, which can make natural variation look like real change.2 The month after a bad month would probably have looked better whatever you did. Change the bundle in between and the recovery gets credited to the change.

This is why Kohavi and colleagues describe controlled experiments as the best scientific design for establishing that a change caused an outcome, and set them apart from after-the-fact analysis of what happened over time.1 The book that three of the field’s practitioners later wrote for Cambridge University Press carries the same idea in its title: the aim is experiments you can trust.3

The second trap: looking too early

A fork solves the calendar problem. It does not stop you misreading the result. The most common misreading has a name, the peeking problem, and it catches careful people.

Start with what "significant" means. When a testing tool says a result is significant at the 5% level, it means this: if the two versions were really identical, a gap this large would turn up by chance only about one time in twenty. That promise holds on one condition. You look once, at a moment you fixed in advance.

Nobody does that. The numbers are on a dashboard and you check them every morning. Each look is another chance for random noise to cross the line, and if you stop the first time it does, you have chosen the luckiest moment in the test and reported it as typical. Evan Miller’s 2010 essay "How Not To Run an A/B Test" put figures on this. In his worked example, two identical versions are checked after every visitor and the test is stopped at the first significant reading or after 150 visitors. The false alarm rate is not 5%. It is 26.1%. He calls that close to a worst case, and also not unheard of.4

How peeking erodes a 5% threshold. If you check a test several times and stop at the first significant reading, the reading has to be far stricter than 5% for the real false alarm rate to stay at 5%.
Times you lookReading you need to see for a true 5%
12.9% 4
22.2% 4
31.8% 4
51.4% 4
101.0% 4

The academic treatment came from researchers at Stanford working with the testing platform Optimizely. Their paper "Peeking at A/B Tests", presented at the KDD conference in 2017, states the problem plainly: the standard readouts are unreliable when people watch them continuously while a test runs, which is exactly how people use them. Their answer is a different kind of readout, one built to stay valid whenever you choose to stop.5 The practical point for anyone without that machinery is Miller’s: decide how long the test will run before it starts, and wait.

A test you stop the moment it looks good is a guess with a chart attached.

How long is long enough

How long depends on how many shoppers see the bundle and how big a difference you are trying to detect. The rule of thumb in Kohavi’s survey says the number of visitors each version needs rises with how noisy the measure is and falls with the square of the difference you want to see.1 The square is the part that matters. Ask for twice the precision and you need four times the traffic.

Their own example is a shop where 5% of visitors buy. To detect a 5% relative change in that conversion rate, the formula asks for a little under 122,000 visitors per version. Relax the target to a 20% change and the figure falls to 7,600. Measuring revenue per visitor is harder still, because order values vary so much: the same shop needs more than 409,000 visitors per version to detect a 5% change in revenue.1 Conversion is the steadier measure, and for most bundles it is the one to decide on.

Three habits follow. Work out the length before starting, using a calculator such as Miller’s,6 and round up to whole weeks. Choose the measure in advance, because picking the best-looking number afterwards is another way of finding effects that are not there.1 And prefer forks with a large difference between the two versions. A different price point, a different lead product or a different way of expressing the saving can produce a gap a modest store is able to see. A reworded button usually cannot.

When the result looks too good

Experimenters keep one more rule to hand, known as Twyman’s law: any figure that looks interesting or different is usually wrong. Kohavi, Tang and Xu give it a chapter.3 A fork that shows one version earning half as much again should prompt a check before a celebration. Were shoppers split evenly? Did one version go live a day late? Did a single large order land on one side? If the result survives the questions, run it once more.

What is worth forking on a bundle

If every fork costs weeks, the choice of what to test matters more than the testing. The pricing literature offers some candidates: places where a change that costs the merchant nothing, or nearly nothing, has moved sales in published studies.

Published findings on levers a bundle can fork on. None of these studies was run on an online bundle, so each is a reason to test and not a forecast.
LeverStudyWhat was found
More product or money offChen, Marmorstein, Tsiros and Rao (2012), a field experiment in a retail store, a survey and lab studiesA bonus pack sold substantially more than a price discount of the same economic value 7
Price endingsAnderson and Simester (2003), three field experimentsA price ending in 9 raised demand in all three, more for new items, less alongside "Sale" signs 8
Which item carries the discountJaniszewski and Cunha (2004), six studiesThe same saving is valued differently depending on which product in the bundle it is attached to 9
A decoy optionHuber, Payne and Puto (1982), with a failed extension by Frederick, Lee and Baskin (2014)An inferior third option lifted the option it resembled, but not when products were shown in pictures or experienced 10 11

How the saving is expressed. Chen and colleagues argue that shoppers tend to read a percentage without attending to what it is a percentage of, which they call base value neglect. "50% more free" sounds larger than "33% off", though the price per unit is identical. In their field experiment a bonus pack outsold an economically equivalent discount by a wide margin.7 A build-a-box has the same choice to make. "Buy five, get a sixth free" and "16.7% off six" are one offer described two ways, and there is published reason to think they will not sell alike.

The last digit. Anderson and Simester varied price endings in three field experiments and found that a price ending in 9 raised demand each time. The effect was stronger for new items than for ones the retailer had sold before, and seemed weaker when a "Sale" sign was also present.8 For a bundle, which is often a new item at a new price, that is an inexpensive fork.

Where the discount sits. A bundle discount can be shown as a saving on the whole or pinned to one product inside it. Janiszewski and Cunha ran six studies on this and found that the framing changes how attractive the bundle looks, because each product carries its own reference point in the shopper’s mind.9 "Kettle free with the set" and "£30 off the set" can be the same money and a different offer.

The decoy. Huber, Payne and Puto’s 1982 paper showed that adding an option nobody should choose can push people towards the option it most resembles.10 It is the source of a great deal of three-tier pricing advice. It is also contested. Frederick, Lee and Baskin reported in 2014 that the effect appeared when products were described by numbers alone and generally did not appear when people could see a picture of the product or try it, and they questioned its practical significance.11 The original authors replied that the effect is robust when the original conditions are reproduced.12 A bundle page is made of photographs. That places it on the side of the argument where the effect has been hardest to find, so a decoy tier is something to test and not something to assume.

Working one fork through

Suppose a coffee merchant sells a starter set: a dripper, filters and two bags of beans, with £8 off the combined price. She suspects the saving would land better as a free third bag, which costs her about the same. The fork looks like this.

  1. 01One difference. Version A says "Save £8". Version B says "Third bag free". Products, photographs, order and everything else stay identical, so whatever gap appears has one possible cause.
  2. 02One measure, chosen first. She picks the share of visitors who buy the bundle, and notes revenue for each version as a check, since a free bag and a discount do not cost exactly the same.
  3. 03A length, fixed first. Her traffic and the size of gap she hopes for give her a number of weeks. She writes it down, in whole weeks, avoiding a stretch with a big promotion at one end of it.
  4. 04No early verdict. She can look at the numbers as often as she likes. She does not act on them until the date she wrote down.
  5. 05A sceptical reading. If one version wins by a believable margin, she keeps it. If it wins by an astonishing one, she checks the split and runs it again.

Then the winner becomes the bundle, and the next fork starts from there. A bundle that has been forked five times is a different bundle, and each difference in it was earned.

What is not known

The findings above come from shops, catalogues and laboratories. None of the studies we cite was run on an online bundle builder, and effects of this kind are known to vary with the product, the shopper and the setting. The decoy dispute shows how a result that is solid in one presentation can fade in another. Treat each published effect as a reason to run a fork, never as a forecast of what the fork will show.

A test also describes only the shoppers who arrived while it ran. Kohavi and colleagues note that a new design can do unusually well or badly at first simply because it is new, and that results may not carry across seasons or countries.1 A fork won in November may deserve a second run in February.

And there is the limit that no method removes. A store with modest traffic cannot detect small differences in any reasonable time. That is not a failing of the store or of the tool. It is the square in the formula. The honest response is to ask bigger questions, or fewer.

Where Kitenzo’s work sits

The fork itself is live. In Kitenzo a merchant can split a bundle into two versions, send shoppers to both over the same period, and see the results for each version side by side. That removes the calendar from the comparison, which is the largest single error in how bundles are usually judged.

What the product does not yet do is tell you how far to trust the gap between the two. A statistical confidence readout is in the lab and has not shipped, and we would rather say so than let a pair of numbers imply more than they can. Until it does, the patience described here is the merchant’s to apply: fix the length first, run in whole weeks, do not call it early, and be suspicious of anything spectacular.

We are working on making that discipline part of the tool. The reading we want a merchant to get is the plain one: this version is better, this is how sure you can be, and this is how much longer it would take to be surer.

References

  1. 1Kohavi, R., Longbotham, R., Sommerfield, D. and Henne, R. M. (2009). Controlled experiments on the web: survey and practical guide. Data Mining and Knowledge Discovery, 18(1), 140–181.
  2. 2Barnett, A. G., van der Pols, J. C. and Dobson, A. J. (2005). Regression to the mean: what it is and how to deal with it. International Journal of Epidemiology, 34(1), 215–220.
  3. 3Kohavi, R., Tang, D. and Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press.
  4. 4Miller, E. (2010). How Not To Run an A/B Test. evanmiller.org.
  5. 5Johari, R., Koomen, P., Pekelis, L. and Walsh, D. (2017). Peeking at A/B Tests: Why it matters, and what to do about it. Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1517–1525.
  6. 6Miller, E. (n.d.). Sample Size Calculator. Evan’s Awesome A/B Tools, evanmiller.org.
  7. 7Chen, H., Marmorstein, H., Tsiros, M. and Rao, A. R. (2012). When More Is Less: The Impact of Base Value Neglect on Consumer Preferences for Bonus Packs over Price Discounts. Journal of Marketing, 76(4), 64–77.
  8. 8Anderson, E. T. and Simester, D. I. (2003). Effects of $9 Price Endings on Retail Sales: Evidence from Field Experiments. Quantitative Marketing and Economics, 1(1), 93–110.
  9. 9Janiszewski, C. and Cunha, M., Jr. (2004). The Influence of Price Discount Framing on the Evaluation of a Product Bundle. Journal of Consumer Research, 30(4), 534–546.
  10. 10Huber, J., Payne, J. W. and Puto, C. (1982). Adding Asymmetrically Dominated Alternatives: Violations of Regularity and the Similarity Hypothesis. Journal of Consumer Research, 9(1), 90–98.
  11. 11Frederick, S., Lee, L. and Baskin, E. (2014). The Limits of Attraction. Journal of Marketing Research, 51(4), 487–507.
  12. 12Huber, J., Payne, J. W. and Puto, C. P. (2014). Let’s Be Honest About the Attraction Effect. Journal of Marketing Research, 51(4), 520–525.