
← ResearchEssay11 min read
A bundle is a path, not a page
Why the bundle you launch should not be the bundle you are running a year later, and what two decades of published experiments say about it.
Abstract
Most bundles are built once and left alone. We ask what that costs. The published record from companies that test every change is blunt: only about a third of well-made ideas improve the thing they were meant to improve, the winners are usually small, and they only add up if someone keeps looking for them. A bundle that never changes cannot find its winners, so we treat a bundle as a path with a next step, not a finished page.
Key findings
- 01At Microsoft, only about one third of carefully built ideas improved the metric they were designed to improve; at Google and Bing the published figure is 10% to 20%.
- 02The gains that are found are mostly small, and they compound: Bing credits a steady stream of them with raising revenue per search by 10% to 25% a year.
- 03Demand moves with seasons, stock and promotions, so a bundle that was right at launch drifts out of date even if nobody touches it.
- 04Testing has real limits for a single store: small samples, false positives and inflated wins all need care.
- 05The useful question about a bundle is not how the page looks but what its next change should be.
Most bundles are built once. Someone chooses the products, sets the price, designs the page and ships it. From that moment the bundle is a fixed object: the same products in the same order at the same price, for as long as it stays live.
Nothing else in a store works like this. Stock moves. Seasons turn. A product that was a quiet extra in spring is the reason people buy in winter. The store changes every day and the bundle page, however beautiful, stays exactly where it was left.
This essay asks a plain question in merchant terms. If you launch a bundle and never change it, what are you giving up? The answer does not come from us. It comes from twenty years of published work by the companies that test almost everything they ship, and it is less comfortable than most people expect.
The flat line
If you could watch a static bundle over a year, the line would be flat. Not because it performs badly, but because it never gets the chance to perform better. It cannot find out that one pairing is doing most of the work, or that a lower price would have earned more, or that the fourth product has not been picked in a month.
Drag the slider below. The dashed line is a bundle page that is shipped and left alone. The solid line is the same page when every change is noticed, suggested, tested and kept only if it wins.
Drag to move through a year
Bundle revenue over a year
- On Kitenzo
- Static page
- Losing test
The same page, on Kitenzo
- Day 1Launched. Same six products, same price, same order as the static page.
Illustrative. Some of these changes rely on capabilities still in the lab.
The figure is an illustration, not a measurement. The rest of this essay is about whether the published evidence supports its shape. It does, with some honest caveats that we set out further down.
What the research says: most ideas do not work
The strongest evidence comes from online controlled experiments, better known as A/B tests. Visitors are split at random between the current version and a changed one, and the two groups are compared. Because the split is random, a difference in what the groups do can be credited to the change and not to the weather, the season or a marketing email.
In 2009 Ron Kohavi and colleagues at Microsoft looked back over the experiments run on the company’s experimentation platform. These were not careless ideas. Each was designed and built by a team that believed it would help. Their finding: “only about one-third were successful at improving the key metric”.1 Kohavi and Stefan Thomke later gave the fuller split in Harvard Business Review: at Microsoft as a whole, one third of experiments prove effective, one third have neutral results and one third have negative results.2
Other companies report the same or worse. The same article puts the share of experiments with positive results at Google and Bing at about 10% to 20%.2 A 2013 paper from the Bing team collects similar statements from Netflix, Etsy and others, and sums up the lesson in one sentence: “we are poor at assessing the value of ideas”.3
| Who | What was counted | Share that helped |
|---|---|---|
| Microsoft, experimentation platform (2009) | Well-designed experiments meant to improve a key metric | About one third 1 |
| Microsoft as a whole (2017) | Experiments: effective, neutral, negative | One third each 2 |
| Google and Bing (2017) | Experiments that generate positive results | About 10% to 20% 2 |
| Bing; Booking.com, Google Ads and Netflix; Airbnb Search (compiled 2022) | Historical success rates gathered from published sources | 15%; 10%; 8% 4 |
| A commercial testing platform, 2,766 experiments (2022) | Effects estimated to be truly zero | About 70% had no real effect 5 |
The last row matters for anyone who is not a search engine. Ron Berman and Christophe Van den Bulte studied 4,964 effects from 2,766 experiments run by ordinary businesses on a commercial testing tool, and estimated that about 70% of the changes tested did nothing at all.5 The pattern is not special to Microsoft.
Now apply that to a bundle. The choice of products, their order, the price, the discount and the size of the box are each an idea. Nobody would claim a bundle team is better at guessing than the engineers at Bing. So some of the choices in any launched bundle are probably wrong, some make no difference, and some are right. From the page alone you cannot tell which is which. A bundle that is never tested does not avoid this problem. It just never finds out.
Small gains, kept
If most ideas fail, why do the companies that test the most keep doing it? Because the minority that work can be kept, the ones that hurt can be thrown away, and the kept gains stack on top of each other.
Kohavi and Thomke report that Microsoft, Amazon, Booking.com, Facebook and Google each run more than 10,000 online controlled experiments a year, and that at Bing this habit finds dozens of revenue-related improvements every month, which together have raised revenue per search by 10% to 25% each year.2 They are clear about where the progress comes from: not from a few grand redesigns but from “getting many small changes right”. Booking.com has described more than ten years of product development run this way, with the tools opened up so that anyone in the company can test a change.6
There are occasional big wins too. The best-known is a small change to how Bing displayed advertising headlines. It was judged a low priority and waited more than six months, and when an engineer finally tested it, it raised revenue by 12%, worth more than $100 million a year in the United States alone.2 The point of the story is not that small changes are secretly huge. It is that nobody, including the people closest to the product, could tell in advance.
The page is the part your customers see. The path is the part that earns.
Explore or exploit
There is a name for the choice a merchant faces every week. To exploit is to keep selling what you already know works. To explore is to try something that might work better and might not. Every visitor shown an untested version of a bundle is a small bet: you may lose a little on that visitor, and you may learn something that pays on every visitor after.
James March set out the tension in a 1991 paper that is still widely cited. His argument was that organisations naturally drift towards exploiting, because its rewards are quick and certain while the rewards of exploring are slow and uncertain. That drift works well in the short run and badly in the long run.7 A bundle that is never changed is the extreme case: all exploit, no explore.
Retail researchers have studied the same trade-off directly. Felipe Caro and Jérémie Gallien modelled the problem faced by fast-fashion retailers such as Zara, which decide what to put on the shelf during the season as sales come in, not once before it starts. They treat it as what statisticians call a multi-armed bandit: a row of slot machines with unknown payouts, where the only way to learn which is best is to spend some pulls finding out. Their paper shows how a retailer can balance showing proven sellers against trying unproven ones.8 The same idea is used well beyond retail. A team at Yahoo used a version of it to choose which news stories to show on the front page and reported a 12.5% lift in clicks over a simpler method.9
You do not need the mathematics to use the lesson. A healthy bundle spends most of its traffic on the version that is known to work and a little on finding the next one.
The ground moves
Even a bundle that was exactly right on launch day does not stay right, because the shoppers and the shelf both change. Robert Fildes, Shaohui Ma and Stephan Kolassa reviewed the research on retail demand forecasting and describe how hard product-level demand is to pin down: promotions and other drivers add “considerable complexity”, and for new products there is little evidence that any forecasting approach works well.10 If professional forecasters with years of data struggle to say what a product will sell next month, a bundle fixed last year is resting on a guess that has aged.
Data scientists call the general problem concept drift: the world a decision was based on changes, so the decision slowly stops fitting, with no error message to say so.11 For a bundle the drift is easy to picture. A summer product goes quiet in October. A bestseller sells out and the box is left with a gap. A new product arrives that half your customers would rather have. The page looks the same throughout.
A worked example
Take a coffee roaster selling a pick-your-six box: six bags chosen from twelve, at a fixed price. As a page, it is finished on launch day. As a path, it has a list of open questions, and each one is a step.
- •Which product leads? The roaster put the house blend first. Shoppers may build more boxes when a single-origin leads. Only a side-by-side test can say.
- •Is six the right size? A box of four at a lower price may bring in more first-time buyers. A box of eight may earn more per order. Both are plausible. Going by the published record, at least one of those guesses is probably wrong.
- •What changed this month? The decaf has stopped being picked. A seasonal roast is nearly out of stock. Neither shows on the page.
- •What goes together? If shoppers who choose the dark roast almost always add the espresso blend, that pairing could be offered up front.
None of these has an answer that can be worked out at a desk, which is what the Microsoft figures tell us. Each has an answer that can be found. Find one a month, keep the winners, and the roaster a year later is running a different and better box, even though no customer ever saw a redesign.
- Beans
- Grinder
- Dripper
- Kettle
This is the idea the rest of our research hangs from. Finding what sells together tells you which steps belong on the path. Experiments tell you which version of the path wins. Signals tell you when a step is about to give way. Advice puts all of it in front of the one person who knows the business.
Limits, and what is not known
The evidence above comes mostly from very large software companies. A merchant should know where it does not transfer cleanly.
Small stores have small samples. Kohavi, Deng and Vermeer work through an example: to reliably detect a 10% relative improvement on a 3.7% conversion rate, the standard formula asks for about 41,600 visitors in each version.4 Many bundles will not see that traffic in a season. Smaller tests can still be worth running, but they can only detect large differences, and a result that looks dramatic on a few hundred visitors is more likely to be luck than insight. The same paper explains why: when a test is too small, the wins it does find tend to be exaggerated, a problem known as the winner’s curse.4
A “significant” result can still be wrong. In the Berman and Van den Bulte study, between 18% and 25% of results that passed the usual 5% significance test were estimated to be false discoveries, which means roughly one “winner” in five was not real.5 Checking a running test every day and stopping the moment it looks good makes this worse, as Ramesh Johari and colleagues have shown. They also describe methods that stay valid under constant checking.12
The right amount of testing is unsettled. Eduardo Azevedo and co-authors, using data from Bing, argue that when most of the value comes from rare big wins, it pays to test many ideas quickly on smaller samples and not a few ideas very carefully.13 That is encouraging for smaller stores, but it is one study of one search engine. Nobody has published the equivalent for product bundles.
Most of this is not about bundles at all. The published experiments concern search results, booking flows and news pages. We think the lesson carries over, because a bundle is a set of guesses about what people want and guesses are what the evidence is about. But that is an argument, not a measurement. The classic guide to the field by Kohavi, Diane Tang and Ya Xu is frank that getting numbers is easy and getting numbers you can trust is hard.14 We would sooner say so than pretend otherwise.
What it means for a bundle, and for teams who code their own
Plenty of teams can build a beautiful bundle page, and more of them every month. That is exactly why the page is no longer the advantage. A hand-coded page that never learns is the flat line with better typography.
In Kitenzo today a merchant can run two versions of a bundle side by side and compare them in reporting, hide products that have sold out so the path does not lead to a dead end, and set rules for what can go in the box. On Enterprise, pairing analysis shows which products are chosen together across every bundle. Suggestions for what to add, swap or drop are still in the lab, and so are statistical confidence readouts on tests, which is why the limits above are worth reading before you call a winner.
Built on Kitenzo, the same page keeps its design and gains a path underneath it. The first version of a bundle is a starting guess. The evidence says it is probably wrong in ways nobody can see yet, and that the stores which keep taking the next step are the ones that find out.
References
- 1Kohavi, R., Crook, T. and Longbotham, R. (2009). Online Experimentation at Microsoft. Third Workshop on Data Mining Case Studies and Practice Prize.
- 2Kohavi, R. and Thomke, S. (2017). The Surprising Power of Online Experiments. Harvard Business Review, September to October 2017.
- 3Kohavi, R., Deng, A., Frasca, B., Walker, T., Xu, Y. and Pohlmann, N. (2013). Online Controlled Experiments at Large Scale. Proceedings of KDD 2013.
- 4Kohavi, R., Deng, A. and Vermeer, L. (2022). A/B Testing Intuition Busters: Common Misunderstandings in Online Controlled Experiments. Proceedings of KDD 2022.
- 5Berman, R. and Van den Bulte, C. (2022). False Discovery in A/B Testing. Management Science, 68(9), 6762 to 6782.
- 6Kaufman, R. L., Pitchforth, J. and Vermeer, L. (2017). Democratizing online controlled experiments at Booking.com. arXiv:1710.08217 (Conference on Digital Experimentation, MIT).
- 7March, J. G. (1991). Exploration and Exploitation in Organizational Learning. Organization Science, 2(1), 71 to 87.
- 8Caro, F. and Gallien, J. (2007). Dynamic Assortment with Demand Learning for Seasonal Consumer Goods. Management Science, 53(2), 276 to 292.
- 9Li, L., Chu, W., Langford, J. and Schapire, R. E. (2010). A Contextual-Bandit Approach to Personalized News Article Recommendation. Proceedings of WWW 2010; arXiv:1003.0146.
- 10Fildes, R., Ma, S. and Kolassa, S. (2022). Retail forecasting: Research and practice. International Journal of Forecasting, 38(4), 1283 to 1318.
- 11Žliobaitė, I. (2010). Learning under Concept Drift: an Overview. arXiv:1010.4784.
- 12Johari, R., Pekelis, L. and Walsh, D. J. (2015). Always Valid Inference: Bringing Sequential Analysis to A/B Testing. arXiv:1512.04922.
- 13Azevedo, E. M., Deng, A., Montiel Olea, J. L., Rao, J. and Weyl, E. G. (2020). A/B Testing with Fat Tails. Journal of Political Economy, 128(12).
- 14Kohavi, R., Tang, D. and Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press.


