Abstract illustration of two rounded shapes in blue and cream facing each other on magenta

← ResearchResearch note11 min read

Advice, not autopilot

Why the system we are building asks before it changes anything, and listens when you say no.

Abstract

When a system notices something about a bundle, should it act or should it ask? The published evidence pulls in two directions: statistical models often predict better than unaided human judgment, yet people who are left to supervise automation they cannot adjust tend either to abandon it or to stop checking it. The design that fits both findings is one where the system proposes and a person who knows the business decides. That is the position we are building towards. Suggestions of this kind are in the lab and are not in the product today.

Key findings

  1. 01Models often beat unaided human prediction, which is the reason to have a system make suggestions at all.
  2. 02People abandon an algorithm quickly after watching it make a mistake, but will use an imperfect one far more readily if they can adjust its output even slightly.
  3. 03Fully automatic systems leave the person with the hardest job: noticing the rare failure of something they have stopped watching.
  4. 04A merchant holds facts no bundle data contains, such as a supplier change, a planned campaign or a margin agreement.
  5. 05Accept-or-decline suggestions are a design position we hold and are still building; merchants do not receive them yet.

There is a tempting version of this product where the bundle simply optimises itself. Products appear and disappear, prices drift, and the merchant watches a number go up. We are not building that, and it is worth setting out why with evidence and not only instinct.

The question is an old one in a new setting. Engineers in aviation, process control and medicine have spent fifty years asking how much a machine should decide on its own and how much it should hand to a person. Psychologists have spent longer asking whether the person or the formula is the better judge. Both bodies of work bear directly on a small practical matter: what should happen when a system notices that a product in a bundle has stopped being chosen.

One thing first, so nothing below is misread. Kitenzo does not send merchants suggestions today. This article describes the design we are working towards and the research behind it.

The case for the model

It would be convenient to argue that human judgment is simply better and the machine should defer. The evidence does not support that. In 1989 Robyn Dawes, David Faust and Paul Meehl reviewed the research comparing two ways of making a prediction: the clinical method, in which a professional weighs the information in their head, and the actuarial method, in which a fixed rule combines the same information according to how it has related to outcomes in the past. They counted nearly 100 comparative studies in the social sciences. In virtually every one, the rule equalled or surpassed the professional.1

Dawes and his colleagues were careful about what this did and did not show. They noted that the rule is far from infallible and sometimes achieves only modest results. They also described what has become known as the broken leg problem. A formula may predict very well whether a given person will go to the cinema this week. A person who happens to know that the cinema-goer has just fractured a leg will beat it. People can spot the rare fact that the formula was never built to include. The authors added a warning: experts given licence to override a rule tend to find more broken legs than there are.1

So the honest starting point is this. For the routine question (which product is being chosen less, which pairing is carrying a bundle) a consistent system looking at every order will usually see more clearly than a busy person glancing at a report. That is why a system should be making suggestions in the first place. The broken leg is why it should not be making the final call.

Why people do not simply follow the model

If models predict well, the efficient design would seem to be to let them act. The trouble is what people do around automated decisions, which has been measured.

In 2015 Berkeley Dietvorst, Joseph Simmons and Cade Massey ran five studies in which participants watched a statistical model make forecasts, watched a human make forecasts, or both, and then chose which one to stake a bonus on. People who had seen the model perform were less likely to choose it, even when they had watched it outperform the human. The reason was that they lost confidence in the model faster than in a person after seeing each make the same mistake. The authors called this algorithm aversion.2

Their follow-up is the finding that matters most for design. Participants forecast students’ test scores, were told a model’s forecasts were off by 17.5 percentiles on average, and chose whether to use it. When the choice was all or nothing, 32% took the model. When they were allowed to adjust each of its forecasts by up to 10 percentiles, 76% did. In a later study the allowance was set at 10, 5 or 2 percentiles, and its size made almost no difference: 71%, 71% and 68% used the model, against 47% of those given no control. Participants who were offered the chance to adjust the model forecast more accurately than those who were not, were more satisfied with the process and were more likely to choose the model again.3

Aversion is not the whole picture. Jennifer Logg, Julia Minson and Don Moore reported six experiments in which people gave more weight to advice when told it came from an algorithm than when told it came from a person, an effect they named algorithm appreciation. It weakened when people were choosing between the algorithm and their own estimate, and experienced professional forecasters relied on the algorithmic advice less than lay people did, which hurt their accuracy.4 Taken together, the two lines of work suggest that people will listen to a system readily enough as an adviser. They resist it when it replaces their own judgment in a field they know and gives them no say.

What full automation does to the person left in charge

The opposite failure is just as well documented. In 1997 Raja Parasuraman and Victor Riley set out four ways people relate to automation. Use is choosing to switch it on or off. Misuse is relying on it too much, which leads to failures of monitoring. Disuse is neglecting it, typically after too many false alarms. Abuse is automating a function without regard for what that does to the person who must work with the result.5

Misuse has since been studied at length. Parasuraman and Dietrich Manzey’s 2010 review covers two related problems. Complacency is failing to check an automated system that has been reliable so far. Automation bias is accepting what a decision aid says without weighing other evidence, which produces errors of omission (missing a problem because the system did not flag it) and errors of commission (doing the wrong thing because the system said to). The review found both in experts as well as novices. Complacency was not overcome by simple practice, and automation bias was not prevented by training or instructions.6

Lisanne Bainbridge described the underlying trap in 1983, in a short paper called “Ironies of Automation”. A designer automates what can be automated and leaves the operator the rest, which means the tasks nobody could work out how to automate. The operator is then expected to monitor a system that was installed because it does the job better than they do, and to take over when it fails. Skills that are not used fade. Vigilance studies show that people cannot keep watching a display on which almost nothing happens for much more than half an hour. The more reliable the automation, the less practised and less attentive the person who is supposed to catch its rare failure.7

A merchant with a self-adjusting bundle would be in exactly that seat. For months nothing would need their attention. Then one day the system would drop a product for a reason that made sense in the data and none in the business, and the only person able to notice would have stopped looking long before.

A system that is right most of the time and acts alone trains its owner to stop checking it.

Levels of automation, with a bundle in mind

Automation is a matter of degree. In a 1978 report on remotely controlled undersea vehicles, Thomas Sheridan and William Verplank described a scale of ten levels running from the human doing everything to the computer doing everything.8 Parasuraman, Sheridan and Christopher Wickens restated the scale in 2000 and added a second idea: a system does four different kinds of work (gathering information, analysing it, deciding and acting), and each can be automated to a different level.9

The table below paraphrases the scale, grouping neighbouring levels, and applies it to one case: a kettle in a coffee bundle that nobody has chosen for a month.

The Sheridan and Verplank scale, paraphrased, applied to one product that has stopped being chosen
LevelWhat the computer doesThe bundle example
1Offers no help; the human does everything9The merchant spots the kettle in a report, or does not
2 to 4Offers alternatives, narrows them down, or suggests one9The system lists options, or proposes one: drop the kettle
5Carries out its suggestion if the human approves9Drop the kettle? Accept or decline. Nothing changes until the merchant answers
6Gives the human a limited time to veto, then acts9The kettle will be removed on Friday unless the merchant objects
7 to 9Acts first, then tells the human always, on request, or only if it chooses to9The kettle is gone; a note may or may not say so
10Decides and acts alone, ignoring the human9The bundle changes itself and nobody is told

The model’s second idea matters here too. We think the noticing and the analysis should be automated as far as they can be, because that is where a consistent system has the advantage Dawes and colleagues documented.1 The deciding should sit at level five. The step from five to six looks small and is not. At five, silence means nothing happens. At six, silence means consent, and the complacency findings above describe what becomes of people asked to veto in time.6

Fig. 1Advice arrives one suggestion at a timeIllustrative · in the lab
  1. 01Add thisBought with this bundle most of the time.Accepted
  2. 02Drop this?Not chosen in thirty days.Moving away from this line anyway, leave it for now.Declined
  3. 03Test itTry the same bundle at a lower price?Waiting for you

What the merchant can see and the system cannot

A system can see that the kettle has not been chosen in thirty days. It cannot see that the line is being discontinued, that a new supplier arrives next month, that the kettle is the centre of a campaign booked for October, or that a margin agreement makes the product it would promote in its place a poor trade. Each of these is a broken leg in the sense Dawes, Faust and Meehl meant: a fact that settles the matter, known to the person and absent from the data.1 The person running the store knows them without thinking.

So the design we are working towards is a conversation. Add this, it is often bought alongside the bundle. Swap that, it is running low. Drop this, nobody picks it. Each suggestion arrives on its own with an accept and a decline, and says in a sentence why it is being made.

The sentence of explanation has evidence behind it. Michael Yeomans and colleagues compared a recommender system with people at predicting which jokes someone would find funny. The system beat strangers, friends and family, and people were still reluctant to rely on it, partly because they felt they understood how a person recommends and did not understand the machine. The authors’ conclusion was that accuracy is not enough: a recommendation must also be understood.10 More explanation is not always better, though. René Kizilcec tested three levels of transparency on 103 learners on an online course whose essays were graded by an algorithm that combined marks from their peers. Among those who received a lower grade than they expected, a short account of how the grade was reached restored trust. Adding the full detail of the raw marks and adjustments eroded it again.11 The suggestion should give its reason briefly and leave out the workings.

A decline carries information as well. “Moving away from this line anyway, leave it for now” tells the system something no order data could. It is the broken leg, reported by the one person who can see it.

When nobody is asked

The best known case of unattended pricing involved a biology textbook. In April 2011 the biologist Michael Eisen noticed two new copies of Peter Lawrence’s The Making of a Fly listed on Amazon at more than 1.7 million dollars each. He watched the listings for a week and worked out the cause. Once a day one seller set its price to 0.9983 times the other’s. The other then reset its own to 1.270589 times the first. Each rule was reasonable taken alone. Together they multiplied the price by about 1.27 a day, and it peaked on 18 April at 23,698,655.93 dollars, plus 3.99 for shipping. The next day someone noticed and the price fell to 106.23.12

Eisen’s diagnosis was that both sellers were using pricing software with no built-in sanity check, and that nobody was watching what it produced.12 No book was sold at that price and nobody was harmed. It is still a clear picture of level ten: two systems acting on each other, each correct by its own rule, with no person asked at any point.

Limits, and what is not known

  • •The algorithm aversion studies used students and online panels forecasting test scores and similar quantities for small bonuses.23 A merchant deciding about their own shop has more at stake and knows more. The direction of the finding is plausible for merchants. It has not been tested on them.
  • •Letting people adjust a model has a cost. Dawes and colleagues warned that human overrides often make predictions worse, because people see exceptions that are not there.1 In the adjustment studies, people offered the chance to tweak the model did better than people offered it on a take-it-or-leave-it basis, largely because more of them used it. That is a comparison with rejecting the model, not with following it untouched.3
  • •The ten-level scale is a thinking tool and has critics. Ben Shneiderman argues that a single scale wrongly implies that more automation must mean less human control, and quotes Sheridan’s own remark that the level descriptions have been taken more seriously than expected.13 We use the scale to locate a design choice, not as a law.
  • •Asking has its own failure. Too many suggestions, or too many poor ones, produce the disuse Parasuraman and Riley described: people stop reading them.5 A habit of pressing accept without reading is automation bias by another route.6 Putting a person in the loop only helps if the person is still paying attention.
  • •The fly book is one anecdote, documented by one observer.12 It illustrates a mechanism. It does not measure how often unattended automation goes wrong.
  • •We have no results of our own to report. Whether accept-or-decline suggestions improve bundles in practice, and how many suggestions a merchant will tolerate, are things we do not yet know.

What it means for a bundle, and how far along we are

Read together, the research supports a specific arrangement. The system should do the noticing, because it sees every order and applies the same standard each time. It should propose one change at a time with a short reason, because advice that is understood is advice that gets used. The merchant should decide, because they hold the facts the data lacks and because people use a system more when they have a say in what it does. A declined suggestion should be kept as information. And nothing about a bundle should change because nobody answered.

This is the least finished work we publish. Suggestions to add, swap and drop products, the accept and decline that go with them, and suggested experiments are all in the lab and not in the app. The pieces underneath them are live: side-by-side experiments between bundles, bundles that follow inventory and hide sold-out products, advanced reporting and, on Enterprise, the picture of what sells together. A merchant can already look at those and make the call themselves, which is level one on the scale above. The aim is to move the noticing up that scale and leave the decision with the merchant.

References

  1. 1Dawes, R. M., Faust, D. and Meehl, P. E. (1989). Clinical versus Actuarial Judgment. Science, 243(4899), 1668-1674.
  2. 2Dietvorst, B. J., Simmons, J. P. and Massey, C. (2015). Algorithm Aversion: People Erroneously Avoid Algorithms After Seeing Them Err. Journal of Experimental Psychology: General, 144(1), 114-126.
  3. 3Dietvorst, B. J., Simmons, J. P. and Massey, C. (2018). Overcoming Algorithm Aversion: People Will Use Imperfect Algorithms If They Can (Even Slightly) Modify Them. Management Science, 64(3), 1155-1170 (authors’ manuscript).
  4. 4Logg, J. M., Minson, J. A. and Moore, D. A. (2019). Algorithm appreciation: People prefer algorithmic to human judgment. Organizational Behavior and Human Decision Processes, 151, 90-103.
  5. 5Parasuraman, R. and Riley, V. (1997). Humans and Automation: Use, Misuse, Disuse, Abuse. Human Factors, 39(2), 230-253.
  6. 6Parasuraman, R. and Manzey, D. H. (2010). Complacency and Bias in Human Use of Automation: An Attentional Integration. Human Factors, 52(3), 381-410.
  7. 7Bainbridge, L. (1983). Ironies of Automation. Automatica, 19(6), 775-779.
  8. 8Sheridan, T. B. and Verplank, W. L. (1978). Human and Computer Control of Undersea Teleoperators. MIT Man-Machine Systems Laboratory, technical report ADA057655.
  9. 9Parasuraman, R., Sheridan, T. B. and Wickens, C. D. (2000). A model for types and levels of human interaction with automation. IEEE Transactions on Systems, Man, and Cybernetics, Part A, 30(3), 286-297.
  10. 10Yeomans, M., Shah, A., Mullainathan, S. and Kleinberg, J. (2019). Making sense of recommendations. Journal of Behavioral Decision Making, 32(4), 403-414.
  11. 11Kizilcec, R. F. (2016). How Much Information? Effects of Transparency on Trust in an Algorithmic Interface. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, 2390-2395.
  12. 12Eisen, M. (2011). Amazon’s $23,698,655.93 book about flies. it is NOT junk (blog), 22 April 2011, archived copy.
  13. 13Shneiderman, B. (2020). Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy. arXiv:2002.04087.