A Summary of John List's book on making great ideas scale
TL;DR
John List's "The Voltage Effect" illustrates the pivotal elements necessary for successfully scaling ideas. He draws on examples from his time at Uber, and running early childhood intervention programmes in Chicago.
Taking inspiration from ‘the Anna Karenina principle’, he asserts that “Successfully scaled ideas are all alike, every idea that fails to scale fails in its own way.”
To scale a great idea, focus on:
Data integrity and audience analysis: Ensure scalability by validating initial successes beyond controlled settings with independent replication, and assessing the representativeness of your initial audience.
Preservation non-negotiables: Identify and preserve essential elements that are critical to an idea's successful growth. Use ‘backwards induction’ to identify whether it is possible to have the non-negotiables at scale. Focus on average talent and scalable systems over individual brilliance during testing, to maintain the project's core value across different markets and contexts.
Management of spillovers and externalities: Prepare for and manage spillovers as projects expand beyond original contexts. These potential unintended consequences could include ‘general equilibrium effects’ caused by large-scale changes not present during initial testing, social-side behavioural impacts as the ‘treatment group’ interact with others, or network effects which can cause wider momentum to gather as an initiative scales.
Cost management: While economies of scale can reduce per-unit costs, diseconomies can emerge due to increased operational complexity and exhausting ‘low hanging fruit’. Factor in a 10% contingency on operating costs, and assume the cost of hiring and retaining outstanding talent increases with scale.
Incentive design to reduce the reliance on individual people’s behaviour: Create incentives that scale by leveraging humans’ loss aversion and sociability. These can tap into people’s concerns about losing face, losing respect for themselves, and ‘losing’ in comparison to others’ performance. Financial incentives structured as an up-front payment which can potentially be lost for failure to deliver are particularly potent.
‘Marginal thinking’ and ‘optimal quitting’: Focus on the impact of the last unit of resource spent, explicitly consider alternatives, and recognise when continuing the current course of action is less beneficial. When any of the ‘5 warning signs’ in chapters 1-5 are present, avoid ‘ambiguity aversion’ and pivot to another initiative where there is more comparative advantage.
Scaling culture: Incentivise cross-functional collaboration. As scale increases, opportunity costs increase with the greater opportunities for partnership. The concept of 'coopetition'—a blend of cooperation and competition, within and between teams—can drive internal innovation and efficiency.
↓↓ Read on for detailed chapter summaries ↓↓
Chapter 1: “Dupers and False Positives”
False positives are misleading successes that appear promising in initial, controlled settings. They fail when scaled to a broader context, often resulting in significant financial and opportunity costs.
Inaccurate results can often stem from random statistical outliers and unrepresentative sampling, even in well-designed studies, as well as calculation errors.
Cognitive biases can drive misinterpretation - faulty conclusions from accurate data. These include:
Confirmation bias, where we view data through our own assumptions, and struggle to ‘see’. conclusions other than those expectations
Bandwagon bias, or social pressure. This isn’t just a danger with leaders' opinions - it applies any time anyone expresses preferences early on
Winner’s curse, the risk that, where there is financial competition over an asset of uncertain value, the ‘winner’ often overpays. Sunk cost fallacy means we then continue investing in losers. Only a bidder with ‘comparative advantage - a ‘secret sauce’ for scaling the asset - should be comfortable with winning an auction like this
The duper effect is more sinister, where misleading results have been created by falsification. 5% of academic economists admit falsifying data, while the Theranos debacle demonstrates this risk in the VC world. These are incentivised by the fact that those doing the research had something to gain from achieving ‘eye-catching’ results.
Requiring multiple independent replications, where people with no vested interest in your result achieve similar results with the same type of population, helps avoid false positives. Long term employee profit share schemes, and ‘Devil’s advocate’ deputies, teams or functions are also proposed.
Chapter 2 - “Know Your Audience”
Understanding the current audience is essential before attempting to scale. Is it representative of the wider population, or limited to a specific demographic or context? For example, utility companies showing customers usage relative to their neighbours dramatically lowered waste… but this only scaled to environmentally conscious places.
Selection bias can drive unrepresentative initial samples. These form where those most likely to benefit from the product opt into initial pilots. These can also be the result of extrapolating assumptions about WEIRD people (Western, Educated, Industrialised, Rich, Democratic) as universal, rather than culturally specific, realities.
Effective membership scaling via promotions needs a favourable ratio between two groups of their audiences. The first are incentivised to buy even more products because the deal is good value - a win for them and the seller. The second only buy as much as they did before, but pay less for it. To scale profitably, the balance must favour the latter over the former.
Tactics for broadening audience appeal include:
Ensuring the datasets that are being extrapolated from are reliable, including by conducting trials with multiple statistically significant, randomised populations. Better still, the evidence would be drawn from ‘natural field experiments’, gathering data from activities that are already occurring, rather than participants opting in
Conducting conjoint analysis and customer surveys to work out what services the target segment would pay for
Allowing for adaptability of the idea if possible, to ensure it resonates with a broader audience and its varied needs. For example, the ‘K-mart Blue Light Special’ flash sale worked brilliantly when each store manager could tailor it to what inventory they had unsold, and where their store was… but failed when set in advance, at the national HQ
Driving down production / distribution costs, or selecting more attractive markets if adaptability of the offering (e.g. expanding the product range) is too expensive to scale
Chapter 3: "Is It the Chef or the Ingredients?"
Understanding the non-negotiable elements of an initial success is crucial for scaling. What contextual factors, or specific individuals were critical to this early success? The non-negotiable elements of the initiative must be maintained to preserve the project's core value ("voltage") as it scales.
‘Non-negotiables’ related to the unique capabilities of individuals ("the chef") are potentially deadly for scaling the project, rather than scalable processes and systems ("the ingredients"). Attempting to replicate personal charisma and exceptional skills across a larger operation often leads to a dilution of quality, ‘mission drift’, or other failures. Lines of code are hugely scalable, and brand recognition can be, but outstanding human talent is not.
‘Backward induction’ is a technique that forces consideration of whether the essential, non-negotiable elements are available at scale, early in the design stage. This is called ‘reverse engineering’ elsewhere: asking ‘what would need to be true’).
Methods for ensuring the organisation’s teams and users maintain fidelity to the non-negotiables at scale include user-centred technology design, building constraints into initial pilots, and retaining the project’s original instigators within implementation teams:
To preserve ‘voltage', users must experience the benefits that will keep them engaged in the long term as soon as possible. Building and testing for the real-world behaviour you will encounter at scale, over idealistic personas (potentially informed by early adopters) is key: assume users will undo default settings, ignore notifications and disobey instructions. Technology interventions should make compliance with the ‘non-negotiables’ as easy as possible.
Pilots should be designed with constraints that consider ‘what would need to be true’ for success. For instance, if a project requires 50,000 teachers at scale, the design should not test with, or assume the availability of, exceptional teachers. Instead, its pilots and design should plan for scaling with a larger number of average performers, as exceptional performers will be harder to find as time goes on.
To guard against mission drift, ensure those who are implementing the scaling deeply understand the rationale. The better they understand the ‘why’, the more they will adhere to it. One way to do this is to include people with a personal interest in maintaining fidelity at scale in the implementation team e.g. the researcher who discovered the scientific breakthrough
Chapter 4: “Spillovers”
"Spillovers" are the unintended consequences —both positive and negative— of scaling initiatives beyond the context in which they were initially tested. In simple terms, unexpected knock-on effects have much more likelihood of occurring when scaling to lots of people. These spillovers can range from beneficial network effects, where the value of a service increases as more users engage, to detrimental outcomes such as increased traffic congestion from popular ride-sharing services. The Peltzman Effect is one example, where initiatives to enhance safety lead to the target audience engaging in more risky behaviours, counteracting the intended benefits (see also ‘moral hazard’ in an insurance context).
Large-scale changes - not present during initial testing - can cause ‘general equilibrium effects’. These effects, unnoticed or minimal on a small scale, occur and become visible as markets adjust in other areas to accommodate for the new supply or demand condition. These potentially undo the initial advantages of a scaled idea. For example, improving Uber drivers’ pay later led to more drivers joining, leading to an oversupply of drivers, thus reducing the number of journeys each driver received, negating the pay increase's impact. General equilibrium effects require us to think about the ‘second order impacts’ of scaling - what ‘externalities’ in the surrounding ecosystem might change as a result?
Social-side behavioural spillovers can impact scaling positively or negatively. These can occur directly within the treatment groups, where participants spur one another on. They can also pass on indirectly, where the others learn to emulate the benefit acquired by the treatment group (e.g. the skills taught in an education programme) by interacting with them socially:
List uses a case study about a team gathering charitable donations to illustrate a behavioural spill-over called ‘resentful demoralisation’. In the team, members who performed at the same level as higher paid peers continued to do so, until they found out what the others were paid. On the other hand, wage transparency is positive where it is disclosed for people’s managers, as it gives trust that they will be paid more in future.
The ‘John Henry effect’ functions in the opposite direction, where people in a control group put more effort in than normal because they know they are being compared to those in the treatment group.
Network effect spillovers (or ‘network externalities’) describe the wider momentum that gathers as an initiative scales. In the context of a digital service, this parabolic - rather than linear - growth can lead to ‘lock-in’, due to the fact that the value of a network disproportionately increases as more users engage (see also Metcalfe’s Law). In the context of public health, spillover effects can also increase ‘voltage’. For example, while immunising one person helps protect their health, at scale this can also provide ‘herd immunity’.
Chapter 5: “The Cost Trap”
Economies of scale can reduce per-unit costs as production increases, allowing ‘voltage gains’. These help pay off high up front fixed costs (e.g. R&D, infrastructure). The less spent on upfront costs, the less needs to be charged to users, likely increasing demand. Offsetting up-front costs can become a crucial accelerators (e.g. crude oil companies sell their waste byproduct - natural gas - as an additional source of revenue)
Diseconomies of scale can emerge due to increased operational complexity and exhausting ‘low hanging fruit’, resulting in ‘voltage drops’. As well as the impact of managing more locations or employees, because quality resources (e.g. skilled teachers) become harder to find the more you scale - and therefore become more costly - these can erode profitability.
List advises on forecasting operating costs to be 10% higher than initial calculations and aligning them against projected revenues to determine the breakeven point. Where substantial scale economies are anticipated, offering a discount will likely be worthwhile, as margins will improve over time without having to raise the price (e.g. freemium software). Knowing the ‘choke price’ - where demand drops to zero - and running some tests to work out how pricing influences demand, is crucial.
The cost of hiring and retaining outstanding talent increases with scale, further demonstrating the "Chef vs. Ingredients" theme, and the importance of backwards induction. However, while it makes sense to build cost assumptions based on hiring ‘average’ people in the scaled part of the operation, paying for exceptional talent which can create scalable impact is worthwhile (e.g. an outstanding software developer).
Chapter 6: "Incentives That Scale"
List assets that “If we get incentives right, character becomes irrelevant”. Relying on humans restricts scaling from both a quality and cost perspective. However, the right incentives can remove the reliance on specific individuals, and motivate desired behaviours from people, at scale. Properly designed incentives, he argues, can scale almost infinitely
Loss aversion can be leveraged to create powerful incentives. Humans tend to avoid losses more than they pursue equivalent gains. Effective incentives often involve making the potential losses of not engaging in desired behaviours appear more significant than the gains from alternative actions. These come in three forms:
Social dynamics, where the threat of a loss of social standing with others motivates action to adhere to desired norms (e.g. The Dominican Republic threatening to publicly publish the details of tax avoiders proved to be an effective, scalable incentive). The Hawthorne Effect is a related phenomenon, where changes in behaviour can occur merely due to the awareness of being observed. This underscores the importance of environmental factors in designing incentives.
Self-image, where incentives tap into internal pressure, leveraging individuals' self-perception (e.g. Virgin Atlantic pilots saved fuel when given a report and a target, as they responded to their view of themselves as responsible professionals being challenged. The pro-social financial incentive provided - a donation to charity - didn't make any difference, and the study increased their job satisfaction). Even requiring people to report their voting behaviour, even though they can lie, improves turnout.
Social comparison, where even anonymous comparisons with others can drive behaviour by triggering aversion to social loss. People innately want to ‘keep up with the Jones’ rather than fall behind. (e.g. Reduced energy consumption was achieved when utility companies shared a smart meter report anonymously comparing customers to other households. This effect persisted long after the direct monitoring had ceased - repeated messaging can be counterproductive). This can also be achieved at a divisional or organisational level too, by subscribing to an external benchmark.
Financial incentives structured as a ‘clawback’ can be particularly potent. This is an up-front payment which can potentially be lost for failure to deliver, rather than a bonus rewarding good behaviour that has just taken place. These require careful design to ensure the correct data is available, to avoid unethically setting unrealistic targets, and to avoid applying them to populations with repeated exposure to loss (e.g. asset traders, who begin to encode loss in a different part of the brain).
The timeliness of incentives is crucial. For instance, while the rewards of studying hard during school years are massive, the rewards are often too far in future to motivate children as they should.
Chapter 7: "Revolution on the Margins"
Cost-benefit calculations based on averages do not take diminishing returns into consideration. Use ‘marginalism’ instead, focusing on the impact of the final pound spent. The value of the final unit paid for (its marginal utility) is rarely the same as the average of the value of all the units. Instead, focusing on the impact of the last pound spent allows decision-makers to identify the point at which investments yield diminishing returns, and optimise accordingly. List uses doughnuts as an illustration: the marginal utility of consuming additional doughnuts decreases to the point of detriment - the first one tastes great, the fourth might make you sick!
These effects also apply to human resources. List gives an example of a factory experiencing diminishing marginal productivity, in a seasonal industry. Workers hired later in the year were each less productive than the last. To make matters worse, the assembly line could only go at the speed of the slowest person, magnifying the impact.
Disregard past costs that cannot be recovered to help avoid sunk cost bias. This principle uses marginal thinking to focus on how best to allocate the next available pound, rather than on the spending, correct or otherwise, that created the current situation.
A healthy culture - with strong psychological safety - is needed to effectively implement marginal thinking. This environment enables individuals to acknowledge past mistakes unearthed via this thinking without fear, thus improving future resource allocation.
Chapter 8: “Quitting is for winners”
‘Optimal quitting’ means minimising opportunity costs by actively considering alternatives. Our cognitive biases often mean we fail to consider viable alternatives unless they are explicitly presented to us. List gives the example of choosing between a $325 baseball bat and a $200 one - the decision becomes easier when it’s presented as a $325 bat or $200 bat with a $125 glove. This can be combined with marginal thinking, framing a choice as ‘What is the opportunity cost of putting the next pound here rather than somewhere else?’
Most of all, we tend not to think of opportunity cost of time. ‘Optimal quitting’ includes recognising that the sooner you give up, the lower the opportunity cost you’ll pay in time and money.
‘Affective forecasting’ hampers our ability to optimally quit: people often overestimate how future outcomes will make them feel based on their current emotions, skewing rational decision-making.
‘Ambiguity aversion’ is a cognitive bias where fear of the unknown can prevent beneficial changes. A Freakonomics ‘coin flip’ study showed that for people who were ‘on the fence’ with a big decision, those who’d made a change rather than continuing were more likely to be happier 2 and 6 months later. This suggests that sometimes the simple act of making a change can lead to greater satisfaction, though we often fear it in the moment.
Grit is the emotional resilience to quit judiciously rather than doggedly persisting in unfruitful endeavours. It's about losing small battles to win the war.
When any of the 5 warning signs in chapters 1-5 are present, pivot to something else where you have more comparative advantage, where you can be the best at something that enough people need and want. As Reid Hoffman put it “It’s easy to kill a product that is failing; it’s much harder, and more strategic, to kill one that lacks the potential to truly scale.”
We cannot be sure we have ‘optimally quit’ as the counterfactual is not available. The closest proxy is observing a competitor that chooses the alternative route, and compare their results to whatever we spent the intervening time doing instead.
Chapter 9: "Scaling culture"
A culture of deep trust is a powerful factor in enabling organisations to scale; how people work influences whether this is achieved. List draws heavily on a study of two different Brazilian fishing villages, to show collective work practices rather than individual work practices result in significant differences in other contexts: more trust, cooperation, and contribution to the greater good. The way people work impacts whether they prize trust or suspicion, individualism or collectivism, fear or security, workaholism or ‘work life balance’.
Maintaining performance in a ‘meritocracy’ hinges on trust that leaders will assess efforts fairly - this is hard to achieve at scale. More scalable values need to be instituted. Studies show that managers in ‘meritocratic’ organisations - based on individual achievement - are less prone to self-examination on gender and race bias. This is because they are convinced that the ‘meritocracy’ in which they operate works.
As organisations scale, designing incentives to encourage cross-functional collaboration becomes crucial; opportunity costs increase with the greater opportunities for partnership. A competitive culture that served well in the early years can be a disaster at scale. The concept of 'coopetition'—a blend of cooperation and competition, within and between teams—can drive internal innovation and efficiency, as evidenced by practices seen at companies like Netflix.
Diverse hiring leads to better innovation and resilience due to cognitive diversity. To achieve this at scale:
Avoid equality and diversity statements on job listings as a study showed these discourage racial minorities from applying, due to fears of tokenism. This was more pronounced the more senior the candidate, and the more white their city
Deploying CSR messaging in recruitment as it tends to enhance the candidate pool. However, in existing employees this can both improve performance and result in moral licence (i.e. a negative spillover effect, where some team members use this perception of the good they have been involved in to justify bad behaviour)
State salaries are negotiable. Men are 8x more likely to negotiate on salary offers without such a statement. Worse still, low performing men tend to bargain the most when the salary is vague, while the most exceptional women tend to hold back! Increased numbers of women apply when a salary is stated as negotiable, and negotiate at least as hard as men in that scenario.
Have longer candidate shortlists to address systemic bias of informal network based recruitment, and the implicit bias of selecting ‘top of mind candidates’. Once given the chance, the more diverse candidates are more likely to secure the position.
As scale increases, the opportunities for mistakes increase - how these are dealt with matters. List references a piece of Uber ‘identical twins’ research on mistakes and apologies (i.e. analysing the outcomes of pairs of people who had similar characteristics until one had a bad experience). More contrite apologies led to better outcomes (“I’m sorry for what i did” not “I’m sorry you feel that way”). Including a discount was better regardless of the wording, as it demonstrated ‘putting your money where your mouth is'. However, apologising for 3 or more mistakes in 90 days was worse than not saying anything at all. List asserts that apologies should be reserved for significant, unforeseen errors to maintain credibility.
Recap
List underscores the importance of the 'five vital signs':
Avoiding false positives from early tests,
Ensuring representativeness of future situations and populations during pilots,
Making sure non-negotiables can be preserved,
Anticipating spillovers, and
Managing costs.
These components are critical in determining whether a concept will maintain its effectiveness when expanded.
Concepts to improve scalability include:
Incentive design which leverages humans’ loss aversion and sociability,
‘Optimal quitting’ to minimise opportunity cost,
Marginal thinking to direct focus on the returns from the most recent spend rather than average spend, and
A culture of 'coopetition' to optimise cross-functional interactions.
Two reflections on the COVID-19 pandemic are also offered by the author:
“If any enterprise has weaknesses it will be revealed at scale”, and
“Scalable ideas and solutions remain our most valuable resource for addressing the world’s most urgent problems… There is only one way to meaningfully change the world - at scale.”
If you're interested in learning more, Tim Ferriss' interview with John List is excellent, and you can buy Voltage Effect here.
Kommentarer