Superforecasting: The art and science of prediction
“What matters most is how the forecaster thinks…superforecasting demands thinking that is open-minded, careful, curious, and – above all – self-critical.” Philip Tetlock
Super Forecasting: The art and science of prediction
Did you know that right now hundreds of people around the world are predicting what will happen in weeks, months and years from now to gain nothing more than personal satisfaction?
In a fascinating study funded by the US Government’s intelligence agency, the Good Judgement project set challenges that anyone with an interest in forecasting could contribute to. Still running as a public competition, the website sets questions as random as “Before 31 December 2017, will President Trump sign legislation removing the Affordable Care Act’s individual mandate to purchase health insurance?” to “When will Fitch, Moody’s or S&P next downgrade China’s long term local or foreign currency issuer ratings?” The questions helped identify people who were highly skilled at forecasting, and then put them into superforecasting teams to improve their performance even more. The results showed that superforecaster volunteers performed as much as 78% better than control groups, and 30% better than expert academics or professional intelligence analysts with access to classified data. Designed by author professor Philip Tetlock, the study has identified elite forecasters who are not well known on TV or to business or governments and who don’t necessarily possess outstanding academic achievements. These often humble individuals come from a wide range of backgrounds, sometimes retired or homemakers, but their unique approach to forecasting makes them far more successful than even the highest paid experts.
Tetlock’s work first came to prominence when he published the results of a twenty year study analysing the predictions of 284 experts – those who had 12 years or more experience in studying political and economical developments. His main conclusion was that the average expert is no better at predicting the future than a chimp throwing darts. Although he has now tired of the joke (it became a shorthand for saying all forecasting was useless), he does believe that any intelligent, open-minded and hardworking person can cultivate skills to see into the future. For leaders looking to move from good to superb, Tetlock argues that using the superforecasting model to help make decisions will make organisations smarter, more adaptable and effective.
Accuracy in choice of words (or much better, in actual numbers/probabilities) and details is important in forecasting. Steve Ballmer, CEO of Microsoft in 2007, made the now infamous forecast that “There’s no chance the iPhone is going to get any significant market share. No chance.” While that remark has made it into the ‘ten worst tech predictions of all time’ hall of shame, closer inspection of his words in context tells a different story. He was talking about a global market, not just the US, and had balanced it by saying that Apple may still make a lot of money. In fact, even today, Apple’s smartphones still only have around 13% market share worldwide.
Tetlock and Gardner’s 11 commandments for improving your own forecasting skills include:
- Triage: Focus on questions where your hard work is likely to pay off. For example, don’t waste time on asking who will win the elections in 2028 – that’s impossible to predict. Predicting what might happen in one year’s time is much more achievable
- Break seemingly intractable problems into tractable sub-problems: Decompose the problem into its knowable and unknowable parts. For example, Peter Backus, a lonely man in London looking for love guesstimated the number of potential female partners (approx. 6 million), winnowed the number down by how many might be single (50%), the proportion in the right age range (about 20%), the proportion he finds attractive (5%) and the proportion likely to be compatible with him (10%). Conclusion? Roughly 26 eligible women – a daunting but not impossible search task
- Strike the right balance between inside and outside views: Superforecasters are in the habit of asking the outside-view question: How often do things of this sort happen in situations of this sort?
- Strike the right balance between under and overreacting to evidence: Keep updating your prediction as time moves on and new evidence comes to light even if it is just by tiny margins
- Look for the clashing causal forces at work in each problem: Consider all counter arguments. Don’t let your belief about, for example, whether military action works, influence whether you think it will work in a very specific situation
- Strive to distinguish as many degrees of doubt as the problem permits but no more: Few things are certain and maybe isn’t informative. Nuance matters – practice translating hunches into numeric probabilities
- Strike the right balance between under- and overconfidence, between prudence and decisiveness: Superforecasters understand the risks both of rushing to judgement and of dawdling too long near a verdict of maybe
- Look for the errors behind your mistakes but beware of rearview-mirror hindsight biases: Own your failures! And don’t forget to do post mortems on your successes too. Not all successes imply your reasoning was right – you may just have lucked out
- Bring out the best in others and let others bring out the best in you: Master the fine art of team management, especially understanding the perspective of others, precision questioning and constructive confrontation
- Master the error-balancing bicycle: Learning requires doing – superforecasting is the product of deep, deliberate practice
- Don’t treat commandments as commandments – It is impossible to lay down binding rules – superforecasting requires constant mindfulness
In uncertain times, accurately predicting what might happen becomes ever more important. To improve your organisation’s future prospects, why not learn from the best at the coalface of forecasting, whatever area of life you choose to focus your crystal ball on.
About the authors
Philip Tetlock is Leonore Annenberg University Professor of Psychology at the University of Pennsylvania. He is the author of several books on politics and psychology, including Counterfactual Thought Experiments in World Politics and the award-winning Expert Political Judgment: How Good Is It? How Can We Know?
Dan Gardner is a journalist, author and lecturer. He is the best-selling author of Future Babble: Why Expert Predictions Fail and Why We Believe them Anyway and Risk: The Science and Politics of Fear, which was published in 11 countries and 7 languages. He lives in Ottawa, Canada.
“The techniques and habits of mind set out in this book are a gift to anyone who has to think about what the future might bring. In other words, to everyone.” The Economist
“A terrific piece of work that deserves to be widely read … Highly recommended.” Independent
“Full of excellent advice – it is the best thing I have read on predictions … Superforecasting is an indispensable guide to this indispensable activity.” The Times
“Fascinating and breezily written.” Sunday Times
“Superforecasting is a very good book. In fact it is essential reading.” Management Today
Author: Philip Tetlock and Dan Gardner