The amateurs won — hands down. Their forecasts were more accurate more often, and the confidence they had in their forecasts — as measured by the odds they set on being right — was more accurately tuned.
The top 2%, whom Prof. Tetlock dubs “superforecasters,” have above-average — but rarely genius-level — intelligence. Many are mathematicians, scientists or software engineers; but among the others are a pharmacist, a Pilates instructor, a caseworker for the Pennsylvania state welfare department and a Canadian underwater-hockey coach.
The forecasters competed online against four other teams and against government intelligence experts to answer nearly 500 questions over the course of four years: Will the president of Tunisia go into exile in the next month? Will the gold price exceed $1,850 on Sept. 30, 2011? Will OPEC agree to cut its oil output at or before its November 2014 meeting?
It turned out that, after rigorous statistical controls, the elite amateurs were on average about 30% more accurate than the experts with access to classified information. What’s more, the full pool of amateurs also outperformed the experts.
The most careful, curious, open-minded, persistent and self-critical — as measured by a battery of psychological tests — did the best.
“What you think is much less important than how you think,” says Prof. Tetlock; superforecasters regard their views “as hypotheses to be tested, not treasures to be guarded.”
Most experts — like most people — “are too quick to make up their minds and too slow to change them,” he says. And experts are paid not just to be right, but to sound right: cocksure even when the evidence is sparse or ambiguous.
Tetlock explains the kto-kogo status quo of political forecasting:
Like many hardball operators before and since, Vladimir Lenin insisted politics, defined broadly, was nothing more than a struggle for power, or as he memorably put it, “kto, kogo?” That literally means “who, whom” and it was Lenin’s shorthand for “Who does what to whom?” Arguments and evidence are lovely adornments but what matters is the ceaseless contest to be the kto, not the kogo. It follows that the goal of forecasting is not to see what’s coming. It is to advance the interests of the forecaster and the forecaster’s tribe. Accurate forecasts may help do that sometimes, and when they do accuracy is welcome, but it is pushed aside if that’s what the pursuit of power requires. Earlier, I discussed Jonathan Schell’s 1982 warning that a holocaust would certainly occur in the near future “unless we rid ourselves of our nuclear arsenals,” which was clearly not an accurate forecast. Schell wanted to rouse readers to join the swelling nuclear disarmament movement. He did. So his forecast was not accurate, but did it fail? Lenin would say it did exactly what it was supposed to do.
Dick Morris — a Republican pollster and former adviser to President Bill Clinton — underscored the point days after the presidential election of 2012. Shortly before the vote, Morris had forecast a Romney landslide. Afterward, he was mocked. So he defended himself. “The Romney campaign was falling apart, people were not optimistic, nobody thought there was a chance of victory and I felt that it was my duty at that point to go out and say what I said,” Morris said. Of course Morris may have lied about having lied, but the fact that Morris felt this defense was plausible says plenty about the kto-kogo world he operates in.
(Hat tip to Steve Sailer.)
Are the superforcasters accurate enough to base real decisions on? The set of easily-measured variables often doesn’t overlap the set of important variables. Favourite recent example: consumer debt is going up. Easily measurable, but what we want to know is how much anguish excessive debt is causing. Have to measure lost utils – good luck with that.
Might well be that even superforecasters aren’t good enough to serve unless they have a bias, a propaganda angle, used to support their paymaster.