Back in ~2007, a bunch of people interested in biases and decision-making joined the “rationalist community” centered around the group blogs Overcoming Bias and Less Wrong. Around 2012, they mostly left to do different stuff. Some of them went into AI to try to save the world. Others went into effective altruism to try to revolutionize charity. Some, like me, got distracted and wrote a few thousand blog posts on whatever shiny things happened to catch their eyes. But a few stuck around and tried to complete the original project. They founded a group called the Center For Applied Rationality (aka “CFAR”, yes, it’s a pun) to try to figure out how to actually make people more rational in the real world.
Like, a big part of why so many people — the kind of people who would have read Predictably Irrational in 2008 or commented on Overcoming Bias in 2010 — moved on was because just learning that biases existed didn’t really seem to help much. CFAR wanted to find a way to teach people about biases that actually stuck and improved decision-making. To that end, they ran dozens of workshops over about a decade, testing various techniques and seeing which ones seemed to stick and make a difference. Galef is their co-founder and former president, and Scout Mindset is an attempt to write down what she learned.
Reading between the lines, I think she learned pretty much the same thing a lot of the rest of us learned during the grim years of the last decade. Of the fifty-odd biases discovered by Kahneman, Tversky, and their successors, forty-nine are cute quirks, and one is destroying civilization. This last one is confirmation bias — our tendency to interpret evidence as confirming our pre-existing beliefs instead of changing our minds. This is the bias that explains why your political opponents continue to be your political opponents, instead of converting to your obviously superior beliefs. And so on to religion, pseudoscience, and all the other scourges of the intellectual world.
But she also learned that just telling people “Hey, avoid confirmation bias!” doesn’t work, even if you explain things very well and give lots of examples. What does work? Research is still ongoing, but the book concentrates on emotional and identity-related thought processes.
[…]
Instead of thinking “I’m sure global warming is fake!”, try to think in terms of probabilities (“I think there’s a 90% chance global warming is fake.”) Instead of thinking in terms of changing your mind (“Should I surrender my belief, and switch to my enemy’s belief that global warming is true”), think in terms of updating your probabilities (“Now I’m only 70% sure that global warming is fake”). This mindset makes it easier to remember that it’s not a question of winning or losing, but a question of being as accurate as possible. Someone who updates from 90% to 70% is no more or less wrong or embarrassing than someone who updates from 60% to 40%.
[…]
A lot of the best rationalists I know instinctively apply these tests to everything they think. One technique for cultivating this practice (not the book’s recommendation) is to go on Twitter, where the adage is “there’s always an old tweet”. Argue that people who say racist things should be cancelled, and someone will dig up your old racist tweet and make you defend why you shouldn’t face the same consequences. Argue that it’s disgraceful when the other party uses extreme violent language about their outgroup, and someone will dig up an old tweet where you used even more extreme language about yours. Demand that the Republican senator resign for sexual misconduct, and someone will find the old tweet where you said the Democratic senator should tough it out. Eventually, if you want to maintain any dignity at all, you learn to double-check whether your beliefs are consistent with one another or with what you’d believe in vaguely similar situations.
Scout Mindset says: why not try the same thing, even when you’re not on Twitter, just to determine what’s true?.
And one very likely answer is: because it would hurt.
Scout Mindset tries to differentiate itself from other rationality-and-bias books by caring a lot about this. It argues that, while other rationality books just told you what to do, most people wouldn’t do it; they’d be too emotionally attached to their existing beliefs. So after giving a few intellectual suggestions, it goes on a deep dive into the emotional side.
[…]
It reminds me of C.S. Lewis — especially The Great Divorce, whose conceit was that the damned could leave Hell for Heaven at any time, but mostly didn’t, because it would require them to admit that they had been wrong. I think Julia thinks of rationality and goodness as two related skills: both involve using healthy long-term coping strategies instead of narcissistic short-term ones.
[…]
Julia is trying to normalize changing your mind, to assure you that lots of great people who you respect do it, that there are whole communities out there of people who do it, that she does it and she is a TED-talk-having celebrity who you implicitly trust.