For the last 60 years or so, Adam Mastroianni notes, science has been running an experiment on itself:
The experimental design wasn’t great; there was no randomization and no control group. Nobody was in charge, exactly, and nobody was really taking consistent measurements. And yet it was the most massive experiment ever run, and it included every scientist on Earth.
Most of those folks didn’t even realize they were in an experiment. Many of them, including me, weren’t born when the experiment started. If we had noticed what was going on, maybe we would have demanded a basic level of scientific rigor. Maybe nobody objected because the hypothesis seemed so obviously true: science will be better off if we have someone check every paper and reject the ones that don’t pass muster. They called it “peer review.”
This was a massive change. From antiquity to modernity, scientists wrote letters and circulated monographs, and the main barriers stopping them from communicating their findings were the cost of paper, postage, or a printing press, or on rare occasions, the cost of a visit from the Catholic Church. Scientific journals appeared in the 1600s, but they operated more like magazines or newsletters, and their processes of picking articles ranged from “we print whatever we get” to “the editor asks his friend what he thinks” to “the whole society votes.” Sometimes journals couldn’t get enough papers to publish, so editors had to go around begging their friends to submit manuscripts, or fill the space themselves. Scientific publishing remained a hodgepodge for centuries.
(Only one of Einstein’s papers was ever peer-reviewed, by the way, and he was so surprised and upset that he published his paper in a different journal instead.)
That all changed after World War II. Governments poured funding into research, and they convened “peer reviewers” to ensure they weren’t wasting their money on foolish proposals. That funding turned into a deluge of papers, and journals that previously struggled to fill their pages now struggled to pick which articles to print. Reviewing papers before publication, which was “quite rare” until the 1960s, became much more common. Then it became universal.
Now pretty much every journal uses outside experts to vet papers, and papers that don’t please reviewers get rejected. You can still write to your friends about your findings, but hiring committees and grant agencies act as if the only science that exists is the stuff published in peer-reviewed journals. This is the grand experiment we’ve been running for six decades.
The results are in. It failed.
[…]
Here’s a simple question: does peer review actually do the thing it’s supposed to do? Does it catch bad research and prevent it from being published?
It doesn’t. Scientists have run studies where they deliberately add errors to papers, send them out to reviewers, and simply count how many errors the reviewers catch. Reviewers are pretty awful at this. In this study reviewers caught 30% of the major flaws, in this study they caught 25%, and in this study they caught 29%. These were critical issues, like “the paper claims to be a randomized controlled trial but it isn’t” and “when you look at the graphs, it’s pretty clear there’s no effect” and “the authors draw conclusions that are totally unsupported by the data.” Reviewers mostly didn’t notice.
In fact, we’ve got knock-down, real-world data that peer review doesn’t work: fraudulent papers get published all the time.
[…]
When one editor started asking authors to add their raw data after they submitted a paper to his journal, half of them declined and retracted their submissions. This suggests, in the editor’s words, “a possibility that the raw data did not exist from the beginning.”
[…]
If you look at what scientists actually do, it’s clear they don’t think peer review really matters.
First: if scientists cared a lot about peer review, when their papers got reviewed and rejected, they would listen to the feedback, do more experiments, rewrite the paper, etc. Instead, they usually just submit the same paper to another journal.
[…]
Second: once a paper gets published, we shred the reviews. A few journals publish reviews; most don’t. Nobody cares to find out what the reviewers said or how the authors edited their paper in response, which suggests that nobody thinks the reviews actually mattered in the first place.
And third: scientists take unreviewed work seriously without thinking twice. We read “preprints” and working papers and blog posts, none of which have been published in peer-reviewed journals. We use data from Pew and Gallup and the government, also unreviewed. We go to conferences where people give talks about unvetted projects, and we do not turn to each other and say, “So interesting! I can’t wait for it to be peer reviewed so I can find out if it’s true.”
[…]
Lack of effort isn’t the problem: remember that our current system requires 15,000 years of labor every year, and it still does a really crappy job. Paying peer reviewers doesn’t seem to make them any better. Neither does training them.
He got some nasty comments and came up with some reasons why people got so nasty:
First: the third-person effect, which is people’s tendency to think that other people are susceptible to persuasion. I am a savvy consumer; you are a knucklehead who can be duped into buying Budweiser by a pair of boobs. I evaluate arguments rationally; you listen to whoever is shouting the loudest. I won’t be swayed by a blog post; you will.
[…]
And second: social dominance. Scientists may think they’re egalitarian because they don’t believe in hierarchies based on race, sex, wealth, and so on. But some of them believe very strongly in hierarchy based on prestige. In their eyes, it is right and good for people with more degrees, bigger grants, and fancier academic positions to be above people who have fewer of those things. They don’t even think of this as hierarchy, exactly, because that sounds like a bad word. To them, it’s just the natural order of things.
(To see this in action, watch what happens when two academic scientists meet. The first things they’ll want to know about each are are 1) career stage — grad student, postdoc, professor, etc., and 2) institution. These are the X and Y coordinates that allow you to place someone in the hierarchy: professor at elite institution gets lots of status, grad student at no-name institution gets none. Older-looking graduate students sometimes have the experience of being mistaken for professors, and professors will chat to them amiably until they realize their mistake, at which point they will, horrified, high-tail it out of the conversation.)
People who are all-in on a hierarchy don’t like it when you question its central assumptions. If peer review doesn’t work or is even harmful to science, it suggests the people at the top of the hierarchy might be naked emperors, and that’s upsetting not just to the naked emperors themselves, but also the people who are diligently disrobing in the hopes of becoming one. In fact, it’s more than upsetting — it’s dangerous, because it could tip over a ladder that has many people on it.
A useful book on this topic is “Science Fictions: How Fraud, Bias, Negligence, and Hype Undermine the Search for Truth“, by Stuart Ritchie (2020).
Almost anyone who has ever been involved in “peer review” has a large dose of contempt for the process. It seems the underlying cause is what President Eisenhower warned about in his long-ago Farewell Speech — the baleful influence of government funding. Publish-or-perish researchers need to toe the political line to get grants, and then they need to publish to get more grants. Meanwhile, universities dependent on government money need a stable of academics with stacks of publications.
The fact that most publications are hardly ever read and do not meaningfully advance science does not matter.
Solution is to kill the government funding.
Some years ago I came to comprehend the meaning of Franklin Delano Roosevelt’s greatest quote, which will forever live in infamy, “In politics, nothing happens by accident. If it happens, you can bet it was planned that way.” This is no superficial message; Mr. Roosevelt was a cunning cripple bastard, a shark among men, and he knew that the world was run by supremely intelligent men possessed of great powers of dissimulation.
When one comes to terms with the fact that things more or less are as they was meant to be by the men with the power to make them be, one begins to appreciate the fact that all effects have causes, and that most causes are rooted, ultimately, in the acquisition or retention by such men of money, power, or money and power.
It follows that, only by blinding ourselves to the stated intentions of a policy and observing the actual outcomes of the policy, can we draw appropriate conclusions as to the purpose of the policy. In this case, the question of peer review, it is observed that, after World War II, a newly-spun social “tradition” or “practice” or “construct” was grafted onto the scientific establishment.
What was the nature of its implementation? Was it applied selectively, with experimental and control groups, and observed scientifically, to determine whether it yielded results superior or inferior? No: it was simply imposed, everywhere the same, all at once.
What were its outcomes? We don’t know, scientifically, because it wasn’t applied scientifically—from our perspective. We can only follow our intuitive powers of reason.
But we can be reasonably confident that the outcomes are about what were intended by those who engineered the underlying incentive structure.
“We are governed, our minds are molded, our tastes formed, our ideas suggested, largely by men we have never heard of.” — Edward Bernays
Gavin Longmuir,
Thanks for the book recommendation. I got a copy and started reading it. The author starts by trashing Bem and applauding Wiseman. I distrust Wiseman so I might be biased against the author. But of course, Wiseman and the author are probably biased against Bem.
In 2016, Bem recanted his claims. It may be that Bem’s analysis of his data really was as bad as Ritchie claims. Certainly we can say that Bem should have gathered more data — we can always say that about any experiment. Unfortunately the upshot is to give Wiseman more talking points. This does not mean that Wiseman is correct.
So, Jim, what you’re saying is that peer review is a plot by the powerful to… what? Make scientists dependent on government funding, and ultimately on the whims of the people in charge?
At the same time, you claim that we don’t know what the outcome of peer review has been or will be. If we’re to ‘look to the outcome of a policy, rather than its stated objective’, well, we have to know what that outcome is. You are suggesting that peer review was a conspiracy, but that nobody knows, exactly, what that conspiracy set out to achieve.
So, which is it? It’s a plot to achieve a very specific outcome, or it was a rushed experiment, the outcome of which we do not know?
Vague conspiratorial murmurings are best avoided for this reason. They offer an attractive way to repackage the world, and make it simpler, so that every effect has a cause that is human in origin — rather than simply accepting that, most of the time, we don’t know what the fuck we’re doing, and unintended consequences rule our lives. Which seems to be the case here.
The government did push peer review, but they didn’t have some long-term vision for how it would make the scientific class subservient to the political class. All the time we unleash things into the world that exceed our control or even understanding. The idea that this was just a hamfisted, unmethodical graft onto science, because the government wanted to make sure it was giving its money to credible researchers, seems like the more natural explanation. It sounded good, and failed in practice. Common occurrence.
And anyway I find you hard to understand. On another post, you were chastising another person for saying essentially the same thing that you are saying here. It was a long and funny exchange, but still.
David, I thank you sincerely for your earnest attempt to make sense of my posting, but if you are to understand anything, understand mainly that we are ruled by cunning men, the all-importance of cui bono, and that I could be a dog.