Isegoria

None of the leading hypotheses about the purpose of dreaming are convincing, Erik Hoel explains:

E.g., some scientists think the brain replays the day’s events during dreams to consolidate the day’s new memories with the existing structure. Yet, such theories face the seemingly insurmountable problem that only in the most rare cases do dreams involve specific memories. So if true, they would mean that the actual dreams themselves are merely phantasmagoric effluvia, a byproduct of some hazily-defined neural process that “integrates” and “consolidates” memories (whatever that really means). In fact, none of the leading theories of dreaming fit well with the phenomenology of dreams—what the experience of dreaming is actually like.

First, dreams are sparse in that they are less vivid and detailed than waking life. As an example, you rarely if ever read a book or look at your phone screen in dreams, because the dreamworld lacks the resolution for tiny scribblings or icons. Second, dreams are hallucinatory in that they are often unusual, either by being about unlikely events, or involve nonsensical objects or borderline categories. People who are two people, places that are both your home and a spaceship. Many dreams could be short stories by Kafka, Borges, Márquez, or some other fabulist. A theory of dreams must explain why every human, even the most unimaginative accountant, has within them a surrealist author scribbling away at night.

To explain the phenomenology of dreams I recently outlined a scientific theory called the Overfitted Brain Hypothesis (OBH). The OBH posits that dreams are an evolved mechanism to avoid a phenomenon called overfitting. Overfitting, a statistical concept, is when a neural network learns overly specifically, and therefore stops being generalizable. It learns too well. For instance, artificial neural networks have a training data set: the data that they learn from. All training sets are finite, and often the data comes from the same source and is highly correlated in some non-obvious way. Because of this, artificial neural networks are in constant danger of becoming overfitted. When a network becomes overfitted, it will be good at dealing with the training data set but will fail at data sets it hasn’t seen before. All learning is basically a tradeoff between specificity and generality in this manner. Real brains, in turn, rely on the training set of lived life. However, that set is limited in many ways, highly correlated in many ways. Life alone is not a sufficient training set for the brain, and relying solely on it likely leads to overfitting.

Common practices in deep learning, where overfitting is a constant concern, lend support to the OBH. One such practice is that of “dropout,” in which a portion of the training data or network itself is made sparse by dropping out some of the data, which forces the network to generalize. This is exactly like the spareness of dreams. Another example is the practice of “domain randomization,” where during training the data is warped and corrupted along particular dimensions, often leading to hallucinatory or fabulist inputs. Other practices include things like feeding the network its own outputs when it’s undergoing random or biased activity.

What the OBH suggests is that dreams represent the biological version of a combination of such techniques, a form of augmentation or regularization that occurs after the day’s learning—but the point is not to enforce the day’s memories, but rather combat the detrimental effects of their memorization. Dreams warp and play with always-ossifying cognitive and perceptual categories, stress-testing and refining. The inner fabulist shakes up the categories of the plastic brain. The fight against overfitting every night creates a cyclical process of annealing: during wake the brain fits to its environment via learning, then, during sleep, the brain “heats up” through dreams that prevent it from clinging to suboptimal solutions and models and incorrect associations.

The OBH fits with the evidence from human sleep research: sleep seems to be associated not so much with assisting pure memorization, as other hypotheses about dreams would posit, but with an increase in abstraction and generalization. There’s also the famous connection between dreams and creativity, which also fits with the OBH. Additionally, if you stay awake too long you will begin to hallucinate (perhaps because your perceptual processes are becoming overfitted). Most importantly, the OBH explains why dreams are so, well, dreamlike.

This connects to another question. Why are we so fascinated by things that never happened?

If the OBH is true, then it is very possible writers and artists, not to mention the entirety of the entertainment industry, are in the business of producing what are essentially consumable, portable, durable dreams. Literally. Novels, movies, TV shows—it is easy for us to suspend our disbelief because we are biologically programmed to surrender it when we sleep.

[…]

Just like dreams, fictions and art keep us from overfitting our perception, models, and understanding of the world.

[…]

There is a sense in which something like the hero myth is actually more true than reality, since it offers a generalizability impossible for any true narrative to possess.

Some people get famous for discovering one thing, Adam Mastroianni notes, like Gregor Mendel:

Some people get super famous for discovering several things, like Einstein and Newton.

So surely if one person came up with a ton of different things — say, correlation, standard deviation, regression to the mean, “nature vs. nurture,” questionnaires, twin studies, the wisdom of the crowd, fingerprinting, the first map of Namibia, synesthesia, weather maps, anticyclones, the best method to cut a round cake, and eugenics (yikes) — they’d be super DUPER famous.

But most people have never heard of Sir Francis Galton (1822-1911). Psychologists still use many of the tools he developed, but the textbooks barely mention him. Charles Darwin, Galton’s half-cousin, seems to get a new biography every other year; Galton has had three in a century.

Galton’s disappearance from collective memory would have been surprising to his contemporaries. Karl Pearson (of regression coefficient fame) thought Galton might ultimately be bigger than Darwin or Mendel:

Twenty years ago, no one would have questioned which was the greater man [...] If Darwinism is to survive the open as well as covert attacks of the Mendelian school, it will only be because in the future a new race of biologists will arise trained up in Galtonian method and able to criticise from that standpoint both Darwinism and Mendelism, for both now transcend any treatment which fails to approach them with adequate mathematical knowledge [...] Darwinism needs the complement of Galtonian method before it can become a demonstrable truth…

So, what happened? How come this dude went from being mentioned in the same breath as Darwin to never being mentioned at all? Psychologists are still happy to talk about the guy who invented “penis envy,” so what did this guy do to get scrubbed from history?

I started reading Galton’s autobiography, Memories of My Life, because I thought it might be full of juicy, embarrassing secrets about the origins of psychology. I’m telling you about it today because it is, and it’s full of so much more. There are adventures in uncharted lands, accidental poisonings, brushes with pandemics, some dabbling in vivisection, self-induced madness, a dash of blood and gore, and some poo humor for the lads. And, ultimately, a chance to wonder whether moral truth exists and how to find it.

Readers of this blog — certainly the ones of proper breeding — will already know what Galton did “wrong” to end up down the memory hole.

I felt a bit embarrassed that I’d never read his biography, but I doubt I’ve ever come across a physical copy.

For the last 60 years or so, Adam Mastroianni notes, science has been running an experiment on itself:

The experimental design wasn’t great; there was no randomization and no control group. Nobody was in charge, exactly, and nobody was really taking consistent measurements. And yet it was the most massive experiment ever run, and it included every scientist on Earth.

Most of those folks didn’t even realize they were in an experiment. Many of them, including me, weren’t born when the experiment started. If we had noticed what was going on, maybe we would have demanded a basic level of scientific rigor. Maybe nobody objected because the hypothesis seemed so obviously true: science will be better off if we have someone check every paper and reject the ones that don’t pass muster. They called it “peer review.”

This was a massive change. From antiquity to modernity, scientists wrote letters and circulated monographs, and the main barriers stopping them from communicating their findings were the cost of paper, postage, or a printing press, or on rare occasions, the cost of a visit from the Catholic Church. Scientific journals appeared in the 1600s, but they operated more like magazines or newsletters, and their processes of picking articles ranged from “we print whatever we get” to “the editor asks his friend what he thinks” to “the whole society votes.” Sometimes journals couldn’t get enough papers to publish, so editors had to go around begging their friends to submit manuscripts, or fill the space themselves. Scientific publishing remained a hodgepodge for centuries.

(Only one of Einstein’s papers was ever peer-reviewed, by the way, and he was so surprised and upset that he published his paper in a different journal instead.)

That all changed after World War II. Governments poured funding into research, and they convened “peer reviewers” to ensure they weren’t wasting their money on foolish proposals. That funding turned into a deluge of papers, and journals that previously struggled to fill their pages now struggled to pick which articles to print. Reviewing papers before publication, which was “quite rare” until the 1960s, became much more common. Then it became universal.

Now pretty much every journal uses outside experts to vet papers, and papers that don’t please reviewers get rejected. You can still write to your friends about your findings, but hiring committees and grant agencies act as if the only science that exists is the stuff published in peer-reviewed journals. This is the grand experiment we’ve been running for six decades.

The results are in. It failed.

[…]

Here’s a simple question: does peer review actually do the thing it’s supposed to do? Does it catch bad research and prevent it from being published?

It doesn’t. Scientists have run studies where they deliberately add errors to papers, send them out to reviewers, and simply count how many errors the reviewers catch. Reviewers are pretty awful at this. In this study reviewers caught 30% of the major flaws, in this study they caught 25%, and in this study they caught 29%. These were critical issues, like “the paper claims to be a randomized controlled trial but it isn’t” and “when you look at the graphs, it’s pretty clear there’s no effect” and “the authors draw conclusions that are totally unsupported by the data.” Reviewers mostly didn’t notice.

In fact, we’ve got knock-down, real-world data that peer review doesn’t work: fraudulent papers get published all the time.

[…]

When one editor started asking authors to add their raw data after they submitted a paper to his journal, half of them declined and retracted their submissions. This suggests, in the editor’s words, “a possibility that the raw data did not exist from the beginning.”

[…]

If you look at what scientists actually do, it’s clear they don’t think peer review really matters.

First: if scientists cared a lot about peer review, when their papers got reviewed and rejected, they would listen to the feedback, do more experiments, rewrite the paper, etc. Instead, they usually just submit the same paper to another journal.

[…]

Second: once a paper gets published, we shred the reviews. A few journals publish reviews; most don’t. Nobody cares to find out what the reviewers said or how the authors edited their paper in response, which suggests that nobody thinks the reviews actually mattered in the first place.

And third: scientists take unreviewed work seriously without thinking twice. We read “preprints” and working papers and blog posts, none of which have been published in peer-reviewed journals. We use data from Pew and Gallup and the government, also unreviewed. We go to conferences where people give talks about unvetted projects, and we do not turn to each other and say, “So interesting! I can’t wait for it to be peer reviewed so I can find out if it’s true.”

[…]

Lack of effort isn’t the problem: remember that our current system requires 15,000 years of labor every year, and it still does a really crappy job. Paying peer reviewers doesn’t seem to make them any better. Neither does training them.

He got some nasty comments and came up with some reasons why people got so nasty:

First: the third-person effect, which is people’s tendency to think that other people are susceptible to persuasion. I am a savvy consumer; you are a knucklehead who can be duped into buying Budweiser by a pair of boobs. I evaluate arguments rationally; you listen to whoever is shouting the loudest. I won’t be swayed by a blog post; you will.

[…]

And second: social dominance. Scientists may think they’re egalitarian because they don’t believe in hierarchies based on race, sex, wealth, and so on. But some of them believe very strongly in hierarchy based on prestige. In their eyes, it is right and good for people with more degrees, bigger grants, and fancier academic positions to be above people who have fewer of those things. They don’t even think of this as hierarchy, exactly, because that sounds like a bad word. To them, it’s just the natural order of things.

(To see this in action, watch what happens when two academic scientists meet. The first things they’ll want to know about each are are 1) career stage — grad student, postdoc, professor, etc., and 2) institution. These are the X and Y coordinates that allow you to place someone in the hierarchy: professor at elite institution gets lots of status, grad student at no-name institution gets none. Older-looking graduate students sometimes have the experience of being mistaken for professors, and professors will chat to them amiably until they realize their mistake, at which point they will, horrified, high-tail it out of the conversation.)

People who are all-in on a hierarchy don’t like it when you question its central assumptions. If peer review doesn’t work or is even harmful to science, it suggests the people at the top of the hierarchy might be naked emperors, and that’s upsetting not just to the naked emperors themselves, but also the people who are diligently disrobing in the hopes of becoming one. In fact, it’s more than upsetting — it’s dangerous, because it could tip over a ladder that has many people on it.