The most relevant aspect of Neal Stephenson‘s work to an audience of AI researchers was the Young Lady’s Illustrated Primer from his 30-year-old novel The Diamond Age:
At the beginning of this book we see a conversation between Lord Finkle-McGraw, who is an Equity Lord in a futuristic neo-Victorian society, and John Hackworth, an engineer who works in one of his companies.
Finkle-McGraw is a classic founder. He didn’t come from a privileged background, except insofar as having a stable family and a decent basic education confers privilege. But when he was young he was brilliant, ambitious, hard-working, and had a vision. He built that into something valuable and as a result became rich and powerful. As so often happens, he used his money to make life good for his children by sending them to the right schools, connecting them to the right people, and so on.
He wasn’t entirely happy with the results. His kids didn’t end up having the traits that had made him successful. He suspects it’s because they didn’t have to work hard and overcome obstacles. Now he has a granddaughter. He knows that the parents are going to raise this girl in the same way, with the same results. He can’t interfere in a heavy-handed way. But the parents can’t possibly object if he gives his granddaughter an educational book. So he commissions Hackworth to make the Young Lady’s Illustrated Primer, an interactive book that will adapt as the user grows and learns. This book is powered by molecular nanotechnology, but any present-day reader will immediately recognize it as an AI system.
As the plot unfolds, three copies of the Primer are made and bestowed on girls from very different backgrounds. In two cases the result is a sort of fizzle. The Primer works as it’s supposed to for a while, but these girls lose interest and set it aside. The third copy falls into the hands of a girl from an abusive and underprivileged background, and it ends up giving her close to superhuman abilities.
Thirty years on, I think I have enough distance on this to grade my performance. I’m happy with the fact that the Primer, as described in the novel, doesn’t invariably produce great results. That seems like a measured and realistic outcome. Nevertheless it’s clear that when I wrote this thing I was influenced by a strain of techno-utopian thinking that was widespread in the mid-1990s, when the Internet was first becoming available to a mass audience. In those days, a lot of people, myself included, assumed that making all the world’s knowledge available to everyone would unlock vast stores of pent-up human potential.
That promise actually did come true to some degree. It’s unquestionably the case that anyone with an Internet connection can now learn things that they could not have had access to before. But as we now know, many people would rather watch TikTok videos eight hours a day.
[…]
The gist of it is that the system we’ve traditionally used for evaluating students’ performance — homework and tests — just happens to be exquisitely vulnerable to being hacked by students who simply use conversational AI systems to do all the work for them. And they are doing so on a massive scale, to the point where conventional education has essentially stopped functioning. The only way to fairly evaluate how much a student has learned now is by marching them into a classroom with no electronics, handing them a pencil and a blank blue book, and assigning them an essay to write or a math problem to solve. Even this is impractical given that many students never really learned to write by hand.
[…]
This question sent me down a rabbit hole on the topic of self-reliance. After all, if AI-driven education does nothing more than make students even more reliant on AI, then it’s not education at all. It’s just a vocational education program teaching them how to be of service to AIs. The euphemism for this role is “prompt engineer” which seems to be a way of suggesting that people who feed inputs to AIs are achieving something that should be valorized to the same degree as designing airplanes and building bridges.
If such a system actually did its job it would have the paradoxical effect of making students less, rather than more, reliant on the use of AI technology.
[…]
My thought last week was that Self-Reliance might contain some wisdom applicable to the challenge of how to educate people in the modern world to rely upon their own knowledge and skill set rather than using AI all the time.
Reader, I did not find anything like that upon re-reading this essay. More the opposite. The overall drift of what Emerson is saying here — and he says it over and over — is that each mind is uniquely positioned to see certain insights. The self-reliant person shouldn’t ignore those merely because they don’t match the conventional wisdom. “The eye was placed where one ray should fall, that it might testify of that particular ray…God will not have his work made manifest by cowards….He who would gather immortal palms (i.e. be honored for great achievements) must not be hindered by the name of goodness, but must explore if it be goodness. Nothing is at last sacred but the integrity of your own mind.”
That is all intoxicating stuff for a smart young man who styles himself as a free thinker and nonconformist, which is why, when I was in my early twenties, I inhaled it like fentanyl fumes off hot foil. But during the same years as I was poring over this essay and jotting down quotes in my notebook, I was writing by far the worst novel I have ever written—a book that has never been published and never should be.
Emerson grew up in Boston, attended Boston Latin and Harvard, then traveled around Europe and visited England where he hung out with Wordsworth, Coleridge, and Thomas Carlyle. His brain was preloaded with the best knowledge base that could possibly have been given a young person of that era. He’d been trained to think systematically and rigorously and to express himself with great fluency in English and probably Latin and other languages as well.
So, yes, when an idea popped into Emerson’s head, chances are it was a pretty damned good one. His own advice about self-reliance was actually worth taking in his own case. And I’d guess that the audience for this essay was similarly well educated. By the time any young person happened upon Self-Reliance, they were probably 99% of the way to being an intellectually mature, highly capable person, and just wanted a bit of self confidence to follow through on good ideas that were coming into their heads—as a result of being that well educated and trained.
When the same advice falls on the ears of people who are not as well informed and not as good at thinking systematically, though, it’s rubbish.
When I first read Self-Reliance, only a few years had passed since the premier of the first Star Wars movie. There’s a pivotal moment in that film when Luke Skywalker is piloting his fighter through the trench on the Death Star, making his bombing run against impossible odds, and he hears Obi-Wan Kenobi’s voice in his head telling him to use the Force. Luke switches off his targeting computer to the consternation of the brass in the ops center. We all know the outcome. It’s a great moment in cinema, and it perfectly encapsulates a certain way of thinking emblematic of the 1970s late hippie scene: the seductive proposition that no one needs a targeting computer, that all we need to do is trust our feelings. Who doesn’t love to hear that? I loved hearing it from Ralph Waldo Emerson, and spent a couple of years of my life building a terrible novel on that foundation.
I have discussed The Diamond Age before.
He should suspect regression to the mean, which is to say, noisy breeding.
As for this:
There has never been a sharper K-shaped divergence in child development. Those whose infant minds were raped by the Algorithm will be the slaves of those who simply weren’t exposed.