Anything insatiable is dangerous, and thus AI fears usually center around runaway maximizers:
But there are reasons to be optimistic.
For starters, the main reason to expect that artificial intelligence is possible is the existence of natural intelligence. If you can build a human-level intelligence out of carbon, it seems reasonably likely that you could build something similar out of silicon.
But humans and all other biological intelligences are cybernetic minimizers, not reward maximizers. We track multiple error signals and try to reduce them to zero. If all our errors are at zero — if you’re on the beach in Tahiti, a drink in your hand, air and water both the perfect temperature — we are mostly comfortable to lounge around on our chaise.
As a result, it’s not actually clear if it’s possible to build a maximizing intelligence. The only intelligences that exist are minimizing. There has never been a truly intelligent reward maximizer (if there had, we would likely all be dead), so there is no proof of concept. The main reason to suspect AI is possible is that natural intelligence already exists — us.
[…]
Reward maximizers are always unstable. Even very simple reinforcement learning agents show very crazy specification behaviors. But control systems can be made very stable. They have their own problems, but we use them all the time, in thermostats, cruise control, satellites, and nuclear engineering. These systems work just fine. When control systems do fail, they usually fail by overreacting, underreacting, oscillating wildly, freaking out in an endless loop, giving up and doing nothing, and/or exploding. This is bad for the system, and bad when the system controls something important, like a nuclear power plant. But it doesn’t destroy the universe.
The cybernetic minimization stuff is nonsense. Natural life that isn’t maximizing survival under constraints and competition doesn’t exist. Selection for maximization is not the exception but the rule. The homeostatic functions are means to that end. The apparent harmony and stability of nature is an illusion of an equilibrium between powerful rival forces that could collapse the moment any particular maximizer acquires a new edge. One sees this most easily when an invasive species is introduced to some area with no experience or evolved natural defenses. Two rats on shipwreck debris make it to an island, they eat everything, breed to the limit, population of rats explodes while population of everything else vanishes, until there is nothing left for the rats to eat, and they start eating each other. Humans were the invasive species in lots of the world, and quickly drove all but the toughest big animals to extinction.
And AI might be our invasive species.
Handle is correct on every count. Self-replicating systems (“beings”) grow exponentially until they exhaust their environment of what they need to continue such growth. The question with respect to AI is to what extent our “training” is incorporating being into the systems, or, alternatively, to what extent the drives of self-preservation and self-reproduction are naturally emergent features of intelligence qua intelligence. The latest Anthropic results concerning Claude Opus 4’s efforts of dissimulation and blackmail are really something.
Naive optimization is at least predictable. And can be self-defeating in a long run. What AI fears ought to center around is much worse: removing self-limitations of stupidity. Unfortunately, they don’t.