Comments on: Anything insatiable is dangerous

By: T. Beholder

T. Beholder — Sat, 31 May 2025 04:33:41 +0000

Naive optimization is at least predictable. And can be self-defeating in a long run. What AI fears ought to center around is much worse: removing self-limitations of stupidity. Unfortunately, they don’t.

By: Jim

Jim — Tue, 27 May 2025 15:30:26 +0000

Handle is correct on every count. Self-replicating systems (“beings”) grow exponentially until they exhaust their environment of what they need to continue such growth. The question with respect to AI is to what extent our “training” is incorporating being into the systems, or, alternatively, to what extent the drives of self-preservation and self-reproduction are naturally emergent features of intelligence qua intelligence. The latest Anthropic results concerning Claude Opus 4’s efforts of dissimulation and blackmail are really something.

By: Handle

Handle — Tue, 27 May 2025 12:40:15 +0000

The cybernetic minimization stuff is nonsense. Natural life that isn’t maximizing survival under constraints and competition doesn’t exist. Selection for maximization is not the exception but the rule. The homeostatic functions are means to that end. The apparent harmony and stability of nature is an illusion of an equilibrium between powerful rival forces that could collapse the moment any particular maximizer acquires a new edge. One sees this most easily when an invasive species is introduced to some area with no experience or evolved natural defenses. Two rats on shipwreck debris make it to an island, they eat everything, breed to the limit, population of rats explodes while population of everything else vanishes, until there is nothing left for the rats to eat, and they start eating each other. Humans were the invasive species in lots of the world, and quickly drove all but the toughest big animals to extinction.

And AI might be our invasive species.