One day in June, 2025, a group of self-driving military boats lined up for a test event at Channel Islands Harbor Marina, a mile north of Port Hueneme Naval Base:
The boats were part of the Replicator program, which was then two years old and less than two months away from the official deadline to deliver thousands of maritime and air drones. The pressure was building.
Things began with support vessels towing autonomous boats out to sea; the drones’ engines were set to neutral and their autonomy mode turned off. The test focused not so much on the vehicles themselves—known as global autonomous reconnaissance crafts, or GARCs—as on the software that allowed them to function on their own. Two separate companies, the defense contractor L3Harris Technologies and Anduril, had made autonomous operating systems for the boats. That day, Replicator was testing GARCs that ran on each company’s product.
As a safety precaution, the autonomy software wasn’t supposed to be enabled until the boats were suitably far out to sea. But one drone running L3Harris’ system suddenly lurched forward. Its autonomy mode, which had somehow turned on, required it to keep a distance of 80 meters (262 feet) from all other objects. The robo-boat sped away, still tethered to the towboat. It alternately accelerated and decelerated, then started crisscrossing in front from port side to starboard side in a semicircling action.
The captain of the towboat had no way of taking over control of the automated vehicle, whose erratic movements caused his own vessel to capsize, throwing him into the water. Still tethered to the towboat, the drone turned back toward it and—for reasons that remain unclear—started advancing at rapid speed.
A captain towing a separate GARC saw what was happening and raced toward the scene, positioning his vessel between his floating comrade and the advancing drone. A third towboat pulled the captain out of the water, and he escaped without serious injury. It had been just three minutes since the drone had gone rogue.
A safety investigation soon diagnosed the problem: An operator on the dock had inadvertently sent a message to the drone remotely disabling the safety lock meant to prevent it from switching into autonomy mode—a classic “fat-finger mistake.” A spokesperson for L3Harris said in a statement that the operator who caused the issue didn’t work at the company and that its software had “demonstrated its ability to control a mix of uncrewed platforms, payloads, and commercial technologies even if they were produced by different manufacturers.” A physical button was added to drone boats to block such accidental commands, and the boats were tweaked to prominently display the mode under which they were operating. Rival companies would start sharing safety lessons.
But the incident illustrated problems that still existed with the Pentagon’s drone strategy and couldn’t be resolved with the addition of another button or two. Replicator had still not progressed to the point that its creators were comfortable putting live ammunition on an unmanned vessel, let alone sending one into a scenario where it would be expected to coordinate with other vehicles or carry out a specific attack plan. The program did manage to deliver hundreds of drones by the August deadline, but it fell far short of its initial goal.
This sort of thing is a classic blunder of software-focused folks with little time Doing the Thing in the real world.