What happens to the human role in war as we arm ever more intelligent, more capable, and more autonomous robots?
When this issue comes up, both specialists and military folks tend to change the subject or speak in absolutes. “People will always want humans in the loop,” says Eliot Cohen, a noted military expert at Johns Hopkins who served in the State Department under President George W. Bush. An Air Force captain similarly writes in his service’s professional journal, “In some cases, the potential exists to remove the man from harm’s way. Does this mean there will no longer be a man in the loop? No. Does this mean that brave men and women will no longer face death in combat? No. There will always be a need for the intrepid souls to fling their bodies across the sky.”
All the rhetoric ignores the reality that humans started moving out of “the loop” a long time before robots made their way onto battlefields. As far back as World War II, the Norden bombsight made calculations of height, speed, and trajectory too complex for a human alone when it came to deciding when to drop a bomb. By the Persian Gulf War, Captain Doug Fries, a radar navigator, could write this description of what it was like to bomb Iraq from his B-52: “The navigation computer opened the bomb bay doors and dropped the weapons into the dark.”
In the Navy, the trend toward computer autonomy has been in place since the Aegis computer system was introduced in the 1980s. Designed to defend Navy ships against missile and plane attacks, the system operates in four modes, from “semi- automatic,” in which humans work with the system to judge when and at what to shoot, to “casualty,” in which the system operates as if all the humans are dead and does what it calculates is best to keep the ship from being hit. Humans can override the Aegis system in any of its modes, but experience shows that this capability is often beside the point, since people hesitate to use this power. Sometimes the consequences are tragic.
The most dramatic instance of a failure to override occurred in the Persian Gulf on July 3, 1988, during a patrol mission of the U.S.S. Vincennes. The ship had been nicknamed “Robo- cruiser,” both because of the new Aegis radar system it was carrying and because its captain had a reputation for being overly aggressive. That day, the Vincennes’s radars spotted Iran Air Flight 655, an Airbus passenger jet. The jet was on a consistent course and speed and was broadcasting a radar and radio signal that showed it to be civilian. The automated Aegis system, though, had been designed for managing battles against attacking Soviet bombers in the open North Atlantic, not for dealing with skies crowded with civilian aircraft like those over the gulf. The computer system registered the plane with an icon on the screen that made it appear to be an Iranian F-14 fighter (a plane half the size), and hence an “assumed enemy.”
Though the hard data were telling the human crew that the plane wasn’t a fighter jet, they trusted the computer more. Aegis was in semi- automatic mode, giving it the least amount of autonomy, but not one of the 18 sailors and officers in the command crew challenged the computer’s wisdom. They authorized it to fire. (That they even had the authority to do so without seeking permission from more senior officers in the fleet, as their counterparts on any other ship would have had to do, was itself a product of the fact that the Navy had greater confidence in Aegis than in a human- crewed ship without it.) Only after the fact did the crew members realize that they had accidentally shot down an airliner, killing all 290 passengers and crew, including 66 children.
The tragedy of Flight 655 was no isolated incident. Indeed, much the same scenario was repeated a few years ago, when U.S. Patriot missile batteries accidentally shot down two allied planes during the Iraq invasion of 2003. The Patriot systems classified the craft as Iraqi rockets. There were only a few seconds to make a decision. So machine judgment trumped any human decisions. In both of these cases, the human power “in the loop” was actually only veto power, and even that was a power that military personnel were unwilling to use against the quicker (and what they viewed as superior) judgment of a computer.
The point is not that the machines are taking over, Matrix-style, but that what it means to have humans “in the loop” of decision making in war is being redefined, with the authority and autonomy of machines expanding. There are myriad pressures to give war- bots greater and greater autonomy. The first is simply the push to make more capable and more intelligent robots. But as psychologist and artificial intelligence expert Robert Epstein notes, this comes with a built-in paradox. “The irony is that the military will want [a robot] to be able to learn, react, etc., in order for it to do its mission well. But they won’t want it to be too creative, just like with soldiers. But once you reach a space where it is really capable, how do you limit them? To be honest, I don’t think we can.”
Simple military expediency also widens the loop. To achieve any sort of personnel savings from using unmanned systems, one human operator has to be able to “supervise” (as opposed to control) a larger number of robots. For example, the Army’s long- term Future Combat Systems plan calls for two humans to sit at identical consoles and jointly supervise a team of 10 land robots. In this scenario, the humans delegate tasks to increasingly autonomous robots, but the robots still need human permission to fire weapons. There are many reasons, however, to believe that this arrangement will not prove workable.
Researchers are finding that humans have a hard time controlling multiple units at once (imagine playing five different video games simultaneously). Even having human operators control two UAVs at a time rather than one reduces performance levels by an average of 50 percent. As a NATO study concluded, the goal of having one operator control multiple vehicles is “currently, at best, very ambitious, and, at worst, improbable to achieve.” And this is with systems that aren’t shooting or being shot at. As one Pentagon- funded report noted, “Even if the tactical commander is aware of the location of all his units, the combat is so fluid and fast paced that it is very difficult to control them.” So a push is made to give more autonomy to the machine.
And then there is the fact that an enemy is involved. If the robots aren’t going to fire unless a remote operator authorizes them to, then a foe need only disrupt that communication. Military officers counter that, while they don’t like the idea of taking humans out of the loop, there has to be an exception, a backup plan for when communications are cut and the robot is “fighting blind.” So another exception is made.
Even if the communications link is not broken, there are combat situations in which there is not enough time for the human operator to react, even if the enemy is not functioning at digital speed. For instance, a number of robot makers have added “counter sniper” capabilities to their machines, enabling them to automatically track down and target with a laser beam any enemy that shoots. But those precious seconds while the human decides whether to fire back could let the enemy get away. As one U.S. military officer observes, there is nothing technical to prevent one from rigging the machine to shoot something more lethal than light. “If you can automatically hit it with a laser range finder, you can hit it with a bullet.”
This creates a powerful argument for another exception to the rule that humans must always be “in the loop,” that is, giving robots the ability to fire back on their own. This kind of autonomy is generally seen as more palatable than other types. “People tend to feel a little bit differently about the counterpunch than the punch,” Noah Shachtman notes. As Gordon Johnson of the Army’s Joint Forces Command explains, such autonomy soon comes to be viewed as not only logical but quite attractive. “Anyone who would shoot at our forces would die. Before he can drop that weapon and run, he’s probably already dead. Well now, these cowards in Baghdad would have to pay with blood and guts every time they shot at one of our folks. The costs of poker went up significantly. The enemy, are they going to give up blood and guts to kill machines? I’m guessing not.”
Each exception, however, pushes one further and further from the absolute of “never” and instead down a slippery slope. And at each step, once robots “establish a track record of reliability in finding the right targets and employing weapons properly,” says John Tirpak, executive editor of Air Force Magazine, the “machines will be trusted.”
The reality is that the human location “in the loop” is already becoming, as retired Army colonel Thomas Adams notes, that of “a supervisor who serves in a fail- safe capacity in the event of a system malfunction.” Even then, he thinks that the speed, confusion, and information overload of modern-day war will soon move the whole process outside “human space.” He describes how the coming weapons “will be too fast, too small, too numerous, and will create an environment too complex for humans to direct.” As Adams concludes, the new technologies “are rapidly taking us to a place where we may not want to go, but probably are unable to avoid.”
The irony is that for all the claims by military, political, and scientific leaders that “humans will always be in the loop,” as far back as 2004 the U.S. Army was carrying out research that demonstrated the merits of armed ground robots equipped with a “quick-draw response.” Similarly, a 2006 study by the Defense Safety Working Group, in the Office of the Secretary of Defense, discussed how the concerns over potential killer robots could be allayed by giving “armed autonomous systems” permission to “shoot to destroy hostile weapons systems but not suspected combatants.” That is, they could shoot at tanks and jeeps, just not the people in them. Perhaps most telling is a report that the Joint Forces Command drew up in 2005, which suggested that autonomous robots on the battlefield would be the norm within 20 years. Its title is somewhat amusing, given the official line one usually hears: Unmanned Effects: Taking the Human Out of the Loop.
So, despite what one article called “all the lip service paid to keeping a human in the loop,” auton omous armed robots are coming to war. They simply make too much sense to the people who matter.