Complex, tightly-coupled systems — like nuclear power plants and our modern financial system — are subject to normal accidents:
It might seem obvious that the way to make a complex system safer is to install some safety measures. Engineers have long known that life is not so simple. In 1638, Galileo described an early example of unintended consequences in engineering. Masons would store stone columns horizontally, lifted off the soil by two piles of stone. The columns often cracked in the middle under their own weight. The “solution” – a third pile of stone in the centre – didn’t help. The two end supports would often settle a little, and the column, balanced like a see-saw on the central pile, would then snap as the ends sagged.
Galileo had found a simple example of a profound point: a new safety measure or reinforcement often introduces unexpected ways for things to go wrong. This was true at Three Mile Island. It was also true during the horrific accident on the Piper Alpha oil and gas platform in 1988, which was aggravated by a safety device designed to prevent vast seawater pumps from starting automatically and killing the rig’s divers. The death toll was 167.
In 1966, at the Fermi nuclear reactor near Detroit, a partial meltdown put the lives of 65,000 people at risk. Several weeks after the plant was shut down, the reactor vessel had cooled enough to identify the culprit: a zirconium filter the size of a crushed beer can, which had been dislodged by a surge of coolant in the reactor core and then blocked the circulation of the coolant. The filter had been installed at the last moment for safety reasons, at the express request of the Nuclear Regulatory Commission.
The problem in all of these cases is that the safety system introduced what an engineer would call a new “failure mode” – in other words, a new way for things to go wrong. And that was precisely the problem in the financial crisis.
[...]
The 1979 crisis at Three Mile Island remains the closest the American nuclear industry has come to a major disaster. It would have been far less grave had the operators understood what was happening. Coolant pumps were useless because a maintenance error had trapped them behind closed valves. Another valve jammed in the open position, allowing pressurised radioactive water at more than 1,000° C to shoot into the sump below the reactor, eventually exposing the reactor core itself and risking a complete and catastrophic meltdown.The operators were baffled by the confusing instrumentation in the control room. One vital warning light was obscured by a paper repair tag hanging from a nearby switch. The control panel seemed to show the jammed-open valve had closed as normal – in fact, it merely indicated that the valve had been “told” to close, not that it had responded. Later, the supervisor asked an engineer to check a temperature reading that would have revealed the truth about the jammed valve, but the engineer looked at the wrong gauge and mistakenly announced that all was well.
All these errors were understandable given the context. More than 100 alarms were filling the control room with an unholy noise. The control panels were baffling: they displayed almost 750 lights, each with letter codes, some near the relevant flip switch and some far. Red lights indicated open valves or active equipment; green indicated closed valves or inactive equipment. But since some of the lights were typically green and others were normally red, it was impossible even for highly trained operators to scan the winking mass of lights and immediately spot trouble.
I asked Philippe Jamet, the head of nuclear installation safety at the International Atomic Energy Agency, what Three Mile Island taught us. “When you look at the way the accident happened, the people who were operating the plant, they were absolutely, completely lost,” he replied.
Jamet says that since Three Mile Island, much attention has been lavished on the problem of telling the operators what they need to know in a format they can understand. The aim is to ensure that never again will operators have to try to control a misfiring reactor core against the sound of a hundred alarms and in the face of a thousand tiny winking indicator lights.
(Hat tip to David Foster.)