Despite treating a caricature of a caricature with trivial algorithms

Monday, February 12th, 2018

A few months back, while I was stepping through Techniques of Systems Analysis, Scipio Americanus recommended Keith R. Tidman’s The Operations Evaluation Group as “a fine overview of the development of the OR/OA field from the naval perspective.” I immediately ordered a copy, read it, and failed to achieve OR Enlightenment, so I didn’t get around to preparing a post full of insights from the book.

Gwern, on the other hand, did follow through and did produce just such a review, which summarizes my experience, too:

So overall: reasonably well-written, covers intrinsically interesting topics like ASW in WWII and Vietnam air tactics; compromised by official history purpose to recount thoroughly uninteresting internal details while omitting too much of both context and technical detail for my tastes and suspiciously hamstrung in certain areas like nuclear strategy or Harpoon.

Here is Gwern’s list of insights from the book:

  • after compiling all data about U-boat sinkings, Edison found merchant shipping routes were unchanged despite the risk, 94% of sinkings were during the day, and <4% of ships carried listening devices or radios. Edison set up a simple war game on a map simulating a merchant vs U-boat, proving that travel by night from port to port would largely eliminate sinkings. Unfortunately, his findings were ignored.
  • Blackett’s Circus used animal experiments demonstrating that lethality of air pressure blasts was overestimated 5x, reducing over-optimistic estimates of the effect of bombing campaigns in Germany
  • Depth charges were set to explode at 100 feet depth, on the assumption that U-boats would be that deep after being spotted; analysis indicated that half had not even submerged when depth-charged, much less reached 100 feet, and the optimal setting was 20 feet (which the depth charges didn’t even allow as a setting), which “new setting [of 35 feet] at least quadrupled their destructive capability.”
  • Big convoys turned out to have half the loss rate of small convoys, due to U-boats being unable to amass more, leading to a shift away from small convoys
  • A British naval program in the Mediterranean armed merchant ships with AA guns to reduce losses from aerial bombing; the program was going to be canceled because only 4% of attacking planes were being shot down and the deployment of scarce guns looked like a waste, however, an OR re-analysis of ship rather than aircraft losses showed that the armed ships had a 10% loss rate versus unarmed ships’ 25% loss rate. The latter was clearly a more relevant end-metric.
  • initial attempts by naval researchers to record underwater ship sounds to fool sound-based naval mines failed as the device invented to make ship-like sounds turned out to not sound much like a ship at all; this device serendipitously turned out to be nearly perfect for fooling the German sound-seeking homing torpedoes, largely scuppering their deployment (and freaking out the U-boat crews by its bizarre sounds, who were sure that the “singing saws” were “some powerful, dangerous weapon”)
  • William Shockley (taking a break from electronics research to serve as an OEG analyst during WWII), deployed to England to observe ASW there and was struck by an incident in which a plane attempted to bomb a discovered U-boat but the bomb jammed due to rust, then, fixed, went out again 2 days later only to crash in the fog. Shockley found that “on the average, an aircrew had just one opportunity to kill a submarine before its own members were either killed or wounded or at least moved on to another assignment. An aircrew thus had little or no opportunity to learn on the job.” ASW could only be developed institutionally and given the nature of search over large areas, statistically.
  • OEG was deeply involved in the early development of optimal search theory (Bayesian or otherwise), developing models of what probability a search plane had of spotting U-bots or periscopes under various conditions and altitudes. This then allowed development of optimal search patterns and setting up barrier patrols, which, when deployed in the Strait of Gibraltar, caught 3 U-boats in 4 months and then sealed off the Mediterranean; this was followed by capture or destruction of 4 of 5 German blockade-runners carrying vital rubber/tin supplies from Malaysia/Japan (the equivalent of “a year and a half” of German supplies).
  • study of U-boats off the US East Coast and also the Caribbeans showed that air patrols were staying far too close to land and needed to be outfitted with radios and spotlights; the patrol patterns were changed.
  • on the other side of the Atlantic, radar+spotlights on even a few planes around France proved to be a potent combination in striking U-boats at night when they typically surfaced to rest & travel rapidly, forcing them to shift travel to during the already-dangerous day. “In sum, the night flying of 2 squadrons had increased the effectiveness of antisubmarine operations in the Bay [of Biscay] by more than 7 squadrons of day flying.” (And also prompting the introduction of radar-detectors, which led to radar-detector-detectors etc.) A similar scenario played out in the Pacific: analysis demonstrated that US subs were lost at the same rate regardless of using their radar, so the Japanese planes did not have radar-detectors, and US subs could go back to using radar full-time.
  • The existence of radar-detectors led Caribbean pilots, when outfitted with a new radar that regularly revealed vanishing contacts, to assume they were being detected by U-boats, and to abandon use of the highly effective radar, crippling their submarine hunting. OEG didn’t believe radar-detectors could have been deployed so fast by the Germans and investigated; the vanishing contacts turned out to be glitch in the radar and the pilots resumed use.
  • US submarines were being lost at high rate in the Pacific for unknown reasons, as few survived long enough to report the cause; study of US submarine miss rates in attacking Japanese subs (which able to report back) revealed that contrary to the US Navy’s belief, most of the US subs were being killed by Japanese subs and not airplanes or surface ships. Immediately, tactics and sound equipment were revised to emphasize anti-torpedo tactics, and “By the close of the war, several commanders had credited the modified torpedo detection equipment and new tactics with saving their submarines from destruction.” While they were at it, they modeled mine fields and appropriate counter-tactics, and “of 12 submarines assigned to operate in the Sea of Japan, none was lost to the mines that heavily dotted the straits leading into and out of the area.”
  • anti-kamikaze tactics were likewise worked out (evasive maneuvers: big ships yes, small no; turn towards a high-diving but away from a low-diving)
  • Analysis of Korean fighter-bomber strikes showed the F4U was much more vulnerable than the F9F, due to tactics like going much lower and more often in range of AA (and even small-arms fire). It also showed pilots were wrong about their belief that the last airplane in a strike ran the largest risks due to loss of the element of surprise (it actually ran the least risk). Changes reduced the F4U losses.
  • a 1958 OEG study found a ‘window of vulnerability’ of the US to USSR pre-emptive strikes 1961-1963 and a ‘missile gap’. Tidman defends the report, noting that it made a number of suggestions for eliminating the ‘window’, many of which were taken: “…the hardening and dispersal of fixed weapons sites, a program of continuous flights by SAC bombers, the sped-up procurement of available weapons systems (such as mobile cruise missiles), and the increased preparedness of naval air. OEG also recommended that emphasis remain on the development of mobile and concealable forces, rather than on fixed-site forces. Polaris, for example, was spotlighted as meriting accelerated production. The defense policies of two administrations were greatly influenced by this expectation of a possible low point in U.S. deterrence…As soon as John F. Kennedy took over the presidency, however, he decided to embark on an extensive program of strengthening American strategic forces. Mirroring much of what OEG’s study had recommended three years earlier, he increased the production rate of Polaris submarines by several months, and added 10 submarines to the original planned total. He also doubled the capability for producing Minuteman and improved the alert status of SAC’s B-52s.”
  • during the Cuban Missile Crisis, Kennedy’s order to blockade Cuba was implemented by the US Navy based heavily on OEG-researched doctrines and with active OEG assistance; OEG further studied data during the blockade about intercept rates, helping confirm that the blockade was tight and intercepting almost all Soviet vessels.
  • this blockade research would be further used in Vietnam as part of the “Operation Market Time” blockade/intercept line, where OEG optimized it & showed that the blockade there too was highly effective in eliminating Vietcong supplies
  • around 1966, OEG “conducted a study of surface-to-surface missiles that led directly to the development of the Harpoon antiship cruise missile.” (Unfortunately, Tidman doesn’t go into more detail about Harpoon other than to note later OEG involvement in finetuning Harpoon based on field exercises and against what was known of Russian ship defenses.)

Here is Gwern’s more meta insight:

Kahn makes an interesting point: one often sees an argument (particularly in conservative/libertarian circles) about ‘Chesterton’s fence’ and variants thereof — that societies have evolved rich and highly effective tactics through vast experience & evolution that mere humans cannot hope to improve upon nor understand; yet, as OR has proved many times, it is possible — easy, even (“it was found that almost any honest, technically competent person could turn out worthwhile and interesting results”) — to apply a little statistics to a problem and despite treating a caricature of a caricature with trivial algorithms or even none at all beyond basic arithmetic, improve, possibly quite a bit, over the carefully-considered judgments of humans in the field with decades of experience. And of course we can add many examples of human judgment being exceeded in areas like chess or Go or math despite millennia of study, or entire areas of human knowledge turning out to be almost 100% wrong (religion, medicine) before the introduction of methods like ‘record all data’ or ‘flip a coin to decide whether to administer a medicine to see if it works’.

Kahn ascribes this in part to technological change (no one is competent to understand how to hunt German submarines in WWII because it is too novel a problem for any folk wisdom to have evolved), and while that’s certainly a problem (witness Shockley’s anecdote of why no air crews could develop real expertise), we also have to note the presence of systematic biases and error in human reasoning demonstrated throughout OR. The problem with Chesterton’s fence is that everything does change, people can’t learn the right thing in the first place, and from an information-theoretic & genetics perspective, there just is not enough reliable transmission of information nor selection within or between societies to maintain more than a few traditional practices with cryptic efficiency. (If societies were a bacteria with a genome, they would succumb to mutational meltdown almost instantaneously.)


  1. Gaikokumaniakku says:

    Edison was a recognized genius. He had wealth. He had social connections to people who wanted to keep Americans alive. And yet people ignored him. Such is the idiocy of centralized government in the wrong hands.

    Shockley was not recognized as a towering genius when he made his discoveries, but still the thundering idiots ignored him.

    What hope can there be for human rationality when the likes of Edison and Shockley are ignored during times of crisis?

  2. Candide III says:

    Gwern is treating a caricature of a caricature of Chesterton’s fence argument. This is easy to prove by reading one paragraph of Chesterton’s text laying out his argument. (Talk about people not being able to learn the right thing in the first place, and about there not being enough reliable transmission of information within societies!) In fact, Chesterton didn’t argue that mere humans cannot hope to improve upon nor understand evolved mechanisms, or that evolved mechanisms for dealing with realities much more complex than mathematically trivial board games or ASW are rich and highly effective. He wrote,

    There exists in such a case a certain institution or law; let us say, for the sake of simplicity, a fence or gate erected across a road. The more modern type of reformer goes gaily up to it and says, “I don’t see the use of this; let us clear it away.” To which the more intelligent type of reformer will do well to answer: “If you don’t see the use of it, I certainly won’t let you clear it away. Go away and think. Then, when you can come back and tell me that you do see the use of it, I may allow you to destroy it.

  3. Alrenous says:

    I noticed that discrepancy as well.

    Although Chesterton was not one of them, there are a large number of people who hold the view Gwern criticizes.

  4. Lu An Li says:

    “armed ships had a 10% loss rate versus unarmed ships’ 25% loss rate”

    The mere PRESENCE of anti-aircraft weaponry on the ships caused the attacking warplanes to fly higher and take some sort of evasive action that made the bomb run less accurate and effective. Even if not one shot fired by the defender!

  5. Adar says:

    “blockade research would be further used in Vietnam as part of the ‘Operation Market Time’ blockade/intercept line, where OEG optimized it & showed that the blockade there too was highly effective in eliminating Vietcong supplies”

    Market Time and Game Warden that blockade one of the few [??] successes in Nam?

Leave a Reply