Yamamoto surprised the US Navy in two ways: Firstly, he completely inverted the controlling doctrine of the entire Japanese Navy. Secondly, he developed technology to enact that doctrine before anyone else thought to do so.
Japanese navel doctrine in the post-WW1 era was dictated by their belief that any future naval conflict would repeat the battle of Tsurigami, i.e., a US fleet would arrive in or near Japanese home waters, where a single decisive battle would take place in which one fleet or the either would be annihilated and the war would then come to a negotiated end. In such a conflict, long-range vessels would be unnecessary. This doctrine arose in part because prior to the 1930s Japan would have been hard-pressed to get enough oil to fight a long-range conflict.
The home-water doctrine controlled absolutely everything in the Japanese Navy, from technology to training to indoctrination. Every ship was designed from the keel upward for an intense, short-ranged and short-duration conflict. Japanese ships usually had only half the fuel capacity of US or British vessels. All training focused on fighting that one climactic battle. Such a battle was portrayed as not only the only practical solution but the only moral one as well. This concept so dominated Japanese naval thinking that, after Yamamoto’s death, the Japanese navy instantly reverted to it. Making the conceptual leap to a radically different strategy was no trivial feat, and neither was convincing everyone else to go along. American naval planners were well aware of all this and they filed any possible long-range Japanese attacks by capital ships in the highly unlikely file.
After Yamamoto broke the pattern for the controlling doctrine of the Japanese navy, he next had to overcome the technological limitations. He had three major problems: (1) Fueling long-range operations, (2) developing air-dropped torpedoes that wouldn’t bottom out in the relatively shallow water of Pearl Harbor and (3) developing air-dropped bombs that could reliably penetrate the armor of capital ships.
As late as January 1941, none of that technology existed. American planners in December of 1941 assumed it still didn’t exist.
Yamamoto couldn’t just put a bunch of oilers (tankers) with the carrier fleet and set sail. The oilers were just large merchant vessels that couldn’t keep up with the fleet. The normal pattern was for tankers to stay around some island and for the warships to sally out and come back to tranquil waters to refuel. Nobody planned to bring tankers along on a sneak attack. Yamamoto solved the problem by constructing some oilers on some old cruiser keels which made an oiler that could reasonably pace a fleet.
The other problem was that refueling in even moderately heavy seas was tricky. Ships getting bounced around a tug at the wrong time could send flammable fuel everywhere. Yamamato solved that problem with a new kind of coupling system that could safely disengage.
This coupling technology gave the Japanese capital fleets unlimited striking range at good speed. Unknown to anyone outside the upper levels of the Japanese naval command, the Japanese carriers could strike West-East from Madagascar to the Panama canal and North-South from Alaska to Australia. The surprising series of naval air strikes that controlled the first few months of the war depended on the Japanese navy’s ability to refuel on the fly.
Yamamoto solved the other two problems with typical Japanese elegance and simplicity. Attaching wooden fins to the torpedoes allowed them to enter the water at a much shallower angle so they wouldn’t plow into the bottom of the harbor but would run true. Attaching fins to existing armor piercing cannon shells turned them into armor penetrating air delivered bombs.
American planners also took into consideration that even with the technical ability to strike plus the element of surprise, a carrier attack on Hawaii was very dangerous for the Japanese.
The Japanese had no more idea of the location of the US carriers than the US did about the location of the Japanese carriers. The Japanese fleet could have been counter ambushed and overwhelmed by the combined force of the US carriers, battleships and land-based planes from Hawaii. Admiral Nagumo failed to launch follow up attacks on the oil storage and dry docks of Pearl Harbor in part because of this realistic fear of a devastating counterattack.
American planners didn’t believe the Japanese would risk so many capital ships and aircraft in such a risky attack.
The combination of all these factors meant that even though Admiral Kimmel, General Short and others understood the theoretical dangers of a carrier attack on Pearl Harbor, they didn’t think it a likely enough scenario to take counter measures against, especially if that meant exposing Pearl Harbor to more likely forms of attack.
When they began actively preparing for war with Japan in early November 1941, they did not irresponsibly plan for an almost “impossible” carrier strike but instead responsibly planned for likely modes of attacks that the America navy thought the Japanese could carry out: Submarine attacks on ships, submarine shelling of the shore, submarine-landed commandos, aerial bombing from lumbering seaplanes and sabotage attacks by covert agents.
Kimmel seriously ramped up anti-submarine defenses around the harbor. Short put the coastal artillery on high alert. Both configured air defenses to repel a high-altitude attack from large seaplanes. Both guarded all land assets from commando or saboteur attacks. Most famously, both the Navy and the Army tightly clustered all their aircraft together on the airfields so they could be easily protected from a ground attack by light infantry or saboteurs.
Like competent baseball coaches, Kimmel and Short had covered all their bases. Unfortunately, the Japanese were playing football.
Most historical works conflate the surprise of the general public at Pearl Harbor with the surprise of the military. The Roosevelt administration worked tirelessly to downplay the risk of attack from Japan because FDR didn’t want attention distracted from Europe. Negotiations were still underway, and Americans of that era assumed that no one would attack during negotiations. The military, however, was actively preparing for war with Japan and was not particularly surprised that it broke out. They were only surprised by a radical change in Japanese doctrine and capabilities.
