Some proposed designs for pebble-bed reactors could conceivably better the thrust-to-weight ratio considerably, removing as they do the need for heavy reinforcing structure for the reactor core.

Using hydrogen propellant, a solid-core design typically delivers specific impulses (Isp) on the order of 850 to 1000 seconds, about twice that of liquid hydrogen-oxygen designs such as the Space Shuttle Main Engine.
[...]
Immediately after World War II, the weight of a complete nuclear reactor was so great that it was feared that solid-core engines would be hard-pressed to achieve a thrust-to-weight ratio of 1:1, which would be needed to overcome the gravity of the Earth on launch. The problem was quickly overcome, however, and over the next twenty-five years U.S. nuclear thermal rocket designs eventually reached thrust-to-weight ratios of approximately 7:1. Still, the lower thrust-to-weight ratio of nuclear thermal rockets versus chemical rockets (which have thrust-to-weight ratios of 70:1) and the large tanks necessary for liquid hydrogen storage mean that solid-core engines are best used in upper stages where vehicle velocity is already near orbital, in space “tugs” used to take payloads between gravity wells, or in launches from a lower gravity planet, moon or minor planet where the required thrust is lower. To be a useful Earth launch engine, the system would have to be either much lighter, or provide even higher specific impulse. The true strength of nuclear rockets currently lies in solar system exploration, outside Earth’s gravity well.

Of all the types of propulsion we could use right now, without waiting for physics breakthroughs, Orion is unique in offering much higher thrust and much higher efficiency. And it is more efficient the bigger you make it up to millions of tons of payload.

Maybe others can’t see a reason for an 8 million ton reference Orion design capable of taking hundreds of people to Saturn in a couple months. But I think that really is a failure of imagination on their part.

NERVA engines are efficient but offer low thrust.

Anyway, fallout was the chief concern at the time, but EMP is a recognized problem:

The launch of such an Orion nuclear bomb rocket from the ground or from low Earth orbit would generate an electromagnetic pulse that could cause significant damage to computers and satellites, as well as flooding the van Allen belts with high-energy radiation. This problem might be solved by launching from very remote areas, because the EMP footprint would be only a few hundred miles wide. The Earth is well shielded by the Van Allen belts. In addition, a few relatively small space-based electrodynamic tethers could be deployed to quickly eject the energetic particles from the capture angles of the Van Allen belts.

And for trips around the solar system (at least to and from anyplace potentially useful) wouldn’t NERVA engines work well enough and be much cheaper to operate?