That may be true for nuclear rockets, but what about nuclear pulse propulsion?
I suppose many of the proposals I've seen for really effective nuclear rockets use some sort of magnetic containment to skirt the materials limitations. That's probably a shaky thing to deal with, if nuclear fusion attempts are any indication.