They should all join the Non Aligned Movement and then cancel their UN membership. Security Council would look pretty impotent if they were the only members left, and NAM was where decisions were made.

I know every non-nuclear power is afraid of then being attacked, but that's what making alliances are for.