You can also opt to set up your citadel in an appropriate location to begin with, far from said ne'erdowells. Fending against nature is often far easier than fending against sociey and state.