That's why you have to announce your failover npub on your current account, which an impersonator can't do.