The curious question is, why do people who have already left still have access and have access to the code? If you don't want to change your hardware wallet, I recommend reading up on Multisig instead.