I didn’t loose anything, your followers have your npub in their contact list, adding/removing relays won’t change their contact list so they are technically still following your account. Remove the relays one by one and check as you do it if that makes you feel more confident, if you notice something off you can just add the relay back