you can't lose follows. It's impossible because nostr is decentralized. So depend of every single person to follow you. The count of followers is always unreal into your nostr client, is approx., an estimated number, not the real number because is impossible count your real follower in nostr protocol.