It doesn't remove them, but you will see different numbers depending on a lot of factors. I don't know if anyone has a complete picture of what exactly is going on, too many moving parts and not enough experience yet, but a follow is an event that the follower writes to some relay. If you no longer read events from that relay, you won't see the follower. But if they still read events from relays you write to, they'll see your notes regardless.

To follow = to post a signed msg saying "I follow you"

If your follower posted this to relays A, B, C and you're connecting to relays X, Y, Z then you'll never know they follow you.

There are other ways to "follow" silently, without publishing anything.