It doesn't remove them, but you will see different numbers depending on a lot of factors. I don't know if anyone has a complete picture of what exactly is going on, too many moving parts and not enough experience yet, but a follow is an event that the follower writes to some relay. If you no longer read events from that relay, you won't see the follower. But if they still read events from relays you write to, they'll see your notes regardless.