Because your followers are not actually associated with your account in any way. The number is just counting how many people your client can see who have you on their follow lists.

So, if I drop relay X, anyone who follows me, who only has their follow list published on that relay, will disappear from the count.

This is actually a useful metric when deciding which relays to keep. If your number goes down significantly, there is a chance that relay was the only one you had in common with those people, and they may no longer be seeing your content.