The client is kinda doing a “follower query” in each relay, kinda like searching for things under the sand on a beach. Every time finds a “thing” it’s displayed in your follower count. Something this “things” have problems to communicate to the cliente (Damus, etc…) maybe the person that follows you move to another relay, maybe the relay has too much traffic, others…

My best practice for looking at this is go to Nostr.band on the web and search yourself. It usually give you the most precise number of followers plus other interesting stats. Talking from experience 🫡