If you are using a relay set app such as Damus, if you remove a relay or few from your list your follower count will reduce.
Conversely, if you add a few new relays your follower count will increase.
This is a differentiator va twitter due to nostr’s non-centralized nature.