The followers are collected from relays where others may published following you. So the count depends on which relays your current client is connected to and what relays your follower used to publish their contact list.