Yeah it's all new and very different than the centralized services we r used to!
So the way I understand it is you've got a list of people you follow attached to your nsec. To see your follower count, your client will poll each relay you're connected to. That relay will then poll each user's follow list. If someone has you on their list, they'll count them.
So if there's any users that follow you but aren't on any of the same relays as you, they won't get counted since your client won't know to poll their relay and retrieve the count.