It can take a while to go thru your social graph to count the followers. I would worry about it too much.