Because anyone somewhere that decides to follow you just publishes that fact to some relays. There is no central point where that information is stored.

A client may cache the count and list of accounts following you and update that the next time you use it. It could even generate notifications from that and I've seen that in some client a while ago.