They have to be fetched from the relays. You usually have to instruct the client to do this, for exampl on Damus you click the arrow next to followers and it begins counting them.
Discussion
I guess there are a lot of relays which is good. Don't relays fo with your key though?