The people you follow ist stored in a single note on the relays. So a client just has to find the latest note of that kind to retrieve that list.
To get the people following you, a client has to find all notes of that kind referencing you. So these are many and possibly distributed on different relays.
