People who follow you are kept track of by relays, they’ll vary. People you follow can disappear sometimes though, but you can restore them, some clients keep snapshots.