Apparently if you follow or unfollow someone while your client is struggling to pull your followers from the relay it'll overwrite your followers with a blank list.

#[2] Perhaps there can be an added attempt to pull the follow list immediately before adding or removing a new follow?