Follows are a replaceable event, so each time you add a follow, it sends your whole follow list again. If your client 'forgets' to do this, it replaces your follow list with an empty list.