I had this issue with following accounts in the early days. Never could figure it out exactly but thought it might have something to do with the relays I was connected to syncing. So maybe that?