There’s note kind 3 which contains your contact list. This kind is updated by relays which receive an update so should have the most up to date view if it was online when you last updated your following list. Followers works differently, it’s determined based on other users contact lists.

Strange your following list would fluctuate. 🤔