Typically happens when you're changing clients, or changing your relay list in fundamental ways.

Solution 1: Use clients that have local follows, like Gossip.

Solution 2: Avoid changing clients. If you find one you like, stick with it. If you want to try new ones, try them with a different key.

Many clients don't work well with others, which leads to some accidentally over-riding what you've done in the other. Then they publish this to the relays, and your other clients assume this is your new data (new profile, new follow list, new block list, etc).