Pretty sure that’s the issue.

Try adding them back.

Then do a backup+broadcast of your notes with nostrsync.

After the backup is complete, try and remove the relays you want to get rid off. That will most likely fix it

https://nostrsync.live/