You can lose followers if you remove relays. I don’t know exactly how it works but that’s a thing, they’re tied to relays somehow. Maybe mutual relays? I dunno. Also things get weird if you use multiple clients.