As I understood the counter of followers never correct on nostr. You can't be sure it is someone unfollowed or just a fucking relay with bunch of your followers is off.