I assume clients subscribe to kind 3 with their own pubkey as tag. So they won't get that note, as the pubkey is no longer in the list. Clients need to either listen to all or requery all kind 3 events, or periodically check the kind 3 events of who they think are following them.