Yeah I’ve had that happen too.

If you leave and come back I think it still registers your removal. If I remember correctly.