I did that to in WoS every now and then I still get a zap to that wallet even though I quit using it as a lightning address. Some relay didn't get changed.