I'm not talking about a 2FA sent over SMS or a pin sent over SMS.

This is the pin/password you set when you first register a phone number to Signal and have to remember it. The pin doesn't recover any chat history or contacts on Signal either. All it does is prevent someone from registering that phone number on Signal again.

So ownership of the phone number doesn't matter, because if they try registering it with Signal again they won't be able to without the pin/password that was set. And even in the extremely unlikely chance they get your pin/password somehow, it won't contain any chat history or contacts.