this is why cryptography exists, to do things like this without having to reveal your number to anyone. If you are paranoid about it you shouldn’t use a sim card. Also this doesn’t have to be a phone number, it could be any piece of data shared among friends. Phone # is just the obvious one that is in contact lists.