If privacy is your concern you wouldn't use the contradictional app signal.
Discussion
You are quite right.
In my opinion, phone number privacy is the least one can expect from apps which claim privacy.
For serious privacy concerns, I would recommend apps like Session, Element, etc.
Phone number is an identifier the only reason signal requires it is if theres a federal case going on you can track down the signal account holder by the phone number. Element is partially safe sure chat msges can't be read in a e2ee room but as an hs admin I can see what ip addresses are in e2ee rooms every users ip who is in a room with one person from your own homeserver & you can read the file names of all media attachments in the room from an acp.
Session & SimpleX are the 2 closest to being the most private secure chats. I prefer simplex over session only because Lokinet the routing method session uses is actually a really small network compared to tor safer yes but less devices to route off of makes it easier to narrow down. But the only thing you have to be concerned about is possibly 5eyes controlling exit nodes. But whos to know Lokinet is in Australia.
SimpleX may be a bit harder to receive messages but defintely securer no identifiers import/export your own profile database & you can use it in terminal interface.