If you use Signal, you can use a burner number; moreover, I'm not forced to give the app access to my whole contact list (i.e., my social graph, which includes family, etc.). I get that it's your app, so you can do what you like.
Personally, I would be someone who wants to use your app, but I don't because, as a privacy advocate, this is a big no go for me. I assume many other privacy sensitive users feel the same way, and they are normally your target audience, in my opinion.
Why not offer an alternative via a Nostr identity? Most people I know from meetups, conferences, etc. share Npubs or usernames for other messengers when they meet, but they rarely exchange phone numbers. In fact, many people don't even have a SIM with a number attached anymore.
Or even offer a way via geohash, where people can use only the first two characters of the are. This makes the location only roughly precise, but it still shows the liquidity for that area, allowing users to decide on their own whether they're comfortable meeting without already knowing each other.
The average Joe might still use WhatsApp with a phone number and doesn't care about sharing his entire private contact list with you, especially since Meta and Google already have it. However, I doubt that these are the kind of people who prefer p2p trades, they tend to buy non custodial ious on Coinbase.