There's not much to distrust: it's a FOSS client with end-to-end encryption.
In worst case scenario, you leak metadata to Signal servers. This is still orders of magnitude better than WhatsApp.
Not allowing competing servers and federation is understandable. Doing otherwise would put them in a similar bag as Matrix (which is trash btw). Also, service centralization allows quicker iterations without accidental breakage.
nostr:npub1exv22uulqnmlluszc4yk92jhs2e5ajcs6mu3t00a6avzjcalj9csm7d828 and Threema are of course far superior but convincing friends and family to use those niche apps just for me it's a hard sell.
Signal has proven to be a nice middle ground where I already have a considerable network effect in my social circle.