I mean, I don't trust the FBI as far as I can throw them but I'm also not the sort that thinks every little thing is part of a larger agenda.
Regarding phone numbers, you can effectively throw a password ("Signal PIN") onto your Signal profile so that even if they have access to your phone number, the literal ONLY thing they're capable of getting from that information is simply when you were online last, and the time that you send a message. They really can't find out much other than that if you have a Signal PIN, massively thanks to Sealed Sender.