Nah, don't worry about it. There's a lot of FUD around Signal (and almost none of it is valid) but it's completely fine unless your life depends on total anonymity.