It's possible to deduce who messages whom (timing / correlation attack). All user contacts are uploaded to Signal servers (they say it's stored in SGX - which may be broken). Groups also store some data on Signal servers. And - most important - Signal relies on phone numbers.