The problem is the metadata, it is possible to associate your phone number and other people's numbers to know when you have spoken.