Identifying someone by breathing patterns picked up through their headphones would be much harder than identifying them through audio of their voice or images of their face

And having the headphones unplugged most of the time means it's more possible to find 2 people having a conversation where neither of them has a device on them monitoring the audio