No. The leak happens on the signing part and the passphrase only changes the signing key you use.

The problem is that signing requires a random number and if that number is not perfectly random, a party that knows about the bias can learn about the private key that was used. If the attacker knows the full "random number" it can extract the private key from just two signatures.