IMO the passphrase probably doesn't even need to be that complicated either and could be something that's super easy to memorize. So you'd have to forget it and lose access to your password manager.