It would compromise this specific key and maybe the xpub as well (privacy related issue). Generally Ledgers are not considered secure for multisig setups. As they create a lot of security issues in such a setup. They are not designed for this. You can find a lot of resources about it.