Would it compromise my multisig setup if one of the devices were a Ledger?
Discussion
It would compromise this specific key and maybe the xpub as well (privacy related issue). Generally Ledgers are not considered secure for multisig setups. As they create a lot of security issues in such a setup. They are not designed for this. You can find a lot of resources about it.
Here is a detailed report about #Ledger issues for multisig https://medium.com/shiftcrypto/the-pitfalls-of-multisig-when-using-hardware-wallets-9b0e98e4c19c