Yeah. I’ve done a set up like this.
When you want to sign with a specific passphrase you can add the passphrase and then you can remove it or change it to “move” to signing different wallets.
The addresses have zero correlation with each other and will effectively be totally different xpriv and xpub keys.
Just exercise caution on protecting those seed words because those being compromised may not directly expose your coin but then there is just the passphrase to guess. Trade offs…