I was also speculating about this just yesterday. I think a possible solution is applying Shamir's Secret Sharing to the passphrase. This way you have a n-of-m access to the wallet without the derivation paths overload. I need to do some experiments.
Discussion
I found https://github.com/trezor/python-shamir-mnemonic?tab=readme-ov-file
But SLIP-0039 aims to replace BIP-39, while would be nice using SSS on top of it.