I was also speculating about this just yesterday. I think a possible solution is applying Shamir's Secret Sharing to the passphrase. This way you have a n-of-m access to the wallet without the derivation paths overload. I need to do some experiments.