I think the issue has to do with bip 32 vs bip 39, but not sure. May just end up doing a normal transaction instead.