Doesn't sound very great convenient.
A cleaner solution would be (imho) to use a master seed (maybe with passphrase), then bip85 derive seeds for each person, then give them either the seed or the xpub. This is easily done with a coldcard for example.
So he can control everything with his seed but there is clear separation and no address reuse.