Let's say he's right, and remote work is much less productive than in-person office work. I might even be convincible in some cases, for certain kinds of work and the potential for driving consensus faster. I'm not sure net-net it's less productive overall. But we can wring-hands about it for hours.

Whether he's right about that or not does not nexus on a moral claim. Why is something that is "less productive" equate to something that is immoral? By that measure, people should go to jail for taking vacations.