I use a layered config for this. I have a “is_desktop = true” flag that turns on x11 and all the desktop tools i use. I share the same config on my servers, laptop, and desktop, with this flag turned on or off. Also per machine configs if further customization is needed