I achieved this with KDE by doing an absolute minimum Debian install and not installing a desktop environment.

There's also usually a minimal installation that's just the bare bones without all the extra bloat for most desktop environments. So it takes a little bit of extra work But the end result is completely clean install.