Umbrel already offers to run an LLM but I don’t want to strain its resources.

I think within a couple of years self-hosted AI boxes with open-source models will be ubiquitous.