Training is a whole other ball game. But just installing a small pre-packaged LLM can be just a docker container.

https://github.com/getumbrel/llama-gpt