has anyone started using a dedicated server for using open source ai stuff? is there something like umbrel, but for ai stuff? umbrel has llamagpt, but it takes up a lot of resources, so prob not a good idea to run that on the same box as a bitcoin/lightning node.
Discussion
Most advanced AI stuff needs GPUs. In theory huggingface but run locally.
i just wanna run open source llms for privacy and to not have to pay companies to use their apis
For llms use ollama.
my macbook is too shitty. that's why i want to get a dedicated machine. would prefer to just buy a machine with all the necessary hardware rather than building it piece by piece.
I don't think there is anything similar like umbrel for ai stuff. But anyone can install ollama on ther desktop and try things out before deciding between upgrading their GPU, building dedicated server with multiple GPUs, or just pay for AI services that support API call.
my macbook is old and can barely run the primal web app. so i doubt it would be good enough to run anything. 😂
Depending on how old it is, the best thing to try now is to install ollama, try a small-ish model such as phi3-mini, and see if you like it
Check out nostr:npub126ntw5mnermmj0znhjhgdk8lh2af72sm8qfzq48umdlnhaj9kuns3le9ll / StartOS with FreeGPT. Also Venice AI is worth looking at from privacy preservation perspective.
doesn't it take up a lot of resources to run? doesn't seem like a good idea to run it on a start9 along with a bitcoin/lightning node.
The newer 7B models like Llama3 or Mistrel work great on well-resourced devices like the ones we sell.
No real harm in running it alongside Bitcoin and Lightning, but you also don't have to. It's your server, only install what you want.
So it runs smoothly on the Server One with both a bitcoin node and an AI chat app hooked up to Llama 3.1 8B? Is OpenUI available on Start9?
Yeah that model runs just fine. Nothing compared to ChatGPT speeds but it's affordable consumer hardware. No, nobody has packaged OpenUI for StartOS yet, looks cool though.
Buy an expensive GPU, stick it in a box, install Pinokio.computer & holesail.io and you have a remote window into all the ai tools you want