You can use FreeGPT-2 on StartOS to find and run open source LLMs locally and interact with them through a ChatGPT-like user interface. It works great for models up to 13B parameters. The drawback is it's much slower than something like ChatGPT. The solution would be to attach a high powered, expensive GPU to accelerate inference, but that is not yet supported by StartOS. It will be though, and when hopefully by then GPUs are coming down in price as well. But for now, have fun with it, try it out, it really does work well!
#asknostr
Dear nostr:nprofile1qqs9df4h2deu3aae83fmet5xmrlm4w5l9gdnsy3q2n7dklem7ezmwfcpz4mhxue69uhhyetvv9ujuerpd46hxtnfduhszythwden5te0dehhxarj9emkjmn99uah95g0 I'm a huge fan of your home server and it was not hard to set up as a salesperson non engineer.
Since I'm not a technical wizard I need some advice/thoughts. If I wanted to build a self sovereign LLM to run and host from my house. What is the best hardware to purchase with cash or non KYC corn? Once I I have the appropriate hardware what models can I download and run locally?
Want to teach my kids how to leverage this tech but don't want them to end up in the machines blob.
Is it best to add to my start9 system, purchase apple machines, or other PCs?
Discussion
No replies yet.