Thinking of building a server that can run a local AI. I'm not super interested in running something like stable diffusion. Do I still need a bunch of GPUs?
#asknostr
Discussion
No, can run quantized models on one decent GPU. Like Zephyr-7b-alpha or Mistral.
Sick so like one decent mid to high grade GUP would probably suit my needs and then just make sure I have a bunch of RAM and cores in my CPU.
Look at LM Studio or H2O GPT.
Having multiple GPUs can for sure speed up many AI-related tasks, but it’s not strictly necessary for running a local AI server, especially if you’re not working with highly intensive models or tasks.
Ideally I would like to run something that's roughly equivalent to GPT-3.5.
Well, that is probably a lot of computing power 🫣💕
I run llama2 on a M2 macbook and earn some sats with it.
I just want to run a good LLM on my phone. I’ve tried a few options but we’re still early.