Replying to Avatar Enki

Thinking of building a server that can run a local AI. I'm not super interested in running something like stable diffusion. Do I still need a bunch of GPUs?

#asknostr
Avatar
jascha 2y ago

No, can run quantized models on one decent GPU. Like Zephyr-7b-alpha or Mistral.

Reply to this note

Please Login to reply.

Discussion

Avatar
Enki 2y ago

Sick so like one decent mid to high grade GUP would probably suit my needs and then just make sure I have a bunch of RAM and cores in my CPU.

Avatar
Enki 2y ago

GPU*

Avatar
jascha 2y ago

Yes, can fine tune LLMs on decent GPU with 12GB RAM like a RTX 4060 if just running smaller models (13B parameters or less) can be less beefy.

Avatar
Enki 2y ago

Om thanks that clears up some questions I had.