i dont get local ai, which language model should i run if i got 16gb ram and core 5 13500h?
so far the models I've tried have been completely retarded. They cant answer simple questions how to learn certain calisthenics moves.
#ai #localAI #tech #privacy
Its your GPU that matters. KoboldCpp (based on llamacpp) can do it regardless though.
You want to use its high priority mode because otherwise it may use your e-cores and performance tanks.
Now if you have a GPU I recommend a Q4_K_S model that fully fits your GPU in file size. Keep in mind context also takes up space. So for 8GB of vram no higher than 11B Q4_K_S at like 4K context. For 12GB that becomes 13B. For 16GB its up to 20B + Mistrals 24B. 24GB its 30B.
If you dont it depends a lot at what speeds you can tollerate but Gemma 3N may be a good starting point.
