Replying to Avatar S!ayer

oh yeah?

Nothing crazy, I just want to mess around locally/offline use cases for LLMs , I've been looking around but never really deployed locally yet
Avatar
ynniv 1y ago

If you don't have a GPU or a Mac, try LLM Farm (iOS) or mllm (Android)

Reply to this note

Please Login to reply.

Discussion

Avatar
S!ayer 1y ago

I have a PC - 6GB card and 16GB RAM.

I mean, for the life of me I can't seem to find the right model to use. Llama seems great, in fact, imo it runs better than ChatGPT but that's because they have more nlp from Facebook/Meta platforms...

Is there a way to run lllama on local without the need for AWS or Azure or a 4090 GTX

Avatar
ynniv 1y ago

You should be able to run https://ollama.com with Microsoft's `phi3:mini-128k` or `llama3.1:8b-instruct-q3_K_M`. You can see the pre-configured models and their rough sizes at https://ollama.com/library.