nostr:npub1lj3lrprmmkjm98nrrm0m3nsfjxhk3qzpq5wlkmfm7gmgszhazeuq3asxjy I’m running llama2 and code lama locally on my laptop. Lot of fun. I think only the 7b models. Wonder if I could run 13b I have 24 gb ram.
Really want to be able to feed it docs pdfs etc. currently only runn inch in command line via Ollama
You could give GPT4ALL a try. It has a built in plugin that can reference local docs. I find it does a good job summarizes concepts, but not so great at pulling out specific information.
24gb is sufficient to run 13B models at 4 or 8 bit quantization, and some will fit at 16bit 👍
Thank you for the response and insight. I’ll try out some 13b models today.
I noticed there are some ways to hook llama2/codellama into vs studio. I think using “continue” was one of them.
I’d like to do that and have it evaluate some nostr protocol code.
Following you for more llm discussion 🤙
