Is anybody using local AIs like chat gpt and can recommend one? Preferably open source, and isolated from the internet...
#asknostr
Discussion
Koboldcpp is a good llm front-end that I use. The github had pretty good instructions for how to install. You then get an llm from huggingface (I use mixtral) you just run koboldcpp with the path to your model as an argument and open the web front-end in your browser.
Thanks!
client GPT4ALL (there's more out there now)
Been using them for year+
Local LLM's for general purpose I've found Falcon-7b does pretty good. Run larger Falcon if you have RAM for that
I've also spent the time to finetune the smaller models for my use cases. Takes time but feed it your local data and you'll thank your past self for the work you put in
very cool, what are you favorite use cases?
my immediate thought is that it would be helpful as an editor and brainstorm engine for some longer pieces I want to write
One use-case for me (that I haven't set up yet) is to generate embeddings for all my notes so that I can search them semantically.
Tbh use it for everything. Been slacking using Maximus (my local agents nameš ) as my daily journal, but exactly what you said.
- Idea generation
- Crafting posts/emails
- Reading & summarizing articles, videos, T&C
- Legal docs (as templates, not final form of course, real humans vet all docs produced and edit accordingly)
- coding (I switch to better coding LLM's for heavier work)
- figma layout generation
- crafting mid journey prompts
- etc
Recently picking back up connecting Maximus to voice for 'speech-to-prompt', then prompt triggers switch for local network IoT devices
Example:
"Hey Maximus, turn off the hallway light please." Etc.
No Alexa, siri, etc nonsense
All local
I use gpt4all all the time.
https://github.com/LostRuins/koboldcpp/releases
All work well. Get an uncensored model here by the Bloke:
That didn't format the way I intended. Hopefully you can see the 3 separate links to 3 different front ends.
Also, in my experience, sometimes these programs need to be connected to the internet momentarily at start up for some reason but after they are loaded you can unplug ethernet cable if you like and be offline. Either way, none of these communicate anything out and are all local, but i've had issues booting up with network turned off at the beginning. Probably because some of them use a browser interface and even though you are connecting locally they don't work if you shut it all down.
This is what Copilot suggested:
Certainly! If youāre interested in offline AI tools, there are several options available that operate locally on your device without requiring a continuous internet connection. Let me introduce a few:
1. Freedom GPT: Freedom GPT is an open-source AI chat interface that can run locally on your own device. Similar to ChatGPT, it can generate text, translate languages, and answer questions. The key difference is that Freedom GPT ensures your conversations and inputs stay within your computer, providing privacy. It leverages Stanfordās Alpaca models and features a simple chat-based interface. Keep in mind that itās not as powerful as ChatGPT due to memory limitations on personal computers1.
2. Offline AI Assistant Apps:
ā¢ Google Assistant: While Google Assistant primarily relies on the internet, some features work offline, such as setting alarms or sending texts.
ā¢ Robin: Robin offers voice-based navigation and local search even without an internet connection.
ā¢ Hound: Hound is a voice assistant that can handle complex queries offline.
ā¢ Siri: Appleās Siri can perform certain tasks offline, like setting reminders or sending messages.
ā¢ Cortana: Microsoftās Cortana has offline capabilities for basic tasks.
ā¢ Dragon Mobile Assistant: Dragon supports voice commands offline.
ā¢ Bestee Offline Virtual Assistant: Bestee provides offline voice assistance for various tasks2.
3. Quantized LLMs: These are locally loaded language models (LLMs) that you can use without an internet connection. Theyāre available on platforms like HuggingFace, H20, Text Gen, and GPT4All. These LLMs offer flexibility and security, making them a great choice for various applications3.
Remember to choose the one that best suits your needs and enjoy the benefits of offline AI! š¤š
Thanks for the thoughtful answers everyone, raising my takeaways for awareness:
Offline clients & LLMs
-Chatgpt4all seems to be the most popular client with a few different llms behind it
-Koboldcpp was also mentioned as a good front end
-Huggingface came up multiple times as a good place to find llms
Google/online replacement
-perplexity