Avatar
Umbrel ☂️ 2y ago

Introducing LlamaGPT — a self-hosted, offline and private AI chatbot, powered by Llama 2, with absolutely no data leaving your device. 🔐

Yes, an entire LLM. ✨

Your Umbrel Home, Raspberry Pi (8GB) Umbrel, or custom umbrelOS server can run it with just 5GB of RAM!

Word generation benchmarks:

Umbrel Home: ~3 words/sec

Raspberry Pi (8GB RAM): ~1 word/sec

→ Watch the demo: https://youtu.be/iu3_1a8SzeA

→ Install on umbrelOS: https://apps.umbrel.com/app/llama-gpt

→ GitHub: https://github.com/getumbrel/llama-gpt

Reply to this note

Please Login to reply.

Discussion

Avatar
Vítor M. 2y ago

Can't wait to try it out!

Avatar
Vítor M. 2y ago

nostr:npub1aghreq2dpz3h3799hrawev5gf5zc2kt4ch9ykhp9utt0jd3gdu2qtlmhct ufff, I have 2tb nvme ssd, 24gb of ram (18 used atm) and an i7 10510U and it's reaaaly slow (less than 1 word/s).

Any tweaks to improve this?

Avatar
Shawn 2y ago

This is super cool, and your marketing is 🔥.

Avatar
Schlaus Kwab 2y ago

Nice! Can't wait to try out.

Avatar
Patrick 2y ago

Props to the maintainers for this!

Avatar
blank 2y ago

Can it run any llama2 fine tuned model?

Avatar
Mayank ☂️ 2y ago

Currently it uses the Nous Hermes Llama 2 (7B). If you’re technical, you can customize the Dockerfile to run a different Llama model: https://github.com/getumbrel/llama-gpt/blob/c76225a6fc26a000fc07b074223a69b0d65b7bcf/api/Dockerfile#L6

Avatar
YEGHRO 2y ago

Whoa! nostr:npub1aghreq2dpz3h3799hrawev5gf5zc2kt4ch9ykhp9utt0jd3gdu2qtlmhct stepping up their game with new addition of LLM AiChat bot.

nostr:note1c0yxqnnmjpe74j0ku9sm7nkgh93jjlmdjqdvtljzy02y58pa95aqkuss92

Avatar
Gracialo679 2y ago

Too much for bitcoin full node

Avatar
Xavier "The Label Guy" Fiechter 2y ago

Doesn't work for me on Umbrel Home.

Avatar
Derek Ross 2y ago

I'll be spinning up this docker container later today to play with it. Thank you!

Avatar
🦢 Cardxyz 🦬 2y ago

Umbrel ha sacado LlamaGPT, una IA que puedes descargar a tu ordenador y usar de manera privada y personal, tu información no sale de tu ordenador.

Estas son las iniciativas que me hacen creer en un mejor futuro para la tecnología e internet ✨😎

nostr:note1c0yxqnnmjpe74j0ku9sm7nkgh93jjlmdjqdvtljzy02y58pa95aqkuss92

Avatar
lazar 2y ago

On hetzner ARM machines it works really well. On CAX41 and Nous Hermes Llama 2 13B (GGML q4_0) i get like 9 tokens/s, cool for quick messing arround.

Avatar
Seneca 2y ago

Great idea, I was eagerly waiting for it. I installed it overnight on my Pi 8 GB, but was very disappointed when I started it this morning. It was VERY slow and could not answer my first two questions (first was relating historic exchange rates and second regarding stoic philosophy). Llama via nostrnet.work from nostr:npub1cmmswlckn82se7f2jeftl6ll4szlc6zzh8hrjyyfm9vm3t2afr7svqlr6f answered both questions quickly. Also, the other apps on my Pi became slower. So I deinstalled the app ... No offence to nostr:npub1aghreq2dpz3h3799hrawev5gf5zc2kt4ch9ykhp9utt0jd3gdu2qtlmhct, you did a great job in offering this one click install app, but anyone has to be realistic ... You cannot run a fully functional and competitive LLama on a Raspi Pi ...

Avatar
Tico 🇨🇷 2y ago

Had a flashback with that name:

"Winamp: It really whips the llama's ass" 🦙