What are the system requirements for the 405B and 70B versions? Having trouble parsing the website myself.
I'm trying to understand how to replace my not-very-private but useful ChatGPT 4.0 subscription with Llama 3.1.
ChatGPT translated the system requirements for https://llama.meta.com/ into slightly less confusing versions of "your beefy desktop is by far not enough".
So if I still need a compute cluster that would sit idle 99% of the time if I ran it just for myself, I'm kind of back at square one. I'd have to find a way to share these resources efficiently and privately.
Where can I use a powerful AI in a privacy preserving way? I want to pay with eCash and use it via TOR without any email or other accounts attached.
Discussion
It's weird they don't prominently put these models side by side to help pick one.
I want the best and I want privacy. Meta itself gave me some numbers here:
With 64GB RAM I’d guess you could locally run the 70B version just fine with a 4000 series Nvidia GPU. Not sure about the 405B version. Supposedly the 5000 series GPUs are releasing beginning of 2025. I wonder if they will have LLMs in mind in their design.
I assume the GPUs are needed for performance but not for quality of the results, right? So if for my own private use I don't have any GPU, I could still get the same results but would have to wait a minute instead of a second.
With the 405B model? I don't think you can run this with only 64gb and no GPU, not with acceptable wait times.
There is no way you will ever run the 405bn parameter model on consumer hardware.
For all the smaller models, a M3 MacBook pro with maxed out RAM works incredibly well. The M3 chip architecture makes it so that all the RAM is available to the GPUs as well, so you can get up to 96GB of RAM.
I can run all the models with ollama with crazy speed.
I wouldn’t rule out consumer LLM focused cards with tons of VRAM
Huggingface?
I see there are many more models on huggingface but can't see good data on what to expect depending on which model you run and on which hardware. I assume the 405B model produces qualitatively other results than the 8B model so I probably want the highest possible for my programming tasks.