Actually i think it’s significantly higher than 180/m, i was looking at the wrong model. Would be more like $800 per month from the price ive been seeing online. Yeah maybe not worth it for now. Will wait till these get cheaper.
671B param deepseek r1 running at 11tok/s on your desk with two of these. $180 a month for 2 years to pay this off, about the same as a chatgpt pro subscription. It’s a bit slower but you get privacy. Hmm…
https://x.com/alexocheema/status/1899735281781411907?s=46 nostr:note1rn95d949hlzjuaef20pvwly36wy090wpnqaypsn3g6q2jmc2lc2qeucxkd
Discussion
Yeah, was just going to say those boxes are $10k *each*. A $2k Epyc is slow, but not that slow
epyc has faster inference then an m3 ultra?
No, but you can get a few tokens per second for $2k
I went with two sockets and 1TB of RAM, but haven't gotten twice the performance of this yet: https://digitalspaceport.com/how-to-run-deepseek-r1-671b-fully-locally-on-2000-epyc-rig/