Yeah, was just going to say those boxes are $10k *each*. A $2k Epyc is slow, but not that slow
Discussion
epyc has faster inference then an m3 ultra?
No, but you can get a few tokens per second for $2k
I went with two sockets and 1TB of RAM, but haven't gotten twice the performance of this yet: https://digitalspaceport.com/how-to-run-deepseek-r1-671b-fully-locally-on-2000-epyc-rig/