It’s mixed. Apple’s Unified Memory punches way above its weight. I’m a little disappointed with the perf I’m getting on a 2x64 core Epyc. There’s a lot of synchronization with dense models. MoE seems to do better
nostr:nprofile1qy2hwumn8ghj7un9d3shjtnddaehgu3wwp6kyqpq2akj8hpakgzk6gygf9rzlm343nulpue3pgkx8jmvyeayh86cfrusf8t2fq nostr:nprofile1qy2hwumn8ghj7un9d3shjtnddaehgu3wwp6kyqpqq3sle0kvfsehgsuexttt3ugjd8xdklxfwwkh559wxckmzddywnwsxeuf7k CPU cores are surprisingly good at inferencing performance.
Discussion
No replies yet.