Replying to Avatar Raul007

Awesome. I was wondering whether it was worth it to grab some extra system RAM to try a 70B model on CPU only, but it runs just as slow as my CPU w/ GPU offloading.

🤷 I'll just have to wait for used 3090s to come down in price.
Avatar
techonsapevole âš¡ 2y ago

it's a server, i didn't want a gpu to increase the energy consumption.. 48Gb @ 5200 is a lot but you can use it for VMs and a smaller LLM at the same time

Reply to this note

Please Login to reply.

Discussion

No replies yet.