New toys :)
Discussion
GPU Twins! Plans to local LLM much? :)
Feeling cute might load gpt-oss up later :)
Id like to run vgpu and hopefully share that gpu compute around if I can manage it.
Let me know how you experiments go! If this a datacenter GPU, or maybe older consumer grade one? (NVIDIA locks the new consumer stuff so hard the even simple passthrough has been sorta painful)
Will do! They are p100s. So they should be DC, not sure about firmware though if that's what you are suggesting. I had Titan X maxwells installed and just swapped these in.
I've seen PopOS! dealing with multiple cards (rtx in this case) and sharing resources nicely. So I think this is built-in in (proprietary) drivers.
The problem isn't having multiple cards. NVIDIA makes it hard to use "enterprise" features like virtualisation, passthrough, etc. It’s all doable AFAIK, but even with AMD hardware, getting proper "hot" passthrough from a Fedora host to a Windows guest and back was somewhat painful (Wayland itself makes it tricky since it really clings to the GPU and doesn’t want to let it go :)).
oh, yes. passing through video cards to vms is definetely not fun.
I use poman to containerize these things. podman-desktop has even AI extension which makes it a breeze ...
I'll receive mine tomorrow
Oh yeah! Congrats. This is an upgrade from 24gb to 32gb which is enough for me right now. Hopefully the FP16 was worth the 32 vs 48gb memory cut.
Keep us posted!
I'm bit sceptical that 24g will be enough so I might endup buying one more card 🫣
Will do! gpt-oss does well (about 16gb) Gemma 27b, qwen, mistral and devstral all run with room to spare. Unleses your running like 36b+ without quantization I think youll be fine! IMO The larger models from under 20b to under 30b aren't a massive improvment overall, Im not sure depending on your workload going past a mid 30b something param model will be worth the money spent on hardware.
Not sure if I will really hit some bottlenecks, but things may get ugly with parallel workloads.
I'd like to Uncle Jim my hw.
lmao I see. Yeah I haven't had the opportunity to try that, mostly because I know I can't squeeze another model into memory and my CPU/Memory are old (ivy bridge ddr3) so running on CPU is painful and power hungry.