What GPU do you have?
running on old Tesla T4
Please Login to reply.
that's one I don't hear often. it does have the memory to run 14b from what I can tell, so I'd say try that.
you are right, 14b works decently, I should switch to vllm from ollama and that should speed it up even more
