If you have a recent, fairly powerful machine with 16B+ RAM and are comfortable tinkering with command line stuff, try ollama first.
Discussion
Oef thanks will have a look
Hit me up if you want to talk through. It’s a really impressive project.
Wondering how much RAM people should generally put into new upper-end PCs these days given (local) AI models and modern games. Thinking 64 gigs.
I’m wishing I had more than 16, but it’s still workable, especially for 7B models.
Another ambitious project for people go look at: