Nostr Web Client

Cool. What LLM are you running and at what resolution? How’s the performance (tps) and accuracy on the single 3090? Do you think it can replace Claude?

Ben Weeks ⚡ 5mo ago

Trying a few different ones at the moment, such as Llama3.2:3b, qwen2.5:3b, mistral:7b, deepseek-r1:7b and deepseek-r1:14b. Right now qwen2.5:3b (Q4_K_M) is doing around 144 TPS (according to Claude Code).

Deepseek-r1:14b was too much for it.

Here’s what I’m working on:

https://github.com/knowall-ai/Nod.ie

I.e. an agent that runs on your machine that you can just talk to, all running locally, that ultimately can call MCP Servers concerning the operation of a Bitcoin Lightening Node. Under the hood it’s using a fork of Unmute for the backend services, but I’ve added MCP tool calling capability and “thinking”.

However, this is also using a lot of other GPU activities such as Speech-To-Text (TTS) and Text-to-Speech (TTS) and I was even trying to add video processing stuff (agent facial animation) using MuseTalk but it can’t quite all squeeze on a single RYX 3090 (24GB). Looking at options to run a second graphics card (until then putting animated AI on hold):-)

I don’t think it will replace Claude Code (I use that a lot to build it) because of the clever way they grab context of a project code base and alike. If you’re referring to Claude Desktop, I’m using Open WebUI and seems pretty comparable (at least to Open AI - you can even get results from your local Ollama instance and compare it to results from OpenAI API which is pretty cool).

Reply to this note

Please Login to reply.

Discussion

unixmonks 5mo ago

Have you tried goose?

Ben Weeks ⚡ 5mo ago

No, I haven’t actually, just read about it. So it’s nostr:nprofile1qqsgydql3q4ka27d9wnlrmus4tvkrnc8ftc4h8h5fgyln54gl0a7dgspp4mhxue69uhkummn9ekx7mqpzemhxue69uhhyetvv9ujumn0wvh8xmmrd9skcju9hvt's baby AI :-) Looks like a similar concept with locally running AI, full open source stack. Even uses an Electron App for the UI too. Will check it out thanks! nostr:nprofile1qqsd0hut8c2pveuk4zkcws9sdap8465am9dh9cp8d2530yssuflcracprpmhxue69uhhyetvv9ujuumwdae8gtnnda3kjctvqyt8wumn8ghj7un9d3shjtnwdaehgu3wvfskueqqyh5dl

unixmonks 5mo ago

It’s more along the lines of Claude code

Ben Weeks ⚡ 5mo ago

Ah ok. Will give it a try. Using AI to build AI is pretty surreal especially how you can see it adjusting the way it’s “brain” works.

Ben Weeks ⚡ 5mo ago

Being able to talk to Claude Code rather than type and then realise half of what you wrote didn’t get input as it asked a question would be nice. Perhaps Goose does that. Would allow for much more context to be passed to the agent.