Looks like there are some comments in the video that talk about the model choice being not ideal for a cluster like that. I wonder what kind of performance he can get with a model better suited to clustering
Framework Desktop mini-cluster by Jeff Geerling
https://blossom.primal.net/0fe3dc896d1e82c67d7865f48844f87c3503e2d4a05b3fcfff58484f81dbe114.mp4
Discussion
testing zaps for this note… we made two attempts to⚡zap this note, at benalleng@mutiny.plus, over a period of about 60 seconds. in each case, we found that your lightning address service or server did not respond correctly. if you wanted to fix this... you could try getting a free rizful lightning address -- https://rizful.com ... if u get it set up, pls reply here so we can do this ⚡zap test again.
Hmmm. Idk enough about LLM stuff to know.