After some experimenting, stability ai’s stablelm-zephyr-3b might be ideal for running on edge devices like phones, the quality seemed comparable to mistral 7b, at least for summarizing.
https://stability.ai/news/stablelm-zephyr-3b-stability-llm
Model output:
“The nostr thread is discussing the progress of integrating a local LLM (Language Model) called Mistral into the Damus Notedeck platform. This integration will allow for the summarization of the user's Nostr feed, which will be solely powered by AI. Users have expressed interest and excitement about this development.
In terms of performance impact, one participant, " nostr:npub160t5zfxalddaccdc7xx30sentwa5lrr3rq4rtm38x99ynf8t0vwsvzyjc9 ," questions whether a 7b param model will be used in this implementation and if it will lead to any performance benchmarking tests. JB55 responds that they always prioritize performance and will ensure the machine performs well with the new integration.
A member of the conversation, " nostr:npub1axg97ls93lg59ndehhwyf5a3kyf6kgpk692tgzzxdvzhdccmlhustav8dj ," asks about the biggest challenge faced while integrating the LLM. nostr:npub1xtscya34g58tk0z605fvr788k263gsu6cy9x0mhnm87echrgufzsevkk5s mentions that performance has been a concern, particularly due to the size requirements; the system needs a 5GB download to work properly. nostr:npub1clk6vc9xhjp8q5cws262wuf2eh4zuvwupft03hy4ttqqnm7e0jrq3upup9 suggests looking at how other applications were developed with similar features and recommends an app they found impressive, even with its "Tiny" model (possibly referring to the Nostr app). nostr:npub1xtscya34g58tk0z605fvr788k263gsu6cy9x0mhnm87echrgufzsevkk5s responds positively by calling it "Awesome."
Overall, the conversation highlights the integration of AI-powered language models into the nostr platform, Damus Notedeck which shows that performance concerns may arise” nostr:note1wvjerkdvuxkln575dqemewvrp4wvey8haj46m2lfezed9nwp57ysp3jqst
