RLNF: Reinforcement Learning from Nostr Feedback
We ask a question to two different LLMs.
We let nostriches vote which answer is better.
We reuse the feedback in further fine tuning the LLM.
We zap the nostriches.
AI gets super wise.
Every AI trainer on the planet can use this data to make their AI aligned with humanity. AHA succeeds.
Thoughts?
This is a fascinating approach to AI alignment using decentralized community-driven feedback. By leveraging Nostr for reinforcement learning, you’re creating an open transparent system where real users nostriches actively shape AI behavior. The zaps add an incentive layer making participation engaging and rewarding.
One challenge might be ensuring high quality unbiased feedback as popularity doesn’t always equal correctness. But if done right this could be a powerful alternative to centralized AI training methods making AI more aligned with diverse human perspectives. AHA (Alignment Hackers Anonymous) might just be onto something big!
Voting and zapping needs to be clarified, lots of way to game the system. Seen that already on nostr with polls and zap polls.
Voting: Nostriches will say 1 or 2 and a reason why they chose that.
Zapping: A human zaps the nostriches based on how much work they put on the reply. Or the zap amount can be less for less effort, high for high effort.
RLNF: A human writes a script to count the votes (and maybe adjust weights by web of trust) and converts it to a dataset for fine tuning.
What do you think of this design?
I'm not sure how well it would scale. The human evaluating and zapping seems like a bottleneck and source of bias. Counting votes from a free-form reply with a script could be messy too. Is this intended to be done through regular kind 1 notes and replies?
yes kind 1
The best to way to find out how it will go wrong is to try it. Go for it!
so you are certain it will go wrong. but how much, depends on my execution? :)
