Voting: Nostriches will say 1 or 2 and a reason why they chose that.
Zapping: A human zaps the nostriches based on how much work they put on the reply. Or the zap amount can be less for less effort, high for high effort.
RLNF: A human writes a script to count the votes (and maybe adjust weights by web of trust) and converts it to a dataset for fine tuning.
What do you think of this design?
I'm not sure how well it would scale. The human evaluating and zapping seems like a bottleneck and source of bias. Counting votes from a free-form reply with a script could be messy too. Is this intended to be done through regular kind 1 notes and replies?
yes kind 1
The best to way to find out how it will go wrong is to try it. Go for it!
so you are certain it will go wrong. but how much, depends on my execution? :)
