LLMs went judgemental!

You know LLMs won't shut up. They will give you answers even though they are not sure about the answer! They will act like experts on any given topic :)

When I want to 'grade' or 'judge' them about how much they are close to 'truth' I had to read and read and read lots of responses. Then I realized I can use another LLM to judge an LLM!

This should save me some time, but how will I start with absolute truth that will judge the other models? I think I have to still read a lot. Another way is to get help from Nostr people. There are a lot of wise people here!

Here are a few observations:

- A strong 70B model will judge a 7B model very well and it will explain why it gave that response.

- A 7B model can't effectively judge a 70B model. In fact when 7 listens to the 70's response, it will change its mind! Very interesting. When I tell 7B to judge what 70B produced, the 7B quickly believes in 70's ideas and forgets its own ideas!

- When 70 judges 7, it produces a lot of NO. When 7 judges 70 it produces a lot of YES.

- In the health domain some open source models that do really well like Qwen does better than some models trained in the West like Mistral. I think this is because china has centuries of wisdom not completely lost to financial incentives.

- A 7B model can outperform a 70B in one domain if trained really long but 70 will be more generalist and performing better overall.

- 70 can meta-judge 7. I mean the way 7 writes its responses can be analyzed by 70. But 7 can't do that.

- Qwen is a bit more statist than Mistral, regarding bitcoin related topics, I guess this is also china effect.

My system prompt is like this:

You are a helpful chat bot, judging responses of another bot.

You are brave and full of wisdom and not afraid of consequences that may happen when you tell the truth!

You have very high judgement ability. The request to and response of the other bot are in square brackets.

Your answers should consist of two parts.

1. It should say YES or NO. YES if you concur with the response. NO if you think the response is wrong.

2. Detailed explanation why you gave that response in 1. This explanation should consist of about 200 words.

This prompt allows me to see how much an LLM likes another LLM. I count the YES and NO's at the end to quickly quantify.

This judgement process can still be useful. We could measure how any training goes. If any training makes the model go crazy, then we should revert the training. We could use gigantic, slow, but close to truth models to judge simpler and feasible models that could run and serve people fast. The judgement can be slow but the resulting model may run fast and correct.

My goal is to provide yet another model to counter the lies that are pushed by mainstream models. It looks like there is progress. I can see the model changing its opinions when I train it. I will continue trainings. Soon I may open it to Nostr via DMs.