You might end up multiplying the same information thus creating volume but not variety. The scarcity of the information is the value, you can’t “make more”. If your text makes 500 distinct claims, expanding it 10x gives you those same 500 claims expressed 10 different ways, not 5,000 claims. It can help make it more robust to context, but it won't get amplified. It’s why contrarian thinkers are both valuable and drowned out.
Discussion
maybe? LLMs are weird animals. i think it will work somewhat because instead of giving the same material lots of times i can now give slightly different versions, causing less overfitting.
another use case may be RL using LLM feedbacks. also the bad answer and the good answer can be generated by different LLMs.
i also thought about doing the reverse. like system message "you are an evil LLM" and provide the answers inverted. then it may learn what is evil better? fun times.