Indeed
I heard people putting screenshots of bitcoin price charts and asking AI for price predictions
In that case, you're not really asking the AI for its price prediction. Instead, you're asking it to predict what "technical analysts" would say about the price
And that's like asking astrologers. In a given field of (non-)expertise, the AI doesn't know how reliable the field is. It just predicts what humans would say in that field
AI is very good at writing software because the humans behind the AI know how to point the AI at reliable resources (like StackOverflow and Wikipedia) and because it's easier to see if a particular piece of software is working or not
The humans behind AI software want LLMs that can write good software, so that the LLM can improve itself, and therefore they constantly test and optimise that