I think they'd probably just regurgitate the commentary of someone who's actually thought about it. No LLM actually thinks for itself.
Discussion
Same thought. It's a mistake to attibute consciousness to a computer in the way I have been seeing lately.