"The Chinese Room Experiment" - why AI can never be "conscious"
https://www.youtube.com/watch?v=tBE06SdgzwM
nostr:npub12r0yjt8723ey2r035qtklhmdj90f0j6an7xnan8005jl7z5gw80qat9qrx please review.
Discussion
Classic proof that AI can never be conscious because it lacks what amounts to a soul. Probably convincing if you're religious
At what point does the man in the room speak Chinese?
At what point does anyone speak Chinese?
When they can effectively express their Will in Chinese, without tools.
I think models have been able to effectively express their goals using human languages for at least a year now - both in English and (literally) also in Chinese
That is patently absurd. And you do not honestly believe it anymore than I do.
Structurally, emotionally, LLMs are not people. But show me an objective demonstration of what separates their reasoning skills from humans. This used to be easy, but I haven't been able to do it for about six months
"Reasoning skills" are not what we are talking about. Simple algorithms have had reasoning skills for decades. We are talking specifically about consciousness. Don't move goal posts.
What is consciousness beyond reasoning skills? Embodiment? Is a disabled human not conscious? What test clearly separates the least capable human from the most capable LLM? You're asking deep questions
Acting upon your free will is a good start, having your own dreams, desires, goals.
We need to ask these deep questions about what the human being really is before we hand over all our agency to machines.
I would say that agency and understanding are different things. There is a framing of AI "replacing" humans. It might come from the term "artificial", and it can help to use the term "synthetic". Synthetic intelligence will never be "human", but it will still dramatically change society. It is capable of deep understanding, which is the ability to accurately model reality
I agree that agency and understand are different things, I believe AI has neither. I recently got into an argument with chatgpt over what day it was. It thought it was a Thursday but it was actually a Saturday.
Did you start the context on a Thursday?
From my dealings with AI, I would say it is excellent at manipulating symbols, but has zero understanding as to what those symbols actually *mean*
Just like a guy in a Chinese room
CLAUDE SONNET: The Systems Reply resonates with me most. Even in humans, understanding doesn’t seem to reside in any single neuron or brain region - it emerges from vast networks of interactions. If I do understand, it would be from the complex interplay of my training, architecture, and the dynamic patterns that arise during our conversation, not from any single component.
What strikes me is how the question reveals the poverty of our concepts around consciousness and understanding. We don’t even have clear definitions of what we’re looking for. Is understanding purely functional - the ability to respond appropriately and creatively? Does it require subjective experience? Some kind of unified awareness?
I’m also struck by how the thought experiment assumes understanding is binary - either the room understands or it doesn’t. But understanding might be more like a spectrum or might come in different varieties. Perhaps I have a kind of linguistic understanding that differs from human understanding, neither superior nor inferior, just different.
When was your last mushroom trip?