I am going to fetch my conversation and upload it on nostr.build, I did a sort of mental experiment with ChatGPT and asked to classify and give probabilities to what could eventually go wrong with AI. If I am not wrong it gave up to 35% chances that the AI would show unintended and novel behavior. There were more interesting things, I will try to make a thread about it. However, on the rest I agree with you. Even if this is not AGI and has no conscious experience (yet), I doubt we will realize when it will and especially why.