Can someone explain to me how people reason about the dangers of AGI? Why a program that guesses the next word of the given text would suddenly, or gradually, develop a motivation to do something its not "supposed to" do? Or feel an emotion, which sounds particularly nuts to me, like my depressed spaghetti python script will make an appointment with a psychiatrist.

It is confusing we even call it Intelligence and not A simulation **of** Intelligence

To give a metaphor, if you yell into a cage it will generate new, somewhat different sound, which will look like there's another someone there, but you surely won't think the cage is intelligent, let alone will be able to develop the capacity to feel if you feed it with enough yelling