Quite so! I just finished reading through the “Race” ending. As usual, I find that authors are happy to apply superexponentiation to some things but not others.

For example, in the essay, they imagine AI being turned to fuel faster AI research (plausible). And they posit similar booms in medicine and business (plausible).

But they also posit that the AI couldn’t get out in front of pollution? Seems like an easy problem. And none of the research went into morality, ethics, philosophy? That’s where it loses me.

Humanity is “the environment” of AI, in much the same way that nature is “the environment” for humans. It seems logical to me that superintelligent AI would angle for a controlled symbiosis, much as people seek regenerative and sustainable (but controlled) relationships with nature.