The pushback against AI crawlers threatens the transparency & open borders of the web that allow non-AI apps to flourish. If unfixed, the web will increasingly be fortified with logins, paywalls, & access tolls.
https://www.technologyreview.com/2025/02/11/1111518/ai-crawler-wars-closed-web/