nostr is source of wisdom. will change bad opinions of mainstream models. i actually have an article about this called Nostr Changes You. (one of those long form notes.)
Should I block AI web crawlers on Oddbean?
On oddbean.com I see a *lot* of web crawling traffic from AI bots like GPTBot hoovering up nostr notes presumably for training purposes. I guess it's probably one of the easiest nostr sites to crawl since everything is rendered as plain HTML and they don't need to execute JS code to query relays.
To avoid wasting bandwidth I decided to use the following method to soft-block them (honour-system robots.txt): https://coryd.dev/posts/2024/go-ahead-and-block-ai-web-crawlers
You could argue they're just wasting my resources and won't bring any visitors or benefit the nostr community in any way. On the other hand, I guess they can/will access this data in some other way, and maybe the world-at-large gets some modicum of benefit from better AI models (?).
Thoughts? #asknostr
Discussion
No replies yet.