Maybe it helps Nostr developers if future gen models "know" more about Nostr, or improves llms for content moderation?
Should I block AI web crawlers on Oddbean?
On oddbean.com I see a *lot* of web crawling traffic from AI bots like GPTBot hoovering up nostr notes presumably for training purposes. I guess it's probably one of the easiest nostr sites to crawl since everything is rendered as plain HTML and they don't need to execute JS code to query relays.
To avoid wasting bandwidth I decided to use the following method to soft-block them (honour-system robots.txt): https://coryd.dev/posts/2024/go-ahead-and-block-ai-web-crawlers
You could argue they're just wasting my resources and won't bring any visitors or benefit the nostr community in any way. On the other hand, I guess they can/will access this data in some other way, and maybe the world-at-large gets some modicum of benefit from better AI models (?).
Thoughts? #asknostr
Discussion
No replies yet.