Nostr Web Client

Should I block AI web crawlers on Oddbean?

On oddbean.com I see a *lot* of web crawling traffic from AI bots like GPTBot hoovering up nostr notes presumably for training purposes. I guess it's probably one of the easiest nostr sites to crawl since everything is rendered as plain HTML and they don't need to execute JS code to query relays.

To avoid wasting bandwidth I decided to use the following method to soft-block them (honour-system robots.txt): https://coryd.dev/posts/2024/go-ahead-and-block-ai-web-crawlers

You could argue they're just wasting my resources and won't bring any visitors or benefit the nostr community in any way. On the other hand, I guess they can/will access this data in some other way, and maybe the world-at-large gets some modicum of benefit from better AI models (?).

Thoughts? #asknostr

Reply to this note

Please Login to reply.

Discussion

Felix 1y ago

Maybe it helps Nostr developers if future gen models "know" more about Nostr, or improves llms for content moderation?

ᴛʜᴇ ᴅᴇᴀᴛʜ ᴏꜰ ᴍʟᴇᴋᴜ 1y ago

i classify the following events as "directory" events that help everyone get notes and user metadata and stuff:

0 ProfileMetadata,

3 FollowList,

5 EventDeletion,

1984 Reporting,

10002 RelayListMetadata,

10000 MuteList,

10050 DMRelaysList,

these help people find each other, i provide read access to them by default in #realy and i hope everyone will eventually follow this because it should be obvious that this glues the network together, especially relay lists and profile metadata, the amount of people i can find their notes but not their profiles on a regular basis is kinda retarded

ᴛʜᴇ ᴅᴇᴀᴛʜ ᴏꜰ ᴍʟᴇᴋᴜ 1y ago

i should probably add that in addition to this, i have designed an access control system that depends on follow lists, you designate "owner" accounts that designate follows, and these followed accounts, and follows of those follows are all whitelisted to read and write to the relay

this scheme also is facilitated by actively spidering other relays for that list of events especially the relay and follow lists, the profile metadata is superficial but helpful for providing clients with display names and avatars

someone 1y ago

nostr is source of wisdom. will change bad opinions of mainstream models. i actually have an article about this called Nostr Changes You. (one of those long form notes.)