Replying to 21823843...

Should I block AI web crawlers on Oddbean?

On oddbean.com I see a *lot* of web crawling traffic from AI bots like GPTBot hoovering up nostr notes presumably for training purposes. I guess it's probably one of the easiest nostr sites to crawl since everything is rendered as plain HTML and they don't need to execute JS code to query relays.

To avoid wasting bandwidth I decided to use the following method to soft-block them (honour-system robots.txt): https://coryd.dev/posts/2024/go-ahead-and-block-ai-web-crawlers

You could argue they're just wasting my resources and won't bring any visitors or benefit the nostr community in any way. On the other hand, I guess they can/will access this data in some other way, and maybe the world-at-large gets some modicum of benefit from better AI models (?).

Thoughts? #asknostr

Avatar
ᴛʜᴇ ᴅᴇᴀᴛʜ ᴏꜰ ᴍʟᴇᴋᴜ 1y ago

i classify the following events as "directory" events that help everyone get notes and user metadata and stuff:

0 ProfileMetadata,

3 FollowList,

5 EventDeletion,

1984 Reporting,

10002 RelayListMetadata,

10000 MuteList,

10050 DMRelaysList,

these help people find each other, i provide read access to them by default in #realy and i hope everyone will eventually follow this because it should be obvious that this glues the network together, especially relay lists and profile metadata, the amount of people i can find their notes but not their profiles on a regular basis is kinda retarded

Reply to this note

Please Login to reply.

Discussion

Avatar
ᴛʜᴇ ᴅᴇᴀᴛʜ ᴏꜰ ᴍʟᴇᴋᴜ 1y ago

i should probably add that in addition to this, i have designed an access control system that depends on follow lists, you designate "owner" accounts that designate follows, and these followed accounts, and follows of those follows are all whitelisted to read and write to the relay

this scheme also is facilitated by actively spidering other relays for that list of events especially the relay and follow lists, the profile metadata is superficial but helpful for providing clients with display names and avatars