Which IP is the one misbehaving? At nos.social, we have a couple of scrapers. One respects rate limits, though we might need to adjust the backoff settings. The other one is more likely to be causing the issue, as we haven’t had time to implement rate-limiting there yet. This could be a good wake-up call for us to prioritize that.
for the first time, since way back in the day, i took some ip blocking actions for nostr1.com.
scrapers have gone wild.
please make sure your scraper is nice, and listens to error responses with exponential backoff. nostr1 will send a 403 for "relay is gone".
many scrapers (i count 4+) are connecting to relays that do not exist in a tight loop and while usually i can handle this traffic, now it annoys me.
nostr:nprofile1qqs8d3c64cayj8canmky0jap0c3fekjpzwsthdhx4cthd4my8c5u47spzfmhxue69uhhqatjwpkx2urpvuhx2ucpz3mhxue69uhhyetvv9ujuerpd46hxtnfduq3vamnwvaz7tmjv4kxz7fwdehhxtnnda3kjctvll3q2p nostr:nprofile1qqs99d9qw67th0wr5xh05de4s9k0wjvnkxudkgptq8yg83vtulad30gpp4mhxue69uhkummn9ekx7mqpz3mhxue69uhhyetvv9ujuerpd46hxtnfduq32amnwvaz7tmgd9ehgtnwdaehgu3wd3skueqyluj3x lmk if u can fix it and ill unblock. the others i have no idea who they are, ..
Discussion
connect.nos.social is one, i stopped looking up dns after a while.. the list is maybe 40 IPs now, im not 100% sure why these scrapers are obsessed with gone relays .. i appreciate you looking into it, lmk what you find.