What is the best tool to scrape all relays for specific events as a one-off task?
For example I have a list of 50 relays and want to know which version (event id) of my profile is stored on each of them.
My current approach is to go to https://nostr.info/relayr/ , patch the `ws.onmessage` callback to `console.log` what I need and then to either use the UI or further call on the dev console what the UI would call.
Fellow nostr devs, how do you do this?
Good question.
I wrote a director/producer/consumer setup
producer
connects to a bunch of relays, sends queries and pipes the results to a PUBSUB on redis (sharded by kind number)
director
orchestrates sending queries to the producers
consumers
does *something* with the data on the PUBSUB
(e.g. feeds nostr:npub1pages5yhmncmmfxuza9f9fx68fju0nxnewtlffzrapsufax5mv0sznuw9d, updates my neo4j, etc)
that's how I do it 🤷♂️ 😂
Doesn't sound like a tool anyone could try out quickly :(
if you feel like a golang code example (or just command line tool)
For what it's worth, I opted for Deno. The following finds all t he accounts that have accepted badges on the Damus relay:
```
#!/bin/sh
//bin/true; exec deno run -A "$0" "$@"
import { Nostr, Relay, NostrKind } from "https://deno.land/x/nostr_deno_client@v0.2.7/mod.ts";
// import { Nostr, Relay, NostrKind } from "../nostr.ts";
const nostr = new Nostr();
nostr.relayList.push({
url: 'wss://relay.damus.io'
});
nostr.on('relayConnected', (relay: Relay) => console.log('Relay connected.', relay.name));
nostr.on('relayError', (err: Error) => console.log('Relay error;', err));
nostr.on('relayNotice', (notice: string[]) => console.log('Notice', notice));
await nostr.connect();
const set = {};
var round = 1;
var d = Math.round(Date.now() / 1000);
while (true) {
const filter = {
kinds: [30008],
until: d - 1,
limit: 200
};
const allNotes = await nostr.filter(filter).collect();
console.log(`Round ${round++}. We had ${Object.keys(set).length} users and found ${allNotes.length}, probing until ${new Date(d * 1000)} ...`)
if (allNotes.length === 0) {
break;
}
d = allNotes.reduce( (result, item) => {
return Math.min(result, item.created_at)
}, d)
for await (const note of allNotes) {
set[note.pubkey] = note.tags.length;
}
}
for (const [k, v] of Object.entries(set)) {
console.log(`${k} ${v}`);
}
Deno.exit();
```
1249 accounts accepted badges. How many of them might be interested in my "Badges Badge" https://badges.page/b/naddr1qqx5jgrvd94k2grzv9jxwetnqgsydl97xpj74udw0qg5vkfyujyjxd3l706jd0t0w0turp93d0vvungrqsqqqafew90d6s ?
