When you start crawling relay lists to aggregate relays, you realize how all sorts of fucked up a lot of relay lists are.
Parsing all this and throwing out everything that's not a valid clearnet websocket server is a lot of crap to clean up. Getting closer though. Finding about 890 online clearnet relays.
https://tenor.com/view/rookie-numbers-gif-26135237
#crawlr
Have fun! And good luck
Man, this journey you are on is really fascinating!
Yep... Welcome to the club. That number sounds right
nostr:npub1zmc6qyqdfnllhnzzxr5wpepfpnzcf8q6m3jdveflmgruqvd3qa9sjv7f60 Are you deduping with NIP-11? There's all kinds of fun in there.
I just worked up something this morning to get NIP11 but it's so early. I was just curious which implementations were most prevelant.
Cool! Have you considered running a monitor and publishing NP-66 events by chance? Also: 
Only after the crawlr and blastr are done and WoT is implemented in grain. Grain is my priority. After I feel like it's in good shape, I'll make the crawlr do more.
Looks cool! Something like Khatru?
Waaaay more than Khatru. Here's the config for a good idea of the current capability. My objective is a highly configurable relay to cover a multitude of use cases. Uses mongodb but more databases are planned, way later...
https://github.com/0ceanSlim/grain/blob/main/app%2Fstatic%2Fexamples%2Fconfig.example.yml
Ah, ok, so much different. Khatru is more of a framework whereas this looks something closer to nostream where there are baked in policies you can configure. I like that there's fine grained control over resources. Can't wait to try it!
I want to do some more resource control with database size but I'm waiting until I get the majority of everything else done and add multi db support since I'll have to do different functions to check the size depending on the database you use.
Then I can tie in purging with a database size limit.
How does your result overlap with the nostr.watch data? Are they looking at fewer relays or are they filtering out more of them, to get their 700ish result?
Mines probably got some in there that aren't actually online. It's very buggy and gets a lot of junk still. But 700-900 seems about the right range.
