So, basically I am doing the following recursive process:
1. crawl the nostr network 24/7/365.
2. if an event is from a pubkey not in the db, skip
3. if instead it's from a known pubkey, save it
4. if it is a follow-list, and it includes new pubkeys, add them to the database.
So, inclusion in the database requires only one follows. There are 385k users already indexed, which covers the overwhelming majority of Nostr, minus spam.
Then, on the follow graph I run pagerank, to get reputational scores.
When you search on npub.world, the result is a combination of:
- text similarity (e.g. "jacky" is closer to "jack" than "mario")
- pagerank
Here is exactly how it does it.
https://github.com/vertex-lab/crawler_v2/blob/f2d7a654ea4c19e9aebee5c8fa950df77e8b9103/pkg/pipe/engine.go#L209
This is great. Is the data available via API?
not API (yet), but as DVMs.
It's very easy to use, check this out
https://vertexlab.io/docs/services/search-profiles/
Ah, OK. So all the services are metered and need to be prepaid?
There is actually a generous free tier:
- first, you just have to tell me the pubkey you'll be using for signing the requests and I'll get you 10k credits on the house
- secondly, you can use the pubkey of the user to sign requests, and if they are minimally reputable they'll get automatically 100 credits every day. To give contex, that amounts to +100k users, and with 100 credits they'll be able to search for many hours most likely.
That’s helpful to understand. I’ll think a bit about what I might want to do with this. Let me know if you have some suggestions on how to create a zombie ranking engine. I’d also like to see if we can figure out who the Nostr users are with the largest follow counts who would benefit from a purge the most.
oh that's easy, you can use RankProfiles and specify sort=followersCount, so you'll get the users with the highest amount of followers.
You can even use Vertex for providing personalized recommendations for people.
It's more a question of what feature you want to build, rather than how to do it
I’m not interested in the number of followERS, just the number of npubs they’re followING, to identify and rank users with the largest amount of zombies.
Thread collapsed
Thread collapsed
Thread collapsed
Thread collapsed
Thread collapsed
Thread collapsed
Thread collapsed
