Nostr Web Client

nostr:npub176p7sup477k5738qhxx0hk2n0cty2k5je5uvalzvkvwmw4tltmeqw7vgup Considering taking you up on that offer now. How does your search engine work? Are you maintaining a database of the most active npubs?

Pip the WoT guy 2mo ago

So, basically I am doing the following recursive process:

1. crawl the nostr network 24/7/365.

2. if an event is from a pubkey not in the db, skip

3. if instead it's from a known pubkey, save it

4. if it is a follow-list, and it includes new pubkeys, add them to the database.

So, inclusion in the database requires only one follows. There are 385k users already indexed, which covers the overwhelming majority of Nostr, minus spam.

Then, on the follow graph I run pagerank, to get reputational scores.

When you search on npub.world, the result is a combination of:

- text similarity (e.g. "jacky" is closer to "jack" than "mario")

- pagerank

Here is exactly how it does it.

https://github.com/vertex-lab/crawler_v2/blob/f2d7a654ea4c19e9aebee5c8fa950df77e8b9103/pkg/pipe/engine.go#L209

Reply to this note

Please Login to reply.

Discussion

The Daniel 🖖 2mo ago

This is great. Is the data available via API?

Pip the WoT guy 2mo ago

not API (yet), but as DVMs.

It's very easy to use, check this out

https://vertexlab.io/docs/services/search-profiles/

The Daniel 🖖 2mo ago

Ah, OK. So all the services are metered and need to be prepaid?

Pip the WoT guy 2mo ago

There is actually a generous free tier:

- first, you just have to tell me the pubkey you'll be using for signing the requests and I'll get you 10k credits on the house

- secondly, you can use the pubkey of the user to sign requests, and if they are minimally reputable they'll get automatically 100 credits every day. To give contex, that amounts to +100k users, and with 100 credits they'll be able to search for many hours most likely.

The Daniel 🖖 2mo ago

That’s helpful to understand. I’ll think a bit about what I might want to do with this. Let me know if you have some suggestions on how to create a zombie ranking engine. I’d also like to see if we can figure out who the Nostr users are with the largest follow counts who would benefit from a purge the most.

Pip the WoT guy 2mo ago

oh that's easy, you can use RankProfiles and specify sort=followersCount, so you'll get the users with the highest amount of followers.

You can even use Vertex for providing personalized recommendations for people.

It's more a question of what feature you want to build, rather than how to do it

The Daniel 🖖 2mo ago

I’m not interested in the number of followERS, just the number of npubs they’re followING, to identify and rank users with the largest amount of zombies.

Pip the WoT guy 2mo ago

ahn, got it.