Appreciate it! Yeah, we are cooking a much better WoT service
The current one is also failing very often.
btw I enjoyed this rip of nostr:npub10atn74wcwh8gahzj3m0cy22fl54tn7wxtkg55spz2e3mpf5hhcrs4602w3 with you and nostr:npub1qny3tkh0acurzla8x3zy4nhrjz5zd8l9sy9jys09umwng00manysew95gx. I’m looking forward to seeing reputation and WoT on zapstore!
Discussion
I’m also cooking up a WoT service. Still have a few bugs to work out but right now it calculates PageRank and GrapeRank scores for your follows network, which for my pubkey extends out by 8 hops and is 173k users. PageRank uses follows, GrapeRank uses follows and mutes and will soon incorporate reports as well. It takes 5 or 6 minutes to calculate it the first time for a new user but the results are cached and available by API.
That's cool. Why do you use two algos together? Like why not just Graperank?
I want to compare and contrast those two algos so I can better appreciate their relative strengths and weaknesses. Also, I’m using the FOSS graph database neo4j and neo4j graph data science which makes calculation of PageRank as easy as execution of a single cypher query, so no reason not to.
I do believe that GrapeRank is ultimately going to be better for most applications. It allows you to incorporate data from any source whatsoever (zaps, ratings, attestations, etc), calculate different types of scores for different contexts, tally trust-weighted votes and do weighted averages. Most importantly, it’s designed to tailor your WoT to the preferences, beliefs, and values of the end user. Over the past week I’ve figured out how to tweak some of the GrapeRank parameters to do a better job of stratification. It will improve further once I incorporate kind 1985 reports.
Very interesting. I'm currently running memgraph with their pagerank implementation but it's quite garbage. And yes a personalized view of the network is a must.
Is Graperank something you designed or it was an established algo previously?
My design. I’d love to find that someone has already implemented it bc I hate to reinvent the wheel, but if a similar centrality algo already exists, I have not yet found it.
What are the problems you’re running into with memgraph?
The biggest headache with neo4j is dealing with java garbage collection, but that’s not an insurmountable problem, just a bit of a learning curve. Other than that I’ve been happy with it. Getting a list of followers is instantaneous, something unthinkable with sql. Finding the shortest path between pubkey1 and pubkey2, is also pretty much instantaneous, even when they’re 7 or 8 hops away.
Oops, meant to say kind 1984.
5-6 min? Do you build the graph as well?
I maintain a cache of kinds 3, 1984 and 10000 notes that is updated continuously. I also maintain a graph database with pubkeys as nodes, and each follow and mute forms an edge. (Reports will be next.) Whenever a new kind 3 or 10000 note replaces an old one, the edges of the graph database get updated, individual edges being removed or added as needed.
There are tradeoffs maintaining a nostr cache versus downloading events every time on the fly. The former is more performant, the latter is perhaps more “pure.”
yeah, that's how we do it too. I don't get the 5min. You shared previously that pagerank takes about 15s. What am I missing?
Yup, about 15 seconds. The 5 min (probably a little less) is to run three algos:
1. Calculate DoS, degree of separation from the reference user to every other user
2. PageRank for every user, currently takes about 15 seconds.
3. GrapeRank for every user, currently takes about 60 seconds.
Then consolidation of all scores into a single ~ 20 MB json that can be accessed by API, put into a table for exploration.
So maybe more like ~ 3 min for now. (GrapeRank might take longer under certain conditions. Or I might get GrapeRank to go faster if I implement it using neo4j graph data science pregel but I haven’t tackled that yet.)
You can try it and tell me if it works for you! (Works for me on my laptop but I haven’t tested it thoroughly so i can’t promise it won’t break.)