Well it’s definitely a lot slower running right now, but as long as it is accurate I guess it’ll be worth it.
This may be more of a couple hour process for anyone who is following a lot.
Discussion
if you want to speed it up, first find which npubs have which relays in common (cluster), then do a single req with all the authors for each relay
Well it seems to have performed well. The only npub that proved to be an exception is nostr:nprofile1qqsph5e28p48hehk3zeaclzgpm7zrnv5ddp74s2t5ja8sd9vw73ru6gprpmhxue69uhhyetvv9ujuumwdae8gtnnda3kjctvqyg8wumn8ghj7mn0wd68ytnhd9hx2ev76ap but other than that it seems entirely accurate
Sorry Alan, but I don’t know why I can’t prove you’ve done anything in 140days. Even though I know you just posted.
Any ideas?
Well, I went ahead and tried to update it with what you said, but it seems that it is just as slow somehow when utilizing the cluster of relays.
"Clustered Outbox Analysis
Running
Discovers user relays, clusters users by common relays, then batch queries each relay cluster for maximum speed and accuracy
Checking: Relay clustering + batch queries"
Have any more ideas of what it might be doing inefficiently?
The optimization works by:
Phase 1: Discover relays for each user (NIP-65, NIP-02, activity-based)
Phase 2: Create relay clusters Map
Phase 3: For each relay cluster, make 3 optimized batch queries
Phase 4: Distribute results back to individual users with deduplication
the discovery should probably just happen once a day on the backend job, then each user is just served a cache based map on their existing list. basically your backend should know 99% of all npub's outbox relays without checking for each user.
then a user comes with a follow list, you should be able to use that to return a cluster map nearly instantly
then req all latest notes limit 1 per author to every relay at once, a browser can handle easily 50 websockets