Anybody here wants to work on the data querying together?
I have a few things built using DBT(data build tool) for nostr analytics.
Discussion
could you share your progress?
Sure, I'm gonna write a lengthier post later but essentially I have the following:
1- I vibe coded an event listener client that stores all events from a relay into a data warehouse. It needs some work still as I suspect it is dropping part of the events.
2 - using the data from the above I started a DBT project (which can become a community template) for getting basic analytics from nostr events in the warehouse. I.e. the follow list of an account, top reacted posts and other aggregations.
The idea was to use the data also to create an open source a few models. Such as: how to flag an account as a bot, or to build a social graph of an account.
This is the same data just plotted on my metabase instance. Just connecting all the dots.
Could you share what use cases you are thinking of when you mention "quering data"?
Could you share the use cases you have in mind for nostr querying?
many use cases:
- pagination
- aggregation (how many likes does this post have?)
- better filtering (show me all replies to me)
- trending posts
- web of trust stuff
querying nostr events feels like working with stone age technology and nobody seems to care enough to improve the relays. people have gotten used to painful development, wasted bandwidth, and client-side filtering. it's not normal.
What I'm working on addresses aggregation directly.
It's still a bit messy but I'm sure we can build an api layer on top of it and leverage bigquery for a lot of this.
https://github.com/FKrauss/dbt_nostream
