When Derek mentioned there were some performance issues, I started tracking the messages returned by the server and saw it mentioned database error on a few REQs. Do you know if when you built and deployed it manually had you enabled read replica support in D1? It also seemed to be edited a bit to support some new filters/tags? I see you’ve since switched to a different implementation now, that’s fine. In Cloudflare, you can add “members” that can view a particular project, but don’t have admin privs. If you’d like, I’d love to check the logs of the worker and the original config. Happy to help any way. Maybe there’s still some use for Nosflare in the future. Cheers 🤙
Hey Nostr,
I need your help. Divine.video as you might have seen is a new video nostr app that i've been working on for the last 3 months.
It got MUCH more attention than I was expecting. Hundreds of millions of people viewed, liked, or shared videos about it. I've got some of the biggest original Viners in my DM's begging to get back on it. The TestFlight hit it's 10k limit in a few hours.
I'm excited but also really stessed out. We've had lots of bugs and Apple and google have been their usual black boxes when it comes to app review.
A bunch of folks have stepped up to help, nostr:nprofile1qqsr7acdvhf6we9fch94qwhpy0nza36e3tgrtkpku25ppuu80f69kfq9q9kky got the android build working for example.
Lots of things have broken, nobody really knows how survive a flash flood. I'm sharing this because I need help. We've got a chance to really grow nostr, the idea of a video app that's not got AI slop and does focus on something more human is resonating. People hate what's happening to tiktok, instagram, and youtube shorts where algorithms and the platforms love of AI generated content going viral is taking over. Instead of fighting back we see AI only platforms like MetaAI and Sora. This is an assault on the very idea that people are central to social media. I think big companies see the shine of AI generated content and dream of a world without all these pesky rabble making demands of platforms. If only they could replace the creators with bots.
This call to action felt right to me, but holy shit I had no idea it'd go so viral.
The app has lots of bugs, and we need appstore approval, but at the moment the biggest problem I have is relays. I need you, the nostr community's help. I started out with strfry which we know scales but lacks search. So i started using nosflare, https://github.com/Spl0itable/nosflare , by nostr:nprofile1qqsdfx5syw3pmwsm8jpsdj3kn0ejg0vtgju0pdk3r9nq0aasny863hcpf4nss which worked pretty good when we had dozens of users but has had scaling issues and has been hard to debug. But Nosflare is cool. I was able to easily add nip-50 search support, and because it runs on cloudlfare i hoped would scale horizontally. When I told nostr:nprofile1qqsdfx5syw3pmwsm8jpsdj3kn0ejg0vtgju0pdk3r9nq0aasny863hcpf4nss I was using nosflare, he said i should have told him... but again I didn't think this would escalate so quickly. So then we tried using the ditto relay https://github.com/andotherstuff/otherstuff-relay by nostr:nprofile1qqsqgc0uhmxycvm5gwvn944c7yfxnnxm0nyh8tt62zhrvtd3xkj8fhggpt7fy and put a bunch of really beefy servers behind it. Even then it's struggling to keep up.
The thing is, we're pre-launch, we have 10k users in testflight and a mostly read only site at divine.video which is a react app.
I'm a really terrible sysadmin. Yes I've helped run my own mail server since the 90's but I hate it and i'm not good at it. I know my way around my command line, I've compiled my own kernel from source, but fuck i hate it. And now i've got to setup and scale servers to realize the dream of something i've worked on for the last 8 years. I need your help, but maybe i'll digress...
In 2017 I decided to learn crypto, i joined a startup, quantstamp, and built their testnet, a SAT solver to verify smart contracts. I quit because I came to see how scamy the world of ICO's and tokens were. I'm not the only Nostr dev to have explored the 'darkside'. I started my company to build decentralized social, initially trying to take secure scuttlebutt to the mainstream. I built planetary.social, and worked with amazing dev's like nostr:nprofile1qqsdpg0lhpmph96va39rh6xtevhfdfcfph85vhl74jpe4fx2yry6t8s4rw4p8 and others we saw Nostr arrive and we pivoted! We built Nos.social, which i'm really proud of but it never took off.
A few months ago I was in talks to help start andotherstuff, but i was also very frustrated with running a company, I wanted to build stuff myself. So I stopped managing people, started a podcast, and really dove in to building with agentic programming. I built a bunch of things I threw away. A lot of bad experiments. In the course of the revolution.social podcast i kept hearing about Vine. I listed to the "Vine 6 seconds that changed the world" podcast: https://vine-six-seconds-that.captivate.fm/ and I talked to people about this social media platform that was shutdown when nostr:nprofile1qqsd96tlwvs92nsnq6235l9whcx9493vgex32yeyajtqv4dna2dy6xc66zx6m was trying to save Twitter when he returned as CEO.
I thought, well Vine is cool, I know folks like nostr:nprofile1qqs04xzt6ldm9qhs0ctw0t58kf4z57umjzmjg6jywu0seadwtqqc75s8fsrrg and others have build nostr video apps, how hard could it be to make a nostrvine app. I started coding, that's why the repo is still called nostrvine: https://github.com/rabble/nostrvine Turns out that it wasn't that had to make something that sort of worked.
Then I thought, it'd be cool to dig up some old vines. I searched the internet, found some on youtube, some on the way back machine, and I thought oh cool, i found a couple hundred popular old vines. Then I hit the motherlode, a community internet preservation project called archiveteam had run crawlers to archive the site: https://wiki.archiveteam.org/index.php/Vine they had about 2.7 TB of vine data, but in these very hard to work with WARC files that are 40GB each! I spent a month or more learning to parse and extract the files. I realized i had the meta data for most vine users, millions of comments, and hundreds of thousands of actual vine videos! It was a nightmare to parse because of the size of the files, the messiness of the data, and the like. But it was a consuming fun project, a puzzle.
At the same time, I was learning about flutter, I've had to rewrite the nostrvine codebase many times as i learned about riverpod, figured out how to get the UI to update smoothly while interacting with nostr. Getting the app to run fast and smooth was really hard. I also had to figure out how to host the damned videos in a way that works. I tried google cloud, cloudflare, and bunny. I made TONS of workers to run all of these services to make the system working. I also was seeing how much people, myself included are frustrated by AI slop, taking over social media. I have an old friend who runs a non-profit tech org, The Guardian Project, they'd make a tool for verifying videos are real for documenting human rights abuses. I thought, hell i could use this proofmode thing they've got to verify that videos are real. People like realness.
Over the last few weeks the pieces came together, I was scheduled to speak at WebSummit with nostr:nprofile1qqsr9cvzwc652r4m83d86ykplrnm9dg5gwdvzzn8ameanlvut35wy3g4h5cp7 and also to interview nostr:nprofile1qqszrptd47zv9e89q55savj7xzpmq4zm3sp749acnqc3zl8lp8ad7rgh59grd on the main stage talking about enshittification of the internet, and how we can resist it, by building things like Divine.
I talked to a reporter from Tech Crunch who'd written a positive article about AndOtherStuff, and she was excited to write an in-depth piece about my vine clone. Once the date was set, I had no choice to go forward. Was the app ready, NO NOT AT ALL. I was literally coding up releases on the plane while flying to Web Summit in Lisbon. I started submitting the app, and getting rejected. It got much better really fast, and basically works.
On the stage at WebSummit when I introduced diVine, the audience clapped politely. I showed the app to people and they sometimes said "oh this will be big" and wanted to play with it. But if you've ever made software, and you show it to people, everybody always finds something nice to say about it.
I had so little faith in diVine taking off that I was planning on taking a few days off to explore Morocco before heading to a non-profit software dev meetup in SF. It was only the last minute that I decided I might need be near a computer and internet connection post launch to see how things are going. Until diVine launched I thought the highlight of my trip and WebSummit would be that a podcast network wanted to pick up revolution.social and help me build an audience around the podcast.
I have never seen anything like this excitement. Just look at tiktok: https://www.tiktok.com/search?q=divine there is a wave of people excited about it. There's a wave of news about it: https://news.google.com/search?q=divine%20vine&hl=en-US&gl=US&ceid=US%3Aen Folks are saying that I'm taking on TikTok, and it's been on the evening TV news all over the place.
This is a dream. More excitement than I ever could have dreamed of. Creating a social media app that reflects all the values I laid out in rights.social . Building something people love and are excited about. When the app's been up, the new videos are amazing, so funny, so creative. When Jack launched Twttr, we didn't get this reaction. It took a lot of time for twitter to emerge as a star. The scaling issues didn't even show up until a year after twitter launched. When Kevin launched Instagram it got 150K signups in the first few days, and I was blown away at how fast it was growing.
If it hadn't been for my messing up getting in to the appStore, and having my relays collapse under the traffic, diVine would have grown much faster. Somehow it hits a nerve.
This is where I need your help, the Nostr community. I've already got help from a ton of folks like the folks from nostr:nprofile1qqs8sxs4yuz47axp7uprpugrs3sfkdz5379tdg9xe2n5qfvz070a4egc9mrhy and nostr:nprofile1qqsfln36agetx43hsw8mgkm4hce9j46zu94m8er59nyzhv74p7gg0esdgpa8a and others i'm forgetting right now... But we need more help. Let's do this as a community.
We're building a permissionless, open future that can't be shutdown by corporate owners. But we only get there if the tech works. We don't get to integrate cashu and show users how there's another business model for social media if we don't make an experience that people enjoy using.
Here's where we are. We've got the new nip for replaceable video events, which is supported by divine and amethyst... https://github.com/nostr-protocol/nips/pull/2072/files we've got the proofmode verification spec i proposed: https://github.com/nostr-protocol/nips/pull/2109 and my weird fork of nosflare which adds the ability to do filter requests that sort on things other than timestamp, it lets us find the most popular old vines: https://github.com/rabble/nosflare
The blossom server for media running on cloudflare mostly works, bunny is mostly working to scale serving the content. But fuck our relays are having trouble. Partially it's because divine doesn't optimize how many relay connections it does, so help with that would be appreciated.. but mostly it's we need to scale the relays, we need to work fast, and reliably. I'm trying to not talk much about Nostr and not make users understand anything about how nostr or keys or relays work.
We need a network of relays, we can dedicate for this, scale horizontally, which respond quickly, and support search. We could have search relays + normal content ones, but doing that requires updates to the released app, which is hard to do because we've got a delay of a day or more per release. So it's best if we can put this all behind relay.divine.video.
In terms of content moderation, my tactic is to provide a pretty heavily moderated experience on the primary relay and media server. But users own their keys, and the app lets users change or add relays and switch media servers. That way we can provide both freedom and the curated experience of users we're enticing away from centralized corporate social. And all of this is open source.
So help! I need nostr sysadmins and scaling folks. Please help. We don't have much time to catch this wave, and I'm in over my head. If you can help, reach out, rabble@rabblelabs.com or send me a DM, i'll add you to a slack room, and we'll figure it out.
Join me and we'll make a social media revolution to make revolution possible.
Discussion
The bottleneck seemed to be the D1 database. Replication was enabled. But how could sqlite ever possibly support such a high level of traffic? After a certain number of events were filled in it, it would slow down and eventually stop working (unless the D1 instance was nuked and restarted)
Yes, in terms of writes, it should have theoretically sustained about 50k concurrents or maybe 1000/sec events. That’s of course all theory. But with read replicas it should be able to sustain much more overall traffic with regards to REQs. I wonder if it could be rebuilt to use Cloudflare’s Hyperdrive with an external DB. Tried to benchmark it all, but obviously have no insight into what amount of traffic were actually hammering the relay from the web and apps. In all honesty, I think they might need a multi-relay setup and break up certain parts handled by different relays in order to support a level of traffic a centralized system like TikTok or IG would handle.
Under a single-writer ceiling how could there be any chance? This multi-relay/specialised-types would be a neat but at some point though you have to question why nostr in the first place. There's a limit to the number of nostr qualities you can sacrifice before you get to the point where you're in a worst of both worlds situation, what could have been a performant centralised architecture is held back by a bunch of nostr tech debt that isn't enabling anything nostr-like in the first place.
A proper SQLite database shouldn't have issues with this amount of traffic. D1 isn't just SQLite though, it's SQLite + an API.
I would try using SQLite-in-DO instead of using D1. There's a good blog post about it here: https://blog.cloudflare.com/sqlite-in-durable-objects/
Secondly I would look at splitting up the database into multiple databases.
Yes, am already using SQLite-in-DO for subscription management with the Websocket Hibernation API. As visitors are geo-steered towards a regional DO. But I haven’t tested trying to handle the data independently across all of the DOs instead of a single D1, but I think that would add some serious latency and make it unnecessarily complex. I did just rollout an update that adds Cloudflare Queues to all EVENT messages, which should help write throughout and handle burst up to 5000/sec.
Cloudflare , actually I don’t know about this until I read this. The idea was great during hibernate period like me in the winter ( sleep a lot) still be able to allowing JavaScript to run Those durable object ( DO) taking from memory .
How can I help ?
I'd be curious to see how this runs on a bare metal VPS - I would expect way way higher throughput.
Another thing to try would be to batch writes - instead of writing every time something happens you collect writes in a queue and write every 50ms in one big transaction. This should help a lot with write throughput.
Reading can be done in the same way.
If you want to avoid parameter limits you can use JSON1.
Last but not least, analyze the queries using `EXPLAIN QUERY PLAN` to make sure your indices are correctly used.
Using a VPS is sort of antithetical to what I was trying to build 😅. By being entirely “serverless” it wouldn’t have a single-point of failure or other downsides such as needing to ensure you harden it, manage updates, fault tolerance, smalllr resource ceilings, etc. There’s none of the traditional nuisances with Cloudflare Workers, etc. Everything is managed.
With adding Cloudflare Queues, the write throughput is now 5x better and it does batches of 100 to the D1. The next thing on my list is to ensure further optimizing the handling of all queries and improve read performance even more efficiently than just with indices and read replicas.
Yeah - totally get that, just curious 😅
Nice! Hope it works out 💪
Thanks! I've also been working on a non-SQL version as well. Will is right and wrong about SQL for Nostr. At scale, SQL does start to crumble. With a medium sized relay (or even semi high trafficked), it's completely fine when indices are used. But at the scale of what diVine was trying to do it wouldn't be the best performance. There were also a lot of things wrong from the beginning too where customizations were added, but accompanying indices not built for the filters/tags that would be used for every REQ.
I built a working Postgres relay (Ditto) but the only way tag queries would work was with a jsonb column on nostr_events table with a gin index. I have never seen tag queries work correctly if the tags table is separate and needs a join. It only works if the data is together in the same table.
sql for nostr doesn’t work. It’s been tried.
I see! Is a document based DB the way to go? In memory?
strfry/nostrdb model is the only thing that has worked. lmdb + indexes that cover common use cases:
- created
- pubkeyKindCreated
- kindCreated
- tagKindCreated
Bonus points for storing json string and flatbuffer version of the note for optimal performance
Intersting! Did not occur to me to use a kv solution for this.