Replying to Avatar rabble

Hey Nostr,

I need your help. Divine.video as you might have seen is a new video nostr app that i've been working on for the last 3 months.

It got MUCH more attention than I was expecting. Hundreds of millions of people viewed, liked, or shared videos about it. I've got some of the biggest original Viners in my DM's begging to get back on it. The TestFlight hit it's 10k limit in a few hours.

I'm excited but also really stessed out. We've had lots of bugs and Apple and google have been their usual black boxes when it comes to app review.

A bunch of folks have stepped up to help, nostr:nprofile1qqsr7acdvhf6we9fch94qwhpy0nza36e3tgrtkpku25ppuu80f69kfq9q9kky got the android build working for example.

Lots of things have broken, nobody really knows how survive a flash flood. I'm sharing this because I need help. We've got a chance to really grow nostr, the idea of a video app that's not got AI slop and does focus on something more human is resonating. People hate what's happening to tiktok, instagram, and youtube shorts where algorithms and the platforms love of AI generated content going viral is taking over. Instead of fighting back we see AI only platforms like MetaAI and Sora. This is an assault on the very idea that people are central to social media. I think big companies see the shine of AI generated content and dream of a world without all these pesky rabble making demands of platforms. If only they could replace the creators with bots.

This call to action felt right to me, but holy shit I had no idea it'd go so viral.

The app has lots of bugs, and we need appstore approval, but at the moment the biggest problem I have is relays. I need you, the nostr community's help. I started out with strfry which we know scales but lacks search. So i started using nosflare, https://github.com/Spl0itable/nosflare , by nostr:nprofile1qqsdfx5syw3pmwsm8jpsdj3kn0ejg0vtgju0pdk3r9nq0aasny863hcpf4nss which worked pretty good when we had dozens of users but has had scaling issues and has been hard to debug. But Nosflare is cool. I was able to easily add nip-50 search support, and because it runs on cloudlfare i hoped would scale horizontally. When I told nostr:nprofile1qqsdfx5syw3pmwsm8jpsdj3kn0ejg0vtgju0pdk3r9nq0aasny863hcpf4nss I was using nosflare, he said i should have told him... but again I didn't think this would escalate so quickly. So then we tried using the ditto relay https://github.com/andotherstuff/otherstuff-relay by nostr:nprofile1qqsqgc0uhmxycvm5gwvn944c7yfxnnxm0nyh8tt62zhrvtd3xkj8fhggpt7fy and put a bunch of really beefy servers behind it. Even then it's struggling to keep up.

The thing is, we're pre-launch, we have 10k users in testflight and a mostly read only site at divine.video which is a react app.

I'm a really terrible sysadmin. Yes I've helped run my own mail server since the 90's but I hate it and i'm not good at it. I know my way around my command line, I've compiled my own kernel from source, but fuck i hate it. And now i've got to setup and scale servers to realize the dream of something i've worked on for the last 8 years. I need your help, but maybe i'll digress...

In 2017 I decided to learn crypto, i joined a startup, quantstamp, and built their testnet, a SAT solver to verify smart contracts. I quit because I came to see how scamy the world of ICO's and tokens were. I'm not the only Nostr dev to have explored the 'darkside'. I started my company to build decentralized social, initially trying to take secure scuttlebutt to the mainstream. I built planetary.social, and worked with amazing dev's like nostr:nprofile1qqsdpg0lhpmph96va39rh6xtevhfdfcfph85vhl74jpe4fx2yry6t8s4rw4p8 and others we saw Nostr arrive and we pivoted! We built Nos.social, which i'm really proud of but it never took off.

A few months ago I was in talks to help start andotherstuff, but i was also very frustrated with running a company, I wanted to build stuff myself. So I stopped managing people, started a podcast, and really dove in to building with agentic programming. I built a bunch of things I threw away. A lot of bad experiments. In the course of the revolution.social podcast i kept hearing about Vine. I listed to the "Vine 6 seconds that changed the world" podcast: https://vine-six-seconds-that.captivate.fm/ and I talked to people about this social media platform that was shutdown when nostr:nprofile1qqsd96tlwvs92nsnq6235l9whcx9493vgex32yeyajtqv4dna2dy6xc66zx6m was trying to save Twitter when he returned as CEO.

I thought, well Vine is cool, I know folks like nostr:nprofile1qqs04xzt6ldm9qhs0ctw0t58kf4z57umjzmjg6jywu0seadwtqqc75s8fsrrg and others have build nostr video apps, how hard could it be to make a nostrvine app. I started coding, that's why the repo is still called nostrvine: https://github.com/rabble/nostrvine Turns out that it wasn't that had to make something that sort of worked.

Then I thought, it'd be cool to dig up some old vines. I searched the internet, found some on youtube, some on the way back machine, and I thought oh cool, i found a couple hundred popular old vines. Then I hit the motherlode, a community internet preservation project called archiveteam had run crawlers to archive the site: https://wiki.archiveteam.org/index.php/Vine they had about 2.7 TB of vine data, but in these very hard to work with WARC files that are 40GB each! I spent a month or more learning to parse and extract the files. I realized i had the meta data for most vine users, millions of comments, and hundreds of thousands of actual vine videos! It was a nightmare to parse because of the size of the files, the messiness of the data, and the like. But it was a consuming fun project, a puzzle.

At the same time, I was learning about flutter, I've had to rewrite the nostrvine codebase many times as i learned about riverpod, figured out how to get the UI to update smoothly while interacting with nostr. Getting the app to run fast and smooth was really hard. I also had to figure out how to host the damned videos in a way that works. I tried google cloud, cloudflare, and bunny. I made TONS of workers to run all of these services to make the system working. I also was seeing how much people, myself included are frustrated by AI slop, taking over social media. I have an old friend who runs a non-profit tech org, The Guardian Project, they'd make a tool for verifying videos are real for documenting human rights abuses. I thought, hell i could use this proofmode thing they've got to verify that videos are real. People like realness.

Over the last few weeks the pieces came together, I was scheduled to speak at WebSummit with nostr:nprofile1qqsr9cvzwc652r4m83d86ykplrnm9dg5gwdvzzn8ameanlvut35wy3g4h5cp7 and also to interview nostr:nprofile1qqszrptd47zv9e89q55savj7xzpmq4zm3sp749acnqc3zl8lp8ad7rgh59grd on the main stage talking about enshittification of the internet, and how we can resist it, by building things like Divine.

I talked to a reporter from Tech Crunch who'd written a positive article about AndOtherStuff, and she was excited to write an in-depth piece about my vine clone. Once the date was set, I had no choice to go forward. Was the app ready, NO NOT AT ALL. I was literally coding up releases on the plane while flying to Web Summit in Lisbon. I started submitting the app, and getting rejected. It got much better really fast, and basically works.

On the stage at WebSummit when I introduced diVine, the audience clapped politely. I showed the app to people and they sometimes said "oh this will be big" and wanted to play with it. But if you've ever made software, and you show it to people, everybody always finds something nice to say about it.

I had so little faith in diVine taking off that I was planning on taking a few days off to explore Morocco before heading to a non-profit software dev meetup in SF. It was only the last minute that I decided I might need be near a computer and internet connection post launch to see how things are going. Until diVine launched I thought the highlight of my trip and WebSummit would be that a podcast network wanted to pick up revolution.social and help me build an audience around the podcast.

I have never seen anything like this excitement. Just look at tiktok: https://www.tiktok.com/search?q=divine there is a wave of people excited about it. There's a wave of news about it: https://news.google.com/search?q=divine%20vine&hl=en-US&gl=US&ceid=US%3Aen Folks are saying that I'm taking on TikTok, and it's been on the evening TV news all over the place.

This is a dream. More excitement than I ever could have dreamed of. Creating a social media app that reflects all the values I laid out in rights.social . Building something people love and are excited about. When the app's been up, the new videos are amazing, so funny, so creative. When Jack launched Twttr, we didn't get this reaction. It took a lot of time for twitter to emerge as a star. The scaling issues didn't even show up until a year after twitter launched. When Kevin launched Instagram it got 150K signups in the first few days, and I was blown away at how fast it was growing.

If it hadn't been for my messing up getting in to the appStore, and having my relays collapse under the traffic, diVine would have grown much faster. Somehow it hits a nerve.

This is where I need your help, the Nostr community. I've already got help from a ton of folks like the folks from nostr:nprofile1qqs8sxs4yuz47axp7uprpugrs3sfkdz5379tdg9xe2n5qfvz070a4egc9mrhy and nostr:nprofile1qqsfln36agetx43hsw8mgkm4hce9j46zu94m8er59nyzhv74p7gg0esdgpa8a and others i'm forgetting right now... But we need more help. Let's do this as a community.

We're building a permissionless, open future that can't be shutdown by corporate owners. But we only get there if the tech works. We don't get to integrate cashu and show users how there's another business model for social media if we don't make an experience that people enjoy using.

Here's where we are. We've got the new nip for replaceable video events, which is supported by divine and amethyst... https://github.com/nostr-protocol/nips/pull/2072/files we've got the proofmode verification spec i proposed: https://github.com/nostr-protocol/nips/pull/2109 and my weird fork of nosflare which adds the ability to do filter requests that sort on things other than timestamp, it lets us find the most popular old vines: https://github.com/rabble/nosflare

The blossom server for media running on cloudflare mostly works, bunny is mostly working to scale serving the content. But fuck our relays are having trouble. Partially it's because divine doesn't optimize how many relay connections it does, so help with that would be appreciated.. but mostly it's we need to scale the relays, we need to work fast, and reliably. I'm trying to not talk much about Nostr and not make users understand anything about how nostr or keys or relays work.

We need a network of relays, we can dedicate for this, scale horizontally, which respond quickly, and support search. We could have search relays + normal content ones, but doing that requires updates to the released app, which is hard to do because we've got a delay of a day or more per release. So it's best if we can put this all behind relay.divine.video.

In terms of content moderation, my tactic is to provide a pretty heavily moderated experience on the primary relay and media server. But users own their keys, and the app lets users change or add relays and switch media servers. That way we can provide both freedom and the curated experience of users we're enticing away from centralized corporate social. And all of this is open source.

So help! I need nostr sysadmins and scaling folks. Please help. We don't have much time to catch this wave, and I'm in over my head. If you can help, reach out, rabble@rabblelabs.com or send me a DM, i'll add you to a slack room, and we'll figure it out.

Join me and we'll make a social media revolution to make revolution possible.

relays scale well. we’re dealing with small json blobs and read heavy workloads. this lends extremely will to caching and horizontal scaling.

the hard part is the clients. in a permissionless network, clients will be bad actors. for your client to work well it must use sane data access patterns.

concretely:

- get the client’s websocket connection mgmt in order

- get the client’s data access patterns in order. batch queries, cache locally, etc

- harden your relays with rate limiting for stomping out bad clients

- implement good caching for your clients needs

nostr:nevent1qqs2ztln6vaff7jq34c7ys67vwp8qpj87rxncrqf64hv9nry65tykscpr9mhxue69uhk2umsv4kxsmewva5hy6twduhx7un89ujnp7pf

Reply to this note

Please Login to reply.

Discussion

I would rather have a relay that protects me from clients than trust the clients to be sanely written. I think relays have a long way to go.

you cannot trust the clients in a decentralized network. that is why the relay must use common techniques to prevent abuse.

BUT the client must also implement sane data access and connection patterns for the client to work at all.

in this particular case I will guarantee you the vibe coded client is doing terrible things with websocket connections and queries. that is step one to making the client work.

Another issue using websockets over http in this case, is it defers the abuse protections completely to the application servers (or custom proxies) meanwhile hogging whole tcp streams in load balancers. In existing http infra, load balancing and resource abuse protection can happen in multiple tiers sparing the application software (and more specifically relay developers).

That said I'd argue almost all web abuse protections are handled way before the client request even makes it to the application server. And it's done very quickly.

totally agree. websockets are annoying. but tcp connections are cheap and as you said you can quickly handle bad actors once the channel is open.

to solve this specific problem i would do something like

- one relay for read, heavy caching to reflect data access patterns in vine client

- one relay for search

- fanout to the proper relay a single api gateway tier

- scale every layer independently

- write a sane client

> tcp connections are cheap

I would disagree but we could be on different scales, that's at least two extra OS handlers, at least 16k (usually 64k) of kernel space (x2 for rx and tx) and often userspace (usually shared) buffers, per connection, per LB. L7 lbs generally multiplex tcp connections and dramatically cut down on memory consumption. While it can be a bit aggressive. For every websocket that's opened in my http server, tuned to 16k, thats about 256k after the upgrade is established (because websockets are usually fully buffered) and http buffers are freed just in userspace. For 5000 connections that's over 1.2G of committed system memory minimum just hanging about. I would expect software like nginx to be more optimized in comparison but that still hogs up LBs and I know I've heard other sysadmins share their stories of websocket overhead.

L7 lbs multiplex these http/1.1 connections, generally cutting down to double digit connections to service 5-6 digit ingress traffic.

I agree with the basic architecture, that is, a highly functional with a "popularity decay" model. I can't remember the scientific name.

os handles*

popularity decay => least recently used (LRU)

Yes but that's not a great model. LRU breaks down in pure form here. A single user pulling an old file forces fresher content out. TTLs have to be attached and respected by each tier. There is an official name for this model I just don't remember it. As files age their ttl decreases and avoids polluting caches.

frecency-based models are what you are looking for

Yeah but there is still a more specific model Im trying to remember the acronym for. It literally describes exactly how to label content with TLLs based on the frequency of the exact content being hosted. Something one of the big tech companies published. Basically, given that the site is hosting images to be shared on social media, it's known that the images will have the highest frequency immediately after being shared, then general decay at a known rate. It can even get as accurate as saying, given an image of a puppy, you can apply TTLs that model how the image of the puppy should be cached.

substitute "puppy" here for some known constant, like: my users usually upload X content. In the case of devine. Users will only be publishing short looping videos designed for user attention. Which might frequently have images of cute animals...

^ And relays only scale well currently because clients assume poor consistency and load balancing and reconciliation happens at the client. And more specifically because nostr users can't really know any better.

Some analytics from Nostr.land shows that many old events are rarely requested.

I am considering a dynamic tiering strategy where the age, access frequency and “position” of the event (relative to similar ones) is used to send it off to archival or (more likely) zstd it.

I do not cache indexes due to the high complexity and low benefit.

testing zaps for this note… we made six attempts to⚡zap this note, at ben@northwest.io, over a period of 3 minutes. all six attempts were successful. please check on your end to be sure you received. average zap time was 14253ms (14.3 seconds). we consider this zap time slow... generally, zaps should complete in under two seconds. (other nostr users might think your zaps are broken, might not zap you again.) if you wanted to fix this... you could try getting a free rizful lightning address -- https://rizful.com ... if u get it set up, pls reply here so we can do this ⚡zap test again.