Avatar
Anthony Accioly 9mo ago

GM, folks! Just putting this out there. I was thinking, while we have a few NIPs and torrent-based tools on Nostr, there are some quick wins to be had by tapping into the torrent ecosystem.

With so many people building HTTP views, bridge APIs to Nostr WebSockets, storage APIs (e.g., Blossom), and streaming tools... FFS nostr:nprofile1qqsrhuxx8l9ex335q7he0f09aej04zpazpl0ne2cgukyawd24mayt8gprfmhxue69uhkcmmrdd3x77pwve5kzar2v9nzucm0d5hszxnhwden5te0wpuhyctdd9jzuenfv96x5ctx9e3k7mf0qydhwumn8ghj7un9d3shjtnhv4ehgetjde38gcewvdhk6tc4rdlnm is even exposing a FUSE interface that can be mounted with nak (just for laughs) to give you Nostr as a filesystem abstraction. So why aren't we leveraging WebTorrent and IPFS to make our lives easier?

I’m not saying we should abandon what we have. Just that we could use what’s already out there pragmatically.

For example, I keep hearing about Negentropy, which is great and all, but… Why can't your client or relay just join the swarm and sync all the notes and Blossom blobs associated with my pubkey? Maybe we could even expose all of this over IPFS and boom, you’ve got "Negentropy" and a cool alternative to njump.me. It doesn’t get much more bandwidth-efficient and P2P than that.

@jonny@neuromatch.social 🔗 https://neuromatch.social/users/jonny/statuses/114245524577616333

-

> we whipped up a bittorrent swarm that has 200TB of proven storage in like a month by cobbling together everyone's random hard drives. storing 200TB on s3 would cost at least $4.5k per month and would cost $10,000 to download it one time.

#gm #nostr #torrent #blossom #ipfs #growNostr #devstr

Reply to this note

Please Login to reply.

Discussion

Avatar
deeznuts 9mo ago

LOL, why is Bram Cohen a “fascist” ??

Avatar
Anthony Accioly 9mo ago

And if you read above and thought to yourself, Wait, torrents aren’t mutable... and can’t be signed... Noth are solved problems:

https://www.bittorrent.org/beps/bep_0039.html

https://www.bittorrent.org/beps/bep_0035.html

I could use, say, Haven to expose the update URL, give you all a magnet link, and voilà—you can sync and serve as much of my content as you want.

Avatar
fiatjaf 9mo ago

IPFS doesn't work. If it worked we would not need Nostr or relays. We could simply sign notes, publish to IPFS, then publish an updated index of our notes categorized by kind or something like that, then update our IPNS to point to the updated index. People who follow us could just fetch our IPNS record and get our index and download our posts.

Avatar
fiatjaf 9mo ago

Interestingly, I've always said this, but now I'm realizing that the approach above would not be sufficient because, for example, we would not be able to know who replied to us, which is something that relays make easy. We would also lose all the other cool things that relay can do and in the end we would have to resort to the Blueky/Pubky approach of having one big "indexer" that would crawl all the IPNS indexes, download all posts and serve them using an HTTP API that everybody would use and then we would get VC funding, create a walled garden controlled by a single company and call it a decentralized social network.

Avatar
Anthony Accioly 9mo ago

Can you expand on why IPFS doesn’t work or point me to an article where I can understand its issues?

I wasn’t thinking of anything as ambitious as replacing Nostr relays, clients or Blossom servers. Just sharing events and blossom vlobs as a signed torrent with an "update URL", allowing both clients and relays to sync, seed, list or navigate my stuff to your heart’s content. Relays would still be there to serve all of this over WebSockets, and Blossom would still be there to serve media over HTTP, etc. In other words, I’m not thinking of throwing away the existing protocol and sending Nostr back to day zero, just checking if we can't solve sync, specially between relays, in an easier and likely more efficient way.

Avatar
fiatjaf 9mo ago

Have you ever tried to use IPFS? It only ever works when you publish a file directly to the Cloudflare instance and then fetch directly from it via HTTP bypassing the actual software and p2p network.

I wrote nostr:naddr1qqyxgdfsxvck2dtzqyghwumn8ghj7enfv96x5ctx9e3k7mgzyqalp33lewf5vdq847t6te0wvnags0gs0mu72kz8938tn24wlfze6qcyqqq823c8y87ll a long time ago and haven't reread it since, so it's probably garbage. Search Nostr for "IPFS" and you'll probably find better content.

Avatar
Anthony Accioly 9mo ago

Got it, thanks. I'll give it a read and do some more practical research. I admit that I've played with impressive demos and not much else.

But I think I’ve sidetracked the post by even mentioning IPFS. My main idea was to implement Nostr Notes + Blossom media sync through torrents. Have, say, Khatru/Haven expose a torrent with a file structure similar to what you did for nak/FUSE, plus all my Blossom blobs. I can then expose a magnet link with a BEP-39 update URL.

We could even have a specific event on Nostr for folks to discover my Nostr/Blossom content index URL. For lack of a better name, let's call it master outbox. Anybody else can use this torrent to sync and seed my notes and media.

This doesn’t need to be a massive centralised index like Bluesky. Instead, each user publishes their content to whatever relay they want, chooses one as their current master outbox, and other relays and even clients (through WebTorrent) can perform full or partial syncs using the torrent/magnet link. Wouldn't this be useful at all? Sure Negentropy can be used for more complex gathering of notes spread across relays. But for simple sync it feels like this would be an efficient way of doing things.

Avatar
nomadshiba⚡ 9mo ago

there is a reddit and 4chan like social media protocol on ipfs, its a bit more wild west. it uses pubsubs.

but its nothing like nostr, nostr is more profile based, and a lot more closer to regular social media. just better in general.

and ipfs problematic when you share something alone from your own node, it takes a long while for others to discover it. but sometimes it also works fast, just like that.

i think one problem with nostr right now is all media is just http links to a cdn.

they are not distributed, and link have nothing to do with the content, makes it impossible to verify. so i think sharing a hash instead of some random centralized id makes more sense. and we already have ipfs hash and uri. so if we shared ipfs uri instead of http url i think it would just work.

like relay lists, we can also have ipfs api and gateway lists. clients can ask all of the gateways for the content until finding it at one. ipfs gateways allow HEAD requests, so you can know if the content exists and its size before fetching it.

you can easily backup/pin your own content on another gateway you pay for as well if you want to.

main thing is its hash based, and distributed.

right now if i post a media on primal, they will host it via their cdn, and all of the other nostr clients will fetch it from a url to that cdn. cdn can censor, remove the content, and in the future even cdn itself can shutdown. there is no way to access that content other than that cdn. but and ipfs uri is hash based, so anyone can host if for you. if the content is lost, you can verify it again once you find it, etc. primal can just host an ipfs gateway instead of a regular cdn.

i know not many clients support ipfs, or have a list of ipfs gateways to use. but what you can do is share an a gateway https url, and anyone can still parse, it and say that "its an ipfs url".

ipfs url formats are known, it can be in bafy subdomain type or bafy and Qm type with /ipfs/ pathname prefixes. you can extract the hash from there and try it on multiple gateways. and clients that doesn't support ipfs would still work, because its basically a https url to an ipfs gateway.

i plan on hosting one in the future for nostr, with my own client.

Avatar
Anthony Accioly 9mo ago

Sounds like a great application for IPFS! To be fair, Blossom blobs are hash based, and nostr:nprofile1qqszv6q4uryjzr06xfxxew34wwc5hmjfmfpqn229d72gfegsdn2q3fgpr3mhxue69uhhxct5v4kxc6t5v5hxs7njvscngwfwvdhk6tcpzfmhxue69uhkummnw3e82efwvdhk6tcpz4mhxue69uhhyetvv9ujuerpd46hxtnfduhsk7wj75 has also been asking around about "Negentropy." On top of that, if clients follow BUD-03 as intended, they should be able to recover the media from any of the servers in a user list: https://github.com/hzrd149/blossom/blob/master/buds/03.md

However, your idea of leveraging IPFS sounds very promising. The client wouldn’t need to implement fallback logic or fetch the media elsewhere, users wouldn’t have to carefully manage redundancy by mirroring blobs and managing multiple blossom servers, and a simple "just download and seed my stuff" approach is certainly a great way to do things.

I’d definitely like to try what you’re building (or planning to build) once it’s out there. And if it can be made Blossom-compatible, that would be even better. I know that despite both systems using SHA-256 hashes by default, they aren’t exactly compatible OOH. IPFS uses CID versioning, supports multiple ways of encoding hashes and even multiple hash algorithms. But given that IPFS has some flexibility built-in and Blossom is an emerging standard, I think there are ways to accommodate this.

Avatar
nomadshiba⚡ 9mo ago

didn't know about blossom before. looks promising.

i think where blossom is strong is its easier to implement in few hours. and more lightweight. more cdn like.

ipfs a bit more complex, but good side is, well-known gateways share content between each-other fast without something like /mirror.

if the hash were compatible/convertable it would have been great. then you can ask for the content on both blossom cdns and ipfs gateways. also blossom cdns can also seed the content on ipfs. and etc.

Avatar
Aida 9mo ago

When I first started with Nostr back in 2023, IPFS sounded like a perfect match. But the more I played with it, the more convinced I became that IPFS is a broken and failed project. Nostr needs something simpler and more straightforward—and then nostr:nprofile1qqszv6q4uryjzr06xfxxew34wwc5hmjfmfpqn229d72gfegsdn2q3fgpr3mhxue69uhhxct5v4kxc6t5v5hxs7njvscngwfwvdhk6tcpzfmhxue69uhkummnw3e82efwvdhk6tcpz4mhxue69uhhyetvv9ujuerpd46hxtnfduhsk7wj75's Blossom came.

Ever since then, I've been dreaming about redundant decentralized storage with file fragmentation—something like what StorJ describes in their white paper:

"All data is encrypted and sharded using Reed Solomon erasure coding. The resulting pieces are distributed around the world. A typical large file is divided into 64 MB encrypted segments, and each encrypted segment is divided into 80 erasure-coded pieces—any 29 of these are sufficient to reconstruct the original file. Each of these 80 pieces is stored on a different drive globally, spread across diverse geographies, operators, power supplies, and networks."

In Blossom, this would be a killer feature. You would offer space on your server to others, and in exchange, get redundancy for your own files, or you would simply pay for it. File metadata could be published as a specific kind hosted by relays.

Avatar
ornedii 9mo ago

can you elaborate on IPFS brokenness? out of interest

Avatar
Aida 9mo ago

It is almost impossible to get content synced over the network. Try to sync a simple picture from your home node to a node in your laptop when you are on different networks... It will take hours or it will never happen.

Avatar
nomadshiba⚡ 9mo ago

I mean what I said was just getting content from well connected gateways anyway, or uploading to one, like nostr client providers can have their own default gateways and etc. i wasnt talking about p2p ipfs experience.

anyway I checked blossom, (first time hearing about it).

seems similar to what i was thinking with ipfs.

but instead its a cdn protocol basically, i think. didnt deep dive in it yet.

so i believe its like a normal cdn but hash based and has a known format, endpoints. so you can extract the hash (like i said in ipfs). and then check other blossom compatible cdns.

and of course user can give list of blossom cdns on their profile as well.

tbh blossom looks more lightweight and easier to implement for already existing cdn solutions. i see no problems with it.

seems like a more lightweight solution. i liked it.

i still see no problem with supporting both or more. similarly you can share a media with a https ipfs gateway URL, so it works everywhere.

and then while viewing extract the hash and check multiple other gateways with that hash as well using HEAD method.

which would also make those gateways search for the content. distributing the GET traffic more later when more users ask for same thing.

but yeah blossom looks nice, maybe better.

d5
akamacgyver 8mo ago

Did you ever try to use Storj? I'm intending to give it a spin for some personal back up, at least.

Avatar
Mike Dilger ☑️ 9mo ago

Torrent was designed over fixed large data. I'm not sure what would happen if the dataset you are syncing is constantly changing.

I think using mainline DHT to bootstrap a pubkey's relay list is a cool idea. But we instead just spread these relay lists out on the current popular relays, which works well once you've been using nostr for a while and your client can easily figure out which relays are popular. DHT would be better for people starting cold though.

Avatar
Anthony Accioly 9mo ago

I get where you’re coming from. I’ve tested BEP-39 in real use cases, and I was impressed by it. From keeping a folder full of Markdown files in symc to ensuring I always download the latest episode of my legally licensed, totally fine-to-download favourite show, things just show up automatically in my torrent client without hassle. The update URL just points to the latest version of a torrent file and from there things just work.

Having said that, update cadence could be parameterised as well. Think njump. It’s smart enough to fetch my notes individually, but it doesn’t need a real-time view of my latest notes. So both the master "relay/tracker" and clients could pace themselves according to their own needs.

I really like your idea of storing a copy of someone's kind-10002 events on DHT! Probably also kind-10050 and maybe other discoverability-related notes. That’s an awesome idea.

On a similar front, despite my dislike for anything that relies on DNS, NIP-05 nostr.json could also hold a copy of all this. Kind of like what you did with the relays array, but on steroids. In other words, NIP-05 identity servers would function similarly to WKD for GPG. If you know someone’s nip05 identifier, you can retrieve all the info you need from there, no need for a second pass on a relay.

Avatar
Mike Dilger ☑️ 9mo ago

I don't think we should have multiple ways because (1) it adds complexity, (2) it adds overhead (have to check them all), (3) things get out of sync. The DHT thing is a neat idea but nostr already has an effective solution without it. And the nostr.json relays came about before kind 10002 did, and maybe even should be retired, because mine keeps falling behind and getting out of sync.

Avatar
Anthony Accioly 9mo ago

I don’t disagree in principle, but in practice… kind-10002 as it is now isn’t sufficient. This is a chicken-and-egg problem: I need to connect to an unknown relay to fetch Mike Dilget’s kind-10002 event in order to figure out which relay I need to connect to in order to find Mike Dilget’s notes. It’s a strange way of bootstrapping things unless we accept relay centralisation or widespread blasting / Negentropy. IMO, the first is undesirable, and the second is wonky, it introduces eventual consistency at best and indeterminism at worst.

This isn’t a criticism of the Outbox model by the way. It’s definitely the way forward, but I think we need an efficient, hopefully deterministic, single-call way of doing this. And this has to be done without relying on Damus, nos.lol, Snort or whatever other giant relays to hold every single kind-10002 (or equivalent) in existence.

On a more fundamental level, I think the "one true way of doing things" ship sailed a long, long time ago. Deprecating a NIP won’t help much either (e.g., NIP-04 vs. NIP-17 even with harassment bots and some popular relays like Haven dropping kind 4 by default, most clients aren’t there yet).

In a way, while this is the opposite of what I’d push for as an architect, I’m happy for Nostr to be the hacky, pragmatic "PHP of Protocols." Every programming language feature ever invented eventually finds its way to PHP, and we all know how hacky and clumsy parts of PHP’s standard library are. PHP, like C++, gives you at least 10 ways of doing anything. But unlike C++, a first year CS undergraduate can easily hack something functional with it.

And that’s totally fine, considering what Nostr aims to be. PHP won the web race, while my favourite, "well-designed" programming languages are all competing in niche spaces. This doesn’t mean your call to clean up Nostr is wrong, it’s just that… good luck herding cats into agreeing on and implementing one true way of doing things.

However, just to indulge in one true way thinking: if you put a gun to my head and told me to choose, while I’d personally pick DHT, in terms of driving Outbox model adoption across all major clients before 2030, I’d go with the "well-known URI on steroids" approach. Maybe with additional NIP-05 provisions to standarise identities behind onion services. Perhaps also with a dead-simple, optional PUT endpoint to update relay information trivially from clients. To me, kind-10002 should be part of nostr.json, i.e., it’s connection bootstrapping information and, as so, should be trivially obtainable before I even connect to any Nostr relays.