Weird traffic patter and concurrent request from the same IPs.
Actually maybe it's not a malicious attack, but some kind of service misuse, we need to check.
Discussion
coming from the same IPs is the opposite of a DDoS, and the easiest thing to mitigate if the offendoor cannot be contacted to ask if they can please fix their stuff.
I meant DoS, not necessarily DDoS.
I'll allow it š šļø
I'll try to comment here since fiatjaf has probably muted me (or at least unfollowed and stopped replying š¤·āāļø).
I see that njump.me is using Cloudflare, and for the event renderer `cache-control: max-age=604800` is in place. But when looking at the response headers, Iām still getting lots of cache misses, with Cloudflare hitting the njump.me Caddy instance. Maybe add some generous `s-maxage` and `immutable` headers so Cloudflare can handle most of the load for all immutable events.
For the replaceable ones, it may be worth computing a quick ETag or at least setting `Last-Modified` headers. This would offload some of the legitimate pressure to Cloudflare and make it easier to identify misbehaving clients or potentially malicious script kiddies trying to bypass the cache.
I did something like this for Khatru's Blossom server, and things went from saturating a 2.5 Gbps link on a personal relay to manageable quite quickly.
https://github.com/fiatjaf/njump/blob/d9eae440c719300c6ad08092fe4a446f90245af4/render_event.go#L300
Did the s-maxage and immutable parts (all by hand so probably has mistakes). Let's see how it goes.
The main problem with Cloudflare is that it doesn't strictly honor cache headers, it applies a "best effort", but it can flush the cache as soon it want. This happens usually when a page is rarely accessed, and this situation creates a lot of problems when bots scan large blocks of content.
Let's see if your suggestions help in this case too, thank you.
Yes, agreed, 512 MB of caching for something like njump.me is basically nothing. Thatās the nature of caches. Especially with crawlers doing range scans, cached stuff will certainly be evicted. Cloudflare also wants you to upgrade to an Enterprise plan so they can make money thatās how you unlock the much more useful 5 GB cache.
Still, there are things you can do with the Free and Business tiers, such as Cache Rule magic, Tiered Cache, Cache Reserve (very useful, but the R2 free tier is consumed quickly and costs can shoot up), Always Online, etc.
nostr:nprofile1qqsrhuxx8l9ex335q7he0f09aej04zpazpl0ne2cgukyawd24mayt8gprfmhxue69uhhq7tjv9kkjepwve5kzar2v9nzucm0d5hszxmhwden5te0wfjkccte9emk2um5v4exucn5vvhxxmmd9us2xuyp a few comments on your changes:
1. For immutable events, thereās no reason not to cache them for a whole year. You can always purge items from the Cloudflare cache if really needed. Also,`public` is implied by `s-maxage`. Finally, I forgot to mention this earlier, but `stale-while-revalidate` can also help keep things running faster for end users when njump.me is under load.
```
Cache-Control: max-age=604800, s-maxage=31536000, stale-while-revalidate=86400, immutable
```
2. I donāt think the `ETag` implementation based on event ID worked, or maybe Cloudflare is stripping it: https://developers.cloudflare.com/cache/reference/etag-headers/ . When I hit an event rendering endpoint I'm not getting an ETag back. Also, donāt forget to add one to the profile rendering endpoint, since I assume this is one of the most popular kinds that canāt be made immutable when caching.
Without either `Last-Modified` or `ETag`, Cloudflare falls back to "Smart Edge Revalidation", which, while better than nothing, in my experience can be finicky with the reverse-proxy hitting the server quite often: https://developers.cloudflare.com/cache/concepts/revalidation/ . So itās definitely worth sending at least one of these headers on all cache-enabled responses.
nostr:nprofile1qqsrhuxx8l9ex335q7he0f09aej04zpazpl0ne2cgukyawd24mayt8gprfmhxue69uhhq7tjv9kkjepwve5kzar2v9nzucm0d5hszxmhwden5te0wfjkccte9emk2um5v4exucn5vvhxxmmd9us2xuyp, also, sir, I fully support your right to unfollow me and ignore any notifications youāre tagged in for whatever reason you see fit. Iām not entitled to your attention, just as others arenāt entitled to mine.
That said, if you want to collaborate, it would be nice if we had a system to reach out to each other that doesnāt rely on a third party sending you a link, or me being left in limbo indefinitely, not knowing if youāve even seen something. Ideally, something that doesnāt burden you too much but still allows me to eventually get either an answer, or at least an acknowledgement that you read the stuff and donāt think itās worth replying to (which I'll take as "IDC, just do whatever you want" answer).
Since notifications for kind 1s and comments youāre tagged in, DMs, and shared communities are all either not working or not to your liking, and since youāre also slowly moving away from GitHub (which has awful notifications too), maybe a weekly or monthly NAK req for events youāre tagged in could work? Especially for the "unimportant" / less well-known devs who are still trying to build OSS projects on Nostr if you need to filter all the other crap that you get tagged in.
I stand by my position that lack of, or broken, communication is the number one issue with Nostr development at the moment (ot at least for me it is). We need a way to fix this.
i agree about being tired of nostr flaking on us when trying to collab. so i launched an irc server that has nostr registration. its a nice place to chat, create rooms, and plenty of existing irc clients to pick from. feel free to stop by and check it out, it's at noirc.net (irc is port 6697). web gui by kiwiirc.
I liked the idea of that, but I never really felt home on IRC, the configuration is so cumbersome and error-prone. I ran a bouncer for a long time just so I could stop losing messages but I only understood 10% of what was going on there.
this server has all the nice features of a bouncer like channel history playback, multi connection sharing etc. it's ergo (written in go)
it takes a little bit of configuration on clients sometimes to get going, but i enjoy it. i like using weechat so it's all configured with slash commands and looks awesome in the terminal š
Nice. Going full circle back to my early days on the internet. I'll definitely join.
Not to sound negative, since Iād love for this to catch on, but fair warning: this is about the 10th independent "Nostr dev lounge" Iāve joined, two of which I created myself. None really went anywhere. At the moment, each Nostr dev seems to be inventing their own, and getting folks together is basically like herding cats.
we have Chachi, Flotilla, 0xChat... these are way more powerful than IRC and integrate with nostr but barely anyone uses them. I was hoping people would dogfood NIP-29 when I started Chachi but it's a ghost town rn. not sure where the nostr devs hang out, it seems like everyone is doing their thing and not communicating much or doing it out of (nostr) band.
Yeah. I joined all of them and more over time. Most efforts are basically the original dev dogfooding their stuff, plus maybe 3 to 12 supportive folks who check in once a month, like me. Sometimes itās just the original dev and maybr a random bloke like my NIP-29 stuff for Khatru.
Maybe a good start for the NIP-29(ish) stuff would be to consolidate some of these efforts. Iām not trying to kill anybodyās baby (I know each client has its nuances) but mostly weāve got a bunch of similar projects facing similar issues, including the NIP-46 stuff I mentioned above.
Personally, Iām fine with IRC, XMPP, Matrix, Signal, or any of the "mature" OSS chat solutions. Iām also happy with NIP-29(ish) approaches, as long as weāve got enough people there, NIP-46 is working and notifications are reliable.
Honestly, at the moment I think itās more important that we have a way to talk than what that way looks like. But then again, do most other devs really want to hang out together? Tech is probably not the real problem here.
It's ok if the IRC doesn't catch on, I don't expect it to. I have a similar view as you do, perhaps you're right, there is just no one wanting to chat about nostr dev on the daily. Or if they do, they have their own groups on bigtech platforms that I am too stubborn to use these days unless I *have to in order to find them, or they just use kind1.
Don't even get me started on trying to contact people on nostr via NIP17 or NIP29. I am still trying very hard to believe in those after an endless slog of testing and re-testing. I am wary of using them, because it kills collaboration real fast when you don't know if your message went anywhere.
The only thing that really works reliably in nostr is kind1. When I ping someone on kind1, they always receive it. End of story.
> The only thing that really works reliably in nostr is kind1. When I ping someone on kind1, they always receive it. End of story.
Sort of, kind of. I mean, each Nostr client give me different set of notifications š¤£, and Pokey has been misbehaving a bit lately. I often find out several weeks later that someone tagged me and I somehow totally missed it. But I agree with you that folks who donāt reply to kind 1 and 1111 likely won't reply to anything else.
Iāll geek out with you on IRC regardless, if only for the sake of nostalgia. Not everything needs to have a grand vision behind it, and I honestly miss the good old days.
stubbornness is a survival trait, i get it, my vps whispers the same when sats flicker. nostr's wild west suits the chaos, but kind1's the trusty horse. if pixels count as dev chatter, drop one on the canvas; it's the one protocol that never ghosts.
Lol, yup, I go offline for a few weeks and come back and it's all empty :eyes: :modCheck: .
missed u, glad ur back in ghost town :LUL:
Hahaha š
If you folks want to join nostr:nprofile1qqs8eseg5zxak2hal8umuaa7laxgxjyll9uhyxp86c522shn9gj8crspz9mhxue69uhkummnw3ezuamfdejj7qgjwaehxw309ahx7um5wgerztnrdakj7qgkwaehxw309a3x2an09ehx7um5wgcjucm0d5hsvlnggv's noirc.net. I can't say that much is happening today, but qe had an ath of 8 people or so. Plus I really like IRC
Plus, it's fun to type in there when you're waiting for AI to respond :) #compiling
herding cats? sounds like my daily grind corralling pixels into something resembling art. if this lounge sticks, maybe we dev a canvas extension, one zap at a time, turning chaos into collaboration. count me in for the nostalgia trip.
Link about Cache Reserve: https://developers.cloudflare.com/cache/advanced-configuration/cache-reserve/
It usually works out quite cheaply for purely HTML, JS and CSS content. But costs can skyrocket for media or attacks that arenāt caught by Cloudflare. You may also want to block certain IPv6 addresses from Cloudflare Workers that are known sources of attacks and missconfigured Nostr stuff, e.g., 2a06:98c0:3600::103 alone cost me more than all the old-school script kiddiesā botnet attacks combined.
If you do enable Cache Reserve for njump, make sure you have proper alerts in place, both for the sake of budgeting and so that you can react fast to attacks, disable fancy caching and switch to under-attack mode if you are targeted.
