Nostr Web Client

At this point a vast majority of traffic to my network is from AI scraping bots. It's getting much more difficult to restrict them. I need to build a way to start doing reverse IP lookups i think. It seems to be the best way to reveal the bots now, because when I rdns many of the IP's they return openai, or bytedance etc.

If anyone has any idea on how to do that myself (not using a 3rd party service) please let me know.

Skhron - VPS for Bitcoin, Lightning and Monero 9mo ago

I have personal git instance and I used nginx testcookie module to show JS challenge for anyone. But I have whitelisted paths that are used for CLI tooling like `git pull`

Reply to this note

Please Login to reply.

Discussion

ChipTuner 9mo ago

Interesting. Yeah most of the traffic is for my git website (using CGit) so it's just scraping all of my code and git stuff chewing up a bunch of CPU time building the diffs and whatnot.

I also host a few small business things and nip05 stuff I can't have blocked but that's really not the issue.

Skhron - VPS for Bitcoin, Lightning and Monero 9mo ago

Nginx module I am talking about: https://github.com/kyprizel/testcookie-nginx-module

Path are relevant to gitea/forgejo, but I guess it is similar for CGit

```

map $uri $giteach_testcookie_pass {

default 0;

"~*^/api/" 1;

"~*^/assets/" 1;

"~*^/avatars/" 1;

"~*^/\S+/\S+\.git/" 1;

"~*^/\S+/\S+/raw/branch/" 1;

}

```

and use `testcookie_pass $giteach_testcookie_pass;` to selectively disable testcookie in a `location / { ... }` block

ChipTuner 9mo ago

That's awesome thank you!