Interesting. What is your LLM prompt to achieve that?
Have you tried using text transformer embedding model and use vector distance to measure similarity for spam rate limit?
Discussion
sharing the prompt may have two consequences:
1. relay ops copy it and notes are censored similarly, effectively a note being banned everywhere, effectively nostr becoming a censorship place
2. spammers can find leaks in the prompt and evade the checks
but i could dm it to you. you have a relay too? how do you filter out nsfw? what kind of tools do you use?
no i didn't do vector distance. i may fine tune the model with 1984 notes later. another thing could be actually downloading the images to memory and doing a check on the image.. .
Oh, thank you, if you don't mind to share it by DM. Agree with that, especially for the spammers abuse. I use regex filter and anti duplication policy for spam management.
Yes I have run custom aggregator relay with filter parameters/settings (like filter.nostr.wine )
This is the package code for my relay
https://github.com/atrifat/nostr-filter-relay
which needs several separated modules (Language detection, SFW detection, Toxicity/hate speech detection) listed as its dependencies. All modules can be run using CPU.
is the proxy connecting to multiple relays?
are you actually looking at contents of the image file for nsfw detection?
Yes, wss://nfrelay.app (demo relay) has been connected to major relays (20+) including nos.lol, nostr.mom, nostr.band
Yes, currently the code were implemented by looking image url file in notes text -> temporary download -> preprocess/resize image -> classify image using tensorflow pretrained model. Quite exhausting to process all notes 😅
wow thats awesome!
since keeping the notes in the ram is getting costlier, i want to divide nos.lol into two. one recent, one archive. the archive can run a different implementation than strfry or it may be strfry too. i don't know yet.
i need a proxy that does this:
- it will connect to two strfry instances for reading (REQ)
1. nos.lol recent notes
2. nos.lol archived notes
and combine the notes and give them to client.
- it will write to 1.
- it will also pass x-real-ip (for spam combat).
can you do it?
> since keeping the notes in the ram is getting costlier, i want to divide nos.lol into two.
Oh, did you set "noReadAhead = false" in lol and mom strfry.conf? I have change it to "true" since my relay database has already bigger than server RAM 😅
What I did as trick to keep some events fresh in memory cache was running cron script that fetch events every hour:
nosdump --kinds 0,1,6,7,9735 --since 24h ws://172.17.0.100:7777 > /dev/null 2>&1
It will force/trick strfry and makes 24 hour last common events fresh in it's memory cache.
> i need a proxy that does this:
I think the one that probably fulfill some of your requirements is bostr proxy relay. It can read from multiple relays. I'm not sure if it can be set to write only 1 relay but read from 2 relays like you said. If not then you need to modify it a little bit.
https://github.com/yonle/bostr
Unfortunately, the proxy relay I'm using currently only support in one server as it has already fulfill my requirements for now. I think I will also probably implement that in future.