#asknostr among the problems that Nostr faces, the child porn problem is a very, very, very bad problem.

A VERY bad problem.

What is the current thinking among developers about how to deal with this?

Nobody likes censorship, but the only solution I can think of (SO FAR) is running an image identification service that labels dangerous stuff like this, and then broadcasts a list of (images, notes, users?) who are scoring high on the "oh shit this is child porn" metric. Typically these systems just output a float between zero and 1, which is the score....

Is anyone working on this currently?

I have a good deal of experience of running ML services like image identification at scale, so this could be something interesting to work on for the community. (I also have a lot GPU power, and anyway, if you do it right, this actually doesn't take a ton of GPUs to do even for millions of images per day....)

It would seem straightforward to subscribe to all the nostr image uploaders, generate a score with 100 being "definite child porn" and 1 being "not child porn", and then broadcast maybe events of some kind to relays with this "opinion" about the image/media?

Maybe someone from the major clients like nostr:npub1yzvxlwp7wawed5vgefwfmugvumtp8c8t0etk3g8sky4n0ndvyxesnxrf8q or #coracle or nostr:npub12vkcxr0luzwp8e673v29eqjhrr7p9vqq8asav85swaepclllj09sylpugg or nostr:npub18m76awca3y37hkvuneavuw6pjj4525fw90necxmadrvjg0sdy6qsngq955 has a suggestion on how this should be done.

One way or another, this has to be done. 99.99% percent of normies, the first time they see child porn on #nostr ... if they see it once, they'll never come back.....

Is there an appropriate NIP to look at? nostr:npub180cvv07tjdrrgpa0j7j7tmnyl2yr6yr7l8j4s3evf6u64th6gkwsyjh6w6 ? nostr:npub1l2vyh47mk2p0qlsku7hg0vn29faehy9hy34ygaclpn66ukqp3afqutajft ? nostr:npub16c0nh3dnadzqpm76uctf5hqhe2lny344zsmpm6feee9p5rdxaa9q586nvr ?

Reply to this note

Please Login to reply.

Discussion

Something we can call community notes. NOSTR already has support for reporting this type of profile. It depends on the relays identifying the profiles and not returning their notes.

OK, but I think that's not fast enough. The bad guys move too fast, you'll never be able to actually prevent n00bs from seeing the very bad stuff like this. (I think.)

And most important Relays should report all logs about these accounts to the police. It is not as if this is freedom of speech and many just do not like it. It is common sense, that people sharing those should be nowhere, but behand bars.

We need basically an open standard, a NIP, for any user to broadcast "scores" for any content. Clients of course don't have to use or look at the data if they don't want. But if your client is targeting new Nostr users, maybe you would look at the scores or allow users to opt-in or opt-out of the scores. And anyone could publish their own scores -- they are probably just a {image_url: ___, score: _ } format or something. Score is between 0 and 1, with 1 being "very very likely child porn" and 0 being "ok"}

NIP-36 is all we really have for any kind of pornographic content filtering. For me, that is usually enough, because I don't want to see ANY pornographic content in my feed, CSAM or otherwise. Let the degenerates worry about how to specify what level of degeneracy they are ok with seeing.

Stopping people from posting it in the first place is going to be a game of whack-a-mole that relay operators and media-hosters are going to have to tackle. Hence why the only relays I operate are ones that only I can post to, or that only folks within my web-of-trust can post to.

I use Nostrify moderation polices to block all porn and it's very effective!

https://nostrify.dev/policy/

So are you doing this only by keyword? Or something else?

Basically! The hashtag policy catches the overwhelming majority of it.

Thank you! This is the first I'm hearing about this.

very easy: don't follow CASM accounts, only view notes from those you follow.

That's kind of the solution indeed. But we still have to fix reply spam.

I think I'm the only one who noticed, but someone dropped 2 naked toddler pics in the replies to this thread.

Yeah, it's totally fucked up. We need to fix this.

By convention replies are sent to the thread root's defined "read" relays.

I think it's fair to assign responsibility to the thread creator for picking unsafe relays. But of course it could also be your client's fault for using random public relays or something like that. Relays are the only way to filter content properly on Nostr, clients and users should start to be more mindful of that.

In my case I use Gossip with the nuclear option of only reading replies from the relays I specify manually, so even if the thread creator is evil I won't see evil replies, but this is not the ideal solution either.

Filtering out replies from people with low WoT score like Coracle does also works fine. It can't prevent your computer from beeing flooded with spam, but at least in the short term it would have filtered all these evil targeted attackers.

Yeah, it was Amethyst. I don't see any garbage on Nostrudel or Voyage, but Amethyst seems to pull replies from anyone, anywhere.

Ditto with web of trust policy solves all these .....issues

What if I want to pull replies from anyone anywhere?

The problem starts, when you use search and hashtags.

They use popular hashtags sometimes

Hashtags are stupid.

Search I don't know, never got any such results from any search I made. But anyway search relays should be more strict on content they accept, they can't be just open to anyone to flood, this will never work.

'stupid' maybe, but how do you find other wrestlers/fishermen/the seamstress union/ whatever?

I looked for BJJ and there was CSAM tagged with [hash]BJJ (besides other tags)

'Normal' People will use tags at least when they're new to nostr just to find a handfull of people to start with.

How do you follow new users if I can only look at notes from users I follow? 🤔

Npubs you follow will sometimes repost notes from other people. And you can look at their follow lists. Or even "view this user's feed", which is supported in some clients

Try hastags

Relays have to become more whitelisted and less open, and clients have to implement outbox model and stop relying on 2 or 3 big relays, then we can just stop worrying about this.

If you have a server that anyone is free to write to on the internet this kind of stuff will always happen. The obvious solution is to not have this kind of server.

You can also have this kind of server but disallow links. That will probably go a long way too.

The obvious solution is to just give up working on it and spend the next few years smoking weed and going snowboarding instead. But the actual solution I think has to be some kind of distributed scoring system.

I drafted a spec to auto block/show/ask for content previews from specific domain providers. Clients can automatically block previews from a domain, or let the users decide on what to do with it. You can aggregate these lists from different users into a rating system for domains as well. Would appreciate any feedback.

```

{

"kind": 10099,

"content": "",

"tags": [

["d", "domain_lists"], // identifier

["white", "nostr.build"],

["white", "void.cat"],

["black", "malicious-site.net"],

["black", "scam-domain.com"]

["unknown", "ask"] // Options: "load" | "block" | "ask"

]

}

```

https://github.com/limina1/nips/blob/extend56/56.md

I just don’t know if a “domain blacklist” will

work well. I think it will be too slow, too incomplete, and ineffective. I think the only way to do this at scale is that relays have a way to score images and videos and simply be sure to delete and not re-broadcast any which get a bad score.

Wonderful that some arse decided to drop some pics in this particular feed. Charming content, brought to me by the public Einundzwanzig relay.

Not sure if you are serious or just trolling the idea. But -- like each individual relay implements its own scoring system? Seems like a ton of duplicated effort.

I am not trolling.

I do think it would be good to have a system for identifying harmful stuff. It would be a nice workaround that would work today and I would definitely adopt it at https://njump.me/ because we keep getting reports from Cloudflare. I tried some things but they didn't work very well, so if you know how to do it I'm interested.

However the long-term solution is paid relays, community relays, relays that only give access to friends of friends of friends, that kind of stuff.

so why do we even need nostr then?

we have mastodon

Because Nostr isn't written in Ruby.

OK, so thinking about it more, in light of what nostr:npub1q3sle0kvfsehgsuexttt3ugjd8xdklxfwwkh559wxckmzddywnws6cd26p says ... 1) Obviously the spec to use would be the LABEL spec nip-32 -- not sure why I didn't figure that out to begin with... https://github.com/nostr-protocol/nips/blob/master/32.md 2) My original idea of "publicly publish a score for each image" is completely impossible and terrible idea... because, of course, the bad guys could actually just use the service in the reverse way that it's intended to be used! ....... Anyway, 1/2 of the problem -- running a service which produces scores -- is completely something I could do -- basically process millions of images and spit out scores for them -- but the other 1/2 ... how to let clients or relays use these scores WITHOUT also giving them a "map to all the bad stuff" at the same time...? I'm not smart enough currently to come up with a solution. It might involve something fancy involving cryptography or "zero knowledge proofs" or things that are generally out of my intellectual league.

> Relays have to become more whitelisted and less open

No.

And then everyone runs a personal relay (I'll take care of making that trivially easy for people) and everything is perfect!

Broadcasting public notes that identify CSAM is probably illegal, because it could be construed as "advertising" that content. I think the only option we really have long-term, at least in the US, is for someone(s) to run a service that crawls the network, matches images against microsoft's hash database product (closed source, but for good reasons, since hash databases can be reverse engineered), and reports matches to NCMEC. A bonus would be to do the same thing but analyze note text for exploitation keywords. Privately hosted and encrypted content are pretty much immune to this fortunately/unfortunately. nostr:nprofile1q9n8wumn8ghj7enfd36x2u3wdehhxarj9emkjmn99ah8qatzx96r2amr8p5rxdm4dp4kzafew3ehwwpjwd48smnywycrgepndcu8qd3nx36hguryvem8xdr5d56hsmt5xfehzemtxejxkeflvfex7ctyvdshxapaw3e82egprfmhxue69uhhyetvv9ujumn0wd68yanfv4mjucm0d5hszrnhwden5te0dehhxtnvdakz7qg3waehxw309ahx7um5wgh8w6twv5hsz9nhwden5te0wfjkccte9ekk7um5wgh8qatz9uqzpxvf2qzp87m4dkzr0yfvcv47qucdhcdlc66a9mhht8s52mprn7g98p5le2 currently checks a hash database for all images uploaded, and I believe they report matches.

As non-cypherpunk as this all is, I think it's the only real option we have unless Ross Ulbricht's ZKANN idea gets built. We need to demonstrate to anyone watching that we take the problem seriously and take measures to self-regulate. This is similar to the bitcoin KYC/AML argument. If we don't want financial surveillance or legal restrictions on social media, we should help law enforcement actually chase down the people who are the problem rather than presenting ourselves as the scapegoat. See iftas.org for some work being done in the fediverse on this.

Agreed on the "broadcasting identifications are probably illegal". If only there was a provably safe way to do it where the exact location of the content (i.e., URL) , wasn't communicated but you gave clients a 99% certainty of being able to block it, still. basically, somehow you give clients the power to test any url to decide if it was bad, but you don't provide the actual scores in such a way as the scores could be directly used in a search or other content discovery exercise... Maybe this is just impossible, because, what's to stop someone else from running a script to test every image and reproduce the score, and then try to use or access the bad scores?

Right, that's basically how microsoft's product works. They don't release the database because you could then craft images that foil the fuzzy hashing algorithm they use.

Right. But FYI, you don't need microsoft's service, you can roll your own with open source models that will return a confidence score between 0 and 1. And a lot of those models are totally open source -- https://huggingface.co/docs/transformers/en/tasks/image_classification They are just classification models which return a value between 0 and 1. And they're pretty fast & efficient since Google and other have been fighting this issue for 20+ years and have developed very good and efficient models. (Which work 99.5% of the time. I think it's impossible to get to 100%).

We use one of these models.

Right. That's what I thought. And it works for you, right? You've been able to measure the benefits in terms of fewer complaints or something?

Yes, easier and quicker to identify, less we have to do. We still use PhotoDNA for their hashes, but is the best solution so far.

It's a totally different approach, but maybe you're right, good LLMs are relatively new and maybe could be considered to supersede fuzzy hashing. But the main problem of reverse-engineering the compression algorithm (which is one way to think about llms) still exists. If you're thinking of working on this I'm happy to see what I can do to help.

We tried CloudFlare’s integrated service and Microsoft’s PhotoDNA, they are ok, but only compare to existing hashes and only supported images, not videos.. AI models scan it all, searches existing hashes and recognizes unreported patterns.

Here's an example: https://huggingface.co/Falconsai/nsfw_image_detection ... putting one of these (or actually multiple, and averaging the results...) behind an API endpoint is not too difficult, and I'd be happy to do it for any service which has a **way to measure the effectiveness** ... since I will not be reviewing any images manually (!) , and YOU will not be reviewing any images manually (1) and I will be deleting all data a few milliseconds after it hits the model and returns a score, you must have SOME way of deciding if the service is useful. Like, user complaints, or blocks, or something like that.... ideally if you run a big enough service where you can measure "complaints/blocks per day" and see that the "number goes down" when you start using the scores that I provide.

As discussed in this thread, making these scores public is potentially dangerous, but providing a service that simply scores images, especially if that service is only offered to a small number of entities who can be trusted to use it only to help them delete something .... is something Microsoft has been doing for decades, I can't see any particular risk in it.

But I only want to build this if someone can say "yes, I'll be able to measure the effectiveness somehow"... because doing this without measurement of any kind of useless, right?

That's a good approach, but could be tricky on nostr. Maybe you could scrape NIP 56 reports? I know those still get published by some clients (which is awful, but I can't convince the devs to stop).

It's not awful.

It's illegal

Awful is very different from illegal.

Ok, but legality has bearing on awfulness

You don't have to worry about definitions. These models are very smart and are happy to provide you with a float between zero and one. And then you just set a threshold on the what scores you will tolerate. No need to engage further with the question.

Semantics aside, I'd use that if I ran a relay. I'm not sure it makes sense to bake it into a client? Especially since it would be sending the data to a third party server

Right. Actually baking it into a client is a thing that would be the most dangerous too.

Yes, more than often what is awful is legal and what is good is illegal.

We use an AI model to recognize and report it, if it is not sure we manually confirm and report. It’s not tolerated in any way and we will do what is needed to rid it of nostr.

Same as only getting notes from specific relays, one could only load media from specific domains that do csam filtering, like nostr build.

Maybe the client app could do run some local AI model & scan for the "other" domains that host the media.

ATproto folks use this apparently

https://www.thorn.org/solutions/for-platforms/

Anyone can run a relay, it's cheap:

Probably because of that cost Bluesky say they'll mix in something AI from this newly-launched Roost.

https://blog.mozilla.org/en/mozilla/ai/roost-launch-ai-safety-tools-nonprofit/

https://github.com/hzrd149/blossom/blob/master/buds/09.md

reporting those medias and events. relays and clients must care about them.

primal seems to be the worst offender

I don’t think it needs to be handled at the protocol level - more like individually

Yeh, building an AI powered CP search engine probably isn't a good idea.

No idea how you implement any of it, but so, so glad you're talking about it. Protect the kids, eliminate the abusers.

I don't have images displayed by default and I don't have any way to tell (other than the person I'm following) whether to display the image or not. Once I see it I cannot unsee it.

I think encouraging self-publication of honest text descriptions of non-text content is the way to go. I recently encountered someone who did not wish to publish a content warning so my recourse was to mute that person. If we repeat this process continuously, where both clients and relays can try to evaluate whether a description of media is honest or not, and block dishonest descriptions, that would go a long way.

In other words, use classification systems not to identify one particular type of content on a scale from 0.0 to 1.0, but rather have it judge whether the attached description of an image or video is honest on a scale from 0.0 to 1.0. Then everybody can block dishonest npubs and filter what they wish to see or not see based on descriptions. If an image or video URL does not have a description alongside it, score it 0.0. I don't know which NIPs this would use or add. And the classification systems would not be required to be used by anyone, but they might help identify dishonest sources.

TLDR: encourage honest text captions.

This is basically a war against illegal and dangerous content. You can’t just nicely ask the other side to politely label their weaponry.

The very purpose of a censorship-resistant communications network is to permit some forms of "illegal and dangerous" content because "illegal" varies with time and jurisdiction and "dangerous" varies with time and culture. Nostr doesn't host non-text media, websites do. I am not trying to minimize the real issue at hand. My suggestion is not about nicely and politely. I suggest we build incentives into clients and relays for people to honestly label their content. Those who do not label at all can be easily exiled. Even posts without a label can be easily blocked. Those who label dishonestly can be (admittedly less easily) exiled.

Unpopular opinion: Because of this problem, because of the refusal of relays block image links which contain CSAM, several of the major Nostr apps, possibly including nostr:npub1yzvxlwp7wawed5vgefwfmugvumtp8c8t0etk3g8sky4n0ndvyxesnxrf8q , nostr:npub12vkcxr0luzwp8e673v29eqjhrr7p9vqq8asav85swaepclllj09sylpugg , and nostr:npub18m76awca3y37hkvuneavuw6pjj4525fw90necxmadrvjg0sdy6qsngq955 will be kicked off the Play Store and App Store. This might happen fairly soon, too. Web applications will be fine and won't be impacted. Relying on "user reports" or "web of trust" is a joke and won't prevent these apps from getting killed by Apple or Google. Sorry.

damus does everything it can to block this stuff, and we are continuing to do so:

https://github.com/damus-io/damus/issues/2853

https://github.com/damus-io/damus/issues/2854

every platform has to deal with this stuff, it's not just nostr clients.

the apps have been in the stores for years. What is the reason why you believe that something happens soon?

Fair point. I guess, I'm fairly new to Nostr, just a few months, and I've seen some pretty horrible things, and then I post about it, and it turns out a lot of OTHER people have seen some pretty horrible things, and recently. So at least I know that:

1) There is a problem

2) The problem is unsolved

It's true I don't know much more than this.

@YakiHonne @fiatjaf @PABLOF7z @miljan @ hodlbod This proof of concept shows real-time events that we've found on Nostr relays which are likely bad enough that Google or Apple, if they discover them being "served" by an app like @primal or @Damus or @YakiHonne might decide to take action and remove these apps from the App Store and Google Play. Here is the dashboard where you can see the events: https://nostr-media-alert.com/ .... .... There is definitely an argument to be made that the best course of action is just "do nothing" and wait and see if the apps get removed. From another perspective, it might be nice (in my opinion) if NORMIES, when they first sit down with a Nostr app, aren't immediately blasted with porn. One way to do this would be for relays to consume these "scores" we are producing (and others could produce also), and allow new users to "opt-in" to seeing hardcore porn in their feeds, instead of just like assuming that they will want to see it?