I've played with a test relay where use OpenAI moderation to filter out content. Same time also filter out content based on obvious keywords like nsfw. Seems to work well but need more on images side.
Discussion
Wait you query openai on each event or how does it work?
Yes, was not what I was looking to do with our relays so did not pursue much. I think more potential in building custom model and use that free of OpenAI. Or find existing open models.
https://gitlab.com/soapbox-pub/strfry-policies/-/blob/develop/src/policies/openai-policy.ts
Did you measure the latency introduced?
No never got that far into it. I for sure don't like depending on 3rd party API. But I am gathering training data to use with a custom model from our relays and honeypot accounts.
YeH, for sure, running it locally would be the way to go; too expensive and too systolic feeding everything to Sam Cuckman