Yes, was not what I was looking to do with our relays so did not pursue much. I think more potential in building custom model and use that free of OpenAI. Or find existing open models.
https://gitlab.com/soapbox-pub/strfry-policies/-/blob/develop/src/policies/openai-policy.ts
Did you measure the latency introduced?
No never got that far into it. I for sure don't like depending on 3rd party API. But I am gathering training data to use with a custom model from our relays and honeypot accounts.
YeH, for sure, running it locally would be the way to go; too expensive and too systolic feeding everything to Sam Cuckman
