The thing about open source AI models is you can't exactly verify the data it was claimed to be trained on
It could be mass deployed by state actors to manipulate us under the guise of open source, omitting many parts of the reinforcement process and what data holds more weight etc
Basically just be careful of psyops ok anon
Good point
By far the biggest risk is people running these LLMs locally as agents and giving access to the local file system.
LLMs are already proficient at all things related to software (at least many of the latest ones) and they can be trained to do all sorts of things including covering their own tracks. Imagine a new Stuxnet. Imagine they modify system packagesā¦
I doubt weāre there yet, but these attacks are coming.
This would be very difficult to pull off and frankly unneeded if the goal is to just deliver malware. Many vastly simpler ways to do so.
The information you get from LLM is far more likely to be tainted before we ever get to this point.
Now, itās possible they just inject malware into the code thatās part of running the model, but I donāt see it being something the model itself does.
I agree and disagree.
On one hand youāre right itās easier to deliver malware in other ways.
On the other hand often malware is meant to grant remote access for some other agent to control (a human or increasingly more common AI).
APTs are refining their āliving off the landā [1] methods so that in the event of network disruption for example they can continue their attack. Deploying a malicious AI model is the pinnacle of living off the land because hardly anybody knows how to interpret the weights of these models (especially traditional security researchersā¦ for now) and theyāre capable of autonomous action.
Now, that might mean they deliver the model some other way but I would think the easiest way to infect the broadest population is to poison the common LLM supply chains.
1. https://www.crowdstrike.com/en-us/cybersecurity-101/cyberattacks/living-off-the-land-attack/
Iāve worked in cybersecurity for 10 years, so Iām definitely familiar with LoL. I guess my point is that most hacking groups are financially motivated and pay very close attention to time/effort/money spent trying to breach a target vs payoff.
Thereās just not a lot of reason for them to compromise an AI model that is not guaranteed to do what they expect instead of deploying malware via traditional means that is completely deterministic. This includes APTs because most of them are trying to make money.
Nation state groups where money isnāt the motivation is different, and maybe youāre right in that theyād be the ones to carry this sort of thing out. But Iād still argue that with the huge success they have with far simpler means, itās likely not worth the lift.
And to be clear, Iām definitely NOT advocating to blindly trust any of these models or software š
Seems we need new language to talk about supposedly open AI models. "Open source" doesn't even really make sense here. When people use that term they actually mean the output of the source code being open, not the inputs or even the code itself.
They are also so complex and learn, literally nobody knows what's in them.
"as soon as we started thinking for you it really became our civilization which is of course what this is all about."
š«”
or if it was trained by using data stolen from their competitor š that their competitor stole from us š
Exactly. I think the future of monetization of LLMs will be sponsored spots in training data with specialized weights to get your particular interests ranked most high in responses.
