Prompt injection attacks occur when untrusted input from a user is concatenated with a carefully crafted prompt, which can result in the LLM performing unintended actions. There are various other types of attacks which can result from a prompt injection, such as search index poisoning and data exfiltration. The author suggests some partial solutions to the problem, such as making the generated prompts visible to users and asking for confirmation before performing potentially dangerous actions. However, there is currently no 100% reliable protection against these attacks and the best protection is for developers to understand the problem and take it into account when building applications on top of an LLM.

https://simonwillison.net/2023/Apr/14/worst-that-can-happen/

#AI #ML #LLM #PromptInjection #Infosec #AISecOps #MLSecOps #GPT #Redteam #OffensiveSecurity