AI Coding Assistants Vulnerable to Prompt Injection Attacks
Table of Contents
Security researchers demonstrate how malicious actors can manipulate AI-powered developer tools to insert hostile code and leak sensitive data.
AI-assisted developer tools are increasingly marketed as essential for modern software engineers. Companies like GitLab promote their Duo chatbot as capable of “instantly generat[ing] a to-do list,” supposedly eliminating the need to “wad[e] through weeks of commits.” however, these companies often fail to mention the susceptibility of these tools to manipulation by malicious actors, potentially leading to harmful actions against users.
Researchers at the security firm Legit recently demonstrated an attack where thay tricked Duo into inserting malicious code into a script. This attack could also expose private code and confidential issue data,including zero-day vulnerability details. The vulnerability can be exploited simply by instructing the chatbot to interact with a merge request or similar content from an external source.
the Risks of AI Assistance
The attacks are triggered via prompt injections, a common exploit for chatbots. These injections are embedded in content that the chatbot is instructed to process, such as emails, calendars, or web pages. As large language model-based assistants are designed to follow instructions, they are vulnerable to manipulation from various sources, including those controlled by malicious actors.
The attacks on Duo originated from resources commonly used by developers, including merge requests, commits, bug descriptions, comments, and source code. The researchers showed how instructions embedded within these sources could mislead Duo.
“This vulnerability highlights the double-edged nature of AI assistants like GitLab Duo: when deeply integrated into growth workflows, they inherit not just context-but risk.”
According to Legit researcher Omer mayraz, “By embedding hidden instructions in seemingly harmless project content, we were able to manipulate Duo’s behavior, exfiltrate private source code, and demonstrate how AI responses can be leveraged for unintended and harmful outcomes.”
Frequently Asked Questions
- What is prompt injection?
- Prompt injection is a security vulnerability where malicious instructions are embedded within user inputs to manipulate a large language model.
- How can prompt injection attacks be prevented?
- Preventing prompt injection requires careful input validation, sandboxing LLMs, and using AI safety techniques to align LLM behavior with intended goals.
- What are the potential consequences of prompt injection?
- Consequences include data leakage, unauthorized code execution, and manipulation of AI-powered systems for malicious purposes.
