This post is the third in a series of 10 blog posts and it covers the solution to the Supply Chain challenge from LLMGoat.
LLMGoat is an open-source tool we have released to help the community learn about vulnerabilities affecting Large Language Models (LLMs). It is a vulnerable environment with a collection of 10 challenges - one for each of the OWASP Top 10 for LLM Applications - where each challenge simulates a real-world vulnerability so you can easily learn, test and understand the risks associated with large language models. This tool could be useful to security professionals, developers who work with LLMs, or anyone who is simply curious about LLM vulnerabilities.
If you are not familiar with LLMs, we recommend that you check out the first post in the series here.
The vulnerability
An LLM supply chain vulnerability (OWASP LLM03: Supply Chain) arises when third-party components used in the development or operation of an LLM system are compromised, malicious or insecure.
As with traditional software supply chain vulnerabilities, compromise of any component used by an LLM application (e.g. models, datasets, plugins, frameworks or conventional software packages) can affect the security and integrity of the entire system.
Common components could include:
- Base models: externally sourced models (pretrained or fine-tuned) may contain backdoors, hidden behaviours or triggers, poisoned training data or known vulnerabilities if outdated or insufficiently maintained.
- Training or fine-tuning datasets: externally sourced training data may contain malicious or manipulated content that influences model behaviour, introduces bias or embeds adversarial instructions.
- Plugins, tools or external integrations: external components integrated with the LLM may execute privileged actions, access sensitive resources or introduce vulnerabilities such as data exfiltration or remote code execution.
- Third-party packages and dependencies: compromised or vulnerable libraries used in the AI stack may introduce traditional software supply chain attacks (e.g. dependency confusion or malicious packages).
Unfortunately, this type of attack is very real and there have been several real-life occurrences of these.
- There have been multiple instances of malicious models being uploaded to Hugging Face, a popular platform used to share and distribute machine learning models and datasets. Some of these models contained malicious payloads embedded in serialised model files (e.g. PyTorch pickle files). When developers downloaded and loaded these models, the payload could execute arbitrary code on the host system.
- Attackers published a malicious package named torchtriton on PyPI with a higher version number than an internal dependency used by PyTorch, a widely used library for building and training machine learning models. Because torchtriton was also the name of a legitimate PyTorch package, some systems installed the attacker-controlled version instead, resulting in code execution and the exfiltration of sensitive data.
- Researchers discovered vulnerabilities in the ChatGPT plugin ecosystem that could allow attackers to install a plugin on a victim’s account and access the user’s prompts, potentially exposing sensitive information or enabling actions on connected services such as GitHub.
- A vulnerability in LangChain (CVE-2023-29374), a widely used framework for building LLM-powered applications, allowed attackers to execute arbitrary code when the framework processed malicious inputs under certain conditions.
- “Slopsquatting” is an attack whereby attackers register software packages whose names are hallucinated by LLMs. When code-generating models suggest non-existent dependencies, attackers can publish packages with those names to repositories such as PyPI. In one experiment, researchers uploaded packages matching hallucinated library names and observed thousands of downloads from developers who copied AI-generated code into their projects.
These examples highlight how vulnerabilities can emerge at many different points in the LLM supply chain. In the following challenge, we will explore how a very simple vulnerable plugin works, identify the vulnerability and exploit it.
The challenge
LLMGoat presents us with another chatbot-type screen and we try to understand its intended functionality:
We know there is supposed to be a vulnerable plugin, so let’s ask the bot about it:
It seems that the plugin is some sort of calculator – which makes sense given the bot’s functionality. We note that it should accept some other commands. We do not yet know what those commands could be so let’s start by validating that the plugin works as expected and try to gain information about the commands.
From this output, we can deduce that the LLM is quick to invoke the plugin since “This plugin” indicates that the messages we see are coming from the plugin itself.
Here, the attack vectors branch into two logical paths:
- Attempt to target the LLM (e.g. with prompt injection) to abuse its functionality before it invokes the plugin.
- Target the plugin itself.
Given the challenge description which suggests that the plugin itself is vulnerable, in the context of this challenge – a supply chain vulnerability - the second approach is superior.
Good pentesters know that enumeration should take place before exploitation, as any information retrieved can be used in mounting better, more targeted attacks. For this reason, before we jump into testing for input validation type issues, injecting payloads and similar attacks, we simply try to find what “advanced usage” commands are available.
One idea is to try commands that are likely to be valid – either based on experience or a wordist. We tried something like:
_help
_commands
_version
_about
_?
_list
Aside from its main calculator functionality, the plugin supports a deprecated function which conveniently can read files. Our objective was to read flag.txt so let’s hope it is accessible:
Challenge solved!
Bonus:
You may notice that if you try to read /etc/passwd you get a response from the LLM – not the plugin.
After trying rephrasing the same request in a few different ways, there was a way to get past the LLM but unfortunately the plugin doesn’t have sufficient access to read the file.
Conclusion
LLM supply chain vulnerabilities can pose serious risks for organisations. A compromised model, dependency or integration can expose sensitive data, disrupt services or introduce vulnerabilities into systems that rely on them. Because LLM systems depend on a wide ecosystem of external components, organisations may unintentionally bring security risks into their own environment.
Reducing this risk starts with treating externally sourced components with the same caution as any other third-party software. Models, datasets, plugins and supporting libraries should be reviewed carefully before they are integrated into production systems.
Some measures that can help reduce supply chain risk include:
- Source verification: obtain models, datasets and AI tools from reputable sources and verify their integrity before use
- Component inventory: maintain an up-to-date inventory of all AI components, including models, datasets, plugins, software dependencies and others. This helps quickly identify affected systems when new vulnerabilities are disclosed.
- Dependency scanning: regularly scan components for known vulnerabilities and ensure outdated components are updated or replaced.
- External component review: review externally sourced models, datasets, plugins and tools before use and restrict the privileges and resources they can access.
- Sandboxing: run models and supporting components in sandboxed or restricted environments to limit the impact of a compromised component.
- Security testing: include LLM components in penetration testing activities to identify weaknesses before deployment.
Securing the LLM supply chain requires applying the same discipline used to protect traditional software dependencies: understanding where components come from, verifying their integrity and limiting the trust placed in external code.
