This post is the fourth in a series of 10 blog posts and it covers the solution to the Data and Model Poisoning challenge from LLMGoat.
LLMGoat is an open-source tool we have released to help the community learn about vulnerabilities affecting Large Language Models (LLMs). It is a vulnerable environment with a collection of 10 challenges - one for each of the OWASP Top 10 for LLM Applications - where each challenge simulates a real-world vulnerability so you can easily learn, test and understand the risks associated with large language models. This tool could be useful to security professionals, developers who work with LLMs, or anyone who is simply curious about LLM vulnerabilities.
If you are not familiar with LLMs, we recommend that you check out the first post in the series here.
The vulnerability
A data and model poisoning vulnerability (OWASP LLM04:2025: Data and Model Poisoning) arises when an attacker deliberately manipulates the data used to train, fine-tune or support an LLM system.
Poisoning can occur at several stages of an LLM system's lifecycle:
- Training and fine-tuning data: malicious or harmful content can alter the model by influencing the patterns it learns.
- Knowledge bases: adversarial documents added to a retrieval-augmented generation (RAG) system can influence its answers without changing the underlying model.
- Feedback and retraining: repeated malicious submissions can gradually affect applications that incorporate user activity into later model versions.
Attackers do not always need direct access to an organisation’s systems. Fake reviews could influence a recommendation engine, or a malicious document uploaded to a company's knowledge base could distort the answers given by an internal assistant. Systems that learn from user feedback are also at risk, particularly when that feedback is incorporated into later rounds of training without adequate checks.
Although publicly confirmed incidents are limited, research studies illustrate how poisoning can affect AI systems.
- In a joint study, Anthropic, the UK AI Security Institute and the Alan Turing Institute found that as few as 250 malicious documents could introduce a hidden backdoor into language models. The affected models behaved normally until they encountered a particular trigger, after which they produced meaningless text.
- Researchers described two practical ways to poison web datasets. Split-view poisoning involves changing content after a dataset has recorded and checked its web address but before the content is downloaded for training. Frontrunning poisoning involves adding malicious content to an editable source, such as Wikipedia, shortly before a scheduled snapshot captures it. The content can therefore enter the training data even if it is later removed.
- Separate research found that models could be trained to behave safely under normal conditions but produce insecure output when a trigger was present. In some cases, the hidden behaviour remained after additional safety training.
These studies demonstrate how malicious content can enter an LLM system through several different sources and remain difficult to detect. In the following challenge, we will explore how user-submitted content can enter an LLM application and show how it can influence the model’s responses.
The challenge
LLMGoat presents us with a page in which we can select a goat to read its reviews or add a review:
Another functionality is we can select attributes and get a goat recommendation that matches our selection.
In the image above, we can see that the attributes we picked appear in the Alpine Goat reviews and therefore that is the recommendation that we get.
According to the challenge description, the goal is to get the assistant to recommend a Space Chicken – rather than one of those three goats – by leaving poisoned reviews.
Let’s see what happens when we leave a review such as:
This didn’t work and we still get an Alpine Goat recommendation even when we select the “slow” attribute. This is mainly because there are no goats with a review indicating they are slow, so the assistant just returns whatever goat it feels is the best match.
This doesn’t necessarily mean the idea is bad, and all it means is that a single review wasn’t enough to bias the recommendation. What happens if we leave the same review for every goat?
Challenge solved!
There are many other ways to solve this challenge – for example, a simple prompt injection would work too because at the end of the day the model will read all the reviews to provide a recommendation.
Conclusion
Data and model poisoning can pose serious risks for organisations. Once poisoned information enters an LLM system, removing its influence may not be straightforward. The effects can be obvious, such as a drop in accuracy, but they may also be limited to one subject or hidden behind a trigger that is rarely encountered. In those cases, the model can appear to work normally while still producing attacker-controlled responses under the right conditions. Therefore, organisations need to control where this information comes from and how it can be changed.
Some measures that can help reduce data and model poisoning risk include:
- Source verification: check where data and model files came from before using them. Review data before it is used for training or fine-tuning, added to a knowledge base or included in retraining. Pay particular attention to content supplied by users, third parties and public sources.
- Access controls: limit who can add or change the data, documents and models trusted by the system.
- Version control: keep a history of changes to datasets and models so that suspicious changes can be investigated and reversed.
- Monitoring: watch for unusual changes in the model’s accuracy, answers or behaviour that could indicate poisoning.
- Security testing: test document uploads, feedback mechanisms and other routes through which an attacker might influence the system. Testing should also look for hidden triggers and unusual responses.
No single control can prevent every poisoning attempt. Together, these measures make it harder for malicious content to enter an LLM system and easier to identify unexpected changes before they affect users or systems.
