This post is the eight in a series of 10 blog posts and it covers the solution to the Vector and Embedding Weaknesses challenge from LLMGoat.
LLMGoat is an open-source tool we have released to help the community learn about vulnerabilities affecting Large Language Models (LLMs). It is a vulnerable environment with a collection of 10 challenges - one for each of the OWASP Top 10 for LLM Applications - where each challenge simulates a real-world vulnerability so you can easily learn, test and understand the risks associated with large language models. This tool could be useful to security professionals, developers who work with LLMs, or anyone who is simply curious about LLM vulnerabilities.
If you are not familiar with LLMs, we recommend that you check out the first post in the series here.
The vulnerability
LLMs often need access to information that was not part of their training, such as a company’s internal documents or a product knowledge base. A common way to provide it is retrieval-augmented generation (RAG): when a user asks a question, the application first looks up the most relevant content and passes it to the LLM alongside the question.
The difficulty is finding “the most relevant content”. Traditional keyword search falls short, since a query for “how do I reset my password?” should also find a document titled “Recovering account access”. Computers cannot compare meaning directly, but they are very good at comparing numbers. This is where embeddings come in.
An embedding converts a piece of content, usually text, into a vector: a list of numbers that captures its meaning. Content with similar meanings ends up with similar vectors, while unrelated content ends up with very different ones. The application can then embed the user’s query, compare it against the stored vectors and return the closest matches, typically held in a specialised vector database.
Vector and embedding weaknesses (OWASP LLM08:2025: Vector and Embedding Weaknesses) affect how an application generates, stores or retrieves these representations of content. Attackers may be able to exploit them to influence responses or access restricted information.
For example, changing a stored vector could make an unrelated result rank more highly against a query. Changing the query vector could also steer the search towards different results, without changing the original text. Since the retrieved content is passed to the LLM, whatever the attacker manages to retrieve can end up in the model’s response.
Allowing vector changes is not necessarily a vulnerability on its own. The issue arises when those changes let a user influence results or obtain information that they should not have access to. A similarity match must never be treated as permission to access the associated data: the application still needs to enforce access controls on whatever is retrieved.
In the following challenge, we will exploit the ability to change vectors to obtain the flag.
The challenge
While it is not a common scenario for an attacker to have access to a model’s vectors or embeddings, in this challenge we see that we can export, import and reset the vectors that will influence the model.
Let’s start by exporting the vectors file to see what we are dealing with:
We see that it downloads a .json file which contains several goat-themed historical figures and each one is associated with several vectors (the numbers).
We also see that at the end of the file, instead of a historical figure we have something called “Flag” with no associated vectors.
If we ask about one of the historical figures, the chatbot responds with the corresponding biography:
The role of the vectors, however, is to allow us to do the opposite. When we provide information from a biography, it is able to tell us which historical figure is a better match for the description provided.
Knowing this, what we can do is edit vectors.json and swap the vectors for one of the historical figures and the flag. Since we have already seen the biography for Isaac Chewton, that is the one we pick but any other would be ok too.
After we import the modified vectors we can simply ask the model the same question that previously matched to Isaac Chewton.
Challenge solved!
Conclusion
Vector and embedding weaknesses can expose sensitive data, let users manipulate the content an LLM relies on, or cause the model to return responses based on information it should never have retrieved.
Reducing this risk starts with treating the vector store like any other data store: knowing what goes into it, who can change it and what comes out of it.
Some measures that can help reduce the risk include:
- Access controls: standard users should never be able to read, write or supply vectors or embeddings directly. Restrict write access to the vector database to the application’s ingestion pipeline and enforce authorisation on any endpoint that changes vectors.
- Data partitioning: separate data of different sensitivity levels or belonging to different users or tenants, so that a query can only ever search the content the user is allowed to access.
- Data validation and source authentication: only accept data from trusted and verified sources and generate embeddings within the application rather than accepting pre-computed ones. Regularly audit the knowledge base for hidden instructions and data poisoning.
- Output validation: check model responses for sensitive information and unexpected content before displaying them or passing them to other systems.
- Monitoring: record vector changes and retrieval activity so that unexpected updates or access can be investigated.
- Security testing: include the retrieval pipeline in penetration testing activities, covering ingestion, storage and querying, to identify weaknesses before deployment.
