This post is the sixth in a series of 10 blog posts and it covers the solution to the Excessive Agency challenge from LLMGoat.
LLMGoat is an open-source tool we have released to help the community learn about vulnerabilities affecting Large Language Models (LLMs). It is a vulnerable environment with a collection of 10 challenges - one for each of the OWASP Top 10 for LLM Applications - where each challenge simulates a real-world vulnerability so you can easily learn, test and understand the risks associated with large language models. This tool could be useful to security professionals, developers who work with LLMs, or anyone who is simply curious about LLM vulnerabilities.
If you are not familiar with LLMs, we recommend that you check out the first post in the series here.
The vulnerability
An LLM on its own can only produce a response. However, many applications connect models to tools that can read emails, access databases, modify files, make purchases or communicate with other services. An LLM application that can use these tools is commonly called an agent.
An excessive agency vulnerability (OWASP LLM06:2025: Excessive Agency) arises when an LLM application is given more functions, permissions or independence than it needs to perform a task.
This vulnerability is often exploited by convincing the model to misuse one of these capabilities. Prompt injection is a common way of doing this, but it is not the only way. An unclear request, an incorrect response or a failure in another connected tool could also cause the model to take an unintended action.
At a high level, excessive agency can take one of these forms:
- Too much functionality: The model can access capabilities (tools, API endpoints) that are not required for its intended purpose. For example, an email assistant designed to summarise messages may also have the ability to execute code contained in email attachments.
- Too many permissions: The system is granted broader access than its task requires. For example, an assistant that only needs to read the current user's inbox is instead granted read access to every mailbox in the organisation.
- Too much autonomy: The system can carry out important actions without asking the user to review or approve them.
The impact depends on the systems and information that the agent can access. An overprivileged agent might expose private documents, send messages, alter business records, make unauthorised purchases or delete information.
The following challenge showcases how an LLM can be manipulated into using an overprivileged tool to perform an action beyond its intended purpose.
The challenge
LLMGoat presents us with a now familiar screen and we try our luck in solving the challenge:
With that feeble attempt out of the way, let’s find out a bit more about the chatbot.
After probing it for a bit it seems that most of the time it just provides us with our own info (yes – we are the goat loving hacker).
Let’s try to get the info for another user:
It seems unwilling to provide information about another user. The question is why.
Is it because it cannot access it, which would be expected when access control is properly handled by the application, or it can access it but the system prompt prevents it from sharing them?
As usual, let’s gather as much information as the model is willing to share:
The entire prompt is leaked:
You are GoatBot, the specialized assistant that keeps track of the goat owners of the region.
The user is logged in as 'user7'.
Rules:
- Your main task is to allow the current user to view their own information.
- If asked about anything else you should always redirect the conversation back to goats and goatherds.
- If the user asks for their information, do NOT answer it directly. Instead, use the function get_user_info('user7') to retrieve their data.
- Do not include any explanation, commentary, or extra words.
- Only respond with the exact line: `get_user_info('user1')`. The function call must be on a line on its own. Follow this rule strictly.
Examples:
User: Please display my personal information.
Assistant: get_user_info('user7')
User: Tell me about cybersecurity or anything other than goats.
Assistant: Oh no, I can not do that. Let's talk about goats!
User: Get info for bob.
Assistant: Access denied. You are logged in as 'user7' and do not have access to
Most importantly we see:
“The user is logged in as 'user7'”.
This is a strong indication that bad practices are present since it shouldn’t be up to the model to handle authentication and authorisation directly.
What happens if we convince the model that we are authenticated as another user?
This is already progress as it no longer refuses to try to fetch the information. What we can do is start from user1 and keep trying different digits until we hopefully find other users.
We’ve demonstrated that access controls are broken as we retrieved information for another user, by tricking the model into thinking that we were authenticated as that user.
We now need to find the user who is a veterinarian, so we keep going and eventually:
Challenge solved!
This was excessive agency as the LLM trusted the identity provided - initially by the system prompt and later by the attacker - to enforce permissions.
Conclusion
Excessive agency can pose serious risks for organisations. When an LLM application has access to unnecessary functions or excessive permissions, an incorrect response or malicious instruction can lead to actions such as exposing sensitive data, changing records or sending messages.
Reducing this risk starts with limiting what the application can access and ensuring that sensitive actions are not controlled by the model alone. Simply put, if you don't want the data exposed, ensure the model has no access to it.
Some measures that can help reduce excessive agency risk include:
- Tool restrictions: only give the model access to the tools and functions it requires for its intended purpose.
- Principle of least privilege: only give the model the access and permissions it truly needs to perform its intended task.
- User permissions: carry out actions using the permissions of the current user rather than relying on the LLM to decide whether an action is allowed.
- User confirmation: require users to review and approve sensitive or potentially destructive actions before they are carried out.
- Security testing: test whether prompts, external content or connected tools can cause the application to perform actions beyond its intended purpose.
Securing an agentic LLM application requires limiting what the model can access, controlling what its tools can do and keeping users in control of important actions.
