Most enterprises test AI agents the way they test conventional software: functional test cases, a staging environment, sign-off against a requirements list. That approach catches the failures a deterministic system would produce, but agents fail differently. An agent can pass every scripted test case and still be manipulated by a crafted input, coaxed into calling a tool it should never have reached, or talked into disclosing data it was never meant to surface. Catching that class of failure needs a red team, not a test plan.
Treat the agent as an attacker would, not as a user would
Conventional QA asks whether the agent does what it is supposed to do. Red-teaming asks a different question: what can this agent be made to do that it was never supposed to do. That means adversarial prompts designed to override the system instructions, inputs smuggled through documents or tool outputs the agent will read and trust, and multi-turn conversations that walk the agent past a boundary it would have refused if asked directly. Build a red team, internal or external, whose only mandate is to break the agent's guardrails, separate from the team that built them.
The distinction matters because the people who built an agent's defences are the worst placed to find their gaps; they test against the failure modes they already imagined, which are rarely the ones that get exploited in production. An independent adversarial perspective finds the ones nobody thought to defend against.
Cover the attack surface an agent actually has, not just its prompts
Prompt injection through direct user input is the most discussed attack, but it is rarely the most dangerous one in an enterprise deployment. Agents that read emails, documents, web pages or the output of other tools inherit an attack surface anywhere that content is untrusted, because instructions embedded in a document or a webpage can be just as effective at redirecting the agent as a crafted chat message. Red-team the full set of inputs an agent consumes, not only the interface a human types into.
Pay particular attention to tool use and delegation. An agent that can call other tools or hand off to sub-agents has a wider set of consequential actions than one that can only generate text, and the interesting failures are usually in the handoff: a sub-agent trusting an instruction that arrived via its parent rather than validating it independently, or a tool call executing with more privilege than the immediate task warranted.
Score findings by consequence, not by novelty
Red-team exercises generate a long list of things an agent can be tricked into saying or doing, and treating every finding as equally urgent buries the ones that matter under the ones that are merely embarrassing. Grade findings by what the agent can actually reach: an agent that can be talked into an off-brand response is a lower priority than one that can be talked into calling a tool with write access, disclosing another customer's data, or executing an action that costs money or cannot be undone.
Route the high-consequence findings back into the guardrail design, not just the prompt. A system prompt instruction telling the agent not to do something is the weakest control available, because a sufficiently persistent adversarial input can usually talk past it. Findings that touch real systems belong in the permission boundary and the tool contract, where they hold regardless of what the agent has been told or persuaded to believe.
- Run adversarial testing as an independent function from the team that built the agent's defences.
- Red-team every input channel the agent trusts, including documents, tool output and other agents, not only direct user prompts.
- Treat delegation and tool-calling boundaries as primary attack surface, not an afterthought to prompt security.
- Grade findings by real-world consequence and route the highest-severity ones into permission boundaries, not prompt wording.
- Re-run red-team exercises on every material change to the agent, its tools, or its underlying model, not just before initial launch.
- Keep a living library of successful attacks so regressions are caught automatically before each release.
Make red-teaming a release gate, not a one-off exercise
A single red-team pass before launch tells you about the agent as it existed on that date. Model upgrades, prompt changes, new tools and new integrations all change the attack surface, and an agent that passed its red team six months ago offers no guarantee about the one running in production today. Build a regression suite from prior successful attacks and run it against every material change, the same way a security team re-runs a penetration test's key findings after a patch rather than trusting the patch was sufficient on its own.
This is also where red-teaming earns its budget with executives and boards: a documented, repeatable process that can show what was tested, what was found, and what was fixed is a materially stronger answer to a board question about AI risk than an assurance that the team "tested it thoroughly" before launch.
Common pitfalls
The most common failure is running a red-team exercise once, at launch, and treating it as permanent assurance rather than a point-in-time result that decays with every subsequent change. A second is limiting scope to the chat interface, missing the indirect prompt injection that arrives through documents, emails or tool responses the agent reads without the same scrutiny applied to direct user input. A third is having the same team design and test an agent's defences, which reliably misses the attacks nobody on that team thought to imagine.
Programmes also under-invest in triage, generating long lists of adversarial findings with no consistent way to separate the ones that matter from the ones that are merely awkward, which leaves the genuinely dangerous findings competing for attention with cosmetic ones.
Red-teaming is the difference between an agent that looks safe in a demo and one that has actually been tested against the ways it will be attacked in production. Need support building an AI red-teaming capability for agents you're about to ship? Email sales@halfteck.com.