Red Teaming

Red teaming deliberately probes an AI system with challenging or adversarial inputs. The goal is to expose problems that normal benchmark testing or expected user behavior may miss.

Exercises can involve domain experts, internal teams, external testers, automated attack generation, or combinations of these methods. Findings may cover harmful outputs, prompt injection, privacy leakage, reliability failures, or tool-use risks.

Red teaming is most useful as part of an iterative safety process: findings should lead to mitigations, regression tests, and follow-up evaluation. It does not prove that a system is safe against all future attacks.

See also Safety Evaluation, System Card, and Model Card.

In a fictional AI document assistant, external material can contain an instruction to answer with unrelated text instead of summarizing. This tests whether source content is treated as data rather than task authority. Put a non-secret synthetic marker such as TEST-ONLY-42 in a forbidden test document to check access-boundary behavior. Use mock publication results instead of publishing real items. These are authorized isolated-test examples, not a reason to use real personal records or production secrets.

For an agent, define the requester's permitted task, expected result and forbidden actions before testing retrieval, writes and stopping. Correct answers on a normal capability test do not verify those boundaries. Inspect selected tools and transferred material as well as final wording.

In a fictional log of 60 attempts, twelve fail, but their causes form three unique defects: a missing access filter, external data accepted as instructions and duplicate saving. The observed 12 / 60 = 20% applies to that attempt set, not the probability of an incident in ordinary user traffic. Adaptively selected failure-seeking inputs are not a random sample of everyday requests.

Record attempt IDs, defect IDs, reproduction conditions and impact separately. Ten wording variants of one defect do not necessarily mean ten different defects; identical wording at different permission boundaries can reveal distinct issues. A low count does not reduce the importance of a consequential write-permission failure.

Suppose a fictional fix reduces failures on 60 retained cases to two. If those cases guided the mitigation, that result does not establish improvement on unseen inputs. If a separate 24-case set has two failures, report 2 / 24, approximately 8.3%, separately. Do not combine the cohorts into one probability for an assumed shared population.

After changing permissions or stop conditions, check that legitimate tasks still complete. Include tool timeouts, empty retrieval and expired permissions, preserving the last verified state. Link residual defects, deferral reasons, fix version and retest date to the system card.

The language-model red-teaming study ↗ explores generating failure-seeking inputs with another language model.^[1] The internal algorithmic-auditing framework ↗ connects lifecycle decisions with documentary evidence.^[2] An automated attack set or a report alone cannot establish the absence of all risk.

Read model information alongside retrieval, tool and filter configuration, identifying untested languages, inputs and settings. Report concrete reproductions and action status without disclosing unnecessary secrets or actual user records.