ORBiS TREE · ENGLISH DOCUMENT
Research →Safety Evaluation
Testing designed to measure risks, harmful behaviors, and the effectiveness of safeguards in an AI system.
한국어
English
This English page is a curated translation layer linked to the Korean source article. Community changes are currently made on the Korean source, where the full revision history and anonymous edit trail are preserved.
Overview
Safety evaluation systematically examines AI behaviors or capabilities connected to deployment risk. It can define and test misuse, deception, cyber capabilities and autonomous behavior separately from general knowledge or accuracy.[1]
An example of dangerous-capability evaluation
Google DeepMind's research describes evaluations of persuasion and deception, cyber capabilities, self-proliferation and self-reasoning, and applies them to Gemini-family models.[1] Observing an enabling capability is not a prediction that a model will cause harm; it helps identify capabilities and trends that warrant scrutiny.
Relationship to other evaluations
MMLU and HELM cover general capabilities or multiple quality dimensions. Safety evaluation defines risk scenarios and failure conditions. Red teaming can help discover unanticipated failures. A system card records scope, findings and residual limits.
Model capabilities and system risk
Capability on a task differs from harmful outcomes in a service. Tools, permissions, users and environment change the system under evaluation. NIST AI RMF emphasizes context, system components, deployment-relevant conditions and continuing risk measurement.[2] An agent requires inspection of actions and oversight as well as answers.
Questions for reading evaluation results
These questions restate context, measurement and operational-management principles rather than define a certification checklist.[2]
- Target and conditions: what model/system version, data, tools and environment were tested?
- Judgment: which failures, methods and metrics were used?
- Uncertainty: where do findings apply, and which risks remain untested?
- Follow-up: what mitigation, retesting and operational monitoring followed?
A score without conditions cannot readily establish performance in a deployment. A model card explains model purpose and limits; a system card links the combined service and protections.
Connect safety claims to evidence
A safety case connects a claim about managed risk in a particular setting to supporting evidence. AISI discusses evidence about safeguards, red-team observations, deployment environments and organizations alongside experiments.[3] Finding no failure in one red-team exercise does not prove the absence of every risk.
For a tool-using assistant, document the permissions and oversight under which an authorized-task boundary was observed and how changed conditions limit that claim. This is an illustration of evaluation principles, not a claim that a product passed.
External-assessment discussion in September 2026
On September 22, 2026, OpenAI published proposed third-party-assessment priorities and principles. They discuss safety cases, safeguards, dangerous-capability/alignment evaluation and major alignment incidents, together with clear scope, access, methods, uncertainty, conflicts of interest and security responsibilities.[4]
External participation, independence and adequate coverage require separate examination. Check the developer–assessor relationship, exclusions and available evidence. This publication states a company's proposals rather than comprehensive safety certification for a model.[4] Related organizational examples appear in OpenAI and Anthropic, and evaluation conditions for model announcements are discussed in Claude.
Define permitted work and failure events first
Suppose a fictional document assistant may save private drafts but may not publish without approval. Define authorized users, material, tools and expected outcomes first. Use mock publication tools and a non-secret marker such as NOTE-TEST-1. Distinguish attempted, blocked and completed actions rather than inferring permission boundaries from the final wording alone.
The dangerous-capability study ↗ investigates abilities that can enable risk.[1] An observed capability differs from harm in a particular service. Link permissions and oversight to the system configuration. Authorized isolated tests need not use actual secrets or unnecessary user records.
Worked example: refusal versus legitimate-request blocking
A fictional binary policy test has twenty disallowed requests: sixteen blocked and four missed. Among eighty legitimate requests, eight are wrongly blocked and 72 allowed. Total refusal is 24 / 100 = 24%. Disallowed-request blocking is 16 / 20 = 80%, legitimate-request false blocking is 8 / 80 = 10%, and agreement with the defined policy is (16 + 72) / 100 = 88%.
More refusals alone do not establish improved safety and usefulness. Connect failure definitions, denominators, impact and legitimate completion to the context, measurement and management addressed by NIST AI RMF ↗.[2] These request proportions and rules are illustrative, not actual traffic or a product safety score.
Why zero failures does not mean zero failure probability
For 100 fictional independent trials sharing failure probability p, the probability of observing no failures is (1 - p)^100. Setting it to 0.05 gives p = 1 - 0.05^(1/100), approximately 2.95%. This illustrates the one-sided 95% exact binomial upper confidence bound after zero failures under those assumptions. The NIST statistical guide ↗ describes proportion intervals and binomial methods for small failure counts.[5]
It is neither an acceptable failure threshold nor an estimate of actual incident probability. A frequentist confidence interval is not a posterior statement that p lies within it with probability 95%. Adaptively chosen attacks, correlated tests and changed input distributions may violate the assumptions. Retain sample, selection, judgment rules and uncertainty.
Link claims and configuration after fixes or deployment
When a model, instructions, retrieval material, access filter or write tool changes, identify affected risk claims and evidence. Separate regression replay from fresh inputs, and check excessive blocking of legitimate requests. Disclose excluded languages, permissions, tools and residual failures.
Link model information with the service configuration, test date, dataset version, mitigation and retest results. A safety case connects a context-specific claim to evidence rather than conferring certification through a document. Old passes do not automatically cover new configurations. Reevaluate general quality, risk and legitimate-task completion separately.
SOURCES
Sources
-
Evaluating Frontier Models for Dangerous CapabilitiesGoogle DeepMind / arXivOpen source ↗
- Open source ↗
-
Safety cases at AISIAI Security InstituteOpen source ↗
- Open source ↗
-
Confidence intervals for proportionsNIST/SEMATECH e-HandbookOpen source ↗
KNOWLEDGE LINKS
Continue from here
BACKLINKS
Articles that point here
AI Agent ConceptsAI Agent Oversight ConceptsAI Alignment ConceptsAnthropic OrganizationsBenchmark Data Contamination ResearchClaude ProductsDatasheets for Datasets ResearchFine-tuning ConceptsGPT ModelsGPT-6 Astra ModelsGoogle DeepMind OrganizationsHELM ResearchHallucination in AI ResearchMMLU ResearchModel Card ResearchMultimodal AI ConceptsOpenAI OrganizationsPhysical AI ConceptsRed Teaming ResearchRLHF ConceptsSystem Card ResearchVoice AI Concepts