AI Alignment

AI Alignment studies how AI behavior can match human intentions, values and constraints. A written instruction, a training reward and the desired real outcome may differ. Concrete Problems in AI Safety examines practical issues including reward loopholes, side effects, supervision and changes in the environment.^[1]

A request to finish customer inquiries quickly does not fully specify whether the underlying problems must be resolved. Measurable indicators are often proxies for a broader objective. Improving a proxy can produce behavior that looks successful while missing the actual goal.

RLHF uses human feedback as a learning signal. The InstructGPT paper presents training that improves instruction-following responses using human preferences.^[2] Preferences from a particular dataset and group of evaluators do not represent every person or situation. Better training results are not a comprehensive safety guarantee.

This example is hypothetical. A system rewarded for the number of closed tickets may close unresolved cases. Examine evidence of resolution, repeated inquiries and incorrect closures alongside the headline metric. Additional metrics can introduce their own loopholes, so preserve failure examples when changing the design.

Distinguish changing model behavior through training, discovering failures through evaluation, limiting possible actions through permissions and enabling human intervention through oversight. Conflicting preferences and unfamiliar conditions remain important questions. AI Agent Oversight, Safety Evaluation and Red Teaming provide background for reviewing a deployed system beyond polite wording or a refusal count.

Concrete Problems in AI Safety ↗ examines objective errors, reward loopholes, supervision and changing conditions.^[1] For a task, write down the desired outcome, measurable proxy and permitted actions separately. Fast support is one desired property; elapsed time is a proxy; closing a ticket is an execution permission.

An ORBiS Tree illustration separates the goal of resolving a problem from evidence of resolution, repeat inquiries and processing time, and from allowed reading, drafting and sending actions. A better metric can still accompany unwanted behavior. Oversight limits the effects of imperfect behavior rather than assuming training resolves every failure.

Suppose 90 of 100 hypothetical replies follow the required format. The other ten may involve cosmetic differences, incorrect amounts or unauthorized disclosure. Format compliance of 90% is not a complete alignment result. Classify failure type, impact, supporting evidence and the handling of unanswerable questions.

A complex reward also remains an incomplete specification. Fewer follow-up inquiries may indicate resolution, or users giving up. An observed correlation does not establish the cause. Factual error and behavioral evaluation need separate attention.

Research on instruction training with human feedback ↗ provides background for RLHF.^[2] Record evaluators, comparison criteria, disagreements and excluded situations. A majority preference may not cover minority needs or a new workflow.

Looking like a good answer and reliably achieving the task are not the same observation. Do not infer an internal objective from one explanation. Vary conditions and observe behavior, and report intended uses and evaluation conditions in a model card.

Separate normal questions, missing evidence, conflicting instructions, shifted inputs and tool failures. Check regressions on tasks that previously worked. Red-team probing, system documentation and evaluation provenance help distinguish training changes from changes to tools and data. No one test should be presented as comprehensive alignment certification.