HELM

HELM is an evaluation framework developed to make language-model comparisons more systematic across tasks, metrics, and deployment concerns. It emphasizes that a single accuracy number does not capture all relevant behavior.

The framework evaluates models across scenarios and can consider dimensions such as accuracy, calibration, robustness, fairness, efficiency, and other properties depending on the release and configuration.

HELM is useful as an example of multi-dimensional evaluation. It reinforces the need to document prompts, model versions, datasets, and metrics when comparing general-purpose models.

See also MMLU, Safety Evaluation, Model Card, and Benchmark Data Contamination.

The HELM study ↗ proposes multi-dimensional evaluation that exposes trade-offs across scenarios and metrics, including accuracy, calibration, robustness, fairness, bias, toxicity and efficiency.^[1] Distinguish measured dimensions from missing ones. An empty cell is unmeasured or undisclosed coverage, not a favorable result.

Two models with similar knowledge scores can behave differently on Korean document questions, long inputs, tools and latency or throughput. Define the purpose and required criteria before adding incompatible numbers into an unexplained overall capability score.

In a fictional classification example, one bin contains 100 predictions, each assigning probability 0.9 to its selected answer; 70 are correct. Another bin contains 100 predictions, each assigning probability 0.6; 60 are correct. The equally weighted calibration gap is 0.5 × |0.9 - 0.7| + 0.5 × |0.6 - 0.6| = 0.10. For these bins, the mean absolute gap is ten percentage points. Overall accuracy is separately 130 / 200 = 65%.

These probabilities are assumed to be numerical scores for the evaluated choices. A sentence saying 90% confident is not automatically an equivalent probability. The neural-network calibration study ↗ examines how predicted probabilities relate to actual correctness.^[2] Calibration depends on the data distribution and binning; it does not guarantee the factual correctness of every generated answer.

Assume a fictional workload using the same 100 requests, hardware and concurrency. Model A has 91% accuracy and a request-to-completed-response p95 of two seconds; B has 94% accuracy and p95 of six seconds. If this workload requires at least 90% accuracy and p95 no greater than three seconds, A meets both criteria while B misses the timing criterion. These are neither universal recommended thresholds nor product measurements.

Separate response completion from first-token timing. Record input/output lengths, concurrency, cost and treatment of errors or refusals. Counting fast incorrect answers as useful throughput distorts the comparison. Repeat matched measurements when the inference configuration changes.

Test equivalent Korean and English wording, whitespace changes, negation, exceptions and evidence position separately. Verify that meaning and the correct answer remain unchanged before treating a changed result as a robustness failure. Retain sample sizes and the distribution of failed cases.

Link model, dataset and metric conditions through a model card, and retrieval, tools and filters through a system card. If safety coverage is absent, general performance metrics do not fill that gap.

The HELM paper ↗, figure 5, section 2 and section 7, describes evaluation through a scenario, a model/adaptation process and metrics.^[1] The same model name with different demonstrations, input formatting or output transformations defines a different run. Prompting adaptation here does not automatically mean changing model weights. Preserve actual prompts and scoring paths with model versions. The studied scope does not guarantee performance across all current models, languages and tools.

This is an ORBiS Tree fictional operational ledger, not an official HELM metric definition. Of 100 planned requests, 90 return, 80 returned outputs are valid and 60 valid outputs succeed. Response availability is 90/100=90%; validity among responses is 80/90≈88.9%; success among valid outputs is 60/80=75%; total request success is 60/100=60%. Reporting 75% as total success drops failures. Separate transport, format and content errors, recording retry time and cost.

This is an ORBiS Tree test design. Rephrasing 'the authentication token is valid' without changing meaning can test whether an AI tool's permission judgment stays stable. Changing it to 'the token has expired' may require a different judgment. Repeating the same answer then does not demonstrate robustness. Define meaning and expected outcomes before testing; report equivalent rephrasing separately from changed conditions. Connect permission and safety tests with the actual system configuration, separating model responses from tool execution.