MMLU

MMLU is a benchmark designed to test knowledge and problem solving across a broad collection of subjects. It contains multiple-choice questions spanning areas such as mathematics, law, history, computer science, and medicine.

Models are evaluated by selecting answers under a defined prompting and scoring procedure. Reported scores can vary with prompt format, few-shot examples, model version, contamination, and evaluation implementation.

MMLU became a widely cited general capability benchmark, but it should not be treated as a complete measure of intelligence or product quality. Broader evaluation suites and task-specific tests remain necessary.

See also HELM, Benchmark Data Contamination, and Safety Evaluation.

The original MMLU study ↗ evaluates broad knowledge and problem solving across 57 tasks.^[1] Record dataset and model versions, included subjects and question counts, demonstration count, instructions, answer-selection or generation method and scoring rules. Evaluation without examples and evaluation with demonstrations have different input conditions.

Specify whether the reported aggregate averages subjects or pools all questions. For a Korean translation, preserve links between original and translated item IDs and verify terminology, options and answer keys. An original-language score does not establish Korean service quality. Include these conditions in the model documentation so readers can judge comparability.

These are two fictional subjects, not actual MMLU measurements. Subject A has 80 correct answers out of 100; subject B has nine out of ten. Their accuracies are 80% and 90%. Giving subjects equal weight yields macro accuracy (80 + 90) / 2 = 85%. Pooling questions yields micro accuracy 89 / 110, approximately 80.9%.

Identical predictions produce different aggregates because the weights and denominators differ. This example does not prescribe a default for every MMLU implementation. Check the implementation's aggregation rule, including when comparing with other evaluation metrics. Scores for small subjects can change substantially after only a few answer changes.

On a fictional shared set of 50 questions, both models answer 42 correctly, A alone answers three correctly, B alone answers four correctly and both miss one. A scores 45 / 50 = 90%; B scores 46 / 50 = 92%. B gains four items but loses three that A solved. A net gain of one question and two percentage points does not establish improvement in every subject.

Join before/after results by item ID, classify errors and report sample sizes and uncertainty. Repeatedly tuning instructions against the reported test set weakens interpretation. Separate configuration selection from final evaluation and inspect data overlap.

The MMLU-Pro study ↗ changes the benchmark through more challenging reasoning questions and a different answer-choice design.^[2] Different data, difficulty and scoring conditions mean that similarly named scores cannot simply be subtracted to prove regression. Report each benchmark version and setup separately.

Multiple-choice knowledge accuracy does not measure tool-task completion, source faithfulness or safe permissions by itself. Evaluate actual workplace questions, retrieval, tools and failure handling. A system card should distinguish model benchmarks from the behavior of the complete service.

The original MMLU paper ↗ evaluates knowledge and problem solving with four choices. MMLU-Pro ↗ adds harder reasoning-focused questions and expands to ten choices.^[1][2] Under the simplified assumption of one correct answer and uniform guessing, chance is 1/4=25% versus 1/10=10%. These are not actual model guessing distributions or ability scores. Questions, difficulty and content also change, so choice count alone cannot explain score differences.

This is an ORBiS Tree evaluation procedure. If a run requests A–D instead of answer text, predefine valid outputs, multiple-answer handling and abstention treatment. Do not retrospectively interpret 'A or C' as a convenient A. Comparing option probabilities and extracting an answer from free generation are different scoring paths. Preserve raw output, parsed answer and ground truth separately, linking adaptation choices to reported conditions.

In this illustrative test design, an originally correct B may move to D after a permutation. Update the answer mapping and compare selected content rather than whether the same letter persists. Record question ID, option text, permutation ID and the new answer mapping; separate disagreements between explanations and final choices. Do not pool variants as independent new questions. Separate settings-selection data from final evaluation and inspect repeated exposure.