Knowledge Distillation

Knowledge distillation is a model-compression approach that transfers useful behavior from a larger model or ensemble to a smaller model, seeking lower computation cost while retaining performance. The original study describes using a teacher's soft output probabilities as training signals for a student.^[1]

A large neural network or foundation model can provide useful capabilities while being expensive to deploy. Distillation trains a smaller model to follow teacher output patterns and can reduce size, latency or serving cost. These outcomes require measurement on the intended task.

Quantization mainly reduces numerical precision. Knowledge distillation focuses on training a separate student. Their goals can overlap, and they can be combined rather than treated as mutually exclusive choices.

The original distillation study ↗ describes transferring a teacher's output distribution into student training.^[1] For two fictional classes, a hard label is (1, 0), while a teacher target might be (0.8, 0.2). This can convey relationships between alternatives without being a validated probability of factual correctness.

For fictional logits ln(4) and zero, softmax at temperature T = 1 gives (4/5, 1/5). Dividing the same logits by T = 2 gives exponentials two and one, hence (2/3, 1/3). Record temperature alongside loss and weighting, and distinguish training distributions from deployment decisions. Student training changes behavior differently from reducing numeric precision through quantization.

On a fictional 50-item binary task, both models are correct on 37 items, only the teacher on three, only the student on one and both give the same wrong answer on nine. Prediction agreement is (37 + 9) / 50 = 92%. Teacher accuracy is 40 / 50 = 80%; student accuracy is 38 / 50 = 76%.

The 46 agreements include nine shared errors. Agreement is not factual accuracy, and imitation success alone does not establish workplace quality. Evaluate important classes, Korean and English, long inputs and unsupported outputs separately. Distinguish inherited teacher errors from additional student errors.

Link original question and answer, teacher version, prompt, output and review status. Separate verified human labels from generated targets. Variants of the same prompt family in both training and evaluation can cause overlap and leakage. Do not approve every generated answer as an answer key.

Check the permitted data scope and usage conditions, using appropriately reviewed material rather than unnecessary sensitive records. Retain uncertainty when provenance is unknown. A teacher's name or public accessibility does not establish student quality or usage conditions. A model card should link provenance, review and residual errors.

DistilBERT ↗ combines distillation and several losses during BERT-family pretraining.^[2] Its reported compression, speed and evaluation results do not describe every student or current device. Architecture, training material and evaluation task affect outcomes.

Compare weights, cache and peak memory during execution, completion time and throughput, formatting and task-specific correctness under consistent conditions. If quantization follows distillation, evaluate the interventions separately to identify their effects. Retain deployed model, tokenizer, preprocessing and failure handling before deciding that a smaller model fits the job.