ORBiS TREE · ENGLISH DOCUMENT
Concepts →Knowledge Distillation
A training method in which a smaller student model learns from the outputs or representations of a larger teacher model.
한국어
English
This English page is a curated translation layer linked to the Korean source article. Community changes are currently made on the Korean source, where the full revision history and anonymous edit trail are preserved.
Knowledge distillation transfers behavior from a teacher model to a student model. The student is trained not only on original labels or data but also on information produced by the teacher, such as probability distributions or generated examples.
How it works
The objective is usually to retain useful capabilities while reducing size, latency, or serving cost. Distillation can be combined with architecture changes, synthetic data, quantization, or task-specific training.
Why it matters
Distillation is valuable when the best available model is too expensive for a target deployment. The student may still inherit weaknesses or biases from the teacher, so independent evaluation remains important.
Related concepts
SOURCES
Sources
-
Distilling the Knowledge in a Neural NetworkHinton et al. / arXivOpen source ↗
KNOWLEDGE LINKS