ORBiS TREE · ENGLISH DOCUMENT
Concepts →Mixture of Experts (MoE)
A model architecture that routes each input to a subset of specialized parameter blocks called experts.
한국어
English
This English page is a curated translation layer linked to the Korean source article. Community changes are currently made on the Korean source, where the full revision history and anonymous edit trail are preserved.
Mixture of Experts, or MoE, increases total model capacity without necessarily activating every parameter for every token. A routing mechanism selects a small subset of expert networks to process each input.
How it works
Sparse MoE systems can therefore have many total parameters while using less compute per token than a similarly sized dense model. Training requires careful routing, load balancing, communication, and capacity management so that experts are used effectively.
Why it matters
MoE is important for scaling large models efficiently, but it introduces engineering complexity. Parameter count alone can also be misleading when comparing sparse and dense systems because only part of an MoE model may be active at once.
Related concepts
SOURCES
Sources
KNOWLEDGE LINKS