Mixture of Experts (MoE)

Mixture of Experts, or MoE, increases total model capacity without necessarily activating every parameter for every token. A routing mechanism selects a small subset of expert networks to process each input.

Sparse MoE systems can therefore have many total parameters while using less compute per token than a similarly sized dense model. Training requires careful routing, load balancing, communication, and capacity management so that experts are used effectively.

MoE is important for scaling large models efficiently, but it introduces engineering complexity. Parameter count alone can also be misleading when comparing sparse and dense systems because only part of an MoE model may be active at once.

See also Large Language Model (LLM), Inference, and Latency and Throughput.

These are fictional dimensions, not a product specification. Eight experts with one million parameters each plus a shared three million, including the router, give 8×1 million + 3 million = 11 million total parameters. Selecting two experts for one token gives a simplified active count of 2×1 million + 3 million = 5 million. A batch can use more experts when its tokens take different routes.

If every weight resides in memory, 32-bit raw weights occupy 11 million×4 bytes = 44 million bytes, or 44 decimal MB. Counting only the active five million as 20 MB misses the other experts. Caches, activations, training state and runtime headroom are additional. Consider precision, the cache and batches when estimating actual inference memory.

In a separate fictional top-1 setup, four experts each accept two tokens. An eight-token batch routes five, one, one and one tokens to them. Total capacity is eight, but the first expert exceeds its capacity by 5 - 2 = 3 tokens. Free slots in other experts do not automatically resolve those assignments.

Check the implementation's overflow behavior, such as reassignment or skipping that expert computation. Skipping an expert calculation differs from deleting the request or token itself. The Switch Transformer study ↗ investigates one-expert routing, capacity and load balancing.^[1] The illustrative counts and policies above are neither its measured model settings nor universal MoE defaults.

Distributing experts across devices requires dispatching token representations and gathering results. Less expert computation can still encounter transfer overhead, waiting and inefficient small batches. Actual completion time and throughput need not improve proportionally. Equal total or active parameter counts do not imply equal runtime.

GShard ↗ connects conditional computation with distributed sharding and uses a top-2 routing configuration.^[2] Distinguish that from Switch's single selection. Record expert count, selected count, devices, precision, batch, input/output lengths, routing policy and quality. A speed ratio measured on different research hardware is not automatically a service's speed ratio.

Do not assume experts have predetermined human labels such as mathematics or Korean. Measure selection frequency and failures by language and task. Improved load balance and improved answers are separate findings. Identify which parts of the architecture use MoE.

A model card should specify total/active counting conditions and evaluation. A system card connects serving, communication and failure handling. Do not infer undisclosed expert counts from product names. Memory fit, actual completion time and Korean-task success together determine whether an architectural benefit helps the workload.