Latency and Throughput

Latency is time for an individual request or step. Throughput is work completed over a time interval. Both depend on workload and may trade off.

LLM services distinguish first-token time, decoding intervals, total completion and concurrent work. Batching changes scheduling, waiting and memory requirements. MLPerf uses scenarios and accuracy conditions when comparing serving performance.^[1] PagedAttention highlights batching and cache management as LLM serving concerns.^[2]

Record realistic arrival patterns, input/output lengths, hardware, software and accepted quality. An isolated fast request and a high-volume workload answer different questions. Include failed and interrupted work when judging the user experience.

In a fictional sample of 100 completed requests, 90 take one second and ten take ten seconds. The mean is (90×1+10×10)/100 = 1.9 seconds. Using the nearest-rank method, the 95th sorted value gives a p95 of ten seconds. A mean can hide a minority's long waits. State the percentile method and sample size; these are not service measurements.

Keeping only completed requests can omit timeouts and failures. In a separate hypothetical one-minute interval, 92 of 100 responses meet acceptance criteria: useful completion throughput is 92/60 ≈ 1.53 tasks per second. Counting every response gives 100/60 ≈ 1.67, a different measure. Use the same completion criteria for cost per successful task.

Browser request to first display, server receipt to first generation, and last-token generation to delivery are distinct intervals. Check engine definitions in vLLM's operational metrics ↗.^[3] Server figures do not automatically represent user-facing latency. Record excluded retrieval, network and tool intervals.

For voice tasks, consider end of utterance, first playback and stopping after interruption. For tool tasks, distinguish a generated answer from confirmed execution. Choose the endpoint for the actual goal first.

The MLPerf Datacenter guide ↗ provides scenario- and quality-conditioned comparison context.^[1] Retain lengths, arrivals, concurrency, hardware, engine, precision and cache state. Higher batching throughput can coexist with excessive first-response or tail waits.

Do not generalize short-input results to long documents. Compare means, upper percentiles, timeouts and useful completions on the same sources and quality. Include retries and review within the same cost boundary. Peak throughput alone cannot determine an operating configuration.