ORBiS TREE · ENGLISH DOCUMENT
Concepts →Latency and Throughput
Two core serving metrics describing response delay and the amount of work a system completes over time.
한국어
English
This English page is a curated translation layer linked to the Korean source article. Community changes are currently made on the Korean source, where the full revision history and anonymous edit trail are preserved.
Latency measures how long a request or generation step takes, while throughput measures how much work a system completes in a period of time. In AI serving, both matter and often trade off against one another.
How it works
Language-model systems may track time to first token, time per output token, requests per second, or tokens per second. Batching can improve throughput while increasing waiting time, and longer contexts can raise computational and memory costs.
Why it matters
Production systems should measure latency and throughput under realistic traffic, prompt lengths, output lengths, and hardware conditions rather than relying on one isolated benchmark number.
Related concepts
SOURCES
Sources
KNOWLEDGE LINKS