ORBiS Tree
KO

ORBiS TREE · ENGLISH DOCUMENT

Concepts →

Inference

The process of running a trained model on new input to produce predictions, embeddings, or generated output.

한국어 English
#Deployment#Inference#LLM#Serving
Korean source Edit source
English updated 10월 10, 2026 Source article updated 10월 10, 2026 4 sources
This English page is a curated translation layer linked to the Korean source article. Community changes are currently made on the Korean source, where the full revision history and anonymous edit trail are preserved.

Overview

Inference executes a trained model on new inputs to compute predictions or generated outputs. Training changes model parameters; inference normally serves requests with fixed parameters.

Serving and LLM generation

Production serving combines request scheduling, memory management, latency and throughput. MLPerf Inference defines benchmark scenarios and conditions for comparing model execution.[1] In autoregressive LLMs, prefill processes the supplied prompt and decoding generates subsequent tokens. Input and output lengths therefore affect different parts of the timeline.

Batching, memory and hardware

Batching can raise throughput but also changes memory requirements and individual waiting time. vLLM's PagedAttention study addresses dynamically changing KV-cache allocation and sharing.[2] Its measured gains are conditional on the tested workloads, rather than guaranteed gains on every GPU. Hardware selection includes operations, memory, software support and batch conditions. Cloud TPU documentation explains workloads suited to its execution model.[3] A hardware name or peak operation count alone does not establish request cost.

Diagnose an apparently slow response

In a hypothetical example, a long document delays the first visible output while later text arrives quickly. Measure queueing, retrieval, input processing, generation and delivery separately to identify the actual cause. Record first-token time, subsequent token rate, total completion time and success rate. Compare different input lengths, output lengths and concurrency before applying an intervention.

Cost per successful task

This is illustrative arithmetic, not pricing. Method A spends 100 cost units on 100 requests and completes 80 tasks to the acceptance criteria: 100÷80 = 1.25 units per successful task. Method B spends 120 units and completes 100: 1.2 units. A has a lower per-request cost, but B has the lower cost per successful task. State whether retrieval, retries, tools and human review are included.

Computation caching and stored state

Prompt caching can reuse computation for matching input prefixes under supported conditions.[4] Verify actual hits and relevant model conditions instead of assigning one saving percentage to every request. Caching does not correct source errors or guarantee truthful answers. KV Cache covers computation state, while AI Memory covers retained user or application state.

Worked example: calculate first-token and total time

These are hypothetical timings, not a service measurement. A request waits 0.2 seconds, then takes 0.8 seconds from input processing to the first token. Time to first token is 1.0 second from request start. Another 99 tokens delivered at 50 tokens per second take 1.98 seconds, giving 2.98 seconds for 100 tokens. Dividing all tokens by total elapsed time gives about 33.6 tokens per second, a different measure from the post-first-token rate of 50.
Define timestamps before comparing speed. Add retrieval and delivery intervals to the user-facing timeline where applicable. Record per-request times and completed concurrent work with latency and throughput, and input tokens and material with context conditions.

Worked example: estimate KV-cache memory

A hypothetical model has 32 layers, eight KV heads per layer, head dimension 128, cached length 4,096 tokens and two bytes per value. Storing K and V gives the simplified estimate 2×32×8×128×4,096×2 = 536,870,912 bytes, or 512 MiB per request. Eight equally long requests sum to 4 GiB. This is not a model specification and excludes weights, activations, metadata, alignment and other runtime costs.
Actual representation, sharing and allocation depend on the engine. The PagedAttention paper ↗ addresses waste and sharing in dynamically changing KV caches.[2] Do not automatically substitute the number of query heads for KV heads. Check cache structure and storage precision, then compare with engine measurements.

Match the intervention to the bottleneck

  • Growing queue time: inspect arrival rates, concurrency and batch waiting.
  • Slow first tokens only on long inputs: inspect source selection and prefill.
  • Long generation: inspect required output length and decoding.
  • Repeated memory exhaustion: separate weights, KV cache and concurrency.
  • Slow tools: check whether a model optimization changes the tool interval at all.
Device utilization alone does not identify the cause. Change one condition at a time. A hardware comparison should retain engine, precision and software versions. Recheck missing content and failures after a speed improvement.

A cache hit is not a completed task

The prompt-caching guide ↗ describes reuse for identical input prefixes.[4] Compare document versions, tool definitions and input ordering, then record actual hits. A computation cache does not permanently store user preferences or automatically refresh source information. Separate its purpose and lifetime from user state.
For a task with tool execution, add model, retrieval, retry and review costs, then divide by tasks meeting the same completion criteria. Call prices and cache-hit rates do not replace the total cost per completed task.

SOURCES

Sources

  1. MLPerf Inference BenchmarkMLCommons / arXiv
    Open source ↗
  2. Efficient Memory Management for Large Language Model Serving with PagedAttentionUC Berkeley / arXiv
    Open source ↗
  3. Introduction to Cloud TPUGoogle Cloud
    Open source ↗
  4. Prompt cachingOpenAI
    Open source ↗

KNOWLEDGE LINKS

AI Agent Oversight → AI Memory → Context Window → GPU → KV Cache → Latency and Throughput → Quantization → Tensor Processing Unit (TPU) → Large Language Model (LLM) →

BACKLINKS

AI Agent ConceptsContext Window ConceptsGPT ModelsGPT-6 Astra ModelsGPU ConceptsHELM ResearchKV Cache ConceptsKnowledge Distillation ConceptsLatency and Throughput ConceptsMistral AI OrganizationsMixture of Experts (MoE) ConceptsMultimodal AI ConceptsOpenAI OrganizationsProject Suncatcher InfrastructureQuantization ConceptsSovereign AI ConceptsTensor Processing Unit (TPU) InfrastructureTokenization ConceptsTransformer Concepts

KEEP EXPLORING

Related articles

Vector SearchSearch that retrieves items by similarity between numerical vector representations.EmbeddingA numerical vector representation used to encode semantic or structural properties of data.Fine-tuningAdditional training that adapts a pre-trained model to a narrower task, domain, or behavior.Retrieval-Augmented Generation (RAG)A method that retrieves external information and supplies it to a generative model as context.Knowledge DistillationA training method in which a smaller student model learns from the outputs or representations of a larger teacher model.Diffusion ModelA generative model that learns to reverse a gradual noise process to create data such as images.
ORBiS Tree English · source-linked translation
Korean source & revision history · Report an issue · Editorial policy