Quantization

Quantization reduces the precision used to store or compute model values, for example moving from higher-precision floating point to lower-bit integer or floating-point formats.

Lower precision can reduce memory footprint, bandwidth, and inference cost. Methods differ in whether they quantize weights, activations, or both, and whether quantization is applied after training or incorporated during training.

Quantization is important for deploying large models on constrained hardware and for improving throughput, but aggressive compression can reduce accuracy or create task-specific regressions that must be measured.

See also Inference, GPU, LoRA, and Latency and Throughput.

This simplified calculation assumes a fictional model with exactly seven billion weights. At 16 bits, or two bytes, per weight, raw storage is 14,000,000,000 bytes = 14GB. Tightly packed four-bit weights occupy 3,500,000,000 bytes = 3.5GB. GB here is the decimal unit of one billion bytes, not GiB. Comparing raw bit counts does not establish that 3.5GB of device memory can run the model.

Scales, zero points, alignment, remaining higher-precision values, KV cache, activations, workspaces and runtime overhead can add memory. Four-bit weights do not imply that every operation, activation or KV-cache entry also uses four bits. Separate loading successfully from serving the intended concurrency within device memory.

A fictional evaluation has 100 questions: 80 short and 20 long. The baseline answers 77 short and 18 long correctly, totaling 95 / 100 = 95%. A quantized model answers 79 short and 12 long correctly, totaling 91%. Aggregate accuracy drops four percentage points, while long-question accuracy falls from 18 / 20 = 90% to 12 / 20 = 60%, a 30-percentage-point loss.

These figures do not describe a real quantization method. They illustrate how aggregates can hide a subgroup regression. Separate Korean, code, long-input, negation and exception cases, considering actual workload shares and error impact. Hold input length and questions fixed when comparing compression effects.

Calibration material should represent the intended input distribution, but data used to select settings should not simply be reused as final performance evidence. Record the original model version, algorithm, bit width and grouping conditions, calibration-data version, tokenizer and runtime. Source records help identify omitted languages or workloads.

Use the same questions and generation settings to check correctness, exceptions and citations first, then measure first-token time, p95 latency, throughput and memory. Lower bit width is not automatically faster: kernel support, conversion overhead, hardware and concurrency affect the observed result. Keep the original or previous verified bundle as a recovery point.

GPTQ ↗ proposes post-training low-bit weight quantization for generative Transformers.^[1] AWQ ↗ studies activation-aware low-bit weight quantization.^[2] Check each study's model, data and hardware conditions instead of transferring its results into guarantees for the deployed service.

For out-of-memory failures during inference, inspect context length, cache and concurrent requests as well as weights. For quality failures, compare the same original model and evaluation conditions, checking whether errors concentrate in particular languages or long inputs. Model file size alone is not a deployment-quality measure.