GPU

A GPU (Graphics Processing Unit) is a processor designed to perform many operations in parallel. It developed from graphics processing and is used to accelerate deep-learning training and inference, including parallel matrix and vector computation.

NVIDIA CUDA is a parallel-computing platform and programming model for using GPU resources for general-purpose computation.^[1] Its programming guide describes thread groups, memory hierarchies and synchronization.^[1]

The name GPU alone does not establish LLM performance. Available memory, bandwidth, numerical precision, model architecture, batch size and software stack all affect execution. Hardware comparisons need the model and measurement conditions.

This is a fictional memory budget, not a GPU specification. Seven billion parameters at 16-bit precision occupy 7 billion×2 bytes = 14 billion bytes, or 14 decimal GB of raw weights. Assume another 4 GB for caches and 2 GB for other runtime memory. The total is 20 GB. A fictional device with 16 GB available cannot be declared suitable from weight size alone.

Distinguish GB from GiB, reserved memory from available capacity, and peak use during loading, input processing and generation. Reduced precision can change weight storage without reducing every cache or intermediate by the same ratio. The cache depends on context and concurrent requests. Check device/host memory and transfers described in the CUDA guide ↗.^[1]

A fictional non-overlapping request spends 40 ms preprocessing, 10 ms transferring data and 50 ms computing on the GPU: 100 ms total. Reducing compute to 25 ms yields 40 + 10 + 25 = 75 ms. Overall speedup is 100 / 75, approximately 1.33-fold. A twofold compute improvement is not a twofold request improvement.

Real pipelines can overlap transfers and computation, so verify this summation assumption. Separate waiting, preparation, transfer, computation and postprocessing. Distinguish first-output time from completion. High utilization need not mean more correctly completed tasks. Keep model, input/output length, concurrency and quality conditions comparable.

The NVIDIA performance guide ↗ compares arithmetic intensity, operations per accessed byte, with the ratio of compute capability to bandwidth.^[2] For a fictional 800 GFLOP/s compute ceiling and 80 GB/s bandwidth, the boundary is 10 FLOP/byte. Intensity five gives a simplified ceiling min(800, 80×5) = 400 GFLOP/s; intensity twenty gives min(800, 80×20) = 800 GFLOP/s.

These are illustrative ceilings under fixed numeric-format and memory-access assumptions, not measured attainable performance. Small workloads, insufficient parallelism, repeated reads and kernel overhead can invalidate the simple estimate. Changing model operations or batches can change the bottleneck on the same GPU.

Record device/software versions, precision, batch, token counts, cache state, warmup, repetitions and treatment of failed requests. Check memory fit, completion within the task deadline and preserved quality separately. Advertised peak operations do not directly establish inference throughput.

Short classification and long sequential generation can encounter different limits on the same device. Mark unmeasured properties as unknown and link the deployed model, cache and serving configuration in a system card. Capacity, bandwidth, compute and completed-work outcomes serve different comparison purposes.