Transformer

The Transformer was introduced in the 2017 paper “Attention Is All You Need.” It replaced recurrent sequence processing with attention-centered blocks, making it easier to train models in parallel and to capture relationships between tokens over long ranges.^[1]

Transformer blocks combine attention, feed-forward layers, residual connections, and normalization. Encoder-style models such as BERT focus on representation learning, while decoder-style models such as GPT generate tokens autoregressively. Many systems use variations of these original designs.

The architecture became a core building block of modern language models and later expanded into vision, audio, and multimodal systems. Understanding Transformers helps explain concepts such as Attention, KV Cache, and Context Window.

See also Attention, BERT, GPT, and Large Language Model (LLM).

The original Transformer study ↗ combines attention with position-wise feed-forward layers, residual connections and normalization.^[1] The following dimensions are teaching assumptions, not the paper's model specification. Six tokens with representation width eight form a 6×8 input. If two heads each project Q, K and V to width four, each matrix is 6×4, QK^T scores are 6×6 and the weighted output is 6×4.

Concatenating the two outputs gives 6×8 before output projection. The token count remains six while representations are mixed. The feed-forward layer after attention separately transforms each position. Distinguish this calculation from the input representation and token splitting to identify shape errors separately from semantic errors.

An encoder reads positions in its supplied input. The original decoder reads permitted previous output positions and encoder outputs. BERT representation tasks differ from GPT next-token generation. The architecture name alone does not specify every variant's masks or output behavior.

During training, the known target sequence can be shifted by one position to supply inputs, while a causal mask blocks future targets. During generation, the next token does not yet exist, so later steps depend on earlier generated results. Parallel position-wise training does not mean every generated token is determined simultaneously. A cache of earlier keys and values does not reveal future answers.

For one fictional head using standard full self-attention, length 16 gives 16² = 256 score positions; length 32 gives 32² = 1,024. With the same layer and head counts, potential score positions quadruple. This does not establish four times the actual price or total runtime. Masks, projection and feed-forward work, memory access, batches and implementation also affect cost.

FlashAttention ↗ proposes exact attention that reduces reads and writes across memory levels.^[3] An optimized implementation need not store every complete score matrix in large device memory. Measure full-input processing separately from cached token-by-token generation and record actual latency and throughput.

Record tokenizer, input length, masks, positional scheme, model version, precision, hardware, concurrency and output length. Identical text can consume different token counts within a context budget. Verify that translated questions preserve meaning and answer keys, and retain separate Korean and English results.

Check whether a faster implementation preserves task success and uses evidence correctly in long inputs. An architecture description does not establish present product performance. Linking configuration and evaluation coverage in a model card replaces name-based comparisons with evidence about the actual workload.