ORBiS TREE · ENGLISH DOCUMENT
Concepts →Attention
A mechanism that lets a neural network weigh the relevance of different positions when computing a representation.
한국어
English
This English page is a curated translation layer linked to the Korean source article. Community changes are currently made on the Korean source, where the full revision history and anonymous edit trail are preserved.
Attention computes how strongly one element of a sequence should use information from other elements. In Transformers, self-attention lets each token build a representation from other tokens in the same context.
How it works
A standard attention operation constructs query, key, and value representations. Similarity between queries and keys determines weights, and the weighted values are combined into the output. Multi-head attention performs this process in parallel across multiple learned projections.
Why it matters
Attention is central to the Transformer architecture and explains both its ability to connect distant context and much of its computational cost as sequence length grows.
Related concepts
SOURCES
Sources
-
Neural Machine Translation by Jointly Learning to Align and TranslateBahdanau et al. / arXivOpen source ↗
-
Attention Is All You NeedGoogle / NeurIPS / arXivOpen source ↗
KNOWLEDGE LINKS
Continue from here
BACKLINKS