BERT

BERT is a Transformer encoder model introduced by Google researchers in 2018. It demonstrated that deeply bidirectional pre-training could produce strong language representations that transfer effectively to many natural-language understanding tasks.

BERT is trained to build contextual token representations using surrounding text rather than generating text one token at a time like a typical autoregressive decoder. Fine-tuned BERT variants became widely used for classification, question answering, retrieval, and information extraction.

BERT helped establish pre-training plus task adaptation as a dominant NLP pattern and remains a reference point for understanding encoder models, embeddings, and transfer learning.

See also Transformer, Embedding, Fine-tuning, and Tokenization.

The original BERT paper ↗ selects 15% of WordPiece positions for prediction. Among selected positions, it uses [MASK] with probability 80%, a random token with probability 10% and the unchanged token with probability 10%.^[1] This differs from replacing every selected position with [MASK].

Applying these proportions to 1,000 eligible positions gives illustrative expectations of 150 prediction targets, 120 [MASK] replacements, 15 random replacements and 15 unchanged targets. One sample need not have exactly these counts; inspect implementation-specific rounding and selection. The target remains the original token. Token counts differ from character or word counts. Original BERT's MLM and NSP objectives do not describe every later variant.

A fictional inquiry classifier can form input representations and learn an output layer for fixed labels such as shipping, returns and other. Token labeling predicts a label at each position, while extractive question answering identifies an answer's start and end within supplied text. The original study evaluates task-specific output layers through fine-tuning.

These tasks have different output conditions from free-form autoregressive generation. Asking old weights to supply rules that change daily raises a separate evidence problem. Consider retrieval of current material or structured rules. Specify input, labels, failure handling and workplace success criteria before selecting by model-family name.

This is a fictional binary classifier, not a BERT product benchmark. Of 20 return inquiries, it identifies twelve and misses eight. Of 80 other inquiries, it correctly rejects 76 and incorrectly labels four as returns. Overall accuracy is (12 + 76) / 100 = 88%. Return recall is 12 / 20 = 60%. Precision among the 16 predicted returns is 12 / 16 = 75%.

Different denominators mean high overall accuracy does not establish good coverage of important inquiries. When changing a decision threshold, inspect misses alongside false positives and separate configuration-selection data from final evaluation. Link other evaluation metrics and retain class counts, precision and recall.

Record model and tokenizer versions, the verified scope of Korean training material, input limits, label definitions, duplicate handling and document- or author-level splits. Paraphrases of the same inquiry in training and evaluation can inflate results. Inspect data overlap and language-specific failures; an English research score is not a Korean workplace score.

A model card should describe intended use, evaluation conditions and residual errors. A system card links preprocessing, rules and review paths. Tokenizer changes and truncation can change behavior even when the model file remains unchanged. Retest negation, exceptions and long inquiries as well as short examples.