Tokenization

Tokenization converts raw text into units called tokens. A token may represent a whole word, part of a word, punctuation, bytes, or other learned units depending on the tokenizer.

Modern language models commonly use subword or byte-oriented tokenization so a finite vocabulary can represent a wide range of text. The tokenizer maps tokens to numeric IDs that are then converted into vectors before entering the model.

Tokenization affects context length, cost, multilingual behavior, and how efficiently different strings are represented. Character count and token count are therefore not interchangeable.

See also Embedding, Context Window, and Large Language Model (LLM).

The string formed by Korean syllables U+C11C and U+C6B8, a space, A and I contains five Unicode code points and occupies nine UTF-8 bytes. Each Korean syllable occupies three bytes; the space, A and I occupy one each. These counts do not determine the number of model tokens. That requires the chosen tokenizer's vocabulary, normalization and segmentation rules.

Visually similar accented text can also have different encodings: precomposed é, U+00E9, is one code point and two UTF-8 bytes; e followed by combining accent U+0301 is two code points and three bytes. These are encoding examples, not token measurements for a particular model. Distinguish displayed characters, code points, bytes and tokens, and use measured tokens for the context budget.

Record the model version, tokenizer files and vocabulary, special tokens and message format as one deployment bundle. The same integer ID can represent different text in another vocabulary. Do not substitute a similarly named tokenizer or forward token IDs unchanged to a different model.

Check whether roles, separators and tool definitions contribute to the actual input. Counting one plain-text fragment may differ from counting the complete request. Length budgeting and provider billing or reasoning-token accounting are separate conditions; a fixed character multiplier does not establish API cost. Keep matching input-processing versions when generating document embeddings too.

A fictional AI product guide says internet access is required, except that a separately installed local mode works offline. Removing the exception can change the answer. Shorten input at meaningful sentence or document boundaries while retaining conditions and original evidence locations. Check tokenizer and string-decoding behavior instead of blindly cutting at a character or byte position.

A hypothetical allocation uses 200 instruction tokens, 500 conversation tokens, 300 source tokens and 200 reserved output tokens: 1,200 total. Adding 150 source tokens makes 1,350. Reserved output is not actual generated output, and this illustration states neither a model limit nor a billing formula. Verify retained negation and exceptions alongside total length.

Test equivalent Korean and English text, product identifier X-20, line breaks, code indentation, combining characters and emoji separately. Compare encoded-then-decoded text with the original, first documenting any intentional normalization. Not every tokenizer reproduces original bytes unchanged. Compare input length and response timing under matched input conditions.

SentencePiece ↗ describes learning subword models from raw sentences.^[1] The subword translation study ↗ represents rare words as sequences of smaller units.^[2] Its specific translation results do not establish efficiency for every current Korean-language model.