ORBiS TREE · ENGLISH DOCUMENT
Concepts →Tokenization
The process of converting text or other input into discrete units that a model can process.
한국어
English
This English page is a curated translation layer linked to the Korean source article. Community changes are currently made on the Korean source, where the full revision history and anonymous edit trail are preserved.
Tokenization converts raw text into units called tokens. A token may represent a whole word, part of a word, punctuation, bytes, or other learned units depending on the tokenizer.
How it works
Modern language models commonly use subword or byte-oriented tokenization so a finite vocabulary can represent a wide range of text. The tokenizer maps tokens to numeric IDs that are then converted into vectors before entering the model.
Why it matters
Tokenization affects context length, cost, multilingual behavior, and how efficiently different strings are represented. Character count and token count are therefore not interchangeable.
Related concepts
SOURCES
Sources
KNOWLEDGE LINKS
Continue from here
BACKLINKS