01 · Understand the idea
In simple words
A language model does not simply receive text as one giant sentence. A tokenizer splits it into units called tokens. Think of dividing a chocolate bar into pieces before counting it, except token pieces do not have a fixed size. A short word might be one piece; another word might use several.
Two useful terms
Token: one unit processed by a model. Tokenizer: the method that splits text into those units.
Build your understanding
Tokenisation converts text into smaller units a model can process. Depending on the tokenizer, a unit may be a word, part of a word, punctuation or bytes. Token counts are model-specific. A whitespace split is an educational baseline, not an accurate substitute for a language model tokenizer.
Sign in to save your reading progress.
Sign in