What is tokenisation?

02 · See it in action

Follow the steps

1. Write examples containing English, Hindi, emoji and source code.
2. Compare a simple whitespace split with the tokenizer for your chosen model.
3. Record which strings create surprisingly many tokens.
4. Estimate your input budget using that tokenizer before sending large documents.

Worked example

"AI is useful".split() -> ["AI", "is", "useful"]
This demonstrates whitespace splitting only.
A model tokenizer may split "usefulness" into several pieces.

Read the example once, then explain each part aloud in your own words. If it includes Python, you can understand the idea first and run the code later in a local Python environment. Code is shown as text; this page does not execute it.

Sign in to save your reading progress.