03 · Practise and check
Try this yourself
Compare "hello", "Hello!" and "नमस्ते" with the same tokenizer. Explain why character length alone does not determine token count.
Check your understanding
The same text can produce different token counts for different tokenizers. You must specify the tokenizer when reporting a count.
A mistake to avoid
Do not assume one token equals one word, especially for multilingual text, URLs and code.
Before looking at the answer, write a short explanation and one example of your own. Finish by naming one limitation of the method. Use Mark as read when you are ready, or return to the previous lesson if a term is unclear.
Sign in to save your reading progress.
Sign in