NLP Engineer
Start here. Build forward.
Build language pipelines that handle real text, ambiguity and varied users.
Follow the stages in order, or open a topic to check what you already know. Each panel includes learning content, a practice task and a role checkpoint. Save Learning, Done or Skipped as you go. Skipped topics do not count as completed.
Build your foundations
Python, files and debugging
Understand the concept
Python is the working language for many AI experiments. Learn to inspect small values before connecting a large model. Variables, lists, dictionaries, loops and functions are enough for a first useful script. Errors are information: read the traceback and isolate the smallest failing input.
Learn it step by step
- Create a function with explicit inputs and outputs.
- Read a small JSON file and validate required keys.
- Handle a missing file and malformed input.
- Write checks for normal and empty input.
Build a small proof
Build a script that counts tutorial topics from ten fictional records and reports invalid rows separately.
Ready to move on?
You can explain each line, reproduce one error and fix its cause without hiding the exception. Keep the expected output with the script.
Common mistake
Copying a large generated program without understanding its inputs makes even small changes difficult to verify.
Your role checkpoint
For the NLP Engineer path, connect this topic to your capstone: Build a multilingual text-classification baseline and document annotation rules, errors and deployment constraints.
What is NLP?
In simple words
NLP stands for Natural Language Processing. It is the area of AI that works with human language. Your email spam filter, a translation tool and a system that finds names in documents are different language applications. The difficult part is that people use slang, mixed languages and context. “That was sick” can mean something positive or something medical.
Two useful terms
Text classification: assigning a category to text. Information extraction: finding specific facts such as a name or date.
Build your understanding
Natural language processing turns language into something software can analyse or generate. Start by defining the task: sentiment, topic classification, search and information extraction have different outputs. A good dataset reflects the spelling, languages and ambiguity of real users, including code-switching.
Follow the steps
- Collect ten non-sensitive sample support messages.
- Choose one task: classify topic into billing, access or other.
- Write label guidelines before assigning labels.
- Keep uncertain examples in a review group and compare disagreements.
Worked example
Input: "Payment ho gaya but course locked hai"
Possible topics: billing + access
Policy: classify by requested resolution -> access
Evidence: "course locked"Read the example once, then explain each part aloud in your own words. If it includes Python, you can understand the idea first and run the code later in a local Python environment. Code is shown as text; this page does not execute it.
Try this yourself
Label ten English and Hinglish messages. Ask a friend to label them independently and discuss disagreements.
Check your understanding
If annotators disagree about your label rule, the model cannot fix that ambiguity for you. Clarify the intended meaning before training.
A mistake to avoid
Aggressively removing punctuation, negation or every non-English token can erase the information your task needs.
Before looking at the answer, write a short explanation and one example of your own. Finish by naming one limitation of the method. Use Mark as read when you are ready, or return to the previous lesson if a term is unclear.
Your role checkpoint
For the NLP Engineer path, connect this topic to your capstone: Build a multilingual text-classification baseline and document annotation rules, errors and deployment constraints.
What is tokenisation?
In simple words
A language model does not simply receive text as one giant sentence. A tokenizer splits it into units called tokens. Think of dividing a chocolate bar into pieces before counting it, except token pieces do not have a fixed size. A short word might be one piece; another word might use several.
Two useful terms
Token: one unit processed by a model. Tokenizer: the method that splits text into those units.
Build your understanding
Tokenisation converts text into smaller units a model can process. Depending on the tokenizer, a unit may be a word, part of a word, punctuation or bytes. Token counts are model-specific. A whitespace split is an educational baseline, not an accurate substitute for a language model tokenizer.
Follow the steps
- Write examples containing English, Hindi, emoji and source code.
- Compare a simple whitespace split with the tokenizer for your chosen model.
- Record which strings create surprisingly many tokens.
- Estimate your input budget using that tokenizer before sending large documents.
Worked example
"AI is useful".split() -> ["AI", "is", "useful"]
This demonstrates whitespace splitting only.
A model tokenizer may split "usefulness" into several pieces.Read the example once, then explain each part aloud in your own words. If it includes Python, you can understand the idea first and run the code later in a local Python environment. Code is shown as text; this page does not execute it.
Try this yourself
Compare "hello", "Hello!" and "नमस्ते" with the same tokenizer. Explain why character length alone does not determine token count.
Check your understanding
The same text can produce different token counts for different tokenizers. You must specify the tokenizer when reporting a count.
A mistake to avoid
Do not assume one token equals one word, especially for multilingual text, URLs and code.
Before looking at the answer, write a short explanation and one example of your own. Finish by naming one limitation of the method. Use Mark as read when you are ready, or return to the previous lesson if a term is unclear.
Your role checkpoint
For the NLP Engineer path, connect this topic to your capstone: Build a multilingual text-classification baseline and document annotation rules, errors and deployment constraints.
Learn the core skills
What is TF-IDF?
In simple words
If every library book mentions “the”, that word will not help you choose a book. A less common word such as “photosynthesis” tells you more about the topic. TF-IDF uses this idea to assign word weights. It is useful for simple search and for turning text into numeric features.
Two useful terms
Term frequency: how often a word appears in a document. Inverse document frequency: a weight related to how uncommon it is across documents.
Build your understanding
TF-IDF represents documents using word weights. Words frequent in one document but less common across the collection often receive more weight. It is a useful lexical-search baseline and is easy to inspect. It cannot reliably understand synonyms or meaning when the words do not overlap.
Follow the steps
- Create a small collection of three topic descriptions.
- Fit a TF-IDF vectorizer to those documents.
- Transform a query using the fitted vectorizer.
- Rank documents by cosine similarity and inspect the top match.
Worked example
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
docs = ["learn python coding", "learn plant biology", "python data analysis"]
v = TfidfVectorizer()
x = v.fit_transform(docs)
scores = cosine_similarity(v.transform(["python"]), x)[0]
print(scores.argsort()[::-1])Read the example once, then explain each part aloud in your own words. If it includes Python, you can understand the idea first and run the code later in a local Python environment. Code is shown as text; this page does not execute it.
Try this yourself
Try "programming" instead of "python". Explain why relevant results may receive zero similarity when that word is absent.
Check your understanding
Zero lexical overlap does not establish that a document is irrelevant. It exposes a limitation of this representation.
A mistake to avoid
For supervised evaluation, fit the vectorizer on training text only. Learning the vocabulary from held-out data can leak information.
Before looking at the answer, write a short explanation and one example of your own. Finish by naming one limitation of the method. Use Mark as read when you are ready, or return to the previous lesson if a term is unclear.
Your role checkpoint
For the NLP Engineer path, connect this topic to your capstone: Build a multilingual text-classification baseline and document annotation rules, errors and deployment constraints.
What is sentiment analysis?
In simple words
A shop receives “Loved the bag” and “The zip broke”. Sentiment analysis tries to identify the opinion expressed in each message. It becomes harder with “Nice colour, terrible quality” because different aspects receive different opinions. A computer prediction about the text is not a direct measurement of the writer’s feelings.
Two useful terms
Sentiment: the expressed opinion or attitude. Aspect: the specific thing being discussed, such as delivery or quality.
Build your understanding
Sentiment analysis estimates expressed opinion. Labels depend on context: a message can praise delivery while criticising the product. Before modelling, decide whether you need one overall label, labels for individual aspects or an uncertain category. Sarcasm and code-switching make simple word lists unreliable.
Follow the steps
- Define positive, negative, mixed and uncertain in plain language.
- Label twenty short fictional product reviews.
- Build a simple keyword baseline and record errors.
- Test negation, sarcasm and mixed opinions separately.
Worked example
"Fast delivery, but the charger stopped working."
Overall: mixed
Delivery aspect: positive
Product reliability aspect: negativeRead the example once, then explain each part aloud in your own words. If it includes Python, you can understand the idea first and run the code later in a local Python environment. Code is shown as text; this page does not execute it.
Try this yourself
Write five reviews containing "not", such as "not bad" and "not useful". Compare what a keyword counter would predict.
Check your understanding
The word "bad" alone does not determine the sentiment of "not bad". Evaluate complete phrases and your chosen annotation policy.
A mistake to avoid
Do not present inferred sentiment as a person’s true emotional state. It is a fallible interpretation of a specific text.
Before looking at the answer, write a short explanation and one example of your own. Finish by naming one limitation of the method. Use Mark as read when you are ready, or return to the previous lesson if a term is unclear.
Your role checkpoint
For the NLP Engineer path, connect this topic to your capstone: Build a multilingual text-classification baseline and document annotation rules, errors and deployment constraints.
What are embeddings?
In simple words
Imagine placing ideas on a map so that related ideas are often near one another. Embeddings are numeric representations that make this kind of comparison possible. A search system may connect “affordable laptop” with “budget computer” without an exact word match. The map is learned and imperfect; nearby items are not necessarily interchangeable.
Two useful terms
Vector: a list of numbers. Similarity: a measure used to compare two representations.
Build your understanding
An embedding maps an input to a numeric vector. Similarity can help retrieve related passages even when they use different words. The usefulness of distance depends on the embedding model, task and data. Similarity scores are ranking signals, not probabilities that a claim is true.
Follow the steps
- Choose one embedding model and use it for both documents and queries.
- Embed a small set of sentences about distinct topics.
- Compare cosine similarity to rank candidates.
- Inspect a synonym case, an unrelated case and a negated sentence.
Worked example
Illustrative vectors only:
A = [1, 0] B = [0.8, 0.6] C = [0, 1]
cosine(A, B) = 0.8
cosine(A, C) = 0
Real embeddings have more dimensions.Read the example once, then explain each part aloud in your own words. If it includes Python, you can understand the idea first and run the code later in a local Python environment. Code is shown as text; this page does not execute it.
Try this yourself
Compare "the course is free" and "the course is not free". Explain why similar wording may create a dangerous retrieval match.
Check your understanding
A high similarity score can reflect shared topic while missing a critical difference such as negation. Read the retrieved passage before answering.
A mistake to avoid
Do not mix document vectors from one model with query vectors from another, even if their dimensions happen to match.
Before looking at the answer, write a short explanation and one example of your own. Finish by naming one limitation of the method. Use Mark as read when you are ready, or return to the previous lesson if a term is unclear.
Your role checkpoint
For the NLP Engineer path, connect this topic to your capstone: Build a multilingual text-classification baseline and document annotation rules, errors and deployment constraints.
Practise with real decisions
What is a transformer in AI?
In simple words
A transformer is a model architecture that uses attention along with other components. Attention helps combine information from different positions in a sequence. In “river bank” and “bank account”, the surrounding words give different clues. This is an intuition for context, not a complete explanation of how a trained model works.
Two useful terms
Architecture: the organisation of a model. Attention: learned computation that weights information from different positions.
Build your understanding
Attention lets a model combine information from different positions using learned relevance weights. Transformers use attention with other components to process sequences. The representation of a word can change with context. This tutorial is an intuition exercise, not a full implementation of training or inference.
Follow the steps
- Write two sentences where the same word has different meanings.
- Identify nearby words that disambiguate it.
- Imagine assigning different relevance weights to those words.
- Explain how a context-sensitive representation differs from a fixed dictionary lookup.
Worked example
"She sat by the river bank."
"She deposited money at the bank."
The tokens "river" and "money" provide different context.
Attention is learned numeric computation, not a human reading rule.Read the example once, then explain each part aloud in your own words. If it includes Python, you can understand the idea first and run the code later in a local Python environment. Code is shown as text; this page does not execute it.
Try this yourself
Use "bat" in a sports sentence and an animal sentence. Identify the evidence that distinguishes the meanings.
Check your understanding
Context can change the useful representation of the same token. Attention weights alone are not a reliable explanation of a model’s complete decision.
A mistake to avoid
Do not interpret a fluent generated continuation as verified knowledge. Sequence modelling and fact-checking are different tasks.
Before looking at the answer, write a short explanation and one example of your own. Finish by naming one limitation of the method. Use Mark as read when you are ready, or return to the previous lesson if a term is unclear.
Your role checkpoint
For the NLP Engineer path, connect this topic to your capstone: Build a multilingual text-classification baseline and document annotation rules, errors and deployment constraints.
Fine-tuning and adaptation
Understand the concept
Fine-tuning changes a model’s parameters using task-specific training examples. It is different from supplying documents at inference time. First establish whether prompting or retrieval solves the problem. A training dataset needs permission, consistent labels and an evaluation split that reflects intended use.
Learn it step by step
- Define a baseline without tuning.
- Prepare a small, reviewed example set.
- Separate training and held-out evaluation.
- Compare usefulness, regressions and cost.
Build a small proof
Write an experiment plan for consistent support-response formatting using synthetic examples.
Ready to move on?
Your plan has a reason tuning might help and a test that could show it did not. Include cases outside the narrow training pattern.
Common mistake
Fine-tuning is not a dependable way to keep rapidly changing facts current.
Your role checkpoint
For the NLP Engineer path, connect this topic to your capstone: Build a multilingual text-classification baseline and document annotation rules, errors and deployment constraints.
What is AI model evaluation?
In simple words
A polished answer is not enough to know whether an AI system works. Evaluation checks specific requirements using examples and scoring rules. For a course assistant, you might check whether it finds the correct course, admits missing information and respects account permissions. Keep failure cases as part of your test collection.
Two useful terms
Test case: an input with an expected behavior. Metric: a defined measurement of performance.
Build your understanding
An AI evaluation checks whether a system meets a defined task. Build examples around realistic successes and failures, not only easy demonstrations. Keep the test set stable enough to compare versions and record the model, prompt and settings used. A single aggregate score can hide critical mistakes.
Follow the steps
- Define success in observable terms.
- Create normal, ambiguous, missing-information and adversarial examples.
- Write expected answers or scoring rules before running the system.
- Compare versions and inspect regressions before accepting a change.
Worked example
FAQ evaluation columns:
question | expected source | required facts | forbidden claim | result
Example: fees unknown | FAQ-3 | say unknown | invented fee | pass/failRead the example once, then explain each part aloud in your own words. If it includes Python, you can understand the idea first and run the code later in a local Python environment. Code is shown as text; this page does not execute it.
Try this yourself
Write five tests for an enrolment assistant. Include a user without Pro access and a request for a nonexistent course.
Check your understanding
A response should not promise enrolment if server-side permissions deny it. Include permission-sensitive cases, not just wording checks.
A mistake to avoid
Do not judge a model solely by another model’s unreviewed score. Human checks and deterministic validation are useful for important requirements.
Before looking at the answer, write a short explanation and one example of your own. Finish by naming one limitation of the method. Use Mark as read when you are ready, or return to the previous lesson if a term is unclear.
Your role checkpoint
For the NLP Engineer path, connect this topic to your capstone: Build a multilingual text-classification baseline and document annotation rules, errors and deployment constraints.
Ship, evaluate and improve
AI application security and threat modelling
Understand the concept
Threat modelling starts with what must be protected, who can access it and where untrusted data enters. Model outputs and retrieved documents are untrusted inputs. A language instruction is not an authorisation boundary; server-side policy decides what a user or tool may do.
Learn it step by step
- List assets and trust boundaries.
- Separate public and private data.
- Validate tool inputs and enforce permissions.
- Test malicious instructions using synthetic fixtures in your own sandbox.
Build a small proof
Draw a document assistant showing where access checks happen before retrieval and before tool execution.
Ready to move on?
An unauthorised user cannot retrieve another account’s private document, even when a prompt asks for it.
Common mistake
Do not test real third-party systems without permission or expose secrets in your demonstration.
Your role checkpoint
For the NLP Engineer path, connect this topic to your capstone: Build a multilingual text-classification baseline and document annotation rules, errors and deployment constraints.
Deployment, containers and rollback
Understand the concept
Deployment makes a working experiment available to users. It needs configuration, a reproducible environment, health checks and a way to roll back. A successful launch does not establish ongoing reliability. Monitor behavior after release and keep a known-good version.
Learn it step by step
- Document dependencies and configuration.
- Run a health check separately from a model call.
- Package a minimal service.
- Practise replacing it with the previous version.
Build a small proof
Deploy a mock prediction service locally in a container and simulate an unavailable dependency.
Ready to move on?
You can rebuild the environment and recover after a failed release without editing production files by hand.
Common mistake
Do not bake secrets into container images or assume the development machine matches production.
Your role checkpoint
For the NLP Engineer path, connect this topic to your capstone: Build a multilingual text-classification baseline and document annotation rules, errors and deployment constraints.
Monitoring quality, latency and cost
Understand the concept
AI monitoring needs more than uptime. Track response time, failures, input volume and task quality using a suitable sample of reviewed outcomes. Quality can change when users or source documents change. Log enough to diagnose failures without retaining unnecessary sensitive input.
Learn it step by step
- Define a service-level target.
- Record request outcome and latency.
- Track a stable evaluation set over releases.
- Create an alert and an investigation checklist.
Build a small proof
Produce a small report comparing two releases on success rate, p95 latency and estimated per-request cost.
Ready to move on?
You can tell whether a change improved usefulness while increasing cost or response time, and explain the trade-off.
Common mistake
An average hides slow tails and rare severe failures. Examine distributions and critical error categories.
Your role checkpoint
For the NLP Engineer path, connect this topic to your capstone: Build a multilingual text-classification baseline and document annotation rules, errors and deployment constraints.
Build something you can explain.
Build a multilingual text-classification baseline and document annotation rules, errors and deployment constraints.
Role expectations vary by company. Use this path as a starting point and build evidence of your skills.