Data Engineer
Start here. Build forward.
Build dependable pipelines and tables that support analytics and AI products.
Follow the stages in order, or open a topic to check what you already know. Each panel includes learning content, a practice task and a role checkpoint. Save Learning, Done or Skipped as you go. Skipped topics do not count as completed.
Build your foundations
Python, files and debugging
Understand the concept
Python is the working language for many AI experiments. Learn to inspect small values before connecting a large model. Variables, lists, dictionaries, loops and functions are enough for a first useful script. Errors are information: read the traceback and isolate the smallest failing input.
Learn it step by step
- Create a function with explicit inputs and outputs.
- Read a small JSON file and validate required keys.
- Handle a missing file and malformed input.
- Write checks for normal and empty input.
Build a small proof
Build a script that counts tutorial topics from ten fictional records and reports invalid rows separately.
Ready to move on?
You can explain each line, reproduce one error and fix its cause without hiding the exception. Keep the expected output with the script.
Common mistake
Copying a large generated program without understanding its inputs makes even small changes difficult to verify.
Your role checkpoint
For the Data Engineer path, connect this topic to your capstone: Build an idempotent ingestion pipeline with validation, analytical queries, failure recovery and a data dictionary.
SQL and relational data
Understand the concept
SQL queries structured tables. Start with filtering, grouping and joins. A join can multiply rows when the relationship is not one-to-one, so check counts before trusting totals. Data engineers and scientists both need to understand what one row represents.
Learn it step by step
- Create fictional learners and enrolments tables.
- Select only necessary columns.
- Join using stable IDs.
- Compare row counts and inspect unmatched records.
Build a small proof
Calculate enrolment counts by course, including a course with zero learners.
Ready to move on?
Explain why an inner join may drop empty courses and how a left join changes that result. Verify totals against the input rows.
Common mistake
Joining on a non-unique display name can double-count or merge different people.
Your role checkpoint
For the Data Engineer path, connect this topic to your capstone: Build an idempotent ingestion pipeline with validation, analytical queries, failure recovery and a data dictionary.
Git and reproducible experiments
Understand the concept
Version control records changes so you can compare experiments and restore a known working state. A commit should describe a coherent change. Keep source code and small configuration files; credentials and large private datasets need separate handling.
Learn it step by step
- Initialise a repository and add a README.
- Commit a working baseline.
- Create a branch for one experiment.
- Compare the change and document how to reproduce it.
Build a small proof
Store two versions of a prompt or data-cleaning script and explain why the second version exists.
Ready to move on?
Another person can run the baseline from your instructions and identify exactly what changed between versions.
Common mistake
Do not commit secrets, private learner records or generated cache folders. Removing a secret from the latest file does not erase Git history.
Your role checkpoint
For the Data Engineer path, connect this topic to your capstone: Build an idempotent ingestion pipeline with validation, analytical queries, failure recovery and a data dictionary.
Learn the core skills
What is a dataset?
In simple words
A dataset is a collection of observations used for analysis or modelling. In a table, a row often represents one observation and a column one attribute. Define these meanings before doing calculations. The collection process determines which cases you have, which you lack and what conclusions are reasonable.
Words to know
Observation: one recorded case. Schema: the expected structure and meaning of fields.
Pause here and explain the definition in one sentence without copying it. The aim is to describe what the system does, not to memorise an impressive phrase.
A familiar example
A fictional plant table could contain plant_id, hours_of_sunlight and height_cm. One row represents one plant measured on one date. If you measure the same plant repeatedly, add a date and remember those rows are related.
Break it down
- Identify the input: what information is available?
- Describe the operation or pattern being used.
- Identify the output and how you would check it.
- Name one mistake the system could make.
Change one detail in this example and consider whether the same idea still applies. This helps separate the underlying concept from one specific application.
Your turn
Design a six-row fictional dataset about a non-sensitive topic. Write a definition and unit for each column.
Answer check
A height of 10 means little without a unit. A row also needs a clear identity: one plant, one measurement or one household are different units of observation.
Explain it to a friend
Give a second example from college, a hobby or a fictional workplace. State what information goes in, what comes out and one limitation. If you can explain those three things in plain language, you are ready to move on. Mark this lesson as read to save your progress when signed in.
Your role checkpoint
For the Data Engineer path, connect this topic to your capstone: Build an idempotent ingestion pipeline with validation, analytical queries, failure recovery and a data dictionary.
What is data cleaning?
In simple words
Imagine a registration list where Pune appears as “Pune”, “pune” and “ Pune”. Cleaning can standardise these values so your analysis counts them consistently. But cleaning is not deleting everything unusual. Some unusual entries are real, and some missing entries have a meaning you need to preserve.
Two useful terms
Missing value: an entry that is absent or unknown. Duplicate: a repeated record according to a defined identity rule.
Build your understanding
Cleaning transforms raw observations into consistent inputs. Keep the raw data and document your decisions so another person can reproduce the result. Correct obvious formatting issues, but do not silently erase unusual values that might represent real cases.
Follow the steps
- Standardise obvious whitespace and spelling variants in category names.
- Identify missing values separately from zero values.
- Detect duplicates using a meaningful record identifier.
- Define how to flag questionable records for review.
Worked example
Raw cities: [" Pune", "pune", "PUNE", "Mumbai"]
Normalised: ["pune", "pune", "pune", "mumbai"]
Record ID, not matching city alone, determines duplicates.Read the example once, then explain each part aloud in your own words. If it includes Python, you can understand the idea first and run the code later in a local Python environment. Code is shown as text; this page does not execute it.
Try this yourself
Design a cleaning log with columns for row ID, original value, new value and reason. Apply it to five fictional rows.
Check your understanding
Two records from the same city are not duplicates merely because their city field matches. Duplicate rules depend on what one row represents.
A mistake to avoid
If cleaning uses learned statistics such as means, fit those statistics on training data only and apply them unchanged to validation and test data.
Before looking at the answer, write a short explanation and one example of your own. Finish by naming one limitation of the method. Use Mark as read when you are ready, or return to the previous lesson if a term is unclear.
Your role checkpoint
For the Data Engineer path, connect this topic to your capstone: Build an idempotent ingestion pipeline with validation, analytical queries, failure recovery and a data dictionary.
Data modelling and analytical tables
Understand the concept
An analytical model organises information around questions users need to answer. Define the grain of each table, keys and relationships. A fact table may record events, while dimensions describe things such as courses or dates. Preserve history deliberately rather than overwriting it accidentally.
Learn it step by step
- Write the grain in one sentence.
- Identify primary and foreign keys.
- Define how changes are represented.
- Validate totals through sample queries.
Build a small proof
Design tables for fictional course enrolments and completions that support monthly reports.
Ready to move on?
You can count learners and enrolments separately and explain why their totals differ.
Common mistake
Mixing one-row-per-user and one-row-per-event data without care creates misleading aggregates.
Your role checkpoint
For the Data Engineer path, connect this topic to your capstone: Build an idempotent ingestion pipeline with validation, analytical queries, failure recovery and a data dictionary.
Practise with real decisions
Batch pipelines and orchestration
Understand the concept
A pipeline moves data through repeatable stages such as extraction, validation, transformation and loading. Each stage should have a clear input and output. A rerun should not corrupt results or create duplicates. Start with a scheduled batch before adding streaming complexity.
Learn it step by step
- Define source and destination schemas.
- Validate required fields at ingestion.
- Make output writes repeatable.
- Record run status and failed rows.
Build a small proof
Build a local CSV-to-table pipeline that can rerun the same day’s input safely.
Ready to move on?
Run it twice and prove that row counts and values remain correct. Explain how late or invalid records are handled.
Common mistake
Appending every rerun without stable keys silently duplicates data.
Your role checkpoint
For the Data Engineer path, connect this topic to your capstone: Build an idempotent ingestion pipeline with validation, analytical queries, failure recovery and a data dictionary.
Enterprise integration and data contracts
Understand the concept
Integration connects a solution to existing systems without losing control of data ownership and permissions. Agree schemas, IDs, update behavior and failure handling. A model should not silently invent missing records or bypass the source system’s rules.
Learn it step by step
- Define a schema and ownership for every field.
- Use stable external IDs.
- Design retries and duplicate handling.
- Test partial failure and reconciliation.
Build a small proof
Connect two local mock services with a queue of fictional requests and document a retry policy.
Ready to move on?
Replaying the same request does not duplicate the business operation, and failed records can be inspected.
Common mistake
Do not treat an API returning HTTP 200 as proof that the intended business change happened.
Your role checkpoint
For the Data Engineer path, connect this topic to your capstone: Build an idempotent ingestion pipeline with validation, analytical queries, failure recovery and a data dictionary.
What are training, validation and test data?
In simple words
When preparing for an exam, you study examples, use practice tests to improve and finally take an unseen exam. ML evaluation uses a similar separation. Training fits the model, validation guides your choices and testing checks the final design on held-out examples. The separation helps make your score more honest.
Two useful terms
Training set: examples used to fit a model. Held-out data: examples kept separate from fitting.
Build your understanding
A model needs evaluation on data it did not learn from. Training data fits parameters, validation data supports model choices and test data provides a final held-out check. The split must reflect how the system will be used: time, users and groups can create dependencies between rows.
Follow the steps
- Define what an independent example means for your task.
- Separate the test set before experimenting.
- Use validation data or cross-validation to choose settings.
- Evaluate the final design once on the held-out test set and report limitations.
Worked example
For 1,000 independent toy examples:
600 training + 200 validation + 200 test
For repeated users: keep each user in only one split.
For forecasting: use chronological splits.Read the example once, then explain each part aloud in your own words. If it includes Python, you can understand the idea first and run the code later in a local Python environment. Code is shown as text; this page does not execute it.
Try this yourself
Explain why randomly splitting multiple messages from the same person can overestimate performance on entirely new people.
Check your understanding
The model may learn person-specific patterns shared across splits. A group-based split better tests generalisation to new people.
A mistake to avoid
Do not repeatedly tune choices against the final test score. That gradually turns the test set into another validation set.
Before looking at the answer, write a short explanation and one example of your own. Finish by naming one limitation of the method. Use Mark as read when you are ready, or return to the previous lesson if a term is unclear.
Your role checkpoint
For the Data Engineer path, connect this topic to your capstone: Build an idempotent ingestion pipeline with validation, analytical queries, failure recovery and a data dictionary.
Ship, evaluate and improve
AI application security and threat modelling
Understand the concept
Threat modelling starts with what must be protected, who can access it and where untrusted data enters. Model outputs and retrieved documents are untrusted inputs. A language instruction is not an authorisation boundary; server-side policy decides what a user or tool may do.
Learn it step by step
- List assets and trust boundaries.
- Separate public and private data.
- Validate tool inputs and enforce permissions.
- Test malicious instructions using synthetic fixtures in your own sandbox.
Build a small proof
Draw a document assistant showing where access checks happen before retrieval and before tool execution.
Ready to move on?
An unauthorised user cannot retrieve another account’s private document, even when a prompt asks for it.
Common mistake
Do not test real third-party systems without permission or expose secrets in your demonstration.
Your role checkpoint
For the Data Engineer path, connect this topic to your capstone: Build an idempotent ingestion pipeline with validation, analytical queries, failure recovery and a data dictionary.
Deployment, containers and rollback
Understand the concept
Deployment makes a working experiment available to users. It needs configuration, a reproducible environment, health checks and a way to roll back. A successful launch does not establish ongoing reliability. Monitor behavior after release and keep a known-good version.
Learn it step by step
- Document dependencies and configuration.
- Run a health check separately from a model call.
- Package a minimal service.
- Practise replacing it with the previous version.
Build a small proof
Deploy a mock prediction service locally in a container and simulate an unavailable dependency.
Ready to move on?
You can rebuild the environment and recover after a failed release without editing production files by hand.
Common mistake
Do not bake secrets into container images or assume the development machine matches production.
Your role checkpoint
For the Data Engineer path, connect this topic to your capstone: Build an idempotent ingestion pipeline with validation, analytical queries, failure recovery and a data dictionary.
Monitoring quality, latency and cost
Understand the concept
AI monitoring needs more than uptime. Track response time, failures, input volume and task quality using a suitable sample of reviewed outcomes. Quality can change when users or source documents change. Log enough to diagnose failures without retaining unnecessary sensitive input.
Learn it step by step
- Define a service-level target.
- Record request outcome and latency.
- Track a stable evaluation set over releases.
- Create an alert and an investigation checklist.
Build a small proof
Produce a small report comparing two releases on success rate, p95 latency and estimated per-request cost.
Ready to move on?
You can tell whether a change improved usefulness while increasing cost or response time, and explain the trade-off.
Common mistake
An average hides slow tails and rare severe failures. Examine distributions and critical error categories.
Your role checkpoint
For the Data Engineer path, connect this topic to your capstone: Build an idempotent ingestion pipeline with validation, analytical queries, failure recovery and a data dictionary.
Build something you can explain.
Build an idempotent ingestion pipeline with validation, analytical queries, failure recovery and a data dictionary.
Role expectations vary by company. Use this path as a starting point and build evidence of your skills.