Azure Databricks Data Engineering
Build a governed lakehouse on Azure — ingestion through reporting
About this course
A hands-on data engineering course built around one governed lakehouse on Azure Databricks. You start from how Spark distributes work, provision a workspace and compute you can afford, secure access to Data Lake Storage Gen2 through a workload identity, and make Unity Catalog the governance target for everything that follows. From there you ingest delimited and nested sources into bronze with declared schemas and quality gates, conform them into silver with joins and window functions, publish gold tables in Spark SQL, and convert the pipeline to Delta for transactional writes, merge-based upserts, schema control and time travel. The final phase schedules the pipeline as a parameterised workflow, orchestrates it from Azure Data Factory, publishes to Power BI, and closes the loop with lineage, audit and group-based access control. The capstone is a synthetic operations analytics platform for a fictional parcel carrier, built end to end across bronze, silver and gold. Streaming and machine learning are out of scope and this course makes no promise about either.
What you’ll be able to do
- Explain how Spark distributes work across driver and executors, and read a slow job for the reason
- Provision an Azure Databricks workspace with compute sized and configured for the workload, and keep spend predictable
- Give Databricks access to Data Lake Storage Gen2 through a workload identity with secrets held outside the code
- Model and create a Unity Catalog layout — catalogs, schemas, managed and external tables, external locations — as the governance target
- Ingest delimited, nested and multi-file sources into bronze with declared schemas, audit columns and a defensible partitioning choice
- Make ingestion trustworthy: handle malformed records, enforce key expectations, and build loads that are safe to re-run
- Conform bronze into silver with joins, aggregations and window functions at a grain you assert rather than assume
- Publish gold tables and views in Spark SQL that answer a reporting question without further joining
- Use Delta Lake for transactional writes, merge-based upserts and deletes, schema enforcement and recovery by time travel
- Schedule the pipeline as a parameterised multi-task workflow with retries, alerting, repair and backfill
- Orchestrate Databricks from Azure Data Factory and publish the gold layer to Power BI
- Operate the governed platform day to day: discovery, audit, lineage and least-privilege access granted to groups
- Recognise legacy access patterns in an inherited estate and plan a safe migration away from them
Before you start
What this course assumes you can already do.
- Working SQL — joins, grouping and subqueries — and Python fundamentals, including functions and working with lists and dictionaries.
- No Spark, Databricks or Azure experience is assumed.
- You will need your own Azure subscription for the labs; Azure consumption is billed by Microsoft directly to you and is not included in this course.
The curriculum
Open any module to see its objective, every teaching topic, the practice brief and the checkpoint.
Recaps and reference material are included in the LMS.
PART 1Platform Foundations
01Distributed Processing and the Lakehouse
Explain how Spark splits work across a cluster, and place Databricks and the lakehouse pattern accurately against the data warehouse and data lake approaches that came before them.
- Why a single machine stops being enough for analytics workloadsConcept
- Spark cluster anatomy: driver, executors, cores and tasksConcept
- Jobs, stages and shuffles: how your code becomes workDemonstration
- Lazy evaluation, transformations and actionsHands-on
- Warehouse, lake, lakehouse: what problem each one solvedConcept
Read a cluster like an engineer
Write a one-page sizing note for a fictional nightly load of roughly two hundred million rows. State how many partitions you expect, how many task slots you would provision, where a shuffle is likely, and which of driver memory, executor memory or slot count you would raise first if the job ran long.
Self-check questions to test your understanding
02Azure Workspace, Compute and Cost Control
Provision an Azure Databricks workspace and configure compute that fits the workload, then keep a learning or team environment inside a known budget.
- How the service is split: managed control plane and your data planeConcept
- Provisioning a workspace and finding your way around itHands-on
- All-purpose compute versus job compute, and when each is rightConcept
- Compute configuration: access mode, runtime version, autoscaling, auto-terminationHands-on
- Choosing an instance family: general purpose, memory, storage and compute optimisedConcept
- The consumption model, pools and policies for predictable spendHands-on
Build and defend a lab budget
Produce a compute standard for your own lab environment: one interactive configuration for development and one job configuration for scheduled runs. For each, record instance family, node count, autoscaling bounds, auto-termination window and the reason behind every value.
Self-check questions to test your understanding
03Notebooks and the Developer Workflow
Work productively in Databricks notebooks — mixed languages, utilities, parameters and modular structure — so that later pipeline code is reusable rather than a single long cell.
- Notebook anatomy, attached compute and execution orderHands-on
- Mixing Python, SQL and Markdown with magic commandsHands-on
- Workspace utilities for files, directories and listingsHands-on
- Parameterising a notebook with widgets and typed defaultsHands-on
- Modularising work: shared setup notebooks and notebook callsHands-on
Refactor a monolithic notebook
Take any long notebook you have written and restructure it into three parts: a shared setup notebook holding configuration and helpers, a parameterised worker notebook that does the actual work, and a thin caller that runs the worker for two different parameter values. Every path, date and environment name must arrive as a parameter with a documented default.
Self-check questions to test your understanding
PART 2Secure, Governed Platform
04Azure Identity, Storage and Secure Access
Give Databricks access to Azure Data Lake Storage Gen2 using a managed identity or service principal with secrets held outside the code, and recognise the legacy credential patterns you will meet in older estates.
- Data Lake Storage Gen2: hierarchical namespace, containers and the abfss pathHands-on
- Microsoft Entra ID identities: users, groups, service principals and managed identitiesConcept
- Authorising storage access with role assignments and ACLsHands-on
- Connecting Databricks to storage with a workload identity — the pattern we useHands-on
- Secrets: key vault backed scopes and referencing them safelyHands-on
- Legacy credential patterns you will inherit — and how to migrate off themConcept
Migrate one legacy access path
Given an inherited notebook that reaches storage through an account key held in a configuration cell, write the migration plan that removes it. Identify the workload identity to use, the role assignment and scope needed, where the remaining secret belongs, the code that changes, and how you would prove the old credential is no longer reachable before rotating it.
Self-check questions to test your understanding
05Unity Catalog as the Governance Target
Model and create a governed catalog — metastore, catalogs, schemas, managed and external tables, storage credentials and external locations — as the destination for everything built later in the course.
- What governance has to deliver, and where a workspace-local metastore falls shortConcept
- The object model: metastore, catalog, schema, table, view and volumeConcept
- Attaching a governed metastore and enabling compute to use itHands-on
- Storage credentials and external locations: governed access to your own containersHands-on
- Managed versus external tables: ownership, lifecycle and when to choose whichHands-on
- Designing catalogs and schemas for environments and medallion layersHands-on
Design your catalog layout
Design and create the governed layout you will use for the remainder of the course. Decide whether environments become separate catalogs or separate schemas, justify the choice, and record which layers hold external tables and which hold managed tables. Add one external location per storage container with the narrowest scope that still works.
Self-check questions to test your understanding
PART 3Building the Pipeline
06Ingestion with PySpark: Readers, Schemas and Writers
Ingest delimited, nested and multi-file sources into a columnar bronze layer with explicit schemas, deliberate write modes and a partitioning choice you can defend.
- The reader API and why you declare a schema instead of inferring oneHands-on
- Delimited files: headers, delimiters, encodings and honest type castingHands-on
- Nested sources: single-record and multi-line JSON, arrays and structsHands-on
- Reading many files at once: folder patterns, mixed layouts and source-file trackingHands-on
- Shaping the frame: selecting, renaming, deriving and adding audit columnsHands-on
- The writer API: save modes, columnar output and what each mode really doesHands-on
- Partitioning: choosing a column, and the small-files trapHands-on
Ingest three shapes into bronze
Ingest three sources into your bronze layer: one delimited file, one nested JSON document and one folder of many daily files. Each must use a declared schema, carry the same four audit columns, and write columnar output under a partitioning choice you justify in one sentence. Then re-run every ingestion a second time and state exactly what happened to the target under the save mode you chose.
Self-check questions to test your understanding
07Data Quality and Reload Patterns
Make ingestion trustworthy and repeatable: handle malformed records deliberately, enforce key expectations, and choose between full and incremental load patterns for a given source.
- Malformed records: permissive, drop and fail-fast, and keeping the rejectsHands-on
- Key expectations: nulls, duplicates and deciding what a row meansHands-on
- Writing checks that fail loudly instead of quietlyHands-on
- Full load versus incremental load: what each assumes about the sourceConcept
- Idempotent reloads and the hybrid case: full file, incremental targetHands-on
Make one ingestion re-runnable and honest
Take one of your bronze ingestions and harden it. Add a quarantine path for malformed records, four assertions covering row count, key uniqueness, a required relationship and one value range, and a run log row recording outcome and counts. Then make the load idempotent, so running it twice on the same input leaves the target identical. Prove it by running twice and comparing counts.
Self-check questions to test your understanding
08Transformations: Filters, Joins, Aggregations and Windows
Turn bronze tables into a conformed silver layer using joins that keep the rows you expect, aggregations at an agreed grain, and window functions for ranking and period comparison.
- Filtering and column expressions that read like the requirementHands-on
- Join types in practice: inner, left, full, semi and antiHands-on
- When a join is wrong: duplicate keys, row explosion and unmatched rowsHands-on
- Making joins fast: partitioning, broadcast of small tables and skewDemonstration
- Aggregations at a declared grain, with groupings that document themselvesHands-on
- Window functions: ranking, running totals and period-on-period changeHands-on
Build one conformed silver table
Join at least three bronze tables into a single silver table that answers one clear question. Before the join, assert key uniqueness on each side; after it, assert that the output row count matches the grain you declared in a comment at the top. Add one ranking column and one period-on-period comparison using window functions.
Self-check questions to test your understanding
09Spark SQL and the Serving Layer
Register and query governed tables and views in SQL, and assemble gold tables shaped for reporting rather than for the pipeline that produced them.
- Catalogs, schemas, tables and views in SQL, and the three-level nameHands-on
- Creating tables from queries, and choosing managed or external in SQLHands-on
- Query fundamentals for the lakehouse: predicates, sorting and expressionsHands-on
- Aggregates, joins and window functions in SQL, side by side with the DataFrame APIHands-on
- Views, temporary views and gold tables designed for the reportHands-on
Publish a gold table for one question
Choose one reporting question and publish a gold table that answers it without further joining. Write it in SQL from your silver layer, name every column as a reader would, and document grain, refresh expectation and owner in the table comment. Add one view that pre-filters the table for a specific audience, and note why it is a view and not another table.
Self-check questions to test your understanding
PART 4Delta, Orchestration & Delivery
10Delta Lake: ACID, Upserts, Evolution and Time Travel
Convert the pipeline to Delta tables and use the guarantees that follow — atomic writes, merge-based upserts and deletes, schema enforcement, and version history for recovery and audit.
- What plain files cannot give you: partial writes, inconsistent reads, no deletesConcept
- How Delta works: data files plus an ordered transaction logConcept
- Creating and converting tables, and reading a table historyHands-on
- Merge: upserts, deletes for data subject requests, and conditional updatesHands-on
- Schema enforcement and controlled evolutionHands-on
- Time travel, restore, file retention and compactionHands-on
Recover from a bad load
Convert one silver table to Delta, then stage a realistic incident: load a batch containing corrupted values, notice it through your own checks, and recover. Use table history to identify the offending version, restore, then re-load correctly using a merge rather than an overwrite. Separately, delete one individual record and show it is gone from the current version.
Self-check questions to test your understanding
11Jobs, Workflows and Data Factory Orchestration
Schedule the pipeline as a parameterised multi-task workflow with retries, alerting and recoverable reruns, and orchestrate it from Azure Data Factory alongside non-Databricks activities.
- From notebook to task: parameters, defaults and safe rerunsHands-on
- Multi-task workflows: dependencies, shared compute and fan-outHands-on
- Scheduling, retries, timeouts and failure alertingHands-on
- Recovering a failed run: repair, partial rerun and backfillHands-on
- Data Factory fundamentals: linked services, datasets, activities and pipelinesConcept
- Calling Databricks from a Data Factory pipeline with parametersHands-on
- Triggers, tumbling windows, dependencies and backfilling from the orchestratorHands-on
Schedule the pipeline end to end
Wire your bronze, silver and gold notebooks into one parameterised workflow with correct dependencies, a retry policy, a timeout and a failure notification. Trigger it from a Data Factory pipeline that passes a run date. Then do three things: force a mid-pipeline failure and confirm the alert, repair the run so only the failed branch re-executes, and backfill two past dates.
Self-check questions to test your understanding
12Serving, Lineage and Access Control
Publish the gold layer to Power BI and operate the governed platform day to day — discovery, audit, lineage and a privilege model granted to groups.
- Connecting Power BI to a SQL warehouse, and choosing import or direct queryHands-on
- Shaping gold tables for a semantic model, and what the report should never doConcept
- Discovery: comments, tags and making a table findable by someone who did not build itHands-on
- The privilege model: securable objects, inheritance, groups and ownershipHands-on
- Audit: answering who read what, when, and from whereHands-on
- Lineage: reading upstream and downstream impact, and its known limitsHands-on
Hand the platform over
Produce the handover pack a second engineer would need. Document each gold table with grain, refresh, owner and known caveats. Define three groups with different needs and grant each the narrowest privileges that let it work, then verify by querying as one of them.
Self-check questions to test your understanding
Metroline Logistics operations analytics platform
Metroline Logistics is a fictional regional parcel carrier running six depots, a mixed delivery fleet and a rotating driver roster. You receive a synthetic daily drop of operational extracts — depots, vehicles, routes, delivery scans, driver shifts and fuel transactions — in mixed delimited and nested formats, with the inconsistencies a real feed has. Build the governed lakehouse that turns it into three reporting answers the operations director asks for every Monday: on-time delivery rate by depot and week, cost per delivered parcel by route, and driver shift utilisation. The dataset and the scenario are synthetic and generated for this course; no real carrier data is involved.
Platform and governance setup — Stand up the workspace, storage and governed catalog structure the rest of the capstone writes into, with access through a workload identity only.
Bronze ingestion — Land all six source feeds into bronze with declared schemas, consistent audit columns and a quarantine path for records that do not parse.
Quality gates and reload strategy — Make the pipeline trustworthy: agree a business key per feed, enforce it, and choose and implement a load pattern per source with reasons.
Silver conformance — Join the feeds into conformed Delta tables at declared grains, resolving late scans, roster overlaps and vehicle reassignment honestly.
Gold reporting layer — Publish the three Monday answers as gold tables that need no further joining, each documented with grain, refresh and owner.
Orchestration and recovery — Schedule the whole pipeline as a parameterised workflow with alerting, then prove you can recover a failed run and backfill history.
Serving, governance and handover — Publish to Power BI, secure the platform with group-based privileges, and hand it over so another engineer can operate it.
Next course
Tell us about yourself or your team and we’ll let you know how and when you can take this course.