Azure Cloud Data Engineering
SQL, Python and Spark through Data Factory, Databricks, Snowflake, Power BI and Fabric to a governed, deployed data platform
About this course
A broad, hands-on data engineering programme on Azure. It builds the foundations first — warehousing concepts, advanced SQL, Python and PySpark — and then applies them across the Azure data estate: Data Factory for orchestration, Azure Databricks and the medallion architecture for the lakehouse, Snowflake and Microsoft Fabric as analytics platforms, and Power BI for serving.
Later modules add the practices that separate a working pipeline from a production one: Unity Catalog governance, advanced Delta Lake, CI/CD and release management, and a first look at how generative AI and retrieval fit into a data engineer’s work. The course ends with an end-to-end capstone on a fictional retailer’s data.
What you’ll be able to do
- Model data for analytics, including dimensional schemas, slowly changing dimensions and incremental loads
- Write and optimise advanced SQL, and choose between relational, NoSQL and graph stores
- Write tested, well-structured Python and PySpark for data engineering
- Orchestrate batch pipelines in Azure Data Factory with parameters, control flow and incremental patterns
- Build a governed medallion lakehouse on Azure Databricks with Unity Catalog and Delta Lake
- Serve analytics through Snowflake, Microsoft Fabric and Power BI with row-level security
- Ship data pipelines through CI/CD with branching and release discipline
Before you start
What this course assumes you can already do.
- Comfort with a computer and spreadsheets, and some exposure to any programming language.
- SQL, Python and cloud concepts are taught from the start.
The curriculum
Open any module to see its objective, every teaching topic, the practice brief and the checkpoint.
Recaps and reference material are included in the LMS.
PART 1Data and Cloud Foundations
01Data Warehousing Foundations
Explain warehouse architecture, compare warehouses with lakes and virtualisation, and design dimensional schemas with the right change-handling and loading strategy.
- Warehouse basics, architecture and componentsConcept
- Warehouse versus lake, and warehouse versus data virtualisationConcept
- Schemas and designs: star, snowflake and dimensional modellingHands-on
- Slowly changing dimensions in practiceHands-on
- Data loading strategies: full, incremental and change captureConcept
Design a sales mart with history
Design a dimensional model for a retailer’s sales process with at least one fact table and four dimensions. State the grain in one sentence, choose a slowly changing dimension type for customer and product with a reason, and choose a loading strategy for each table. Implement the customer dimension as type 2 and show a query that reports sales by the customer’s region at the time of sale.
Knowledge checkpoint with explained answers
02Big Data and Cloud Computing
Describe what makes data "big", compare processing frameworks and serverless options, and choose cloud service models and providers for data workloads.
- Big data characteristics and processing frameworksConcept
- Server versus serverless for data processingConcept
- Cloud basics and service models: IaaS and PaaSConcept
- Comparing cloud providers for data workloadsConcept
Pick a platform shape for three workloads
For three workloads — a nightly finance load, a clickstream feed arriving every second and an ad-hoc data science sandbox — describe the data characteristics, choose between provisioned and serverless processing, and choose a service model. Explain which provider service you would start with and what would make you change your mind.
Knowledge checkpoint with explained answers
03Agile Delivery for Data Teams
Work in a Scrum team on data work, using its roles, ceremonies and artefacts to deliver usable increments.
- The Agile Manifesto and its principlesConcept
- The Scrum framework and its rolesConcept
- Scrum ceremonies for a data teamHands-on
- Backlogs and increments for data workHands-on
Turn a data request into a sprint
A sales director asks for "a dashboard of regional performance". Write the sprint goal, run a mock stand-up script with one blocker, and note what the review should demonstrate.
Knowledge checkpoint with explained answers
04Generative AI for Data Engineers
Explain generative AI and retrieval in a data engineer’s terms, use an AI assistant responsibly for SQL and Python, and follow an AI project lifecycle.
- AI, machine learning and generative AI: what is differentConcept
- Applications and industry use casesConcept
- Ethical considerations and responsible useConcept
- Tools and frameworks for generative AIConcept
- Generating and reviewing SQL and Python with an approved AI assistantHands-on
- The AI project development lifecycleConcept
- Retrieval-augmented generation concepts for data engineersConcept
Trust, but verify, generated SQL
Use an approved AI assistant to draft five SQL queries and two Python functions for a sample sales dataset, including one window function and one date calculation. Before accepting each, write the expected result by hand or with a simpler query, run both and record whether they match. Fix anything wrong and write three rules your team should follow when using generated code.
Knowledge checkpoint with explained answers
PART 2SQL and Python
05SQL Foundations and Data Modelling
Model and build relational tables, manipulate data safely inside transactions, and package logic as views, procedures and functions.
- Relational databases and the basics of data modellingConcept
- SQL and SQL Server Management StudioHands-on
- DDL, DML and TCLHands-on
- Transactions and ACID propertiesConcept
- Views, stored procedures and functionsHands-on
A safe order-processing procedure
Create tables for customers, products, orders and order lines with keys and constraints. Write a stored procedure that places an order, decrements stock and rolls back completely if any line would take stock below zero. Add a view of daily order value and a function that calculates line totals with tax. Prove the rollback works with a failing order.
Knowledge checkpoint with explained answers
06Advanced SQL and NoSQL
Write analytical SQL with window functions, set operations and CTEs, tune slow queries, and know when a NoSQL or graph store fits better.
- Window functions for advanced analysisHands-on
- Advanced joins and set operationsHands-on
- Subqueries and common table expressionsHands-on
- Debugging and optimising queriesHands-on
- NoSQL overview and SQL versus NoSQLConcept
- An introduction to graph databasesConcept
Answer five analytical questions and make one fast
Using a sales dataset of at least a million rows, answer five questions: top three products per region, month-over-month growth, customers who bought in January but not February, a running total of revenue and an org-chart rollup with a recursive CTE. Then take the slowest query, read its plan, and make it measurably faster without changing its result.
Knowledge checkpoint with explained answers
07Python Foundations
Set up a clean Python environment and write readable, tested Python with sound control flow, functions, file handling and logging.
- Environment setup: IDE, virtual environments and a code repositoryHands-on
- Variables, operators, data types and modulesHands-on
- Control flow: conditions and loopsHands-on
- Functions: built-in and user-definedHands-on
- File handling, exceptions and loggingHands-on
- Unit testing with pytest and debugging techniquesHands-on
A tested file-cleaning utility
Write a Python command-line utility that reads a folder of messy CSV files, standardises column names and dates, rejects rows that fail validation into a separate file, and logs a summary. Structure it as small functions in a module, handle missing or unreadable files gracefully, and write pytest tests that cover at least three edge cases.
Knowledge checkpoint with explained answers
08Python for Data Engineering
Use object-oriented design, the right data structures, pandas and HTTP clients to build maintainable data utilities.
- Object-oriented Python: classes, inheritance, methods and packagesHands-on
- Data structures and collectionsHands-on
- Lambda expressions and regular expressionsHands-on
- pandas for data engineeringHands-on
- Calling REST APIs with requestsHands-on
An API ingestion component
Write a reusable Python class that pulls records from a paginated public REST API, retries on transient errors, respects rate limits, validates fields with regular expressions and writes a clean Parquet file with pandas. Package it so another script can import it, and write tests that simulate a failed page and a malformed record.
Knowledge checkpoint with explained answers
PART 3Spark and the Lakehouse
09PySpark
Explain how Spark executes work and write efficient PySpark that reads, transforms and writes data at scale.
- Spark cluster architecture and the Spark UIConcept
- SparkSession, configurations and RDDsHands-on
- DataFrame transformations and Spark SQLHands-on
- Reading and writing ORC, Parquet, Avro and JSONHands-on
- Partitioning, shared variables, caching and broadcastHands-on
- User-defined functions and when to avoid themHands-on
- Debugging, best practices and performanceHands-on
Make a slow Spark job fast, and prove it
Take a provided PySpark job that joins a large transactions table to a small product table, uses a Python UDF and writes JSON. Profile it in the Spark UI, then improve it: replace the UDF, broadcast the small table, fix partitioning and write Parquet. Record the runtime and shuffle size before and after each change.
Knowledge checkpoint with explained answers
10Medallion Architecture
Design and implement bronze, silver and gold layers with clear contracts, and explain what each layer is for.
- Medallion architecture, and why lakes and warehouses both matterConcept
- Bronze: landing raw data faithfullyHands-on
- Silver: cleaning and conformingHands-on
- Gold: aggregated, consumer-ready tablesHands-on
- Use cases for each layerConcept
- Implementing medallion architecture in DatabricksHands-on
Three layers with contracts
Build bronze, silver and gold layers for a synthetic orders feed that includes duplicates, bad dates and late-arriving records. Write a one-paragraph contract for each layer, quarantine failing rows in silver, and build one gold table that answers weekly revenue by region. Re-run the whole flow and prove nothing duplicates.
Knowledge checkpoint with explained answers
11Azure Databricks
Use the Azure Databricks workspace, clusters, notebooks, Delta Lake and jobs to build, share and schedule data pipelines.
- Azure Databricks overview, benefits and workspaceConcept
- Clusters and notebooksHands-on
- Apache Spark on Databricks: ingestion and transformationHands-on
- Delta Lake overview, DataFrames and DatasetsHands-on
- Collaboration, visualisation and BI integrationHands-on
- Jobs and workflows: creating, scheduling, monitoring and debuggingHands-on
A scheduled, monitored Databricks job
Build a three-task Databricks job that ingests daily files into a Delta table, transforms them and refreshes a summary table used by a Power BI report. Configure retries, a failure alert and job compute with justified settings. Break the input once, debug the failed run from its logs and document the fix.
Knowledge checkpoint with explained answers
12Databricks Governance and Advanced Platform Features
Govern data with Unity Catalog, use advanced Delta Lake features, automate deployment, and understand Genie, Databricks Apps and Lakebase.
- Unity Catalog and data governanceHands-on
- Advanced Delta Lake: merge, time travel, optimisation and change data feedHands-on
- CI/CD for Databricks with bundles and Git foldersHands-on
- Natural-language analytics with AI/BI GenieDemonstration
- Building data applications with Databricks AppsDemonstration
- Lakebase: an operational database beside the lakehouseConcept
Govern, version and deploy a pipeline
Move the job from module 11 into a Unity Catalog structure with separate catalogs for development and production and group-based grants. Convert one table load to a merge-based upsert and use time travel to recover from a deliberate bad write. Package the job as an asset bundle and deploy it to both environments from a pipeline.
Knowledge checkpoint with explained answers
PART 4Azure Data Services
13Azure Fundamentals and Data Solution Design
Choose Azure storage, identity and processing services for a data solution, and design it for cost, performance and security.
- Overview of Azure services for dataConcept
- Azure Blob Storage and Data Lake Storage basicsHands-on
- Identity and access management for data platformsHands-on
- Relational and non-relational storage choicesConcept
- Batch processing and streaming analytics on AzureConcept
- Designing, implementing and optimising an Azure data solutionConcept
- Azure Synapse Analytics and where it fitsConcept
A reference design with no keys in it
Design an Azure data platform for a retailer with nightly store sales, a near-real-time web clickstream and a customer table updated by a CRM. Choose storage, processing and serving services, draw the design, and specify identities and role assignments so no service uses an account key. Add a cost note and two optimisation levers you would try first.
Knowledge checkpoint with explained answers
14Azure Data Factory Foundations
Build, parameterise, trigger and monitor Data Factory pipelines that move data between Azure stores.
- What Azure Data Factory is forConcept
- Creating a data factory, linked services and datasetsHands-on
- Building, debugging and running a pipelineHands-on
- Triggers and monitoringHands-on
- Parameters and the expression builderHands-on
- Copy activity: blob to blob, data lake, Azure SQL and MySQLHands-on
One parameterised pipeline, four destinations
Build a single parameterised Data Factory pipeline that copies a dated file from blob storage to a destination chosen by parameter: another blob container, the data lake, Azure SQL or MySQL. Use managed identity for every linked service, add a schedule trigger and a failure alert, and demonstrate a rerun after a deliberate failure.
Knowledge checkpoint with explained answers
15Azure Data Factory: Data Flows, Control Flow and Incremental Loads
Transform data with mapping data flows, drive pipelines with metadata and control flow, and implement deduplication, SCD and incremental patterns.
- Mapping data flows: aggregationHands-on
- Mapping data flows: alter row and database updatesHands-on
- Get Metadata, Filter and ForEach activitiesHands-on
- ForEach with a stored procedure, and If Condition with copyHands-on
- Removing duplicate recordsDemonstration
- Slowly changing dimensions in Data FactoryDemonstration
- Incremental loading patterns: watermarks and change captureHands-on
- Building ETL processes across Azure servicesHands-on
A metadata-driven incremental load
Build a pipeline that reads a control table listing source tables and their watermark columns, loads only changed rows for each with ForEach, logs every run with a stored procedure and updates the watermark only on success. Add a data flow that upserts one table with alter row. Simulate a late-arriving row and a failed run, and show both are handled.
Knowledge checkpoint with explained answers
PART 5Analytics Platforms and Serving
16Snowflake on Azure
Load, optimise and integrate data in Snowflake on Azure, and understand how its built-in AI features and retrieval patterns work.
- Snowflake overview and architectureConcept
- Loading data with COPYHands-on
- Working with unstructured dataHands-on
- Performance optimisationHands-on
- Cloud integration with Azure storageHands-on
- Snowflake Cortex and AI capabilitiesConcept
- Retrieval-augmented generation patterns in SnowflakeHands-on
Load, tune and ask questions in place
Load a month of synthetic sales files from Azure storage into Snowflake through an external stage with a storage integration. Build a reporting table, read its query profile and cut a slow query’s runtime or credit use measurably. Then use a built-in AI function to classify a column of customer comments and report the result by product.
Knowledge checkpoint with explained answers
17Power BI
Model, secure, publish and design Power BI reports over data from Azure, SQL Server, Excel and Snowflake.
- Power BI overviewConcept
- Loading and transforming data with Power QueryHands-on
- Data modelling and DAX basicsHands-on
- Reports, dashboards and visualisation best practiceHands-on
- Row-level securityHands-on
- Publishing to the Power BI serviceHands-on
- Connecting to Excel, Azure, SQL Server and Snowflake sourcesHands-on
A secure regional sales report
Build a Power BI model over sales data in Azure SQL and targets in an Excel file, with a star schema and five DAX measures including year-to-date and variance to target. Design a one-page report for regional managers, add row-level security so each manager sees only their region, and publish it with scheduled refresh.
Knowledge checkpoint with explained answers
18Microsoft Fabric
Explain Fabric’s OneLake architecture and use its lakehouse, warehouse, real-time and Power BI experiences together.
- Microsoft Fabric overview and OneLake architectureConcept
- Lakehouses and notebooksHands-on
- Data warehousing in FabricHands-on
- Power BI integrationHands-on
- Real-Time Intelligence with KQLHands-on
- Machine learning and Copilot in FabricConcept
One lake, three experiences
In a Fabric workspace, load a sales dataset into a lakehouse and transform it with a notebook, build a small warehouse table with T-SQL for finance, and publish a Power BI report over both. Add a streaming feed of store footfall to an eventhouse and write one KQL query that compares live footfall with yesterday’s sales.
Knowledge checkpoint with explained answers
PART 6DevOps and Capstone
19DevOps for Data Engineering
Apply Azure DevOps, Git branching, CI/CD and release management to data pipelines and platform code.
- Azure DevOps fundamentalsConcept
- Git and branching strategiesHands-on
- CI/CD concepts and implementation for data pipelinesHands-on
- Release managementConcept
Promote a pipeline through three environments
Put a Data Factory pipeline and a Databricks job in one repository with a branching strategy you justify. Build a CI/CD pipeline that validates and tests on pull request, deploys to development on merge and promotes to test and production with approvals. Include one schema change and a rollback plan in the release notes.
Knowledge checkpoint with explained answers
20Capstone Project
Plan, build, govern and present an end-to-end Azure data platform for a fictional retailer.
- Scoping the problem and designing the platformHands-on
- Building ingestion and transformationHands-on
- Serving, security and governanceHands-on
- Deploying, presenting and handing overHands-on
A capstone plan you can defend
Before building, write the capstone plan: the three business questions you will answer, the sources and their quality issues, the target dimensional model, the Azure services for each layer with reasons, the identity and access design, and the CI/CD approach. Present it to a peer team for five minutes and revise it from their challenge.
Knowledge checkpoint with explained answers
Tallowby Retail omnichannel sales and inventory platform
Tallowby Retail is a fictional home-goods retailer with forty stores and an online shop. You receive synthetic daily extracts of store sales, online orders, returns, stock levels and a customer table that changes over time. Build a governed Azure data platform that answers three questions the trading team asks every week: sales and margin by channel and region, stock cover by product, and return rates by product and customer segment. All data and the organisation are synthetic.
Design and environment — Agree the questions, model and architecture, and stand up secure environments.
Ingestion and orchestration — Land every source reliably with metadata-driven, incremental loads.
Silver and gold layers — Clean, conform and aggregate the data into trusted, consumer-ready tables.
Serving and security — Serve the answers in Power BI or Fabric with row-level security and governed access.
CI/CD, governance and handover — Deploy through a gated pipeline and hand the platform over with evidence.
Next course
Tell us about yourself or your team and we’ll let you know how and when you can take this course.