Databricks AI Lakehouse Engineering
Streaming pipelines, Unity Catalog governance, DevOps, MLflow and AI agents on the Databricks platform
About this course
An advanced six-module course for engineers who already know Spark and Delta Lake basics. It moves quickly through platform fundamentals and spends most of its time where production lakehouses get hard: performance at scale, streaming and declarative pipelines, Unity Catalog governance of data and AI assets, CI/CD with bundles, and the machine learning lifecycle with MLflow.
The final module builds generative AI on the platform — vector search, agents with governed tools, evaluation with LLM judges, serving and monitoring — and a capstone joins streaming data, a forecasting model and a retrieval agent for a fictional energy company. It goes beyond introductory Databricks data engineering and does not repeat it.
What you’ll be able to do
- Choose and tune Databricks compute, and diagnose Spark performance from evidence
- Build streaming and declarative pipelines with data quality expectations and incremental ingestion
- Govern data, models, functions and vector indexes with Unity Catalog, including fine-grained access and lineage
- Deliver Databricks projects through CI/CD with bundles, tests and environment promotion
- Run the machine learning lifecycle with feature tables, MLflow tracking, the Unity Catalog model registry, serving and monitoring
- Build, evaluate, deploy and monitor retrieval-grounded AI agents with governed tools
Before you start
What this course assumes you can already do.
- Working SQL and Python, and prior hands-on experience with Spark DataFrames and Delta tables, for example from an introductory Databricks data engineering course.
- Basic machine learning vocabulary helps in module 5.
The curriculum
Open any module to see its objective, every teaching topic, the practice brief and the checkpoint.
Recaps and reference material are included in the LMS.
PART 1Platform and Engineering Core
01Databricks Fundamentals and Lakehouse Concepts
Revisit the lakehouse through an AI lens and set up workspace structure and compute that suit engineering, analytics and machine learning together.
- The lakehouse revisited for AI workloadsConcept
- Control plane, compute plane and serverlessConcept
- Workspace structure: Git folders, jobs, SQL and AI surfacesHands-on
- Open formats: Delta Lake, Iceberg interoperability and Delta SharingConcept
- Choosing compute for engineering, analytics and AIHands-on
A workspace fit for AI work
Set up a workspace structure for a team building pipelines, dashboards and models: Git folders, a naming convention, compute policies for engineering and machine learning, and a serverless SQL warehouse. Write a one-page note justifying each compute choice by workload and cost, and list what each compute policy restricts and why.
Knowledge checkpoint with explained answers
02Spark, PySpark and SQL at Scale
Diagnose and tune Spark workloads, use Photon and SQL warehouses well, and build reliable Structured Streaming jobs.
- Adaptive query execution and join strategiesHands-on
- Diagnosing skew, spill and shuffle in the Spark UIHands-on
- Photon and SQL warehouses for analytical SQLHands-on
- Advanced SQL: higher-order functions and semi-structured dataHands-on
- Structured Streaming: triggers, checkpoints and watermarksHands-on
- Python UDFs, pandas UDFs and ArrowHands-on
Tune a batch job and harden a stream
Profile a skewed batch join in the Spark UI, fix the skew and measure the gain. Then build a Structured Streaming job that reads events, aggregates them in ten-minute windows with a watermark and writes to Delta. Kill and restart the stream and prove from the output that nothing was lost or duplicated.
Knowledge checkpoint with explained answers
03Delta Lake and Data Engineering Pipelines
Use advanced Delta Lake features and build declarative, quality-checked, incremental pipelines with orchestration.
- Delta internals: the log, optimisation, liquid clustering and deletion vectorsHands-on
- Merge, change data feed and slowly changing dimensionsHands-on
- Declarative pipelines: streaming tables and materialised viewsHands-on
- Data quality with expectations and quarantineHands-on
- Incremental ingestion with Auto Loader and managed connectorsHands-on
- Orchestrating multi-task jobsHands-on
A declarative pipeline with quality gates
Build a declarative pipeline that ingests smart-meter files with Auto Loader into a streaming table, applies expectations that quarantine invalid readings, and maintains a type 2 meter dimension with change data feed downstream. Orchestrate it with a job that alerts on failure. Introduce a schema change and bad records, and show how each is handled.
Knowledge checkpoint with explained answers
PART 2Governance and Delivery
04Governance, Unity Catalog and DevOps
Govern data and AI assets end to end in Unity Catalog and deliver Databricks projects through tested CI/CD.
- Unity Catalog: metastores, catalogs, external locations and volumesHands-on
- Fine-grained access: row filters, column masks and tag-based policiesHands-on
- Lineage, audit logs and system tablesHands-on
- Governing AI assets: models, functions and vector indexesHands-on
- Asset bundles and CI/CDHands-on
- Testing pipelines and promoting across environmentsHands-on
Governed and deployed by pipeline
Move the module 3 pipeline into a Unity Catalog layout with development and production catalogs, group-based grants, a row filter by region and a column mask on customer identifiers. Package the pipeline and job as a bundle, add unit and integration tests, and deploy through a CI/CD pipeline to both environments. Show lineage for one gold table.
Knowledge checkpoint with explained answers
PART 3Machine Learning and AI
05Machine Learning and MLflow
Run the machine learning lifecycle on the lakehouse: features, experiments, registry, serving and monitoring.
- Feature engineering and feature tablesHands-on
- Experiment tracking with MLflowHands-on
- Training at scale: distributed tuning and AutoMLHands-on
- The model registry: versions, aliases and lineageHands-on
- Model serving and batch inferenceHands-on
- Monitoring model quality and data driftHands-on
A forecasting model from feature to endpoint
Build a point-in-time-correct feature table for daily energy demand, track at least five training runs with MLflow and register the best model with a champion alias. Serve it on an endpoint and run batch inference for next week. Set up monitoring on the inference table and simulate drift to show an alert firing.
Knowledge checkpoint with explained answers
06Generative AI, Mosaic AI, AI Agents and Capstone
Build, evaluate, deploy and monitor retrieval-grounded AI agents with governed tools on Databricks, and plan the capstone.
- Foundation models on Databricks: pay-per-token, provisioned throughput and external modelsConcept
- Vector search over governed documentsHands-on
- Building agents with governed toolsHands-on
- Evaluating agents with MLflow and LLM judgesHands-on
- Deploying and monitoring agents: serving, tracing and guardrailsHands-on
- Choosing between custom agents, low-code builders and natural-language analyticsConcept
- Capstone planning and reviewHands-on
An evaluated field-operations agent
Index a set of synthetic maintenance manuals with vector search and register two Unity Catalog functions that look up asset history and open work orders. Build an agent that uses them, evaluate it on twenty known-answer questions with an LLM judge and a human spot-check, and deploy it with tracing. Report accuracy, cost per answer and one failure traced to its cause.
Knowledge checkpoint with explained answers
Aldermoor Energy demand and field-operations platform
Aldermoor Energy is a fictional regional electricity distributor. You receive a synthetic stream of smart-meter readings, weather files, an asset register with maintenance history and a library of equipment manuals. Build a governed Databricks platform that forecasts next-day demand by substation and gives field engineers an evaluated AI agent that answers maintenance questions from manuals and asset data. All data and the organisation are synthetic.
Governed platform and delivery — Stand up catalogs, access and CI/CD before building features.
Streaming ingestion and quality — Ingest meter readings and weather continuously with quality gates.
Demand forecasting model — Train, register and serve a next-day demand forecast.
Model monitoring
Field-operations agent — Answer maintenance questions from manuals and asset data with an evaluated, governed agent.
Next course
Tell us about yourself or your team and we’ll let you know how and when you can take this course.