AWS Cloud Data Engineering
SQL, Python and Spark through S3, Glue, Athena, Kinesis, Redshift, Databricks, dbt and Snowflake to a working analytics platform
About this course
A broad, hands-on data engineering programme on AWS. It builds foundations first — warehousing concepts, SQL, Python and its data and machine learning libraries, and PySpark — then applies them across the AWS data services: S3 and IAM, Glue and Athena for batch, Lambda, Kinesis and Step Functions for event-driven and streaming work, and Redshift for warehousing.
The second half adds the lakehouse with Databricks on AWS and the medallion architecture, transformation engineering with dbt, Snowflake as a cloud warehouse and BI tools for serving. The course ends with an end-to-end capstone on a fictional mobility operator’s data.
What you’ll be able to do
- Model data for analytics and write analytical SQL with window functions and CTEs
- Write tested Python for data work and use NumPy, pandas and scikit-learn appropriately
- Build batch pipelines with S3, Glue, Athena and Step Functions, and streaming pipelines with Kinesis and Lambda
- Load, query and serve data in Redshift and Snowflake with integrations that avoid embedded keys
- Build a medallion lakehouse with PySpark and Databricks on AWS
- Engineer tested, documented transformations with dbt
- Design dashboards in QuickSight, Power BI or Tableau that answer specific questions
Before you start
What this course assumes you can already do.
- Comfort with a computer and spreadsheets, and some exposure to any programming language.
- SQL, Python and cloud concepts are taught from the start.
The curriculum
Open any module to see its objective, every teaching topic, the practice brief and the checkpoint.
Recaps and reference material are included in the LMS.
PART 1Data and Cloud Foundations
01Data Warehousing Foundations
Explain warehouse architecture, compare it with lakes and virtualisation, and design star and snowflake schemas.
- Warehouse basics, architecture and componentsConcept
- Warehouse versus lake, and warehouse versus data virtualisationConcept
- Schemas and designs: star and snowflakeHands-on
- Mapping warehouse concepts onto AWS servicesConcept
Design a trips mart
Design a dimensional model for a ride-hire business with a trips fact table and at least four dimensions, including time, zone, vehicle and customer plan. State the grain in one sentence, choose star or snowflake for each dimension with a reason, and sketch which AWS service would hold each layer of the design.
Knowledge checkpoint with explained answers
02Big Data and Cloud Computing
Describe what makes data "big", compare frameworks and serverless options, and choose cloud service models and providers.
- Big data characteristics and processing frameworksConcept
- Server versus serverless for data processingConcept
- Cloud basics and service models: IaaS and PaaSConcept
- Comparing cloud providers for data workloadsConcept
Pick a platform shape for three workloads
For three workloads — a nightly billing load, a stream of vehicle telemetry every few seconds and an analyst sandbox — describe the data characteristics, choose between provisioned and serverless processing and choose a service model. Name the AWS service you would start with for each and what evidence would make you change your mind.
Knowledge checkpoint with explained answers
03Agile Delivery and an Introduction to Generative AI
Work in a Scrum team on data work, and explain generative AI, its applications, ethics and tooling in a data engineer’s terms.
- The Agile Manifesto and its principlesConcept
- The Scrum framework and its rolesConcept
- Ceremonies and artefacts in a data teamHands-on
- AI, machine learning and generative AIConcept
- Generative AI applications and ethical considerationsConcept
- Tools and frameworks for generative AIConcept
A sprint with an AI assistant in it
Mark which items an AI assistant could help draft, what a human must verify for each, and one data type that must never be pasted into an external tool.
Knowledge checkpoint with explained answers
PART 2SQL and Python
04SQL including Advanced SQL
Build and manipulate relational data safely and write analytical, well-performing SQL.
- Relational databases, SQL and SQL Server Management StudioHands-on
- DDL, DML and TCLHands-on
- Window functions for advanced analysisHands-on
- Subqueries and common table expressionsHands-on
- Views, stored procedures and functionsHands-on
- Debugging and optimising queriesHands-on
Analytical questions on a trips table
Using a synthetic trips table of at least a million rows, answer five questions: the busiest hour per zone, week-over-week trip growth, riders active last month but not this month, a running total of revenue and the top three vehicles per zone by distance. Then take the slowest query, read its plan and make it measurably faster without changing the result.
Knowledge checkpoint with explained answers
05Python Foundations
Write readable, tested Python with sound control flow, functions, file handling and logging.
- Python basics: variables, operators, data types and modulesHands-on
- Control flow: conditions and loopsHands-on
- Functions: built-in and user-definedHands-on
- File handling, exceptions and loggingHands-on
- Testing your code with pytestHands-on
A tested log-parsing utility
Write a Python command-line utility that reads a folder of vehicle event logs, parses each line into structured records, rejects malformed lines into a separate file and writes a daily summary. Structure it as small functions in a module, handle missing or unreadable files gracefully, and write pytest tests for at least three edge cases.
Knowledge checkpoint with explained answers
06Python for Data Engineering
Use object-oriented design, the right data structures, pandas and HTTP clients to build maintainable data utilities.
- Object-oriented Python: methods, classes and packagesHands-on
- Data structures: lists, tuples, sets and dictionariesHands-on
- Lambda expressions and regular expressionsHands-on
- pandas overview for data engineeringHands-on
- Calling REST APIs with requestsHands-on
An API-to-S3 ingestion component
Write a reusable Python class that pulls records from a paginated public REST API, retries on transient errors, respects rate limits, validates fields with regular expressions and writes a Parquet file to S3 with pandas. Package it for import by other scripts, and write tests that simulate a failed page and a malformed record.
Knowledge checkpoint with explained answers
07Python Libraries for Data and Machine Learning
Use NumPy, pandas and scikit-learn for data and classical machine learning, and understand where TensorFlow and PyTorch fit.
- NumPy for numerical workHands-on
- pandas beyond the basicsHands-on
- Classical machine learning with scikit-learnHands-on
- Deep learning libraries: TensorFlow and PyTorchConcept
A feature table a data scientist can trust
From a synthetic trips dataset, build a feature table with pandas and NumPy — trips per rider per week, average distance and days since last trip — with no leakage from the future. Train a simple scikit-learn model that predicts churn from it and report precision and recall. Document each feature’s definition and the date it is computed as of.
Knowledge checkpoint with explained answers
PART 3AWS Data Engineering
08AWS Foundations for Data: S3, IAM and Boto3
Organise data in S3, choose storage classes, secure access with IAM and automate with Boto3.
- Overview of AWS services for data and analyticsConcept
- S3 basics and operationsHands-on
- S3 storage classes and lifecycle rulesConcept
- IAM for data platformsHands-on
- Boto3 and validating S3 access from PythonHands-on
A lake layout with provable access
Define two IAM roles — an ingestion role that writes only to raw and an analyst role that reads only serving — and write a Boto3 script that proves each role’s allowed and denied actions.
Knowledge checkpoint with explained answers
09Batch Pipelines with Glue, Athena and EMR
Catalogue, transform and query lake data in batch with Glue and Athena, and know when EMR is warranted.
- The Glue Data Catalog and crawlersHands-on
- Glue ETL jobs and job bookmarksHands-on
- Orchestrating batch pipelines with Glue workflowsHands-on
- Querying the lake with AthenaHands-on
- EMR for large-scale SparkDemonstration
A nightly batch pipeline that only does new work
Build a Glue workflow that crawls a raw prefix, runs a bookmarked job to convert new JSON files to partitioned Parquet and updates the catalog. Query the result with Athena and record scanned bytes before and after partitioning. Add a new day of files, rerun and prove only the new files were processed.
Knowledge checkpoint with explained answers
10Serverless and Streaming: Lambda, Kinesis and Step Functions
Build event-driven and streaming pipelines with Lambda and Kinesis, and orchestrate multi-step processes with Step Functions.
- Lambda functions and invocation patternsHands-on
- Kinesis basicsConcept
- Building a streaming pipeline with KinesisHands-on
- Orchestrating with Step FunctionsHands-on
Telemetry in, alerts out
Stream synthetic vehicle telemetry into Kinesis, process it with Lambda to flag low-battery vehicles and land all events in S3 by hour. Build a Step Functions state machine that runs every hour, aggregates the landed data with Athena and publishes a summary. Inject duplicate and late events and show how your design handles them.
Knowledge checkpoint with explained answers
11Amazon Redshift
Load, query and serve data from Redshift using its core features, clients, BI tools and automation interfaces.
- Redshift core components and featuresConcept
- Loading data into RedshiftHands-on
- Querying with the query editor and SQL clientsHands-on
- Connecting BI toolsHands-on
- Automating with the CLI and APIsHands-on
A reporting schema that performs
Load a month of synthetic trips and a zone dimension into Redshift from S3 with an IAM role. Choose distribution and sort keys for the fact table, justify them, and compare a key report query’s runtime against a naive design. Automate the daily load with the Data API from a script and connect a BI tool to a reporting view.
Knowledge checkpoint with explained answers
PART 4Spark and the Lakehouse
12PySpark
Explain how Spark executes work and write efficient PySpark that reads, transforms and writes data at scale.
- Spark cluster architectureConcept
- SparkSession, configurations and RDDsHands-on
- DataFrame transformations, functions and Spark SQLHands-on
- File formats: ORC, Parquet, Avro and JSONHands-on
- Partitioning, shared variables, caching and broadcastHands-on
- Debugging and performanceHands-on
Make a slow Spark job fast, and prove it
Take a provided PySpark job that joins a large trips table to a small zone table, groups by hour and writes JSON. Profile it in the Spark UI, then improve it by broadcasting the small table, fixing partitioning, removing an unnecessary shuffle and writing Parquet. Record the runtime and shuffle size before and after each change.
Knowledge checkpoint with explained answers
13Medallion Architecture
Design and implement bronze, silver and gold layers with clear contracts, and explain what each layer is for.
- Medallion architecture, and why lakes and warehouses both matterConcept
- Bronze: landing raw data faithfullyHands-on
- Silver: cleaning and conformingHands-on
- Gold: aggregated, consumer-ready tablesHands-on
- Use cases for each layerConcept
- Implementing medallion architecture in DatabricksHands-on
Three layers with contracts
Build bronze, silver and gold layers for a synthetic trips feed that includes duplicates, invalid timestamps and late-arriving trips. Write a one-paragraph contract for each layer, quarantine failing rows in silver and build one gold table for trips and revenue by zone and hour. Re-run the full flow and prove nothing duplicates.
Knowledge checkpoint with explained answers
14Databricks on AWS
Use the Databricks workspace on AWS to ingest, transform, visualise and schedule data with Delta Lake.
- Databricks overview, benefits and workspaceConcept
- Notebooks and Apache Spark on DatabricksHands-on
- Ingestion, transformation and Delta LakeHands-on
- Collaboration, visualisation and BI integrationHands-on
- Jobs: creating, scheduling, monitoring and debuggingHands-on
A scheduled lakehouse job on AWS
Build a three-task Databricks job that incrementally ingests trip files from S3 into Delta, transforms them into silver and gold, and refreshes a summary used by a BI tool. Configure retries, a failure alert and job compute with justified settings. Break the input once, debug the failed run from its logs and document the fix.
Knowledge checkpoint with explained answers
PART 5Transformation, Warehousing and BI
15Transformation Engineering with dbt
Structure, test, document and extend dbt projects with incremental models and macros.
- dbt project structureConcept
- Models, sources, tests and documentationHands-on
- Advanced techniques: incremental models, snapshots and packagesHands-on
- Extending dbt with your own and third-party macrosHands-on
A tested dbt project over the trips data
Build a dbt project over the trips data with sources, staging models, an incremental fact model and a snapshot of the rider plan table. Add at least eight tests, including uniqueness, relationships and one custom test, and write one macro that standardises zone codes. Generate the documentation site and show the lineage of your main mart.
Knowledge checkpoint with explained answers
16Snowflake
Load, program, optimise and integrate Snowflake with AWS, including Snowpark and stored procedures.
- Snowflake architectureConcept
- Loading data with COPYHands-on
- Working with unstructured dataHands-on
- Snowpark for PythonHands-on
- Stored procedures in Python and JavaScriptHands-on
- Performance optimisationHands-on
- Cloud integration with Amazon S3Hands-on
Load, transform in place and tune
Load a month of trip files from S3 into Snowflake through a storage integration. Write a Snowpark job that builds a daily zone summary and a Python stored procedure that logs each run. Read the query profile of the slowest step, change one thing, and show the effect on runtime and credits.
Knowledge checkpoint with explained answers
17BI and Data Visualisation on AWS
Choose a BI tool, connect it to several sources and design charts and dashboards that answer specific questions.
- Choosing a BI tool: QuickSight, Power BI and TableauConcept
- Pulling data from various sourcesHands-on
- Rendering graphs and charts that answer a questionHands-on
- Building a dashboard in Amazon QuickSightHands-on
An operations dashboard with a point
Build a dashboard for a fleet operations manager that answers three questions: where demand is unmet, which vehicles need maintenance soon and how utilisation is trending. Source data from at least two of Athena, Redshift and Snowflake. Add row-level security by city, and test each chart by asking a colleague what it tells them.
Knowledge checkpoint with explained answers
PART 6Capstone
18Capstone Project
Plan, build, govern and present an end-to-end AWS data platform for a fictional mobility operator.
- Scoping the problem and designing the platformHands-on
- Building batch and streaming ingestionHands-on
- Modelling, serving and securing the dataHands-on
- Presenting and handing overHands-on
A capstone plan you can defend
Before building, write the capstone plan: the three business questions, the sources and their quality issues, the target model, the AWS services for each layer with reasons, the IAM design and how you will test the data. Present it to a peer team for five minutes, record their strongest challenge and revise the plan in response.
Knowledge checkpoint with explained answers
Corvane Mobility fleet and trips analytics platform
Corvane Mobility is a fictional shared e-bike and scooter operator in three cities. You receive synthetic trip records, a live stream of vehicle telemetry, maintenance logs and a rider table whose plans change over time. Build an AWS data platform that answers what the operations team asks daily: utilisation by zone and hour, vehicles due for maintenance, and revenue per trip by rider plan. All data and the organisation are synthetic.
Design and secure foundations — Agree the questions, model and architecture, and set up a secure lake.
Batch ingestion and catalog — Land trips, maintenance and rider data reliably with incremental reruns.
Streaming telemetry — Process vehicle telemetry as it arrives and land it for analysis.
Tested models and marts — Transform the data into tested, documented marts that answer the three questions.
Serving, dashboard and handover — Serve the answers securely and hand the platform over.
Next course
Tell us about yourself or your team and we’ll let you know how and when you can take this course.