AI Infrastructure Engineering
Data layers, RAG, GPU platforms, model serving, MLOps, security, agents, observability and FinOps for enterprise AI
About this course
A five-phase programme for infrastructure and platform engineers who need to run AI workloads, not just call them. It starts with how models behave and what they cost, then builds the data layer for retrieval, the cloud platform and GPU compute that hosts models, the pipelines that deploy them safely, and the security and governance controls enterprise AI requires.
Later phases cover self-hosted model serving and inference optimisation, agents that automate infrastructure operations, and the observability, reliability and FinOps practices that keep AI systems healthy and affordable. Team sprints punctuate each phase, and the capstone asks you to design, deploy and defend a complete solution for one of five fictional case studies. Azure is the primary platform, with AWS equivalents covered throughout.
What you’ll be able to do
- Make compute, storage and platform decisions for AI workloads from their inference and training profiles
- Build ingestion, embedding and hybrid retrieval pipelines, and evaluate RAG systems with measured quality
- Provision secure AI platforms with infrastructure as code, private networking and managed identity
- Deploy models through CI/CD with registries, canary releases and rollback
- Serve open-weight models and tune inference against explicit latency, throughput and cost targets
- Secure AI endpoints against injection and misuse, and apply governance and privacy controls
- Operate agents, observability, SLOs and FinOps for AI systems in production
Before you start
What this course assumes you can already do.
- Working experience with cloud infrastructure, networking basics, the command line and scripting in Python.
- Kubernetes and infrastructure-as-code experience helps from phase 3 onwards.
The curriculum
Open any module to see its objective, every teaching topic, the practice brief and the checkpoint.
Recaps and reference material are included in the LMS.
PART 1Foundation and AI Fluency
01AI and ML Fundamentals for Infrastructure Engineers
Understand model architectures and workload characteristics well enough to make informed compute, storage and platform choices for AI.
- Model types and inference patterns: language, vision, speech and embedding modelsConcept
- Tokenisation, context windows and temperatureConcept
- Training versus inference compute profiles and their costConcept
- Multi-modal AI services: document intelligence, vision and speechHands-on
- API-based AI consumption patternsConcept
- Responsible AI principles for infrastructure teamsConcept
- Mapping AI service types to infrastructure requirementsHands-on
A workload profile you can plan capacity from
Pick three AI workloads — a chat assistant, a nightly document-processing batch and an embedding refresh of a large corpus. For each, estimate tokens or units per day, peak concurrency, latency requirement and whether it needs GPUs you manage. Call a hosted model for a sample of each and record real latency and usage, then recommend a consumption pattern.
Knowledge checkpoint with explained answers
02Prompt Engineering and Programmatic AI Interaction
Write, test and evaluate structured prompts for infrastructure scenarios and call models programmatically with production-grade reliability.
- Prompt anatomy: system, user and context rolesConcept
- Few-shot and chain-of-thought prompting for technical tasksHands-on
- Structured JSON output and tool use through the APIHands-on
- Prompt evaluation and iterationHands-on
- Azure OpenAI integration patterns in PythonHands-on
- Prompt injection risks and basic mitigationsConcept
- A structured prompt library and an alert-classification prototypeHands-on
Team sprint · AI fluency: prompt-driven alert triage
In a team, build a prompt-driven triage system that reads synthetic infrastructure alerts, classifies severity, suggests the owning team and explains its reasoning in structured output from a hosted model. Test it against a labelled sample, document the prompt design and API usage, and present the design rationale to another team in five minutes.
Knowledge checkpoint with explained answers
PART 2AI Data Layer and RAG Architecture
03Data Infrastructure for AI: Pipelines, Embeddings and Vector Stores
Design and operate pipelines that ingest, process, embed and store content for AI, and configure vector stores for enterprise retrieval.
- AI data pipeline architecture and design patternsConcept
- Ingestion and preprocessing: cleaning, chunking and metadataHands-on
- Embedding models and vector similarity searchHands-on
- Vector stores compared: Azure AI Search, pgvector and WeaviateConcept
- Hybrid search: keyword plus vector retrievalHands-on
- Feature stores for machine learning pipelinesConcept
- Data Factory and Blob Storage for AI data, with AWS equivalentsHands-on
An ingestion-to-retrieval pipeline over runbooks
Build a pipeline that ingests a synthetic corpus of runbooks and incident reports, chunks them two different ways, embeds them and loads them into Azure AI Search with metadata. Write twenty questions with known source passages and measure top-five retrieval hit rate for vector-only and hybrid search under each chunking strategy. Handle a deleted document correctly.
Knowledge checkpoint with explained answers
04RAG Architecture and LLM Integration Patterns
Architect, build and evaluate retrieval-augmented generation systems and integrate models into enterprise applications with proven patterns.
- RAG versus fine-tuning: a decision frameworkConcept
- Advanced RAG: query expansion, HyDE, re-ranking and contextual compressionHands-on
- Hybrid and multi-vector retrieval strategiesHands-on
- Evaluating RAG with RAGAS and TruLensHands-on
- Managed "on your data" integration for enterprise searchHands-on
- Integration patterns: synchronous, streaming and asynchronousConcept
- Multi-modal RAG and AWS equivalentsConcept
Team sprint · RAG build for an IT operations knowledge base
In a team, build a working RAG application that answers infrastructure questions from a synthetic corpus of runbooks and incident reports. Design the data pipeline and vector store configuration, tune retrieval and prompts, evaluate answer quality on a fixed question set, and estimate the monthly infrastructure cost of the data layer from current published pricing.
Knowledge checkpoint with explained answers
PART 3Platform, Deployment and Security
05Cloud AI Platform and Compute Architecture
Design and provision AI platform infrastructure — compute, networking, platform services and infrastructure as code — with cost governance built in.
- Azure Machine Learning workspaces and the Foundry platformConcept
- AKS for AI workloads: GPU node pools and autoscalingHands-on
- GPU and accelerator selection for training, inference and embeddingHands-on
- Networking and security foundations: private endpoints, VNets and managed identityConcept
- Infrastructure as code for AI stacks with Bicep and TerraformHands-on
- Platform cost governance: tagging, budgets and AdvisorHands-on
- AWS equivalents: SageMaker, EKS, GPU instances and BedrockConcept
An AI platform in code
Write infrastructure as code that provisions a machine learning workspace, an AKS cluster with an autoscaling GPU node pool that can scale to zero, private endpoints for storage and the model service, and managed identities with the roles they need. Apply tagging and a budget alert. Tear it down and rebuild it to prove the code is complete.
Knowledge checkpoint with explained answers
06AI Workload Deployment and MLOps
Deploy models to production and manage their lifecycle with CI/CD, registries, safe release strategies and rollback.
- MLOps fundamentals and model lifecycle managementConcept
- CI/CD for AI with Azure DevOps and GitHub ActionsHands-on
- Model versioning, registry and packagingHands-on
- Canary and blue-green deployments for AI endpointsHands-on
- Environments and promotion gatesConcept
- Performance monitoring, retraining triggers and rollbackHands-on
- AWS equivalents: CodePipeline, ECR and SageMaker endpointsConcept
A canary release you can roll back
Build a pipeline that packages a model in a container, pushes it to a registry, registers the version and deploys it to a staging endpoint behind an evaluation gate. Promote it to production as a ten percent canary, compare error rate and latency with the current version, and then roll back on purpose. Record how long each step took.
Knowledge checkpoint with explained answers
07AI Model Serving and Inference Optimisation
Deploy, benchmark and tune managed and self-hosted serving to meet explicit latency, throughput and cost targets.
- Managed APIs versus self-hosted servingConcept
- Deploying open-weight models on AKS GPU node pools with vLLMHands-on
- Inference servers: vLLM, Triton, TensorRT-LLM, KServe and Ray ServeConcept
- Throughput: continuous batching, KV cache and PagedAttentionHands-on
- Latency: streaming, prefix caching and speculative decodingConcept
- Serving-time quantisation and its accuracy trade-offsHands-on
- Multi-model and multi-tenant serving, routing and GPU autoscalingConcept
- Benchmarking: tokens per second, time to first token, tail latency and cost per million tokensHands-on
- AWS equivalents: SageMaker containers, Bedrock and EKS with KServeConcept
Hit a latency and cost target
Given a target of p95 latency under a stated threshold at a stated request rate, deploy an open-weight model with vLLM on a GPU node pool and benchmark it. Tune batching, cache settings and quantisation until the target is met or shown impossible, checking task accuracy after each change. Compare cost per million tokens with a managed endpoint.
Knowledge checkpoint with explained answers
08AI Security, Governance and Responsible AI
Identify and mitigate AI-specific threats and implement identity, network, privacy, safety and governance controls for enterprise AI.
- The AI threat landscape: prompt injection, data poisoning, model inversion and adversarial inputsConcept
- Identity and access for AI: managed identity, RBAC and Entra IDHands-on
- Network security: private endpoints, firewalls and API gateway controlsHands-on
- Data privacy and PII in AI pipelines: pseudonymisation and residencyConcept
- Content safety filters, fairness monitoring and explainabilityHands-on
- Governance frameworks: the EU AI Act and organisational AI policiesConcept
- Security testing for AI endpoints, including injection detectionHands-on
Team sprint · Platform, security and MLOps
Within a fixed time, a team provisions a security-hardened AI platform in code — AKS with a GPU node pool, private networking, managed identity and role assignments — and builds a CI/CD pipeline that deploys a model endpoint to it. Tagging and budgets must be in place, and the team documents its security posture and a cost estimate from current published pricing.
Knowledge checkpoint with explained answers
PART 4Agents, Reliability and Cost Engineering
09AI Agent Frameworks and Automation Operations
Design, build and deploy agents and multi-agent pipelines that automate infrastructure operations reliably and safely.
- Agent frameworks: a managed agent service, Semantic Kernel and AutoGenConcept
- Single-agent workflows with function callingHands-on
- Connecting agents to infrastructure tools with the Model Context ProtocolHands-on
- Multi-agent orchestration: supervisor, peer and hierarchicalConcept
- Agentic workflow design for infrastructure operationsConcept
- Stateful versus stateless agentsConcept
- Human-in-the-loop for high-stakes automation, and AWS equivalentsConcept
Team sprint · Agent build for alert triage
In a team, build and deploy an agent that diagnoses synthetic infrastructure incidents, queries a runbook knowledge base and recommends remediation for human approval. Choose and justify the framework, implement function calling and tool integration, add multi-agent coordination only if it helps, and test reliability against malformed and adversarial inputs.
Knowledge checkpoint with explained answers
10AI System Observability, Reliability and Cost Engineering
- Monitoring AI workloads: Azure Monitor, Application Insights and PrometheusHands-on
- SLIs, SLOs and SLAs for AI servicesConcept
- Incident response runbooks and diagnosing a latency spikeHands-on
- Performance tuning: latency, batching, caching and quantisationHands-on
- FinOps for AI: showback, chargeback, commitments and spot computeConcept
- GPU utilisation and rightsizingHands-on
- Cost anomaly detection, budgets and alertingHands-on
- Guardrails in production monitoring, and AWS equivalentsConcept
Team sprint · Ops, reliability and cost recovery
Each team receives a deliberately degraded AI system with latency problems, cost overruns and a missing alert configuration. Within a fixed time, diagnose the root causes from dashboards and logs, stabilise and optimise the system, restore alerting, and produce a FinOps cost reduction plan with an incident write-up and post-mortem.
Knowledge checkpoint with explained answers
PART 5Architecture Communication and Capstone
11AI Infrastructure Architecture and Stakeholder Communication
Structure and communicate AI infrastructure decisions to technical and non-technical audiences, and produce architecture documentation others can act on.
- Structuring a technical narrative for mixed audiencesConcept
- Presenting infrastructure decisions and trade-offsHands-on
- Architecture documentation: the C4 model and decision recordsHands-on
- Handling objections and defending design choicesHands-on
- Written design rationale: one-pagers and decision logsHands-on
- Communication patterns for stakeholder engagementsConcept
One design, two audiences
Take your module 10 service and produce two artefacts: C4 context and container diagrams with three architecture decision records for engineers, and a one-page rationale for a budget holder that states the decision, the cost, the risks and the alternative you rejected. Present the one-pager in three minutes and answer five challenge questions from peers.
Knowledge checkpoint with explained answers
Full-stack AI infrastructure solution for Orrin Managed Cloud
Orrin Managed Cloud is a fictional managed-services provider running AI workloads for several business units. Choose one of the five applied case studies, then design, provision, deploy, secure and present a complete AI infrastructure solution for it, working from synthetic datasets. Every phase of the course contributes a component. No live production system or real organisation is involved.
Requirements and solution design — Analyse the case study’s requirements and design the complete solution before provisioning anything.
Platform provisioning and deployment — Provision the platform in code and deploy the solution end to end.
Security posture and cost governance — Demonstrate the security controls and the cost model with evidence.
Stakeholder panel presentation — Present the solution to a mock stakeholder panel and defend its decisions under questioning.
Next course
Tell us about yourself or your team and we’ll let you know how and when you can take this course.