AI Engineer
Become an AI engineer — LLM APIs, prompt engineering, RAG systems, agents, vector databases, evaluation, guardrails, and production AI applications. 18 phases from foundations to deployed AI systems.
Your Progress
Complete all 18 phases to master this track
Become an AI engineer — LLM APIs, prompt engineering, RAG systems, agents, vector databases, evaluation, guardrails, and production AI applications. 18 phases from foundations to deployed AI systems.
Phase 1: AI Engineering Landscape
Situate LLM engineering within the broader ML field and identify where AI features create real product value.
AI Engineering vs ML Engineering
How LLM-powered product work differs from classical model training workflows.
Common LLM Application Archetypes
Survey chat assistants, RAG search, extraction, summarization, and automation patterns.
Foundation Model Landscape
Compare major frontier model families, open-weight options, and their trade-offs.
Identifying Valuable AI Use Cases
Score product ideas by feasibility, risk, and measurable business impact before building.
Responsible AI Foundations
Cover privacy, bias, transparency, and human-in-the-loop defaults for AI features.
Phase 2: Python for AI Engineering
Sharpen the Python skills needed to call APIs, parse JSON, and ship tested AI services.
Python Data Structures for API Work
Use dicts, lists, dataclasses, and comprehensions to shape model inputs and outputs.
Async HTTP with Python
Call LLM APIs concurrently using asyncio, httpx, and proper timeout handling.
Environments and Dependency Hygiene
Set up reproducible virtual environments and lock dependencies for AI projects.
JSON Handling and Pydantic Models
Validate and parse model output with schemas that fail loudly on malformed data.
Testing AI Code
Write unit tests with mocked model calls plus deterministic regression assertions.
Phase 3: LLM API Fundamentals
Make your first calls to major LLM providers and control every aspect of the request.
Your First LLM API Call
Send a chat completion request and read the response envelope end to end.
OpenAI SDK Essentials
Navigate the chat completions client, parameters, and error types in the OpenAI SDK.
Anthropic SDK Essentials
Send messages to Claude models with system prompts, max tokens, and stop sequences.
Messages, Roles, and Turn Structure
Structure system, user, and assistant turns so conversations stay coherent over time.
Sampling Parameters and Determinism
Tune temperature, top_p, and seed to balance creativity, consistency, and control.
Streaming Responses
Consume server-sent events to render tokens as they arrive instead of waiting.
Phase 4: Tokenization and Context Windows
Understand how text becomes tokens and budget finite context for reliable results.
How Tokenizers Work
See how byte-pair encoding splits text into subword tokens across languages and code.
Context Window Budgeting
Allocate the prompt, history, retrieved docs, and output within the model's limit.
Estimating Token Costs
Compute input and output token spend per feature to set budgets and pricing.
Strategies for Long Inputs
Summarize, truncate, or retrieve over documents that exceed the usable window.
Phase 5: Prompt Engineering Patterns
Design, iterate, and version prompts that produce consistent, correct model behavior.
System Prompt Design
Write standing instructions that fix tone, scope, rules, and output format.
Few-Shot Example Selection
Choose representative input-output pairs that steer format, tone, and edge cases.
Zero-Shot Instruction Writing
Achieve correct outputs from clear instructions alone when examples are impractical.
Chain-of-Thought Prompting
Elicit step-by-step reasoning for math, logic, and multi-constraint decisions.
Prompt Iteration Workflow
Build a small eval set, change one variable at a time, and measure regressions.
Versioning Prompts in Codebases
Store prompts as reviewed artifacts with changelogs instead of scattered strings.
Phase 6: Structured Outputs and Tool Calling
Force machine-readable JSON from models and connect them safely to real tools.
JSON Mode Outputs
Enable provider JSON mode and handle the malformed output that still slips through.
Schema-Constrained Generation
Bind output to JSON Schema so enums, types, and required fields are enforced.
Defining Tools for LLMs
Write tool names, descriptions, and parameter schemas the model can choose correctly.
Running Multi-Step Tool Flows
Execute tool calls in a loop, feeding results back until the model finishes.
Shaping Tool Results for Models
Format, truncate, and error-handle tool output so it stays digestible downstream.
Phase 7: Embeddings and Vector Search
Turn text into vectors and measure semantic similarity for retrieval workloads.
Embedding Models Explained
Map documents and queries into dense vectors that place related text nearby.
Similarity Metrics for Vectors
Compare cosine similarity, dot product, and Euclidean distance and when each applies.
Batching Embedding Requests
Embed corpora efficiently with batches, retries, and incremental indexing jobs.
Hybrid Search Basics
Combine keyword BM25 scores with vector similarity for robust retrieval.
Evaluating Embedding Quality
Measure retrieval quality with labeled pairs, recall at k, and downstream gains.
Phase 8: Chunking and Retrieval Strategies
Split documents intelligently and retrieve the smallest set of passages that answers the query.
Fixed-Size Chunking
Split text by token or character counts with overlap to preserve context boundaries.
Semantic and Structural Chunking
Split on headings, paragraphs, or embedding breakpoints instead of raw counts.
Metadata Filtering in Retrieval
Attach structured fields to chunks and pre-filter by tenant, date, or category.
Query Rewriting and Expansion
Transform user questions into retrieval-friendly queries with synonyms and sub-queries.
Reranking with Cross-Encoders
Re-score top candidates with a reranker for precision beyond first-stage retrieval.
Fusing Multiple Retrievers
Merge keyword, vector, and reranked lists with reciprocal rank fusion.
Phase 9: RAG Architecture End-to-End
Assemble ingestion, retrieval, prompting, and citation into one reliable QA system.
RAG Pipeline Overview
Map the stages from document ingestion through retrieval to grounded generation.
Building the Ingestion Pipeline
Parse PDFs and HTML, clean text, chunk it, and upsert vectors with provenance.
Assembling Context for the Model
Order, dedupe, and trim retrieved passages to maximize usable signal per token.
Citations and Grounded Answers
Require the model to cite passage IDs and verify claims trace back to sources.
Diagnosing RAG Failure Modes
Attribute bad answers to retrieval, context, or generation and fix the right stage.
Iterating on a RAG System
Run offline evals, tune one component, and promote changes with measurable gains.
Phase 10: Vector Databases in Depth
Choose, configure, and operate a vector store that meets scale, latency, and cost targets.
Vector Search with pgvector
Store embeddings in Postgres and query them alongside your relational data.
Managed Vectors with Pinecone
Use a hosted vector database with namespaces, metadata filtering, and autoscaling.
Approximate Nearest Neighbor Indexes
Understand HNSW, IVF, and how recall trades off against query latency.
Index and Collection Operations
Manage upserts, deletes, migrations, and multi-environment vector stores safely.
Choosing a Vector Store
Compare pgvector, Pinecone, Weaviate, Qdrant, and Milvus on real requirements.
Phase 11: Memory and Conversation State
Persist conversations and give models durable recall without leaking stale context.
Managing Conversation History
Store turns, cap history length, and decide what the model sees on each request.
Summarizing Long Conversations
Roll old turns into running summaries so long sessions stay within token limits.
Semantic Memory with Vectors
Store durable facts as embeddings and retrieve relevant memories during turns.
Session Persistence Patterns
Back sessions with Redis or a database and handle expiry, resumes, and concurrency.
Evaluating Memory Quality
Test whether a system recalls the right facts and forgets what it should not keep.
Phase 12: Agent Loops and Planning
Build autonomous loops that plan, act, observe, and stop within safe boundaries.
The Core Agent Loop
Implement think-act-observe cycles that drive a model toward a task goal.
Planning Strategies for Agents
Compare ReAct, plan-then-execute, and hierarchical decomposition approaches.
Reflection and Self-Critique
Have agents grade and revise their own outputs before returning results.
Stopping Conditions and Budgets
Define iteration caps, goal checks, and token budgets so agents terminate reliably.
Error Recovery in Agent Runs
Handle tool failures, timeouts, and malformed actions without losing progress.
Phase 13: Multi-Tool Agents and MCP
Give agents broad tool access through registries and the Model Context Protocol safely.
Designing a Tool Registry
Catalog tools with schemas, versions, and ownership metadata for discovery.
Tool Selection Under Many Options
Keep selection accurate when agents face dozens of available tools.
Model Context Protocol Overview
Understand MCP clients, servers, tools, resources, and standard message flows.
Building an MCP Server
Wrap internal APIs as MCP tools with typed schemas and granular permissions.
Bounding Agent Capabilities
Apply allowlists, sandboxes, and human approval gates to dangerous actions.
Phase 14: Evaluation and Datasets
Build datasets, judges, and offline and online metrics that tell you if changes help.
Building Eval Datasets
Curate representative examples, edge cases, and golden answers for scoring.
LLM-as-Judge Techniques
Use a strong model to score outputs with rubrics, pairwise comparison, and calibration.
Offline Evaluation Harness
Automate batch runs over datasets with scoring, diffs, and CI integration.
Online Metrics for AI Features
Define live signals like thumbs-up rate, task completion, and deflection.
Regression Suites for Model Changes
Lock known-good behavior so provider model updates cannot silently break features.
Human Review Workflows
Sample traffic for expert labeling and feed corrections back into datasets and prompts.
Phase 15: Guardrails, Safety, and Prompt Injection Defense
Defend against hostile inputs, content policy violations, and unauthorized tool use.
Prompt Injection Attack Patterns
Study direct, indirect, and tool-mediated injection techniques used against LLM apps.
Input and Output Guardrails
Screen prompts and responses for banned content, PII, and injection signatures.
Content Policy Enforcement
Apply provider moderation endpoints and custom policy rules consistently.
Permissioned Tool Access
Scope tools by user identity, tenant, and action risk before the model can call them.
Red-Teaming AI Systems
Run structured adversarial exercises to surface failures before attackers do.
Phase 16: Observability, Tracing, and Cost Control
Trace every model call, watch latency and spend, and catch regressions in production.
Tracing LLM Calls
Capture spans for each prompt, retrieval, and tool call with full request context.
Metering Token Usage
Attribute token spend to features, customers, and teams for chargeback and alerts.
Profiling End-to-End Latency
Break response time into retrieval, model generation, and post-processing segments.
Budgeting AI Spend
Forecast monthly costs, set kill switches, and pick cheaper models per task tier.
Responding to AI Incidents
Handle model outages, quality regressions, and data exposure with runbooks.
Phase 17: Serving Patterns for AI Applications
Design APIs and infrastructure that stream, queue, cache, and degrade gracefully.
Streaming APIs for AI Backends
Expose SSE or WebSocket endpoints that relay token streams to browser clients.
Queue-Based AI Processing
Move slow generations onto queues with workers, retries, and job status endpoints.
Caching Model Responses
Cache deterministic results with semantic keys and invalidate on prompt changes.
Rate Limiting and Retry Policy
Protect provider quotas with client-side limits, backoff, and circuit breakers.
Model Fallback and Degradation
Chain fallback models and degraded feature modes to survive provider failures.
A/B Testing AI Features
Run controlled experiments on prompts and models with safe rollout and guardrails.
Phase 18: Production Capstone and Interview Prep
Ship a complete AI system and articulate its design in system design interviews.
Capstone: Production RAG Assistant
Build a full document assistant with ingestion, retrieval, evals, and monitoring.
AI Deployment Checklist
Verify secrets, quotas, monitoring, rollbacks, and docs before a production launch.
AI System Design Interviews
Structure answers covering requirements, architecture, evals, cost, and failure modes.
Portfolio Case Studies
Write case studies with metrics, trade-offs, and lessons from shipped AI projects.
Production Incident Drills
Practice responding to hallucinations, outages, and injection attacks under pressure.