AI System Design¶
Interview-grade and production-grade design of LLM-powered systems: RAG, assistants, agent platforms, gateways, serving, evaluation, document processing, search, coding agents, capacity planning, and classic ML ranking. 12 topics, about 38 hours, phases 2-5.
At a glance
Goal: design any "build an LLM-powered X" system in 45 minutes with numbers, evals and safety built in. Checkpoints: phase 2 (framework, RAG, AI search), phase 3-4 (assistant, agents, gateway, evals, capacity), phase 5 (serving, coding agent, recsys).
Reading order¶
Read top to bottom; each page's You're done when line is the exit check before moving on.
- AI system design interview framework — P0, ~2 h
- Design an enterprise RAG system — P0, ~4 h
- Design a ChatGPT-style assistant — P0, ~3 h
- Design a multi-agent platform — P0, ~4 h
- Design a multi-tenant LLM gateway — P0, ~3 h
- Design an LLM inference/serving platform — P1, ~4 h
- Design an LLM evaluation & observability platform — P0, ~3 h
- Design an intelligent document processing pipeline — P1, ~3 h
- Design AI-powered semantic search — P1, ~3 h
- Design a coding agent / code-review bot — P1, ~3 h
- Classic ML system design: recommendation & ranking — P2, ~4 h
- LLM capacity, latency & cost planning — P0, ~2 h
How AI system design interviews differ¶
| Classic system design | AI system design (2026) |
|---|---|
| Correctness is deterministic | Quality is probabilistic; you must propose evals and error analysis |
| Cost is a footnote | Tokens, GPU or API cost per request is a core NFR with a worked estimate |
| Latency = p99 in ms | TTFT, TPOT, streaming UX, multi-step agent latency |
| Threats: AuthN/Z, injection into SQL | Prompt injection (direct and indirect), tool misuse, data exfiltration, memory poisoning (OWASP LLM and Agentic Top 10) |
| Change = deploy code | Change = prompts, models, indexes, tools; each needs versioning and eval gates |
| Scale knob: replicas | Scale knobs: routing, caching (prompt/semantic), batching, quantisation |
| Interviewer probes storage and consistency | Interviewer probes context construction, retrieval quality, agent boundaries, "how do you know it works?" |
Classic skills (queues, idempotency, rate limiting, multi-tenancy, observability) remain the substrate; spend roughly a third of the interview there and the rest on the AI-specific parts. Start with the framework.
Topics¶
| Topic | Priority | Complexity | Phase | Hours |
|---|---|---|---|---|
| AI system design interview framework | P0 | ⅖ | 2 | 2 |
| Design an enterprise RAG system | P0 | ⅘ | 2 | 4 |
| Design AI-powered semantic search | P1 | ⅘ | 2 | 3 |
| Design a ChatGPT-style assistant | P0 | ⅘ | 3 | 3 |
| Design a multi-agent platform | P0 | 5/5 | 3 | 4 |
| Design a multi-tenant LLM gateway | P0 | ⅘ | 4 | 3 |
| Design an LLM evaluation & observability platform | P0 | ⅘ | 4 | 3 |
| Design an intelligent document processing pipeline | P1 | ⅗ | 4 | 3 |
| LLM capacity, latency & cost planning | P0 | ⅗ | 4 | 2 |
| Design an LLM inference/serving platform | P1 | 5/5 | 5 | 4 |
| Design a coding agent / code-review bot | P1 | ⅘ | 5 | 3 |
| Classic ML system design: recommendation & ranking | P2 | ⅘ | 5 | 4 |
Also: the question bank (graded questions, scenarios and rapid-fire).
Recommended path¶
flowchart LR
F[Framework] --> R[RAG system]
R --> S[AI search]
S --> C[Chat assistant]
C --> A[Agent platform]
A --> G[LLM gateway]
G --> E[Evaluation platform]
E --> P[Capacity & cost planning]
P --> D[Document processing]
D --> V[LLM serving platform]
V --> K[Coding agent]
K --> M[Recsys / classic ML]- Phase 2: framework, RAG, AI search. Pair with the agentic-ai topics RAG fundamentals, Hybrid search & reranking and Evals I.
- Phase 3: chat assistant, agent platform. Pair with MCP, Multi-agent systems, Durable execution & HITL.
- Phase 4: gateway, evaluation platform, capacity planning, document processing. Pair with Guardrails & security and Model routing & gateways.
- Phase 5: serving, coding agent, recsys. Pair with Inference serving.
- Weeks 4/8/12/16/20/24: run a timed mock (45 minutes) from the question bank scenarios and record yourself.
Classic system design foundations: Interview framework & estimation, Caching, Rate limiting, Observability & SLOs.
Books and courses¶
| Resource | Notes |
|---|---|
| AI Engineering (Chip Huyen) | Best 2025+ coverage of evals, RAG, agents, inference; the primary text. |
| Generative AI System Design Interview (Aminian & Sheng) | Good interview-format walkthroughs (2024) but light on agents, MCP/A2A and evals; use it for structure, then supplement with this track. |
| Hello Interview: ML System Design in a Hurry | Delivery framework and classic ML design. |
| Designing Machine Learning Systems (Chip Huyen) | Data, features, deployment, monitoring; pairs with the recsys topic. |
| ML System Design Interview (Aminian & Xu) | Classic ML case studies (ranking, search, recsys). |
| Patterns for Building LLM-based Systems (Eugene Yan) | Free pattern catalogue. |
Full curated list with metadata: data/resources/ai-system-design.yml.
Practice for this chapter¶
These designs are exercised directly by the Agentic AI capstone — the gateway, eval platform and RAG topics above are what M2–M4 of that project builds.
Interview prep (25 AI-system-design prompts) lives in Staff+ → Interview prep, scored with the unified rubric.
Conventions used on these pages¶
- Estimates are order-of-magnitude and prices are assumptions as of Sept 2026; replace with measured numbers from load tests and traces.
- Framework and product status as of Sept 2026: LangGraph 1.x, Pydantic AI, MCP spec 2025-11-25 and 2026-07-28 revision, A2A v1.0, OTel GenAI semconv (Development status), Microsoft Foundry, Amazon Bedrock AgentCore, OWASP Top 10 for LLM Apps and for Agentic Applications 2026.
- Each design page follows: At a glance, Problem, Clarifying questions, Requirements, Estimation, Architecture, Component deep dives, Evaluation, Observability, Failure modes, Scaling and cost, Staff-level additions, Resources, Follow-up questions (L2-L4), Use cases, Checklist.