Week 07 — Evals I — error analysis, LLM-as-judge¶

At a glance

Phase 2: RAG + Evals · Starts: Mon 09 Nov 2026 · Planned: 16h 45m

Build: Capstone: eval harness — error analysis on 100 traces, failure taxonomy, LLM-as-judge aligned to labels

By day¶

System Design · 45 min · Partitioning: key-range vs hash, consistent hashing with vnodes, hot keys and rebalancing → Partitioning & sharding, consistent hashing w07-sd-1
DSA · 25 min · Copy List with Random Pointer → Linked list · resource w07-dsa-1
Python · 20 min · Rep: descriptors — build a validated field descriptor; how properties are descriptors → Decorators, descriptors & metaclasses · resource w07-py-1
Communication · 30 min · Grammar: Relative clauses and reported speech → Week 07 drills · day 1 w07-comm-1

Agentic AI · 60 min · Read Hamel & Shreya evals FAQ: error analysis → open/axial coding → failure taxonomy; apply to 50 capstone traces → Evals I: error analysis, LLM-as-judge, eval-driven development · resource w07-ai-1
DSA · 30 min · Add Two Numbers + Find the Duplicate Number → Linked list · resource w07-dsa-2
Communication · 30 min · Vocabulary: Persuasion words: compelling, tangible, rationale → Week 07 drills · day 2 w07-comm-2

DSA · 25 min · LRU Cache (hash map + doubly linked list) → Linked list · resource w07-dsa-3
Architecture · 45 min · DDD tactical: aggregates, invariants, value objects, domain events — model the Incident aggregate → DDD tactical: aggregates, entities, value objects, domain events w07-arch-1
Python · 20 min · Rep: message bus dispatching domain events to handlers (Cosmic Python ch8) → Architecture patterns in Python (repository, UoW, message bus) · resource w07-py-2
Communication · 30 min · Speaking drill: Explaining technical ideas to non-technical people → Week 07 drills · day 3 w07-comm-3

Agentic AI · 60 min · LLM-as-judge: binary pass/fail judges, measure judge TPR/TNR against 40 hand labels; iterate judge prompt → Evals I: error analysis, LLM-as-judge, eval-driven development · resource w07-ai-2
Java/Spring AI · 30 min · Spring AI evaluation: RelevancyEvaluator / FactCheckingEvaluator on the Java RAG path → RAG with Spring AI & pgvector w07-java-1
Communication · 30 min · Idioms & phrasal verbs: Teamwork: pull your weight, have each other's back → Week 07 drills · day 4 w07-comm-4

System Design · 45 min · Consistency models map: linearisable, sequential, causal, eventual; CAP vs PACELC — use Jepsen consistency pages → Consistency models, CAP & PACELC · resource w07-sd-2
DSA · 25 min · Merge K Sorted Lists → Linked list · resource w07-dsa-4
Python · 20 min · Rep: testcontainers-python spinning pgvector for integration tests → Testing: pytest, fixtures, Hypothesis, testcontainers w07-py-3
Communication · 30 min · Writing: Technical explanations with analogies → Week 07 drills · day 5 w07-comm-5

Agentic AI · 120 min · Capstone build: eval harness (pytest-style) with golden set, retrieval metrics (recall@k, MRR) + generation judge; run on RAG v0 vs hybrid → Evals I: error analysis, LLM-as-judge, eval-driven development · resource w07-ai-3
DSA · 30 min · Reverse Nodes in k-Group → Linked list · resource w07-dsa-5
Java/Spring AI · 30 min · Testcontainers (pgvector) integration test for the Spring AI RAG endpoint with a fixed 10-query assertion set → RAG with Spring AI & pgvector w07-java-2
Communication · 30 min · Record & shadow: Explaining technical ideas to non-technical people → Week 07 drills · day 6 w07-comm-6

System Design · 45 min · Written design: news feed / timeline — fan-out on write vs read, celebrity problem, ranking, cache layers → News feed / timeline w07-sd-3
Architecture · 30 min · Write ADR 007 for capstone: Incident aggregate boundaries and domain events (IncidentTriaged, TicketCreated) → DDD tactical: aggregates, entities, value objects, domain events · resource w07-arch-2
Staff+ · 30 min · Write a 1-page exec summary of RAG v0 vs hybrid results (decision, numbers, risk, ask) for a non-technical stakeholder → Communicating with executives & stakeholders w07-staff-1
Communication · 30 min · Soft skills + weekly review: Influencing without authority → Week 07 drills · day 7 w07-comm-7
Review · 60 min · Explain-it-back: consistent hashing + consistency models in 3 min each; flashcards on eval terms (TPR/TNR, axial coding) → Consistency models, CAP & PACELC w07-rev-1