Cracking the ML System Design Interview: A Repeatable Framework with 3 Worked Examples

Why this round is different

Unlike coding rounds with a right answer, ML system design is open-ended: 'Design a recommendation system for our app.' It tests whether you can take a vague business goal and turn it into a concrete, end-to-end ML system — data, features, model, serving, evaluation, and iteration. It's where mid and senior offers are won or lost.

The mistake candidates make is diving straight into model architecture. Strong candidates spend the first five minutes on the problem framing, because that's what the interviewer is actually grading.

The 8-step framework

1. Clarify the problem & objective. What business metric matters? What's the scale and latency budget?
2. Frame it as an ML problem. Classification, ranking, regression? What's the label?
3. Data & features. What data exists, how is the label generated, what features and how are they served?
4. Model choice. Start simple (a baseline), then justify complexity.
5. Training. Splits (time-based?), imbalance, retraining cadence.
6. Evaluation. Offline metrics that align with the business metric, plus an online A/B plan.
7. Serving. Batch vs. real-time, latency, feature store, caching.
8. Monitoring & iteration. Drift, feedback loops, guardrails.

Memorise these eight headings; the full framework with detail lives on the ML System Design page.

Worked example 1 — A recommendation system

Clarify: recommend items to maximise long-term engagement, not just clicks; serve in <100ms at millions of requests. Frame: a ranking problem over candidate items. Data: user-item interactions, item metadata, context. Approach: two-stage — a cheap candidate generator (embeddings / collaborative filtering) then a heavier ranker (gradient boosting or a neural ranker) on the top few hundred candidates.

Evaluation: offline NDCG/recall@k that correlates with online engagement, then an A/B test on the real metric. Serving: precompute candidate embeddings, rank in real time with a feature store. Monitoring: watch for feedback loops and popularity bias.

Worked example 2 — Fraud detection

Clarify: flag fraudulent transactions in real time; the cost of a false negative (missed fraud) far exceeds a false positive, and classes are extremely imbalanced. Frame: binary classification with a heavy class imbalance and a strict latency budget.

Data & features: transaction attributes, velocity features (count/sum over recent windows), device and location signals. Model: gradient boosting as a strong baseline; calibrate probabilities. Evaluation: precision-recall and cost-weighted metrics, not accuracy. Serving: a real-time feature store for velocity features and a low-latency model service. Monitoring: fraud patterns shift fast, so drift detection and frequent retraining are essential.

Worked example 3 — A RAG knowledge assistant

Clarify: answer employee/customer questions grounded in a document corpus, with citations and low hallucination. Frame: retrieval + generation, evaluated on faithfulness and answer quality. Data: the document corpus, chunked and embedded.

Approach: hybrid retrieval (keyword + vector) with a re-ranker, then a generation step with citation. Evaluation: an offline eval set scored for faithfulness/relevance (LLM-as-judge + human spot-checks), then online deflection rate. Serving: cache frequent queries, route simple questions to cheaper models. Monitoring: track hallucination reports and cost-per-query.

Signals interviewers reward

• Starting simple. Proposing a baseline before a fancy model shows maturity.
• Tying offline metrics to the business metric. The most common gap in weak answers.
• Thinking about serving and latency, not just modelling.
• Naming failure modes — drift, feedback loops, leakage — unprompted.
• Driving the conversation with structure instead of waiting to be asked.

How to practise

Pick five prompts (recommendations, fraud, search ranking, churn prediction, a RAG assistant) and talk through all eight steps out loud, ideally to another person or a recording. The goal is fluency with the framework so that under pressure you have a structure to fall back on.

Work the case studies on the ML System Design page, and make sure your underlying algorithm knowledge is solid via the ML Algorithms guide so you can defend every model choice.

Your prep checklist

1. Memorise the 8-step framework cold.
2. Prepare baselines for the five canonical problems.
3. For each, know one offline metric that aligns with the business goal and your A/B plan.
4. Rehearse naming serving constraints and failure modes unprompted.

Do this and the most feared round becomes the one where you stand out.