AI / ML / Data Science Take-Home Assignments

📦 Deliverables

• A trained model reported with a metric appropriate for class imbalance (PR-AUC or F1, not accuracy)
• Top churn drivers explained with SHAP or coefficients
• A written targeting recommendation with a rough ROI estimate

✅ What you'd be graded on

• Metric choice is appropriate for the class imbalance
• Drivers are correctly interpreted, not just listed by importance score
• The recommendation ties model output to a specific business action

💡 Tip

A model with great AUC but no clear 'so who do we call, and is it worth it' answer will not pass a real interview.

📦 Deliverables

• An elasticity estimate with a stated confidence range
• A specific recommended price change
• A discussion of at least one confounder (promotions, seasonality, competitor pricing)

✅ What you'd be graded on

• Elasticity is estimated with a defensible method, not a single correlation
• The recommendation includes uncertainty, not a false-precision number
• Confounders are addressed, not ignored

💡 Tip

Naive correlation between price and quantity is almost always confounded — this is the trap the exercise is testing for.

📦 Deliverables

• A model evaluated with PR-AUC and a cost-weighted confusion matrix
• A justified classification threshold, not the 0.5 default
• A discussion of resampling or class-weighting choices made

✅ What you'd be graded on

• Accuracy is not used as the headline metric
• Threshold choice is tied to a stated cost trade-off (false positive vs. false negative cost)
• Resampling/weighting decisions are explained, not just applied by default

💡 Tip

If your writeup reports accuracy as the main metric on a 0.5%-positive dataset, that alone signals a miss to most graders.

📦 Deliverables

• A time-based (not random) train/test split
• A forecast compared against a naive baseline (e.g. last-value or seasonal-naive)
• A prediction interval, not a single point estimate

✅ What you'd be graded on

• Split respects time order — no leakage from the future
• The model is shown to beat the baseline on a real metric (MAPE/RMSE), not just presented alone
• A prediction interval is included and explained

💡 Tip

A forecast without a baseline comparison tells the interviewer nothing about whether the extra modeling effort was worth it.

📦 Deliverables

• A recomputation of the claimed significance
• An assessment of practical significance, not just statistical significance
• A written verdict: trust it, or not, and why

✅ What you'd be graded on

• Correctly identifies whether the sample size supports the claimed effect
• Distinguishes statistical from practical significance
• Verdict is clearly justified, not just asserted

💡 Tip

This tests skepticism as much as statistics — a good analyst questions a suspiciously good result before acting on it.

📦 Deliverables

• A working API serving predictions (e.g. FastAPI)
• Input validation with clear error responses on bad input
• A small test suite covering the happy path and one edge case

✅ What you'd be graded on

• API design uses sensible status codes and request/response shapes
• Validation actually rejects malformed input rather than crashing
• Tests genuinely exercise the running service

💡 Tip

A model.predict() wrapped in a route with zero validation is the most common way this exercise is failed.

📦 Deliverables

• A written root-cause diagnosis
• The fix applied, with a before/after metric comparison
• A regression test that would have caught the bug

✅ What you'd be graded on

• Diagnosis identifies the true root cause, not just a symptom
• The fix measurably changes the metric in the expected direction
• The regression test is meaningful, not superficial

💡 Tip

Classic planted bugs: fitting a scaler on the full dataset before the train/test split, or leaking the target through a derived feature.

📦 Deliverables

• A one-page design document
• A clear recommendation justified on latency, cost, and data-freshness needs
• A monitoring and rollback plan

✅ What you'd be graded on

• The recommendation matches the scenario's actual latency/freshness needs
• Trade-offs are explicitly reasoned, not just asserted
• The monitoring/rollback plan is concrete, not generic

💡 Tip

The 'right' answer depends entirely on the scenario — graders are testing your reasoning, not a memorized default.

📦 Deliverables

• A working CI configuration (e.g. GitHub Actions)
• An automated data-schema check
• A metric-regression gate that fails the build on a real regression

✅ What you'd be graded on

• The pipeline actually runs and fails correctly on a deliberately regressed metric
• The schema check catches a genuine schema violation
• The gate's threshold is sensible, not arbitrary

💡 Tip

Test that your gate actually fails by deliberately breaking something — an untested CI check is worse than no check.

📦 Deliverables

• Shared feature-computation code used by both training and serving paths
• A test proving the two paths produce identical values
• A short writeup explaining what skew this prevents

✅ What you'd be graded on

• Training and serving genuinely share the same computation, not two similar copies
• The equivalence test is real and passes
• The writeup correctly explains the skew risk being solved

💡 Tip

The bug this exercise tests for is subtle: two 'equivalent-looking' implementations that quietly diverge on edge cases like time-zone boundaries.

📦 Deliverables

• A working chatbot that cites its sources
• A written explanation of the chunking/retrieval strategy chosen
• A demonstrated case where it correctly says it doesn't know

✅ What you'd be graded on

• Answers are grounded and citations point to the correct source
• Chunking choices are justified, not arbitrary defaults
• The fallback genuinely triggers on an out-of-scope question rather than hallucinating

💡 Tip

Graders will deliberately ask something not in the documents — make sure your fallback actually works before you submit.

📦 Deliverables

• A log of the 5 attack attempts and their outcomes before patching
• The defenses applied
• Re-run results showing the attacks now fail

✅ What you'd be graded on

• Attacks are genuinely distinct, not five variations of one idea
• Defenses address the actual mechanism of the attack, not just the specific wording used
• Re-test evidence is included, not just claimed

💡 Tip

A defense that only blocks the exact phrase you tested, rather than the underlying pattern, will be caught by a grader trying a slightly different wording.

📦 Deliverables

• A before/after cost-per-request comparison
• The specific optimizations applied (caching, model routing, prompt trimming, etc.)
• Evidence that output quality didn't materially regress

✅ What you'd be graded on

• The cost reduction is real and measured, not estimated
• At least one structural optimization is used, not just a smaller model swap
• Quality is actually re-evaluated after the change, not assumed unchanged

💡 Tip

Swapping to a cheaper model alone is the obvious answer — the strongest submissions also restructure the prompt or add caching.

📦 Deliverables

• The eval harness code
• A scored comparison table for both prompts
• A recommendation backed by the scores, not intuition

✅ What you'd be graded on

• The same 20 test cases are used fairly for both prompts
• Scoring is consistent and explained (exact match, rubric, or LLM-judge)
• The recommendation is evidence-based

💡 Tip

This is exactly the muscle real teams want: 'prompt engineering with evidence, not vibes.'

📦 Deliverables

• A working MCP server exposing at least one tool
• Confirmation the client discovers it with zero custom integration code
• A short writeup of what the server exposes and why

✅ What you'd be graded on

• The server correctly implements the MCP schema for tool discovery
• The tool actually executes and returns a correct result end-to-end
• The writeup demonstrates understanding of why MCP matters, not just that it works

💡 Tip

The point of this exercise is proving you understand the discovery/schema layer, not just calling an API from Python.

📦 Deliverables

• A working Dockerfile and built image
• A deployed, reachable service
• A health-check endpoint that reflects real service state

✅ What you'd be graded on

• The image builds and runs cleanly from scratch
• The service is actually reachable, not just running locally
• The health check meaningfully reflects whether the service can serve requests

💡 Tip

A health check that always returns 200 regardless of model state defeats the purpose — make it check something real.

📦 Deliverables

• A drift metric computed per feature (e.g. PSI or KS statistic)
• A dashboard or log showing drift over time
• An alert that fires when a deliberately shifted sample is fed through

✅ What you'd be graded on

• The drift metric is appropriate and correctly computed
• The alert genuinely fires on the shifted sample and not on unshifted data
• Thresholds are justified, not arbitrary

💡 Tip

Test your alert with a batch you know is shifted — an alert that never fires in testing is a bug, not a safe default.

📦 Deliverables

• A one-page rollout plan (traffic split, ramp schedule)
• A specific, measurable rollback trigger (not 'if it looks bad')
• A description of what's logged to decide the rollback

✅ What you'd be graded on

• The traffic ramp is concrete and reasonable
• The rollback trigger is a specific metric and threshold
• The plan would actually be actionable by an on-call engineer at 2am

💡 Tip

'Roll back if performance degrades' is not a plan — name the exact metric, threshold, and who/what acts on it.

📦 Deliverables

• A pinned dependency file
• Confirmed identical results across two separate runs
• Versioned artifacts for the data and the trained model

✅ What you'd be graded on

• Two independent runs produce identical (or near-identical) metrics
• Dependencies are actually pinned, not just listed loosely
• Artifacts are versioned in a way someone else could trace back

💡 Tip

Run it twice yourself, on a clean environment, before you claim it's reproducible — this is the step most people skip.

📦 Deliverables

• The top 3 cost drivers, ranked and quantified
• A specific reduction proposal for each, with an estimated savings
• One trade-off each proposal introduces (latency, reliability, or complexity)

✅ What you'd be graded on

• Cost drivers are correctly ranked by actual magnitude, not guessed
• Proposals are specific and technically sound
• Trade-offs are honestly stated, not omitted

💡 Tip

The strongest answers name the trade-off a cost cut introduces — free lunches are a red flag to experienced graders.