Learn machine learning, with practice after every lesson
8 lessons, about 137 minutes of reading, and 24 multiple-choice questions. Each lesson names the mistake that most often costs people the interview, because that is where the hours actually go. Part of the free coding course.
What a model actually learns
BasicsA model learns a mapping from inputs to an output by looking at examples where the answer is already known. You give it a thousand past loans with the outcome attached, and it works out which combinations of income, age and history tended to precede default. That is supervised learning, and it is the overwhelming majority of machine learning done in industry — the glamorous parts of the field are a small slice of the actual work.
The distinction to be clear about is training data versus everything else. A model that has seen an example can reproduce its answer trivially, which tells you nothing. So you hold data back: train on 80%, test on the 20% the model has never seen, and judge it only on that. Anyone who reports accuracy on the training set has measured memory, not learning, and it is one of the fastest ways to fail an interview screen.
The other early idea is the baseline. Before any model, ask what a stupid answer scores. If 97% of transactions are legitimate, a model that predicts 'legitimate' every time is 97% accurate and completely useless. Knowing that number first is what stops you presenting a result that sounds impressive and beats nothing.
Syntax
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
# Hold data back BEFORE doing anything else. stratify keeps the rare class
# represented in both halves, which matters when it is only 3% of rows.
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y)
# The baseline first — what does 'always guess the common answer' score?
baseline = (y_test == y_test.mode()[0]).mean()
print(f"baseline: {baseline:.3f}") # 0.970 <- the number to beat
model = LogisticRegression(max_iter=1000).fit(X_train, y_train)
print(f"model: {accuracy_score(y_test, model.predict(X_test)):.3f}")
# 0.971 would be a FAILURE dressed as success.Key points
- Split before you explore, not after. Looking at the test set to decide which features to use leaks its information into your choices, and the score stops being honest.
- Always compute the baseline first. It converts 'the model is 97% accurate' into 'the model beats guessing by 0.1%', which is the sentence that matters.
- stratify=y keeps a rare class present in both halves. Without it a 3% class can land almost entirely in one split and the evaluation becomes meaningless.
Practice challenge
97% of transactions are legitimate. Your fraud model scores 97.1% accuracy. State the baseline, state whether the model is useful, and name a metric that would actually show its performance.
Baseline: 97% — predict 'legitimate' every time
Useful: no, it beats guessing by 0.1%
Better metric: recall on the fraud class (or precision/recall, F1, confusion matrix)Baseline: ______
Useful?: ______
Better metric: ______Show a hint
- Compute what the laziest possible model would score first
- Accuracy on rare-event problems is dominated by the common class
Check yourself
1. Why hold back a test set?
Show answer
B. Because a model can reproduce examples it has already seen, which measures memory not learning
2. Why compute a baseline before modelling?
Show answer
B. So you know what a trivial answer scores and whether the model beats it
3. Why is accuracy a poor metric for fraud detection?
Show answer
B. The common class dominates it, so predicting 'not fraud' always scores well
Overfitting, and the metric that matters
Working levelOverfitting is a model learning the noise in your training data rather than the pattern. It shows up as a large gap between training and test performance: 99% on data it has seen, 71% on data it has not. The model has effectively memorised, and memorisation does not generalise to next month's customers. Every practical decision in machine learning is a trade against this.
The tools against it are simple and worth naming: more data, fewer features, simpler models, regularisation, and cross-validation to check the score was not luck. Cross-validation splits the data several ways and averages the result, which matters because a single split can flatter a model by accident — and a candidate who mentions it unprompted signals having been burned by exactly that.
Choosing the metric is the part that separates useful work from technically correct work, and it is a business question rather than a mathematical one. Precision asks: of the cases I flagged, how many were real? Recall asks: of the real cases, how many did I catch? A cancer screen wants recall, because a missed case is fatal and a false alarm is an extra test. A spam filter wants precision, because a missed spam is an annoyance and a false positive loses somebody's job offer to the junk folder.
Syntax
from sklearn.model_selection import cross_val_score
from sklearn.metrics import classification_report, confusion_matrix
# One split can flatter you by luck. Five tells you whether it was real.
scores = cross_val_score(model, X_train, y_train, cv=5, scoring="f1")
print(f"f1 across folds: {scores.mean():.3f} +/- {scores.std():.3f}")
# The gap between these two IS the overfitting measurement:
print("train", model.score(X_train, y_train)) # 0.99
print("test ", model.score(X_test, y_test)) # 0.71 <- memorised
# Look at the confusion matrix, not just the score.
print(confusion_matrix(y_test, model.predict(X_test)))
# [[940 12] 12 false alarms
# [ 21 9]] 21 MISSED real cases — is that acceptable? business question
print(classification_report(y_test, model.predict(X_test)))Key points
- The train-minus-test gap is the overfitting measurement. Quote both numbers, never just the good one.
- Precision versus recall is a business decision about which error is more expensive, not a technical preference. Be able to say which one your problem needs and why.
- Cross-validation guards against a lucky split. A single hold-out score with no variance estimate is a number without an error bar.
Practice challenge
For each, say whether you optimise for precision or recall, and why in one line: (a) cancer screening, (b) a spam filter for a jobseeker's inbox.
a) recall — a missed case can be fatal, a false alarm costs one extra test
b) precision — a missed spam is an annoyance, a false positive sends a job offer to junka) ______ because ______
b) ______ because ______Show a hint
- Ask which error is more expensive, not which number is higher
- It is a business question, not a mathematical one
Check yourself
1. What does a large train-test gap indicate?
Show answer
B. Overfitting — it learned noise rather than pattern
2. A cancer screening model should optimise for…
Show answer
B. Recall, because a missed case is far more costly than an extra test
3. Why is repeatedly tuning against the test set a problem?
Show answer
B. Its information leaks into your choices and the score stops being honest
Getting a model into production
AdvancedA model in a notebook is not a product. Production means something calls it, gets an answer within a latency budget, and keeps working when the input looks different from what you trained on. This transition is where most machine learning projects die, and it is why the job market pays engineers who can do it more than it pays people who can only model.
The failure that defines production machine learning is drift. The world changes — prices rise, a competitor launches, a marketing campaign brings in a different kind of customer — and a model trained on last year's data quietly degrades. Nothing errors. Predictions keep arriving, they are simply worse each week. That is why monitoring input distributions matters as much as monitoring uptime.
The other thing to get right is training-serving skew: the features you compute at prediction time must match how they were computed during training. If training used a 30-day average calculated in a batch job and serving computes it live over a slightly different window, the model receives inputs it never saw and performance drops for reasons no metric explains. Feature stores exist mainly to make that one class of bug impossible.
Syntax
# Serving: same preprocessing as training, or the model gets inputs it never saw.
# Bundling the transform WITH the model is what prevents that.
from sklearn.pipeline import Pipeline
pipe = Pipeline([("scale", StandardScaler()), ("clf", LogisticRegression())])
pipe.fit(X_train, y_train)
joblib.dump(pipe, "model.joblib") # transform + model travel together
# --- serving ---
@app.post("/predict")
def predict(payload: dict):
features = build_features(payload) # SAME code path as training
proba = pipe.predict_proba([features])[0][1]
log_prediction(features, proba) # log inputs, not just outputs
return {"risk": float(proba)}
# Drift check, run on a schedule — compare live inputs to the training profile
# if psi(training_dist["income"], last_7d_dist["income"]) > 0.2:
# alert("income distribution shifted — model may need retraining")Key points
- Bundle preprocessing with the model. A pipeline saved as one object cannot drift apart from its own transformations.
- Log the input features, not only the prediction. When performance drops you need to know what the model was actually given.
- Monitor input distributions on a schedule. Drift produces no errors — predictions keep arriving and are simply wrong more often.
Practice challenge
A model launched at 91% accuracy. Six months later nobody has seen an error, no alert has fired, and the business says the predictions feel wrong. Name the likely cause and two things you should have been monitoring.
Cause: drift — the world changed and the model was trained on old data
Monitor: input feature distributions vs the training profile
Monitor: prediction outcomes against actuals once labels arriveCause: ______
Monitor 1: ______
Monitor 2: ______Show a hint
- Nothing errored because nothing is broken in a software sense
- One monitor watches the inputs, the other watches whether it was right
Check yourself
1. What is model drift?
Show answer
B. The world changing so a model trained on old data quietly degrades
2. Why bundle preprocessing with the model?
Show answer
B. So serving cannot compute features differently from training
3. Why log input features in production?
Show answer
B. Because diagnosing a performance drop needs to know what the model was given
Ship a model and know when it breaks
Job-readyA model that lives in a notebook has not been shipped. Shipping means something calls it, gets an answer inside a latency budget, and somebody finds out when it starts being wrong. That last part is the one people skip, and it is the one interviews probe hardest, because a silently degrading model is worse than no model — the business keeps trusting it.
The minimum viable production setup is smaller than it sounds: the model serialised with the exact preprocessing that produced it, behind an interface that takes raw input and returns a prediction; the training data profile stored alongside it; and a log of every prediction with its inputs. Skipping the preprocessing pairing is the classic failure — training-serving skew, where the model sees differently-scaled features in production and quietly performs worse than in your notebook.
Then decide what "broken" means before it happens. Input drift: are the features arriving still shaped like the ones you trained on? Outcome drift: when labels eventually arrive, is accuracy holding? And a fallback: what does the system do when the model is unavailable or unconfident, because "return an error" is rarely the right business answer.
Syntax
# The pairing that prevents training-serving skew:
# the transformer and the model are ONE artifact, versioned together.
from sklearn.pipeline import Pipeline
import joblib, json, datetime
pipe = Pipeline([("scale", scaler), ("model", clf)])
pipe.fit(X_train, y_train)
joblib.dump(pipe, "churn_v3.joblib")
json.dump({ # the profile drift is measured against
"version": "v3",
"trained": str(datetime.date.today()),
"n_rows": len(X_train),
"baseline": float((y_train == 1).mean()), # what beating it means
"feature_means": X_train.mean().to_dict(),
"metric": {"recall_fraud": 0.71}
}, open("churn_v3.meta.json", "w"))
# In production: log the inputs AND the prediction, or you can never
# answer "was it already drifting last month?"
log.info({"v": "v3", "features": row, "pred": float(p), "ts": now})Key points
- Serialise the preprocessing with the model as one artifact. Separating them is how training-serving skew gets in, and it degrades quality without raising an error.
- Store the training data profile next to the model. Drift is meaningless without a baseline to be measured against.
- Log inputs and predictions from day one. Without that log you can never reconstruct when the decay started.
Practice challenge
Your model scores 0.89 in the notebook and noticeably worse in production, with no errors anywhere. Name the most likely cause, the change that prevents it, and the two things you should have been logging from day one.
Cause: training-serving skew - the production path preprocesses features differently from the training path (different scaler, different missing-value handling, different column order).
Prevention: serialise the preprocessing and the model as ONE artifact - a pipeline - and version them together, so production cannot use a different transform.
Log 1: the input features of every prediction
Log 2: the prediction itself, with the model version - without both you can never establish when the gap openedLikely cause: ______
Prevention: ______
Log 1: ______
Log 2: ______Show a hint
- Nothing errored, so it is not a crash - something is quietly different between the two paths
- The fix is about what gets saved together, not about the model
Check yourself
1. What is training-serving skew?
Show answer
B. Production features processed differently from training features
2. Why store the training data profile with the model?
Show answer
B. Drift can only be measured against a baseline
3. Why log inputs alongside predictions?
Show answer
B. To reconstruct when a decay started
How the data is split, and why your 99% is fake
BasicsA model that scores 99% on the data it was trained on has told you nothing, because it may simply have memorised it. The only meaningful score comes from data the model has never seen, which is why the data is split before any training happens: a training set to learn from, a validation set to make decisions with, and a test set that you look at once, at the end. Three sets rather than two, because every time you tune something based on a score, that data has influenced the model and stops being an honest estimate.
How you split matters as much as that you split. A random split is right when rows are independent, and wrong in two common cases. With time-series data a random split lets the model learn from the future to predict the past, which cannot happen in production, so you split by time. And when rows are grouped — several visits by the same patient, several sessions by the same user — a random split puts the same entity on both sides, and the model recognises the entity rather than learning the pattern, so you split by group.
Leakage is the general name for information reaching the model that will not exist at prediction time, and it always shows up as a suspiciously excellent score. It hides in preprocessing more often than in the features: scaling or imputing on the full dataset before splitting lets the training set learn the mean of the test set. It hides in obvious columns too, like a 'refund_issued' flag when predicting fraud, or a customer's future purchase count. The habit is simple and reliable — when a score is far better than the problem should allow, look for leakage first, because that is what it almost always is.
Syntax
# THREE SETS, NOT TWO
# train the model learns from this
# validation you tune and choose using this
# test you look at this ONCE, at the very end
# Every decision made on a set spends its honesty.
from sklearn.model_selection import train_test_split
X_tr, X_tmp, y_tr, y_tmp = train_test_split(X, y, test_size=0.3, random_state=42)
X_val, X_te, y_val, y_te = train_test_split(X_tmp, y_tmp, test_size=0.5, random_state=42)
# WRONG for time series -- learns from the future
train_test_split(X, y, shuffle=True)
# RIGHT -- split at a date
train = df[df.date < "2026-06-01"]
test = df[df.date >= "2026-06-01"]
# WRONG when rows are grouped -- same patient on both sides
# RIGHT
from sklearn.model_selection import GroupShuffleSplit
# LEAKAGE HIDES IN PREPROCESSING
# WRONG: scaler sees the test set
X_scaled = StandardScaler().fit_transform(X) # <-- before the split
# RIGHT: fit on train only, inside a pipeline
from sklearn.pipeline import make_pipeline
pipe = make_pipeline(StandardScaler(), LogisticRegression())
pipe.fit(X_tr, y_tr)
# 0.99 on a hard problem is a bug report, not a result.Key points
- Use three sets. The validation set is spent by tuning, so only a test set you have looked at once gives an honest estimate of new-data performance.
- Split by time for time series and by group when rows share an entity. A random split in either case lets the model see what it could never see in production.
- Fit scalers and imputers on the training data only, inside a pipeline. Preprocessing the full dataset before splitting is the most common invisible leak.
Practice challenge
A churn model scores 0.99 AUC. The pipeline scales all features with StandardScaler before splitting, the data has several rows per customer, and one feature is days_since_last_login computed as of today. Name the three problems and give the corrected split strategy.
Problem 1: scaling before the split lets the training set absorb statistics of the test set - fit the scaler inside a pipeline on train only
Problem 2: several rows per customer with a random split puts the same customer on both sides, so the model recognises the customer rather than the pattern - split by group
Problem 3: days_since_last_login as of today uses information from after the prediction point - it must be computed as of the prediction date
Correct: GroupShuffleSplit by customer, all preprocessing inside the pipeline, features computed point-in-timeProblem 1: ______
Problem 2: ______
Problem 3: ______
Correct approach: ______Show a hint
- 0.99 on a hard problem is a bug report, not a result
- Ask of each feature whether it would exist, with that value, at prediction time
Check yourself
1. Why keep a separate test set as well as a validation set?
Show answer
B. Tuning against validation spends its honesty; test stays untouched until the end
2. Rows are multiple visits by the same patients. What split is correct?
Show answer
B. By group, so a patient appears on only one side
3. You scaled the full dataset then split. What happened?
Show answer
B. Training data absorbed statistics of the test set — leakage
Features, and the one that gives away the answer
Working levelFeature engineering is usually where a model gets good, far more than model choice. It is the work of turning raw columns into things that carry signal: a raw timestamp is nearly useless, while hour-of-day, day-of-week and is-holiday derived from it are often strongly predictive. Ratios beat raw counts because they normalise for size — spend per order tells you something that total spend and order count separately do not. Aggregates over a window give context, so 'transactions in the last hour' turns a single event into a pattern.
Categorical variables need a representation and the choice has consequences. One-hot encoding creates a column per value and is fine at low cardinality; at ten thousand values it produces an unusable matrix. Target encoding, replacing a category with the mean outcome for that category, is compact and powerful and leaks badly if computed on the full training set, so it must be done within cross-validation folds. And every encoder must have an answer for the category it has never seen before, because production will send one.
The rule that keeps all this honest is a point-in-time question you ask of every feature: would this value have been available, with this value, at the moment the prediction is made? A 'total_refunds' column when predicting fraud fails it, because refunds happen after. A customer's lifetime value fails it if computed from all history including the future. These features look brilliant in testing and are worthless in production, and the failure is silent — the model does not error, it simply performs far worse than the number you promised.
Syntax
# RAW COLUMNS ARE RARELY THE FEATURES
df["hour"] = df.ts.dt.hour
df["dow"] = df.ts.dt.dayofweek
df["is_weekend"] = df.dow >= 5
# ratios normalise for size; counts do not
df["spend_per_order"] = df.total_spend / df.order_count.clip(lower=1)
# windows turn one event into a pattern
df["txn_last_1h"] = (df.groupby("user_id")
.rolling("1h", on="ts").txn_id.count()
.reset_index(drop=True))
# CATEGORICALS
# low cardinality -> one-hot
# high cardinality -> target encoding, but ONLY inside CV folds
# unseen category -> every encoder needs an answer; production
# will send one on day one
from sklearn.preprocessing import OneHotEncoder
enc = OneHotEncoder(handle_unknown="ignore")
# THE POINT-IN-TIME TEST -- ask of EVERY feature:
# "At prediction time, would this value exist, with THIS value?"
#
# total_refunds -> refunds happen AFTER fraud. FAILS.
# lifetime_value -> computed over all history. FAILS.
# account_age_days -> known at the time. PASSES.
# txn_last_1h -> known at the time. PASSES.
#
# A feature that fails this test does not error.
# It scores beautifully in testing and is worthless in production.Key points
- Derived features usually beat model choice. Hour-of-day from a timestamp, ratios instead of raw counts, and rolling windows are where most of the gain comes from.
- Target encoding must be computed inside cross-validation folds, and every encoder needs a defined behaviour for a category it has never seen.
- Apply the point-in-time test to every feature: would this value exist, with this value, at prediction time? Anything computed with hindsight fails silently.
Practice challenge
For a model that scores a transaction for fraud at the moment it is submitted, mark each feature PASS or FAIL and give a one-line reason: account_age_days, total_refunds_on_account, transactions_in_last_hour, customer_lifetime_value, merchant_category, chargeback_flag.
account_age_days: PASS - known at the moment of the transaction
total_refunds_on_account: FAIL - refunds occur after; if computed over all history it encodes the outcome
transactions_in_last_hour: PASS - computable from data that already exists at that instant
customer_lifetime_value: FAIL if computed over all history including the future; PASS only if computed as of that date
merchant_category: PASS - a static attribute of the merchant
chargeback_flag: FAIL - a chargeback is the outcome itself, arriving weeks lateraccount_age_days: ______
total_refunds_on_account: ______
transactions_in_last_hour: ______
customer_lifetime_value: ______
merchant_category: ______
chargeback_flag: ______Show a hint
- The question is always: would this value exist, with THIS value, at prediction time
- Two of these are the outcome wearing a different name
Check yourself
1. Why is 'total_refunds' a bad feature for predicting fraud?
Show answer
B. Refunds occur after the prediction point — it is not available then
2. A categorical column has 10,000 distinct values. What is the problem with one-hot?
Show answer
B. It creates 10,000 columns, which is unusable
3. When must target encoding be computed?
Show answer
B. Within cross-validation folds, to avoid leakage
Thresholds, imbalance and the cost of being wrong
AdvancedMost classifiers do not output a class, they output a probability, and something must turn that into a decision. The default is 0.5 and it is almost never the right number. Moving the threshold trades the two kinds of error against each other: lower it and you catch more of the positives while raising false alarms, raise it and your alerts become more reliable while more cases slip through. There is no threshold that is correct in the abstract, only one that is correct given what each mistake costs.
That is why accuracy is such a poor headline on imbalanced data. If one transaction in a thousand is fraudulent, a model that predicts 'not fraud' every time is 99.9% accurate and completely useless. Precision asks what fraction of your alerts were real, recall asks what fraction of the real cases you caught, and they move in opposite directions as the threshold changes. Which one dominates is a business question: for a cancer screen a missed case is catastrophic and a false alarm means another test, while for an email filter a false positive that hides a real invoice is worse than letting some spam through.
So the threshold should be derived from cost, not chosen by convention. Assign a rough figure to a false positive and a false negative — analyst minutes, refunded fraud, a lost customer — and pick the threshold minimising total cost, and the answer often lands somewhere unintuitive like 0.12. Handle imbalance with class weights or resampling if the model is not learning the minority class at all, but note carefully that resampling distorts the predicted probabilities, so a model trained on a rebalanced set needs calibration before those probabilities mean anything.
Syntax
# 1 in 1,000 is fraud. "Predict never" = 99.9% accurate, useless.
# Accuracy is the wrong headline on imbalanced data.
from sklearn.metrics import precision_recall_curve, confusion_matrix
prob = model.predict_proba(X_val)[:, 1]
# DERIVE the threshold from cost, do not accept 0.5
COST_FP = 5 # analyst reviews a clean case: 5 minutes
COST_FN = 400 # missed fraud: average loss
best = min(
((( (prob >= t) & (y_val == 0) ).sum() * COST_FP
+ ( (prob < t) & (y_val == 1) ).sum() * COST_FN), t)
for t in [i/100 for i in range(1, 100)]
)
print(best) # (14830, 0.12) <- not 0.5
# PRECISION vs RECALL -- they move in opposite directions
# threshold 0.5 precision 0.91 recall 0.34 <- misses most fraud
# threshold 0.12 precision 0.44 recall 0.87 <- 2x the alerts,
# catches most of it
# WHICH ERROR IS WORSE IS A BUSINESS QUESTION
# cancer screen -> recall (a miss is catastrophic)
# spam filter -> precision (a hidden invoice is worse than spam)
# If the model ignores the minority class entirely:
LogisticRegression(class_weight="balanced")
# NOTE: resampling/weighting distorts predicted probabilities.
# Calibrate before treating them as probabilities.
from sklearn.calibration import CalibratedClassifierCVKey points
- 0.5 is a default, not a decision. Derive the threshold from the cost of each error type and it frequently lands nowhere near the middle.
- Accuracy is meaningless under imbalance — predicting the majority class always can score 99.9%. Use precision and recall, and say which one the business is optimising.
- Resampling and class weights change the predicted probabilities. If those probabilities are used downstream, calibrate the model afterwards.
Practice challenge
A fraud model outputs probabilities. A false positive costs 5 minutes of analyst time; a missed fraud costs about 400. At threshold 0.5 precision is 0.91 and recall is 0.34; at 0.12 precision is 0.44 and recall is 0.87. Say which threshold to use and why, and explain why accuracy is the wrong metric when 1 in 1,000 rows is fraud.
Threshold: 0.12
Why: a missed fraud costs 80 times a false positive, so recall dominates. Moving from 0.34 to 0.87 recall catches most of the fraud, and the extra false positives cost analyst minutes rather than losses.
Why accuracy is wrong: predicting 'not fraud' for everything scores 99.9% accurate and never catches a single case - accuracy is dominated by the majority class and conceals exactly the failure the model exists to preventThreshold: ______
Why: ______
Why accuracy is wrong here: ______Show a hint
- Compare the two error costs before looking at the metrics
- 0.5 is a default, not a decision
Check yourself
1. 1 in 1,000 rows is positive. A model predicts negative always. Its accuracy?
Show answer
B. 99.9%, and it is useless
2. Lowering the decision threshold does what?
Show answer
B. Raises recall, lowers precision
3. You trained with class_weight='balanced' and want to use the probabilities. What is needed?
Show answer
B. Calibration — weighting distorts predicted probabilities
Explaining a model to whoever is accountable for it
Job-readyA model that nobody senior understands does not get deployed, or worse, gets deployed and then withdrawn the first time it is questioned. The person accountable is not asking about architecture; they are asking what it does, how often it is wrong, what happens when it is wrong, and who is responsible then. Answering in terms of AUC does not address any of those. Answering in terms of 'it flags about 40 cases a day, roughly 4 in 10 are genuine, and a missed case costs us around 400' does.
Global and local explanations answer different questions and you will be asked both. Globally: which features drive the model overall, which permutation importance answers honestly by measuring how much performance drops when a feature is shuffled. Locally: why this particular case was flagged, which SHAP values answer by attributing a prediction to its features. The local one matters most in practice, because that is what a customer disputes, what a regulator asks about, and what an analyst needs in order to act on an alert rather than merely receive it.
Then there is the question of whether to train a model at all, which has changed. For text tasks — classification, extraction, summarisation — a general-purpose language model with a good prompt is often adequate immediately, with no training data, and is the sensible baseline to beat. A trained model wins when you need low latency, low unit cost at volume, stable behaviour that does not change under you, or a decision you must be able to explain and defend. Deciding that deliberately, and being able to say why, is more valuable than any modelling technique.
Syntax
# WHAT THEY ARE ACTUALLY ASKING
# "What does it do?" -> flags ~40 transactions/day for review
# "How often is it wrong?" -> ~4 in 10 flags are genuine (precision .44)
# ~13% of real fraud is missed (recall .87)
# "What happens when it is?" -> FP: 5 analyst minutes
# FN: ~400 average loss
# "Who is responsible?" -> a human approves every block;
# the model never acts alone
# Do NOT open with AUC 0.94.
# GLOBAL: which features drive it? (honest, model-agnostic)
from sklearn.inspection import permutation_importance
r = permutation_importance(model, X_val, y_val, n_repeats=10)
# LOCAL: why THIS case? -- what a customer disputes and a
# regulator asks about
import shap
sv = shap.TreeExplainer(model).shap_values(X_val)
# "Flagged because: 14 transactions in 1h (+0.31),
# new device (+0.22), amount 6x the user's median (+0.18)"
# SHOULD YOU TRAIN A MODEL AT ALL?
# LLM + a good prompt wins when: text, no labelled data,
# requirements still moving, volume modest
# A trained model wins when: low latency, low unit cost at
# scale, stable behaviour, an explanation you must defend
#
# Baseline first. "We beat the simple approach by X" is a result;
# "we built a model" is not.Key points
- Translate metrics into consequences: alerts per day, share that are genuine, cost of each error type. That is the language in which the decision is actually made.
- Permutation importance answers 'what drives the model'; SHAP answers 'why this case'. The local explanation is what disputes, regulators and analysts require.
- An LLM with a good prompt is a legitimate baseline for text tasks. Train a model when you need latency, unit cost, stability or a defensible explanation — and say which.
Practice challenge
You are presenting the fraud model to the operations director who must sign off. Rewrite 'AUC is 0.94, F1 is 0.58' into the four things they actually need, using: 40 flags/day, precision 0.44, recall 0.87, FP costs 5 analyst minutes, FN costs 400. Then name which technique explains a single flagged transaction.
What it does: flags about 40 transactions a day for human review
How often wrong: roughly 4 in 10 flags are genuine fraud, and it misses about 13% of real fraud
What happens: a false flag costs about 5 minutes of analyst time; a missed one costs about 400 on average
Who is responsible: a human approves every block - the model never acts on its own
Technique: SHAP values, which attribute one prediction to its features (permutation importance is global, not per case)What it does: ______
How often it is wrong: ______
What happens when it is: ______
Who is responsible: ______
Technique for one case: ______Show a hint
- They are not asking about architecture - they are asking what they are signing off on
- Local versus global explanation is the distinction
Check yourself
1. Which explains a single prediction?
Show answer
B. SHAP values
2. How should model performance be presented to a business owner?
Show answer
B. Alerts per day, share genuine, and the cost of each error type
3. When does a trained model beat an LLM prompt for a text task?
Show answer
B. When you need low latency, low unit cost at volume, or a defensible explanation
Common questions
Do I need any background to start Machine Learning?
No. This track begins at its own beginning and assumes nothing, and the first lesson explains what the thing is before showing you any syntax.
How long does the Machine Learning track take?
About 137 minutes of reading across 8 lessons, plus the practice challenges and 24 multiple-choice questions, which is where the time actually goes.
Is it free?
Yes, and there is no account. Everything runs in your browser.
More: all 15 tracks · what employers actually ask for · the full syllabus