What the letters mean

Imagine a spreadsheet of Lincoln plating jobs. Every row is a job. Every column is something you measured about that job. There's one column you care about predicting. That's all four letters.

n — the number of rows. How many jobs are in your dataset. If you have 50 plating jobs from last quarter, n = 50.

p — the number of columns (other than the target). How many things you measured per job. Bath temperature, line speed, surface prep score, plating thickness, age of solution, operator skill, customer tier, part complexity. p = 8.

X — the whole table of inputs. Capital X = matrix. n rows by p columns.

y — the column you want to predict. Lower-case y = a single column. For each row, one value.

Quantitative vs categorical

One more fork before you can read anything in the field. The type of y decides which family of algorithms you need.

Quantitative — y is a number. Plating yield (87%), thickness (28 microns), cycles since service (120). Use regression methods.

Categorical — y is a label. PASS or FAIL. Customer tier A, B, or C. Use classification methods.

You'll see "regression vs classification" mentioned constantly. This is what they mean.

Reducible vs irreducible error

Your model's total error has two parts.

Reducible error — improves when you pick a better model or add better features. This is the part you fight against. Most of ML practice is shrinking it.

Irreducible error — locked in by the world. Random measurement noise, things you can't observe, factors that vary day-to-day. Lincoln framing: even a perfect model can't predict scrap rate to the decimal — vibration, humidity, operator focus, micro-power-fluctuations all carry noise no spreadsheet captures.

Knowing the difference saves your sanity. When your model isn't getting better, you have to ask: am I fighting reducible error (try harder) or irreducible error (this is the floor)?

Two reasons we estimate f

Same equation, two very different goals. Knowing which goal you have changes which model you should choose.

Parametric — pick the shape, fit the numbers

Linear regression is the classic example. You assume f is a straight line. Now you only have two numbers to find: slope and intercept. Cheap, fast, easy to explain.

The tax: if the truth is curved and you assumed a line, you'll always be a little wrong. Doesn't matter how much data you throw at it. Your assumption is the ceiling.

Non-parametric — let the data lead

No assumed shape. The fit can wiggle as much as it needs to follow the data. KNN, decision trees, splines — all non-parametric.

The tax: you need a lot more data to get a stable answer. With 30 jobs you can fit a straight line confidently. With 30 jobs and a wiggly non-parametric model, you're chasing noise.

The flexibility / interpretability tradeoff

Here's the rule that matters in practice: more flexible models are harder to explain.

A linear regression gives you a one-line answer: "yield rises 0.4% for every degree increase in bath temp." Your boss can defend it. A wiggly non-parametric fit gives you "the model says so." Your boss is unlikely to defend that.

Choose your model based on who's asking and what they need. Sometimes accuracy wins, sometimes explainability wins. There's no universal right answer.

MSE — Mean Squared Error

The standard score for a regression model. For each prediction, take (prediction − actual), square it, then average across all predictions.

Why squared? So big misses count more than small ones. A prediction that's off by 10 is much worse than ten predictions off by 1. The square makes that bite.

Lower MSE = better. Period. But: which data did you measure it on? That's the whole question.

The train/test split

Hold out 20% of your data before you train. Train the model on 80%. Score it on the 20% it has never seen. That score is your honest estimate of how it'll do in the real world.

Anything else is cheating. If the model has seen the data during training, asking it to predict that data is like giving a student the answer key during the exam. They'll ace it. They'll teach you nothing.

Overfitting — the trap

A super-flexible model can memorize the training data. It can fit every noisy point exactly. Train MSE → near zero. The model looks brilliant. You ship it.

Then production data arrives. The same model is suddenly catastrophic. Why? Because it learned the noise, not the signal. Train MSE was a lie.

The only honest metric is the one measured on data the model hasn't seen. That's the rule. Memorize it.

Bias — error from being too simple

Bias is what happens when your model can't bend enough to follow the truth. You assume defect rate is a flat line, but it's actually a curve. No matter how much data you give it, the line will always be wrong in the middle. That's bias.

High bias = the model is too rigid. It misses real patterns. The error is built into the assumption.

Variance — error from being too clingy

Variance is what happens when your model bends too eagerly. It hugs every data point, including the noisy ones. Re-train on a slightly different sample and the fit changes wildly. That instability is variance.

High variance = the model is too sensitive. It's chasing noise. Predictions on new data are unstable.

The U-shape

As you increase a model's flexibility, two things happen at once. Bias drops (more flexible models can follow the truth). Variance rises (more flexible models chase noise).

Total error = bias² + variance + irreducible noise. Add it up across flexibility levels and you get a U-shape. The bottom of the U is the sweet spot — flexible enough to capture the pattern, not so flexible that it's chasing noise.

Every modeling decision is just trying to find the bottom of that U.

How it works in one sentence

Given a new point, find the K closest points in your training data, take a majority vote. That's it.

Lincoln framing: a new plating job arrives with bath temp 82°C and line speed 14 parts/min. Look up the 10 most similar past jobs. If 7 passed and 3 failed, predict PASS. If 3 passed and 7 failed, predict FAIL. Done.

That's a real, defensible model. No coefficients. No training. The "model" is just your past data.

Picking K

K = 1 means you trust whoever's standing closest, even if they're an outlier. One weird past job dominates every prediction near it. Wiggly decision boundary. High variance.

K = 100 means you average over basically your whole dataset. Too smooth to catch local patterns. Blurry decision boundary. High bias.

The right K is somewhere in between, and yes — it's a bias-variance tradeoff. The U-shape from Lesson 6 shows up here too. Sound familiar?

The problem

50 machines on the floor. Some will fail in the next 7 days, some won't. We have 8 features per machine — age, cycles since service, vibration, bath temp drift, operator skill, part complexity, customer tier, and one decoy that looks important but isn't. Build a KNN classifier that predicts failure on a held-out test set.

What you'll do

Step 1. Pick which features go into the model. Some help. One is a trap.

Step 2. Pick K. (Remember Lesson 6 and 7 — find the U-shape sweet spot.)

Step 3. Watch test accuracy and the confusion matrix update live as you tweak.

Step 4. When you're happy, generate the spec sheet — your shippable artifact.

Why it matters

If you can flag a failure 7 days before it happens, you schedule maintenance during a planned slot instead of an emergency stop. That's literally Planning + Cost Savings + Operations all at once. It's the kind of model leadership notices.

The shape of a tree

Every node asks one yes/no question about one feature. Yes goes left, no goes right. You keep splitting until you hit a leaf — that leaf is your prediction.

For your sourcing data, a tree might learn this:

if lot_purity_pct < 97.0:
    if humidity_pct > 75:
        predict FAIL    ← leaf
    else:
        predict PASS    ← leaf
else:
    if bath_temp_f < 163:
        predict FAIL    ← leaf
    else:
        predict PASS    ← leaf

You can read this. If the model predicted FAIL on a new job, you can trace the exact path. That's defensibility you literally cannot get from a neural network.

How does a tree decide where to split?

The algorithm tries every possible split (every feature × every threshold) and picks the one that separates passes from fails the most. The mathematical name for "how mixed up are these groups" is Gini impurity (classification) or residual sum of squares (regression). You don't need the math; you need the intuition: a good split makes the two resulting groups as pure as possible.

Then it does the same thing on each resulting group. Recursively. Until you say stop — or every leaf is pure.

The overfitting trap

A tree with no depth limit will keep splitting until each leaf contains a single training job. Training accuracy: 100%. Test accuracy on new jobs: bad. The model memorized the noise instead of the signal — same overfitting story you saw in Lesson 6.

Your purity-vs-thickness scatter has noise around the true linear relationship. A deep tree will draw boundaries that zig-zag around every noisy point. New jobs land between those zig-zags and get predicted wrong.

Two ways to control tree size

Pre-pruning — stop early. Set a max_depth (e.g., 4 levels) or min_samples_leaf (e.g., at least 10 jobs per leaf) before training. Simple and fast.

Post-pruning — let the tree grow huge, then prune back nodes that don't help test accuracy. The textbook calls this cost-complexity pruning (ISLR2 §8.1.2). More principled but more compute.

For your Lincoln-sized data (hundreds of jobs, not millions), pre-pruning with max_depth = 3 to 5 usually beats the more elaborate methods. Don't over-engineer.

What's different (and what isn't)

The structure is identical: yes/no splits, leaves at the bottom. The only thing that changes is what the leaves contain and how the splits are chosen.

• Leaves contain numbers (the average of the training jobs in that leaf), not class labels.
• Splits minimize variance within each leaf, not Gini impurity. The math: reduce the squared error.

That's it. ISLR2 §8.1.1 covers this in 3 pages because there isn't much more to say.

Trees vs linear regression — head to head

If your data is genuinely linear, linear regression wins. It's the simplest model that fits a line.

If your data has thresholds, interactions, or non-linear chunks ("things change once temperature exceeds 167°F"), a tree wins. Trees handle these natively because every split IS a threshold.

Your sourcing data has both — purity → thickness is mostly linear, but humidity has a threshold effect on QC, and Supplier D introduces a step-change. A tree captures this without you specifying the form.

Why 500 trees beat 1 tree

A single decision tree is twitchy — change one data point and it can split differently. High variance.

If you train 500 trees, each on a slightly different sample (with replacement), each one will overfit in a slightly different way. Average their predictions and the noise cancels out. The signal survives. The variance drops dramatically.

This trick is called bootstrap aggregating ("bagging" — ISLR2 §8.2.1). Random forests add one more twist: at each split, only a random subset of features is considered. This forces the trees to be diverse — without it they'd all look identical.

Feature importance — the sourcing engineer's bonus

Random forests give you something a single tree can't: a clean answer to "which features actually matter?"

For every feature, the forest counts how much it improved predictions across all 500 trees. The result is a ranked list. This is the answer to "where should I focus as a sourcing engineer?" If lot_purity_pct ranks #1 and operator ranks #11, you stop blaming operators and start tightening supplier contracts.

Bagging vs boosting in one sentence each

Bagging (random forest) — train many independent trees on different samples in parallel, average the predictions. Reduces variance.

Boosting — train one shallow tree. Look at what it got wrong. Train the next tree to fix those mistakes. Repeat. Reduces both bias and variance — but can overfit if you let it run too long.

Why everyone reaches for gradient boosting

On tabular data with hundreds to millions of rows, gradient boosting (the modern incarnation of the idea) usually wins. Why:

• Handles mixed features natively. Numeric, categorical, missing values — boosters eat them all.
• Fast inference. Predictions are still just walking a tree (many trees, but each is shallow).
• Strong out-of-the-box. Default settings beat heavily-tuned linear models on most real datasets.
• Built-in feature importance. Same bonus you got from random forests.

The trade: harder to defend than a single tree, more knobs to tune than a random forest, can overfit silently if you don't validate honestly. ISLR2 §8.2.3 covers the math.

Grid search vs random search

Grid search = try every combination of every knob you care about. Exhaustive, slow, but thorough. Fine for 2-3 hyperparameters on a small dataset like yours.

Random search = sample N random combinations from the hyperparameter space. Surprisingly often beats grid search per unit of compute because most knobs don't matter much; sampling lets you waste less time on the unimportant ones.

Bayesian optimization (Optuna, scikit-optimize) = smarter random search that learns from each trial. Overkill for Lincoln-sized data; use it once you're past 100K rows.

Cross-validation — the only honest tuning method

You CAN'T pick hyperparameters by looking at training accuracy — they'd all overfit. You CAN'T pick them by looking at your test set either — you'd be using the "honest" set to make a choice, contaminating it.

The fix: k-fold cross-validation on your TRAINING data only. Split the train set into k chunks (commonly 5). For each candidate setting: train on 4 chunks, validate on the 5th. Rotate. Average the 5 validation scores. Pick the setting with the best average. THEN evaluate the winner once on the held-out test set. ISLR2 §5.1 covers it.

This sounds elaborate; it's two lines of code in scikit-learn. The discipline matters more than the syntax.

Drift — the slow killer

Data drift — your inputs change shape. New supplier joins, humidity baseline rises, line speed target shifts (Month 6, remember?). The model still works mechanically but its accuracy decays because the world moved.

Concept drift — the relationship between inputs and outputs changes. Maybe Supplier E's purity number is now measured differently, or the QC spec was tightened by Quality. Same inputs, different "right" answer.

You only catch either if you're watching. The discipline: log every prediction + every eventual outcome. Compare weekly.

When to retrain — three triggers

• Time-based. "Retrain monthly" — simple, defensible, no surprises. Default for stable processes.
• Drift-based. Set a threshold on your monitoring metric (e.g., "rolling 30-day accuracy drops 5 percentage points below baseline"). Retrain when triggered. More efficient but needs monitoring infrastructure.
• Event-based. Known regime change: new supplier, new equipment, new spec. Retrain on the day of the event. Sourcing engineer's instinct says this matters — trust it.

In practice: use all three. Time-based as the default, drift-based as the safety net, event-based when you know something changed.

What you're building

A model that, given a new incoming plating job's features (supplier, purity, bath conditions), predicts pass_qc. You'll pick from three model families and tune two key hyperparameters:

• Single decision tree — most defensible. Tune max_depth.
• Random forest — most accurate "default." Tune n_estimators + max_depth.
• Gradient boosting — current world champion. Tune n_estimators + max_depth + learning_rate.

The workbench below lets you swap between models and see test accuracy on a held-out 96-job test set (last 20% of the data). Find the best honest accuracy you can. Then write your spec.