CE49X: Introduction to Computational Thinking and Data Science for Civil Engineers

Week 6: Introduction to Machine Learning

Instructor: Dr. Eyuphan Koc Department of Civil Engineering, Bogazici University Semester: Spring 2026

Table of Contents
1. What is Machine Learning?
2. The Scikit-Learn Workflow
3. The Accuracy Trap
4. The Confusion Matrix
5. Precision, Recall, and the Tradeoff
6. ROC Curves and AUC
7. Overfitting
8. Data Leakage
9. Putting It Together

In [1]: import numpy as np

import pandas as pd
import [Link] as plt
from sklearn.linear_model import LogisticRegression, LinearRegression
from sklearn.model_selection import train_test_split, cross_val_score
from [Link] import (accuracy_score, confusion_matrix, classification_report,
precision_recall_curve, roc_curve, auc,
mean_squared_error, r2_score)
%matplotlib inline

We begin by importing the libraries we will use throughout this lecture. Don't worry if you don't recognize all of these yet — we will introduce each one as we need it.

1. What is Machine Learning?

After the devastating 1999 Izmit earthquake, engineers had borehole data from thousands of sites across the Marmara region. They wanted to answer a critical question: which sites are at risk of soil liquefaction in a future
quake?

They could try writing explicit rules — "if SPT blow count is below 15 and the groundwater table is shallow, flag it" — but how do you set those thresholds? What about the interactions between soil type, depth, seismic intensity,
and dozens of other factors?

Machine learning takes a different approach: instead of programming rules, you provide examples and let the algorithm discover the patterns.

Definition: Machine Learning

Machine Learning is about building mathematical models to understand data. Models learn from examples rather than explicit programming.

"Instead of coding rules for predicting soil behavior, we provide examples of sites that did and didn't liquefy — the model learns the relationship."

Categories of Machine Learning

Supervised Learning Unsupervised Learning

Data Learn from labeled data Learn from unlabeled data

Setup Have input-output pairs No predefined outputs

Goal Predict outputs for new inputs Discover structure

Type 1 Classification: Predict discrete labels Clustering: Group similar items

Type 2 Regression: Predict continuous values Dimensionality Reduction: Compress data

Key Insight: The Key Difference Supervised: "Here are examples with answers — learn the pattern" vs Unsupervised: "Find patterns in this data on your own"

Supervised Learning: Classification vs Regression

Classification Regression

Predict Discrete categories Continuous values

Output A label or class A number

CE Examples Soil: will it liquefy in an earthquake? Building heating load from geometry

Pipe segment: will it fail this year? Traffic speed from vehicle density

Visualizing Classification and Regression

The following plot shows the key difference between classification (left) and regression (right).

In [2]: fig, axes = [Link](1, 2, figsize=(12, 5))

# Classification: Liquefied vs Stable soil samples

ax = axes[0]
# Stable sites (high SPT blow count, deeper)
[Link]([25, 30, 35, 28, 32], [3, 8, 12, 5, 10],
c='steelblue', s=80, label='Stable', edgecolors='k', zorder=3)
# Liquefied sites (low SPT blow count, shallower)
[Link]([5, 8, 12, 7], [2, 4, 3, 6],
c='indianred', s=80, label='Liquefied', edgecolors='k', zorder=3)
ax.set_xlabel('SPT Blow Count (N)', fontsize=12)
ax.set_ylabel('Depth (m)', fontsize=12)
ax.set_title('Classification: Liquefaction Risk', fontsize=14)
[Link](fontsize=10)
[Link](True, alpha=0.3)

# Regression: Wall area vs heating load

ax = axes[1]
wall_pts = [Link]([250, 280, 320, 360, 400])
heat_pts = [Link]([13, 17, 23, 30, 38])
[Link](wall_pts, heat_pts, c='indianred', s=80, edgecolors='k', zorder=3, label='Buildings')
x_line = [Link](240, 420, 100)
y_line = -19 + 0.14 * x_line
[Link](x_line, y_line, 'steelblue', linewidth=2, label='Fit line')
ax.set_xlabel('Wall Area (m²)', fontsize=12)
ax.set_ylabel('Heating Load (kWh/m²)', fontsize=12)
ax.set_title('Regression: Energy Efficiency', fontsize=14)
[Link](fontsize=10)
[Link](True, alpha=0.3)

plt.tight_layout()
[Link]()

Machine Learning Workflow

Every ML project follows the same high-level pipeline:

[Raw Data] --> [Preprocess & Clean] --> [Feature Engineering]

|
v
[Validate] <-- [Train Model] <-- [Choose Model]
| ^
| | (adjust)
+----------------+
|
v
[Deploy]

Key Steps:

1. Data Collection & Preprocessing: Gather, clean, normalize data

2. Feature Engineering: Select/create informative features
3. Model Selection & Training: Choose algorithm, fit to data
4. Validation: Test on unseen data, tune hyperparameters
5. Deployment: Use model in production

[QUICK] For each scenario below, decide: supervised or unsupervised? Classification or regression?

1. Predicting a building's annual heating load from its wall area, roof area, and glazing percentage
2. Grouping soil samples into clusters based on grain size, plasticity, and moisture (no labels)
3. Deciding whether a soil layer will liquefy during an earthquake based on borehole data
4. Estimating traffic speed on a highway from vehicle density measurements

2. The Scikit-Learn Workflow

Scikit-learn is Python's premier machine learning library. Its most powerful feature is a consistent API — learning one model teaches you all models.

Definition: The Estimator API

All scikit-learn models follow the same three-step pattern:

from sklearn.some_module import SomeModel

model = SomeModel(hyperparameters) # 1. Choose model

[Link](X, y) # 2. Fit to data
predictions = [Link](X_new) # 3. Apply to new data

Data Representation in Scikit-Learn

Definition: Features Matrix and Target Array

| | Features Matrix: X | Target Array: y | |---|---|---| | Shape | [n samples , nfeatures ] | [n samples ] | | Content | Each row = one sample, each column = one feature | Labels (classification) or values (regression) | | Type |
2D NumPy array or pandas DataFrame | 1D NumPy array or pandas Series |

[TOGETHER] Example: Predicting Building Heating Load

Problem: Predict a building's heating load (kWh/m²) from its wall area.

This is inspired by the UCI Energy Efficiency Dataset — a real dataset of 768 buildings analyzed for energy performance. Let's see the Estimator API in action.

In [3]: # Building wall area vs heating load (inspired by UCI Energy Efficiency dataset)
X_wall = [Link]([[245], [280], [318], [350], [390], [416]])
y_heat = [Link]([12, 16, 22, 28, 35, 41]) # Heating load in kWh/m²

# 1. Choose model
model_energy = LinearRegression()

# 2. Fit model
model_energy.fit(X_wall, y_heat)

# 3. Make predictions
X_new_wall = [Link]([[300], [375]])
predictions_energy = model_energy.predict(X_new_wall)

print(f"Predictions: {predictions_energy.round(1)} kWh/m²")

print(f"Slope: {model_energy.coef_[0]:.4f} kWh/m² per m² of wall area")
print(f"Intercept: {model_energy.intercept_:.2f}")

Predictions: [20. 32.8] kWh/m²

Slope: 0.1704 kWh/m² per m² of wall area
Intercept: -31.12

In [4]: # Visualize the regression fit

X_line_wall = [Link](230, 430, 100).reshape(-1, 1)
y_line_wall = model_energy.predict(X_line_wall)

fig, ax = [Link](figsize=(10, 6))

[Link](X_wall, y_heat, color='steelblue', s=100, alpha=0.7,
label='Training Data', edgecolors='k', zorder=3)
[Link](X_line_wall, y_line_wall, color='indianred', linewidth=2, label='Regression Line')
[Link](X_new_wall, predictions_energy, color='green', s=120, marker='s',
label='Predictions', zorder=5, edgecolors='k')
ax.set_xlabel('Wall Area (m²)', fontsize=12)
ax.set_ylabel('Heating Load (kWh/m²)', fontsize=12)
ax.set_title('Linear Regression: Wall Area vs Building Heating Load', fontsize=14)
[Link](fontsize=10)
[Link](True, alpha=0.3)
plt.tight_layout()
[Link]()

Key Insight: The Estimator API is Universal

Whether you're doing linear regression, logistic regression, decision trees, or neural networks — the pattern is always fit() → predict() → score() . Master this pattern once, and you can use any model
in scikit-learn.

[QUICK] What would you change to predict heating load from multiple building features (wall area, roof area, glazing percentage, overall height)? Hint: Only the shape of X needs to change — the API stays the
same.

3. The Accuracy Trap

Now that you know how to train a model, the natural question is: how good is it?

Every beginner's first question about a model is: "What's the accuracy?"

This section shows why that question, on its own, can be dangerously misleading — especially in geotechnical engineering, where soil failures are rare but catastrophic.

Example: Everyday Accuracy Traps

Before we look at any code, consider these scenarios:

A weather model that always predicts "no rain" in Izmir would be right ~90% of the time. But you'd never trust it during the rainy season.
A spam filter that never flags anything as spam has 100% accuracy on legitimate emails — but lets every phishing attack through.
A medical screening test that always says "healthy" would have >99% accuracy, because most people ARE healthy. But it would miss every case of disease.

The pattern: when one outcome is much rarer than the other, a lazy model that always predicts the common outcome gets high accuracy for free.

In [5]: # (continuing with the same imports from above)

In [6]: [Link](42)

# Soil liquefaction dataset — inspired by SPT-based records from the 1999 Izmit earthquake
# HIGHLY IMBALANCED: most sites remain stable
n_stable = 900
n_liquefied = 100

# Features for stable sites (higher SPT N, deeper groundwater, lower PGA)
X_stable = np.column_stack([
[Link](8, 45, n_stable), # spt_n: SPT blow count
[Link](1, 20, n_stable), # depth (m)
[Link](0, 70, n_stable), # fines_content (%)
[Link](0.05, 0.45, n_stable), # pga: peak ground acceleration (g)
[Link](1, 10, n_stable) # gw_depth: groundwater depth (m)
])

# Features for liquefied sites (low SPT N, shallow GW, high PGA, high fines)
X_liquefied = np.column_stack([
[Link](2, 25, n_liquefied),
[Link](1, 15, n_liquefied),
[Link](10, 90, n_liquefied),
[Link](0.15, 0.55, n_liquefied),
[Link](0, 6, n_liquefied)
])

X = [Link]([X_stable, X_liquefied])
y = [Link]([0] * n_stable + [1] * n_liquefied) # 0=stable, 1=liquefied

print(f"Dataset: {len(X)} borehole sites (Marmara region)")

print(f" Stable: {n_stable} ({n_stable/len(X)*100:.1f}%)")
print(f" Liquefied: {n_liquefied} ({n_liquefied/len(X)*100:.1f}%)")
print(f"\nFeatures: SPT blow count, depth, fines content, PGA, groundwater depth")

Dataset: 1000 borehole sites (Marmara region)

Stable: 900 (90.0%)
Liquefied: 100 (10.0%)

Features: SPT blow count, depth, fines content, PGA, groundwater depth

In [7]: # A model that ALWAYS predicts "stable"

class AlwaysStableModel:
def predict(self, X):
return [Link](len(X))

dumb_model = AlwaysStableModel()
y_pred_dumb = dumb_model.predict(X)
accuracy_dumb = accuracy_score(y, y_pred_dumb)
print(f"Accuracy of the 'always stable' model: {accuracy_dumb:.1%}")
print(f"\nBut how many liquefiable sites did it catch?")
print(f"Liquefied sites detected: {[Link](y_pred_dumb[y == 1]):.0f} out of {n_liquefied}")

Accuracy of the 'always stable' model: 90.0%

But how many liquefiable sites did it catch?

Liquefied sites detected: 0 out of 100

In [8]: # Let's visualize this disconnect

accuracy = accuracy_dumb * 100
sites_caught = int([Link](y_pred_dumb[y == 1] == 1))
total_liq = [Link](y == 1)

fig, axes = [Link](1, 2, figsize=(12, 5))

# Left: Accuracy bar

axes[0].bar(['Model Accuracy'], [accuracy], color='steelblue', width=0.4, edgecolor='black', linewidth=1.2)
axes[0].text(0, accuracy + 2, f'{accuracy:.1f}%', ha='center', fontsize=16, fontweight='bold', color='steelblue')
axes[0].set_ylim(0, 110)
axes[0].set_ylabel('Accuracy (%)', fontsize=12)
axes[0].set_title('Looks Great!', fontsize=13)
axes[0].grid(True, alpha=0.3, axis='y')

# Right: Detection bar

axes[1].bar(['Liquefiable Sites\nDetected'], [sites_caught], color='indianred', width=0.4, edgecolor='black', linewidth=1.2)
axes[1].text(0, sites_caught + 2, f'{sites_caught} out of {total_liq}', ha='center', fontsize=16, fontweight='bold', color='indianred')
axes[1].set_ylim(0, total_liq + 10)
axes[1].set_ylabel('Count', fontsize=12)
axes[1].set_title('...But Actually Useless', fontsize=13)
axes[1].grid(True, alpha=0.3, axis='y')

[Link]('The Accuracy Illusion: A "90% Accurate" Model That Catches Nothing', fontsize=13, fontweight='bold')
plt.tight_layout()
[Link]()

Key Insight: Accuracy is Meaningless When Classes are Imbalanced

The model above has 90% accuracy — and it's completely useless. It missed ALL 100 liquefiable sites.

In geotechnical engineering, soil failures are rare (thankfully). But that makes accuracy a terrible metric for safety-critical applications.

[DISCUSS] Can you think of other engineering scenarios where the "rare event" is the one you care about most? Consider:

Earthquake damage assessment — most buildings survive, but the damaged ones need urgent attention
Water pipe failure — most pipes are fine, but a burst main floods entire neighborhoods
Landslide prediction — most slopes are stable, but a single failure can destroy a highway
Dam safety monitoring — most readings are normal, but anomalies can precede catastrophic failure

[QUICK] If you were a geotechnical engineer assessing 1,000 borehole sites across the Marmara region and your liquefaction model had 90% accuracy, would you trust it? What additional information would you
demand before approving construction on a "stable" site?

4. The Confusion Matrix

The confusion matrix tells us how the model is wrong — not just whether it's wrong. This distinction matters enormously when different types of errors have different consequences.

Building the Confusion Matrix Step by Step

Imagine you have tested 100 borehole sites. You know the truth (from detailed geotechnical investigation after the earthquake) and you have the model's prediction. Let's sort every site into one of four buckets:

1. Model says "stable," site IS stable — great, safe to build (True Negative)
2. Model says "liquefiable," site IS stable — unnecessary ground improvement, costs money (False Positive)
3. Model says "stable," site IS liquefiable — building on dangerous ground (False Negative)
4. Model says "liquefiable," site IS liquefiable — caught it, apply ground improvement (True Positive)

The confusion matrix simply counts how many sites fell into each bucket.

In [9]: # Let's see this with 10 specific borehole sites

print("Manual Confusion Matrix: Sorting 10 Borehole Sites into Buckets")
print("=" * 65)

# Simulated results for 10 sites

site_ids = ['S001', 'S002', 'S003', 'S004', 'S005', 'S006', 'S007', 'S008', 'S009', 'S010']
actual_labels = [0, 0, 1, 0, 0, 1, 0, 0, 0, 0 ]
predicted = [0, 1, 0, 0, 0, 1, 0, 1, 0, 0 ]
label_map = {0: 'Stable', 1: 'Liquefied'}

tn, fp, fn, tp = 0, 0, 0, 0

for sid, actual, pred in zip(site_ids, actual_labels, predicted):
if actual == 0 and pred == 0:
bucket = "True Negative (correct: stable)"
tn += 1
elif actual == 0 and pred == 1:
bucket = "False Positive (false alarm)"
fp += 1
elif actual == 1 and pred == 0:
bucket = "FALSE NEGATIVE (MISSED LIQUEFACTION!)"
fn += 1
else:
bucket = "True Positive (caught liquefaction)"
tp += 1
print(f" {sid}: Actual={label_map[actual]:10s} Predicted={label_map[pred]:10s} -> {bucket}")

print(f"\nFinal counts: TN={tn}, FP={fp}, FN={fn}, TP={tp}")

print(f"\nNotice: Site S003 was liquefiable but predicted stable — that's the dangerous one!")

Manual Confusion Matrix: Sorting 10 Borehole Sites into Buckets

=================================================================
S001: Actual=Stable Predicted=Stable -> True Negative (correct: stable)
S002: Actual=Stable Predicted=Liquefied -> False Positive (false alarm)
S003: Actual=Liquefied Predicted=Stable -> FALSE NEGATIVE (MISSED LIQUEFACTION!)
S004: Actual=Stable Predicted=Stable -> True Negative (correct: stable)
S005: Actual=Stable Predicted=Stable -> True Negative (correct: stable)
S006: Actual=Liquefied Predicted=Liquefied -> True Positive (caught liquefaction)
S007: Actual=Stable Predicted=Stable -> True Negative (correct: stable)
S008: Actual=Stable Predicted=Liquefied -> False Positive (false alarm)
S009: Actual=Stable Predicted=Stable -> True Negative (correct: stable)
S010: Actual=Stable Predicted=Stable -> True Negative (correct: stable)

Final counts: TN=6, FP=2, FN=1, TP=1

Notice: Site S003 was liquefiable but predicted stable — that's the dangerous one!

Definition: Confusion Matrix Quadrants

| | Predicted Negative | Predicted Positive | |---|---|---| | Actually Negative | True Negative (TN) | False Positive (FP) | | Actually Positive | False Negative (FN) | True Positive (TP) |

TN: Correctly identified as stable soil

FP: Stable site flagged as liquefiable ("false alarm" — triggers unnecessary ground improvement)
FN: Liquefiable site missed ("missed detection" — building on dangerous ground)
TP: Correctly identified as liquefiable

Now let's use sklearn to compute the confusion matrix on the full dataset. Note the class_weight='balanced' parameter — this tells the model to pay extra attention to the minority class (liquefied sites), counteracting
the imbalance in our dataset.

In [10]: X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42, stratify=y)

model = LogisticRegression(random_state=42, class_weight='balanced')

[Link](X_train, y_train)
y_pred = [Link](X_test)

cm = confusion_matrix(y_test, y_pred)

fig, ax = [Link](figsize=(7, 6))

im = [Link](cm, cmap='Blues')
ax.set_xticks([0, 1])
ax.set_yticks([0, 1])
ax.set_xticklabels(['Predicted Stable', 'Predicted Liquefied'], fontsize=11)
ax.set_yticklabels(['Actually Stable', 'Actually Liquefied'], fontsize=11)
for i in range(2):
for j in range(2):
label = f'{cm[i, j]}'
if i == 0 and j == 0: label += '\n(TN)'
elif i == 0 and j == 1: label += '\n(FP)'
elif i == 1 and j == 0: label += '\n(FN)'
else: label += '\n(TP)'
[Link](j, i, label, ha='center', va='center', fontsize=14, fontweight='bold')
[Link](im, ax=ax)
ax.set_title('Confusion Matrix — Soil Liquefaction Classifier', fontsize=13)
plt.tight_layout()
[Link]()

Key Insight: In Civil Engineering, Errors are NOT Symmetric

| Error Type | Meaning | Consequence | |---|---|---| | False Negative (FN) | Liquefiable site predicted as stable | Building collapses in next earthquake | | False Positive (FP) | Stable site predicted as liquefiable |
Unnecessary ground improvement (costs money) |

A false negative in liquefaction assessment can be catastrophic — as seen in Adapazarı during the 1999 Izmit earthquake, where buildings on liquefiable soil collapsed. A false positive is merely expensive.

Example: The Fire Alarm Analogy

Think of a fire alarm in your building:

True Positive: Fire alarm goes off, there IS a fire. Life saved.
False Positive: Fire alarm goes off, it's just burnt toast. Annoying but harmless.
False Negative: Fire alarm stays silent during a fire. Fatal.
True Negative: No alarm, no fire. Normal day.

In most safety systems, we tolerate false positives (nuisance alarms) to minimize false negatives (missed dangers). This is exactly the tradeoff in structural health monitoring.

In [11]: # Two models, same accuracy, VERY different confusion matrices

# Let's actually build them using different thresholds
y_proba = model.predict_proba(X_test)[:, 1]

# Model A: High threshold (conservative — rarely flags liquefaction)

y_pred_A = (y_proba >= 0.8).astype(int)
cm_A = confusion_matrix(y_test, y_pred_A)
acc_A = accuracy_score(y_test, y_pred_A)

# Model B: Low threshold (aggressive — flags liquefaction more readily)

y_pred_B = (y_proba >= 0.2).astype(int)
cm_B = confusion_matrix(y_test, y_pred_B)
acc_B = accuracy_score(y_test, y_pred_B)

fig, axes = [Link](1, 2, figsize=(14, 5))

labels = [['TN', 'FP'], ['FN', 'TP']]

for ax, cm_plot, title, acc in zip(axes, [cm_A, cm_B],

['Model A: Conservative (high threshold)', 'Model B: Aggressive (low threshold)'],
[acc_A, acc_B]):
im = [Link](cm_plot, cmap='Blues')
ax.set_xticks([0, 1])
ax.set_yticks([0, 1])
ax.set_xticklabels(['Pred Stable', 'Pred Liquefied'], fontsize=10)
ax.set_yticklabels(['Actually Stable', 'Actually Liquefied'], fontsize=10)
for i in range(2):
for j in range(2):
[Link](j, i, f'{cm_plot[i, j]}\n({labels[i][j]})',
ha='center', va='center', fontsize=13, fontweight='bold')
ax.set_title(f'{title}\nAccuracy: {acc:.1%}', fontsize=11)

[Link]('Which Model Would You Deploy for a Real Construction Project?', fontsize=13, fontweight='bold')
plt.tight_layout()
[Link]()

# Print the key comparison

liq_caught_A = cm_A[1, 1]
liq_caught_B = cm_B[1, 1]
total_liq = cm_A[1, 0] + cm_A[1, 1]
print(f"Model A — Accuracy: {acc_A:.1%}, Liquefiable sites caught: {liq_caught_A}/{total_liq}")
print(f"Model B — Accuracy: {acc_B:.1%}, Liquefiable sites caught: {liq_caught_B}/{total_liq}")

Model A — Accuracy: 91.3%, Liquefiable sites caught: 22/30

Model B — Accuracy: 66.3%, Liquefiable sites caught: 29/30

[QUICK] Sklearn provides a full classification report with precision, recall, and F1 for each class. Let's see what it looks like.

In [12]: # Full classification report from sklearn

print("Classification Report — Soil Liquefaction Classifier")
print("=" * 55)
print(classification_report(y_test, y_pred, target_names=['Stable', 'Liquefied']))

Classification Report — Soil Liquefaction Classifier

=======================================================
precision recall f1-score support

Stable 0.99 0.81 0.89 270

Liquefied 0.35 0.93 0.51 30

accuracy 0.82 300

macro avg 0.67 0.87 0.70 300
weighted avg 0.93 0.82 0.85 300

[PRACTICE] Look at the classification report above. Identify:

1. Which class (Stable or Liquefied) has higher precision?

2. Which class has higher recall?
3. Why does this make sense given the class imbalance? (Hint: think about what happens when there are very few positive examples — the model has very little to learn from.)

5. Precision, Recall, and the Tradeoff

Now that we understand the confusion matrix, we can define two critical metrics. But before the math, let's build intuition:

Think of a smoke detector:

Precision answers: "When the alarm rings, should I trust it?" — Of all the times it warned me, how often was there actually a fire?
Recall answers: "Am I catching everything dangerous?" — Of all the real fires, how many did the detector catch?

A very sensitive smoke detector has high recall (catches every fire) but low precision (also goes off when you cook). A very conservative detector has high precision (only alarms on real fires) but low recall
(might miss a slow-burning fire).

Definition: Precision and Recall

Precision = TP / (TP + FP) — "Of all alarms raised, how many were real?"

Recall = TP / (TP + FN) — "Of all actual positives, how many did we catch?"

These two metrics are in tension. Improving one typically hurts the other.

The Classification Threshold

When a classification model outputs a probability (e.g., "there is a 0.73 chance this sample is positive"), we still need to make a binary decision: positive or negative. The threshold is the cutoff value that converts
probabilities into predictions:

1 if P (positive) ≥ threshold
y
^ = {
0 if P (positive) < threshold

By default, most classifiers use a threshold of 0.5, but this is not always the best choice.

Key Insight: Changing the threshold shifts the balance between precision and recall:

| Threshold | Effect | |-----------|--------| | Low (e.g., 0.1) | More samples predicted positive → higher recall (fewer missed positives) but lower precision (more false alarms) | | High (e.g., 0.9) | Only very confident
predictions are positive → higher precision but lower recall (more missed positives) |

There is no free lunch — improving one metric typically comes at the cost of the other.

Example: Civil Engineering Consider a model that predicts whether a bridge component will fail. A low threshold means we flag more components for inspection (costly but safe). A high threshold means we
only flag components the model is very confident about (cheaper but riskier — we might miss a real failure).

In [13]: y_proba = model.predict_proba(X_test)[:, 1]

thresholds = [Link](0.1, 0.95, 0.05)

precisions = []
recalls = []

for t in thresholds:
y_pred_t = (y_proba >= t).astype(int)
tp = [Link]((y_pred_t == 1) & (y_test == 1))
fp = [Link]((y_pred_t == 1) & (y_test == 0))
fn = [Link]((y_pred_t == 0) & (y_test == 1))
prec = tp / (tp + fp) if (tp + fp) > 0 else 0
rec = tp / (tp + fn) if (tp + fn) > 0 else 0
[Link](prec)
[Link](rec)

# Show a few key thresholds

print(f"{'Threshold':>10} {'Precision':>10} {'Recall':>10} {'Interpretation':>40}")
print("-" * 75)
for i in range(0, len(thresholds), 3):
t = thresholds[i]
p = precisions[i]
r = recalls[i]
interp = "high recall (catch more)" if t < 0.3 else ("balanced" if t < 0.6 else "high precision (fewer false alarms)")
print(f"{t:>10.2f} {p:>10.2f} {r:>10.2f} {interp:>40}")

Threshold Precision Recall Interpretation

---------------------------------------------------------------------------
0.10 0.18 1.00 high recall (catch more)
0.25 0.25 0.97 high recall (catch more)
0.40 0.31 0.97 balanced
0.55 0.38 0.90 balanced
0.70 0.44 0.77 high precision (fewer false alarms)
0.85 0.67 0.67 high precision (fewer false alarms)

In [14]: # Visualize the tradeoff at 6 different thresholds

fig, axes = [Link](2, 3, figsize=(16, 10))
threshold_vals = [0.1, 0.2, 0.4, 0.5, 0.7, 0.9]

for ax, t in zip([Link], threshold_vals):

y_pred_t = (y_proba >= t).astype(int)
tp = [Link]((y_pred_t == 1) & (y_test == 1))
fp = [Link]((y_pred_t == 1) & (y_test == 0))
fn = [Link]((y_pred_t == 0) & (y_test == 1))
tn = [Link]((y_pred_t == 0) & (y_test == 0))
prec = tp / (tp + fp) if (tp + fp) > 0 else 0
rec = tp / (tp + fn) if (tp + fn) > 0 else 0

# Plot bars for TP, FP, FN, TN

categories = ['TN', 'FP', 'FN', 'TP']
values = [tn, fp, fn, tp]
colors = ['steelblue', 'sandybrown', 'indianred', 'mediumseagreen']
[Link](categories, values, color=colors, edgecolor='black', linewidth=0.8)
for i_bar, v in enumerate(values):
if v > 0:
[Link](i_bar, v + 0.3, str(v), ha='center', fontsize=10, fontweight='bold')
ax.set_title(f'Threshold = {t:.1f}\nPrec={prec:.2f}, Recall={rec:.2f}', fontsize=11)
ax.set_ylim(0, max(tn + 5, 20))
[Link](True, alpha=0.3, axis='y')

[Link]('How Changing the Threshold Affects the Confusion Matrix', fontsize=14, fontweight='bold')
plt.tight_layout()
[Link]()

[TOGETHER] Look at the 6 plots above. As the threshold moves from low (0.1) to high (0.9):

1. What happens to the number of false alarms (FP)?

2. What happens to the number of missed detections (FN)?
3. At which threshold would you feel comfortable deploying this model for a real construction project?
4. Is there a threshold that makes BOTH FP and FN zero? Why or why not?

The Precision–Recall Tradeoff in Detail

Both metrics share TP in the numerator, but their denominators pull in opposite directions:

To increase recall, we lower the threshold so we catch more true positives — but this also lets in more false positives, which decreases precision.
To increase precision, we raise the threshold so we only predict positive when we are very confident — but this causes us to miss some true positives, which decreases recall.

The only way to improve both simultaneously is to build a better model (one that assigns higher probabilities to true positives and lower probabilities to true negatives).

The F1 Score: A Single Balanced Metric

When neither precision nor recall alone captures what we care about, the F1 score provides a single number. But why not just average them?

Consider a model with Precision = 1.0 and Recall = 0.01 (almost never predicts positive, but when it does, it's always right):

1.0 + 0.01
Arithmetic mean = = 0.505 (looks decent!)
2

1.0 × 0.01
F1 = 2 ⋅ = 0.02 (correctly shows this model is nearly useless)
1.0 + 0.01

The harmonic mean penalizes extreme imbalances — the F1 score is high only when both metrics are reasonably high.

F1 Score Interpretation

1.0 Perfect — no false positives and no false negatives

> 0.8 Strong classifier for most practical applications

0.5 – 0.8 Moderate — may need tuning depending on the application

< 0.5 Weak — the model is making many errors of one type or both

Weighted Variants: F β

Sometimes false negatives and false positives are not equally costly. The generalized F score lets you control the balance:
β

Precision × Recall
2
Fβ = (1 + β ) ⋅
2
(β ⋅ Precision) + Recall

Variant β Emphasis Use case

F0.5 0.5 Weighs precision more When false alarms are costly (e.g., dispatching emergency crews)

F1 1.0 Equal weight Default balanced metric

F2 2.0 Weighs recall more When missing positives is dangerous (e.g., structural failure screening)

Key Insight: The F1 score is the best default when you have no strong reason to prefer precision over recall. In real engineering applications, always think about what type of error is more costly before choosing
your evaluation metric.

In [15]: precision_curve, recall_curve, _ = precision_recall_curve(y_test, y_proba)

fig, ax = [Link](figsize=(8, 6))

[Link](recall_curve, precision_curve, 'b-', linewidth=2)
ax.set_xlabel('Recall (How many liquefiable sites we catch)', fontsize=12)
ax.set_ylabel('Precision (How many alarms are real)', fontsize=12)
ax.set_title('Precision-Recall Tradeoff', fontsize=13)
[Link](True, alpha=0.3)
ax.set_xlim([0, 1.05])
ax.set_ylim([0, 1.05])
plt.tight_layout()
[Link]()

Key Insight: You Can't Maximize Both

Optimize Recall when missing a positive is catastrophic → structural safety monitoring

Optimize Precision when false alarms are costly → routine maintenance scheduling
F1 Score = harmonic mean of precision and recall — use when you want one number that balances both

Example: Civil Engineering Applications

Liquefaction screening → High recall (don't miss a dangerous site, even if you trigger some unnecessary ground improvement)
Routine pipe inspection → High precision (don't waste crew time inspecting healthy pipes)
Emergency earthquake response → High recall (better to over-evacuate than miss a damaged building)

Example: Flood Early Warning System

You build a model that predicts whether a river will flood in the next 24 hours.

High recall strategy (threshold = 0.2): You evacuate neighborhoods whenever there is even a 20% chance of flooding. Many false evacuations, but you never miss a real flood. Cost: public trust erodes over
time from false alarms.
High precision strategy (threshold = 0.8): You only evacuate when the model is 80%+ confident. Fewer unnecessary evacuations, but you might miss a real flood. Cost: lives at risk.
The tradeoff is real and there is no "correct" answer — it depends on the consequences.

[PRACTICE] Let's compute precision, recall, and F1 manually for our liquefaction model to reinforce the formulas.

In [16]: # Compute F1 score manually

# F1 = 2 * (precision * recall) / (precision + recall)

tp = [Link]((y_pred == 1) & (y_test == 1))

fp = [Link]((y_pred == 1) & (y_test == 0))
fn = [Link]((y_pred == 0) & (y_test == 1))

precision_val = tp / (tp + fp) if (tp + fp) > 0 else 0

recall_val = tp / (tp + fn) if (tp + fn) > 0 else 0
f1_val = 2 * (precision_val * recall_val) / (precision_val + recall_val) if (precision_val + recall_val) > 0 else 0

print(f"Precision: {precision_val:.3f}")
print(f"Recall: {recall_val:.3f}")
print(f"F1 Score: {f1_val:.3f}")
print()
if recall_val > precision_val:
print("This model leans toward recall — it catches more liquefiable sites")
print("but raises more false alarms. For geotechnical safety, this is usually preferred.")
else:
print("This model leans toward precision — its alarms are more trustworthy")
print("but it may miss some liquefiable sites. For safety applications, be cautious.")

Precision: 0.354
Recall: 0.933
F1 Score: 0.514

This model leans toward recall — it catches more liquefiable sites

but raises more false alarms. For geotechnical safety, this is usually preferred.

[TOGETHER] Would you accept this F1 score for a liquefaction screening system? What F1 score would you consider the minimum for deploying such a model in practice?

6. ROC Curves and AUC

So far, we've evaluated our model at a specific threshold. But what if we want to compare two models across all possible thresholds simultaneously? That's what the ROC curve does.

The Receiver Operating Characteristic (ROC) curve gives us a way to evaluate a classifier across every threshold at once. The Area Under the Curve (AUC) summarizes this into a single number — think of it as a report card
GPA for your model: it doesn't tell you how the model performed on any single exam (threshold), but it summarizes the overall performance.

Building the ROC Curve One Point at a Time

Remember that Recall (also called True Positive Rate) answers: "Of all liquefiable sites, how many did we catch?"

Now we need one more metric: False Positive Rate (FPR) = FP / (FP + TN) — "Of all stable sites, how many did we falsely flag?"

Here's how the ROC curve is built:

1. At threshold = 0.5, our liquefaction model produces a specific confusion matrix. From that matrix, compute TPR and FPR. That gives us one point on the ROC plot: (FPR, TPR).
2. Change the threshold to 0.3. New confusion matrix → new TPR and FPR → another point.
3. Repeat for every possible threshold from 0 to 1. Connect the dots.
4. That's the ROC curve.

In [17]: # Let's build the ROC curve point by point

print("Building the ROC curve step by step:")
print("=" * 70)

demo_thresholds = [0.1, 0.3, 0.5, 0.7, 0.9]

roc_points = []

for t in demo_thresholds:
y_pred_t = (y_proba >= t).astype(int)
tp = [Link]((y_pred_t == 1) & (y_test == 1))
fp = [Link]((y_pred_t == 1) & (y_test == 0))
fn = [Link]((y_pred_t == 0) & (y_test == 1))
tn = [Link]((y_pred_t == 0) & (y_test == 0))
tpr = tp / (tp + fn) if (tp + fn) > 0 else 0
fpr = fp / (fp + tn) if (fp + tn) > 0 else 0
roc_points.append((fpr, tpr))
print(f"\nThreshold = {t:.1f}:")
print(f" Confusion matrix: TN={tn}, FP={fp}, FN={fn}, TP={tp}")
print(f" FPR = {fp}/({fp}+{tn}) = {fpr:.3f} | TPR = {tp}/({tp}+{fn}) = {tpr:.3f}")
print(f" → Plot point: ({fpr:.3f}, {tpr:.3f})")

# Now plot these individual points AND the full smooth curve
fpr_full, tpr_full, _ = roc_curve(y_test, y_proba)

fig, ax = [Link](figsize=(8, 7))

[Link](fpr_full, tpr_full, color='steelblue', linewidth=1.5, alpha=0.4, label='Full ROC curve')
[Link]([0, 1], [0, 1], 'k--', linewidth=1, alpha=0.5, label='Random guessing')

for i, (fpr_pt, tpr_pt) in enumerate(roc_points):

t = demo_thresholds[i]
[Link](fpr_pt, tpr_pt, s=120, zorder=5, edgecolors='black', linewidth=1.5)
[Link](f't={t}', (fpr_pt, tpr_pt), textcoords="offset points",
xytext=(10, -10), fontsize=10, fontweight='bold')

ax.set_xlabel('False Positive Rate (stable sites falsely flagged)', fontsize=12)

ax.set_ylabel('True Positive Rate = Recall (liquefiable sites caught)', fontsize=12)
ax.set_title('ROC Curve Built Point by Point', fontsize=13)
[Link](fontsize=10)
[Link](True, alpha=0.3)
plt.tight_layout()
[Link]()

print("\nEach dot is one threshold. Connect them all → you get the ROC curve!")

Building the ROC curve step by step:

======================================================================

Threshold = 0.1:
Confusion matrix: TN=131, FP=139, FN=0, TP=30
FPR = 139/(139+131) = 0.515 | TPR = 30/(30+0) = 1.000
→ Plot point: (0.515, 1.000)

Threshold = 0.3:
Confusion matrix: TN=191, FP=79, FN=1, TP=29
FPR = 79/(79+191) = 0.293 | TPR = 29/(29+1) = 0.967
→ Plot point: (0.293, 0.967)

Threshold = 0.5:
Confusion matrix: TN=219, FP=51, FN=2, TP=28
FPR = 51/(51+219) = 0.189 | TPR = 28/(28+2) = 0.933
→ Plot point: (0.189, 0.933)

Threshold = 0.7:
Confusion matrix: TN=241, FP=29, FN=7, TP=23
FPR = 29/(29+241) = 0.107 | TPR = 23/(23+7) = 0.767
→ Plot point: (0.107, 0.767)

Threshold = 0.9:
Confusion matrix: TN=264, FP=6, FN=15, TP=15
FPR = 6/(6+264) = 0.022 | TPR = 15/(15+15) = 0.500
→ Plot point: (0.022, 0.500)

Each dot is one threshold. Connect them all → you get the ROC curve!

Comparing Multiple Models with ROC

One of the best uses of ROC curves is comparing different classifiers on the same problem. Let's try three models:

Logistic Regression — fits a linear decision boundary (the model we've been using)
Decision Tree — splits data using a sequence of if/else rules on individual features
K-Nearest Neighbors (KNN) — classifies a sample based on the majority vote of its closest neighbors in feature space

We won't dive deep into how these models work — that's for future weeks. For now, the point is that different models can produce very different ROC curves on the same data.

In [18]: from [Link] import DecisionTreeClassifier

from [Link] import KNeighborsClassifier

# Train multiple models on the liquefaction data

models = {
'Logistic Regression': LogisticRegression(random_state=42, class_weight='balanced'),
'Decision Tree': DecisionTreeClassifier(random_state=42, class_weight='balanced'),
'K-Nearest Neighbors': KNeighborsClassifier(n_neighbors=5)
}

fig, ax = [Link](figsize=(8, 7))

[Link]([0, 1], [0, 1], 'k--', linewidth=1, label='Random Guessing (AUC=0.50)')

for name, m in [Link]():

[Link](X_train, y_train)
if hasattr(m, 'predict_proba'):
y_score = m.predict_proba(X_test)[:, 1]
else:
y_score = m.decision_function(X_test)
fpr, tpr, _ = roc_curve(y_test, y_score)
roc_auc = auc(fpr, tpr)
[Link](fpr, tpr, linewidth=2, label=f'{name} (AUC={roc_auc:.2f})')

ax.set_xlabel('False Positive Rate', fontsize=12)

ax.set_ylabel('True Positive Rate', fontsize=12)
ax.set_title('ROC Curves — Comparing Liquefaction Models', fontsize=13)
[Link](fontsize=10)
[Link](True, alpha=0.3)
plt.tight_layout()
[Link]()

Key Insight: AUC Tells You How Well Your Model Separates Classes

AUC = 1.0: Perfect separation (every liquefiable site ranked higher than every stable site)
AUC = 0.5: Random guessing (the diagonal — no better than flipping a coin)
Higher AUC = better model, regardless of threshold choice

Think of it like a GPA: it summarizes overall performance without telling you the score on any single test.

In [19]: # Compare AUC values numerically

print("Liquefaction Model Comparison Summary")
print("=" * 55)
for name, m in [Link]():
y_pred_m = [Link](X_test)
if hasattr(m, 'predict_proba'):
y_score_m = m.predict_proba(X_test)[:, 1]
else:
y_score_m = m.decision_function(X_test)
fpr_m, tpr_m, _ = roc_curve(y_test, y_score_m)
roc_auc_m = auc(fpr_m, tpr_m)
acc_m = accuracy_score(y_test, y_pred_m)
print(f"{name:30s} Accuracy={acc_m:.3f} AUC={roc_auc_m:.3f}")

Liquefaction Model Comparison Summary

=======================================================
Logistic Regression Accuracy=0.823 AUC=0.942
Decision Tree Accuracy=0.930 AUC=0.828
K-Nearest Neighbors Accuracy=0.917 AUC=0.849

[DISCUSS] A colleague tells you: "My liquefaction model has AUC = 0.95, so it's great." What questions would you ask before trusting this claim for a real construction project?

Consider:

How imbalanced are the classes? (Most sites don't liquefy)

What specific threshold will they deploy at?
What kind of errors matter most — missing a liquefiable site, or triggering unnecessary ground improvement?
Was the AUC computed on truly unseen test data, or on sites from the same earthquake event?

Key Insight: When to Use Which Curve

ROC curve: Best when classes are roughly balanced. The FPR axis can be misleading when negatives vastly outnumber positives (a small FPR can still mean many false alarms).
Precision-Recall curve: Better for imbalanced datasets (like our liquefaction data with 90% stable sites). It focuses on the minority class.
For most geotechnical safety problems, the precision-recall curve is more informative than ROC.

7. Overfitting — When Your Model Memorizes

Analogy: Studying for an Exam

Imagine two students preparing for a civil engineering exam:

Student A memorizes every answer from past exams word-for-word. On a practice test using those same past questions, they score 100%. But on the real exam with new questions, they fail.
Student B studies the underlying concepts (equilibrium, compatibility, material behavior). They score 85% on practice tests but also 82% on the real exam.

Student A is overfitting. Student B is generalizing. We want our models to be Student B.

A model that performs perfectly on training data but poorly on new data has memorized the noise in the training set rather than learning the underlying pattern. This is one of the most fundamental challenges in machine
learning.

In [20]: # Simple memorization demo: 5 speed-density measurements

[Link](42)

# Training data: 5 traffic sensor readings (vehicles/km → speed km/h)

density_train = [Link]([10, 30, 60, 90, 120])
speed_train = [Link]([105, 88, 55, 25, 8]) + [Link](0, 2, 5)

# Fit a degree-4 polynomial (passes through all 5 points exactly)

coeffs = [Link](density_train, speed_train, 4)

# Test on 3 new sensor readings

density_test = [Link]([20, 50, 100])
# True speeds from the Underwood model
speed_test = 110 * [Link](-density_test / 70)
speed_pred = [Link](coeffs, density_test)

train_error = [Link]((speed_train - [Link](coeffs, density_train))**2)

test_error = [Link]((speed_test - speed_pred)**2)

print("Memorization Demo: Traffic Speed-Density")

print("=" * 45)
print(f"Training error (5 sensors): {train_error:.6f} <- practically zero!")
print(f"Test error (3 new sensors): {test_error:.4f} <- much worse!")
print(f"\nThe model memorized the 5 training readings perfectly,")
print(f"but fails on new traffic data it hasn't seen before.")

Memorization Demo: Traffic Speed-Density

=============================================
Training error (5 sensors): 0.000000 <- practically zero!
Test error (3 new sensors): 140.7395 <- much worse!

The model memorized the 5 training readings perfectly,

but fails on new traffic data it hasn't seen before.

In [21]: [Link](42)
n_pts = 30

# Traffic speed-density relationship (Underwood model: speed = v_f * exp(-density/k_opt))

# v_f = 110 km/h (free-flow speed), k_opt = 70 veh/km (optimum density)
x = [Link]([Link](5, 130, n_pts)) # density (vehicles/km)
y_true = 110 * [Link](-x / 70) # true speed (km/h)
y_noisy = y_true + [Link](0, 6, n_pts) # noisy measurements

fig, axes = [Link](1, 4, figsize=(20, 4))

x_plot = [Link](5, 130, 300)

for ax, degree in zip(axes, [1, 3, 10, 20]):

coeffs = [Link](x, y_noisy, degree)
y_fit = [Link](coeffs, x_plot)

[Link](x, y_noisy, s=40, color='steelblue', edgecolors='k', zorder=3)

[Link](x_plot, y_fit, 'r-', linewidth=2)
[Link](x_plot, 110 * [Link](-x_plot / 70), 'g--', linewidth=1, alpha=0.5, label='True relationship')
ax.set_title(f'Degree {degree}', fontsize=12)
ax.set_ylim(-20, 130)
ax.set_xlabel('Density (veh/km)', fontsize=9)
ax.set_ylabel('Speed (km/h)', fontsize=9)
[Link](True, alpha=0.3)
if degree == 1: [Link](fontsize=9)

[Link]('Polynomial Fits: From Underfitting to Overfitting (Traffic Speed-Density)', fontsize=14, fontweight='bold')

plt.tight_layout()
[Link]()

C:\Users\AFT\AppData\Local\Temp\ipykernel_20412\[Link]: RankWarning: Polyfit may be poorly conditioned

coeffs = [Link](x, y_noisy, degree)

Definition: Underfitting vs. Overfitting

Underfitting (degree 1): A straight line cannot capture the exponential decay of speed with density. High bias, low variance.
Just right (degree 3): Model captures the smooth decay without fitting sensor noise.
Overfitting (degree 20): Model chases every noisy speed reading. Low bias, high variance.

The goal is to find the sweet spot — complex enough to capture real patterns, simple enough to generalize.

Why Train-Test Split?

The polynomial plots above used all 30 data points for both fitting and evaluation. That's like grading a student on the exact questions they practiced — of course they'll do well.

To get an honest evaluation, we need unseen data: data the model has never seen during training.

Definition: Train-Test Split

Divide your dataset into two parts:

Training set (~70-80%): used to fit the model

Test set (~20-30%): held back, used only for evaluation

Never use test data during training or hyperparameter tuning. The test set estimates how well your model will perform on truly new data.

In [22]: fig, ax = [Link](figsize=(10, 2))

# Training data rectangle

[Link](0, 80, left=0, height=0.5, color='steelblue', alpha=0.4, edgecolor='k')
[Link](40, 0, 'Training Data (80%)', ha='center', va='center', fontsize=13, fontweight='bold')

# Test data rectangle

[Link](0, 20, left=80, height=0.5, color='indianred', alpha=0.4, edgecolor='k')
[Link](90, 0, 'Test (20%)', ha='center', va='center', fontsize=13, fontweight='bold')

ax.set_xlim(-2, 102)
ax.set_ylim(-0.6, 0.6)
ax.set_xlabel('All Available Data', fontsize=12)
ax.set_yticks([])
[Link]['top'].set_visible(False)
[Link]['right'].set_visible(False)
[Link]['left'].set_visible(False)

plt.tight_layout()
[Link]()

In [23]: from [Link] import PolynomialFeatures

from [Link] import make_pipeline

# Split data
x_train_ov, x_test_ov, y_train_ov, y_test_ov = train_test_split(
x, y_noisy, test_size=0.3, random_state=42
)

degrees = range(1, 16)

train_errors = []
test_errors = []

for d in degrees:
pipe = make_pipeline(PolynomialFeatures(d), LinearRegression())
[Link](x_train_ov.reshape(-1, 1), y_train_ov)

train_pred = [Link](x_train_ov.reshape(-1, 1))

test_pred = [Link](x_test_ov.reshape(-1, 1))

train_errors.append([Link](mean_squared_error(y_train_ov, train_pred)))
test_errors.append([Link](mean_squared_error(y_test_ov, test_pred)))

# Cap very large test errors for visualization (values above 5 are truncated)
test_errors_capped = [min(e, 5) for e in test_errors]

fig, ax = [Link](figsize=(10, 6))

[Link](list(degrees), train_errors, 'o-', color='steelblue', linewidth=2, label='Training Error (RMSE)')
[Link](list(degrees), test_errors_capped, 's-', color='indianred', linewidth=2, label='Test Error (RMSE)')
[Link](x=3, color='green', linestyle='--', alpha=0.5, label='Sweet spot')
ax.set_xlabel('Polynomial Degree', fontsize=12)
ax.set_ylabel('RMSE', fontsize=12)
ax.set_title('Train vs Test Error: The Overfitting Curve', fontsize=13)
[Link](fontsize=11)
[Link](True, alpha=0.3)
if max(test_errors) > 5:
[Link]('(test errors above 5 truncated)', xy=(0.98, 0.98),
xycoords='axes fraction', ha='right', va='top', fontsize=9, color='gray')
plt.tight_layout()
[Link]()

[TOGETHER] Look at the U-curve above:

1. At degree 1, both training error and test error are high. What does that mean?
2. Around degree 3, test error is lowest. What does that mean?
3. Beyond degree 8, training error approaches zero but test error explodes. What does that mean?
4. The green dashed line marks the sweet spot. In practice, how do you find this spot?

Key Insight: A Model That's Perfect on Training Data is Suspicious, Not Impressive

Training error always decreases with complexity

Test error decreases, then increases (the U-curve)
The gap between train and test error = overfitting

Example: Civil Engineering

You collect speed-density data from 50 highway sensors during one week on the O-4 motorway near Istanbul. Your degree-12 polynomial fits the training data beautifully (R² = 0.99).

Then you deploy it for the following week's traffic management system. At moderate densities, your model predicts negative speeds — a physical impossibility. The polynomial oscillations that fit the noise in
training data produce absurd predictions on new data.

The model memorized the specific traffic patterns of that one week, not the general physics of traffic flow. This is overfitting in the real world.

[TOGETHER] Let's examine the exact train and test error numbers at each polynomial degree. Watch how the status changes from underfitting to good fit to overfitting.

In [24]: # Let's look at the numbers

print(f"{'Degree':>8} {'Train RMSE':>12} {'Test RMSE':>12} {'Gap':>10} {'Status':>15}")
print("-" * 60)
for d, tr_e, te_e in zip(degrees, train_errors, test_errors):
gap = te_e - tr_e
if tr_e > 3.0:
status = "underfitting"
elif gap < 0.5:
status = "good fit"
elif gap < 2.0:
status = "overfitting"
else:
status = "SEVERE overfit"
te_display = min(te_e, 99.99)
print(f"{d:>8d} {tr_e:>12.4f} {te_display:>12.4f} {gap:>10.4f} {status:>15}")

Degree Train RMSE Test RMSE Gap Status

------------------------------------------------------------
1 7.6313 9.3552 1.7239 underfitting
2 5.1146 5.2616 0.1470 underfitting
3 5.1067 5.2376 0.1309 underfitting
4 5.0912 5.2202 0.1290 underfitting
5 4.9190 5.4730 0.5539 underfitting
6 4.9065 5.0998 0.1934 underfitting
7 4.9091 6.1438 1.2347 underfitting
8 4.8094 8.3035 3.4941 underfitting
9 5.0364 11.0746 6.0382 underfitting
10 5.8070 14.1465 8.3395 underfitting
11 7.8784 16.3866 8.5082 underfitting
12 8.4976 30.8951 22.3975 underfitting
13 9.0742 60.5908 51.5167 underfitting
14 9.5977 99.9900 107.4713 underfitting
15 10.0632 99.9900 210.4499 underfitting

Cross-Validation: A More Robust Estimate

The U-curve above depends on which 30% of data ended up in the test set. A different random split might give a different "best degree." How do we get a more reliable answer?

Definition: K-Fold Cross-Validation

1. Split data into k equal parts (folds)

2. Train on k − 1 folds, test on the remaining fold
3. Repeat k times — each fold serves as the test set once
4. Average the k scores for a robust performance estimate

Why it works: Every data point gets used for both training and testing, and the average smooths out the luck of any single split.

In [25]: fig, ax = [Link](figsize=(10, 5))

n_folds = 5
colors_train = 'steelblue'
colors_test = 'indianred'

for i in range(n_folds):
y_pos = n_folds - 1 - i
for j in range(n_folds):
x_start = j * (1.0 / n_folds)
width = 1.0 / n_folds
if j == i:
color = colors_test
alpha = 0.5
label = 'Test'
else:
color = colors_train
alpha = 0.4
label = 'Train'
rect = [Link]((x_start, y_pos - 0.35), width, 0.7,
facecolor=color, alpha=alpha, edgecolor='k', linewidth=1)
ax.add_patch(rect)
if j == i:
[Link](x_start + width / 2, y_pos, 'Test', ha='center', va='center',
fontsize=10, fontweight='bold')
[Link](-0.08, y_pos, f'Fold {i+1}:', ha='right', va='center', fontsize=11)

# Legend
from [Link] import Patch
legend_elements = [Patch(facecolor=colors_train, alpha=0.4, edgecolor='k', label='Train'),
Patch(facecolor=colors_test, alpha=0.5, edgecolor='k', label='Test')]
[Link](handles=legend_elements, loc='upper right', fontsize=11)

ax.set_xlim(-0.15, 1.05)
ax.set_ylim(-0.6, n_folds - 0.2)
ax.set_xticks([])
ax.set_yticks([])
ax.set_title('5-Fold Cross-Validation', fontsize=14)
[Link]['top'].set_visible(False)
[Link]['right'].set_visible(False)
[Link]['bottom'].set_visible(False)
[Link]['left'].set_visible(False)

plt.tight_layout()
[Link]()

In [26]: # Cross-validation on our polynomial regression problem

from [Link] import PolynomialFeatures
from [Link] import make_pipeline

# Use the polynomial data from the overfitting demo above

print("Cross-Validation Scores by Polynomial Degree")
print("=" * 55)
print(f"{'Degree':>8} {'Mean R²':>10} {'Std Dev':>10} {'Verdict':>15}")
print("-" * 55)

for degree in [1, 2, 3, 5, 10]:

pipe = make_pipeline(PolynomialFeatures(degree), LinearRegression())
scores = cross_val_score(pipe, [Link](-1, 1), y_noisy, cv=5, scoring='r2')

if [Link]() > 0.7 and [Link]() < 0.15:

verdict = "Good"
elif [Link]() < 0.3:
verdict = "Underfitting"
else:
verdict = "Overfitting"

print(f"{degree:>8d} {[Link]():>10.3f} {[Link]():>10.3f} {verdict:>15}")

print()
print("Note: High std dev across folds = model is sensitive to which data it sees = overfitting")

Cross-Validation Scores by Polynomial Degree

=======================================================
Degree Mean R² Std Dev Verdict
-------------------------------------------------------
1 -10.044 17.842 Underfitting
2 -0.280 1.304 Underfitting
3 -10.807 22.026 Underfitting
5 -13.346 17.285 Underfitting
10 -16021.269 32038.421 Underfitting

Note: High std dev across folds = model is sensitive to which data it sees = overfitting

Key Insight: Cross-Validation is Your Defense Against Overfitting

Cross-validation is precisely the tool you use to detect overfitting in practice: if your model performs well on all K folds (not just the training set), you have evidence that it generalizes. If performance varies wildly
across folds, your model may be overfitting to specific subsets of the data.

8. Data Leakage — The Silent Killer

Data leakage occurs when information from outside the training dataset is used to create the model. It leads to overly optimistic performance estimates that collapse in production.

This is arguably the most common and most dangerous mistake in applied machine learning.

Analogy: The Exam Answer Key

Imagine you're studying for a final exam and you accidentally find the answer key beforehand. You ace the exam with a perfect score. But you didn't actually learn anything — your "performance" is an illusion.

On the next exam (with different questions), you fail miserably because the leaked answers are no longer available.

Data leakage is the same: your model "saw the answers" during training, so it looks brilliant. But in production (the next exam), it fails.

In [27]: [Link](42)
n = 200

# Earthquake building damage data — inspired by the 2023 Kahramanmaraş earthquake

n_floors = [Link](1, 10, n).astype(float)
building_age = [Link](0, 80, n)
soil_score = [Link](1, 5, n) # 1=rock, 5=soft clay

# True damage score (0-100) based on pre-earthquake features

damage_score = (5 * n_floors + 0.4 * building_age + 8 * soil_score
+ [Link](0, 10, n))
damage_score = [Link](damage_score, 0, 100)

# Post-earthquake observation — only measurable AFTER the event!

observed_crack_width = 0.5 * damage_score + [Link](0, 3, n)
observed_crack_width = [Link](observed_crack_width, 0, None)

df_leak = [Link]({
'n_floors': n_floors,
'building_age': building_age,
'soil_score': soil_score,
'crack_width_mm': observed_crack_width, # LEAKED — post-earthquake measurement!
'damage_score': damage_score # TARGET
})

print("Features available:")
print(df_leak.[Link]())
print("\n'crack_width_mm' was measured AFTER the earthquake — this is data leakage!")
print("You can't know crack widths before the earthquake happens.")

Features available:
['n_floors', 'building_age', 'soil_score', 'crack_width_mm', 'damage_score']

'crack_width_mm' was measured AFTER the earthquake — this is data leakage!

You can't know crack widths before the earthquake happens.

In [28]: # WITH leakage (including post-earthquake crack width)

X_leaked = df_leak[['n_floors', 'building_age', 'soil_score', 'crack_width_mm']].values
y_target = df_leak['damage_score'].values
X_tr, X_te, y_tr, y_te = train_test_split(X_leaked, y_target, test_size=0.2, random_state=42)
model_leaked = LinearRegression().fit(X_tr, y_tr)

# WITHOUT leakage (only pre-earthquake features)

X_clean = df_leak[['n_floors', 'building_age', 'soil_score']].values
X_tr2, X_te2, y_tr2, y_te2 = train_test_split(X_clean, y_target, test_size=0.2, random_state=42)
model_clean = LinearRegression().fit(X_tr2, y_tr2)

print(f"WITH leaked feature (crack width): R² = {model_leaked.score(X_te, y_te):.3f} <- Too good to be true!")
print(f"WITHOUT leaked feature: R² = {model_clean.score(X_te2, y_te2):.3f} <- The real performance")
print(f"\nThe leaked model looks amazing but would fail in practice,")
print(f"because you can't measure crack widths before an earthquake.")

WITH leaked feature (crack width): R² = 0.921 <- Too good to be true!
WITHOUT leaked feature: R² = 0.738 <- The real performance

The leaked model looks amazing but would fail in practice,

because you can't measure crack widths before an earthquake.

In [29]: # Visualize: Predicted vs Actual for both models

fig, axes = [Link](1, 2, figsize=(14, 6))

# Leaked model
y_pred_leaked = model_leaked.predict(X_te)
axes[0].scatter(y_te, y_pred_leaked, alpha=0.6, color='indianred', edgecolors='k', s=50)
axes[0].plot([0, 100], [0, 100], 'k--', linewidth=1.5, label='Perfect prediction')
axes[0].set_xlabel('Actual Damage Score', fontsize=11)
axes[0].set_ylabel('Predicted Damage Score', fontsize=11)
axes[0].set_title(f'WITH Leaked Feature (R² = {model_leaked.score(X_te, y_te):.3f})\nSuspiciously perfect!', fontsize=12)
axes[0].legend(fontsize=10)
axes[0].grid(True, alpha=0.3)

# Clean model
y_pred_clean = model_clean.predict(X_te2)
axes[1].scatter(y_te2, y_pred_clean, alpha=0.6, color='steelblue', edgecolors='k', s=50)
axes[1].plot([0, 100], [0, 100], 'k--', linewidth=1.5, label='Perfect prediction')
axes[1].set_xlabel('Actual Damage Score', fontsize=11)
axes[1].set_ylabel('Predicted Damage Score', fontsize=11)
axes[1].set_title(f'WITHOUT Leaked Feature (R² = {model_clean.score(X_te2, y_te2):.3f})\nRealistic scatter', fontsize=12)
axes[1].legend(fontsize=10)
axes[1].grid(True, alpha=0.3)

[Link]('Data Leakage: The Difference is Visible', fontsize=14, fontweight='bold')

plt.tight_layout()
[Link]()

Key Insight: Data Leakage Gives You False Confidence

Always ask: "Would I have this feature at the time I need to make my prediction?"

3 Common Leakage Patterns:

1. Target leakage: A feature derived from or strongly correlated with the target. Example: Using "observed crack width" (measured after the earthquake) to predict earthquake damage.
2. Train-test contamination: Scaling/normalizing before splitting. Example: Computing mean building height across ALL buildings (including test set) to normalize.
3. Temporal leakage: Using future data to predict the past. Example: Using 2024 repair records to predict which buildings were damaged in 2023.

[PRACTICE] Let's see what happens when we accidentally let test data statistics leak into training through feature scaling.

In [30]: # Train-test contamination example

from [Link] import StandardScaler

# WRONG way: scale before split (test data statistics leak into training)
scaler_wrong = StandardScaler()
X_scaled_wrong = scaler_wrong.fit_transform(X_clean) # fit on ALL data
Xw_tr, Xw_te, yw_tr, yw_te = train_test_split(X_scaled_wrong, y_target, test_size=0.2, random_state=42)
model_wrong = LinearRegression().fit(Xw_tr, yw_tr)

# RIGHT way: scale after split (only use training data statistics)
Xr_tr, Xr_te, yr_tr, yr_te = train_test_split(X_clean, y_target, test_size=0.2, random_state=42)
scaler_right = StandardScaler()
Xr_tr_scaled = scaler_right.fit_transform(Xr_tr) # fit ONLY on training
Xr_te_scaled = scaler_right.transform(Xr_te) # transform test with train stats
model_right = LinearRegression().fit(Xr_tr_scaled, yr_tr)

print("Train-Test Contamination Demo")

print("=" * 50)
print(f"WRONG (scale before split): R² = {model_wrong.score(Xw_te, yw_te):.4f}")
print(f"RIGHT (scale after split): R² = {model_right.score(Xr_te_scaled, yr_te):.4f}")
print(f"\nIn this case the difference is small because we have few features")
print(f"and plenty of data. With more features and less data, contamination")
print(f"can drastically inflate scores.")

Train-Test Contamination Demo

==================================================
WRONG (scale before split): R² = 0.7379
RIGHT (scale after split): R² = 0.7379

In this case the difference is small because we have few features

and plenty of data. With more features and less data, contamination
can drastically inflate scores.

Example: Civil Engineering — The Timeline Test

Imagine you are predicting building damage in a future earthquake. Ask yourself: "When does each feature become available?"

| Feature | Available before the earthquake? | Why? | |---|---|---| | Number of floors | Yes | Known from building records | | Building age | Yes | Known from permit records | | Soil type at site | Yes | Known from
geological surveys | | Observed crack width | No | Only measurable after the earthquake | | Differential settlement | No | Only measurable after the earthquake | | Repair cost estimate | No — this is essentially
the target! | Circular reasoning |

This is directly relevant to Turkey's post-earthquake building screening program (Law 6306). The goal is to identify at-risk buildings before the next earthquake — so only pre-earthquake features are valid.

[PRACTICE] You are building a model to predict landslide risk along a highway corridor in the Black Sea region. Your dataset includes:

slope_angle , elevation , rainfall_monthly , vegetation_index , soil_type , distance_to_road

road_closure_status (whether the road was closed due to a landslide)
landslide_occurred (target: yes/no)

Which feature(s) should you NOT use? Why?

(Hint: Apply the timeline test — would you have this feature BEFORE the landslide happens?)

9. Putting It Together
Model Evaluation Checklist
Before you report results, ask yourself:

# Question If "No", Then...

1 Is my dataset balanced? Accuracy is misleading — use precision, recall, F1

2 Do I know the cost of each error type? Define: What's worse, FP or FN?

3 Am I evaluating on truly unseen data? Use train-test split or cross-validation

4 Does my model generalize? Check train vs test error gap

5 Could there be data leakage? Audit every feature: would I have it at prediction time?

Mini Capstone: Water Pipe Failure Prediction

Istanbul's water utility ISKI manages one of the largest urban pipe networks in Europe — over 20,000 km of water mains serving 16 million people. Aging infrastructure means pipe failures are inevitable, but inspecting every
pipe is impossible.

You've been hired to build a model that predicts which pipe segments are most likely to fail in the next year. Your dataset: 5,000 pipe segments — 4,750 operational and 250 with recorded failures.

Let's walk through the full evaluation pipeline with this scenario.

In [31]: # Generate the water pipe failure dataset

[Link](42)

n_pipes = 5000
n_operational = 4750
n_failed = 250

# Features for operational pipes

X_oper = np.column_stack([
[Link](5, 80, n_operational), # pipe_age (years)
[Link](100, 600, n_operational), # diameter_mm
[Link](5, 500, n_operational), # length_m
[Link]([0, 1, 2, 3, 4], n_operational, p=[0.15, 0.25, 0.30, 0.20, 0.10]), # material (0=cast iron, 1=ductile iron, 2=PVC, 3=steel, 4=concrete)
[Link](2, 8, n_operational), # pressure_bar
[Link]([1, 2, 3], n_operational, p=[0.4, 0.4, 0.2]), # soil_corrosivity (1=low, 2=medium, 3=high)
[Link](0.8, 3.0, n_operational), # burial_depth_m
[Link](1, n_operational) # prev_breaks
])

# Features for failed pipes (older, smaller, cast iron, more corrosive soil, more breaks)
X_fail = np.column_stack([
[Link](40, 120, n_failed), # older pipes
[Link](75, 300, n_failed), # smaller diameter
[Link](5, 500, n_failed), # length_m
[Link]([0, 1, 2, 3, 4], n_failed, p=[0.45, 0.25, 0.10, 0.15, 0.05]), # mostly cast iron
[Link](3, 8, n_failed), # pressure_bar
[Link]([1, 2, 3], n_failed, p=[0.15, 0.35, 0.50]), # more corrosive soil
[Link](0.8, 2.5, n_failed), # burial_depth_m
[Link](4, n_failed) # more previous breaks
])

X_pipe = [Link]([X_oper, X_fail])

y_pipe = [Link]([0] * n_operational + [1] * n_failed)

# Shuffle
shuffle_idx = [Link](n_pipes)
X_pipe, y_pipe = X_pipe[shuffle_idx], y_pipe[shuffle_idx]

feature_names = ['pipe_age', 'diameter_mm', 'length_m', 'material', 'pressure_bar',

'soil_corrosivity', 'burial_depth_m', 'prev_breaks']

print(f"Dataset: {n_pipes} pipe segments ({n_operational} operational, {n_failed} failed)")

print(f"Class balance: {n_failed/n_pipes*100:.1f}% failed")
print(f"Features: {', '.join(feature_names)}")

Dataset: 5000 pipe segments (4750 operational, 250 failed)

Class balance: 5.0% failed
Features: pipe_age, diameter_mm, length_m, material, pressure_bar, soil_corrosivity, burial_depth_m, prev_breaks

In [32]: # Train model and evaluate

X_pipe_train, X_pipe_test, y_pipe_train, y_pipe_test = train_test_split(
X_pipe, y_pipe, test_size=0.3, random_state=42, stratify=y_pipe
)

pipe_model = LogisticRegression(random_state=42, class_weight='balanced', max_iter=1000)

pipe_model.fit(X_pipe_train, y_pipe_train)
y_pipe_pred = pipe_model.predict(X_pipe_test)
y_pipe_proba = pipe_model.predict_proba(X_pipe_test)[:, 1]

# Results
acc_pipe = accuracy_score(y_pipe_test, y_pipe_pred)
cm_pipe = confusion_matrix(y_pipe_test, y_pipe_pred)

print("Water Pipe Failure Prediction — Model Results")

print("=" * 60)
print(f"\nAccuracy: {acc_pipe:.1%}")
print(f"\nConfusion Matrix:")
print(f" TN={cm_pipe[0,0]:4d} FP={cm_pipe[0,1]:4d}")
print(f" FN={cm_pipe[1,0]:4d} TP={cm_pipe[1,1]:4d}")
print(f"\nClassification Report:")
print(classification_report(y_pipe_test, y_pipe_pred, target_names=['Operational', 'Failed']))

Water Pipe Failure Prediction — Model Results

============================================================

Accuracy: 95.9%

Confusion Matrix:
TN=1367 FP= 58
FN= 4 TP= 71

Classification Report:
precision recall f1-score support

Operational 1.00 0.96 0.98 1425

Failed 0.55 0.95 0.70 75

accuracy 0.96 1500

macro avg 0.77 0.95 0.84 1500
weighted avg 0.97 0.96 0.96 1500

precision recall f1-score support

Operational 1.00 0.96 0.98 1425

Failed 0.55 0.95 0.70 75

accuracy 0.96 1500

macro avg 0.77 0.95 0.84 1500
weighted avg 0.97 0.96 0.96 1500

In [33]: # ROC curve for the pipe failure model

fpr_pipe, tpr_pipe, _ = roc_curve(y_pipe_test, y_pipe_proba)
auc_pipe = auc(fpr_pipe, tpr_pipe)

fig, ax = [Link](figsize=(7, 6))

[Link](fpr_pipe, tpr_pipe, color='steelblue', linewidth=2, label=f'Model (AUC={auc_pipe:.2f})')
[Link]([0, 1], [0, 1], 'k--', linewidth=1, label='Random guessing')
ax.set_xlabel('False Positive Rate', fontsize=12)
ax.set_ylabel('True Positive Rate', fontsize=12)
ax.set_title('ROC Curve — Water Pipe Failure Model', fontsize=13)
[Link](fontsize=11)
[Link](True, alpha=0.3)
plt.tight_layout()
[Link]()

[TOGETHER] Looking at the pipe failure model results above:

1. Is the accuracy misleadingly high? (Check the class balance — 95% of pipes are operational.)
2. What does the confusion matrix tell you about false negatives? How many pipe failures did we miss?
3. For this application (water infrastructure), should we optimize for precision or recall? Consider: a missed pipe failure can flood a neighborhood, but sending a crew to a healthy pipe wastes resources.
4. If ISKI can only inspect 500 pipe segments this year, how would you use the model's probability scores to prioritize?

[TOGETHER] If ISKI can inspect 500 pipe segments this year, which threshold gives the best balance between catching failures and not wasting crew time?

In [34]: # Threshold analysis: What happens at different decision thresholds?

print("Threshold Analysis — Water Pipe Failure Model")
print("=" * 75)
print(f"{'Threshold':>10} {'Precision':>10} {'Recall':>10} {'FP (false alarms)':>18} {'FN (missed)':>12}")
print("-" * 75)

for t in [0.1, 0.2, 0.3, 0.5, 0.7]:

y_t = (y_pipe_proba >= t).astype(int)
tp = [Link]((y_t == 1) & (y_pipe_test == 1))
fp = [Link]((y_t == 1) & (y_pipe_test == 0))
fn = [Link]((y_t == 0) & (y_pipe_test == 1))
prec = tp / (tp + fp) if (tp + fp) > 0 else 0
rec = tp / (tp + fn) if (tp + fn) > 0 else 0
print(f"{t:>10.1f} {prec:>10.2f} {rec:>10.2f} {fp:>18d} {fn:>12d}")

print(f"\nTotal failed pipes in test set: {[Link](y_pipe_test)}")

print(f"Total operational pipes in test set: {[Link](y_pipe_test == 0)}")

Threshold Analysis — Water Pipe Failure Model

===========================================================================
Threshold Precision Recall FP (false alarms) FN (missed)
---------------------------------------------------------------------------
0.1 0.31 0.97 165 2
0.2 0.38 0.95 117 4
0.3 0.43 0.95 93 4
0.5 0.55 0.95 58 4
0.7 0.61 0.89 42 8

Total failed pipes in test set: 75

Total operational pipes in test set: 1425

In [35]: # Self-check quiz

print("Quick Self-Check")
print("=" * 50)
print()
print("Q1: Your pipe failure model has 95% accuracy.")
print(" Should you celebrate? Why or why not?")
print(" # Your answer: ")
print(" # Hint: What percentage of pipes are operational?")
print()
print("Q2: Your model has perfect R² on training data.")
print(" What should you suspect?")
print(" # Your answer: ")
print(" # Hint: Think about Student A vs Student B from Section 5")
print()
print("Q3: You're predicting water pipe failure. Which is worse:")
print(" predicting 'operational' when it's about to burst (FN), or")
print(" sending a crew to an intact pipe (FP)?")
print(" # Your answer: ")
print(" # Hint: What are the consequences of each error?")
print()
print("Q4: Your model uses 'repair_cost' as a feature. This cost")
print(" was recorded AFTER the pipe failed. Is this a problem?")
print(" # Your answer: ")
print(" # Hint: Apply the timeline test from Section 6")

Quick Self-Check
==================================================

Q1: Your pipe failure model has 95% accuracy.

Should you celebrate? Why or why not?
# Your answer:
# Hint: What percentage of pipes are operational?

Q2: Your model has perfect R² on training data.

What should you suspect?
# Your answer:
# Hint: Think about Student A vs Student B from Section 5

Q3: You're predicting water pipe failure. Which is worse:

predicting 'operational' when it's about to burst (FN), or
sending a crew to an intact pipe (FP)?
# Your answer:
# Hint: What are the consequences of each error?

Q4: Your model uses 'repair_cost' as a feature. This cost

was recorded AFTER the pipe failed. Is this a problem?
# Your answer:
# Hint: Apply the timeline test from Section 6

Connection to the CE49X Course

This notebook gives you the evaluation framework for:

Week 07 (Naive Bayes): Before reporting "my Naive Bayes has 94% accuracy," ask — accuracy at what cost?
Week 08 (SVM): Before choosing an SVM kernel, check — am I overfitting to the training set?

Key Insight: Model Evaluation is Not a Step — It's a Mindset

Evaluation pervades the entire workflow. From data collection (is this representative?) to deployment (is the model still accurate on new data?).

Machine Learning in Civil Engineering

Today we explored ML across five CE sub-disciplines:

1. Energy & Sustainability — predicting building heating loads from geometry

2. Geotechnical Engineering — soil liquefaction screening from borehole data
3. Transportation Engineering — modeling traffic speed-density relationships
4. Structural/Earthquake Engineering — detecting data leakage in damage prediction
5. Water Infrastructure — prioritizing pipe inspections with limited budgets

The Right Question

Don't ask: "What's my model's accuracy?"

Ask: "What's the cost of my model's errors, and can I live with it?"

This question should follow you into every ML project you work on — in this course and beyond.

Questions?
Dr. Eyuphan Koc [Link]@[Link]