PYTHON PROGRAMMING FOR DATA SCIENCE
Classification & Model evaluation
Logistic Regression · Decision Trees · Confusion Matrix · Cross-validation
1. From regression to classification
Last session you predicted a number (the final exam score). This session you predict a
category. Using the same students and the same features, we ask a yes/no question instead:
• Regression (S12): how many points will the student score? -> final_score = 78
• Classification (today): will the student pass (score >= 75)? -> passed = 1
Both are supervised learning. The difference is only the kind of answer: a continuous number vs
a discrete label.
Data and a leakage trap
Dataset students_pass.csv: 220 students, 156 pass / 64 fail (about 71% / 29%). Features:
study_hours, attendance_pct, prior_gpa. Target: passed.
Do NOT use final_score as a feature: passed is derived from it, so feeding it in is data leakage
(the model would “cheat”). Predict from study habits only.
2. Logistic regression
A linear model can output any number, but a class needs a probability between 0 and 1. Logistic
Regression passes the linear output through the sigmoid function, then applies a threshold (0.5
by default): probability >= 0.5 -> Pass, else Fail.
Despite its name, Logistic Regression is a classification model. In scikit-learn:
Python
from sklearn.linear_model import LogisticRegression
model = LogisticRegression(max_iter=2000)
[Link](X_train, y_train)
[Link](X_test) # 0 / 1 labels
model.predict_proba(X_test) # probability of each class
predict vs predict_proba
predict_proba returns the probability of each class (e.g. 0.82 chance of Pass). predict applies
the 0.5 threshold and returns the label directly.
3. Decision tree
A Decision Tree asks one yes/no question at a time, splitting students into purer groups until it
can guess Pass or Fail. Here is a tree (depth 2) trained on our data:
Python
from [Link] import DecisionTreeClassifier
tree = DecisionTreeClassifier(max_depth=3, random_state=42)
[Link](X_train, y_train)
[Link](X_test)
max_depth controls complexity
A shallow tree gives simple rules but may underfit; a very deep tree memorises the training data
and overfits. Limiting depth keeps it honest - more on overfitting in Section 8.
4. Why accuracy is not enough
Our classes are imbalanced: 71% of students pass. A lazy model that predicts “Pass” for
everyone is already 71% accurate - and useless, because it never flags a struggling student.
The key question
Accuracy alone can hide failure on the minority class. We need metrics that ask: of those
predicted Pass, how many really passed (precision)? Of those who really passed, how many did
we catch (recall)?
5. The confusion matrix
Every test prediction lands in one of four cells. For Logistic Regression on our 44 test students:
Predicted Fail Predicted Pass
Actual Fail TN = 11 FP = 2
Actual Pass FN = 4 TP = 27
• TP / TN - correct predictions (Pass called Pass, Fail called Fail).
• FP - a Fail wrongly called Pass. FN - a real Pass we missed.
Accuracy = (TP + TN) / all = (27 + 11) / 44 = 0.86.
6. Precision, Recall, F1
These three look past accuracy. Definitions (for the positive class, Pass):
• Precision = TP / (TP + FP) - of those predicted Pass, how many really passed?
• Recall = TP / (TP + FN) - of those who really passed, how many did we catch?
• F1-score = harmonic mean of precision and recall - one balanced number.
scikit-learn prints all of them per class with one call:
Python
from [Link] import classification_report
print(classification_report(y_test, [Link](X_test)))
report
precision recall f1-score support
Fail 0.73 0.85 0.79 13
Pass 0.93 0.87 0.90 31
accuracy 0.86 44
Watch the minority class
Read the smaller class (Fail). Its recall (0.85 here) tells you how many at-risk students the model
actually catches - far more useful than overall accuracy.
7. Comparing two models
On the same test set:
Model Accuracy Precision Recall F1
Logistic Regression 0.86 0.93 0.87 0.90
Decision Tree (depth 3) 0.89 0.93 0.90 0.92
The tree edges ahead here, but a small gap on a single split is not proof. The honest
comparison uses cross-validation (next).
8. Overfitting, underfitting, and cross-validation
A model can be too simple (underfit) or too complex (overfit). The goal is the middle - capturing
the real pattern while ignoring noise.
A single train/test split can be lucky or unlucky. k-fold cross-validation rotates the test fold k
times and averages the scores for a steadier estimate:
Python
from sklearn.model_selection import cross_val_score
cross_val_score(model, X, y, cv=5).mean()
5-fold accuracy
0.877
Averaged over 5 folds, Logistic Regression scores about 0.88 - more trustworthy than any
single split.
9. A Second example: Loan default
The exact same workflow applies to finance. Using [Link] (320 loans, ~37% default), we
predict whether a loan will default from income, loan amount, asset value, and term.
Loan default - Logistic Regression Value
Accuracy 0.73
Precision (default) 0.69
Recall (default) 0.48
Why recall matters here
Recall on defaults is only 0.48 - the model misses about half the loans that actually default. For a
lender a missed default costs far more than a false alarm, so recall is the metric to push up. Same
code, different stakes: the right metric depends on the cost of each error.
10. Common mistakes
• Trusting accuracy on imbalanced data. Check precision and recall, especially on the
minority class.
• Data leakage. Using a feature derived from the target (like final_score) inflates every
metric.
• Unlimited tree depth. A maxed-out tree memorises the training data. Limit depth.
• Ignoring the costly error. Decide whether a false positive or false negative hurts more,
then optimise for it.
Exercises
Dataset students_pass.csv; the last exercise uses [Link].
Exercise 1
Load students_pass.csv. Define X with the three features (not final_score!) and y = passed. Split
80/20 (random_state=42), fit a Logistic Regression, and print the test accuracy
Exercise 2 - Confusion matrix
Compute the confusion matrix on the test set with confusion_matrix(y_test,
[Link](X_test)). Identify TN, FP, FN, TP in the output.
Exercise 3 - Precision & recall
Print classification_report. What is the recall for the Fail class, and why does it matter more
than overall accuracy here?
Exercise 4 - Decision Tree
Train a DecisionTreeClassifier(max_depth=3, random_state=42) on the same split. Compare its
accuracy, precision, recall, and F1 with the Logistic Regression. Which would you choose, and
why?
Exercise 5 - Cross-validate the depth
Use cross_val_score(..., cv=5) to compare Logistic Regression against trees of depth 2, 3, and
8. Which depth generalises best? What happens as depth grows large?
Summary
• Classification predicts a category; Logistic Regression uses the sigmoid to output a
probability.
• A Decision Tree splits with yes/no questions; limit its depth to avoid overfitting.
• The confusion matrix (TP, TN, FP, FN) is the basis of precision, recall, and F1.
• On imbalanced data, look past accuracy to precision and recall on the minority class.
• Cross-validation averages over folds for an honest score.