Lab Part 1: Statistical Learning Theory
Customer Churn Prediction. Generalization in Machine Learning
1. Definition of Generalization
Generalization in statistical learning theory is about a models ability to work
well on data that it hasn't seen before. This new data was not part of its training
set. A model generalizes well if it learns patterns from old data not just noise or
weird things about the training data.
In customer churn prediction, a model that generalizes well doesn't just
remember which customers left in the past. Instead it figures out the reasons
customers leave, like low engagement or poor onboarding. Then it uses that
understanding to predict which customers will leave in the future.
We measure generalization by looking at the difference between how the model
fits old data and how well it works on new data. If the model does well on both
that's generalization.
2. Structuring Model Development to Maximize Generalization
Step 1. Data Splitting
Split your data into three parts:
* Training set (about 70%): used to fit the model
* Validation set (about 15%): used to tweak the model and compare models
* Test set (about 15%): not used until the very end to see how the model really
performs
This way the test set shows how the model will really work and its not used to
make decisions about the model.
Step 2. Exploratory Data Analysis (EDA)
Before modeling look at the data carefully. Understand if one type of customer
is more common than the other check for missing values and see how features
are related. This helps prevent training on misleading data.
Step 3. Feature Engineering
Make features that really capture how customers behave, like how its been
since they logged in or how much they spend on average. Good features that
make sense help the model generalize better than models on raw data.
Step 4. Cross-Validation
Use kfold cross-validation (usually k=5 or k=10) during training. Split the data
into k parts train on k-1 parts and validate on the part. Rotate through all parts.
This gives an idea of how well the model generalizes.
Step 5. Regularization
Use techniques to prevent the model from fitting too closely to the training
data:
* L1 (Lasso) regularization: makes unimportant features zero
* L2 (Ridge) regularization: penalizes big coefficients
* Tree-based models: control complexity with parameters
Step 6. Final Evaluation
Report metrics like AUC-ROC, precision, recall and F1-score on the test set and
only do it once at the very end. Using the test set times gives too good of
performance estimates.
3. Detecting and Addressing Overfitting and Underfitting
Detection
The main clue is the difference between training and validation performance:
Problem | Signal
---------|--------
Overfitting High training accuracy, much lower validation accuracy
Underfitting | Low accuracy on both training and validation
Plotting learning curves. Model performance versus training data size. Helps
diagnose. Overfitting shows a gap between training and validation curves.
Underfitting shows both curves plateauing at a level.
Adjustments for Overfitting
* Increase regularization strength
* Reduce model complexity
* Gather training data
* Apply dropout (for neural networks)
* Use methods like bagging
Adjustments for Underfitting
* Use a complex model
* Engineer informative features
* Reduce regularization strength
* Allow the model to train longer
4. Real-World Business Impact of Poor Generalization
Scenario
Imagine a churn model trained during a discount campaign. Customers who left
did so because of the discount. The model learns to associate behaviors with
churn.
When deployed normally these patterns don't hold. The model fails to identify
at-risk customers and flags the ones.
Business Consequences
* Decision-making: The retention team sends offers to the customers wasting
budget.
* Customer retention: Real churners go unidentified and leave without
intervention.
* Business performance: Management trusts the models accuracy but churn
rates rise. This erodes trust in data-driven decisions.
A model that performs in backtesting but fails in production is often more
dangerous than no model. It creates false confidence, in flawed predictions.