Statistical Encoding
Statistical encoding is a category of data encoding techniques that transforms categorical
variables into numerical representations based on the statistical relationship between the category
and the target variable. Unlike methods such as one-hot encoding or label encoding, which do
not consider the target, statistical encoding creates more informative features that often improve
the performance of machine learning models.
Common statistical encoding methods
Mean (or Target) Encoding: This technique replaces each category with the average (mean) of
the target variable for that category. It is a powerful method for handling high-cardinality
features, where a large number of unique categories would cause one-hot encoding to create too
many columns.
o How it works (binary target): For a binary classification problem, each category is replaced
with the proportion of the target variable that is 1. For a regression problem, it is replaced with
the mean of the target variable.
o Smoothing: To prevent overfitting on categories with very few data points, a smoothing
technique is often applied. This involves blending the category's mean with the overall mean of
the target variable, especially for low-frequency categories.
Weight of Evidence (WoE) Encoding: Primarily used for binary classification, WoE
encoding measures the predictive power of a categorical variable with respect to the target
variable.
How it works: It calculates the logarithm of the ratio of the proportion of "good"
outcomes (e.g., non-defaults) to the proportion of "bad" outcomes (e.g., defaults) for each
category.
Benefit: The resulting WoE-transformed variable has a direct relationship with the log-
odds of the target, making it well-suited for logistic regression.
Leave-One-Out (LOO) Encoding: A variant of mean encoding that prevents target leakage
by calculating the mean of the target for a category using all other rows, but not the current row.
This technique is more computationally intensive but can be useful for reducing overfitting.
M-estimator Encoding: A type of target encoding that uses a blend of the target mean for a
specific category and the overall target mean, controlled by a parameter (m).
CatBoost Encoding: A variation of target encoding used by the CatBoost gradient boosting
algorithm. It calculates target statistics dynamically during training by using only the history of
observed data, which helps to prevent target leakage and overfitting.
Advantages
Dimensionality Reduction: Statistical encoding consolidates a high-cardinality categorical
feature into a single, highly predictive numerical feature, avoiding the curse of dimensionality
caused by one-hot encoding.
Information Capture: It embeds predictive power directly into the feature by capturing the
relationship between the category and the target variable.
Better Model Performance: The encoded features often provide tree-based models with more
signal, leading to higher accuracy compared to other encoding methods.
Feature Interpretability: For methods like WoE, the encoded feature offers a clear,
interpretable representation of a category's influence on the target odds.
Disadvantages
Risk of Overfitting: Without proper regularization (like smoothing or cross-validation),
encoding on small datasets can cause the model to memorize the training data.
Target Leakage: Statistical encoding is considered a supervised technique, and improper
implementation (e.g., encoding the entire dataset before splitting) can cause the model to use
information from the test set during training, leading to overly optimistic results.
Issues with New Categories: Models can't handle categories not seen during training. This can
be addressed by assigning a default value (like the overall mean) or grouping rare labels into a
single "Other" category.
Only for Supervised Learning: Since these methods rely on the target variable, they are not
applicable for unsupervised tasks.