21/11/2025, 17:05 Feature Engineering Techniques and Best Practices - DeepSeek

[Link]
PDF 728.27KB

Take the attached file and summarize the main ideias (I need to
understand what the concepts mean) of each sections and
subsections with topics considering the main ideas page by page. If
needed to express the main ideas and not lose good information, the
subsections can be described using a couple of topics. do all page.
Extract also the code from each page in a formatted way with
reserved words colored.

Of course. Here is a detailed, page-by-page summary of the main ideas and

concepts from the provided chapter, along with any code snippets extracted and
formatted.

Page 1

Main Idea: Introduction to Feature Engineering

Feature engineering is critically important for ML model performance, often

providing a bigger boost than advanced algorithms.

A significant portion of ML jobs involves creating new, useful features.

The chapter will cover common techniques and a major pitfall: data leakage.

Subsections:

Learned Features vs. Engineered Features: Introduces the debate on whether

deep learning (feature learning) eliminates the need for manual feature
engineering. Concludes that while deep learning automates feature extraction for
some data types (text, images), it's not universal, and many production systems
still rely on handcrafted features.

Page 2

Main Idea: Manual Feature Engineering Example (N-grams)

Illustrates the old, manual way of engineering features for text (NLP) before deep
learning.
[Link] 1/12
21/11/2025, 17:05 Feature Engineering Techniques and Best Practices - DeepSeek

N-gram: A contiguous sequence of n items (words, characters) from a text

sample.

1-grams (unigrams): ["I", "like", "food"]

2-grams (bigrams): ["I like", "like food"]

Page 3

Main Idea: Vectorizing Text with N-grams

After generating n-grams, you create a vocabulary that maps each n-gram to an
index.

Each text sample is then converted into a vector where each element represents
the count of a specific n-gram from the vocabulary in that sample.

Page 4

Main Idea: The Shift with Deep Learning & Beyond Simple Data

Deep learning has automated much of the manual text preprocessing

(lemmatization, etc.) by using tokenization and embeddings.

However, ML systems often need data beyond just text or images.

Feature Engineering is the process of choosing what information to use and how
to extract it into a format usable by ML models. This can involve domain expertise
and result in millions of features for complex tasks.

Page 5

Main Idea: Common Feature Engineering Operations

Lists fundamental techniques to prepare data for models.

Key operations include:

Handling Missing Values

Scaling

[Link] 2/12
21/11/2025, 17:05 Feature Engineering Techniques and Best Practices - DeepSeek

Discretization

Encoding Categorical Features

Feature Crossing

Positional Embeddings

Page 6

Main Idea: Types of Missing Values

Not all missing values are the same. Three types are defined:

MNAR (Missing Not at Random): The value is missing because of the true value
itself (e.g., high-income earners refuse to disclose income).

MAR (Missing at Random): The value is missing due to another observed variable
(e.g., people of a specific gender don't like disclosing age).

MCAR (Missing Completely at Random): There is no pattern; the missingness is

random (very rare).

Page 7

Main Idea: Handling Missing Values - Deletion

Deletion is a simple but often suboptimal method.

Column Deletion: Remove a variable/feature if it has too many missing values.

Risk: Losing important information.

Row Deletion: Remove a sample/row if it has any missing values. Risk: Only works
if data is MCAR and the number of deletions is small; otherwise, it can introduce
bias and remove valuable information.

Page 8

Main Idea: Handling Missing Values - Imputation

Imputation means filling in missing values, which is preferred over deletion to

avoid information loss and bias.
[Link] 3/12
21/11/2025, 17:05 Feature Engineering Techniques and Best Practices - DeepSeek

Common but risky practices:

Fill with defaults (e.g., empty string "" ).

Fill with mean, median, or mode.

Pitfall: Avoid filling with a possible value (e.g., filling missing number_of_children
with 0 ), as it becomes impossible to distinguish between "missing" and "true
zero".

There is no perfect solution; both deletion and imputation have risks, including
data leakage.

Page 9

Main Idea: Feature Scaling

Why Scale? Features with vastly different ranges (e.g., Age vs. Annual Income)
can cause models to incorrectly prioritize features with larger values.

Min-Max Scaling (Normalization): Rescales features to a range, typically [0, 1].

Formula for [0, 1] range:

python

# Min-Max Scaling to [0, 1]

x_scaled = (x - min(x)) / (max(x) - min(x))

Formula for arbitrary [a, b] range:

python

# Min-Max Scaling to [a, b]

x_scaled = a + ( (x - min(x)) * (b - a) ) / (max(x) - min(x))

Page 10

Main Idea: Standardization and Log Transformation

Standardization: Rescales features to have a mean of 0 and a standard deviation

of 1. Useful if features are (or are assumed to be) normally distributed.

[Link] 4/12
21/11/2025, 17:05 Feature Engineering Techniques and Best Practices - DeepSeek

Formula:

python

# Standardization (Zero Mean, Unit Variance)

x_standardized = (x - mean(x)) / std(x)

Log Transformation: Applies the log function to reduce skewness in the data
distribution.

Important Notes on Scaling:

1. It's a common source of data leakage. Statistics (min, max, mean) must be
calculated only on the training data and applied to validation/test data.

2. It requires global statistics, so models must be retrained if the data distribution

changes over time.

Page 11

Main Idea: Discretization (Binning/Quantization)

Discretization converts continuous features into discrete categories/buckets.

Why use it? It simplifies the learning task for the model by reducing the infinite
possible values of a continuous feature into a finite set (e.g., Low, Middle, Upper
income).

Downsides:

Introduces hard boundaries (e.g., $34,999 is "Low" but $35,000 is "Middle").

Choosing the right bucket boundaries can be challenging (use histograms,

quantiles, domain knowledge).

Page 12

Main Idea: The Challenge of Dynamic Categorical Features

In production, categories are often not static; new categories appear all the time
(new brands, new users, new IPs).

[Link] 5/12
21/11/2025, 17:05 Feature Engineering Techniques and Best Practices - DeepSeek

A naive approach of assigning a unique number to each category fails when new,
unseen categories appear in production, causing model crashes or poor
performance.

Page 13

Main Idea: Solution - The Hashing Trick

The Hashing Trick is a solution for handling dynamic or high-cardinality

categorical features.

How it works: A hash function maps each category (even unseen ones) to a fixed
number of buckets (the hash space). This fixes the feature dimension in advance.

Problem: Collisions: Different categories can be hashed to the same bucket.

Mitigation: Collisions are often random, and their impact on model performance is
usually minimal. A larger hash space reduces collision probability.

Page 14

Main Idea: Feature Crossing

Feature Crossing combines two or more features to create a new feature that
captures their interaction.

Why use it? Essential for modeling nonlinear relationships in models that are
inherently linear (like Linear/Logistic Regression) or weak at learning interactions
(like basic trees).

Caveats:

It can cause the feature space to explode (Feature A: 100 values x Feature B:
100 values = 10,000 new features).

It increases the risk of overfitting.

Page 15

(Continuation of Feature Crossing with an example table)

[Link] 6/12
21/11/2025, 17:05 Feature Engineering Techniques and Best Practices - DeepSeek

Page 16

Main Idea: Introduction to Positional Embeddings

Embedding: A vector that represents a piece of data (words, products, users).

Positional Embedding: Necessary for models that process data in parallel (like
Transformers) to provide information about the order or position of elements in a
sequence.

Simply using raw position indices (0, 1, 2,...) doesn't work well because they are
not scaled and neural networks struggle with non-unit-variance inputs.

Page 17

Main Idea: Learned vs. Fixed Positional Embeddings

Learned Positional Embeddings: Treat position indices like words. An embedding

matrix is created where each position (0, 1, 2,...) has a trainable vector. These
vectors are learned during training.

Fixed Positional Embeddings: The embedding vector for each position is

predefined using mathematical functions (sine and cosine), not learned.

Page 18

Main Idea: Fourier Features for Continuous Positions

Fixed positional embedding is a specific case of Fourier Features.

This technique can be generalized to handle continuous coordinates (e.g., 3D

points on an object), not just discrete positions.

General Formula for Fourier Features:

python

# Generalized Fourier Features for coordinate v

# a_i, b_i are parameters, m is the number of frequency components
gamma(v) = [ a1 * cos(2*pi * b1^T * v), a1 * sin(2*pi * b1^T * v),

[Link] 7/12
21/11/2025, 17:05 Feature Engineering Techniques and Best Practices - DeepSeek

...,
am * cos(2*pi * bm^T * v), am * sin(2*pi * bm^T * v) ]

Page 19

Main Idea: Introduction to Data Leakage

Data Leakage: A critical problem where information from the label "leaks" into the
features during training, but that same information is unavailable during inference,
causing models to fail spectacularly in production.

Examples:

A model predicting COVID-19 risk learned from the patient's position in a scan
(lying down vs. standing up), which was correlated with illness severity.

A model learned to recognize text fonts from specific hospitals, which were
correlated with COVID-19 caseloads.

Page 20

Main Idea: Common Cause 1 - Incorrect Data Splitting

Problem: Randomly splitting time-correlated data (e.g., stock prices, user clicks)
instead of splitting by time.

Consequence: Future information "leaks" into the training set, allowing the model
to "cheat" by seeing data from what is effectively its future. This gives an
unrealistically high performance during evaluation that won't hold in production.

Solution: Always split data by time (e.g., use first 4 weeks for training, the next
week for validation/test).

Page 21

Main Idea: Common Cause 2 - Scaling Before Splitting

Problem: Calculating scaling statistics (mean, min, max) using the entire dataset
before splitting it into train/validation/test.

[Link] 8/12
21/11/2025, 17:05 Feature Engineering Techniques and Best Practices - DeepSeek

Consequence: Information from the test set "leaks" into the training process via
these global statistics.

Solution: Split data first, then calculate scaling statistics only on the training set.
Apply these training-set statistics to scale the validation and test sets.

Page 22

Main Idea: Common Causes 3 & 4 - Imputation and Duplication

Cause 3: Imputation with Test Statistics: Similar to scaling, using the entire
dataset's mean/median to fill missing values leaks test information. Solution: Use
statistics from only the training set.

Cause 4: Poor Handling of Data Duplication: If duplicates exist across train and
test splits, the model is effectively evaluated on data it has already seen. Solution:
Check for and remove duplicates before splitting. If oversampling, do it after
splitting.

Page 23

Main Idea: Common Causes 5 & 6 - Group and Data Generation Leakage

Cause 5: Group Leakage: Highly correlated samples (e.g., multiple CT scans of

the same patient, photos of the same object) are split between train and test sets.
Solution: Understand data generation and ensure such groups are entirely in one
split.

Cause 6: Leakage from Data Generation Process: The data collection method
itself introduces a correlation with the label (e.g., different CT machines used for
sick vs. healthy patients). Solution: Understand the data pipeline, normalize data
from different sources, and involve subject matter experts.

Page 24

Main Idea: Detecting Data Leakage

Monitor for leakage throughout the entire ML project lifecycle.

[Link] 9/12
21/11/2025, 17:05 Feature Engineering Techniques and Best Practices - DeepSeek

Detection Methods:

1. Measure Feature-Label Correlation: Investigate features with unusually high

predictive power.

2. Ablation Studies: Remove a feature and see if performance drops significantly.

Investigate why critical features are so important.

3. Monitor New Features: A new feature that causes a massive performance

boost might be leaking label information.

4. Be Careful with the Test Set: Only use the test set for final evaluation. Using it
for feature idea generation or hyperparameter tuning causes leakage.

Page 25

Main Idea: Engineering Good Features - The Trade-offs

While more features often help, there are downsides:

Increased risk of data leakage and overfitting.

Higher memory and computational cost for serving the model.

Increased inference latency.

Technical debt from maintaining useless features.

It's good practice to remove features that are no longer useful.

Page 26

Main Idea: Feature Importance

Feature Importance measures how much a feature contributes to a model's

predictions.

Methods:

Model-specific: Use built-in functions (e.g., from XGBoost).

Model-agnostic: Use methods like SHAP (SHapley Additive exPlanations)

which can measure overall feature importance and contribution to individual
predictions.

[Link] 10/12
21/11/2025, 17:05 Feature Engineering Techniques and Best Practices - DeepSeek

Page 27

(Continuation of Feature Importance with SHAP diagrams)

Page 28

Main Idea: Feature Generalization

Features must work well on unseen data.

Generalization is less scientific to measure than importance and requires intuition

and domain knowledge.

Two key aspects are Coverage and Value Distribution.

Page 29

Main Idea: Coverage and Value Distribution

Coverage: The percentage of samples that have a value for a feature.

Low-coverage features are often less generalizable, unless the "missingness"

itself is informative (MNAR).

Drastic coverage differences between train and test splits signal a data
distribution problem.

Value Distribution: The set of values a feature takes.

If the values in the training set have no overlap with the values in the test set,
the feature will not generalize and may harm performance (e.g., training on
weekdays, testing on Sunday).

Trade-off: There's a trade-off between generalization (e.g., IS_RUSH_HOUR ) and

specificity (e.g., HOUR_OF_THE_DAY ).

Page 30

Main Idea: Summary and Best Practices

Feature engineering is crucial and requires investment.

[Link] 11/12
21/11/2025, 17:05 Feature Engineering Techniques and Best Practices - DeepSeek

Learning comes from experience and studying expert techniques (e.g., Kaggle
winners).

Summary of Best Practices:

Split data by time.

Oversample after splitting.

Scale after splitting, using only training data statistics.

Understand your data lineage and involve domain experts.

Understand feature importance and generalization.

Remove useless features.

Page 31

Main Idea: The Iterative Nature of Feature Engineering

Data and feature engineering are never done. It's a continuous process as long as

[Link] 12/12