III Semester Elective

(AI & ML)

Subject: Feature Engineering
Subject Code: 24BCAE1

UNIT I: Introduction to Feature Engineering

UNIT I: Introduction to Data and Features: Importance of Features in Machine
Learning. Data types and features: Numerical, Categorical, Ordinal, Discrete,
Continuous, Interval and Ratio. Basic Feature Pre-processing: Handling Missing
Data, Data Cleaning, Feature Scaling, Normalization, and Transformation.

 INTRODUCTION

What is Artificial Intelligence (AI)?

Artificial Intelligence (AI) is a branch of computer science focused on creating systems or
machines that can perform tasks requiring human intelligence.
 AI tries to simulate human cognitive functions.
 Examples of AI include virtual assistants (Siri, Alexa), autonomous vehicles, etc.

What is Machine Learning (ML)?

Machine Learning (ML) is a subset of AI concerned with building systems that learn from
data and improve their performance automatically without being explicitly
programmed for every scenario.
 Instead of following fixed instructions, ML models learn patterns from data and make
predictions or decisions.
 ML is widely used in speech recognition, image classification, fraud detection, etc.

How does Machine Learning Work?

1. Data Collection: ML starts with collecting relevant data (images, text, numbers).
2. Training: The model learns from this data by finding patterns.
3. Prediction: Once trained, the model can predict or classify new, unseen data.
4. Feedback and Improvement: Models improve over time as more data is provided, similar
to human learning.

Types of Machine Learning

1. Supervised Learning
 Uses labelled data (inputs with correct output).
 The model learns to map inputs to known outputs.
 Example: Predicting if an email is “spam” or “not spam” based on previous emails
labelled accordingly.
2. Unsupervised Learning
 Works with unlabelled data.
 The model finds hidden patterns or groups in data.
 Example: Grouping customers by buying behaviour for targeted marketing without
predefined categories.
3. Reinforcement Learning
 An agent learns by interacting with the environment, receiving rewards or penalties
based on its actions.
 Example: Training a robot to navigate a maze by rewarding successful moves and
penalizing wrong ones.

Suitable Real-World Examples

 AI: Self-driving cars use AI to interpret sensor data and safely navigate roads.
 ML: Netflix recommends movies based on users’ previous watching habits.
 Speech Recognition: Voice assistants like Google Assistant convert spoken words into
text and respond accordingly.
 Healthcare: ML models analyse medical images to detect diseases early.

Data : Data refers to raw facts, figures, or observations collected for analysis. In machine
learning, data is the input that algorithms learn from to make predictions, classifications,
or decisions. This data may come in various forms such as numbers, text, images, or
audio.
Feature : A feature is a measurable property or characteristic of the data relevant to the
task at hand. Features act as the input variables that the model uses to learn and identify
patterns. Each column in a dataset commonly represents a feature.

 Importance of Features in Machine Learning

 Feature Relevance: Relevant features help models capture the essential characteristics
of the problem, leading to accurate and effective learning.
 Performance Impact: The choice and quality of features significantly affect model
accuracy, computational efficiency, and generalization ability on new data.
 Reducing Noise: Irrelevant or noisy features can confuse the model, cause poor
predictions, or lead to overfitting where the model memorizes training data but fails on
unseen data.
 Model Interpretability: Clear and understandable features make models more
interpretable and trustworthy, important for fields like healthcare, credit scoring, and
legal decisions.
 Feature Engineering: Creating new features or transforming existing ones based on
domain knowledge enhances model power.
Detailed Examples
1. House Price Prediction
Features like location, number of rooms, and area size are crucial because they
strongly influence the price, whereas features such as colour of the front door may be
irrelevant. Properly selected features help the model predict prices accurately.
2. Spam Email Detection
Features like presence of certain keywords, sender address, and email
length contribute greatly to classification performance. Extraneous features like email
font size are less significant.
3. Medical Diagnosis
Recording age, blood pressure, and symptoms provide essential features to train
models for disease detection. Features unrelated to health do not help and add noise.
The importance of features in machine learning cannot be overstated. Selecting and
engineering the right features is foundational for building reliable, high-performance
machine learning models. It requires understanding the problem, the data, and the
domain combined with exploratory analysis and transformation techniques.

 Data types and features

1. Numerical Data: Numerical data comprises features measured as numbers and used for
mathematical operations in machine learning models.
 Discrete Data: These are countable values, typically integers. For example: the number
of students in a class, number of cars, or tickets sold.
Example: Number of children in a family (e.g., 2, 3, 4).

 Continuous Data: These can take any value within a range and are often measurements.
For example: height (in cm), temperature (in Celsius), or weight (in kg).
Example: Student exam scores (e.g., 73.5, 89.2).
[Link] Data: Categorical data represent features that describe categories or groups,
identified by labels or names instead of numbers. Arithmetic operations are typically not
meaningful.
 Nominal Data: Categories without intrinsic ordering. For example: gender (male,
female), blood type (A, B, AB, O), colours (red, blue, green).
Example: Types of fruits (apple, banana, orange).

 Ordinal Data: Categorical data with a meaningful order. For example: education level
(High School, Bachelor, Master), customer satisfaction (Poor, Fair, Good, Excellent).
Example: Shirt size (Small, Medium, Large).

3. Interval and Ratio Data: These are advanced numerical categories based on their
mathematical properties.
 Interval Data: Ordered numerical data with equal intervals, but no absolute zero.
Example: Temperature in Celsius or Fahrenheit, dates on a calendar. Zero is arbitrary
and does not imply absence.
Example: Temperature (Celsius, Fahrenheit), IQ score.

 Ratio Data: Like interval data, but with a meaningful absolute zero, allowing all
arithmetic operations. Example: height (zero means absence of height), age, weight in
kg, income. Here, zero means none of the quantity exists.
Example: Distance (meters), salary (zero means no salary earned).
Summary Table

Data Type Description Example

Numerical Numbers, measurable Age, Income, Height

Discrete Countable integers Number of students, cars

Continuous Any value in a range Weight, Score, Temperature

Categorical Labels, groups Gender, Fruit type

Nominal Categories, no order Blood type, Eye colour

 Data Type Description Example

Ordinal Categories, with order T-shirt size, Star ratings (1-

Temperature (C), Calendar

Interval Ordered, no true zero date

Ratio Ordered, true zero present Length, Weight, Age, Income

 Feature engineering : Feature engineering is a machine learning technique that

leverages data to create new variables that aren’t in the training set. It can produce
new features for both supervised and unsupervised learning, with the goal of
simplifying and speeding up data transformations while also enhancing model accuracy.

Feature engineering "improvizes" the features in such a way that they describe the structures
inherent in the provided dataset. Involving better features for the training leads to
following advantages:
▶ More flexibility: Good features provide more flexibility, which helps to analyze datasets
with much rigid features which keep on adding to various constraints to the model,
making it more complex in terms of understanding as well as execution.
▶ Simpler and fast Models: As features are more flexible, less complex models can be
built which are faster to run, easier to understand and easier to maintain. It allivates the
need to chose overfitting models with maximally screened and optimized parameters.
▶ Better results: Models made from better features are well balanced, i.e., neither
underfitted nor overfitted with the training dataset. This lead to better results during
testing.

 Feature engineering pre-processing:

Feature engineering consists of various processes, as listed below:
▶ Feature Creation: Creating features can involve creating new parameters by adding or
removing some features by using relation among them.
▶ Transformations: It is a function that transforms features from one representation to
another. Such transformations are chosen which make the model more flexible and be
able to take variety of data as input. It helps to speed up training and increase the
accuracy of the model.
▶ Feature Extraction: Involves extracting features from the raw data to identify useful
information and generate new variables to be used in the model. It basically compresses
the data and reduces them into manageable quantities for modelling, without distorting
the original dependencies and significant information.
▶ Feature Selection: All above processes help us to either create or identify the appropriate
features useful for building the model. As features are selected, any redundant feature
which may negatively impact the model by reducing the overall performance and
accuracy are removed.

 Steps in feature engineering: Raw data is acquired from various resources and is
prepared into a suitable format in order to be used in the model. It involves various sub-
procedures as listed:
• Cleaning: Removal of incorrect, corrupted, incorrectly formatted, duplicate, or
incomplete data within a dataset.
• Delivery: Process of sending the dataset from the data source to the system in which the
model will be trained.
• Augmentation: Set of techniques for artificially increase the amount of data by
generating data points from existing data.
 • Fusion: Involves integration of the data obtained and delivered from several data
sources for producing consistent, accurate and useful dataset.
• Ingestion/Loading: Final process of importing and loading data for the model training.

 Handling Missing Data:

The term missing data refers to the absence of records or values or observations usually
expected to be present in a dataset. Missing data must be addressed during the data pre-
processing stage prior to feeding it into the ML model otherwise it will render the model
too complex and affect its performance. This may create biased outcomes or predictions.
Therefore using feature engineering methods like mean, mode, k-nearest neighbor
(KNN), logistic regression (LR), SVM, Random Forest (RF), Naïve Bayes (NB) and
Artificial Neural Network (ANN)

Missing data can significantly impact analysis accuracy. Methods to address missing
data include:
 Removing rows/columns with missing values when such gaps are minimal and do not
affect model usefulness.
 Imputing missing values by replacing gaps with statistical estimates such as mean,
median, or mode for numerical features, or the most frequent category for categorical
features.
 Algorithmic imputation using techniques like regression or k-nearest neighbors (KNN)
to predict and fill missing entries.
These approaches keep data analysis reliable and minimize bias introduced by
incomplete datasets.

Data Cleaning: Data cleaning is the process of fixing or removing incorrect, corrupted,
duplicated, or incomplete data within a dataset.
Key techniques include:
 Removing duplicates to avoid repetitive entries that can skew analysis.
 Correcting inconsistencies, such as standardizing formats, converting date/time values,
and fixing spelling or capitalization errors.
 Dealing with outliers by either removing or transforming extreme values that can
mislead statistical results.
 Validating accuracy by checking data against rules (e.g., email or numerical format
validation).
 Standardizing data (e.g., consistent units, date formats) ensures all data is easily
comparable and ready for modeling.
Automation tools like Pandas and Open Refine assist in efficient data cleaning for large
datasets.
Feature Scaling: Feature scaling adjusts the range and distribution of numerical features to
facilitate fair treatment by machine learning algorithms.
Common techniques:
 Min-Max Scaling: Rescales features to a 0-1 range
using (x−min(x))(max(x)−min(x))(max(x)−min(x))(x−min(x)).
 Standardization: Centers features around a mean of 0 and scales to unit variance,
using (x−μ)σσ(x−μ) where μμ is mean and σσ is standard deviation.
Scaling is especially important for algorithms like K-Nearest Neighbors (KNN) and
Support Vector Machines (SVM), as differences in scale can bias their results.

Normalization: Normalization shrinks data into a specified range or applies

transformations for consistent measurement. The primary goal is to bring all features to
similar scales, so that no feature dominates due to its magnitude.
Key methods:
 Min-Max Normalization: Already covered; places values between 0 and 1.
 Unit Vector Transformation: Scales data such that the Euclidean norm (length) is 1.
 Log or Power Transformation: Used for highly skewed data to reduce the effect of
extreme values and normalize distributions.

Feature Transformation: Feature transformation modifies data to improve model

performance or make features more usable:
 Binning: Converts continuous data into discrete bins or intervals (e.g., age groups).
 Feature extraction: Creates new features from raw data, like extracting "weekday" from
a date.
 Dimensionality reduction: Techniques like Principal Component Analysis (PCA)
reduce the number of variables while preserving essential information.
 Encoding categorical variables: Converts categories to numbers (e.g., one-hot encoding)
for algorithms that require numeric input.
Transformations often make raw data compatible with modeling techniques and can
reveal hidden patterns.

These pre-processing steps—addressing missing data, cleaning and transforming features,

scaling, and normalizing—are essential foundations for successful machine learning and
analytics projects in any business or research context.

*********************
 UNIT 02
FEATURE ENGINEERING TECHNIQUES

UNIT II: Feature Engineering Techniques [15 Hours] Techniques for Numerical Data:
Binning and Discretization, Polynomial and Interaction Features. Categorical Data
Techniques: One Hot Encoding, Label Encoding. Feature extraction vs. feature
selection, Steps in feature selection. Feature Selection Methods: Filter, Wrapper, and
Hybrid. Feature Reduction: Introduction and application of Principal Components
Analysis.

 Key techniques in feature engineering:

● Feature extraction – Deriving new features from existing data
● Feature transformation – Applying mathematical functions such as log scaling,
polynomial transformations, or power transformations.
● Feature encoding – Converting categorical variables into numerical representations.
● Feature aggregation – Computing summary statistics over time windows or groups.

 Discretization, Polynomial & Interaction Features in Feature Engineering

1. Discretization :Discretization (or binning) is the process of transforming

continuous variables into discrete categories or bins. This is commonly used to:
 Enable linear models to capture non-linear relationships by providing
stepwise features
 Reduce noise and increase signal clarity
 Make algorithms (like decision trees) more interpretable
Techniques
 Equal-width Binning: Divides the range of a feature into intervals of the
same width.
 Equal-frequency Binning: Splits the feature so that each bin has
approximately the same number of observations.
 K-means Discretization: Uses clustering algorithms to group similar
values into bins.
 Decision Tree Discretization: Leverages the tree’s optimal cut-points to
split continuous values into coherent intervals.
Pros & Cons
Pros Cons
Finds non-linear patterns for Increases feature count (risk of
simple models overfitting)

Improves robustness to outliers/noise May lose detailed information

Not always beneficial for linearly

Enhances interpretability
separable data

2. Polynomial Features: Polynomial features expand the set of features by

creating higher-order (squared, cubed, etc.) versions of the original features. This
allows linear models to learn non-linear relationships.

Mathematical Idea
If you have two features x1x_1x1 and x2x_2x2, polynomial expansion adds:
 x12,x22,x1x2x_1^2, x_2^2, x_1 x_2x12,x22,x1x2, etc.
This enables linear models to approximate quadratic surfaces, curves, and
more complex patterns.
 Set desired degree (commonly 2 or 3).
 Fit and transform your features for model training.

Pros & Cons

Pros Cons
Allow linear models to fit Can create many features (combinatorial growth)
non-linear trends leading to Greater risk of overfitting

Enhance model accuracy Increased computation and interpretation

in complex data complexity

3. Interaction Features: Interaction features capture the combined effect of two

or more variables on the target. These are constructed by multiplying or
combining two (or more) features.
Example:
Suppose x1x_1x1 (age), x2x_2x2 (income). The product x1×x2x_1 , x_2x1×x2 is an
interaction feature, representing how the combined influence of both affects the
outcome.
 Key Considerations
 As with polynomials, large numbers of interactions can cause overfitting
 Best used with domain knowledge (e.g., certain interactions support real-world
mechanisms)

Summary Table

Technique Purpose Typical Impact

Convert continuous to Improves model accuracy

Discretization categorical/bin features; and interpretability; risk:
discover non-linearities feature explosion

Polynomial Capture non-linear patterns Boosts linear model power;

Features with higher-order terms risk: high dimensionality

Reveals joint influences;

Interaction Model combined effects of
risk: overfitting if too many
Features multiple variables
interactions

 Categorical Data Techniques

1. ONE-HOT ENCODING : One-Hot Encoding converts categorical variables into

binary indicators.

▶ Simplest and basic categorical-column encoding method.

▶ Encoding procedure involves the representation of the element of any finite set by
the index of that element in the set, i.e., the element under consideration is assigned
to index "1", where all other elements are assigned value as “0”.
▶ As it assigns a unique binary number of multiple digits for each possible case or
category, making it different from other binary encoding schemes.

Considering we have a feature as color, we need 3 many bits and they are Red,
Blue, and Green. Now to encode them we will follow the below procedure.
For every feature value a fixed position in the array is assigned which is 1 and the rest is
0. It is like a vector where the basis is formed by the feature value and the coefficients
form the encoded combination.

2. ONE COLD ENCODING : In the previous combination if the 0 are replaced with
1 and the 1 is replaced with 0. This type of encoding is known as one-cold encoding.
The encoding is shown in the table below.

Advantages:
▶ Categorical column is mapped into multiple binary columns, which are easy-to-use and
faster to parse through.

▶ Features encoded by one-hot encoding can be easily applied into the models since each
of the new binary column corresponds to a category in the original column.

Disadvantages:
▶ Any weight for each of the category, or any ordinal relation between the categories are
removed. Making them not much useful for models involving Hierarchial classification.

▶ Features with very high cardinality in terms of categories are encoded with very large
amount of dimensions corresponding to each of the binary column.
▶ This leads to various undesired issues like high training variance, decrease in accuracy
and significant consumption of memory and computation.
Used in tree-based models (e.g., Random Forest, XGBoost).

3. Label Encoding: Label encoding assigns integer values to categories.

Example: Color Encoded Value
Red 0
Blue 1
Green 2

 Feature Creation vs Feature Exrtraction vs Feature Selection

Feature Extraction : Feature extraction involves transforming or combining

the original features to create new, more informative features, often in a lower-
dimensional space. This is especially useful for high-dimensional or complex datasets
such as images and text. Common techniques include Principal Component Analysis
(PCA), Linear Discriminant Analysis (LDA), and neural network-based methods
such as autoencoders.
 Alters the feature space, often making features less directly interpretable.
 Captures hidden patterns and relationships.

Feature Creation : Feature creation, sometimes called feature generation, is the

process of inventing new features from raw data or combining existing features to
encode relevant business, domain, or mathematical knowledge. This can include
synthesizing features using arithmetic, binning, domain knowledge, or encoding
categorical variables. Examples include creating an 'age group' from 'age', or
generating a 'total purchase' field by summing multiple purchase records.
 Adds fresh dimensions to your data, not just reduces or transforms them

Feature Selection : Feature selection refers to the process of identifying and

choosing a subset of the most relevant features from your original dataset, while
discarding those that are irrelevant or redundant. This helps reduce overfitting,
improves model accuracy and interpretability, and speeds up training. Common
feature selection methods include filter methods (statistical ranking), wrapper
methods (model-based selection), and embedded methods (feature selection built into
model training).
 Keeps original feature values and meanings intact.
 Useful when you have domain knowledge or want to interpret model results
directly

Key techniques in feature selection:

● Filter Methods – Selecting features based on statistical measures such as
correlation coefficients, mutual information, or variance thresholds.
● Wrapper Methods – Using iterative model training (e.g., Recursive Feature
Elimination, Forward/Backward Selection) to identify the best feature subset.
● Embedded Methods – Selecting features within model training (e.g., Lasso
Regression, Tree-based feature importance).

Key Differences Between FeatureCreation, Feature Exrtraction and

Feature Selection
Feature
Aspect Feature Selection Feature Creation
Extraction

Picks a subset of Transforms and

Generates new
relevant features combines
Approach features from raw or
from existing features into new
existing data
features ones

New features added

New, composite
Output Subset of original using domain
features; altered
Features features, unchanged knowledge, math, or
meaning
encoding

Can increase
Reduces Reduces or
Effect on dimensionality if
dimensionality by transforms
Dimensionality more features are
removing features dimensionality
created

High; original feature Low; transformed Variable; depends on

Interpretability names/meanings features may lack design and domain
retained direct meaning knowledge
 Feature
Aspect Feature Selection Feature Creation
Extraction

Arithmetic,
Typical Filter, Wrapper, PCA, LDA, ICA, aggregation,
Techniques Embedded, Voting Autoencoders time/binning,
encoding, interaction

Overcomplicating
May lose
May drop useful info model if too
Risk interpretability,
if selection is poor many/irrelevant
introduce noise
features added

When reducing When raw features

When handling
overfitting and costs, miss important
When to Use complex/high-
retaining patterns or business
dimensional data
interpretability rules

Feature Reduction : Principal Component Analysis (PCA) in Feature

Engineering

1. Introduction: Principal Component Analysis (PCA) is one of the most widely used
techniques in feature engineering and dimensionality reduction. It transforms high-
dimensional datasets into a smaller set of features called principal components, which
preserve most of the important information. The key motivation for PCA is to simplify
complex data, remove redundancy, and improve the performance and interpretability of
machine learning models.
Developed by Karl Pearson (1901) and later formalized by Harold Hotelling (1930s),
PCA uses linear algebra to uncover hidden patterns in correlated data by finding new
axes (directions) that best explain the data's variance.

2. Why PCA is Used in Feature Engineering

Feature engineering aims to improve model performance by creating efficient and
informative input features. PCA helps in this process by:
1. Reducing dimensionality: By selecting the most informative components, PCA
removes irrelevant or redundant features.
2. Improving computational efficiency: Fewer features mean faster model training and
prediction.
3. Avoiding overfitting: By eliminating noise and redundant information, PCA
improves generalization.
4. Handling multicollinearity: When two or more variables are highly correlated, PCA
replaces them with uncorrelated principal components.
5. Visualizing high-dimensional data: PCA projects data into lower dimensions (2D or
3D) for pattern discovery.

3. Steps in PCA Process

1. Standardize the data: Scale the features to have a mean of 0 and a standard deviation
of 1. This is crucial because PCA is sensitive to the scale of features.
2. Calculate the covariance matrix: Compute the covariance matrix of the
standardized data to understand the relationships between variables.
3. Compute eigenvectors and eigenvalues: Find the eigenvectors and eigenvalues of
the covariance matrix. The eigenvectors represent the directions (principal
components), and the eigenvalues represent the magnitude of variance along those
directions.
4. Sort eigenvectors: Sort the eigenvectors in descending order based on their
corresponding eigenvalues. This ranks the principal components from most to least
important in terms of variance explained.
5. Select principal components: Decide how many components to keep. This is often
based on the cumulative explained variance, keeping enough components to retain a
desired percentage (e.g., 95%) of the total variance.
6. Transform the data: Project the original standardized data onto the new, selected
eigenvectors to create the final set of principal components.

4. Interpreting PCA Components

1. Principal Component 1 (PC₁): Captures the direction of maximum variance.
2. Principal Component 2 (PC₂): Captures the next highest variance orthogonal to PC₁.
3. Subsequent Components: Each additional component is orthogonal to all previous
ones and explains remaining variance.

5. PCA in the Feature Engineering Workflow

PCA acts as a preprocessing step before model training. Typical steps:
1. Data Cleaning: Handle missing values and scale features.
2. Feature Standardization: Make sure each variable contributes equally.
3. Apply PCA: Fit and transform the data to generate principal components.
4. Select Components: Retain components explaining significant variance (e.g., 95%).
5. Model Building: Train models using transformed, reduced data.
6. Evaluation: Check for accuracy and interpretability changes compared to the original
features.
6. Advantages of PCA in Feature Engineering
 Simplifies data by reducing dimensionality.
 Removes multicollinearity between features.
 Speeds up training and inference processes.
 Focuses on signal over noise, improving model generalization.
 Helps visualize patterns in high-dimensional data.

7. Limitations of PCA
 Loss of interpretability: Principal components are combinations of initial features,
making interpretation difficult.
 Assumes linear relationships: PCA only captures linear patterns.
 Scaling sensitivity: Without proper standardization, results can be misleading.
 Not ideal for categorical features without numeric encoding.

******************