Week # 8

“Before building any AI/ML model, you must understand your data and improve it
(feature engineering) so the model can perform better”

Topic: Understand Your Data Better

This part is fundamental in Data Science and Machine Learning. It is asking you to
develop the ability to analyse, interpret, and structure raw data before applying any
model. If you skip this step, even the best algorithm will give poor results.

Data & Datasets

Data refers to raw facts, measurements, or
observations collected from real-world sources. It
can exist in multiple formats such as numeric values,
text, images, or time-based records. On its own, data
has limited meaning; it becomes useful only after
processing and analysis.

A dataset is a structured collection of

data, typically organized in tabular form
with rows and columns. Each row
represents an individual record (instance),
while each column represents a variable
or attribute.
Types of Datasets (Conceptual
Understanding)
Structured Data → organized (tables,
databases)
Unstructured Data → not organized
(images, videos, raw text)
Semi-structured Data → partially organized (JSON, XML)
Datasets are commonly managed using tools like Microsoft Excel or programming
libraries such as Pandas.
Student_ID Name Age Study_Hours Attendance (%) Mark Result
s
101 Ali 20 3 85 78 Pass
102 Sara 21 5 92 88 Pass
103 Ahmed 19 2 70 60 Pass
104 Zain 22 1 60 45 Fail
105 Ayesha 20 4 90 91 Pass

Features are the individual attributes or columns in a dataset that describe the data.
They serve as the input variables used by machine learning models to identify
patterns and make predictions.
In the dataset above, the following are considered features:
Age , Study_Hours , Attendance (%) , Marks
These features provide meaningful information about each student and help in
analyzing performance.

Input Features and Target Variable

In supervised learning, features are divided into:
Input Features (X): Variables used to make predictions
Target Variable (y): The output variable the model aims to predict
From the dataset:
Input Features (X): Age, Study_Hours, Attendance
Target Variable (y): Result (Pass/Fail)
This means the model will use the input features to predict whether a student will
pass or fail.
Types of Data in the Dataset
Numeric Data: Age, Study_Hours, Marks
Categorical Data: Result (Pass/Fail), Name
Percentage Data: Attendance
Understanding data types is critical because each type requires different
pre_processing techniques.

Why Understanding Data is Important

1. Understanding your data ensures that:
2. Irrelevant or redundant features are removed
3. Important patterns are identified early
4. Data quality issues (missing values, errors) are handled
5. The machine learning model performs accurately
6. Poor understanding of data leads to incorrect assumptions and weak model
performance.

Model

A model is a function that learns a relationship between input variables (features)

and an output (target), then uses that relationship to make predictions.
“Model learns from data → then predicts new results”

A machine learning model is used to learn patterns from existing data and then
make predictions on new data. Consider a simple real-world scenario where we
want to predict whether a student will pass or fail. The model studies past student
records and identifies relationships between study behaviour and results. Instead of
memorizing data, the model learns general patterns such as students who study
more hours, have better attendance, and higher previous marks are more likely to
pass. This learned pattern is then applied to new students to predict their outcomes.

K-Means — Step-by-Step Distance Calculation (Euclidean)

Dataset (2 Features)

We reduce to 2 numeric features for clarity:

Student Study_Hours Marks

A 1 40
B 2 45
C 3 60
D 4 75
E 5 90

Choose Centroids (K = 2)
We randomly select:

 Centroid 1 (C1) = (1, 40)

 Centroid 2 (C2) = (5, 90)

Student Cluster
A C1
B C1
C C2
D C2
E C2

K-Means assigns data points to clusters by calculating the Euclidean distance

between each point and cluster centroids, then grouping each point with the nearest
centroid.
Feature Engineering

Feature Engineering is the process of selecting, creating or modifying features

like input variables or data to help machine learning models learn patterns more
effectively. It involves transforming raw data into meaningful inputs that improve
model accuracy and performance.

This step may include handling missing values, encoding categories, scaling
numbers, creating new features or combining existing ones. It helps turn messy
real-world data into a form that models can understand and use for better
predictions.

Importance of Feature Engineering

Feature engineering can significantly influence model performance. By refining

features, we can:
 Improve accuracy: Choosing the right features helps the model learn better,
leading to more accurate predictions.
 Reduce overfitting: Using fewer, more important features helps the model
avoid memorizing the data and perform better on new data.
 Boost interpretability: Well-chosen features make it easier to understand how
the model makes its predictions.
 Enhance efficiency: Focusing on key features speeds up the model’s training
and prediction process, saving time and resources.
Processes Involved in Feature Engineering
Lets see various features involved in feature engineering:

1. Feature Creation:
Feature creation is the process of generating new features (variables) from existing
data to improve the performance of a machine learning model. It uses domain
knowledge, data patterns, or feature combinations to create meaningful inputs.
“Create new useful columns from existing data.”

1. Domain-Specific Feature Creation

These features are created using industry knowledge, expert rules, or business
logic. They are not derived from data patterns but from understanding of the
problem domain.

Example
Incom Loan
Banking Dataset: e
50000 20000
New Feature:
Loan_to_Income = Loan / Income = 0.4

Banks use this ratio to measure risk level of a customer. This rule comes from
financial knowledge, not from data patterns.
2. Data-Driven Feature Creation

These features are created by analysing patterns, trends, or distributions in the

dataset. No external rule is used — data itself guides the feature creation.

Example

Student Dataset:

Study Hours Result

1 Fail
2 Fail
4 Pass
5 Pass

Observation:
Students studying 3+ hours usually pass

New Feature:
Study_Level = Low / High

Study Hours Study_Level

1 Low
4 High

This feature helps the model easily separate passing and failing students.

Another Example:

Date
2026-04-23

New features:

 Day = 23
 Month = 4
 Weekend = Yes/No
The model can understand that sales are higher on weekends. Since this feature is
created by observing patterns in the data, it is called data-driven.

3. Synthetic Feature Creation

These features are created by combining two or more existing features

mathematically.

Example

Driver Dataset:

Distanc Time
e
100 km 2 hr

New Feature:
Speed = Distance / Time = 50 km/h

Speed gives more meaningful information than distance or time alone.

[Link] Transformation
Feature transformation is the process of modifying existing features into a better
format so that a machine learning model can learn patterns more effectively.
“Change data into a form that the model can understand easily.”

Types of Feature Transformation

1. Normalization & Scaling

Different features often have different ranges (for example, age vs salary).
Scaling adjusts all features to a similar range, so no feature dominates others.

Example

Original Data:

Age Salary
20 20000
40 80000
Salary values are much larger than age → model gives more importance to salary

After Scaling (0 to 1 range):

Age Salary  Balances all features

0.2 0.25  Improves model accuracy
0.4 1.0  Helps algorithms like distance-based models

2. Encoding (Categorical → Numerical)

Machine learning models cannot understand text data, so categorical values must
be converted into numbers.

Example

Original Data:

City
Lahore
Karachi

Lahore Karachi One-Hot Encoding:

1 0
 Makes data understandable for models
0 1
 Prevents confusion between categories

3. Mathematical Transformations

Sometimes data is skewed (uneven distribution), meaning some values are

extremely large. Mathematical transformations (like logarithms) help to reduce this
imbalance.

Example

Original Data (Skewed):

Income
1000
2000
100000
Very large value disturbs the model

After Log Transformation:

Income  Reduces effect of extreme values

3.0  Makes data more balanced
3.3  Improves learning performance
5.0

3. Feature Extraction
Feature extraction is the process of deriving meaningful features from raw data
while often reducing the number of features (dimensionality). It helps improve
model performance by keeping only the most important information.
“Reduce data size but keep important information.”

Types of Feature Extraction

1. Dimensionality Reduction

Sometimes datasets have too many features (columns), which can:

 Slow down the model

 Cause overfitting
 Increase complexity

Dimensionality reduction techniques reduce the number of features while

preserving useful information.

Example

Original Data (many features):

Math Physic Chemistry Biology

s
80 75 70 85

Instead of using all subjects separately, we can combine them into fewer features.
Technique:

One common method is Principal Component Analysis (PCA)

PCA converts multiple features into fewer new features (components) that still
capture most of the information.

2. Aggregation & Combination

This method creates new features by combining or summarizing existing features,

such as: Sum, Average, Count

Example

Student Dataset:

Math Physics Chemistry

80 70 90

New Feature:
Average_Marks = (80 + 70 + 90) / 3 = 80

Another Example

Sales Dataset:

Day1 Day2 Day3

100 200 300

New Feature:
Total_Sales = 100 + 200 + 300 = 600

 Simplifies data
 Reduces number of features
 Makes patterns easier to learn

[Link] Selection
Feature selection is the process of choosing the most relevant features (columns)
from a dataset and removing unnecessary or irrelevant ones.
“Keep only important features and remove useless data.”

 Reduces overfitting
 Improves model accuracy
 Decreases training time
 Makes the model simpler and easier to understand

Types of Feature Selection Methods

1. Filter Methods

Filter methods select features based on statistical measures without using any
machine learning model. They check how strongly each feature is related to the
target variable.

Example

Dataset: Age Salar Random_Number Purchased

y
Observation:

 Age and Salary are related to Purchased

 Random_Number has no relation

Remove Random_Number

Technique:

One common method is Correlation

High correlation → keep feature

Low correlation → remove feature

2. Wrapper Methods

Wrapper methods select features by testing different feature combinations using a

machine learning model. The model is trained multiple times to find the best
feature subset.

Example
Features: [Age, Salary, Experience]

Try different combinations:

 [Age, Salary]
 [Salary, Experience]
 [Age, Experience]

Choose the combination that gives highest accuracy

3. Embedded Methods

Embedded methods perform feature selection during model training. The model
automatically decides which features are important.

Example

Models like decision trees assign importance scores to features:

Feature Importance
Salary High
Age Medium
Random_Num Low

Remove features with low importance

Example Model:

Decision Tree (automatically selects important features)

[Link] Scaling
Feature scaling is the process of adjusting the range of numerical features so that
all features contribute equally to a machine learning model.

“Bring all values to the same scale so no feature dominates.”

Why Feature Scaling is Important

 Prevents one feature from dominating others

  Improves model accuracy
 Speeds up training
 Very important for distance-based models (e.g., KNN, K-means)

Types of Feature Scaling

1. Min-Max Scaling

Min-Max scaling rescales values into a fixed range, usually 0 to 1.

Example

Original Data:

Marks
50
80
100

Here:
Min = 50, Max = 100

After Min-Max Scaling:

Marks
0.0
0.6
1.0

Why useful?

 Keeps values within a fixed range

 Easy to understand
 Works well when data has no extreme outliers

2. Standard Scaling (Z-score Normalization)

Standard scaling transforms data so that it has:

 Mean = 0
 Variance = 1

Example

Original Data:

Values
10
20
30

Mean = 20

After Standard Scaling:

Values
-1
0
1

Why useful?

 Centers data around zero

 Handles wide ranges better
 Commonly used in many ML algorithms

Topic: Feature Engineering on Numeric Data

Feature engineering on numeric data involves modifying and improving numerical
features (numbers) so that machine learning models can learn patterns more
effectively.
“Improve and prepare number-based data for better predictions.”

Numeric data consists of numbers that represent measurable quantities.

Examples:

 Age = 25
 Salary = 50000
 Marks = 85

Common Techniques Used on Numeric Data

1. Scaling (Normalization / Standardization)

Numeric features often have different ranges (e.g., age vs salary).

Scaling ensures all values are on a similar scale.

2. Handling Outliers

Outliers are extremely large or small values that can disturb the model.

Example

Salary
20000
25000
1000000 ❌

1000000 is an outlier

3. Feature Creation (Numeric)

Create new numeric features from existing ones.

4. Binning (Discretization)

Convert continuous numeric data into categories (groups).

Example

Age
10
25
 Age Group
Child
Adult
60 Senior

Convert into:

Feature Engineering on Categorical Data

Feature engineering on categorical data is the process of converting non-numeric
labels (categories) into numerical form so that machine learning models can
understand and process them.

Categorical Data

Categorical data represents labels or groups that do not have a numerical meaning.

Examples:

 Gender → Male, Female

 City → Lahore, Karachi
 Product Type → Electronics, Clothing

Encoding

Encoding is the process of transforming categorical data into numbers so that

machine learning models can use it.

One-Hot Encoding

One-hot encoding is a method where each category is converted into a separate

column, and values are represented using 0 and 1.

Example:

Original Data:

City
Lahore
Karachi

After One-Hot Encoding:

Lahore Karachi
 1 0
0 1

Feature Engineering on Text Data

Feature engineering on text data is the process of extracting useful information
from raw text so that machine learning models can understand patterns in
language.
“Convert text into meaningful features that represent its information.”

Text Data

Text data includes unstructured language information such as:

 Sentences
 Reviews
 Emails
 Comments

Example:

 “The product is very useful”

 “Service was not good”

Feature Engineering Techniques in Text Data

1. Word-based Features

Text is broken into meaningful parts like words to understand structure.

Example:

 Sentence length (number of words)

 Word count in a review
2. Sentiment-based Features

Text is analyzed to identify emotional meaning.

Example:

 Positive sentiment → “good”, “excellent”

 Negative sentiment → “bad”, “poor”

Feature created:

 Sentiment Score (Positive / Negative / Neutral)

3. Structural Features

These features are based on the format and structure of text.

Example:

 Number of sentences in a review

 Presence of punctuation (!)
 Capital letters usage

4. Keyword-based Features

Important words are identified to represent meaning.

Example:

 “cheap”, “discount”, “free” → indicate promotional intent

Feature Engineering on Temporal Data

Feature engineering on temporal data is the process of extracting meaningful
information from time-based data (date and time) to help machine learning models
understand patterns over time.
“Convert date/time into useful features for better prediction.”

Temporal Data

Temporal data is data that is related to time or sequence of events.

Example:

 2026-04-26 10:30 AM
 15 Jan 2025
 Order timestamps

Feature Engineering Techniques in Temporal Data

1. Date-Based Features

Extract useful parts from a date.

Example:

 Day = 26
 Month = April (4)
 Year = 2026

2. Time-Based Features

Extract information from time values.

Example:

 Hour = 10 AM
 Morning / Evening / Night

3. Week Structure Features

Identify patterns in days of the week.

Example:

 Monday to Friday → Working days

 Saturday/Sunday → Weekend
Sales may increase on weekends.

4. Seasonal Features

Identify long-term time patterns.

Example:

 Summer → High ice cream sales

 Winter → High jacket sales

Feature Engineering on Image Data

Feature engineering on image data is the process of extracting meaningful
information from images so that machine learning models can understand visual
patterns.
“Convert images into useful visual features for prediction.”

Image Data

Image data is visual information made up of pixels.

Example:

 Photos (faces, animals, objects)

 X-ray images
 Satellite images

Feature Engineering Techniques in Image Data

1. Pixel-Based Features

An image is broken into small units called pixels, and each pixel contains numeric
values.

Example:

 Black & white image → intensity values

  RGB image → Red, Green, Blue values

2. Edge Detection Features

Identify boundaries or shapes inside an image.

Example:

 Detecting face outline

 Detecting object edges in a photo

Helps model understand structure of objects

3. Texture Features

Capture the surface patterns in an image.

Example:

 Smooth surface (glass)

 Rough surface (tree bark)

4. Shape Features

Extract geometric structures from images.

Example:

 Circle (ball)
 Rectangle (building)