Module 3 Feature Engineering
Module 3 Feature Engineering
Example
Raw Data:
Date Salary City
2026-06-19 50000 Chennai
Engineered Features:
• Day = Friday
• Month = June
• Salary scaled
Feature Creation
package module3.pkg01;
import [Link];
import [Link];
public class CSVReaderDemo {
public static void main(String[] args) {
String file =
"[Link]";
1
try {
BufferedReader br = new BufferedReader(new FileReader(file));
String line;
// Skip header row
[Link]();
[Link]("FEATURE ENGINEERING OUTPUT");
[Link]("------------------------------------------");
while((line = [Link]()) != null)
{
String[] data = [Link](",");
// Check column count
if([Link] < 3)
{
continue;
}
// Skip empty Data_value
if(data[2].trim().isEmpty())
{
continue;
}
String period = data[1];
double value = [Link](data[2]);
// Feature Transformation
double valueInThousands = value / 1000;
[Link]("Period: " + period +" | Original: " + value + " | Thousands: " +
valueInThousands);
}
[Link]();
[Link]("\nFeature Engineering Completed Successfully");
}catch(Exception e){
[Link]();
}
}}
Output
2
2. Feature Selection – Useful vs Irrelevant Features
Feature Selection is the process of selecting only the most important features from a dataset while
removing irrelevant or unnecessary features. Useful features help improve prediction accuracy, while
irrelevant features add noise and reduce model performance. Feature Selection helps reduce
overfitting, improves model efficiency, reduces training time, and increases accuracy.
1. Filter Method
The Filter Method uses statistical techniques to select important features before training the model.
Examples:
• Correlation
• Chi-Square Test
• Information Gain
2. Wrapper Method
The Wrapper Method uses Machine Learning algorithms to test different combinations of features
and select the best set of features.
Examples:
• Forward Selection
• Backward Elimination
• Recursive Feature Elimination (RFE)
3. Embedded Method
In the Embedded Method, feature selection is performed automatically during model training.
Examples:
• Decision Tree
• Random Forest
• Lasso Regression
Dataset
Feature Selection
3
Useful Features:
Age , Salary
Irrelevant Features:
• ID, Name
Example
package module3.pkg01;
import [Link];
import [Link];
public class FeatureSelectionDemo {
public static void main(String[] args) {
String file = "[Link]";
try {
BufferedReader br = new BufferedReader(new FileReader(file));
String line;
// Skip Header
[Link]();
[Link]("FEATURE SELECTION OUTPUT");
[Link]("--------------------------------");
while ((line = [Link]()) != null) {
String[] data = [Link](",");
// Check minimum columns
if ([Link] < 3) {
continue;
}
// Skip empty values
if (data[2].trim().isEmpty()) {
continue;
}
// Useful Features
String period = data[1];
String dataValue = data[2];
[Link]("Useful Features");
[Link]("Period : " + period);
[Link]("Data Value : " + dataValue);
// Irrelevant Features
[Link]("Ignored Features");
[Link]("Series Reference : " + data[0]);
[Link]("--------------------------------");
}
[Link]();
[Link]("Feature Selection Completed Successfully");
} catch (Exception e) {
4
[Link]();
}
}
}
Output
3. Importance of Features in ML
Feature importance refers to measuring how much each feature contributes to the prediction made
by a machine learning model. Some features strongly affect output, while others have little or no
impact. Identifying important features helps improve model accuracy and interpretability. It also
helps remove unnecessary features and reduce complexity.
Types of Feature Importance Methods
1. Model-Based Importance
2. Permutation Importance
3. Statistical Importance
Example
Feature Importance
Area High
Location High
5
Age Medium
ID None
[Link]
ID,StudyHours,Attendance,Marks
1,2,60,45
2,4,70,55
3,5,80,65
4,6,85,72
5,8,95,90
6,7,90,85
7,3,65,50
8,9,98,95
Example code
package module3.pkg01;
import [Link];
import [Link];
import [Link];
public class FeatureImportanceDemo {
// Method to calculate Pearson Correlation
public static double correlation(ArrayList<Double> x, ArrayList<Double> y) {
int n = [Link]();
double sumX = 0;
double sumY = 0;
double sumXY = 0;
double sumX2 = 0;
double sumY2 = 0;
for(int i=0;i<n;i++){
sumX += [Link](i);
sumY += [Link](i);
sumXY += [Link](i) * [Link](i);
sumX2 += [Link](i) * [Link](i);
sumY2 += [Link](i) * [Link](i);
}
6
double numerator = (n * sumXY) - (sumX * sumY);
double denominator = [Link](((n * sumX2) - (sumX * sumX)) * ((n * sumY2) - (sumY *
sumY)));
return numerator / denominator;
}
public static void main(String[] args) {
String file = "[Link]";
ArrayList<Double> studyHours = new ArrayList<>();
ArrayList<Double> attendance = new ArrayList<>();
ArrayList<Double> marks = new ArrayList<>();
try{
BufferedReader br = new BufferedReader(new FileReader(file));
String line;
[Link](); // Skip Header
while((line = [Link]()) != null){
String[] data = [Link](",");
[Link]([Link](data[1]));
[Link]([Link](data[2]));
[Link]([Link](data[3]));
}
[Link]();
double studyImportance = [Link](correlation(studyHours, marks));
double attendanceImportance = [Link](correlation(attendance, marks));
[Link]("===== FEATURE IMPORTANCE =====");
[Link]();
[Link]("Study Hours Importance : %.3f\n", studyImportance);
[Link]("Attendance Importance : %.3f\n", attendanceImportance);
[Link]();
if(studyImportance > attendanceImportance){
[Link]("Most Important Feature : Study Hours");
}
else{
[Link]("Most Important Feature : Attendance");
}
}
catch(Exception e){
[Link](e);
}
}
}
7
Output
FEATURE TRANSFORMATION
1. Feature Transformation
Feature Transformation is the process of modifying, converting, or mathematically transforming
existing features into a new form that helps Machine Learning algorithms understand the data better.
The transformed feature contains the same information as the original feature but in a format that is
easier for the model to process.
1. Skewed Data
Some values occur much more frequently than others.
Example:
Income
20000
25000
30000
35000
500000
The last value is much larger than the others, causing a skewed distribution.
8
2. Large Numerical Value
Large numbers may dominate calculations and make training difficult.
Example:
House Price
1000000
2500000
5000000
3. Presence of Outliers
Outliers are unusually large or small values.
Example:
Age
20
25
30
35
150
4. Non-Normal Distribution
Many Machine Learning algorithms perform better when data follows a normal distribution.
Feature transformation helps make data closer to a normal distribution.
• Reduce skewness
• Reduce variance
• Handle outliers
• Improve model accuracy
• Improve training speed
• Make data more normally distributed
• Improve feature relationship
1. Log Transformation
Log Transformation is a mathematical technique used to reduce the impact of very large values in
a dataset. It converts data into a smaller scale using logarithmic functions, making the distribution
9
more balanced. This transformation is commonly applied to highly skewed data where a few values
are much larger than the rest. It helps machine learning algorithms learn patterns more effectively.
Formula: 𝑿′ = 𝒍𝒐𝒈(𝑿)
Example
Sales
1000
5000
10000
50000
Advantages
• Reduces skewness
• Compresses large values
• Improves model performance
• Handles outliers better
Log Transformation –Example Code
package module3.pkg01;
import [Link];
10
import [Link];
public class LogTransformation {
public static void main(String[] args) {
String file = "salary_data.csv";
try {
BufferedReader br = new BufferedReader(new FileReader(file));
[Link](); // skip header
String line;
[Link]("LOG TRANSFORMATION");
OUTPUT
Square Root Transformation is used to reduce moderate skewness and variance in numerical data. It
transforms each value into its square root, which helps compress larger values while preserving the
overall structure of the data. This transformation is less aggressive than Log Transformation and is
useful when data contains moderate outliers. It can improve the distribution of data and model
performance
Formula: 𝑿′ = √𝑿
11
Example
Original Value Square Root
25 5
64 8
100 10
Advantages
• Reduces variance
• Stabilizes data distribution
• Less aggressive than Log Transformation
Example Code
package module3.pkg01;
import [Link];
import [Link];
public class SqrtTransformation {
public static void main(String[] args) {
String file = "salary_data.csv";
try {
BufferedReader br = new BufferedReader(new FileReader(file));
[Link]();
String line;
[Link]("SQUARE ROOT TRANSFORMATION");
while ((line = [Link]()) != null) {
String[] data = [Link](",");
double salary = [Link](data[1]);
double result = [Link](salary);
[Link]("Salary: " + salary +
" -> Sqrt: " + result);
}
[Link]();
} catch (Exception e) {
[Link]();
}
}
}
Output
12
1. Reciprocal Transformation
Reciprocal Transformation converts each value into its inverse by taking 1 divided by the original
value. It is mainly used when datasets contain extremely large values or strong positive skewness.
This transformation significantly reduces the influence of outliers and helps create a more balanced
data distribution. It is useful in situations where large values dominate the dataset.
𝟏
Formula: 𝑿′ =
𝑿
Example
Advantages
Comparison of Transformations
Transformation Purpose
Log Reduce high skewness
Square Root Reduce moderate skewness
Reciprocal Handle extreme values
Scaling Bring values to same range
Example Code
package module3.pkg01;
import [Link];
import [Link];
Output
2. Feature Scaling
Feature Scaling is the process of adjusting the range of numerical features so that all variables
contribute equally during model training. Different features often have different scales, and larger
values can dominate smaller values in some algorithms. Scaling ensures fair comparison among
features and improves the efficiency of learning algorithms. It is especially important for distance-
based machine learning models.
Age Salary
25 50000
30 60000
35 70000
0 to 100
14
Salary ranges from:
0 to 100000
Some algorithms calculate distance between data points. Larger values can dominate smaller
values and lead to incorrect predictions.
• Faster training
• Better accuracy
• Equal importance to features
• Improved convergence
• Better optimization
Types of Scaling
1. Min-Max Scaling
Feature Scaling is the process of adjusting the range of numerical features so that all variables
contribute equally during model training. Different features often have different scales, and larger
values can dominate smaller values in some algorithms. Scaling ensures fair comparison among
features and improves the efficiency of learning algorithms. It is especially important for distance-
based machine learning models.
𝑿−𝑴𝒊𝒏
Formula: 𝑿′ =
𝑴𝒂𝒙−𝑴𝒊𝒏
Example
Original values:
10, 20, 30
After scaling:
0, 0.5, 1
Advantages
• Easy to understand
• Preserves relationships
• Values remain in fixed range
15
Example Code
package module3.pkg01;
import [Link];
import [Link];
import [Link];
import [Link];
Output
16
2. Standard Scaling
Standard Scaling is a technique that transforms data so that it has a mean of 0 and a standard
deviation of 1. It measures how far each value is from the average value of the dataset. This method
is useful when features have different units or scales. Many machine learning algorithms perform
better when data is standardized using this approach.
𝑿−𝝁
Formula: 𝒁 =
𝝈
Where:
• μ = Mean
• σ = Standard Deviation
Result
After scaling:
Mean = 0
Standard Deviation = 1
Advantages
Example Code
17
package module3.pkg01;
import [Link];
import [Link];
import [Link];
Output
18
1. Label Encoding
Label Encoding is a technique used to convert categorical text values into numerical values. Since
machine learning algorithms work with numbers rather than text, categories must be represented
numerically before training. Each unique category is assigned a unique integer value. It is simple,
memory-efficient, and commonly used for ordinal categorical data.
Example
Before Encoding
Color
Red
Blue
Green
After Encoding
Color Value
Red 0
Blue 1
Green 2
Therefore, text categories must be transformed into numerical form before model training.
Advantages
• Simple implementation
• Low memory usage
• Faster processing
• Useful for ordinal data
Disadvantages
Example:
Education University = 2
School = 0
College = 1
19
This order makes sense.
But for:
Gender
Male = 0
Female =1
There is no actual ranking.
Example Code
package module3.pkg01;
import [Link];
import [Link];
Output
20
One-Hot Encoding
One-Hot Encoding is a method of converting categorical data into multiple binary columns.
Instead of assigning a single number to each category, it creates a separate column for every unique
category. The presence of a category is represented by 1, while its absence is represented by 0.
This approach avoids creating unwanted relationships between categories and is widely used in
machine learning.
Gender
Male
Female
Male
After Encoding
Male Female
1 0
0 1
1 0
Advantages
• No false ordering
• Better for nominal categories
• Widely used in Machine Learning
Example Code
package module3.pkg01;
import [Link];
import [Link];
21
public class OneHotEncoding {
public static void main(String[] args) {
String file = "salary_data.csv";
try {
BufferedReader br = new BufferedReader(new FileReader(file));
[Link]();
String line;
[Link]("ONE HOT ENCODING (Level)");
while ((line = [Link]()) != null) {
String[] data = [Link](",");
double exp = [Link](data[0]);
String level = exp < 2 ? "Junior" : exp < 3 ? "Mid" : "Senior";
int j = [Link]("Junior") ? 1 : 0;
int m = [Link]("Mid") ? 1 : 0;
int s = [Link]("Senior") ? 1 : 0;
[Link](level + " -> " + j + " " + m + " " + s);
}
[Link]();
} catch (Exception e) {
[Link]();
}
}
}
22
Output
23
Feature Engineering
1. Feature Normalization
Definition
Feature Normalization is the process of scaling data values into a fixed range (usually 0 to 1) so
that all features contribute equally to the model.
Why Normalization is Needed?
In datasets, features may have different ranges:
• Salary → 50,000
• Age → 20
• Experience → 5
Large values (Salary) dominate small values (Age) Normalization fixes this problem.
𝑋−𝑋𝑚𝑖𝑛
Formula (Min-Max Scaling): 𝑋 ′ = 𝑋
𝑚𝑎𝑥 −𝑋𝑚𝑖𝑛
Example
Value Normalized
10 0.0
20 0.5
30 1.0
Advantages
• Brings all features to same scale
• Improves model performance
• Useful for KNN, Neural Networks
Disadvantages
• Sensitive to outliers
• Changes original data distribution
Example code:
package module3.pkg01;
import [Link].*;
Output
25
What it does:
• Converts all values into range 0 to 1
• Example: Salary, Age, Experience scaled equally
2. Standardization
Standardization is a data preprocessing technique used in Machine Learning to transform
numerical values so that the dataset has a mean of 0 and a standard deviation of 1. It converts
each value into a Z-score, which indicates how far the value is from the average of the
dataset. Standardization helps features with different scales contribute equally to the model
and improves the performance of algorithms such as Linear Regression, Logistic Regression,
SVM, and Neural Networks.
Mean = 0
Standard Deviation = 1
𝒙−𝝁
Formula (Z-Score): 𝒛 = 𝝈
Explanation
Each value shows how far it is from the mean.
• Positive value → above mean
• Negative value → below mean
Example
If:
• Mean = 50
• Standard Deviation = 10
Then:
• 60 → (60-50)/10 = 1
• 40 → (40-50)/10 = -1
Advantages
• Works well with Gaussian data
• Less affected by outliers than normalization
• Preferred in Linear Regression, Logistic Regression, SVM
Disadvantages
• Does not bound values (can be negative or >1)
26
• Harder to interpret
Example Code
package module3.pkg01;
import [Link].*;
import [Link].*;
27
Output
What it does:
Converts data into:
Mean = 0
Standard deviation = 1
Why it is needed?
Real datasets may have:
• 100 features
• 1000 features
• Even more
Problems:
• Slow training
• Overfitting
• High complexity
28
Example
Before:
• Age
• Salary
• Experience
• Education level
• City
• Gender
After reduction:
• 2 or 3 important combined features
Advantages
• Faster training
• Less memory usage
• Reduces overfitting
• Removes noise
Disadvantages
• Some information is lost
• Hard to interpret new features
Example code
package module3.pkg01;
import [Link];
import [Link];
29
[Link]("Original Features:");
[Link]("SepalLength, SepalWidth, PetalLength, PetalWidth");
[Link]();
[Link]("Reduced Features:");
[Link]("SepalLength, PetalLength");
[Link]("----------------------------------");
What it does:
• Reduces number of columns
• Keeps only important features
Advantages of PCA
• Reduces high-dimensional data
• Removes multicollinearity
• Improves model speed
• Useful for visualization (2D/3D)
Disadvantages
• Hard to interpret new features
• Sensitive to scaling
• Information loss possible
Example Code
package module3.pkg01;
import [Link];
import [Link];
import [Link];
public class PCAOverview {
public static void main(String[] args) {
String file = "[Link]";
ArrayList<Double> pc1List = new ArrayList<>();
31
ArrayList<Double> pc2List = new ArrayList<>();
try {
BufferedReader br = new BufferedReader(new FileReader(file));
String line;
[Link](); // Skip Header
while ((line = [Link]()) != null) {
String[] data = [Link](",");
double sepalLength = [Link](data[0]);
double sepalWidth = [Link](data[1]);
double petalLength = [Link](data[2]);
double petalWidth = [Link](data[3]);
// Simulated Principal Components
double pc1 = (sepalLength + petalLength) / 2.0;
double pc2 = (sepalWidth + petalWidth) / 2.0;
[Link](pc1);
[Link](pc2);
}
[Link]();
[Link]("PCA OVERVIEW");
[Link]("PC1\tPC2");
for (int i = 0; i < [Link](); i++) {
[Link]( [Link](i) + "\t" + [Link](i));
}
} catch (Exception e) {
[Link]();
}
}
Output
What it does:
• Converts correlated features into: PC1, PC2, PC3...
• Keeps maximum variance
32