UNIT IV

Regression Vs Segmentation(Clustering)

Feature Regression Segmentation(Clustering)

Purpose Predict continuous values Group similar data points
Type Supervised learning Unsupervised learning
Input data Labeled data Unlabeled data
Output Continuous numerical value Discrete groups/clusters
Algorithms Linear regression, Polynomial K-Means, Hierarchical, DBSCAN
regression, Decision Trees, etc. etc.
Evaluation MAE, MSE, RMSE, R-squared Silhouette Score, Inertia, DB Index
Metrics
Applications Price prediction, trend analysis Customer segmentation, image
processing

Supervised and Unsupervised Learning

Supervised Learning Unsupervised Learning

Supervised learning algorithms are trained Unsupervised learning algorithms are trained
using labeled data. using unlabeled data.

Supervised learning model takes direct Unsupervised learning model does not take any
feedback to check if it is predicting correct feedback.
output or not.

Supervised learning model predicts the Unsupervised learning model finds the hidden
output. patterns in data.

In supervised learning, input data is In unsupervised learning, only input data is

provided to the model along with the provided to the model.
output.

The goal of supervised learning is to train The goal of unsupervised learning is to find the
the model so that it can predict the output hidden patterns and useful insights from the
when it is given new data. unknown dataset.

Supervised learning needs supervision to Unsupervised learning does not need any
train the model. supervision to train the model.

Supervised learning can be categorized Unsupervised Learning can be classified

in Classification and Regression problems. in Clustering and Associations problems.

Supervised learning can be used for those Unsupervised learning can be used for those cases
cases where we know the input as well as where we have only input data and no
corresponding outputs. corresponding output data.

Supervised learning model produces an Unsupervised learning model may give less
accurate result. accurate result as compared to supervised
learning.

Supervised learning is not close to true Unsupervised learning is more close to the true
Artificial intelligence as in this, we first Artificial Intelligence as it learns similarly as a
train the model for each data, and then child learns daily routine things by his
only it can predict the correct output. experiences.

It includes various algorithms such as

Linear Regression, Logistic Regression,
It includes various algorithms such as Clustering,
Support Vector Machine, Multi-class
KNN, and Apriori algorithm.
Classification, Decision tree, Bayesian
Logic, etc.

Tree Building/Decision Tree

What is a Decision Tree?

A decision tree falls under the category of supervised learning. They can be used to solve both
classification and regression problems. It is a tree that helps in making decisions. It starts with a
question (like "Is it raining?") and leads you through different paths based on your answers (like
"Yes" or "No"). At the end of each path, you reach a decision or an outcome. It contains 3 types
of nodes:
1. Root Node: It is the top most node in the tree. The data which is inside the node is called
attribute.
2. Internal Nodes: It denotes a test on attribute. Nodes which are in between root node and
lead node are called as internal nodes.
3. Leaf Nodes: It represents the final outcomes or predictions.
Advantages of Decision Trees
1. Easy to understand because it can be visualized like a flowchart.
2. It can be used for both classification (categories) and regression (numbers).
3. No need to scale or normalize the data.
4. It can handle non-linear relationships between inputs and outputs.

Disadvantages of Decision Trees

1. Overfitting: The tree might become too detailed and work well on training data but
poorly on new data.
2. Unstable: Small changes in the data can lead to a completely different tree.
3. Bias Toward Features with More Levels: Features with many categories may
dominate the decision-making process.

Applications of Decision Trees

1. Loan Approval in Banking: A bank uses a decision tree to decide whether to approve a
loan.
2. Medical Diagnosis: A hospital uses a decision tree to predict whether a patient has
diabetes.
3. Predicting Exam Results in Education: A school uses a decision tree to predict if a
student will pass or fail.
4. Weather Forecasting: A decision tree predicts whether it will rain tomorrow.

Example:
Consider the following set of training examples:
(a)What is the entropy of this collection of training examples with respect to the target function
classification?

(b)What is the information gain of a1,a2 relative to these training examples?

(c ) Draw decision tree for the given dataset.

Solution:
 Regression,Classification

Regression Algorithm Classification Algorithm

In Regression, the output variable must be of In Classification, the output variable must be a
continuous nature or real value. discrete value.
The task of the regression algorithm is to map The task of the classification algorithm is to
the input value (x) with the continuous output map the input value (x) with the discrete output
variable (y). variable (y).
Regression Algorithms are used with Classification Algorithms are used with
continuous data. discrete data.
In Regression, we try to find the best fit line,
In Classification, we try to find the decision
which can predict the output more accurately. boundary, which can divide the dataset into
different classes.
Regression algorithms can be used to solve the Classification Algorithms can be used to solve
regression problems such as Weather classification problems such as Identification
Prediction, House price prediction, etc. of spam emails, Speech Recognition,
Identification of cancer cells, etc.
The regression Algorithm can be further The Classification algorithms can be divided
divided into Linear and Non-linear Regression. into Binary Classifier and Multi-class
Classifier.
Evaluation metrics used are MAE,MSE etc Evaluation metrics used are
Accuracy,Precision,Recall,F1-Score

Overfitting, Pruning and Complexity

Tree pruning is a technique used in decision tree learning to simplify the tree and prevent
overfitting. Overfitting occurs when a model becomes too complex and starts capturing noise or
anomalies in the training data, which can lead to poor performance on new, unseen data. Tree
pruning helps by removing unnecessary branches that don't contribute much to the accuracy of
the [Link] goal is to make the tree smaller, simpler, and more generalizable to new data.

Two Common Approaches to Tree Pruning

1. Pre-pruning (Stopping Early)

The tree is “pruned” by halting its construction early .Yet some criteria (like statistical
significance, information gain, or Gini index) decides when to stop splitting further.
 Example: Suppose you have a node where splitting the data would result in only
a small improvement in accuracy. Instead of continuing to split, you stop there
and make that node a leaf (a final decision point).
Thus the tree becomes simpler from the start, avoiding unnecessary complexity.
2. Post-pruning (Trimming Later)

First build a full, complex tree and then remove branches that don't help [Link]
building the full tree, evaluate each branch to see if removing it improves the tree's performance
on a validation [Link] the final tree becomes simpler and better at handling new data.

The cost complexity of a tree is defined as a function of two factors:

 Number of leaves: A measure of the tree's complexity.
 Error rate: The percentage of tuples misclassified by the tree.
The pruning process starts from the bottom of the tree .For each node 𝑁, compute:

 The cost complexity of the subtree rooted at 𝑁.

 The cost complexity if the subtree were pruned (replaced by a leaf).
If pruning reduces the cost complexity, prune the subtree. Otherwise, keep it.
Instead of using a separate pruning set, pessimistic pruning uses the training data itself but
adjusts the error rates to account for [Link] adds a penalty to the error rates estimated from
the training data to make them more realistic.
Rather than pruning trees based on error rates,we can prune trees based on the number of
bits required to encode them. The “best” pruned tree is the one that minimizes the number of
encoding bits. This method adopts the Minimum Description Length(MDL) principle. The
basic idea is that the simplest solution is preferred.
Decision trees can become large and complex, even after pruning. The two common
issues that make decision tree difficult to interpret are
1. Repetition, where an attribute is repeatedly tested along a given branch of the tree (e.g.,
age) and
2. Replication, where duplicate subtrees exist within a tree (e.g., the subtree headed by the
node “credit rating?”)
 Multiple Decision Trees/Random Forest

Random Forest is a machine learning algorithm that uses multiple decision trees to make
predictions. It combines their outputs (by voting) to improve accuracy and reduce overfitting.

Random Forest Algorithm:

Step 1: Select random samples from a given data or training set.

Step 2: This algorithm will construct a decision tree for every training data.

Step 3: Voting will take place by averaging the decision tree.

Step 4: Finally, select the most voted prediction result as the final prediction result.

Why Use Random Forest?

1. Reduces Overfitting and training time.
2. Offers high level of accuracy
3. Can handle large datasets
4. Produces more accurate predictions

Advantages
 Works for both classification (e.g., identifying fruits) and regression (e.g., predicting
house prices).
 Can handle large datasets efficiently.
 Reduces overfitting compared to a single decision tree.
  Produces more accurate predictions.

Disadvantages
 Requires more computational power and time than a single decision tree.
 Can be harder to interpret when there are too many trees.

Real-Life Applications
1. Banking:
Predicts whether a customer will repay a loan or commit fraud.
2. Healthcare:
Helps diagnose diseases by analyzing patient data.
3. E-commerce:
Predicts what products a customer is likely to buy based on past behavior.
4. Stock Market:
Identifies trends and predicts stock prices.

Time Series Methods: Arima

ARIMA (AutoRegressive Integrated Moving Average) is a powerful tool for predicting future
values in time series data, like sales figures, stock prices, or weather patterns.

What is Time Series Data?

Time series data is any data collected over time, such as:
 Daily temperatures: 30°C, 32°C, 31°C, etc.
 Monthly sales: 5000,5200, $5300, etc.
 Stock prices: 100,102, $101, etc.
The goal of ARIMA is to analyze past trends and predict future values.

Key Components of ARIMA

ARIMA has three main parts, represented by the parameters p, d, and q:
1. AutoRegression (AR): Looks at how past values influence the current value. For
example, today’s temperature might depend on yesterday’s temperature.
p: Number of past values to consider.
2. Integrated (I): Makes the data "stationary" by removing trends (like upward or
downward movements).
d: Number of times we "difference" the data to make it stationary.
 3. Moving Average (MA): Considers past forecast errors to improve predictions.
q: Number of past errors to include.

Example: Predicting Monthly Sales

Let’s say you run a small business and want to predict next month’s sales based on historical
data.
Step 1: Collect Data
Your monthly sales data looks like this:

Month Sales ($)

January 5000
February 5200
March 5300
April 5400
Step 2: Check if the Data is Stationary
Stationary data has no clear trend or seasonality. If your sales are steadily increasing, the data
isn’t stationary yet.
 Use a test like the Augmented Dickey-Fuller (ADF) test:
o If p-value > 0.05, the data isn’t stationary.
o If p-value ≤ 0.05, the data is stationary.
For our sales data, let’s assume the ADF test shows p-value = 0.10 (not stationary).
Step 3: Make the Data Stationary
To remove the upward trend, we "difference" the data:
 Subtract each value from the previous one:
o February: 5200 - 5000 = 200
o March: 5300 - 5200 = 100
o April: 5400 - 5300 = 100
Now, the differenced data is stationary.
Step 4: Determine ARIMA Parameters (p, d, q)
We use plots like ACF (Autocorrelation Function) and PACF (Partial Autocorrelation
Function) to find the best values for p and q:
 ACF Plot: Helps determine q (number of past errors to use).
 PACF Plot: Helps determine p (number of past values to use).
For our sales data, let’s assume:
  p = 1: Use the last month’s sales.
 d = 1: Differenced the data once.
 q = 1: Use the last forecast error.
So, our ARIMA model is ARIMA(1, 1, 1).

Step 5: Build and Fit the Model

Using Python’s statsmodels library, we fit the ARIMA model:
from [Link] import ARIMA

# Historical sales data

data = [5000, 5200, 5300, 5400]
# Fit ARIMA model
model = ARIMA(data, order=(1, 1, 1)) # p=1, d=1, q=1
model_fit = [Link]()

# Print summary
print(model_fit.summary())

Step 6: Make Predictions

Now, we can predict next month’s sales:
# Forecast next month's sales
forecast = model_fit.forecast(steps=1)
print(f"Next month's predicted sales: ${forecast[0]:.2f}")

If the model predicts 5500, it means next month’s sales will likely be $5500.

Step 7: Evaluate the Model

To check how accurate the model is, compare the predicted values with actual values using
metrics like:
 Mean Absolute Error (MAE): Average difference between predicted and actual values.
 Root Mean Squared Error (RMSE): Penalizes larger errors more heavily.
For example:
 If actual sales were 54500 and the model predicted 5500, the error is small, so the
model is good.

Real-Life Applications
1. Business Planning: Predict holiday sales to stock up on popular items.
2. Weather Forecasting: Predict rainfall for farmers.
3. Finance: Predict stock prices for investors.
 4. Energy: Predict electricity demand during summer peaks.
Measures of Forecast Accuracy

Forecast accuracy measures how close predictions are to actual values. Since perfect forecasts
are nearly impossible, we use different metrics to evaluate how well a forecasting model
performs.

Example:
Imagine you own a store and want to predict weekly sales. You use a simple forecasting method
where you assume next week's sales will be the same as this week's sales (called the naïve
forecasting method).

Week Actual Sales Forecasted Sales Forecast Error

Week 1 20 - -
Week 2 21 20 +1
Week 3 19 21 -2
Week 4 22 19 +3
Here, the forecast error is the difference between actual and predicted sales:
 For Week 2: 21 − 20 = +1
 For Week 3: 19 − 21 = −2
Measures of Forecast Accuracy
To evaluate the model, we use several metrics:
1. Mean Forecast Error (MFE):
 Measures whether the model tends to overestimate or underestimate.
Sum of all forecast errors
 Formula: MFE = Number of forecasts

Example:
+1 + −2 + +3 2
MFE = = ≈ 0.67
3 3
A positive MFE means the model underestimates on average.
2. Mean Absolute Error (MAE):
 Ignores whether the error is positive or negative and focuses on the size of the error.
Sum of absolute forecast errors
 Formula: MAE = Number of forecasts

Example:
 ∣ +1 ∣ +∣ −2 ∣ +∣ +3 ∣ 1 + 2 + 3
MAE = = =2
3 3
3. Mean Squared Error (MSE):
 Penalizes larger errors more heavily because it squares the errors.
Sum of squared forecast errors
 Formula: MSE = Number of forecasts

Example:
12 + −2 2
+ +3 2
1+4+9
MSE = = = 4.67
3 3
4. Root Mean Squared Error (RMSE):
 The square root of MSE, giving error in the same units as the data.
 Formula: RMSE = MSE
Example:

RMSE = 4.67 ≈ 2.16

5. Mean Absolute Percentage Error (MAPE):
 Measures error as a percentage of actual values, making it useful for comparing across
datasets.
Absolute Error
Sum of ×100

Actual Value
Formula: MAPE = Number of forecasts

Example:
∣+1∣
For Week 2: × 100 = 4.76%
21
∣−2∣
For Week 3: × 100 = 10.53%
19
∣+3∣
For Week 4: × 100 = 13.64%
22

4.76 + 10.53 + 13.64

MAPE = ≈ 9.64%
3
Thus, lower values for MAE, MSE, RMSE, and MAPE gives better accuracy.

STL approach

STL (Seasonal and Trend decomposition using Loess) is a technique used to break down time
series data into three main parts: trend, seasonality, and noise. This helps us understand patterns
in the data and make better predictions.

Three Components
1. Trend: The long-term direction of the data (e.g., sales increasing over years).
 2. Seasonality: Repeating patterns over a fixed period (e.g., higher sales during holidays
every year).
3. Noise: Random fluctuations or irregularities that don’t follow a pattern.

How Does STL Work?

STL uses a method called Loess Smoothing to separate these components step by step:
1. Extracting the Trend
 The trend shows the overall movement of the data over time.
 To find it, STL smooths out short-term ups and downs using a technique called Loess
Smoothing.
 Example: If monthly sales data shows an upward trend, the trend component will capture
this steady growth.
2. Extracting Seasonality
 Seasonality captures repeating patterns, like spikes in sales during December every year.
 To isolate seasonality:
1. Subtract the trend from the original data.
2. Group the remaining data into "seasonal subseries" (e.g., all January data points
together).
3. Smooth each subseries to capture the repeating pattern.
 Example: For coffee shop sales, seasonality might show peaks in December due to
holiday demand.
3. Extracting Noise
 Noise is what’s left after removing the trend and seasonality.
 It represents random fluctuations that don’t fit into the other two components.
 Noise is computed by
1. Subtracting both the trend and seasonal components from the original
data.
2. R = Y — T — S: Mathematically, the noise (R) is the difference between
the original time series (Y) and the sum of the trend (T) and seasonal (S)
components
 Example: A sudden drop in sales one month due to bad weather would be part of the
noise.
When to Use STL?
1. Business Planning to understand sales trends and seasonal patterns to plan inventory
and marketing.
2. Anomaly Detection,spot unusual spikes or drops in data after removing normal patterns.
3. Forecasting, improve prediction accuracy by modeling trends and seasonality separately.

Extract features from generated model as Height, Average Energy etc and Analyze for
prediction

To extract features such as Height, Average Energy, or other domain-specific attributes from a
generated model (e.g., ARIMA, Random Forest, STL decomposition, etc.), and use them for
prediction, we need to follow a structured approach.
Step 1: Understand the Dataset and Features
Before extracting features, ensure you understand the dataset and the variables involved. For
instance:
 Height: Could represent the magnitude of a signal, stock price, or sales volume.
 Average Energy: Could represent the average activity level, power consumption, or
volatility in time series data.
Example:
Suppose you are analyzing monthly energy consumption data for a factory:
 Height: Peak energy usage during the month.
 Average Energy: Mean energy consumption over the month.
 Other features might include Trend, Seasonality, and Noise (from STL decomposition).
Step 2: Preprocess the Data
Preprocessing ensures the data is clean and ready for feature extraction.
Example Code:
import pandas as pd
# Load dataset
data = pd.read_csv("energy_consumption.csv", parse_dates=["Date"], index_col="Date")
# Handle missing values
data = [Link](method="ffill") # Forward fill missing values

Step 3: Extract Features

Use domain knowledge and statistical methods to extract meaningful features.
Feature 1: Height
Height can be calculated as the maximum value in the time series for each period.
# Extract Height (Peak Energy Usage)
data["Height"] = data["Energy"].resample("M").max() # Monthly peak

Feature 2: Average Energy

Average energy can be calculated as the mean value in the time series for each period.
# Extract Average Energy (Mean Energy Usage)
data["Average_Energy"] = data["Energy"].resample("M").mean() # Monthly
average

Feature 3: Trend, Seasonality, and Noise

Use STL decomposition to extract these components.
from [Link] import STL
# Perform STL decomposition
stl = STL(data["Energy"], period=12) # Assuming monthly data with yearly
seasonality
result = [Link]()

# Add components to the dataset

data["Trend"] = [Link]
data["Seasonality"] = [Link]
data["Noise"] = [Link]

Feature 4: Variance
Variance can indicate the spread or volatility of energy usage.
# Extract Variance
data["Variance"] = data["Energy"].resample("M").var() # Monthly variance

Step 4: Analyze Features

Analyze the extracted features to identify patterns or relationships.
Example Analysis:
[Link] Analysis: Check how features like Height, Average Energy, and Variance
correlate with the target variable (e.g., future energy consumption).
# Correlation matrix
correlation_matrix = data[["Height", "Average_Energy", "Variance", "Trend"]].corr()
print(correlation_matrix)
[Link]: Plot the features to observe trends and relationships.
import [Link] as plt

# Plot features
fig, axes = [Link](3, 1, figsize=(10, 8), sharex=True)
axes[0].plot([Link], data["Height"], label="Height", color="blue")
axes[1].plot([Link], data["Average_Energy"], label="Average Energy",
color="green")
axes[2].plot([Link], data["Variance"], label="Variance", color="red")
plt.tight_layout()
[Link]()

Step 5: Use Features for Prediction

Once the features are extracted and analyzed, use them as inputs for a predictive model like
Random Forest or ARIMA.
Example with Random Forest:
from [Link] import RandomForestRegressor
from sklearn.model_selection import train_test_split
from [Link] import mean_squared_error

# Prepare features and target

X = data[["Height", "Average_Energy", "Variance", "Trend"]].dropna()
y = data["Energy"].shift(-1).dropna() # Predict next month's energy
consumption

# Train-test split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,
shuffle=False)

# Train Random Forest model

model = RandomForestRegressor(n_estimators=100, random_state=42)
[Link](X_train, y_train)

# Predict and evaluate

y_pred = [Link](X_test)
mse = mean_squared_error(y_test, y_pred)
print(f"Mean Squared Error: {mse}")

Step 6: Interpret Results

Interpret the importance of features to understand their contribution to predictions.
# Feature importance
importances = model.feature_importances_
feature_names = [Link]
for name, importance in zip(feature_names, importances):
print(f"{name}: {importance:.2f}")

Example Application
Suppose the analysis reveals:
 Height and Average Energy have high correlation with future energy consumption.
 Variance indicates periods of instability, which might require additional resources.
One can use this insight to:
1. Optimize resource allocation during peak periods.
2. Plan maintenance schedules during low-energy months.