Scribd
Upload
4 views15 pages

Uber Fare Prediction Analysis

The document outlines a project to predict Uber ride prices using a dataset, detailing steps such as data pre-processing, outlier detection, correlation analysis, and the implementation of linear and random forest regression models. The random forest model outperformed the linear regression model, achieving an R-squared value of approximately 0.701 and an RMSE of about 5.575. Additionally, it mentions a separate task for email spam classification using K-Nearest Neighbors and Support Vector Machine.

Uploaded by

banrushi7
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
4 views15 pages

Uber Fare Prediction Analysis

The document outlines a project to predict Uber ride prices using a dataset, detailing steps such as data pre-processing, outlier detection, correlation analysis, and the implementation of linear and random forest regression models. The random forest model outperformed the linear regression model, achieving an R-squared value of approximately 0.701 and an RMSE of about 5.575. Additionally, it mentions a separate task for email spam classification using K-Nearest Neighbors and Support Vector Machine.

Uploaded by

banrushi7
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

# Predict the price of the Uber ride from a given pickup point to the agreed drop-off location.

# Perform following tasks:


# 1. Pre-process the dataset.
# 2. Identify outliers.
# 3. Check the correlation.
# 4. Implement linear regression and random forest regression models.
# 5. Evaluate the models and compare their respective scores like R2, RMSE, etc.
# Dataset link: [Link]

# Import necessary libraries


import pandas as pd
import numpy as np
import [Link] as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from [Link] import RandomForestRegressor
from [Link] import r2_score, mean_squared_error

# Load the dataset


data = pd.read_csv("[Link]")

data

Unnamed:
key fare_amount pickup_datetime pickup_longitude pickup_latitude dropoff_longitude dropoff_latitud
0

2015-05-07 2015-05-07
0 24238194 7.5 -73.999817 40.738354 -73.999512 40.72321
19:52:06.0000003 19:52:06 UTC

2009-07-17 2009-07-17
1 27835199 7.7 -73.994355 40.728225 -73.994710 40.75032
20:04:56.0000002 20:04:56 UTC

2009-08-24 2009-08-24
2 44984355 12.9 -74.005043 40.740770 -73.962565 40.77264
21:45:00.00000061 21:45:00 UTC

2009-06-26 2009-06-26
3 25894730 5.3 -73.976124 40.790844 -73.965316 40.80334
08:22:21.0000001 08:22:21 UTC

2014-08-28 2014-08-28
4 17610152 16.0 -73.925023 40.744085 -73.973082 40.76124
17:47:00.000000188 17:47:00 UTC

... ... ... ... ... ... ... ...

2012-10-28 2012-10-28
199995 42598914 3.0 -73.987042 40.739367 -73.986525 40.74029
10:49:00.00000053 10:49:00 UTC

2014-03-14 2014-03-14
199996 16382965 7.5 -73.984722 40.736837 -74.006672 40.73962
01:09:00.0000008 01:09:00 UTC

2009-06-29 2009-06-29
199997 27804658 30.9 -73.986017 40.756487 -73.858957 40.69258
00:42:00.00000078 00:42:00 UTC

2015-05-20 2015-05-20
199998 20259894 14.5 -73.997124 40.725452 -73.983215 40.69541
14:56:25.0000004 14:56:25 UTC

2010-05-15 2010-05-15
199999 11951496 14.1 -73.984395 40.720077 -73.985508 40.76879
04:08:00.00000076 04:08:00 UTC

200000 rows × 9 columns

# 1. Pre-process the dataset

# Remove unnecessary column


data["pickup_datetime"] = pd.to_datetime(data["pickup_datetime"])

missing_values = [Link]().sum()
print("Missing values in the dataset:")
print(missing_values)

# Handle missing values


# We can choose to drop rows with missing values or fill them with appropriate values.

[Link](inplace=True)

# To fill missing values with the mean value of the column:


# [Link]([Link](), inplace=True)

# Ensure there are no more missing values


missing_values = [Link]().sum()
print("Missing values after handling:")
print(missing_values)

# 2. Identify outliers
# visualization to detect outliers.
[Link](x=data["fare_amount"])
[Link]()

Missing values in the dataset:


key 0
fare_amount 0
pickup_datetime 0
pickup_longitude 0
pickup_latitude 0
dropoff_longitude 1
dropoff_latitude 1
passenger_count 0
dtype: int64
Missing values after handling:
key 0
fare_amount 0
pickup_datetime 0
pickup_longitude 0
pickup_latitude 0
dropoff_longitude 0
dropoff_latitude 0
passenger_count 0
dtype: int64

# Calculate the IQR for the 'fare_amount' column


Q1 = data["fare_amount"].quantile(0.25)
Q3 = data["fare_amount"].quantile(0.75)
IQR = Q3 - Q1

# Define a threshold (e.g., 1.5 times the IQR) to identify outliers


threshold = 1.5
lower_bound = Q1 - threshold * IQR
upper_bound = Q3 + threshold * IQR

# Remove outliers
data_no_outliers = data[(data["fare_amount"] >= lower_bound) & (data["fare_amount"] <= upper_bound)]

# Visualize the 'fare_amount' distribution without outliers


[Link](x=data_no_outliers["fare_amount"])
[Link]()
[Link](kind="box",subplots=True, layout=(7, 2), figsize=(15, 20))

fare_amount AxesSubplot(0.125,0.786098;0.352273x0.0939024)
pickup_longitude AxesSubplot(0.547727,0.786098;0.352273x0.0939024)
pickup_latitude AxesSubplot(0.125,0.673415;0.352273x0.0939024)
dropoff_longitude AxesSubplot(0.547727,0.673415;0.352273x0.0939024)
dropoff_latitude AxesSubplot(0.125,0.560732;0.352273x0.0939024)
passenger_count AxesSubplot(0.547727,0.560732;0.352273x0.0939024)
dtype: object

# 3. Check the correlation


# Determine the correlation between features and the target variable (fare_amount).
correlation_matrix = [Link]()
[Link](correlation_matrix, annot=True)
[Link]()
# 4. Implement linear regression and random forest regression models
# Split the data into features and target variable
X = data[['pickup_longitude', 'pickup_latitude', 'dropoff_longitude', 'dropoff_latitude', 'passenger_count']]
y = data['fare_amount'] #Target

0 7.5
1 7.7
2 12.9
3 5.3
4 16.0
...
199995 3.0
199996 7.5
199997 30.9
199998 14.5
199999 14.1
Name: fare_amount, Length: 199999, dtype: float64

# Split the data into training and testing sets


X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Create and train the linear regression model


lr_model = LinearRegression()
lr_model.fit(X_train, y_train)

LinearRegression()

# Create and train the random forest regression model


rf_model = RandomForestRegressor(n_estimators=100, random_state=42)
rf_model.fit(X_train, y_train)

RandomForestRegressor(random_state=42)

# 5. Evaluate the models


# Predict the values
y_pred_lr = lr_model.predict(X_test)
y_pred_lr
print("Linear Model:",y_pred_lr)
y_pred_rf = rf_model.predict(X_test)
print("Random Forest Model:", y_pred_rf)

Linear Model: [11.29237916 11.29171388 11.5718662 ... 11.29183291 11.43252639


11.29190248]
Random Forest Model: [ 9.262 5.043 12.547 ... 6.8087 11.279 8.315 ]

# Calculate R-squared (R2) and Root Mean Squared Error (RMSE) for both models
r2_lr = r2_score(y_test, y_pred_lr)
rmse_lr = [Link](mean_squared_error(y_test, y_pred_lr))
# Compare the scores
print("Linear Regression - R2:", r2_lr)
print("Linear Regression - RMSE:", rmse_lr)

Linear Regression - R2: 0.00034152697863043535


Linear Regression - RMSE: 10.197470623964248

r2_rf = r2_score(y_test, y_pred_rf)


rmse_rf = [Link](mean_squared_error(y_test, y_pred_rf))

print("Random Forest Regression R2:", r2_rf)


print("Random Forest Regression RMSE:",rmse_rf)

Random Forest Regression R2: 0.7011790407391916


Random Forest Regression RMSE: 5.575350372469675

# Overall Analysis

# The Random Forest Regression model has significantly improved the predictive performance.
# An R-squared (R2) value of approximately 0.701 and a Root Mean Squared Error (RMSE)
# of approximately 5.575 indicate that the Random Forest model is capturing a substantial portion
# of the variance in the target variable and providing more accurate predictions compared to the linear regression mo

Loading [MathJax]/jax/output/CommonHTML/fonts/TeX/[Link]
# Classify the email using the binary classification method. Email Spam detection has two
# states: a) Normal State – Not Spam, b) Abnormal State – Spam. Use K-Nearest Neighbors and
# Support Vector Machine for classification. Analyze their performance.
# Dataset link: The [Link] dataset on the Kaggle
# [Link]

# Import necessary libraries


import pandas as pd
from sklearn.model_selection import train_test_split
from [Link] import KNeighborsClassifier
from [Link] import SVC
from [Link] import accuracy_score, classification_report

# Load the dataset


data = pd.read_csv("[Link]") # Replace with the actual path to the dataset
data

Email
the to ect and for of a you hou ... connevey jay valued lay infrastructure military allowing ff dry Predic
No.

Email
0 0 0 1 0 0 0 2 0 0 ... 0 0 0 0 0 0 0 0 0
1

Email
1 8 13 24 6 6 2 102 1 27 ... 0 0 0 0 0 0 0 1 0
2

Email
2 0 0 1 0 0 0 8 0 0 ... 0 0 0 0 0 0 0 0 0
3

Email
3 0 5 22 0 5 1 51 2 10 ... 0 0 0 0 0 0 0 0 0
4

Email
4 7 6 17 1 5 2 57 0 9 ... 0 0 0 0 0 0 0 1 0
5

... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...

Email
5167 2 2 2 3 0 0 32 0 0 ... 0 0 0 0 0 0 0 0 0
5168

Email
5168 35 27 11 2 6 5 151 4 3 ... 0 0 0 0 0 0 0 1 0
5169

Email
5169 0 0 1 1 0 0 11 0 0 ... 0 0 0 0 0 0 0 0 0
5170

Email
5170 2 7 1 0 2 1 28 2 0 ... 0 0 0 0 0 0 0 1 0
5171

Email
5171 22 24 5 1 6 5 148 8 2 ... 0 0 0 0 0 0 0 0 0
5172

5172 rows × 3002 columns

# 1. Data Preprocessing - Handle missing values if necessary


[Link](['Email No.'],axis=1, inplace=True)
# 2. Feature Selection/Engineering - Select relevant features

# 3. Split the data into training and testing sets


X = [Link]("Prediction", axis=1) # Features
y = data["Prediction"] # Target variable
print("Features: ",X)
print("Target: ",y)
Features: the to ect and for of a you hou in ... enhancements \
0 0 0 1 0 0 0 2 0 0 0 ... 0
1 8 13 24 6 6 2 102 1 27 18 ... 0
2 0 0 1 0 0 0 8 0 0 4 ... 0
3 0 5 22 0 5 1 51 2 10 1 ... 0
4 7 6 17 1 5 2 57 0 9 3 ... 0
... ... .. ... ... ... .. ... ... ... .. ... ...
5167 2 2 2 3 0 0 32 0 0 5 ... 0
5168 35 27 11 2 6 5 151 4 3 23 ... 0
5169 0 0 1 1 0 0 11 0 0 1 ... 0
5170 2 7 1 0 2 1 28 2 0 8 ... 0
5171 22 24 5 1 6 5 148 8 2 23 ... 0

connevey jay valued lay infrastructure military allowing ff dry


0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 1 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 1 0
... ... ... ... ... ... ... ... .. ...
5167 0 0 0 0 0 0 0 0 0
5168 0 0 0 0 0 0 0 1 0
5169 0 0 0 0 0 0 0 0 0
5170 0 0 0 0 0 0 0 1 0
5171 0 0 0 0 0 0 0 0 0

[5172 rows x 3000 columns]


Target: 0 0
1 0
2 0
3 0
4 0
..
5167 0
5168 0
5169 1
5170 1
5171 0
Name: Prediction, Length: 5172, dtype: int64

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# 4. Model Building
# K-Nearest Neighbors
knn_model = KNeighborsClassifier(n_neighbors=5)
knn_model.fit(X_train, y_train)

# Support Vector Machine


svm_model = SVC()
svm_model.fit(X_train, y_train)

SVC()

# 5. Model Evaluation
# K-Nearest Neighbors
knn_predictions = knn_model.predict(X_test)
knn_accuracy = accuracy_score(y_test, knn_predictions)
knn_report = classification_report(y_test, knn_predictions)

C:\Users\rohit\anaconda3\lib\site-packages\sklearn\neighbors\_classification.py:228: FutureWarning: Unlike other


reduction functions (e.g. `skew`, `kurtosis`), the default behavior of `mode` typically preserves the axis it ac
ts along. In SciPy 1.11.0, this behavior will change: the default value of `keepdims` will become False, the `ax
is` over which the statistic is taken will be eliminated, and the value None will no longer be accepted. Set `ke
epdims` to True or False to avoid this warning.
mode, _ = [Link](_y[neigh_ind, k], axis=1)

print(knn_predictions)

[0 0 1 ... 0 0 0]

# Print or visualize the evaluation results


print("K-Nearest Neighbors Accuracy:")
print(knn_accuracy)
print("K-Nearest Neighbors Classification Report:")
print(knn_report)
K-Nearest Neighbors Accuracy:
0.8608247422680413
K-Nearest Neighbors Classification Report:
precision recall f1-score support

0 0.93 0.87 0.90 1097


1 0.73 0.83 0.78 455

accuracy 0.86 1552


macro avg 0.83 0.85 0.84 1552
weighted avg 0.87 0.86 0.86 1552

# Support Vector Machine


svm_predictions = svm_model.predict(X_test)
svm_accuracy = accuracy_score(y_test, svm_predictions)
svm_report = classification_report(y_test, svm_predictions)

print(svm_predictions)

[0 0 1 ... 0 0 0]

print("Support Vector Machine Accuracy:")


print(svm_accuracy)
print("Support Vector Machine Classification Report:")
print(svm_report)

Support Vector Machine Accuracy:


0.803479381443299
Support Vector Machine Classification Report:
precision recall f1-score support

0 0.79 0.99 0.88 1097


1 0.92 0.36 0.52 455

accuracy 0.80 1552


macro avg 0.85 0.67 0.70 1552
weighted avg 0.83 0.80 0.77 1552

Loading [MathJax]/jax/output/CommonHTML/fonts/TeX/[Link]
# Given a bank customer, build a neural network-based classifier that can determine whether
# they will leave or not in the next 6 months.
# Dataset Description: The case study is from an open-source dataset from Kaggle.
# The dataset contains 10,000 sample points with 14 distinct features such as
# CustomerId, CreditScore, Geography, Gender, Age, Tenure, Balance, etc.
# Link to the Kaggle project:
# [Link]
# Perform following steps:
# 1. Read the dataset.
# 2. Distinguish the feature and target set and divide the data set into training and test sets.
# 3. Normalize the train and test data.
# 4. Initialize and build the model. Identify the points of improvement and implement the same.
# 5. Print the accuracy score and confusion matrix (5 points)

import pandas as pd
from sklearn.model_selection import train_test_split
from [Link] import StandardScaler
from [Link] import accuracy_score, confusion_matrix
import tensorflow as tf
from tensorflow import keras

# 1. Read the dataset


data = pd.read_csv("Churn_Modelling.csv") # Replace with the actual path to the dataset
data

---------------------------------------------------------------------------
ModuleNotFoundError Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_31424\[Link] in <module>
4 from [Link] import accuracy_score, confusion_matrix
5
----> 6 from tensorflow import keras
7
8 # 1. Read the dataset

ModuleNotFoundError: No module named 'tensorflow'

# 2. Distinguish features and target


X = [Link]("Exited", axis=1) # Features
y = data["Exited"] # Target variable

RowNumber CustomerId Surname CreditScore Geography Gender Age Tenure Balance NumOfProducts HasCrCard IsActi

0 1 15634602 Hargrave 619 France Female 42 2 0.00 1 1

1 2 15647311 Hill 608 Spain Female 41 1 83807.86 1 0

2 3 15619304 Onio 502 France Female 42 8 159660.80 3 1

3 4 15701354 Boni 699 France Female 39 1 0.00 2 0

4 5 15737888 Mitchell 850 Spain Female 43 2 125510.82 1 1

... ... ... ... ... ... ... ... ... ... ... ...

9995 9996 15606229 Obijiaku 771 France Male 39 5 0.00 2 1

9996 9997 15569892 Johnstone 516 France Male 35 10 57369.61 1 1

9997 9998 15584532 Liu 709 France Female 36 7 0.00 1 0

9998 9999 15682355 Sabbatini 772 Germany Male 42 3 75075.31 2 1

9999 10000 15628319 Walker 792 France Female 28 4 130142.79 1 1

10000 rows × 13 columns

# 2. Divide the dataset into training and test sets


X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# 3. Normalize the data


# scaler = StandardScaler()
# X_train = scaler.fit_transform(X_train)
# X_test = [Link](X_test)
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_31424\[Link] in <module>
1 # 3. Normalize the data
2 scaler = StandardScaler()
----> 3 X_train = scaler.fit_transform(X_train)
4 X_test = [Link](X_test)

~\anaconda3\lib\site-packages\sklearn\[Link] in fit_transform(self, X, y, **fit_params)


850 if y is None:
851 # fit method of arity 1 (unsupervised transformation)
--> 852 return [Link](X, **fit_params).transform(X)
853 else:
854 # fit method of arity 2 (supervised transformation)

~\anaconda3\lib\site-packages\sklearn\preprocessing\_data.py in fit(self, X, y, sample_weight)


804 # Reset internal state before fitting
805 self._reset()
--> 806 return self.partial_fit(X, y, sample_weight)
807
808 def partial_fit(self, X, y=None, sample_weight=None):

~\anaconda3\lib\site-packages\sklearn\preprocessing\_data.py in partial_fit(self, X, y, sample_weight)


839 """
840 first_call = not hasattr(self, "n_samples_seen_")
--> 841 X = self._validate_data(
842 X,
843 accept_sparse=("csr", "csc"),

~\anaconda3\lib\site-packages\sklearn\[Link] in _validate_data(self, X, y, reset, validate_separately, **check_


params)
564 raise ValueError("Validation should be done on X, y or both.")
565 elif not no_val_X and no_val_y:
--> 566 X = check_array(X, **check_params)
567 out = X
568 elif no_val_X and not no_val_y:

~\anaconda3\lib\site-packages\sklearn\utils\[Link] in check_array(array, accept_sparse, accept_large_spar


se, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimato
r)
744 array = [Link](dtype, casting="unsafe", copy=False)
745 else:
--> 746 array = [Link](array, order=order, dtype=dtype)
747 except ComplexWarning as complex_warning:
748 raise ValueError(

~\anaconda3\lib\site-packages\pandas\core\[Link] in __array__(self, dtype)


2062
2063 def __array__(self, dtype: [Link] | None = None) -> [Link]:
-> 2064 return [Link](self._values, dtype=dtype)
2065
2066 def __array_wrap__(

ValueError: could not convert string to float: "P'an"

Loading [MathJax]/jax/output/CommonHTML/fonts/TeX/[Link]
# Implement K-Nearest Neighbors algorithm on [Link] dataset. Compute confusion
# matrix, accuracy, error rate, precision and recall on the given dataset.
# Dataset link : [Link]

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from [Link] import StandardScaler
from [Link] import KNeighborsClassifier
from [Link] import confusion_matrix, accuracy_score, precision_score, recall_score, f1_score

data=pd.read_csv("[Link]")
data

Pregnancies Glucose BloodPressure SkinThickness Insulin BMI DiabetesPedigreeFunction Age Outcome

0 6 148 72 35 0 33.6 0.627 50 1

1 1 85 66 29 0 26.6 0.351 31 0

2 8 183 64 0 0 23.3 0.672 32 1

3 1 89 66 23 94 28.1 0.167 21 0

4 0 137 40 35 168 43.1 2.288 33 1

... ... ... ... ... ... ... ... ... ...

763 10 101 76 48 180 32.9 0.171 63 0

764 2 122 70 27 0 36.8 0.340 27 0

765 5 121 72 23 112 26.2 0.245 30 0

766 1 126 60 0 0 30.1 0.349 47 1

767 1 93 70 31 0 30.4 0.315 23 0

768 rows × 9 columns

X = [Link]("Outcome", axis=1) # Features


y = data["Outcome"] # Target variable

Pregnancies Glucose BloodPressure SkinThickness Insulin BMI DiabetesPedigreeFunction Age

0 6 148 72 35 0 33.6 0.627 50

1 1 85 66 29 0 26.6 0.351 31

2 8 183 64 0 0 23.3 0.672 32

3 1 89 66 23 94 28.1 0.167 21

4 0 137 40 35 168 43.1 2.288 33

... ... ... ... ... ... ... ... ...

763 10 101 76 48 180 32.9 0.171 63

764 2 122 70 27 0 36.8 0.340 27

765 5 121 72 23 112 26.2 0.245 30

766 1 126 60 0 0 30.1 0.349 47

767 1 93 70 31 0 30.4 0.315 23

768 rows × 8 columns

# 2. Split the dataset into training and test sets


X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# 3. Normalize the data


scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = [Link](X_test)

X_train
array([[-0.52639686, -1.15139792, -3.75268255, ..., -4.13525578,
-0.49073479, -1.03594038],
[ 1.58804586, -0.27664283, 0.68034485, ..., -0.48916881,
2.41502991, 1.48710085],
[-0.82846011, 0.56687102, -1.2658623 , ..., -0.42452187,
0.54916055, -0.94893896],
...,
[ 1.8901091 , -0.62029661, 0.89659009, ..., 1.76054443,
1.981245 , 0.44308379],
[-1.13052335, 0.62935353, -3.75268255, ..., 1.34680407,
-0.78487662, -0.33992901],
[-1.13052335, 0.12949347, 1.43720319, ..., -1.22614383,
-0.61552223, -1.03594038]])

# 4. Implement K-Nearest Neighbors (KNN)


k = 3 # Choose the number of neighbors (k) based on your needs
knn = KNeighborsClassifier(n_neighbors=k)
[Link](X_train, y_train)

KNeighborsClassifier(n_neighbors=3)

# 5. Predict and Evaluate


y_pred = [Link](X_test)
y_pred

C:\Users\rohit\anaconda3\lib\site-packages\sklearn\neighbors\_classification.py:228: FutureWarning: Unlike other


reduction functions (e.g. `skew`, `kurtosis`), the default behavior of `mode` typically preserves the axis it ac
ts along. In SciPy 1.11.0, this behavior will change: the default value of `keepdims` will become False, the `ax
is` over which the statistic is taken will be eliminated, and the value None will no longer be accepted. Set `ke
epdims` to True or False to avoid this warning.
mode, _ = [Link](_y[neigh_ind, k], axis=1)
array([0, 0, 0, 1, 1, 0, 0, 1, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 1, 0, 1, 0,
0, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 1, 0,
0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 1,
0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0,
0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1,
0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1,
0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0],
dtype=int64)

# Compute the confusion matrix


conf_matrix = confusion_matrix(y_test, y_pred)

# Calculate accuracy, error rate, precision, and recall


accuracy = accuracy_score(y_test, y_pred)
error_rate = 1 - accuracy
precision = precision_score(y_test, y_pred)
recall = recall_score(y_test, y_pred)

print("Confusion Matrix:")
print(conf_matrix)
print("Accuracy:", accuracy)
print("Error Rate:", error_rate)
print("Precision:", precision)
print("Recall:", recall)

Confusion Matrix:
[[81 18]
[27 28]]
Accuracy: 0.7077922077922078
Error Rate: 0.29220779220779225
Precision: 0.6086956521739131
Recall: 0.509090909090909

# Accuracy: This measures the overall correctness of the classifier's predictions. In this case, the model is about 7

# Error Rate: The error rate is the complement of accuracy (1 - accuracy), representing the proportion of incorrect p

# Precision: Precision measures the ratio of true positive predictions to the total number of positive predictions (t

# Recall: Recall measures the ratio of true positive predictions to the total number of actual positive instances (tr
Loading [MathJax]/jax/output/CommonHTML/fonts/TeX/[Link]
# Implement K-Means clustering/ hierarchical clustering on sales_data_sample.csv dataset.
# Determine the number of clusters using the elbow method.
# Dataset link : [Link]

import pandas as pd
import [Link] as plt
from [Link] import KMeans
# from [Link] import KElbowVisualizer

data = pd.read_csv('sales_data_sample.csv', sep = ',', encoding = 'Latin-1')

data

ORDERNUMBER QUANTITYORDERED PRICEEACH ORDERLINENUMBER SALES ORDERDATE STATUS QTR_ID MONTH_ID

2/24/2003
0 10107 30 95.70 2 2871.00 Shipped 1 2
0:00

1 10121 34 81.35 5 2765.90 5/7/2003 0:00 Shipped 2 5

2 10134 41 94.74 2 3884.34 7/1/2003 0:00 Shipped 3 7

8/25/2003
3 10145 45 83.26 6 3746.70 Shipped 3 8
0:00

10/10/2003
4 10159 49 100.00 14 5205.27 Shipped 4 10
0:00

... ... ... ... ... ... ... ... ... ...

12/2/2004
2818 10350 20 100.00 15 2244.40 Shipped 4 12
0:00

1/31/2005
2819 10373 29 100.00 1 3978.51 Shipped 1 1
0:00

2820 10386 43 100.00 4 5417.57 3/1/2005 0:00 Resolved 1 3

3/28/2005
2821 10397 34 62.24 1 2116.16 Shipped 1 3
0:00

2822 10414 47 65.52 9 3079.44 5/6/2005 0:00 On Hold 2 5

2823 rows × 25 columns

# Prepare the data as needed (feature selection, preprocessing, etc.)

# Step 2: Select relevant features for clustering (e.g., 'QUANTITYORDERED', 'PRICEEACH')


selected_features = data[['QUANTITYORDERED', 'PRICEEACH']]
selected_features

QUANTITYORDERED PRICEEACH

0 30 95.70

1 34 81.35

2 41 94.74

3 45 83.26

4 49 100.00

... ... ...

2818 20 100.00

2819 29 100.00

2820 43 100.00

2821 34 62.24

2822 47 65.52

2823 rows × 2 columns

# Step 3: Normalize the data (if needed)


from [Link] import StandardScaler
scaler = StandardScaler()
normalized_features = scaler.fit_transform(selected_features)
normalized_features

array([[-0.52289086, 0.5969775 ],
[-0.11220131, -0.11445035],
[ 0.60650538, 0.54938372],
...,
[ 0.81185016, 0.81015797],
[-0.11220131, -1.06186404],
[ 1.2225397 , -0.89925195]])

# Step 4: Determine the optimal number of clusters using the elbow method
wcss = [] # Within-cluster sum of squares
for i in range(1, 11):
kmeans = KMeans(n_clusters=i, init='k-means++', max_iter=300, n_init=10, random_state=0)
[Link](normalized_features)
[Link](kmeans.inertia_)

# Plot the elbow graph


[Link](range(1, 11), wcss)
[Link]('Elbow Method')
[Link]('Number of clusters')
[Link]('WCSS')
[Link]()

# Step 5: Choose the optimal number of clusters (elbow point) and perform K-Means clustering
optimal_clusters = 3 # Adjust based on the elbow point in the graph
kmeans = KMeans(n_clusters=optimal_clusters, init='k-means++', max_iter=300, n_init=10, random_state=0)
cluster_labels = kmeans.fit_predict(normalized_features)

# Step 6: Visualize the clusters (if possible)


[Link](normalized_features[:, 0], normalized_features[:, 1], c=cluster_labels, cmap='viridis')
[Link]('QUANTITYORDERED')
[Link]('PRICEEACH')
[Link]('K-Means Clustering')
[Link]()
Loading [MathJax]/jax/output/CommonHTML/fonts/TeX/[Link]

Common questions

Powered by AI

Using both linear regression and random forest models allows us to compare different approaches to prediction and assess their effectiveness. Linear regression, being a simple model, might provide a baseline understanding of how the features combine to predict the target variable. Random forest regression, on the other hand, is a more complex model capable of capturing non-linear relationships and interactions between features. In this context, the random forest model significantly improves predictive performance with an R-squared value of 0.701 compared to 0.00034 of the linear model, indicating it's able to capture more variance in the fare amounts .

R2 and RMSE provide insights into model performance. The R2 value indicates how much of the variance in the target variable is explained by the model. A higher R2 means a better fit, with the Random Forest achieving 0.701, suggesting it explains over 70% of the variance in fares. RMSE measures how concentrated the data is around the line of best fit; a lower RMSE indicates better predictive accuracy. The Random Forest's RMSE of 5.575 shows it makes smaller prediction errors compared to linear regression's 10.197, underscoring its superior predictive capability .

To ensure missing values do not affect predictions, data preprocessing involves handling missing data appropriately. In the Uber dataset, missing values were addressed by either dropping rows containing them or filling them with the mean value of the column. This ensures that the dataset used for model training is complete and consistent, thereby enhancing the model's ability to make accurate predictions .

The Random Forest model has a higher R-squared and lower RMSE because it is a more complex, ensemble-based model that captures non-linear relationships and feature interactions more effectively than a linear model. Linear regression assumes a straight-line relationship between predictors and the target variable, which might not hold true for the Uber dataset that involves complex geographic and temporal patterns. The Random Forest's ability to average over many decision trees helps in reducing variance and improving robustness to overfitting compared to a single linear model .

Feature selection is critical in email spam classification as it involves identifying and using only those features that contribute most to the prediction accuracy. By selecting relevant features, models like K-Nearest Neighbors and Support Vector Machines can be trained more efficiently, reducing the complexity and potential overfitting while improving prediction accuracy. This process involves evaluating the significance of each feature with respect to the target outcome, thus ensuring that each input contributes constructively to classification performance .

Ensemble models like Random Forest enhance predictive reliability by combining multiple decision trees to correct for overfitting issues present in single models. This method captures diverse data patterns, leading to improved generalization across unseen data, as evidenced by superior performance in Uber fare predictions with higher R-squared values and lower errors. Ensemble methods increase robustness by reducing the likelihood of errors from any single model's variability, thus offering a more stable and accurate prediction across different datasets .

Correlation analysis helps identify the strength and direction of linear relationships between features and the target variable, fare amount. By examining the correlation matrix, we can understand which features might have a stronger impact on pricing. For example, highly correlated features with the target variable are likely to have more predictive power, and understanding these relationships can inform feature selection and engineering processes .

Data normalization ensures that the features of the dataset have a standard scale, typically between 0 and 1, which is crucial for the performance of neural network models. This prevents certain features from having undue influence over the network's training process due to differing scales and ensures the model converges more efficiently. Normalization can help in improving the speed, performance, and convergence rate of the model by stabilizing the learning process .

Precision and recall offer insights beyond what accuracy provides by focusing on the balance between correctly identified positive observations (true positives) and those incorrect predictions. Precision measures the ratio of true positives against all positive predictions, indicating reliability in positive identification, while recall assesses the model's ability to find all relevant instances (sensitivity). These metrics are especially useful in imbalanced datasets, helping to understand trade-offs between catching true positives and avoiding false positives, which accuracy alone may not reflect adequately .

Identifying and removing outliers allows the models to focus on the prevalent trends in the data, improving their predictive accuracy. Outliers can skew the results, particularly for linear models, which can be highly sensitive to them. By calculating the Interquartile Range (IQR) and defining a threshold to remove data points outside this range, the dataset becomes more representative of normal conditions, which aids both in model training and in reducing error from anomalies .

You might also like