Data Analysis

with Python
Importing Datasets
Python Packages for Data Science
Scientific Computing Visualization Algorithmic
Libraries Libraries Libraries

Pandas Matplotlib Statsmodel

Data structures & tools Plots & graphs Statistical models

NumPy Seaborn Scikit-learn

Arrays & matrices Plots: heatmaps, Machine learning
violin plots

SciPy Folium Keras

Integrals, differential Plots: geospatial data, Deep learning
equations, optimization chloropleth maps
Importing Data
 Process of loading or reading data into Python environment from
various sources.
Database Cloud

Folder API
Importing Data
 Two important factors:

Format Location or Path

 .csv  PC
 .json  Web
 .xlsx
 .hdf
Importing CSV into Python

import pandas as pd

url = “[Link]

df = pd.read_csv(url)
Printing Data in Python
 df returns (prints out) the
entire dataset (DataFrame)

 [Link](n) returns
(prints out) the first n rows
of a DataFrame

 [Link](n) returns
(prints out) the last n rows
of a DataFrame
Exporting Data to CSV
 Save modified or processed dataset

 Format and location (path) also matter

path = “C:/Windows/User/Downloads/[Link]”

df.to_csv(path)
Exporting to Different Formats in Python

Data Format Read Save

CSV pd.read_csv() df.to_csv()

JSON pd.read_json() df.to_json()

EXCEL pd.read_excel() df.to_excel()

SQL pd.read_sql() df.to_sql()

Basic Insight of Dataset
 Understand the data before analysis

 Should check:
 Data Types
 Data Distribution
 Locate potential issues with the data
Basic Insight of Dataset – Data Type

Pandas Type Native Python Type Description

object string Text characters

int64 int Numeric characters

float64 float Numeric with decimals

datetime64 datetime module Time data

Basic Insight of Dataset – Data Type
 Why check data types?
 potential info and type mismatch
 compatibility with python Pandas methods
Basic Insight of Dataset – Data Type
 In Pandas, we use [Link] to check data types

[Link]
Basic Insight of Dataset – Data Distribution
 In Pandas, [Link] returns a statistical summary

[Link]()
Basic Insight of Dataset – Data Distribution
[Link](include=“all”)

 [Link](include=“all”)
returns summary statistics
Data Wrangling
Data Preprocessing
 The process of transforming raw data into a format that is suitable
and effective for further analysis (machine learning algorithm,
statistical analysis, visualization)
Data Preprocessing Steps
 Identify and handle missing values

 Data formatting

 Data normalization

 Data binning

 Turning categorical values to numeric values

Missing Values
 Missing values occur when no data is stored for a variable (feature)
in an observation

 Represented as NaN in Pandas

How To Deal with Missing Data
 Check with the data collection source

 Drop missing values

 Drop the variables (columns) with missing values
 Drop the rows with missing values

 Replace missing values

 Replace with an average
 Replace with most frequently occurring value (mode)

 Leave it as missing data

How To Drop Rows with Missing Values
 Use [Link]()

[Link](subset=[“price”], axis=0, inplace=True)

axis=0 – drops the entire row

axis=1 – drops the entire column

df = [Link](subset=[“price”], axis=0)
How To Replace Missing Values
 Use [Link](missing_value, new_value)

mean = df[“age”].mean()

[Link]([Link], mean)
Data Formatting
 Bringing data into a common standard expression

City City
Non-formatted NY New York Formatted
New York New York
N.Y New York
N.Y New York
Data Formatting
 Correct data types using [Link]()

df[“price”] = df[“price”].astype(“int”)
Data Normalization
age income
age income
scale [20,45] [20000,500000]
20 100000
impact small large
30 20000
40 500000
45 150000
 age and income are in different ranges

 income will influence the result more

Data Normalization

age income age income

20 100000 0.2 0.2
30 20000 0.3 0.04
40 500000 0.4 1
45 150000 0.45 0.25
Methods for Data Normalization
𝑥𝑜𝑙𝑑
Simple Feature Scaling 𝑥𝑛𝑒𝑤 =
𝑥𝑚𝑎𝑥

𝑥𝑜𝑙𝑑 − 𝑥𝑚𝑖𝑛
Min-Max 𝑥𝑛𝑒𝑤 =
𝑥𝑚𝑎𝑥 − 𝑥𝑚𝑖𝑛

𝑥𝑜𝑙𝑑 − 𝜇
Z-Score 𝑥𝑛𝑒𝑤 =
𝜎
Simple Feature Scaling

length width 𝑥𝑜𝑙𝑑 length width

𝑥𝑛𝑒𝑤 =
168.8 64.1 𝑥𝑚𝑎𝑥
0.81 64.1
168.8 64.1 0.81 64.1
180.0 65.5 0.87 65.5
… … … …

df[“length”] = df[“length”] / df[“length”].max()

Min-Max

length width 𝑥𝑛𝑒𝑤 =

𝑥𝑜𝑙𝑑 − 𝑥𝑚𝑖𝑛 length width
𝑥𝑚𝑎𝑥 − 𝑥𝑚𝑖𝑛
168.8 64.1 0.41 64.1
168.8 64.1 0.41 64.1
180.0 65.5 0.58 65.5
… … … …

df[“length”] = (df[“length”] - df[“length”].min()) /

(df[“length”].max() – df[“length”].min())
Z-Score

length width 𝑥𝑜𝑙𝑑 − 𝜇 length width

𝑥𝑛𝑒𝑤 =
168.8 64.1 𝜎
-0.034 64.1
168.8 64.1 -0.034 64.1
180.0 65.5 0.039 65.5
… … … …

df[“length”] = (df[“length”]–df[“length”].mean())/df[“length”].std()
Binning
 Grouping of values into “bins”

 Convert numeric values into categorical

price 5000, 10000, 12000 30000, 31000 40000, 44000, 44500

bins Low Mid High

Binning in Python
price price price-binned
13495 13495 Low
16500 16500 Low
18920 18920 Medium
41315 41315 High

bins = [Link](df[“price”].min(), df[“price”].max(), 4)

group_names = [“Low”, “Medium”, “High”]
df[“price-binned”] = [Link](df[“price”], bins,
labels=group_names,
include_lowest=True)
Categorical Variables
 Most statistical models cannot take in the object/string as input

car fuel
A gas
B diesel
C gas
D gas
Categorical -> Numeric
 Add dummy variables for each unique categories
 Assign 0 or 1 in each category
car fuel gas diesel
A gas 1 0
B diesel 0 1
C gas 1 0
D gas 1 0

One-hot encoding
Dummy Variable in Python
 Use pandas.get_dummies() method to convert categorical
variables to dummy variables (0 or 1)
fuel gas diesel
gas 1 0
diesel 0 1
gas 1 0
gas 1 0

pd.get_dummies(df[“fuel”])
Exploratory Data Analysis
Exploratory Data Analysis (EDA)
 Preliminary step in data analysis to:
 Summarize main features of the data
 Gain better understanding of the dataset
 Uncover relationships between features
 Extract important features
Exploratory Data Analysis (EDA)
 Typically involves the following steps:
 Descriptive statistics
 Grouping data using GroupBy
 Correlation
Descriptive Statistics
 Describe basic features of the sample data and give short
summaries about it
Generate summary statistics

Check distribution of categorical variables

Visually summarize distribution of data

Visually examine relationship between numeric variables

Descriptive Stats – Summary Statistics
 Generate summary statistics of sample data using pandas
describe() method

[Link]()
Descriptive Stats–Categorical Variable Distribution
 Summarize categorical variables by using value_counts()
method

df[‘age-bracket’].describe()
Descriptive Stats–Data Distribution with Boxplots

100
90 Upper Extreme
80 Whisker
Upper Quartile
70
60
50 Median
40
30 Lower Quartile
20
10 Lower Extreme
Outlier/Data point
0
Descriptive Stats–Data Distribution with Boxplots
 Generate boxplot of sample data using seaborn’s boxplot()
method

[Link](
x='smoker’,
y=‘bmi’,
data=df
)
Descriptive Stats – Relationships with Scatterplot

 Scatterplot shows the relationship between two variables

 Predictor/Independent variable
 Target/Dependent variable
Descriptive Stats – Relationships with Scatterplot

 Generate scatterplot of sample data using [Link]’s

scatter() method

[Link](
df['engine-size’],
df['price’]
)
Grouping Data
 Use pandas [Link]() method to group categorical variables
into distinct groups/categories
GroupBy with One Categorical Variable
[Link]('sex')['bmi'].mean()
GroupBy with Multiple Categorical Variables
[Link](['sex', 'smoker'])['bmi'].mean()
Grouping Data - Pandas pivot_table Method
 Use pandas df.pivot_table() method to create pivot tables
 One can choose what variables to display along the columns and
rows of the pivot table
df.pivot_table(
index='sex’,
columns='smoker’,
values='bmi’
)
Correlation
 Measures to what extent different numeric variables are
interdependent

Lung cancer Smoking

Correlation is not causation

Rain Umbrella Correlation does not mean that one variable
caused the outcome of the other variable

Sales Profit
Correlation
[Link](x='engine-size', y='price', data=df)

 The straight line shows that there is a

positive linear relationship between
the two variables.
 With increase in values of engine-size,
values of price go up as well.
 Hence, a correlation exist between the
two variables.
Correlation
[Link](x='highway-mpg', y='price', data=df)

 The straight line shows that there is a

negative linear relationship between the
two variables.
 With increase in values of highway-mpg,
values of price go down as well.
 Hence, a correlation exist between the
two variables.
Correlation
[Link](x=‘peak-rpm', y='price', data=df)

 The straight line shows that there is a

weak correlation between the two
variables.
 Both low and high values of peak-rpm,
have both high and low values of price.
 Hence, peak-rpm cannot be used to
predict values of price.
Pearson Correlation
 This is one way to measure correlation, and it gives two values:
 Correlation coefficient
 p-value
Correlation coefficient
 Close to +1: large positive relationship
 Close to -1: large negative relationship
 Close to 0: no relationship

p-value
 p-value < 0.001: strong certainty in the result
 p-value < 0.05: moderate certainty in the result
 p-value < 0.1: weak certainty in the result
 p-value > 0.1: no certainty in the result
Pearson Correlation - Example
coef, p_value = [Link](
autos['horsepower’],
autos['price’]
)

 coef: 0.81  With a correlation coefficient of 0.81,

which is close to 1, there is a strong
positive correlation.
 p_value: 6.37e-48
 The p-value of 6.37e-48 (6.37 * 10-48), is
very small, much smaller than 0.001, so
there is strong certainty about the strong
positive correlation.
Correlation
Heatmap
With a heatmap, we can
look at the correlation
between multiple or all
numeric variables all at
once.
Association
 Measures to what extent different categorical variables are
interdependent.
 Chi-square Test for Association is one test out of a few used for this.
 Chi-square tests how likely it is that an observed distribution is due
to chance.
 Chi-square tests a null hypothesis that the variables are
independent.
 Chi-square does not tell you the type of relationship that exists
between both variables; only that a relationship exists.
Association
 Is there an association between fuel-type and aspiration?

contingency_table = [Link](
autos['fuel-type’],
autos['aspiration’]
)
Association
[Link].chi2_contingency(contingency_table,
correction = True)

 Conclusion
 Chi-square test value is 30.03
 The p-value is 4.24e-08 (4.24 * 10-8) (very close to zero)
 Since p-value is close to zero, we reject the null hypothesis that the two variables are
independent and conclude that there is evidence of association between the two variables
Model Development
Model Development
 A model is a mathematical equation used to predict a value, given
one or more other values.
 Relates one or more independent variables to one dependent
variable.
 Value(s) of the independent variable(s) are a given to the model and
used by the model to predict a value for the dependent variable.
 engine size This car is
$3000
make

mileage

horsepower
Independent variables Dependent variable or
or features target variable

Model
‘highway-mpg’ ‘price’
55 mpg $5000

prediction
 Usually, the more relevant data you have the more accurate your
model is

‘horsepower’

Model
‘engine-size’ ‘price’
‘drive-wheels’
$5400
‘highway-mpg’
Linear Regression
 Simple linear regression
Independent
will refer to one variable Simple Linear prediction
independent variable to 𝑥1
Regression
𝑦
make a prediction

 Multiple linear regression

will refer to multiple Independent
Multiple Linear
variables prediction
independent variables to Regression
make a prediction 𝑥1 , … , 𝑥𝑛 𝑦
Simple Linear Regression
 Method to measure the relationship between two variables:
 The independent (predictor) variable - 𝑥
 The dependent (target) variable - 𝑦

𝑦 = 𝑏0 + 𝑏1 𝑥
 b0 : the intercept
 b1 : the slope or coefficient
Simple Linear Regression - Fit
 After ascertaining that there is a linear relationship between the
two variables, what we do next is to determine the
line/equation/model that best represents the relationship.
 The process of doing so is called fitting.
 Hence, we try to find the line of best fit.
 ‘price’
Preprocessed data ‘highway-mpg’

Fit data to the model

𝑦 = 𝑏0 + 𝑏1 𝑥

(b0 , b1)

𝑦 = 38423 − 821𝑥
Simple Linear Regression – Prediction
 b0 : 38423
 b1 : -821
𝑦 = 38423 − 821𝑥

𝑦 = 38423 − 821𝑥
𝑦 = 38423 − 821(20)
𝑦ො = 22003
predictions = y_hat = 𝒚
ෝ
Fitting a Simple Linear Model Estimator
from sklearn.linear_model import LinearRegression

model = LinearRegression()

X = df[[‘highway-mpg’]]

y = df[‘price’]

[Link](X, y)

y_hat = [Link](X)
Estimated Linear Model
model.intercept_ b0
38423.31

model.coef_
b1
-821.73

 The relationship between price and highway-mpg is given by:

 price = 38423.31 - 821.73 * highway-mpg

 𝑦 = 𝑏0 + 𝑏1 𝑥

price = 38423.31 - 821.73 * highway-mpg

Multiple Linear Regression
 Method to measure the relationship between:
 Two or more independent (predictor) variables - 𝑥
 The dependent (target) variable - 𝑦

𝑦 = 𝑏0 + 𝑏1 𝑥1 + 𝑏2 𝑥2+ 𝑏3 𝑥3

 b0 : the intercept  b2 : the coefficient of x2

 b1 : the coefficient of x1  b3 : the coefficient of x3
Fitting a Multiple Linear Model Estimator
from sklearn.linear_model import LinearRegression

model = LinearRegression()

X = df[[‘highway-mpg’, ‘horsepower’, ‘engine-size’]]

y = df[‘price’]

[Link](X, y)

y_hat = [Link](X)
Model Evaluation using Visualization
 Regression plot

 Residual plot

 Distribution plot
Regression Plot
 Regression plot shows a combination of:
 A scatterplot
 The fitted linear regression line
Regression Plot
import seaborn as sns
[Link](x='highway-mpg', y='price', data=df)
Residual Plot
 Residuals are the differences between actual values of y and
predicted values of y
ෝ
𝒚−𝒚
 We then plot the values of the independent variable against the
residuals to obtain a residual plot
Residual Plot
import seaborn as sns
[Link](x='highway-mpg', y='price', data=df)
Distribution Plot
 Distribution plot shows a comparison of:
 The distribution of actual values of y
 The distribution of predicted values of y
Distribution Plot

ax1 = [Link](
autos['price’],
hist=False,
color=‘r’,
label='Actual Value’
)

[Link](yhat,
hist=False,
color='b’,
label='Predicted Values’,
ax=ax1
)
error = residuals = actual values – predicted values = y - ŷ
Polynomial Regression
 A special case of general linear regression

 Useful for describing curvilinear relationships

Polynomial Regression
 Quadratic – 2nd order
2
𝑦ො = 𝑏0 + 𝑏1 𝑥1 + 𝑏2 𝑥1
 Cubic – 3rd order
2 3
𝑦ො = 𝑏0 + 𝑏1 𝑥1 + 𝑏2 𝑥1 + 𝑏3 𝑥1
 Higher order
2 3
𝑦ො = 𝑏0 + 𝑏1 𝑥1 + 𝑏2 𝑥1 + 𝑏3 𝑥1 + ..
Polynomial Regression
 We can also have multi dimensional polynomial linear regression

2 2
𝑦ො = 𝑏0 + 𝑏1 𝑥1 + 𝑏2 𝑥2 + 𝑏3 𝑥1 𝑥2 + 𝑏4 𝑥1 + 𝑏5 𝑥2
Polynomial Regression
from [Link] import PolynomialFeatures

pr = PolynomialFeatures(degrees=2, include_bias=False)

x_pr = pr.fit_transform(
df[[‘highway-mpg’, ‘horsepower’]]
)
ෝ = 𝒃𝟎 + 𝒃𝟏 𝒙𝟏 + 𝒃𝟐 𝒙𝟐 + 𝒃𝟑 𝒙𝟏 𝒙𝟐 + 𝒃𝟒 𝒙𝟏 𝟐 𝟐
𝒚 + 𝒃𝟓 𝒙𝟐
Methods for Data Normalization
𝑥𝑜𝑙𝑑
Simple Feature Scaling 𝑥𝑛𝑒𝑤 =
𝑥𝑚𝑎𝑥

𝑥𝑜𝑙𝑑 − 𝑥𝑚𝑖𝑛
Min-Max 𝑥𝑛𝑒𝑤 =
𝑥𝑚𝑎𝑥 − 𝑥𝑚𝑖𝑛

𝑥𝑜𝑙𝑑 − 𝜇
Z-Score 𝑥𝑛𝑒𝑤 =
𝜎
Preprocessing – Data Normalization
 We can normalize multiple features simultaneously:

from [Link] import StandardScaler

scaler = StandardScaler()

X_scaled = scaler.fit_transform(
df[[‘highway-mpg’, ‘horsepower’]]
)

StandardScaler uses Z-Score normalization 𝑥𝑜𝑙𝑑 − 𝜇

method to transform data
𝑥𝑛𝑒𝑤 =
𝜎
Pipelines
 There are many steps to getting a prediction

Polynomial
Normalization Regression
Transformation

 Pipelines automate all these steps, thereby simplifying the whole

process of prediction
Pipelines
from sklearn.linear_model import LinearRegression

from [Link] import PolynomialFeatures

from [Link] import StandardScaler

from [Link] import Pipeline

Pipelines
input = [
(‘polynomial’, PolynomialFeatures(degree=2)),
(‘scaler’, StandardScaler()),
(‘model’, LinearRegression())
]
pipe = Pipeline(input)
Pipelines
[Link](df[[‘highway-mpg’, ‘horsepower’]], y)

Polynomial
Normalization Regression
Transformation
Pipelines
y_hat = [Link](df[[‘highway-mpg’, ‘horsepower’]])
Measures for In-Sample Evaluation
 A way to numerically determine how good a model fits a dataset

 Two key measures to determine the fit of a model:

 Mean Squared Error (MSE)
 R-Squared (R2)
error = residuals = actual values – predicted values = y - ŷ
Mean Squared Error (MSE)

Actual Predicted Error Squared  We now calculate the Average of the

Squared Error (MSE)
values values Error
(y) (ŷ) (y-ŷ) (y-ŷ)2
1000+3600+100+100
150 50 100 10000 =
120 60 60 3600 4
90 100 -10 100
= 3450
140 130 10 100
Mean Squared Error (MSE)
from [Link] import mean_squared_error

mean_squared_error(df[‘price’], y_hat)

3163502.9446
R-Squared (R2)
 Also referred to as Coefficient of Determination

 It determines how close the data is to the fitted regression line

 It is the percentage of variation of the target variable that is

explained by the linear model
 It is typically a comparison of the regression model and a simple
model, i.e., the mean of the data points
R-Squared (R2)

𝑀𝑆𝐸 𝑜𝑓 𝑟𝑒𝑔𝑟𝑒𝑠𝑠𝑖𝑜𝑛 𝑙𝑖𝑛𝑒

𝑅 =1-
2
𝑀𝑆𝐸 𝑜𝑓 𝑚𝑒𝑎𝑛 𝑜𝑓 𝑦
R-Squared (R2)
MSE regression line MSE mean of y

Actual Predicted Error Squared Actual Mean Error Squared

values values Error values value Error
(y) (ŷ) (y-ŷ) (y-ŷ)2 (y) (𝒚ഥ) (y-𝒚ഥ) (y-𝒚ഥ )2
150 50 100 10000 150 125 25 625
120 60 60 3600 120 125 -5 25
90 100 -10 100 90 125 -35 1225
140 130 10 100 140 125 15 225

10000+3600+100+100 625+25+1225+225
= =
4 4
= 3450 = 525
R-Squared (R2)

𝑀𝑆𝐸 𝑜𝑓 𝑟𝑒𝑔𝑟𝑒𝑠𝑠𝑖𝑜𝑛 𝑙𝑖𝑛𝑒

𝑅 =1-
2
𝑀𝑆𝐸 𝑜𝑓 𝑚𝑒𝑎𝑛 𝑜𝑓 𝑦

3450
𝑅 =1-
2
525

𝑅 = 1 - 6.57
2
R-Squared (R2)
 Generally values of R-Squared are between 0 and 1

 A value near 1 means that the line is a good fit for the data

 A value near 0 means that the line is not a good fit for the data

 If your R-Squared is negative, it can be due to over fitting

R-Squared (R2)
X = df[[‘highway-mpg’]]

y = df[‘price’]

[Link](X, y)

[Link](X, y)

0.496591188
Model Evaluation
Model Evaluation
 There two types of model evaluation:
 In-sample evaluation
 Out-of-sample evaluation
 In-sample evaluation Out-of-sample evaluation

Both training and test data Training data Test data

In-sample data In-sample data Out-of-sample data

Model Evaluation
 In-sample evaluation tells us how well our model will fit the data
used to train it
 However, it does not tell us how well the trained model can be used
to predict new data
Model Evaluation
 Hence, we split the data into 2 sets:
 In-sample data (training data)
 Out-of-sample data (test data)
 Evaluation done with out-of-sample data is referred to as out-of-
sample evaluation
Training and Test Sets
Height Weight State
Height Weight State
150 50 New York
150 50 New York 120 60 Lagos
90 100 Paris
Training
120 60 Lagos data
140 130 New York
90 100 Paris 160 80 California
140 130 New York 155 70 Cape Town
160 80 California 130 60 Lagos

155 70 Cape Town

130 60 Lagos
Height Weight State
147 80 Arizona
147 80 Arizona Test
131 100 Abuja 131 100 Abuja data
145 121 New York 145 121 New York
Training and Test Sets
Data:
Training and Test Sets
Data:

Training set (70%) Test set (30%)

 Build and train the model with training set

 Assess model performance with test set
train_test_split() Function
 Use train_test_split function in scikit-learn package to split
data into training and test sets
train_test_split() Function
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(

X, y,
test_size=0.3,
random_state=0
)

 X: features or independent variables  test_size: percentage of data for testing

 y: target variable  random_state: number generator used for

number sampling
 X_train, y_train: training set

 X_test, y_test: test set

Generalization Performance
 Generalization error is a measure of how well a model does at
predicting unseen data
 The error we obtain using the test set is an approximation of this
error
 It is directly influenced by bias and variance
𝑮𝒆𝒏𝒆𝒓𝒂𝒍𝒊𝒛𝒂𝒕𝒊𝒐𝒏 𝑬𝒓𝒓𝒐𝒓 = 𝑩𝒊𝒂𝒔𝟐 + 𝑽𝒂𝒓𝒊𝒂𝒏𝒄𝒆 + 𝑰𝒓𝒓𝒆𝒅𝒖𝒄𝒊𝒃𝒍𝒆 𝑬𝒓𝒓𝒐𝒓

 Bias: Error due to overly simplistic assumptions in the model

(underfitting)
 Variance: Error due to excessive sensitivity to small fluctuations in
training data (overfitting)
Think of your
model as a
Basketballer…

…being observed
taking 100 shots.
 Scenario 1: Using a Lot of Training Data (90% Training, 10% Testing)

 You watch the player take 100 shots (90 for training, 10 for testing)
 First try: They make 8/10 test shots → 80% accuracy
 Second try: They make 7/10 → 70% accuracy
 Third try: They make 9/10 → 90% accuracy

 What’s happening?
 Each test is close to their true generalization error, but the results vary a lot
(low precision)
 This is because the test set is small (only 10 shots), so luck plays a big role
70 80 90

The results vary a lot (low precision)

 Scenario 2: Using More Test Data (10% Training, 90% Testing)

 You watch the player take 100 shots (10 for training, 90 for testing)
 First try: They make 72/90 → 80% accuracy
 Second try: They make 75/90 → 83% accuracy
 Third try: They make 70/90 → 78% accuracy

 What’s happening?
 The results are more consistent (higher precision) because the test set is
larger
 But you had to use less training data (only 10 shots), so your model might
not learn as well
 78 80 83

The results are more consistent (higher precision)

Their true skill might actually be 85%, but the model (trained poorly) keeps estimating ~80%.
So, the results are consistent but wrong (far from true generalization error)
 Scenario 1 Scenario 2

70 80 90 78 80 83

 More Training Data → High Accuracy  More Testing Data → High Precision
(Low Bias in Model) (Low Variance in Estimates)

 Less Testing Data → High Precision  Less Training Data → Low Accuracy
(High Variance in Estimates) (High Bias in Model)
True Generalization
Error
 Scenario 1 Scenario 2

70 80 90 78 80 83
 Scenario 1 Scenario 2

 Low Bias in Model  Low Variance in Estimates

 High Variance in Estimates  High Bias in Model

 Bias-Variance Tradeoff

 Low Bias in Model  Low Variance in Estimates

 High Variance in Estimates  High Bias in Model

Cross Validation
 To estimate and manage bias and variance effectively, we use cross
validation

 It is the gold standard for model evaluation in Machine Learning,

ensuring that performance metrics are reliable and models
generalize well

 In cross validation, each observation is both used for training and

testing
Cross Validation
Data Training

Test

1st time
1st fold 2nd fold 3rd fold 4th fold

2nd time

3rd time

4th time
 𝑀𝑆𝐸 𝑜𝑓 𝑟𝑒𝑔𝑟𝑒𝑠𝑠𝑖𝑜𝑛 𝑙𝑖𝑛𝑒
𝑅2 = 1 -
𝑀𝑆𝐸 𝑜𝑓 𝑚𝑒𝑎𝑛 𝑜𝑓 𝑦

Model
cross_val_score() Function
from sklearn.model_selection import cross_val_score

scores = cross_val_score(model, X, y, cv=4)

[Link](scores)
Overfitting and Underfitting
 Overfitting is where the model is too flexible and fits the noise
rather than the function
 Underfitting is where the model is too simple to fit the data
Overfitting and Underfitting

Underfitting Overfitting

 High training error  Low training error

 High test error  High test error

Training error is error measured (MSE) on training set

Test error is error measured (MSE) on test set
Ridge Regression
 Ridge Regression is a type of linear regression that prevents
overfitting by adding a penalty term (L2 regularization) to the
model’s coefficients.
 It’s perfect for datasets with many features or when features are
highly correlated (multicollinearity).
 The strength of the penalty term is controlled by a parameter
known as alpha (ʎ)
Ridge Regression

𝑦ො = 1 + 2𝑥 − 3𝑥2 − 2𝑥3 − 12𝑥4 − 40𝑥5 + 80𝑥6 + 71𝑥7 − 141𝑥8 − 38𝑥9 + 75𝑥10

Alpha x x2 x3 x4 x5 x6 x7 x8 x9 x10

0 2 -3 -2 -12 -40 80 71 -141 -38 75

0.001 2 -3 -7 5 4 -6 4 -4 4 6
0.01 1 -2 -5 -0.04 0.15 -1 1 -0.5 0.3 1
1 0.5 -1 -1 -0.614 0.7 -0.38 -0.56 -0.21 -0.5 -0.1
10 0 -0.5 -0.3 -0.37 -0.3 -0.3 -0.22 -0.22 -0.22 -0.17
Ridge Regression
from sklearn.linear_model import Ridge

model = Ridge(alpha=0.1)

[Link](X, y)

y_hat = [Link](X)
Ridge Regression
Data:
Ridge Regression
Data:

Training set Validation set Test set

This is similar to test data, but it is used to select parameters like alpha
Ridge Regression

Alpha R2
0.1 Model 1 Model 1 0.5

Train Evaluate

1 Model 2 Model 2 0.75

Train Evaluate

10 Model 3 Model 3 0.55

Train Evaluate
Hyperparameters
 Because we are manually setting different values for alpha just to
select the best alpha value which maximizes the R2, we call alpha a
hyperparameter.
 Hyperparameters are the "knobs" or "settings" you manually tune
before training a machine learning model. They control how the
model learns from data.
 Hyperparameters are not estimated from the data but are instead
chosen by the practitioner or optimized via external methods
Grid Search
 Scikit-learn has a means of automatically iterating over these
hyperparameters using cross-validation called Grid Search.
Grid Search
Ridge

Scoring
Method
(R2, MSE)
Alpha 1 10 100 1000
Grid Search CV R2 0.74 0.35 0.073 0.008
Number
of folds

Alpha 1 10 100 1000

from sklearn.linear_model import Ridge
from sklearn.model_selection import GridSearchCV
parameters = [{‘alpha’: [0.1, 1, 10, 100, 1000, 10000]}]
model = Ridge()
grid = GridSearchCV(model, parameters, cv=4)
[Link](X, y)
print(grid.best_estimator_)
scores = grid.cv_results_
print(scores[‘mean_test_score’])
That’s a Wrap!