Sarita Chauhan

Roll no: 06

MCA Sem III

Data Science and Analytics Lab
Index

Sr. Page
No Topics number
Learning Objectives, Understanding theDomain, Understanding 3-50
1 the Dataset, Python package for data science, Importing and
Exporting Data in Python,Basic Insights from Datasets.
Data collection : Importing and Exporting Data in Python,Basic 51-57
2 Insights from Datasets. Identify and Handle Missing Values, Data
Formatting, Data Normalization Sets, Binning, Indicator variables.

3 Static Dashboard 58-66

4 Dynamic Dashboard 67-72

5 Basic of Grouping, ANOVA Model Evaluation Using 73-78

Visualization,
6 Simple Linear Regression, And Multiple Linear Regression, 79-97
Model Evaluation Using Visualization
7 Prediction and Decision Making 98-108

8 Model Evaluation, Over-fitting, Under-fitting and Model, 109-116

1
Sarita Chauhan
Roll no: 06

Lab 1: Learning Objectives, Understanding the Domain, Understanding the

Dataset, Python package for data science, Importing and Exporting Data in
Python, Basic Insights from Datasets.

2
Sarita Chauhan
Roll no: 06

3
Sarita Chauhan
Roll no: 06

4
Sarita Chauhan
Roll no: 06

5
Sarita Chauhan
Roll no: 06

6
Sarita Chauhan
Roll no: 06

7
Sarita Chauhan
Roll no: 06

8
Sarita Chauhan
Roll no: 06

9
Sarita Chauhan
Roll no: 06

10
Sarita Chauhan
Roll no: 06

11
Sarita Chauhan
Roll no: 06

12
Sarita Chauhan
Roll no: 06

13
Sarita Chauhan
Roll no: 06

14
Sarita Chauhan
Roll no: 06

15
Sarita Chauhan
Roll no: 06

16
Sarita Chauhan
Roll no: 06

17
Sarita Chauhan
Roll no: 06

18
Sarita Chauhan
Roll no: 06

19
Sarita Chauhan
Roll no: 06

20
Sarita Chauhan
Roll no: 06

21
Sarita Chauhan
Roll no: 06

22
Sarita Chauhan
Roll no: 06

23
Sarita Chauhan
Roll no: 06

24
Sarita Chauhan
Roll no: 06

25
Sarita Chauhan
Roll no: 06

26
Sarita Chauhan
Roll no: 06

27
Sarita Chauhan
Roll no: 06

28
Sarita Chauhan
Roll no: 06

29
Sarita Chauhan
Roll no: 06

30
Sarita Chauhan
Roll no: 06

31
Sarita Chauhan
Roll no: 06

32
Sarita Chauhan
Roll no: 06

33
Sarita Chauhan
Roll no: 06

34
Sarita Chauhan
Roll no: 06

35
Sarita Chauhan
Roll no: 06

36
Sarita Chauhan
Roll no: 06

37
Sarita Chauhan
Roll no: 06

38
Sarita Chauhan
Roll no: 06

39
Sarita Chauhan
Roll no: 06

40
Sarita Chauhan
Roll no: 06

41
Sarita Chauhan
Roll no: 06

42
Sarita Chauhan
Roll no: 06

43
Sarita Chauhan
Roll no: 06

44
Sarita Chauhan
Roll no: 06

45
Sarita Chauhan
Roll no: 06

46
Sarita Chauhan
Roll no: 06

47
Sarita Chauhan
Roll no: 06

48
Sarita Chauhan
Roll no: 06

49
Sarita Chauhan
Roll no: 06

50
Sarita Chauhan
Roll no: 06

51
Sarita Chauhan
Roll no: 06

52
Sarita Chauhan
Roll no: 06

53
Sarita Chauhan
Roll no: 06

54
Sarita Chauhan
Roll no: 06

55
Sarita Chauhan
Roll no: 06

56
Sarita Chauhan
Roll no: 06

57
Sarita Chauhan
Roll no: 06

58
Sarita Chauhan
Roll no: 06

59
Sarita Chauhan
Roll no: 06

60
Sarita Chauhan
Roll no: 06

61
Sarita Chauhan
Roll no: 06

62
Sarita Chauhan
Roll no: 06

Lab 2: Data collection : Importing and Exporting Data in Python, Basic

Insights from Datasets. Identify and Handle Missing Values,
Data Formatting, Data Normalization Sets, Binning, Indicator
variables.

1. Data Preprocessing Techniques :

Identify missing values using methods like .isnull() or .info().

Handle missing values:

• Drop rows/columns containing missing data (dropna).
• Impute values using mean, median, mode, or forward/backward fill.

Code:
import pandas as pd
import numpy as np

# Initial dataset
data = {
63
Sarita Chauhan
Roll no: 06

'Name': ['Alice', 'Bob', 'Charlie', 'David', [Link]],

'Age': [25, 30, [Link], 35, 40],
'Salary': [50000, 60000, 70000, None, 90000]
}
df = [Link](data)

print("Original DataFrame:")
print(df)

print("\nMissing value summary:")

print([Link]().sum())

print("\nDataset Info:")
print([Link]())

# Step 1: Fill missing Age with mean

age_mean = df['Age'].mean()
df['Age'].fillna(age_mean, inplace=True)
print("\nAfter filling missing 'Age' with mean:")
print(df)

# Step 2: Fill missing Salary with median

salary_median = df['Salary'].median()
df['Salary'].fillna(salary_median, inplace=True)
print("\nAfter filling missing 'Salary' with median:")
print(df)

# Step 3: Drop rows where Name is missing

[Link](subset=['Name'], inplace=True)
print("\nAfter dropping rows with missing 'Name':")
print(df)

# Reset index
df.reset_index(drop=True, inplace=True)
print("\nFinal DataFrame:")
print(df)

Output:

Original DataFrame:
Name Age Salary
0 Alice 25.0 50000.0
64
Sarita Chauhan
Roll no: 06

1 Bob 30.0 60000.0

2 Charlie NaN 70000.0
3 David 35.0 NaN
4 NaN 40.0 90000.0

--- Missing value summary (.isnull().sum()) ---

Name 1
Age 1
Salary 1
dtype: int64

--- Info (.info()) ---

<class '[Link]'>
RangeIndex: 5 entries, 0 to 4
Data columns (total 3 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 Name 4 non-null object
1 Age 4 non-null float64
2 Salary 4 non-null float64
dtypes: float64(2), object(1)
memory usage: 252.0+ bytes
None

After Step 1: Filled missing 'Age' with mean (32.50):

Name Age Salary
0 Alice 25.0 50000.0
1 Bob 30.0 60000.0
2 Charlie 32.5 70000.0
3 David 35.0 NaN
4 NaN 40.0 90000.0

After Step 2: Filled missing 'Salary' with median (65000.00):

Name Age Salary
0 Alice 25.0 50000.0
1 Bob 30.0 60000.0
2 Charlie 32.5 70000.0
3 David 35.0 65000.0
65
Sarita Chauhan
Roll no: 06

4 NaN 40.0 90000.0

After Step 3: Dropped rows with missing 'Name':

Name Age Salary
0 Alice 25.0 50000.0
1 Bob 30.0 60000.0
2 Charlie 32.5 70000.0
3 David 35.0 65000.0

Final DataFrame (index reset):

Name Age Salary
0 Alice 25.0 50000.0
1 Bob 30.0 60000.0
2 Charlie 32.5 70000.0
3 David 35.0 65000.0

2. Data Formatting
Ensuring data types are consistent (e.g., converting object to numeric or date).

Code:
df['Salary'] = pd.to_numeric(df['Salary'], errors='coerce')
df['Salary'].fillna(0, inplace=True)
print("Data Types:\n", [Link])

Output:

66
Sarita Chauhan
Roll no: 06

3. Data Normalization
Scaling data to a range (e.g., 0–1) for uniformity.

Code:
from [Link] import MinMaxScaler

# Select only numeric columns for scaling

scaler = MinMaxScaler()
df[['Age', 'Salary']] = scaler.fit_transform(df[['Age', 'Salary']])

print("\nNormalized Data:\n", df)

Output:

4. Binning
Categorizing numeric data into intervals (bins).

Code:
# Binning Age into categories: Young, Middle-aged, Senior
bins = [0, 0.33, 0.67, 1]
67
Sarita Chauhan
Roll no: 06

labels = ['Young', 'Middle-aged', 'Senior']

df['Age_Group'] = [Link](df['Age'], bins=bins, labels=labels,

include_lowest=True)

print("\nData after Binning:\n", df)

Output:

5. Indicator Variables (Dummy Variables)

Converting categorical variables into numeric indicators.

Code:
# Create dummy variables for Age_Group
df = pd.get_dummies(df, columns=['Age_Group'], prefix='Group')

print("\nData with Dummy Variables:\n", df)

Output:

6. Visualizations
Histogram of Age Distribution

68
Sarita Chauhan
Roll no: 06

Code:
import [Link] as plt

[Link](df['Age'], bins=4, edgecolor='black',color='green')

[Link]('Age Distribution')
[Link]('Age (Normalized)')
[Link]('Frequency')
[Link]()

output:

Bar Chart of Age Groups

Code:
# Create Age Groups again (if not already created)
bins = [0, 0.33, 0.67, 1]
labels = ['Young', 'Middle-aged', 'Senior']
df['Age_Group'] = [Link](df['Age'], bins=bins, labels=labels)

# Plot Bar Chart

df['Age_Group'].value_counts().plot(kind='bar', color='pink',edgecolor='black')
[Link]('Age Group Distribution')
[Link]('Age Group')
69
Sarita Chauhan
Roll no: 06

[Link]('Count')
[Link]()

Output:

Updated Dataset :

Name Age (Normalized) Salary (Normalized) Age_Group

Alice 0.000000 0.000000 Young

Bob 0.416667 0.285714 Middle-aged

Charlie 0.250000 0.571429 Young

David 0.625000 0.428571 Middle-aged

70
Sarita Chauhan
Roll no: 06

Lab 3: Static Dashboard

For textual/analytics charts for each type of data.

1. Categorical/Text Columns
Columns: Program Name, Mode, Institutes, Medium, Location
Charts:
• Bar/Horizontal Bar: Program count by institute, mode, or location.
• Pie/Donut: Proportion by Mode or Medium.
• Treemap/Sunburst: Hierarchical view (Institute → Mode → Program).
2. Textual/Description Columns
Columns: Eligibility, Admission Procedure, Objectives, Career Prospects
Charts:
• Word Cloud: Frequent keywords.
• Bar/Bigrams/Trigrams: Top words and phrases.
• Sentiment Chart: Positive vs negative vs neutral distribution.
3. Numeric Columns
Columns: Duration, Annual Intake, Fees
Charts:
• Histogram: Distributions of duration, intake, and fees.
• Box Plot: Compare numeric data by category.
• Scatter/Bubble: Fees vs duration with intake as bubble size.
4. Combined Views / Dashboard
Charts:
• Stacked/Grouped Bar: Mode-based program comparison.
• Heatmap: Keyword frequency by institute.
• Streamlit Dashboard: Interactive filters + word cloud + hoverable charts.

Dashboard Output (from code):

• Treemap → hierarchical distribution.
• Word Cloud → top keywords.
• Sunburst → category proportions.
• Bubble Chart → numerical relationships.

Code:

# ================== IMPORTS ==================

import streamlit as st
import pandas as pd
71
Sarita Chauhan
Roll no: 06

import numpy as np
import os
import [Link] as px
import [Link] as plt
from wordcloud import WordCloud

# ================== PAGE CONFIG ==================

st.set_page_config(
page_title="Text Dataset Dashboard",
layout="wide",
)

# ================== LIGHT PROFESSIONAL BACKGROUND

==================
[Link]("""
<style>
[data-testid="stAppViewContainer"] {
background: linear-gradient(135deg, #ffffff, #e6f2ff); /* light white to blue
gradient */
}
[data-testid="stSidebar"] > div:first-child {
background: #e6f7ff; /* light sidebar */
}
h1, h2, h3, h4, h5, h6, p, span, label, div {
color: #000000; /* dark text */
}
</style>
""", unsafe_allow_html=True)

[Link]("<h1 style='text-align:center;'>Static Dashboard</h1>",

unsafe_allow_html=True)

# ================== LOAD CSV/TXT FILES ==================

DATASET_FOLDER = "datasets"

if not [Link](DATASET_FOLDER):
[Link](f" Folder '{DATASET_FOLDER}' not found!")
[Link]()

datasets = {}
for f in [Link](DATASET_FOLDER):
if [Link](".csv") or [Link](".txt"):
72
Sarita Chauhan
Roll no: 06

try:
df = pd.read_csv([Link](DATASET_FOLDER, f), encoding="utf-8",
on_bad_lines="skip")
except:
df = pd.read_csv([Link](DATASET_FOLDER, f),
encoding="latin1", on_bad_lines="skip")
df["Dataset_Name"] = f
datasets[f] = df

if len(datasets) == 0:
[Link]("⚠ No CSV/TXT files found in folder.")
[Link]()

# ================== MULTI-DATASET SELECTOR

==================
selected_files = [Link](
"Select Dataset(s)",
list([Link]())
)
if len(selected_files) == 0:
[Link]("Please select at least one dataset.")
[Link]()

# Merge selected datasets

df = [Link]([datasets[f] for f in selected_files], ignore_index=True)

# ================== DATA CLEANING ==================

def clean_dataset(df):
df = [Link](how='all')
[Link](["", " ", "unknown", "Unknown", "NA", "na"], [Link],
inplace=True)
df = [Link](how='any')
df.reset_index(drop=True, inplace=True)
for col in [Link]:
if df[col].dtype == "object":
try:
df[col] = pd.to_numeric(df[col])
except:
pass
return df

df = clean_dataset(df)
73
Sarita Chauhan
Roll no: 06

# ================== PARAMETER SELECTOR ==================

all_columns = [Link]()
selected_param = [Link]("Select a Text Column / Parameter",
all_columns)

# ================== ADD WORD AND CHAR COUNT

==================
df["word_count"] = df[selected_param].astype(str).apply(lambda x: len([Link]()))
df["char_count"] = df[selected_param].astype(str).apply(lambda x: len(x))

# ================== GRID LAYOUT 2x2 FOR ALL CHARTS

==================
figsize = (6,4) # Smaller figure size to fit 4 charts on single page
color_seq = [
"#FF5733", "#33FF57", "#3357FF", "#F333FF", "#FFC300",
"#FF33A6", "#33FFF6", "#A633FF", "#FF8C33", "#33FF8C",
"#228B22", "#FFD700", "#FF69B4", "#00CED1", "#8A2BE2"
]

row1_cols = [Link](2)
row2_cols = [Link](2)

# ---------------- TREEMAP ----------------

with row1_cols[0]:
[Link]("### Treemap")
try:
hierarchy_cols = [Link][:3].tolist()
numeric_cols = df.select_dtypes(include=['int','float']).[Link]()
value_col = numeric_cols[0] if numeric_cols else "word_count"

fig = [Link](
df,
path=hierarchy_cols,
values=value_col,
color=value_col,
hover_data=[Link],
color_discrete_sequence=color_seq,
template="plotly_white"
)
fig.update_traces(textfont_color="black")
fig.update_layout(margin=dict(t=50, l=25, r=25, b=25))
74
Sarita Chauhan
Roll no: 06

st.plotly_chart(fig, use_container_width=True)
except:
[Link]("Treemap could not be generated.")

# ---------------- WORD CLOUD ----------------

with row1_cols[1]:
[Link]("### Word Cloud")
try:
text = " ".join(df[selected_param].astype(str))
wc = WordCloud(
width=800,
height=400,
background_color="#ffffff",
colormap="tab20",
max_words=250,
contour_color='steelblue',
contour_width=1
).generate(text)
[Link](figsize=figsize)
[Link](wc, interpolation="bilinear")
[Link]("off")
[Link](plt)
except:
[Link]("Word Cloud could not generate.")

# ---------------- SUNBURST ----------------

with row2_cols[0]:
[Link]("### Sunburst Chart")
try:
hierarchy_cols = [Link][:3].tolist()
numeric_cols = df.select_dtypes(include=['int','float']).[Link]()
value_col = numeric_cols[0] if numeric_cols else "word_count"

fig = [Link](
df,
path=hierarchy_cols,
values=value_col,
color=value_col,
color_discrete_sequence=color_seq,
template="plotly_white"
)
fig.update_traces(textfont_color="black")
75
Sarita Chauhan
Roll no: 06

st.plotly_chart(fig, use_container_width=True)
except:
[Link]("Sunburst could not be generated.")

# ---------------- BUBBLE CHART ----------------

with row2_cols[1]:
[Link]("### Bubble Chart")
try:
numeric_cols = df.select_dtypes(include=['int','float']).[Link]()
if len(numeric_cols) < 2:
numeric_cols = ["word_count","char_count"]
size_col = numeric_cols[1] if len(numeric_cols) > 1 else numeric_cols[0]

fig = [Link](
df,
x=numeric_cols[0],
y=size_col,
size=size_col,
color=numeric_cols[0],
hover_data=[Link],
color_discrete_sequence=color_seq,
template="plotly_white"
)
fig.update_layout(font_color="black")
st.plotly_chart(fig, use_container_width=True)
except:
[Link]("Bubble chart could not be generated.")

output:

76
Sarita Chauhan
Roll no: 06

Treemap:

77
Sarita Chauhan
Roll no: 06

Wordcloud:

Sunburst:

78
Sarita Chauhan
Roll no: 06

Bubble chart:

79
Sarita Chauhan
Roll no: 06

Conclusion:

The Text Dataset Dashboard provides an interactive and visual way to explore
textual datasets in CSV or TXT format. After loading and cleaning multiple
datasets, the dashboard computes basic metrics such as word count and
character count for each text entry. Using these metrics and other numeric data
in the dataset, the dashboard generates several visualizations to help understand
the structure, distribution, and patterns within the data:
1. Treemap – Shows hierarchical relationships within the dataset and allows
comparison of values across multiple categorical levels.
2. Word Cloud – Highlights the most frequent words in the selected text
column, giving quick insights into common terms and themes.
3. Sunburst Chart – Visualizes hierarchical data with proportions, providing
an alternative perspective to the treemap.
4. Bubble Chart – Displays relationships between numeric variables, with
bubble size representing a third metric, allowing identification of clusters
or outliers.

Results
• Word counts and character counts are computed for all text entries,
enabling quantitative analysis of text length.
• Visualizations are generated dynamically based on the dataset, providing
interactive insights:
o Treemap and Sunburst help identify hierarchical and categorical
patterns.
80
Sarita Chauhan
Roll no: 06

o Word Cloud highlights the most frequent terms.

o Bubble Chart shows numeric relationships and potential correlations.

Lab 4: Dynamic Dashboard

Code:

# ================== IMPORTS ==================

import streamlit as st
import pandas as pd
import numpy as np
import [Link] as px
import re

# ================== PAGE CONFIG ==================

st.set_page_config(page_title="SNDT Programs Dashboard", layout="wide",
page_icon=" ")
[Link]("Dynamic SNDT programmes Dashboard")

# ================== DATA ==================

data = [
{"Program": "[Link]. Nursing", "Duration": "4 Years", "Fees": 765, "Campus":
"Churchgate", "Department": "Nursing"},
{"Program": "[Link] Special Education: Intellectual Disability", "Duration": "2
Years", "Fees": 35000, "Campus": "Juhu", "Department": "Special Education"},
{"Program": "[Link] Special Education: Learning Disability", "Duration": "2
81
Sarita Chauhan
Roll no: 06

Years", "Fees": 35000, "Campus": "Juhu", "Department": "Special Education"},

{"Program": "[Link] Special Education: Visual Impairment", "Duration": "2
Years", "Fees": 35000, "Campus": "Juhu", "Department": "Special Education"},
{"Program": "[Link]", "Duration": "4 Years", "Fees": 0, "Campus":
"Juhu", "Department": "Pharmacy"},
{"Program": "[Link] Computer Science & Technology", "Duration": "4
Years", "Fees": 146141, "Campus": "Juhu", "Department": "Engineering"},
{"Program": "BBA LLB", "Duration": "5 Years", "Fees": 0, "Campus": "Juhu",
"Department": "Law & Management"},
{"Program": "M.A. Marathi", "Duration": "2 Years", "Fees": 7215, "Campus":
"Churchgate", "Department": "Arts"},
{"Program": "M.A. Hindi", "Duration": "2 Years", "Fees": 7215, "Campus":
"Churchgate", "Department": "Arts"},
{"Program": "M.A. English", "Duration": "2 Years", "Fees": 7215, "Campus":
"Churchgate", "Department": "Arts"},
{"Program": "[Link]. Nursing", "Duration": "2 Years", "Fees": 81025,
"Campus": "Churchgate", "Department": "Nursing"},
{"Program": "PG Diploma in Travel and Tourism", "Duration": "1 Year",
"Fees": 0, "Campus": "Juhu", "Department": "Tourism"},
{"Program": "Certificate in Travel and Tourism", "Duration": "6 Months",
"Fees": 0, "Campus": "Juhu", "Department": "Tourism"},
{"Program": "M.A. Women’s Studies", "Duration": "2 Years", "Fees": 7215,
"Campus": "Churchgate", "Department": "Women’s Studies"},
{"Program": "[Link]. Food Science & Nutrition", "Duration": "2 Years", "Fees":
26410, "Campus": "Juhu", "Department": "Food Science"},
{"Program": "[Link]. Clinical Nutrition & Dietetics", "Duration": "2 Years",
"Fees": 66310, "Campus": "Juhu", "Department": "Food Science"},
{"Program": "[Link]. Resource Management & Ergonomics", "Duration": "2
Years", "Fees": 0, "Campus": "Juhu", "Department": "Home Science"},
{"Program": "[Link]. Extension & Communication", "Duration": "2 Years",
"Fees": 22000, "Campus": "Juhu", "Department": "Communication"},
{"Program": "M.A. Media and Communication", "Duration": "2 Years",
"Fees": 43000, "Campus": "Juhu", "Department": "Media"},
{"Program": "[Link]. Human Development", "Duration": "2 Years", "Fees":
15965, "Campus": "Juhu", "Department": "Human Development"},
{"Program": "[Link]. Early Childhood Education", "Duration": "2 Years",
"Fees": 0, "Campus": "Juhu", "Department": "Early Childhood"},
{"Program": "[Link]. Textile Science & Apparel Design", "Duration": "2
Years", "Fees": 30000, "Campus": "Juhu", "Department": "Textile & Apparel"},
{"Program": "[Link]. Fashion Design & Textile", "Duration": "2 Years", "Fees":
60000, "Campus": "Juhu", "Department": "Fashion"},
{"Program": "[Link] Special Education: Intellectual Disability", "Duration": "2
82
Sarita Chauhan
Roll no: 06

Years", "Fees": 35000, "Campus": "Juhu", "Department": "Special Education"},

]

df = [Link](data)

# ================== CLEAN DATA ==================

def duration_to_months(dur):
dur = [Link]()
years = [Link](r"(\d+)\s*year", dur)
months = [Link](r"(\d+)\s*month", dur)
total = 0
if years: total += int(years[0])*12
if months: total += int(months[0])
return total

df['Duration_Months'] = df['Duration'].apply(duration_to_months)

# ================ STREAMLIT WIDGETS ==================

dept_options = df['Department'].unique().tolist()
selected_dept = [Link]("Select Department(s):", options=dept_options,
default=dept_options)
filtered_df = df[df['Department'].isin(selected_dept)].reset_index(drop=True)

# ================== BUBBLE SIZE SLIDER ==================

min_bubble_size = [Link]("Min Bubble Size", min_value=1, max_value=20,
value=1)
max_bubble_size = [Link]("Max Bubble Size", min_value=5, max_value=50,
value=10)

def normalize_sizes(values, min_size=min_bubble_size,

max_size=max_bubble_size):
min_val, max_val = [Link](values), [Link](values)
if min_val == max_val:
return np.full_like(values, (min_size + max_size) / 2)
return min_size + (values - min_val) / (max_val - min_val) * (max_size -
min_size)

filtered_df['Size'] = normalize_sizes(filtered_df['Fees'].values)

# ============ CREATE ANIMATED FRAMES ==================

num_frames = 50
frames_list = []
83
Sarita Chauhan
Roll no: 06

[Link](42)
x_start = filtered_df['Duration_Months'].values + [Link](-5, 5,
size=len(filtered_df))
y_start = filtered_df['Fees'].values + [Link](-2000, 2000,
size=len(filtered_df))

for i in range(num_frames):
x_start += [Link](-2, 2, size=len(filtered_df))
y_start += [Link](-1000, 1000, size=len(filtered_df))
frame = filtered_df.copy()
frame['x'] = x_start
frame['y'] = y_start
frame['frame'] = i
frames_list.append(frame)

animated_df = [Link](frames_list, ignore_index=True)

# =========== PLOTLY ANIMATED FIGURE ===========

fig = [Link](
animated_df,
x='x',
y='y',
animation_frame='frame',
size='Size',
color='Department',
hover_name='Program',
hover_data=['Campus', 'Fees', 'Duration'],
size_max=max_bubble_size,
color_discrete_sequence=[Link].Dark24 +
[Link].Set3 + [Link],
)

fig.update_traces(marker=dict(opacity=0.6, line=dict(width=1, color='black')))

fig.update_layout(
title=dict(text="SNDT University Programs", font=dict(color='black',
size=18)),
xaxis=dict(title=dict(text="Duration (Months)", font=dict(color='black',
size=14)), tickfont=dict(color='black')),
yaxis=dict(title=dict(text="Fees (INR)", font=dict(color='black', size=14)),
tickfont=dict(color='black')),
84
Sarita Chauhan
Roll no: 06

legend=dict(title=dict(text="Department", font=dict(color='black')),
font=dict(color='black')),
plot_bgcolor='rgb(245,245,245)',
paper_bgcolor='rgb(245,245,245)',
)

# ========== DISPLAY IN STREAMLIT ========

st.plotly_chart(fig, use_container_width=True)

[Link]("Program Details")
[Link](filtered_df[['Program','Department','Campus','Duration','Fees']])

Output:

85
Sarita Chauhan
Roll no: 06

Conclusion:

The SNDT Programs Dashboard successfully demonstrates how interactive data

visualization can enhance the understanding of academic program information.
By transforming raw program details into a dynamic animated bubble chart, the
system allows users to visually explore relationships between program duration,
fees, campus, and department.
The inclusion of a data-cleaning pipeline ensures that inconsistent duration
formats (years/months) are standardized automatically. Interactive components
such as department filters, adjustable bubble sizes, and real-time animation
provide a user-friendly analytical experience.
Overall, the dashboard integrates effective data processing, visualization, and UI
design to offer a clear, engaging, and insightful overview of SNDT University’s
academic programs. This practical also demonstrates the applicability of Streamlit
and Plotly in creating modern analytical dashboards suitable for educational,
administrative, and decision-making environments.

86
Sarita Chauhan
Roll no: 06

Lab 5: Basic of Grouping, ANOVA Model Evaluation Using Visualization.

Q.1. The times required by three workers to perform an assembly-line task

were recorded on five randomly selected occasions. Here are the times, to the
nearest minute.

Implement the One-way ANOVA table in python and explain the output with
respect to p- value for α = 0.5

Code:
import numpy as np
from scipy import stats

# Productivity data of three workers

worker1 = [Link]([8, 10, 9, 11, 10])
worker2 = [Link]([8, 9, 9, 8, 10])
worker3 = [Link]([10, 9, 10, 11, 9])

# Perform one-way ANOVA

f_statistic, p_value = stats.f_oneway(worker1, worker2, worker3)

# Significance level
alpha = 0.05

# Output results
print("F-statistic:", f_statistic)
print("P-value:", p_value)

# Hypothesis Decision
if p_value < alpha:
print("Reject the null hypothesis: There is a significant difference between the
87
Sarita Chauhan
Roll no: 06

worker groups.")
else:
print("Fail to reject the null hypothesis: There is no significant difference
between the worker groups.")

Output:

Explanation:
ANOVA is used to compare differences of means among more than two groups.
It does this by looking at variation in the data and where that variation is found
(hence its name). Specifically, ANOVA compares the amount of variation
between groups with the amount of variation within groups.

we first define the data for each worker's performance. We then use the f_oneway
function from [Link] to perform the one-way ANOVA test. The F-statistic
and the p-value are calculated.

The null hypothesis (H0) in this test is that there is no significant difference
between the worker groups, and the alternative hypothesis (Ha) is that there is a
significant difference.

The p-value represents the probability that the observed differences in means are
due to random chance. If the p-value is less than the chosen significance level
(alpha), which is 0.05 in this case, you would reject the null hypothesis, indicating
that there is a significant difference between the worker groups.

So, in your case, you would analyze the p-value, and if it's less than 0.05, you
would conclude that there is a significant difference in the performance of the
three workers.

Q.2 .Four brands of flashlight hatteries are to be compared by testing each

brand in five flashlights. Twenty flashlights are randomly selected and
divided randomly into four groups of five flashlights each. Then each group
of flashlights uses a different brand of battery. The lifetimes of the batteries,

88
Sarita Chauhan
Roll no: 06

to the nearest hour, are as follows

Preliminary data analyses indicate that the independent samples come from
normal populations with equal standard deviations. At the 5% significance
level, does there appear to be a difference in m lifetime among the four
brands of hatteries?

Answer hint-Since 0.7393 <3.24 (0.10>0.05), fail to reject the null hypothesis.

State conclusion in words At the a=0.05 level of significance, there is not enough
evidence to conclude that the mean lifetimes of the brands of batteries differ.

Code:
import numpy as np
from scipy import stats

# Data for each brand

brand_D = [Link]([42, 30, 39, 28, 29])
brand_A = [Link]([28, 25, 27, 24, 36])
brand_R = [Link]([28, 36, 31, 32])
brand_C = [Link]([28, 28, 33, 38, 28])

# Perform one-way ANOVA

f_statistic, p_value = stats.f_oneway(brand_D, brand_A, brand_R, brand_C)

# Significance level
alpha = 0.05

# Output results
print("F-statistic:", f_statistic)
print("P-value:", p_value)

# Decision
if p_value < alpha:
89
Sarita Chauhan
Roll no: 06

print("Reject the null hypothesis: There is a significant difference among the

brands.")
else:
print("Fail to reject the null hypothesis: There is no significant difference
among the brands.")

Output:

Explanation:
we first organize the data for each brand of batteries. Then, we use the f_oneway
function from [Link] to perform the one-way ANOVA test. The F-statistic
and the p-value are calculated.

At the 5% significance level (alpha = 0.05), you compare the p-value to the
significance level. If the p-value is less than 0.05, you would reject the null
hypothesis, indicating that there is a significant difference among the brands'
battery lifetimes. If the p-value is greater than or equal to 0.05, you would fail to
reject the null hypothesis, suggesting that there is no significant difference among
the brands.

Q.3. Manufacturers of golf balls always seem to be claiming that their ball
goes the farthest. A writer for a sports magazine decided to conduct an
impartial test. She randomly selected 20 golf professionals and then
randomly assigned four golfers to each of five brands. Each golfer drove the
assigned brand of ball. The driving distances, in yards, are displayed in the
following table.

Preliminary data analyses indicate that the independent samples come from
normal populations with equal standard deviations. Do the data provide
sufficient evidence to conclude that a difference exists in mean weekly
earnings among nonsupervisory workers in the five industries? Perform the
required hypothesis test using a = 0.05. (Note: T-L117, 7- 1128. T-1089. T-
90
Sarita Chauhan
Roll no: 06

1099.

Verify the below statement -:

Since 2.47303.06 (p-value> 0.05), fail to reject the null hypothesis. Step 6:
State conclusion in words At the a=0.05 level of significance, there is not
enough evidence to conclude that the mean driving distances of the brands of
golf balls differ.

Program:
import numpy as np
from scipy import stats

# Data for each brand

brand1 = [Link]([286, 276, 281, 274])
brand2 = [Link]([279, 277, 284, 288])
brand3 = [Link]([270, 262, 277, 280])
brand4 = [Link]([284, 271, 269, 275])
brand5 = [Link]([281, 293, 276, 292])

# Perform one-way ANOVA

f_statistic, p_value = stats.f_oneway(brand1, brand2, brand3, brand4, brand5)

# Significance level (alpha)

alpha = 0.05

# Output the results

print("F-statistic:", f_statistic)
print("P-value:", p_value)

# Decision
if p_value < alpha:
print("Reject the null hypothesis: There is a significant difference in mean
driving distances among the brands.")
else:
print("Fail to reject the null hypothesis: There is no significant difference in
mean driving distances among the brands.")

Output:

91
Sarita Chauhan
Roll no: 06

Explanation:
To analyze the data correctly and test if there is a difference in mean driving
distances among the five brands of golf balls, you can perform a one-way
ANOVA test. Here's the correct
procedure:

[Link] up the hypotheses:

Null Hypothesis (H0): There is no significant difference in the mean driving

distances among the five brands of golf balls.

Alternative Hypothesis (Ha): There is a significant difference in the mean driving

distances among the five brands of golf balls.

[Link] the one-way ANOVA test

[Link] the results:

In the provided output, if the p-value is less than 0.05 (alpha), you would reject
the null hypothesis. If the p-value is greater than or equal to 0.05, you would fail
to reject the null hypothesis.

The conclusion should be about the mean driving distances of the golf ball
brands, not weekly earnings. Please use the correct context and hypothesis testing
for the golf ball problem.

92
Sarita Chauhan
Roll no: 06

Lab 6: Simple Linear Regression, And Multiple Linear Regression, Model

Evaluation Using Visualization

Objectives –
To Understanding regression
To able to differentiate between Simple and Multiple Linear Regression
To implement Simple and Multiple Linear Regression using Python
To work with Regression case study.
To understand the Polynomial Regression
To implement Polynomial Regression in python

Write Answer to the following questions

6.1 What is regression?
Answer:

6.2 Why and when it is used?

93
Sarita Chauhan
Roll no: 06

Ans:

6.3 Explain the difference between Simple and Multiple Linear Regression
Answer:

6.4 Write python code for Simple and Multiple Linear Regression?
Answer:
Code:
# Import necessary libraries
import numpy as np
from sklearn.linear_model import LinearRegression
import [Link] as plt
# Generate random sample data
[Link](0)
X = 2 * [Link](100, 1)
y = 4 + 3 * X + [Link](100, 1)
# Create a Simple Linear Regression model
simple_reg = LinearRegression()
simple_reg.fit(X, y)
# Make predictions
X_new = [Link]([[2.5]]) # New data point to predict
y_pred = simple_reg.predict(X_new)
# Plot the data and regression line
[Link](X, y, label='Data')
[Link](X, simple_reg.predict(X), color='red', label='Regression Line')
[Link](X_new, y_pred, color='green', label='Prediction')
[Link]()
[Link]('Independent Variable (X)')
[Link]('Dependent Variable (y)')
[Link]('Simple Linear Regression')
[Link]()
# Display regression coefficients
print("Simple Linear Regression Coefficients:")
print("Intercept (b0):", simple_reg.intercept_[0])
print("Slope (b1):", simple_reg.coef_[0][0])

Output:

94
Sarita Chauhan
Roll no: 06

Explanation:
In the Simple Linear Regression code:

1. We import the necessary libraries, including NumPy for data generation, scikit-
learn for the regression model, and Matplotlib for visualization.

2. We generate random sample data `X` and `y`, where `X` is the independent
variable, and `y` is the dependent variable with some random noise.

3. We create a Simple Linear Regression model using `LinearRegression()` from

scikit-learn.

4. We fit the model to the data using `simple_reg.fit(X, y)`.

95
Sarita Chauhan
Roll no: 06

5. We make predictions for a new data point, `X_new`, using

`simple_reg.predict(X_new)`.

6. We plot the data points, the regression line, and the predicted data point using
Matplotlib.

7. The code displays the coefficients for the regression line (intercept `b0` and
slope `b1`).

Expected Output for Simple Linear Regression:

The output includes a scatter plot of the data points, the regression line (a straight
line), and a green point representing the predicted value for the new data point.
Additionally, the coefficients for the regression line are displayed:

- Intercept (b0): It represents the y-intercept of the regression line.

- Slope (b1): It indicates the slope of the regression line.

The output visually shows the linear relationship between the independent and
dependent variables.

Multiple Linear Regression:

Code :
# Import necessary libraries
import numpy as np
from sklearn.linear_model import LinearRegression
import [Link] as plt

# Generate random sample data

[Link](0)
X = 2 * [Link](100, 2)
y = 4 + 3 * X[:, 0] + 2 * X[:, 1] + [Link](100)

# Create a Multiple Linear Regression model

multi_reg = LinearRegression()
multi_reg.fit(X, y)

# Make predictions for a new data point

X_new = [Link]([[2.5, 1.5]]) # New data point
96
Sarita Chauhan
Roll no: 06

y_pred = multi_reg.predict(X_new)

# Display regression coefficients

print("Multiple Linear Regression Coefficients:")
print("Intercept (b0):", multi_reg.intercept_)
print("Slopes (b1, b2):", multi_reg.coef_)
print("Predicted value for new data point:", y_pred[0])

# Plot the data and regression plane

fig = [Link]()
ax = fig.add_subplot(111, projection='3d')

[Link](X[:, 0], X[:, 1], y, label='Data')

ax.set_xlabel('Independent Variable 1 (X1)')
ax.set_ylabel('Independent Variable 2 (X2)')
ax.set_zlabel('Dependent Variable (y)')
ax.set_title('Multiple Linear Regression')
ax.view_init(elev=20, azim=45)

# Create a meshgrid for the prediction surface

x1, x2 = [Link]([Link](0, 2, 20), [Link](0, 2, 20))
x1_flat = [Link]()
x2_flat = [Link]()

X_grid = np.column_stack((x1_flat, x2_flat))

y_pred_surface = multi_reg.predict(X_grid)
y_pred_surface = y_pred_surface.reshape([Link])

# Plot the regression surface

ax.plot_surface(x1, x2, y_pred_surface, alpha=0.6, cmap='viridis')

# Plot the new prediction point

[Link](X_new[0, 0], X_new[0, 1], y_pred, color='green', s=100,
label='Prediction')

# Add legend properly

[Link]()

[Link]()

Output:

97
Sarita Chauhan
Roll no: 06

Explanation:
In the Multiple Linear Regression code:

98
Sarita Chauhan
Roll no: 06

1. We import the necessary libraries, generate random sample data with two
independent variables (`X`) and a dependent variable (`y`).

2. We create a Multiple Linear Regression model using `LinearRegression()`.

3. We fit the model to the data using `multi_reg.fit(X, y)`.

4. We make predictions for a new data point, `X_new`, using

`multi_reg.predict(X_new)`.

5. The code displays the coefficients for the regression model, including the
intercept and slopes for each independent variable.

6. We create a 3D scatter plot of the data points, including both independent

variables and the dependent variable. We also visualize the regression surface as a
plane and the predicted data point.

Expected Output for Multiple Linear Regression:

The output includes a 3D scatter plot of the data points, a plane representing the
regression surface, and a green point indicating the predicted value for the new
data point. The coefficients for the regression model are displayed.

- Intercept (b0): It represents the y-intercept of the regression plane.

- Slope for X1 (b1): It indicates the impact of the first independent variable.

- Slope for X2 (b2): It indicates the impact of the second independent variable.

The output visually illustrates the relationship between multiple independent

variables and the dependent variable in a three-dimensional space.

6.5 Run your python code for the case study given below. Explain the output.

99
Sarita Chauhan
Roll no: 06

Code:
import numpy as np
import [Link] as stats

# Provided data
temperature = [Link]([10, 20, 30, 40, 50, 60, 70, 80, 90])
life_time = [Link]([420, 365, 285, 220, 176, 117, 69, 34, 5])

# Perform linear regression

slope, intercept, r_value, p_value, std_err = [Link](temperature,
life_time)

# Number of observations
n = len(temperature)

# Correct formula for SE of slope

SE_slope = std_err / [Link]([Link]((temperature - [Link](temperature)) ** 2))

# Degrees of freedom
df = n - 2

# Critical t-value for 95% CI

t_critical = [Link](0.975, df)

# Margin of error
100
Sarita Chauhan
Roll no: 06

MOE = t_critical * SE_slope

# 95% Confidence Interval

CI_lower = slope - MOE
CI_upper = slope + MOE
confidence_interval = (CI_lower, CI_upper)

# Print results
print("Slope (β1):", slope)
print("Standard Error of the Slope (SE(β1)):", SE_slope)
print("Degrees of Freedom (df):", df)
print("Critical t-value:", t_critical)
print("Margin of Error (MOE):", MOE)
print("95% Confidence Interval for the Slope:", confidence_interval)

Output:

Explanation:

1. Import Necessary Libraries:

- We start by importing the required libraries: `numpy` for numerical operations

and `[Link]` for statistical functions.

2. Provided Data:

- We define the provided data in two NumPy arrays: `temperature` and

`life_time`. These arrays represent the temperature in Celsius and the
corresponding life time in hours.

3. Perform Linear Regression:

- We use the `[Link]` function from `[Link]` to perform a linear

regression analysis on the data. This function returns several statistics, including
101
Sarita Chauhan
Roll no: 06

the slope, intercept, correlation coefficient, p-value, and standard error of the
regression.

4. Calculate Standard Error of the Slope (SE(β1)):

- The standard error of the slope (SE(β1)) is calculated manually using the
formula:

- SE(β1) = std_err / sqrt(Σ(ti - mean(t))2)

- `std_err` is obtained from the linear regression analysis, and `mean(t)`

represents the mean of the temperature values.

5. Determine Degrees of Freedom (df):

- The degrees of freedom (df) are calculated as the number of data points minus 2
(df = n - 2). In this case, `n` represents the number of data points.

6. Find Critical t-value:

- We use the `[Link]` function to find the critical t-value for a 95% confidence
interval. The `0.975` argument corresponds to the desired confidence level (95%),
and `df` represents the degrees of freedom.

7. Calculate Margin of Error (MOE):

- The margin of error (MOE) is calculated by multiplying the critical t-value by

the standard error of the slope:

- MOE = t_critical * SE(β1)

8. Calculate Confidence Interval for the Slope:

- The confidence interval for the slope is calculated by adding and subtracting the
margin of error from the estimated slope:

- Confidence Interval = (slope - MOE, slope + MOE)

9. Print Results:

- The code prints out various results, including the estimated slope (β1), standard
error of the slope (SE(β1)), degrees of freedom (df), critical t-value, margin of
102
Sarita Chauhan
Roll no: 06

error (MOE), and the 95% confidence interval for the slope.

This code allows you to analyze the linear relationship between temperature and
life time, providing a confidence interval for the slope, which indicates the effect
of temperature on life time.

6.6 Run your python code for the case study given below. Explain the output.

Code:
import numpy as np
from [Link] import linregress
# Provided data
temperature = [Link]([0, 25, 50, 75, 100])
yield_grams = [Link]([14, 38, 54, 76, 95])
# Perform linear regression
slope, intercept, r_value, p_value, std_err = linregress(temperature, yield_grams)
# Equation of the linear relationship
equation = f"y = {intercept:.2f} + {slope:.2f} * x"
# Print results
print("Linear Regression Results:")
print("Slope (β1):", slope)
print("Intercept (β0):", intercept)
print("Equation of the Linear Relationship:", equation)
103
Sarita Chauhan
Roll no: 06

Ouput:

Explanation:

1. Import the necessary libraries:

- We import NumPy for numerical operations and `linregress` from `[Link]`

for performing linear regression.

2. Define the provided data:

- We create NumPy arrays `temperature` and `yield_grams` to store the provided

temperature and yield values.

3. Perform linear regression:

- We use the `linregress` function to perform a simple linear regression analysis

on the data.

This function returns several statistics, including the slope (β1), intercept (β0),
correlation coefficient (r_value), p-value (p_value), and standard error of the
regression (std_err).

4. Calculate the equation of the linear relationship:

- We create a string variable `equation` to represent the linear relationship

equation. This equation is constructed using the estimated intercept and slope.

5. Print the results:

- The code prints out the results, including the estimated slope (β1), intercept
(β0), and the equation of the linear relationship between temperature and yield.

The output of the code will include the slope, intercept, and the equation of the
linear relationship that describes how temperature (x) influences the yield (y) in
this chemical process.

104
Sarita Chauhan
Roll no: 06

POLYNOMIAL REGRESSION

model is a machine learning model that can capture non-linear relationships

between variables by fitting a non-linear regression line, which may not be
possible with simple linear regression. It is used when linear regression models
may not adequately capture the complexity of the relationship.

Linear regression is having x to the power 1

Nonlinear equation means when the variable x is having power more than 1.

[Link]
polynomial-regression/

explain well about polynomial regression.

For sloved example in python Use the above and below link.

[Link]

Q) Explain well about polynomial regression.

Polynomial regression is a type of regression analysis used to model the

relationship between the independent variable(s) and the dependent variable in a
nonlinear manner. Unlike simple linear regression, which models the relationship
as a straight line, polynomial regression uses a polynomial equation to fit a curve
to the data. This allows for more complex and flexible modeling of data with
nonlinear patterns.

Key characteristics and concepts of polynomial regression:

1. Polynomial Equation: The core idea of polynomial regression is to use a

polynomial equation of the form:

105
Sarita Chauhan
Roll no: 06

Here, \( y \) represents the dependent variable, \( x \) is the independent variable,

\( β0 \) is the intercept, \( β1, β2, β3, ... \) are the coefficients of the polynomial
terms, \( n \) is the degree of the polynomial, and \( ε \) represents the error term.

2. Degree of the Polynomial: The degree (\( n \)) of the polynomial represents
the highest power of \( x \) in the equation. The degree determines the complexity
of the curve that can be fitted to the data. A higher degree allows for more
flexibility in modeling complex relationships but can also lead to overfitting if not
chosen carefully.

3. Overfitting vs. Underfitting: One of the key challenges in polynomial

regression is finding the right balance between underfitting and overfitting.
Overfitting occurs when the model is too complex (high degree) and fits the noise
in the data, leading to poor generalization. Underfitting occurs when the model is
too simple (low degree) and cannot capture the underlying patterns in the data.
The goal is to choose an appropriate degree that fits the data well without
overcomplicating the model.

4. Data Transformation: Polynomial regression can be used to capture nonlinear

relationships in data. It's particularly useful when there's a curvilinear or
polynomial pattern in the data. However, it's important to ensure that the data is
transformed appropriately, especially if the relationship is not inherently
polynomial. This may involve transforming the independent variable(s) or the
dependent variable to create a more linear relationship.

5. Model Evaluation: Common evaluation metrics for polynomial regression

include Mean Squared Error (MSE), R-squared (R2), and others. These metrics
help assess the goodness of fit and the predictive performance of the polynomial
model.

6. Visualization: Visualization is essential for understanding the relationship

between the variables. Scatterplots of the data and the fitted polynomial curve can
provide insights into the data patterns and the quality of the model.

7. Applications: Polynomial regression is used in various fields, including

physics, engineering, economics, and social sciences. It's particularly valuable
when modeling phenomena with complex, nonlinear behavior.

Polynomial regression is a flexible and powerful technique for modeling

relationships in data that exhibit nonlinear patterns. By choosing an appropriate
106
Sarita Chauhan
Roll no: 06

degree and carefully evaluating the model, you can capture complex relationships
and make accurate predictions in a wide range of applications.

Lab 7: Prediction and Decision Making

Q.1) What is the need of evaluating the model?

Ans:
Evaluating a machine learning model is a critical step in the model development
process
for several important reasons:
1. Performance Assessment: Evaluating a model helps you understand how well
it is performing on a specific task. It provides a quantifiable measure of the
model's quality and effectiveness in making predictions or classifications.

2. Model Selection: When working with different algorithms or variations of a

model, evaluation allows you to compare their performance and select the best
one for a particular problem. This is crucial for choosing the most suitable model
or approach for a given task.

3. Hyperparameter Tuning: Model evaluation guides the process of

hyperparameter tuning. By assessing the model's performance under different
settings, you can identify the best hyperparameter values that optimize the
107
Sarita Chauhan
Roll no: 06

model's performance.

4. Bias and Variance Analysis: Evaluation helps you analyze the trade-off
between bias (underfitting) and variance (overfitting). It allows you to find the
right balance to ensure that the model generalizes well to new, unseen data.

5. Generalization Assessment: The primary goal of a machine learning model is

to make accurate predictions on new, unseen data. Evaluation measures how well
a model generalizes beyond the training data. It helps you verify that the model is
not just memorizing the training data (overfitting) but is capturing meaningful
patterns.

6. Quality Control: Evaluation helps identify potential issues with the model,
such as underperformance, overfitting, or instability. It allows you to catch and
address these issues before deploying the model in a real-world application.

7. Decision Support: Machine learning models are often used to support

decision-making. Evaluation provides the necessary information to make
informed decisions based on the model's predictions. For example, it can help in
medical diagnoses, financial risk assessments, or recommendation systems.

8. Continuous Improvement: Model evaluation is an ongoing process.

Regularly assessing a model's performance over time allows you to monitor its
effectiveness and make adjustments as necessary. This is particularly important as
data distributions or the nature of the problem can change over time.

9. Interpretability and Transparency: Evaluation helps in understanding how a

model makes predictions. It allows you to assess the model's transparency and the
impact of different features on its predictions, which is important for explainable
AI.

10. Resource Allocation: Evaluating a model's performance can help allocate

resources effectively. For example, you can decide whether further data
collection, feature engineering, or model refinement is necessary based on the
model's performance.

In summary, model evaluation is a fundamental step in the machine learning

workflow. It ensures that models meet the desired performance criteria, supports
informed decision- making, and guides the continuous improvement and
refinement of machine learning systems.

108
Sarita Chauhan
Roll no: 06

Q.2) Explain R-squared and MSE?

Ans:
R-squared (R2) and Mean Squared Error (MSE) are common metrics used in
regression analysis to evaluate the performance of predictive models. They
provide different
perspectives on the quality of the model's predictions:

Mean Squared Error (MSE):

- MSE is a measure of the average squared difference between the predicted
values and the actual values in a regression model.

- It quantifies how far off the model's predictions are from the true values, with
larger errors contributing more to the score due to squaring.

- Mathematically, MSE is calculated as the average of the squared differences

between the actual values (y) and the predicted values (ŷ):

The value of MSE is always positive. A value close to zero will represent better
quality of the estimator/predictor (regression model).

- MSE is always a non-negative value, and a lower MSE indicates a better fit of
the model to the data. In other words, a lower MSE means that the model's
predictions are closer to the actual values.

Use Cases: MSE is widely used in regression tasks for assessing prediction
accuracy. It provides a measure of how well the model's predictions match the
actual data.

R-squared (R2):
- It is also known as “Coefficient of determination”.

- It is a statistical metric used to evaluate the goodness of fit of a regression

model.
109
Sarita Chauhan
Roll no: 06

- R-squared is a statistical measure that represents the proportion of the variance

in the dependent variable (the target) that is explained by the independent
variables (features) in a regression model.

- R-squared ranges from 0 to 1, where 0 means that the model does not explain
any variance, and 1 means that the model perfectly explains all the variance in the
target variable.

- Mathematically, R-squared is calculated as:

R2 = 1 - (SSR / SST), where SSR is the sum of squared residuals (the squared
differences between the actual values and the predicted values), and SST is the
total sum of squares (the squared differences between the actual values and the
mean of the actual values).

- R-squared can be interpreted as the proportion of the total variation in the data
that is captured by the model. A higher R-squared value indicates a better fit of
the model to the data.

Here's a simple way to understand the difference between MSE and R-squared:

- MSE focuses on the accuracy of individual predictions, measuring how close or

far the model's predictions are from the true values.

- R-squared, on the other hand, focuses on the goodness of fit and explains how
much of the variation in the target variable can be attributed to the model's
features. It provides insight into how well the model as a whole explains the data.

In practice, both metrics are valuable when evaluating regression models. A

model with a low MSE is making accurate predictions, and a model with a high
R-squared value is explaining a significant portion of the variance in the target
variable. However, it's essential to consider both metrics together and in the
context of the specific problem to gain a comprehensive understanding of the
model's performance.
110
Sarita Chauhan
Roll no: 06

Q.3) Which one to use when?

Ans:

The choice between using Mean Squared Error (MSE) or R-squared (R2) depends
on the specific goals and context of your regression analysis. Here are some
guidelines on when to use each of these metrics:

Use MSE when:

1. Prediction Accuracy Matters: MSE is a suitable choice when your primary

concern is the accuracy of individual predictions. It quantifies how close or far
the model's predictions are from the actual values. Lower MSE values indicate
better prediction accuracy.

2. Balancing Errors: MSE is particularly useful when you want to weigh

prediction errors evenly. It treats both overestimations and underestimations
symmetrically because it squares the errors. This can be important when
overestimation and underestimation have similar costs or implications.

3. No Assumptions about Linearity: MSE is applicable to a wide range of

regression models, including those that don't necessarily assume a linear
relationship between the features and the target variable.

Use R-squared (R2) when:

1. Model Interpretability: R2 provides insights into how well the model as a

whole explains the variance in the target variable. It is useful when you want to
understand how much of the variation in the target variable is captured by the
independent variables.

2. Comparing Models: R2 is helpful for model selection and comparison. When

you have multiple regression models, you can use R2 to assess which model
provides a better fit to the data and explains more of the variance.

3. Goodness of Fit: R2 is commonly used to assess the goodness of fit in linear

regression models, particularly when you want to evaluate the overall
performance of the model in explaining the observed variability.

4. Communicating Results: R2 is easier to communicate and interpret for non-

technical audiences. It's often presented as a percentage, representing the
111
Sarita Chauhan
Roll no: 06

proportion of variance explained. Higher R2 values indicate better explanatory

power.

In summary, the choice between MSE and R2 depends on whether you prioritize
prediction accuracy (MSE) or the model's ability to explain the variance in the
target variable (R2). In practice, it's often a good idea to use both metrics to gain a
comprehensive understanding of your regression model's performance. However,
the relative importance of each metric should be determined by the specific
objectives and requirements of your analysis or the domain in which you are
working.

Q.4) What is Mean Squared Error (MSE)?

Ans:

Mean Squared Error (MSE) is a commonly used metric in statistics and machine
learning to measure the average squared difference between the predicted values
and the actual values in a dataset. It is often used in regression analysis to
evaluate the accuracy and quality of predictive models. MSE quantifies how well
a model's predictions align with the true values, with smaller MSE values
indicating more accurate predictions.

The formula for calculating MSE is as follows:

Key characteristics of MSE:

1. Non-Negative Value: MSE is always a non-negative value. Since it involves

squaring the differences, all individual errors contribute positively to the score.

2. Sensitivity to Outliers: MSE is sensitive to outliers or extreme values in the

data because it squares the differences. Outliers can have a substantial impact on
the overall score.

3. Units of Measurement: The units of MSE are the square of the units of the
112
Sarita Chauhan
Roll no: 06

target variable. This can make interpretation less intuitive in some cases.

4. Lower MSE is Better: A lower MSE indicates a better fit of the model to the
data, with smaller prediction errors. In other words, a lower MSE means that the
model's predictions are closer to the actual values.

MSE is a versatile metric used in various regression tasks, such as linear

regression, polynomial regression, and machine learning models that predict
continuous numerical values. It is often employed in model evaluation, model
selection, and hyperparameter tuning to assess and compare the performance of
different models or parameter settings. However, it's essential to consider the
context and specific goals of your analysis when interpreting
MSE, as it may not be the only metric used for evaluation.

Q.5) What is R-Squared?

Ans:

R-squared (R2), also known as the coefficient of determination, is a statistical

measure used in regression analysis to assess the goodness of fit of a regression
model to the observed data. R2 quantifies the proportion of the variance in the
dependent variable (the target) that is explained by the independent variables
(features) in the model. In other words, it provides information about how well
the model accounts for the variability in the data.

R-squared values range between 0 and 1, with the following interpretations:

- R2 = 0: The model does not explain any of the variability in the dependent
variable. It is equivalent to a model that always predicts the mean of the target
variable.

- R2 = 1: The model explains all of the variability in the dependent variable. It

perfectly predicts the observed values.

Mathematically, R2 is calculated as:

R2 = 1 - (SSR / SST)

113
Sarita Chauhan
Roll no: 06

Where:
- R2 is the coefficient of determination.

- SSR is the sum of squared residuals, which represents the squared differences
between the actual values and the predicted values from the model.

- SST is the total sum of squares, which represents the squared differences
between the actual values and the mean of the actual values.

Key points about R-squared:

1. Interpretation: R-squared measures the proportion of variance in the target
variable that is explained by the independent variables. For example, if R2 is
0.75, it means that 75% of the variance in the dependent variable is accounted for
by the model.

2. Values between 0 and 1: R-squared values fall between 0 and 1, indicating the
fraction of variance explained. Higher R2 values suggest a better fit of the model
to the data.

3. Limitations: While R-squared is a valuable measure for assessing the

goodness of fit, it has limitations. It doesn't provide information about the
appropriateness of the model's functional form or the relevance of specific
predictors. It also does not distinguish between a meaningful model and a model
that captures noise.

4. Comparative Use: R-squared is often used for model selection and

comparison. When comparing multiple regression models, a higher R-squared
indicates a better fit to the data.

5. R2 Adjusted: In multiple regression analysis, a related metric called the

adjusted R-squared is often used. Adjusted R-squared accounts for the number of
predictors in the model and can be more informative in cases with many
independent variables.

In summary, R-squared is a valuable tool for evaluating the goodness of fit of

regression models, especially linear regression. It provides a simple and intuitive
measure of how well the model explains the variability in the data, making it a
widely used metric in regression analysis.

114
Sarita Chauhan
Roll no: 06

Q.6) Implement the MSE or R-Squared Python Code on the IRIS data set.
Code:

115
Sarita Chauhan
Roll no: 06

Output :

Explanation:
1. Import Libraries:

- We start by importing necessary libraries, including scikit-learn for dataset

handling, model creation, and evaluation metrics.

2. Load the Iris Dataset:

- We load the Iris dataset using scikit-learn's `datasets.load_iris()` function. This

dataset contains features (sepal length and sepal width) and target labels (species).

3. Data Splitting:

- The data is divided into training and testing sets using `train_test_split`. We
allocate 20% of the data for testing while keeping 80% for training. The
`random_state` parameter ensures reproducibility.

4. Model Creation:

116
Sarita Chauhan
Roll no: 06

- We create a linear regression model using `LinearRegression()` from scikit-

learn. This model will attempt to predict species based on sepal length and width.

5. Model Fitting:

- The model is trained (fitted) on the training data using `[Link](X_train,

y_train)`.

6. Prediction:

- The trained model makes predictions on the test data using

`[Link](X_test)`.

7. MSE Calculation:

- We calculate the Mean Squared Error (MSE) by comparing the model's

predictions (`y_pred`) with the true target values (`y_test`) using
`mean_squared_error`. MSE quantifies the average squared difference between
predicted and actual values.

8. R-squared (R2) Calculation:

- We calculate the R-squared (R2) score to assess how well the model explains
the variance in the target variable. R2 is computed with `r2_score` by comparing
predicted values to actual values.

9. Model Evaluation Interpretation:

- We provide an interpretation of the MSE and R2 values based on the results:

- If R2 is 1, it means the model perfectly explains the variance in the target

variable.

- If R2 is greater than 0, it explains a certain percentage of the variance, with

higher values indicating a better fit.

- If R2 is 0 or negative, the model does not explain the variance well.

- A lower MSE indicates that the model's predictions are closer to the actual
values, suggesting better accuracy.

- If MSE is 0, it means the model's predictions are perfectly accurate.

117
Sarita Chauhan
Roll no: 06

Output Interpretation:

The code calculates the MSE and R2 for the linear regression model applied to
the Iris dataset. The interpretation is provided based on the calculated values. The
specific MSE and R2 values will depend on the random splitting of the data, but
the interpretation will remain consistent.

This output indicates that the model has an R2 of approximately 0.75, which
means it explains about 80.69% of the variance in the target variable, and the
MSE is around 0.16, indicating reasonably accurate predictions.

118
Sarita Chauhan
Roll no: 06

Lab 8: Model Evaluation, Over-fitting, Under- fitting and Model.

Q.1) What is a Good Fit in Machine Learning?

Ans:
In machine learning, the concept of a "good fit" refers to how well a model's
predictions match the actual values in the dataset it was trained on or on new,
unseen data. A good fit is an indication that the model has learned the underlying
patterns in the data and can make accurate predictions. Achieving a good fit is a
key goal in machine learning, as it reflects the model's ability to generalize from
the training data to make reliable predictions on new, unseen data.

Here are some indicators of a good fit in machine learning:

 Low Training Error: A model with low training error means that it can
accurately
predict the target variable on the data it was trained on. However, low training
error

alone is not sufficient; the model should also perform well on new data.
 Low Validation Error: Validation error, also known as cross-validation error,
measures how well the model generalizes to new, unseen data. A good fit is
indicated by low validation error, which suggests that the model is not overfitting
the training data.
 High Test Accuracy: Testing the model on a separate test dataset (different
from
the training and validation data) and achieving high accuracy or low error rates is
a
sign of a good fit.

 Generalization: A model that generalizes well means it can make accurate

119
Sarita Chauhan
Roll no: 06

predictions on data it has never seen before. It's important to avoid overfitting,
where the model memorizes the training data but fails to generalize to new data.
Consistency: A good fit is consistent with the underlying patterns in the data. If
the
model's predictions are consistent with domain knowledge and expectations, it's a
positive sign.
 Low Bias and Low Variance: A good fit typically strikes a balance between
low
bias (the model is not too simplistic) and low variance (the model is not overly
complex). Bias refers to the model's error on the training data, while variance
refers
to its sensitivity to small changes in the training data.
 Proper Evaluation Metrics: The choice of evaluation metrics should align with
the
problem you're trying to solve. For example, if you're working on a classification
problem, accuracy, precision, recall, F1-score, or ROC AUC may be relevant
metrics to consider.
Ultimately, a good fit is a balance between underfitting (oversimplified model)
and
overfitting (overly complex model) and should be tailored to the specific needs of
the
problem at hand.

Q.2) How to Detect Overfitting or Underfitting

Ans:
Detecting overfitting and underfitting in a machine learning model is essential for
ensuring
that the model generalizes well to new, unseen data. Here are some common
methods to
detect overfitting or underfitting:

120
Sarita Chauhan
Roll no: 06

1. Visual Inspection of Learning Curves:

 Training and Validation Curves: Plot the learning curves for both the training
and
validation datasets. The training error should decrease as the model learns, while
the
validation error should also decrease initially but then stabilize or increase
slightly. If
the training error is significantly lower than the validation error, it's a sign of
overfitting. If both errors are high, it's a sign of underfitting.
 Bias-Variance Trade-off: Observe the gap between the training and validation
error
curves. A large gap indicates overfitting, while a small gap suggests a good fit.
2. Model Complexity:

 Simpler Models: If your model is too simple, it may underfit the data. In this
case,
consider using a more complex model, adding more features, or tuning
hyperparameters.
 Complex Models: If your model is overly complex and captures noise in the
training
data, it's likely overfitting. You may want to simplify the model, reduce the
number of
features, or apply regularization techniques.
3. Cross-Validation: Use techniques like k-fold cross-validation to assess the
model's
performance on different subsets of the data. If the model consistently performs
well
on all folds, it's a good sign. If there's significant variability in performance
between
folds, it could indicate overfitting.
4. Regularization: Apply regularization techniques like L1 (Lasso) or L2 (Ridge)
regularization to penalize the model for having overly complex weights.
Regularization can help mitigate overfitting.
5. Feature Selection: Remove irrelevant or redundant features from your dataset.
Having
too many features can lead to overfitting, especially when the number of features
is
much greater than the number of data points.
6. Hyperparameter Tuning: Experiment with hyperparameters such as learning
rate, the
number of layers or units in a neural network, or the depth of a decision tree.
121
Sarita Chauhan
Roll no: 06

Hyperparameter tuning can help find a balance between underfitting and

overfitting.
7. Validation and Test Data Performance:
 After training your model, assess its performance on a separate validation
dataset. If
the model performs well on the training set but poorly on the validation set, it's
overfitting.
 Evaluate the model on a held-out test dataset to ensure that it can generalize to
new,
unseen data. If the test performance is significantly worse than the validation
performance, it's a strong sign of overfitting.
8. Early Stopping: Implement early stopping during training. This involves
monitoring
the validation error and stopping training when it starts to increase. This prevents
the
model from continuing to learn the noise in the data.
9. Ensemble Methods: Consider using ensemble methods like bagging (e.g.,
random
forests) or boosting (e.g., AdaBoost) to combine multiple models. Ensembles can
reduce the risk of overfitting by aggregating the predictions of multiple models.
10. Domain Knowledge: Use domain knowledge to guide your model selection
and
feature engineering. Understanding the problem you're solving can help you avoid
overfitting by eliminating irrelevant or misleading features.

Detecting overfitting and underfitting is often an iterative process that may

require fine-
tuning the model and its hyperparameters until a satisfactory balance is achieved
between

bias and variance.

Q.3) How to Prevent Overfitting and Underfitting in Models.

Ans:
Preventing overfitting and underfitting in machine learning models is crucial to
ensure that
the model generalizes well to new, unseen data. Here are some strategies to help
you
prevent overfitting and underfitting:
1. Collect More Data:
- One of the most effective ways to prevent overfitting is to have a larger and
more
122
Sarita Chauhan
Roll no: 06

diverse dataset. More data can help the model learn the underlying patterns
without
memorizing noise.
2. Cross-Validation:
- Use techniques like k-fold cross-validation to assess the model's performance on
different subsets of the data. Cross-validation helps you understand how well the
model
generalizes to new data.
3. Proper Data Preprocessing:
- Clean and preprocess your data carefully. Remove outliers, handle missing
values, and
scale or normalize features as needed. Proper data preprocessing can make your
model
more robust to overfitting.
4. Feature Selection:
- Identify and select relevant features while discarding irrelevant or redundant
ones.
Feature selection can simplify the model and reduce the risk of overfitting.
5. Model Complexity:
- Adjust the complexity of your model based on the data and problem. If your
model is
too simple (underfitting), consider using a more complex model or adding more
features.
If it's too complex (overfitting), simplify it, reduce the number of features, or use
regularization techniques.
6. Regularization:
- Apply L1 (Lasso) or L2 (Ridge) regularization to penalize large weights and
prevent
overfitting. These regularization techniques encourage the model to have smaller,
more
balanced weights.
7. Dropout (Neural Networks):
- If you're working with neural networks, consider using dropout layers. Dropout
randomly deactivates a fraction of neurons during training, which helps prevent
overfitting.

8. Early Stopping:
- Implement early stopping during training. Monitor the validation error, and stop
training when it starts to increase or plateaus. This prevents the model from
overfitting the
training data.
9. Cross-Validation with Grid Search:
123
Sarita Chauhan
Roll no: 06

- Use cross-validation in combination with hyperparameter optimization

techniques like
grid search. This helps you find the right set of hyperparameters that balance
model
complexity and performance.
10. Ensemble Methods:
- Combine multiple models using ensemble techniques such as bagging (e.g.,
random
forests) or boosting (e.g., AdaBoost). Ensembles can reduce overfitting by
aggregating the
predictions of multiple models.
11. Bias-Variance Trade-off:
- Understand the bias-variance trade-off. Striking the right balance between bias
(underfitting) and variance (overfitting) is essential. It often involves adjusting the
model
complexity and tuning hyperparameters.
12. Domain Knowledge:
- Leverage domain knowledge to guide your model selection and feature
engineering. A
deep understanding of the problem can help you make informed decisions and
avoid
overfitting.
13. Evaluation Metrics:
- Choose appropriate evaluation metrics that align with your problem. For
example, use
precision, recall, or F1-score for classification tasks and mean squared error or R-
squared
for regression tasks.
14. Test on Unseen Data:
- Evaluate your model on a held-out test dataset to ensure that it generalizes well
to new,
unseen data. This is a crucial step to confirm that overfitting is minimized.

Q.4) Explain Model Fit: Underfitting vs Overfitting with one example

Ans:
Model fit, in the context of machine learning, refers to how well a predictive
model
matches the underlying patterns in the data. Model fit can be categorized into two
extremes: underfitting and overfitting. Let's explain these concepts with an
example.
124
Sarita Chauhan
Roll no: 06

Suppose you are building a model to predict a student's exam performance based
on the
number of hours they study. You collect data from 50 students and create a scatter
plot to
visualize the relationship between study hours and exam scores.
1. Underfitting:

- Underfitting occurs when a model is too simple to capture the underlying

patterns in
the data. In this case, if you fit a linear regression model to the data, it might
result in
underfitting.
- Imagine you fit a straight line (a linear model) to the data points, but the line
doesn't fit
the data well. It has a low training error but high test error. The linear model is
too
simplistic to capture the true relationship between study hours and exam scores.
- As a result, the model's predictions are inaccurate, as it fails to capture the
nuances in
the data, such as the possibility that exam scores may improve more steeply with
additional study hours for some students.
- This underfit model might look like a flat line, indicating a poor fit to the data.
2. Overfitting:

125
Sarita Chauhan
Roll no: 06

- Overfitting occurs when a model is overly complex and fits not only the
underlying
patterns in the data but also the noise or random fluctuations. For this example,
let's say
you fit a high-degree polynomial regression model to the data.
- The high-degree polynomial fits the training data extremely well, passing
through
every data point. However, it exhibits significant oscillations and variations,
capturing
even the noise in the data.
- While this overfit model achieves very low training error, it has high test error.
When
you use it to predict the exam scores of new students, it may make wildly
inaccurate
predictions because it's modeling the noise in the training data.
- The overfit model might result in a curve that zigzags through the data points,
showing
a poor fit to the underlying trend.
Underfitting is characterized by a model that is too simple and fails to capture the
data's
underlying patterns, leading to poor predictive performance. Overfitting is
characterized
by a model that is too complex and fits the training data too closely, capturing
noise and
leading to poor generalization to new data. The goal in machine learning is to find
the
right balance between these two extremes to achieve a good model fit that
126
Sarita Chauhan
Roll no: 06

generalizes well
to unseen data.

127