Data Science Lab

CIE-405P

Faculty name: Prof. M. L. Sharma Student name : Chirag Yadav

Roll No. : 21614803121
Semester :7
Group : IT (FSD-4-A)

Maharaja Agrasen Institute of Technology, PSP Area, Sector –

22, Rohini, New Delhi – 110085
 MAHARAJA AGRASEN INSTITUTE OF TECHNOLOGY

VISION OF THE INSTITUTE

To attain global excellence through education, innovation, research, and work ethics with
the commitment to serve humanity.

MISSION OF THE INSTITUTE

M1. To promote diversification by adopting advancement in science, technology,

management, and allied discipline through continuous learning
M2. To foster moral values in students and equip them for developing sustainable
solutions to serve both national and global needs in society and industry.
M3. To digitize educational resources and process for enhanced teaching and effective
learning.
M4. To cultivate an environment supporting incubation, product development,
technology transfer, capacity building and entrepreneurship.
M5. To encourage faculty-student networking with alumni, industry, institutions,
and other stakeholders for collective engagement.
1 MAHARAJA AGRASEN INSTITUTE OF TECHNOLOGY

DEPARTMENT OF INFORMATION TECHNOLOGY

VISION OF THE DEPARTMENT

To establish a centre of excellence promoting Information Technology related education and

research for preparing technocrats and entrepreneurs with ethical values.

MISSION OF THE DEPARTMENT

M1. To excel in the field by imparting quality education and skills for software
development and applications.
M2. To establish a conducive environment that promotes intellectual growth and
research.
M3. To facilitate students to acquire entrepreneurial skills for innovation and product
development.
M4. To encourage students to participate in competitive events and industry
interaction with a focus on continuous learning.
CO’s and its mapping with PO’s and PSO’s

Program Specific Outcomes (PSOs)

PSO1: Proficient at discerning suitable data structures and algorithms for designing, executing, and
validating efficient solutions for research-oriented challenges.
PSO2: Attaining excellence in programming, technical projects and handling technical challenges.
PSO3: Acquiring adequate knowledge in Information Technology for a successful professional life,
entrepreneurship and higher education.

Program Outcomes (POs) Engineering Graduates will be able to:

1. Engineering knowledge: Apply the knowledge of mathematics, science, engineering fundamentals,
and an engineering specialization to the solution of complex engineering problems.
2. Problem analysis: Identify, formulate, review research literature, and analyze complex engineering
problems reaching substantiated conclusions using first principles of mathematics, natural sciences, and
engineering sciences.
3. Design/development of solutions: Design solutions for complex engineering problems and design
system components or processes that meet the specified needs with appropriate consideration for the
public health and safety, and the cultural, societal, and environmental considerations.
4. Conduct investigations of complex problems: Use research-based knowledge and research methods
including design of experiments, analysis and interpretation of data, and synthesis of the information to
provide valid conclusions.
5. Modern tool usage: Create, select, and apply appropriate techniques, resources, and modern
engineering and IT tools including prediction and modelling to complex engineering activities with an
understanding of the limitations.
6. The engineer and society: Apply reasoning informed by the contextual knowledge to assess societal,
health, safety, legal and cultural issues and the consequent responsibilities relevant to the professional
engineering practice.
7. Environment and sustainability: Understand the impact of the professional engineering solutions in
societal and environmental contexts, and demonstrate the knowledge of, and need for sustainable
development.
8. Ethics: Apply ethical principles and commit to professional ethics and responsibilities and norms of
the engineering practice.
9. Individual and team work: Function effectively as an individual, and as a member or leader in diverse
teams, and in multidisciplinary settings.
10. Communication: Communicate effectively on complex engineering activities with the engineering
community and with society at large, such as, being able to comprehend and write effective reports and
design documentation, make effective presentations, and give and receive clear instructions.
11. Project management and finance: Demonstrate knowledge and understanding of the engineering
and management principles and apply these to one’s own work, as a member and leader in a team, to
manage projects and in multidisciplinary environments.
12. Life-long learning: Recognize the need for, and have the preparation and ability to engage in
independent and life-long learning in the broadest context of technological change
 Department of Information Technology
Rubrics for Lab Assessment

Rubrics 0 1 2 3
Missing Inadequate Needs Improvement Adequate
R1 Is able to No mention is An attempt is The problem to be The problem to be
identify the made of the made to identify solved is described solved is clearly stated.
problem to problem to be the problem to be but there are minor Objectives are
be solved and solved. solved but it is omissions or vague complete, specific,
define the described in a details. Objectives are concise, and
objectives of confusing manner, conceptually correct measurable. They are
the objectives are not and measurable but written using correct
experiment. relevant, may be incomplete in technical terminology
objectives contain scope or have and are free from
technical/ linguistic errors. linguistic errors.
conceptual errors
or objectives are
not measurable.
R2 Is able to The experiment The experiment The experiment The experiment solves
design a does not solve the attempts to solve attempts to solve the the problem and has a
reliable problem. the problem but problem but due to high likelihood of
experiment due to the nature the nature of the producing data that will
that solves of the design the design there is a lead to a reliable
the problem. data will not lead moderate chance the solution.
to a reliable data will not lead to a
solution. reliable solution
R3 Is able to Diagrams are Diagrams are Diagrams and/or Diagrams and/or
communicate missing and/or present but experimental experimental procedure
the details of experimental unclear and/or procedure are present are clear and complete.
an procedure is experimental but with minor
experimental missing or procedure is omissions or vague
procedure extremely vague. present but details.
clearly and important details
completely. are missing.
R4 Is able to Data are either Some important All important data are All important data are
record and absent or data are absent or present, but recorded present, organized and
represent incomprehensible. incomprehensible. in a way that requires recorded clearly.
data in a some effort to
meaningful comprehend.
way.
R5 Is able to No discussion is A judgment is An acceptable An acceptable judgment
make a presented about made about the judgment is made is made about the
judgment the results of the results, but it is about the result, but result, with clear
about the experiment. not reasonable or the reasoning is reasoning. The effects
results of the coherent. flawed or incomplete. of assumptions and
experiment. experimental
uncertainties are
considered.
 INDEX

PAPER : DATA SCIENCE LAB

PAPER CODE : CIE‐405P

NAME OF THE STUDENT : Ankit Prakash

UNIVERSITY ROLL NO. : 04014803121

BRANCH : Information Technology

SECTION/ GROUP : FSD-III-A

PRACTICAL DETAILS:

S Date Experiment Name Marks (0-3) Total Signature

No. Marks
(15)
a) Experiments according to the list provided by GGSIPU R1 R2 R3 R4 R5

1 Describing data, viewing and

manipulating data.

2 To plot the probability distribution

curve through Python.

3 To perform chi square test on

various data sets.

4 To use Python as a programming

tool for the analysis of data
structures.
5 To perform various operations
such as data storage, analysis and
visualization
6 To perform descriptive statistics
analysis and data visualization.

7 To perform Principal Component

Analysis on datasets.

8 To perform linear regression on

datasets.

9 To perform Data Aggregation and

Group-wise Operations.
S No. Date Experiment Name Marks (0-3) Total Signature
Marks
(15)

a) Experiments Additional the list provided by GGSIPU R1 R2 R3 R4 R5

1 Analyze and forecast time series

data using statistical and
machine learning techniques.
2 Apply and evaluate clustering
techniques to group similar
data points.
 EXPERIMENT 1

AIM: Describing data, viewing and manipulating data.

THEORY:

Python is a powerful tool for data analysis, and it offers a range of functions for describing,
viewing, and manipulating datasets. Understanding these capabilities is essential for effective
data exploration and preparation.

1. Describing Data:
Descriptive statistics help summarize and describe the main features of a dataset. R provides
built-in functions to gain insights into the data's structure and key statistics:

● summary(): Summarizes each column in a dataset, displaying metrics like

mean, median, minimum, and maximum.
● str(): Shows the structure of an object, including data types and
sample values.
● dim(): Displays the dimensions (rows and columns) of a data frame.
● nrow() and ncol(): Return the number of rows and columns, respectively.

2. Viewing Data:
Python allows users to take a closer look at datasets using various functions:

● head(): Displays the first few rows of a dataset, making it easy to inspect initial
data.
● tail(): Shows the last few rows, useful for checking data at the end of
the dataset.
● Indexing and slicing (data[1:10, ]): Extracts specific rows and columns for
detailed viewing.
3. Manipulating Data:
Python is equipped with tools for data transformation and manipulation:

● Subsetting: Extracts specific parts of a dataset using functions like subset().

● Filtering: Uses logical conditions to select rows, e.g.,
data[data$column > value, ].
● Adding Columns: New columns can be added by simple assignment, such as
data$new_column <- data$existing_column * 2.
● Modifying Data: Data can be modified directly by referencing the
specific rows and columns that need changes.
● Packages for Enhanced Manipulation: The dplyr package offers a more intuitive
syntax for data manipulation tasks, such as filtering (filter()), selecting columns
(select()), and summarizing data (summarize()).

Applications: Understanding how to describe, view, and manipulate data is critical for any
data analysis process, as it allows users to clean, organize, and prepare data for further
statistical analysis and visualization. This step ensures that data is accurate, well-structured,
and ready for deeper insights.
 SOURCE CODE:

# Importing required libraries

import pandas as pd

# Step 1: Load data from an online source (example: Titanic dataset from GitHub)

url = "[Link]

data = pd.read_csv(url)

# Step 2: Describe the data - Get a summary of descriptive statistics

print("Data Description:")

print([Link]()) # Summary of numerical columns

print([Link](include='object')) # Summary of categorical columns

# Step 3: View data - Check the first few rows and column information

print("\nFirst 5 rows of data:")

print([Link]()) # View first few rows

print("\nData Information:")

print([Link]()) # Data type and non-null count for each column

# Step 4: Manipulate data - Example: Create a new column, fill missing values, and filter data

# Fill missing Age values with the median age

data['Age'].fillna(data['Age'].median(), inplace=True)

# Add a new column to categorize passengers as "Child" if Age < 18, else "Adult"

data['Age_Group'] = data['Age'].apply(lambda x: 'Child' if x < 18 else 'Adult')

# Filter data to show only female passengers

female_passengers = data[data['Sex'] == 'female']

print("\nData with Age_Group column and only female passengers:")

print(female_passengers[['Name', 'Sex', 'Age', 'Age_Group']].head())

OUTPUT:
VIVA VOCE:

1. Q: What function would you use to read a CSV file into a Pandas DataFrame, and
how would you specify the delimiter if it’s not a comma?
A: You would use the pd.read_csv() function from Pandas. To specify a different delimiter,
you can use the sep parameter. For example, to read a tab-delimited file, you would use
pd.read_csv('file_path.csv', sep='\t').

2. Q: Which function is used to obtain a quick summary of the statistics for numerical
columns in a DataFrame? What kind of output does it provide?
A: The describe() function is used to obtain a summary of statistics for numerical columns.
It provides output including count, mean, standard deviation, minimum, maximum, and
quartiles (25th, 50th, and 75th percentiles).

3. Q: What are two methods you can use in Pandas to handle missing values in a
DataFrame, and what are the implications of each method?
A: Two methods are:
a. dropna(): This method removes any rows or columns with missing values. The
implication is that you may lose valuable data.
b. fillna(): This method fills missing values with a specified value (e.g., mean,
median). The implication is that it may introduce bias if the chosen fill value
does not represent the data well.

4. Q: Which library would you commonly use for data visualization in Python, and
what is a simple command to create a histogram of a column in a DataFrame?
A: The matplotlib library is commonly used for data visualization. A simple command to
create a histogram of a column named column_name in a DataFrame df is:
[Link](df['column_name']).

5. Q: After manipulating a DataFrame, what command would you use to save it back
to a CSV file?
A: You would use the to_csv() method. For example: df.to_csv('output_file.csv',
index=False) to save the DataFrame df to a CSV file without the index.
 EXPERIMENT 2

AIM: To plot the probability distribution curve .

THEORY:

Probability distribution curves are graphical representations of the likelihood of different

outcomes in a random variable. Python provides several libraries to visualize these distributions,
which can help in understanding the underlying characteristics of the data.
You can plot probability distribution curves for various distributions, such as the normal,
binomial, and Poisson distributions using functions from the [Link] module.

The basic functions used to plot these curves include:

 [Link]() for the normal distribution,
 [Link]() for the binomial distribution,
 [Link]() for the Poisson distribution.

Steps to Plot a Probability Distribution Curve in Python

1. Define the Range of Values: Create a sequence of values over which to evaluate the
distribution using numpy.
2. Calculate the Probability Density Function (PDF): Use the appropriate functions from
[Link] to calculate the PDF (for continuous distributions) or the PMF (for discrete
distributions) for each value.
3. Plot the Curve: Use matplotlib to create the graph and add the curve.
SOURCE CODE:
# Importing libraries
import numpy as np
import seaborn as sns
import [Link] as plt

# Step 1: Generate sample data from a normal distribution

data = [Link](loc=0, scale=1, size=1000) # mean=0, std=1, 1000 data points

# Step 2: Plot the probability distribution curve using Seaborn

[Link](data, shade=True, color="blue")

# Step 3: Customize and display the plot

[Link]("Probability Distribution Curve")
[Link]("Value")
[Link]("Density")
[Link]()

OUTPUT:
VIVA VOCE:

1. Q: Which Python libraries would you typically import to plot probability

distribution curves, and what are their primary functions?
A: You would typically import matplotlib for general plotting and seaborn for statistical
visualizations. Matplotlib's primary function is [Link]() for basic plots, while Seaborn
provides functions like [Link]() for plotting density estimates.

2. Q: How do you create a histogram of a dataset using Matplotlib, and what

parameter would you adjust to change the number of bins?
A: To create a histogram using Matplotlib, you would use the command [Link](data,
bins=num_bins), where num_bins is the parameter you adjust to change the number of bins.

3. Q: What is a Kernel Density Estimate (KDE) plot, and how can you create one
using Seaborn for a given dataset?
A: A Kernel Density Estimate (KDE) plot is a smooth representation of the distribution of
data points. You can create one using Seaborn with the command: [Link](data).

4. Q: If you want to plot multiple probability distribution curves on the same graph,
how would you do it? What function allows you to plot multiple lines?
A: You can use the [Link]() function from Matplotlib to overlay multiple distribution
curves on the same graph. You would call [Link]() for each distribution curve you want to
visualize.
 EXPERIMENT 3

AIM: To perform a chi-square test on various data sets.

THEORY:

The Chi-Square test is a statistical method used to determine if there is a significant

association between two categorical variables. It compares the observed frequencies in each
category to the frequencies expected if the two variables are independent. The test calculates
the Chi-Square statistic, which can then be compared to a critical value from the Chi-Square
distribution to determine significance.

1. Steps for Performing Chi-Square Test:

1. Create a Contingency Table: Organize the data into a matrix format showing
the frequency distribution of the variables.
2. Calculate the Chi-Square Statistic: Use the formula: χ2=∑(O−E)2E\chi^2 = \
sum \frac{(O - E)^2}{E}χ2=∑E(O−E)2 where OOO is the observed frequency
and EEE is the expected frequency.
[Link] Degrees of Freedom: The degrees of freedom (df) is calculated as:
df=(r−1)(c−1)df = (r - 1)(c - 1)df=(r−1)(c−1)
where rrr is the number of rows and ccc is the number of columns.
4. Interpret the Results: Compare the calculated Chi-Square value with the critical
value from the Chi-Square distribution table based on the degrees of freedom to
determine if the result is statistically significant.
Source Code:
# Import libraries
import pandas as pd
import [Link] as stats

# Load the Titanic dataset

url = "[Link]
data = pd.read_csv(url)

# Create a contingency table for 'Sex' and 'Survived' columns

contingency_table = [Link](data['Sex'], data['Survived'])

# Perform the Chi-Square test for independence

chi2, p, dof, expected = stats.chi2_contingency(contingency_table)

# Display results
print("Chi-Square Statistic:", chi2)
print("p-value:", p)
print("Degrees of Freedom:", dof)
print("Expected Frequencies:\n", expected)

# Interpretation
if p < 0.05:
print("Reject the null hypothesis: There's a significant association between gender and
survival.")
else:
print("Fail to reject the null hypothesis: No significant association between gender and
survival.")

OUTPUT:
VIVA VOCE:

1. Q: What is a chi-square test, and what types of data is it typically used to analyze?
A: A chi-square test is a statistical method used to determine whether there is a significant
association between categorical variables. It is typically used to analyze nominal or ordinal
data.

2. Q: When performing a chi-square test, how should the data be structured? Explain
the requirements for the observed and expected frequency tables.
A: The data should be structured in a contingency table format, where rows represent one
categorical variable and columns represent another. The observed frequency table contains
counts of actual observations, while the expected frequency table contains the counts
expected under the null hypothesis, calculated based on the marginal totals.

3. Q: Which function is used to create a contingency table from a Pandas DataFrame

to prepare for a chi-square test?
A: The [Link]() function is used to create a contingency table from a Pandas
DataFrame, which summarizes the relationship between two categorical variables.

4. Q: How is the chi-square goodness-of-fit test performed, and how is it different

from the test for independence?
A: The chi-square goodness-of-fit test is performed to determine if a sample distribution fits
a theoretical distribution. It compares observed frequencies to expected frequencies for one
categorical variable. In contrast, the test for independence evaluates whether two categorical
variables are related by comparing observed frequencies in a contingency table.

5. Q: What kind of plots can be useful for visualizing the results of a chi-square test,
and which libraries are used to create them?
A: Bar plots and mosaic plots are useful for visualizing chi-square test results. Libraries such
as Matplotlib and Seaborn can be used to create these visualizations.
 EXPERIMENT 4

AIM: To use Python as a programming tool for the analysis of data structures.

THEORY:

Python is a versatile programming language widely used for data analysis due to its readability,
simplicity, and the availability of powerful libraries like pandas, NumPy, and matplotlib. These
libraries provide efficient data structures and functions for handling, manipulating, and
analyzing data.

● pandas: Provides data structures like DataFrame and Series, which make it easy to
work with structured data.
● NumPy: Offers support for arrays and a wide range of mathematical
functions.
● matplotlib and seaborn: Used for data visualization.

2. Key Python Libraries for Data Analysis:

1. pandas:
○ Core Structure: DataFrame, a 2D labeled data structure akin to a
spreadsheet or SQL table, and Series, a 1D array-like structure.
○
Use Cases: Data cleaning, manipulation, joining/merging data,
and handling missing data.
○ Example Operations: Reading data from CSVs, performing group-by
operations, and data aggregation.
 2. NumPy:
○ Core Structure: ndarray, a powerful n-dimensional array object.
○ Use Cases: Supports mathematical operations on large data
arrays, efficient computations, and handling multidimensional data.
○ Example Operations: Array slicing, matrix operations, and
statistical computations.

3. matplotlib and seaborn:

○ Core Structures: Graph plotting tools that help visualize data through
charts, histograms, scatter plots, and more.
○ Use Cases: Data exploration through visual analysis, plotting
data distributions, and trend analysis.
○ Example Operations: Line graphs for time series data, bar charts for
categorical data comparisons, and scatter plots for correlation analysis.
SOURCE CODE:
# Importing statistics library
import statistics

# Sample list of data

data = [12, 15, 20, 21, 24, 17, 30, 19, 21, 25]

# Sorting the list

sorted_data = sorted(data)

# Basic statistical analysis

mean = [Link](data)
median = [Link](data)
mode = [Link](data)

# Output results
print("Sorted Data:", sorted_data)
print("Mean:", mean)
print("Median:", median)
print("Mode:", mode)

# Define a binary tree node class

class TreeNode:
def __init__(self, data):
[Link] = data
[Link] = None
[Link] = None

# Function for in-order traversal

def in_order_traversal(root):
if root:
in_order_traversal([Link])
print([Link], end=" ")
in_order_traversal([Link])
# Building a sample binary tree
root = TreeNode(10)
[Link] = TreeNode(5)
[Link] = TreeNode(15)
[Link] = TreeNode(3)
[Link] = TreeNode(7)

# Traversing the tree

print("In-Order Traversal of Tree:")
in_order_traversal(root)

OUTPUT:
VIVA VOCE:

1. Q: What are the primary built-in data structures in Python, and what are their key
characteristics?
A: The primary built-in data structures in Python are:
List: Ordered, mutable, can contain duplicate elements.
Tuple: Ordered, immutable, can contain duplicate elements.
Set: Unordered, mutable, no duplicate elements allowed.
Dictionary: Unordered, mutable, key-value pairs with unique keys.

2. Q: How are nested data structures (e.g., a list of dictionaries) created in Python,
and how can a specific value within it be accessed?
A: Nested data structures are created by defining a list that contains dictionaries. For
example: data = [{'name': 'Alice', 'age': 25}, {'name': 'Bob', 'age': 30}]. To access a specific
value, you can use indexing and key access: data[0]['name'] would return 'Alice'.

3. Q: In the context of Pandas, explain the difference between a DataFrame and a

Series. In what scenarios is one preferred over another?
A: A DataFrame is a two-dimensional, tabular data structure with labeled axes (rows and
columns), while a Series is a one-dimensional array-like structure that can hold any data
type. A DataFrame is preferred for handling multi-dimensional data, while a Series is
suitable for handling single columns or one-dimensional data.

4. Q: How can two dictionaries be merged in Python? What syntax is used to ensure
that values from both dictionaries are retained?
A: You can merge two dictionaries using the update() method or the {**dict1, **dict2}
syntax in Python 3.5 and above. To ensure that values from both dictionaries are retained,
you can handle conflicts by using a custom approach, like combining values in a list:
dict1 = {'a': 1, 'b': 2}
dict2 = {'b': 3, 'c': 4}
merged = {key: [[Link](key), [Link](key)] for key in set(dict1) | set(dict2)}

5. Q: Which libraries are used in Python for handling and analyzing large datasets,
and what are some of their key features?
A: Common libraries include:
a. Pandas: For data manipulation and analysis, offering DataFrame and Series
structures.
b. Dask: For parallel computing with larger-than-memory datasets, extending
Pandas functionality.
c. Vaex: For lazy loading and efficient memory usage with large datasets,
particularly in out-of-core computing.
 EXPERIMENT 5

AIM: Performing Data Storage, Analysis, and Visualization

THEORY:

Python is a versatile programming language widely used for data analysis, visualization, and
data manipulation. Its powerful libraries, such as pandas, numpy, matplotlib, and seaborn, make
it an excellent tool for managing and interpreting data.

Python provides capabilities for data storage through various file formats, efficient data analysis
using built-in and external packages, and comprehensive data visualization to showcase insights.

Key Features of Python for Data Management:

 Data Storage: Python can read from and write to various file formats, including CSV,
Excel, JSON, and databases (using libraries like sqlite3 or SQLAlchemy).
 Data Analysis: Python includes functions for data manipulation (using pandas),
descriptive statistics (with numpy and scipy), and applying complex statistical models
(using libraries like statsmodels and scikit-learn).
 Visualization: Libraries such as matplotlib and seaborn provide extensive tools for
creating high-quality plots and graphs, making it easy to visualize data trends and
insights.
SOURCE CODE:
import pandas as pd

# Load data from a CSV file

url = "[Link]
df = pd.read_csv(url)

# Save the DataFrame to a new CSV file

df.to_csv("iris_data.csv", index=False)

print("Data loaded and saved successfully.")

### 2. **Data Analysis**

# Python, with the help of Pandas and NumPy, can perform a variety of analytical operations,
from descriptive statistics to more complex calculations.

#### Example: Basic Data Analysis

# Display the first few rows of the dataset

print("First 5 rows of the dataset:\n", [Link]())

# Summary statistics for numerical columns

print("\nSummary Statistics:\n", [Link]())

# Count occurrences of each species

species_count = df['species'].value_counts()
print("\nSpecies Count:\n", species_count)

### 3. **Data Manipulation**

# Pandas offers powerful functions to filter, group, and manipulate data.

#### Example: Filtering and Grouping Data

# Filter data for a specific species

setosa_data = df[df['species'] == 'setosa']
print("\nSetosa Species Data:\n", setosa_data.head())
# Group data by species and calculate the mean of each group
mean_by_species = [Link]('species').mean()
print("\nMean values by species:\n", mean_by_species)

### 4. **Data Visualization**

# Visualization is essential for understanding patterns and trends. Libraries like Matplotlib and
Seaborn provide a range of plotting options.

import [Link] as plt

import seaborn as sns

# Histogram of petal lengths

[Link](figsize=(6, 4))
[Link](df['petal_length'], bins=20, kde=True)
[Link]("Histogram of Petal Lengths")
[Link]("Petal Length")
[Link]("Frequency")
[Link]()

# Scatter plot of petal length vs petal width by species

[Link](figsize=(6, 4))
[Link](data=df, x='petal_length', y='petal_width', hue='species')
[Link]("Petal Length vs. Petal Width by Species")
[Link]("Petal Length")
[Link]("Petal Width")
[Link](title='Species')
[Link]()

### 5. **Combining Storage, Analysis, and Visualization**

# Python’s ability to handle data storage, analysis, and visualization in one workflow makes it
ideal for end-to-end data processing tasks. Here’s a complete example:

# ```python
# Load the dataset
df = pd.read_csv("[Link]
# Analyze the data: Calculate the mean petal length for each species
mean_petal_length = [Link]('species')['petal_length'].mean()
print("\nMean Petal Length by Species:\n", mean_petal_length)

# Visualize the mean petal length by species

mean_petal_length.plot(kind='bar', color=['skyblue', 'orange', 'lightgreen'])
[Link]("Mean Petal Length by Species")
[Link]("Species")
[Link]("Mean Petal Length")
[Link](rotation=0)
[Link]()

OUTPUT:
VIVA VOCE:

1. Q: What are the main descriptive statistics measures that can be calculated using
Pandas, and how can they be obtained from a DataFrame?
A: Main descriptive statistics measures include:
a. Mean: Average value (use [Link]()).
b. Median: Middle value (use [Link]()).
c. Standard Deviation: Measure of dispersion (use [Link]()).
d. Count: Number of non-null entries (use [Link]()).
e. Minimum and Maximum: Lowest and highest values (use [Link]() and [Link]()).
f. Quantiles: Values below which a certain percentage of data falls (use
[Link](q)).

2. Q: What methods can be used to handle missing values in a dataset when

performing descriptive statistics in Python, and what are their implications?
A: Methods include:
a. Drop Missing Values: Using [Link](), which may lead to loss of information
if many rows are dropped.
b. Fill Missing Values: Using [Link](value) to replace missing values, which can
introduce bias if not done thoughtfully.
c. Imputation: Replacing missing values with calculated statistics (e.g., meanor
median), which can distort the actual distribution.

3. Q: What is a box plot, and how can you create one using Seaborn? What
information does a box plot convey about the data?
A: A box plot is a graphical representation that shows the distribution of a dataset based on
five summary statistics: minimum, first quartile, median, third quartile, and maximum. It can
be created using Seaborn with:
import seaborn as sns
[Link](x='column_name', data=df)
A box plot conveys information about the central tendency, variability, and presence of
outliers in the data.

4. Q: Which functions can be used for computation and visualization of a correlation

matrix for a DataFrame? What kind of plot is commonly used to display this
matrix?
A: The [Link]() function computes the correlation matrix. For visualization, a heatmap can
be created using Seaborn:
import seaborn as sns
corr_matrix = [Link]()
[Link](corr_matrix, annot=True, cmap='coolwarm')
A heatmap is commonly used to display the correlation matrix visually.

5. Q: After generating a plot, how can it be saved to a file using Matplotlib? Provide
an example command to save a plot.
A: You can save a plot using the savefig() function in Matplotlib. For example:
import [Link] as plt
[Link]('[Link]')
This command saves the current plot as a PNG file.
 EXPERIMENT 6

AIM: To perform descriptive statistics analysis and data visualization in python.

THEORY:
Descriptive statistics provide summaries about the main characteristics of a dataset. This
includes measures such as mean, median, standard deviation, and quartiles. Data visualization
complements this by providing a graphical representation of data, making it easier to identify
patterns, trends, and outliers. Python, with its rich set of libraries like pandas, matplotlib, and
seaborn, is well-suited for both descriptive analysis and visualization.

Descriptive statistics summarize and provide insights about the characteristics of a dataset. This
includes:

● Measures of Central Tendency:

○ Mean (Average): The sum of all values divided by the count, giving a
central value for the data.
○ Median: The middle value when data points are arranged in
ascending order. It is less affected by outliers than the mean.
○ Mode: The value that appears most frequently in the dataset.

● Measures of Dispersion:
○ Range: The difference between the maximum and minimum values in the
dataset.
○ Standard Deviation: A measure of how spread out the
numbers in a dataset are relative to the mean. A small standard
deviation indicates that the values are close to the mean, while a large standard deviation
shows that they are more spread out.
○ Variance: The square of the standard deviation, representing
the spread of the data points.

● Percentiles and Quartiles:

○ Percentiles: Indicate the value below which a certain percentage
of data points fall.
○ Quartiles: Divide the data into four parts, with Q1 (25th
percentile), Q2 (median or 50th percentile), and Q3 (75th percentile). The interquartile range
(IQR) is the difference between Q3 and Q1 and represents the middle 50% of the data.

These statistics give an overview of data distribution, highlighting how values are spread and
clustered.
SOURCE CODE:

import pandas as pd
import numpy as np
import [Link] as plt
import seaborn as sns
# Load dataset from Seaborn's repository
url = "[Link]
df = pd.read_csv(url)

# Display the first few rows

[Link]()
# Get a statistical summary for numerical columns
summary_stats = [Link]()
print("Descriptive Statistics:\n", summary_stats)

# Histogram of petal length

[Link](figsize=(6, 4))
[Link](df['petal_length'], bins=20, kde=True, color='skyblue')
[Link]("Distribution of Petal Length")
[Link]("Petal Length")
[Link]("Frequency")
[Link]()
# Box plot for petal width by species
[Link](figsize=(6, 4))
[Link](data=df, x='species', y='petal_width', palette="Set3")
[Link]("Box Plot of Petal Width by Species")
[Link]("Species")
[Link]("Petal Width")
[Link]()
# Calculate and visualize the correlation matrix
correlation_matrix = [Link]()
[Link](figsize=(8, 6))
[Link](correlation_matrix, annot=True, cmap="coolwarm", fmt=".2f")
[Link]("Correlation Matrix of Iris Dataset")
[Link]()
OUTPUT:
 EXPERIMENT 7

AIM: To perform Principal Component Analysis on datasets.

THEORY:

Principal Component Analysis (PCA) is a dimensionality reduction technique used to reduce

the number of variables in a dataset while retaining most of the original variability. It
transforms the original variables into a new set of uncorrelated variables called principal
components. These components are ordered such that the first principal component
accounts for the maximum variance, the second accounts for the second highest variance,
and so on.

3. Purpose of PCA:

● Simplification: Reduces the complexity of high-dimensional data.

● Visualization: Helps visualize data in 2D or 3D for better
interpretation.
● Feature Selection: Identifies the most significant variables that
contribute to variance.
● Noise Reduction: Removes less informative features that can lead to noise in data
analysis.

4. Mathematical Concepts:

● Covariance Matrix: Measures how changes in one variable are associated

with changes in another.
● Eigenvalues and Eigenvectors: Used to determine the principal
components. The eigenvectors correspond to the direction of maximum variance, while the
eigenvalues represent the magnitude of the variance in each direction.
 ● Principal Components: Computed as linear combinations of the original
variables.

SOURCE CODE:

# Step 1: Import Required Libraries

import pandas as pd
import numpy as np
from [Link] import PCA
from [Link] import StandardScaler
import [Link] as plt
import seaborn as sns

# Step 2: Load a Dataset

url = "[Link]
df = pd.read_csv(url)

# Display the first few rows

print([Link]())

# Step 3: Standardize the Data

features = [Link](columns=['species'])
target = df['species']

scaler = StandardScaler()
scaled_features = scaler.fit_transform(features)

# Step 4: Apply PCA

pca = PCA(n_components=2)
principal_components = pca.fit_transform(scaled_features)

# Create a DataFrame for the principal components

pca_df = [Link](data=principal_components, columns=['PC1', 'PC2'])
pca_df['species'] = target

# Step 5: Visualize the Results

[Link](figsize=(8, 6))
[Link](data=pca_df, x='PC1', y='PC2', hue='species', palette='Set1')
[Link]("PCA of Iris Dataset")
[Link]("Principal Component 1")
[Link]("Principal Component 2")
[Link](title='Species')
[Link]()
# Step 6: Explained Variance Ratio
explained_variance = pca.explained_variance_ratio_
print("Explained Variance by each Principal Component:", explained_variance)

# Plot cumulative explained variance

[Link](figsize=(8, 6))
[Link]([Link](explained_variance), marker='o', linestyle='--')
[Link]("Number of Components")
[Link]("Cumulative Explained Variance")
[Link]("Explained Variance vs. Number of Components")
[Link]()

OUTPUT:
VIVA VOCE:

1. Q: Why is it important to standardize data before performing PCA, and which

function from scikit-learn is used for this purpose?
A: Standardizing data is important before PCA to ensure that each feature contributes
equally to the analysis, as PCA is sensitive to the scale of the data. The StandardScaler from
scikit-learn is used for this purpose:

2. Q: After creating a PCA object, how is it fitted to the data, and what method is
used to transform the original data into the principal component space?
A: After creating a PCA object with PCA(), it is fitted to the data using the fit() method. To
transform the data, the transform() method is used:
from [Link] import PCA
pca = PCA(n_components=2)
[Link](data)
transformed_data = [Link](data)

3. Q: How is the optimal number of principal components determined for further

analysis, and what techniques can be employed to visualize this decision?
A: The optimal number of principal components can be determined using techniques like the
explained variance ratio, which indicates how much variance each principal component
captures. This can be visualized using a scree plot:
import [Link] as plt
[Link](range(1, len(explained_variance) + 1), explained_variance)
[Link]('Number of Components')
[Link]('Explained Variance')

4. Q: After transforming the data using PCA, how can the transformation be reversed
to return to the original feature space, and which method is utilized for this
purpose?
A: The transformation can be reversed using the inverse_transform() method of the PCA
object.

5. Q: What challenges may arise when applying PCA to high-dimensional datasets,

and how can PCA help alleviate some of these issues?
A: Challenges include the curse of dimensionality, where distance measures become less
meaningful. PCA helps alleviate these issues by reducing dimensionality while retaining
variance, making the dataset easier to visualize and analyze.
 EXPERIMENT 8

AIM: To perform linear regression on datasets.

THEORY:

Linear regression is a statistical technique used to model and analyze the relationships between
a dependent variable (target) and one or more independent variables (predictors). The goal is to
find the best-fitting line that describes the linear relationship between these variables.

5. Key Concepts:

● Dependent Variable (Y): The outcome variable we are trying to predict or

explain.
● Independent Variable (X): The predictor variable(s) used to explain
changes in the dependent variable.
● Regression Line: The line of best fit represented by the equation
Y=β0+β1X+ϵY = \beta_0 + \beta_1X + \epsilonY=β0+β1X+ϵ, where:
○ β0\beta_0β0 is the intercept.
○ β1\beta_1β1 is the slope of the line.
○ ϵ\epsilonϵ is the error term.

6. Assumptions of Linear Regression:

1. Linearity: The relationship between the dependent and independent variables is

linear.
2. Independence: Observations are independent of each other.
3. Homoscedasticity: The residuals have constant variance at every level of XXX.
4. Normality: Residuals are normally distributed.

7. Metrics to Evaluate Linear Regression:

 ● R-squared (R²): Indicates the proportion of variance in the dependent
variable that is predictable from the independent variable(s). Values range
from 0 to 1, with values closer to 1 indicating a better fit.
● Adjusted R-squared: A modified version of R-squared that
accounts for the number of predictors in the model.
● p-value: Tests the hypothesis that the coefficients are significantly different from
zero.
SOURCE CODE:
# Step 1: Import Required Libraries
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from [Link] import mean_absolute_error, mean_squared_error, r2_score
import [Link] as plt

# Step 2: Load Dataset

# Using an online dataset for example (Boston Housing Dataset)
url = "[Link]
df = pd.read_csv(url)

# Display the first few rows of the dataset

print([Link]())

# Step 3: Select Features and Target Variable

X = [Link](columns=['medv']) # All columns except the target variable 'medv'
y = df['medv'] # Target variable (house prices)

# Step 4: Split Data into Training and Testing Sets

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Step 5: Create and Train the Model

model = LinearRegression()
[Link](X_train, y_train)

# Step 6: Make Predictions

y_pred = [Link](X_test)

# Step 7: Evaluate the Model

mae = mean_absolute_error(y_test, y_pred)
mse = mean_squared_error(y_test, y_pred)
rmse = mean_squared_error(y_test, y_pred, squared=False)
r2 = r2_score(y_test, y_pred)

print("Mean Absolute Error (MAE):", mae)

print("Mean Squared Error (MSE):", mse)
print("Root Mean Squared Error (RMSE):", rmse)
print("R-squared (R2):", r2)
# Step 8: Visualization
[Link](figsize=(8, 6))
[Link](y_test, y_pred, color="blue", alpha=0.6)
[Link]([[Link](), [Link]()], [[Link](), [Link]()], 'r--', lw=2)
[Link]("Actual Prices")
[Link]("Predicted Prices")
[Link]("Actual vs Predicted Prices")
[Link]()

OUTPUT:
VIVA VOCE:

1. Q: Which Python libraries are commonly used for performing linear regression,
and what are their key functionalities?
o A: Common libraries include:
 scikit-learn: Provides tools for regression analysis, including the
LinearRegression class.
 statsmodels: Offers detailed statistical output and model diagnostics for
linear regression.
 TensorFlow/Keras: Can be used for regression tasks, especially in neural
network contexts.
2. Q: What steps should be taken to prepare a dataset for linear regression analysis,
including handling missing values and feature scaling?
o A: Steps include:
 Handling Missing Values: Use techniques like imputation or deletion.
 Feature Scaling: Normalize or standardize features, especially if they are
on different scales, to improve convergence during training.
3. Q: How can the coefficients obtained from a linear regression model be interpreted
in terms of their effect on the target variable?
o A: Each coefficient represents the expected change in the target variable for a
one-unit increase in the corresponding feature, holding all other features
constant. Positive coefficients indicate a direct relationship, while negative
coefficients indicate an inverse relationship.
4. Q: What metrics can be used to evaluate the performance of a linear regression
model, and how is each metric calculated?
o A: Common metrics include:
 Mean Absolute Error (MAE): Average of absolute differences between
predicted and actual values.
 Mean Squared Error (MSE): Average of squared differences between
predicted and actual values.
 R-squared: Proportion of variance in the target variable that is predictable
from the independent variables.
5. Q: What are the key assumptions underlying linear regression, and why is it
important to check these assumptions before interpreting the results?
o A: Key assumptions include:
 Linearity: The relationship between predictors and the target variable is
linear.
 Independence: Observations are independent of each other.
 Homoscedasticity: Constant variance of errors.
 Normality: Residuals should be normally distributed. Checking these
assumptions is important to ensure the validity of the regression results
and to avoid misleading conclusions.
 EXPERIMENT 9

AIM: To perform Data Aggregation and Group Wise Operations.

THEORY:

Data aggregation involves summarizing data by grouping it based on one or more attributes.
This technique is essential in data analysis for extracting insights and making the data more
interpretable.
Group-wise operations allow you to apply functions (e.g., mean, sum, count) to specific
groups within a dataset.

8. Applications:

● Summarizing sales data by region or product category.

● Calculating average test scores per student group.
● Analyzing financial data to find total revenue by quarter.

9. R Functions for Group-wise Operations:

● aggregate(): Used to compute summary statistics for subgroups of

data.
● tapply(): Applies a function to subsets of a vector.
● dplyr package functions:

○ group_by(): Groups the data by one or more columns.

○ summarise(): Computes summary statistics for grouped data.
SOURCE CODE:
# Step 1: Import Required Libraries
import pandas as pd
import [Link] as plt

# Step 2: Load Dataset

# Using an online dataset (for example, the Titanic dataset)
url = "[Link]
df = pd.read_csv(url)

# Display the first few rows of the dataset

print([Link]())

# Step 3: Data Aggregation

# Grouping by 'Pclass' and calculating the average 'Age' and 'Fare'
agg_df = [Link]('Pclass').agg({
'Age': 'mean', # Calculate mean age
'Fare': 'mean', # Calculate mean fare
'Survived': 'sum' # Calculate total number of survivors
}).reset_index()

# Display the aggregated DataFrame

print("\nAggregated DataFrame:")
print(agg_df)

# Step 4: Visualize Aggregated Results

# Bar plot for average age and fare by passenger class
fig, ax1 = [Link](figsize=(10, 6))

# Create bar plot for average age

[Link](agg_df['Pclass'], agg_df['Age'], color='b', alpha=0.6, label='Average Age')
ax1.set_xlabel('Passenger Class')
ax1.set_ylabel('Average Age', color='b')
ax1.tick_params(axis='y', labelcolor='b')

# Create a second y-axis for average fare

ax2 = [Link]()
[Link](agg_df['Pclass'], agg_df['Fare'], color='r', marker='o', label='Average Fare')
ax2.set_ylabel('Average Fare', color='r')
ax2.tick_params(axis='y', labelcolor='r')
# Adding title and legend
[Link]('Average Age and Fare by Passenger Class')
[Link](loc='upper left')
[Link](loc='upper right')

[Link]()

# Step 5: Group-wise Operations

# Count of survivors by gender
survivor_count = [Link]('Sex')['Survived'].sum().reset_index()

# Display the count of survivors by gender

print("\nCount of Survivors by Gender:")
print(survivor_count)

OUTPUT:
VIVA VOCE:

1. Q: What are some common aggregate functions that can be applied to grouped
data in Pandas, and how are they used?
A: Common aggregate functions include:
a. sum(): Calculates the total for each group.
b. mean(): Calculates the average for each group.
c. count(): Counts the number of entries in each group.
d. max() and min(): Find the maximum and minimum values in each group.

2. Q: How can multiple aggregation functions be applied to different columns of a

DataFrame simultaneously using Pandas? Provide an example of the syntax.
A: Multiple aggregation functions can be applied using the agg() method with a
dictionary specifying column-function pairs.

3. Q: How can custom functions be applied to groups in a DataFrame, and which

method allows for this functionality?
A: Custom functions can be applied using the apply() method in combination with
groupby(). result = [Link]('group_column').apply(custom_function)

4. Q: What is a pivot table, and how can it be created using Pandas? What are the
key parameters to consider when constructing a pivot table?
A: A pivot table is a data processing tool that allows for the summarization of data in a
DataFrame, often used to aggregate values. It can be created using the pivot_table()
function, with key parameters including:
a. data: The DataFrame to pivot.
b. index: The column(s) to use as the row index.
c. columns: The column(s) to use as the column index.
d. values: The column

5. Q: How can the results of an aggregation be sorted in ascending or descending

order after performing a group-wise operation?
A: The sort_values() method can be used to sort the results of an aggregation in ascending
or descending order.
 EXPERIMENT 1

AIM: Analyze and forecast time series data using statistical and machine learning
techniques.
1. Load and visualize a time series dataset, such as stock prices or weather data.
2. Perform seasonal decomposition of time series data using seasonal_decompose from
statsmodels.
3. Implement a simple moving average model for time series forecasting.
4. Apply ARIMA (AutoRegressive Integrated Moving Average) model for time series
forecasting.
5. Evaluate time series forecasting performance using metrics like MAE and RMSE.
6. Plot autocorrelation and partial autocorrelation functions to identify patterns in time
series data.
7. Perform time series cross-validation using rolling windows

CODE:
### 1. Install the required libraries (if not already installed)
```python
!pip install yfinance statsmodels matplotlib
```

### 2. Import the necessary libraries

```python
import yfinance as yf
import [Link] as plt
import pandas as pd
from [Link] import seasonal_decompose
from [Link] import ARIMA
from [Link] import acf, pacf
from [Link] import mean_absolute_error, mean_squared_error
import numpy as np
```

### 3. Load and visualize the time series data (Apple stock prices)
```python
# Load stock data from Yahoo Finance
data = [Link]('AAPL', start='2020-01-01', end='2023-01-01', progress=False)
data = data['Close'] # Using the 'Close' price as time series data

# Visualize the time series data

[Link](figsize=(10, 4))
[Link](data, label='AAPL Stock Price')
[Link]('Apple Stock Prices')
[Link]('Date')
[Link]('Close Price')
[Link]()
[Link]()
```

### 4. Perform seasonal decomposition

```python
# Perform seasonal decomposition
decomposition = seasonal_decompose(data, model='additive', period=30)
[Link]()
[Link]()
```

### 5. Implement a Simple Moving Average Model

```python
# Simple Moving Average (SMA)
data['SMA_30'] = [Link](window=30).mean()

# Plot the SMA with the original data

[Link](figsize=(10, 4))
[Link](data['Close'], label='Original')
[Link](data['SMA_30'], label='30-day SMA', color='orange')
[Link]('Simple Moving Average (30 days)')
[Link]('Date')
[Link]('Price')
[Link]()
[Link]()
```

### 6. Apply ARIMA Model for Forecasting

```python
# Fit ARIMA model
model = ARIMA(data, order=(5, 1, 0))
arima_result = [Link]()

# Forecast for the next 30 days

forecast = arima_result.forecast(steps=30)

# Plot the forecast

[Link](figsize=(10, 4))
[Link](data, label='Historical')
[Link](forecast, label='Forecast', color='red')
[Link]('ARIMA Model Forecast')
[Link]('Date')
[Link]('Price')
[Link]()
[Link]()
```
### 7. Evaluate Forecast Performance (using MAE and RMSE)
```python
# Evaluate model performance
train = data[:'2022']
test = data['2022':]

# Predict on test data

model = ARIMA(train, order=(5, 1, 0))
arima_result = [Link]()
forecast = arima_result.forecast(steps=len(test))

# Calculate MAE and RMSE

mae = mean_absolute_error(test, forecast)
rmse = [Link](mean_squared_error(test, forecast))
print(f"Mean Absolute Error: {mae}")
print(f"Root Mean Squared Error: {rmse}")
```

### 8. Plot Autocorrelation and Partial Autocorrelation

```python
from [Link] import plot_acf, plot_pacf

# Autocorrelation
plot_acf([Link](), lags=30)
[Link]()

# Partial Autocorrelation
plot_pacf([Link](), lags=30)
[Link]()
```

### 9. Perform Time Series Cross-Validation using Rolling Windows

```python
# Rolling forecast with ARIMA
window = 30
predictions = []
for i in range(window, len(test)):
train_window = data[:'2022'].iloc[i - window:i]
model = ARIMA(train_window, order=(5, 1, 0))
result = [Link]()
[Link]([Link]()[0])

# Plot rolling predictions vs. actuals

[Link](figsize=(10, 4))
[Link]([Link][window:], test[window:], label='Actual')
[Link]([Link][window:], predictions, label='Rolling Forecast', color='red')
[Link]('Date')
[Link]('Price')
[Link]('Rolling Window Forecast')
[Link]()
[Link]()
```

OUTPUT:

VIVA VOCE:
 10. Q: What is the purpose of seasonal decomposition in time series analysis?
A: Seasonal decomposition separates a time series into its trend, seasonal, and residual
(noise) components. This helps in understanding underlying patterns, like repeating seasonal
effects, which are important for accurate forecasting.

11. Q: What are the main components of the ARIMA model?

A: ARIMA stands for AutoRegressive Integrated Moving Average. It has three components:
o AR (AutoRegressive): Depends on past values.
o I (Integrated): Represents differencing to make the series stationary.
o MA (Moving Average): Relies on past errors.

12. Q: How do you interpret autocorrelation and partial autocorrelation plots?

A: Autocorrelation shows the correlation of a series with its own lagged values, identifying
repeating patterns. Partial autocorrelation measures the correlation of a series with its lags,
excluding the effects of intermediate lags, helping to determine the AR and MA terms in
ARIMA models.

13. Q: What metrics are commonly used to evaluate time series forecasting models?
A: Common metrics include MAE (Mean Absolute Error), which calculates the average
absolute errors, and RMSE (Root Mean Squared Error), which emphasizes larger errors. Both
indicate forecast accuracy.

14. Q: What is the difference between a simple moving average model and ARIMA?
A: A simple moving average (SMA) averages data over a fixed window, smoothing
fluctuations but not adapting to trends or seasonality. ARIMA, however, models both the
trend and seasonality through its AR, I, and MA components, making it more suitable for
complex patterns.
 EXPERIMENT 2

AIM: Clustering Techniques

Apply and evaluate clustering techniques to group similar data points.
1. Implement K-Means clustering and visualize the clusters.
2. Apply Hierarchical Clustering and create a dendrogram.
3. Use DBSCAN (Density-Based Spatial Clustering of Applications with Noise) for
clustering.
4. Perform clustering using Gaussian Mixture Models (GMM).
5. Evaluate clustering performance using metrics like Silhouette Score.
6. Apply clustering algorithms to image data and visualize results.
7. Compare the performance of K-Means and DBSCAN on a dataset.
8. Perform dimensionality reduction before clustering using PCA.
9. Analyze the impact of different values of K in K-Means clustering.
10. Use t-SNE for visualizing high-dimensional clustering results

CODE:

### 1. Install Required Libraries (if not installed)

```python
!pip install matplotlib seaborn scikit-learn scipy
```

### 2. Import Libraries

```python
import numpy as np
import pandas as pd
import [Link] as plt
import seaborn as sns
from [Link] import KMeans, DBSCAN, AgglomerativeClustering
from [Link] import GaussianMixture
from [Link] import silhouette_score
from [Link] import PCA
from [Link] import make_blobs
from [Link] import dendrogram, linkage
from [Link] import TSNE
```

### 3. Generate or Load a Sample Dataset

Let's use a synthetic dataset for simplicity.
```python
# Generate synthetic dataset
data, _ = make_blobs(n_samples=300, centers=4, cluster_std=0.60, random_state=0)
```
### 4. Implement K-Means Clustering and Visualize Clusters
```python
# Apply K-Means clustering
kmeans = KMeans(n_clusters=4, random_state=0)
labels_kmeans = kmeans.fit_predict(data)

# Visualize clusters
[Link](data[:, 0], data[:, 1], c=labels_kmeans, cmap='viridis')
[Link](kmeans.cluster_centers_[:, 0], kmeans.cluster_centers_[:, 1], s=200, c='red',
marker='X')
[Link]('K-Means Clustering')
[Link]()
```

### 5. Apply Hierarchical Clustering and Create a Dendrogram

```python
# Perform Hierarchical Clustering
linked = linkage(data, 'ward')
[Link](figsize=(10, 7))
dendrogram(linked, truncate_mode='lastp', p=12)
[Link]('Hierarchical Clustering Dendrogram')
[Link]()
```

### 6. Use DBSCAN for Clustering

```python
# Apply DBSCAN
dbscan = DBSCAN(eps=0.5, min_samples=5)
labels_dbscan = dbscan.fit_predict(data)

# Visualize clusters
[Link](data[:, 0], data[:, 1], c=labels_dbscan, cmap='plasma')
[Link]('DBSCAN Clustering')
[Link]()
```

### 7. Perform Clustering using Gaussian Mixture Models (GMM)

```python
# Apply Gaussian Mixture Model
gmm = GaussianMixture(n_components=4, random_state=0)
labels_gmm = gmm.fit_predict(data)

# Visualize clusters
[Link](data[:, 0], data[:, 1], c=labels_gmm, cmap='coolwarm')
[Link]('Gaussian Mixture Model Clustering')
[Link]()
```
### 8. Evaluate Clustering Performance Using Silhouette Score
```python
# Calculate silhouette scores
silhouette_kmeans = silhouette_score(data, labels_kmeans)
silhouette_dbscan = silhouette_score(data, labels_dbscan)
silhouette_gmm = silhouette_score(data, labels_gmm)

print(f"Silhouette Score for K-Means: {silhouette_kmeans}")

print(f"Silhouette Score for DBSCAN: {silhouette_dbscan}")
print(f"Silhouette Score for GMM: {silhouette_gmm}")

```

### 9. Apply Clustering Algorithms to Image Data and Visualize Results

For simplicity, let's cluster pixel values in an image. Replace `image_path` with any valid
image URL.

```python
from skimage import io

# Load image data

image = [Link]('[Link]
image = image / 255.0 # Normalize

# Reshape image for clustering

pixels = [Link](-1, 3)

# Apply K-Means to pixels

kmeans_image = KMeans(n_clusters=3, random_state=0).fit(pixels)
segmented_img = kmeans_image.cluster_centers_[kmeans_image.labels_]
segmented_img = segmented_img.reshape([Link])

# Visualize original and segmented images

[Link](figsize=(10, 5))
[Link](1, 2, 1)
[Link](image)
[Link]('Original Image')
[Link]('off')
[Link](1, 2, 2)
[Link](segmented_img)
[Link]('Segmented Image (K-Means)')
[Link]('off')
[Link]()
```

### 10. Compare the Performance of K-Means and DBSCAN on the Dataset
```python
print(f"Silhouette Score for K-Means: {silhouette_kmeans}")
print(f"Silhouette Score for DBSCAN: {silhouette_dbscan}")
# This comparison shows which model fits better on this dataset.
```

### 11. Perform Dimensionality Reduction Using PCA Before Clustering

```python
# Apply PCA
pca = PCA(n_components=2)
data_pca = pca.fit_transform(data)

# Apply K-Means on reduced data

kmeans_pca = KMeans(n_clusters=4, random_state=0)
labels_pca = kmeans_pca.fit_predict(data_pca)

# Visualize
[Link](data_pca[:, 0], data_pca[:, 1], c=labels_pca, cmap='viridis')
[Link]('K-Means Clustering after PCA')
[Link]()
```

### 12. Analyze the Impact of Different Values of K in K-Means Clustering

```python
inertia = []
k_values = range(1, 10)

for k in k_values:
kmeans = KMeans(n_clusters=k, random_state=0)
[Link](data)
[Link](kmeans.inertia_)

[Link](k_values, inertia, marker='o')

[Link]('Number of Clusters K')
[Link]('Inertia')
[Link]('Elbow Method to Determine Optimal K')
[Link]()
```

### 13. Use t-SNE for Visualizing High-Dimensional Clustering Results

```python
# Apply t-SNE
tsne = TSNE(n_components=2, random_state=0)
data_tsne = tsne.fit_transform(data)

# Plot t-SNE results with K-Means labels

[Link](data_tsne[:, 0], data_tsne[:, 1], c=labels_kmeans, cmap='viridis')
[Link]('t-SNE Visualization of K-Means Clusters')
[Link]()

OUTPUT:
VIVA VOCE:

1. Q: What is the main idea behind the K-Means clustering algorithm?

A: K-Means aims to partition data into KKK clusters, where each data point belongs to the
cluster with the nearest mean. It works by iteratively updating centroids and assigning points
to their closest centroid until convergence.

2. Q: How does Hierarchical Clustering differ from K-Means?

A: Hierarchical clustering builds a hierarchy of clusters, often visualized using a dendrogram,
without requiring a predefined number of clusters. K-Means requires a fixed number of
clusters as an input and iteratively optimizes the centroids for those clusters.

3. Q: What is the role of the eps parameter in DBSCAN?

A: In DBSCAN, eps defines the radius within which points must lie to be considered part of
the same cluster. It helps determine the density around each point, which is used to identify
clusters and noise.

4. Q: Why is the Silhouette Score used, and what does it indicate?

A: The Silhouette Score evaluates clustering performance by measuring how similar points
are within the same cluster compared to points in other clusters. Higher scores indicate well-
separated clusters.

5. Q: Why is dimensionality reduction (e.g., PCA) often performed before

clustering?
A: Dimensionality reduction helps reduce the complexity of high-dimensional data, making
clustering algorithms faster and potentially improving clustering quality by eliminating noise
and redundant features.