UNIT-II

Exploratory Data Analysis (EDA)

Exploratory Data Analysis (EDA) is a crucial step in the data science pipeline. It involves
examining a dataset to summarize its key characteristics, often with visual methods. The goal
is to gain an understanding of the dataset, discover patterns, detect anomalies, test
assumptions, and check for data quality issues. In Python, EDA is performed using libraries
like Pandas, Matplotlib, Seaborn, and NumPy.

Steps for EDA in Data Science with Python

1. Load the Data

o The first step is loading your data into a Pandas DataFrame.
o You can load data from CSV, Excel, SQL databases, or other formats.

Example:

import pandas as pd
import numpy as np
import [Link] as plt
import seaborn as sns
# Load the dataset (Seaborn provides built-in datasets)
df = sns.load_dataset('iris')
print(df)

Output:

2. Data Cleaning
o Data might have missing values, duplicates, or inconsistent data types.
Handling these is essential before proceeding with analysis.
o You can fill missing values, drop rows with missing values, or convert data
types as necessary.

Example:

Prepared by: B. Sai Hemanth,

Assistant Professor, Dept. Of MCA
 import pandas as pd
import numpy as np
import [Link] as plt
import seaborn as sns
# Load the dataset (Seaborn provides built-in datasets)
df = sns.load_dataset('iris')
# 1. Overview of the Data
print("First 5 rows of the dataset:")
print([Link]())

3. Data Transformation
o This step includes normalizing data, encoding categorical variables, and
creating new features if necessary.
4. Descriptive Statistics
o Understanding central tendency (mean, median, mode) and spread (variance,
standard deviation) of the data is important.
5. Data Visualization
o Visualization helps to explore the dataset and understand the distribution,
relationships, and patterns in the data.
o Common plots include histograms, scatter plots, box plots, and correlation
heatmaps.
6. Correlation Analysis
o Understanding the relationship between different variables can help identify
dependencies, correlations, or multicollinearity.
7. Outlier Detection
o Identifying and handling outliers ensures your data is clean for further
modeling.
8. Feature Engineering
o You may create new features or modify existing ones to make your model
more effective later on.

[Link] Science Life Cycle

The Data Science Life Cycle is a structured approach that guides data scientists through the
steps of a project. It spans from the initial problem definition to the final deployment of a
model or system. The life cycle can vary slightly depending on the project or organization,
but generally, it consists of the following stages:

Visual Representation of the Data Science Life Cycle

1. Problem Definition

 Objective: The first step in any data science project is to define the problem clearly
and understand the goals of the business or stakeholders.
 Tasks:
Prepared by: B. Sai Hemanth,
Assistant Professor, Dept. Of MCA
 o Understand the business context and problem domain.
o Define the project goals and metrics for success.
o Develop a hypothesis or set of questions that the analysis will attempt to
answer.
 Example: A company wants to predict customer churn to improve retention strategies.

2. Data Collection

 Objective: Gather the relevant data from multiple sources. This step involves
identifying and collecting data that can answer the problem at hand.
 Tasks:
o Identify sources of data (internal systems, public datasets, APIs, etc.).
o Determine the necessary features (variables) to solve the problem.
o Gather data from databases, spreadsheets, APIs, web scraping, or sensors.
 Example: For the customer churn prediction problem, data might include customer
demographics, transaction history, usage patterns, customer service interactions, etc.

3. Data Cleaning and Preprocessing

 Objective: Raw data is often messy and incomplete. The goal is to clean and
transform the data into a format suitable for analysis.
 Tasks:
o Handling Missing Data: Fill, interpolate, or drop missing values.
o Outlier Detection: Identify and remove outliers that could skew analysis.
o Data Transformation: Convert data types, normalize/scale features, encode
categorical variables (e.g., one-hot encoding).
o Data Integration: Combine data from different sources, if necessary.
 Example: If some rows in the customer data are missing churn labels, you could
either impute the missing values or exclude the rows.

4. Exploratory Data Analysis (EDA)

 Objective: Explore the dataset through visualizations and summary statistics to

understand patterns, relationships, and the general structure of the data.
 Tasks:
o Visualize the data using histograms, scatter plots, box plots, and heatmaps.
o Calculate basic statistics like mean, median, standard deviation, etc.
o Look for correlations, trends, and potential outliers.
o Identify the distribution of features (normal, skewed, etc.).
 Example: You could visualize the distribution of the customer ages, income, and
usage frequency to understand customer behavior.

5. Feature Engineering

 Objective: Create new features or modify existing ones to improve the performance
of the model.
 Tasks:
Prepared by: B. Sai Hemanth,
Assistant Professor, Dept. Of MCA
 o Feature Creation: Create new variables from existing ones (e.g., creating an
"Age Group" feature from the "Age" column).
o Feature Transformation: Apply transformations like scaling or logarithmic
transformations.
o Feature Selection: Identify which features are most important for the model,
and drop irrelevant ones.
 Example: You might create a new feature called "Average Purchase Value" based on
the frequency and value of customer transactions.

6. Model Building

 Objective: Train machine learning or statistical models to make predictions or

classify data.
 Tasks:
o Choose appropriate machine learning algorithms (e.g., regression,
classification, clustering).
o Split the data into training and testing sets (often 70% train, 30% test).
o Train the model on the training set and evaluate its performance using
validation techniques (e.g., cross-validation).
o Tune the hyperparameters to optimize model performance.
 Example: If predicting customer churn, you might use algorithms like Logistic
Regression, Random Forests, or XGBoost.

7. Model Evaluation

 Objective: Assess the performance of the model using appropriate metrics and test
data.
 Tasks:
o Use performance metrics relevant to the problem (e.g., accuracy, precision,
recall, F1-score for classification, RMSE for regression).
o Compare model performance against a baseline or previous models.
o Visualize the performance of the model using confusion matrices, ROC curves,
or other relevant plots.
o Check for overfitting or underfitting.
 Example: In customer churn prediction, you would assess how well the model
identifies customers who are likely to churn using accuracy, recall, and precision
metrics.

8. Model Tuning and Optimization

 Objective: Improve the model's performance through fine-tuning and optimizations.

 Tasks:
o Hyperparameter Tuning: Use techniques like grid search or random search
to find the optimal hyperparameters for your model.
o Feature Tuning: Revisit feature engineering and transformation steps to
create more impactful features.
o Ensemble Methods: Combine multiple models to improve performance (e.g.,
bagging, boosting).

Prepared by: B. Sai Hemanth,

Assistant Professor, Dept. Of MCA
  Example: Fine-tune the hyperparameters of a Random Forest model or an XGBoost
model to improve its performance.

9. Model Deployment

 Objective: Deploy the trained model into a production environment, where it can start
making predictions on new, unseen data.
 Tasks:
o Integrate the model into the existing application or system (e.g., a web app, a
recommendation engine, or a chatbot).
o Set up the model to make real-time predictions or batch predictions on new
data.
o Monitor model performance over time and check if it degrades (concept drift).
o Automate the model update process (e.g., retraining with new data).
 Example: The customer churn prediction model is deployed as part of a business
dashboard that alerts managers when a customer is at risk of churning.

10. Model Monitoring and Maintenance

 Objective: Ensure that the model continues to perform well and make updates when
necessary.
 Tasks:
o Monitoring: Track model performance in production over time to detect any
degradation or unexpected behavior.
o Feedback Loop: Collect user feedback or results to retrain the model with
new data.
o Model Retraining: Periodically retrain the model to ensure it stays up-to-date
with new data patterns.
 Example: If the customer churn model starts to perform poorly due to seasonal
changes or shifts in customer behavior, it may need to be retrained with new data.

Descriptive Statistics in Data Science with Python

Descriptive statistics is the foundation of data analysis, providing an overview of the data's
main characteristics through summary measures such as central tendency, dispersion, and
shape of the distribution. In data science, descriptive statistics help us understand the basic
properties of the dataset, identify patterns, and detect anomalies or outliers before proceeding
to more complex analysis or model building.

In Python, libraries like Pandas, NumPy, and SciPy are commonly used to perform
descriptive statistical analysis. Visualization libraries such as Matplotlib and Seaborn help
in presenting the results in graphical form.

Key Descriptive Statistics Measures

1. Measures of Central Tendency: These measures describe the center of a data

distribution.
o Mean: The average of all values.
o Median: The middle value in the data when it is sorted.

Prepared by: B. Sai Hemanth,

Assistant Professor, Dept. Of MCA
 o Mode: The most frequent value(s) in the dataset.
2. Measures of Dispersion: These describe the spread or variability of the data.
o Range: The difference between the maximum and minimum values.
o Variance: The average of squared differences from the mean, measuring the
data’s spread.
o Standard Deviation: The square root of variance, providing a more
interpretable measure of spread.
o Interquartile Range (IQR): The range between the 25th percentile (Q1) and
75th percentile (Q3), representing the middle 50% of the data.
3. Shape of the Distribution: These measures describe the symmetry or asymmetry of
the data.
o Skewness: Measures the asymmetry of the data distribution.
o Kurtosis: Measures the "tailedness" or extreme values of the distribution.

Steps to Calculate Descriptive Statistics in Python

1. Import Libraries

We will primarily use Pandas and NumPy for descriptive statistics and Matplotlib or
Seaborn for visualizations.

import pandas as pd
import numpy as np
import [Link] as plt
import seaborn as sns
2. Load a Sample Dataset

For this example, we'll use the Iris dataset, which is built into the Seaborn library. You can
use any dataset for this analysis.

# Load the Iris dataset

df = sns.load_dataset('iris')
3. Measures of Central Tendency

Mean, Median, and Mode can be easily calculated using Pandas:

# Mean
mean = df['sepal_length'].mean()
print("Mean:", mean)

# Median
median = df['sepal_length'].median()
print("Median:", median)

# Mode
mode = df['sepal_length'].mode()[0] # Returns the first mode if there are multiple modes
print("Mode:", mode)

Prepared by: B. Sai Hemanth,

Assistant Professor, Dept. Of MCA
4. Measures of Dispersion

 Range: The difference between the max and min values.

 Variance and Standard Deviation: var() and std() methods in Pandas.

# Range
range_value = df['sepal_length'].max() - df['sepal_length'].min()
print("Range:", range_value)

# Variance
variance = df['sepal_length'].var()
print("Variance:", variance)

# Standard Deviation
std_dev = df['sepal_length'].std()
print("Standard Deviation:", std_dev)

# Interquartile Range (IQR)

iqr = df['sepal_length'].quantile(0.75) - df['sepal_length'].quantile(0.25)
print("IQR:", iqr)
5. Shape of Distribution

 Skewness and Kurtosis can be calculated using SciPy:

from [Link] import skew, kurtosis

# Skewness
skewness = skew(df['sepal_length'])
print("Skewness:", skewness)

# Kurtosis
kurt = kurtosis(df['sepal_length'])
print("Kurtosis:", kurt)
6. Summary of Descriptive Statistics

Pandas provides the describe() method, which calculates a quick summary of all numerical
columns:

# Summary Statistics for all numeric columns

summary_stats = [Link]()
print(summary_stats)
Visualization of Descriptive Statistics

Visualizations are important in descriptive statistics as they help to quickly identify the
underlying patterns, distributions, and outliers in the data.

Prepared by: B. Sai Hemanth,

Assistant Professor, Dept. Of MCA
 1. Histogram: Shows the frequency distribution of a variable.

[Link](figsize=(8, 6))
[Link](df['sepal_length'], kde=True, bins=20)
[Link]('Distribution of Sepal Length')
[Link]('Sepal Length (cm)')
[Link]('Frequency')
[Link]()

2. Box Plot: Visualizes the spread of the data, highlighting the median, quartiles, and
potential outliers.

[Link](figsize=(8, 6))
[Link](x=df['sepal_length'])
[Link]('Boxplot of Sepal Length')
[Link]('Sepal Length (cm)')
[Link]()

3. Pairplot: Visualizes the pairwise relationships and distributions between multiple

variables.

[Link](df)
[Link]()

Example: Full Code for Descriptive Statistics in Python

import pandas as pd
import numpy as np
import seaborn as sns
import [Link] as plt
from [Link] import skew, kurtosis

# Load the Iris dataset

df = sns.load_dataset('iris')

# 1. Measures of Central Tendency

mean = df['sepal_length'].mean()
median = df['sepal_length'].median()
mode = df['sepal_length'].mode()[0]

print("Mean:", mean)
print("Median:", median)
print("Mode:", mode)

# 2. Measures of Dispersion
range_value = df['sepal_length'].max() - df['sepal_length'].min()
variance = df['sepal_length'].var()
std_dev = df['sepal_length'].std()
iqr = df['sepal_length'].quantile(0.75) - df['sepal_length'].quantile(0.25)
Prepared by: B. Sai Hemanth,
Assistant Professor, Dept. Of MCA
print("Range:", range_value)
print("Variance:", variance)
print("Standard Deviation:", std_dev)
print("IQR:", iqr)

# 3. Shape of Distribution
skewness = skew(df['sepal_length'])
kurt = kurtosis(df['sepal_length'])

print("Skewness:", skewness)
print("Kurtosis:", kurt)

# 4. Summary Statistics
summary_stats = [Link]()
print(summary_stats)

# 5. Visualization
# Histogram
[Link](figsize=(8, 6))
[Link](df['sepal_length'], kde=True, bins=20)
[Link]('Distribution of Sepal Length')
[Link]('Sepal Length (cm)')
[Link]('Frequency')
[Link]()

# Boxplot
[Link](figsize=(8, 6))
[Link](x=df['sepal_length'])
[Link]('Boxplot of Sepal Length')
[Link]('Sepal Length (cm)')
[Link]()

# Pairplot
[Link](df)
[Link]()

Basic Tools in Data Science with Python

Python is one of the most popular programming languages in the data science field because
of its simplicity, flexibility, and the vast ecosystem of libraries and tools that it provides.
These tools cover various aspects of the data science process, from data collection to model
building and evaluation. Below is an overview of the basic tools and libraries in Python
commonly used for data science.

1. Data Manipulation and Analysis Tools

These libraries are used for handling, cleaning, transforming, and analyzing data.

Prepared by: B. Sai Hemanth,

Assistant Professor, Dept. Of MCA
1.1. Pandas

 Purpose: Pandas is the most widely used library for data manipulation and analysis in
Python. It provides powerful data structures like DataFrame and Series to handle
structured data.
 Key Functions:
o DataFrames: Tabular data with rows and columns.
o Series: One-dimensional labeled arrays.
o Common operations: Importing data, filtering, grouping, aggregating,
merging, reshaping, and handling missing data.
 Example:

import pandas as pd

# Creating a DataFrame
data = {'Name': ['Alice', 'Bob', 'Charlie'],
'Age': [25, 30, 35],
'Salary': [50000, 60000, 70000]}
df = [Link](data)

# Basic operations
print([Link]()) # First few rows
print([Link]()) # Summary statistics
1.2. NumPy

 Purpose: NumPy is a powerful library for numerical computing in Python. It provides

support for arrays, matrices, and high-level mathematical functions.
 Key Functions:
o Arrays: NumPy arrays (ndarray) allow you to store large datasets in a more
efficient and faster way than traditional Python lists.
o Mathematical operations: NumPy provides functions for mathematical and
statistical operations such as sum, mean, dot products, and more.
 Example:

import numpy as np

# Creating an array
arr = [Link]([1, 2, 3, 4, 5])

# Operations on arrays
print([Link](arr)) # Mean
print([Link](arr, arr)) # Dot product

2. Data Visualization Tools

Visualization is crucial in data science to interpret and present data insights clearly. Python
offers powerful libraries for creating various types of plots.
Prepared by: B. Sai Hemanth,
Assistant Professor, Dept. Of MCA
2.1. Matplotlib

 Purpose: Matplotlib is the most widely used library for creating static, interactive,
and animated plots and visualizations in Python.
 Key Functions:
o Create basic plots such as line, bar, and scatter plots.
o Customize charts with titles, labels, and legends.
 Example:

import [Link] as plt

# Simple line plot

x = [1, 2, 3, 4, 5]
y = [1, 4, 9, 16, 25]
[Link](x, y)
[Link]('Simple Line Plot')
[Link]('X Axis')
[Link]('Y Axis')
[Link]()
2.2. Seaborn

 Purpose: Seaborn is built on top of Matplotlib and offers a high-level interface for
creating attractive and informative statistical graphics.
 Key Features:
o Built-in themes for aesthetics.
o Advanced plots like box plots, violin plots, pair plots, and heatmaps.
o Works seamlessly with Pandas DataFrames.
 Example:

import seaborn as sns

# Loading the Iris dataset

iris = sns.load_dataset('iris')

# Scatter plot
[Link](data=iris, x='sepal_length', y='sepal_width', hue='species')
[Link]()

3. Machine Learning Libraries

These libraries help data scientists build and evaluate machine learning models.

3.1. Scikit-learn

 Purpose: Scikit-learn is the most popular library for machine learning in Python. It
provides a simple and efficient way to implement various machine learning
algorithms and tools for data preprocessing, model evaluation, and optimization.
 Key Features:
Prepared by: B. Sai Hemanth,
Assistant Professor, Dept. Of MCA
 oClassification, regression, and clustering algorithms (e.g., Logistic Regression,
Decision Trees, K-Means, etc.).
o Tools for model evaluation (e.g., cross-validation, confusion matrix, and
performance metrics).
o Preprocessing tools (e.g., scaling, encoding categorical variables).
 Example:

from [Link] import load_iris

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from [Link] import accuracy_score

# Load dataset
iris = load_iris()
X = [Link]
y = [Link]

# Train-test split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3,
random_state=42)

# Model training
model = LogisticRegression(max_iter=200)
[Link](X_train, y_train)

# Prediction and evaluation

y_pred = [Link](X_test)
print("Accuracy:", accuracy_score(y_test, y_pred))
3.2. TensorFlow & Keras

 Purpose: TensorFlow (with Keras as a high-level API) is an open-source library for

deep learning and neural networks. It is widely used for training complex machine
learning models, especially in deep learning (e.g., CNNs, RNNs).
 Key Features:
o Supports both CPU and GPU computation.
o Helps build neural networks with layers, activations, and loss functions.
o Pretrained models and easy model deployment.
 Example:

import tensorflow as tf
from [Link] import Sequential
from [Link] import Dense

# Define the model

model = Sequential()
[Link](Dense(64, activation='relu', input_shape=(X_train.shape[1],)))
[Link](Dense(3, activation='softmax')) # 3 output classes (Iris species)

Prepared by: B. Sai Hemanth,

Assistant Professor, Dept. Of MCA
 # Compile the model
[Link](optimizer='adam', loss='sparse_categorical_crossentropy',
metrics=['accuracy'])

# Train the model

[Link](X_train, y_train, epochs=50, batch_size=32)

# Evaluate the model

loss, accuracy = [Link](X_test, y_test)
print("Accuracy:", accuracy)

4. Data Cleaning and Preprocessing Tools

Data cleaning and preprocessing are crucial in real-world projects, as raw data is often noisy
and messy.

4.1. Pandas (Data Preprocessing)

Pandas offers several functions to clean and preprocess data:

 Handling missing values: fillna(), dropna()

 Encoding categorical data: get_dummies()
 Feature scaling: StandardScaler from scikit-learn

4.2. OpenCV

 Purpose: OpenCV is a computer vision library used for image processing, feature
detection, and visual analysis.
 Example:

import cv2

# Read an image
image = [Link]('[Link]')

# Convert to grayscale
gray_image = [Link](image, cv2.COLOR_BGR2GRAY)

# Show the image

[Link]('Grayscale Image', gray_image)
[Link](0)
[Link]()

5. Statistical Tools

Python provides several libraries for statistical analysis, which are essential for hypothesis
testing, statistical modeling, and exploratory data analysis.

Prepared by: B. Sai Hemanth,

Assistant Professor, Dept. Of MCA
5.1. SciPy

 Purpose: SciPy is used for scientific and technical computing. It includes functions
for optimization, statistics, integration, and linear algebra.
 Key Functions:
o Statistical functions: Mean, variance, normality tests, etc.
o Optimization: Find the minimum of a function.
 Example:

from scipy import stats

Data visualization
Data visualization is a crucial aspect of data analysis in data science. Python provides various
libraries for creating different types of visualizations such as scatter plots, bar charts,
histograms, boxplots, and heatmaps. Below is an overview of each visualization type and an
example of how to create them in Python using libraries like matplotlib, seaborn, and
pandas.

1. Scatter Plot

A scatter plot is used to visualize the relationship between two continuous variables. It is
helpful to identify correlations or patterns between them.

Example:

import [Link] as plt

import seaborn as sns
import pandas as pd

# Sample data

data = {'x': [1, 2, 3, 4, 5], 'y': [5, 4, 3, 2, 1]}

df = [Link](data)

# Scatter plot using matplotlib

[Link](df['x'], df['y'])

[Link]('Scatter Plot')

[Link]('X')

[Link]('Y')

[Link]()

Prepared by: B. Sai Hemanth,

Assistant Professor, Dept. Of MCA
# Scatter plot using seaborn

[Link](data=df, x='x', y='y')

[Link]('Scatter Plot with Seaborn')

[Link]()

2. Bar Chart

Bar charts are used to compare discrete categories. It helps visualize categorical data,
showing the size of each category.

Example:

# Sample data

categories = ['A', 'B', 'C', 'D']

values = [10, 20, 30, 40]

# Bar chart using matplotlib

[Link](categories, values)

[Link]('Bar Chart')

[Link]('Categories')

[Link]('Values')

[Link]()

# Bar chart using seaborn

[Link](x=categories, y=values)

[Link]('Bar Chart with Seaborn')

[Link]()

3. Histogram

A histogram shows the distribution of a single continuous variable, divided into bins. It's
useful for understanding the frequency distribution of data.

Example:

import numpy as np

# Sample data
Prepared by: B. Sai Hemanth,
Assistant Professor, Dept. Of MCA
data = [Link](1000)

# Histogram using matplotlib

[Link](data, bins=30, edgecolor='black')

[Link]('Histogram')

[Link]('Values')

[Link]('Frequency')

[Link]()

# Histogram using seaborn

[Link](data, kde=True) # kde=True adds a Kernel Density Estimate

[Link]('Histogram with Seaborn')

[Link]()

4. Boxplot

A boxplot displays the distribution of data based on five summary statistics: minimum, first
quartile, median, third quartile, and maximum. It's great for detecting outliers.

Example:

# Sample data

data = [Link](100)

# Boxplot using matplotlib

[Link](data)

[Link]('Boxplot')

[Link]('Values')

[Link]()

# Boxplot using seaborn

[Link](data=data)

[Link]('Boxplot with Seaborn')

[Link]()

Prepared by: B. Sai Hemanth,

Assistant Professor, Dept. Of MCA
5. Heatmap

A heatmap is used to represent data in matrix form, where individual values are represented
by different colors. It's especially useful for showing correlations between multiple variables
or visualizing large data sets like a correlation matrix.

Example:

# Sample data

data = [Link](10, 12) # 10x12 matrix of random numbers

# Heatmap using seaborn

[Link](data, annot=True, cmap='coolwarm', linewidths=0.5)

[Link]('Heatmap with Seaborn')

[Link]()

Prepared by: B. Sai Hemanth,

Assistant Professor, Dept. Of MCA