• Exploratory Data Analysis (EDA) is an approach to analyzing datasets to

summarize their main characteristics, often employing visual methods.

• EDA is a crucial step in the data analysis process, helping analysts
understand the data's underlying structure, detect anomalies, and
formulate hypotheses.
•Key Objectives:
• Discover patterns and relationships in data.
• Identify anomalies or outliers.
• Test assumptions.
• Develop insights for further analysis or modeling.
Importance of EDA
• Data Understanding: Gain a comprehensive understanding of data before
proceeding to modeling.
• Data Quality Assessment: Detect errors, missing values, and
inconsistencies.
• Hypothesis Generation: Formulate questions and hypotheses based on
data patterns.
• Feature Selection: Identify relevant features for predictive modeling.
• Visualization: Present data insights in an interpretable manner.
 Steps in EDA
1. Data Collection
• Sources: CSV files, databases, APIs, web scraping.
• Loading Data: Use pandas.read_csv(), pandas.read_excel(), etc.

import pandas as pd
# Load dataset
df = pd.read_csv('data/your_dataset.csv')

2. Data Cleaning
Handling Missing Values:
• Detection: [Link]().sum()
• Imputation: Fill missing values with mean, median, mode, or use
interpolation.
• Removal: Drop rows or columns with excessive missing data.
# Detect missing values
missing_values = [Link]().sum()
# Fill missing values with median
df['column_name'] =
df['column_name'].fillna(df['column_name'].median())
Handling Duplicates:
• Remove duplicate rows using df.drop_duplicates().

# Remove duplicate rows

df = df.drop_duplicates()
# Detect missing values
missing_values = [Link]().sum()
# Fill missing values with median
df['column_name'] =
df['column_name'].fillna(df['column_name'].median())
Handling Duplicates:
• Remove duplicate rows using df.drop_duplicates().
# Remove duplicate rows
df = df.drop_duplicates()
Correcting Data Types:
• Convert columns to appropriate types using astype().
# Convert column to datetime
df['date_column'] = pd.to_datetime(df['date_column'])
 3. Data Profiling
Descriptive Statistics:
• Summary statistics using [Link]().

# Descriptive statistics
desc_stats = [Link]()
print(desc_stats)

Understanding Data Distribution:

• Use histograms and box plots to understand distributions and detect
outliers.
4. Data Visualization
•Univariate Analysis: Analyzing individual variables.
• Histograms: Distribution of a single variable.
• Box Plots: Distribution and outliers.
import [Link] as plt
import seaborn as sns
# Histogram
[Link](df['numeric_column'], bins=30, color='skyblue')
[Link]('Histogram of Numeric Column')
[Link]('Value')
[Link]('Frequency')
[Link]()
# Box Plot
[Link](x=df['numeric_column'])
[Link]('Box Plot of Numeric Column')
[Link]()
Bivariate Analysis: Relationship between two variables.
• Scatter Plots: Relationship between two numerical variables.
• Bar Plots: Comparison between categorical variables.

# Scatter Plot
[Link](x='feature1', y='feature2', data=df)
[Link]('Feature1 vs Feature2')
[Link]()

# Bar Plot
[Link](x='categorical_feature', y='numeric_feature', data=df)
[Link]('Categorical vs Numeric Feature')
[Link]()
Multivariate Analysis: Relationships among multiple variables.
• Pair Plots: Pairwise relationships in the dataset.
• Heatmaps: Correlation matrices.

# Pair Plot
[Link](df)
[Link]()

# Heatmap of Correlation Matrix

corr = [Link]()
[Link](corr, annot=True, cmap='coolwarm')
[Link]('Correlation Matrix')
[Link]()
5. Feature Engineering
• Creating New Features: Derive new variables from existing ones.
• Encoding Categorical Variables: Convert categorical data into numerical
formats using one­hot encoding or label encoding.
• Scaling and Normalization: Standardize numerical features for modeling.
from [Link] import OneHotEncoder, StandardScaler
# One­Hot Encoding
df_encoded = pd.get_dummies(df, columns=['categorical_feature'])
# Scaling
scaler = StandardScaler()
df_scaled = scaler.fit_transform(df[['numeric_feature1',
'numeric_feature2']])
 Advanced EDA Techniques
Handling High­Dimensional Data
•Dimensionality Reduction: Techniques like PCA (Principal Component
Analysis) to reduce the number of variables.
from [Link] import PCA
pca = PCA(n_components=2)
principal_components = pca.fit_transform(df_scaled)
df_pca = [Link](data=principal_components, columns=['PC1',
'PC2'])
Time Series Analysis
•Trend Analysis: Identifying trends over time.
•Seasonality Detection: Observing seasonal patterns.
# Line Plot for Time Series
[Link](df['date_column'], df['value_column'])
[Link]('Time Series Plot')
[Link]('Date')
[Link]('Value')
[Link]()
Handling Imbalanced Data
•Visualization: Use count plots to visualize class distribution.
•Techniques: Resampling methods like SMOTE.
from imblearn.over_sampling import SMOTE

# SMOTE for oversampling

smote = SMOTE()
X_resampled, y_resampled = smote.fit_resample(X, y)
 Scikit­Learn for Data Analytics
• We will go through real­time exercises that demonstrate how to apply Scikit­
Learn for various data analytics tasks using Python.
Setup & Data Preprocessing
Importing Required Libraries
import pandas as pd
import numpy as np
import seaborn as sns
import [Link] as plt
# Scikit­learn specific imports
from sklearn.model_selection import train_test_split
from [Link] import StandardScaler, LabelEncoder
from [Link] import accuracy_score, confusion_matrix
Loading a Dataset
• let’s use the famous Iris Dataset that comes preloaded in Scikit­learn.

from [Link] import load_iris

# Load the iris dataset
data = load_iris()
df = [Link]([Link], columns=data.feature_names)
df['target'] = [Link]
Data Preprocessing
• Before applying machine learning models, we must preprocess the data.
Handling missing data: Even though the Iris dataset has no missing values, in
real­world datasets, you may encounter missing values.
# Example handling missing values
[Link]([Link](), inplace=True)
Encoding Categorical Data:
• If your dataset has categorical features, you need to convert them into
numerical data using techniques like One­Hot Encoding or Label
Encoding.
le = LabelEncoder()
df['target'] = le.fit_transform(df['target'])
Feature Scaling: Applying scaling to ensure all features contribute equally to
the analysis.
scaler = StandardScaler()
df_scaled = scaler.fit_transform([Link]('target', axis=1))

Train­Test Split: Splitting the dataset into training and testing sets.

X_train, X_test, y_train, y_test = train_test_split(df_scaled, df['target'],

test_size=0.2, random_state=42)
Classification Using Logistic Regression
Model Building
• Let’s build a logistic regression model to classify the species of Iris flowers
based on their features.
from sklearn.linear_model import LogisticRegression
# Initialize the model
log_reg = LogisticRegression()
# Train the model
log_reg.fit(X_train, y_train)
Model Prediction & Evaluation
• After training the model, we evaluate its performance on the test data.
# Make predictions
y_pred = log_reg.predict(X_test)
# Accuracy score
accuracy = accuracy_score(y_test, y_pred)
print(f"Accuracy: {accuracy * 100:.2f}%")
# Confusion matrix
conf_matrix = confusion_matrix(y_test, y_pred)
[Link](conf_matrix, annot=True, fmt="d", cmap="Blues")
[Link]()
Accuracy: 100.00%
Decision Tree Classifier
• Another popular machine learning algorithm is the Decision Tree. Let’s
implement it on the Iris dataset.
Model Building
from [Link] import DecisionTreeClassifier
# Initialize the model
tree_clf = DecisionTreeClassifier()
# Train the model
tree_clf.fit(X_train, y_train)
Model Prediction & Evaluation
# Make predictions
y_pred_tree = tree_clf.predict(X_test)

# Accuracy score
accuracy_tree = accuracy_score(y_test, y_pred_tree)
print(f"Decision Tree Accuracy: {accuracy_tree * 100:.2f}%")

# Confusion Matrix
conf_matrix_tree = confusion_matrix(y_test, y_pred_tree)
[Link](conf_matrix_tree, annot=True, fmt="d", cmap="Greens")
[Link]()
Decision Tree Accuracy: 100.00%
 Visualizing the Decision Tree

from [Link] import plot_tree

[Link](figsize=(12, 8))
plot_tree(tree_clf, filled=True, feature_names=data.feature_names,
class_names=data.target_names)
[Link]()
Hyperparameter Tuning using GridSearchCV
• Hyperparameter tuning is essential for improving model performance. We can perform
hyperparameter optimization using GridSearchCV.
Applying GridSearchCV on a Random Forest Classifier
from sklearn.model_selection import GridSearchCV Output
from [Link] import RandomForestClassifier
# Define the model
Best Parameters:
rf_clf = RandomForestClassifier() {'max_depth': 5,
# Define the hyperparameter grid 'max_features':
param_grid = {
'n_estimators': [10, 50, 100],
'sqrt',
'max_depth': [5, 10, None], 'n_estimators':
'max_features': ['sqrt', 'log2', None] 100} Random Forest
}
# Initialize GridSearchCV
Accuracy after
grid_search = GridSearchCV(estimator=rf_clf, param_grid=param_grid, cv=5) Hyperparameter
# Train with GridSearchCV Tuning: 100.00%
grid_search.fit(X_train, y_train)
# Best parameters
print("Best Parameters:", grid_search.best_params_)
# Predict with the best estimator
y_pred_rf = grid_search.best_estimator_.predict(X_test)
accuracy_rf = accuracy_score(y_test, y_pred_rf)
t(f"Random ft { acy_rf * 2f}%")
Feature Importance in Random Forest
• Once we train the Random Forest model, we can also look at the feature
importance to understand which features contribute the most to the
predictions.
# Extract feature importances
importances = grid_search.best_estimator_.feature_importances_
# Plot feature importances
[Link](x=importances, y=data.feature_names)
[Link]("Feature Importance in Random Forest")
[Link]()
Cross­Validation for Model Evaluation
• Instead of a single train­test split, we can use cross­validation to evaluate
the model’s performance more reliably.
Cross­Validation with Logistic Regression
from sklearn.model_selection import cross_val_score
# Perform 5­fold cross­validation
cross_val_scores = cross_val_score(log_reg, df_scaled, df['target'], cv=5)
print("Cross­Validation Scores:", cross_val_scores)
print("Mean Cross­Validation Score:", [Link](cross_val_scores))

Output:
Cross-Validation Scores: [0.96666667 1. 0.93333333
0.9 1. ] Mean Cross-Validation Score:
0.9600000000000002
Clustering with K­Means
• Clustering is an unsupervised learning technique. We’ll apply K­Means
clustering to group the data points into clusters.
K­Means Model Building
from [Link] import KMeans
# Initialize KMeans with 3 clusters
kmeans = KMeans(n_clusters=3, random_state=42)
# Fit the model
[Link](df_scaled)
# Cluster centers
print("Cluster Centers:", kmeans.cluster_centers_)
# Predicted clusters
clusters = [Link](df_scaled)
# Plot clusters
[Link](df_scaled[:, 0], df_scaled[:, 1], c=clusters, cmap='viridis')
[Link]()
Cluster Centers: [[ 0.57100359 -0.37176778
0.69111943 0.66315198] [-0.81623084 1.31895771 -
1.28683379 -1.2197118 ] [-1.32765367 -0.373138 -
1.13723572 -1.11486192]]
 Pipelines in Scikit­Learn
• A Pipeline helps automate workflows by chaining multiple steps like
preprocessing and modeling.
Creating a Pipeline for Logistic Regression
from [Link] import Pipeline Output:
# Define the pipeline Pipeline
pipeline = Pipeline([ Accuracy:
('scaler', StandardScaler()), 100.00%
('log_reg', LogisticRegression())
])
# Train the pipeline
[Link](X_train, y_train)
# Make predictions
y_pred_pipe = [Link](X_test)
accuracy_pipe = accuracy_score(y_test, y_pred_pipe)
print(f"Pipeline Accuracy: {accuracy_pipe * 100:.2f}%")