Advance Data Science

Syllabus
UNIT I: Data Science Fundamentals
Introduction to data science, PYTHON libraries for data science, data, types of
variables continuous and discrete variables, Data sampling, Pandas Data
Structures. Statistics for Data Science, Data visualization, Matplotlib,
Histograms, line charts, pie charts, multiple bar graphs, box plots, scatter plots,
SEABORN Plots, BOKEH plots

UNIT II: Data Collection and Pre-Processing

Data Collection, Data Cleaning, Data munging, Web Scrapping, Rescaling and
dimensionality reduction, Feature Selection, Feature Extraction, Principal
component analysis and Linear Discriminant Analysis
UNIT III: Supervised Machine Learning
Understanding a decision tree, creating trees for different purposes, working with a Random
Forest classifier and Random Forest regressor, Optimizing Random Forest, Boosting
Predictions: Setting a gradient boosting classifier and Running a gradient boosting regressor
Using GBM hyperparameters, Understanding Support Vector Machines, classifying with SVC,
Performing regression with SVR.

UNIT IV : Unsupervised Machine Learning

Clustering with K means, understanding centroid based algorithms, performing Hierarchical
Clustering, Moving Beyond the Round-Shaped Clusters: DBScan. Considering Detection of
Outliers, understanding anomalies and novel data, examining a Simple Univariate Method,
developing a Multivariate Approach, Automating outliers’ detection with SVM.

UNIT V: Reinforcement Learning

Introduction to RL terminology, Markov property, Markov chains, Markov reward process
(MRP), Bellman equations in MRP, Markov decision process (MDP), state and action value
functions, Bellman expectation equations, optimality of value functions and policies, Bellman
 Aim of the Course
To gain hands-on experience in data wrangling, cleaning, and exploration techniques.

To use Python libraries for data visualization and statistical analysis.

To implement supervised and unsupervised machine learning algorithms.

To understand and apply reinforcement learning concepts.

To work with real-world datasets for deriving actionable insights.

 This Session is Designed To:
Introduce key concepts of Supervised Learning / Data Visualization /
Feature Engineering

Explain the theoretical foundations behind the techniques discussed

Demonstrate practical implementation using Python and relevant

libraries

hands-on coding exercises or case studies

Analyze real-world datasets to extract insights and draw conclusions

 Learning Outcomes
Explain fundamental concepts of data science, including data types and
visualization techniques.

Perform data collection, cleaning, transformation, and

dimensionality reduction.

Develop and evaluate machine learning models for classification

and regression.

Apply unsupervised learning techniques such as clustering and

anomaly detection.
UNIT I: Data Science Fundamentals
 Syllabus: UNIT I: Data Science Fundamentals
• Introduction to data science, • Matplotlib,
• PYTHON libraries for data • Histograms,
science, • line charts,
• data, types of variables • pie charts,
continuous and discrete
variables, • multiple bar graphs,
• Data sampling, • box plots,
• Pandas Data Structures. • scatter plots,
• Statistics for Data Science, • SEABORN Plots,
• Data visualization, • BOKEH plots
 Key Concepts

I. Introduction of Data Science

II. Applications of Data Science

[Link] science Life Cycle

[Link] Science tools and technologies

V. Advantages and Disadvantages

The slides presented here are obtained from the authors of
prescribed textbooks, references and from various other
contributors. We hereby acknowledge all the contributors for their
material and input. We have modified and added slides to suit the
requirements of the course.
 Key Concepts
I. Introduction of Data Science

II. Applications of Data Science

[Link] science Life Cycle

[Link] Science tools and technologies

V. Advantages and Disadvantages

Introduction to Data Science
 What is Data Science?
➢Data science involves using data to answer questions and extract insights and
knowledge from various sources.
 What is Data Science?
➢Data science involves using data to answer questions and extract insights and
knowledge from various sources.
➢Data science uncovers actionable insights hidden in an organization’s data.
These insights can be used to guide decision making and strategic planning.
 What is Data Science?
➢Data science involves using data to answer questions and extract insights and
knowledge from various sources.
➢Data science uncovers actionable insights hidden in an organization’s data.
These insights can be used to guide decision making and strategic planning.
➢It combines disciplines such as computer science, statistics, advanced
analytics, artificial intelligence (AI), Machine learning (ML) and domain
expertise to analyze and interpret large and complex datasets, ultimately
informing decision-making and solving real-world problems.
 What is Data Science?
➢Data science involves using data to answer questions and extract insights and
knowledge from various sources.
➢Data science uncovers actionable insights hidden in an organization’s data.
These insights can be used to guide decision making and strategic planning.
➢It combines disciplines such as computer science, statistics, advanced
analytics, artificial intelligence (AI), Machine learning (ML) and domain
expertise to analyze and interpret large and complex datasets, ultimately
informing decision-making and solving real-world problems.
The term “Data scientist” was coined as recently as 2008 when companies
realized the need for data professionals who are skilled in organizing and
analyzing massive amounts of data.
 What is Data Science?
➢Data science starts with data, which can range from a simple array of a few
numeric observations to a complex matrix of millions of observations with
thousands of variables.
 What is Data Science?
➢Data science starts with data, which can range from a simple array of a few
numeric observations to a complex matrix of millions of observations with
thousands of variables.

➢Data science utilizes certain specialized computational methods in order to

discover meaningful and useful structures within a dataset.
 What is Data Science?
➢Data science starts with data, which can range from a simple array of a few
numeric observations to a complex matrix of millions of observations with
thousands of variables.

➢Data science utilizes certain specialized computational methods in order to

discover meaningful and useful structures within a dataset.

➢The discipline of data science coexists and is closely associated with a

number of related areas such as database systems, data engineering,
visualization, data analysis, experimentation, and business intelligence (BI).
DATA SCIENCE VOCABULARY
AI, MACHINE LEARNING, AND
DATA SCIENCE
Traditional Program and Machine Learning
Building Representative Models

• In statistics, a model is the representation of a

relationship between variables in a dataset. It
describes how one or more variables in the data are
related to other variables.

• Modeling is a process in which a representative

abstraction is built from the observed dataset. For
example, based on credit score, income level, and
requested loan amount, a model can be developed to
determine the interest rate of a loan.
 Introduction to Data Science

Data Science is a multidisciplinary field to extract knowledge and insights from data

Categories of Data:

STRUCTURED DATA - WELL SEMI-STRUCTURED DATA - A UNSTRUCTURED DATA – IT

ORGANIZED DATA IN THE STRUCTURED DATA THAT IS REQUIRES ADVANCE TOOLS
FORM OF TABLES UNORGANISED. AND SOFTWARE’S TO ACCESS
INFORMATION
 Introduction to Data Science

Diversity of data brings in new challenges such as handling special structures

e.g., sequences, trees, graphs, and networks and
specific semantics such as ordering, image, audio and video contents, and connectivity
 Introduction to Data Science
Data can be
▪ Time-related or sequence data
e. g: historical records, stock exchange data, and timeseries and biological sequence data.
▪ Data streams
e.g: video surveillance and sensor data, which are continuously transmitted
▪ spatial data
e.g: maps
▪ Engineering design data
e.g: the design of buildings, system components, or integrated circuits
▪ Hypertext and multimedia data -- e g: text, image, video, and audio data)
▪ graph and networked data ----e.g., social and information networks and
▪ the Web---a widely distributed information repository.
 Applications of Data Science

USE CASES
 Real world applications
• Financial Modeling
• Business Analytics
• Risk Management
• Market Segmentation
• Fraud Detection
• Customer Churn Prediction
• Algorithmic Trading
• Sales Forecasting
• Healthcare Informatics
• Social Network Analysis
• Disease Diagnosis
• Community Detection
• Patient Outcome Prediction
• Sentiment Analysis
• Drug Discovery.
• Influence Mapping
 Applications of Data Science in HR Analytics
• Talent Acquisition
• Predictive modeling for candidate sourcing
• Resume parsing and screening automation
• Behavioral analysis for cultural fit assessment
• Employee Retention
• Attrition prediction using machine learning algorithms
• Identifying factors contributing to employee turnover
• Personalized retention strategies based on risk assessment
• Performance Management
• Predictive analytics for performance forecasting
• Feedback sentiment analysis for performance reviews
• Individualized development plans based on skill gap analysis
 Applications of Data Science in HR Analytics

• Workforce Planning
• Demand forecasting for future workforce needs
• Skills inventory analysis for succession planning
• Optimal staffing levels based on workload prediction
• Employee Engagement
• Sentiment analysis of employee feedback
• Engagement score prediction using NLP techniques
• Customized engagement initiatives based on sentiment analysis results
• Diversity and Inclusion
• Bias detection in recruitment and promotion processes
• Diversity metrics tracking and reporting
• Personalized inclusion programs to address diversity gaps
 Applications of Data Science in Finance
• Risk Management
• Predictive modeling for credit risk assessment
• Fraud detection using anomaly detection algorithms
• Market risk analysis and stress testing
• Algorithmic Trading
• Automated trading strategies based on machine learning
• Sentiment analysis of news and social media for trading signals
• High-frequency trading optimization
• Customer Segmentation
• Behavioral analysis for personalized marketing campaigns
• Churn prediction to retain high-value customers
• Customer lifetime value prediction for targeted acquisition
 Applications of Data Science in Finance

• Portfolio Optimization
• Asset allocation using modern portfolio theory
• Risk-return optimization with Monte Carlo simulations
• Dynamic rebalancing based on market trends and portfolio goals
• Fraud Detection
• Transaction monitoring for detecting fraudulent activities
• Pattern recognition in financial transactions
• Network analysis to uncover complex fraud schemes
• Regulatory Compliance
• Anti-money laundering (AML) compliance using predictive analytics
• GDPR and CCPA compliance through data governance frameworks
• Automated reporting for regulatory filings
 APPLICATIONS OF DATA SCIENCE IN BANKING
• Credit Scoring
• Predictive modeling for assessing creditworthiness
• Automated loan approval processes
• Risk-based pricing strategies
• Fraud Detection
• Anomaly detection algorithms for identifying suspicious activities
• Real-time transaction monitoring
• Behavioral analysis for fraud pattern recognition
• Customer Relationship Management (CRM)
• Personalized product recommendations
• Churn prediction and customer retention strategies
• Customer segmentation for targeted marketing campaigns
• Risk Management
• Market risk analysis using predictive analytics
• Stress testing and scenario analysis
• Compliance monitoring and regulatory reporting
• Operations Optimization
• Process automation for efficiency gains
• Predictive maintenance for ATM networks
• Supply chain optimization for inventory management
• Sentiment Analysis
• Social media and customer feedback analysis
• Brand sentiment tracking
• Reputation management and crisis response
•
 Applications of Data Science in Supply Chain
• Demand Forecasting
• Predictive analytics for accurate demand prediction
• Seasonal trend analysis for demand fluctuations
• Machine learning models for dynamic forecasting

• Inventory Optimization
• Real-time inventory tracking and management
• Automated replenishment based on demand forecasting
• Optimization algorithms to minimize carrying costs

• Supply Chain Visibility

• Real-time monitoring of shipments and inventory levels
• Predictive analytics for identifying potential disruptions
• Block chain technology for enhanced transparency and traceability
APPLICATIONS OF DATASCIENCE IN SUPPLY CHAIN

Supplier Relationship Management

• Predictive analytics for supplier performance evaluation
• Risk assessment and mitigation strategies
• Supplier collaboration platforms for improved communication

Transportation and Logistics Optimization

• Route optimization for efficient transportation
• Dynamic scheduling to minimize delays and costs
• Fleet management optimization for resource efficiency
 Data Science Life Cycle

[Link]
DATA SCIENCE TOOLS AND TECHNOLOGIES

A. Programming Languages
• Python
• R
B. Data Visualization Tools

• Tableau
• Power BI
C. Machine Learning Libraries
• Scikit-learn
• TensorFlow
D. Big Data Frameworks
• Apache HadooP
• Apache Spark
DATA SCIENCE CLASSIFICATION
Tasks Description Algorithms Examples

Classification Predict if a data point belongs to Decision Trees, Neural Assigning voters into known
one of predefined classes. The networks, Bayesian buckets by political parties eg:
prediction will be based on models, Induction rules, soccer moms. Bucketing new
learning from known data set. K nearest neighbors customers into one of known
customer groups.

Regression Predict the numeric target label Linear regression, Predicting unemployment rate
of a data point. The prediction Logistic regression for next year. Estimating
will be based on learning from insurance premium.
known data set.

Anomaly Predict if a data point is an outlier Distance based, Density Fraud transaction detection in
detection compared to other data points in based credit cards. Network intrusion
the data set. detection.
Tasks Description Algorithms Examples
Time series Predict if the value of the Exponential Sales forecasting, production
target variable for future smoothing, forecasting, virtually any
time frame based on history ARIMA, growth phenomenon that needs
values. regression to be extrapolated

Clustering Identify natural clusters K means, density Finding customer segments in a

within the data set based on based clustering - company based on transaction,
inherit properties within the DBSCAN web and customer call data.
data set.

Association Identify relationships within FP Growth, Find cross selling opportunities

analysis an itemset based on Apriori for a retailor based on
transaction data. transaction purchase history.
 Advantages of Data Science
[Link] decision-making: It organizations make better decisions by
providing insights & predictions based on analysis of data.

[Link]-effecitve: Right tools and techniques can reduce costs by identifying

areas of inefficiency and optimizing processes.
[Link]: To identify new opportunities for innovation and to develop new
products and services.
[Link] advantage: Organizations effectively can gain a competitive
advantage by making better decisions, improving efficiency, and identifying new
opportunities by using data science.

[Link]: It also helps organizations personalize their products or

services to better meet the needs of individual customers.
 Disadvantages of Data Science
[Link] quality: The accuracy and quality of the data used in data
science can have a significant impact on the results obtained.
[Link] concerns: The collection and use of data can raise privacy
concerns, particularly if the data is personal or sensitive.
[Link]: Data science can be a complex and technical field that
requires specialized skills and expertise.
[Link]: Data science algorithms can be biased if the data used to train
them is biased, which can lead to inaccurate results.
[Link]: Interpreting data science results can be challenging,
particularly for non-technical stakeholders who may not understand the
underlying assumptions and methods used.
ADVANCED DATA SCIENCE
 UNIT I: Data Science Fundamentals
• Introduction To Data Science,
• PYTHON Libraries For Data Science,
• Data, Types Of Variables Continuous And Discrete Variables,
• Data Sampling, Pandas Data Structures.
• Statistics For Data Science,
• Data Visualization, Matplotlib, Histograms, Line Charts,
• Pie Charts, Multiple Bar Graphs, Box Plots, Scatter Plots, Seaborn
Plots, Bokeh Plots
 Introduction to Data Science
• Definition
• Key Components
• Why Data Science?
• Roles and Responsibilities of a Data Scientist
• Data Science Lifecycle (Step-by-Step)
• Skills Required for Data Science
• Tools Commonly Used in Data Science
• Real-World Applications of Data Science
 Data Science =
Domain Expertise + Programming + Statistics
What is Data Science?
 What is Data Science?
Data Science is a multidisciplinary field that uses scientific methods,
algorithms, processes, and systems to extract insights and knowledge
from structured and unstructured data.
 Key Components:
• Mathematics & Statistics

• Programming (mostly in Python/R)

• Domain Knowledge

• Machine Learning

• Data Visualization
Why Data Science?
 Why Data Science?

We generate 2.5 quintillion bytes of data daily (from social

media, IoT, e-commerce, etc.)

2500000000 gigabytes
➢ Traditional data processing tools are not efficient
for such massive data.

Data Science helps organizations:

o Predict customer behavior
o Optimize operations
o Automate tasks
Make informed decisions
Roles and Responsibilities of a Data
Scientist
Roles and Responsibilities of a Data
Scientist
Roles and Responsibilities of a Data Scientist

Interpret
Perform Apply
Understand Collect and results and
statistical machine
business clean raw communicate
analysis and learning
problems data them to
visualizations models
stakeholders
 Data Science Lifecycle
Problem Data Cleaning Exploratory
Data Collection
Definition (Preprocessing) Data Analysis

Feature
Deployment Evaluation Model Building
Engineering

Monitoring
 Data Science Lifecycle
Problem Definition:
• Define the objective (e.g., “Predict employee attrition” or “Classify emails as spam or not”).
Data Collection:
• Gather data from sources like databases, APIs, web scraping, IoT devices, etc.
Data Cleaning (Preprocessing):
• Handle missing values, outliers, and incorrect formats
• Normalize or standardize values
• Encode categorical variables
Exploratory Data Analysis (EDA):
• Use summary statistics and visualization tools
Identify trends, patterns, and relationships
 Data Science Lifecycle
Feature Engineering:
• Select and transform relevant input variables (features)
• Create new features that improve model performance
Model Building:
• Choose and train a machine learning model (e.g., Decision Tree, Logistic Regression)
Evaluation:
• Use metrics like accuracy, precision, recall, F1-score, RMSE to test model performance
Deployment:
• Use the trained model in a real-world system (e.g., web app, mobile app)
Monitoring:
• Track the model’s performance over time and retrain if needed
 Skills Required for Data Science
Skill Type Examples
Programming Python, R
Statistics Probability, distributions, testing
Machine Learning Regression, clustering, trees

Tools Jupyter, Git, SQL, Excel

Visualization Matplotlib, Seaborn, Tableau
Tools Commonly Used in Data Science
Programming Languages: Python, R, SQL
Libraries:
❑ NumPy (arrays & math)
❑ Pandas (data handling)
❑ Matplotlib & Seaborn (visualization)
❑ Scikit-learn (machine learning)
❑ TensorFlow / PyTorch (deep learning)
Environments: Jupyter Notebooks, Google Colab, VS Code
 Python Libraries for Data Science
NumPy (Numerical Python)
Pandas (Panel Data System)
Matplotlib
Seaborn
SciPy (Scientific Python)
Scikit-learn
Bokeh
 Library Main Use Common Use Cases
NumPy

Pandas

Matplotlib

Seaborn

SciPy

Scikit-learn

Bokeh
 Library Main Use Common Use Cases
NumPy Numerical computations Arrays, math functions

Pandas

Matplotlib

Seaborn

SciPy

Scikit-learn

Bokeh
 Library Main Use Common Use Cases
NumPy Numerical computations Arrays, math functions

Pandas Data manipulation Cleaning, grouping, filtering

Matplotlib

Seaborn

SciPy

Scikit-learn

Bokeh
 Library Main Use Common Use Cases
NumPy Numerical computations Arrays, math functions

Pandas Data manipulation Cleaning, grouping, filtering

Matplotlib Static 2D plots Line, bar, scatter plots

Seaborn

SciPy

Scikit-learn

Bokeh
 Library Main Use Common Use Cases
NumPy Numerical computations Arrays, math functions

Pandas Data manipulation Cleaning, grouping, filtering

Matplotlib Static 2D plots Line, bar, scatter plots

Seaborn Statistical visualizations Pairplots, heatmaps, distributions

SciPy

Scikit-learn

Bokeh
 Library Main Use Common Use Cases
NumPy Numerical computations Arrays, math functions

Pandas Data manipulation Cleaning, grouping, filtering

Matplotlib Static 2D plots Line, bar, scatter plots

Seaborn Statistical visualizations Pairplots, heatmaps, distributions

SciPy Scientific computations Stats, integration, optimization

Scikit-learn

Bokeh
 Library Main Use Common Use Cases
NumPy Numerical computations Arrays, math functions

Pandas Data manipulation Cleaning, grouping, filtering

Matplotlib Static 2D plots Line, bar, scatter plots

Seaborn Statistical visualizations Pairplots, heatmaps, distributions

SciPy Scientific computations Stats, integration, optimization

Scikit-learn Machine learning models Classification, regression

Bokeh
 Library Main Use Common Use Cases
NumPy Numerical computations Arrays, math functions

Pandas Data manipulation Cleaning, grouping, filtering

Matplotlib Static 2D plots Line, bar, scatter plots

Seaborn Statistical visualizations Pairplots, heatmaps, distributions

SciPy Scientific computations Stats, integration, optimization

Scikit-learn Machine learning models Classification, regression

Bokeh Interactive browser-based plots Dashboards, web apps

Understanding Data
and
Variable Types
What is Data?
 What is Data?
▪ Data is a collection of facts or information used for reference, analysis, or
calculation.
 What is Data?
▪ Data is a collection of facts or information used for reference, analysis, or
calculation.

These facts can take many forms:

 What is Data?
▪ Data is a collection of facts or information used for reference, analysis, or
calculation.

These facts can take many forms:

• Numbers: 98.6, 2000, 0.99

• Words: "Pass", "Fail", "Red"
• Measurements: Height, Weight, Temperature
• Observations: "The weather is cloudy", "Machine stopped at 3 PM"
Why Understanding Data Types Matters?
 Why Understanding Data Types Matters?

➢ It determines the type of analysis or visualization you can perform.

➢ It affects how the data is stored and processed in code.
➢ It guides your choice of statistical methods and machine learning models.
 Classification of Data
Description Example
Data Type
Qualitative Describes qualities or Gender, Color, Type
categories of Car

Quantitative Describes quantities or Age, Height,

measurements Temperature
 Quantitative Data:

Continuous vs Discrete
Variables
Continuous Variables
Continuous Variables
• Variables that can take any value within a range. They are measurable
and often have decimal points.
 Continuous Variables
• Variables that can take any value within a range. They are measurable
and often have decimal points.

Characteristics: Examples: Visualization Tools:

• Can take infinite values • Temperature: 36.5°C, 37.01°C • Histograms
• Represented by real numbers • Height: 5.6 ft, 170.2 cm • Line Charts
• Usually measured, not counted • Time: 3.5 hours, 0.25 sec • Box Plots
• Weight: 65.4 kg
Discrete Variables
Discrete Variables
• Variables that take specific, countable values. They cannot take
fractions or decimals between values.
 Discrete Variables
• Variables that take specific, countable values. They cannot take
fractions or decimals between values.

Examples:
Characteristics: Visualization Tools:
• Number of students in a class: 30, 31
• Finite or countably infinite values • Bar Graphs
• Defects in a product: 0, 1, 2, ...
• Typically counted, not measured • Pie Charts
• Number of cars in a parking lot
Types of Qualitative (Categorical) Data

Nominal:
• Categories with no natural order
• E.g., Colors (Red, Green, Blue), Gender

Ordinal:
• Categories with a meaningful order, but differences are not measurable
• E.g., Ratings (Bad, Average, Good), Education level
 Data Type Identification – Real Examples

Variable Type Subtype

Temperature (°C)

Number of siblings

Blood group

Customer satisfaction

Student ID
 Data Type Identification – Real Examples

Variable Type Subtype

Temperature (°C) Quantitative Continuous

Number of siblings

Blood group

Customer satisfaction

Student ID
 Data Type Identification – Real Examples

Variable Type Subtype

Temperature (°C) Quantitative Continuous

Number of siblings Quantitative Discrete

Blood group

Customer satisfaction

Student ID
 Data Type Identification – Real Examples

Variable Type Subtype

Temperature (°C) Quantitative Continuous

Number of siblings Quantitative Discrete

Blood group Qualitative Nominal

Customer satisfaction

Student ID
 Data Type Identification – Real Examples

Variable Type Subtype

Temperature (°C) Quantitative Continuous

Number of siblings Quantitative Discrete

Blood group Qualitative Nominal

Customer satisfaction Qualitative Ordinal

Student ID
 Data Type Identification – Real Examples

Variable Type Subtype

Temperature (°C) Quantitative Continuous

Number of siblings Quantitative Discrete

Blood group Qualitative Nominal

Customer satisfaction Qualitative Ordinal

Student ID Qualitative Nominal (even if numeric)

Data Sampling
What is Data Sampling?
 What is Data Sampling?

• Data sampling is the process of selecting a subset of data from a larger dataset
(population) to perform analysis and make inferences about the entire population.
Why is Sampling Important in Data Science?
Why is Sampling Important in Data Science?

Purpose Explanation
Efficiency Speeds up analysis by working with smaller data

Feasibility Some datasets are too large to process fully

Cost-effectiveness Less storage and processing power needed

Model Training Used in training machine learning models on subsets

Statistical Estimation Estimate population parameters (mean, variance, etc.)

Key Terminology in Data Sampling

1. Population
Key Terminology in Data Sampling

1. Population

Definition: The entire set of individuals, items, or data points that you're
interested in studying.

Example: If you're analyzing student performance in Telangana, the

population could be all students enrolled in schools across Telangana.
Key Terminology in Data Sampling

2. Sample
Key Terminology in Data Sampling

2. Sample
Definition: A subset of the population selected for analysis.

Purpose: To make inferences about the population without analyzing every

member.

Example: Selecting 500 students from different schools in Telangana to

study their performance.
Key Terminology in Data Sampling

3. Sampling Frame

Definition: A list or database that includes all members of the population

from which the sample is drawn.

Importance: Ensures that the sample is representative and unbiased.

Example: A government database listing all registered students in

Telangana schools.
Key Terminology in Data Sampling

4. Sampling Unit

Definition: The individual element or group of elements considered for

selection in the sample.

Example: Each student in the database is a sampling unit.

Key Terminology in Data Sampling

5. Sampling Error

Definition: The difference between the characteristics of the sample and

those of the population.

Cause: Occurs because only a part of the population is studied.

Example: If the average score of the sample is 75 but the actual population
average is 78, the sampling error is 3.
Key Terminology in Data Sampling

6. Sample Size

Definition: The number of observations or data points in the sample.

Impact: Larger sample sizes generally lead to more accurate and reliable
results.

Example: A sample size of 1,000 students is more reliable than a sample of

100.
Types of Sampling Methods
Types of Sampling Methods

A. Probability Sampling
1. Simple Random Sampling
2. Systematic Sampling
3. Stratified Sampling
4. Cluster Sampling

B. Non-Probability Sampling
1. Convenience Sampling
2. Judgmental (Purposive) Sampling
3. Snowball Sampling
Probability Sampling (Randomized Sampling)
 Probability Sampling (Randomized Sampling)

In probability sampling, every member of the population has a known, non-zero

chance of being selected. This method is more statistically sound and allows for
generalization to the population.
 Probability Sampling (Randomized Sampling)

In probability sampling, every member of the population has a known, non-zero

chance of being selected. This method is more statistically sound and allows for
generalization to the population.

Type Description

1. Simple Random Sampling Each item has equal chance. Example: [Link]() in Python

2. Systematic Sampling Every k-th element is selected. e.g., 2nd, 4th, 6th...

Population divided into strata (groups), samples are taken from each
3. Stratified Sampling
stratum

4. Cluster Sampling Population divided into clusters; entire clusters are randomly selected
 Simple Random Sampling
Python Code
• import random

• # Sample population data (IDs from 1 to 20)

• population = list(range(1, 21))

• # Randomly select 5 samples

• sample = [Link](population, 5)

• print("Population:", population)
• print("Simple Random Sample:", sample)
 Simple Random Sampling
Python Code
• import random

• # Sample population data (IDs from 1 to 20)

• population = list(range(1, 21))

• # Randomly select 5 samples

• sample = [Link](population, 5)

• print("Population:", population)
• print("Simple Random Sample:", sample)

Output Example

Population: [1, 2, ..., 20]

Simple Random Sample: [14, 3, 18, 7, 11]
 Probability Sampling (Randomized Sampling)

In probability sampling, every member of the population has a known, non-zero

chance of being selected. This method is more statistically sound and allows for
generalization to the population.

Type Description

1. Simple Random Sampling Each item has equal chance. Example: [Link]() in Python

2. Systematic Sampling Every k-th element is selected. e.g., 2nd, 4th, 6th...

Population divided into strata (groups), samples are taken from each
3. Stratified Sampling
stratum

4. Cluster Sampling Population divided into clusters; entire clusters are randomly selected
 Systematic Sampling

Python Code
import numpy as np

# Sample population
population = list(range(1, 21)) # Population of 20

k = 4 # Select every 4th item

start = [Link](0, k) # Random start index between
0 and k-1

# Get systematic sample

systematic_sample = population[start::k]

print("Population:", population)
print(f"Systematic Sample (every {k}th starting at {start}):",
systematic_sample)
 Systematic Sampling

Python Code Output

import numpy as np
Population: [1, 2, ..., 20]
Systematic Sample (every 4th starting at 2): [3, 7, 11, 15, 19]
# Sample population
population = list(range(1, 21)) # Population of 20

k = 4 # Select every 4th item

start = [Link](0, k) # Random start index between
0 and k-1

# Get systematic sample

systematic_sample = population[start::k]

print("Population:", population)
print(f"Systematic Sample (every {k}th starting at {start}):",
systematic_sample)
 Probability Sampling (Randomized Sampling)

In probability sampling, every member of the population has a known, non-zero

chance of being selected. This method is more statistically sound and allows for
generalization to the population.

Type Description

1. Simple Random Sampling Each item has equal chance. Example: [Link]() in Python

2. Systematic Sampling Every k-th element is selected. e.g., 2nd, 4th, 6th...

Population divided into strata (groups), samples are taken from each
3. Stratified Sampling
stratum

4. Cluster Sampling Population divided into clusters; entire clusters are randomly selected
 Stratified Sampling

Python Code
import pandas as pd

# Create a sample DataFrame with two strata (e.g., Gender)

data = {
'ID': range(1, 21),
'Gender': ['Male']*10 + ['Female']*10
}
df = [Link](data)

# Stratified sampling: sample 3 from each gender group

stratified_sample = [Link]('Gender',
group_keys=False).apply(lambda x: [Link](3))

print("Original DataFrame:\n", df)

print("\nStratified Sample:\n", stratified_sample)
 Stratified Sampling

Python Code Output

import pandas as pd
Stratified Sample:
# Create a sample DataFrame with two strata (e.g., Gender) ID Gender
2 3 Male
data = {
6 7 Male
'ID': range(1, 21), 9 10 Male
'Gender': ['Male']*10 + ['Female']*10 10 11 Female
} 12 13 Female
df = [Link](data) 15 16 Female

# Stratified sampling: sample 3 from each gender group

stratified_sample = [Link]('Gender',
group_keys=False).apply(lambda x: [Link](3))

print("Original DataFrame:\n", df)

print("\nStratified Sample:\n", stratified_sample)
 Probability Sampling (Randomized Sampling)

In probability sampling, every member of the population has a known, non-zero

chance of being selected. This method is more statistically sound and allows for
generalization to the population.

Type Description

1. Simple Random Sampling Each item has equal chance. Example: [Link]() in Python

2. Systematic Sampling Every k-th element is selected. e.g., 2nd, 4th, 6th...

Population divided into strata (groups), samples are taken from each
3. Stratified Sampling
stratum

4. Cluster Sampling Population divided into clusters; entire clusters are randomly selected
 Cluster Sampling
Python Code Output

import pandas as pd Selected Clusters: ['B', 'D']

import random Cluster Sample:
ID Cluster
# Assume population is grouped into 4 clusters of 5 items each 5 6 B
data = { 6 7 B
'ID': range(1, 21), ...
'Cluster': ['A']*5 + ['B']*5 + ['C']*5 + ['D']*5 19 20 D
}
df = [Link](data)

# Randomly choose 2 clusters

selected_clusters = [Link](df['Cluster'].unique().tolist(), 2)

# Select all members from chosen clusters

cluster_sample = df[df['Cluster'].isin(selected_clusters)]

print("Selected Clusters:", selected_clusters)

print("\nCluster Sample:\n", cluster_sample)
Types of Sampling Methods

A. Probability Sampling
1. Simple Random Sampling
2. Systematic Sampling
3. Stratified Sampling
4. Cluster Sampling

B. Non-Probability Sampling
1. Convenience Sampling
2. Judgmental (Purposive) Sampling
3. Snowball Sampling
 Non-Probability Sampling

Non-Probability Sampling is a sampling technique in which not all

members of the population have a known or equal chance of being
selected.

Unlike probability sampling, which is based on randomness, non-

probability sampling is based on subjective judgment, convenience,
or specific criteria chosen by the researcher.
 Key Characteristics

Feature Description
Randomness Not random
Chance of Selection Unknown or unequal for each population member
Bias Risk Higher compared to probability sampling
Use Cases Exploratory research, qualitative research, early testing
Cost & Time Usually faster and cheaper
 Common Types of Non-Probability
Sampling
Method Description
1. Convenience Sampling Sample is taken from people who are easiest to reach

2. Judgmental Sampling Researcher selects based on their judgment or criteria

Existing participants recruit future ones (used in hidden

3. Snowball Sampling
populations)

Sampling until a quota (number) from each category is met

4. Quota Sampling
(non-random)
 1. Convenience Sampling

✓ The sample is taken from a group that is easy to access.

✓ It's quick and cost-effective, but may lead to bias.

✓ Often used in early-stage research or pilot studies.

 1. Convenience Sampling
Python Code Output

import pandas as pd Convenience Sample:

ID Name Score
# Sample dataset of 10 students 0 1 Alice 85
data = { 1 2 Bob 78
'ID': range(1, 11), 2 3 Charlie 92
'Name': ['Alice', 'Bob', 'Charlie', 'David', 'Eva', 'Fay', 'George', 3 4 David 67
'Hannah', 'Ian', 'Jane'], 4 5 Eva 88
'Score': [85, 78, 92, 67, 88, 73, 81, 90, 77, 84]
}
df = [Link](data)

# Convenience sampling: Selecting the first 5 entries

convenience_sample = [Link](5)

print("Original Data:\n", df)

print("\nConvenience Sample (first 5 students):\n",
convenience_sample)
 Judgmental (Purposive) Sampling

✓ The researcher selects the sample based on personal judgment or

expertise.

✓ Chosen individuals are believed to represent the population best.

✓ Useful in case studies, expert interviews, or rare-event studies.

 Judgmental (Purposive) Sampling
Python Code Output

# Select students with score > 85 as they are "high performers" Judgmental Sample:
judgmental_sample = df[df['Score'] > 85] ID Name Score
1 3 Charlie 92
4 5 Eva 88
print("Judgmental Sample (Score > 85):\n", judgmental_sample)
7 8 Hannah 90
 Snowball Sampling

Definition: Existing participants recruit future participants.

Example: Studying a rare disease by asking patients to refer others.
Advantages:
Useful for hard-to-reach populations
Disadvantages:
Non-random and biased
# Creating a social network of people (simplified)
network = {
'Alice': ['Bob', 'Charlie'],
'Bob': ['David', 'Eva'],
'Charlie': ['Fay'],
'David': ['George'],
'Eva': ['Hannah'], Snowball Sample:
'Fay': [], {'Alice', 'Bob', 'Charlie', 'David', 'Eva', 'Fay',
'George': [], 'George', 'Hannah'}
'Hannah': []
}

# Snowball Sampling: Starting from Alice

snowball_sample = set()
to_explore = ['Alice'] # Initial subject

while to_explore:
person = to_explore.pop()
if person not in snowball_sample:
snowball_sample.add(person)
to_explore.extend(network[person]) # Add referrals

print("Snowball Sample (network traversal):\n", snowball_sample)

Pandas Data Structures
 Pandas Data Structures
✓ In data science, Pandas is a powerful Python library used for data manipulation
and analysis.
 Pandas Data Structures
✓ In data science, Pandas is a powerful Python library used for data manipulation
and analysis.

✓ Pandas is a powerful open-source Python library.

✓ It provides fast, flexible, and expressive data structures that are designed to
make working with structured (tabular or labeled) data easy and intuitive.
It provides two primary data structures:
 It provides two primary data structures:

Data
Description Analogy
Structure

Series

DataFrame
 It provides two primary data structures:

Data
Description Analogy
Structure

1D labeled
Series A single column in Excel
array

2D labeled
DataFrame An Excel spreadsheet or table
table
Series
 Series
A Series is a one-dimensional labeled array capable of holding any
data type (integers, strings, floats, etc.).

It has:
•Values: the actual data
•Index: labels for each value (like row labels)
Example:
 Pandas DataFrame
A DataFrame is a two-dimensional labeled data structure with rows
and columns.

You can think of it as a table where:

•Each column is a Series

•It has row and column labels (index and columns)
Example:
Accessing Data in DataFrame

print("Names column:")
print(df['Name'])
Access a Row by Index: Access Multiple Columns:

print("Second row:") print("Names and Marks:")

print([Link][1]) print(df[['Name', 'Marks']])
 Accessing Data in DataFrame

print("Names column:")
print(df['Name'])
 Adding and Modifying Columns

Add New Column:

 Adding and Modifying Columns

Add New Column: Modify Existing Column:

 Useful DataFrame Functions

Function Description
[Link]()
[Link]()
[Link]()
[Link]()
[Link]
[Link]
 Useful DataFrame Functions

Function Description
[Link]() Shows first 5 rows
[Link]()
[Link]()
[Link]()
[Link]
[Link]
 Useful DataFrame Functions

Function Description
[Link]() Shows first 5 rows
[Link]() Shows last 5 rows
[Link]()
[Link]()
[Link]
[Link]
 Useful DataFrame Functions

Function Description
[Link]() Shows first 5 rows
[Link]() Shows last 5 rows
[Link]() Summary stats (mean, std, min, max, etc.)
[Link]()
[Link]
[Link]
 Useful DataFrame Functions

Function Description
[Link]() Shows first 5 rows
[Link]() Shows last 5 rows
[Link]() Summary stats (mean, std, min, max, etc.)
[Link]() Structure of the DataFrame
[Link]
[Link]
 Useful DataFrame Functions

Function Description
[Link]() Shows first 5 rows
[Link]() Shows last 5 rows
[Link]() Summary stats (mean, std, min, max, etc.)
[Link]() Structure of the DataFrame
[Link] Rows and columns count
[Link]
 Useful DataFrame Functions

Function Description
[Link]() Shows first 5 rows
[Link]() Shows last 5 rows
[Link]() Summary stats (mean, std, min, max, etc.)
[Link]() Structure of the DataFrame
[Link] Rows and columns count
[Link] Data types of each column
import pandas as pd
# Summary statistics
# Sample data dictionary print("\n [Link]():")
data = { print([Link]())
'Name': ['Alice', 'Bob', 'Charlie', 'David', 'Eva', 'Frank'],
'Age': [25, 30, 22, 28, 35, 27], # Structure of the DataFrame
'Marks': [85, 90, 78, 92, 88, 80] print("\n [Link]():")
} [Link]()

# Create DataFrame # Rows and columns count

df = [Link](data) print("\n [Link]:")
print([Link])
# Display the first 5 rows
print(" [Link]():") # Data types of each column
print([Link]()) print("\n [Link]:")
print([Link])
# Display the last 5 rows
print("\n [Link]():")
print([Link]())
Statistics for Data Science
 Statistics for Data Science
Statistics is a fundamental pillar of Data Science. It helps in:

✓ Understanding data
✓ Drawing meaningful conclusions
✓ Making informed decisions
✓ Building predictive models
 Descriptive Statistics

Describes and summarizes data using:

✓ Measures of Central Tendency (Mean, Median, Mode)
✓ Measures of Dispersion (Variance, Standard Deviation, Range)
✓ Distribution Shape (Skewness, Kurtosis)
 Common Descriptive Measures:

Concept Python Function/Method

Mean [Link](data)
Median [Link](data)
Mode [Link](data)
Variance [Link](data)
Std Deviation [Link](data)
Correlation [Link](x, y)
Covariance [Link](x, y)
t-Test stats.ttest_ind(group1, group2)
Percentiles [Link](data, q)
Box Plot [Link]()
 Topic Purpose
Descriptive Statistics Summarize data
Probability Measure chance
Distributions Model data behavior
Inference Make predictions from samples
Correlation Understand relationships
Sampling Efficient data selection
Outlier Detection Identify anomalies
Bayesian Stats Incorporate prior beliefs
Mean (Average)
1. Mean (Average)
Shows the average of the dataset.
Median
 Median
Middle value when data is sorted.

When to prefer median: When data has outliers (e.g., income data),
median gives a better center.
Mode
c. Mode
 Mode
• The most frequent value.
Minimum (min) and Maximum (max)
Minimum (min) and Maximum (max)
 Range
Range = Max − Min
Shows the total spread in the dataset.

Use:
•Quick way to understand the variability
•Doesn't account for distribution of data
Variance
 Variance

Variance tells us how spread out the numbers are in a dataset.

 Variance

Variance tells us how spread out the numbers are in a dataset.

•If all numbers are close to the mean, the variance is small.
 Variance

Variance tells us how spread out the numbers are in a dataset.

•If all numbers are close to the mean, the variance is small.
•If the numbers are far from the mean, the variance is large
 Variance

Variance tells us how spread out the numbers are in a dataset.

•If all numbers are close to the mean, the variance is small.
•If the numbers are far from the mean, the variance is large

Real-Life Example:

Imagine students’ marks in a class. If everyone's score is close to the class average
(mean), there's low variance. If some score very high and others very low, the
variance is high.
Mathematical Formula:
 What Is Variance in Real Life?

✓ Variance tells us how much values differ from the average.

✓ If values are very close to each other, variance is low.

✓ If values differ a lot, variance is high.

Real-Life example: Exam Scores of Two Classes
 Real-Life example: Exam Scores of Two Classes
Imagine I am teaching two classes: Class A and Class B. Both classes wrote the same exam.
 Real-Life example: Exam Scores of Two Classes
Imagine I am teaching two classes: Class A and Class B. Both classes wrote the same exam.

Class A Scores: [70, 72, 68, 71, 69]

Class B Scores: [50, 90, 40, 100, 70]

Step 1: Calculate Mean (Average)
Step 1: Calculate Mean (Average)

Class A Mean = (70 + 72 + 68 + 71 + 69) / 5 =

Step 1: Calculate Mean (Average)

•Class A Mean = (70 + 72 + 68 + 71 + 69) / 5 = 70

Step 1: Calculate Mean (Average)

•Class A Mean = (70 + 72 + 68 + 71 + 69) / 5 = 70

•Class B Mean = (50 + 90 + 40 + 100 + 70) / 5 =
Step 1: Calculate Mean (Average)

•Class A Mean = (70 + 72 + 68 + 71 + 69) / 5 = 70

•Class B Mean = (50 + 90 + 40 + 100 + 70) / 5 = 70

Step 1: Calculate Mean (Average)

•Class A Mean = (70 + 72 + 68 + 71 + 69) / 5 = 70

•Class B Mean = (50 + 90 + 40 + 100 + 70) / 5 = 70

Both classes have the same average score = 70

Step 2: Compare the Spread of Scores
Step 2: Compare the Spread of Scores
Step 2: Compare the Spread of Scores
So, even though the average score is the same in both classes, Class A's students
performed consistently, while Class B had high variation — some scored low, others
very high.
Standard Deviation (σ)
 Standard Deviation
➢ Standard Deviation (SD) is simply the square root of variance.
 Standard Deviation
➢ Standard Deviation (SD) is simply the square root of variance.

➢ It’s used to express spread in the same unit as the data (unlike variance,
which is squared).
 Standard Deviation
➢ Standard Deviation (SD) is simply the square root of variance.

➢ It’s used to express spread in the same unit as the data (unlike variance,
which is squared).

Why Take Square Root?

 Standard Deviation
➢ Standard Deviation (SD) is simply the square root of variance.

➢ It’s used to express spread in the same unit as the data (unlike variance,
which is squared).

Why Take Square Root?

➢ Variance is in squared units (e.g., meters², marks²), which is not

intuitive.
 Standard Deviation
➢ Standard Deviation (SD) is simply the square root of variance.

➢ It’s used to express spread in the same unit as the data (unlike variance,
which is squared).

Why Take Square Root?

➢ Variance is in squared units (e.g., meters², marks²), which is not

intuitive.

➢ Taking the square root brings it back to the original unit (meters,
marks), making it easier to interpret.
 Standard Deviation (σ)
• Measures spread of data around the mean.

It’s the square root of variance, so it brings the unit back to the
original — making it easier to interpret than variance.
Real-Life example: Food delivery apps
Real-Life example: Food delivery apps

➢ Imagine two food delivery apps — App A and App B.

➢ You're tracking how long (in minutes) they take to deliver food.
Real-Life example: Food delivery apps
Imagine two food delivery apps — App A and App B. You're tracking how long (in
minutes) they take to deliver food.

App A: [30, 32, 31, 29, 30]

App B: [10, 60, 30, 50, 20]

 Step 1: Find the Mean

✓App A mean = (30+32+31+29+30)/5 =

✓App B mean = (10+60+30+50+20)/5 =

 Step 1: Find the Mean

✓App A mean = (30+32+31+29+30)/5 = 30.4 minutes

✓App B mean = (10+60+30+50+20)/5 = 34 minutes

 Step 2: Calculate the Deviation from Mean
App A:

➢ Deviations from mean: [-0.4, 1.6, 0.6, -1.4, -0.4]

➢ Squared deviations: [0.16, 2.56, 0.36, 1.96, 0.16]
➢ Variance = 1.04
➢ Standard Deviation = √1.04 ≈ 1.02
 Step 2: Calculate the Deviation from Mean
App A:

➢ Deviations from mean: [-0.4, 1.6, 0.6, -1.4, -0.4]

➢ Squared deviations: [0.16, 2.56, 0.36, 1.96, 0.16]
➢ Variance = 1.04
➢ Standard Deviation = √1.04 ≈ 1.02

App B:
➢Deviations: [-24, 26, -4, 16, -14]
➢Squared deviations: [576, 676, 16, 256, 196]
➢Variance = 344
➢Standard Deviation = √344 ≈ 18.52
Even though the average delivery times are somewhat close, App A is
reliable, and App B is unpredictable.
Why is Standard Deviation Important in Data Science?

✓ Helps understand spread of data

✓ A low σ = data is clustered near mean → more predictable
✓ A high σ = data is widely spread → more uncertainty
✓ Used in outlier detection, risk analysis, and model performance evaluation
 What are Outliers?
An outlier is a data point that significantly differs from other observations in a dataset.
 What are Outliers?
An outlier is a data point that significantly differs from other observations in a dataset.

Outlier = A value that is very different from the rest of the data.
 What are Outliers?
An outlier is a data point that significantly differs from other observations in a
dataset.

It could be:
• A result of data entry errors :
 What are Outliers?
An outlier is a data point that significantly differs from other observations in a
dataset.

It could be:
• A result of data entry errors : e.g., someone typing 1000 instead of 100
Example:

Let’s say we have exam scores for a group of students:

scores = [65, 70, 68, 72, 69, 71, 250]

Here, 250 is an outlier — it’s way off

compared to the others .
 What are Outliers?
An outlier is a data point that significantly differs from other observations in a
dataset.

It could be:
• A result of data entry errors : e.g., someone typing 1000 instead of 100
• A rare event (valid but extreme):
 What are Outliers?
An outlier is a data point that significantly differs from other observations in a
dataset.

It could be:
• A result of data entry errors : e.g., someone typing 1000 instead of 100
• A rare event (valid but extreme): e.g., one student scoring 100 while
everyone else scores 60–70
 What are Outliers?
An outlier is a data point that significantly differs from other observations in a
dataset.

It could be:
• A result of data entry errors : e.g., someone typing 1000 instead of 100
• A rare event (valid but extreme): e.g., one student scoring 100 while
everyone else scores 60–70
• Or due to measurement error: faulty device or sensor
Why Detect and Remove Outliers?
➢They can skew statistical analyses (like mean, regression).

➢May lead to overfitting in machine learning models.

➢Can mislead visualizations and affect model accuracy.

 Types of Outliers
1. Global Outliers – Far from the rest of the data
 Types of Outliers
1. Global Outliers – Far from the rest of the data

2. Contextual Outliers – Outliers in a specific context (e.g.,

temperature anomaly in a specific month)
 Types of Outliers
1. Global Outliers – Far from the rest of the data

2. Contextual Outliers – Outliers in a specific context (e.g.,

temperature anomaly in a specific month)

3. Collective Outliers – A group of outliers behaving abnormally

together
These are groups of data points that deviate together — individually, they may seem fine, but together
they form an unusual pattern.

Example:
A sudden spike in network traffic (e.g., [100, 105, 110, 500, 520, 530]) during midnight might signal a
cyber attack.
 Topic Purpose
Descriptive Statistics Summarize data
Probability Measure chance
Distributions Model data behavior
Inference Make predictions from samples
Correlation Understand relationships
Sampling Efficient data selection
Outlier Detection Identify anomalies
Bayesian Stats Incorporate prior beliefs
Probability Basics:
 Probability Basics:
Probability quantifies the likelihood of events.
 Probability Basics:
Probability quantifies the likelihood of events.

Probability = Number of favorable outcomes / Total outcomes

Types: Independent, Dependent Events

 Probability Distributions

Distribution Use Case

Binomial Pass/fail outcomes (classification)
Normal Most real-world data
Poisson Events in a fixed interval
 What is Binomial Distribution?
The Binomial Distribution models the number of successes in a fixed
number of independent Bernoulli trials, where each trial has only
two outcomes: success or failure.
 What is Binomial Distribution?
The Binomial Distribution models the number of successes in a fixed
number of independent Bernoulli trials, where each trial has only
two outcomes: success or failure.

Key Characteristics:
▪ n = number of trials
▪ p = probability of success in each trial
▪ q = 1 - p = probability of failure
▪ x = number of successes
 What is Binomial Distribution?
The Binomial Distribution models the number of successes in a fixed
number of independent Bernoulli trials, where each trial has only
two outcomes: success or failure.

Key Characteristics:
▪ n = number of trials
▪ p = probability of success in each trial
▪ q = 1 - p = probability of failure
▪ x = number of successes
Suppose a multiple-choice quiz has 10 questions (each with only 1 correct
answer), and a student guesses randomly with a 25% chance of getting any
question right. What’s the probability they get exactly 4 answers, correct?
Suppose a multiple-choice quiz has 10 questions (each with only 1 correct
answer), and a student guesses randomly with a 25% chance of getting any
question right. What’s the probability they get exactly 4 answers, correct?
Plot the Distribution:
Plot the Distribution:
Binomial Distribution
Use Case: Classification (e.g., pass/fail, success/failure)
 Applications of Binomial Distribution:

Field Example Scenario

Machine Learning Success rate of a binary classifier on test samples
Medicine Effectiveness of a new drug (success/failure per patient)
Quality Control Number of defective items in a batch of products
Marketing Email campaign response (clicked vs. not clicked)
 Probability Distributions

Distribution Use Case

Binomial Pass/fail outcomes (classification)
Normal Most real-world data
Poisson Events in a fixed interval
 Normal Distribution
The Normal Distribution, also called the Gaussian distribution, is a continuous probability
distribution that is symmetric and bell-shaped. Most real-world data—like height, weight,
exam scores, etc.—follow this distribution when enough samples are collected.
 Normal Distribution
The Normal Distribution, also called the Gaussian distribution, is a continuous probability
distribution that is symmetric and bell-shaped. Most real-world data—like height, weight,
exam scores, etc.—follow this distribution when enough samples are collected.
Suppose the exam scores of students in a class are normally distributed with a
mean of 70 and a standard deviation of 10. What is the probability that a
student scores between 60 and 80?
 Normal Distribution
Use Case: Real-world continuous data (e.g., height, scores)
 Probability Distributions

Distribution Use Case

Binomial Pass/fail outcomes (classification)
Normal Most real-world data
Poisson Events in a fixed interval
 What is Poisson Distribution?
The Poisson Distribution is a discrete probability distribution that describes the
number of events occurring in a fixed interval of time or space, given a constant
average rate, and assuming that events occur independently.
 What is Poisson Distribution?
The Poisson Distribution is a discrete probability distribution that describes the
number of events occurring in a fixed interval of time or space, given a constant
average rate, and assuming that events occur independently.
 Example:
A call center receives an average of 5 calls per hour.
What is the probability that exactly 3 calls come in a randomly selected hour?
 Poisson Distribution

Use Case: Count of events in a fixed interval (e.g., customer arrivals)

 Applications of Poisson Distribution:

Field Example Scenario

Call Centers Number of incoming calls per minute/hour
Healthcare Number of patients arriving at a hospital per hour
Traffic Analysis Number of cars passing through a toll booth per minute
IT Operations Number of server crashes or failures in a time period
Biology Number of mutations in a strand of DNA
 Inferential Statistics
Drawing conclusions about a population from a sample.

•Hypothesis Testing
•Confidence Intervals
•p-value
 Inferential Statistics
Drawing conclusions about a population from a sample.

•Hypothesis Testing
•Confidence Intervals
•p-value

A company claims that the average salary of their employees is ₹50,000.

You collect a sample of 30 employees and want to test this claim.
Step 1: Import Required Libraries
Step 2: Sample Data Simulation
Step 3: Hypothesis Testing (One Sample t-Test)
Step 4: Confidence Interval (95%)
 Concept Output Example
Sample Mean ₹48,366.98

t-Statistic -1.855

p-value 0.074

Confidence Interval ₹46,946.19 to ₹49,787.77

 What is Data Visualization?

Data Visualization is the graphical representation of information

and data. It allows us to:

•Quickly understand data trends

•Identify patterns, correlations, and outliers

•Communicate findings effectively in Data Science and Machine

Learning projects
Why Use Data Visualization in Data Science?

Purpose Benefit

Explore data Spot trends and anomalies

Compare variables See relationships between multiple features

Present results Share insights with non-technical stakeholders

Model evaluation Understand performance visually

 Python Libraries for Visualization

Library Description

Matplotlib Basic plots like line, bar, pie, histograms

Seaborn Statistical plots with beautiful default styles

Bokeh Interactive visualizations for web apps

1. Matplotlib
 1. Matplotlib

Matplotlib is a powerful and widely-used Python library for data visualization.

It allows you to create a wide variety of static, animated, and interactive plots.
 1. Matplotlib
Matplotlib is a powerful and widely-used Python library for data visualization.
It allows you to create a wide variety of static, animated, and interactive plots.

✓ Think of it as the "Excel chart tool" for Python—but much more flexible.
✓ It is often used with NumPy, Pandas, and Seaborn for data analysis tasks.
 1. Matplotlib
Matplotlib is a powerful and widely-used Python library for data visualization.
It allows you to create a wide variety of static, animated, and interactive plots.

✓ Think of it as the "Excel chart tool" for Python—but much more flexible.
✓ It is often used with NumPy, Pandas, and Seaborn for data analysis tasks.

pip install matplotlib

 Core Components

✓ pyplot: Module inside matplotlib for basic plotting (like line, bar, pie, etc.)

✓ figure(): Main container for the plot

✓ plot(): Function to draw line charts

✓ show(): Displays the plot

Other Plot Types in Matplotlib

➢Line Chart,
➢Bar Chart,
➢Pie Chart,
➢Histogram,
➢Box Plot
 Line Chart
A Line Chart is a type of plot used to display information as a series of data points
connected by straight lines. It is commonly used to visualize trends over time (time
series) or to show how a value changes continuously.
Line Chart
 Line Chart
A Line Chart is a type of plot used to display information as a series of data points
connected by straight lines. It is commonly used to visualize trends over time (time
series) or to show how a value changes continuously.

Key Features:
•Shows relationship between two variables
•Useful for trend analysis
•X-axis: usually time or categories
•Y-axis: numerical values
Visualize the monthly sales of a company over 6 months.
 Line Chart

import [Link] as plt

# Sample data
x = [1, 2, 3, 4, 5]
y = [10, 15, 13, 18, 16]
# Line Chart
[Link](x, y)
[Link]("Line Chart")
[Link]("X-axis")
[Link]("Y-axis")
[Link]()
Pie Chart
 What is a Pie Chart?
A Pie Chart is a circular statistical graphic divided into slices to illustrate numerical
proportions. Each slice of the pie represents a category's contribution to the whole
(100%).
 What is a Pie Chart?
A Pie Chart is a circular statistical graphic divided into slices to illustrate numerical
proportions. Each slice of the pie represents a category's contribution to the whole
(100%).

Key Features:
•Best for showing percentage or part-of-whole relationships
•Total of all segments should equal 100%
•Often used for categorical data
Show the market share of five smartphone brands.
Show the market share of five smartphone brands.
What is a Histogram?
 What is a Histogram?

A Histogram is a type of bar chart that shows the frequency

distribution of a continuous numerical variable. It groups data into
ranges (called bins) and counts how many values fall into each bin.

Key Features:
✓ Represents distribution of data
✓ X-axis: intervals or bins of values
✓ Y-axis: frequency (number of values in each bin)
✓ Best used for understanding data spread, central tendency, and outliers
# Histogram
import numpy as np
data = [Link](1000)
[Link](data, bins=30, color='purple')
[Link]("Histogram using Matplotlib")
[Link]("Value")
[Link]("Frequency")
[Link]()
Multiple Bar Graph
 Multiple Bar Graph

A Multiple Bar Graph (also called a grouped bar chart) displays two or more sets of
data side-by-side for the same categories.

It’s great for comparing multiple variables across the same categories (e.g., comparing
marks of 2 students in 5 subjects).
import [Link] as plt
import numpy as np
# Subjects (categories)
subjects = ['Math', 'Physics', 'Chemistry', 'Biology', 'English']
# Marks of two students
student_A = [85, 90, 78, 88, 76]
student_B = [80, 85, 82, 79, 83]
# Number of categories
x = [Link](len(subjects))
# Width of each bar
width = 0.35
# Create the grouped bar chart
[Link](x - width/2, student_A, width, label='Student A', color='skyblue')
[Link](x + width/2, student_B, width, label='Student B', color='orange')
# Labels and title
[Link]("Subjects")
[Link]("Marks")
[Link]("Marks Comparison of Two Students")
[Link](x, subjects) # Set category labels
[Link]()
# Display the plot
[Link](axis='y', linestyle='--', alpha=0.6)
plt.tight_layout()
[Link]()
Box Plot?
 Box Plot?

A Box Plot (also called a box-and-whisker plot) is a standardized way to display the
distribution of a dataset based on five summary statistics:
✓ Minimum
✓ First Quartile (Q1) – 25th percentile
✓ Median (Q2) – 50th percentile
✓ Third Quartile (Q3) – 75th percentile
✓ Maximum
It also shows outliers, which are values that fall significantly outside the typical range.
Example: Multiple Box Plots
Scatter Plot?
 What is a Scatter Plot?
A Scatter Plot is used to display the relationship (correlation) between two continuous
variables.

Each point in the plot represents one data pair (x, y).

Key Features:
➢ Shows patterns, trends, or correlation
➢ Used for bivariate data
➢ Ideal for identifying:
✓ Linear or non-linear relationships
✓ Clusters
✓ Outliers
✓ Each dot represents one student.
✓ As study hours increase, exam scores also increase — indicating a positive correlation.
Regression Line (Trend Line):
 Real-World Applications:

Domain Use Case

Education Study time vs test scores
Finance Company revenue vs expenses
Marketing Ad budget vs conversions
Healthcare Age vs blood pressure
Data Science Feature correlation in machine learning datasets
Seaborn?
 What is Seaborn?
Seaborn is a Python data visualization library built on top of Matplotlib.
It integrates well with Pandas DataFrames and simplifies the creation of
complex plots with beautiful default styles.

Why Use Seaborn?

➢ Cleaner and more attractive visualizations than plain Matplotlib

➢ Works directly with DataFrames
➢ Supports statistical plotting
➢ One-liner plots for:
▪ Distribution
▪ Categorical comparisons
▪ Correlations
▪ Heatmaps
Line Plot
Box Plot
Scatter Plot
import seaborn as sns
import pandas as pd
# Create a sample DataFrame
df = sns.load_dataset("tips")
# Boxplot
[Link](x='day', y='total_bill',
data=df)
[Link]("Box Plot using Seaborn")
[Link]()
# Scatter plot with hue
[Link](x='total_bill', y='tip',
hue='sex', data=df)
[Link]("Scatter Plot using Seaborn")
[Link]()
# Violin Plot
[Link](x='day', y='total_bill',
data=df)
[Link]("Violin Plot using Seaborn")
[Link]()
Bokeh?
 What is Bokeh?

Bokeh is an interactive visualization library in Python that targets

modern web browsers for presentation. It allows creation of
interactive, zoomable, and hoverable plots directly in HTML.
from [Link] import figure, show,
output_notebook
from [Link] import ColumnDataSource
from [Link] import output_file
output_notebook() # For Jupyter. Use
output_file() if you want HTML output.
# Sample data with string x-values
x = ['A', 'B', 'C', 'D']
y = [23, 45, 56, 78]
# Convert to ColumnDataSource
source = ColumnDataSource(data=dict(x=x,
y=y))
# Create a figure with categorical x-axis
p = figure(title="Interactive Bar Chart
using Bokeh",
x_range=x,
width=600, height=400,
toolbar_location=None)
# Create bar chart
[Link](x='x', top='y', width=0.5,
source=source, color="orange")
# Axis labels
[Link].axis_label = "Category"
[Link].axis_label = "Values"
# Show plot
show(p)
x = [1, 2, 3, 4]
y = [23, 45, 56, 78]
source = ColumnDataSource(data=dict(x=x,
y=y))
p = figure(title="Bar Chart with Numeric
X", width=600, height=400)
[Link](x='x', top='y', width=0.5,
source=source)
show(p)
The End