Module 5
Exploratory Data Analysis
CO5:Choose and apply appropriately from a wider range of exploratory and inferential
methods for analyzing data and will be able to evaluate and interpret the results
contextually.
Data Analysis
The process of cleaning,transferring,interpreting analysis and then visualising this
data to extract useful information and gain valuable insights to make more
effective business decisions is called Data analysis.
Types:
[Link] data analysis
[Link] data analysis
[Link] data analysis
[Link] data analysis
[Link] data analysis
[Link] data analysis
Exploratory Data Analysis
➔ helps to determine how best to manipulate data sources to get the
answers you need, making it easier for data scientists to discover
patterns, spot anomalies, test a hypothesis, or check
assumptions.
➔ used to see what data can reveal beyond the formal modeling or
hypothesis testing task and provides a better understanding of
data set variables and the relationships between them.
➔ Originally developed by American mathematician John Tukey in the
1970s, EDA techniques continue to be a widely used method in the
data discovery process today.
➔ Exploratory data analysis (EDA) is used by data scientists to analyze
and investigate data sets and summarize their main characteristics,
often employing data visualization methods.
Why is exploratory data analysis important in data science?
➔ Important role in data Science.
➔ The main purpose of EDA is to help look at data before making any
assumptions.
➔ It can help identify obvious errors, as well as better understand
patterns within the data, detect outliers or anomalous events, find
interesting relations among the variables.
➔ EDA can help answer questions about standard deviations, categorical
variables, and confidence intervals.
➔ Data scientists can use exploratory analysis to ensure the results
they produce are valid and applicable to any desired business
outcomes and goals.
➔ EDA also helps stakeholders by confirming they are asking the
right questions.
➔ Once EDA is complete and insights are drawn, its features can
then be used for more sophisticated data analysis or modeling,
including machine learning.
The main reasons to use EDA:
• detection of mistakes
• checking of assumptions
• preliminary selection of appropriate models
• determining relationships among the explanatory variables
• assessing the direction and rough size of relationships between
explanatory and outcome variables.
Five Number Summary
The five number summary is a set of five descriptive [Link] make useful
starting point for any exploratory data analysis.
It includes the fixed most common sample percentiles.
1. The sample minimum(Smallest observation)
2. The lower quartile
3. The median
4. The upper quartile
5. The sample maximum
This helps to describe the overall spread of the data
Five Number Summary
Exploratory data analysis tools
Specific statistical functions and techniques can perform with EDA tools include:
➔ Clustering and dimension reduction techniques, which help create graphical
displays of high-dimensional data containing many variables.
➔ Univariate visualization of each field in the raw dataset, with summary
statistics.
➔ Bivariate visualizations and summary statistics that allow you to assess the
relationship between each variable in the dataset and the target variable .
➔ Multivariate visualizations, for mapping and understanding interactions
between different fields in the data.
➔ K-means Clustering is a clustering method in unsupervised learning where
data points are assigned into K groups, i.e. the number of clusters, based on
the distance from each group’s centroid.
➔ K-means Clustering is commonly used in market segmentation, pattern
recognition, and image compression.
➔ Predictive models, such as linear regression, use statistics and data to
predict outcomes.
➔Qualitative (Categorical) data:
They are an observed phenomenon and cannot be measured with numbers.
Examples: a race, age group, gender, origin, and so on.
➔Quantitative data:
➔ The quantities of things or the things we can measure. expressed in terms of
numbers.
➔It is also known as numerical data and includes statistical data analysis.
Examples: height, water, distance, and so on.
Types of exploratory data analysis
➔ Univariate non-graphical
➔ Univariate graphical
➔ Multivariate non-graphical
➔ Multivariate graphical
Univariate graphical EDA
➔involves creating charts and graphs to explore a single variable. This can help
you understand the distribution of the data and identify any outliers.
➔Common sorts of univariate graphics are:
➔Histogram
➔Stem-and-leaf plots
➔Box Plots
➔Quantile-normal plots
● Univariate analysis means analysis of one variable or one feature.
● Univariate analysis basically tells us how data in each feature is distinguished
and also tells us about the central tendencies like median ,mean and mode
Methods of univariate Distribution
1. Frequency Distribution
2. Bar-Charts
3. Bar Graph
4. Histogram
5. Pie-diagram
6. Frequency Polygon
[Link]
Represent the adjoining distribution of marks of 100
students in the examination by a histogram
Marks Obtained No. of students
Less than 10 4
Less than 20 6
Less than 30 24
Less than 40 46
Less than 50 67
Less than 60 86
Less than 70 96
Less than 80 99
Solution:First we convert the given cumulative frequency distribution into the frequency
distribution of marks in each subject
Marks No. of students
0- 10 4
10- 20 6-4=2
20- 30 24-6=18
30- 40 46-24=22
40-50 67-46=21
50-60 86-67=19
60-70 96-86=10
70-80 99-96=3
80-90 100-99=1
Draw Histogram
No. Of
students
Marks
The following data represent expenditure by a state government
for the year 1997-98. Draw a Pie -diagram
Items Agriculture Industries and Health and Miscellaneous
and rural urban education
development development
Proposed 4200 1500 1000 500
expediting in
million Rs.
Item Proposed expenditure(in Angle at the centre
million Rs.)
1) 2) 3) (2/7200)*360
Agriculture and rural 4200 4200/7200 *360=210
development
Industries and urban 1500 75
development
Health and education 1000 50
Miscellaneous 500 25
Total 7200 360
The following data show the number of accidents sustained by
313 drivers of a public utility company over a period of 5 years.
Draw the frequency polygon
No. 0 1 2 3 4 5 6 7 8 9 10 11
of
accid
ents
No of 80 44 68 41 25 20 13 7 5 4 3 2
Drive
rs
Multivariate Analysis
Types of Multivariate analysis:
[Link] Tree
[Link] correlation Analysis
[Link] Analysis
[Link] Analysis
[Link] correspondence analysis
[Link] algorithm
ANOVA Examples
[Link]
Sum of squares between groups :SSB
Sum of squares of errors :SSE
Total sum of squares: SST = SSB + SSE
Degrees of freedom between groups, df1 = k - 1. Here, k denotes the number of groups.
Degrees of freedom of errors, df2 = N - k, where N denotes the total number of observations across k groups.