lOMoARcPSD|59726953

Unit-3 DS

[Link]. Computer Science (Bharathiar University)

Scan to open on Studocu

Studocu is not sponsored or endorsed by any college or university

Downloaded by Aditya Joshi (adityakjoshi01@[Link])
 lOMoARcPSD|59726953

DATA SCIENCE – STUDY MATERIALS

UNIT-III

Exploratory Data Analytics: Descriptive Statistics - Mean, Standard Deviation, Skewness and Kurtosis -
Box Plots - Pivot Table - Heat Map - Correlation Statistics – ANOVA

EXPLORATORY DATA ANALYTICS (EDA)

Exploratory Data Analysis (EDA) is an approach that is used to analyze the data and discover trends,
patterns, or check assumptions in data with the help of statistical summaries and graphical
representations.

Exploratory Data Analysis (EDA) is an approach that is used to analyze the data and discover trends,
patterns, or check assumptions in data with the help of statistical summaries and graphical
representations.

Types of EDA

Depending on the number of columns we are analyzing we can divide EDA into two types.

1. Univariate Analysis – In univariate analysis, we analyze or deal with only one variable at a time.
The analysis of univariate data is thus the simplest form of analysis since the information deals
with only one quantity that changes. It does not deal with causes or relationships and the main
purpose of the analysis is to describe the data and find patterns that exist within it.
2. Bi-Variate analysis – This type of data involves two different variables. The analysis of this type
of data deals with causes and relationships and the analysis is done to find out the relationship
between the two variables.
3. Multivariate Analysis – When the data involves three or more variables, it is categorized under
multivariate.

Depending on the type of analysis we can also subcategorize EDA into two parts.

1. Non-graphical Analysis – In non-graphical analysis, we analyze data using statistical tools like
mean median or mode or skewness
2. Graphical Analysis – In graphical analysis, we use visualizations charts to visualize trends and
patterns in the data

DESCRIPTIVE STATISTICS

Downloaded by Aditya Joshi (adityakjoshi01@[Link])

 lOMoARcPSD|59726953

Descriptive statistics summarizes or describes the characteristics of a data set. Descriptive statistics
consists of three basic categories of measures: measures of central tendency, measures of variability (or
spread), and frequency distribution.

 Descriptive statistics summarizes or describes the characteristics of a data set.

 Descriptive statistics consists of three basic categories of measures: measures of central tendency,
measures of variability (or spread), and frequency distribution.
 Measures of central tendency describe the center of the data set (mean, median, mode).
 Measures of variability describe the dispersion of the data set (variance, standard deviation).
 Measures of frequency distribution describe the occurrence of data within the data set (count).

Types of Descriptive Statistics

All descriptive statistics are either measures of central tendency or measures of variability, also known as
measures of dispersion.

Central Tendency
Measures of central tendency focus on the average or middle values of data sets, whereas measures of
variability focus on the dispersion of data. These two measures use graphs, tables and general discussions
to help people understand the meaning of the analyzed data.

Measures of Variability
Measures of variability (or the measures of spread) aid in analyzing how dispersed the distribution is for a
set of data. For example, while the measures of central tendency may give a person the average of a data
set, it does not describe how the data is distributed within the set.

Consider the following data set: 5, 19, 24, 62, 91, 100. The range of that data set is 95, which is calculated
by subtracting the lowest number (5) in the data set from the highest (100).

Distribution

Distribution (or frequency distribution) refers to the quantity of times a data point occurs. Alternatively, it
is the measurement of a data point failing to occur. Consider a data set: male, male, female, female,
female, other. The distribution of this data can be classified as:

 The number of males in the data set is 2.

Downloaded by Aditya Joshi (adityakjoshi01@[Link])

 lOMoARcPSD|59726953

 The number of females in the data set is 3.

 The number of individuals identifying as other is 1.
 The number of non-males is 4.

MEAN, STANDARD DEVIATION

MEAN

Mean is used to find the average value around which your data values range. Generally, when working
with data, you may want to know the average data value. This will give you a term that incorporates every
data value from the dataset.

The mean (average) of a data set is found by adding all numbers in the data set and then dividing by the
number of values in the set. The median is the middle value when a data set is ordered from least to
greatest.

What is mean and mode in data science?

To find the mean, add up the values in the data set and then divide by the number of values that you
added. To find the median, list the values of the data set in numerical order and identify which value
appears in the middle of the list. To find the mode, identify which value in the data set occurs most often.

Example:

Mean is the average of the given numbers and is calculated by dividing the sum of given numbers by the
total number of numbers. Mean = (Sum of all the observations/Total number of observations)

STANDARD DEVIATION

A standard deviation (or σ) is a measure of how dispersed the data is in relation to the mean. Low
standard deviation means data are clustered around the mean, and high standard deviation indicates data
are more spread out.

Downloaded by Aditya Joshi (adityakjoshi01@[Link])

 lOMoARcPSD|59726953

The standard deviation is a summary measure of the differences of each observation from the mean.
If the differences themselves were added up, the positive would exactly balance the negative and so
their sum would be zero. Consequently the squares of the differences are added.

SKEWNESS AND KURTOSIS

Skewness and Kurtosis are used to describe the spread and height of your normal distribution. Skewness
is used to denote the horizontal pull on the data. It tells you how spread out the data is, and Kurtosis is
used to find the vertical pull or the peak's height.

What is the difference between kurtosis and skewness?

Skewness and kurtosis are both important measures of a distribution's shape. Skewness measures the
asymmetry of a distribution. Kurtosis measures the heaviness of a distribution's tails relative to a normal
distribution.

Skewness measures the degree of asymmetry of the distribution, while Kurtosis measures the degree of
peakedness and flatness of a distribution. Skewness is the third measure of moments, while kurtosis is the
fourth measure of moments. The value of both Skewness and Kurtosis ranges from -infinity to +infinity.

BOX PLOTS

What Is a Boxplot?

A boxplot is a graph that gives a visual indication of how a data set’s mean, median, mode, minimum,
maximum and outlier values are spread out and compare to each other.

Downloaded by Aditya Joshi (adityakjoshi01@[Link])

 lOMoARcPSD|59726953

Box plot in Data Science

A boxplot is a standardized way of displaying the distribution of data based on its five-number summary
(“minimum”, first quartile [Q1], median, third quartile [Q3] and “maximum”). Boxplots can tell you
about your outliers and their values, if your data is symmetrical, how tightly your data is grouped and if
and how your data is skewed.

A boxplot, also known as a box plot, box plots or box-and-whisker plot, is a standardized way of
displaying the distribution of a data set based on its five-number summary of data points: the “minimum,”
first quartile [Q1], median, third quartile [Q3] and “maximum.” Here’s an example.

 Median (Q2/50th percentile): The middle value of the data set

 First Quartile (Q1/25th percentile): The middle number between the smallest number (not the
“minimum”) and the median of the data set

Downloaded by Aditya Joshi (adityakjoshi01@[Link])

 lOMoARcPSD|59726953

 Third Quartile (Q3/75th percentile): The middle value between the median and the highest
value (not the “maximum”) of the dataset
 Interquartile Range (IQR): 25th to the 75th percentile
 Whiskers (shown in blue)
 Outliers (shown as green circles)
 “Minimum”: Q1 - 1.5*IQR
 “Maximum”: Q3 + 1.5*IQR

Boxplots can tell you about your outliers and what their values are. They can also tell you if your data is
symmetrical, how tightly your data is grouped and if and how your data is skewed.

PIVOT TABLE

A pivot table is a statistics tool that summarizes and reorganizes selected columns and rows of data in a
spreadsheet or database table to obtain a desired report.

A pivot table is a statistics tool that summarizes and reorganizes selected columns and rows of data in a
spreadsheet or database table to obtain a desired report. The tool does not actually change the spreadsheet
or database itself, it simply “pivots” or turns the data to view it from different perspectives.

Importance of Pivot table in Excel

A PivotTable is a powerful tool to calculate, summarize, and analyze data that lets you see comparisons,
patterns, and trends in your data. PivotTables work a little bit differently depending on what platform you
are using to run Excel.

For Example:

Downloaded by Aditya Joshi (adityakjoshi01@[Link])

 lOMoARcPSD|59726953

Excel Pivot Table Calculated Field formula to be used like sumproduct

A pivot table is a powerful data analysis tool that allows you to summarize and aggregate data based on
different dimensions. In Python, you can create pivot tables using the pandas library, which provides
flexible and efficient tools for data manipulation and analysis.

HEAT MAP

A heat map (or heatmap) is a 2-dimensional data visualization technique that represents the magnitude of
individual values within a dataset as a color. The variation in color may be by hue or intensity.

A heat map helps you visualize density. And in the case of web design and analysis, it helps you visualize
how far people scroll on your site, where they click and even sometimes where they're looking.

Why use heatmap in data science?

By definition, Heat Maps are graphical representations of data that utilize color-coded systems. The
primary purpose of Heat Maps is to better visualize the volume of locations/events within a dataset and
assist in directing viewers towards areas on data visualizations that matter most. But they're much more
than that.

Heat Map Adaptations

Retail Matrix

Imagine working as an analyst for a large, multi-national retail corporation that operates a chain of large
department stores. You’re responsible for analyzing their flagship store, which is located in the downtown
core of a large metropolis. What you’re interested in, is determining whether or not specific dates and
times receive more traffic, in order to better allocate in-store resources.

Downloaded by Aditya Joshi (adityakjoshi01@[Link])

 lOMoARcPSD|59726953

Retail Matrix in Dundas BI

This Heat Map is displaying the dates and times whereby the store is most heavily trafficked. With this,
we’re able to immediately identify which dates and hours are busiest. For example, it’s clear that the store
experiences an influx of traffic on Monday’s beginning at 10:00 AM, which slowly dissipates throughout
the day. This pattern is repeated throughout the week, with minor daily variances (for example, Friday’s
tend to experience prolonged periods of traffic, whereas Sunday’s tend to be lighter).

Armed with these insights, the retailer is better positioned to maximize their operations by better
allocating key resources; for example, by scheduling additional Sales Associates during lunch rushes, or
by scheduling inventory shipments during down periods earlier in the week.

Manufacturing Diagram

Any manufacturer intent on improving their operations must be willing to analyze data relevant to their
production bays. In addition to having access to crucial KPIs such as Production Count, Downtime, or
Cycle Time, manufacturers must also be able to monitor the performance of their production bays in real-
time in order to identify critical issues as they arise and intervene as required. After all, production bays
are most effective when all components (machines, equipment, and labor) are firing on all cylinders.

Manufacturing Diagram in Dundas BI

This Custom Diagram of a multi-bay production facility, is displaying which production bays are
contributing the most defects from a monetary perspective. In this example, the darker the production
bay, the higher the loss attributable to defects. It’s easy to immediately see that production bays 1 and 11

Downloaded by Aditya Joshi (adityakjoshi01@[Link])

 lOMoARcPSD|59726953

are producing defects at an alarming rate, while production bays 3, 4, and 7 are performing within their
expected thresholds.

This rendition of a Heat Map is great to use on a high-level dashboard as it allows for at-a-glance
monitoring of which production bays are in good condition, which require attention, and which are at risk.
Once the statuses have been identified, the manufacturer is then prompted to perform deeper analysis and
identify the root causes.

POPULATION MAP

As mentioned earlier, Heat Maps work wonders on literal maps, because of the defined boundaries that
are relevant to the data. In these scenarios, Heat Maps can be used to represent population density, per-
capita income, average temperatures, and so on. Take the following Bar Chart for example, which
features the populations of American states and Canadian provinces. It’s equally as difficult (if not more)
to analyze as a Bar Chart, as you’d imagine it would be as a Table.

Population Bar Chart in Dundas BI

The best way to interpret this data is by using a Map visualization; specifically one that incorporates Heat
Map components. The map leverages our knowledge of geography in place of text labels, and the colors
attributed by the Heat Map, indicate approximate values.

Figure 4: Population Map in Dundas BI

CORRELATION STATISTICS

Correlation refers to the statistical relationship between the two entities. It measures the extent to which
two variables are linearly related.

Example

Downloaded by Aditya Joshi (adityakjoshi01@[Link])

 lOMoARcPSD|59726953

Correlation refers to the statistical relationship between the two entities. It measures the extent to which
two variables are linearly related. For example, the height and weight of a person are related, and taller
people tend to be heavier than shorter people. You can apply correlation to a variety of data sets.

There are three types of correlation:

 Positive Correlation: A positive correlation means that this linear relationship is positive, and
the two variables increase or decrease in the same direction.
 Negative Correlation: A negative correlation is just the opposite. The relationship line has a
negative slope, and the variables change in opposite directions, i.e., one variable decreases while
the other increases.
 No Correlation: No correlation simply means that the variables behave very differently and thus,
have no linear relationship.

What is Correlation Coefficient?

 Correlation coefficients give you the measure of the strength of the linear relationship between
two variables.
 The letter r denotes the value, and it ranges between -1 and +1
 If r < 0, it implies negative correlation
 If r > 0, it implies positive correlation
 If r = 0, it implies no correlation
 Calculating the correlation coefficient takes time; therefore, data is entered into a calculator,
computer, or statistics program to calculate the correlation coefficient.

Correlation examples are 3 types of correlation research: Naturalistic observation research, survey
research and archival research.

ANOVA

ANOVA (Analysis of variance) is to test for differences among the means of the population by examining
the amount of variation within each sample, relative to the amount of variation between the samples.
Analyzing variance tests the hypothesis that the means of two or more populations are equal.

One-way example of ANOVA

The dependent variable would be “weight loss,” measured in kilograms, and the independent variable
would be “diet type,” which has three groups or levels: “diet type 1”, “diet type 2,” and “diet type 3”.

Downloaded by Aditya Joshi (adityakjoshi01@[Link])

 lOMoARcPSD|59726953

 Data samples should be taken from the normally distributed population.

 Samples should be selected randomly to ensure that observations in one sample do not depend on
the observations in other samples. In essence, samples should be independent.
 The dependent variable should be continuous.
 The variance of data belonging to different groups should be the same.
 In a group, there should not be any data point exhibiting an unusual pattern.
 The independent variable should contain three or more levels.

Applications of ANOVA

ANOVA is used in statistics when you are testing a hypothesis to understand how different groups
respond to each other by making connections between independent and dependent variables.

Advantage of ANOVA

ANOVA, or its non-parametric counterparts, allow you to determine if differences in mean values
between three or more groups are by chance or if they are indeed significantly different. ANOVA is
particularly useful when analyzing the multi-item scales common in market research.

Downloaded by Aditya Joshi (adityakjoshi01@[Link])