Data exploration and
analysis
1
What is data exploration?
• Highlight the relevant features of each attribute
contained in a dataset
• Using graphical methods and calculating summary
statistics
• Identify the intensity of the underlying relationships
among the attributes
2
Today’s contents
• Univariate analysis
• Bivariate analysis
• Multivariate analysis
3
Univariate analysis
• Used to study the behavior of each attribute
• Univariate analysis plays a key role in pointing out
anomalies and non-standard values
• For the purpose of a graphical representation, there
is different between categorical and numerical
attributes
4
Graphical analysis of categorical attributes
• The most natural representation for the graphical
analysis of a categorical attribute is a vertical bar
chart
5
Graphical analysis of numerical attributes
• Numerical attributes assuming a finite and limited
number of values
• Density or frequency histogram is used
6
Example
7
Example (results)
8
Frequency vs. Density histogram
9
Measures of central tendency
and dispersion for numerical attributes
• Mean
• Median
• Mode
• Variance
• Standard Deviation
10
A sample of dogs was measured
The heights (at the shoulders) are:
600mm, 470mm, 170mm, 430mm and 300mm.
Image from: [Link]
Find the MEAN
MEAN ( ): average of all data points
Mean height is 394 mm
Image from: [Link]
FIND the MEDIAN
The heights (at the shoulders) are:
600mm, 470mm, 170mm, 430mm and 300mm.
.
Image from: [Link]
MEDIAN:
Data point that is found in the middle when all
the measurements are lined up from in order
from high to low
(or an average of middle two values if there is
an even number of data points
170mm 300mm 430mm 470 mm 600mm
.
FIND the MODE
The heights (at the shoulders) are:
600mm, 470mm, 170mm, 430mm and 300mm.
.
Image from: [Link]
MODE: Most common data point
170mm 300mm 430mm 470 mm 600mm
There is no mode for these data
The heights (at the shoulders) are:
600mm, 470mm, 170mm, 430mm and 300mm.
Determine the VARIANCE.
How far is each measurement away from the mean?
Image from: [Link]
VARIANCE (= s2)
s2 = (600 – 394)2 + ( 470- 394) 2 + ( 170- 394)2 + ( 430-394)2 +( 470- 394) 2
5-1
s2 = 206 2 + 76 2 + (-224)2 + 36 2 + (-94) 2
4
s2 = 21,704
[Link]
STANDARD DEVIATION (s)
Watch Bozeman Biology video
Standard Deviation= square root of Variance
σ = √21,704 = 147.32... = 147mm
Standard Deviation is useful. Now we can show which heights are
within one Standard Deviation (147mm) of the Mean:
Rottweilers are tall dogs and Dachshunds are a bit short
Image from: [Link]
STANDARD DEVIATION
the standard deviation of the sample is the degree to
which individuals within the sample differ from the sample
mean.
STANDARD ERROR OF THE MEAN
the standard error of the mean is an estimate of how far
the sample mean is likely to be from the population mean
Bozeman Biology Standard Error
In a normal distribution
68.2% within 1 standard deviation from mean
95.4% within 2 standard deviations from mean
99.7% within 3 standard deviations from mean
[Link]
Bivariate analysis
• It is useful to distinguish three cases that may occur
within bivariate analysis
• Both attributes are numerical
• One attribute is numerical and the other is categorical
• Both attributes are categorical.
22
Graphical analysis
Scatter plot
Loess plot 23
Graphical analysis (cont.)
Time series
24
Multivariate analysis
• The purpose of multivariate analysis is to extend the
concepts introduced for the bivariate case in order
to assess the relationships existing among multiple
attributes in a dataset
25
Graphical analysis
Scatter plot matrix
26
Graphical analysis (cont.)
Star plots
27
Next lecture
• Regression analysis
28
References
• Vercellis, Carlo. Business intelligence: data mining
and optimization for decision making. New York:
Wiley, 2009
• [Link]
de/120/
• [Link]
[Link]
29