Descriptive Statistics

18CSE202- Foundations of Data Science

Today’s discussion…
— Introduction

— Data summarization
— Measurement of location
— Mean, median, mode, midrange, etc.

— Measure of dispersion
— Range, Variance, Standard Deviation, etc.

— Other measures
— MAD, AAD, Percentile, IQR, etc.

• Graphical summarization
• Box plot

18CSE202- Foundations of Data Science

TRP: An example
— Television rating point (TRP) is a tool provided
to judge which programs are viewed the most.
— This gives us an index of the choice of the people
and also the popularity of a particular channel.

— For calculation purpose, a device is attached to the TV sets in few

thousand viewers’ houses in different geographic and demographic sectors.
— The device is called as People's Meter. It reads the time and the programme
that a viewer watches on a particular day for a certain period.

— An average is taken, for example, for a 30-days period.

— The above further can be augmented with a personal interview survey

(PIS), which becomes the basis for many studies/decision making.
— Essentially, we are to analyze data for TRP estimation.
18CSE202- Foundations of Data Science
Defining Data
Definition 3.1: Data

A set of data is a collection of observed values representing one or more

characteristics of some objects or units.

Example: For TRP, data collection consist of the following attributes.

— Age: A viewer’s age in years
— Gender: A viewer’s gender coded 1 for male and 0 for female
— Happy: A viewer’s general happiness
— NH for not too happy
— PH for pretty happy
— VH for very happy
— TVHours: The average number of hours a respondent watched TV during a day

18CSE202- Foundations of Data Science

Defining Data
Viewer# Age Gender Happy TVHours
… … … … …
… … … … …
55 34 F VH 5
… … … … …

Note:
— A data set is composed of information from a set of units.

— Information from a unit is known as an observation.

— An observation consists of one or more pieces of information about a unit; these are
called variables.

18CSE202- Foundations of Data Science

Defining Population
Definition 3.2: Population

A population is a data set representing the entire entities of interest.

Example: All TV Viewers in the country/world.

Note:
1. All people in the country/world is not a population.

2. For different survey, the population set may be completely different.

3. For statistical learning, it is important to define the population that we intend to study
very carefully.

18CSE202- Foundations of Data Science

Defining Sample
Definition 3.3: Sample

A sample is a data set consisting of a population.

Example: All students studying in Class XII is a sample, whereas those students
belong to a given school is population.

Note:
— Normally a sample is obtained in such a way as to be representative of the population.

18CSE202- Foundations of Data Science

Defining Statistics
Definition 3.4: Statistics

A statistics is a quantity calculated from data that describes a particular

characteristics of a sample.

Example: The sample mean (denoted by 𝑦)! is the arithmetic mean of a

variable of all the observations of a sample.

18CSE202- Foundations of Data Science

Defining Statistical Inference
Definition 3.5: Statistical inference

Statistical inference is the process of using sample statistics to make

decisions about population.

Example: In the context of TRP

— Overall frequency of the various levels of happiness.

— Is there a relationship between the age of a viewers and his/her general happiness?

— Is there a relationship between the age of the viewer and the number of TV hours
watched?

18CSE202- Foundations of Data Science

Data Summarization
— To identify the typical characteristics of data (i.e., to have an overall picture).

— To identify which data should be treated as noise or outliers.

— The data summarization techniques can be classified into two broad

categories:

— Measures of location

— Measures of dispersion

18CSE202- Foundations of Data Science

Measurement of location
— It is also alternatively called as measuring the central tendency.
— A function of the sample values that summarizes the location information into a single
number is known as a measure of location.

— The most popular measures of location are

— Mean
— Median
— Mode
— Midrange

— These can be measured in three ways

— Distributive measure
— Algebraic measure
— Holistic measure

18CSE202- Foundations of Data Science

Distributive measure
— It is a measure (i.e. function) that can be computed for a given data set by
partitioning the data into smaller subsets, computing the measure for each
subset, and then merging the results in order to arrive at the measure’s value
for the original (i.e. entire) data set.

Example
ü sum(), count()

18CSE202- Foundations of Data Science

Algebraic measure
— It is a measure that can be computed by applying an algebraic function to one or
more distributive measures.

— Example
#$%( )
average =
()$*+( )

18CSE202- Foundations of Data Science

Holistic measure
— It is a measure that must be computed on the entire data set as a whole.

— Example
Calculating median
What about mode?

18CSE202- Foundations of Data Science

Mean of a sample
,. Different mean measurements
— The mean of a sample data is denoted as 𝒙
known are:
— Simple mean
— Weighted mean
— Trimmed mean

— In the next few slides, we shall learn how to calculate the mean of a sample.

— We assume that given 𝑥0 , 𝑥1 , 𝑥2 ,….., 𝑥3 are the sample values.

18CSE202- Foundations of Data Science

Simple mean of a sample
— Simple mean
It is also called simply arithmetic mean or average and is abbreviated as
(AM).

Definition 3.6: Simple mean

ü If 𝑥0 , 𝑥1 , 𝑥2 ,….., 𝑥3 are the sample values, the simple mean is

defined as
𝒏
𝟏
, = 7 xi
𝒙
𝒏
𝒊9𝟏

18CSE202- Foundations of Data Science

Weighted mean of a sample
— Weighted mean
It is also called weighted arithmetic mean or weighted average.

Definition 3.7: Weighted mean

When each sample value 𝑥: is associated with a weight 𝑤: , for i =

1,2,…,n, then it is defined as
∑3:90 wixi
,= 𝒏
𝒙
∑𝒊9𝟏 wi

Note
When all weights are equal, the weighted mean reduces to simple mean.
18CSE202- Foundations of Data Science
Trimmed mean of a sample
— Trimmed Mean
If there are extreme values (also called outlier) in a sample, then the mean is
influenced greatly by those values. To offset the effect caused by those
extreme values, we can use the concept of trimmed mean

Definition 3.8: Trimmed mean

Trimmed mean is defined as the mean obtained after chopping off

values at the high and low extremes.

18CSE202- Foundations of Data Science

Properties of mean
— Lemma 3.1
If 𝒙𝒊 , i = 1,2,…,m are the means of m samples of sizes 𝒏𝟏 , 𝒏𝟐 ,….., 𝒏𝒎
respectively, then the mean of the combined sample is given by:-
∑𝒎
𝒊9𝟏 𝒏𝒊 𝒙𝒊
𝒙=
∑𝒎
𝒊9𝟏 𝒏𝒊
(Distributive Measure)
— Lemma 3.2
ü If a new observation 𝒙𝒌 is added to a sample of size n with mean 𝒙, the
new mean is given by
@ 𝒏 𝒙 + 𝒙𝒌
𝒙 =
𝒏+𝟏

18CSE202- Foundations of Data Science

Properties of mean
— Lemma 3.3
If an existing observation 𝒙𝒌 is removed from a sample of size n with mean 𝒙,
the new mean is given by
@ 𝒏 𝒙 − 𝒙𝒌
𝒙 =
𝒏−𝟏
— Lemma 3.4
If m observations with mean 𝒙𝒎, are added (removed) from a sample of size n
with mean 𝒙𝒏 , then the new mean is given by

𝒏 𝒙𝒏 ± 𝒎 𝒙𝒎
𝒙=
𝒏±𝒎

18CSE202- Foundations of Data Science

Properties of mean
— Lemma 3.5
If a constant c is subtracted (or added) from each sample value, then the mean
of the transformed variable is linearly displaced by c. That is,
@
𝒙 = 𝒙∓𝒄
— Lemma 3.6
If each observation is called by multiplying (dividing) by a non-zero constant,
then the altered mean is given by
@
𝒙 = 𝒙∗𝒄
Where, * is x (multiplication) or ÷ (division) operator.

18CSE202- Foundations of Data Science

Mean with grouped data
Sometimes data is given in the form of classes and frequency for each class.
Class à 𝑥0 - 𝑥1 𝑥1 - 𝑥2 ….. 𝑥: - 𝑥:G0 ….. 𝑥3H0 - 𝑥3
Frequency à 𝑓0 𝑓1 ….. 𝑓: ….. 𝑓3

There three methods to calculate the mean of such a grouped data.

• Direct method

• Assumed mean method

• Step deviation method

18CSE202- Foundations of Data Science

Direct method

— Direct Method
∑𝒏𝒊9𝟏 fi xi
,=
𝒙
∑𝒏𝒊9𝟏 fi

𝟏 xi+ xi+1
Where, xi = (lower limit + upper limit) of the ith class, i.e., xi =
𝟐 𝟐
(also called class size), and fi is the frequency of the ith class.

Note
∑ fi (xi - 𝒙
,) = 0

18CSE202- Foundations of Data Science

Assumed mean method
— Assumed Mean Method

∑𝒏𝒊9𝟏 fi di
,=𝑨+
𝒙
∑𝒏𝒊9𝟏 fi

x+x
where, A is the assumed mean (it is usually a value xi = i i+1
𝟐
chosen in the middle of the groups di = (𝑨 - xi ) for each i )

18CSE202- Foundations of Data Science

Step deviation method

— Step deviation method

∑𝒏𝒊9𝟏 fi ui
,=𝑨+
𝒙 𝒏 𝒉
∑𝒊9𝟏 fi
where,
A = assumed mean
h = class size (i.e., 𝐱 𝐢G𝟏 - 𝐱 𝐢 for the ith class)
x −A
ui = i
𝒉

18CSE202- Foundations of Data Science

Mean for a group of data
— For the above methods, we can assume that…
— All classes are equal sized

— Groups are with inclusive classes, i.e., xi = 𝐱 𝐢H𝟏 (linear limit of a class
is same as the upper limit of the previous class)

10 - 19 20 - 29 30 - 39 40 − 49

Data with exclusive classes

9.5 – 19.5 19.5 – 29.5 29.5 – 39.5 39.5 – 49.5

Data with inclusive classes

18CSE202- Foundations of Data Science

Ogive: Graphical method to find mean
— Ogive (pronounced as O-Jive) is a cumulative frequency polygon graph.
— When cumulative frequencies are plotted against the upper (lower) class
limit, the plot resembles one side of an Arabesque or ogival architecture,
hence the name.
— There are two types of Ogive plots
— Less-than (upper class vs. cumulative frequency)
— More than (lower class vs. cumulative frequency)

Example:
Suppose, there is a data relating the marks obtained by 200 students in an
examination

444, 412, 478, 467, 432, 450, 410, 465, 435, 454, 479, …….

(Further, suppose it is observed that the minimum and maximum marks

are 410, 479, respectively.)

18CSE202- Foundations of Data Science

Ogive: Cumulative frequency table
444, 412, 478, 467, 432, 450, 410, 465, 435, 454, 479, …….

Step 1: Draw a cumulative frequency table

Conversion
into No. of Cumulative
Marks
exclusive students Frequency
series
(x) (f) (C.M)
410-419 409.5-419.5 14 14
420-429 419.5-429.5 20 34
430-439 429.5-439.5 42 76
440-449 439.5-449.5 54 130
450-459 449.5-459.5 45 175
460-469 459.5-469.5 18 193
470-479 469.5-479.5 7 200

18CSE202- Foundations of Data Science

Ogive: Graphical method to find mean
Conversion
into No. of Cumulative
Marks
exclusive students Frequency
series
(x) (f) (C.M)
410-419 409.5-419.5 14 14
420-429 419.5-429.5 20 34
430-439 429.5-439.5 42 76
440-449 439.5-449.5 54 130
450-459 449.5-459.5 45 175
460-469 459.5-469.5 18 193
470-479 469.5-479.5 7 200

Step 2: Less-than Ogive graph

Cumulative
Upper class
Frequency
Less than 419.5 14
Less than 429.5 34
Less than 439.5 76
Less than 449.5 130
Less than 459.5 175
Less than 469.5 193
Less than 479.5 200

18CSE202- Foundations of Data Science

Ogive: Graphical method to find mean
Conversion
into No. of Cumulative
Marks
exclusive students Frequency
series
(x) (f) (C.M)
410-419 409.5-419.5 14 14
420-429 419.5-429.5 20 34
430-439 429.5-439.5 42 76
440-449 439.5-449.5 54 130
450-459 449.5-459.5 45 175
460-469 459.5-469.5 18 193
470-479 469.5-479.5 7 200

Step 3: More-than Ogive graph

Cumulative
Upper class
Frequency
More than 409.5 200
More than 419.5 186
More than 429.5 166
More than 439.5 124
More than 449.5 70
More than 459.5 25
More than 469.5 7

18CSE202- Foundations of Data Science

Information from Ogive
— Mean from Less-than Ogive — Mean from More-than Ogive

— A % C freq of .65 for the third class 439.5.....449.5 means that 65% of all
scores are found in this class or below.

18CSE202- Foundations of Data Science

Information from Ogive
— Less-than and more-than Ogive approach

A cross point of two Ogive plots gives the mean of the sample

18CSE202- Foundations of Data Science

Some other measures of mean
— There are three mean measures of location:

— Arithmetic Mean (AM)

— Geometric mean (GM)

— Harmonic mean (HM)

18CSE202- Foundations of Data Science

Some other measures of mean
— Arithmetic Mean (AM)
— Harmonic Mean (HM)
— 𝑆: 𝑥0 , 𝑥1
— 𝑆: 𝑥0 , 𝑥1
\ G\
̅ ] ^
— 𝑥= 1
1 — 𝑥` = ] ]
— 𝑥̅ − 𝑥0 = 𝑥1 − 𝑥̅ G
a] a^
1 0 0
— = +
\` \] \^

— Geometric mean (GM)

— 𝑆: 𝑥0 , 𝑥1
— 𝑥_ = 𝑥0 . 𝑥1
\] \_
— =
\_ \^

18CSE202- Foundations of Data Science

???
— Is there any generalization for AM (, 𝒙), GM (b
𝒙) and
HM (c 𝒙) calculations for a sample of size ≥ 2?

— In which situation, a particular mean is applicable?

— If there is any interrelationship among them?

18CSE202- Foundations of Data Science

Geometric mean
Definition 3.9: Geometric mean

Geometric mean of n observations (none of which are zero) is defined as:

𝒏 𝟏/𝒏

b=
𝒙 g xi
𝒊9𝟏
where, n ≠ 0

Note
— GM is the arithmetic mean in “log space”. This is because, alternatively,
𝒏
𝟏
𝒍𝒐𝒈b
𝒙 = 7 𝒍𝒐𝒈 𝒙𝒊
𝒏
𝒊9𝟏

— This summary of measurement is meaningful only when all observations are > 0
— If at least one observation is zero, the product will itself be zero! For a negative value, root is not real
18CSE202- Foundations of Data Science
Harmonic mean
Definition 3.10: Harmonic mean

If all observations are non zero, the reciprocal of the arithmetic mean of the
reciprocals of observations is known as harmonic mean.

For ungrouped data

𝒏
c=
𝒙
𝟏
∑𝒏𝒊9𝟏
xi
For grouped data
∑𝒏𝒊9𝟏 fi
c=
𝒙
fi
∑𝒏𝒊9𝟏
xi
where, fi is the frequency of the ith class with xi as the center value of the
ith class.

18CSE202- Foundations of Data Science

Significant of different mean calculations
— There are two things involved when we consider a sample
— Observation
— Range

Example: Rainfall data

Rainfall (in r1 r2 … rn
mm)
Days d1 d2 … dn
(in number)
— Here, rainfall is the observation and day is the range for each element in
the sample

— Here, we are to measure the mean “rate of rainfall” as the measure of

location

18CSE202- Foundations of Data Science

Significant of different mean calculations
— Case 1: Range remains same for each observation

Example: Having data about amount of rainfall per week, say.

Rainfall 35 18 … 22
(in mm)
Days 7 7 … 7
(in number)

18CSE202- Foundations of Data Science

Significant of different mean calculations
— Case 2: Ranges are different, but observation remains same

Example: Same amount of rainfall in different number of days, say.

Rainfall 50 50 … 50
(in mm)
Days 1 2 … 7
(in number)

18CSE202- Foundations of Data Science

Significant of different mean calculations
— Case 3: Ranges are different, as well as the observations

Example: Different amount of rainfall in different number of days, say.

Rainfall 21 34 … 18
(in mm)
Days 5 3 … 7
(in number)

18CSE202- Foundations of Data Science

Rule of thumbs for means
— AM: When the range remains same for each observation
Example: Case 1

Rainfall 35 18 … 22
(in mm)
Days 7 7 … 7
(in number)

3
1
𝑟̅ = 7 𝑟:
𝑛
0

18CSE202- Foundations of Data Science

Rule of thumbs for means
— HM: When the range is different but each observation is same
— Example: Case 2

Rainfall 50 50 … 50
(in mm)
Days 1 2 … 7
(in number)

𝑛
𝑟̃ =
1
∑30
𝑟:

18CSE202- Foundations of Data Science

Rule of thumbs for means
— GM: When the ranges are different as well as the observations
— Example: Case 3

Rainfall 21 34 … 18
(in mm)
Days 5 3 … 7
(in number)

0
3 3
𝑟̂ = g 𝑟:
0

18CSE202- Foundations of Data Science

Rule of thumbs for means
— The important things to recognize is that all three means are simply the
arithmetic means in disguise!

— Each mean follows the “additive structure”.

— Suppose, we are given some abstract quantities {x1, x2, …, xn}

— Each of the three means can be obtained with the following steps

1. Transform each xi into some yi

2. Taking the arithmetic mean of all yi’s

3. Transforming back the to the original scale of measurement

18CSE202- Foundations of Data Science

Rule of thumbs for means
— For arithmetic mean
— Use the transformation yi = xi
— Take the arithmetic mean of all yi s to get 𝑦!
— Finally, 𝑥̅ = 𝑦!

— For geometric mean

— Use the transformation 𝒚𝒊 = 𝐥𝐨𝐠 𝒙𝒊
— Take the arithmetic mean of all yi s to get 𝑦!
c = 𝒆𝒚,
— Finally, 𝒙

— For harmonic mean

𝟏
— Use the transformation 𝒚𝒊 =
𝒙𝒊
— Take the arithmetic mean of all yi s to get 𝑦!
𝟏
— Finally, b𝒙 = ,
𝒚

18CSE202- Foundations of Data Science

Relationship among means
— A simple inequality exists between the three means related summary measure
as

AM ≥ GM ≥ HM

18CSE202- Foundations of Data Science

Median of a sample
Definition 3.12: Median of a sample

Median of a sample is the middle value when the data are arranged in
increasing (or decreasing) order. Symbolically,

𝒙(𝒏G𝟏)/𝟐 𝒊𝒇 𝒏 𝒊𝒔 𝒐𝒅𝒅

c = r𝟏
𝒙 𝒊𝒇 𝒏 𝒊𝒔 𝒆𝒗𝒆𝒏
𝒙𝒏/𝟐 + 𝒙(𝒏G𝟏)
𝟐 𝟐

18CSE202- Foundations of Data Science

Median of a sample
Definition 3.12: Median of a grouped data

Median of a grouped data is given by

𝑵
− 𝒄𝒇
c=𝒍+
𝒙 𝟐 𝒉
𝒇
where h = width of the median class
N = ∑𝒏𝒊9𝟏 𝒇𝒊
𝒇𝒊 is the frequency of the ith class, and n is the total number of groups
cf = the cumulative frequency
N = the total number of samples
l = lower limit of the median class
Note
A class is called median class if its cumulative frequency is just greater
than N/2

18CSE202- Foundations of Data Science

Mode of a sample
— Mode is defined as the observation which occurs most frequently.

— For example, number of wickets obtained by bowler in 10 test matches are as

follows.
1 2 0 3 2 4 1 1 2 2
— In other words, the above data can be represented as:-
0 1 2 3 4
# of matches 1 3 4 1 1

— Clearly, the mode here is “2”.

18CSE202- Foundations of Data Science

Mode of a grouped data
Definition 3.13: Mode of a grouped data

Select the modal class (it is the class with the highest frequency). Then
the mode 𝒙b is given by:
∆𝟏
b=l+
𝒙 h
∆𝟏 G∆𝟐
where,
h is the class width
∆𝟏 is the difference between the frequency of the modal class and the
frequency of the class just after the modal class
∆𝟐 is the difference between the frequency of the modal class and the class
just before the modal class
l is the lower boundary of the modal class

Note
If each data value occurs only once, then there is no mode!

18CSE202- Foundations of Data Science

Relation between mean, median and mode

— A given set of data can be categorized into three categories:-

— Symmetric data
— Positively skewed data
— Negatively skewed data

• To understand the above three categories, let us consider the following

• Given a set of m objects, where any object can take values 𝒗𝟏 , 𝒗𝟐 ,…..,𝒗𝒌.
Then, the frequency of a value 𝒗𝒊 is defined as

𝑵𝒖𝒎𝒃𝒆𝒓 𝒐𝒇 𝒐𝒃𝒋𝒆𝒄𝒕𝒔 𝒘𝒊𝒕𝒉 𝒗𝒂𝒍𝒖𝒆 𝒗𝒊

Frequency(𝒗𝒊 ) =
𝒏
for i = 1,2,…..,k

18CSE202- Foundations of Data Science

Symmetric data
— For symmetric data, all mean, median and mode lie at the same point

18CSE202- Foundations of Data Science

Positively skewed data
— Here, mode occurs at a value smaller than the median

18CSE202- Foundations of Data Science

Negatively skewed data
— Here, mode occurs at a value greater than the median

18CSE202- Foundations of Data Science

Empirical Relation!
— There is an empirical relation, valid for moderately skewed data

Mean – Mode = 3 * (Mean – Median)

18CSE202- Foundations of Data Science

Midrange
— It is the average of the largest and smallest values in the set.

Steps
1. A percentage ‘p’ between 0 and 100 is specified.
2. The top and bottom of (p/2)% of the data is thrown out
3. The mean is then calculated in the normal way

— Thus, the median is trimmed mean with p = 100% while the traditional mean
corresponds to p = 0%

Note
— Trimmed mean is a special case of Midrange

18CSE202- Foundations of Data Science

Measures of dispersion
— Location measure are far too insufficient to understand data.
— Another set of commonly used summary statistics for continuous data are
those that measure the dispersion.
— A dispersion measures the extent of spread of observations in a sample.

— Some important measure of dispersion are:

— Range
— Variance and Standard Deviation
— Mean Absolute Deviation (MAD)
— Absolute Average Deviation (AAD)
— Interquartile Range (IQR)

18CSE202- Foundations of Data Science

Measures of dispersion
Example
— Suppose, two samples of fruit juice bottles from two companies A and B. The
unit in each bottle is measured in litre.

Sample A 0.97 1.00 0.94 1.03 1.06

Sample B 1.06 1.01 0.88 0.91 1.14

— Both samples have same mean. However, the bottles from company A with more
uniform content than company B.
— We say that the dispersion (or variability) of the observation from the average is
less for A than sample B.
— The variability in a sample should display how the observation spread out from the average
— In buying juice, customer should feel more confident to buy it from A than B

18CSE202- Foundations of Data Science

Range of a sample
Definition 3.14: Range of a sample

Let X = 𝐱 𝟏 , 𝐱 𝟐 , 𝐱 𝟏 ,….., 𝐱 𝐧 be n sample values that are arranged in

increasing order.

The range R of these samples are then defined as:

R = max(X) – min(X) = 𝐱 𝐧 - 𝐱 𝟏 z

— Range identifies the maximum spread, it can be misleading if most of the

values are concentrated in a narrow band of values, but there are also a
relatively small number of more extreme values.

— The variance is another measure of dispersion to deal with such a situation.

18CSE202- Foundations of Data Science
Variance and Standard Deviation

Definition 3.15: Variance and Standard Deviation

Let X = { 𝐱 𝟏 , 𝐱 𝟐 , 𝐱 𝟏 ,….., 𝐱 𝐧 } are sample values of n samples. Then,

variance denoted as σ² is defined as :-
𝐧
𝟐
𝟏 𝟐
𝛔 = 7 𝐱𝐢 − 𝐱
𝐧−𝟏
𝐢9𝟏
where, x denotes the mean of the sample

The standard deviation, σ, of the samples is the square root of the

variance 𝛔𝟐

18CSE202- Foundations of Data Science

Coefficient variation
— Basic properties
— σ measures spread about mean and should be chosen only when the mean is
chosen as the measure of central tendency
— σ = 0 only when there is no spread, that is, when all observations have the
same value, otherwise σ > 0

Definition 3.16: Coefficient variation

A related measure is the coefficient of variation CV, which is defined as

follows
σ
CV = 𝐱
× 100

This gives a ratio measure to spread.

18CSE202- Foundations of Data Science

Variance and Standard Deviation
— Lemma 3.8
𝐱H𝐚
— If data are transformed as 𝐱 @ = , the variance is transformed as
𝐜
𝟐 𝟏 𝟐
𝛔@ = 𝟐 𝛔
𝐜
Proof
@ 𝐱H𝐚
The new mean 𝐱 = 𝐜
0 @ 1 0 * ˆ‰ HŠ ˆHŠ 1
∑*‡90 x‡@ − x = ∑ − (
* * ‡90 (
0 * 1
= ∑ x‡ − a − x − a
(^ * ‡90
0 * 1
= ∑ x − x
(^ * ‡90 ‡
0 1
= σ [PROVED]
(^
18CSE202- Foundations of Data Science
Mean Absolute Deviation (MAD)
— Since, the mean can be distorted by outlier, and as the variance is computed
using the mean, it is thus sensitive to outlier. To avoid the effect of outlier,
there are two more robust measures of dispersion known. These are:

— Mean Absolute Deviation (MAD)

MAD (X) = median 𝐱𝟏 − 𝐱 , … . . , 𝐱𝐧 − 𝐱

— Absolute Average Deviation (AAD)

𝟏
AAD(X) = ∑𝐧𝐢9𝟏 𝐱 𝐢 − 𝐱
𝐧

where, X = {𝐱 𝟏 , 𝐱 𝟐 ,…..,𝐱 𝐧}is the sample values of n observations

18CSE202- Foundations of Data Science

Interquartile Range
— Like MAD and AAD, there is another robust measure of dispersion known, called
as Interquartile range, denoted as IQR
— To understand IQR, let us first define percentile and quartile

— Percentile
— The percentile of a set of ordered data can be defined as follows:

o Given an ordinal or continuous attribute x and a number p between 0 and

100, the pth percentile 𝐱 𝐩 is a value of x such that p% of the observed
values of x are less than 𝐱 𝐩

o Example: The 50th percentile is that value 𝐱 𝟓𝟎% such that 50% of all values
of x are less than 𝐱 𝟓𝟎% .

— Note: The median is the 50th percentile.

18CSE202- Foundations of Data Science

Interquartile Range
— Quartile
— The most commonly used percentiles are quartiles.
— The first quartile, denoted by 𝐐𝟏 is the 25th percentile.
— The third quartile, denoted by 𝐐𝟑 is the 75th percentile
— The median, 𝐐𝟐 is the 50th percentile.

— The quartiles including median, give some indication of the center, spread
and shape of a distribution.

— The distance between 𝐐𝟏 and 𝐐𝟑 is a simple measure of spread that gives the
range covered by the middle half of the data. This distance is called the
interquartile range (IQR) and is defined as
IQR = 𝐐𝟑 - 𝐐𝟏

18CSE202- Foundations of Data Science

Application of IQR
— Outlier detection using five-number summary

— A common rule of the thumb for identifying suspected outliers is to single

out values falling at least 1.5 × IQR above 𝐐𝟑 and below 𝐐𝟏 .

— In other words, extreme observations occurring within 1.5 × IQR of the

quartiles

18CSE202- Foundations of Data Science

Application of IQR
— Five Number Summary
— Since, 𝐐𝟏 , 𝐐𝟐 and 𝐐𝟑 together contain no information about the endpoints
of the data, a complete summary of the shape of a distribution can be
obtained by providing the lowest and highest data value as well. This is
known as the five-number summary
— The five-number summary of a distribution consists of :
— The Median 𝐐𝟐
— The first quartile 𝐐𝟏
— The third quartile 𝐐𝟑
— The smallest observation
— The largest observation
These are, when written in order gives the five-number summary:
Minimum, 𝐐𝟏 , Median (𝐐𝟐 ), 𝐐𝟑 , Maximum

18CSE202- Foundations of Data Science

Box plot
— Graphical view of Five number summary

Maximum

Q3

Median

Q1

Minimum

18CSE202- Foundations of Data Science

 Reference

— The detail material related to this lecture can be found in

Probability and Statistics for Enginneers and Scientists (8th Ed.)

by Ronald E. Walpol, Sharon L. Myers, Keying Ye (Pearson), 2013 .

18CSE202- Foundations of Data Science