shortene
d
🞂 Notation
🞂 Measures of Location
🞂 Measures of Dispersion
🞂 Standardization
🞂 Proportions for Categorical
Variables
🞂 Measures of Association
🞂 Outliers
🞂 Population - all items of interest for a
particular decision or investigation
- all married drivers over 25 years old
- all subscribers to Netflix
🞂 Sample - a subset of the population
- a list of individuals who rented a
comedy from Netflix in the past year
🞂 Thepurpose of sampling is to obtain
sufficient information to draw a valid
conclusion about a population.
Is the Netflix sample above a good sample?
Why?
Other ways to select a sample?
🞂 We typically label the elements of a data set using
subscripted variables, x1, x2 , … , and so on, where xi
represents the ith observation. Upper-case letters like X
represent often random variables.
🞂 It is common practice in statistics to use
◦ Greek letters, such as (mu; mean), (sigma; std. deviation), and
(pi;
◦ italic letters such as by 𝑥ҧ (called x-bar), s, and p to represent
proportion), to represent population measures and
sample statistics.
🞂 N represents the number of items in a population and
n represents
the number of observations in a sample.
🞂 Notation
🞂 Measures of Location
🞂 Mean
🞂 Median
🞂 Measures of Dispersion
🞂 Standardization
🞂 Proportions for Categorical
Variables
🞂 Measures of Association
🞂 Outliers
🞂 Population mean:
🞂 Sample mean:
🞂 Excelfunction: =AVERAGE(data range)
🞂 Property of the mean:
🞂 Outlierscan affect the value of the mean.
🞂 Mean valid for interval/ratio variables and often
questionable for ordinal variables.
Purchase Orders
database
🞂 Using formula:
=SUM(B2:B95)/
COUNT(B2:B95)
Mean = $2,471,760/94
= $26,295.32
Using Excel AVERAGE
Function
=AVERAGE(B2:B95)
Person Age Person Age
1 17 1 17
2 21 2 21
3 15 3 15
4 18 4 18
5 999 5
6 22 6 22
7 11 7 11
8 25 8 25
Mean 141.00 Mean 18.43
Wikipedia: In statistics, an outlier is an observation point that is
distant from other observations. An outlier may be due to
variability in the measurement or it may indicate experimental error;
the latter are sometimes excluded from the data set.
🞂The median specifies the middle value when the data are
arranged
from least to greatest.
◦ Half the data are below the median, and half the data are
above it.
◦ For an odd number of observations, the median is the
middle of the sorted numbers.
◦ For an even number of observations, the median is the
mean of the two
middle numbers.
🞂 We could use the Sort option in Excel to rank-order the
data and then determine the median. The Excel function
=MEDIAN(data range) could also be used.
🞂 The median is meaningful for ratio, interval, and ordinal
data.
🞂 Not affected by outliers.
🞂 Sort the data from smallest to largest.
Since we have 90 observations, the
median is the average of the 47th and 48th
observation.
Median =
($15,562.50 + $15,750.00)/2
= $15,656.25
=MEDIAN(B2:B94)
Person Age
1 17.00
2 21.00
3 15.00
4 18.00
5 999.00
6 22.00
7 11.00
8 25.00
Mean 141.00
Median 19.50
Median is insensitive to
outliers!
The Excel file Computer Repair Times includes
250 repair times for customers.
🞂 What repair time
would be reasonable
to quote to a new
customer?
🞂 Median repair time is 2
weeks; mean and mode
are about 15 days.
🞂 Examine the histogram.
90% are completed within 3
weeks
Distribution is important!
🞂 Notation
🞂 Measures of Location
🞂 Measures of Dispersion
🞂 Range
🞂 Interquartile Range
🞂 Variance
🞂 Standard Deviation
🞂 Empirical Rules
🞂 Standardization
🞂 Proportions for Categorical
Variables
🞂 Measures of Association
🞂 Outliers
🞂 Dispersion refers to the degree of
variation in the data; that is, the
numerical spread (or compactness) of
the data.
🞂 Key measures:
◦ Range
◦ Interquartile range
◦ Variance
◦ Standard deviation
🞂 The range is the simplest and is the
difference between the maximum value
and the minimum value in the data set.
🞂 InExcel, compute as =MAX(data
range) - MIN(data range).
🞂 Therange is affected by outliers, and is
often used only for very small data sets.
🞂 Purchase Orders data
🞂 For the cost per order data:
◦ Maximum = $127,500
◦ Minimum = $68.78
🞂 Range
= $127,500 - $68.78 =
$127,431.22
🞂 The interquartile range (IQR), or the
midspread is the difference between the
first and third quartiles, Q3 – Q1.
🞂 This
includes only the middle 50% of the
data and, therefore, is not influenced by
extreme values.
🞂 Purchase Orders data
🞂 For the Cost per order data:
🞂 Third Quartile = Q3 = $27,593.75
🞂 First Quartile = Q1 = $6,757.81
🞂 Interquartile Range = $27,593.75 –
$6,757.81
=$20,835.94
🞂 The variance is the “average” of the
squared deviations from the mean.
🞂 For a population:
◦ In Excel: =VAR.P(data range)
🞂 For a sample:
◦ In Excel: =VAR.S(data range)
🞂 Note the difference in denominators!
🞂 The
standard deviation is the square root of
the variance.
◦ Note that the dimension of the variance is the square
of the dimension of the observations, whereas the
dimension of the standard deviation is the same as
the data. This
makes the standard deviation more practical to use
in applications.
🞂 For a population:
◦ In Excel: =STDEV.P(data range)
🞂 For a sample:
◦ In Excel: =STDEV.S(data range)
Excel file: Closing
Stock Prices
Intel (INTC):
Mean = $18.81
Standard deviation =
$0.50
General Electric
(GE): Mean =
$16.19
Standard deviation
= $0.35
INTC is a higher
risk
investment than
GE.
🞂 For
many data sets encountered in
practice:
🞂 Approximately 68% of the observations fall within
one
standard deviation of the mean
🞂 Approximately 95% fall within two standard
deviations of
the mean
🞂 Approximately 99.7% fall within three standard
deviations
of the mean
🞂 These rules are commonly used to
characterize the natural variation in
🞂The empirical Rule comes from the normal
distribution.
Most data does not follow a normal distribution!
🞂 For any data set (any distribution), the
proportion of values that lie within +/- k
(k > 1) standard deviations of the mean
is at least 1 – 1/k2
🞂 Examples:
◦ For k = 2: at least ¾ or 75% of the data lie
within two
standard deviations of the mean
◦ For k = 3: at least 8/9 or 89% of the data lie
within
three standard deviations of the mean
🞂 Notation
🞂 Measures of Location
🞂 Measures of Dispersion
🞂 Standardization
🞂 Proportions for Categorical
Variables
🞂 Measures of Association
🞂 Outliers
🞂 A standardized value, commonly called a z-
score, provides a relative measure of the
distance an observation is from the mean,
which is independent of the units of
measurement.
🞂 The z-score for the ith observation in a data
set is calculated as follows:
◦ Excel function: =STANDARDIZE(x, mean,
standard_dev).
Standardized data is needed by many predictive
methods since it makes variables comparable.
🞂 Purchase Orders Cost per order
data
=(B2 - $B$97)/$B$98, or
=STANDARDIZE(B2,$B$97,$B$9
8).
0
1
🞂 Notation
🞂 Measures of Location
🞂 Measures of Dispersion
🞂 Standardization
🞂 Proportions for Categorical
Variables
🞂 Measures of Association
🞂 Outliers
🞂 Theproportion, denoted by p, is the
fraction of data that have a certain
characteristic.
🞂 Proportions are key descriptive
statistics for categorical data, such as
defects or errors in quality control
applications or consumer preferences
in market research.
🞂 Example:Proportion of female
students is 60%.
🞂 Proportion
of orders placed by Spacetime
Technologies
=COUNTIF(A4:A97, “Spacetime
Technologies”)/94
= 12/94 = 0.128
🞂 Notation
🞂 Measures of Location
🞂 Measures of Dispersion
🞂 Standardization
🞂 Proportions for Categorical
Variables
🞂 Measures of Association
🞂 Correlation
🞂 Outliers
🞂 Twovariables have a strong statistical
relationship with one another if they appear
to “move” together.
🞂 When two variables appear to be
related, you might suspect a cause-and-
effect relationship.
🞂 Caution:
Correlation does not prove causation!
Statistical relationships may exist even
though a change in one variable is not
caused by a change in the other.
🞂 Covariance is a measure of the linear association between
two variables, X and Y. Like the variance, different formulas
are used for populations and samples.
🞂 Population covariance:
◦ Excel function:
=COVARIANCE.P(array1,array2)
🞂 Sample covariance:
◦ Excel function: =COVARIANCE.S(array1,array2)
🞂 The covariance between X and Y is the average of the
product of the deviations of each pair of observations
from their respective means.
FORMULA: =SUM/(COUNT-1)
🞂 Collegesand
Universities
data
🞂 Correlation is a measure of the linear relationship
between two variables, X and Y, which does not depend
on the units of measurement.
🞂 Correlation is measured by the correlation coefficient, also
known as the Pearson product moment correlation coefficient.
🞂 Correlation coefficient for a population:
🞂 Correlation coefficient for a sample:
🞂 The correlation coefficient is scaled between -1
and 1.
🞂 Excel function: =CORREL(array1,array2)
Why is correlation important?
🞂 Colleges and Universities data
🞂 Is a schools graduation rate related to the SAT
score of
incoming students?
Is there a causal
relationship?
Data >
Data Analysis >
Correlation
🞂 Excel
computes the correlation coefficient
between all pairs of variables in the Input
Range. Input Range data must be in
contiguous columns.
🞂Colleges and Universities
data
◦ Moderate negative correlation between acceptance
rate and graduation rate, indicating that schools with
lower acceptance rates have higher graduation rates.
◦ Acceptance rate is also negatively correlated with
the median SAT and Top 10% HS, suggesting that
schools with lower acceptance rates have higher
student profiles.
◦ The correlations with Expenditures/Student suggest
that schools with higher student profiles spend
more money per student.
Value Field Settings include several
statistical measures:
🞂 Average
🞂 Max and Min
🞂 Product
🞂 Standard deviation
🞂 Variance
🞂 Credit Risk Data
🞂 First, create a PivotTable.
🞂 In the PivotTable Field List, move Job to the Row
Labels field and Checking and Savings to the
Values field. Then change the field settings from
“Sum of Checking” and “Sum of Savings” to the
averages.
🞂 Notation
🞂 Measures of Location
🞂 Measures of Dispersion
🞂 Standardization
🞂 Proportions for Categorical
Variables
🞂 Measures of Association
🞂 Outliers
🞂 There is no standard definition of what constitutes an
outlier!
🞂 Wikipedia: “In statistics, an outlier is an observation point that is
distant from other observations. […] Outliers can occur by
chance in any distribution, but they often indicate either
measurement error or that the population has a heavy-tailed
distribution.”
🞂 If the outlier is due to a measurement error then we
often want to
exclude it from the analysis.
🞂 Some typical rules of thumb:
🞂 Normal distribution: z-scores greater than +3 or less than -3
🞂 Boxplot:
🞂 Extreme outliers are more than 3*IQR to the left of Q1 or right of
Q3
🞂 Mild outliers are between 1.5*IQR and 3*IQR to the left of Q1 or
🞂Home Market Value
data
🞂 None of the z-scores exceed 3. However,
while individual variables might not
exhibit outliers, combinations of them
might.
◦ The last observation has a high market value
($120,700) but a relatively small house size (1,581
square feet) and may be an outlier.
🞂 Excel file Surgery Infections
◦ Is month 12 simply random variation or some
explainable phenomenon?
🞂Three-standard deviation empirical
rule:
🞂 There is only a 0.3% (for normally distributed data) or a
11% (for any
distribution) chance to see an observation outside +/- 3
[Link].
🞂 This suggests that month 12 is statistically different from
the rest of