Biostatistics Notes

Week 1
Statistics tries to explain random phenomena, based on observations of a limited sample material.

Statistics tries to describe data with numbers.

Mathematical statistics is a subfield of statistics, where the aim is to develop and improve statistical
methods.

Applied statistics is a subfield of statistics, where the aim is to apply the methods established by
mathematical statistics to related areas.

Biostatistics is a specific branch of applied statistics, where the aim is to apply the methods
established by mathematical statistics to medical and biological problems.

Most of the time, the first step of a statistical analysis is to summarize the data in hand with several
descriptive statistics, in an effort to have a basic understanding about what is being dealt with.

Inferential statistics deals with the problem of revealing the information about the population, by
analyzing the limited sample data.

Descriptive statistics can be numerical or graphical.

Measures of Location
Measures of location are numerical descriptive statistics that are used to characterize the center, or
the middle of the sample data.

The Arithmetic Mean:

Arithmetic mean (average, mean, or sample mean) is denoted by x̅.

It is defined as:
n
1
x= ∑x
n i=1 i
Note1: The Σ (sigma) character is the summation sign. If a and b are integers, where a < b, then:
b

∑ x i=x a + x a +1+ …+ x b
i=a

Note2: One property of summation sign is:

b b

∑ cx i=c (∑ x i)
i=a i=a

Arithmetic mean is a very natural measure of location. However, it is very sensitive to extreme values,
and may not always indicate the location where most of the sample points are aggregated.
In extreme cases, only one sample point may be on the one side of the arithmetic mean, while all the
rest are on the other side.

Although in these cases the arithmetic mean is a poor measure of central location, it is still the most
commonly used one.

Some Properties of Arithmetic Mean:

1. If , y i=x i +c ,i=1 , … , nthen , y =x+ c

This is useful because sometimes it is convenient to calculate the mean after translating the
data, and then translating the result back, to obtain the sample mean.
2. If , y i=cx i , i=1 ,… ,n then , y =c x
The same reasoning applies here.
3. If , y i=c 1 x i +c 2 ,i=1, … , nthen , y =c 1 x +c 2
And it works even when the data is scaled (by c1) and shifted (c2) at the same time.

The Median:

The median, or more precisely the sample median, is an alternative measure of location. For a
sample set consisting of n observations, which are ordered from smallest to largest, the median is
defined as:

1. ( n+12 ) th largest sample point , if n is odd

The average of ( ) th∧( + 1) th largest sample point if n is even .
n n
2.
2 2
The aim is to make sure that there are equal number of observations on each side of median.

Median is more robust to outliers and extreme values than mean.

However, it is also insensitive to actual numerical values of the data points, since it is defined by the
middle points in the sample.

If a distribution is symmetrical, the relative position of the points on each side of the sample median
is the same. Also, the arithmetic mean is roughly equal to the sample median.

If a distribution is positively skewed (right-skewed) the points above the median tend to be farther
from the median in absolute value than points below the median. Also, the arithmetic mean tends to
be greater than the sample median.

If a distribution is negatively skewed (left-skewed) the points below the median tend to be farther
from the median in absolute value than points above the median. Also, the arithmetic mean tends to
be less than the sample median.

The Mode:

The mode is another widely used measure of location. It is defined as the most frequently observed
value in the sample.
If the number of possible values in a given sample are large, and each value is infrequently repeated;
then the mode either will not represent the center of the sample, or it may not even exist.

The Geometric Mean:

If the data is in the form of exponential numbers, for example the growth of a bacterial culture
doubling after every time interval t, the arithmetic mean will not be as useful, since the data will be
strongly skewed.

In this case the logarithm of each value can be calculated, to obtain more equally spaced sample
points with a weaker skew, and then the arithmetic mean can be calculated. However, it is more
convenient to take the antilogarithm of this result, to keep working in the original scale. Thus, the
geometric mean can be defined as:

The antilogarithm of l o g x , where

n
1
l o g x= ∑ log x i
n i=1
Or, if we work the formula out:

√n x i∗x i+1∗…∗x n
The result of geometric mean is the same regardless of the base of the logarithm. However, the same
bases should be used for the log and antilog steps. Commonly used bases are base e, and base 10.

Measures of Spread
Measures of spread describe the variation in the data. Many samples can be well described by a
combination of a measure of location and a measure of spread.

The Range:

The range is the difference between the largest and the smallest observations in the sample.

It can be easily calculated when the sample points are ordered, however it is extremely sensitive to
outliers.

Moreover, it depends on the sample size, as the larger the sample size the greater the range tends to
be. Thus, it is hard to compare the ranges of the samples with differing sizes.

The Variance and Standard Deviation:

Deviation is the distance of each value to the center of the sample, or to the sample mean.

If the deviation of each value is summed up to summarize the spread of the sample, the following
would be obtained:
 n
1
d= ∑ (x ¿−x )=0 ¿
n i=1 i
Because, mean defined as the value which is equally distant from each data point in the sample.
Thus, the sum of the deviations of each data point in the sample around the sample mean is always
zero, and d is not usable as a measure of spread.

An alternative is the mean deviation:

n
1
∑ ¿ x −x∨¿
n i=1 i
However, if square of the deviation used in the calculation, instead of the absolute value, the values
will be kept positive, while the ones that deviate more will be punished more as well. The resulting
measure of spread is called the sample variance. Instead of n in the denominator, however, (n-1) is
commonly used. This is because the decreased degrees of freedom due to mean calculation. Thus,
the sample variance, s2 is defined as:
n
1
2
s= ∑
n−1 i=1
( x i−x )
2

Finally, the square root of the sample variance gives the sample standard deviation:

√
n
1
s= ∑ ( x −x )2= √ s2
n−1 i=1 i

Some Properties of The Variance and The Standard Deviation:

How does the variance and the standard deviation respond to translation of the data?

1. Let there be two samples, x 1 , … , x n∧ y 1 , … , y nwhere , y i=x i +c ,i=1 , … , n

If the respective sample variances ares2x ∧s 2y ,Then s 2y =s 2x
2. Let there be two samples, x 1 , … , x n∧ y 1 , … , y nwhere , y i=c x i , i=1 , … , n , c> 0
2 2 2
Then s y =c s x s y =c s x
Thus, the changes in the origin (shifting the data) does not affect the variance. However, the changes
in the scale of data, by a factor of c will affect the variance by a factor of c2 while affecting the
standard deviation by a factor of c. This means that the standard deviation and the arithmetic mean
are in the same units.

The arithmetic mean and the standard deviation are the most commonly used measures. An
important reason why is that the normal distribution is defined specifically by these two parameters,
and the normal distribution has wide applications in biological and medical settings.

The Coefficient of Variation (CV):

The coefficient of variation is used to relate arithmetic mean and the standard deviation, and
compare the spread of samples with different means. Because, a higher variation is expected with a
higher mean, and CV accounts for this when two samples with different means are compared. CV is
also useful to compare the reproducibility of different variables.

The coefficient of variation is defined as:

100∗s
CV =
x
Quantiles:

Quantiles are an alternative way to quantify the spread of a data. A q-quantile divides sorted data
into q parts. The pth q-quantile of a sample of n observations is defined as:

th n∗p
1. ( k +1 ) largest value if
is not anintager .
q
n∗p
(Where k is the largest intager less than )
q
2. The average of
n∗p
q ( ) (
th∧
np
q )
+1 thlargest values if
n∗p
q
is anintager .

The spread of a sample can be characterized as the difference between two q-quantiles, in a sorted
data in ascending order, for example, 90th and 10th percentiles (or 100-quantiles).

When this is applied on 3rd and 1st quartiles (or 4-quntiles), this is called The Interquartile Range
(IQR), or middle spread, or middle fifty.

Given a sorted data∈ascending order ,IQR=Q 3−Q1

Compared to the range, the quantiles are less sensitive to outliers, and are not affected as much by
the sample size (n).

It may be hard to order large samples before calculating quantiles. However, stem-and-leaf plot may
be helpful for this purpose.

Specialized Quantiles:

2. The only 2-quantile is called the median,

3. The 3-quantiles are called tertiles or terciles (T),
4. The 4-quantiles are called quartiles (Q),
5. The 5-quantiles are called quintiles (QU),
6. The 6-quantiles are called sextiles (S),
7. The 7-quantiles are called septiles (SP),
8. The 8-quantiles are called octiles (O),
10. The 10-quantiles are called deciles (D),
12. The 12-quantiles are called duo-deciles or dodeciles (DD),
16. The 16-quantiles are called hexadeciles (H),
20. The 20-quantiles are called ventiles or vigintiles (V),
100. The 100-quantiles are called percentiles (P),
1000. The 1000-quantiles are called permilles (PR),
Grouped Data
The sample size can be too large to display all raw data. Also, the degree of accuracy for the exact
measurement of a quantity can be lacking due to measurement error or imprecise patient recall. In
these cases, the data is grouped.

The simplest way to group data is the frequency distribution.

A frequency distribution is an ordered display of each value in a data set along with the number of
times that value occurs in the data set. The percentage of sample points that take on a particular
value is also usually given.

If the number of unique values is large in a data set, a frequency distribution can still be too detailed.

In this case, the data can be grouped/binned into broader categories, as follows:

1. Subdivide the data into k intervals, starting at a lower bound y1 and ending at an upper bound
yk+1. (The number of groups will be one less than the number of unique values.)
2. The first interval is from y1 inclusive to y2 exclusive; the second interval is from y2 inclusive to
y3 exclusive; …; the kth interval is from yk inclusive to yk+1 exclusive. (The group intervals will
include all the possible values, and will not overlap.)
3. Intervals are generally equal, although it depends on the subject-matter.
4. For each interval, the number of points that fall into the interval, that is the frequency of that
interval, are counted.
5. The group intervals are then displayed in a table, or plot.

Graphic Methods
Graphics methods are handy to give a quick overall impression of the data, when it is not so easy to
achieve with the numerical measures.

What makes a good graphical or numerical display?

 The material must be sufficient enough by itself for viewers to understand without reading
any text.
 Clear labelling of attributes.
 The quantity of the material displayed should be neither too few, nor too many.

Bar Graphs:

Bar graph is one of the most commonly used method for displaying grouped data.

A bar graph can be constructed as follows:

1. Group the data as mentioned.

2. For each group, draw a rectangle with a base of constant width, and height proportional to
the frequency withing that group
3. The rectangles are generally equally dispersed.

Stem-and-Leaf Plots:
In the bar graphs; (1) the definition of groups is arbitrary, and (2) the actual sample points are lost
within the groups.

Stem-and-leaf plot is a graphical method that can work around these problems. To construct a stem-
and-leaf plot:

1. Separate each data point into a stem component and a leaf component, where the stem is all
but the rightmost digit of the value, and the leaf is the remaining rightmost digit.
2. Place the smallest stem to the top-left corner of the plot.
3. Place the next smallest stem under the previous one.
4. Repeat step 3 until the largest stem is placed.
5. Draw a vertical line to the right of the stems column.
6. Foreach value in the dataset, the leaves are placed to the right of the vertical line at the level
of their stem, in ascending order.

The leaves show the general form of the distribution of the sample points, and the actual values are
also visible while the data is displayed in groups.

It is also easy to define median and other quantiles on a stem-and-leaf plot.

In some variations, leaves can consist of more than one digit, and the digits of the same leaf are
underlined. Also, if the number of leaves is large, each stem can get more than one line in the plot.
Moreover, the order of stems can be reversed.

Finally, a multiplication factor (m) can be added to the bottom of the plot, to allow representation of
decimal numbers. If m is present, the numbers assumed to be in the form of [Link] * 10m.

Histogram:

A histogram is a frequency distribution plot (or a probability distribution plot.).

They are used to visualize the distribution of values in a dataset.

Symmetrical and Non-symmetrical Distributions:

Non-symmetrical distributions are also called “skewed”.

Box Plots (Box-and-Whisker Plot):

A box plot uses the relative positions of median, upper quartile and lower quartile to display the
skewness of a distribution.

1. For a symmetrical distribution, the upper and lower quartiles will be roughly equally
distanced from the median.
2. For a positively skewed distribution, the lower quartile will be closer to the median than the
upper quartile.
3. For a negatively skewed distribution, the upper quartile will be closer to the median than the
lower quartile.

Also, the difference between upper and lower quartiles (Q3 – Q1, IQR) can be used to describe the
spread of the data.

Moreover, it is possible to identify outliers and extreme values, as follows:

Outliers:

x > ( upper quartile+(1.5∗IQR) ) ,∨¿ x <(lower quartile−(1.5∗IQR))

 Extreme values:
x > ( upper quartile+(3.0∗IQR) ) ,∨¿x <(lower quartile−(3.0∗IQR))

However, these definitions for outliers and extreme values are sensitive to sample size. The larger the
sample size, the greater the number of outliers and/or extreme values.

To construct a box plot:

1. Draw three parallel horizontal lines: the first for the upper quartile, the second for the
median, and the third, the bottom-most, for the lower quartile. (The data is being
represented in descending order from top to bottom.)
2. Draw two parallel lines, perpendicular to the horizontal lines, connecting the ends of the
horizontal lines. (This completes the box.)
3. Based on the definition above, draw a perpendicular line from the upper quartile to the
largest non-outlying value; and a perpendicular line from the lower quartile to the smallest
non-outlying value. (This completes the whiskers.)
4. Then, represent the outliers with “o” and extreme values with “x” if any.

The whiskers can also be drawn to represent the range, 2nd and 98th percentiles, or standard
deviation. Then, the data points outside of the whiskers, if any, would be outliers.
In a variable width box plot, the width of the box, w, is proportional to the sample size, n. Usually,
the relationship is in the form of:

w=√ n
In a notched box plot, the box has a notch (narrowing) of height, nh, proportional to the IQR, and
inversely proportional to the sample size. Usually, the relation shape is in the form of:

1.5∗IQR
nh=±
√n

If the notches of two box plots overlap, that means the difference between their medians is not
expected to be significant.

Violin plot:

A violin plot is another tool to display the distribution of data. It is in the form of a distribution
rotated 90° and mirrored along the vertical axis.

Other Pots:
Week 2 + 3
Sample Space: All possible outcomes.

Event: Outcomes of interest or outcomes of a trial. Denoted by { }.

Frequency: The number of times an event occurs.

Relative Frequency: The ratio of frequency of events to number of trials. Probability.

Probability: Relative frequency of an outcome (event) after a large number of trials.

Empirical Probability: Probability of an event based on experimental results.

Probability of an event E is denoted by Pr(E) or P(E).

0 ≤ P( E )≤ 1
Events A and B are mutually exclusive if they cannot happen at the same time.

A ∪ B= A∨B
A ∩ B= A∧B
A∧B are independent events ,if :
P ( A ∩B )=P ( A )∗P ( B )
Multiplication Law of Probability:

If A1 , A 2 , … , Ak are mutually independent events , then

P ( A 1 ∩ A 2 ∩ … ∩ A k ) =P (A 1 )∗P( A 2)∗…∗P( A k )

Addition Law of Probability:

If A∧B are any events , thenP ( A ∪ B )=P ( A ) + P ( B ) −P ( A ∩ B )

Moreover , if A∧B are independent , thenP ( A ∪ B )=P ( A ) + P ( B )∗[ 1−P ( A ) ]
Lastly ,¿ extend this ¿ more than two events ,
P ( A ∪ B ∪C )=P ( A ) + P ( B )+ P ( C )−P ( A ∩ B )−P ( A ∩C )−P (C ∩B )+ P( A ∩ B∩C )
Conditional Probability:

Conditional probability is helpful when we want to calculate the probability of several dependent
events occurring simultaneously.

P(A∩B)
Conditional probability of B given A is denoted by , P ( B| A )=
P( A)
If A∧B are independent , then P ( B| A )=P ( B )=P ( B| A ) If A∧B are dependent , then
'

P ( B| A ) ≠ P ( B ) ≠ P ( B|A ' ) For any events A∧B , P ( B )=P ( B| A )∗P ( A ) + P ( B| A' )∗P ( A ' )
The relative risk of B given A is , P ( B| A ) / P ( B| A ' )

A set of events A1, A2, …, Ak are exhaustive, if at least one of these events must occur. In other
words, no events in the universe except from these.

Total-Probability Rule:
Let A1, A2, …, Ak be mutually exclusive and exhaustive events. P(B) can be defined as a weighted
average of p(B|Ai) with weight as P(Ai).
k
P ( B )=∑ P ( B|A i )∗P ( A i )
i=1

Generalized Multiplication Law of Probability:

Let A1, A2, …, Ak be any events, then

P ( A 1 ∩ A 2 ∩ … ∩ A k ) =P ( A 1 )∗P ( A2| A 1)∗P ( A 3| A2 ∩ A1 )∗…∗P( Ak ∨A k−1 ∩ … ∩ A 2 ∩ A 1)

Bayes’ Rule:

PV+ or Predictive Value Positive of a screening test is P(disease | test+).

PV- or Predictive Value Negative of a screening test is P(no-disease | test-).

The higher the PVs of a test, the more reliable the test is.

Sensitivity of a test is P(test+ | disease).

Specificity of a test is P(test- | no-disease).

A false negative is a negative test result in the presence of a condition.

A false positive is a positive test result in the absence of a condition.

Predictive Value Positive, P(B|A), can be described based on Sensitivity, P(A|B) and Specificity, (A’|
B’) terms.
P( A ∩B)
+¿= P ( B|A ) = ¿
Let A be the test ∧B be the condition: If we substitute ,
P(A)
PV
P ( A ∩B ) → P ( A|B )∗P ( B ) , [ ¿ definition of cond . prob. ]∧¿
P ( A ) → P ( A|B )∗P ( B ) + P ( A|B' )∗P ( B' ) , [ ¿ total prob . rule ] we get :
P ( A|B )∗ P ( B )
+¿= P ( B|A ) = ¿
P ( A|B) ∗P ( B) + P ( A|B )∗ P (B )
' '

PV
If we express the final equation in words:

Sensitivity∗x
Predictive Value Positive=
Sensitivity∗x + ( 1−Specificity )∗( 1−x )
Similarly, Predictive Value Negative would be:

P(B ∩ A ) P ( A |B )∗P ( B )
' ' ' ' '
−¿=P ( B |A )= −¿=P ( B |A )=
' ' ' '
¿ ¿
P( A ) P ( A |B )∗P ( B )+ P ( A |B )∗P ( B )
' ' ' ' '

PV PV

In words, that would be:

Specificity∗( 1−x )
Predictive Value Negative=
Specificity∗(1−x )+ ( 1−Sensitivity )∗x
Generalized Bayes’ Rule:

Let B1 , B2 , … , Bk be a set of mutually exclusive∧exhaustive conditions .

( at least one condtion must occur ,∧only one condition must occur ) Let A be the test .
P ( A|Bi )∗P ( B i)
P ( B i| A ¿= k
∑ P ( A∨B j )∗P(B j ¿)¿
j =1

Prior and Posterior Probabilities:

P ( A|Bi )∗P ( B i)
P ( B i| A ¿= k
In the Baye’s Rule Formula, for a condition B and a test A,
∑ P ( A∨B j )∗P(B j ¿)¿
j =1

The term P(Bi) is also called the Prior Probability. That is the probability of a condition occurs,
guessed by the observer, without data. Can be a single number or a range of values possibly with a
weight for each one.

The term P(Bi|A) is also called the Posterior Probability. That is the probability of a condition
determined after collecting empirical data. It is obtained by relating additional data to the
information from the prior probability.

In other words, we update our knowledge on the probability of a condition, based on new data.

Receiving Operator Characteristic (ROC) Curves:

Sometimes a test may yield multi-categorical results, instead of simple positive and negative. Or the
result may be a continues variable. In these cases, an arbitrary cut-off point is necessary to assess
the result as positive or negative.

ROC curve is a plot of sensitivity on the y-axis versus (1- specificity) on the x-axis. Each point
corresponds to a cut-off point.

The area under the ROC curve is used to summarize the accuracy of the test. (Calculated with
trapezoidal rule)

In general, larger area means more accurate test. Unless some specific sensitivity or specificity values
are important for the test.
Week 4
Discrete Probability Distributions
Permutation: It gives the number of ways to choose n items out of k, when the order of items is
important.

n!
n P k=
( n−k ) !
Combination: It gives the number of ways to choose n items out of k, when the order of items is
NOT important.

()
n!
n Ck = n =
k k ! ( n−k ) !
A feature of combination is:

n Ck =n C n−k

Pascal Triangle can be used to find nCk:

A constant has a fixed value; e, π, etc.

A random variable is a function that that assigns values to different events in a sample space.

If there are a discrete set of values that can be assigned; such as 1, 2, 3, … but nothing in between
them; then the random variable called a discrete random variable.
On the other hand, if the possible values of a random variable cannot be enumerated; such as [0,1]
interval, which has infinite number of values in between 0 and 1; then the random variable is called a
continuous random variable.

A probability-mass function, also called a probability distribution, assigns the probability P(X = r) for
any possible value r of a discrete random variable X, for the r values with positive probability.

A probability-mass function or a probability distribution can be expressed either in the form of a

table, or as a mathematical expression.

For any probability-mass function:

0< P ( X =r ) <1 ,∧¿ ∑ P (X=r)=1, for all r with positive probability

If a random variable has a large number of values with positive probabilities, then measures of
location and spread can be calculated to better summarize the random variable, as in samples.

Expected Value

The mean of a random variable is called the expected value E(X), or the population mean (µ). It is
calculated as:
R
E ( X ) ≡ µ=∑ xi P ( X=x i ) ,where x i are all the possible values of X with a positive probability .
i=1

This can be thought of as a weighted average of possible values, where the weights are the
theoretical probability for each value. As long as the individual values are distinct from each other, or
they are discrete, R can be finite or infinite.

Population Variance

Variance of a random variable Var(X) is called the population variance σ2. It can be calculated as:
R
Var ( X ) ≡σ 2=∑ ( x i−µ ) P ( X=x i ) ,∨¿ Var ( X ) ≡σ 2=E ( X 2 ) −( E ( X ) )
2 2

i=1
R
¿ E ( X −µ ) =∑ x 2i P ( X =xi ) −µ2
2

i=1
where x i are all the possible values of X with a positive probability .

Analogous to the sample standard deviation, the standard deviation of a random variable sd(X) or σ
is:

√ Var (x)
Approximately 95% of the probability mass is represented within two standard deviation (2σ) range
of the mean of a random variable.

Cumulative-distribution function (cdf) of a random variable X, F(X), assigns each possible value of X,
the sum of probabilities of all the values that are no larger than the considered value. So, for a
specific value x of a random variable X, it is defined as:
k
P ( X ≤ x k ) =∑ P ( X =xi )
i=1
Bernoulli Distribution
The simplest of the discrete probability distributions.

It represents the outcome of one Bernoulli trial.

The outcome is binary. 1 or 0, yes or no, success or failure.

Result of a single coin toss: P(X = heads) = 0.5, P(X = tails) = 0.5, result of a normal coin toss is
either heads or tails with a 0.5 probability for each.

GC content of a DNA molecule: P(X = AT) = 0.6, P(X = GC) = 0.4, the GC content of human genome is
approximately 50%.

Binomial Distribution
Multiple Bernoulli Trials:

Let GC = p = 0.4, and AT = q = 0.6.

If we choose 5 consecutive bases in a DNA molecule, what is the chance that it will be:

GACGT
CTGCA

Bernoulli trials are mutually independent. Thus, we can calculate this probability by multiplication
rule.

P(DNA = GACGT) = P(B1 = GC) * P(B2 = AT) * P(B3 = GC) * P(B4 = GC) * P(B5 = AT)
P(DNA = GACGT) = p * q * p * p * q
P(DNA = GACGT) = p3 * q2
P(DNA = GACGT) = 0.43 * 0.62
P(DNA = GACGT) = 0.023

A general formula can be derived from here:

Let success= p ,∧failure=q ,

what is the probability that we get k success∧( n−k ) failure ∈n Bernouli trials ?
n ( n−k )
p ∗q , but there are n C k ways ¿ choose n success∧( n−k ) failure , so :
k n−k
P ( X=k )=n C k p q , p=probability of success , q=1− p
So, in a binomial distribution:

 Trials are independent of each other.

 Order of success does not matter.
 There are 3 parameters:
o Number of trials (n),
o Number of desired success (k),
o Probability of success in one trial (p).

Since n C k =n Cn−k , P ( X=k )=n C k pk q n−k =( X =k )=n C n−k qn −k pk =P (Y =(n−k ))

The expected value E(X) of a Binomial distribution is np, and

The variance Var(X) of a Binomial distribution is npq.

Multinomial Distribution
Let’s keep following the GC content examples for the Bernoulli and Binomial distributions.

The possible outcomes were only 2, AT or GC.

What if we want to calculate the probability of a 15 amino acid long protein?

For each residue aai, there will be 20 different possible values.

If we generalize this question:

What is the probability of observing y items ki times each, given probability pi of observing each item
separately?

This is the multinomial distribution:

y
n!
P ( k 1 , k 2 , k 3 ,… ,k y )= y ∏ p ki ,where :k + k + k +…+ k =n ,∧¿ p + p + p
i

+…+ p y =1
∏ ki !
i=1 1 2 3 y 1 2 3

i=1

Poisson Distribution
Poisson distribution is related to rare events.

There are three assumptions for Poisson distribution, where t is a long period of time and Δt is a
small sub interval of t:

1- Assume that
 P(1 event) = λΔt for some constant λ.
 P(0 event) ≈ 1 – λΔt
 P(>1 event) = 0
2- Stationarity the number of events occurring in a unit time does not change over t. This
assumption is less likely to hold for overly long periods of t.
3- Independence if an event occurs in a Δt, it has no effect on the probability of occurrence of
another event in the next Δt.

Then, the probability of k events occurring in a time period of t for a Poisson random variable with
parameter λ is:
−µ k
( ) e µ
P X=k = , k=0 , 1 , 2, …where µ=λt ∧e 2.71828
k!
Poisson distribution depends on a single parameter µ = λt. Where λ represents the expected number
of events per unit time, and µ represents the expected number of events over time period t.

In Binomial distribution, there are a definite number of trials, n, and the number of events can be no
larger than n.
However, in Poisson distribution, the number of trials is infinite, and the number of events can be
indefinitely large. The probability of k events becomes very small as k increases.

The expected value E(X) of a Poisson distribution is µ, and

The variance Var(X) of a Poisson distribution is µ.

E(X) ≈ Var(X) for a Poisson distribution. Since it is often hard to predict whether the assumptions for
the Poisson distribution are satisfied, this feature can help to preliminarily identify a distribution as
Poisson before using other tests to confirm this hypothesis.

Since the expected value E(X) of a Poisson distribution is µ, the parameter µ can be estimated from
the observed mean number of events over time t. If the data are not available, other sources can be
used to estimate µ.

Poisson approximation of a Binomial Distribution

For a Binomial distribution, the expected value and the variance are given as np and npq respectively.
In the case of a Poisson distribution with parameters as a large n (≥ 100) and a small p (≤ 0.01),

q=1− p ≈ 1 , sonp ≈ npq

In this case, this Binomial distribution can be accurately approximated by a Poisson distribution with
parameter µ = np.

This is useful because Poisson distribution is easier to work with then Binomial distribution.

Week 6
Negative Binomial Distribution
In Binomial distribution, the probability in question was:
What is the probability of k successes in n independent trials, with a probability of p for success?

In Negative Binomial distribution, however, the question is:

What is the probability of making n independent trials to achieve k success, with a probability of p
for success?

In other words, the number of trials, n, is fixed in Binomial distribution, while the number of
successes, k, is fixed in the Negative Binomial distribution.

For n trials, the probability of k – 1 success in n – 1 trials is calculated, and then multiplied by the
probability of a success.

The idea is:

Since the probability of having to try n times to get k success is the question, the last trial has to be
success, but k – 1 success can be in any order in the rest n – 1 trials, so this is normal Binomial
distribution.
 k−1 ( n −k )
P ( X=n )=n−1 Ck −1 p q p,
k n−k
¿ n−1 C k−1 p q , where

n=number of trials , k =number of success , p=the probability of an individual success ,∧q=1− p

E(X) of Negative Binomial distribution is k/p

Var(X) of Negative Binomial distribution is k(1-p)/p2

Geometric Distribution
Geometric distribution is a special case of Negative Binomial distribution, where k = 1.

That means, we want the probability of getting the first success after n independent trials, where
the probability for each success is p.

Then, if we begin with the PDF of Negative Binomial distribution:

k−1 ( n −k )
P ( X=n )=n−1 Ck −1 p q p,
And replace k = 1:
0 ( n−1)
P ( X=n )=n−1 C0 p q p,
The term nC0 becomes 1, because there is only 1 way to choose 0 elements.
Also, the term p0 becomes 1, because any number raised to the power of 0 is 1.

Then we have:
( n−1 )
P ( X=n )=q p , where
p=the probability of a success , q=1− p ,∧n=the number of trials until the first success
E(X) of Geometric distribution is 1/p

Var(X) of Geometric distribution is q/p2

Note: When there is replacement in the trials, then the conditions turn into conditional probabilities,
where each trial has a probability of success p, which is affected by the outcome of last trial.

If there are 3 blue and 2 red balls in a bag, what is the probability of getting a blue ball?
This is simply is 3/5.

Given that the first ball was blue, what is the probability of getting another blue without replacing the
first one?
If the blue ball is not replaced, then there are 2 blue and 2 red balls in the bag.
The probability of getting a blue in the second trial becomes 2/4 = 1/2

Hyper Geometric Distribution

Hyper Geometric Distribution is similar to Binomial distribution, but there is no replacement.

That means, the trials are not independent, so we cannot simply use Binomial distribution to model
this sampling.

If we have 3 blue and 2 red balls in a bag, what is the probability of picking 2 blue balls in 3 picks?

This behavior can be represented graphically as follows:

At the last step we summed each combination of 2 blue and 1 red balls:
3 3 2
∗2 ∗2 ∗3
5 5 5
∗2 ∗2 ∗2
4 4 4
+ +
3 3 3
3∗3
∗2
5
∗2
4
¿ , follow the ¿ ¿
3
3!
∗3!
2!
∗2 !
1!
∗2 !
1!
¿
5!
3!
∗2 !
1!∗2 !
∗2!∗3 !
1!
¿
5!
3!
∗2 !
2!∗1 !
1 ! 1!
¿
5!
2 !∗3 !

3 C 2∗2 C 1
¿
5 C3

Thus, we can easily calculate the probability of getting 2 blue balls in 3 trials with no replacement,
from a pool of 3 blue and 2 red balls as combination of 2 blue balls out of all three, multiplied by
combination of 1 red ball out of 2, all divided by combinations of 3 balls out of all 5.

If we generalize this formula with the following parameters:

 N = total number of cases

 m = desired cases
 k = number of successes
 s = number of trials

We get the PDF of a Hyper Geometric Distribution:

m Ck ∗N −m C s− k
P ( X=k )= , fork =max ( 0 , s +m−N ) , … , min(m, s )
NCs

m
E(X) of Hyper Geometric distribution is s
N

Var(X) of Hyper Geometric distribution is s (

m N−m
N N )( NN−s
−1 )
Lastly,

If the sampled fraction of the population, s, is no more than 5% of the main population, N; picking
items from the population without replacement will change the probabilities so slightly. That means,
the probability for success at each independent trial will remain somewhat constant, and the Hyper
Geometric distribution will behave like Binomial distribution.

In this case Hyper Geometric distribution can be approximated by the Binomial distribution.

Multivariate Hyper Geometric Distribution

Similar to Multinomial distribution, Multivariate Hyper Geometric distribution is also used to model
the probability of success in the case of multiple choices. However, there is no replacement.

The idea is the same as in Hyper Geometric distribution. We just multiply all miCki for each different
choice, and then divide the result by combinations of total of kis in whole population, NCs.
y

∏ mi C k i
m1 +m2+ …+mi=N , total population
P ( X =k 1 , k 2 ,… , k y )= i=1 , for
Cs N
k 1+ k 2+ …+k i=s , numebr of sample /trials
Week 7
Continuous Probability Distributions
In a continuous range, such as [a, b], there are infinitely many values.

Thus, unlike a discrete random variable, a continuous random variable is not countable; and the
probability of picking a single specific value from a continuous range is practically 0 (1/infinity).
However, the probability of a value falling in sub-range of the continuous range [a, b] can be
calculated.

Uniform / Rectangular Distribution

Uniform Distribution is a kind of probability distribution where the probability of each outcome is
equal.

Throwing a die follows a discrete uniform distribution.

P(X = x) = 1/6, x = {1, 2, 3, 4, 5, 6}

Let’s say a bus is expected to arrive at the bus step in 6 minutes.

Then the bus can arrive at any minute interval in the next 6 minutes. Thus, the bus follows a
continuous uniform distribution.

P(X = x) = p, x = [0, 6]

The probability of the bus arriving in a given time interval can be calculated as the area under the
curve in that time interval.
But the probability that the bus will arrive at a specific time, say 1 minutes 45 seconds and 84
milliseconds later, is 0.

The area under the probability density function of a random variable X, between any two points a
and b, gives the probability that the value of X falls between a and b.

The total area under a pdf is equal to 1.

The cumulative distribution function for a random variable X is evaluated at point a, and gives the
probability that the value of X is smaller than or equal to a. Thus, it is calculated as the area to the left
of a, under the pdf function.
P(X ≤ a) = P(X < a) because P(X = a) = 0

Expected value, E(X) or µ, of a random variable X is the average value that is taken on by X

Variance, Var(X) or σ2, of a random variable X is the average squared distance of each value of X from
the E(X), which is given by E(X - µ)2 or E(X2) - µ2.

The standard deviation, σ, of a random variable X is the square root of its variance, that is:

σ =√ Var (X ).

Normal Distribution
The normal distribution is the most widely used continuous probability distribution in statistics.

It is also called Gaussian distribution.

The normal distribution is denoted as N(µ, σ2).

The pdf of a normal distribution:

1
e
[ −1
2σ
2
( x−µ )
2
], ∞
−¿< x<∞
+ ¿,¿
¿
for parameters µ∧σ , where σ >0.
√2 π σ 2
The probability density function for a normal distribution follows a bell-shaped curve.

The peak of the curve, in other words the mode of the distribution, is at µ, and the most frequently
occurring values around µ.

The curve is symmetrical about µ with points of inflection, meaning the points at which
the slope of the curve changes direction, on either side of µ at µ - σ and µ + σ.
µ and σ2 are used to define a normal distribution, because µ and σ 2 are expected
value, and variance and the variance of a normal distribution, respectively.
The entire shape of the curve is also determined by µ and σ 2.
A special case of anormal distribution is when µ = 0 and σ 2 = 1, N(0, 1). A normal
distribution of this form is called the standard normal distribution.
Substituting µ and σ2 for 0 and 1 respectively, in the probability density function
for a normal distribution:

1
e
−
[ ( ) ],∞
1
2σ2
( x−µ )
2

−¿<x <∞
+¿ ,¿
¿

√2 π σ 2

2
let µ=0∧σ =1 , then:
−1 2
1 x +¿ ¿

e 2
, ∞−¿<x <∞ ¿

√2 π
The standard normal distribution is symmetrical about 0, because f(x) = f(-x).

An interesting feature of the standard normal distribution probability density function:

 about 68% of the area under the curve is in between -1 and +1,
 about 95% of the area under the curve is in between -2 and +2, and
 about 99% of the area under the curve is in between -3 and +3.

This can be expressed more exactly as:

 P(-1 < X < +1) = 0.6827,

 P(-1.96 < X < +1.96) = 0.95, and
 P(-2.576 < X < +2.576) = 0.99.

Thus, the absolute values greater than 3 are unlikely.

Note: ~ symbol means ‘distributed as …’, ‘follow the

distribution of …’. For example, X ~ N(0,1), the random
variable X follows the standard normal distribution.

The cumulative-distribution function for a standard

normal distribution is denoted as:
φ (x) = P(X ≤ x), where X ~ N(0, 1)
Unlike the discrete cumulative-distribution functions, where we have close form formulas to calculate
cumulative probabilities, the area under the cumulative-distribution functions of continuous
distributions must be calculated using numerical methods.

As we stated before, the normal distribution, is symmetrical about its µ. Thus,

φ (−x )=P ( X ≤−x ) =P ( X ≥ x )=1−P ( X ≤ x )=1−φ ( x ) ,φ (−x )=1−φ ( x )

In words, this simply means that:

 The area under the cdf curve to the left of -x is equal to the area under the cdf curve to the
right of the x. Or,
 The probability that X is smaller than or equal to -x is equal to the probability that X is greater
than or equal to x.

So, knowing the probabilities for only 1 side of the distribution, i.e. X < µ or X > µ, is enough.

P(a ≤ X ≤ b) = P(X ≤ b) – P(X ≤ a) = φ(b) - φ(a)

One possible way to define a normal range for some values, say the normal birth weight for
newborns, is to use arrange within x standard deviations for some specified value of x.

Another way is to use percentiles of the distribution, for example upper and lower 5th percentiles.
th
The ( 100∗u ) percentile of a standard normal distributionis denoted by z u
P ( Z < z u ) =u , where Z N (0 ,1)

zu is sometimes referred to as the invers normal function. This is because we first find the area u in
the table, to then find the corresponding value zu.

Note: If u < 0.5, we have to use symmetry properties of the normal distribution to obtain zu = -z(1-u)
Actually, it is possible to describe the pth percentile of a general normal distribution (x) as the
percentiles of the standard normal distribution:

x=µ+ z p σ

Any X N ( µ ,σ 2 ) can be converted to Z N (0 ,1).

Note: this is only applicable if random variable X follows the normal distribution, in other words if
random variable X satisfies the assumption of normality. This means, the conversion only scales the
distribution.

By scaling a normal distribution to standard normal distribution, we can employ powerful statistical
tests that are only applicable to standard normal distributions.

X−µ
if X N ( µ , σ ) ,∧Z=
2
,then Z N ( 0 , 1 ) ,∧¿
σ
P ( a ≤ X ≤b )=P (
a−µ
σ
<Z<
b−µ
σ
=φ ) ( ) ( )
b−µ
σ
−φ
a−µ
σ
The idea is, if the random variable X follows a random distribution N(µ, σ2), then:

 We can first subtract the mean from the X to bring the center of the distribution to 0:
o X–µ

 Then, we can divide this value by σ to scale the spread, so that points of inflection are at µ – 1
and µ + 1, or σ = σ2 = 1:
o (X – µ) / σ

Thus, X~ N(µ, σ2) is converted to an equivalent probability distribution Z~N(0, 1).

This example was skipped during the class but worth knowing about:
Linear Combinations of Random Variables
This concept is also known as Linear Contrast, Linear Transformation, and Linear Mapping.

A linear combination Lof the random variables X 1 , X 2 , … , X n

is defined as any function ∈the form of : L=c1 X 1 +c 2 X 2+ …+c n X n

Often, we need to compute expected value, and the variance for the linear combinations.

Expected Value:

Expected value of the ∑ of n random variablesis the ∑ of therespective expected values :

E ( L )=E ( c1 X 1 +c 2 X 2+ …+c n X n )¿ E ( c 1 X 1 ) +…+ E ( c n X n) =c 1 E ( X 1 ) +…+ cn E ( X n )Thus ,
n n
if L=∑ c i X i , then E ( L )=∑ c i E ( X i)
i=1 i=1

Variance:

Variance of the ∑ of n random variables if the random variables are INDEPENDENT ‼

is the ∑ of therespective variances :Var ( L )=Var ( c 1 X 1+ c 2 X 2 +…+ c n X n )
2 2 2
¿ Var ( c 1 X 1 )+ …+Var ( c n X n )=c 1 Var ( X 1 ) +…+ c n Var ( X n ) Because , Var ( c i X i )=c i Var (X i )Thus ,
n n
if L=∑ c i X i , where X 1 , … , X n are independent ,then Var ( L ) =∑ c 2i Var ( X i)
i=1 i=1

Any linear combination of independent normal random variables will also follow a normal
distribution.
 if X 1 , X 2 , … , X n are independent normal random variabiles
2 2 2
with expected values µ 1 , µ 2 , … , µn∧variances σ 1 , σ 2 , … , σ n ,
n
¿ Lis any linear combination=∑ c i X i ,Then Lis normally distributed with:
i=1
n n
Expected value , E ( L )=∑ c i µi ,∧¿Variance , Var ( L )=∑ c 2 σ 2i
i=1 i=1

Week 8
Midterm

Week 9
Sampling
A random sample is a selection of members of a population, where each member has an
independent non-zero probability of being selected.

A simple random sample is a random sample where each member has the same probability of being
selected.

The group under study, or the group where the sample is selected from, is called; reference, target
or study population.

Cluster sampling is a sampling method where:

 1- The population is divided into clusters,
2- Random clusters are selected,
3- Random members are sampled from each cluster.
4- The totality of the selected members forms the cluster sample.

A random number, or a random digit, is a random variable X that takes on the values 0, 1, 2, …, 9
with equal probability:

1
P ( X=0 )=P ( X =1 )=P ( X=2 )=…=P ( X=9 )=
10
Computer-generated random numbers are collection of digits that satisfy:

 Each digit is equally likely to occur (above expression),

 The value of each digit is independent of each other.

These are also called pseudo random numbers because they satisfy these conditions approximately.

Random selection is when a sample is chosen randomly to represent the population. Data from
random selection can be used for generalization.

Random assignment is when the sample is randomly assigned to groups (i.e., treatment/control).
Data from random assignment can be used for causality.
A Randomized Clinical Trial (RCT) is a type of research design to compare different treatments.
Randomization is the process where the patients are randomly assigned to a specific treatment. If
the samples are large enough, the treatment groups will include similar types of patients. If the
samples are small, the treatment groups may not be comparable. A table of characteristics can be
presented for each treatment group to check if randomization worked well in RCTs.

Block randomization is a process where a block size, of kn, is predetermined for k treatment groups;
and for each kn patients entering the study, n patients are randomly assigned to 1st treatment, n
patients are randomly assigned to 2nd treatment, …, and n patients are randomly assigned to kth
treatment. This is the other way to refer to random assignment.

For every kn patients, there will be equal number of patients assigned to in each group, and the sizes
of treatment groups will be equal throughout the study period. This ensures comparability of
treatment groups over short periods of time as the study procedures change and are adjusted.

The randomization process may get apparent over time and physicians may take decisions that will
affect the randomization (i.e., prioritizing certain patients, delaying their entry to study until a
treatment gets more likely to work). To avoid this, variable block size is sometimes preferred.

In stratification, patients are divided into groups, or strata, based on characteristics that could be
important for the outcome of trials; such as age, gender, or over all clinical condition. Then, random
selection or random assignment can be used in each stratum. Separate randomization lists are
maintained for each stratum to ensure comparable patient populations in each stratum.

Blinding:

 If neither the patients nor the physicians know which treatment is applied to which group,
this is called a double-blind study.
 If only patients don’t know what treatment they get, but the physicians are aware of the
study plan, this is called a single-blind study.

The gold standard is a randomized double-blinded study.

Degrees of freedom, is the number of free parameters of an underdetermined system.

Degrees of freedom, is the number of parameters that can be freely decided on, until an
underdetermined system becomes determined.

Underdetermined means there is not enough evidence (data) for certainty.

x + y−z=10x=? , y =? , z=? x=10− y + z

Here, for example y and z are free parameters. The value of x depends on y and z. The equation can
be rearranged for y or z too. So, the number of free parameters = the degree of freedom = 2.

In general:

degrees of freedom=the number of variables−thenumber of equations

There are 3 variables and 2 equations, so the df is:

3–2=1

Indeed, if x1 or x2 is known in the below equation, the

whole system can become determined.

There are 3 variables and only 1 equation, so the df is:

3–1=2
 This time we have sample mean and sample
variance equations as a system.

If the above equation, the sample mean,

considered alone the df is 3 – 1 = 2. Similarly,
if the below equation, the sample variance, considered alone the df is 4 – 1 = 3. However, if we
consider the system as a whole, the df becomes 4 – 2 = 2. This is why we use N-1 degrees of freedom
when calculating the sample variance, if we also calculate the mean using the same data.

Estimation – I (Mean)
How is a specific random sample x1, x2, …, xn, of a population can be used to estimate the µ σ2 of the
underlying distribution.

Inferring the properties of a distribution underlying the data at hand involves inductive reasoning
(tüme varım), rather than deductive reasoning (tümden gelim).

Statistical inference can be divided into:

 Estimation: Estimating the values of specific population parameters (mean, variance, etc.).
 Hypothesis Testing: Testing if the value of a population parameter is equal to some specific
value.

When a specific value is estimated for a parameter, this is called point estimate.

When a range of values within which the values of a parameter are likely to fall, this is called
interval estimate. If this range is narrow, then the point estimate can be considered good.

Point Estimation

A natural estimator for the estimation of population mean µ is the sample mean.
n
Xi
X =∑
i=1 n
Sampling distribution of X , is the distribution of values of x over all possible samples of size n, that
could have been selected from the reference population.

The values of X , in each of these samples will be denoted x 1 , x 2 ,… and so forth.

For any population, regardless of the underlying distribution:

Let X 1 , X 2 , … , X n be a random sample drawn ¿ a population withmean μ .Then ,

for the sample mean X , E ( X ) =μ .
Thus, X is an unbiased estimator of µ.

More generally:

Anestimator θ^ for a parameter θ is unbiased if E ( θ^ ) =θ .

θ^
The average value of large number of random samples of ¿ is θ .
a
The ideal estimator is unbiased and also minimum variance.
Standard Error of Mean

The larger the sample size, the more precise the estimator X is.

Let X1, X2, …, Xn be a random sample from a population with mean µ and variance σ2.

( )
n n
1
Var ( X )= 2 ∗Var
n
∑ X i Var ( X )= n12 ∗∑ Var ( X i ) ,∧Var ( X i ) =σ 2
i=1 i =1

√
2 2
1 nσ σ σ2 σ
Thus , Var ( X )= 2
∗( σ
2
+σ
2
+…+ σ
2
) Var ( X ) = 2
= ,∧¿ SD=√ Var ( X )= =
n n n n √n

The set of sample means in repeated random samples of size n from this population has variance
σ /n. The standard deviation of this set of sample means is σ /√ n and is called the standard error of
2

the mean (sem), or standard error (se).

However, often times, the population variance σ2 is unknown. Thus, sem is estimated by s/ √ n.

Shortly, standard error of the mean, represents the estimated standard deviation of a set of sample
means, which are obtained by repeated random samples of size n from a population with variance
σ2.

Thus, the greater the sample size n, or the smaller the population variance σ2; the lower the
standard error of the mean, and the more precise the estimate of µ.

Central – Limit Theorem

If the population is distributed normally, then X also follows a normal distribution with mean µ
and variance σ 2 /n:
2
X N ( μ , σ / n)
Moreover, even when the underlying distribution is not normal, for a large sample size n, X
approximately follows a normal distribution with mean µ and variance σ 2 /n:

Let X 1 , X 2 , … , X n be a random sample ¿ some population ,with mean μ∧variance σ 2 .

2
Then , for a large n : X ∼
˙ N (μ , σ /n)
∼
˙ means approximately distributed as.
This is the central-limit theorem, and lets us perform statistical tests based on approximate normality
of the sample mean, despite the non-normality of the population.

Interval Estimation

Often times, along with a precise point estimate, we also want an interval of acceptable values for
the parameter being estimated. This is interval estimation. According to central-limit theorem,
interval estimates hold exactly if the population is distributed normally and approximately if the
population is not distributed normally.

If X N ( μ , σ 2 / n), and if we knew µ and σ2, then we could precisely know the behavior of the set of
sample means over a large number of samples of size n. More precisely:

95 % of such sample means would fall withinthe interval(μ−(1.96 σ /√ n) , μ+(1.96 σ /√ n))

This is because:

And 95 % of the values of Z is within theinterval (−1.96 ,+1.96).

However, σ is rarely known in practice.

We already discussed that, when σ is unknown, the standard error of the mean σ /√ n can be
estimated using sample variance by s/ √ n.

However, we cannot use the above property with s/ √ n, because (X −μ)/(s/ √ n) does not follow a
normal distribution anymore.

(X −μ)/(s/ √ n) is referred to as Student’s t distribution, and the shape of the distribution depends
on the sample size n. t distribution is a family of distributions indexed by the degrees of freedom
(df) of the distribution.

If X1, …, Xn ~ N(µ, σ2) and are independent, then (X −μ)/(s/ √ n)is distributed as t distribution with
(n-1) df.

The 100 x uth percentile of a t distribution with d degrees of freedom is denoted by td, u and given as:

P ( t d <t d , u ) ≡ u

As d becomes large, the corresponding t distribution, td, converges to an N(0, 1) distribution. This is
because d = (n-1) and for larger sample size n s2 approximates better to σ2.

The difference between t distribution and the standard normal distribution is the greatest for n < 30.

To use this information practically

If σ is unknown replace it with s and obtain the t statistic:

t=( X −μ ) /(s /√ n)
This quantity should follow a t distribution with (n-1) degrees of freedom, t(n-1).

95% of the statistic in repeated samples of size n should fall between 2.5th and 97.5th percentiles of a
t(n-1) distribution, shortly:

P ( t n−1 , 0.025< t<t n−1 ,97.5 ) =95 %

This can also be generalized as, 100% x (1-α) of the t statistics should fall between lower and upper
α/2 percentiles of the t(n-1) distribution in repeated samples of size n, shortly:

P t
( n−1,
α
2
< t<t
n−1 ,1−
α
2
)=1−α
The inequality can be divided into two inequalities:
 X −μ
∧X−μ
s /√ n
t α < <t α
n−1 ,
2
s /√ n n−1 ,1−
2

Then, multiply each side by (s/ √ n) and add µ to each side, to obtain:

t
n−1 ,
α
2
( √sn )+ μ< X ∧X <t n−1 ,1−
α
2
( √sn )+ μ
Finally, subtract t
n−1 ,
α
2
( √sn ) from both sides in the first inequality, and t
n−1 ,1−
α
2
( √sn ) from both sides

in the second one, to obtain:

μ< X−t
n−1 ,
α
2
( √sn )∧X −t n−1, 1−
α
2
( √sn )< μ
Expressed as a single inequality, as in the beginning, we obtain:

X −t
n−1 , 1−
α
2
( √sn )< μ< X−t ( √sn ) n−1 ,
α
2

The t distribution is also symmetrical, such that tn-1, α/2 = -t n-1, 1-α/2. So, the inequality can be written as:

X −t
n−1 , 1−
α
2
( √sn )< μ< X +t n−1 ,1−
α
2
( √sn )
Thus, it can be said that:

P X−t
( n−1 ,1−
α
2
( √sn )< μ< X +t n−1 ,1−
α
2
( √sn ))=1−α
(
The interval X−t
n−1 ,1−
α
2
( √sn ), X + t n−1 ,1−
α
2
( √sn )) is referred to as the 100% * (1 – α) confidence

interval (CI) for the mean µ of a normal distribution with unknown variance.

A shorthand notation for this is:

X ±t
n−1 , 1−
α
2
( √sn )
The boundaries of the interval depend on the sample mean and sample variance, and vary from one
sample to another. For A% CI for the µ of a normal distribution with unknown variance, A% of such
intervals that could be constructed from repeated random samples of size n, will contain population
parameter µ.

Moreover, the length of the CI gives some idea about the precision of the point estimate x , and is
given by:

2∗t
n−1 ,1−
α
2
( √sn )
From this, we understand that the length of a CI is determined by 3 variables:
  n: as the sample size n increases, the length of the CI decreases.
 s: as the sample standard deviation s increases the length of the CI increases.
 α: as the alpha decreases, the desired confidence increase (1- α), and the length of the CI
increases.

If the sample size is sufficiently large, x > 200, then the distribution converges to a normal distribution
and their percentiles are virtually the same. In this case 100% * (1 – α) CI for µ of a normal
distribution with unknown variance can be approximated as:

X±z
1−
α
2
( √sn )
This interval can also be used if the variance is known, by replacing s with σ.

Week 10
Hypothesis Testing – I (Mean)
Hypothesis testing enables us to make decisions based on a consistent uniform criterion.

Two types of hypothesis testing:

 one-sample problem: hypotheses are built on a single distribution.

 two-sample problem: two different distributions are compared.

Null Hypothesis, denoted H0, is the hypothesis that is being tested.

Alternative Hypothesis, denoted H1, is the opposite of null hypothesis.

In other words, alternative hypothesis is the extreme condition that we investigate, and the null
hypothesis is the opposite of alternative hypothesis which we expect to observe in the population
already.

For example, if we think that the mean of a group is smaller than the other, our null and alternative
hypotheses can be expressed shortly as:

H 0 : μ=μ 0 vs . H 1 : μ< μ 0

We try to decide whether H0 or H1 is true.

For simplicity, all outcomes stated around null hypothesis, H0.

If we decide H0 is true, we say we accept H0.

If we decide H1 is true, we say we reject H0.

This results in 4 outcomes:

1- We accept H0, and H0 is true.

2- We accept H0, and H1 is true.
3- We reject H0, and H0 is true.
4- We reject H0, and H1 is true.

Practically it is impossible to use statistical testing methods and prove that the null hypothesis is
true. Thus, if we accept H0, then we say we failed to reject H0.
Note: Why it is impossible to prove null hypothesis?

Because we assume the samples are coming from some distributions, and we cannot really now the
true distributions with exact parameters. Thus, we can only claim that there is not enough data to
reject H0, or show it is false. Hence, we fail to reject H0.

It is simpler to understand with an analogy.

Imagine that someone committed a crime, but we don’t know about it. This person is suspected to be
guilty, and we try to prove this is true. In this case:

H 0 :accused is innocent vs . H 1 :accused is guilty

So, if we gather enough evidence, we can reject H0, and say accused is actually guilty.

However, if we can’t gather enough data, that does not prove accused is innocent. We just fail to
falsify that accused is innocent.

Therefore, since we do not now the true nature of the underlying distribution, there is always a small
chance that H0 is not true, and thus we can’t prove H0 is true, but we can only fail to reject H0.

Correct decision:

 H0 is true, and H0 is accepted.

 H1 is true, and H0 is rejected.

The other two cases are errors:

 Type I error:
H0 is true, and H0 is rejected.
P ( reject H 0∨H 0 istrue )=P ( type I error ) =α =significance of the test

 Type II error:
H1 is true, and H0 is accepted.
P ( type II error )=β ,∧¿1−P ( accept H 0∨H 1 is true )=1−P ( type II error )
¿ 1−β= power of the test=P ( reject H 0∨H 1 istrue )
We aim to make α and β as small as possible for the statistical tests. However, making α small means
rejecting H0 less often, and making β small means accepting H0 less often. Moreover, as α decreases,
β increases, and vice versa. A general strategy is to fix α at some specific level and to use the test
that minimizes β, thus maximizing the power (1 – β).

ONE-SAMPLE TEST FOR THE MEAN OF A NORMAL DISTRBUTION: ONE SIDED ALTERNATIVES

Say we want to test if a sample has lower mean than a population, assuming they both follow a
normal distribution with mean µ and an unknown variance σ2.

The best, or most powerful test would be based on the sample mean x , which has the highest
power (1 – β) for a fixed significance (α).

If x is smaller enough than µ0, then H0 is rejected. Otherwise H0 is failed to reject.

That makes sense because if H0 is true, then the most likely values of x tend to cluster around µ0,
whereas if H1 is true then the most likely values of x tend to cluster around µ1.

The acceptance region: is the range of values of x for which we fail to reject H0.

The rejection region: is the range of values of x for which we reject H0.

One-tailed test: the values of the parameter being studied (in this case µ) under H1, are allowed be
either smaller than or greater than the values of the parameter under H 0, but not both.

How small the x should be to reject H0?

Remember that the significance of the test is set to be α.

Suppose H0 is rejected for all values of x < c, and accepted otherwise. The value c should be selected
so that the probability of type I error = α.

If we define the criteria based on the standardized terms, instead of x , for convenience:
X−μ 0
S t= t n −1
if we subract μ0∧divide by , we get s
√n
√n
¿ the definition of the percentilesof a t distribution: P ( t< t n−1 , α ) =α

So, we will calculate t values, and reject H0 for t < tn-1, α, and accept it otherwise.

The t value here is called a test statistic because the test procedure is based on this statistic.

tn-1, α here is called the critical value. The outcome of the test depends on the relation of test statistic
and the critical value. In this case, for example, we reject H0 if test statistic < critical value, t < tn-1, α.

This general approach, where a test statistic is calculated and compared to a critical value,
determined by the significance of the test (α, probability of type I error), to decide on the outcome
of the test is called the critical-value method of hypothesis testing.
Procedure: One – Sample t Test for The Mean of A Normal Distribution with Unknown Variance,
One – Sided (Alternative Mean < Null Mean)

To test the hypothesis H 0 : μ=μ 0 , σ unknown vs . H 1 :μ < μ0 , σ unknown with a significance level of
α, we compute:

X−μ 0
t=
S
√n
If t <t n −1 , α , then we reject H0.

If t ≥ t n−1 ,α , then we fail to reject H0.

What should the α be?

The level of significance, α, depends on the relative importance of type I and type II errors. This is
because, for a fixed sample size (n), as α gets smaller, β gets greater and the power of the test (1 – β)
drops.

Generally, significance levels greater than 0.05 are not welcomed, and α = 0.05 is used most
frequently.

Several statistical tests at different α levels can be performed and whether H0 would be rejected or
accepted could be noted. A lot of work, and unnecessary.

Significance tests can be effectively performed at all α levels, by calculating the p-value for the test.

p-value for any hypothesis test is the α level at the borderline of acceptance and rejection regions.
In other words, we would neither accept not reject H0 at the α level of p-value.

If the p-value is used as the criterion in above test, then:

reject H 0 for t <t n−1, p ,accept H 0 for t >t n−1 , p ,no choice for t=t n−1 , p

If we solve for p as a function of t, we get: p=P ( t n−1 ≤ t )

Thus, p is the area to the left of t under a tn-1 distribution.

p-value can also be defined as the probability of obtaining a test statistic as extreme as or more
extreme than the actual test statistic computed, given that H0 is true.

Guidelines for Judging the Significance of p-value:

 If p < 0.001, the results are very highly significant,

 If 0.001 ≤ p < 0.01, the results are highly significant,
 If 0.01 ≤ p < 0.05, the results are significant,
 If p > 0.05, the results are not statistically significant (NS).

However, if 0.05 < p < 0.1, the trend towards statistical significance is sometimes noted.

If, instead of the critical value method, p-value for the test is calculated, it is referred to as the p-
value method. If p < 0.05, H0 is rejected and the results are declared statistically significant.
Otherwise, the H0 is accepted and the results are declared to be NS.

Sometimes, the results of a study may be statistically significant but scientifically not important.

This may occur if slight differences found to be statistically significant because the sample size is
large.

In contrast, some results may not be statistically significant but can be scientifically important. This
may encourage researchers to perform larger studies to confirm the results, and possibly reject H0
with a larger sample size.

Procedure: One – Sample t Test for The Mean of A Normal Distribution with Unknown Variance,
One – Sided (Alternative Mean > Null Mean)

To test the hypothesis H 0 : μ=μ 0 , σ unknown vs . H 1 :μ > μ0 , σ unknown with a significance level of
α, we compute:

X−μ 0
t=
S
√n
If t >t n −1 , 1−α, then we reject H0.

If t ≤ t n−1, 1−α , then we fail to reject H0.

The p-value for this test is given by:

p=P (t n−1 >t)

ONE-SAMPLE TEST FOR THE MEAN OF A NORMAL DISTRBUTION: TWO SIDED ALTERNATIVES

In a two-tailed test, the values of the parameter being studied (in this case µ) under H1 are allowed
be on both sides of (either greater or less than) the values of the parameter under H0 (µ0).

The best test is again based on x .

In a two-tailed test, we will reject H0 if the alternative values of the parameter on two sides of the
null are too small or too large.

In other words, H0 will be rejected either for t > c1 or t < c2 for some constants c1, c2 and H0 will be
accepted if c1 ≤ t ≤ c2.

Values for c1 and c2 are again determined by α.

P ( reject H 0| H 0 is true ) =P ( t< c1 ∨t> c2|H 0 is true )

¿ P ( t< c 1∨H 0 is true ) + P ( t >c 2∨H 0 is true )=α

Half of the α is arbitrarily assigned to each tail, so that:

α
P ( t< c 1∨H 0 is true ) =P ( t> c 2∨H 0 is true ) =
2
Since t follows a tn-1 distribution under H0, tn-1, α/2 and tn-1, 1 – (α/2) are the lower and upper 100 x (α/2)
percentiles of a tn-1 distribution. So:

α
(
P t< t
n−1 ,
α
2 ) (
=P t>t
n−1 ,1−
α
2 )
=
2

Therefore:

c 1=t α ,−t α ∧c 2=t α

n−1 , n−1 ,1− n−1 ,1−
2 2 2
Procedure: One – Sample t Test for The Mean of A Normal Distribution with Unknown Variance,
Two – Sided

To test the hypothesis H 0 : μ=μ 0 vs . H 1 : μ ≠ μ 0, with a significance level of α, the best test is based
on:

X−μ 0
t=
S
√n
If |t |>t n−1 ,1− α , then we reject H0.
2

If |t |≤t n−1 ,1− α , then we fail to reject H0.

2

The p-value for the one-sample t test for the mean of a normal distribution, two-sided:

X−μ 0
Let t=
S p=
√n
{
2∗P ( t n −1 <t ) , if t ≤ 0
2∗[ 1−P ( t n−1< t ) ] , if t >0

If sample size is large enough (n > 200, but Dr. Pir says even > 30 is enough), then the t distribution

( t>t ) used in determining the critical values can be approximated by the percentiles of N(0,1)
n−1 ,1−
α
2

distribution ( z ). Similarly, I computing p-values, p = P(t < t) can be approximated by P[N(0, 1) <
1−
α n-1
2
t] = φ(t).

ONE-SAMPLE z Test

In some cases, the variance may be assumed to be known from previous studies. In this case the test
statistic t can be replaced by z=(X−μ 0)/(σ / √ n). Also, the critical values that were based on the t
distribution can be replaced by the corresponding critical values from the z distribution. This leads
to the following procedure.
Procedure: One – Sample z Test for The Mean of A Normal Distribution with Known Variance, One –
Sided (µ1 < µ0)

To test the hypothesis H 0 : μ=μ 0 vs . H 1 : μ< μ 0 with a significance level of α, where the underlying
standard deviation σ is known, the best test is based on:

X−μ 0
z=
σ
√n
If z < z α , then we reject H0.

If z ≥ z α , then we fail to reject H0.

The p-value for this test is given by:

p=ϕ (z)
Procedure: One – Sample z Test for The Mean of A Normal Distribution with Known Variance, One –
Sided (µ1 > µ0)

To test the hypothesis H 0 : μ=μ 0 vs . H 1 : μ> μ 0 with a significance level of α, where the underlying
standard deviation σ is known, the best test is based on:

X−μ 0
z=
σ
√n
If z > z 1−α, then we reject H0.

If z ≤ z 1−α , then we fail to reject H0.

The p-value for this test is given by:

p=1−ϕ (z )

Procedure: One – Sample z Test for The Mean of A Normal Distribution with Known Variance, Two –
Sided

To test the hypothesis H 0 : μ=μ 0 vs . H 1 : μ ≠ μ 0, with a significance level of α, the best test is based
on:

X−μ 0
z=
σ
√n

If z < z α ,∨z> z 1− α then we reject H0.

2 2
zα ≤ z≤ z α then we fail to reject H0.
1−
2 2

To compute the two-sided p-value:

p=
{ 2∗ϕ ( z ) , if z ≤ 0
2∗[ 1−ϕ ( z ) ] , if z >0

Relationship Between Hypothesis Testing and CI

Suppose that we are testing H 0 : μ=μ 0 vs . H 1 : μ ≠ μ 0.

t α ∗s
n −1 , 1−
The two-sided 100% x (1 – α) CI for µ = (c1, c2) = x ± 2 .
√n
Suppose we reject H0 at level α. That means either t <−t n−1 , 1− α or t >t n −1 , 1− α .
2 2

Suppose that:

t=( x−μ0 )/( s/√ n)←t α

n−1 ,1−
2

Multiply both sides with s/ √ n:

t α ∗s
n−1 , 1−
2
x−μ0 ←
√n
Add μ0 to both sides:

t α ∗s
n−1 ,1−
2
x < μ0−
√n
or
t α ∗s
n−1 ,1−
2
μ0 > x + =c 2
√n
Similarly, for α:
t >t
n −1 , 1−
2
t α ∗s
n−1 ,1−
2
x−μ0 >
√n
or
t α ∗s
n−1 ,1−
2
μ0 < x− =c 1
√n
This means that, if we reject H0 at level α using a two-sided test, then either µ0 <
c1 or µ0 > c2; That is, µ0 must fall outside of the two-sided 100% x (1 – α) CI for µ.
It can also be shown, similarly, that if we accept H0 at level α using a two-sided test, then µ0
must fall within the two-sided 100% x (1 – α) CI for µ (or c1 < µ0 < c2).

A similar relationship exists between one-sided hypothesis tests and one-sided

CI for parameter µ.

The p-value from a hypothesis test tells us how statistically significant the
results are.

However, the significant results may not be scientifically important because the
difference between means are not so great and the significance is due to large
sample size.

A CI for µ gives additional and complementary information. CI provides the

possible range of values within which the µ is likely to fall.

The Power of A Test

The power is calculated to plan a study before any data have been obtained.

A small preliminary study, called a pilot study, can be performed to asses the
variables required to calculate power.

Also, the standard deviation is predicted/a projection is made concerning the

standard deviation, without any data to estimate it.

Thus, the standard deviation is assumed to be known and the power calculations
are based on one-sample z tests.

One – Sided Alternatives:

Suppose we are testing H 0 : μ=μ 0 vs . H 1 : μ< μ 0, assume that the underlying

distribution is normal and the population variance is known.

The best test at significance level α is based on:

x−μ 0
σ
z=
√n
we reject H 0 if z< z α ,

we accept H 0 if z > z α

¿ the p−value for this test is givenby : p=ϕ( z)

So far, we have seen that for testing, let’s say, H 0 : μ=μ 0 vs . H 1 : μ ≠ μ 0, the form of
the best test does not depend on the specific value of alternative mean, as long
as this value is less than the null mean.
However, the specific value of alternative mean affects the power of the test,
which is:
1 – P(type II error).
Power=1−P ( type II error )=1−P(accept H 0∨H 0 is false)

¿ P ( reject H 0| H 0 is false ) =P ( Z < z|μ=μ 1)

( | )
x−μ0
¿P < z α μ=μ1
σ
√n

(
¿ P X < μ0 +
z α ∗σ
√n | )
μ=μ1

We know that under H 1 , X N μ 1 ,

( , if we standardize the limits:
)
σ2
√n

[(
Power=ϕ μ 0+
z α∗σ
√n ) ] [
−μ1 /(σ / √ n) =ϕ z α +
μ0−μ1
σ
∗√n ]
Which is the area to the left of z α + ∗√ n under N(0, 1) distribution.
μ0 −μ1
σ

The area to the left of μ0 + under the H0 distribution is the significance level
z α∗σ
√n
= α, whereas the area to the left of μ0 + under the H1 distribution is the
z α∗σ
√n
power = 1 – β.

The power tells us how likely it is to find a statistically significant difference

based on a finite sample size n, if the alternative hypothesis true.
If the power is too low, there is little chance of finding such a difference, and
nonsignificant results are likely even if there is a difference between the mean of
the sample being studied and the null mean.

An insufficient sample size is usually the cause of low power.

If we were testing H 0 : μ=μ 0 vs . H 1 : μ> μ 0 instead, again assuming the underlying

distribution is normal and the population variance is known;

The best test at significance level α would be a one-sample, one-sided z test.

We would reject H0 for z > z1- α and accept it for z ≤ z1- α.

With a similar reasoning to previous one, the power of this test would be:

[
Power=ϕ −z 1−α +
μ1−μ 0
σ ] [
∗√n = ϕ z α +
μ 1−μ0
σ ]
∗√ n , if μ1 > μ0

Power of a statistical test – One-sided: In summary

The power of the test for the hypothesis:

H 0 : μ=μ 0 vs . H 1 : μ< μ 0

Where the underlying distribution is normal and the population variance (σ2) is
assumed to known is given by:

[
Power=ϕ −z 1−α +
|μ0 −μ1|
σ ] [
∗√ n =ϕ z α +
|μ0 −μ 1|
σ
∗√ n ]
The power depends on 4 factors:
 1- If α decreases, zα decreases, and hence the power decreases.
2- If |µ0 – µ1| increases, the power increases.
3- If sigma increases σ, the power decreases.
4- If n increases, the power increases.

Two – Sided Alternatives

Suppose we are testing H 0 : μ=μ 0 vs . H 1 : μ=μ 0, assume that the underlying

distribution is normal and the population variance is known.

The best test at significance level α is based on:

x−μ 0
σ
z=
√n
if z α < z< z α , we reject H
1− 0
2 2

if z α ≤ z ≤ z α then we fail to reject H0.

1−
2 2

The exact power is given by:

Power=ϕ −z
[ 1−
α
2
+
( μ0−μ 1 )
σ
∗√ n +ϕ −z
] [ 1−
α
2
+
( μ1−μ 0 )
σ
∗√ n
]
And approximated by:

Power=ϕ −z
[ 1−
α
2
+
|μ0 −μ 1|
σ
∗√ n
]
If µ1 < µ0, the second term of the equation is negligible compared to the first one, and usually only the
first term used as an approximation.

If µ0 < µ1, the first term of the equation is negligible compared to the second one, and usually only the
second term used as an approximation.
Sample Size Determination

To plan a study, we need an idea about the appropriate sample size.

If the required sample size is too big, knowing this before starting the study will save us a lot of effort
and resources, and we can replan our study to make it more practically possible.

The appropriate sample size can be verbally defined as:

For one-sided

The sample size that is needed to be able to detect a significant difference with probability 1 – β,
given that a one-sided statistical test will be performed at significance level α, and that the true
alternative mean is µ1.

And derived from the power formula:

[
Power=ϕ z α +
|μ 0−μ1|
σ
∗√ n =1−β]
We want to solve for n in terms of α , β ,|μ0−μ1|,∧σ .

From properties of percentiles of z distribution, we know ϕ ( z 1−β ) =1−β . Therefore:

|μ 0−μ1|
zα+ ∗√ n=z1− β
σ
Subtract z α from each side:

|μ0 −μ1|
∗√ n=z 1− β−z α
σ
σ
Multiply each side by :
|μ0 −μ1|
z 1−β −z α
√ n= ∗σ
|μ0 −μ 1|
From properties of percentiles of z distribution, we know z 1−α =−z α . Therefore:

z 1−β + z1−α
√ n= ∗σ
|μ 0−μ1|
Finally, square both sides to get n, and thus the sample size required to perform a one-sided
hypothesis test at significance level α and detect a statistically significant difference with a
probability of 1 – β is obtained as:
2
( z 1− β + z 1−α ) 2
n= 2
∗σ
( μ0−μ1 )
The sample size is always rounded up, to make sure we achieve at least the required level of power.

The sample size is extremely sensitive to alternative mean chosen.

The sample size is inversely proportional to ( μ0 −μ1 ) . If |μ0 −μ1| is halved, then the required sample
2

size is 4 times as large. If, on the other hand, |μ0 −μ1| is doubled, then the required sample size is 4
times as small.

Factors affecting the sample size:

1- As σ2 increases, the sample size n increases.

2- As α decreases, the sample size n increases.
3- As (1 – β) increases, the sample size n increases.
4- As |μ0 −μ1| decreases, the sample size n increases.

Estimation of parameters for calculating the sample size:

 μ0 is usually easy to specify.

 α is usually set to 0.05.
 1−β parameter is less clear; however, it is usually ≥ 0.80.
2
 The appropriate values for μ1∧σ are usually unknown.
o Can be obtained from previous work.
o Can be obtained from a pilot study.
o μ1 can be estimated considering; w-hat would be a scientifically important
difference |μ0 −μ1| in the context of the problem studied.
2
Because of the inaccuracy in the estimation of parameters μ1∧σ , most of the sample size estimates
are nothing more than rough guesses, and used only to check if the proposed sample size is close to
what is actually needed, rather than identifying a precise sample size.

For two-sided:

The sample size required to perform a two-sided hypothesis test at significance level α and detect a
statistically significant difference with a probability of 1 – β is obtained as:

( ) ∗σ
2
z1− β +z α
1−
2 2
n= 2
( μ0−μ1 )

This sample size is always larger than the one-sided alternative because z 1−α < z1− α .
2

Sample Size Estimation Based on CI Width

Sometimes it is already known that there will be a significant result (i.e. physiological effect of a
drug). The interest is focused on estimating this result with a given degree precision.

Suppose we quantify the precision of the estimation by the width of the two-sided 100% * (1 – α) CI.

The two-sided 100% * (1 – α) CI for parameter µ is:

 t α ∗s
n −1 , 1−
2
x±
√n
The width of this interval is:

( )
t α ∗s
n−1 , 1−
2
2∗
√n
We eish this interval to be no wider than a constant L:

( )
t α ∗s
n−1 , 1−
2
2∗ =L
√n

Multiply the both side of the equation by

√n:
L

( )
t α ∗s
n−1 ,1−
√ n=2∗ 2
L

Square the both sides to get n:

( )
2
t α ∗s
n−1 ,1−
n=22∗ 2
L
2 2
4∗t α ∗s
n−1, 1−
2
n= 2
L

Usually, t n−1 ,1− α is approximated by z 1− α :

2 2

2 2
4∗z α ∗s
1−
2
n= 2
L
Thus, the sample size n for the estimation of the mean of a normal distribution with sample variance
s2, and with a two-sided 100% * (1 – α) CI for µ no wider than L is calculated.

Estimation II (Variance)
Estimation of The Variance of A Distribution

Point estimation

Let X1, X2, …, Xn be a random sample from some population with mean µ and variance σ2. The sample
variance S2 is an unbiased estimator of σ2 over all possible random samples of size n that could have
been drawn from that population.

E ( S2 ) =σ 2
This estimation holds for any underlying distribution.

Remember:
n
1
s=2
∑
n−1 i=1
( x i−x )
2

The Chi-Square Distribution

n
If G=∑ X i
2

i=1

Where X1, X2, …, Xn ~ N(0, 1),

and the Xi’s are independent, then G is said to follow a chi-square distribution with n degrees of
2
freedom. The distribution is often noted by χ n .

Like t distribution, chi-square distribution is also a family of distribution indexed by the parameter n
referred to as degrees of freedom (df).

However, unlike the t distribution, chi-square distribution only takes on positive values and always
skewed to the right.

The skewness declines as n increases.

2
The expected value of a χ n distribution is n and the variance is 2n.
2 2
The uth percentile of a χ n distribution is denoted by χ n , u:
2 2
P( χ ¿ ¿ n < χ n ,u )≡ u ¿
Interval Estimation

We often want to obtain the interval estimate of the variance.

The interval estimates will hold exactly, only if the underlying distribution is normal, as in the interval
estimates of the mean. However, they perform poorly if the underlying distribution is not normal, in
the case of variance. Thus, they should be used with caution.
2
Assume that X 1 , X 2 , … , X n N ( µ , σ ).

X−μ
If X N (µ , σ 2 ), and Z= , then Z N ( 0 , 1 ). Thus:
σ
2
n n
( x i−μ )
∑ Z =∑ 2
i
σ 2
χ 2n
i=1 i=1

We usually don’t know μ and estimate it by x . However, if we substitute x for μ, we lose one 1 df
resulting in:
2
n
( x i−x )
∑ σ 2
χ 2n−1
i=1

Quickly remember that the sample variance S2 is:

n
1
S2 = ∑
n−1 i=1
( x i−x )
2

If we multiply both sides by (n – 1):

n
S2∗(n−1)=∑ ( x i−x )
2

i=1

And if we substitute this back into the expression above, we get:

2
S ∗( n−1 ) 2
χ n−1
σ2
 2
σ
If we multiply both sides of the equation by :
n−1
2
2 σ 2
S ∗χ n−1
( n−1 )
2
Thus, we can show that S2 follows a chi-square distribution with (n – 1) df ( χ n−1), multiplied by a
2
σ
constant
( n−1 )
Now, we can use this to express:

( )
2 2
σ 2 2 σ 2
P ∗χ α < S < ∗χ =1−α
( n−1 ) n−1 ,
2 ( n−1 ) n−1 ,1− α2

The inequality in the probability can be expressed as two inequalities:

2 2 2 2
σ ∗χ α σ ∗χ α
n−1 , n−1 , 1−
2 2 2 2
< S ∧S <
( n−1 ) ( n−1 )
n−1
If the both sides of the first equation multiplied by χ 2 , and the both sides of the second equation
α
n−1 ,
2
n−1
multiplied by χ 2 :
α
n−1 ,1−
2

( n−1 )∗S 2
∧( n−1 )∗S2
χ2 α
n−1 ,
σ 2< 2
2
< σ2
χ α
n−1 , 1−
2

If we combine and put these back into the original expression:

[ ]
2 2
( n−1 )∗S 2 ( n−1 ) ∗S
P 2
<σ < 2 =1−α
χ α χ α
n−1 , 1− n−1 ,
2 2

Thus, a 100% * (1 – α) CI for σ2 is given by:

[ ( n−1 )∗s 2 ( n−1 )∗s 2

χ
2
α
n−1 ,1−
, 2
χ
2
α
n −1 ,
2
]
And a 100% * (1 – α) CI for σ is given by (the square root of above interval):
 [√ χ α
n−1 ,1−
χ
2
√
( n−1 )∗s2 ( n−1 )∗s 2
2
, 2
α
n−1 ,
2
]
Hypothesis Testing – II (Variance)
If x1, x2, …, xn are a random sample, then we can base the test for variance on s2, because s2 is an
unbiased estimator of σ2.

If x 1 , x 2 ,… , x n are a random sample from N ( μ , σ 2 ), then under H0:

( n−1 )∗s 2 2
2
X = χ n−1
σ2
Therefore,

α
( ) ( )
2 2 2 2
P X <χ α = =P X > χ α
n−1 ,
2
2 n−1 , 1−
2

The test procedure is given as:

One-Sample χ 2 Test for The Variance of A Normal Distribution, Two-Sided

We compute the test statistic:

( n−1 )∗s 2
X2=
σ 20
2 2 2 2
If X < χ n−1, α or X > χ n−1, 1− α , then H0 is rejected.
2 2

2 2 2
If χ n−1 , 1− α ≤ X ≤ χ n−1 , α , then H0 is accepted.
2 2

2 2 2 2
The p-value depends whether s <σ 0 or s >σ 0.
p-Value for a One-Sample χ2 Test for the
Variance of a Normal Distribution, Two-Sided

2 ( n−1 )∗s 2
Let the test statistic X = .
σ 20

If s ≤ σ 0, then p−value=2∗P ( X < χ n−1) .

2 2 2 2

If s >σ 0, then p−value=2∗P ( X > χ n−1) .

2 2 2 2

If the assumption of normality is not satisfied for

the hypothesis testing and CI estimations of
variances, then the critical regions, p-values and
the confidence limits will not be valid.

Week 11
Estimation – III (Discrete: Binomial and Poisson)
Estimation for the Binomial Distribution

Point estimation

Remember: The expected value E(X) and the variance Var(X) of a binomial random variable were:
E(X) = np and Var(X) = npq, respectively.

The estimator for the population parameter p of a binomial distribution is:

n
1 X
^p= ∑ X i=
n i=1 n
Similar to previous discussion:

√
2
E ( ^p )=E ( X i ) ≡ μ= p ,∧¿Var ( ^p )= σ = pq ,∧se ( ^p )= pq
n n n

Lastly, the exact standard error se ( ^p )=

√ pq is estimated by
n √ ^p q^
n^
.

Assume that the normal approximation to the binomial distribution holds.

Then, the number of events X observed out of n cases will be approximately normally distributed
X
with mean np and variance npq. Moreover, the proportion of events ^p= is normally distributed
n
pq
with mean p and variance .
n
This is the case when the sample size n is large enough that the central limit theorem holds. Then:
 ^p N p ,( pq
n )
Also, since the number of successes in n Bernoulli trials X =n ^p , if above equation is multiplied by n:

X N ( np , npq )
This assumption holds only if:

npq ≥ 5
If the binomial parameter p is unknown, it can be estimated by the sample proportion of events ^p,
and q can be estimated as 1− ^p =^q . Thus, the condition for the assumption to hold can be
approximated as,

n^
pq ≥5
Interval Estimation

Normal – Theory Method for Obtaining a CI for Binomial Parameter p (Wald Method)

An approximate 100% * (1 – α) CI for the Binomial parameter p based on the normal approximation
to the binomial distribution is given by:

^p ± z
1−
α
2 √
∗
^p q^
n
This method is valid only whenn ^p q^ ≥ 5.

A more accurate CI can be computed, especially when normal approximation is not valid, i.e. npq ≥ 5
or n ^p q^ ≥ 5.

Exact Method for Obtaining CI for the Binomial Parameter p (Clopper – Pearson Method)

An exact 100% * (1 – α) CI for the binomial parameter p is given by (p1, p2), where:
n x
α
() α
P ( X ≥ x| p= p1 )= =∑ n ∗p1k∗( 1− p1 ) P ( X ≤ x| p=p 2 )= =∑ n ∗pk2∗( 1− p2 )
2 k=x k
n−k

2 k=0 k
n−k
()
This method is always valid .

x
Computing the expressions such as ∑
k=0
(nk)∗p ∗(1− p )
k
2 2
n−k
is cumbersome. Hence, computer

programs are employed.

Also, since the exact confidence limits are accurate to a fixed number of decimal points, we cannot
exactly satisfy the above equation. Thus, we use a more conservative approach, such that:

α α
max ( p1 ) for P ( X ≥ x| p= p 1) ≤ ,∧¿min ( p2 ) for P ( X ≤ x| p= p2 ) ≤
2 2
One Sided Confidence Intervals

In some cases, we are only interest in one bound of the interval.

For example, if we want to know whether a new drug significantly increases the life expectancy of
cancer patients compared 30% increase by a generic drug, we would only test if the 30% is below the
lower bound. If we could show that 30% is indeed below the lower bound, we would be able to
conclude that the new drug significantly increases the life expectancy.

Upper One-Sided 100% * (1 – α) CI for the Binomial Parameter p – Normal-Theory Method

An upper one-sided 100% * (1 – α) CI is in the form of p> p1, such that:

P ( p> p1 )=1−α

If the binomial approximation holds true, this CI is given approximately by:

p> ^p −z1−α∗
√ ^p∗q^
n
Upper One-Sided 100% * (1 – α) CI for the Binomial Parameter p – Normal-Theory Method

A lower one-sided 100% * (1 – α) CI is in the form of p< p2, such that:

P ( p< p2 )=1−α

If the binomial approximation holds true, this CI is given approximately by:

p< ^p + z 1−α∗
√ ^p∗q^
n

These interval estimators should only be used if ^p q^ n ≥ 5.

Notice that z α is used for two-sided intervals where z 1−α is used for one-sided intervals.

Similar methods can be used to obtain one-sided Cis for the mean and variance of a normal
distribution, for the binomial parameter p using exact methods, and for the Poisson expectation μ.

Estimation for the Poisson Distribution

Point estimation

A person – year is a unit of time defined as 1 person being followed for 1 year in a study.

Let’s assume that the number of events X over T person years is Poisson distributed with parameter
μ= λt . An unbiased estimator of λ is given by:
 ^λ= X , where X =number of events person− years.
T T

If λ is the incidence rate of events over T person years, T = number of person – years of follow up, and
we assume a Poisson distribution for the events X over T person – years, then the expected value of X
is given by:

E ( X ) λT
E ( X )=λT , thereforeE ( ^λ )= = =λ
T T
Thus, ^λ is an unbiased estimator of λ .

Interval estimation

Normal Approximation Method for Obtaining a CI for Poisson Parameter λ

If we assume ( observed number of events ) X N ( μ , μ ), then a large-sample 100% * (1 – α) CI for µ is

given by:

x± z α ∗√ x .
1−
2

However, for a Poisson distribution, ln(x) is more closely normally distributed than x, and an
approximate variance of ln(x) is 1/x.

Hence, 100% * (1 – α) CI for ln(µ) is given by:

z α
1−
2
ln ( x ) ± ≡ ( c 1 ,c 2 )
√x
And the corresponding 100 * (1 – α) CI for µ is given by:

[ exp ( c1 ) , exp ( c 2 ) ]

Exact Method for Obtaining a CI for Poisson Parameter λ

An exact 100% * (1 – α) CI for the Poisson parameter λ is given by ( μT , μT )

1 2
, where μ1∧μ2 satisfy the

equations:
∞ x−1 −μ1 k − μ1 k
α e μ1 e μ1
P ( X ≥ x|μ=μ1 )= =∑ ¿ 1−∑
2 k=x k ! k=0 k!
x −μ2 k
α e μ2
P ( X ≤ x|μ=μ2 )= =∑
2 k=0 k !
And x = observed number of events, T = number of person-years of follow-up.
Table 7 in the appendix gives exact CI for 90%, 95%, 98%, 99%, and 99.8% for μ, if observed number
of events (x) ≤ 50.

For small x exact CIs for μ of Poisson distribution is not symmetrical about x.

In some cases, for a rare event which is assumed Poisson distributed, the number of person-years is
unknown/not reported. In these situations, it is still possible to obtain a CI for μ, but not for λ , since

the CI for μ is given as ( μ1 μ 2

,
T T ).

Hypothesis Testing – III (Discrete: Binomial and Poisson)

One-Sample Inference for the Binomial Distribution – Normal
Approximation Method
The test is based on the observed proportion of cases in the sample ^p.

Assume that ^p q^ n ≥ 5, that is the normal approximation holds.

Under H 0:

^p N p0 , ( p0 q 0
n )
If we standardize ^p, subtracting the expected value E( ^p) and dividing by the se of ^p:

^p− p0
z=

√ (p ¿ ¿ 0 q0 )
n
¿

Now, under H 0:

z N ( 0 , 1)

To obtain a better approximation of the binomial by the normal distribution, we apply a continuity
correction.

1
^p − p0 +
2n
if ^p < p0 : z= if ^p ≥ p 0 :

√ (p ¿ ¿ 0 q0 )
n
¿

1
^p − p0−
2n
z=

√ (p ¿ ¿ 0 q0 )
n
¿
This expression can be generalized as:

1
| ^p − p0|− 2
z corr=

√ ( p ¿ ¿ 0 q 0)
n
¿

Thus, z corr is our test statistic.

Procedure: One-Sample Test for a Binomial Proportion – Normal-Theory Method (Two-Sided)

Let the test statistic be:

1
| ^p − p0|− 2 n
z corr=

√ ( p ¿ ¿ 0 q 0)
n
¿

If z corr > z 1− α , reject H 0.

2

If z corr < z 1− α , accept H 0.

2

p−value=2∗[ 1−ϕ ( z corr ) ]

This procedure should only be used if ^p q^ n ≥ 5.

Additionally:

( )
2
1
|^p − p0|− 2n
2 2
x corr = χ1
p0q0
n
With the p-value:

p=( χ 21 > X 2corr )

 2
This p-values obtained with both z corr and x corr are equal.

One-Sample Inference for the Binomial Distribution – Exact Method

If ^p q^ n ≥ 5 is not satisfied, then the normal approximation is not satisfied.

In this case, the p-value can be calculated as the sum of the probabilities of all events that are as
extreme as or more extreme than the sample result obtained.

This calculation depends on the ^p ≤ p 0 or ^p > p0 .

Procedure: Computation of the p-value for the One-Sample Binomial Test – Exact Method (Two-
Sided)

[ ]
x

()
if ^p ≤ p 0 , p=2∗P ( X ≤ x ) =min 2∗∑ n ∗pk0∗( 1− p0 )n−k , 1
k=0 k

[ ]
n

()
if ^p ≥ p 0 , p=2∗P ( X ≥ x ) =min 2∗∑ n ∗pk0∗( 1− p0 ) n−k , 1
k=x k

Procedure: The Power and the Sample Size of Estimates of a Binomial Test

The power of a normal test can be estimated using large sample procedures.

The power for a one-sample binomial test for the hypothesis

H 0 : p= p0 vs . H 1 : p ≠ p 0

For the specific alternative hypothesis

H 1 : p= p1

Is given by:

power=ϕ
[√ (p 0∗q 0
p 1∗q 1
∗
z α ∗| p0 −p 1|∗√ n
2

√ p 0∗q 0 )]
Then, using this power, we can calculate the sample size needed to conduct a two-sided test at the
significance level α and the probability of finding significant results 1 – β (power), as:

( √ )
2
p ∗q
p 0∗q 0∗ z α + z 1−β∗ 1 1
1−
2
p 0∗q 0
n= 2
( p 1−p 0 )
To calculate one-sided version, it is enough to substitute α is substituted for α /2.
One-Sample Inference for the Poisson Distribution – Large Sample Method
Let μ be the expected value for a Poisson random variable. To test the hypothesis

H 0 : μ=μ 0 vs . H 1 : μ ≠ μ 0

(1) Compute x = observed number of events in the study population.

(2) Compute the test statistic:

( )
2
|x−μ0|− 12
2 2 2
X =X corr = χ 1 under H 0
μ0
(3) For a two-sided test at level α , H 0 is rejected if ,
2 2
X corr > χ 1 ,1−α
and H 0 is accepted if ,
2 2
X corr ≤ χ 1 ,1−α
(4) The p-value is given by:
P ( χ 21 > X 2corr )
(5) This test should only be used if μ0 ≥10 .

One-Sample Inference for the Poisson Distribution – Small Sample Method

We well use critical value method to test:

H 0 : μ=μ 0 vs . H 1 : μ ≠ μ 0

The two sided 100% * (1 – α) CI interval for Poisson parameter µ (c1, c2) contains all values µ0 for
which we would accept H0 at confidence level α.

Thus, if c1 ≤ µ ≤ c2 we accept H0, and if c1 > µ or c2 < µ we reject H0.

Table 7 in the Appendix contains exact confidence limits for the Poisson expectation µ.

Procedure: One-Sample Test for Poisson Parameter µ, Small-Sample Method.

Let X be a Poisson random variable with expected value µ.

To test the hypothesis:

H 0 : μ=μ 0 vs . H 1 : μ ≠ μ 0

With a two-sided test at significant level α

(1) Compute x = observed value of X.

(2) Obtain the two-sided 100 * (1 – α) CI for Poisson parameter µ (c1, c2), based on x.
(3) If c1 ≤ µ ≤ c2 we accept H0,
If c1 > µ or c2 < µ we reject H0.
(4) The exact p-value depends on whether µ0 ≤ x or not, and is given by:

( ) ( ( ))
x − μ0 k x−1 k − μ0
e μ0 e μ0
min 2∗∑ ,1 if x< μ 0min 2∗ 1−∑ , 1 if x ≥ μ 0
k=0 k! k=0 k!
Week 12
Midterm II

Week 13
Hypothesis Testing – III (Two-Sample Inference)
In two-sample hypothesis testing the underlying parameters of two populations, with unknown
values, are compared.

In a longitudinal or a follow-up study, the same group of people is followed over time. This
represents a paired-sample design, where each participant is used as their own control.

In a cross-sectional study, the participants in compared groups are evaluated at the same time. This
represents an independent-sample design, where the data points in one sample are unrelated to the
data points in the second sample

Paired t-test (t-test for the difference in means of two paired samples)
Test statistic:

√ [ (∑ (∑ ) ) ]
n n 2
d 2
d− di
, where i
sd i=1 i=1 n=the number of matched pairs
√n n
sd = ,∧¿
n−1
If t> t α ∨t <−t α , reject H 0,
n−1 ,1− n−1 ,1−
2 2

If −t α < t<t α , accept H 0

n−1 ,1− n−1 ,1−
2 2
 p−value=
{ if t <0 , p=2∗P ( t n −1 <t )
if t ≥0 , p=2∗[ 1−P ( t ( n−1) <t ) ]

Two-sided 100% * (1 – α) CI for the true mean difference (Δ) between two
paired samples
t α ∗sd t α ∗s d
n−1 ,1− n−1 ,1−
2 2
d− ,d+
√n √n

Two-sample t-test for Independent Samples

The two-sample t-test to compare the means of two independent samples, µ1 and µ2, has two forms.
We either assume that the underlying variances, σ1 and σ2, are equal or assume that they are
different.

Thus, before performing a two-sample t-test on the means of independent samples we need a test to
decide if the underlying variances are equal or not.
2
2 2 2 2 s1
The best test for hypothesis H 0 :σ 1=σ 2 vs . H 1 :σ 1 ≠ σ 2 is based on the ratio of variances 2. We
s2
2
s1
reject H 0 if the ratio 2 is too small or too large.
s2
2
s1
The variance ratio 2 follows an F distribution, which is a family of distributions indexed by two
s2
parameters termed the numerator degrees of freedom, and the denominator degrees of freedom.

If the sizes of the tested samples are denoted n1 and n2, then the F distribution with numerator df
of n1 – 1 and denominator df of n2 – 1 is denoted F n −1 , n −1. 1 2

The 100 * pth percentile of an F distribution with d1 and d2 degrees of freedom is denoted by
F d ,d , p.
1 2

Thus:

P ( F d ,d < Fd
1 2 1 ,d 2, p )= p
Also, the lower pth percentile of an F d ,d distribution is the reciprocal of the upper percentile of an
1 2

F d , d distribution:
2 1

1
F d ,d , p=
1 2
F d , d ,1− p
2 1

F-test for Equality of Two Variances

2 2 2 2
We want to test H 0 :σ 1=σ 2 vs . H 1 :σ 1 ≠ σ 2, with significance level α.
Compute the test statistic:
2
s1
F= 2
s2

if F > F α ,∨F < F α , reject H 0,

n1−1 , n2−1 , 1− n1−1, n2−1 ,
2 2

if F α ≤ F≤ F α , accept H 0
n1−1 ,n2−1 , n1−1 , n2−1 , 1−
2 2

{
p−value= if F ≥1 , then p=2∗P(F n −1 ,n −1> F)
if F< 1, then p=2∗P(F n −1 ,n −1 < F)
1

1
2

It does not matter, for a two-sided test, which sample size is in the numerator. Since a variance ratio >
1 is usually more convenient, the larger variance is put in the numerator as a common practice.

If, as a result of the F-test, we decide that the underlying variances of two samples are not
significantly different, then we can perform a two-sample t-test for the difference in means of two
samples with equal variances.

Since the population variance σ2 is usually unknown, we will estimate it from the sample variances
2 2
s1∧s 2.
However, since the sample size is an important factor for the variance, the best estimate of the
population variance σ2 is given by a weighted average of the two sample variances, with degrees of
freedom for each sample as the weights.

This is called the pooled estimate of the variance from two independent samples:

2 ( n1 −1 )∗s12+ ( n2−1 )∗s 22

s=
n1+ n2−2

The degrees of freedom for the variance s2 is then:

( n 1−1 ) + ( n2 −1 )=n 1+ n2−2 df

We will use this estimate for the test.

Two-sample t-test for Independent Samples with Equal Variance

We want to test H 0 : μ1=μ2 vs . H 1 : μ1 ≠ μ2, with a significance level of α for two normally distributed
populations where σ2 is assumed to be the same for both:

Compute the test statistic:

x 1−x 2

√
2 2
t= , where ( n1 −1 )∗s1 + ( n2−1 )∗s 2
s∗
√ 1 1
+
n1 n2
s=
n1+ n2−2
if t >t α ,∨t<−t α , thenreject H 0,
n1+n2−2 ,1− n1 +n2−2 ,1−
2 2

if −t α <t <t α ,then accept H 0

n1+n2−2 ,1− n1 +n2−2 ,
2 2

p−value=
{ if t ≤ 0 ,then p=2∗P ( t n +n −1< t )
1

if t >0 , then p=2∗[ 1−P ( t n +n −1< t ) ]

1
2

Two-sided 100% * (1 – α) CI for the true mean difference µ1 – µ2, based on

two independent samples with equal variances

( x 1−x 2−t
n1+n2−2 ,1−
α
2
∗s∗
√ 1 1
+ , x 1−x 2+ t
n1 n2 n +n1
−2,2
1−
α ∗s∗
2
1 1
+
n1 n 2 √ )
Don’t forget that for both the t-test and the CI, s2 is the pooled variance estimate.

If, as a result of the F-test, we conclude that the underlying variances of the two samples are
significantly different, we will perform a two-sample t-test for independent variables with unequal
variances.

Notice that we don’t use the pooling this time because we assume that the variances are different.

Two-Sample t-Test for Independent Samples with Unequal Variances

(Satterthwaite’s Method)
Compute the test statistic:

x 1−x 2
t=

√
2 2
s1 s2
+
n1 n 2

Compute the approximate degrees of freedom d’’, where:

( )
2 2 2
s1 s2
+
'
n1 n2
d= ,∧¿

( ) ( )
2 2 2 2 ''
s1 s2 d =largest intager ≤ d '
n1 n2
+
n1−1 n2 −1
if t >t α ,∨t ←t α , thenreject H 0,
d' ' ,1− d' ' , 1−
2 2

if −t α <t <t α ,then accept H 0

d' ' , 1− d' ' ,1−
2 2
 p−value=
{ if t ≤ 0 ,then p=2∗P ( t d < t )
''

if t >0 , then p=2∗[ 1−P ( t d < t ) ]

''

Two-sided 100% * (1 – α) CI for the true mean difference µ1 – µ2, based on

two independent samples with unequal variances

( √ √ )
2 2 2 2
s1 s2 s1 s2
x 1−x 2−t α∗ + , x 1−x 2 +t α ∗s∗ +
d ,1−
2
n1 n 2
''
d ' ', 1−
2
n1 n2

The sample size and power can be estimated for two-sample inferences.

The Sample Size Needed for Comparing the Means of Two Normally
Distributed Samples of Equal Size Using a Two-Sided Test with Significance
Level α and Power 1 – β

( σ 21 +σ 22)∗( z1− α + z 1−β )2

2 where , ∆=|μ2−μ 1|
n= 2
=sample ¿ each group ,
∆

The Sample Size Needed for Comparing the Means of Two Normally
Distributed Samples of Unequal Size Using a Two-Sided Test with
Significance Level α and Power 1 – β

( )(
2
2 σ2 2

n1 =
σ1+
k
∗ z α + z 1− β
1−
2 ) =sample ¿ the first group ,
2
∆
( k∗σ +σ )∗ z
2
1
2
2
( 1−
α + z 1−β
) =sample ¿ the second group ,
2

2
n2 = 2
∆
n2
where , ∆=|μ2−μ 1|,∧, k= = predicted ratio of thetwo sample sizes
n1

Power for Comparing the Means of Two Normally Distributed Samples

Using A Significance Level α
To test the hypothesis H 0 : μ1=μ2 vs . H 1 : μ1 ≠ μ2, for the specific alternative |μ1−μ 2|=∆ , with
significance level α:
 ( √ )
∆
Power=ϕ −z α +
1−
2 σ σ 22
2
1
+
n 1 n2

It is possible to detect and treat outliers in the data with different methods.

Extreme Studentized Deviate (ESD statistic)

|x i−x|
ESD=max
i , …, n s

The pth percentile for the ESD statistic based on a sample of size n is denoted by ES D n , p .

ES Dn ,1−α =
t n−2 , p∗( n−1 )
√ n∗(n−2+t 2
n−2, p )
, where p=1− [ ]
α
2n

ESD Single Outlier Procedure

Suppose we have sample x 1 , x 2 , x 3 , … , x n N ( μ , σ 2) .

We suspect there are outliers in the sample. To test the hypothesis H0: that no outliers are present vs.
H1: that a single outlier is present, with significance level α,

(1) Compute the ESD statistic ESD=max

|x i−x|. The sample value x such that ESD=
|x i−x|
i
i , …, n s s
(n )
is denoted x .
(2) .
a. Refer to table 9 in the appendix and obtain the critical value ES D n ,1−α .
b. If n is not in the table, use the formula:

ES Dn ,1−α =
t n−2 , p∗( n−1 )
√ n∗(n−2+t 2
n−2, p )
, where p=1− [ ]
α
2n
(n )
(3) If ESD > ES D n ,1−α ,then we reject H 0∧x is an outlier .
If ESD ≤ ES Dn ,1−α , thenwe accept H 0∧there are no outliers .

In some instances, the standard deviation can get inflated in the presence of multiple outliers. This
reduces the magnitude of the ESD statistic, and it gets harder to identify outliers by this method.

This is called masking.

To overcome this problem, we need a more flexible procedure.

We first need to determine a reasonable upper limit for the number of outliers. This can be given as:

min
([ ] )
n
10
, 5 , where
n
10 [ ]
isthe largets integer ≤
n
10
If there are more than 5 outliers in the dataset, then the underlying distribution is likely to be
nonnormal (given that the sample size is not very large).

ESD Many-Outlier Procedure

Suppose we have sample x 1 , x 2 , x 3 , … , x n N ( μ , σ 2) .

We suspect there may be as many as k outliers in the sample, where:

k =min
([ ] )
n
10
, 5 , where
n
10 [ ]
is the largets integer ≤
n
10

To test the hypothesis H0: that no outliers are present vs. H1: that there are 1 to k outliers, with
significance level α,

(1) Compute the ESD statistic ESD=max

|x i−x|. The sample value x such that ESD=
|x i−x|
i
i , …, n s s
is denoted x (n ).
(2) We remove x (n ) from the sample, and compute the mean, standard deviation, and the ESD
statistic from the remaining n – 1 data points. We denote the ESD statistic from the reduced
sample by ES D (n −1 ).
(3) W repeat these steps by removing the most outlying data point at each iteration, until we
computed k ESD statistics denoted by ES D (n ) , ES D ( n−1 ) , … , ES D(n −k+1) , based on the
original sample size n.
At this point we successively reduced the sample size n−1 ,… ,n−k +1; and identified the
most outlying values as x (n ) , x (n−1 ) , … , x( n−k+1 ).
(4) The corresponding critical values for the ESD statistics are
ES Dn ,1−α , ES D n−1, 1−α , … , ES Dn−k+1 , 1−α .
(5) Then, we use the following decision rule to identify the outliers.
( n−k +1) ( n) ( n−1) ( n−k +1)
If ES D > ES Dn−k +1 ,1−α , then we declare the k values x , x ,… , x , as outliers
.
( n−k+2 ) (n ) ( n−1 ) ( n−k+ 2)
else If ES D > ES Dn −k+2 ,1−α ,then we declare the k−1 values x , x ,…, x , as outliers
.
.
.
.
( n) (n )
else If ES D > ES Dn ,1−α ,then we declare x as the single outlier .
( n)
else If ES D ≤ ES Dn , 1− α , thenwe tdeclare there are outlier s.
Thus, we can declare 0, 1, …, k sample points as outliers.

(6) We should use table 9 from the appendix to implement this procedure only if n ≥ 20.

Testing for Normality

It is possible to test if two data sets are coming from the same distribution.
Aside from other more advanced tests (which will be covered in advanced statistics class), quantile-
quantile plot (Q-Q plot) technique can be used to compare the distributions of two data sets.

For example, we can place the quantiles of our data on the y-axis, and the quantiles from a standard
normal distribution on the x-axis of the Q-Q plot.

Week 14 – Nonparametric Tests

The Sign Test
With sign test we can statistically test if one sample of ordinal data is greater than, equal to, or less
than another one. We compare medians instead of means.

Let’s say a group of people asked to compare two items, A and B.

They can say A is better, B is better, or they are both equally good.

Let’s assume this can be expressed with numbers. For ith person, xi is how good the item A is, yi is how
good the item B is, and di = xi – yi.

We want to test H0: Δ = 0 vs. H1: Δ ≠ 0, where Δ is the median, or the 50th percentile of the underlying
distribution of di.

If Δi > 0, xi > yi, and A is better;

If Δi < 0, xi < yi, and B is better; and

If Δi = 0, xi = yi, both are equally good.

We exclude the people with di = 0 because in this case both items are equally good.

The test will be based on the number of people C for whom di > 0 out of people with nonzero di.

If C is large, then A is better; and

If C is small, then B is better.

Thus, under null hypothesis H0: Δ = 0, we would expect that P(nonzero di > 0) = ½. Because in that
case C would be equal to its counterpart where nonzero di < 0.

Normal Method
If npq ≥ 5,

n∗1
∗1
or in that case if 2 , and when rearranged,
≥5
2
if n ≥ 20, where n is the number of nonzero d is
Then normal approximation to binomial distribution can be assumed.

The Sign Test with Normal Approximation Procedure

To test the hypothesis H 0 :∆=0 vs . H 1 :∆ ≠ 0 at significance level of α, where the number of

nonzero di = n ≥ 20 and C = the number of nonzero di where di > 0, if

C> c 2=
n+1
2 1−
2
n
4√
+ z α ∗ ∨C< c1=
n−1
2
−z α ∗
1−
2
n
4 √
Then H0 is rejected. Otherwise, H0 is accepted.

To calculate the p-value

[ ( √ )]
p=2∗ 1−ϕ
C− ( n+12 )
n
4
if C>
n
2

(√ ) ( n−1
2 )
C−
n
p=2∗ϕ if C <
n 2
4

n
p=1.0 if C=
2
Alternatively:

[
p=2∗ 1−ϕ (|C−D√n|−1 )]if C ≠ D∧ p=1.0 if C=D ,
where C=the number of di > 0∧D=thenumber of d i <0.

Or another alternative:

2 (|C−D|−1 ) 2
Let X = χ 21 under H 0 .
n
Then the two-sided p-value is given as:

p=P ( χ 21> X 2 )

The test is called sign test because we only consider the signs, regardless of the relative magnitude.

Exact method
If n < 20, then exact binomial probabilities must be used to calculate a p-value.
Procedure for Calculation of Exact p-values for Sign Test

( )( ) ( )( )
n C
n 1 n n 1 n n
If C> , p=2∗∑ n If C< , p=2∗∑ n If C= , p=1.0
2 k=C k 2 2 k=0 k 2 2

Wilcoxon Signed-Rank Test

Let’s consider the same comparison of items A and B, but this time we ask people to score them.

They can give a score of 1-10 based on how good an item they think.

In such cases, ordinary data may have numerical values, but these values just represent the relative
rank instead of an actual magnitude.

In this case, we will use Wilcoxon Signed-Rank Test, which is a nonparametric analogous to the paired
two-sample t-test.

Ranking Procedure for Wilcoxon Signed-Rank Test

1. Arrange the differences di in order of absolute value.

2. Count the number of differences with same absolute value.
3. Ignore the observations with di = 0, and rank the remaining observations from 1 for the
observation with the lowest absolute value and n for the observation with the highest
absolute value.
4. If there is a group of several observations with the same absolute value, then find the lowest
rank in the rank in the range = 1 + R and the highest rank in the range G + R, where R =
highest rank used prior to considering this group and G = the number of differences in the -
range of ranks for this group. Assign the average rank = (lowest rank in the range + highest
rank in the range) / 2, as the rank of each difference in the group.

The test is based on the sum of the ranks, or rank sum (R1), for the group of people with positive di.

If the null hypothesis is true, then the expected value and the variance of the rank sum (when there
are no ties) is given by:

n∗( n+1 ) n∗( n+1 )∗( 2 n+1 )

E ( R1 ) = , Var ( R 1) =
4 24
When n is the number of nonzero di.

Normal Method
If the number of nonzero di ≥ 16, then the normal approximation can be used for sampling
distribution R1.
Procedure for Wilcoxon Signed-Rank Test (Normal Approximation Method for Two-Sided Level α
Test)

1. Rank the differences.

2. Compute the rank sum R1 of the positive differences.
3. –
n∗( n+1 )
a. If R1 ≠ and there are no ties, then
4

T=
[|
R 1−
n∗( n+1 ) 1
4
− | ]
2

√ n∗( n+1 )∗( 2n+1 )

24
n∗( n+1 )
b. If R1 ≠ and there are ties, then
4

T=
[|
R 1−
n∗( n+1 ) 1
4
− | ]
2

√
n∗( n+1 )∗( 2n+1 )
g 3
t i −t i
24−∑
i=1 48

Where ti is the number of observations within the same tied group and g is the
number of tied groups.
n∗( n+1 )
c. If R1= , then T = 0
4
4. If T > z1− α ,then reject H 0 , otherwise accept H 0 .
2

5. The p value of the test is fiven by p=2∗[ 1−ϕ ( T ) ]

6. This test should only be used if the number of nonzero di ≥ 16 and if the difference scores
have an underlying continuous symmetric distribution.

Alternatively, the variance can be calculated as:

n
1
Var ( R1 )= ∗∑ r 2j
4 j=1
Where rj = rank of the rank of jth observation and the sum is over all observations.

This formula is easier to compute and valid for both in the absence or presence of ties.

The resulting test statistic and p-value will be the same, if the negative values were used for
computation instead of the positive values. Thus, the procedure can be applied arbitrarily on the
positive or negative differences (preferably on the ones with lower numbers).
Exact Method
If the number of nonzero di < 16, then probability tables such as table 10 in the appendix, must be
used.

The Wilcoxon Rank-Sum Test

The nonparametric alternative for the independent two-sample t-test is the Wilcoxon rank-sum test.

Ranking Procedure for The Wilcoxon Rank-Sum Test

1. Combine the data from two groups, and order the values from lowest to highest.
2. Assign ranks to the individual values.
3. If a group has of observations have the same value, then compute the range of ranks for the
group and assign the average rank for each observation in the group.

The test statistic is the sum of the ranks in the first sample (R1). If this sum is large, then the first
sample has higher scores than the other one; and if this sum is small, then the first sample has lower
scores than the other one.

If the number of observations in two groups are n1 and n2, the expected rank-sum in the first group,
E(R1) = n1 * (n1 + n2 + 1)/2, and variance Var(R1) = n1*n2*(n1 + n2 + 1)/12.

Furthermore, the smaller of the two groups must be of size at least 10, and the variable under study
must have an underlying continuous distribution.

Under these assumptions, the sampling distribution of the rank-sum R1 is approximately normal.

Normal Method
Procedure for Wilcoxon Rank-Sum Test (Normal Method)

1. Rank the observations.

2. Compute the rank-sum R1 in the first sample.
3. –
n1∗( n1 +n2 +1 )
a. If R1 ≠ ∧there are no ties,
2

T=
[|
R 1−
n1∗( n1 +n2 +1 ) 1
2
− | ]
2

(√ n 12∗n )∗( n + n +1)

1 2
1 2

n1∗ ( n1 +n2 +1 )
b. If R1 ≠ ∧there are ties,
2
 T=
[| R1−
2
−| ]
n1∗( n1 +n2 +1 ) 1
2

√( [ ]
g

∑ t2i −ti
n1∗n 2
12 )
∗ n1+ n2 +1− i=1

( 1 2 )∗( n1 +n 2−1 )
n +n
Where ti is the number of observations in the same tied group, and g is the number
of tied groups.
n1∗( n1 +n2 +1 )
c. If R1= then T =0
2
4. If T > z1− α thenreject H 0 . Otherwise accept H 0 .
2

5. p−value=2∗[ 1−ϕ ( T ) ]
6. This should be used if both n1 and n2 are at least 10, and if there is an underlying continuous
distribution.

Alternatively, the variance can be computed as:

( n1∗n2 N
) ( )
2
N +1
Var ( R1 )= ∗∑ r i−
N i=1 2
Where N = n1 + n2 and ri = rank of the ith observation in the combined sample of size of N.

This alternative method is valid in both conditions whether there are ties or not.

Exact Method
If one of the sample sizes is smaller than 10 then exact methods must be used. Table 11 in appendix is
used for this. Under the assumption of there are no ties.