Confidence Intervals and Estimation Methods
Confidence Intervals and Estimation Methods
Define an "estimator" and "estimate" and list the characteristics of a good estimator.
00 Describe the primary methods of point estimation (MoM, MLE, Least Squares).
f
It can be large or small, as long as it includes *all* the items e
I I I e
f
you are studying. I
f
Examples: I I •
I
f Sample
• All voting-age citizens in a country. I
• Size: N
• Size: n
Fixed and
This is the fundamental idea that connects everything: Population Para,eaeter Unknown
3. We calculate a Statistic from our sample (the Statistic, x The above is a diagram of the process of the process of using data to obtain information about a
population. Please answer the following:
). a) What is the difference between populations and samples?
b) What is the difference between parameters and statistics? Explain using an example.
c) How can we obtain representative samples from a population?
4. We use the Statistic to make an educated guess, or d) Describe why parameters are "fixed and unknown" and why statistics are "random
variables"
*inference*, about the unknown Parameter. e) How do we estimate parameters using statistics? What are the things that we need to
take into consideration when estimating parameters using statistics?
Estimator vs. Estimate
Estimator vs. Estimate: Visua
0 2 4 8 10 12
Estimator (recipe) will produce a new Estimate (a new cake).
• •
•
•
•
II
....- Sufficiency:Does it use *all* the information available in the sample?
•
1. Un iase ness
4
I "
f
true population parameter 8.
BIAS - when our statistic
does not accurately estimate
"
our parameter
E(8) =8
In English: The estimator is "correct on average." It doesn't (•) H~h blD&; lowvarta~llliy (b} Low bla&; high vanabl'fty
•
parameter.
-
Example: The sample mean X is an unbiased estimator for
the population mean µ.
•
Visua izin iase ness
s2 =
2. Consistenc
In English: As your sample size gets larger and larger, your - sa r1 piing
0.15 a istrlbutlon of
estimate gets closer and closer to the true value. mean uno er v-alt,
n = 100
0.05
More data = More accuracy. This is a very desirable
property. -0 .. 0~ - -6-5-4-3-2-1 1 2 3 4 S 6
• •
3.E icrenc
60
•
An estimator is efficient if it has the smallest possible 50
•
variance among all unbiased estimators. This is called the • •
Minimum Variance Unbiased Estimator (MVUE).
40
• •• •
30
"
Var ( 81
"
< Var ( 82 •
) ) ••
20
• • •
In English: If we have two unbiased estimators (both •
10
However, the variance of the sample mean is smaller This is a classic trade-off: Efficiencyvs. Robustness.
than the variance of the sample median.
9 A statistic is sufficient if it uses *all* the information in the sample about the parameter .
•
f@ In English:Once you have calculated the sufficient statistic, the original sample data provides no
*additional* information about the parameter.
Example: For a Bernoulli population (coin flips), the number of successes ( k) is a sufficient statistic for
the probability of success ( p). You don't need to know the *order* of the flips (e.g., H, T, H vs. H, H, T) to
estimate p. The total number of heads is "sufficient".
• •
•
•
~ number line with a single red dot labeled 'Point Estimate', showing its precision
but isolation
Example:
O Pros:
Simple, precise, and easy to report.
Cons:
It is virtually guaranteed to be wrong.
~ number line showing a 'Point Estimate' in the center and a 'Margin of Error' bar
extending on both sides, labeled 'Interval Estimate'
Example:
A wide interval tells us we are not very certain; a narrow interval tells us we are very precise.
Cons:
More complex to calculate.
Method of Moments (MoM): A simple, old method that equates sample moments to population
moments.
J!::
••••
Maximum Likelihood Method (MLE): The most popular and powerful method .
VX Method of Least Squares (OLS): The method used to find the best-fit line in regression.
1. Population 2. Sample
Moment Moment 3. Equate and Solve
The first population The first sample Set them equal to find the estimator for µ:
moment is the moment is the -
Easy to calculate.
Cons:
Often *not* the most efficient estimator (can be high variance).
Can produce estimates that are nonsensical (e.g., a negative estimate for a variance).
This is the most common and important method, Likelihood and Log-Likelihood Function of Poisson Function
0.12 0
I
I 20
I
I
I
I
0.06 I
"Given the data we *observed*, what value of the I
I
I
I
I 30
parameter e 0.04
I
I
I
I
I
I
I
40
0.02;
It finds the parameter value that *maximizes* the
o-l----~---===:::::::;:.._---~--~---~---~----=i.50
LikelihoodFunction. 0 0.5 1 1.5 2 2.5 3 3.5
1. Write the LikelihoodFunction L ( 8 ) : This is the joint probability of observing your data, given a
parameter 8. (Often a large product).
2. Write the Log-LikelihoodFunction 1 ( 8): Take the natural log ( ln) of L. This turns products into
sums, which are *much* easier to work with.
3. Differentiate and Solve: Take the derivative of 1 ( e) with respect to 8, set the derivative to zero,
and solve for 8. This finds the maximum (the peak) of the function.
LE: Exam
O Pros:
Generally the "best" estimator.
Invariance Property: The MLE of a function of e is just the function of the MLE of 0. (e.g., MLE of µ2
.IS µ"'2) .
Cons:
Can be mathematically *very* difficult to compute.
"residuals").
,... .
c:
...u
:,
.... 2 -1 •• • •
.·~· . .... • •• • ••
VI
< 25 ••
-2 • •
•
• •
point and the fitted line. 1s=•~~~~~~~~~~~~~~~
20 30 35 40 45 50 20 25 30 40 45
• 50
25 35
n Note that we've colored in a few dots in orange so you can get the sense of how this transformation works.
" 2
Minimize: I: ( Y.1 - Y.1 )
i = 1
O Pros:
The foundation of regression analysis.
Under certain assumptions (Gauss-Markov), OLS estimators are the "BLUE" (Best Linear Unbiased
Estimators).
Cons:
Can be highly sensitive to outliers (since errors are *squared*, big errors have a huge effect).
•••
•••
•••
Min. Chi-Square
~ number line showing a 'Point Estimate' in the center and a 'Margin of Error' bar
extending on both sides, labeled 'Interval Estimate'
CORRECT:
95%
•
"We are 95% confident that this *method* works. If we were I
~ maze
The confidence is in the process, not in any single interval.
RONGInter retation
O WRONG: "There is a 95% probability that the true population meanµ is in our interval [172, 178]."
to
• Why is this wrong? Because the true parameterµ is a *fixed, unknown constant*. It is not a random
variable. It doesn't have a probability of being anywhere. It just "is",
O Either our specific interval [172, 178] *did* captureµ (100% chance) or it *did not* (0% chance). We just
don't know which. The 95% refers to the "success rate" of the method in the long run.
• •
•
•
•
0 Scenario: The "textbook" case. We somehow know the true population variance o2. (Very rare in real
life).
4 Formula (z-interval):
-
x +
- Za I 2 ( Jri )
For a 95% Cl, Za I 2 = 1.96.
The "Critical Value" for a z-interval comes from the standard
normal (Z) distribution.
The z-score that cuts off the top 2.5% of the data is 1.96.
This is our critical value, za 12.
ean Variance a2 Un nown
~Graph comparing the standard normal 'Z-distribution' (blue line) to the 't-
distribution' (red dashed line) which is slightly flatter with fatter tails
4 Formula Ct-interval):
-
x +
- ta I 2 , n - 1 { Jn )
The critical value t depends on the confidence level AND the sample size ( n). As n gets large (e.g., > 100),
the t-distribution becomes almost identical to the z-distribution.
ean: Exam
Problem Solution
We sample 10 students and find their average height 1. df = n - 1 = 9
is x = 175 cm, with a sample std. dev. of s = 5 cm. Find 2. Critical t-value (from t-table for 95%, 9 df) =
the 95% Cl for the true mean height. to.02s, 9 = 2.262
3. Standard Error = s I ...;n = 5 I -!ID= 1.581
4. Margin of Error = 2.262 x 1.581 = 3.576
5. Interval: 175 ± 3.576
[171.4, 178.6]
CI or Variance a2
~
II)
c di= 2
Q) ('!
O O
To estimate the variance a2, we use the sample variance s2. ~ L------=~==:::::;:::====::;=J
0 5 10 15
0
0
~
0
.
di: 4
0
q
This distribution is not symmetric (it is skewed to the 0
0 5 10 15
(I) di: 6
.iii q
c O
Q)
0 ....
q
0
0
0
0
0 5 10 15
Chi Square
CI or Variance Formu a
4 Formula:
( n - 1) s2 ( n - 1 ) s2
2
Xa I 2, n - 1
' x21-a/2 n-1
'
Note the "flip": the larger critical value ( x2 ) goes in the denominator for the *lower* bound.
a/2
• I
•
Comparing µ1 and µ2
CI
..
:.:. Goal: We want to estimate the *difference* between the means of two independent groups (e.g.,
"Treatment" vs. "Control").
-
Point Estimate: ( X1 - •
x
2
)
Interpretation: If the 95% Cl for the difference contains zero (e.g., [-5, +10]), then we are NOT 95%
confident that there is any real difference between the means!
Paired(Dependent) Samples
IndependentSamples
The two groups are related.
The two groups are totally unrelated.
Example: Patient's blood pressure *before* and *after* a
Example: 50 men vs. 50 women.
drug.
I - t wf!h 21 <Mgf"5 0, l'rNQOm
100 *
i
75
'
(I)
"'c0
c. 50
"'
(I)
0::
""9ct lhe ~POhli& of «.AT Retitct the ~[Link] d equal
meeN' ttletest~ h leloe,, "*"" It ths ~ Nllmtic • iarp
25
"""''*'
•
~
0
0 1 3 4
Before After
CI a
0 Scenario: Independent samples. We know both population variances, af and a~. (Again, rare) .
4 Formula:
+
-
CI a
40
30
Scenario: Independent samples. We don't know the
variances, but we assume they are equal ( of = o~ ). ......
c
Q)
~
Q)
20
We calculate a "pooled" sample variance, s~, which is a Cl..
Comparing ai and a~
30
Scenario: Independent samples. We don't know the
variances, but we assume they are equal { ifi = ~ ).
i,
<.:>
;;; 20
We calculate a "pooled" sample variance, s~, which is a a.
Distribution
: The ratio of two sample variances follows an
F-distribution
, which is also skewed and defined by *two*
sets of degrees of freedom. F10,2s
(a) (b)
Interpretation
: If the 95% Cl for the ratio contains 1.0, we
are not confident that the variances are truly different.
.
Unbiased: Correct on average. E ( 0 ) = 0
~ Consistent
: Gets better as sample size n increases.
-
... , Sufficient
: Captures all relevant information from the sample.
CI for of I 02
CI for af I oz
• - ... .. . .. ,
Review: Part 3 & 4 - Methods
X 95o/o Confident: Refers to the long-run success rate of the "method", not the probability for one
interval.
1,,,,,,,,,,,,,,,,,,,1 I ?
' ' A
I
'
•
••
••• I
"
c
D
E
i ./
./
-
-,'.\_
"
c
D
<
••••
••••• l- I w In I ro I I" I"!" I"' I ro J" I"-
0'
1,, ,111111•••••
<Q
.1
1111,,1,,,1 t :,
,/
• •
•
••
• • •
•I •I • •I • I • • I
•••••
••••••••••
•••••••••• ·~·,y --
7 ,/
![Link]..il
D \\'1111.:;·U
-
=i Fl111 otQ -
b11·11(1,,.'\\0,.....·r~1h
I I I I I I LI I I I I I I I I
• f-+-+-1--1-1 ••
12
I " !./ e
f-+-+-+--1-1 • !•
••• I
c. ./ ./ c
0
./.
•••
•••• •
••••• ro] ll~lo I
••••• 1, ,,1,,,1 (d ,
1,,, 1,,,,,,, 3 t :, 2 3 4
' •
• •
• ••••• ./
./ ./ -
• • • •
• •••• •• •
l111l111l111l111l111I
••••••••••
••••••••••
~ ~
D \\'11!1(0:U 0•1111<'\<U -0.1!lt.'ccdl
- [x1m.°tCdf1,.y..·rMh - -
=i~~~ . :;:
S:r~httlO'""l':,:h
t. . [Link]
I •';tr
1i
ti. I t'
i t t Sample Source: [Link]
iilllillillllli [Link]
VS iiiiitiiitiHH
Population
Mean Source: [Link]
- ·- [Link]
- ...y--E::::-:::J=
_____ _ __ --,-· -.
___ .
.......... ...,....
. ..... . -----·---
'! :"''"' !"":--:-~
Source: [Link]
[Link] [Link]
Source: [Link]
[Link]
On4CLCOmJmjzt18VcRzV9PMyvtgxo7xlQ
lol .... _ .. ........, (» ... - .... -
Source: [Link]
[Link]
Source: [Link]
Ima e Sources
[Link]
Source: [Link]
\
[Link] files/figure-html/[Link]
-:• • • ,• .. ,'I·.· ~.·
.: ·~
. ...---·-__p- ••
. /.\. . . . . Source: [Link]
I . .
,1 -..1 ... _ .. _ ...... _ ... _. [Link] [Link] m/vi/h5yrl Dk nCdA/hqdefault.j pg?sqp=-oaymwEm COA DEOgC8quK qQMa8A EB-AH U BoAC4AOKAgwlA BA BGG U gZS hi MA 8 =& rs =AOn4CLCMzYe TOrW 013 Rl b TQx
'
' lgYXdOZ5GA
••
-, .
.. ~ • (l,,I
Source: [Link]
[ 0, 3)
Source: [Link]
[Link]
Source: [Link]
..,.. ..... . .
···~·
..
.c":"~·
,
•
.. .,
. . . . . .. . ,,.,·.. .. . . .
•• •!
.
.
I . . • -· •' ..·~
..
. . .
• '.... ,
.. . .
[Link] ml m/assets Is uppo rt/wp-co ntent/ uploads /2017 /07 /Screen-Shot- 2017-07-19-at-9.46.11-A M. png
I'".·.:.. . . ...
\.
;
. .. . -· ..
•I • • Source: [Link]
Ima e Sources
[Link]
...
~
-- - ....... -. Source: [Link]
[Link]
FCritical_ Values_Ol.png&w=3840&q=75
Source: [Link]
;,:
•
§
,,,
N
l ·-·-
o..
,_....,_
......
ca-w ..
-
-nii-
[Link]
ww,....... '
Source: [Link]
[Link]
J•
a • •••
Source: [Link]
[Link]
Source: [Link]
•! " [Link]
8. 50
i
" Source: [Link]
Ima e Sources
' [Link]
; .
•
Source: [Link]
\ [Link]
•••••
1, 11111,•••••
111 ,1, 111111 I [Link]
•
• •• ••• ••• •• • ••
1 , II I II I I II " I II 111 I I Source: [Link]
[Link]
.....
..••••••.• ·~~-
••••••••••
[Link]?width=600&height=600&name=[Link]
Source: [Link]
https:// pub. m dpi-res. com/water /water-17-01273/article _ deploy/htm 1/i m ages /water-17-01273-gOOl. png?1745671093
Source: [Link]
----·