HSO 402
Computational Psychology
Statistical Inference in Simple Models
Key takeaway:
Statistical inference is equivalent to asking data to
answer questions
Look at the following questions
• Do students perform better with self-study or classroom
teaching?
• How educated are people in this city?
• Does income depend on level of education?
• Do students perform better with self-study or classroom
teaching?
• Simple answer: YES / NO
• Statistical Question:
Is the mean score of students better with self-study or with
classroom teaching?
• Statistical Method : Hypothesis Test
• How educated are people in this city?
• Simple Answer: Most / some / very few etc
• Statistical Question:
What is the mean literacy of the population in this city?
What is the percentage of K-12 pass outs to the population?
• Statistical Method: Estimation
• Does income depend on level of education?
• Simple Answer: LIKELY / DEPENDS / MAYBE
• Statistical Question:
Is a regression model of income with education a better fit than
one without it?
• Statistical Method: Model Comparison
Suppose that 20 students were eligible for a travel-support grant that
had a single award of $1000 for attending an academic conference
or study program. How much would individuals be willing to pay for a
1-in-20 or 5% chance of receiving the $1000 grant as application
fee?
The expected value of applying for this grant in this particular
situation would be $1000 / 20 students = $50. We might think that
students will focus on the size of the grant (i.e. $1000) and thus be
willing to pay more than the expected value (i.e. $50).
But one might also think that students would focus on the chance of
not receiving the grant and thus be willing to pay less than the $50
expected value of applying.
Responses collected from 20 Students -
“How much are they willing to pay for this application?”
PRE (Proportional Reduction in Error)
PRE (Proportional Reduction in Error)
Is this PRE of 20.9% good enough to reject Model C
in favour of Model A ?
Sum of Squares of Residuals (SSR)
Sampling Distribution of the PRE
• Even if model C is true, i.e. real PRE = 0, we are likely to see a
small reduction in SSE (A) compared to SSE (C)
• PRE for Model A can never be less than 0
• Therefore, PRE is not expected to be 0, even if Model A makes
no improvement over Model C
• Criteria of PRE being equal or not equal to 0 cannot be used
Sampling Distribution of the PRE
• Random fluctuation is possible in PRE just as it was in the
estimated mean
• Sampling distribution helps determine if the fluctuation can be
considered significant or not
• Sampling distribution of the PRE tell us if the obtained PRE is
likely (by chance) value or is it unexpected
Key takeaway:
If the obtained PRE, assuming Model C is
correct, is within our expectations then
we don’t reject Model C
Sampling Distribution of the PRE
• It is tedious to develop sampling distribution through simulation (or real
data collection!)
• Statistician’s have developed sampling distribution of PRE using known
assumptions about errors:
• Normally distributed
• Independent and identically distributed (i.i.d)
• Mean = 0, SD = σ
• Sampling distribution of the PRE tell us if the obtained PRE is likely (by
chance) value or is it unexpected
Interpretation
• If Model C ($ 50) is correct, then we are likely to obtain this PRE
(.209) less than 4% of the times
• Traditionally accepted cut-off is 5% or less (p-value)
• Therefore, we can say reject Model C and reasonably say
students are not likely to pay $50 as application fee
Critical Value: The cutoff point on a test statistic's distribution that marks
where we reject the null hypothesis.
obtained PRE (.209) > critical value (.187)
F Test
• PRE is not usually reported in books / analysis
• F Statistic is more common
• Advantages:
• Examines PRE per additional parameter
• PRE that was reduced to the PRE that remains (i.e. 1 – PRE)
PC = number of parameters of the compact model
PA = number of parameters of the augmented model
n = total number of observations (max number of parameters that can be added)
For our example, PRE = .209, n = 20, PA = 1, PC = 0
For our example, PRE = .209, n = 20, PA = 1, PC = 0
Has the F value reached the critical threshold?
Common version of F reported
MSR = mean squares reduced
MSE = mean square error