Statistical Inference Review

Gonzalo Mateos
Dept. of ECE and Goergen Institute for Data Science and AI
University of Rochester
gmateosb@[Link]
[Link]

January 23, 2025

Network Science Analytics Statistical Inference Review 1

Statistical inference and models

Statistical inference and models

Point estimates, confidence intervals and hypothesis tests

Tutorial on inference about a mean

Tutorial on linear regression inference

Network Science Analytics Statistical Inference Review 2

Probability and inference

Probability theory

Data-generating process Observed data

Inference and data mining

▶ Probability theory is a formalism to work with uncertainty

▶ Given a data-generating process, what are properties of outcomes?
▶ Statistical inference deals with the inverse problem
▶ Given outcomes, what can we say on the data-generating process?

Network Science Analytics Statistical Inference Review 3

Statistical inference

▶ Statistical inference refers to the process whereby

⇒ Given observations x = [x1 , . . . , xn ]T from X1 , . . . , Xn ∼ F
⇒ We aim to extract information about the distribution F
▶ Ex: Infer a feature of F such as its mean
▶ Ex: Infer the CDF F itself, or the PDF f = F ′
▶ Often observations are of the form (yi , xi ), i = 1, . . . , n
⇒ Y is the response or outcome. X is the predictor or feature
▶ Q: Relationship between the random variables (RVs) Y and X ?
 
▶ Ex: Learn E Y X = x as a function of x
▶ Ex: Foretelling a yet-to-be observed value y∗ from the input X∗ = x∗

Network Science Analytics Statistical Inference Review 4

Models

▶ A statistical model specifies a set F of CDFs to which F may belong

▶ A common parametric model is of the form F = {f (x; θ) : θ ∈ Θ}
▶ Parameter(s) θ are unknown, take values in parameter space Θ
▶ Space Θ has dim(Θ) < ∞, not growing with the sample size n

▶ Ex: Data come from a Gaussian distribution

 
1 −
(x−µ)2
FN = f (x; µ, σ) = √ e 2σ 2
, µ ∈ R, σ > 0
2πσ 2

⇒ A two-parameter model: θ = [µ, σ]T and Θ = R × R+

▶ A nonparametric model has dim(Θ) = ∞, or dim(Θ) grows with n
▶ Ex: FAll = {All CDFs F }

Network Science Analytics Statistical Inference Review 5

Models and inference tasks

▶ Given independent data x = [x1 , . . . , xn ]T from X1 , . . . , Xn ∼ F

⇒ Statistical inference often conducted in the context of a model
Ex: One-dimensional parametric estimation
▶ Suppose observations are Bernoulli distributed with parameter p
▶ The task is to estimate the parameter p (i.e., the mean)

Ex: Two-dimensional parametric estimation

▶ Suppose the PDF f ∈ FN , i.e., data are Gaussian distributed
▶ The problem is to estimate the parameters µ and σ
▶ May only care about µ, and treat σ as a nuisance parameter

Ex: Nonparametric estimation of the CDF

▶ The goal is to estimate F assuming only F ∈ FAll = {All CDFs F }

Network Science Analytics Statistical Inference Review 6

Regression models

▶ Suppose observations are from (Y1 , X1 ), . . . , (Yn , Xn ) ∼ FYX

⇒ Goal is to learn the relationship between the RVs Y and X
▶ A typical approach is to model the regression function
Z ∞
 
r (x) := E Y X = x = yfY |X (y |x)dy
−∞
 
⇒ Equivalent to the regression model Y = r (X ) + ϵ, E ϵ X = 0
▶ Ex: Parametric linear regression model

r ∈ FLin = {r : r (x) = β0 + β1 x}

▶ Ex: Nonparametric regression model, assuming only smoothness

 Z ∞ 
r ∈ FSob = r : (r ′′ (x))2 dx < ∞
−∞

Network Science Analytics Statistical Inference Review 7

Regression, prediction and classification

▶ Given data (y1 , x1 ), . . . , (yn , xn ) from (Y1 , X1 ), . . . , (Yn , Xn ) ∼ FYX

▶ Ex: xi is the blood pressure of subject i, yi how long she lived
 
▶ Model the relationship between Y and X via r (x) = E Y X = x
⇒ Q: What are classical inference tasks in this context?
Ex: Regression or curve fitting
▶ The problem is to estimate the regression function r ∈ F

Ex: Prediction
▶ The goal is to predict Y∗ for a new patient based on their X∗ = x∗
▶ If a regression estimate rˆ is available, can do y∗ := rˆ(x∗ )

Ex: Classification
▶ Suppose RVs Yi are discrete, e.g. live or die encoded as ±1
▶ The prediction problem above is termed classification

Network Science Analytics Statistical Inference Review 8

Fundamental concepts in inference

Statistical inference and models

Point estimates, confidence intervals and hypothesis tests

Tutorial on inference about a mean

Tutorial on linear regression inference

Network Science Analytics Statistical Inference Review 9

Point estimators

▶ Point estimation refers to making a single “best guess” about F

▶ Ex: Estimate the parameter β in a linear regression model
n o
FLin = r : r (x) = β T x

▶ Def: Given data x = [x1 , . . . , xn ]T from X1 , . . . , Xn ∼ F , a point

estimator θ̂ of a parameter θ is some function

θ̂ = g (X1 , . . . , Xn )

⇒ The estimator θ̂ is computed from the data, hence it is a RV

⇒ The distribution of θ̂ is called sampling distribution
▶ The estimate is the specific value for the given data sample x
⇒ May write θ̂n to make explicit reference to the sample size

Network Science Analytics Statistical Inference Review 10

Bias, standard error and mean squared error
h i
▶ Def: The bias of an estimator θ̂ is given by bias(θ̂) := E θ̂ − θ

▶ Def: The standard error is the standard deviation of θ̂

r h i
se = se(θ̂) := var θ̂

⇒ Often, se depends on the unknown F . Can form an estimate se

ˆ
▶ Def: The mean squared error (MSE) is a measure of quality of θ̂
h i
MSE = E (θ̂ − θ)2

▶ Expected values are with respect to the data distribution

n
Y
f (x1 , . . . , xn ; θ) = f (xi ; θ)
i=1

Network Science Analytics Statistical Inference Review 11

The bias-variance decomposition of the MSE

Theorem h i
The MSE = E (θ̂ − θ)2 can be written as
h i
MSE = bias2 (θ̂) + var θ̂

Proof. h i
▶ Let θ̄ = E θ̂ . Then
h i h i
E (θ̂ − θ)2 = E (θ̂ − θ̄ + θ̄ − θ)2
h i h i
= E (θ̂ − θ̄)2 + 2(θ̄ − θ)E θ̂ − θ̄ + (θ̄ − θ)2
h i
= var θ̂ + bias2 (θ̂)
h i h i
▶ The last equality follows since E θ̂ − θ̄ = E θ̂ − θ̄ = 0

Network Science Analytics Statistical Inference Review 12

Desirable properties of point estimators

▶ Q: Desiderata for an estimator θ̂ of the parameter θ?

h i
▶ Def: An estimator is unbiased if bias(θ̂) = 0, i.e., if E θ̂ = θ
⇒ An unbiased estimator is “on target” on average
p
▶ Def: An estimator is consistent if θ̂n → θ, i.e. for any ϵ > 0
 
lim P |θ̂n − θ| < ϵ = 1
n→∞

⇒ A consistent estimator converges to θ as we collect more data

▶ Def: An unbiased estimator is asymptotically Normal if
! Z x
θ̂n − θ 1 2
lim P ≤x = √ e −u /2 du
n→∞ se 2π −∞

⇒ Equivalently, for large enough sample size then θ̂n ∼ N (θ, se2 )

Network Science Analytics Statistical Inference Review 13

Coin tossing example

Ex: Consider tossing the same coin n times and record the outcomes
▶ Model observations as X1 , . . . , Xn ∼ Ber(p). Estimate of p?
▶ A natural choice is the sample mean estimator
n
1X
p̂ = Xi
n
i=1

▶ Recall that for X ∼ Ber(p), then E [X ] = p and var [X ] = p(1 − p)

▶ The estimator p̂ is unbiased since
" n # n
1X 1X
E [p̂] = E Xi = E [Xi ] = p
n n
i=1 i=1

⇒ Also used that the expected value is a linear operator

Network Science Analytics Statistical Inference Review 14

Coin tossing example (continued)

▶ The standard error is

v " # v
n n
u u r
u 1 X u1 X p(1 − p)
se = var
t Xi = t
2
var [Xi ] =
n n n
i=1 i=1

q
⇒ Unknown p. Estimated standard error is se ˆ = p̂(1−n
p̂)

▶ Since p̂n is unbiased, then MSE = E (p̂n − p)2 = p(1−p)

 
n →0
p
▶ Thus p̂ converges in the mean square sense, hence also p̂n → p
▶ Establishes p̂ is a consistent estimator of the parameter p

▶ Also, p̂ is asymptotically Normal by the Central Limit Theorem

Network Science Analytics Statistical Inference Review 15

Confidence intervals

▶ Set estimates specify regions of Θ where θ is likely to lie on

▶ Def: Given i.i.d. data X1 , . . . , Xn ∼ F , a 1 − α confidence interval
of a parameter θ is an interval Cn = (a, b), where a = a(X1 , . . . , Xn )
and b = b(X1 , . . . , Xn ) are functions of the data such that

P (θ ∈ Cn ) ≥ 1 − α, for all θ ∈ Θ

⇒ In words, Cn = (a, b) traps θ with probability 1 − α

⇒ The interval Cn is computed from the data, hence it is random
▶ We call 1 − α the coverage of the confidence interval
▶ Ex: It is common to report 95% confidence intervals, i.e., α = 0.05

Network Science Analytics Statistical Inference Review 16

Aside on the standard Normal distribution

▶ Let X be a standard Normal RV, i.e., X ∼ N (0, 1) with CDF Φ(x)

Z x
1 u2
Φ(x) = P (X ≤ x) = √ e − 2 du
2π −∞

α/2 1−α α/2

−zα/2 zα/2

▶ Define zα/2 = Φ−1 (1 − (α/2)), i.e., the value such that

P X > zα/2 = α/2 and P −zα/2 < X < zα/2 = 1 − α

Network Science Analytics Statistical Inference Review 17

Normal-based confidence intervals

ˆ 2)
▶ Nice point estimators θ̂n are Normal as n → ∞, i.e., θ̂n ∼ N (θ, se
⇒ Useful property in constructing confidence intervals for θ

Theorem
ˆ 2 ) as n → ∞. Let Φ be the CDF of a standard
Suppose that θ̂n ∼ N (θ, se
Normal and define zα/2 = Φ−1 (1 − (α/2)). Consider the interval

Cn = (θ̂n − zα/2 se,

ˆ θ̂n + zα/2 se).
ˆ

Then P (θ ∈ Cn ) → 1 − α, as n → ∞
▶ These intervals only have approximately (large n) correct coverage

Network Science Analytics Statistical Inference Review 18

Proof

Proof.
▶ Consider the normalized (centered and scaled) RV

θ̂n − θ
Xn =
se
ˆ

▶ By assumption Xn → X ∼ N (0, 1) as n → ∞. Hence,

 
P (θ ∈ Cn ) = P θ̂n − zα/2 se
ˆ < θ < θ̂n + zα/2 se
ˆ
!
θ̂n − θ
= P −zα/2 < < zα/2
se
ˆ


→ P −zα/2 < X < zα/2 = 1 − α

▶ The last equality follows by definition of zα/2

Network Science Analytics Statistical Inference Review 19

Coin tossing example (encore)

Ex: Given observations X1 , . . . , Xn ∼ Ber(p). Estimate of p?

▶ We studied properties of the sample mean estimator
n
1X
p̂ = Xi
n
i=1

▶ By the Central Limit Theorem, it follows that

p̂(1 − p̂)
 
p̂ ∼ N p, as n → ∞
n

▶ Therefore, an approximate 1 − α confidence interval for p is

r r !
p̂(1 − p̂) p̂(1 − p̂)
Cn = p̂ − zα/2 , p̂ + zα/2
n n

Network Science Analytics Statistical Inference Review 20

Hypothesis testing

▶ In hypothesis testing we start with some default theory

▶ Ex: The data come from a zero-mean Gaussian distribution
▶ Q: Do the data provide sufficient evidence to reject the theory?
▶ The hypothesized theory is called null hypothesis, written as H0
⇒ Specify also an alternative hypothesis to the null, H1
▶ Formally, given i.i.d. data x = [x1 , . . . , xn ]T from X1 , . . . , Xn ∼ F
(i) Form a test statistic T (x), i.e., a function of the data
(ii) Define a rejection region R of the form

R = {x : T (x) > c}

▶ If data x ∈ R we reject H0 , otherwise we retain (do not reject) H0

▶ The problem is to select the test statistic T and the critical value c

Network Science Analytics Statistical Inference Review 21

Testing if a coin is fair

Ex: Consider tossing the same coin n times and record the outcomes
▶ Model observations as X1 , . . . , Xn ∼ Ber(p). Is the coin fair?
▶ Let H0 be the hypothesis that the coin is fair, and H1 the alternative
⇒ Can write the hypotheses as

H0 : p = 1/2 versus H1 : p ̸= 1/2

▶ Consider the test statistic given by

n
1 1X 1
T (X1 , . . . , Xn ) = p̂n − = Xi −
2 n 2
i=1

▶ It seems reasonable to reject H0 if (X1 , . . . , Xn ) ∈ R, where

R = {(X1 , . . . , Xn ) : T (X1 , . . . , Xn ) > c}

▶ Will soon see this is a Wald’s test, hence c = zα/2 se.

ˆ More later
Network Science Analytics Statistical Inference Review 22
Tutorial on inference about a mean

Statistical inference and models

Point estimates, confidence intervals and hypothesis tests

Tutorial on inference about a mean

Tutorial on linear regression inference

Network Science Analytics Statistical Inference Review 23

Inference about a mean

▶ Consider a sample of n i.i.d. observations X1 , . . . , Xn ∼ F

▶ Q: How can we perform inference about the mean µ = E [X1 ]?
⇒ Practical and canonical problem in statistical inference
▶ A natural estimator of µ is the sample mean estimator
n
1X
µ̂n = Xi
n
i=1

⇒ Well motivated since by the strong law of large numbers

lim µ̂n = µ almost surely

n→∞

▶ It is a simple example of a method of moments estimator (MME). . .

▶ . . . and also a maximum likelihood estimator (MLE)

Network Science Analytics Statistical Inference Review 24

Moments and sample moments

▶ In parametric inference we wish to estimate θ ∈ Θ ⊆ Rp in

F = {f (x; θ) : θ ∈ Θ}

▶ For 1 ≤ j ≤ p, define the j-th moment of X ∼ F as

Z ∞
 j
αj ≡ αj (θ) = E X = x j f (x; θ)dx
−∞

▶ Likewise, the j-th sample moment is an estimate of αj , namely

n
1X j
α̂j = Xi
n
i=1

⇒ The j-th moment αj (θ) depends on the unknown θ

⇒ But α̂j does not, a function of the data only

Network Science Analytics Statistical Inference Review 25

Method of moments estimator

▶ A first method for parametric estimation is the method of moments

⇒ MMEs are not optimal, yet typically easy to compute
▶ Def: The method of moments estimator (MME) θ̂ n is the solution to

α1 (θ̂ n ) = α̂1
α2 (θ̂ n ) = α̂2
.. .. ..
. . .
αp (θ̂ n ) = α̂p

⇒ This is a system of p (nonlinear) equations with p unknowns

▶ Ex: Back to estimating a mean µ, p = 1 and µ = θ = α1 (θ) so
n
1X
µ̂MM
n = α̂1 = Xi
n
i=1

Network Science Analytics Statistical Inference Review 26

Example: Gaussian data model

Ex: Suppose now X1 , . . . , Xn ∼ N (µ, σ 2 ), i.e., the model is F ∈ FN

▶ Q: What is the MME of the parameter vector θ = [µ, σ 2 ]T ?
▶ The first p = 2 moments are given by

α2 (θ) = E X12 = σ 2 + µ2
 
α1 (θ) = E [X1 ] = µ,

▶ The MME θ̂ n is the solution to the following system of equations

n
1X
µ̂n = Xi
n
i=1
n
1X 2
σ̂n2 + µ̂2n = Xi
n
i=1

▶ The solution is
n n
1X 1X
µ̂n = Xi , σ̂n2 = (Xi − µ̂n )2
n n
i=1 i=1

Network Science Analytics Statistical Inference Review 27

Maximum likelihood estimator

▶ Often “the” method for parametric estimation is maximum likelihood

▶ Consider i.i.d. data X1 , . . . , Xn from a PDF f (x; θ)
▶ The likelihood function Ln (θ) : Θ → R+ is defined by
n
Y
Ln (θ) := f (Xi ; θ)
i=1

⇒ Ln (θ) is the joint PDF of the data, treated as a function of θ

⇒ The log-likelihood function is ℓn (θ) := log Ln (θ)
▶ Def: The maximum likelihood estimator (MLE) θ̂ n is given by

θ̂ n = arg max Ln (θ)

θ

▶ Very useful: The maximizer of Ln (θ) coincides with that of ℓn (θ)

Network Science Analytics Statistical Inference Review 28

Example: Bernoulli data model

▶ Suppose X1 , . . . , Xn ∼ Ber(p). MLE of µ = p?

⇒ The data PMF is f (x; p) = p x (1 − p)1−x , x ∈ {0, 1}
▶ The likelihood function is (define Sn = ni=1 Xi )
P

n
Y n
Y
Ln (p) = f (Xi ; p) = p Xi (1 − p)1−Xi = p Sn (1 − p)n−Sn
i=1 i=1

⇒ The log-likelihood is ℓn (p) = Sn log(p) + (n − Sn ) log(1 − p)

▶ The MLE p̂n is the solution to the equation

∂ℓn (p) Sn n − Sn
= − =0
∂p p=p̂n p̂n 1 − p̂n

▶ The solution is
n
Sn 1X
µ̂ML
n = p̂n = = Xi
n n
i=1

Network Science Analytics Statistical Inference Review 29

Example: Gaussian data model
▶ Suppose X1 , . . . , Xn ∼ N (µ, 1). MLE of µ?
(x−µ)2
⇒ The data PDF is f (x; µ) = √12π exp −

2 ,x ∈R
▶ The likelihood function is (up to constants independent of µ)
n n
Y n X (Xi − µ)2 o
Ln (µ) = f (Xi ; µ) ∝ exp −
2
i=1 i=1
Pn
⇒ The log-likelihood is ℓn (µ) ∝ − i=1 (Xi − µ)2
▶ The MLE µ̂n is the solution to the equation
n
∂ℓn (µ) X
=2 (Xi − µ̂n ) = 0
∂µ µ=µ̂n i=1

▶ The solution is, once more, the sample mean estimator

n
1X
µ̂ML
n = Xi
n
i=1

Network Science Analytics Statistical Inference Review 30

Properties of the MLE

▶ MLEs have desirable properties under loose conditions on f (x; θ)

p
P1) Consistency: θ̂n → θ as the sample size n increases
P2) Equivariance: If θ̂n is the MLE of θ, then g (θ̂n ) is the MLE of g (θ)
ˆ 2)
P3) Asymptotic Normality: For large n, one has θ̂n ∼ N (θ, se
P4) Efficiency: For large n, θ̂n attains the Cramér-Rao lower bound
▶ Efficiency means no other unbiased estimator has smaller variance
▶ Ex: Can use the MLE to create a confidence interval for µ, i.e.,

Cn = µ̂ML ˆ µ̂ML


n − zα/2 se, n + zα/2 se
ˆ

⇒ By asymptotic Normality, P (µ ∈ Cn ) ≈ 1 − α for large n

zα/2
⇒ For the N (µ, 1) model, µ̂ML
n ±
√
n
has exact coverage

Network Science Analytics Statistical Inference Review 31

The Wald test

▶ Consider the following hypothesis test regarding the mean µ

H0 : µ = µ0 versus H1 : µ ̸= µ0

▶ Let µ̂n be the sample mean, with estimated standard error se

ˆ
▶ Def: Given α ∈ (0, 1), the Wald test rejects H0 when

µ̂n − µ0
T (X1 , . . . , Xn ) := > zα/2
se
ˆ

▶ If H0 is true, µ̂n −µ0

se
ˆ ∼ N (0, 1) by the Central Limit Theorem
⇒ Probability of incorrectly rejecting H0 is no more than α
▶ The value of α is called the significance level of the test

Network Science Analytics Statistical Inference Review 32

The p-value

▶ Reporting “reject H0 ” or “retain H0 ” is not too informative

⇒ Could ask, for each α, whether the test rejects at that level
▶ Let Tobs := T (x) be the test statistic value for the observed sample

p/2 p/2

−Tobs Tobs

▶ The probability p := PH0 (|T (X)| ≥ Tobs ) is called the p-value

⇒ Smallest level at which we would reject H0
▶ A small p-value (< 0.05) indicates reduced evidence supporting H0

Network Science Analytics Statistical Inference Review 33

Bayesian inference

▶ Methods discussed so far are termed frequentist, where:

F1: Probability refers to limiting relative frequencies
F2: Parameters are fixed, unknown constants
F3: Statistical procedures offer guarantees on long-run performance
▶ Alternatively, Bayesian inference is based on these postulates:
B1: Probability describes degree of belief, not limiting frequency
B2: We can make probability statements about parameters
B3: A probability distribution for θ is produced to make inferences
▶ Controversial? Inherently embraces a subjective notion of probability
▶ Bayesian methods do not offer long-run performance guarantees
▶ Very useful to combine prior beliefs with data in a principled way

Network Science Analytics Statistical Inference Review 34

The Bayesian method

▶ Bayesian inference is usually carried out in the following way

Step 1: Choose a probability density f (θ) called the prior distribution
▶ The prior expresses our beliefs about θ, before seeing any data

Step 2: Choose a statistical model f (x θ) (compare with f (x; θ))

▶ Reflects our beliefs about the data-generating process, i.e., X given θ

Step 3: Given data X = [X1 , . . . , Xn ]T , we update our beliefs and

calculate the posterior distribution f (θ|X) using Bayes’ rule
n
Y
f (θ|X) ∝ f (Xi θ)f (θ) = Ln (θ)f (θ)
i=1
⇒ Point estimates, confidence intervals obtained from f (θ|X)
▶ Ex: A maximum a posteriori (MAP) estimator θ̂n = arg maxθ f (θ|X)

Network Science Analytics Statistical Inference Review 35

Example: Gaussian data model and prior

▶ Consider X1 , . . . , Xn ∼ N (µ, σ 2 ). Suppose σ 2 is known

⇒ To estimate θ we adopt the prior θ ∼ N (a, b 2 )
▶ Using Bayes’ rule, can show the posterior is also Gaussian where
n
 wX se−2
θ̂nMAP = E θ X =

Xi + (1 − w )a, with w = −2
n se + b −2
i=1

⇒ Weighted average of the sample mean θ̂nML and the prior mean a
√
⇒ Here, se = σ/ n is the standard error for the sample mean
▶ Asymptotics: Note that w → 1 as the sample size n → ∞
⇒ For large n the posterior is approximately N (θ̂nML , se2 )
⇒ Same holds if n is fixed but b → ∞, i.e., prior is uninformative

Network Science Analytics Statistical Inference Review 36

Tutorial on linear regression inference

Statistical inference and models

Point estimates, confidence intervals and hypothesis tests

Tutorial on inference about a mean

Tutorial on linear regression inference

Network Science Analytics Statistical Inference Review 37

Linear regression

▶ Suppose observations are from (Y1 , X1 ), . . . , (Yn , Xn ) ∼ FYX

⇒ Goal is to learn the relationship between the RVs Y and X
▶ A workhorse approach is to model the regression function
Z ∞
 
r (x) = E Y X = x = yfY |X (y |x)dy
−∞

▶ The simple linear regression model specifies that given Xi = xi

y i = β 0 + β1 x i + ϵi , i = 1, . . . , n
▶ The yi ’s are modeled as noisy samples of the line r (x) = β0 + β1 x
▶ Errors ϵi are i.i.d., with E [ϵi |Xi = xi ] = 0 and var [ϵi |Xi = xi ] = σ 2

▶ With the linear model, regression amounts to parametric inference

rˆ(x) ⇔ [β̂0 , β̂1 ]T

Network Science Analytics Statistical Inference Review 38

Multiple linear regression

▶ More generally, suppose we observe data (y1 , x1 ), . . . , (yn , xn )

⇒ Each input xi = [xi1 , . . . , xip ]T is a p × 1 feature vector
▶ The multiple linear regression model specifies
p
X
yi = xij βj + ϵi = β T xi + ϵi , i = 1, . . . , n
j=1
▶ Typically xi1 = 1 for all i, providing an intercept term
▶ Errors ϵi are i.i.d., with E [ϵi |Xi = xi ] = 0 and var [ϵi |Xi = xi ] = σ 2

▶ Can be compactly represented as y = Xβ + ϵ, defining

       
y1 x11 . . . x1p β1 ϵ1
y =  ...  , X =  ... .. ..  , β =  ..  , ϵ =  .. 
  
. .   .   . 
yn xn1 ... xnp βp ϵn

Network Science Analytics Statistical Inference Review 39

Least-squares estimator

▶ A sound estimate β̂ minimizes the residual sum of squares (RSS)

n
X
RSS(β) = (yi − β T xi )2 = ∥y − Xβ∥2
i=1

⇒ Residuals are the distances from yi to hyperplane r (x) = β T x

▶ Def: The least-squares estimator (LSE) β̂ n is the solution to

β̂ n = arg min RSS(β)

β

▶ Carrying out the optimization yields the LSE β̂ n = (XT X)−1 XT y

⇒ Only defined if XT X invertible ⇔ X has full column rank p

Network Science Analytics Statistical Inference Review 40

Geometry of the LSE

▶ In least squares we seek the vector ŷ = Xβ̂ ∈ span(X) closest to y

y y ŷ

ŷ = X ˆ
0
X
span(X)

▶ Solution: Orthogonal projection of y onto span(X), i.e., (let X = UΣVT )

ŷ = PX (y) = X(XT X)−1 XT y = UUT y

▶ The residual y − ŷ lies in the orthogonal complement (span(X))⊥

⇒ This way RSS(β̂) = ∥y − ŷ∥2 is minimum

Network Science Analytics Statistical Inference Review 41

Properties of the LSE

▶ LSE β̂ n = (XT X)−1 XT y is a linear combination of the random y

h i h i
P1) Unbiasedness: E β̂ n X = β with var β̂ n X = σ 2 (XT X)−1
p
P2) Consistency: β̂ n → β as the sample size n increases
P3) Asymptotic Normality: For large n, one has β̂ n ∼ N (β, σ 2 (XT X)−1 )
P4) If errors ϵ ∼ N (0, σ 2 I), then β̂ n ∼ N (β, σ 2 (XT X)−1 ) exactly; and
Efficiency: No other unbiased estimator of β has smaller variance
▶ Ex: Can use the LSE to create confidence intervals for each βj , i.e.,
 
Cn = β̂j − zα/2 se(
ˆ β̂j ), β̂j + zα/2 se(
ˆ β̂j )

⇒ By asymptotic (or exact) Normality, P (βj ∈ Cn ) ≈ 1 − α

ˆ β̂j ) = σ̂ [(XT X)−1 ]jj , where σ̂ 2 = RSS( β̂)
p
⇒ Note that se( n−p

Network Science Analytics Statistical Inference Review 42

Hypothesis testing and prediction

Ex: Consider the hypothesis test regarding the parameter βj

(0) (0)
H 0 : βj = βj versus H1 : βj ̸= βj

▶ By asymptotic (or exact) Normality of the LSE, an α-level test is

(0)
β̂j − βj
Reject H0 if Tj := > zα/2
se(
ˆ β̂j )

Ex: Can predict an unobserved value Y∗ = y∗ from a given x∗ via

y∗ = x T
∗ β̂

▶ May define a notion of standard error for y∗ , and predictive intervals

⇒ Should account for the variability in estimating β and in ϵ∗

Network Science Analytics Statistical Inference Review 43

The LSE as a MLE

▶ Suppose that conditioned on Xi = xi , the errors ϵi are i.i.d. Normal

1
 ϵ2
⇒ The conditional PDF is f (ϵi xi ) = √2πσ 2
exp − 2σi 2
▶ Assume σ 2 is known. The (conditional) likelihood function is
n n
Y n X (yi − β T xi )2 o
Ln (β) = f (yi xi ; β) ∝ exp −
2σ 2
i=1 i=1

⇒ The log-likelihood is ℓn (β) ∝ −RSS(β)

ML
▶ The MLE β̂ n maximizes the log-likelihood function, thus
ML LS
β̂ n = arg max ℓn (β) = arg min RSS(β) = β̂ n
β β

▶ Take-home: Under a linear-Gaussian model the LSE is also a MLE

Network Science Analytics Statistical Inference Review 44

MAP with Gaussian data model and prior

ϵ2i
√ 1

▶ Consider again Gaussian errors, i.e., f (ϵi xi ) = exp − 2σ 2
2πσ 2
⇒ Gaussian prior to model the parameters: β ∼ N (0, τ 2 I)
⇒ Variances σ 2 and τ 2 assumed known. Define λ := ( στ )2
▶ Bayesian approach: posterior Fβ|Y,X is Gaussian, with log-density

n
X p
X
T 2
log f (β Y, X) ∝ − (yi − β xi ) − λ βj2
i=1 j=1

MAP
▶ MAP estimator β̂ n := arg maxβ f (β Y, X) is thus the solution to
MAP
β̂ n = arg min RSS(β) + λ∥β∥22
β

MAP
▶ Carrying out the optimization yields β̂ n = (XT X + λI)−1 XT y
⇒ Recover the LSE as λ → 0 ⇔ Uninformative prior when τ 2 → ∞

Network Science Analytics Statistical Inference Review 45

Ridge regression

▶ Non-Bayesian, ℓ2 -norm penalized LSE also known as ridge regression

ridge
β̂ = arg min RSS(β) + λ∥β∥22
β

ridge
▶ For λ > 0, the ridge estimator β̂ = (XT X + λI)−1 XT y
LS
▶ Differs from the LSE β̂ := arg minβ RSS(β)
ridge
▶ Is biased, and bias(β̂ ) increases with λ
▶ Is well defined even when X is not of full rank
h LS i
▶ In exchange for bias, potential to reduce variance below var β̂
h LS i
▶ Ex: Large var β̂ when X nearly rank-deficient, unstable (XT X)−1

▶ From bias-variance MSE decomposition, fruitful tradeoff may yield

ridge LS
MSE(β̂ ) < MSE(β̂ )

⇒ Tradeoff depends on λ, chosen subjectively or via cross validation

Network Science Analytics Statistical Inference Review 46
Complexity-penalized LSE

▶ Ridge an instance from the general class of complexity-penalized LSE

J
β̂ = arg min RSS(β) + λJ(β)
β

▶ Function J(·) penalizes (i.e., constrains) the parameters in β

▶ Constrained parameter space Θ effects ‘less complex’ models
▶ Tuning λ balances goodness-of-fit and model complexity

▶ Ex: ℓ1 -norm penalized LSE for sparsity, i.e., variable selection

Network Science Analytics Statistical Inference Review 47

Glossary

▶ Consistent estimator
▶ Statistical inference
▶ Confidence interval
▶ Outcome or response
▶ Hypothesis test
▶ Predictor, feature or regressor
▶ Null hypothesis
▶ (Non) parametric model
▶ Test statistic and critical value
▶ Nuisance parameter
▶ Method of moments estimator
▶ Regression function
▶ Maximum likelihood estimator
▶ Prediction
▶ Likelihood function
▶ Classification
▶ Significance level and p − value
▶ Point and set estimation
▶ Prior and posterior distribution
▶ Estimator and estimate
▶ Multiple linear regression
▶ Standard error
▶ Least-squares estimator

Network Science Analytics Statistical Inference Review 48