Modeling Techniques -
Statistical Modeling
Modeling Techniques
Session Outline
Modeling Techniques: how predictive models are constructed
Statistical Modeling for prediction (regression family)
Classes of predictive models: parametric, non-parametric, semi-parametric
What is a statistical model? (conceptual and formal views)
Model comparison: exploratory vs confirmatory analysis
Worked example: nested linear regression models
Model classification: complete vs incomplete; common model families
Modeling Techniques
Modeling Techniques
Modeling Techniques
There are multiple ways to construct predictive models (depending on the goal and data).
Propensity models are often constructed using:
statistical techniques, and
machine learning methods.
Cluster models may be constructed using:
statistical techniques, and/or
heuristic algorithms.
Modeling Techniques
What is Regression?
Regression is a statistical technique used to model and predict the relationship between a dependent variable (output) and
one or more independent variables (inputs).
Regression helps us understand how one variable changes when another variable changes, and it allows us to make
predictions.
❖ One variable is explained or predicted
❖ Other variables are used as explanatory factors
❖ The relationship includes random error (it is not exact)
Suppose we want to understand how exam marks depend on study hours.
1. Study hours → independent variable
2. Exam marks → dependent variable
Regression finds the best mathematical relationship between them.
Simple regression model
Marks = 𝑏0 + 𝑏1 Study Hours + 𝜀
❖ 𝑏0 :baseline marks
❖ 𝑏1 :change in marks per hour of study
❖ 𝜀: unexplained variation (noise)
❖ Why regression is used
❖ To predict future values
❖ To measure the effect of one variable on another
❖ To explain relationships in data
Statistical Modeling
Statistical Modeling: Regression Models for Prediction
Nearly any regression model can be used for prediction, including:
linear regression,
logistic regression,
general linear model (GLM),
robust regression, etc.
These regression families can be adapted to predictive tasks under appropriate
assumptions.
Modeling Techniques
Statistical Modeling
Logistic Regression for Propensity Models
Logistic regression is a widely used modeling technique for propensity models.
Particularly suitable when the response (dependent variable) is binary:
e.g., Yes/No, Success/Failure, Treatment/No treatment.
Interpretation focuses on the probability of the binary outcome given predictors.
Modeling Techniques
Classes of Predictive Models
Predictive Model Classes
Broadly, predictive models can be grouped into:
Parametric models
Non-parametric models
A third class also exists:
Semi-parametric models (combine features of both).
Parametric Non-parametric
(stronger assumptions) (fewer assumptions)
Semi-parametric
(mixed features)
Modeling Techniques
Statistical Model: Meaning and Formalization
What is a Statistical Model?
A statistical model is a formalization of relationships between variables using
mathematical equations.
It describes how one or more random variables are related to other variables.
The relationship is stochastic (not deterministic): variability is explicitly modeled.
Modeling Techniques
Statistical Model: Meaning and Formalization
Statistical Model as a Pair (Y , P)
A statistical model is frequently viewed as a pair (Y , P):
Y : the set of possible observations,
P: the set of possible probability distributions on Y .
It is assumed that there exists a distinct element of P that generated the observed data.
Statistical inference aims to identify which element(s) of P are likely to be the true one.
Modeling Techniques
Statistical Model: Meaning and Formalization
Tests and Models: Common Structure
Many statistical tests can be described in the form of a statistical model.
Example: the Student’s t-test (comparing means of two groups) can be framed as:
checking whether an estimated parameter in the model differs from 0.
A shared feature between models and tests is the presence of assumptions.
In many models, error is assumed to be normally distributed.
Modeling Techniques
Formal Definition (Equations Kept Accurate)
Formal Definition (Statistical Model)
A statistical model is a collection of:
probability distribution functions, or
probability density functions,
collectively referred to as distributions (for brevity).
Modeling Techniques
Model Comparison
Model Comparison: When and Why
Models can be compared to each other:
after exploratory data analysis (EDA), or
during confirmatory data analysis.
In EDA:
formulate multiple candidate models,
select the model that best describes the data.
In confirmatory analysis:
test which pre-specified model(s) fit the data best,
or test whether the only proposed model fits the data.
Modeling Techniques
Model Comparison
Model Comparison in Linear Regression
In linear regression, models can be compared using the amount of variance explained:
R2.
In general, nested models can be compared using a likelihood-ratio test.
Nested models:
a simpler model is obtained from a more complex model
by restricting a parameter in the complex model to be zero.
Modeling Techniques
Worked Example (Nested Models)
Example: Height and Age (Probabilistic Relationship)
Height and age are probabilistically distributed over humans.
They are stochastically related; knowing age changes the chance of observing a certain
height.
A linear regression model can formalize this relationship:
heighti = b0 + b1agei + ϵi ,
b0: intercept b1: parameter multiplying age ϵi : error term i: subject
Prediction is not perfect because the error term captures unexplained variation.
Modeling Techniques
Worked Example (Nested Models)
Example: Adding Sex and Nesting
Additional variables (e.g., sex) can reduce unexplained error and improve fit.
The model becomes:
heighti = b0 + b1agei + b2sexi + ϵi ,
sexi is dichotomous (binary).
This expanded model would typically have a higher R2.
The first model is nested in the second model:
the first model is obtained by restricting b2 = 0.
Modeling Techniques