1.

Regression Techniques
Linear Regression

Linear regression models the relationship between a dependent variable ($Y$) and one or
more independent variables ($X$).

 Ordinary Least Squares (OLS): The standard method for estimating coefficients. It
minimizes the sum of the squared differences (residuals) between observed values and
the values predicted by the linear approximation.
 Multiple Regression: An extension of simple linear regression used when there are
two or more independent variables.
 Assumptions: For OLS results to be valid (BLUE - Best Linear Unbiased Estimator),
the data must meet specific criteria:
o Linearity: The relationship between $X$ and $Y$ is linear.
o Independence: Observations are independent of each other.
o Homoscedasticity: Constant variance of error terms.
o Normality: Errors are normally distributed.
 Multicollinearity: Occurs when independent variables are highly correlated. This
inflates the variance of coefficients, making the model unstable. It is often measured
using the Variance Inflation Factor (VIF).
 Residual Analysis: Examining the difference between observed and predicted values
to check for patterns that suggest model misfit or assumption violations.

Specialized Regression Variables

 Dummy Variables: Categorical data (like "Gender" or "Region") converted into

numerical format (0 or 1) so they can be used in regression models.
 Stepwise Regression: An automated process of selecting the most significant
variables for the model by adding or removing them based on statistical significance
(e.g., AIC or p-values).

2. Classification Techniques (Categorical Outcomes)

Logistic Regression

Used when the dependent variable is binary (e.g., Yes/No, Default/No Default).

 Odds Ratios: Represents the constant effect of a predictor $X$ on the likelihood that
the outcome will occur.
 Log-Likelihood: The function used to estimate the parameters; we seek to maximize
this value to find the best-fitting model.
 Performance Metrics:
o ROC Curve: A plot of the True Positive Rate vs. the False Positive Rate. The
AUC (Area Under the Curve) measures the model's ability to distinguish
between classes.
 o Classification Tables (Confusion Matrix): A table showing correct vs.
incorrect predictions (Accuracy, Precision, Recall).

Discriminant Analysis

Linear Discriminant Analysis (LDA) finds a linear combination of features that characterizes
or separates two or more classes of objects.

 Linear Discriminant Function: Maximizes the ratio of between-class variance to

within-class variance.
 Classification Performance: Evaluated similarly to logistic regression, focusing on
how well the model assigns new observations to the correct group.

3. Dimension Reduction
When dealing with many variables, dimension reduction simplifies the dataset while retaining
as much information as possible.

 Principal Component Analysis (PCA): A technique that transforms correlated

variables into a smaller set of uncorrelated variables called Principal Components. It
focuses on capturing the maximum variance in the data.
 Factor Analysis: Used to identify underlying, unobservable "latent" variables
(factors) that explain the correlations between observed variables.

4. Implementation in R
R is the industry standard for these statistical methods. A typical workflow involves:

1. Linear: lm(y ~ x1 + x2, data = df)

2. Logistic: glm(y ~ x1, family = "binomial", data = df)
3. PCA: prcomp(df, scale = TRUE)

Example Case Study: In a banking domain, you might use Logistic Regression to predict
loan default (Yes/No) and PCA to reduce 50 different credit-scoring metrics into 5 key
components.

Step Technique Key R Output / Metric

Model Fit OLS / Logistic Adjusted $R^2$ (Linear) / AIC (Logistic)

Significance p-values Look for stars (***) indicating $p < 0.05$

Independence Durbin-Watson Tests for autocorrelation in residuals

 Step Technique Key R Output / Metric

Classification Confusion Matrix Sensitivity (Recall) and Specificity

1. Logistic Regression: Deep Dive

Unlike Linear Regression, which predicts a continuous value, Logistic Regression predicts
the probability ($p$) that an observation belongs to a specific category.

 The Concept: It uses the Logit Link Function to map any real-valued input to a value
between 0 and 1. The equation is:

$$\ln\left(\frac{p}{1-p}\right) = \beta_0 + \beta_1 X_1 + ... + \beta_n X_n$$

 Odds Ratios: The "Odds" are defined as the probability of success divided by the
probability of failure ($p / (1-p)$). The Odds Ratio (OR) tells you how the odds
change with a one-unit increase in $X$.
o $OR > 1$: Increasing $X$ increases the odds of the outcome.
o $OR < 1$: Increasing $X$ decreases the odds.
 Log-Likelihood: This is the "engine" of the model. Since we can't use OLS, we use
Maximum Likelihood Estimation (MLE). The log-likelihood measures how well
the model parameters support the observed data. A higher (less negative) log-
likelihood indicates a better fit.
 Classification Table (Confusion Matrix): A cross-tabulation of observed vs.
predicted classes.
o Sensitivity: Ability to correctly identify "Positives."
o Specificity: Ability to correctly identify "Negatives."
 ROC Curves: A visual trade-off between Sensitivity and (1-Specificity). The closer
the curve hugs the top-left corner, the better the model.

2. Discriminant Analysis (LDA)

While Logistic Regression is preferred for its lack of strict assumptions, Linear
Discriminant Analysis (LDA) is often more stable when classes are well-separated or the
sample size is small.

 Linear Discriminant Function: It projects data onto a lower-dimensional space to

maximize the distance between the means of different classes while minimizing the
variance within each class.
 Classification Performance: Evaluated using the Apparent Error Rate (APER) or
Hit Ratio. Unlike Logistic Regression (which uses a 0.5 probability cutoff), LDA
uses a "Decision Boundary" based on the centroids of the groups.
3. Stepwise & Dummy Variable Regression
These are methods used to refine and prepare variables for a final regression model.

Dummy Variable Regression

When you have categorical data (e.g., "Season": Spring, Summer, Fall, Winter), you cannot
plug text into an equation.

 The Rule: If you have $K$ categories, you create $K-1$ dummy variables.
 Reference Category: One category is left out to serve as the baseline for comparison.
For example, if "Spring" is the baseline, the coefficient for "Summer" tells you the
change in $Y$ relative to Spring.

Stepwise Regression

This is an iterative procedure to build the "most efficient" model.

 Forward Selection: Starts with no variables and adds the most significant one at each
step.
 Backward Elimination: Starts with all variables and removes the least significant
one.
 Criteria: Usually uses AIC (Akaike Information Criterion) or BIC. A lower AIC
indicates a better model that isn't overfitted.

R Implementation Snippet: Stepwise & Dummy

Here is how you would handle these in R:

R
# 1. Dummy variables are often handled automatically by factor()
df$Season <- [Link](df$Season)
model_dummy <- lm(Sales ~ Season, data = df)

# 2. Stepwise Regression using the MASS library

library(MASS)
full_model <- lm(Sales ~ ., data = df) # Start with all variables
step_model <- stepAIC(full_model, direction = "both") # Add/Remove
automatically
summary(step_model)

To tie these concepts together, let’s look at how R handles these methods through two
distinct domain-based case studies: Finance (for Regression/Classification) and
Psychology/Marketing (for Dimension Reduction).
1. Case Study: Credit Risk Scoring (Regression &
Classification)
Domain: Finance

Goal: Predict whether a loan applicant will default.

Step-wise & Dummy Variable Regression

In credit scoring, we often have categorical data like Employment_Type (Full-time, Part-
time, Unemployed).1 R converts these into Dummy Variables automatically when you
define them as factors.

R
# Convert to Factor (R creates k-1 dummy variables)
data$Employment <- [Link](data$Employment)

# Step-wise Regression to find the best predictors of 'Credit_Score'

full_model <- lm(Credit_Score ~ ., data = data)
step_model <- step(full_model, direction = "backward")

Logistic Regression & ROC Curves

To predict the binary outcome (Default: Yes/No), we use Logistic Regression. We evaluate it
using the Classification Table and ROC Curve.

R
# Logistic Model
logit_model <- glm(Default ~ Income + Debt_Ratio + Employment, family =
"binomial", data = data)

# Odds Ratios
exp(coef(logit_model))

# ROC Curve & AUC

library(pROC)
prob <- predict(logit_model, type = "response")
roc_curve <- roc(data$Default, prob)
plot(roc_curve, col = "blue", main = "ROC Curve for Credit Default")

Discriminant Analysis (LDA)

If the classes (Default vs. Non-Default) are well-separated and we assume the predictors
follow a multivariate normal distribution, LDA is a powerful alternative.

R
library(MASS)
lda_model <- lda(Default ~ Income + Debt_Ratio, data = data)
plot(lda_model) # Visualizes the separation
2. Case Study: Consumer Behavior (Dimension
Reduction)
Domain: Marketing Research

Goal: Reduce 20 different survey questions about "Brand Loyalty" into a few manageable
themes.

Principal Component Analysis (PCA)

PCA is used to reduce dimensionality while keeping as much variance as possible.2 It is

purely mathematical and doesn't assume an underlying "latent" structure.

 Use Case: Summarizing 50 different stock market technical indicators into 3

"momentum" components.
 R Code:

pca_res <- prcomp(survey_data, scale. = TRUE)

summary(pca_res)
screeplot(pca_res, type = "lines") # Look for the 'elbow'

Factor Analysis (FA)

Unlike PCA, Factor Analysis assumes that your observed variables are influenced by
unobserved latent factors.

 Use Case: In a survey, questions about "Trust," "Satisfaction," and "Repeat Purchase"
might all load onto one latent factor: Brand Loyalty.
 R Code:

# Factanal performs Factor Analysis

fa_res <- factanal(survey_data, factors = 3, rotation = "varimax")
print(fa_res$loadings, cutoff = 0.4)

Comparison: PCA vs. Factor Analysis

Feature PCA Factor Analysis

Explaining correlations (latent

Goal Data reduction (maximize variance).
variables).

Structure Components are linear combinations Variables are linear combinations of

Feature PCA Factor Analysis

of variables. factors.

When to To simplify data for a regression To understand underlying psychological

use? model. constructs.

Summary of Performance Metrics in R

 Linear Regression: Check $R^2$ and Residual Plots.
 Logistic Regression: Check AIC, Log-Likelihood, and AUC.
 Discriminant Analysis: Check the Hit Ratio (Percentage of correct classifications).
 PCA/FA: Check the Proportion of Variance Explained