Regression Analysis (R)
1. Introduction
Regression analysis is a statistical method used to model the relationship between a
dependent variable (response) and one or more independent variables (predictors).
It is widely used for prediction, forecasting, and understanding relationships between
variables.
Applications
Predicting car mileage (mpg) based on weight and horsepower.
Forecasting sales based on advertising spend.
Estimating house prices from size, location, and amenities.
2. Types of Regression
1. Simple Linear Regression
o Relationship between one predictor and one response.
o Model: Y=β0+β1X+ϵ
2. Multiple Linear Regression
o Relationship between multiple predictors and a response.
o Model: Y=β0+β1X1+β2X2+⋯+βnXn+ϵ
3. Polynomial Regression
o Models non-linear relationships using polynomial terms.
o Example: Y=β0+β1X+β2X2+ϵ
4. Logistic Regression (for binary outcomes)
o Predicts categorical response (0/1).
o Uses log-odds transformation.
3. Assumptions of Regression
1. Linearity – Relationship between predictors and response is linear.
2. Independence – Observations are independent of each other.
3. Homoscedasticity – Constant variance of residuals across all levels of predictors.
4. Normality of residuals – Residuals should be approximately normally distributed.
5. No multicollinearity – Predictors should not be highly correlated.
4. Implementation in R
Step 1: Load Dataset
# Load mtcars dataset
mtcars <- [Link]("[Link]")
head(mtcars)
Step 2: Simple Linear Regression
Predict mpg using wt (weight):
# Simple Linear Regression
model1 <- lm(mpg ~ wt, data=mtcars)
summary(model1)
Step 3: Multiple Regression
Predict mpg using wt and hp (horsepower):
# Multiple Linear Regression
model2 <- lm(mpg ~ wt + hp, data=mtcars)
summary(model2)
Step 4: Make Predictions
predicted_mpg <- predict(model2, mtcars)
predicted_mpg
5. Diagnostics and Model Evaluation
1. Residual Analysis
# Plot residuals
plot(model2$residuals)
hist(model2$residuals)
2.
3. Check Linearity
plot(mtcars$wt, mtcars$mpg)
abline(model1, col="red")
3. R-squared and Adjusted R-squared
R² indicates the proportion of variance explained by predictors.
Adjusted R² accounts for number of predictors.
4. RMSE (Root Mean Squared Error)
rmse <- sqrt(mean(model2$residuals^2))
rmse
6. Advanced Topics
Stepwise Regression – Automated variable selection.
Regularization – Ridge and Lasso regression to reduce overfitting.
Polynomial terms – Capture non-linear relationships.
7. Visualization
1. Scatter plot with regression line
plot(mtcars$wt, mtcars$mpg)
abline(model1, col="blue")
2. Diagnostic plots
par(mfrow=c(2,2))
plot(model2)
8. Exercises
1. Predict mpg using wt and hp and evaluate model performance.
2. Plot residuals and check normality.
3. Try adding cyl (number of cylinders) to the model and compare R².
4. Use polynomial regression to model mpg ~ wt + I(wt^2).
5. Interpret coefficients and explain their impact on mpg.
Classification Techniques (R)
1. Introduction
Classification is a supervised machine learning technique used to predict categorical
outcomes based on input features.
It assigns data points to predefined classes or labels.
Applications
Predicting if an email is spam or not spam
Predicting customer churn
Disease diagnosis (positive/negative)
Iris flower species classification
2. Types of Classification Techniques
2.1 Logistic Regression
Predicts a binary outcome (0/1, yes/no)
Uses logit function (log-odds)
Outputs probabilities which can be converted to classes
R Example (Binary Classification)
# Binary classification: setosa vs others
iris <- [Link]('[Link]')
iris$Binary <- ifelse(iris$Species == 'setosa', 1, 0)
log_model <- glm(Binary ~ [Link] + [Link], data=iris,
family=binomial)
summary(log_model)
# Predict probabilities
pred_probs <- predict(log_model, iris, type='response')
pred_class <- ifelse(pred_probs > 0.5, 1, 0)
2.2 Decision Trees
Tree-based structure for multi-class classification
Splits data based on feature thresholds
Easy to interpret
R Example
library(rpart)
tree_model <- rpart(Species ~ [Link] + [Link] + [Link] +
[Link],
data=iris, method='class')
print(tree_model)
plot(tree_model)
text(tree_model, use.n=TRUE)
2.3 Random Forest
Ensemble of decision trees
Reduces overfitting
Better accuracy than a single tree
R Example
library(randomForest)
rf_model <- randomForest(Species ~ ., data=iris)
print(rf_model)
2.4 k-Nearest Neighbors (kNN)
Classifies a point based on majority vote of k nearest neighbors
Distance metrics: Euclidean, Manhattan
R Example
library(class)
train <- iris[1:100, 1:4]
train_labels <- iris[1:100, 5]
test <- iris[101:150, 1:4]
pred <- knn(train, test, train_labels, k=3)
2.5 Support Vector Machine (SVM)
Finds the best hyperplane to separate classes
Works well with high-dimensional data
R Example
library(e1071)
svm_model <- svm(Species ~ ., data=iris)
pred_svm <- predict(svm_model, iris)
3. Model Evaluation Metrics
Accuracy: (TP + TN) / Total
Precision: TP / (TP + FP)
Recall (Sensitivity): TP / (TP + FN)
F1-Score: Harmonic mean of Precision & Recall
Confusion Matrix: Table of predicted vs actual
R Example
table(pred_class, iris$Binary) # Confusion matrix
ROC Curve & AUC for binary classification
library(pROC)
roc_obj <- roc(iris$Binary, pred_probs)
plot(roc_obj)
auc(roc_obj)
4. Steps to Implement Classification in R
1. Load dataset ([Link])
2. Split data into training and testing sets
3. Choose a classification algorithm
4. Train the model on training set
5. Predict on testing set
6. Evaluate performance using accuracy, confusion matrix, ROC/AUC
7. Tune model parameters if necessary
5. Visualization
Decision tree plots (plot() + text())
ROC curve (pROC::roc())
Feature importance (Random Forest: importance())
6. Exercises
1. Build a decision tree classifier for Iris dataset. Plot the tree and interpret splits.
2. Train a random forest model on Iris dataset and check feature importance.
3. Apply logistic regression for binary classification and compute confusion matrix.
4. Split data 70/30 for training/testing and compare accuracy of kNN vs SVM.
5. Plot ROC curve and calculate AUC for logistic regression predictions.
Clustering Techniques (R) – Teaching Notes
1. Introduction
Clustering is an unsupervised machine learning technique used to group similar
observations based on features, without predefined labels.
Applications
Customer segmentation in marketing
Image segmentation
Anomaly detection
Pattern recognition
2. Types of Clustering Techniques
2.1 K-Means Clustering
Partition-based clustering method
Groups data into K clusters based on feature similarity
Minimizes within-cluster variance
Steps:
1. Choose number of clusters K
2. Initialize cluster centroids
3. Assign points to nearest centroid
4. Recalculate centroids
5. Repeat until convergence
R Example
iris_data <- iris[,1:4] # Use only numeric columns
kmeans_model <- kmeans(iris_data, centers=3)
print(kmeans_model$cluster)
Visualization
library(cluster)
clusplot(iris_data, kmeans_model$cluster, color=TRUE, shade=TRUE, labels=2,
lines=0)
2.2 Hierarchical Clustering
Creates a tree (dendrogram) showing nested clusters
Two approaches:
1. Agglomerative (bottom-up) – each observation starts as its own cluster,
merge iteratively
2. Divisive (top-down) – start with all points in one cluster, split iteratively
R Example
dist_mat <- dist(iris_data) # Compute distance matrix
hclust_model <- hclust(dist_mat, method='ward.D2') # Hierarchical
clustering
plot(hclust_model) # Dendrogram
Cut dendrogram into clusters
clusters <- cutree(hclust_model, k=3)
table(clusters)
2.3 Density-Based Clustering (DBSCAN)
Groups points based on density
Can identify noise/outliers
Useful when clusters have irregular sha
R Example
library(dbscan)
dbscan_model <- dbscan(iris_data, eps=0.5, minPts=5)
dbscan_model$cluster
3. Choosing the Number of Clusters
Elbow Method: Plot total within-cluster sum of squares vs K, choose “elbow” point
Silhouette Score: Measures how similar a point is to its cluster vs other clusters
Gap Statistic: Compares total intra-cluster variation with reference random data
R Example (Elbow Method)
wss <- sapply(1:10, function(k){
kmeans(iris_data, k, nstart=10)$[Link]
})
plot(1:10, wss, type='b', xlab='Number of clusters K', ylab='Total within-
clusters sum of squares')
4. Interpretation
Examine cluster centroids (mean values of features in each cluster)
Profile clusters for actionable insights:
o Example: Segment customers by age and spending score
Compare results of different clustering methods
5. Visualization
Scatter plots colored by cluster
library(ggplot2)
iris_data$Cluster <- [Link](kmeans_model$cluster)
ggplot(iris_data, aes(x=[Link], y=[Link], color=Cluster)) +
geom_point()
Dendrogram for hierarchical clustering
Cluster plots (clusplot)
6. Steps to Implement Clustering in R
1. Load dataset ([Link])
2. Standardize features (if needed)
3. Choose clustering technique
4. Apply clustering algorithm
5. Determine optimal number of clusters
6. Visualize clusters and interpret results
7. Compare results across different methods
7. Exercises
1. Apply K-Means clustering on Iris dataset and visualize clusters.
2. Apply Hierarchical clustering on the same dataset and plot dendrogram.
3. Use Elbow method to choose the number of clusters.
4. Cluster a customer dataset (Age vs Spending Score) and interpret clusters.
5. Compare K-Means vs Hierarchical clustering results.
6. Identify potential outliers using DBSCAN.