Building linear models - model fitting, Predict values
using models,Analyzing the fit,Refining the model,
Regression- types, Unusual observation and corrective
measures, Comparison of models, Generalized linear
models - Logistic Regression, Nonlinear least squares
Introduction to Regression
Regression in machine learning is a supervised learning technique used to predict continuous
numerical values by learning relationships between input variables (features) and an output
variable (target). It helps understand how changes in one or more factors influence a measurable
outcome and is widely used in forecasting, risk analysis, decision-making and trend estimation.
● Works with real-valued output variables
● Helps to identify strengths and the type of relationships
● Supports both simple and complex predictive models.
● Used for tasks like price prediction, trend forecasting and risk scoring.
Steps to establish a regression model
Types of Regression Models
● Simple Linear Regression
● Multiple Linear Regression
● Polynomial Regression
● Logistic Regression
● Poisson Regression
● Ridge & Lasso Regression
Linear regression
● Linear regression is a type of supervised machine-learning algorithm that learns from the
labelled datasets and maps the data points with most optimized linear functions which can be
used for prediction on new datasets.
● It assumes that there is a linear relationship between the input and output, meaning the output
changes at a constant rate as the input changes.
● This relationship is represented by a straight line.
For example we want to predict a student's exam score based on how many hours they studied. We observe that
as students study more hours, their scores go up. In the example of predicting exam scores based on hours
studied. Here
● Independent variable (input): Hours studied because it's the factor we control or observe.
● Dependent variable (output): Exam score because it depends on how many hours were studied.
We use the independent variable to predict the dependent variable.
Best Fit Line in Linear Regression
● In linear regression, the best-fit line is the straight line that most accurately represents the
relationship between the independent variable (input) and the dependent variable (output).
● It is the line that minimizes the difference between the actual data points and the predicted
values from the model.
1. Goal of the Best-Fit Line
● The goal of linear regression is to find a straight line that minimizes the error (the
difference) between the observed data points and the predicted values.
● This line helps us predict the dependent variable for new, unseen data.
● Here Y is called a dependent
or target variable
● X is called an independent
variable also known as the
predictor of Y.
● θ1 represents the intercept,
which is the value of Y when
X=0
● θ2 represents the slope, which
shows how much Y changes
for a unit change in X
Types of Linear Regression
● When there is only one independent feature it is known as Simple Linear Regression or Univariate Linear Regression
● When there are more than one feature it is known as Multiple Linear Regression or Multivariate Regression.
Write a R program to solve linear regression and make predictions
# Linear Regression in R using CSV file
# Step 1: Create sample data
height <- c(150, 160, 170, 180, 190) # independent variable (X)
weight <- c(50, 55, 65, 70, 80) # dependent variable (Y)
data <- [Link](height, weight)
# Step 2: Write data to CSV file
[Link](data, "regression_data.csv", [Link] = FALSE)
# Step 3: Read data back from CSV file
dataset <- [Link]("regression_data.csv")
# Step 4: Fit the linear regression model
model <- lm(weight ~ height, data = dataset)
# Step 5: Display model summary
summary(model)
# Step 6: Make predictions
new_heights <- [Link](height = c(175, 185))
predicted_weights <- predict(model, new_heights)
# Step 7: Print predictions
print(predicted_weights)
# Step 8: Plot regression line
plot(dataset$height, dataset$weight, col = "blue", pch = 19,
xlab = "Height (cm)", ylab = "Weight (kg)",
main = "Linear Regression Example (CSV Data)")
abline(model, col = "red", lwd = 2)
# Linear Regression in R using CSV file
# Step 1: Read the dataset from the CSV file into data
dataset <- [Link]("regression_data.csv")
# Step 2: Extract Variables
x <- dataset$height # predictor variable (independent)
y <- dataset$weight # response variable (dependent)
# Step 3: Apply Linear Regression
relation <- lm(y ~ x)
# Step 4: Display Summary
summary(relation)
# Step 5: Predict New Values
new_x <- [Link](x = c(170, 165))
predicted_y <- predict(relation, new_x)
# Step 6: Display Predictions
print(predicted_y)
# Step 7: Plot the Data and Regression Line
plot(x, y, col = "blue", pch = 19,
xlab = "Height (cm)", ylab = "Weight (kg)",
main = "Linear Regression Example (CSV Data)")
abline(relation, col = "red", lwd = 2)
# Step 8: Plot Predicted Points
points(new_x$x, predicted_y, col = "yellow", pch = 19, cex = 1.5)
Explanation
Step 7: Plot the Data and Regression Line
Given two vectors, write an R program to predict the weight of new person using regression model.
Logistic Regression in R
Logistic Regression is a supervised machine learning algorithm used for classification problems. Unlike linear
regression which predicts continuous values it predicts the probability that an input belongs to a specific class.
Key Features
● Binary classification: Often used when the dependent variable has
two categories (e.g., pass/fail, spam/not spam, disease/no disease).
● Sigmoid function: Converts inputs into probabilities between 0 and
1.
● Applications: Widely used in areas like medical diagnosis, spam
detection, credit scoring, and social science research.
How does Logistic Regression work?
Logistic regression model transforms the linear regression function continuous value output into categorical value output using a
sigmoid function which maps any real-valued set of independent variables input into a value between 0 and 1. This function is
known as the logistic function.
What’s Happening
● The model learns a sigmoid
curve mapping study hours to
probability of passing.
● At low hours, probability is small
→ predicts Fail (0).
● At higher hours, probability
crosses 0.5 → predicts Pass
(1).
This shows how Logistic Regression
transforms a linear relationship into a
probabilistic classification.
Types of Logistic Regression
Q: Create a logistic Regression model to classify n observation based on probability function y= 2 + 2*x1
Start Set seed for reproducibility ([Link](123)).
Generate data:
a. Create 100 random values for x1.
b. Generate binary output y using a logistic function (plogis).
c. Combine x1 and y into a data frame.
Fit the model:
Use glm(y ~ x1, data = data, family = binomial).
Display model summary.
Predict new values:
Create new_data and use predict() to get probabilities.
Add predicted probabilities to the main data frame.
Plot results:
a. Plot data points (red for 1, blue for 0).
b. Draw logistic regression curve (green).
c. Add new predicted points (black).
d. Draw threshold line at 0.5 (gray).
e. Add legend.
Stop
Q. Given the dataset containing Name,Age, Estimated Salary, and Purchased (the target
variable), build a Regression model to predict which customers will buy the brand new
SUV based on their age and estimated salary. The target variable Purchased contains
binary values: 0 (customer did not buy the SUV) and 1 (customer bought the
SUV).Visualize all.
Polynomial Regression in R
Polynomial Regression is a form of linear regression where the relationship between the independent variable (x) and
the dependent variable (y) is modelled as an nth degree polynomial. It is useful when the data exhibits a non-linear
relationship allowing the model to fit a curve to the data.
Need for Polynomial Regression
● Non-linear Relationships: Polynomial regression is used when the relationship between the independent variable
(input) and dependent variable (output) is non-linear. Unlike linear regression which fits a straight line, it fits a
polynomial equation to capture the curve in the data.
● Better Fit for Curved Data: When a researcher hypothesizes a curvilinear relationship, polynomial terms are added to
the model. A linear model often results in residuals with noticeable patterns which shows a poor fit. It can capture these
non-linear patterns effectively.
● Flexibility and Complexity: It does not assume all independent variables are independent. By introducing
higher-degree terms, it allows for more flexibility and can model more complex, curvilinear relationships between
variables.
How does a Polynomial Regression work?
Polynomial regression is an extension of linear regression where higher-degree terms are added to model
non-linear relationships. The general form of the equation for a polynomial regression of degree n is:
Real-Life Example for Polynomial Regression
Let’s consider an example in the field of finance where we
analyze the relationship between an employee's years of
experience and their corresponding salary. If we check that the
relationship might not be linear, polynomial regression can be
used to model it more accurately.
1. Clearly, the salary growth is not linear — it accelerates after 4–5 years.
6. Visualization
# Plot data and fitted curve
plot(experience, salary, col="blue", pch=19, xlab="Years of Experience", ylab="Salary")
curve(predict(model, [Link](experience=x)), add=TRUE, col="red", lwd=2)
Poisson Regression in R
● Poisson regression is a type of Generalized Linear Model (GLM) used in R to model
count data (e.g., number of events, number of individuals) where the response
variable is a non-negative integer.
● It uses the glm() function in base R with the family = "poisson" argument.
● A Poisson Regression model is used to model count data and model response variables
(Y-values) that are counts. It shows which X-values work on the Y-value and more
categorically, it counts data: discrete data with non-negative integer values that count
something.
● It shows which explanatory variables have a notable effect on the response variable.
● Poisson Regression involves regression models in which the response variable is in the form
of counts and not fractional numbers.
Implementation in R
The primary function for Poisson regression is glm().
# General syntax
# glm(formula, family = poisson, data)
# Example using the built-in 'warpbreaks' dataset
# The goal is to predict the number of 'breaks' based on 'wool' type and 'tension' level.
# Load the dataset
data(warpbreaks)
# Fit the Poisson regression model
poisson_model <- glm(breaks ~ wool + tension, family = poisson, data = warpbreaks)
# View a summary of the model
summary(poisson_model)
Example:
● Approach:To understand how we can create:
○ We use the data set "warpbreaks".
○ Considering "breaks" as the response variable.
○ The wool "type" and "tension" are taken as predictor variables.
● Create Regression Model
Approach:Creating the poisson regression model:
○ Take the parameters which are required to make model.
○ let's use summary() function to find the summary of the model for data analysis.
When to Use Poisson Regression
Poisson regression is appropriate when the following assumptions are met:
● The response variable is a count per unit of time or space (non-negative integer).
● Observations are independent of one another.
Ridge Regression
Ridge Regression is a regularized version of linear regression that aims to address the problem of multicollinearity and
overfitting in linear models. It modifies the standard least squares loss function by adding a penalty term that is
proportional to the square of the magnitude of the coefficients (also called L2 norm).
Ridge Regression Line
A ridge regression line represents the linear relationship between predictors and the response, while shrinking large
coefficient estimates to stabilize the model. As lambda increases:
● Coefficient values shrink closer to zero.
● Model becomes more stable and less likely to overfit.
Assumptions of Ridge Regression
Ridge Regression assumes the following:
● Linear relationship: Between predictors and target.
● No perfect multicollinearity: It tolerates multicollinearity, but not exact correlation.
● Homoscedasticity: Constant error variance across predictors.
● Normal error terms: Residuals are normally distributed.
● Independent residuals: Errors are uncorrelated.
Lasso Regression
Lasso Regression is a linear modeling technique that uses L1 regularization to improve prediction accuracy and model
interpretability. By adding a penalty equal to the absolute values of the coefficients, it shrinks some of them to zero, effectively
performing feature selection and reducing model complexity, especially in high-dimensional data.
Key Characteristics
● Performs both regularization and feature selection.
● Suitable for high-dimensional data.
● Helps reduce variance while possibly increasing bias slightly.
● Works well when there are many correlated features.
Step-by-Step Implementation in R
To perform Lasso regression in R, you typically follow these steps:
1. Install and Load Packages:
You need the glmnet package for fitting the model and often the caret package for streamlined cross-validation and
hyperparameter tuning.
2. Prepare Data:The glmnet function requires the predictor variables (x) to be a numeric matrix and the response variable (y) to
be a vector. You may also need to standardize your data for better results.
3.
4. Fit the Final Model: Use the optimal value to train the final Lasso model on the entire dataset.
5. Examine Coefficients and Make Predictions: You can view which variables have non-zero coefficients (feature selection) and
use the model for predictions.
Model Fitting in R
● lm() function for linear regression
● glm() function for generalized linear models
● nls() function for nonlinear least squares
This fits a linear regression model predicting mpg from wt and hp in the mtcars dataset.
This fits a logistic
regression model
where the response vs
is binary.
Unusual Observations : unusual observations are data points that don’t behave like the rest of the dataset.
● Outliers
● High leverage points
● Influential observations
● Corrective measures (transformations, robust regression)
The point marked Outlier lies vertically far
away from the regression pattern, but it is
located in the middle range of X.
The point marked High leverage is far to the
right on the [Link] still follows the linear
pattern, but it is located at an extreme X
value.
The point marked Influential:
● has an extreme X value
● and does not follow the overall trend
Removing this point would noticeably change
the slope of the fitted line.