Machine Learning

Shipra Shukla
Manipal University,
Jaipur
 Introduction
• Machine learning is a sub area of Artificial Intelligence (AI).

• It enables a machine to automatically learn from data, improve

performance from experiences, and predict things without being
explicitly programmed.

Shipra Shukla, Manipal University, Jaipur

 Applications
• Currently, machine learning is used in various domains such as self-driving
cars, cyber fraud detection, face recognition, and friend suggestion by
Facebook, etc.
• Various top companies such as Netflix and Amazon have build machine
learning models that are using a vast amount of data to analyze the user
interest and recommend product accordingly.
• Most industries and businesses working with massive amounts of data have
recognized the value of machine learning technology.
Shipra Shukla, Manipal University, Jaipur
Shipra Shukla, Manipal University, Jaipur
 Supervised Machine Learning
• Supervised machine learning is based on supervision. It means in the
supervised learning, we train the machines using the "labelled" dataset, and
based on the training, the machine predicts the output.

• Here, the labelled data specifies that the inputs are mapped to the output.

• The main goal of the supervised learning technique is to map the input
variable(x) with the output variable(y).
Shipra Shukla, Manipal University, Jaipur
Supervised machine learning can be classified into two types of problems:

a) Classification

b) Regression

Shipra Shukla, Manipal University, Jaipur

 Classification
• Classification algorithms are used to identify the category of new
observations on the basis of training data.
• The output variable is categorical, such as "Yes" or No, Male or Female,
Red or Blue, etc.
• Some real-world examples of classification algorithms are Spam
Detection, Email filtering, etc.

Shipra Shukla, Manipal University, Jaipur

 Regression
• Regression algorithms are used to solve prediction or regression
problems.

• These are used to predict continuous output variables, such as market

trends, weather prediction, etc.

Shipra Shukla, Manipal University, Jaipur

 Unsupervised Machine Learning
• As its name suggests, there is no need for supervision. It means, in
unsupervised machine learning, the machine is trained using the
unlabelled dataset.

• The main aim of the unsupervised learning algorithm is to group or

categories the unsorted dataset according to the similarities, patterns,
and differences.

Shipra Shukla, Manipal University, Jaipur

Unsupervised Learning can be further classified into two types of problems:
• Clustering- groups unlabelled data based on their similarities or
differences.

• Association- finds relationship between variables in a given dataset.

• Dimensionality Reduction- reduces the number of features or

dimensions of data to a manageable size while also preserving the
integrity of the dataset as much as possible.
Shipra Shukla, Manipal University, Jaipur
 Reinforcement Learning
• Reinforcement learning is a type of machine learning algorithm that enables
an agent to learn in an interactive environment by trial and error using
feedback from its own actions and experiences.

Shipra Shukla, Manipal University, Jaipur

 Parametric Models
• A learning model that summarizes data with a set of fixed-size parameters (independent
on the number of instances of training).

• Parametric machine learning algorithms optimize the function to a known form.

• Some more examples of parametric machine learning algorithms include:

1) Logistic Regression
2) Linear Discriminant Analysis
3) Perceptron
Shipra Shukla, Manipal University, Jaipur
 Parametric Models
The algorithms involve two steps:
1. Select a form for the function.
2. Learn the coefficients for the function from the training data.

An easy to understand functional form for the mapping function is a line, as is used in
linear regression:
• b0 + b1*x1 + b2*x2 = 0
Where b0, b1 and b2 are the coefficients of the line and x1 and x2 are two input
variables.
Shipra Shukla, Manipal University, Jaipur
 Parametric Models
Pros
• These models are simpler, fast and do not require much training data.

Cons
• More suited to simpler problems.
• Highly Constrained.
Shipra Shukla, Manipal University, Jaipur
 Non-Parametric Models
• Nonparametric machine learning algorithms are those which do not make specific
assumptions about the type of the mapping function.

• Nonparametric methods are good when you have a lot of data and no prior knowledge.

• Some more examples of popular nonparametric machine learning algorithms are:

1) k-Nearest Neighbors
2) Decision Trees like CART and C4.5
3) Support Vector Machines
Shipra Shukla, Manipal University, Jaipur
 Non-Parametric Models
Pros
• Capable of fitting a large number of functional forms.
• No assumptions (or weak assumptions) about the underlying function.
• Can result in higher performance models for prediction.
Cons
• Requires more data.
• Slower to train.
• Risk of overfitting.
Shipra Shukla, Manipal University, Jaipur
 Bias – variance trade-off

• In machine learning, the class value assigned by the learning model built
based on the training data may differ from the actual class value. This error
in learning can be of three types – errors due to ‘bias’, error due to ‘variance’
and irreducible errors.

Shipra Shukla, Manipal University, Jaipur

 Errors due to ‘Bias’
• Errors due to bias arise from simplifying assumptions made by the model to
make the target function less complex or easier to learn.

• Parametric models generally have high bias making them easier to

understand/interpret and faster to learn.

Shipra Shukla, Manipal University, Jaipur

 Errors due to ‘Variance’
• Errors due to variance occur from difference in training data sets used to
train the model.

• The model with high variance has a very complex fit to the training data and
thus is not able to fit accurately on the testing data.

• Non-parametric algorithms demonstrate low bias and high variance.

Shipra Shukla, Manipal University, Jaipur

 Bias – variance trade-off
• The best solution is to have a
model with low bias as well as low
variance.
• However, that may not be possible
in reality.
• Hence, the goal of supervised
machine learning is to achieve a
balance between bias and variance.

Shipra Shukla, Manipal University, Jaipur

 Underfitting
• If the target function is kept too simple, it may not be able to capture the
essential qualities and represent the underlying data well.

• A typical case of underfitting may occur when trying to represent a non-

linear data with a linear model.

Shipra Shukla, Manipal University, Jaipur

 Overfitting
• Overfitting refers to a situation where the model has been designed in such a
way that it emulates the training data too closely.
• In such a case, any specific deviation in the training data, like noise or
outliers, gets embedded in the model. It adversely impacts the performance
of the model on the test data.
• Overfitting, in many cases, occur as a result of trying to fit an excessively
complex model to closely match the training data.

Shipra Shukla, Manipal University, Jaipur

Shipra Shukla, Manipal University, Jaipur
 Model Selection
• Multiple factors play a role when we try to select the model for solving a machine
learning problem.

• The most important factors are :

(i) the kind of problem we want to solve using machine learning and
(ii) the nature of the underlying data.

• In other words, there is no one model that works best for every machine learning
problem. This is what ‘No Free Lunch’ theorem also states.
Shipra Shukla, Manipal University, Jaipur
 Model Selection
• The problem may be related to the prediction of a class value like whether
the next day will be snowy or rainy, etc.
• It may be related to prediction of some numerical value like what the price
of a house should be in the next quarter, what is the expected growth of a
certain IT stock in the next 7 days, etc.
• Certain problems are related to grouping of data like finding customer
segments that are using a certain product, movie genres which have got more
box office success in the last one year, etc.
Shipra Shukla, Manipal University, Jaipur
 Model Selection
There are two main classes of techniques to approximate the ideal case of model
selection:

• Resampling Methods
• Probabilistic Measures

These methods judge a model based two factors:

• Model Performance
• Model Complexity
Shipra Shukla, Manipal University, Jaipur
 Resampling Methods
• Data is resampled into train/test followed by training on train and evaluation
on test set.
• Model chosen from this technique is assessed based on performance, not the
model complexity.
• Performance is computed on out-of-sample data.

• Example: Random train/test splits, Cross-Validation (k-fold, LOOCV, etc.),

Bootstrap.
Shipra Shukla, Manipal University, Jaipur
 Probabilistic Measures
• Probabilistic model selection uses statistical methods.

• It involve analytically scoring a candidate model using both its performance

on the training dataset and the complexity of the model.

• Performance is computed on in-sample data which means test set is not

required and the score is computed on whole train data directly.
Shipra Shukla, Manipal University, Jaipur
 Probabilistic Measures
Some commonly used probabilistic model selection measures include:
• Akaike Information Criterion (AIC), Bayesian Information Criterion (BIC),
Minimum Description Length (MDL) etc.

Shipra Shukla, Manipal University, Jaipur

 Akaike Information Criterion (AIC)
The name of method is given on the name of statistician, Hirotugu Akaike.
An AIC score is a number used to determine which machine learning model is best for
a given data set in situations where one can’t easily test a data set. An AIC test is most
useful when you’re working with a small data set or time series analysis. The lower the
AIC score the better.

• K = number of independent variables or predictors

• L = maximum-likelihood of the model
• N = number of data points in the training set
Shipra Shukla, Manipal University, Jaipur
 Bayesian Information Criterion (BIC)
• BIC was derived from the Bayesian probability concept.
• The below formula is used to calculate the score:

• K = number of independent variables

• L = maximum-likelihood
• N = Number of sampler/data points in the training set

Shipra Shukla, Manipal University, Jaipur

 Cost Function
• A cost function (or error function) helps to measure the extent to which the
model is going wrong in estimating the relationship between X and Y.

• Cost function is a measure of how well a machine learning model predicts

the actual outcomes for a given data set.

• Different types of cost functions are used for different types of machine
learning problems, such as regression, classification, or clustering.
Shipra Shukla, Manipal University, Jaipur
 Cost Function
Some common cost functions are as follows:
• Mean Error (ME)
• Mean Squared Error (MSE)
• Mean Absolute Error (MAE)
• Cross Entropy Cost Function

Shipra Shukla, Manipal University, Jaipur