MODULE - 2

Understanding Data – 2:

2.6 Bivariate Data and Multivariate Data

Bivariate Data involves two variables deals with causes of relationships.

This results in finding relationship among data.
Ex. Data of temperature and sales of sweater

Aim is to find relationship among variables.

The relationship can be used for
 Comparison,
 Finding causes
 Future exploration
To find these the use of scatter graph will be useful
The Line graphs are similar to scatter plot

2.6.1 Bivariate Statistics

Covariance and Correlations are examples of bivariate statistics.

i. Covariance defines the joint probability of random variables.
It can be calculated as

E(X) is mean of X attribute, Similarly E(Y) is mean of Y attribute

ii. Correlation -
Correlation measures the strength and direction of a linear relationship between x and
y
The correlation indicates the relationship between dimensions using its sign. Sign is
important than value
If value is positive – dimensions increase together
If value is negative – one dimension increases and other dimension decreases
If Zero – Both dimensions are independent of each other
Person correlation coefficient is most commonly used to determine the association between
two phonomena.

2.7 Multivariate Statistics

In real world usually dataset is multivariable.

Multivariable data is analysis of more than two observable variables and often thousands of
multiple measurements need to be conducted for one or more subjects.
Ex.

Mean of multivariate data is mean vector {2, 5, 1.33}

The variance of multivariate data becomes the covariance matrix
The mean vector is called centroid and variance is called dispersion matrix.
Multivariate data has three or more variables.
The multivariable statistics can use
regression analysis,
factor analysis and
multivariate analysis of variance
Heatmap
Heatmap is a graphical representation of 2D matrix. It takes a matrix and inputs and colours
it. The darker colours indicates very larger values lighter indicate smaller values

PairPlot
Pairplot or scatter matrix is data visual technique for multivariate data.
 2.8 Essential Mathematics for Multivariate Data

ML involves many mathematical concepts from the domain of

- Linear algebra, - Statistics, - Probability and - Information theory.
Linear Algebra is branch of mathematics that is central for many scientific applications and
other mathematical subjects.
Linear algebra deals with linear equations, vector spaces and transformations

2.8.1 Linear Systems and Gaussian Elimination for Multivariate Data

A linear system of equation is a group of equations with unknow variables.

Let A . x=y then the solution x is given as : x=y/A x=y. A-1
This is true only y and A are not equal to 0.
The logic can be extended for N-set of equations with ‘n’ unknown variable

Computed as
If there is a unique solution, then system is called consistent independent.
If there are various solutions, then the system is called consistent dependent.
If there is no solutions and if equations are contradictory, then system is called
inconsistent

For solving large number of system equations, Gaussian elimination can be used.
Procedure is as follows

This is called as back ward substitution.

The following row operation are applied in Gaussian elimination method
i. Swapping the rows
ii. Multiplying or dividing a row by a constant
iii. Replacing a row by adding or substituting a multiple of another row to it

2.8.2 Matrix Decompositions

It is necessary to reduce a matrix to its constituent parts so that complex matrix operations
can be performed. These methods are also known as matrix factorization methods.
The most popular matrix decomposition is called eigen decomposition.
This is the way to reduce matrix with eigen values and eigen vectors.

LU Decomposition
One of the simplest matrix decomposition is LU decomposition where the matrix A can be
decomposed matrices

L is the lower triangular matrix, U is upper triangular matrix.

The decomposition can be done using Gausian elimination method as discussed.
First an identity matrix is augmented to the given matrix. Then, row operations and
Gaussian elimination is applied to reduce the given matrix to get matrices L and U
Consider the system of equations in three variables:
a11x1 + a12x2 + a13x3 = b1
a21x1 + a22x2 + a23x3 = b2
a31x1 + a32x2 + a33x3 = b3
These can be written in the form of AX = B as:

Here,

Now follow the steps given below to solve the above system of linear equations by LU
Decomposition method

Step 1: Generate a matrix A = LU such that L is the lower triangular matrix with principal
diagonal elements being equal to 1 and U is the upper triangular matrix.
That means,

and

Step 2: Now, we can write AX = B as

LUX = B………………………………………………………..(1)

Step 3: Let us assume UX = Y….(2)

Step 4: From equations (1) and (2), we have;
LY = B
On solving this equation, we get y1, y2, y3.

Step 5: Substituting Y in equation (2), we get UX = Y

By solving equation, we get X, x1, x2, x3.
The above process is also called the Method of Triangularisation.

Example:

Solve the system of equations x1 + x2 + x3 = 1, 3x1 + x2 – 3x3 = 5 and x1 – 2x2 – 5x3 = 10 by

LU decomposition method.
Solution:
Given system of equations are:
x1 + x2 + x3 = 1
3x1 + x2 – 3x3 = 5
x1 – 2x2 – 5x3 = 10
These equations are written in the form of AX = B as:

Step 1: Let us write the above matrix as LU = A.

That means,
By expanding the left side matrices, we get

Thus, by equating the corresponding elements, we get

u11 = 1, u12 = 1, u13 = 1
l21u11 = 3,
l21u12 + u22 = 1,
u21u13 + u23 = -3
l31u11 = 1,
l31u12 + l32u22 = -2,
l31u13 + l32u23 + u33 = -5
Solving these equations, we get;
u22 = -2, u23 = -6, u33 = 3
l21 = 3, l31 = 1, l32 = 3/2

Step 2: LUX = B
Step 3: Let UX = Y
Step 4: From the previous two steps, we have LY = B
Thus,

So,
y1 = 1
3y1 + y2 = 5
y1 + (3/2) y2 + y3 = 10
Solving these equations, we get;
y1 = 1, y2 = 2, y3 = 6
Step 5: Now, consider UX = Y. So,
By expanding this equation, we get;
x1 + x2 + x3 = 1
-2x2 – 6x3 = 2
3x3 = 6
Solving these equations, we can get;
x3 = 2, x2 = -7 and x1 = 6

2.8.3 Machine Learning and Importance of Probability and Statistics

ML is linked with probability and statistics similar to linear algebra, statistics and this is
treated as heart of ML.
The probability play in ML as it is assumed that data probability distribution.
The datasets consists of multiple data generated by many distribution therefore knowledge
of probability distribution and random variables is must for better understanding of
machine learning concepts

Probability Distribution
Probability distribution of variable X is summarized as probability associated with X’s
event. Distribution is parameterized mathematical function that describes relationship
between the observed sample space.
Consider a set of data and it is said follow a distribution if it obeys a mathematical function
that characterizes that distribution The function can be used to calculate the of individual
observation
Probability distributions are of two types:
1. Discrete probability distribution
2. Continuous probability distribution

Continuous probability distribution

The relationships between the events for a continious random variable and their
probabilities is called continious probabilty distribution. It is summarized as Probabilty
Density Function (PDF).
PDF calculates the probability of observing an instance. The plot of PDF shows the shape
of the distribution. Cummulative Distributive Function (CDF) computes the probability of
an obseration <= value.
The probabilty of an event cannot be detected directly. It shoiuld be computed as the area
under the curve for a small interval around the specific outcome. This is defined as CDF
Normal, Rectangular and Exponential distribution fall under this curve
1. Normal Distribution – is continious probbility distribution. This is also known as g
Continious Probability Distributionsaussian distribution or bell-shaped curve
distribution. It is most common distribution function. Here data tends to tends around
central value with no bia on left or right
Example: height of students, blood pressure of population, marks of students

Mostly, one uses the normal distribution curve of mean 0 and SD of 1. In normal
distribution, mean, median and mode are same. The distribution extends from to

One important concept associated with normal distribution is z-score. It can be

computed as

When Mu is zero and SD is 1 the Z-score is same as of x.

2. Rectangular Distribution
This is also know as uniform distribution. It has equal probabilities for all the values in
range a,b

3. Exponential Distribution
This is continious uniform distribution. This probability distribution is used to describe the
 time between the events in Poisson process. Exponential distribution is another special case
of Gamma distribution with fixed parameter of 1.

X is random variable
Lamda is called rate parameter

The mean and SD of exponential distribution is given by Beta , where Beta=1/Lamda

Discrete Distribution
Binomail, Poisson and Bernolli distribution fall under this category
1. Binomial Distribution – This distribution is most encourged in machine learning. It
has two outcomes either success or failure. This is called Bernoulli trial
The objective of this distribution is to find probability of getting success k out of n
trials.

The binomial distribution function is given as follows, where p is probability of

success and probability of failure is (1-p). The probability of success in a certain
number of trials is given as

Combining both bionomial distriution as

2. Poisson Distribution Given an interval of time, this distribution is used to model the
 probability of a given number of events k. The mean rule lamda is inclusive of
previous events.
Example: number of emails received, number of customers visiting the shop, The
number of phone calls received by office.

3. Bernoulli Distribution

2.10 Feature Engineering and Dimensionality Reduction Techniques

Feature Engineering is about determining the subset of features that from an important part of
input that improves the performance of model. The model may classify or any other model in
Machine Learning.

The feature engineering deals with two problems

i. Feature Selection – is extraction of feature and creating new features leading in
increase in performance Ex. Height and weight leading to body mass index (BMI)

ii. Feature transformation – Focuses on selection features to reduce the time but not at
the cost of reliability

The subset selection reduces the dataset size by removing irrelevant features and constructs
minimum set of attributes for machine learning.
For ‘n’ attributes there can be 2n possible subsets. If the value of ‘n’ is high, the problem
becomes intractable. This is called ‘curse of dimensionality’ hence need for the need of
reduction of attributes.
The feature subset selection is typically a graph search problem. These algorithms are based
on greedy strategy. In such algorithms the assumption is made that decision is made initially
and it is assumed that it leads to optimal solution.
Features can be removed based on two aspects

i. Feature Relevancy – Some features contribute more for classification that others
in building model. Example : mole on face is than nose for detection of face. The
feature should be relevant. The relevant features can be determined based on the
information such as mutual information, correlation based features like correlation
coefficient and distance measures
ii. Feature redundancy – There are some features present that are redundant. The
redundant feature should be taken off. Ex. In the database if there is date of birth
and age attributes, the age is redundant.

General procedure in Feature Engineering is

i. Generate all possible subsets

ii. Evaluate the subsets and model performance
iii. Evaluate the results for optimal feature selection

Approaches:

i. Filter based selection uses statistical measures for assessing features. This
approach doesn’t use any algorithms. Correlation and information gain
measures like mutual information and entropy are examples of this app
ii. Wrapper-based method use classifiers to identify the best features. These
identified features are selected and evaluated by learning algorithms. These are
computationally intensive but superior in performance.

Algorithms

Stepwise forward selection

Procedure starts with an empty set of attributes, Till a good reduced set of attributes are
obtained, every time an attribute is tested for statistical significance for best quality and is
added to reduced set.

2.10.1 Stepwise backword selection

 This procedure starts with a complete set of attributes. At every stage, the procedure
removes the worst attribute from the set leads to reduced set

The Combined Approach both forward and back selection methods so that the reduced set
has best attributes.

2.10.2 Principal Component Analysis

The idea of principal component analysis (PCA) or KL transform is to transform a given set
of measurements to new set of features so that the features exhibit high information packing
properties. This leads to a reduced and compact set of features. This uses elimination of
redundant information
/* Step by Step PCA (Example)
There are several steps involved while conducting a PCA. Let’s dive into it.
Step 1: Standardize the Data Set
Let’s say we have a data set with 4 variables and 4 observations, as shown below:

The first step is to standardize, which means transforming all variables as they will have
means of zeros and standard deviations of one, hence variances of one.
This is done to ensure that there is no imbalance in the contribution of the variables due to
unit differences. Otherwise, the variables that have higher variances would contribute more
than the ones with lower variances in identifying the principal components, although it does
not reflect reality. For further explanation, see the PCA Using Correlation & Covariance
Matrix tutorial.
Regarding standardization formula is given below:

z=Value−Mean(μ)StandardDeviation(σ)

For each feature, the mean and the standard deviation were as follows before the
standardization.

As you see, the variability does not vary much for this sample; hence standardization is not
a must in this case. However, for the sake of illustration, the variables were standardized,
and the following values they have taken.
Z=(x-mean)

Step 2: Calculate the Covariance Matrix

In this step, we want to observe the association between the variables in our data set.
Therefore, we will calculate the covariance matrix, which actually corresponds to the
correlation matrix as the variables are standardized. The following formula is used for the
computation.

Cov(x,y)=∑(xi–x¯¯¯)∗(yi–y¯¯¯)/N
The resulting covariance matrix is given below.

Positive covariance implies that the variable pair is positively related. In other words, when
the magnitude of one variable tends to increase (or decrease), the other does too.
Negative covariance implies that the variable pair is inversely related. In other words, when
the magnitude of one variable tends to increase, the other tends to decrease or vice versa.
In our case, only the variables x3 and x4 are negatively correlated, whereas the other
variables are positively correlated.
Step 3: Calculate the Eigenvalues and Eigenvectors of the Covariance Matrix
To determine principal components, we need eigenvectors and eigenvalues, which inform
us about the directions and the magnitude of the spread of our data. The first thing we need
to understand is that they always come in pairs: every eigenvector has an eigenvalue to
describe its magnitude.
As early stated, the “principal components” are the new variables that are formed via the
linear transformations of the original variables. Eigenvectors give the weights to be used in
this linear transformation and eigenvalues tell how much variance is explained by those
newly transformed variables.
Ranking our eigenvectors based on their eigenvalues, from the highest to the lowest, allows
us to select the principal components, which explain most of the variation in the dataset.
The following decomposition is employed for a (nonzero) vector v of dimension N for a N
× N square matrix A to compute the eigenvectors and eigenvalues:

//Aν=λν

In our case, solving the equation leads to the result below:

λ1 = 1.6698239685
λ2 = 1.0144883673
λ3 = 0.3151205250
λ4 = 0.0005671392
Now we can calculate the eigenvectors. The following result is obtained:

The eigenvalues are ranked in descending order as λ1, λ2, λ3, and λ4. Based on the result,
we can choose the top 2 eigenvectors:

For more information on how to select the ideal number of components, you can see our
tutorial.
Step 4: Recast the Data
Now we can reorient the data to the new axes: the ones represented by the principal
components, hence the original (standardized) variables can be expressed in terms of
principal component scores.
Regarding linear transformation is shown below:

The resulting transformed data is as follows:

By this example, it is shown how PCA allows us to reduce the dimensions of our data set
while keeping most of the valuable information.
*/

2.10.3 Linear Discriminant Analysis

Linear Discriminant Analysis (LDA) is reduction technique like PCA. Projects higher
dimension data to line (lower dimension data). LDA is used to classify the data.
Let there be 2 classes c1 and c2 and Mu1 and Mu2 are means of classes
The LDA is to optimize function

V is linear projection
2.10.4 Singular Value Decomposition
 Basic Learning Theory:
Design of Learning System, Introduction to Concept of Learning, Modelling in Machine
Learning.

3.1 Introduction to Learning and its types

Learning is process of acquiring knowledge and expertise through study, experience or

being taught
To make machine learn, there is a requirement to simulate the strategies of human
leaning Machine has limitation.
The machine can solve depends on the nature of problem that cpmuters can solve.

There are two kinds of problems

i. Well-posed problems ii. Ill-poised problems
Well -posed or defined problems have well defined specification and have following
component inheritant to it
i. Class of learning tasks (T)
ii. A measure of performance (P)
iii. A source of experience (E)

Mitchell – A program can learn from experience E, and performance P improves with
experience.

Let x be the input and X be the input space, which is set of all the inputs and Y be the
output space with the answers of ‘yes’ or ‘no’

Let D be the input dataset with examples, (x1,y1), (x2,y2), …(xn,yn) for n inputs

Let the unknown target function be f:XY that maps the input space to output space

The objective of learning program is to pick a function g:XY to approximate hypothesis

f.

All possible formulae form a hypothesis space.

Let H be the set of all formulae from which learning algorithm chooses.
The choice is good when the hypothesis ‘g’ replicates ‘f’ for all samples

Learning Model= Hypothesis Set + Learning Algorithm

Let us assume a problem of predicting a label for given input data.

Let D be the input dataset with both negative and positive examples

Let y be the out with class 0 or 1

The Learning model can be given as

This can be written into single equation as

Where x1, x2, ..xn are components of the input vector

w1,w2 …wn are the weights

+1 and -1 represent the class.

This simple model is called perceptron model. One can simplify this by making w0=b and
fixing it as 1 then we get

3.1.1 Classical and adoptive learning model

3.1.1 Classical and adoptive learning model

- A classical machine learning system

Includes the components such as Inputs, Process and Output.

The Input vlaues are taken from environment directly.

These values are processed and a hypothesis is generated as output model

This model is used for making prediction

The predicted values are consumed by environment

- Adaptive systems

System interacts with inputs for getting labelled data as direct inputs are not available.

The process is called reinforcement learning.

In reinforcement learning, a learning agent interacts with environment and in return gets
feedback.

Based on feedback, the learning agent generates input samples for learning, which are
used for generating the learning model

Such learning agents are not static and change their behaviour according to external
signal received from environment.

The feedback is known as reward and learning here is the ability of the learning agent
adapting to the environment based on the reward. These are the characteristics of an adaptive
system.

The following are the characteristics of an adoptive system

 1. Learn by Memorization or learn by repetition : also called as rote learning
achieved by memorizing without understanding the logic or concept.
2. Learn by example / Learn by experience : Previous knowledge acquired at some
time like finding analogy
3. Learn by being taught: generally called passive learning some expert systems will
help to identify the labels. Special type is active learning these interactively query
expert system to label the unlabelled data instances.
4. Learning by critical thinking : Also called as deductive learning, deduces new facts
or conclusion
5. Self Learning: called as reinforcement learning that learns from mistakes
(punishment) and rewards
6. Learning to solve problem: is cognitive learning where learning happens in mind
and is possible by devising a methodology to achieve a goal.
7. Learning by Generalizing explanations: also called explanation-based learning
(EBL). This exploits the domain knowledge from expert to improve the accuracy of
learned concepts by supervised learning

3.3 Design of a Learning System

A system that is built around a learning algorithm is learning system

The design of system focusses on the steps:

i. Choosing a training experience
ii. Choosing a target function
iii. Representation of target function
iv. Function approximation

3.3.1 Training Experience

Consider designing of chess game.

 In direct experience – individual board state and correct moves of the chess game are
given directly.
 In Indirect system – the move sequences and results are only given.
 The training experience depends on the presence of supervisor these labels all valid
moves for a board state.
  In absence of supervisor, the game agent plays against itself and learn the good moves
 If training distribution is good meaning that cover all scenarios then this is enough for
the performance computation.
 In this case result will be good

3.3.2 Determine the Target Function

The next step is the determination of target function.

Here the type of knowledge that needs to be learnt is determined

In Direct Experience- A board move is selected and is determined whether it is a good move
or not against all other moves.

If best move, then it is chosen as BM, where B and M are legal moves.

In Indirect experience – all legal moves are accepted and a score is generated for each

The move with highest score is chosen and executed.

3.3.3 Determine the Target Function Representation

The representation of knowledge may be a table, collection of rules or a neural network.

The linear combination of these factors can be coined as

V=w0 +w1x1+w2x2+w3x3

Where x1, x2 and x3 represent different board features and w0, w1 w2and w3 represent weights

Choosing an Approximation Algorithm for Target Function

The focus is to choose weights and fit the given training samples effectively.

The aim is to reduce the error is

Here b is the sample and V(cap) is predicted hypothesis. The approximation is carried out as

Computing the error as the difference between trained and expected hypothesis. Let error be
error(b).
Then, for every board feature xi, the weights are updated as

Wi= wi + Mu X errob(b) X xi

Here Mu is constant that moderates the size of the weight update

Thus the learning system has the following components

 A performance system to allow the game to play agaist itself

 A critic system to generate the samples.
 A generalizer system to generate a hypothesis based on samples.
 An Experiment system to generate a new system based on the currently learnt
function.

This is sent to as input to the performance system

3.4 Introduction to Concept Learning

Concept learning is a strategy of aquiring abstract knowledge or inferring a general concept

or deriving a category from given training samples.

It is a process of abstraction and generalization from the data

Concept learning helps to classify an object that has common and relevant feature

It helps to compare abd contrast categories based on similarity and association of positive and
negative instances in the training data to classify object.

The learner tries to aimplify by observing the common feature from training samples and then
apply simplified model to future samples.

This task is also known as learning from experience.

Each concept or category obtained by learning is a boolean valued function that takes a true
or false value.

Ex: humans can identify different kinds of animals based on common relevant features and
categorize all animals based on specific sets of feature.

The features that distiguish one animal from another called concept.

This way of learning categories for object and to recognize new instances of those categories
is called concept learning.
It is formally defined as inferring a Boolean valued function by processing training instances

Concept learning requires three things

i. Input – Training dataset which is set of training instances. Each one is labelled
ii. Output- Target concept ot function f. Mapping function f(x) from input x to
output y.
iii. Test – New instances to test the learned model.

Ex:

3.4.1 Representation of Hypothesis

A hypothesis ‘h’ approximates a target function ‘f’ to represrent the relationship between the
independent attributes and the dependent attribute of training instances

The hypothesis is the predicted approximate model that maps the inputs to outputs.

Each hypothesis is represented as a conjunction of attribute conditions in the antecedent part.

Ex: (Tail=short) ˄ (color=Black) ….

The set of hypothesis (h) in the search space is called as hypotheses (H)
3.4.2 Hypothesis Space

Hypothesis space is the set of all possible hypotheses that approximate the target function ‘f’

f(x) = {h1, h2, h3 …}

The ML algorithm will choose best possible hypothesis that describes trget function ‘f’ or
best fits the output

Every machine learning algorithm would represents a larger hypothesis space in a different
manner. Ex: regrssion- linear system, decision tree – tree

The set of hypotheses that can be generated by learning algorithm can be further reduced by
specifing a language bias

The subset of hypothesis space that consistent with all observed training instances is called as
Version Space.

Version Space represents the only hypotheses that are used for classification.

Dataset mentioned in table 7 variables takes 2 values and one variable takes 3 values so there
can be (2 X 2 X 2 X 2 X 2 X 3 X 2 X 2 ) = 384 instances can be drawn

If we add ? and ɸ for each of attribute we get 81290 instances

By eliminating ɸ we can have 8479 instances

The hypothesis space is much larger and hence we need efficient learning algorithm to search
for the best hypothesis from hypotheses

Hypothesis ordering is also important where in the hypotheses are ordered from most specific
to most general one.

3.4.4 Generalisation and Specialisation

h=<? ? ? ? ? ? ? ?>

I1 = {No Short, Yes, No, No, Black, No, Big> YES

h1 ={ <? ? ? ? ? ? ? ?> all greater than or equal to h

I2={ Yes Short No No No Brown Yes Medium > NO

h1={No, ?, No, }

3.4.5 Hypothesis Space Search by Find-S Algorithm

Limitations of Find-S algorithm

1. Considers only positive instances and if dataset is consistent then Find-S works well
2. Finds only one hyposthesis and there may exists more than one hypothesis consistent
with dataset
3. If dataset is erronious then hypothesis also will be not consistent

3.4.5 Version Spaces

Version Space is subset of Hypotheses. This uses various algorithm that outputs the set of
hypothesis that are consistant with dataset

[Link] List-then-Eliminate algorithm

[Link] Version Space and Candidate Elimination Algorithm

Version Space learning is to generate all consistent hypotheses around. This
algorithm computes the version space by combination of the two cases namely
i. Specific to General learning – Generalise S to include positive examples
ii. General to Specific learning – Specialise G to exclude the negative
example

The algorithm defines two boundaries

‘general boundary’ – Set of all hypothese that are most general

‘Specific boundary’ – Set of hypotheses that are most specific

This algorithm limits the version space for most gereric and most specific.

Algorithm

1. Initialise G, to most general hypotheses

2. Initialise S, to most specific hypotheses
3. For each subsequent new training instance
 If instance is positive
o Generalise S to include positive instance
 Check the attribute value of positive instance and S,
 If the attribute value of positive instance and S are
different fill the value with ?
 If the attribute value of positive instance and S are
same, then do not change
 o Prune G to exclude all inconstistent hypotheses in G with
positive example
 If instance is negative
o Specialise G to exclude the negative instance
 Add to G all minimal specialisation to exclude the
negative example and be consistent with S
 If the attribute value of S and the negative
instance are different then fill that attribute value
with S
 If the attribute value of S and negative instance
are same then no need to update G and fill that
attribute value with ?
o Remove from S all inconsistent hypothesis with
negative instance

Ex:

Initialise G to all ? S to ɸ

STEP 1: G= {
? ? ? ? ? ?
}
STEP 2: S = {
ɸ ɸ ɸ ɸ ɸ ɸ
}
Genraralise the initial hypothesis for the first positive instance I1
I1: >=9 Yes Excellent Good Fast Yes +ve
S1 = >=9 Yes Excellent Good Fast Yes
G1= ? ? ? ? ? ?
STEP 3:
Scan next subsequent instance I2 it is +ve instance
I2: >=9 Yes Good Good Fast Yes +ve
S2= >=9 Yes ? Good Fast Yes
Prune G1 to exclude all inconsistent hypotheses with positive instance, Here no change
G2= ? ? ? ? ? ?
Iteration 2: >=8 No Good Good Fast No -Ve
Since it is -ve instance
Specialise G to exclude the -ve but stay consistent with S2. So generate hypothesis for each
non-matching attribute value of S2 and fill with value of S2. The attributes 1, 2 and 6 differ
with S2 values so generate 3 hypothesis
G3 = < >=9 ? ? ? ? ?>
<? Yes ? ? ? ?>
<? ? ? ? ? Yes >
S3= < >=9 Yes ? Good Fast Yes >

Iteration 3:
Scan I4, it is +ve instance Check for mismatch in S3 with I4
I4= >=9 Yes Good Good slow No +ve
Change S3 tpo S4 as
S4 = < >=9 Yes ? Good ? ?>
Prune G3 to exclude all inconsistant hypothesis with +ve Instant I4
G3 < >=9 ? ? ? ? ?>
<? Yes ? ? ? ?>
<? ? ? ? ? Yes > 
inconsitant
Resulting G4 = < >=9 ? ? ? ? ?>
 <? Yes ? ? ? ?>
Conbining both G4 and S4 by generalising we get version space as
< >=9 Yes ? ? ? ?>
< >=9 ? ? Good ? ?>
<? Yes ? Good ? ?>

3.6 Modelling in Machine Learning

A machine learning model is an abstract of the training dataset that can perform a predictoin
of new data.
Training the model means feeding instances to the machine learning algorithm.
Training dataset are used to fit and tune the model.
After training a machine learning algorithm with training data, a predictive model is
generated as output to which a new data is fed to make predictions.
The process of modelling means training a machine learning algorithm with training dataset,
tuning it to increase performance, validating it and making predictions for a new unseen data.
The major concern in machine learning s what model to select,
i. how to train the model,
ii. time required to train
iii. the dataset to be used
iv. what performance to expect and so on
Learning parameter is the main goal in machine learning algorithm
There are two types of parameters
i. Model parameters – parameters directly from data like regession coefficient,
spliting of attributes in decision tree, weights and bias in neural network
ii. Hyper parameters – parameters are high leavel parameters which cannot be learnit
directly Ex. Regularisation lamda ƛ used in regularisation of regression, Number
of decision trees to include in random forest etc.,
Evaluating the selected machine learning model is also equally important as training the
model.
Dataset is split in to two sets called training and dataset and test dataset. The test dataset is
used to evaluate the model that is trained with dataset.
During prediction, an error occurs when the estimated output doesnot match with the true
output.
Training error is called in-sample error that is obtained at the time trainied samples are
applied on predicted model
The other error out-sample error is one that is drawn when testing data is applied on model
The error function or loss function is Mean Squared Error (MSE), which is average of
squared differences between the true values Yi and predicted values f(x) for an input value Xi
Smaller the value better the model

Machne learning Process

Four basic steps in machine learning are
i. Choose a machine learning algorithm to suit the training data and the problem domain
ii. Input the training dataset and train the machine learning algorithm to learn from the data
and capture the patterns in the data
iii. Tune the parameters of the model to improve the accuracy of learning of the algorithm
iv. Evaluate the learned model once the model is built

3.6.1 Model Selection and Model Evaluation

Challenge to choose algorithm that is suitable for problem.

Deals with two complexities
i. Model Performance : How well model performce on training dataset?
ii. Model Complexity: Complexity involved after training phase
Selecting algorithm of many ML algorithms.
Difficult as each algorithm exhibits predictive error for the problem
Some approaches used
i. Use Resample method and split the dataset as training, testing and validation
dataset. Observe the performance of the model over all phases  suitable for
smaller datasets
ii. Fit a model on training dataset and to compute measures like error or accuracy
iii. Use probabalistic framework and quantification of the performance of the model
as a score
Randomly chooses
[Link] Re-sampling Methods
instances in the dataset
Ex: Cross-Validation (K-Fold, LOOCV), Bootstrap
i. Holdout Method : Trined using training dataset and tested for the test dataset , can
opt the repeatation
ii. K-Fold cross validation: Made as K folds k-1 is trained and 1 fold is validated.
Randomly chooses instances in the dataset
iii. Leave-one-out-cross-Validation
3.6.1 Model Performance