Machine Learning Foundations

Professor. Harish Guruprasad Ramaswamy

Department of Computer Sciences & Engineering
Indian Institute of Technology, Madras
Data, Model and ML Task

(Refer Slide Time: 0:26)

Hello, everyone, and welcome to another video of foundations of Machine Learning. In this
part of the course, we will define some fundamental terms which have which are, what is data?
What is models and what do ML tasks mean? These are the some of the most important
terminologies that you will encounter when talking to people on machine learning.

(Refer Slide Time: 0:46)

The first question that must come to mind is, what is data? Because this is so prevalent in
machine learning and data science that it has the risk of running out of meaning. Data in its
most general sense can pretty much mean any collection of bits or bytes. But in the context of
machine learning, data will almost always mean a collection of vectors.

For example, let us say you have a bunch of houses, you have let us say you have 6 houses,
and each house is represented by a vector. The example house 1 is represented by a vector, a
four-dimensional vector, just 3,9,1.9,5.0. This is an example vector. And similarly now house
2, which is represented by the vector 2,7,2.1,3.2.

Similarly, you have 6 houses, each house is represented by a vector. And you have all of these
put together is a collection of vectors. And this is an example of a data or a data set. By itself,
this is sufficient for a computer to work with, for most reasons, for most purposes, but humans
require a little bit more understanding.

That is, for example, you are saying that house 1 is represented by this vector 3,9,1.9,,5.0, but
what does it even mean? So, this is just a bunch of numbers, but what does it mean in reality?
This information is typically contained in metadata. Metadata is information on the data. It is
self-referential in nature, because data already is information. So, metadata is information on
the information.

For example, in this particular case, it might contain information on what the numbers mean.
For example, it might say the first number refers to the number of rooms in the house, the
second number represents the number, the area in hundreds of square feet, the third number
represents distance to the nearest metro in kilometers, and the fourth number represents price
in tens of lakhs.

House, I mean, representing each house is a vector is abstract, but when you put that together
with a meta data, it tells a story. What this says is house 1 is a three-bedroom house with 900
square feet, at a distance of 1.9 kilometers from the metro and its price is 50 lakhs. For this
metadata makes essentially makes a data human interpretable, but as far as the computer is
concerned, it does not matter.

Because these meaning of these words really is not useful to the computer in any way. The
only thing that matters is that, it is consistent. That is, if you are using the number one and the
first number for representing number of rooms for house 1, you consistently keep doing that
for house 2, house 3 and house 4. As long as you do that the data is useful for the computer. It
does not need to know the information about these numbers, but this, the metadata is still useful
for humans to understand the data.

(Refer Slide Time: 3:56)

Now, next, the next main player in this, in the field of machine learning is a model. What is a
model? So, a model is, has lots of it is based in, it is in fact the core player in most of science.
And in fact, models have been used for thousands of years. For example, the most, the simplest
way to express a model to define a model is simply called a model, a mathematical
simplification of reality. It is almost always representing reality, but it is simpler than reality,
and it is more compact than really.

The most famous model perhaps, that all of you might have heard of is the is the Ideal Gas
model, which is essentially something on the order of pressure into volume is equal to number
of moles, × the gas constant × temperature. So, this is an equation that you might have seen in
your school. This is an equation that is satisfied by an ideal gas. In reality, there exists no ideal
gas, all gases have non-idealities, but it still captures an important aspect of a lot of gases to a
reasonable approximation that this forms a major progress in science.

The other example of a model is, the inverse square law for gravitational attraction that is the
1
Newton's formula, which says that the gravitational attraction force decays there R is the
R2

distance between the two bodies. This is also a model, it tries to model a very real physical
thing, which is gravitational attraction or a gravitational force in terms of mathematical
formula.
We know for a fact that it is not exact because it does not account for example, for the speed
of the object, it does not account for the curvature of space time and so on. It does not do that,
but despite its inaccuracies, it is still very useful that. The Newton's gravitational model is very
useful because it serves a purpose and explains a whole lot of reality, even if not all of reality.

And similarly, there are other laws, for example, the Moore's law for semiconductors simply
states that the number of semiconductors, number of transistors in a chip is going to double
every few months and so on. This is clearly not a physical law, it does not, it is not exact, but
the trend of that is ascribed to Moore's law is still useful, because it pretty concisely captures a
trend which held on for several decades. And models are actually the bread and butter in most
of economics.

The basic idea of economics everyone is aware of, that is, if you increase the supply of a
product, the price goes down, if you increase the demand of a product the price goes up, but
how much. If you increase the supply by 10 percent how much is the price going to go up? If
suddenly this product becomes popular among one group of people how much is the price
going to go up?

These are questions which you do not know the exact answer. And to answer these questions
exactly, economists regularly go to models, which are essentially mathematical simplification
of reality. They are not reality. Reality is not very nicely expressible in a few equations, but
equations can be studied, equations can be analyzed and you can plan with equations, but you
can never plan with the full reality.

So, there is this very famous statement attributed to George Box a famous statistician who said
that all models are wrong, but some are useful. So, no model is exact because models by
definition are simplifications, by models by definition are approximations. But despite their
approximations, some of them are useful. For example, ideal gas model or the gravitational,
Newton's gravitational model are very useful despite their approximation despite their errors,
because they give a sense of law, I mean, give a sense of order to that chaos around.

So, this is model in the method in the scientific or mathematical sense. In the machine learning
world, when we mean a model, it is related to the scientific model sense, but we mean it in a
different slightly different sense.
(Refer Slide Time: 8:11)

That there are two main types of models that are used in machine learning. They are the
predictive model and the probabilistic model. There are several types of predictive models. The
two main ones are the regression model and the classification model. There are several other
types of models, which we will not look into. Now we will see the example, I mean, what do
we mean by regression model and the classification model.

(Refer Slide Time: 8:41)

A regression model is a type of predictive model. What kind of thing does it do? Well, an
example of Regression Model would model the price of a house based on its area and distance
to metro. This is the goal of elevation model here. It models the price of a house based on its
area and distance to a metro. An example good model would be something like the price of a
house is 0.5 × area − distance to metro.

Do not worry too much about the coefficients, they do not matter. What matters is the trend
that is here, you have a situation where the price of the house increases with area and decreases
with distance to metro, which is a reasonable approximation. So, the price of a house will
increase as the horse gets bigger, but it will get cheaper as you move farther and farther away
from the city, which is approximated by distance to metro.

And clearly you can see this is not a law of physics or this is not always solved, means not
always. There are going to be a lot of exceptions to this rule, but despite the exceptions, this is
a general rule this is meaningful. You can, if you are looking for a cheap house, you have two
options now. You can either look for a reasonably big house far away from the city or super
small house in the middle of the city. This a kind of broad conclusions are possible with the
model. With reality, you can never do that.

With reality if you are looking for a reality, you will just have to look at all prices of all the
houses and then have to make a decision. But with a model you can have a good sense of what
to look for. Let us say you have a budget and you can decide, okay, with this budget I can either
look for a large house far away or a small house close by. That is the power of a model.

And a regression model. Why is it called a regression model? Well, regression models predict
a real valued, price is a real value. It lives on a continuum. It can be 5 lakhs, it can be 5 lakh
50,000, it can be 5 lakh 50,223. So, a price it lives on a continuum, it is a, when you want to
predict a real valued variable what you need is a regression model. And regression model is a
type of a predictive model because it predicts the price.

For example, if someone says I have this house, which is 700 square feet at a distance of three
kilometers from the metro, you can use this model to predict the price of this new house. Even
though you have no, you do not have any information on its price, you do have information on
its area and the distance of the house from the metro, you can use that to your advantage along
with a good model to predict the price of this house. So, that is why it is called a predictive
model. You can use the model to make predictions on unknown data on unseen data at least.
(Refer Slide Time: 11:33)

The other type of predictive model is a classification model. A classification model is very
similar to a regression model. The only difference is that the variable that you are predicting is
no longer real value, the variable that you are predicting is discrete in nature. For example, you
might want to predict whether a house is closer than 2 kilometers to a metro based on price and
area. So, that is your, you want, you are only interested in two possible outputs, that is, it is
closer than two kilometers to a metro or farther than two kilometers to a metro. So, that is the
only two things that you are interested in.

An example good model might be you might say, if (2 × rooms – price) < 1, then it is close
otherwise, it is far. So, what is it really telling? It says that if the number of rooms is small, and
the price is high, so that is what this is saying. If you have a small, you have like a one-bedroom
apartment costing 1 crore rupees, it is quite likely that it is close to the city. So, that is what this
model is saying.

And if not, it is perhaps far away from the city. So, in this case, you can see that the variable
that you are predicting has only two possible values, which is either close or far. Unlike the
case of the price, which you are predicting a real valued number. A price is a real valued number
just printing whether it is close or far is just there are only two possible options. That is why it
is called, such models are called classification models.
(Refer Slide Time: 13:00)

Now we'll move to what are probabilistic models. Probabilistic models are slightly different
from predictive models, that is, their goal is not to predict the future. You can use both
classification models and regression models to predict on future data. For example, if someone
gives a new house, the details of a new house, you can use their number of rooms and the price
to predict whether it is close to a metro or not. Even though you have not used, not seeing that
house before you can use the classification model to make a prediction.

Probabilistic models, on the other hand are not tuned towards making a prediction. Their goal
is slightly different. Their goal is to evaluate how likely a certain event or a configuration is.
For example, someone might pick a randomly chosen person from India, and they can ask the
question, what is the probability that a randomly chosen person is in the latitude longitude
given by 25 ° north and 30 ° east.

The probability of this 25 ° north and the degree is east let us say if it is in the middle of the
Sahara Desert is going to be quite low, and if this is in the middle of Bombay, it is going to be
quite high. Based on different configurations of the latitude longitude, some latitude, longitude
are much more likely to have a randomly chosen person rather than other latitude longitudes.

So, this essentially scores reality. Probability models essentially score reality, and you can give
any configuration of reality. For example, 25 ° north and 30 ° east, and it can say how likely is
this particular reality? Another example might be, for example, you might give a tweet and ask
the question, what is the probability that a given tweet was generated by Mister. Chopra? So,
that is the question that they can ask.
And there is, the broad idea is that tweets that are similar to tweets that Mister. Chopra would
tweet would get a high score, and tweets that are completely random collection of digits and
numbers, digits and characters are likely to have a low score. So, a probability model simply
scores different configurations and it has other constraints also, but that is the broad again, you
should take.

That is, if you have a probabilistic model will effectively score different configurations of
reality. So, now we come to the major player, another major player of machine learning, which
is a learning algorithm.

(Refer Slide Time: 15:36)

What is a learning algorithm? A learning algorithm is what converts data into models? In most
machine learning tasks, what you are interested in is, is getting a model. And the learning
algorithms are the tools that learning algorithms are the things that convert data into models.

And how does it do that? Well, typically, they choose from a collection of models. There is a
whole bunch of models, and learning algorithms consider all these models. And among these
models, they give the best model. The typical way it has done is, it chooses from a collection
of models with the same structure, but different parameters.

For example, let us say, you are you are building a learning algorithm for predicting the house
of, price of the house. You can, before you even start, before you even look at any data, you
can say, well, I am going to decide that my price of the house is A × the area of the house, + B
× the number of rooms in the house, + C × the distance to metro.
Before you even see any data, you can just decide this in advance. You do not know A, B and
C. But before even looking at any single data you can decide this. And what the learning
algorithm does is, it takes a look at all the data and then decides the correct values of A, B and
C. And these A, B and C are called parameters that represent or that characterize the model, so
that is a learning curve.

(Refer Slide Time: 17:12)

Now, we can actually put all of this in context with our task hierarchy. In the machine learning
task hierarchy, we said that the tool which transforms input to output is not designed directly
by the human but rather the human, rather only gives a broad outline to a tool design, which in
turn uses data to create to tool.

Now, we have seen enough terminology of machine learning to actually give more precise
names. In our case, the tool is simply the model. The model, for example, let us say the price
of a house prediction model that will take a new house and it will predict a price. So, this is a
house and price. But what machine learning does is you do not, the human does not directly
write the model. It does not say the price of the house is 3 × 1 square feet − 5 × distance to
metro, he does not directly write that down.

The human simply gives broad outlines, and the learning algorithm uses past data of several
houses and prices and decides the right model. It builds the right model with a guideline from
a human and data of several houses and it constructs the model from such data. So, now, this
particular block diagram should give an idea and ground several ideas that several abstract
ideas that we expressed when we add tasks into machine learning terminology.
(Refer Slide Time: 19:04)

With that, I think we can wrap this part of the lecture, and the next part we will give concrete
examples on more details of supervised learning tasks and unsupervised learning tasks.