NAVIGATION for this section:

Chapter 4: Machine Learning

It has been long understood that learning is a key element of intelligence. This holds both for
natural intelligence - we all get smarter by learning - and artificial intelligence.

In Chapter 4, we will cover the following sections. Click below to get started:

Section - Exercises
I. The types of machine learning
II. The nearest neighbor classifier
III. Regression

Please join the Elements of AI community to discuss and ask questions about this chapter.

Start of section 1:

I. The types of machine learning

Handwritten digits are a classic case that is often used when discussing why we use machine
learning, and we will make no exception.

Below you can see examples of handwritten images from the very commonly used MNIST
dataset.

Image: MNIST dataset

The correct label (what digit the writer was supposed to write) is shown above each image. Note
that some of the "correct” class labels are questionable: see for example the second image from
left: is that really a 7, or actually a 4?

Note

MNIST – What’s that?

Every machine learning student knows about the MNIST dataset. Fewer know what the
acronym stands for. In fact, we had to look it up to be able to tell you that the M stands
for Modified, and NIST stands for National Institute of Standards and Technology. Now
you probably know something that an average machine learning expert doesn’t!
In the most common machine learning problems, exactly one class value is correct at a time.
This is also true in the MNIST case, although as we said, the correct answer may often be hard
to tell. In this kind of problem, it is not possible that an instance belongs to multiple classes (or
none at all) at the same time. What we would like to achieve is an AI method that can be given
an image like the ones above, and automatically spits out the correct label (a number between 0
and 9).

Note

How not to solve the problem

An automatic digit recognizer could in principle be built manually by writing down rules
such as:
if the black pixels are mostly in the form of a single loop then the label is 0
if the black pixels form two intersecting loops then the label is 8
if the black pixels are mostly in a straight vertical line in the middle of the figure then the
label is 1 and so on...

This was how AI methods were mostly developed in the 1980s (so called “expert
systems”). However, even for such a simple task as digit recognition, the task of writing
such rules is very laborious. In fact, the above example rules wouldn’t be specific
enough to be implemented by programming – we’d have to define precisely what we
mean by “mostly”, “loop”, “line”, “middle”, and so on.

And even if we did all this work, the result would likely be a bad AI method because as
you can see, the handwritten digits are often a bit so-and-so, and every rule would need
a dozen exceptions.

Three types of machine learning

The roots of machine learning are in statistics, which can also be thought of as the art of
extracting knowledge from data. Especially methods such as linear regression and Bayesian
statistics, which are both already more than two centuries old (!), are even today at the heart of
machine learning. For more examples and a brief history, see the timeline of machine learning
(Wikipedia).

The area of machine learning is often divided in subareas according to the kinds of problems
being attacked. A rough categorization is as follows:

Supervised learning: We are given an input, for example a photograph with a traffic sign, and
the task is to predict the correct output or label, for example which traffic sign is in the picture
(speed limit, stop sign, etc.). In the simplest cases, the answers are in the form of yes/no (we
call these binary classification problems).

Unsupervised learning: There are no labels or correct outputs. The task is to discover the
structure of the data: for example, grouping similar items to form “clusters”, or reducing the data
to a small number of important “dimensions”. Data visualization can also be considered
unsupervised learning.​
​
Reinforcement learning: Commonly used in situations where an AI agent like a self-driving car
must operate in an environment and where feedback about good or bad choices is available
with some delay. Also used in games where the outcome may be decided only at the end of the
game.

The categories are somewhat overlapping and fuzzy, so a particular method can sometimes be
hard to place in one category. For example, as the name suggests, so-called semisupervised
learning is partly supervised and partly unsupervised.

Note

Classification
When it comes to machine learning, we will focus primarily on supervised learning, and
in particular, classification tasks. In classification, we observe in input, such as a
photograph of a traffic sign, and try to infer its “class”, such as the type of sign (speed
limit 80 km/h, pedestrian crossing, stop sign, etc.). Other examples of classification tasks
include: identification of fake Twitter accounts (input includes the list of followers, and the
rate at which they have started following the account, and the class is either fake or real
account) and handwritten digit recognition (input is an image, class is 0,...,9).

Image: teaching a robot

Humans teaching machines: supervised learning

Instead of manually writing down exact rules to do the classification, the point in supervised
machine learning is to take a number of examples, label each one by the correct label, and use
them to “train” an AI method to automatically recognize the correct label for the training
examples as well as (at least hopefully) any other images. This of course requires that the
correct labels are provided, which is why we talk about supervised learning. The user who
provides the correct labels is a supervisor who guides the learning algorithm towards correct
answers so that eventually, the algorithm can independently produce them.

In addition to learning how to predict the correct label in a classification problem, supervised
learning can also be used in situations where the predicted outcome is a number. Examples
include predicting the number of people who will click a Google ad based on the ad content and
data about the user’s prior online behavior, predicting the number of traffic accidents based on
road conditions and speed limit, or predicting the selling price of real estate based on its
location, size, and condition. These problems are called regression. You probably recognize the
term linear regression, which is a classical, still very popular technique for regression.

​
 Note

Example
Suppose we have a data set consisting of apartment sales data. For each purchase, we
would obviously have the price that was paid, together with the size of the apartment in
square meters (or square feet, if you like), and the number of bedrooms, the year of
construction, the condition (on a scale from “disaster“ to “spick and span”). We could
then use machine learning to train a regression model that predicts the selling price
based on these features. See a real-life example here.

Image: houses

Caveat: careful with that machine learning algorithm

There are a couple potential mistakes that we’d like to make you aware of. They are related to
the fact that unless you are careful with the way you apply machine learning methods, you could
become too confident about the accuracy of your predictions, and be heavily disappointed when
the accuracy turns out to be worse than expected.

The first thing to keep in mind in order to avoid big mistakes, is to split your data set into two
parts: the training data and the test data. We first train the algorithm using only the training
data. This gives us a model or a rule that predicts the output based on the input variables.

To assess how well we can actually predict the outputs, we can’t count on the training data.
While a model may be a very good predictor in the training data, it is no proof that it can
generalize to any other data. This is where the test data comes in handy: we can apply the
trained model to predict the outputs for the test data and compare the predictions to the actual
outputs (for example, future apartment sale prices).

Note

Too fit to be true! Overfitting alert

It is very important to keep in mind that the accuracy of a predictor learned by machine
learning can be quite different in the training data and in separate test data. This is the
so-called overfitting phenomenon, and a lot of machine learning research is focused on
avoiding it one way or another. Intuitively, overfitting means trying to be too smart. When
predicting the success of a new song by a known artist, you can look at the track record
of the artist’s earlier songs, and come up with a rule like “if the song is about love, and
includes a catchy chorus, it will be top-20”. However, maybe there are two love songs
with catchy choruses that didn’t make the top-20, so you decide to continue the rule
“...except if Sweden or yoga are mentioned” to improve your rule. This could make your
rule fit the past data perfectly, but it could in fact make it work worse on future test
data.

Machine learning methods are especially prone to overfitting because they can try a
huge number of different “rules” until one that fits the training data perfectly is found.
Especially methods that are very flexible and can adapt to almost any pattern in the data
can overfit unless the amount of data is enormous. For example, compared to quite
 restricted linear models obtained by linear regression, neural networks can require
massive amounts of data before they produce reliable prediction.

Learning to avoid overfitting and choose a model that is not too restricted, nor too flexible, is one
of the most essential skills of a data scientist.

Learning without a teacher: unsupervised learning

Above we discussed supervised learning where the correct answers are available, and the task
of the machine learning algorithm is to find a model that predicts them based on the input data.

In unsupervised learning, the correct answers are not provided. This makes the situation quite
different since we can’t build the model by making it fit the correct answers on training data. It
also makes the evaluation of performance more complicated since we can’t check whether the
learned model is doing well or not.

Typical unsupervised learning methods attempt to learn some kind of “structure” underlying the
data. This can mean, for example, visualization where similar items are placed near each other
and dissimilar items further away from each other. It can also mean clustering where we use
the data to identify groups or “clusters” of items that are similar to each other but dissimilar from
data in other clusters.

​
​ Note

Example
As a concrete example, grocery store chains collect data about their customers’
shopping behavior (that’s why you have all those loyalty cards). To better understand
their customers, the store can either visualize the data using a graph where each
customer is represented by a dot and customers who tend to buy the same products are
placed nearer each other than customers who buy different products. Or, the store could
apply clustering to obtain a set of customer groups such as ‘low-budget health food
enthusiasts’, ‘high-end fish lovers’, ‘soda and pizza 6 days a week’, and so on. Note that
the machine learning method would only group the customers into clusters, but it
wouldn’t automatically generate the cluster labels (‘fish lovers’ and so on). This task
would be left for the user.

Yet another example of unsupervised learning can be termed generative modeling. This has
become a prominent approach over the last few years as a deep learning technique called
generative adversarial networks (GANs) has lead to great advances. Given some data, for
example, photographs of people’s faces, a generative model can generate more of the same:
more real-looking but artificial images of people’s faces.

We will return to GANs and the implications of being able to produce high-quality artificial image
content a bit later in the course, but next we will take a closer look at supervised learning and
discuss some specific methods in more detail.
Start of section 2:

II. The nearest neighbor classifier

The nearest neighbor classifier is among the simplest possible classifiers. When given an item
to classify, it finds the training data item that is most similar to the new item, and outputs its
label. An example is given in the following diagram.

Image: training sets

In the above diagram, we show a collection of training data items, some of which belong to one
class (green) and other to another class (blue). In addition, there are two test data items, the
stars, which we are going to classify using the nearest neighbor method.

The two test items are both classified in the “green” class because their nearest neighbors are
both green (see diagram (b) above).

The position of the points in the plot represents in some way the properties of the items. Since
we draw the diagram on a flat two-dimensional surface – you can move in two independent
directions: up-down or left-right – the items have two properties that we can use for comparison.
Imagine for example representing patients at a clinic in terms of their age and blood-sugar level.
But the above diagram should be taken just as a visual tool to illustrate the general idea, which
is to relate the class values to similarity or proximity (nearness). The general idea is by no
means restricted to two dimensions and the nearest neighbor classifier can easily be applied to
items that are characterized by many more properties than two.

What do we mean by nearest?

An interesting question related to (among other things) the nearest neighbor classifier is the
definition of distance or similarity between instances. In the illustration above, we tacitly
assumed that the standard geometric distance, technically called the Euclidean distance, is
used. This simply means that if the points are drawn on a piece of paper (or displayed on your
screen), you can measure the distance between any two items by pulling a piece of thread
straight from one to the other and measuring the length.

Note

Defining “nearest”
Using the geometric distance to decide which is the nearest item may not
always be reasonable or even possible: the type of the input may, for example,
be text, where it is not clear how the items are drawn in a geometric
representation and how distances should be measured. You should therefore
choose the distance metric on a case-by-case basis.
In the MNIST digit recognition case, one common way to measure image similarity is to count
pixel-by-pixel matches. In other words, we compare the pixels in the top-left corner of each
image to one another and if the more similar color (shade of gray) they are, the more similar the
two images are. We also compare the pixels in the bottom-right corner of each image, and all
pixels inbetween. This technique is quite sensitive to shifting or scaling the images: if we take an
image of a “1” and shift it ever so slightly either left or right, the outcome is that the two images
(before and after the shift) are very different because the black pixels are in different positions in
the two images. Fortunately, the MNIST data has been preprocessed by centering the images
so that this problem is alleviated.

Image: music recommendations

Using nearest neighbors to predict user behavior

A typical example of an application of the nearest neighbor method is predicting user behavior
in AI applications such as recommendation systems.

The idea is to use the very simple principle that users with similar past behavior tend to have
similar future behavior. Imagine a music recommendation system that collects data about users’
listening behavior. Let’s say you have listened to 1980s disco music (just for the sake of
argument). One day, the service provider gets their hands on a hard-to-find 1980 disco classic,
and adds it into the music library. The system now needs to predict whether you will like it or not.
One way of doing this is to use information about the genre, the artist, and other metadata,
entered by the good people of the service provider. However, this information is relatively scarce
and coarse and it will only be able to give rough predictions.

What current recommendation systems use instead of the manually entered metadata, is
something called collaborative filtering. The collaborative aspect of it is that it uses other users’
data to predict your preferences. The word “filter” refers to the fact that you will be only
recommended content that passes through a filter: content that you are likely to enjoy will pass,
other content will not (these kind of filters may lead to the so called filter bubbles, which we
mentioned in Chapter 1. We will return to them later).

Now let’s say that other users who have listened to 80s disco music enjoy the new release and
keep listening to it again and again. The system will identify the similar past behavior that you
and other 80s disco fanatics share, and since other users like you enjoy the new release, the
system will predict that you will too. Hence it will show up at the top of your recommendation list.
In an alternative reality, maybe the added song is not so great and other users with similar past
behavior as yours don’t really like it. In that case, the system wouldn’t bother recommending it to
you, or at least it wouldn’t be at the top of the list of recommendations for you.

The following exercise will illustrate this idea.

Exercise 14
Image: customer behaviour

In the above example, we only had six users’ data and our prediction was probably very
unreliable. However, online shopping sites often have millions of users, and the amount of data
they produce is massive. In many cases, there are a horde of users whose past behavior is very
similar to yours, and whose purchase history gives a pretty good indication of your interests.

These predictions can also be self-fulfilling prophecies in the sense that you are more likely to
buy a product if it is recommended to you by the system, which makes it tricky to evaluate how
well they actually work. The same kind of recommendation systems are also used to
recommend music, movies, news, and social media content to users. In the context of news and
social media, filters created by such systems can lead to filter bubbles.

Exercise 15

Start of section 3:

III. Regression
Our main learning objective in this section is another nice example of supervised learning
methods, and almost as simple as the nearest neighbor classifier too: linear regression. We’ll
introduce its close cousin, logistic regression as well.

​ Note

The difference between classification and regression

There is a small but important difference in the kind of predictions that we should
produce in different scenarios. While for example the nearest neighbor classifier chooses
a class label for any item out of a given set of alternatives (like spam/ham, or 0,1,2,...,9),
linear regression produces a numerical prediction that is not constrained to be an integer
(a whole number as opposed to something like 3.14). So linear regression is better
suited in situations where the output variable can be any number like the price of a
product, the distance to an obstacle, the box-office revenue of the next Star Wars movie,
and so on.

The basic idea in linear regression is to add up the effects of each of the feature variables to
produce the predicted value. The technical term for the adding up process is linear combination.
The idea is very straightforward, and it can be illustrated by your shopping bill.

​
 Note

Thinking of linear regression as a shopping bill

Suppose you go to the grocery store and buy 2.5kg potatoes, 1.0kg carrots, and two
bottles of milk. If the price of potatoes is 2€ per kg, the price of carrots is 4€ per kg, and
a bottle of milk costs 3€, then the bill, calculated by the cashier, totals 2.5 × 2€ + 1.0 × 4€
+ 2 × 3€ = 15€. In linear regression, the amount of potatoes, carrots, and milk are the
inputs in the data. The output is the cost of your shopping, which clearly depends on
both the price and how much of each product you buy.

The word linear means that the increase in the output when one input feature is increased by
some fixed amount is always the same. In other words, whenever you add, say, two kilos of
carrots into your shopping basket, the bill goes up 8€. When you add another two kilos, the bill
goes up another 8€, and if you add half as much, 1kg, the bill goes up exactly half as much, 4€.

​ Key terminology

Coefficients or weights
In linear regression terminology, the prices of the different products would be called
coefficients or weights (this may appear confusing since we measured the amount of
potatoes and carrots by weight, but do not let yourself be tricked by this). One of the
main advantages of linear regression is its easy interpretability: the learned weights may
in fact be more interesting than the predictions of the outputs.

For example, when we use linear regression to predict the life expectancy, the weight of
smoking (cigarettes per day) is about minus half a year, meaning that smoking one
cigarette more per day takes you on the average half a year closer to termination.
Likewise, the weight of vegetable consumption (handful of vegetables per day) has
weight plus one year, so eating a handful of greens every day gives you on the average
one more year.

Exercise 16

In the above exercise, the life expectancy of non-smoking, veggie-hating women, 80 years, was
the starting point for the calculation. The technical term for the starting point is the intercept.
We will return to this below when we discuss how to learn linear regression models from data.
Learning linear regression
Above, we discussed how predictions are obtained from linear regression when both the
weights and the input features are known. So we are given the inputs and the weight, and we
can produce the predicted output.

When we are given the inputs and the outputs for a number of items, we can find the weights
such that the predicted output matches the actual output as well as possible. This is the task
solved by machine learning.

Note

Example
Continuing the shopping analogy, suppose we were given the contents of a number of
shopping baskets and the total bill for each of them, and we were asked to figure out the
price of each of the products (potatoes, carrots, and so on). From one basket, say 1kg of
sirloin steak, 2kg of carrots, and a bottle of Chianti, even if we knew that the total bill is
35€, we couldn’t determine the prices because there are many sets of prices that will
yield the same total bill. With many baskets, however, we will usually be able to solve the
problem.

But the problem is made harder by the fact that in the real world, the actual output isn’t always
fully determined by the input, because of various factors that introduce uncertainty or "noise"
into the process. You can think of shopping at a bazaar where the prices for any given product
may vary from time to time, or a restaurant where the final damage includes a variable amount
of tip. In such situations, we can estimate the prices but only with some limited accuracy.

Finding the weights that optimize the match between the predicted and the actual outputs in the
training data is a classical statistical problem dating back to the 1800s, and it can be easily
solved even for massive data sets.

We will not go into the details of the actual weight-finding algorithms, such as the classical least
squares technique, simple as they are. However, you can get a feel of finding trends in data in
the following exercises.

Visualizing linear regression

A good way to get a feel for what linear regression can tell us is to draw a chart containing our
data and our regression results. As a simple toy example our data set has one variable, the
number of cups of coffee an employee drinks per day, and the number of lines of code written
per day by that employee as the output. This is not a real data set as obviously there are other
factors having an effect on the productivity of an employee other than coffee that interact in
complex ways. The increase in productivity by increasing the amount of coffee will also hold
only to a certain point after which the jitters distract too much.

Image: cups of coffee graph

When we present our data in the chart above as points where one point represents one
employee, we can see that there is obviously a trend that drinking more coffee results in more
lines of code being written (recall that this is completely made-up data). From this data set we
can learn the coefficient, or the weight, related to coffee consumption, and by eye we can
already say that it seems to be somewhere close to five, since for each cup of coffee consumed
the number of lines programmed seems to go up roughly by five. For example, employees who
drink around two cups of coffee per day seem to produce around 20 lines of code per day, and
similarly at four cups of coffee, the amount of lines produced is around 30.

It can also be noted that employees who do not drink coffee at all also produce code, and is
shown by the graph to be about ten lines. This number is the intercept term that we mentioned
earlier. The intercept is another parameter in the model just like the weights are, that can be
learned from the data. Just as in the life expectancy example it can be thought of as the starting
point of our calculations before we have added in the effects of the input variable, or variables if
we have more than one, be it coffee cups in this example, or cigarettes and vegetables in the
previous one.

The line in the chart represents our predicted outcome, where we have estimated the intercept
and the coefficient by using an actual linear regression technique called least squares. This line
can be used to predict the number of lines produced when the input is the number of cups of
coffee. Note that we can obtain a prediction even if we allow only partial cups (like half, 1/4
cups, and so on).

Exercise 17
Exercise 18

It should be pointed out that studies like those used in the above exercises cannot identify
causal relationships. In other words, from this data alone, it is impossible to say whether
studying actually increases life expectancy through a better-informed and healthier life-style or
other mechanisms, or whether the apparent association between life expectancy and education
is due to underlying factors that affects both. It is likely that, for example, in countries where
people tend to be highly educated, nutrition, healthcare, and safety are also better, which
increases life expectancy. With this kind of simple analysis, we can only identify associations,
which can nevertheless be useful for prediction.

Machine learning applications of linear regression

Linear regression is truly the workhorse of many AI and data science applications. It has its
limits but they are often compensated by its simplicity, interpretability and efficiency. Linear
regression has been successfully used in the following problems to give a few examples:

●​ prediction of click rates in online advertising

●​ prediction of retail demand for products
●​ prediction of box-office revenue of Hollywood movies
●​ prediction of software cost
●​ prediction of insurance cost
●​ prediction of crime rates
●​ prediction of real estate prices
Could we use regression to predict labels?
As we discussed above, linear regression and the nearest neighbor method produce different
kinds of predictions. Linear regression outputs numerical outputs while the nearest neighbor
method produces labels from a fixed set of alternatives ("classes").

Where linear regression excels compared to nearest neighbors is interpretability. What do we

mean by this? You could say that in a way, the nearest neighbor method and any single
prediction that it produces are easy to interpret: it’s just the nearest training data element! This is
true, but when it comes to the interpretability of the learned model, there is a clear difference.
Interpreting the trained model in nearest neighbors in a similar fashion as the weights in linear
regression is impossible: the learned model is basically the whole data, and it is usually way too
big and complex to provide us with much insight. So what if we’d like to have a method that
produces the same kind of outputs as the nearest neighbor, labels, but is interpretable like linear
regression?

Logistic regression to the rescue

Well there is good news for you: we can turn the linear regression method’s outputs into
predictions about labels. The technique for doing this is called logistic regression. We will not go
into the technicalities, suffice to say that in the simplest case, we take the output from linear
regression, which is a number, and predict one label A if the output is greater than zero, and
another label B if the output is less than or equal to zero. Actually, instead of just predicting one
class or another, logistic regression can also give us a measure of uncertainty of the prediction.
So if we are predicting whether a customer will buy a new smartphone this year, we can get a
prediction that customer A will buy a phone with probability 90%, but for another, less
predictable customer, we can get a prediction that they will not buy a phone with 55% probability
(or in other words, that they will buy one with 45% probability).

It is also possible to use the same trick to obtain predictions over more than two possible labels,
so instead of always predicting either yes or no (buy a new phone or not, fake news or real
news, and so forth), we can use logistic regression to identify, for example, handwritten digits, in
which case there are ten possible labels.

An example of logistic regression

Let’s suppose that we collect data of students taking an introductory course in cookery. In
addition to the basic information such as the student ID, name, and so on, we also ask the
students to report how many hours they studied for the exam (however you study for a cookery
exam, probably cooking?) – and hope that they are more or less honest in their reports. After
the exam, we will know whether each student passed the course or not. Some data points are
presented below:

Student ID Hours studied Pass/fail

 24 15 Pass

41 9.5 Pass

58 2 Fail

101 5 Fail

103 6.5 Fail

215 6 Pass

Based on the table, what kind of conclusion could you draw between the hours studied and
passing the exam? We could think that if we have data from hundreds of students, maybe we
could see the amount needed to study in order to pass the course. We can present this data in a
chart as you can see below.

Exercise 19

Logistic regression is also used in a great variety of real-world AI applications such as predicting
financial risks, in medical studies, and so on. However, like linear regression, it is also
constrained by the linearity property and we need many other methods in our toolbox. We will
return to the linearity issue later when we discuss neural networks.

The limits of machine learning

To summarize, machine learning is a very powerful tool for building AI applications. In addition
to the nearest neighbor method, linear regression, and logistic regression, there are literally
hundreds, if not thousands, of different machine learning techniques, but they all boil down to
the same thing: trying to extract patterns and dependencies from data and using them either to
gain understanding of a phenomenon or to predict future outcomes.

Machine learning can be a very hard problem and we can’t usually achieve a perfect method
that would always produce the correct label. However, in most cases, a good but not perfect
prediction is still better than none. Sometimes we may be able to produce better predictions by
ourselves but we may still prefer to use machine learning because the machine will make its
predictions faster and it will also keep churning out predictions without getting tired. Good
examples are recommendation systems that need to predict what music, what videos, or what
ads are more likely to be of interest to you.

The factors that affect how good a result we can achieve include:

●​ The hardness of the task: in handwritten digit recognition, if the digits are written very
sloppily, even a human can’t always guess correctly what the writer intended
●​ The machine learning method: some methods are far better for a particular task than
others
●​ The amount of training data: from only a few examples, it is impossible to obtain a good
classifier
●​ The quality of the data

Note

Data quality matters

In the beginning of this chapter, we emphasized the importance of having enough data
and the risks of overfitting. Another equally important factor is the quality of the data. In
order to build a model that generalizes well to data outside of the training data, the
training data needs to contain enough information that is relevant to the problem at
hand. For example, if you create an image classifier that tells you what the image given
to the algorithm is about, and you have trained it only on pictures of dogs and cats, it will
assign everything it sees as either a dog or a cat. This would make sense if the algorithm
is used in an environment where it will only see cats and dogs, but not if it is expected to
see boats, cars, and flowers as well.

We’ll return to potential problems caused by ”biased” data.

It is also important to emphasize that different machine learning methods are suitable for
different tasks. Thus, there is no single best method for all problems ("one algorithm to rule
them all..."). Fortunately, one can try out a large number of different methods and see which one
of them works best in the problem at hand.

This leads us to a point that is very important but often overlooked in practice: what it means to
work better. In the digit recognition task, a good method would of course produce the correct
label most of the time. We can measure this by the classification error: the fraction of cases
where our classifier outputs the wrong class. In predicting apartment prices, the quality measure
is typically something like the difference between the predicted price and the final price for which
the apartment is sold. In many real-life applications, it is also worse to err in one direction than
in another: setting the price too high may delay the process by months, but setting the price too
low will mean less money for the seller. And to take yet another example, failing to detect a
pedestrian in front of a car is a far worse error than falsely detecting one when there is none.

As mentioned above, we can’t usually achieve zero error, but perhaps we will be happy with
error less than 1 in 100 (or 1%). This too depends on the application: you wouldn’t be happy to
have only 99% safe cars on the streets, but being able to predict whether you’ll like a new song
with that accuracy may be more than enough for a pleasant listening experience. Keeping the
actual goal in mind at all times helps us make sure that we create actual added value.

After completing Chapter 4 you should be able to:

●​ Explain why machine learning techniques are used

●​ Distinguish between unsupervised and supervised machine learning scenarios
●​ Explain the principles of three supervised classification methods: the nearest neighbor
method, linear regression, and logistic regression

Please join the Elements of AI community to discuss and ask questions about this
chapter.