CLASSIFICATION Machine

Learning
OUTLINE
1) Dataset
2) Training a Binary Classifier
3) Performance Measures
4) Multiclass Classification
5) Error Analysis
6) Multilabel Classification
7) Multioutput Classification

Machine Learning | Classification 2

DATASET
▪ We need a labeled dataset to train a classification model.
▪ In this chapter, as an example, we will be using the MNIST dataset, which is a set of
70,000 small images of digits handwritten by high school students and employees of
the US Census Bureau.
▪ Each image is labeled with the digit it represents.
▪ Scikit-Learn provides many helper functions to download popular datasets. MNIST is
one of them. The following code fetches the MNIST dataset:

Machine Learning | Classification 3

DATASET
▪ Datasets loaded by Scikit-Learn generally have a similar dictionary structure including:
• A DESCR key describing the dataset
• A data key containing an array with one row/instance and one column/feature
• A target key containing an array with the labels
▪ Let’s look at these arrays:

Machine Learning | Classification 4

DATASET
▪ As mentioned earlier, there are 70,000 images, and each image has 784 features. This
is because each image is 28×28 pixels, and each feature simply represents one pixel’s
intensity, from 0 (white) to 255 (black).
▪ Let’s take a peek at one digit from the dataset. All you need to do is grab an
instance’s feature vector, reshape it to a 28×28 array, and display it using Matplotlib’s
imshow() function:

Machine Learning | Classification 5

DATASET
▪ This looks like a 5, and indeed that’s what the label tells us:

▪ Note that the label is a string. We prefer numbers, so let’s cast y to integers:

Machine Learning | Classification 6

DATASET

A few digits from the MNIST dataset

Machine Learning | Classification 7

DATASET
▪ But wait! You should always create a test set and set it aside before inspecting the
data closely. The MNIST dataset is actually already split into a training set (the first
60,000 images) and a test set (the last 10,000 images):

▪ The training set is already shuffled for us, which is good as this guarantees that all
cross-validation folds will be similar (you don’t want one-fold to be missing some
digits). Moreover, some learning algorithms are sensitive to the order of the training
instances, and they perform poorly if they get many similar instances in a row. Shuffling
the dataset ensures that this won’t happen.

Machine Learning | Classification 8

TRAINING A BINARY CLASSIFIER
▪ Let’s simplify the problem for now and only try to identify one digit—for example, the
number 5.
▪ This “5-detector” will be an example of a binary classifier, capable of distinguishing
between just two classes, 5 and not-5.
▪ Let’s create the target vectors for this classification task:

Machine Learning | Classification 9

TRAINING A BINARY CLASSIFIER
▪ Now, let’s pick a classifier and train it. A good place to start is with a Stochastic
Gradient Descent (SGD) classifier, using Scikit-Learn’s SGDClassifier class.
▪ This classifier has the advantage of being capable of handling very large datasets
efficiently. This is in part because SGD deals with training instances independently, one
at a time (which also makes SGD well suited for online learning), as we will see later.
▪ Let’s create an SGDClassifier and train it on the whole training set:

Machine Learning | Classification 10

TRAINING A BINARY CLASSIFIER
▪ Now you can use it to detect images of the number 5:

▪ The classifier guesses that this image represents a 5 (True). Looks like it guessed right in
this particular case! Now, let’s evaluate this model’s performance.

Machine Learning | Classification 11

PERFORMANCE MEASURES
▪ Evaluating a classifier is often significantly trickier than evaluating a regressor, so we
will spend a large part of this chapter on this topic. There are many performance
measures available:

1) Measuring Accuracy Using Cross-Validation

2) Confusion Matrix
3) Precision, Recall, and F1 score
4) Precision/Recall Tradeoff
5) The ROC Curve

Machine Learning | Classification 12

PERFORMANCE MEASURES > MEASURING ACCURACY USING CROSS-VALID.
▪ A good way to evaluate a model is to use cross-validation, just as we did previously.
▪ Let’s use the cross_val_score() function to evaluate your SGDClassifier model using K-
fold cross-validation, with three folds. Remember that K-fold cross validation means
splitting the training set into K-folds (in this case, three), then making predictions and
evaluating them on each fold using a model trained on the remaining folds:

Machine Learning | Classification 13

PERFORMANCE MEASURES > MEASURING ACCURACY USING CROSS-VALID.
▪ We get above 93% accuracy (ratio of correct predictions) on all cross-validation folds.
This looks amazing, doesn’t it?
▪ Well, before you get too excited, let’s look at a very dumb classifier that just classifies
every single image in the “not-5” class:

Machine Learning | Classification 14

PERFORMANCE MEASURES > MEASURING ACCURACY USING CROSS-VALID.
▪ Can you guess this model’s accuracy? Let’s find out:

▪ That’s right, it has over 90% accuracy! This is simply because only about 10% of the
images are 5s, so if you always guess that an image is not a 5, you will be right about
90% of the time.
▪ This demonstrates why accuracy is generally not the preferred performance measure
for classifiers, especially when you are dealing with skewed datasets (i.e., when some
classes are much more frequent than others).

Machine Learning | Classification 15

PERFORMANCE MEASURES > CONFUSION MATRIX
▪ A much better way to evaluate the performance of a classifier is to look at the
confusion matrix.
▪ The general idea is to count the number of times instances of class A are classified as
class B. For example, to know the number of times the classifier confused images of 5s
with 3s, you would look in the 5th row and 3rd column of the confusion matrix.
▪ To compute the confusion matrix, you first need to have a set of predictions, so they can
be compared to the actual targets. You could make predictions on the test set, but let’s
keep it untouched for now. Instead, you can use the cross_val_predict() function:

Machine Learning | Classification 16

PERFORMANCE MEASURES > CONFUSION MATRIX
▪ Just like the cross_val_score() function, cross_val_predict() performs K-fold cross-
validation, but instead of returning the evaluation scores, it returns the predictions
made on each test fold.
▪ Now you are ready to get the confusion matrix using the confusion_matrix() func‐ tion.
Just pass it the target classes (y_train_5) and the predicted classes (y_train_pred):

Machine Learning | Classification 17

PERFORMANCE MEASURES > CONFUSION MATRIX
▪ Each row in a confusion matrix represents an actual class, while each column represents
a predicted class.
▪ The first row of this matrix considers non-5 images (the negative class): 53,057 of them
were correctly classified as non-5s (they are called true negatives), while the remaining
1,522 were wrongly classified as 5s (false positives).
▪ The second row considers the images of 5s (the positive class): 1,325 were wrongly
classified as non-5s (false negatives), while the remaining 4,096 were correctly
classified as 5s (true positives).
▪ A perfect classifier would have only true positives and true negatives, so its confusion
matrix would have nonzero values only on its main diagonal.

Machine Learning | Classification 18

PERFORMANCE MEASURES > CONFUSION MATRIX
▪ The confusion matrix gives you a lot of information, but sometimes you may prefer a
more concise metric. An interesting one to look at is the accuracy of the positive
predictions; this is called the precision of the classifier:

▪ TP is the number of true positives, and FP is the number of false positives.

Machine Learning | Classification 19

PERFORMANCE MEASURES > CONFUSION MATRIX
▪ A trivial way to have perfect precision is to make one single positive prediction and
ensure it is correct (precision = 1/1 = 100%). This would not be very useful since the
classifier would ignore all but one positive instance. So precision is typically used along
with another metric named recall, also called sensitivity or true positive rate(TPR). This
is the ratio of positive instances that are correctly detected by the classifier:

▪ Here, FN is the number of false negatives.

Machine Learning | Classification 20

PERFORMANCE MEASURES > CONFUSION MATRIX
▪ The figure below illustrates the confusion matrix and its components:

Machine Learning | Classification 21

PERFORMANCE MEASURES > PRECISION, RECALL, AND F1 SCORE
▪ Scikit-Learn provides several functions to compute classifier metrics, including precision
and recall:

▪ Now your 5-detector does not look as shiny as it did when you looked at its accuracy.
When it claims an image represents a 5, it is correct only 72.9% of the time. Moreover,
it only detects 75.6% of the 5s.

Machine Learning | Classification 22

PERFORMANCE MEASURES > PRECISION, RECALL, AND F1 SCORE
▪ It is often convenient to combine precision and recall into a single metric called the F1
score, in particular if you need a simple way to compare two classifiers.
▪ The F1 score is the harmonic mean of precision and recall. Whereas the regular mean
treats all values equally, the harmonic mean gives much more weight to low values. As
a result, the classifier will only get a high F1 score if both recall and precision are high.

Machine Learning | Classification 23

PERFORMANCE MEASURES > PRECISION, RECALL, AND F1 SCORE
▪ To compute the F1 score, simply call the f1_score() function:

Machine Learning | Classification 24

PERFORMANCE MEASURES > PRECISION, RECALL, AND F1 SCORE
▪ The F1 score favors classifiers that have similar precision and recall. This is not always
what you want: in some contexts you mostly care about precision, and in other contexts
you really care about recall.
▪ For example, if you trained a classifier to detect videos that are safe for kids, you
would probably prefer a classifier that rejects many good videos (low recall) but
keeps only safe ones (high precision), rather than a classifier that has a much higher
recall but lets a few really bad videos show up in your product.
▪ On the other hand, suppose you train a classifier to detect shoplifters on surveillance
images: it is probably fine if your classifier has only 30% precision as long as it has
99% recall (sure, the security guards will get a few false alerts, but almost all
shoplifters will get caught).

Machine Learning | Classification 25

PERFORMANCE MEASURES > PRECISION/RECALL TRADEOFF
▪ Increasing precision reduces recall, and vice versa. This is called the precision/recall
tradeoff.
▪ To understand this tradeoff, let’s look at how the SGDClassifier makes its classification
decisions. For each instance, it computes a score based on a decision function, and if
that score is greater than a threshold, it assigns the instance to the positive class, or
else it assigns it to the negative class. The figure below shows a few digits positioned
from the lowest score on the left to the highest score on the right.

Machine Learning | Classification 26

PERFORMANCE MEASURES > PRECISION/RECALL TRADEOFF
▪ Suppose the decision threshold is positioned at the central arrow (between the two 5s):
you will find 4 true positives (actual 5s) on the right of that threshold, and one false
positive (actually a 6). Therefore, with that threshold, the precision is 80% (4/5). But
out of 6 actual 5s, the classifier only detects 4, so the recall is 67% (4/6).
▪ Now if you raise the threshold (move it to the arrow on the right), the false positive (the
6) becomes a true negative, thereby increasing precision (up to 100% in this case), but
one true positive becomes a false negative, decreasing recall down to 50%.
Conversely, lowering the threshold increases recall and reduces precision.

Machine Learning | Classification 27

PERFORMANCE MEASURES > PRECISION/RECALL TRADEOFF
▪ Scikit-Learn does not let you set the threshold directly, but it does give you access to
the decision scores that it uses to make predictions.
▪ Instead of calling the classifier’s predict() method, you can call its decision_function()
method, which returns a score for each instance, and then make predictions based on
those scores using any threshold you want:

Machine Learning | Classification 28

PERFORMANCE MEASURES > PRECISION/RECALL TRADEOFF
▪ The SGDClassifier uses a threshold equal to 0, so the previous code returns the same
result as the predict() method (i.e., True). Let’s raise the threshold:

▪ This confirms that raising the threshold decreases recall. The image actually represents
a 5, and the classifier detects it when the threshold is 0, but it misses it when the
threshold is increased to 8,000.

Machine Learning | Classification 29

PERFORMANCE MEASURES > PRECISION/RECALL TRADEOFF
▪ Now how do you decide which threshold to use? For this you will first need to get the
scores of all instances in the training set using the cross_val_predict() function again,
but this time specifying that you want it to return decision scores instead of predictions:

▪ Now with these scores you can compute precision and recall for all possible thresholds
using the precision_recall_curve() function:

Machine Learning | Classification 30

PERFORMANCE MEASURES > PRECISION/RECALL TRADEOFF
▪ Finally, you can plot precision and recall as functions of the threshold value using
Matplotlib:

Machine Learning | Classification 31

PERFORMANCE MEASURES > PRECISION/RECALL TRADEOFF
▪ Another way to select a good precision/recall tradeoff is to plot precision directly
against recall, as shown here:

Machine Learning | Classification 32

PERFORMANCE MEASURES > PRECISION/RECALL TRADEOFF
▪ You can see that precision really starts to fall sharply around 80% recall. You will
probably want to select a precision/recall tradeoff just before that drop—for
example, at around 60% recall. But, of course, the choice depends on your project.
▪ So, let’s suppose you decide to aim for 90% precision. You look up the first plot and
find that you need to use a threshold of about 8,000. To be more precise you can
search for the lowest threshold that gives you at least 90% precision ([Link]() will
give us the first index of the maximum value, which in this case means the first True
value):

Machine Learning | Classification 33

PERFORMANCE MEASURES > PRECISION/RECALL TRADEOFF
▪ To make predictions (on the training set for now), instead of calling the classifier’s
predict() method, you can just run this code:

▪ Let’s check these predictions’ precision and recall:

▪ Great, you have a 90% precision classifier! It is fairly easy to create a classifier with
any precision you want: just set a high enough threshold. But be careful. A high-
precision classifier is not very useful if its recall is too low!

Machine Learning | Classification 34

PERFORMANCE MEASURES > THE ROC CURVE
▪ The receiver operating characteristic (ROC) curve is another common tool used with
binary classifiers. It is very similar to the precision/recall curve, but instead of plotting
precision versus recall, the ROC curve plots the true positive rate (recall) against the
false positive rate (FPR).
▪ The FPR is the ratio of negative instances that are incorrectly classified as positive. It is
equal to one minus the true negative rate (TNR), which is the ratio of negative instances
that are correctly classified as negative. The TNR is also called specificity. Hence the
ROC curve plots sensitivity (recall) vs. (1 – specificity).

Machine Learning | Classification 35

PERFORMANCE MEASURES > THE ROC CURVE
▪ To plot the ROC curve, you first need to compute the TPR and FPR for various threshold
values, using the roc_curve() function:

▪ Then you can plot the FPR against the TPR using Matplotlib:

Machine Learning | Classification 36

PERFORMANCE MEASURES > THE ROC CURVE

Machine Learning | Classification 37

PERFORMANCE MEASURES > THE ROC CURVE
▪ Once again there is a tradeoff: the higher the recall (TPR), the more false positives
(FPR) the classifier produces. The dotted line represents the ROC curve of a purely
random classifier; a good classifier stays as far away from that line as possible
(toward the top-left corner).
▪ One way to compare classifiers is to measure the area under the curve (AUC). A
perfect classifier will have a ROC AUC equal to 1, whereas a purely random classifier
will have a ROC AUC equal to 0.5. Scikit-Learn provides a function to compute the
ROC AUC:

Machine Learning | Classification 38

PERFORMANCE MEASURES > THE ROC CURVE
▪ Since the ROC curve is so similar to the precision/recall (or PR) curve, you may wonder
how to decide which one to use.
▪ As a rule of thumb, you should prefer the PR curve whenever the positive class is rare
or when you care more about the false positives than the false negatives, and the ROC
curve otherwise.
▪ For example, looking at the previous ROC curve (and the ROC AUC score), you may
think that the classifier is really good. But this is mostly because there are few positives
(5s) compared to the negatives (non-5s). In contrast, the PR curve makes it clear that
the classifier has room for improvement (the curve could be closer to the top right
corner).

Machine Learning | Classification 39

PERFORMANCE MEASURES > THE ROC CURVE
▪ Let’s train a RandomForestClassifier and compare its ROC curve and ROC AUC score
to the SGDClassifier.
▪ First, you need to get scores for each instance in the training set. But due to the way it
works, the RandomForestClassifier class does not have a decision_function() method.
Instead it has a predict_proba() method. Scikit-Learn classifiers generally have one or
the other.
▪ The predict_proba() method returns an array containing a row per instance and a
column per class, each containing the probability that the given instance belongs to the
given class (e.g., 70% chance that the image represents a 5):

Machine Learning | Classification 40

PERFORMANCE MEASURES > THE ROC CURVE
▪ But to plot a ROC curve, you need scores, not probabilities. A simple solution is to use
the positive class’s probability as the score:

▪ Now you are ready to plot the ROC curve. It is useful to plot the first ROC curve as
well to see how they compare:

Machine Learning | Classification 41

PERFORMANCE MEASURES > THE ROC CURVE

Machine Learning | Classification 42

PERFORMANCE MEASURES > THE ROC CURVE
▪ As you can see in the previous figure, the RandomForestClassifier’s ROC curve looks
much better than the SGDClassifier’s. As a result, its ROC AUC score is also
significantly better:

▪ Try measuring the precision and recall scores: you should find 99.0% precision and
86.6% recall. Not too bad! Hopefully you now know how to train binary classifiers,
choose the appropriate metric for your task, evaluate your classifiers using cross-
validation, select the precision/ recall tradeoff that fits your needs, and compare
various models using ROC curves and ROC AUC scores. Now let’s try to detect more
than just the 5s.

Machine Learning | Classification 43

MULTICLASS CLASSIFICATION
▪ Whereas binary classifiers distinguish between two classes, multiclass classifiers (also
called multinomial classifiers) can distinguish between more than two classes.
▪ Some algorithms (such as Random Forest classifiers or naive Bayes classifiers) are
capable of handling multiple classes directly.
▪ Others (such as Support Vector Machine (SVM) classifiers or Linear classifiers) are
strictly binary classifiers.
▪ However, there are various strategies that you can use to perform multiclass
classification using multiple binary classifiers.

Machine Learning | Classification 44

MULTICLASS CLASSIFICATION
▪ For example, one way to create a system that can classify the digit images into 10
classes (from 0 to 9) is to train 10 binary classifiers, one for each digit (a 0-detector, a
1-detector, a 2-detector, and so on). Then when you want to classify an image, you get
the decision score from each classifier for that image and you select the class whose
classifier outputs the highest score. This is called the one-versus-all (OvA) strategy (also
called one-versus-the-rest).
▪ Another strategy is to train a binary classifier for every pair of digits: one to
distinguish 0s and 1s, another to distinguish 0s and 2s, another for 1s and 2s, and so
on. This is called the one-versus-one (OvO) strategy. If there are N classes, you need to
train N × (N – 1) / 2 classifiers. For the MNIST problem, this means training 45 binary
classifiers! When you want to classify an image, you have to run the image through all
45 classifiers and see which class wins the most duels. The main advantage of OvO is
that each classifier only needs to be trained on the part of the training set for the two
classes that it must distinguish.

Machine Learning | Classification 45

MULTICLASS CLASSIFICATION
▪ Some algorithms (such as SVM classifiers) scale poorly with the size of the training set,
so for these algorithms OvO is preferred since it is faster to train many classifiers on
small training sets than training few classifiers on large training sets. For most binary
classification algorithms, however, OvA is preferred.
▪ Scikit-Learn detects when you try to use a binary classification algorithm for a multi‐
class classification task, and it automatically runs OvA (except for SVM classifiers for
which it uses OvO). Let’s try this with the SGDClassifier:

Machine Learning | Classification 46

MULTICLASS CLASSIFICATION
▪ That was easy! This code trains the SGDClassifier on the training set using the original
target classes from 0 to 9 (y_train), instead of the 5-versus-all target classes
(y_train_5). Then it makes a prediction (a correct one in this case). Under the hood,
Scikit-Learn actually trained 10 binary classifiers, got their decision scores for the
image, and selected the class with the highest score.
▪ To see that this is indeed the case, you can call the decision_function() method. Instead
of returning just one score per instance, it now returns 10 scores, one per class:

Machine Learning | Classification 47

MULTICLASS CLASSIFICATION
▪ The highest score is indeed the one corresponding to class 5:

▪ When a classifier is trained, it stores the list of target classes in its classes_ attribute,
ordered by value. In this case, the index of each class in the classes_ array
conveniently matches the class itself (e.g., the class at index 5 happens to be class 5),
but in general you won’t be so lucky.

Machine Learning | Classification 48

MULTICLASS CLASSIFICATION
▪ If you want to force ScikitLearn to use one-versus-one or one-versus-all, you can use the
OneVsOneClassifier or OneVsRestClassifier classes. Simply create an instance and
pass a binary classifier to its constructor. For example, this code creates a multi‐class
classifier using the OvO strategy, based on a SGDClassifier:

Machine Learning | Classification 49

MULTICLASS CLASSIFICATION
▪ Training a RandomForestClassifier is just as easy:

▪ This time Scikit-Learn did not have to run OvA or OvO because Random Forest classifiers can
directly classify instances into multiple classes. You can call predict_proba() to get the list of
probabilities that the classifier assigned to each instance for each class:

▪ You can see that the classifier is fairly confident about its prediction: the 0.9 at the 5th index in
the array means that the model estimates a 90% probability that the image represents a 5. It
also thinks that the image could instead be a 2, a 3 or a 9, respectively with 1%, 8% and 1%
probability.

Machine Learning | Classification 50

MULTICLASS CLASSIFICATION
▪ Now let's evaluate these classifiers. As usual, you want to use cross validation. Let’s
evaluate the SGDClassifier’s accuracy using the cross_val_score() function:

▪ It gets over 84% on all test folds. If you used a random classifier, you would get 10%
accuracy, so this is not such a bad score, but you can still do much better. For example,
simply scaling the inputs increases accuracy above 89%:

Machine Learning | Classification 51

ERROR ANALYSIS
▪ Now, lets assume that you have a promising model and you want to find ways to
improve it. One way to do this is to analyze the types of errors it makes.
▪ First, you can look at the confusion matrix. You need to make predictions using the
cross_val_predict() function, then call the confusion_matrix() function, just like you did
earlier:

Machine Learning | Classification 52

ERROR ANALYSIS
▪ That’s a lot of numbers. It’s often more convenient to
look at an image representation of the confusion
matrix, using Matplotlib’s matshow() function:

▪ This confusion matrix looks fairly good, since most

images are on the main diagonal, which means that
they were classified correctly.
▪ The 5s look slightly darker than the other digits, which
could mean that there are fewer images of 5s in the
dataset or that the classifier does not perform as well
on 5s as on other digits. In fact, you can verify that
both are the case.
Machine Learning | Classification 53
ERROR ANALYSIS
▪ Let’s focus the plot on the errors. First, you need to
divide each value in the confusion matrix by the
number of images in the corresponding class, so
you can compare error rates instead of absolute
number of errors (which would make abundant
classes look unfairly bad):

▪ Now let’s fill the diagonal with zeros to keep only

the errors, and let’s plot the result:

Machine Learning | Classification 54

ERROR ANALYSIS
▪ Now you can clearly see the kinds of errors the classifier makes. The column for class 8
is quite bright, which tells you that many images get misclassified as 8s. However, the
row for class 8 is not that bad, telling you that actual 8s in general get properly
classified as 8s.
▪ As you can see, the confusion matrix is not necessarily symmetrical. You can also see
that 3s and 5s often get confused (in both directions). Analyzing the confusion matrix
can often give you insights on ways to improve your classifier.
▪ Looking at this plot, it seems that your efforts should be spent on reducing the false 8s.
For example, you could try to gather more training data for digits that look like 8s (but
are not) so the classifier can learn to distinguish them from real 8s. Or you could
engineer new features that would help the classifier. Or you could preprocess the
images (e.g., using Scikit-Image, Pillow, or OpenCV) to make some patterns stand out
more.
Machine Learning | Classification 55
MULTILABEL CLASSIFICATION
▪ Until now each instance has always been assigned to just one class. In some cases, you
may want your classifier to output multiple classes for each instance.
▪ For example, consider a face-recognition classifier: what should it do if it recognizes
several people on the same picture? Of course, it should attach one tag per person it
recognizes. Say the classifier has been trained to recognize three faces, Alice, Bob,
and Charlie; then when it is shown a picture of Alice and Charlie, it should output [1, 0,
1] (meaning “Alice yes, Bob no, Charlie yes”). Such a classification system that outputs
multiple binary tags is called a multilabel classification system.

Machine Learning | Classification 56

MULTILABEL CLASSIFICATION
▪ For this purpose, let’s look at a simple example, just for illustration purposes:

▪ This code creates a y_multilabel array containing two target labels for each digit
image: the first indicates whether or not the digit is large (7, 8, or 9) and the second
indicates whether or not it is odd. The next lines create a KNeighborsClassifier instance
(which supports multilabel classification) and we train it using the multiple targets
array.

Machine Learning | Classification 57

MULTILABEL CLASSIFICATION
▪ Now you can make a prediction, and notice that it outputs two labels:

▪ And it gets it right! The digit 5 is indeed not large (False) and odd (True).

Machine Learning | Classification 58

MULTIOUTPUT CLASSIFICATION
▪ There are many ways to evaluate a multilabel classifier, and selecting the right metric
really depends on your project. One approach is to measure the F1 score for each
individual label (or any other binary classifier metric discussed earlier), then simply
compute the average score. This code computes the average F1 score across all labels:

▪ This assumes that all labels are equally important, which may not be the case. In
particular, if you have many more pictures of Alice than of Bob or Charlie, you may
want to give more weight to the classifier’s score on pictures of Alice. One simple
option is to give each label a weight equal to its support (i.e., the number of instances
with that target label). To do this, simply set average="weighted" in the preceding
code.
Machine Learning | Classification 59
MULTIOUTPUT CLASSIFICATION
▪ The last type of classification task we are going to discuss here is called multioutput-
multiclass classification (or simply multioutput classification).
▪ It is simply a generalization of multilabel classification where each label can be
multiclass (i.e., it can have more than two possible values).
▪ To illustrate this, let’s build a system that removes noise from images. It will take as
input a noisy digit image, and it will (hopefully) output a clean digit image,
represented as an array of pixel intensities, just like the MNIST images. Notice that the
classifier’s output is multilabel (one label per pixel) and each label can have multiple
values (pixel intensity ranges from 0 to 255). It is thus an example of a multioutput
classification system.

Machine Learning | Classification 60

MULTIOUTPUT CLASSIFICATION
▪ Let’s start by creating the training and test sets by taking the MNIST images and
adding noise to their pixel intensities using NumPy’s randint() function. The target
images will be the original images:

Machine Learning | Classification 61

MULTIOUTPUT CLASSIFICATION
▪ Let’s take a peek at an image from the test set. On the left is the noisy input image,
and on the right is the clean target image:

▪ Now let’s train the classifier and make it clean this image. Looks close enough to the
target:

Machine Learning | Classification 62

MULTIOUTPUT CLASSIFICATION
▪ This concludes our tour of classification. Hopefully, you should now know how to select
good metrics for classification tasks, pick the appropriate precision/recall tradeoff,
compare classifiers, and more generally build good classification systems for a variety
of tasks.

Machine Learning | Classification 63

SUMMARY
1) Dataset
2) Training a Binary Classifier
3) Performance Measures
4) Multiclass Classification
5) Error Analysis
6) Multilabel Classification
7) Multioutput Classification

Machine Learning | Classification 64

REFERENCES
▪ Géron, Aurélien. Hands-on machine learning with Scikit-Learn, Keras, and TensorFlow:
Concepts, tools, and techniques to build intelligent systems. O'Reilly Media, Inc., 2019.

Machine Learning | Classification 65