Classification
The most common supervised learning tasks are regression (prediction
values) and classification (prediction classes)
- Binary Classification
- Multiclass classification
- Multilabel classification
- multioutput classification
Binary Classification
We will be using the MNIST dataset, which is a set of 70,000 small images of
digits handwritten. Each image is labeled with the digit it represents.
This set is often called the “hello world” of machine learning.
Scikit-learn provides many helper functions to download popular datasets.
[Link] contains three types of functions:
1. fetch_*: such as fetch_openml () to download real datasets.
2. Load_*: to load small toy datasets bundled with sklearn.
3. Make_*: to generate fake datasets, useful for testing.
Make_*: return data as an (X, y) tuple.
fetch_*, load_*: return data as [Link].
All these functions contain:
“DESCR”: A description of the dataset
“data”: the input data, usually as a 2D NumPy array.
“target”: the labels, usually as a 1D NumPy array.
the fetch_openml () function returns the inputs as a pandas DataFrame
and the labels as a Pandas Series. But the data contains images, and
DataFrame isn’t ideal for that, so we set as_frame = False to get the data as
NumPy arrays.
There are 70,000 images, and each image has 784 features. This is
because each image is 28 x 28 pixels. Every pixel represents from 0
(White) to 255 (Black).
Let’s take one digit from the dataset.
grab an instance’s feature vector, reshape it to 28 x 28 pixels. Cmap =
“binary” to get a grayscale color map where 0 is white and 255 is
black. [Link](“off”): remove x and y axis.
X [0]
Print first 100 digits in data set
First 100 digits in MNIST dataset
Binary Classification
We identify one digit for example, the number 5 will be a binary classifier. It is
capable of distinguishing between just two classes, 5 and non-5.
A good place to start is with a stochastic gradient descent classifier. Because
this classifier is capable of handling very large data sets efficiently.
Check our model
Evaluate this model’s performance
Evaluation a classifier is often significantly trickier than evaluating a
regressor, so we will spend a large part to know all performance measures
that available.
1. Measuring accuracy using Cross-Validation
2. Confusing Matrices
3. The ROC curve
Measuring accuracy using Cross-Validation
Using k-fold with three folds. Remember that k-fold means splitting the
training set into k folds, then training the model k times, holding out a different
fold each time for evaluation.
Above 95% accuracy? Well, before you get too excited, let’s look at a dummy
classifier that just classifies every single image in the most frequent class,
which in this case is the negative class (non 5).
Prints False: no 5s detected. Let’s check out the new accuracy.
This is simply because only about 10% of the images are 5s, so if you always
guess that an image is not a 5, you will be right about 90% of the time.
This demonstrates why accuracy isn’t the preferred performance measure for
classifiers, especially when we deal with classes that are much more
frequent than others. A much better way to evaluate is to look at the
confusion matrix.
Confusion Matrix (CM)
The general idea is to count the number of times instances of class A are
classified as class B. For example, to know the number of times the classifier
confused images of 5s with 3s, you would look in the 5th row and 3rd column
of the confusion matrix.
To compute the confusion matrix, you first need to have a set of predictions,
so they can be compared to the actual targets. you can use the
cross_val_predict() function:
Now you are ready to get the confusion matrix using the confusion_matrix ()
Confusion_matrix (the target classes, the predicted classes).
Each row represents the actual class.
Each column represents the predicted class.
Here explains the result of confusion matrix:
TN: True negative, FN: False negative
TP: True positive, FP: False negative
The confusion matrix gives you a lot of information, but sometimes you may
prefer a more concise metric.
An interesting one to look at is the accuracy of the positive predictions, this is
called the precision of the classifier.
𝑇𝑃
Precision =
𝑇𝑃 + 𝐹𝑃
3530
Precision = = 0.83708
3530+687
كام حالة كانت ا، من كل الحاالت اللي اتنبأ بيها النموذج على إنها إيجابية،بمعنى
فعًل إيجابية؟
𝑇𝑃
𝑅𝑒𝑐𝑎𝑙𝑙 =
𝑇𝑃 + 𝐹𝑁
3530
Recall = = 0.651171
3530+1891
كام حالة النموذج قدر يصنفها صح كإيجابية؟، من كل الحاالت اإليجابية الفعلية،بمعنى
The difference between precision and recall:
• Precision )بيهمنا لما يكون غلط التنبؤ بحالة إيجابية له تأثير كبير (زي حاالت اكتشاف المرض
• بيهمنا لما يكون من المهم إننا ما نفوتش حاالت إيجابية (زي اكتشاف المرضى ا
Recall )فعًل
When it claims an image represents a 5, it is correct only 83.7% of the time.
Moreover, it only detects 65.1% of the 5s.
F1 score
The harmonic mean of precision and recall. Whereas the regular mean treats
all values equally, the harmonic mean gives much more weight to low values.
As a result, the classifier will only get a high F1 score if both recall, and
precision are high.
2 𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 × 𝑟𝑒𝑐𝑎𝑙𝑙 𝑇𝑃
𝐹1 = =2 × =
1
+
1 𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 + 𝑟𝑒𝑐𝑎𝑙𝑙 𝑇𝑃 + 𝐹𝑁 + 𝐹𝑃
𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 𝑟𝑒𝑐𝑎𝑙𝑙 2
0.827 × 0.6511
𝐹1 = 2 × = 0.7285
0.827 + 0.6511
F1 score favors classifiers that have similar precision and recall. This is not
always what you want: in some context you mostly care about precision, and
in other contexts you really care about recall.
:مثال
ف تعمل مودل بالمواصفات الجايه.لو انت عايز تصنف الناس اللي بيسرقوا من المحًلت أثناء التسوق
حتى لو هيغلط ف الناس ال مش، ف هنا طبعا مفيش مشكله30% precision and 99% recall
ف الميه لو حد99 ف الميه بس30 بتسرق وصنفهم انهم بيسرقوا عادي ممكن نتفاهم معاهم ال هما بنسبه
.بيسرق هيتمسك
You can’t have high precision and high recall: increasing precision reduces
recall, and vice versa. This is called the precision/recall trade-off
To understand the trade off, let’s look at how the classifier makes its
classification decision. It computes a score based on a decision function.
If the score is greater than a threshold, it assigns the instance to the
positive class.
If the score is lower than a threshold, it assigns the instance to the
negative class.
و نشوف احنا عاوزين أي بالظبطPrecision and recallنفترض ان دا الحد االدني و نحسب ال
هيساوي عدد الخمسات الصح على العدد الكلي في الجنب اليمينPrecision أوال ازاي حسبنا ال
طب وازاي حسبنا Recallهيساوي عدد الخمسات الصح ال ع اليمين على عدد الخمسات الكلي
نجرب نودي الحد االدني يمين شويه هنًلحظ فيه تغير
طب نجيبها شمال
الحظ ان كل م نروح يمين ال Precisionبيزيد بس دا ف االغلب ممكن يقل
وكل م نروح شمال ال Recallبيزيد و دا بيحصل دايما
Scikit-Learn doesn’t let you set the threshold directly, but it does give you
access to the decision scores.
How do you decide which threshold to use?
First, use the cross-validation prediction function to get the scores of all
instances in the training set.
Use the precision recall curve function to compute prediction and recall for
all possible thresholds.
The ROC Curve
The receiver operating characteristic (ROC) is another common tool used
with binary classifiers. The ROC curve plots the true positive rate (another
name for recall) against the false positive rate (called the fall-out).
The false positive rate (FPR) is the ratio of negative instances that are
incorrectly classified as positive.
The true negative rate (TNR) is the ratio of negative instances that are
correctly classified as negative.