Machine

Learning with
Python
Tutorial

by
Bernd Klein

bodenseo
© 2021 Bernd Klein

All rights reserved. No portion of this book may be reproduced or used in any
manner without written permission from the copyright owner.

For more information, contact address: [Link]@[Link]

[Link]
 Python Course
Machine Learning
With Python by
Bernd Klein
Machine Learning Terminology .................................................................................................3
Representation and Visualization of Data ................................................................................15
Loading the Iris Data with Scikit-learn ....................................................................................18
Visualising the Features of the Iris Data Set.............................................................................23
Scatterplot 'Matrices .................................................................................................................27
Datasets in sklearn ....................................................................................................................29
Loading Digits Data..................................................................................................................31
Reading the data and conversion back into 'data' and 'labels'...................................................51
Other Interesting Distributions .................................................................................................54
k-Nearest-Neighbor Classifier ..................................................................................................72
From Dividing Lines to Neural Networks................................................................................96
Neural Networks, Structure, Weights and Matrices ...............................................................141
Running a Neural Network with Python ................................................................................153
Backpropagation in Neural Networks ....................................................................................162
Training a Neural Network with Python ................................................................................169
Softmax as Activation Function .............................................................................................182
Confusion Matrix........................................................................................................................3
Neural Network ......................................................................................................................198
Multiple Runs .........................................................................................................................210
With Bias Nodes .....................................................................................................................216
Networks with multiple hidden layers....................................................................................227
Networks with multiple hidden layers and Epochs ................................................................231
A Neural Network for the Digits Dataset ...............................................................................269
Naive Bayes Classifier with Scikit .........................................................................................316
Regression Trees.....................................................................................................................413
The maths behind regression trees..........................................................................................418
Regression Decision Trees from scratch in Python ................................................................423
Regression Trees in sklearn ....................................................................................................434
TensorFlow .............................................................................................................................437
2
MACHINE LEARNING TERMINOLOGY

CLASSIFIER
A program or a function which maps from unlabeled instances to classes is called a classifier.

CONFUSION MATRIX
A confusion matrix, also called a contingeny table or error matrix, is used to visualize the performance of a
classifier.

The columns of the matrix represent the instances of the predicted classes and the rows represent the instances
of the actual class. (Note: It can be the other way around as well.)

In the case of binary classification the table has 2 rows and 2 columns.

Example:

3
 Confusion Predicted classes
Matrix
male female
classes
Actual

male 42 8

female 18 32

This means that the classifier correctly predicted a male person in 42 cases and it wrongly predicted 8 male
instances as female. It correctly predicted 32 instances as female. 18 cases had been wrongly predicted as male
instead of female.

ACCURACY (ERROR RATE)

Accuracy is a statistical measure which is defined as the quotient of correct predictions made by a classifier
divided by the sum of predictions made by the classifier.

The classifier in our previous example predicted correctly predicted 42 male instances and 32 female instance.

Therefore, the accuracy can be calculated by:

accuracy = (42 + 32) / (42 + 8 + 18 + 32)

which is 0.72

Let's assume we have a classifier, which always predicts "female". We have an accuracy of 50 % in this case.

Confusion Predicted classes

Matrix
male female
classes
Actual

male 0 50

female 0 50

We will demonstrate the so-called accuracy paradox.

A spam recogition classifier is described by the following confusion matrix:

4
 Confusion Predicted classes
Matrix
spam ham
classes
Actual

spam 4 1

ham 4 91

The accuracy of this classifier is (4 + 91) / 100, i.e. 95 %.

The following classifier predicts solely "ham" and has the same accuracy.

Confusion Predicted classes

Matrix
spam ham
classes
Actual

spam 0 5

ham 0 95

The accuracy of this classifier is 95%, even though it is not capable of recognizing any spam at all.

PRECISION AND RECALL

Confusion Predicted classes

Matrix
negative positive
classes
Actual

negative TN FP

positive FN TP

Accuracy: (TN + TP) / (TN + TP + FN + FP)

Precision: TP / (TP + FP)

5
Recall: TP / (TP + FN)

SUPERVISED LEARNING
The machine learning program is both given the input data and the corresponding labelling. This means that
the learn data has to be labelled by a human being beforehand.

UNSUPERVISED LEARNING
No labels are provided to the learning algorithm. The algorithm has to figure out the a clustering of the input
data.

REINFORCEMENT LEARNING
A computer program dynamically interacts with its environment. This means that the program receives
positive and/or negative feedback to improve it performance.

6
EVALUATION METRICS

INTRODUCTION
Not only in machine learning but also in
general life, especially business life, you
will hear questiones like "How accurate is
your product?" or "How precise is your
machine?". When people get replies like
"This is the most accurate product in its
field!" or "This machine has the highest
imaginable precision!", they feel
fomforted by both answers. Shouldn't
they? Indeed, the terms accurate and
precise are very often used
interchangeably. We will give exact
definitions later in the text, but in a
nutshell, we can say: Accuracy is a
measure for the closeness of some
measurements to a specific value, while
precision is the closeness of the measurements to each other.

These terms are also of extreme importance in Machine Learning. We need them for evaluating ML
algorithms or better their results.

We will present in this chapter of our Python Machine Learning Tutorial four important metrics. These metrics
are used to evaluate the results of classifications. The metrics are:

• Accuracy
• Precision
• Recall
• F1-Score

We will introduce each of these metrics and we will discuss the pro and cons of each of them. Each metric
measures something different about a classifiers performance. The metrics will be of outmost importance for
all the chapters of our machine learning tutorial.

ACCURACY
Accuracy is a measure for the closeness of the measurements to a specific value, while precision is the
closeness of the measurements to each other, i.e. not necessarily to a specific value. To put it in other words: If
we have a set of data points from repeated measurements of the same quantity, the set is said to be accurate if
their average is close to the true value of the quantity being measured. On the other hand, we call the set to be
precise, if the values are close to each other. The two concepts are independent of each other, which means
that the set of data can be accurate, or precise, or both, or neither. We show this in the following diagram:

7
CONFUSION MATRIX
Before we continue with the term accuracy , we want to make sure that you understand what a confusion
matrix is about.

A confusion matrix, also called a contingeny table or error matrix, is used to visualize the performance of a
classifier.

The columns of the matrix represent the instances of the predicted classes and the rows represent the instances
of the actual class. (Note: It can be the other way around as well.)

In the case of binary classification the table has 2 rows and 2 columns.

8
We want to demonstrate the concept with an example.

Example:

Confusion Predicted classes

Matrix
cat dog
classes
Actual

cat 42 8

dog 18 32

This means that the classifier correctly predicted a cat in 42 cases and it wrongly predicted 8 cat instances as
dog. It correctly predicted 32 instances as dog. 18 cases had been wrongly predicted as cat instead of dog.

ACCURACY IN CLASSIFICATION
We are interested in Machine Learning and accuracy is also used as a statistical measure. Accuracy is a
statistical measure which is defined as the quotient of correct predictions (both True positives (TP) and True
negatives (TN)) made by a classifier divided by the sum of all predictions made by the classifier, including
False positves (FP) and False negatives (FN). Therefore, the formula for quantifying binary accuracy is:

TP + TN
accuracy =
TP + TN + FP + FN

where: TP = True positive; FP = False positive; TN = True negative; FN = False negative

The corresponding Confusion Matrix looks like this:

Confusion Predicted classes

Matrix
negative positive
classes
Actual

negative TN FP

positive FN TP

We will now calculate the accuracy for the cat-and-dog classification results. Instead of "True" and "False",
we see here "cat" and "dog". We can calculate the accuracy like this:

9
 TP = 42
TN = 32
FP = 8
FN = 18

Accuracy = (TP + TN)/(TP + TN + FP + FN)

print(Accuracy)
0.74

Let's assume we have a classifier, which always predicts "dog".

Confusion Predicted classes

Matrix
cat dog
classes
Actual

cat 0 50

dog 0 50

We have an accuracy of 0.5 in this case:

TP, TN, FP, FN = 0, 50, 50, 0

Accuracy = (TP + TN)/(TP + TN + FP + FN)
print(Accuracy)
0.5

ACCURACY PARADOX
We will demonstrate the so-called accuracy paradox.

A spam recogition classifier is described by the following confusion matrix:

Confusion Predicted classes

Matrix
spam ham
classes
Actual

spam 4 1

ham 4 91

10
 TP, TN, FP, FN = 4, 91, 1, 4
accuracy = (TP + TN)/(TP + TN + FP + FN)
print(accuracy)
0.95

The following classifier predicts solely "ham" and has the same accuracy.

Confusion Predicted classes

Matrix
spam ham
classes
Actual

spam 0 5

ham 0 95

TP, TN, FP, FN = 0, 95, 5, 0

accuracy = (TP + TN)/(TP + TN + FP + FN)
print(accuracy)
0.95

The accuracy of this classifier is 95%, even though it is not capable of recognizing any spam at all.

PRECISION
Precision is the ratio of the correctly identified positive cases to all the predicted positive cases, i.e. the
correctly and the incorrectly cases predicted as positive . Precision is the fraction of retrieved documents
that are relevant to the query. The formula:

TP
precision =
TP + FP

We will demonstrate this with an example.

Confusion Predicted classes

Matrix
spam ham
classes
Actual

spam 12 14

11
 ham 0 114

We can calculate the precision for our example:

TP = 114
FP = 14
# FN (0) and TN (12) are not needed in the formuala!
precision = TP / (TP + FP)
print(f"precision: {precision:4.2f}")
precision: 0.89

Exercise: Before you go on with the text think about what the value precision means. If you look at the
precision measure of our spam filter example, what does it tell you about the quality of the spam filter? What
do the results of the confusion matrix of an ideal spam filter look like? What is worse, high FP or FN values?

You will find the answers indirectly in the following explanations.

Incidentally, the ideal spam filter would have 0 values for both FP and FN.

The previous result means that 11 mailpieces out of a hundred will be classified as ham, even though they are
spam. 89 are correctly classified as ham. This is a point where we should talk about the costs of
misclassification. It is troublesome when a spam mail is not recognized as "spam" and is instead presented to
us as "ham". If the percentage is not too high, it is annoying but not a disaster. In contrast, when a non-spam
message is wrongly labeled as spam, the email will not be shown in many cases or even automatically deleted.
For example, this carries a high risk of losing customers and friends. The measure precision makes no
statement about this last-mentioned problem class. What about other measures?

We will have a look at recall and F1-score .

RECALL
Recall, also known as sensitivity, is the ratio of the correctly identified positive cases to all the actual positive
cases, which is the sum of the "False Negatives" and "True Positives".

TP
recall =
TP + FN

TP = 114
FN = 0
# FT (14) and TN (12) are not needed in the formuala!
recall = TP / (TP + FN)
print(f"recall: {recall:4.2f}")

12
recall: 1.00

The value 1 means that no non-spam message is wrongly labeled as spam. It is important for a good spam
filter that this value should be 1. We have previously discussed this already.

F1-SCORE
The last measure, we will examine, is the F1-score.

2 precision ⋅ recall
F1 = 1 1
=2⋅
precision + recall
recall
+ precision

TF = 7 # we set the True false values to 5 %

print(" FN FP TP pre acc rec f1")
for FN in range(0, 7):
for FP in range(0, FN+1):
# the sum of FN, FP, TF and TP will be 100:
TP = 100 - FN - FP - TF
#print(FN, FP, TP, FN+FP+TP+TF)
precision = TP / (TP + FP)
accuracy = (TP + TN)/(TP + TN + FP + FN)
recall = TP / (TP + FN)
f1_score = 2 * precision * recall / (precision + recall)
print(f"{FN:6.2f}{FP:6.2f}{TP:6.2f}", end="")
print(f"{precision:6.2f}{accuracy:6.2f}{recall:6.2f}{f1_sc
ore:6.2f}")

13
 FN FP TP pre acc rec f1
0.00 0.00 93.00 1.00 1.00 1.00 1.00
1.00 0.00 92.00 1.00 0.99 0.99 0.99
1.00 1.00 91.00 0.99 0.99 0.99 0.99
2.00 0.00 91.00 1.00 0.99 0.98 0.99
2.00 1.00 90.00 0.99 0.98 0.98 0.98
2.00 2.00 89.00 0.98 0.98 0.98 0.98
3.00 0.00 90.00 1.00 0.98 0.97 0.98
3.00 1.00 89.00 0.99 0.98 0.97 0.98
3.00 2.00 88.00 0.98 0.97 0.97 0.97
3.00 3.00 87.00 0.97 0.97 0.97 0.97
4.00 0.00 89.00 1.00 0.98 0.96 0.98
4.00 1.00 88.00 0.99 0.97 0.96 0.97
4.00 2.00 87.00 0.98 0.97 0.96 0.97
4.00 3.00 86.00 0.97 0.96 0.96 0.96
4.00 4.00 85.00 0.96 0.96 0.96 0.96
5.00 0.00 88.00 1.00 0.97 0.95 0.97
5.00 1.00 87.00 0.99 0.97 0.95 0.97
5.00 2.00 86.00 0.98 0.96 0.95 0.96
5.00 3.00 85.00 0.97 0.96 0.94 0.96
5.00 4.00 84.00 0.95 0.95 0.94 0.95
5.00 5.00 83.00 0.94 0.95 0.94 0.94
6.00 0.00 87.00 1.00 0.97 0.94 0.97
6.00 1.00 86.00 0.99 0.96 0.93 0.96
6.00 2.00 85.00 0.98 0.96 0.93 0.96
6.00 3.00 84.00 0.97 0.95 0.93 0.95
6.00 4.00 83.00 0.95 0.95 0.93 0.94
6.00 5.00 82.00 0.94 0.94 0.93 0.94
6.00 6.00 81.00 0.93 0.94 0.93 0.93

We can see that f1-score best reflects the worse case scenario that the FN value is rising, i.e. ham is
getting classified as spam!

14
REPRESENTATION AND VISUALIZATION OF
DATA

Machine learning is about adapting

models to data. For this reason we begin
by showing how data can be represented
in order to be understood by the computer.

At the beginning of this chapter we quoted

Tom Mitchell's definition of machine
learning: "Well posed Learning Problem:
A computer program is said to learn from
experience E with respect to some task T
and some performance measure P, if its
performance on T, as measured by P,
improves with experience E." Data is the
"raw material" for machine learning. It
learns from data. In Mitchell's definition,
"data" is hidden behind the terms
"experience E" and "performance measure
P". As mentioned earlier, we need labeled
data to learn and test our algorithm.

However, it is recommended that you

familiarize yourself with your data before
you begin training your classifier.

Numpy offers ideal data structures to

represent your data and Matplotlib offers great possibilities for visualizing your data.

In the following, we want to show how to do this using the data in the sklearn module.

IRIS DATASET, "HELLO WORLD" OF MACHINE LEARNING

What was the first program you saw? I bet it might have been a program giving out "Hello World" in some
programming language. Most likely I'm right. Almost every introductory book or tutorial on programming
starts with such a program. It's a tradition that goes back to the 1968 book "The C Programming Language" by
Brian Kernighan and Dennis Ritchie!

The likelihood that the first dataset you will see in an introductory tutorial on machine learning will be the
"Iris dataset" is similarly high. The Iris dataset contains the measurements of 150 iris flowers from 3 different
species:

• Iris-Setosa,
• Iris-Versicolor, and

15
 • Iris-Virginica.

Iris Setosa

Iris Versicolor

Iris Virginica

16
The iris dataset is often used for its simplicity. This dataset is contained in scikit-learn, but before we have a
deeper look into the Iris dataset we will look at the other datasets available in scikit-learn.

17
LOADING THE IRIS DATA WITH SCIKIT-
LEARN

For example, scikit-learn has a very straightforward set of data on these iris species. The data consist of the
following:

• Features in the Iris dataset:

1. sepal length in cm
2. sepal width in cm
3. petal length in cm
4. petal width in cm

• Target classes to predict:

1. Iris Setosa
2. Iris Versicolour
3. Iris Virginica

scikit-learn embeds a copy of the iris CSV file along with a helper function to load it into numpy
arrays:

18
 from [Link] import load_iris
iris = load_iris()

The resulting dataset is a Bunch object:

type(iris)
Output: [Link]

You can see what's available for this data type by using the method keys() :

[Link]()
Output: dict_keys(['data', 'target', 'target_names', 'DESCR', 'featur
e_names', 'filename'])

A Bunch object is similar to a dicitionary, but it additionally allows accessing the keys in an attribute style:

print(iris["target_names"])
print(iris.target_names)
['setosa' 'versicolor' 'virginica']
['setosa' 'versicolor' 'virginica']

The features of each sample flower are stored in the data attribute of the dataset:

n_samples, n_features = [Link]

print('Number of samples:', n_samples)
print('Number of features:', n_features)
# the sepal length, sepal width, petal length and petal width of t
he first sample (first flower)
print([Link][0])
Number of samples: 150
Number of features: 4
[5.1 3.5 1.4 0.2]

The feautures of each flower are stored in the data attribute of the data set. Let's take a look at some of the
samples:

# Flowers with the indices 12, 26, 89, and 114

[Link][[12, 26, 89, 114]]

19
Output: array([[4.8, 3. , 1.4, 0.1],
[5. , 3.4, 1.6, 0.4],
[5.5, 2.5, 4. , 1.3],
[5.8, 2.8, 5.1, 2.4]])

The information about the class of each sample, i.e. the labels, is stored in the "target" attribute of the data set:

print([Link])
print([Link])
(150, 4)
(150,)

print([Link])
[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2
2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2
2 2]

import numpy as np

[Link]([Link])
Output: array([50, 50, 50])

Using NumPy's bincount function (above) we can see that the classes in this dataset are evenly distributed -
there are 50 flowers of each species, with

• class 0: Iris Setosa

• class 1: Iris Versicolor
• class 2: Iris Virginica

These class names are stored in the last attribute, namely target_names :

print(iris.target_names)
['setosa' 'versicolor' 'virginica']

20
The information about the class of each sample of our Iris dataset is stored in the target attribute of the
dataset:

print([Link])
[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2
2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2
2 2]

Beside of the shape of the data, we can also check the shape of the labels, i.e. the [Link] :

Each flower sample is one row in the data array, and the columns (features) represent the flower measurements
in centimeters. For instance, we can represent this Iris dataset, consisting of 150 samples and 4 features, a
2-dimensional array or matrix R 150 × 4 in the following format:

[ ]
x (1) x (1) x (1) x (1)
1 2 3 4

x (2) x (2) x (2) x (2)

1 2 3 4
X= .

x ( 150 ) x ( 150 ) x ( 150 ) x ( 150 )

1 2 3 4

The superscript denotes the ith row, and the subscript denotes the jth feature, respectively.

Generally, we have n rows and k columns:

[ ]
x (1) x (1) x (1) … x (1)
1 2 3 k

x (2) x (2) x (2) … x (2)

1 2 3 k
X= .

x (n) x (n) x (n) … x (n)

1 2 3 k

print([Link])

21
 print([Link])
(150, 4)
(150,)

bincount of NumPy counts the number of occurrences of each value in an array of non-negative integers.
We can use this to check the distribution of the classes in the dataset:

import numpy as np

[Link]([Link])
Output: array([50, 50, 50])

We can see that the classes are distributed uniformly - there are 50 flowers from each species, i.e.

• class 0: Iris-Setosa
• class 1: Iris-Versicolor
• class 2: Iris-Virginica

These class names are stored in the last attribute, namely target_names :

print(iris.target_names)
['setosa' 'versicolor' 'virginica']

22
VISUALISING THE FEATURES OF THE IRIS
DATA SET

The feauture data is four dimensional, but we can visualize one or two of the dimensions at a time using a
simple histogram or scatter-plot.

from [Link] import load_iris

iris = load_iris()
print([Link][[Link]==1][:5])

print([Link][[Link]==1, 0][:5])
[[7. 3.2 4.7 1.4]
[6.4 3.2 4.5 1.5]
[6.9 3.1 4.9 1.5]
[5.5 2.3 4. 1.3]
[6.5 2.8 4.6 1.5]]
[7. 6.4 6.9 5.5 6.5]

HISTOGRAMS OF THE FEATURES

import [Link] as plt

fig, ax = [Link]()
x_index = 3
colors = ['blue', 'red', 'green']

for label, color in zip(range(len(iris.target_names)), colors):

[Link]([Link][[Link]==label, x_index],
label=iris.target_names[label],
color=color)

ax.set_xlabel(iris.feature_names[x_index])
[Link](loc='upper right')
[Link]()

23
EXERCISE
Look at the histograms of the other features, i.e. petal length, sepal widt and sepal length.

SCATTERPLOT WITH TWO FEATURES

The appearance diagram shows two features in one diagram:

import [Link] as plt

fig, ax = [Link]()

x_index = 3
y_index = 0

colors = ['blue', 'red', 'green']

for label, color in zip(range(len(iris.target_names)), colors):

[Link]([Link][[Link]==label, x_index],
[Link][[Link]==label, y_index],
label=iris.target_names[label],
c=color)

ax.set_xlabel(iris.feature_names[x_index])
ax.set_ylabel(iris.feature_names[y_index])
[Link](loc='upper left')
[Link]()

24
EXERCISE
Change x_index and y_index in the above script

Change x_index and y_index in the above script and find a combination of two parameters which maximally
separate the three classes.

GENERALIZATION
We will now look at all feature combinations in one combined diagram:

import [Link] as plt

n = len(iris.feature_names)
fig, ax = [Link](n, n, figsize=(16, 16))

colors = ['blue', 'red', 'green']

for x in range(n):
for y in range(n):
xname = iris.feature_names[x]
yname = iris.feature_names[y]
for color_ind in range(len(iris.target_names)):
ax[x, y].scatter([Link][[Link]==color_ind,
x],
[Link][[Link]==color_ind, y],
label=iris.target_names[color_ind],
c=colors[color_ind])

25
 ax[x, y].set_xlabel(xname)
ax[x, y].set_ylabel(yname)
ax[x, y].legend(loc='upper left')

[Link]()

26
SCATTERPLOT 'MATRICES

Instead of doing it manually we can also use the scatterplot matrix provided by the pandas module.

Scatterplot matrices show scatter plots between all features in the data set, as well as histograms to show the
distribution of each feature.

import pandas as pd

iris_df = [Link]([Link], columns=iris.feature_names)

[Link].scatter_matrix(iris_df,
c=[Link],
figsize=(8, 8)
);

27
3-DIMENSIONAL VISUALIZATION
import [Link] as plt
from [Link] import load_iris
from mpl_toolkits.mplot3d import Axes3D
iris = load_iris()
X = []
for iclass in range(3):
[Link]([[], [], []])
for i in range(len([Link])):
if [Link][i] == iclass:
X[iclass][0].append([Link][i][0])
X[iclass][1].append([Link][i][1])
X[iclass][2].append(sum([Link][i][2:]))

colours = ("r", "g", "y")

fig = [Link]()
ax = fig.add_subplot(111, projection='3d')

for iclass in range(3):

[Link](X[iclass][0], X[iclass][1], X[iclass][2], c=colour
s[iclass])
[Link]()

28
DATASETS IN SKLEARN

Scikit-learn makes available a host of

datasets for testing learning algorithms.
They come in three flavors:

• Packaged Data: these small

datasets are packaged with
the scikit-learn installation,
and can be downloaded
using the tools in

[Link].load_*
• Downloadable Data: these larger datasets are available for download, and scikit-learn includes
tools which streamline this process. These tools can be found in
[Link].fetch_*
• Generated Data: there are several datasets which are generated from models based on a random
seed. These are available in the [Link].make_*

You can explore the available dataset loaders, fetchers, and generators using IPython's tab-completion
functionality. After importing the datasets submodule from sklearn , type

datasets.load_<TAB>

or

datasets.fetch_<TAB>

or

datasets.make_<TAB>

to see a list of available functions.

STRUCTURE OF DATA AND LABELS

Data in scikit-learn is in most cases saved as two-dimensional Numpy arrays with the shape (n, m) . Many
algorithms also accept [Link] matrices of the same shape.

29
 • n: (n_samples) The number of samples: each sample is an item to process (e.g. classify). A
sample can be a document, a picture, a sound, a video, an astronomical object, a row in database
or CSV file, or whatever you can describe with a fixed set of quantitative traits.
• m: (n_features) The number of features or distinct traits that can be used to describe each item in
a quantitative manner. Features are generally real-valued, but may be Boolean or discrete-valued
in some cases.

from sklearn import datasets

Be warned: many of these datasets are quite large, and can take a long time to download!

30
LOADING DIGITS DATA

We will have a closer look at one of these datasets. We look at the digits data set. We will load it first:

from [Link] import load_digits

digits = load_digits()

Again, we can get an overview of the available attributes by looking at the "keys":

[Link]()
Output: dict_keys(['data', 'target', 'target_names', 'images', 'DESC
R'])

Let's have a look at the number of items and features:

n_samples, n_features = [Link]

print((n_samples, n_features))
(1797, 64)

print([Link][0])
print([Link])
[ 0. 0. 5. 13. 9. 1. 0. 0. 0. 0. 13. 15. 10. 15. 5. 0.
0. 3.
15. 2. 0. 11. 8. 0. 0. 4. 12. 0. 0. 8. 8. 0. 0. 5.
8. 0.
0. 9. 8. 0. 0. 4. 11. 0. 1. 12. 7. 0. 0. 2. 14. 5. 1
0. 12.
0. 0. 0. 0. 6. 13. 10. 0. 0. 0.]
[0 1 2 ... 8 9 8]

The data is also available at [Link]. This is the raw data of the images in the form of 8 lines and 8
columns.

With "data" an image corresponds to a one-dimensional Numpy array with the length 64, and "images"
representation contains 2-dimensional numpy arrays with the shape (8, 8)

print("Shape of an item: ", [Link][0].shape)

print("Data type of an item: ", type([Link][0]))
print("Shape of an item: ", [Link][0].shape)

31
 print("Data tpye of an item: ", type([Link][0]))
Shape of an item: (64,)
Data type of an item: <class '[Link]'>
Shape of an item: (8, 8)
Data tpye of an item: <class '[Link]'>

Let's visualize the data. It's little bit more involved than the simple scatter-plot we used above, but we can do it
rather quickly.

# set up the figure

fig = [Link](figsize=(6, 6)) # figure size in inches
fig.subplots_adjust(left=0, right=1, bottom=0, top=1, hspace=0.0
5, wspace=0.05)

# plot the digits: each image is 8x8 pixels

for i in range(64):
ax = fig.add_subplot(8, 8, i + 1, xticks=[], yticks=[])
[Link]([Link][i], cmap=[Link], interpolatio
n='nearest')

# label the image with the target value

[Link](0, 7, str([Link][i]))

32
EXERCISES

EXERCISE 1
sklearn contains a "wine data set".

• Find and load this data set

• Can you find a description?
• What are the names of the classes?
• What are the features?
• Where is the data and the labeled data?

EXERCISE 2:
Create a scatter plot of the features ash and color_intensity of the wine data set.

33
EXERCISE 3:
Create a scatter matrix of the features of the wine dataset.

EXERCISE 4:
Fetch the Olivetti faces dataset and visualize the faces.

SOLUTIONS

SOLUTION TO EXERCISE 1
Loading the "wine data set":

from sklearn import datasets

wine = datasets.load_wine()

The description can be accessed via "DESCR":

In [ ]:
print([Link])

The names of the classes and the features can be retrieved like this:

print(wine.target_names)
print(wine.feature_names)
['class_0' 'class_1' 'class_2']
['alcohol', 'malic_acid', 'ash', 'alcalinity_of_ash', 'magnesiu
m', 'total_phenols', 'flavanoids', 'nonflavanoid_phenols', 'proant
hocyanins', 'color_intensity', 'hue', 'od280/od315_of_diluted_wine
s', 'proline']

data = [Link]
labelled_data = [Link]

SOLUTION TO EXERCISE 2:
from sklearn import datasets
import [Link] as plt

34
wine = datasets.load_wine()

features = 'ash', 'color_intensity'

features_index = [wine.feature_names.index(features[0]),
wine.feature_names.index(features[1])]

colors = ['blue', 'red', 'green']

for label, color in zip(range(len(wine.target_names)), colors):

[Link]([Link][[Link]==label, features_index[0]],
[Link][[Link]==label, features_index[1]],
label=wine.target_names[label],
c=color)

[Link](features[0])
[Link](features[1])
[Link](loc='upper left')
[Link]()

SOLUTION TO EXERCISE 3:
import pandas as pd
from sklearn import datasets

wine = datasets.load_wine()
def rotate_labels(df, axes):
""" changing the rotation of the label output,
y labels horizontal and x labels vertical """

35
 n = len([Link])
for x in range(n):
for y in range(n):
# to get the axis of subplots
ax = axs[x, y]
# to make x axis name vertical
[Link].set_rotation(90)
# to make y axis name horizontal
[Link].set_rotation(0)
# to make sure y axis names are outside the plot area
[Link] = 50

wine_df = [Link]([Link], columns=wine.feature_names)

axs = [Link].scatter_matrix(wine_df,
c=[Link],
figsize=(8, 8),
);

rotate_labels(wine_df, axs)

36
SOLUTION TO EXERCISE 4
from [Link] import fetch_olivetti_faces

# fetch the faces data

faces = fetch_olivetti_faces()

[Link]()
Output: dict_keys(['data', 'images', 'target', 'DESCR'])

37
n_samples, n_features = [Link]
print((n_samples, n_features))
(400, 4096)

[Link](4096)
Output: 64.0

[Link]
Output: (400, 64, 64)

[Link]
Output: (400, 4096)

print([Link]([Link]((400, 4096)) == [Link]))

True

# set up the figure

fig = [Link](figsize=(6, 6)) # figure size in inches
fig.subplots_adjust(left=0, right=1, bottom=0, top=1, hspace=0.0
5, wspace=0.05)

# plot the digits: each image is 8x8 pixels

for i in range(64):
ax = fig.add_subplot(8, 8, i + 1, xticks=[], yticks=[])
[Link]([Link][i], cmap=[Link], interpolation='ne
arest')

# label the image with the target value

[Link](0, 7, str([Link][i]))

38
FURTHER DATASETS
sklearn has many more datasets available. If you still need more, you will find more on this nice List of
datasets for machine-learning research at Wikipedia.

39
DATA GENERATION

GENERATE SYNTHETICAL DATA WITH PYTHON

A problem with machine learning,
especially when you are starting out and
want to learn about the algorithms, is that
it is often difficult to get suitable test data.
Some cost a lot of money, others are not
freely available because they are protected
by copyright. Therefore, artificially
generated test data can be a solution in
some cases.

For this reason, this chapter of our tutorial

deals with the artificial generation of data.
This chapter is about creating artificial
data. In the previous chapters of our
tutorial we learned that Scikit-Learn
(sklearn) contains different data sets. On
the one hand, there are small toy data sets,
but it also offers larger data sets that are
often used in the machine learning
community to test algorithms or also serve
as a benchmark. It provides us with data
coming from the 'real world'.

All this is great, but in many cases this is

still not sufficient. Maybe you find the
right kind of data, but you need more data
of this kind or the data is not completely
the kind of data you were looking for, e.g.
maybe you need more complex or less
complex data. This is the point where you
should consider to create the data
yourself. Here, sklearn offers help. It
includes various random sample
generators that can be used to create
custom-made artificial datasets. Datasets
that meet your ideas of size and
complexity.

The following Python code is a simple example in which we create artificial weather data for some German
cities. We use Pandas and Numpy to create the data:

import numpy as np

40
import pandas as pd

cities = ['Berlin', 'Frankfurt', 'Hamburg',

'Nuremberg', 'Munich', 'Stuttgart',
'Hanover', 'Saarbruecken', 'Cologne',
'Constance', 'Freiburg', 'Karlsruhe'
]

n= len(cities)
data = {'Temperature': [Link](24, 3, n),
'Humidity': [Link](78, 2.5, n),
'Wind': [Link](15, 4, n)
}
df = [Link](data=data, index=cities)
df
Output:
Temperature Humidity Wind

Berlin 20.447301 75.516079 12.566956

Frankfurt 27.319526 77.010523 11.800371

Hamburg 24.783113 80.200985 14.489432

Nuremberg 25.823295 76.430166 19.903070

Munich 21.037610 81.589453 17.677132

Stuttgart 25.560423 75.384543 20.832011

Hanover 22.073368 81.704236 12.421998

Saarbruecken 25.722280 80.131432 10.694502

Cologne 25.658240 79.430957 16.360829

Constance 29.221204 75.626223 17.281035

Freiburg 25.625042 81.227281 6.850105

Karlsruhe 26.245587 81.546979 11.787846

41
ANOTHER EXAMPLE
We will create artificial data for four nonexistent types of flowers. If the names remind you of programming
languages and pizza, it will be no coincidence:

• Flos Pythonem
• Flos Java
• Flos Margarita
• Flos artificialis

The RGB avarage colors values are correspondingly:

• (255, 0, 0)
• (245, 107, 0)
• (206, 99, 1)
• (255, 254, 101)

The average diameter of the calyx is:

• 3.8
• 3.3
• 4.1
• 2.9

Flos pythonem Flos Java

(254, 0, 0) (245, 107, 0)

Flos margarita Flos artificialis

(206, 99, 1) (255, 254, 101)

import [Link] as plt

import numpy as np
import pandas as pd

from [Link] import truncnorm

def truncated_normal(mean=0, sd=1, low=0, upp=10, type=int):

return truncnorm(
(low - mean) / sd, (upp - mean) / sd, loc=mean, scale=sd)

def truncated_normal_floats(mean=0, sd=1, low=0, upp=10, num=100):

res = truncated_normal(mean=mean, sd=sd, low=low, upp=upp)
return [Link](num)

def truncated_normal_ints(mean=0, sd=1, low=0, upp=10, num=100):

42
 res = truncated_normal(mean=mean, sd=sd, low=low, upp=upp)
return [Link](num).astype(np.uint8)

# number of items for each flower class:

number_of_items_per_class = [190, 205, 230, 170]
flowers = {}
# flos Pythonem:
number_of_items = number_of_items_per_class[0]
reds = truncated_normal_ints(mean=254, sd=18, low=235, upp=256,
num=number_of_items)
greens = truncated_normal_ints(mean=107, sd=11, low=88, upp=127,
num=number_of_items)
blues = truncated_normal_ints(mean=0, sd=15, low=0, upp=20,
num=number_of_items)
calyx_dia = truncated_normal_floats(3.8, 0.3, 3.4, 4.2,
num=number_of_items)
data = np.column_stack((reds, greens, blues, calyx_dia))
flowers["flos_pythonem"] = data

# flos Java:
number_of_items = number_of_items_per_class[1]
reds = truncated_normal_ints(mean=245, sd=17, low=226, upp=256,
num=number_of_items)
greens = truncated_normal_ints(mean=107, sd=11, low=88, upp=127,
num=number_of_items)
blues = truncated_normal_ints(mean=0, sd=10, low=0, upp=20,
num=number_of_items)
calyx_dia = truncated_normal_floats(3.3, 0.3, 3.0, 3.5,
num=number_of_items)
data = np.column_stack((reds, greens, blues, calyx_dia))
flowers["flos_java"] = data

# flos Java:
number_of_items = number_of_items_per_class[2]
reds = truncated_normal_ints(mean=206, sd=17, low=175, upp=238,
num=number_of_items)
greens = truncated_normal_ints(mean=99, sd=14, low=80, upp=120,
num=number_of_items)
blues = truncated_normal_ints(mean=1, sd=5, low=0, upp=12,
num=number_of_items)
calyx_dia = truncated_normal_floats(4.1, 0.3, 3.8, 4.4,
num=number_of_items)
data = np.column_stack((reds, greens, blues, calyx_dia))
flowers["flos_margarita"] = data

43
# flos artificialis:
number_of_items = number_of_items_per_class[3]
reds = truncated_normal_ints(mean=255, sd=8, low=2245, upp=2255,
num=number_of_items)
greens = truncated_normal_ints(mean=254, sd=10, low=240, upp=255,
num=number_of_items)
blues = truncated_normal_ints(mean=101, sd=5, low=90, upp=112,
num=number_of_items)
calyx_dia = truncated_normal_floats(2.9, 0.4, 2.4, 3.5,
num=number_of_items)
data = np.column_stack((reds, greens, blues, calyx_dia))
flowers["flos_artificialis"] = data

data = [Link]((flowers["flos_pythonem"],
flowers["flos_java"],
flowers["flos_margarita"],
flowers["flos_artificialis"]
), axis=0)

# assigning the labels

target = [Link](sum(number_of_items_per_class)) # 4 flowers
previous_end = 0
for i in range(1, 5):
num = number_of_items_per_class[i-1]
beg = previous_end
target[beg: beg + num] += i
previous_end = beg + num

conc_data = [Link]((data, [Link]([Link][0],

1)),
axis=1)

[Link]("data/strange_flowers.txt", conc_data, fmt="%2.2f",)

import [Link] as plt

target_names = list([Link]())
feature_names = ['red', 'green', 'blue', 'calyx']
n = 4
fig, ax = [Link](n, n, figsize=(16, 16))

colors = ['blue', 'red', 'green', 'yellow']

for x in range(n):

44
 for y in range(n):
xname = feature_names[x]
yname = feature_names[y]
for color_ind in range(len(target_names)):
ax[x, y].scatter(data[target==color_ind, x],
data[target==color_ind, y],
label=target_names[color_ind],
c=colors[color_ind])

ax[x, y].set_xlabel(xname)
ax[x, y].set_ylabel(yname)
ax[x, y].legend(loc='upper left')

[Link]()

45
GENERATE SYNTHETIC DATA WITH SCIKIT-LEARN
It is a lot easier to use the possibilities of Scikit-Learn to create synthetic data.

The functionalities available in sklearn can be grouped into

1. Generators for classifictation and clustering

2. Generators for creating data for regression
3. Generators for manifold learning
4. Generators for decomposition

46
GENERATORS FOR CLASSIFICATION AND CLUSTERING
We start with the the function make_blobs of [Link] to create 'blob' like data
distributions. By setting the value of centers to n_classes , we determine the number of blobs, i.e.
the clusters. n_samples corresponds to the total number of points equally divided among clusters. If
random_state is not set, we will have random results every time we call the function. We pass an int to
this parameter for reproducible output across multiple function calls.

import numpy as np
import [Link] as plt
from [Link] import make_blobs

n_classes = 4
data, labels = make_blobs(n_samples=1000,
centers=n_classes,
random_state=100)

labels[:7]
Output: array([1, 3, 1, 3, 1, 3, 2])

We will visualize the previously created blob custers with matplotlib:

fig, ax = [Link]()

colours = ('green', 'orange', 'blue', "pink")

for label in range(n_classes):
[Link](x=data[labels==label, 0],
y=data[labels==label, 1],
c=colours[label],
s=40,
label=label)

[Link](xlabel='X',
ylabel='Y',
title='Blobs Examples')

[Link](loc='upper right')

47
Output: <[Link] at 0x7f50f92a4640>

The centers of the blobs were randomly chosen in the previous example. In the following example we set the
centers of the blobs explicitly. We create a list with the center points and pass it to the parameter centers :

import numpy as np
import [Link] as plt
from [Link] import make_blobs

centers = [[2, 3], [4, 5], [7, 9]]

data, labels = make_blobs(n_samples=1000,
centers=[Link](centers),
random_state=1)

labels[:7]
Output: array([0, 1, 1, 0, 2, 2, 2])

Let us have a look at the previously created blob clusters:

fig, ax = [Link]()

colours = ('green', 'orange', 'blue')

for label in range(len(centers)):
[Link](x=data[labels==label, 0],
y=data[labels==label, 1],
c=colours[label],
s=40,

48
 label=label)

[Link](xlabel='X',
ylabel='Y',
title='Blobs Examples')

[Link](loc='upper right')
Output: <[Link] at 0x7f50f91eaca0>

Usually, you want to save your artificially created datasets in a file. For this purpose, we can use the function
savetxt from numpy. Before we can do this we have to reaarange our data. Each row should contain both
the data and the label:

import numpy as np

labels = [Link](([Link][0],1))
all_data = [Link]((data, labels), axis=1)
all_data[:7]
Output: array([[ 1.72415394, 4.22895559, 0. ],
[ 4.16466507, 5.77817418, 1. ],
[ 4.51441156, 4.98274913, 1. ],
[ 1.49102772, 2.83351405, 0. ],
[ 6.0386362 , 7.57298437, 2. ],
[ 5.61044976, 9.83428321, 2. ],
[ 5.69202866, 10.47239631, 2. ]])

49
For some people it might be complicated to understand the combination of reshape and concatenate.
Therefore, you can see an extremely simple example in the following code:

import numpy as np

a = [Link]( [[1, 2], [3, 4]])

b = [Link]( [5, 6])
b = [Link](([Link][0], 1))
print(b)

x = [Link]( (a, b), axis=1)

x
[[5]
[6]]
Output: array([[1, 2, 5],
[3, 4, 6]])

We use the numpy function savetxt to save the data. Don't worry about the strange name, it is just for fun
and for reasons which will be clear soon:

[Link]("[Link]",
all_data,
fmt=['%.3f', '%.3f', '%1d'])
all_data[:10]
Output: array([[ 1.72415394, 4.22895559, 0. ],
[ 4.16466507, 5.77817418, 1. ],
[ 4.51441156, 4.98274913, 1. ],
[ 1.49102772, 2.83351405, 0. ],
[ 6.0386362 , 7.57298437, 2. ],
[ 5.61044976, 9.83428321, 2. ],
[ 5.69202866, 10.47239631, 2. ],
[ 6.14017298, 8.56209179, 2. ],
[ 2.97620068, 5.56776474, 1. ],
[ 8.27980017, 8.54824406, 2. ]])

50
READING THE DATA AND CONVERSION
BACK INTO 'DATA' AND 'LABELS'

We will demonstrate now, how to read in the data again and how to split it into data and labels again:

file_data = [Link]("[Link]")

data = file_data[:,:-1]
labels = file_data[:,2:]

labels = [Link](([Link][0]))

We had called the data file [Link] , because we imagined a strange kind of animal living in the
Sahara desert. The x-values stand for the night vision capabilities of the animals and the y-values correspond
to the colour of the fur, going from sandish to black. We have three kinds of squirrels, 0, 1, and 2. (Be aware
that our squirrals are imaginary squirrels and have nothing to do with the real squirrels of the Sahara!)

import [Link] as plt

colours = ('green', 'red', 'blue', 'magenta', 'yellow', 'cyan')

n_classes = 3

fig, ax = [Link]()
for n_class in range(0, n_classes):
[Link](data[labels==n_class, 0], data[labels==n_class,
1],
c=colours[n_class], s=10, label=str(n_class))

[Link](xlabel='Night Vision',
ylabel='Fur color from sandish to black, 0 to 10 ',
title='Sahara Virtual Squirrel')

[Link](loc='upper right')

51
Output: <[Link] at 0x7f545b4d6340>

We will train our articifical data in the following code:

from sklearn.model_selection import train_test_split

data_sets = train_test_split(data,
labels,
train_size=0.8,
test_size=0.2,
random_state=42 # garantees same output fo
r every run
)

train_data, test_data, train_labels, test_labels = data_sets

# import model
from [Link] import KNeighborsClassifier

# create classifier
knn = KNeighborsClassifier(n_neighbors=8)

# train
[Link](train_data, train_labels)

# test on test data:

calculated_labels = [Link](test_data)
calculated_labels

52
Output: array([2., 0., 1., 1., 0., 1., 2., 2., 2., 2., 0., 1., 0.,
0., 1., 0., 1.,
2., 0., 0., 1., 2., 1., 2., 2., 1., 2., 0., 0., 2.,
0., 2., 2., 0.,
0., 2., 0., 0., 0., 1., 0., 1., 1., 2., 0., 2., 1.,
2., 1., 0., 2.,
1., 1., 0., 1., 2., 1., 0., 0., 2., 1., 0., 1., 1.,
0., 0., 0., 0.,
0., 0., 0., 1., 1., 0., 1., 1., 1., 0., 1., 2., 1.,
2., 0., 2., 1.,
1., 0., 2., 2., 2., 0., 1., 1., 1., 2., 2., 0., 2.,
2., 2., 2., 0.,
0., 1., 1., 1., 2., 1., 1., 1., 0., 2., 1., 2., 0.,
0., 1., 0., 1.,
0., 2., 2., 2., 1., 1., 1., 0., 2., 1., 2., 2., 1.,
2., 0., 2., 0.,
0., 1., 0., 2., 2., 0., 0., 1., 2., 1., 2., 0., 0.,
2., 2., 0., 0.,
1., 2., 1., 2., 0., 0., 1., 2., 1., 0., 2., 2., 0.,
2., 0., 0., 2.,
1., 0., 0., 0., 0., 2., 2., 1., 0., 2., 2., 1., 2.,
0., 1., 1., 1.,
0., 1., 0., 1., 1., 2., 0., 2., 2., 1., 1., 1., 2.])

from sklearn import metrics

print("Accuracy:", metrics.accuracy_score(test_labels, calculate

d_labels))
Accuracy: 0.97

53
OTHER INTERESTING DISTRIBUTIONS

import numpy as np

import [Link] as ds
data, labels = ds.make_moons(n_samples=150,
shuffle=True,
noise=0.19,
random_state=None)

data += [Link](-[Link](data[:,0]),
-[Link](data[:,1]))

[Link](data[:,0]), [Link](data[:,1])
Output: (0.0, 0.34649342272719386)

import [Link] as plt

fig, ax = [Link]()

[Link](data[labels==0, 0], data[labels==0, 1],

c='orange', s=40, label='oranges')
[Link](data[labels==1, 0], data[labels==1, 1],
c='blue', s=40, label='blues')

[Link](xlabel='X',
ylabel='Y',
title='Moons')

#[Link](loc='upper right');

54
Output: [Text(0.5, 0, 'X'), Text(0, 0.5, 'Y'), Text(0.5, 1.0, 'Moon
s')]

We want to scale values that are in a range [min, max] in a range [a, b] .

(b − a) ⋅ (x − min)
f(x) = +a
max − min

We now use this formula to transform both the X and Y coordinates of data into other ranges:

min_x_new, max_x_new = 33, 88

min_y_new, max_y_new = 12, 20

data, labels = ds.make_moons(n_samples=100,

shuffle=True,
noise=0.05,
random_state=None)

min_x, min_y = [Link](data[:,0]), [Link](dat

a[:,1])
max_x, max_y = [Link](data[:,0]), [Link](dat
a[:,1])

#data -= [Link]([min_x, 0])

#data *= [Link]([(max_x_new - min_x_new) / (max_x - min_x), 1])
#data += [Link]([min_x_new, 0])

#data -= [Link]([0, min_y])

#data *= [Link]([1, (max_y_new - min_y_new) / (max_y - min_y)])

55
#data += [Link]([0, min_y_new])

data -= [Link]([min_x, min_y])

data *= [Link]([(max_x_new - min_x_new) / (max_x - min_x), (ma
x_y_new - min_y_new) / (max_y - min_y)])
data += [Link]([min_x_new, min_y_new])

#[Link](data[:,0]), [Link](data[:,0])
data[:6]
Output: array([[71.14479608, 12.28919998],
[62.16584307, 18.75442981],
[61.02613211, 12.80794358],
[64.30752046, 12.32563839],
[81.41469127, 13.64613406],
[82.03929032, 13.63156545]])

def scale_data(data, new_limits, inplace=False ):

if not inplace:
data = [Link]()
min_x, min_y = [Link](data[:,0]), [Link](dat
a[:,1])
max_x, max_y = [Link](data[:,0]), [Link](dat
a[:,1])
min_x_new, max_x_new = new_limits[0]
min_y_new, max_y_new = new_limits[1]
data -= [Link]([min_x, min_y])
data *= [Link]([(max_x_new - min_x_new) / (max_x - min_x),
(max_y_new - min_y_new) / (max_y - min_y)])
data += [Link]([min_x_new, min_y_new])
if inplace:
return None
else:
return data

data, labels = ds.make_moons(n_samples=100,

shuffle=True,
noise=0.05,
random_state=None)

scale_data(data, [(1, 4), (3, 8)], inplace=True)

56
data[:10]
Output: array([[1.19312571, 6.70797983],
[2.74306138, 6.74830445],
[1.15255757, 6.31893824],
[1.03927303, 4.83714182],
[2.91313352, 6.44139267],
[2.13227292, 5.120716 ],
[2.65590196, 3.49417953],
[2.98349928, 5.02232383],
[3.35660593, 3.34679462],
[2.15813861, 4.8036458 ]])

fig, ax = [Link]()

[Link](data[labels==0, 0], data[labels==0, 1],

c='orange', s=40, label='oranges')
[Link](data[labels==1, 0], data[labels==1, 1],
c='blue', s=40, label='blues')

[Link](xlabel='X',
ylabel='Y',
title='moons')

[Link](loc='upper right');

import [Link] as ds
data, labels = ds.make_circles(n_samples=100,
shuffle=True,

57
 noise=0.05,
random_state=None)

fig, ax = [Link]()

[Link](data[labels==0, 0], data[labels==0, 1],

c='orange', s=40, label='oranges')
[Link](data[labels==1, 0], data[labels==1, 1],
c='blue', s=40, label='blues')

[Link](xlabel='X',
ylabel='Y',
title='circles')

[Link](loc='upper right')
Output: <[Link] at 0x7f54588c2e20>

print(__doc__)

import [Link] as plt

from [Link] import make_classification

from [Link] import make_blobs
from [Link] import make_gaussian_quantiles

[Link](figsize=(8, 8))
plt.subplots_adjust(bottom=.05, top=.9, left=.05, right=.95)

58
[Link](321)
[Link]("One informative feature, one cluster per class", fontsi
ze='small')
X1, Y1 = make_classification(n_features=2, n_redundant=0, n_inform
ative=1,
n_clusters_per_class=1)
[Link](X1[:, 0], X1[:, 1], marker='o', c=Y1,
s=25, edgecolor='k')

[Link](322)
[Link]("Two informative features, one cluster per class", fonts
ize='small')
X1, Y1 = make_classification(n_features=2, n_redundant=0, n_inform
ative=2,
n_clusters_per_class=1)
[Link](X1[:, 0], X1[:, 1], marker='o', c=Y1,
s=25, edgecolor='k')

[Link](323)
[Link]("Two informative features, two clusters per class",
fontsize='small')
X2, Y2 = make_classification(n_features=2,
n_redundant=0,
n_informative=2)
[Link](X2[:, 0], X2[:, 1], marker='o', c=Y2,
s=25, edgecolor='k')

[Link](324)
[Link]("Multi-class, two informative features, one cluster",
fontsize='small')
X1, Y1 = make_classification(n_features=2,
n_redundant=0,
n_informative=2,
n_clusters_per_class=1,
n_classes=3)
[Link](X1[:, 0], X1[:, 1], marker='o', c=Y1,
s=25, edgecolor='k')

[Link](325)
[Link]("Gaussian divided into three quantiles", fontsize='smal
l')
X1, Y1 = make_gaussian_quantiles(n_features=2, n_classes=3)
[Link](X1[:, 0], X1[:, 1], marker='o', c=Y1,
s=25, edgecolor='k')

59
 [Link]()
Automatically created module for IPython interactive environment

EXERCISES

EXERCISE 1
Create two testsets which are separable with a perceptron without a bias node.

EXERCISE 2
Create two testsets which are not separable with a dividing line going through the origin.

60
EXERCISE 3
Create a dataset with five classes "Tiger", "Lion", "Penguin", "Dolphin", and "Python". The sets should look
similar to the following diagram:

SOLUTIONS

SOLUTION TO EXERCISE 1
data, labels = make_blobs(n_samples=100,
cluster_std = 0.5,
centers=[[1, 4] ,[4, 1]],
random_state=1)

fig, ax = [Link]()

colours = ["orange", "green"]

label_name = ["Tigers", "Lions"]
for label in range(0, 2):
[Link](data[labels==label, 0], data[labels==label, 1],
c=colours[label], s=40, label=label_name[label])

[Link](xlabel='X',
ylabel='Y',
title='dataset')

61
[Link](loc='upper right')
Output: <[Link] at 0x7f788afb2c40>

SOLUTION TO EXERCISE 2
data, labels = make_blobs(n_samples=100,
cluster_std = 0.5,
centers=[[2, 2] ,[4, 4]],
random_state=1)

fig, ax = [Link]()

colours = ["orange", "green"]

label_name = ["label0", "label1"]
for label in range(0, 2):
[Link](data[labels==label, 0], data[labels==label, 1],
c=colours[label], s=40, label=label_name[label])

[Link](xlabel='X',
ylabel='Y',
title='dataset')

[Link](loc='upper right')

62
Output: <[Link] at 0x7f788af8eac0>

SOLUTION TO EXERCISE 3
import [Link] as ds
data, labels = ds.make_circles(n_samples=100,
shuffle=True,
noise=0.05,
random_state=42)

centers = [[3, 4], [5, 3], [4.5, 6]]

data2, labels2 = make_blobs(n_samples=100,
cluster_std = 0.5,
centers=centers,
random_state=1)

for i in range(len(centers)-1, -1, -1):

labels2[labels2==0+i] = i+2

print(labels2)
labels = [Link]([labels, labels2])
data = data * [1.2, 1.8] + [3, 4]

data = [Link]([data, data2], axis=0)

63
[2 4 4 3 4 4 3 3 2 4 4 2 4 4 3 4 2 4 4 4 4 2 2 4 4 3 2 2 3 2 2 3
2 3 3 3 3
3 4 3 3 2 3 3 3 2 2 2 2 3 4 4 4 2 4 3 3 2 2 3 4 4 3 3 4 2 4 2 4
3 3 4 2 2
3 4 4 2 3 2 3 3 4 2 2 2 2 3 2 4 2 2 3 3 4 4 2 2 4 3]

fig, ax = [Link]()

colours = ["orange", "blue", "magenta", "yellow", "green"]

label_name = ["Tiger", "Lion", "Penguin", "Dolphin", "Python"]
for label in range(0, len(centers)+2):
[Link](data[labels==label, 0], data[labels==label, 1],
c=colours[label], s=40, label=label_name[label])

[Link](xlabel='X',
ylabel='Y',
title='dataset')

[Link](loc='upper right')
Output: <[Link] at 0x7f788b1d42b0>

64
DATA PREPARATION

LEARN, TEST AND EVALUATION DATA

You have your data ready and you are eager to start training the
classifier? But be careful: When your classifier will be finished,
you will need some test data to evaluate your classifier. If you
evaluate your classifier with the data used for learning, you may
see surprisingly good results. What we actually want to test is
the performance of classifying on unknown data.

For this purpose, we need to split our data into two parts:

1. A training set with which the learning algorithm

adapts or learns the model
2. A test set to evaluate the generalization
performance of the model

When you consider how machine learning normally works, the idea of a split between learning and test data
makes sense. Really existing systems train on existing data and if other new data (from customers, sensors or
other sources) comes in, the trained classifier has to predict or classify this new data. We can simulate this
during training with a training and test data set - the test data is a simulation of "future data" that will go into
the system during production.

In this chapter of our Python Machine Learning Tutorial, we will learn how to do the splitting with plain
Python.

We will see also that doing it manually is not necessary, because the train_test_split function from
the model_selection module can do it for us.

If the dataset is sorted by label, we will have to shuffle it before splitting.

65
We separated the dataset into a learn (a.k.a. training) dataset and a test dataset. Best practice is to split it into a
learn, test and an evaluation dataset.

We will train our model (classifier) step by step and each time the result needs to be tested. If we just have a
test dataset. The results of the testing might get into the model. So we will use an evaluation dataset for the
complete learning phase. When our classifier is finished, we will check it with the test dataset, which it has not
"seen" before!

Yet, during our tutorial, we will only use splitings into learn and test datasets.

SPLITTING EXAMPLE: IRIS DATA SET

We will demonstrate the previously discussed topics with the Iris Dataset.

The 150 data sets of the Iris data set are sorted, i.e. the first 50 data correspond to the first flower class (0 =
Setosa), the next 50 to the second flower class (1 = Versicolor) and the remaining data correspond to the last
class (2 = Virginica).

If we were to split our data in the ratio 2/3 (learning set) and 1/3 (test set), the learning set would contain all
the flowers of the first two classes and the test set all the flowers of the third flower class. The classifier could
only learn two classes and the third class would be completely unknown. So we urgently need to mix the data.

Assuming all samples are independent of each other, we want to shuffle the data set randomly before we split
the data set as shown above.

66
In the following we split the data manually:

import numpy as np
from [Link] import load_iris
iris = load_iris()

Looking at the labels of [Link] shows us that the data is sorted.

[Link]
Output: array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2])

The first thing we have to do is rearrange the data so that it is not sorted anymore. For this purpose, we will
use the permutation function of the random submodul of Numpy:

indices = [Link](len([Link]))
indices

67
Output: array([ 98, 56, 37, 60, 94, 142, 117, 121, 10, 15, 8
9, 85, 66,
29, 44, 102, 24, 140, 58, 25, 19, 100, 83, 12
6, 28, 118,
50, 127, 72, 99, 74, 0, 128, 11, 45, 143, 5
4, 79, 34,
32, 95, 92, 46, 146, 3, 9, 73, 101, 23, 7
7, 39, 87,
111, 129, 148, 67, 75, 147, 48, 76, 43, 30, 14
4, 27, 104,
35, 93, 125, 2, 69, 63, 40, 141, 7, 133, 1
8, 4, 12,
109, 33, 88, 71, 22, 110, 42, 8, 134, 5, 9
7, 114, 135,
108, 91, 14, 6, 137, 124, 130, 145, 55, 17, 8
0, 36, 61,
49, 62, 90, 84, 64, 139, 107, 112, 1, 70, 12
3, 38, 132,
31, 16, 13, 21, 113, 120, 41, 106, 65, 20, 11
6, 86, 68,
96, 78, 53, 47, 105, 136, 51, 57, 131, 149, 11
9, 26, 59,
138, 122, 81, 103, 52, 115, 82])

n_test_samples = 12
learnset_data = [Link][indices[:-n_test_samples]]
learnset_labels = [Link][indices[:-n_test_samples]]
testset_data = [Link][indices[-n_test_samples:]]
testset_labels = [Link][indices[-n_test_samples:]]
print(learnset_data[:4], learnset_labels[:4])
print(testset_data[:4], testset_labels[:4])
[[5.1 2.5 3. 1.1]
[6.3 3.3 4.7 1.6]
[4.9 3.6 1.4 0.1]
[5. 2. 3.5 1. ]] [1 1 0 1]
[[7.9 3.8 6.4 2. ]
[5.9 3. 5.1 1.8]
[6. 2.2 5. 1.5]
[5. 3.4 1.6 0.4]] [2 2 2 0]

SPLITS WITH SKLEARN

Even though it was not difficult to split the data manually into a learn (train) and an evaluation (test) set, we
don't have to do the splitting manually as shown above. Since this is often required in machine learning, scikit-
learn has a predefined function for dividing data into training and test sets.

68
We will demonstrate this below. We will use 80% of the data as training and 20% as test data. We could just as
well have taken 70% and 30%, because there are no hard and fast rules. The most important thing is that you
rate your system fairly based on data it did not see during exercise! In addition, there must be enough data in
both data sets.

from [Link] import load_iris

from sklearn.model_selection import train_test_split
iris = load_iris()
data, labels = [Link], [Link]

res = train_test_split(data, labels,

train_size=0.8,
test_size=0.2,
random_state=42)
train_data, test_data, train_labels, test_labels = res

n = 7
print(f"The first {n} data sets:")
print(test_data[:7])
print(f"The corresponding {n} labels:")
print(test_labels[:7])
The first 7 data sets:
[[6.1 2.8 4.7 1.2]
[5.7 3.8 1.7 0.3]
[7.7 2.6 6.9 2.3]
[6. 2.9 4.5 1.5]
[6.8 2.8 4.8 1.4]
[5.4 3.4 1.5 0.4]
[5.6 2.9 3.6 1.3]]
The corresponding 7 labels:
[1 0 2 1 1 0 1]

STRATIFIED RANDOM SAMPLE

Especially with relatively small amounts of data, it is better to stratify the division. Stratification means that
we keep the original class proportion of the data set in the test and training sets. We calculate the class
proportions of the previous split in percent using the following code. To calculate the number of occurrences
of each class, we use the numpy function 'bincount'. It counts the number of occurrences of each value in the
array of non-negative integers passed as an argument.

import numpy as np
print('All:', [Link](labels) / float(len(labels)) * 100.0)
print('Training:', [Link](train_labels) / float(len(train_lab
els)) * 100.0)

69
 print('Test:', [Link](test_labels) / float(len(test_labels))
* 100.0)
All: [33.33333333 33.33333333 33.33333333]
Training: [33.33333333 34.16666667 32.5 ]
Test: [33.33333333 30. 36.66666667]

To stratify the division, we can pass the label array as an additional argument to the train_test_split function:

from [Link] import load_iris

from sklearn.model_selection import train_test_split
iris = load_iris()
data, labels = [Link], [Link]

res = train_test_split(data, labels,

train_size=0.8,
test_size=0.2,
random_state=42,
stratify=labels)
train_data, test_data, train_labels, test_labels = res

print('All:', [Link](labels) / float(len(labels)) * 100.0)

print('Training:', [Link](train_labels) / float(len(train_lab
els)) * 100.0)
print('Test:', [Link](test_labels) / float(len(test_labels))
* 100.0)
All: [33.33333333 33.33333333 33.33333333]
Training: [33.33333333 33.33333333 33.33333333]
Test: [33.33333333 33.33333333 33.33333333]

This was a stupid example to test the stratified random sample, because the Iris data set has the same
proportions, i.e. each class 50 elements.

We will work now with the file strange_flowers.txt of the directory data . This data set is created
in the chapter Generate Datasets in Python The classes in this dataset have different numbers of items. First
we load the data:

content = [Link]("data/strange_flowers.txt", delimiter=" ")

data = content[:, :-1] # cut of the target column
labels = content[:, -1]
[Link]
[Link]
Output: (795,)

70
res = train_test_split(data, labels,
train_size=0.8,
test_size=0.2,
random_state=42,
stratify=labels)
train_data, test_data, train_labels, test_labels = res

# [Link] expects non negative integers:

print('All:', [Link]([Link](int)) / float(len(label
s)) * 100.0)
print('Training:', [Link](train_labels.astype(int)) / float(l
en(train_labels)) * 100.0)
print('Test:', [Link](test_labels.astype(int)) / float(len(te
st_labels)) * 100.0)
All: [ 0. 23.89937107 25.78616352 28.93081761 21.3836478 ]
Training: [ 0. 23.89937107 25.78616352 28.93081761 21.3836
478 ]
Test: [ 0. 23.89937107 25.78616352 28.93081761 21.3836478
]

71
K-NEAREST-NEIGHBOR CLASSIFIER

"Show me who your friends are and I’ll

tell you who you are?"

The concept of the k-nearest neighbor

classifier can hardly be simpler described.
This is an old saying, which can be found
in many languages and many cultures. It's
also metnioned in other words in the
Bible: "He who walks with wise men will
be wise, but the companion of fools will
suffer harm" (Proverbs 13:20 )

This means that the concept of the k-

nearest neighbor classifier is part of our
everyday life and judging: Imagine you
meet a group of people, they are all very
young, stylish and sportive. They talk
about there friend Ben, who isn't with them. So, what is your imagination of Ben? Right, you imagine him as
being yong, stylish and sportive as well.

If you learn that Ben lives in a neighborhood where people vote conservative and that the average income is
above 200000 dollars a year? Both his neighbors make even more than 300,000 dollars per year? What do you
think of Ben? Most probably, you do not consider him to be an underdog and you may suspect him to be a
conservative as well?

The principle behind nearest neighbor classification consists in finding a predefined number, i.e. the 'k' - of
training samples closest in distance to a new sample, which has to be classified. The label of the new sample
will be defined from these neighbors. k-nearest neighbor classifiers have a fixed user defined constant for the
number of neighbors which have to be determined. There are also radius-based neighbor learning algorithms,
which have a varying number of neighbors based on the local density of points, all the samples inside of a
fixed radius. The distance can, in general, be any metric measure: standard Euclidean distance is the most
common choice. Neighbors-based methods are known as non-generalizing machine learning methods, since
they simply "remember" all of its training data. Classification can be computed by a majority vote of the
nearest neighbors of the unknown sample.

The k-NN algorithm is among the simplest of all machine learning algorithms, but despite its simplicity, it has
been quite successful in a large number of classification and regression problems, for example character
recognition or image analysis.

Now let's get a little bit more mathematically:

As explained in the chapter Data Preparation, we need labeled learning and test data. In contrast to other
classifiers, however, the pure nearest-neighbor classifiers do not do any learning, but the so-called learning set
LS is a basic component of the classifier. The k-Nearest-Neighbor Classifier (kNN) works directly on the
learned samples, instead of creating rules compared to other classification methods.

72
Nearest Neighbor Algorithm:

Given a set of categories C = {c 1, c 2, . . . c m}, also called classes, e.g. {"male", "female"}. There is also a
learnset LS consisting of labelled instances:

LS = {(o 1, c o ), (o 2, c o ), ⋯(o n, c o )}
1 2 n

As it makes no sense to have less lebelled items than categories, we can postulate that

n > m and in most cases even n ⋙ m (n much greater than m.)

The task of classification consists in assigning a category or class c to an arbitrary instance o.

For this, we have to differentiate between two cases:

• Case 1:
The instance o is an element of LS, i.e. there is a tupel (o, c) ∈ LS
In this case, we will use the class c as the classification result.
• Case 2:
We assume now that o is not in LS, or to be precise:
∀c ∈ C, (o, c) ∉ LS

o is compared with all the instances of LS. A distance metric d is used for the comparisons.
We determine the k closest neighbors of o, i.e. the items with the smallest distances.
k is a user defined constant and a positive integer, which is usually small.
The number k is typically chosen as the square root of LS, the total number of points in the training data set.

To determine the k nearest neighbors we reorder LS in the following way:

(o i , c o ), (o i , c o ), ⋯(o i , c o )
1 i1 2 i2 n in
so that d(o i , o) ≤ d(o i , o) is true for all 1 ≤ j ≤ n − 1
j j+1
The set of k-nearest neighbors N k consists of the first k elements of this ordering, i.e.
N k = {(o i , c o ), (o i , c o ), ⋯(o i , c o )}
1 i1 2 i2 k ik
The most common class in this set of nearest neighbors N k will be assigned to the instance o. If there is no
unique most common class, we take an arbitrary one of these.

There is no general way to define an optimal value for 'k'. This value depends on the data. As a general rule
we can say that increasing 'k' reduces the noise but on the other hand makes the boundaries less distinct.

The algorithm for the k-nearest neighbor classifier is among the simplest of all machine learning algorithms.
k-NN is a type of instance-based learning, or lazy learning. In machine learning, lazy learning is understood
to be a learning method in which generalization of the training data is delayed until a query is made to the
system. On the other hand, we have eager learning, where the system usually generalizes the training data
before receiving queries. In other words: The function is only approximated locally and all the computations
are performed, when the actual classification is being performed.

73
The following picture shows in a simple way how the nearest neighbor classifier works. The puzzle piece is
unknown. To find out which animal it might be we have to find the neighbors. If k=1 , the only neighbor is a
cat and we assume in this case that the puzzle piece should be a cat as well. If k=4 , the nearest neighbors
contain one chicken and three cats. In this case again, it will be save to assume that our object in question
should be a cat.

K-NEAREST-NEIGHBOR FROM SCRATCH

PREPARING THE DATASET

Before we actually start with writing a nearest neighbor classifier, we need to think about the data, i.e. the
learnset and the testset. We will use the "iris" dataset provided by the datasets of the sklearn module.

The data set consists of 50 samples from each of three species of Iris

• Iris setosa,
• Iris virginica and
• Iris versicolor.

Four features were measured from each sample: the length and the width of the sepals and petals, in
centimetres.

import numpy as np
from sklearn import datasets

iris = datasets.load_iris()

74
data = [Link]
labels = [Link]

for i in [0, 79, 99, 101]:

print(f"index: {i:3}, features: {data[i]}, label: {label
s[i]}")
index: 0, features: [5.1 3.5 1.4 0.2], label: 0
index: 79, features: [5.7 2.6 3.5 1. ], label: 1
index: 99, features: [5.7 2.8 4.1 1.3], label: 1
index: 101, features: [5.8 2.7 5.1 1.9], label: 2

We create a learnset from the sets above. We use permutation from [Link] to split the data
randomly.

# seeding is only necessary for the website

#so that the values are always equal:
[Link](42)
indices = [Link](len(data))

n_training_samples = 12
learn_data = data[indices[:-n_training_samples]]
learn_labels = labels[indices[:-n_training_samples]]
test_data = data[indices[-n_training_samples:]]
test_labels = labels[indices[-n_training_samples:]]

print("The first samples of our learn set:")

print(f"{'index':7s}{'data':20s}{'label':3s}")
for i in range(5):
print(f"{i:4d} {learn_data[i]} {learn_labels[i]:3}")

print("The first samples of our test set:")

print(f"{'index':7s}{'data':20s}{'label':3s}")
for i in range(5):
print(f"{i:4d} {learn_data[i]} {learn_labels[i]:3}")

75
 The first samples of our learn set:
index data label
0 [6.1 2.8 4.7 1.2] 1
1 [5.7 3.8 1.7 0.3] 0
2 [7.7 2.6 6.9 2.3] 2
3 [6. 2.9 4.5 1.5] 1
4 [6.8 2.8 4.8 1.4] 1
The first samples of our test set:
index data label
0 [6.1 2.8 4.7 1.2] 1
1 [5.7 3.8 1.7 0.3] 0
2 [7.7 2.6 6.9 2.3] 2
3 [6. 2.9 4.5 1.5] 1
4 [6.8 2.8 4.8 1.4] 1

The following code is only necessary to visualize the data of our learnset. Our data consists of four values per
iris item, so we will reduce the data to three values by summing up the third and fourth value. This way, we
are capable of depicting the data in 3-dimensional space:

#%matplotlib widget

import [Link] as plt

from mpl_toolkits.mplot3d import Axes3D

colours = ("r", "b")

X = []
for iclass in range(3):
[Link]([[], [], []])
for i in range(len(learn_data)):
if learn_labels[i] == iclass:
X[iclass][0].append(learn_data[i][0])
X[iclass][1].append(learn_data[i][1])
X[iclass][2].append(sum(learn_data[i][2:]))

colours = ("r", "g", "y")

fig = [Link]()
ax = fig.add_subplot(111, projection='3d')

for iclass in range(3):

[Link](X[iclass][0], X[iclass][1], X[iclass][2], c=colo
urs[iclass])
[Link]()

76
DISTANCE METRICS
We have already mentioned in detail, we calculate the distances between the points of the sample and the
object to be classified. To calculate these distances we need a distance function.

In n-dimensional vector rooms, one usually uses one of the following three distance metrics:

• Euclidean Distance

The Euclidean distance between two points x and y in either the plane or 3-dimensional
space measures the length of a line segment connecting these two points. It can be calculated
from the Cartesian coordinates of the points using the Pythagorean theorem, therefore it is also
occasionally being called the Pythagorean distance. The general formula is

d(x, y) =
√ ∑ (x i − y i) 2
i=1

• Manhattan Distance

It is defined as the sum of the absolute values of the differences between the coordinates of x
and y:
n

d(x, y) = ∑ | xi − yi |
i=1

• Minkowski Distance

The Minkowski distance generalizes the Euclidean and the Manhatten distance in one distance
metric. If we set the parameter p in the following formula to 1 we get the manhattan distance
an using the value 2 gives us the euclidean distance:

77
 ( )
1
n
p
d(x, y) = ∑ | xi − yi | p
i=1

The following diagram visualises the Euclidean and the Manhattan distance:

The blue line illustrates the Eucliden distance between the green and red dot. Otherwise you can also move
over the orange, green or yellow line from the green point to the red point. The lines correspond to the
manhatten distance. The length is equal.

DETERMINING THE NEIGHBORS

To determine the similarity between two instances, we will use the Euclidean distance.

We can calculate the Euclidean distance with the function norm of the module [Link] :

def distance(instance1, instance2):

""" Calculates the Eucledian distance between two instance
s"""
return [Link]([Link](instance1, instance2))

print(distance([3, 5], [1, 1]))

print(distance(learn_data[3], learn_data[44]))

78
 4.47213595499958
3.4190641994557516

The function get_neighbors returns a list with k neighbors, which are closest to the instance
test_instance :

def get_neighbors(training_set,
labels,
test_instance,
k,
distance):
"""
get_neighors calculates a list of the k nearest neighbors
of an instance 'test_instance'.
The function returns a list of k 3-tuples.
Each 3-tuples consists of (index, dist, label)
where
index is the index from the training_set,
dist is the distance between the test_instance and the
instance training_set[index]
distance is a reference to a function used to calculate the
distances
"""
distances = []
for index in range(len(training_set)):
dist = distance(test_instance, training_set[index])
[Link]((training_set[index], dist, labels[inde
x]))
[Link](key=lambda x: x[1])
neighbors = distances[:k]
return neighbors

We will test the function with our iris samples:

for i in range(5):
neighbors = get_neighbors(learn_data,
learn_labels,
test_data[i],
3,
distance=distance)
print("Index: ",i,'\n',
"Testset Data: ",test_data[i],'\n',
"Testset Label: ",test_labels[i],'\n',
"Neighbors: ",neighbors,'\n')

79
 Index: 0
Testset Data: [5.7 2.8 4.1 1.3]
Testset Label: 1
Neighbors: [(array([5.7, 2.9, 4.2, 1.3]), 0.141421356237309
95, 1), (array([5.6, 2.7, 4.2, 1.3]), 0.17320508075688815, 1), (ar
ray([5.6, 3. , 4.1, 1.3]), 0.22360679774997935, 1)]

Index: 1
Testset Data: [6.5 3. 5.5 1.8]
Testset Label: 2
Neighbors: [(array([6.4, 3.1, 5.5, 1.8]), 0.141421356237309
3, 2), (array([6.3, 2.9, 5.6, 1.8]), 0.24494897427831783, 2), (arr
ay([6.5, 3. , 5.2, 2. ]), 0.3605551275463988, 2)]

Index: 2
Testset Data: [6.3 2.3 4.4 1.3]
Testset Label: 1
Neighbors: [(array([6.2, 2.2, 4.5, 1.5]), 0.264575131106458
6, 1), (array([6.3, 2.5, 4.9, 1.5]), 0.574456264653803, 1), (arra
y([6. , 2.2, 4. , 1. ]), 0.5916079783099617, 1)]

Index: 3
Testset Data: [6.4 2.9 4.3 1.3]
Testset Label: 1
Neighbors: [(array([6.2, 2.9, 4.3, 1.3]), 0.200000000000000
18, 1), (array([6.6, 3. , 4.4, 1.4]), 0.2645751311064587, 1), (arr
ay([6.6, 2.9, 4.6, 1.3]), 0.3605551275463984, 1)]

Index: 4
Testset Data: [5.6 2.8 4.9 2. ]
Testset Label: 2
Neighbors: [(array([5.8, 2.7, 5.1, 1.9]), 0.316227766016837
5, 2), (array([5.8, 2.7, 5.1, 1.9]), 0.3162277660168375, 2), (arra
y([5.7, 2.5, 5. , 2. ]), 0.33166247903553986, 2)]

VOTING TO GET A SINGLE RESULT

We will write a vote function now. This functions uses the class Counter from collections to count
the quantity of the classes inside of an instance list. This instance list will be the neighbors of course. The
function vote returns the most common class:

from collections import Counter

def vote(neighbors):

80
 class_counter = Counter()
for neighbor in neighbors:
class_counter[neighbor[2]] += 1
return class_counter.most_common(1)[0][0]

We will test 'vote' on our training samples:

for i in range(n_training_samples):
neighbors = get_neighbors(learn_data,
learn_labels,
test_data[i],
3,
distance=distance)
print("index: ", i,
", result of vote: ", vote(neighbors),
", label: ", test_labels[i],
", data: ", test_data[i])
index: 0 , result of vote: 1 , label: 1 , data: [5.7 2.8 4.1
1.3]
index: 1 , result of vote: 2 , label: 2 , data: [6.5 3. 5.5
1.8]
index: 2 , result of vote: 1 , label: 1 , data: [6.3 2.3 4.4
1.3]
index: 3 , result of vote: 1 , label: 1 , data: [6.4 2.9 4.3
1.3]
index: 4 , result of vote: 2 , label: 2 , data: [5.6 2.8 4.9
2. ]
index: 5 , result of vote: 2 , label: 2 , data: [5.9 3. 5.1
1.8]
index: 6 , result of vote: 0 , label: 0 , data: [5.4 3.4 1.7
0.2]
index: 7 , result of vote: 1 , label: 1 , data: [6.1 2.8 4.
1.3]
index: 8 , result of vote: 1 , label: 2 , data: [4.9 2.5 4.5
1.7]
index: 9 , result of vote: 0 , label: 0 , data: [5.8 4. 1.2
0.2]
index: 10 , result of vote: 1 , label: 1 , data: [5.8 2.6 4.
1.2]
index: 11 , result of vote: 2 , label: 2 , data: [7.1 3. 5.9
2.1]

We can see that the predictions correspond to the labelled results, except in case of the item with the index 8.

81
'vote_prob' is a function like 'vote' but returns the class name and the probability for this class:

def vote_prob(neighbors):
class_counter = Counter()
for neighbor in neighbors:
class_counter[neighbor[2]] += 1
labels, votes = zip(*class_counter.most_common())
winner = class_counter.most_common(1)[0][0]
votes4winner = class_counter.most_common(1)[0][1]
return winner, votes4winner/sum(votes)

for i in range(n_training_samples):
neighbors = get_neighbors(learn_data,
learn_labels,
test_data[i],
5,
distance=distance)
print("index: ", i,
", vote_prob: ", vote_prob(neighbors),
", label: ", test_labels[i],
", data: ", test_data[i])

82
 index: 0 , vote_prob: (1, 1.0) , label: 1 , data: [5.7 2.8
4.1 1.3]
index: 1 , vote_prob: (2, 1.0) , label: 2 , data: [6.5 3.
5.5 1.8]
index: 2 , vote_prob: (1, 1.0) , label: 1 , data: [6.3 2.3
4.4 1.3]
index: 3 , vote_prob: (1, 1.0) , label: 1 , data: [6.4 2.9
4.3 1.3]
index: 4 , vote_prob: (2, 1.0) , label: 2 , data: [5.6 2.8
4.9 2. ]
index: 5 , vote_prob: (2, 0.8) , label: 2 , data: [5.9 3.
5.1 1.8]
index: 6 , vote_prob: (0, 1.0) , label: 0 , data: [5.4 3.4
1.7 0.2]
index: 7 , vote_prob: (1, 1.0) , label: 1 , data: [6.1 2.8
4. 1.3]
index: 8 , vote_prob: (1, 1.0) , label: 2 , data: [4.9 2.5
4.5 1.7]
index: 9 , vote_prob: (0, 1.0) , label: 0 , data: [5.8 4.
1.2 0.2]
index: 10 , vote_prob: (1, 1.0) , label: 1 , data: [5.8 2.6
4. 1.2]
index: 11 , vote_prob: (2, 1.0) , label: 2 , data: [7.1 3.
5.9 2.1]

THE WEIGHTED NEAREST NEIGHBOUR CLASSIFIER

We looked only at k items in the vicinity of an unknown object „UO", and had a majority vote. Using the
majority vote has shown quite efficient in our previous example, but this didn't take into account the following
reasoning: The farther a neighbor is, the more it "deviates" from the "real" result. Or in other words, we can
trust the closest neighbors more than the farther ones. Let's assume, we have 11 neighbors of an unknown item
UO. The closest five neighbors belong to a class A and all the other six, which are farther away belong to a
class B. What class should be assigned to UO? The previous approach says B, because we have a 6 to 5 vote
in favor of B. On the other hand the closest 5 are all A and this should count more.

To pursue this strategy, we can assign weights to the neighbors in the following way: The nearest neighbor of
an instance gets a weight 1 / 1, the second closest gets a weight of 1 / 2 and then going on up to 1 / k for the
farthest away neighbor.

This means that we are using the harmonic series as weights:

k
1 1 1
∑ 1 / (i + 1) = 1 + 2 + 3 + . . . +
k
i

We implement this in the following function:

83
def vote_harmonic_weights(neighbors, all_results=True):
class_counter = Counter()
number_of_neighbors = len(neighbors)
for index in range(number_of_neighbors):
class_counter[neighbors[index][2]] += 1/(index+1)
labels, votes = zip(*class_counter.most_common())
#print(labels, votes)
winner = class_counter.most_common(1)[0][0]
votes4winner = class_counter.most_common(1)[0][1]
if all_results:
total = sum(class_counter.values(), 0.0)
for key in class_counter:
class_counter[key] /= total
return winner, class_counter.most_common()
else:
return winner, votes4winner / sum(votes)

for i in range(n_training_samples):
neighbors = get_neighbors(learn_data,
learn_labels,
test_data[i],
6,
distance=distance)
print("index: ", i,
", result of vote: ",
vote_harmonic_weights(neighbors,
all_results=True))
index: 0 , result of vote: (1, [(1, 1.0)])
index: 1 , result of vote: (2, [(2, 1.0)])
index: 2 , result of vote: (1, [(1, 1.0)])
index: 3 , result of vote: (1, [(1, 1.0)])
index: 4 , result of vote: (2, [(2, 0.9319727891156463), (1, 0.0
6802721088435375)])
index: 5 , result of vote: (2, [(2, 0.8503401360544217), (1, 0.1
4965986394557826)])
index: 6 , result of vote: (0, [(0, 1.0)])
index: 7 , result of vote: (1, [(1, 1.0)])
index: 8 , result of vote: (1, [(1, 1.0)])
index: 9 , result of vote: (0, [(0, 1.0)])
index: 10 , result of vote: (1, [(1, 1.0)])
index: 11 , result of vote: (2, [(2, 1.0)])

The previous approach took only the ranking of the neighbors according to their distance in account. We can

84
improve the voting by using the actual distance. To this purpos we will write a new voting function:

def vote_distance_weights(neighbors, all_results=True):

class_counter = Counter()
number_of_neighbors = len(neighbors)
for index in range(number_of_neighbors):
dist = neighbors[index][1]
label = neighbors[index][2]
class_counter[label] += 1 / (dist**2 + 1)
labels, votes = zip(*class_counter.most_common())
#print(labels, votes)
winner = class_counter.most_common(1)[0][0]
votes4winner = class_counter.most_common(1)[0][1]
if all_results:
total = sum(class_counter.values(), 0.0)
for key in class_counter:
class_counter[key] /= total
return winner, class_counter.most_common()
else:
return winner, votes4winner / sum(votes)

for i in range(n_training_samples):
neighbors = get_neighbors(learn_data,
learn_labels,
test_data[i],
6,
distance=distance)
print("index: ", i,
", result of vote: ",
vote_distance_weights(neighbors,
all_results=True))

85
index: 0 , result of vote: (1, [(1, 1.0)])
index: 1 , result of vote: (2, [(2, 1.0)])
index: 2 , result of vote: (1, [(1, 1.0)])
index: 3 , result of vote: (1, [(1, 1.0)])
index: 4 , result of vote: (2, [(2, 0.8490154592118361), (1, 0.1
5098454078816387)])
index: 5 , result of vote: (2, [(2, 0.6736137462184478), (1, 0.3
263862537815521)])
index: 6 , result of vote: (0, [(0, 1.0)])
index: 7 , result of vote: (1, [(1, 1.0)])
index: 8 , result of vote: (1, [(1, 1.0)])
index: 9 , result of vote: (0, [(0, 1.0)])
index: 10 , result of vote: (1, [(1, 1.0)])
index: 11 , result of vote: (2, [(2, 1.0)])

ANOTHER EXAMPLE FOR NEAREST NEIGHBOR CLASSIFICATION

We want to test the previous functions with another very simple dataset:

train_set = [(1, 2, 2),

(-3, -2, 0),
(1, 1, 3),
(-3, -3, -1),
(-3, -2, -0.5),
(0, 0.3, 0.8),
(-0.5, 0.6, 0.7),
(0, 0, 0)
]

labels = ['apple', 'banana', 'apple',

'banana', 'apple', "orange",
'orange', 'orange']

k = 2
for test_instance in [(0, 0, 0), (2, 2, 2),
(-3, -1, 0), (0, 1, 0.9),
(1, 1.5, 1.8), (0.9, 0.8, 1.6)]:
neighbors = get_neighbors(train_set,
labels,
test_instance,
k,
distance=distance)

print("vote distance weights: ",

vote_distance_weights(neighbors))

86
vote distance weights: ('orange', [('orange', 1.0)])
vote distance weights: ('apple', [('apple', 1.0)])
vote distance weights: ('banana', [('banana', 0.529411764705882
4), ('apple', 0.47058823529411764)])
vote distance weights: ('orange', [('orange', 1.0)])
vote distance weights: ('apple', [('apple', 1.0)])
vote distance weights: ('apple', [('apple', 0.5084745762711865),
('orange', 0.4915254237288135)])

KNN IN LINGUISTICS
The next example comes from computer linguistics. We show how we can use a k-nearest neighbor classifier
to recognize misspelled words.

We use a module called levenshtein, which we have implemented in our tutorial on Levenshtein Distance.

from levenshtein import levenshtein

cities = open("data/city_names.txt").readlines()
cities = [[Link]() for city in cities]

for city in ["Freiburg", "Frieburg", "Freiborg",

"Hamborg", "Sahrluis"]:
neighbors = get_neighbors(cities,
cities,
city,
2,
distance=levenshtein)

print("vote_distance_weights: ", vote_distance_weights(neighbo

rs))
vote_distance_weights: ('Freiberg', [('Freiberg', 0.8333333333333
334), ('Freising', 0.16666666666666669)])
vote_distance_weights: ('Lüneburg', [('Lüneburg', 0.5), ('Duisbur
g', 0.5)])
vote_distance_weights: ('Freiberg', [('Freiberg', 0.8333333333333
334), ('Freising', 0.16666666666666669)])
vote_distance_weights: ('Hamburg', [('Hamburg', 0.714285714285714
3), ('Bamberg', 0.28571428571428575)])
vote_distance_weights: ('Saarlouis', [('Saarlouis', 0.83870967741
93549), ('Bayreuth', 0.16129032258064516)])

Marvin and James introduce us to our next example:

87
Can you help Marvin and James?

88
You will need an English dictionary and a k-nearest Neighbor classifier to solve this problem. If you work
under Linux (especially Ubuntu), you can find a file with a British-English dictionary under /usr/share/dict/
british-english. Windows users and others can download the file as

[Link]

We use extremely misspelled words in the following example. We see that our simple vote_prob function is
doing well only in two cases: In correcting "holpposs" to "helpless" and "blagrufoo" to "barefoot". Whereas
our distance voting is doing well in all cases. Okay, we have to admit that we had "liberty" in mind, when we
wrote "liberdi", but suggesting "liberal" is a good choice.

words = []
with open("[Link]") as fh:
for line in fh:
word = [Link]()
[Link](word)

89
for word in ["holpful", "kundnoss", "holpposs", "thoes", "innersta
nd",
"blagrufoo", "liberdi"]:
neighbors = get_neighbors(words,
words,
word,
3,
distance=levenshtein)

print("vote_distance_weights: ", vote_distance_weights(neighbo

rs,
all_res
ults=False))
print("vote_prob: ", vote_prob(neighbors))
print("vote_distance_weights: ", vote_distance_weights(neighbo
rs))

90
vote_distance_weights: ('helpful', 0.5555555555555556)
vote_prob: ('helpful', 0.3333333333333333)
vote_distance_weights: ('helpful', [('helpful', 0.555555555555555
6), ('doleful', 0.22222222222222227), ('hopeful', 0.22222222222222
227)])
vote_distance_weights: ('kindness', 0.5)
vote_prob: ('kindness', 0.3333333333333333)
vote_distance_weights: ('kindness', [('kindness', 0.5), ('fondnes
s', 0.25), ('kudos', 0.25)])
vote_distance_weights: ('helpless', 0.3333333333333333)
vote_prob: ('helpless', 0.3333333333333333)
vote_distance_weights: ('helpless', [('helpless', 0.3333333333333
333), ("hippo's", 0.3333333333333333), ('hippos', 0.33333333333333
33)])
vote_distance_weights: ('hoes', 0.3333333333333333)
vote_prob: ('hoes', 0.3333333333333333)
vote_distance_weights: ('hoes', [('hoes', 0.3333333333333333),
('shoes', 0.3333333333333333), ('thees', 0.3333333333333333)])
vote_distance_weights: ('understand', 0.5)
vote_prob: ('understand', 0.3333333333333333)
vote_distance_weights: ('understand', [('understand', 0.5), ('int
erstate', 0.25), ('understands', 0.25)])
vote_distance_weights: ('barefoot', 0.4333333333333333)
vote_prob: ('barefoot', 0.3333333333333333)
vote_distance_weights: ('barefoot', [('barefoot', 0.4333333333333
333), ('Baguio', 0.2833333333333333), ('Blackfoot', 0.283333333333
3333)])
vote_distance_weights: ('liberal', 0.4)
vote_prob: ('liberal', 0.3333333333333333)
vote_distance_weights: ('liberal', [('liberal', 0.4), ('libert
y', 0.4), ('Hibernia', 0.2)])

91
NEURAL NETWORKS

INTRODUCTION
When we say "Neural Networks", we
mean artificial Neural Networks (ANN).
The idea of ANN is based on biological
neural networks like the brain of living
being.

The basic structure of a neural network -

both an artificial and a living one - is the
neuron. A neuron in biology consists of
three major parts: the soma (cell body),
the dendrites and the axon.

The dendrites branch of from the soma in

a tree-like way and become thinner with
every branch. They receive signals
(impulses) from other neurons at synapses. The axon - there is always only one - also leaves the soma and
usually tend to extend for longer distances than the dentrites. The axon is used for sending the output of the
neuron to other neurons or better to the synapsis of other neurons.

BIOLOGICAL NEURON
The following image by Quasar Jarosz, courtesy of Wikipedia, illustrates this:

92
ABSTRACTION OF A BIOLOGICAL NEURON AND ARTIFICIAL NEURON
Even though the above image is already an abstraction for a biologist, we can further abstract it:

A perceptron of artificial neural networks is simulating a biological neuron.

It is amazingly simple, what is going on inside the body of a perceptron or neuron. The input signals get
multiplied by weight values, i.e. each input has its corresponding weight. This way the input can be adjusted
individually for every x i. We can see all the inputs as an input vector and the corresponding weights as the
weights vector.

When a signal comes in, it gets multiplied by a weight value that is assigned to this particular input. That is, if
a neuron has three inputs, then it has three weights that can be adjusted individually. The weights usually get
adjusted during the learn phase.
After this the modified input signals are summed up. It is also possible to add additionally a so-called bias 'b'
to this sum. The bias is a value which can also be adjusted during the learn phase.

Finally, the actual output has to be determined. For this purpose an activation or step function Φ is applied to
the weighted sum of the input values.

93
The simplest form of an activation function is a binary function. If the result of the summation is greater than
some threshold s, the result of Φ will be 1, otherwise 0.

Φ(x) =
{ 1
0
wx + b > s
otherwise

NUMBER OF NEURON IN ANIMALS

We will examine in the following chapters artificial neuronal networks of various sizes and structures. It is
interesting to have a look at the total numbers of neurons some animals have:

• Roundworm: 302
• Jellyfish

94
In [ ]:

95
FROM DIVIDING LINES TO NEURAL
NETWORKS

We will develop a simple neural network in this chapter of our tutorial. A network capable of separating two
classes, which are separable by a straight line in a 2-dimensional feature space.

LINE SEPARATION
Before we start programming a simple neural
network, we are going to develop a different concept.
We want to search for straight lines that separate two
points or two classes in a plane. We will only look at
straight lines going through the origin. We will look
at general straight lines later in the tutorial.

You could imagine that you have two attributes

describing an eddible object like a fruit for example:
"sweetness" and "sourness".

We could describe this by points in a two-

dimensional space. The A axis is used for the values
of sweetness and the y axis is correspondingly used
for the sourness values. Imagine now that we have
two fruits as points in this space, i.e. an orange at
position (3.5, 1.8) and a lemon at (1.1, 3.9).

We could define dividing lines to define the points which are more lemon-like and which are more orange-
like.

In the following diagram, we depict one lemon and one orange. The green line is separating both points. We
assume that all other lemons are above this line and all oranges will be below this line.

96
The green line is defined by

y = mx

where:

m is the slope or gradient of the line and x is the independent variable of the function.

p2
m= x
p1

This means that a point P ′ = (p ′ , p ′ ) is on this line, if the following condition is fulfilled:
1 2

mp ′ − p ′ = 0
1 2

The following Python program plots a graph depicting the previously described situation:

import [Link] as plt

import numpy as np

97
 X = [Link](0, 7)
fig, ax = [Link]()

[Link](3.5, 1.8, "or",

color="darkorange",
markersize=15)
[Link](1.1, 3.9, "oy",
markersize=15)

point_on_line = (4, 4.5)

[Link](1.1, 3.9, "oy", markersize=15)
# calculate gradient:
m = point_on_line[1] / point_on_line[0]
[Link](X, m * X, "g-", linewidth=3)
[Link]()

It is clear that a point A = (a 1, a 2) is not on the line, if m ⋅ a 1 − a 2 is not equal to 0. We want to know more.
We want to know, if a point is above or below a straight line.

98
If a point B = (b 1, b 2) is below this line, there must be a δ B > 0 so that the point (b 1, b 2 + δ B) will be on the
line.

This means that

m ⋅ b 1 − (b 2 + δ B) = 0

which can be rearranged to

m ⋅ b1 − b2 = δB

Finally, we have a criteria for a point to be below the line. m ⋅ b 1 − b 2 is positve, because δ B is positive.

The reasoning for "a point is above the line" is analogue: If a point A = (a 1, a 2) is above the line, there must
be a δ A > 0 so that the point (a 1, a 2 − δ A) will be on the line.

This means that

m ⋅ a 1 − (a 2 − δ A) = 0

which can be rearranged to

m ⋅ a1 − a2 = − δA

99
In summary, we can say: A point P(p 1, p 2) lies

• below the straight line if m ⋅ p 1 − p 2 > 0

• on the straight line if m ⋅ p 1 − p 2 = 0
• above the straight line if m ⋅ p 1 − p 2 < 0

We can now verify this on our fruits. The lemon has the coordinates (1.1, 3.9) and the orange the coordinates
3.5, 1.8. The point on the line, which we used to define our separation straight line has the values (4, 4.5). So
m is 4.5 divides by 4.

lemon = (1.1, 3.9)

orange = (3.5, 1.8)
m = 4.5 / 4

# check if orange is below the line,

# positive value is expected:
print(orange[0] * m - orange[1])

# check if lemon is above the line,

# negative value is expected:
print(lemon[0] * m - lemon[1])
2.1375
-2.6624999999999996

We did not calculate the green line using mathematical formulas or methods, but arbitrarily determined it by
visual judgement. We could have chosen other lines as well.

The following Python program calculates and renders a bunch of lines. All going through the origin, i.e. the
point (0, 0). The red ones are completely unusable for the purpose of separating the two fruits, because in
these cases both the lemon and the orange are on the same side of the straight line. However, it is obvious that
even the green ones might not be too useful if we have more than these two fruits. Some lemons might be
sweeter and some oranges can be quite sour.

import numpy as np
import [Link] as plt

def create_distance_function(a, b, c):

""" 0 = ax + by + c """
def distance(x, y):
"""
returns tuple (d, pos)
d is the distance

100
 If pos == -1 point is below the line,
0 on the line and +1 if above the line
"""
nom = a * x + b * y + c
if nom == 0:
pos = 0
elif (nom<0 and b<0) or (nom>0 and b>0):
pos = -1
else:
pos = 1
return ([Link](nom) / [Link]( a ** 2 + b ** 2), pos)
return distance

orange = (4.5, 1.8)

lemon = (1.1, 3.9)
fruits_coords = [orange, lemon]

fig, ax = [Link]()
ax.set_xlabel("sweetness")
ax.set_ylabel("sourness")
x_min, x_max = -1, 7
y_min, y_max = -1, 8
ax.set_xlim([x_min, x_max])
ax.set_ylim([y_min, y_max])
X = [Link](x_min, x_max, 0.1)

step = 0.05
for x in [Link](0, 1+step, step):
slope = [Link]([Link](x))
dist4line1 = create_distance_function(slope, -1, 0)
Y = slope * X
results = []
for point in fruits_coords:
[Link](dist4line1(*point))
if (results[0][1] != results[1][1]):
[Link](X, Y, "g-", linewidth=0.8, alpha=0.9)
else:
[Link](X, Y, "r-", linewidth=0.8, alpha=0.9)

size = 10
for (index, (x, y)) in enumerate(fruits_coords):
if index== 0:
[Link](x, y, "o",
color="darkorange",
markersize=size)

101
 else:
[Link](x, y, "oy",
markersize=size)

[Link]()

Basically, we have carried out a classification based on our dividing line. Even if hardly anyone would
describe this as such.

It is easy to imagine that we have more lemons and oranges with slightly different sourness and sweetness
values. This means we have a class of lemons ( class1 ) and a class of oranges class2 . This is depicted
in the following diagram.

102
We are going to "grow" oranges and lemons with a Python program. We will create these two classes by
randomly creating points within a circle with a defined center point and radius. The following Python code
will create the classes:

import numpy as np
import [Link] as plt

def points_within_circle(radius,
center=(0, 0),
number_of_points=100):
center_x, center_y = center
r = radius * [Link]([Link]((number_of_points,)))
theta = [Link]((number_of_points,)) * 2 * [Link]
x = center_x + r * [Link](theta)
y = center_y + r * [Link](theta)
return x, y

X = [Link](0, 8)
fig, ax = [Link]()
oranges_x, oranges_y = points_within_circle(1.6, (5, 2), 100)
lemons_x, lemons_y = points_within_circle(1.9, (2, 5), 100)

[Link](oranges_x,
oranges_y,
c="orange",
label="oranges")
[Link](lemons_x,

103
 lemons_y,
c="y",
label="lemons")

[Link](X, 0.9 * X, "g-", linewidth=2)

[Link]()
[Link]()
[Link]()

The dividing line was again arbitrarily set by eye. The question arises how to do this systematically? We are
still only looking at straight lines going through the origin, which are uniquely defined by its slope. the
following Python program calculates a dividing line by going through all the fruits and dynamically adjusts
the slope of the dividing line we want to calculate. If a point is above the line but should be below the line, the
slope will be increment by the value of learning_rate . If the point is below the line but should be above
the line, the slope will be decremented by the value of learning_rate .

import numpy as np
import [Link] as plt
from itertools import repeat
from random import shuffle

X = [Link](0, 8)
fig, ax = [Link]()
[Link](oranges_x,
oranges_y,
c="orange",
label="oranges")
[Link](lemons_x,

104
 lemons_y,
c="y",
label="lemons")

fruits = list(zip(oranges_x,
oranges_y,
repeat(0, len(oranges_x))))
fruits += list(zip(lemons_x,
lemons_y,
repeat(1, len(oranges_x))))
shuffle(fruits)

def adjust(learning_rate=0.3, slope=0.3):

line = None
counter = 0
for x, y, label in fruits:
res = slope * x - y
#print(label, res)
if label == 0 and res < 0:
# point is above line but should be below
# => increment slope
slope += learning_rate
counter += 1
[Link](X, slope * X,
linewidth=2, label=str(counter))

elif label == 1 and res > 0:

# point is below line but should be above
# => decrement slope
#print(res, label)
slope -= learning_rate
counter += 1
[Link](X, slope * X,
linewidth=2, label=str(counter))
return slope

slope = adjust()
[Link](X,
slope * X,
linewidth=2)
[Link]()
[Link]()
[Link]()

105
 print(slope)

[<[Link].Line2D object at 0x7f53b0a22c50>]

Let's start with a different slope from the 'lemon side':

X = [Link](0, 8)
fig, ax = [Link]()
[Link](oranges_x,
oranges_y,
c="orange",
label="oranges")
[Link](lemons_x,
lemons_y,
c="y",
label="lemons")

slope = adjust(learning_rate=0.2, slope=3)

[Link](X,
slope * X,
linewidth=2)
[Link]()
[Link]()
[Link]()

print(slope)

106
 0.9999999999999996

A SIMPLE NEURAL NETWORK

We were capable of separating the two classes with a straight line. One might wonder what this has to do with
neural networks. We will work out this connection below.

We are going to define a neural network to classify the previous data sets. Our neural network will only
consist of one neuron. A neuron with two input values, one for 'sourness' and one for 'sweetness'.

The two input values - called in_data in our Python program below - have to be weighted by weight
values. So solve our problem, we define a Perceptron class. An instance of the class is a Perceptron (or
Neuron). It can be initialized with the input_length, i.e. the number of input values, and the weights, which can
be given as a list, tuple or an array. If there are no values for the weights given or the parameter is set to None,
we will initialize the weights to 1 / input_length.

In the following example choose -0.45 and 0.5 as the values for the weights. This is not the normal way to do
it. A Neural Network calculates the weights automatically during its training phase, as we will learn later.

import numpy as np

107
class Perceptron:

def __init__(self, weights):

"""
'weights' can be a numpy array, list or a tuple with the
actual values of the weights. The number of input values
is indirectly defined by the length of 'weights'
"""
[Link] = [Link](weights)

def __call__(self, in_data):

weighted_input = [Link] * in_data
weighted_sum = weighted_input.sum()
return weighted_sum

p = Perceptron(weights=[-0.45, 0.5])

for point in zip(oranges_x[:10], oranges_y[:10]):

res = p(point)
print(res, end=", ")

for point in zip(lemons_x[:10], lemons_y[:10]):

res = p(point)
print(res, end=", ")
-1.8131460150609238, -1.1931285955719209, -1.3127632381850327,
-1.3925163810790897, -0.7522874009031233, -0.8402958901009828,
-1.9330506389030604, -1.490534974734101, -0.4441170096959772, -1.9
942817372340516, 1.998076257605724, 1.1512784858148413, 2.51418870
799987, 0.4867012212497872, 1.7962680593822624, 0.875162742271260
9, 1.5455925862569528, 1.6976576197574347, 1.4467637066140102, 1.4
634541513290587,

We can see that we get a negative value, if we input an orange and a posive value, if we input a lemon. With
this knowledge, we can calculate the accuracy of our neural network on this data set:

from collections import Counter

evaluation = Counter()
for point in zip(oranges_x, oranges_y):
res = p(point)
if res < 0:
evaluation['corrects'] += 1
else:
evaluation['wrongs'] += 1

108
 for point in zip(lemons_x, lemons_y):
res = p(point)
if res >= 0:
evaluation['corrects'] += 1
else:
evaluation['wrongs'] += 1

print(evaluation)
Counter({'corrects': 200})

How does the calculation work? We multiply the input values with the weights and get negative and positive
values. Let us examine what we get, if the calculation results in 0:

w1 ⋅ x1 + w2 ⋅ x2 = 0

We can change this equation into

w1
x2 = − ⋅ x1
w2

We can compare this with the general form of a straight line

y=m⋅x+c

where:

• m is the slope or gradient of the line.

• c is the y-intercept of the line.
• x is the independent variable of the function.

We can easily see that our equation corresponds to the definition of a line and the slope (aka gradient) m is
w1
− w and c is equal to 0.
2

This is a straight line separating the oranges and lemons, which is called the decision boundary.

We visualize this with the following Python program:

import time
import [Link] as plt
slope = 0.1

X = [Link](0, 8)

109
 fig, ax = [Link]()
[Link](oranges_x,
oranges_y,
c="orange",
label="oranges")
[Link](lemons_x,
lemons_y,
c="y",
label="lemons")

slope = 0.45 / 0.5

[Link](X, slope * X, linewidth=2)

[Link]()
[Link]()

print(slope)

0.9

TRAINING A NEURAL NETWORK

As we mentioned in the previous section: We didn't train our network. We have adjusted the weights to values
that we know would form a dividing line. We want to demonstrate now, what is necessary to train our simple
neural network.

Before we start with this task, we will separate our data into training and test data in the following Python
program. By setting the random_state to the value 42 we will have the same output for every run, which can
be benifial for debugging purposes.

110
 from sklearn.model_selection import train_test_split
import random

oranges = list(zip(oranges_x, oranges_y))

lemons = list(zip(lemons_x, lemons_y))

# labelling oranges with 0 and lemons with 1:

labelled_data = list(zip(oranges + lemons,
[0] * len(oranges) + [1] * len(lemons)))
[Link](labelled_data)

data, labels = zip(*labelled_data)

res = train_test_split(data, labels,

train_size=0.8,
test_size=0.2,
random_state=42)
train_data, test_data, train_labels, test_labels = res
print(train_data[:10], train_labels[:10])
[(2.592320569178846, 5.623712204925406), (4.7943502284049355, 0.88
39613414681706), (2.1239534889189637, 5.377962359316873), (4.13018
3870483639, 3.2036358839244397), (2.5700607722439957, 3.4894903329
620393), (1.1874742907020708, 4.248237496795156), (4.9754099376160
54, 3.258818001021547), (2.4858113049930375, 3.778544332039814),
(0.759896779289841, 4.699741038079466), (1.3275488108562907, 4.204
176294559159)] [1, 0, 1, 0, 1, 1, 0, 1, 1, 1]

As we start with two arbitrary weights, we cannot expect the result to be correct. For some points (fruits) it
may return the proper value, i.e. 1 for a lemon and 0 for an orange. In case we get the wrong result, we have to
correct our weight values. First we have to calculate the error. The error is the difference between the target or
expected value ( target_result ) and the calculated value ( calculated_result ). With this error
we have to adjust the weight values with an incremental value, i.e. w 1 = w 1 + Δw 1 and w 2 = w 2 + Δw 2

111
If the error e is 0, i.e. the target result is equal to the calculated result, we don't have to do anything. The
network is perfect for these input values. If the error is not equal, we have to change the weights. We have to
change the weights by adding small values to them. These values may be positive or negative. The amount we
have a change a weight value depends on the error and on the input value. Let us assume, x 1 = 0 and x 2 > 0.
In this case the result in this case solely results on the input x 2. This on the other hand means that we can
minimize the error by changing solely w 2. If the error is negative, we will have to add a negative value to it,
and if the error is positive, we will have to add a positive value to it. From this we can understand that
whatever the input values are, we can multiply them with the error and we get values, we can add to the
weights. One thing is still missing: Doing this we would learn to fast. We have many samples and each sample
should only change the weights a little bit. Therefore we have to multiply this result with a learning rate
( self.learning_rate ). The learning rate is used to control how fast the weights are updated. Small
values for the learning rate result in a long training process, larger values bear the risk of ending up in sub-
optimal weight values. We will have a closer look at this in our chapter on backpropagation.

We are ready now to write the code for adapting the weights, which means training the network. For this
purpose, we add a method 'adjust' to our Perceptron class. The task of this method is to crrect the error.

import numpy as np
from collections import Counter

class Perceptron:

def __init__(self,
weights,
learning_rate=0.1):
"""
'weights' can be a numpy array, list or a tuple with the
actual values of the weights. The number of input values
is indirectly defined by the length of 'weights'
"""
[Link] = [Link](weights)
self.learning_rate = learning_rate

@staticmethod
def unit_step_function(x):
if x < 0:
return 0
else:
return 1

def __call__(self, in_data):

weighted_input = [Link] * in_data
weighted_sum = weighted_input.sum()
#print(in_data, weighted_input, weighted_sum)

112
 return Perceptron.unit_step_function(weighted_sum)

def adjust(self,
target_result,
calculated_result,
in_data):
if type(in_data) != [Link]:
in_data = [Link](in_data) #
error = target_result - calculated_result
if error != 0:
correction = error * in_data * self.learning_rate
[Link] += correction
#print(target_result, calculated_result, error, in_dat
a, correction, [Link])

def evaluate(self, data, labels):

evaluation = Counter()
for index in range(len(data)):
label = int(round(p(data[index]),0))
if label == labels[index]:
evaluation["correct"] += 1
else:
evaluation["wrong"] += 1
return evaluation

p = Perceptron(weights=[0.1, 0.1],
learning_rate=0.3)

for index in range(len(train_data)):

[Link](train_labels[index],
p(train_data[index]),
train_data[index])

evaluation = [Link](train_data, train_labels)

print(evaluation.most_common())
evaluation = [Link](test_data, test_labels)
print(evaluation.most_common())

print([Link])
[('correct', 160)]
[('correct', 40)]
[-1.68135341 2.07512397]

113
Both on the learning and on the test data, we have only correct values, i.e. our network was capable of learning
automatically and successfully!

We visualize the decision boundary with the following program:

import [Link] as plt

import numpy as np

X = [Link](0, 7)
fig, ax = [Link]()

lemons = [train_data[i] for i in range(len(train_data)) if train_l

abels[i] == 1]
lemons_x, lemons_y = zip(*lemons)
oranges = [train_data[i] for i in range(len(train_data)) if trai
n_labels[i] == 0]
oranges_x, oranges_y = zip(*oranges)

[Link](oranges_x, oranges_y, c="orange")

[Link](lemons_x, lemons_y, c="y")

w1 = [Link][0]
w2 = [Link][1]
m = -w1 / w2
[Link](X, m * X, label="decision boundary")
[Link]()
[Link]()
print([Link])

[-1.68135341 2.07512397]

114
Let us have a look on the algorithm "in motion".

import numpy as np
import [Link] as plt
import [Link] as cm

p = Perceptron(weights=[0.1, 0.1],
learning_rate=0.3)
number_of_colors = 7
colors = [Link]([Link](0, 1, number_of_colors))

fig, ax = [Link]()
ax.set_xticks(range(8))
ax.set_ylim([-2, 8])

counter = 0
for index in range(len(train_data)):
old_weights = [Link]()
[Link](train_labels[index],
p(train_data[index]),
train_data[index])
if not np.array_equal(old_weights, [Link]):
color = "orange" if train_labels[index] == 0 else
"y"
[Link](train_data[index][0],
train_data[index][1],
color=color)
[Link](str(counter),
(train_data[index][0], train_data[index][1]))
m = -[Link][0] / [Link][1]
print(index, m, [Link], train_data[index])
[Link](X, m * X, label=str(counter), color=colors[counte
r])
counter += 1
[Link]()
[Link]()

115
1 -3.0400347553192493 [-1.45643048 -0.4790835 ] (5.18810161174240
7, 1.930278325463612)
2 0.5905980182798966 [-0.73406347 1.24291557] (2.407890035938178
7, 5.739996893315745)
18 6.70051650445074 [-2.03694068 0.30399756] (4.342924008657758,
3.129726697580847)
20 0.5044094409795936 [-0.87357998 1.73188666] (3.87786897216146
7, 4.759630340827767)
27 2.7418853617419434 [-2.39560903 0.87370868] (5.07343016541601
7, 2.8605932860372967)
31 0.8102423930878537 [-1.68135341 2.07512397] (2.3808520725267
2, 4.004717642222739)

Each of the points in the diagram above cause a change in the weights. We see them numbered in the order of
their appearance and the corresponding straight line. This way we can see how the networks "learns".

116
SIMPLE NEURAL NETWORKS

LINEARLY SEPARABLE DATA SETS

As we have shown in the previous chapter of our tutorial on machine
learning, a neural network consisting of only one perceptron was enough to
separate our example classes. Of course, we carefully designed these
classes to make it work. There are many clusters of classes, for whichit will
not work. We are going to have a look at some other examples and will
discuss cases where it will not be possible to separate the classes.

Our classes have been linearly separable. Linear separability make sense
in Euclidean geometry. Two sets of points (or classes) are called linearly
separable, if at least one straight line in the plane exists so that all the
points of one class are on one side of the line and all the points of the other
class are on the other side.

More formally:

If two data clusters (classes) can be separated by a decision boundary in the

form of a linear equation
n

∑ xi ⋅ wi = 0
i=1

they are called linearly separable.

Otherwise, i.e. if such a decision boundary does not exist, the two classes are called linearly inseparable. In
this case, we cannot use a simple neural network.

PERCEPTRON FOR THE AND FUNCTION

In our next example we will program a Neural Network in Python which implements the logical "And"
function. It is defined for two inputs in the following way:

Input1 Input2 Output

0 0 0

0 1 0

1 0 0

117
 Input1 Input2 Output

1 1 1

We learned in the previous chapter that a neural network with one perceptron and two input values can be
interpreted as a decision boundary, i.e. straight line dividing two classes. The two classes we want to classify
in our example look like this:

import [Link] as plt

import numpy as np

fig, ax = [Link]()
xmin, xmax = -0.2, 1.4
X = [Link](xmin, xmax, 0.1)
[Link](0, 0, color="r")
[Link](0, 1, color="r")
[Link](1, 0, color="r")
[Link](1, 1, color="g")
ax.set_xlim([xmin, xmax])
ax.set_ylim([-0.1, 1.1])
m = -1
#[Link](X, m * X + 1.2, label="decision boundary")
[Link]()
Output: []

We also found out that such a primitive neural network is only capable of creating straight lines going through
the origin. So dividing lines like this:

118
 import [Link] as plt
import numpy as np

fig, ax = [Link]()
xmin, xmax = -0.2, 1.4
X = [Link](xmin, xmax, 0.1)
ax.set_xlim([xmin, xmax])
ax.set_ylim([-0.1, 1.1])
m = -1
for m in [Link](0, 6, 0.1):
[Link](X, m * X )
[Link](0, 0, color="r")
[Link](0, 1, color="r")
[Link](1, 0, color="r")
[Link](1, 1, color="g")
[Link]()
Output: []

We can see that none of these straight lines can be used as decision boundary nor any other lines going
through the origin.

We need a line

y=m⋅x+c

where the intercept c is not equal to 0.

For example the line

y = − x + 1.2

119
could be used as a separating line for our problem:

import [Link] as plt

import numpy as np

fig, ax = [Link]()
xmin, xmax = -0.2, 1.4
X = [Link](xmin, xmax, 0.1)
[Link](0, 0, color="r")
[Link](0, 1, color="r")
[Link](1, 0, color="r")
[Link](1, 1, color="g")
ax.set_xlim([xmin, xmax])
ax.set_ylim([-0.1, 1.1])
m, c = -1, 1.2
[Link](X, m * X + c )
[Link]()
Output: []

The question now is whether we can find a solution with minor modifications of our network model? Or in
other words: Can we create a perceptron capable of defining arbitrary decision boundaries?

The solution consists in the addition of a bias node.

SINGLE PERCEPTRON WITH A BIAS

A perceptron with two input values and a bias corresponds to a general straight line. With the aid of the bias
value b we can train the perceptron to determine a decision boundary with a non zero intercept c .

120
While the input values can change, a bias value always remains constant. Only the weight of the bias node can
be adapted.

Now, the linear equation for a perceptron contains a bias:

n

∑ wi ⋅ xi + wn + 1 ⋅ b = 0
i=1

In our case it looks like this:

w1 ⋅ x1 + w2 ⋅ x2 + w3 ⋅ b = 0

this is equivalent with

w1 w3
x2 = − ⋅ x1 − ⋅b
w2 w2

This means:

w1
m= −
w2

and

w3
c= − ⋅b
w2

import numpy as np
from collections import Counter

class Perceptron:

def __init__(self,

121
 weights,
bias=1,
learning_rate=0.3):
"""
'weights' can be a numpy array, list or a tuple with the
actual values of the weights. The number of input values
is indirectly defined by the length of 'weights'
"""
[Link] = [Link](weights)
[Link] = bias
self.learning_rate = learning_rate

@staticmethod
def unit_step_function(x):
if x <= 0:
return 0
else:
return 1

def __call__(self, in_data):

in_data = [Link]( (in_data, [[Link]]) )
result = [Link] @ in_data
return Perceptron.unit_step_function(result)

def adjust(self,
target_result,
in_data):
if type(in_data) != [Link]:
in_data = [Link](in_data) #
calculated_result = self(in_data)
error = target_result - calculated_result
if error != 0:
in_data = [Link]( (in_data, [[Link]]) )
correction = error * in_data * self.learning_rate
[Link] += correction

def evaluate(self, data, labels):

evaluation = Counter()
for sample, label in zip(data, labels):
result = self(sample) # predict
if result == label:
evaluation["correct"] += 1
else:
evaluation["wrong"] += 1
return evaluation

122
We assume that the above Python code with the Perceptron class is stored in your current working directory
under the name '[Link]'.

import numpy as np
from perceptrons import Perceptron

def labelled_samples(n):
for _ in range(n):
s = [Link](0, 2, (2,))
yield (s, 1) if s[0] == 1 and s[1] == 1 else (s, 0)

p = Perceptron(weights=[0.3, 0.3, 0.3],

learning_rate=0.2)

for in_data, label in labelled_samples(30):

[Link](label,
in_data)

test_data, test_labels = list(zip(*labelled_samples(30)))

evaluation = [Link](test_data, test_labels)

print(evaluation)
Counter({'correct': 30})

import [Link] as plt

import numpy as np

fig, ax = [Link]()
xmin, xmax = -0.2, 1.4
X = [Link](xmin, xmax, 0.1)
[Link](0, 0, color="r")
[Link](0, 1, color="r")
[Link](1, 0, color="r")
[Link](1, 1, color="g")
ax.set_xlim([xmin, xmax])
ax.set_ylim([-0.1, 1.1])
m = -[Link][0] / [Link][1]
c = -[Link][2] / [Link][1]
print(m, c)
[Link](X, m * X + c )
[Link]()

123
 -3.0000000000000004 3.0000000000000013
Output: []

We will create another example with linearly separable data sets, which need a bias node to be separable. We
will use the make_blobs function from [Link] :

from [Link] import make_blobs

n_samples = 250
samples, labels = make_blobs(n_samples=n_samples,
centers=([2.5, 3], [6.7, 7.9]),
random_state=0)

Let us visualize the previously created data:

import [Link] as plt

colours = ('green', 'magenta', 'blue', 'cyan', 'yellow', 'red')

fig, ax = [Link]()

for n_class in range(2):

[Link](samples[labels==n_class][:, 0], samples[labels==n_c
lass][:, 1],
c=colours[n_class], s=40, label=str(n_class))

124
n_learn_data = int(n_samples * 0.8) # 80 % of available data point
s
learn_data, test_data = samples[:n_learn_data], samples[-n_learn_d
ata:]
learn_labels, test_labels = labels[:n_learn_data], labels[-n_lear
n_data:]

from perceptrons import Perceptron

p = Perceptron(weights=[0.3, 0.3, 0.3],

learning_rate=0.8)

for sample, label in zip(learn_data, learn_labels):

[Link](label,
sample)

evaluation = [Link](learn_data, learn_labels)

print(evaluation)
Counter({'correct': 200})

Let us visualize the decision boundary:

import [Link] as plt

fig, ax = [Link]()

# plotting learn data

colours = ('green', 'blue')

125
 for n_class in range(2):
[Link](learn_data[learn_labels==n_class][:, 0],
learn_data[learn_labels==n_class][:, 1],
c=colours[n_class], s=40, label=str(n_class))

# plotting test data

colours = ('lightgreen', 'lightblue')
for n_class in range(2):
[Link](test_data[test_labels==n_class][:, 0],
test_data[test_labels==n_class][:, 1],
c=colours[n_class], s=40, label=str(n_class))

X = [Link]([Link](samples[:,0]))
m = -[Link][0] / [Link][1]
c = -[Link][2] / [Link][1]
print(m, c)
[Link](X, m * X + c )
[Link]()
[Link]()
-1.5513529034664024 11.736643489707035

In the following section, we will introduce the XOR problem for neural networks. It is the simplest example of
a non linearly separable neural network. It can be solved with an additional layer of neurons, which is called a
hidden layer.

126
THE XOR PROBLEM FOR NEURAL NETWORKS
The XOR (exclusive or) function is defined by the following truth table:

Input1 Input2 XOR Output

0 0 0

0 1 1

1 0 1

1 1 0

This problem can't be solved with a simple neural network, as we can see in the following diagram:

No matter which straight line you choose, you will never succeed in having the blue points on one side and the
orange points on the other side. This is shown in the following figure. The orange points are on the orange
line. This means that this cannot be a dividing line. If we move this line parallel - no matter which direction,
there will be always two orange and one blue point on one side and only one blue point on the other side. If we
move the orange line in a non parallel way, there will be one blue and one orange point on either side, except
if the line goes through an orange point. So there is no way for a single straight line separating those points.

127
To solve this problem, we need to introduce a new type of neural networks, a network with so-called hidden
layers. A hidden layer allows the network to reorganize or rearrange the input data.

We will need only one hidden layer with two neurons. One works like an AND gate and the other one like an
OR gate. The output will "fire", when the OR gate fires and the AND gate doesn't.

As we had already mentioned, we cannot find a line which separates the orange points from the blue points.
But they can be separated by two lines, e.g. L1 and L2 in the following diagram:

128
To solve this problem, we need a network of the following kind, i.e with a hidden layer N1 and N2

The neuron N1 will determine one line, e.g. L1 and the neuron N2 will determine the other line L2. N3 will
finally solve our problem:

129
The implementation of this in Python has to wait until the next chapter of our tutorial on machine learning.

130
EXERCISES

EXERCISE 1
We could extend the logical AND to float values between 0 and 1 in the following way:

Input1 Input2 Output

x1 < 0.5 x2 < 0.5 0

x1 < 0.5 x2 >= 0.5 0

x1 >= 0.5 x2 < 0.5 0

x1 >= 0.5 x2 >= 0.5 1

Try to train a neural network with only one perceptron. Why doesn't it work?

EXERCISE 2
A point belongs to a class 0, if x 1 < 0.5 and belongs to class 1, if x 1 >= 0.5. Train a network with one
perceptron to classify arbitrary points. What can you say about the dicision boundary? What about the input
values x 2

SOLUTIONS TO THE EXERCISES

SOLUTION TO THE 1. EXERCISE

from perceptrons import Perceptron

p = Perceptron(weights=[0.3, 0.3, 0.3],

bias=1,
learning_rate=0.2)

def labelled_samples(n):
for _ in range(n):
s = [Link]((2,))
yield (s, 1) if s[0] >= 0.5 and s[1] >= 0.5 else (s, 0)

for in_data, label in labelled_samples(30):

[Link](label,

131
 in_data)

test_data, test_labels = list(zip(*labelled_samples(60)))

evaluation = [Link](test_data, test_labels)

print(evaluation)
Counter({'correct': 32, 'wrong': 28})

The easiest way to see, why it doesn't work, is to visualize the data.

import [Link] as plt

import numpy as np

ones = [test_data[i] for i in range(len(test_data)) if test_label

s[i] == 1]
zeroes = [test_data[i] for i in range(len(test_data)) if test_labe
ls[i] == 0]

fig, ax = [Link]()
xmin, xmax = -0.2, 1.2
X, Y = list(zip(*ones))
[Link](X, Y, color="g")
X, Y = list(zip(*zeroes))
[Link](X, Y, color="r")
ax.set_xlim([xmin, xmax])
ax.set_ylim([-0.1, 1.1])
c = -[Link][2] / [Link][1]
m = -[Link][0] / [Link][1]
X = [Link](xmin, xmax, 0.1)
[Link](X, m * X + c, label="decision boundary")

132
Output: [<[Link].Line2D at 0x7fabe8bfbf90>]

We can see that the green points and the red points are not separable by one straight line.

SOLUTION TO THE 2ND EXERCISE

from perceptrons import Perceptron

import numpy as np
from collections import Counter

def labelled_samples(n):
for _ in range(n):
s = [Link]((2,))
yield (s, 0) if s[0] < 0.5 else (s, 1)

p = Perceptron(weights=[0.3, 0.3, 0.3],

learning_rate=0.4)

for in_data, label in labelled_samples(300):

[Link](label,
in_data)

test_data, test_labels = list(zip(*labelled_samples(500)))

print([Link])
[Link](test_data, test_labels)

133
[ 2.03831116 -0.1785671 -0.9 ]
Output: Counter({'correct': 489, 'wrong': 11})

import [Link] as plt

import numpy as np

ones = [test_data[i] for i in range(len(test_data)) if test_label

s[i] == 1]
zeroes = [test_data[i] for i in range(len(test_data)) if test_labe
ls[i] == 0]

fig, ax = [Link]()
xmin, xmax = -0.2, 1.2
X, Y = list(zip(*ones))
[Link](X, Y, color="g")
X, Y = list(zip(*zeroes))
[Link](X, Y, color="r")
ax.set_xlim([xmin, xmax])
ax.set_ylim([-0.1, 1.1])
c = -[Link][2] / [Link][1]
m = -[Link][0] / [Link][1]
X = [Link](xmin, xmax, 0.1)
[Link](X, m * X + c, label="decision boundary")
Output: [<[Link].Line2D at 0x7fabe8bc89d0>]

[Link], m
Output: (array([ 2.03831116, -0.1785671 , -0.9 ]), 11.414819026
425487)

134
The slope m will have to get larger and larger in situations like this.

135
PERCEPTRON CLASS FROM SKLEARN

INTRODUCTION
In the previous chapter, we had
implemented a simple Perceptron class
using pure Python. The module
sklearn contains a Perceptron
class. We saw that a perceptron is an
algorithm to solve binary classifier
problems. This means that a Perceptron is
abinary classifier, which can decide
whether or not an input belongs to one or
the other class. E.g. "spam" or "ham". We
accomplished this by linearly combining
weights with the feature vector, i.e. the
input.

It is amazing that the perceptron algorithm was already invented in the year 1958 by Frank Rosenblatt. The
algorithm was implemented in custom-built hardware, called "Mark 1 perceptron". This hardware was
designed for image recognition.

The invention has been extremely overestimated: In 1958 the New York Times wrote after a press conference
with Rosenblatt: "New Navy Device Learns By Doing; Psychologist Shows Embryo of Computer Designed to
Read and Grow Wiser"

What initially seemed very promising was quickly proved incapable of keeping its promises. Thes perceptrons
could not be trained to recognise many classes of patterns.

EXAMPLE: PERCEPTRON CLASS

We will create with the help of make_blobs a binary testset:

import [Link] as plt

from [Link] import make_blobs

n_samples = 50
data, labels = make_blobs(n_samples=n_samples,
centers=([1.1, 3], [4.5, 6.9]),
random_state=0)

colours = ('green', 'orange')

fig, ax = [Link]()

136
 for n_class in range(2):
[Link](data[labels==n_class][:, 0],
data[labels==n_class][:, 1],
c=colours[n_class],
s=50,
label=str(n_class))

We will split our testset into a learnset and testset:

from sklearn.model_selection import train_test_split

datasets = train_test_split(data,
labels,
test_size=0.2)

train_data, test_data, train_labels, test_labels = datasets

We will use not the Perceptron class of sklearn.linear_model :

from sklearn.linear_model import Perceptron

p = Perceptron(random_state=42)
[Link](train_data, train_labels)
Output: Perceptron(random_state=42)

We can calculate predictions on the learnset and testset and can evaluate the score:

from [Link] import accuracy_score

predictions_train = [Link](train_data)

137
 predictions_test = [Link](test_data)
train_score = accuracy_score(predictions_train, train_labels)
print("score on train data: ", train_score)
test_score = accuracy_score(predictions_test, test_labels)
print("score on train data: ", test_score)
score on train data: 1.0
score on train data: 0.9

[Link](train_data, train_labels)
Output: 1.0

CLASSIFYING THE IRIS DATA WITH PERCEPTRON CLASSIFIER

We want to apply the Perceptron classifier on the iris dataset, which we had already used in our chapter
on k-nearest neighbor

Loading the iris data set:

import numpy as np
from [Link] import load_iris

iris = load_iris()

We have one problem: The Perceptron classifiert can only be used on binary classification problems, but
the Iris dataset consists fo three different classes, i.e. 'setosa', 'versicolor', 'virginica', corresponding to the
labels 0, 1, and 2:

iris.target_names
Output: array(['setosa', 'versicolor', 'virginica'], dtype='<U10')

We will merge the classes 'versicolor' and 'virginica' into one class. This means that only two classes are left.
So we can differentiate with the classifier between

• Iris setose
• not Iris setosa, or in other words either 'viriginica' od 'versicolor'

We accomplish this with the following command:

targets = ([Link]==0).astype(np.int8)
print(targets)

138
 [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0
0 0]

We split the data into a learn and a testset:

from sklearn.model_selection import train_test_split

datasets = train_test_split([Link],
targets,
test_size=0.2)

train_data, test_data, train_labels, test_labels = datasets

Now, we create a Perceptron instance and fit the training data:

from sklearn.linear_model import Perceptron

p = Perceptron(random_state=42,
max_iter=10,
tol=0.001)
[Link](train_data, train_labels)
Output: Perceptron(max_iter=10, random_state=42)

Now, we are ready for predictions and we will look at some randomly chosen random X values:

import random

sample = [Link](range(len(train_data)), 10)

for i in sample:
print(i, [Link]([train_data[i]]))

139
99 [1]
50 [0]
57 [0]
92 [0]
54 [0]
64 [0]
108 [0]
47 [0]
34 [0]
89 [0]

from [Link] import classification_report

print(classification_report([Link](train_data), train_labels))
precision recall f1-score support

0 1.00 1.00 1.00 76

1 1.00 1.00 1.00 44

accuracy 1.00 120

macro avg 1.00 1.00 1.00 120
weighted avg 1.00 1.00 1.00 120

from [Link] import classification_report

print(classification_report([Link](test_data), test_labels))
precision recall f1-score support

0 1.00 1.00 1.00 24

1 1.00 1.00 1.00 6

accuracy 1.00 30
macro avg 1.00 1.00 1.00 30
weighted avg 1.00 1.00 1.00 30

140
NEURAL NETWORKS, STRUCTURE,
WEIGHTS AND MATRICES

INTRODUCTION

We introduced the basic ideas about

neural networks in the previous chapter of
our machine learning tutorial.

We have pointed out the similarity

between neurons and neural networks in
biology. We also introduced very small
articial neural networks and introduced
decision boundaries and the XOR
problem.

In the simple examples we introduced so

far, we saw that the weights are the
essential parts of a neural network. Before
we start to write a neural network with multiple layers, we need to have a closer look at the weights.

We have to see how to initialize the weights and how to efficiently multiply the weights with the input values.

In the following chapters we will design a neural network in Python, which consists of three layers, i.e. the
input layer, a hidden layer and an output layer. You can see this neural network structure in the following
diagram. We have an input layer with three nodes i 1, i 2, i 3 These nodes get the corresponding input values
x 1, x 2, x 3. The middle or hidden layer has four nodes h 1, h 2, h 3, h 4. The input of this layer stems from the
input layer. We will discuss the mechanism soon. Finally, our output layer consists of the two nodes o 1, o 2

The input layer is different from the other layers. The nodes of the input layer are passive. This means that the
input neurons do not change the data, i.e. there are no weights used in this case. They receive a single value
and duplicate this value to their many outputs.

141
The input layer consists of the nodes i 1, i 2 and i 3. In principle the input is a one-dimensional vector, like (2, 4,
11). A one-dimensional vector is represented in numpy like this:

import numpy as np

input_vector = [Link]([2, 4, 11])

print(input_vector)
[ 2 4 11]

In the algorithm, which we will write later, we will have to transpose it into a column vector, i.e. a two-
dimensional array with just one column:

import numpy as np

input_vector = [Link]([2, 4, 11])

input_vector = [Link](input_vector, ndmin=2).T
print("The input vector:\n", input_vector)

print("The shape of this vector: ", input_vector.shape)

The input vector:
[[ 2]
[ 4]
[11]]
The shape of this vector: (3, 1)

142
WEIGHTS AND MATRICES
Each of the arrows in our network diagram has an associated weight value. We will only look at the arrows
between the input and the output layer now.

The value x 1 going into the node i 1 will be distributed according to the values of the weights. In the following
diagram we have added some example values. Using these values, the input values (Ih 1, Ih 2, Ih 3, Ih 4 into the
nodes (h 1, h 2, h 3, h 4) of the hidden layer can be calculated like this:

Ih 1 = 0.81 ∗ 0.5 + 0.12 ∗ 1 + 0.92 ∗ 0.8

Ih 2 = 0.33 ∗ 0.5 + 0.44 ∗ 1 + 0.72 ∗ 0.8

Ih 3 = 0.29 ∗ 0.5 + 0.22 ∗ 1 + 0.53 ∗ 0.8

Ih 4 = 0.37 ∗ 0.5 + 0.12 ∗ 1 + 0.27 ∗ 0.8

Those familiar with matrices and matrix multiplication will see where it is boiling down to. We will redraw
our network and denote the weights with w ij:

143
In order to efficiently execute all the necessary calaculations, we will arrange the weights into a weight matrix.

144
The weights in our diagram above build an array, which we will call 'weights_in_hidden' in our Neural
Network class. The name should indicate that the weights are connecting the input and the hidden nodes, i.e.
they are between the input and the hidden layer. We will also abbreviate the name as 'wih'. The weight matrix
between the hidden and the output layer will be denoted as "who".:

Now that we have defined our weight matrices, we have to take the next step. We have to multiply the matrix
wih the input vector. Btw. this is exactly what we have manually done in our previous example.

()( )( ) ( )
y1 w 11 w 12 w 13 w 11 ⋅ x 1 + w 12 ⋅ x 2 + w 13 ⋅ x 3
x1
y2 w 21 w 22 w 23 w 21 ⋅ x 1 + w 22 ⋅ x 2 + w 23 ⋅ x 3
= x2 =
y3 w 31 w 32 w 33 w 31 ⋅ x 1 + w 32 ⋅ x 2 + w 33 ⋅ x 3
x3
y4 w 41 w 42 w 43 w 41 ⋅ x 1 + w 42 ⋅ x 2 + w 43 ⋅ x 3

We have a similar situation for the 'who' matrix between hidden and output layer. So the output z 1 and z 2 from
the nodes o 1 and o 2 can also be calculated with matrix multiplications:

()
y1

()(
z1
z2
=
wh 11
wh 21
wh 12
wh 22
wh 13 wh 14
wh 23 wh 24 ) ( y2
y3
y4
=
wh 11 ⋅ y 1 + wh 12 ⋅ y 2 + wh 13 ⋅ y 3 + wh 14 ⋅ y 4
wh 21 ⋅ y 1 + wh 22 ⋅ y 2 + wh 23 ⋅ y 3 + wh 24 ⋅ y 4 )
You might have noticed that something is missing in our previous calculations. We showed in our introductory

145
chapter Neural Networks from Scratch in Python that we have to apply an activation or step function Φ on
each of these sums.

The following picture depicts the whole flow of calculation, i.e. the matrix multiplication and the succeeding
application of the activation function.
The matrix multiplication between the matrix wih and the matrix of the values of the input nodes x 1, x 2, x 3
calculates the output which will be passed to the activation function.

The final output y 1, y 2, y 3, y 4 is the input of the weight matrix who:

Even though treatment is completely analogue, we will also have a detailled look at what is going on between
our hidden layer and the output layer:

146
INITIALIZING THE WEIGHT MATRICES
One of the important choices which have to be made before training a neural network consists in initializing
the weight matrices. We don't know anything about the possible weights, when we start. So, we could start
with arbitrary values?

As we have seen the input to all the nodes except the input nodes is calculated by applying the activation
function to the following sum:
n

yj = ∑ w ji ⋅ x i
i=1

(with n being the number of nodes in the previous layer and y j is the input to a node of the next layer)

We can easily see that it would not be a good idea to set all the weight values to 0, because in this case the
result of this summation will always be zero. This means that our network will be incapable of learning. This
is the worst choice, but initializing a weight matrix to ones is also a bad choice.

The values for the weight matrices should be chosen randomly and not arbitrarily. By choosing a random
normal distribution we have broken possible symmetric situations, which can and often are bad for the
learning process.

There are various ways to initialize the weight matrices randomly. The first one we will introduce is the unity
function from [Link]. It creates samples which are uniformly distributed over the half-open interval
[low, high), which means that low is included and high is excluded. Each value within the given interval is
equally likely to be drawn by 'uniform'.

import numpy as np

number_of_samples = 1200
low = -1
high = 0
s = [Link](low, high, number_of_samples)

147
# all values of s are within the half open interval [-1, 0) :
print([Link](s >= -1) and [Link](s < 0))
True

The histogram of the samples, created with the uniform function in our previous example, looks like this:

import [Link] as plt

[Link](s)
[Link]()

The next function we will look at is 'binomial' from [Link]:

binomial(n, p, size=None)

It draws samples from a binomial distribution with specified parameters, n trials and probability p of
success where n is an integer >= 0 and p is a float in the interval [0,1]. ( n may be input as a float, but
it is truncated to an integer in use)

s = [Link](100, 0.5, 1200)

[Link](s)
[Link]()

148
We like to create random numbers with a normal distribution, but the numbers have to be bounded. This is not
the case with [Link](), because it doesn't offer any bound parameter.

We can use truncnorm from [Link] for this purpose.

The standard form of this distribution is a standard normal truncated to the range [a, b] — notice that a and b
are defined over the domain of the standard normal. To convert clip values for a specific mean and standard
deviation, use:

a, b = (myclip_a - my_mean) / my_std, (myclip_b - my_mean) / my_std

from [Link] import truncnorm

s = truncnorm(a=-2/3., b=2/3., scale=1, loc=0).rvs(size=1000)

[Link](s)
[Link]()

149
The function 'truncnorm' is difficult to use. To make life easier, we define a function truncated_normal
in the following to fascilitate this task:

def truncated_normal(mean=0, sd=1, low=0, upp=10):

return truncnorm(
(low - mean) / sd, (upp - mean) / sd, loc=mean, scale=sd)

X = truncated_normal(mean=0, sd=0.4, low=-0.5, upp=0.5)

s = [Link](10000)

[Link](s)
[Link]()

Further examples:

150
X1 = truncated_normal(mean=2, sd=1, low=1, upp=10)
X2 = truncated_normal(mean=5.5, sd=1, low=1, upp=10)
X3 = truncated_normal(mean=8, sd=1, low=1, upp=10)

import [Link] as plt

fig, ax = [Link](3, sharex=True)
ax[0].hist([Link](10000), density=True)
ax[1].hist([Link](10000), density=True)
ax[2].hist([Link](10000), density=True)
[Link]()

We will create the link weights matrix now. truncated_normal is ideal for this purpose. It is a good
idea to choose random values from within the interval

1 1
(− , )
√n √n
where n denotes the number of input nodes.

So we can create our "wih" matrix with:

no_of_input_nodes = 3
no_of_hidden_nodes = 4
rad = 1 / [Link](no_of_input_nodes)

X = truncated_normal(mean=2, sd=1, low=-rad, upp=rad)

wih = [Link]((no_of_hidden_nodes, no_of_input_nodes))
wih

151
Output: array([[-0.41379992, -0.24122842, -0.0303682 ],
[ 0.07304837, -0.00160437, 0.0911987 ],
[ 0.32405689, 0.5103896 , 0.23972997],
[ 0.097932 , -0.06646741, 0.01359876]])

Similarly, we can now define the "who" weight matrix:

no_of_hidden_nodes = 4
no_of_output_nodes = 2
rad = 1 / [Link](no_of_hidden_nodes) # this is the input in thi
s layer!

X = truncated_normal(mean=2, sd=1, low=-rad, upp=rad)

who = [Link]((no_of_output_nodes, no_of_hidden_nodes))
who
Output: array([[ 0.15892038, 0.06060043, 0.35900184, 0.14202827],
[-0.4758216 , 0.29563269, 0.46035026, -0.29673539]])

152
RUNNING A NEURAL NETWORK WITH
PYTHON

A NEURAL NETWORK CLASS

We learned in the previous chapter of our tutorial on neural

networks the most important facts about weights. We saw how
they are used and how we can implement them in Python. We
saw that the multiplication of the weights with the input values
can be accomplished with arrays from Numpy by applying
matrix multiplication.

However, what we hadn't done was to test them in a real neural

network environment. We have to create this environment first.
We will now create a class in Python, implementing a neural
network. We will proceed in small steps so that everything is
easy to understand.

The most essential methods our class needs are:

• __init__ to initialize a class, i.e. we will set

the number of neurons for every layer and
initialize the weight matrices.
• run : A method which is applied to a sample,
which which we want to classify. It applies this
sample to the neural network. We could say, we
'run' the network to 'predict' the result. This
method is in other implementations often known
as predict .
• train : This method gets a sample and the corresponding target value as an input. With this
input it can adjust the weight values if necessary. This means the network learns from an input.
Seen from the user point of view, we 'train' the network. In sklearn for example, this method
is called fit

We will postpone the definition of the train and run method until later. The weight matrices should be
initialized inside of the __init__ method. We do this indirectly. We define a method
create_weight_matrices and call it in __init__ . In this way, the init method remains clear.

We will also postpone adding bias nodes to the layers.

153
The following Python code contains an implementation of a neural network class applying the knowledge we
worked out in the previous chapter:

import numpy as np
from [Link] import truncnorm

def truncated_normal(mean=0, sd=1, low=0, upp=10):

return truncnorm(
(low - mean) / sd, (upp - mean) / sd, loc=mean, scale=sd)

class NeuralNetwork:

def __init__(self,
no_of_in_nodes,
no_of_out_nodes,
no_of_hidden_nodes,
learning_rate):
self.no_of_in_nodes = no_of_in_nodes
self.no_of_out_nodes = no_of_out_nodes
self.no_of_hidden_nodes = no_of_hidden_nodes
self.learning_rate = learning_rate
self.create_weight_matrices()

def create_weight_matrices(self):
rad = 1 / [Link](self.no_of_in_nodes)
X = truncated_normal(mean=0, sd=1, low=-rad, upp=rad)
self.weights_in_hidden = [Link]((self.no_of_hidden_nodes,
self.no_of_in_nodes))
rad = 1 / [Link](self.no_of_hidden_nodes)
X = truncated_normal(mean=0, sd=1, low=-rad, upp=rad)
self.weights_hidden_out = [Link]((self.no_of_out_nodes,
self.no_of_hidden_nodes))

def train(self):
pass

def run(self):
pass

We cannot do a lot with this code, but we can at least initialize it. We can also have a look at the weight
matrices:

simple_network = NeuralNetwork(no_of_in_nodes = 3,

154
 no_of_out_nodes = 2,
no_of_hidden_nodes = 4,
learning_rate = 0.1)
print(simple_network.weights_in_hidden)
print(simple_network.weights_hidden_out)
[[-0.3460287 -0.19427278 -0.19102916]
[ 0.56743476 -0.47164202 -0.06910573]
[ 0.53013469 -0.05117752 -0.430623 ]
[ 0.48414483 0.31263278 -0.08123676]]
[[-0.12645547 0.05260599 -0.36278102 -0.32649173]
[-0.20841352 -0.01456191 -0.13778649 -0.08920465]]

ACTIVATION FUNCTIONS, SIGMOID AND RELU

Before we can program the run method, we have to deal with the activation function. We had the following
diagram in the introductory chapter on neural networks:

The input values of a perceptron are processed by the summation function and followed by an activation
function, transforming the output of the summation function into a desired and more suitable output. The
summation function means that we will have a matrix multiplication of the weight vectors and the input
values.

There are lots of different activation functions used in neural networks. One of the most comprehensive
overviews of possible activation functions can be found at Wikipedia.

The sigmoid function is one of the often used activation functions. The sigmoid function, which we are using,
is also known as the Logistic function.

It is defined as

1
σ(x) =
1 + e −x

Let us have a look at the graph of the sigmoid function. We use matplotlib to plot the sigmoid function:

import numpy as np

155
 import [Link] as plt
def sigma(x):
return 1 / (1 + [Link](-x))

X = [Link](-5, 5, 100)

[Link](X, sigma(X),'b')
[Link]('X Axis')
[Link]('Y Axis')
[Link]('Sigmoid Function')

[Link]()

[Link](2.3, 0.84, r'$\sigma(x)=\frac{1}{1+e^{-x}}$', fontsize=1

6)

[Link]()

Looking at the graph, we can see that the sigmoid function maps a given number x into the range of numbers
between 0 and 1. 0 and 1 not included! As the value of x gets larger, the value of the sigmoid function gets
closer and closer to 1 and as x gets smaller, the value of the sigmoid function is approaching 0.

Instead of defining the sigmoid function ourselves, we can also use the expit function from
[Link] , which is an implementation of the sigmoid function. It can be applied on various data
classes like int, float, list, numpy,ndarray and so on. The result is an ndarray of the same shape as the input
data x.

156
 from [Link] import expit
print(expit(3.4))
print(expit([3, 4, 1]))
print(expit([Link]([0.8, 2.3, 8])))
0.9677045353015494
[0.95257413 0.98201379 0.73105858]
[0.68997448 0.90887704 0.99966465]

The logistic function is often often used in neural networks to introduce nonlinearity in the model and to map
signals into a specified range, i.e. 0 and 1. It is also well liked because the derivative - needed in
backpropagation - is simple.

1
σ(x) =
1 + e −x

and its derivative:

σ ′ (x) = σ(x)(1 − σ(x))

import numpy as np
import [Link] as plt
def sigma(x):
return 1 / (1 + [Link](-x))

X = [Link](-5, 5, 100)

[Link](X, sigma(X))
[Link](X, sigma(X) * (1 - sigma(X)))

[Link]('X Axis')
[Link]('Y Axis')
[Link]('Sigmoid Function')

[Link]()

[Link](2.3, 0.84, r'$\sigma(x)=\frac{1}{1+e^{-x}}$', fontsize=1

6)
[Link](0.3, 0.1, r'$\sigma\'(x) = \sigma(x)(1 - \sigma(x))$', fo
ntsize=16)

[Link]()

157
We can also define our own sigmoid function with the decorator vectorize from numpy:

@[Link]
def sigmoid(x):
return 1 / (1 + np.e ** -x)

#sigmoid = [Link](sigmoid)
sigmoid([3, 4, 5])
Output: array([0.95257413, 0.98201379, 0.99330715])

Another easy to use activation function is the ReLU function. ReLU stands for rectified linear unit. It is also
known as the ramp function. It is defined as the positve part of its argument, i.e. y = max (0, x). This is
"currently, the most successful and widely-used activation function is the Rectified Linear Unit (ReLU)"1 The
ReLu function is computationally more efficient than Sigmoid like functions, because Relu means only
choosing the maximum between 0 and the argument x . Whereas Sigmoids need to perform expensive
exponential operations.

# alternative activation function

def ReLU(x):
return [Link](0.0, x)

# derivation of relu
def ReLU_derivation(x):
if x <= 0:
return 0
else:
return 1

158
import numpy as np
import [Link] as plt

X = [Link](-5, 6, 100)
[Link](X, ReLU(X),'b')
[Link]('X Axis')
[Link]('Y Axis')
[Link]('ReLU Function')
[Link]()
[Link](0.8, 0.4, r'$ReLU(x)=max(0, x)$', fontsize=14)
[Link]()

ADDING A RUN METHOD

We have everything together now to implement the run (or predict ) method of our neural network
class. We will use [Link] as the activation function and rename it to
activation_function :

from [Link] import expit as activation_function

All we have to do in the run method consists of the following.

1. Matrix multiplication of the input vector and the weights_in_hidden matrix.

2. Applying the activation function to the result of step 1
3. Matrix multiplication of the result vector of step 2 and the weights_in_hidden matrix.
4. To get the final result: Applying the activation function to the result of 3

import numpy as np
from [Link] import expit as activation_function

159
from [Link] import truncnorm

def truncated_normal(mean=0, sd=1, low=0, upp=10):

return truncnorm(
(low - mean) / sd, (upp - mean) / sd, loc=mean, scale=sd)

class NeuralNetwork:

def __init__(self,
no_of_in_nodes,
no_of_out_nodes,
no_of_hidden_nodes,
learning_rate):
self.no_of_in_nodes = no_of_in_nodes
self.no_of_out_nodes = no_of_out_nodes
self.no_of_hidden_nodes = no_of_hidden_nodes
self.learning_rate = learning_rate
self.create_weight_matrices()

def create_weight_matrices(self):
""" A method to initialize the weight matrices of the neur
al network"""
rad = 1 / [Link](self.no_of_in_nodes)
X = truncated_normal(mean=0, sd=1, low=-rad, upp=rad)
self.weights_in_hidden = [Link]((self.no_of_hidden_nodes,
self.no_of_in_nodes))
rad = 1 / [Link](self.no_of_hidden_nodes)
X = truncated_normal(mean=0, sd=1, low=-rad, upp=rad)
self.weights_hidden_out = [Link]((self.no_of_out_nodes,
self.no_of_hidden_nodes))

def train(self, input_vector, target_vector):

pass

def run(self, input_vector):

"""
running the network with an input vector 'input_vector'.
'input_vector' can be tuple, list or ndarray
"""
# turning the input vector into a column vector
input_vector = [Link](input_vector, ndmin=2).T
input_hidden = activation_function(self.weights_in_hidden

160
 @ input_vector)
output_vector = activation_function(self.weights_hidden_ou
t @ input_hidden)
return output_vector

We can instantiate an instance of this class, which will be a neural network. In the following example we
create a network with two input nodes, four hidden nodes, and two output nodes.

simple_network = NeuralNetwork(no_of_in_nodes=2,
no_of_out_nodes=2,
no_of_hidden_nodes=4,
learning_rate=0.6)

We can apply the run method to all arrays with a shape of (2,), also lists and tuples with two numerical
elements. The result of the call is defined by the random values of the weights:

simple_network.run([(3, 4)])
Output: array([[0.54558831],
[0.6834667 ]])

FOOTNOTES
1
Ramachandran, Prajit; Barret, Zoph; Quoc, V. Le (October 16, 2017). "Searching for Activation Functions".

161
BACKPROPAGATION IN NEURAL NETWORKS

INTRODUCTION
We already wrote in the previous chapters of our
tutorial on Neural Networks in Python. The networks
from our chapter Running Neural Networks lack the
capabilty of learning. They can only be run with
randomly set weight values. So we cannot solve any
classification problems with them. However, the
networks in Chapter Simple Neural Networks were
capable of learning, but we only used linear networks
for linearly separable classes.

Of course, we want to write general ANNs, which are

capable of learning. To do so, we will have to
understand backpropagation. Backpropagation is a
commonly used method for training artificial neural
networks, especially deep neural networks.
Backpropagation is needed to calculate the gradient,
which we need to adapt the weights of the weight matrices. The weight of the neuron (nodes) of our network
are adjusted by calculating the gradient of the loss function. For this purpose a gradient descent optimization
algorithm is used. It is also called backward propagation of errors.

Quite often people are frightened away by the mathematics used in it. We try to explain it in simple terms.

Explaining gradient descent starts in many articles or tutorials with mountains. Imagine you are put on a
mountain, not necessarily the top, by a helicopter at night or heavy fog. Let's further imagine that this
mountain is on an island and you want to reach sea level. You have to go down, but you hardly see anything,
maybe just a few metres. Your task is to find your way down, but you cannot see the path. You can use the
method of gradient descent. This means that you are examining the steepness at your current position. You
will proceed in the direction with the steepest descent. You take only a few steps and then you stop again to
reorientate yourself. This means you are applying again the previously described procedure, i.e. you are
looking for the steepest descend.

This procedure is depicted in the following diagram in a two-dimensional space.

162
Going on like this you will arrive at a position, where there is no further descend.

Each direction goes upwards. You may have reached the deepest level - the global minimum -, but you might
as well be stuck in a basin. If you start at the position on the right side of our image, everything works out fine,
but from the leftside, you will be stuck in a local minimum.

BACKPROPAGATION IN DETAIL
Now, we have to go into the details, i.e. the mathematics.

We will start with the simpler case. We look at a linear network. Linear neural networks are networks where
the output signal is created by summing up all the weighted input signals. No activation function will be
applied to this sum, which is the reason for the linearity.

The will use the following simple network.

When we are training the network we have samples and corresponding labels. For each output value o i we
have a label t i, which is the target or the desired value. If the label is equal to the output, the result is correct

163
and the neural network has not made an error. Principially, the error is the difference between the target and
the actual output:

ei = ti − oi

We will later use a squared error function, because it has better characteristics for the algorithm:

1
ei = (t − o i) 2
2 i

We want to clarify how the error backpropagates with the following example with values:

We will have a look at the output value o 1, which is depending on the values w 11, w 12, w 13 and w 14. Let's
assume the calculated value (o 1) is 0.92 and the desired value (t 1) is 1. In this case the error is

e 1 = t 1 − o 1 = 1 − 0.92 = 0.08

The eror e 2 can be calculated like this:

e 2 = t 2 − o 2 = 1 − 0.18 = 0.82

164
Depending on this error, we have to change the weights from the incoming values accordingly. We have four
weights, so we could spread the error evenly. Yet, it makes more sense to to do it proportionally, according to
the weight values. The larger a weight is in relation to the other weights, the more it is responsible for the
error. This means that we can calculate the fraction of the error e 1 in w 11 as:

w 11
e1 ⋅
∑ 4 w 1i
i=1

This means in our example:

0.6
0.08 ⋅ = 0.0343
0.6 + 0.1 + 0.15 + 0.25

The total error in our weight matrix between the hidden and the output layer - we called it in our previous
chapter 'who' - looks like this

165
 [ ]
w 11 w 21 w 31
∑ 4 w 1i ∑ 4 w 2i ∑ 4 w 3i
i=1 i=1 i=1

[]
w 12 w 22 w 32
e1
∑ 4 w 1i ∑ 4 w 2i ∑ 4 w 3i
i=1 i=1 i=1
e who = ⋅ e2
w 13 w 23 w 33
∑ 4 w 1i ∑ 4 w 2i ∑ 4 w 3i
e3
i=1 i=1 i=1

w 14 w 24 w 34
∑ 4 w 1i ∑ 4 w 2i ∑ 4 w 3i
i=1 i=1 i=1

You can see that the denominator in the left matrix is always the same. It functions like a scaling factor. We
can drop it so that the calculation gets a lot simpler:

[ ][ ]
w 11 w 21 w 31
e1
w 12 w 22 w 32
e who = ⋅ e2
w 13 w 23 w 33
e3
w 14 w 24 w 34

If you compare the matrix on the right side with the 'who' matrix of our chapter Neuronal Network Using
Python and Numpy, you will notice that it is the transpose of 'who'.

e who = who. T ⋅ e

So, this has been the easy part for linear neural networks. We haven't taken into account the activation function
until now.

We want to calculate the error in a network with an activation function, i.e. a non-linear network. The
derivation of the error function describes the slope. As we mentioned in the beginning of the this chapter, we
want to descend. The derivation describes how the error E changes as the weight w kj changes:

166
 ∂E
∂w kj

The error function E over all the output nodes o i (i = 1, . . . n) where n is the total number of output nodes:

n
1
E= ∑ 2 (t i − o i) 2
i=1

Now, we can insert this in our derivation:

n
∂E ∂ 1
∂w kj
= ∑ (t − o i) 2
∂w kj 2 i = 1 i

If you have a look at our example network, you will see that an output node o k only depends on the input
signals created with the weights w ki with i = 1, …m and m the number of hidden nodes.

The following diagram further illuminates this:

This means that we can calculate the error for every output node independently of each other. This means that
we can remove all expressions t i − o i with i ≠ k from our summation. So the calculation of the error for a node
k looks a lot simpler now:

∂E ∂ 1
= (t − o k) 2
∂w kj ∂w kj 2 k

The target value t k is a constant, because it is not depending on any input signals or weights. We can apply the
chain rule for the differentiation of the previous term to simplify things:

167
 ∂E ∂E ∂o k
= ⋅
∂w kj ∂o k ∂w kj

In the previous chapter of our tutorial, we used the sigmoid function as the activation function:

1
σ(x) =
1 + e −x

The output node o k is calculated by applying the sigmoid function to the sum of the weighted input signals.
This means that we can further transform our derivative term by replacing o k by this function:

m
∂E ∂
= (t k − o k) ⋅ σ( ∑ w h )
∂w kj ∂w kj i = 1 ki i

where m is the number of hidden nodes.

The sigmoid function is easy to differentiate:

∂σ(x)
= σ(x) ⋅ (1 − σ(x))
∂x

The complete differentiation looks like this now:

m m m
∂E ∂
= (t k − o k) ⋅ σ( ∑ w kih i) ⋅ (1 − σ( ∑ w kih i)) ∑ w kih i
∂w kj i=1 i=1
∂w kj i = 1

The last part has to be differentiated with respect to w kj. This means that the derivation of all the products will
be 0 except the the term w kjh j) which has the derivative h j with respect to w kj:

m m
∂E
= (t k − o k) ⋅ σ( ∑ w kih i) ⋅ (1 − σ( ∑ w kih i)) ⋅ h j
∂w kj i=1 i=1

This is what we need to implement the method 'train' of our NeuralNetwork class in the following chapter.

In [ ]:

168
TRAINING A NEURAL NETWORK WITH
PYTHON

INTRODUCTION
In the chapter "Running Neural
Networks", we programmed a class in
Python code called 'NeuralNetwork'. The
instances of this class are networks with
three layers. When we instantiate an ANN
of this class, the weight matrices between
the layers are automatically and randomly
chosen. It is even possible to run such a
ANN on some input, but naturally it
doesn't make a lot of sense exept for
testing purposes. Such an ANN cannot
provide correct classification results. In
fact, the classification results are in no
way adapted to the expected results. The
values of the weight matrices have to be
set according the the classification task.
We need to improve the weight values,
which means that we have to train our network. To train it we have to implement backpropagation in the
train method. If you don't understand backpropagation and want to understand it, we recommend to go
back to the chapter Backpropagation in Neural Networks.

After knowing und hopefully understanding backpropagation, you are ready to fully understand the train
method.

The train method is called with an input vector and a target vector. The shape of the vectors can be one-
dimensional, but they will be automatically turned into the correct two-dimensional shape, i.e.
reshape(input_vector.size, 1) and reshape(target_vector.size, 1) . After this
we call the run method to get the result of the network output_vector_network =
[Link](input_vector) . This output may differ from the target_vector . We calculate the
output_error by subtracting the output of the network output_vector_network from the
target_vector .

import numpy as np
from [Link] import expit as activation_function

169
from [Link] import truncnorm

def truncated_normal(mean=0, sd=1, low=0, upp=10):

return truncnorm(
(low - mean) / sd, (upp - mean) / sd, loc=mean, scale=sd)

class NeuralNetwork:

def __init__(self,
no_of_in_nodes,
no_of_out_nodes,
no_of_hidden_nodes,
learning_rate):
self.no_of_in_nodes = no_of_in_nodes
self.no_of_out_nodes = no_of_out_nodes
self.no_of_hidden_nodes = no_of_hidden_nodes
self.learning_rate = learning_rate
self.create_weight_matrices()

def create_weight_matrices(self):
""" A method to initialize the weight matrices of the neur
al network"""
rad = 1 / [Link](self.no_of_in_nodes)
X = truncated_normal(mean=0, sd=1, low=-rad, upp=rad)
self.weights_in_hidden = [Link]((self.no_of_hidden_nodes,
self.no_of_in_nodes))
rad = 1 / [Link](self.no_of_hidden_nodes)
X = truncated_normal(mean=0, sd=1, low=-rad, upp=rad)
self.weights_hidden_out = [Link]((self.no_of_out_nodes,
self.no_of_hidden_nodes))

def train(self, input_vector, target_vector):

"""
input_vector and target_vector can be tuples, lists or nda
rrays
"""
# make sure that the vectors have the right shape
input_vector = [Link](input_vector)
input_vector = input_vector.reshape(input_vector.size, 1)
target_vector = [Link](target_vector).reshape(target_vec
[Link], 1)

output_vector_hidden = activation_function(self.weights_i
n_hidden @ input_vector)

170
 output_vector_network = activation_function(self.weights_h
idden_out @ output_vector_hidden)

output_error = target_vector - output_vector_network

tmp = output_error * output_vector_network * (1.0 - outpu
t_vector_network)
self.weights_hidden_out += self.learning_rate * (tmp @ ou
tput_vector_hidden.T)

# calculate hidden errors:

hidden_errors = self.weights_hidden_out.T @ output_error
# update the weights:
tmp = hidden_errors * output_vector_hidden * (1.0 - outpu
t_vector_hidden)
self.weights_in_hidden += self.learning_rate * (tmp @ inpu
t_vector.T)

def run(self, input_vector):

"""
running the network with an input vector 'input_vector'.
'input_vector' can be tuple, list or ndarray
"""
# make sure that input_vector is a column vector:
input_vector = [Link](input_vector)
input_vector = input_vector.reshape(input_vector.size, 1)
input4hidden = activation_function(self.weights_in_hidden
@ input_vector)
output_vector_network = activation_function(self.weights_h
idden_out @ input4hidden)
return output_vector_network

def evaluate(self, data, labels):

"""
Counts how often the actual result corresponds to the
target result.
A result is considered to be correct, if the index of
the maximal value corresponds to the index with the "1"
in the one-hot representation,
e.g.
res = [0.1, 0.132, 0.875]
labels[i] = [0, 0, 1]
"""
corrects, wrongs = 0, 0
for i in range(len(data)):
res = [Link](data[i])

171
 res_max = [Link]()
if res_max == labels[i].argmax():
corrects += 1
else:
wrongs += 1
return corrects, wrongs

We assume that you save the previous code in a file called neural_networks1.py . We will use it under
this name in the coming examples.

To test this neural network class we need train and test data. We create the data with make_blobs from
[Link] .

from [Link] import make_blobs

n_samples = 500
blob_centers = ([2, 6], [6, 2], [7, 7])
n_classes = len(blob_centers)
data, labels = make_blobs(n_samples=n_samples,
centers=blob_centers,
random_state=7)

Let us visualize the previously created data:

import [Link] as plt

colours = ('green', 'red', "yellow")

fig, ax = [Link]()

for n_class in range(n_classes):

[Link](data[labels==n_class][:, 0],
data[labels==n_class][:, 1],
c=colours[n_class],
s=40,
label=str(n_class))

172
The labels are wrongly represented. They are in a one-dimensional vector:

labels[:7]
Output: array([2, 2, 1, 0, 2, 0, 1])

We need a one-hot representation for each label. So the labels are represented as

Label One-Hot Representation

0 (1, 0, 0)

1 (0, 1, 0)

2 (0, 0, 1)

We can easily change the labels with the following commands:

import numpy as np

labels = [Link](n_classes) == [Link]([Link], 1)

labels = [Link]([Link])
labels[:7]

173
Output: array([[0., 0., 1.],
[0., 0., 1.],
[0., 1., 0.],
[1., 0., 0.],
[0., 0., 1.],
[1., 0., 0.],
[0., 1., 0.]])

We are ready now to create a train and a test data set:

from sklearn.model_selection import train_test_split

res = train_test_split(data, labels,

train_size=0.8,
test_size=0.2,
random_state=42)
train_data, test_data, train_labels, test_labels = res
train_labels[:10]
Output: array([[0., 0., 1.],
[0., 1., 0.],
[1., 0., 0.],
[0., 0., 1.],
[0., 0., 1.],
[1., 0., 0.],
[0., 1., 0.],
[1., 0., 0.],
[1., 0., 0.],
[0., 0., 1.]])

We create a neural network with two input nodes, and three output nodes. One output node for each class:

from neural_networks1 import NeuralNetwork

simple_network = NeuralNetwork(no_of_in_nodes=2,
no_of_out_nodes=3,
no_of_hidden_nodes=5,
learning_rate=0.3)

The next step consists in training our network with the data and labels from our training samples:

for i in range(len(train_data)):
simple_network.train(train_data[i], train_labels[i])

174
We now have to check how well our network has learned. For this purpose, we will use the evaluate function:

simple_network.evaluate(train_data, train_labels)
Output: (390, 10)

NEURAL NETWORK WITH BIAS NODES

We already introduced the basic idea and necessity of bias nodes in the chapter "Simple Neural Network", in
which we focussed on very simple linearly separable data sets. We learned that a bias node is a node that is
always returning the same output. In other words: It is a node which is not depending on some input and it
does not have any input. The value of a bias node is often set to one, but it can be set to other values as well.
Except for zero, which makes no sense for obvious reasons. If a neural network does not have a bias node in a
given layer, it will not be able to produce output in the next layer that differs from 0 when the feature values
are 0. Generally speaking, we can say that bias nodes are used to increase the flexibility of the network to fit
the data. Usually, there will be not more than one bias node per layer. The only exception is the output layer,
because it makes no sense to add a bias node to this layer.

The following diagram shows the first two layers of our previously used three-layered neural network:

We can see from this diagram that our weight matrix needs one additional column and the bias value has to be
added to the input vector:

175
Again, the situation for the weight matrix between the hidden and the output layer is similar:

The same is true for the corresponding matrix:

The following is a complete Python class implementing our network with bias nodes:

import numpy as np
from [Link] import truncnorm
from [Link] import expit as activation_function

def truncated_normal(mean=0, sd=1, low=0, upp=10):

return truncnorm(
(low - mean) / sd, (upp - mean) / sd, loc=mean, scale=sd)

176
class NeuralNetwork:

def __init__(self,
no_of_in_nodes,
no_of_out_nodes,
no_of_hidden_nodes,
learning_rate,
bias=None):
self.no_of_in_nodes = no_of_in_nodes
self.no_of_hidden_nodes = no_of_hidden_nodes
self.no_of_out_nodes = no_of_out_nodes
self.learning_rate = learning_rate
[Link] = bias
self.create_weight_matrices()

def create_weight_matrices(self):
""" A method to initialize the weight matrices of the neur
al
network with optional bias nodes"""
bias_node = 1 if [Link] else 0
rad = 1 / [Link](self.no_of_in_nodes + bias_node)
X = truncated_normal(mean=0, sd=1, low=-rad, upp=rad)
self.weights_in_hidden = [Link]((self.no_of_hidden_nodes,
self.no_of_in_nodes + bia
s_node))
rad = 1 / [Link](self.no_of_hidden_nodes + bias_node)
X = truncated_normal(mean=0, sd=1, low=-rad, upp=rad)
self.weights_hidden_out = [Link]((self.no_of_out_nodes,
self.no_of_hidden_nodes
+ bias_node))

def train(self, input_vector, target_vector):

""" input_vector and target_vector can be tuple, list or n
darray """

# make sure that the vectors have the right shap

input_vector = [Link](input_vector)
input_vector = input_vector.reshape(input_vector.size,
1)
if [Link]:
# adding bias node to the end of the input_vector
input_vector = [Link]( (input_vector, [[self.b

177
ias]]) )
target_vector = [Link](target_vector).reshape(target_vec
[Link], 1)

output_vector_hidden = activation_function(self.weights_i
n_hidden @ input_vector)
if [Link]:
output_vector_hidden = [Link]( (output_vecto
r_hidden, [[[Link]]]) )
output_vector_network = activation_function(self.weights_h
idden_out @ output_vector_hidden)

output_error = target_vector - output_vector_network

# update the weights:
tmp = output_error * output_vector_network * (1.0 - outpu
t_vector_network)
self.weights_hidden_out += self.learning_rate * (tmp @ ou
tput_vector_hidden.T)

# calculate hidden errors:

hidden_errors = self.weights_hidden_out.T @ output_error
# update the weights:
tmp = hidden_errors * output_vector_hidden * (1.0 - outpu
t_vector_hidden)
if [Link]:
x = (tmp @input_vector.T)[:-1,:] # last row cut of
f,
else:
x = tmp @ input_vector.T
self.weights_in_hidden += self.learning_rate * x

def run(self, input_vector):

"""
running the network with an input vector 'input_vector'.
'input_vector' can be tuple, list or ndarray
"""
# make sure that input_vector is a column vector:
input_vector = [Link](input_vector)
input_vector = input_vector.reshape(input_vector.size, 1)
if [Link]:
# adding bias node to the end of the inpuy_vector
input_vector = [Link]( (input_vector, [[1]]) )
input4hidden = activation_function(self.weights_in_hidden

178
 @ input_vector)
if [Link]:
input4hidden = [Link]( (input4hidden, [[1]]) )
output_vector_network = activation_function(self.weights_h
idden_out @ input4hidden)
return output_vector_network

def evaluate(self, data, labels):

corrects, wrongs = 0, 0
for i in range(len(data)):
res = [Link](data[i])
res_max = [Link]()
if res_max == labels[i].argmax():
corrects += 1
else:
wrongs += 1
return corrects, wrongs

We can use again our previously created classes to test our classifier:

from neural_networks2 import NeuralNetwork

simple_network = NeuralNetwork(no_of_in_nodes=2,
no_of_out_nodes=3,
no_of_hidden_nodes=5,
learning_rate=0.1,
bias=1)

for i in range(len(train_data)):
simple_network.train(train_data[i], train_labels[i])

simple_network.evaluate(train_data, train_labels)
Output: (382, 18)

EXERCISE
We created in the chapter "Data Creation" a file strange_flowers.txt in the folder data . Create a
Neural Network to classify the 'flowers':

The data looks like this:

0.000,240.000,100.000,3.020

179
 253.000,99.000,13.000,3.875
202.000,107.000,6.000,4.1
186.000,84.000,6.000,4.068
0.000,244.000,103.000,3.386
0.000,246.000,98.000,2.955
241.000,103.000,3.000,4.049
236.000,104.000,12.000,3.087
244.000,109.000,1.000,3.111
253.000,97.000,8.000,3.752
231.000,92.000,1.000,3.488
0.000,250.000,103.000,3.379

SOLUTION:
c = [Link]("data/strange_flowers.txt", delimiter=" ")

data = c[:, :-1]

n_classes = [Link][1]
labels = c[:, -1]
data[:5]
Output: array([[242. , 117. , 1. , 3.87],
[236. , 104. , 6. , 4.11],
[238. , 116. , 5. , 3.9 ],
[248. , 96. , 6. , 3.91],
[252. , 104. , 4. , 3.75]])

labels = [Link](n_classes) == [Link]([Link], 1)

labels = [Link]([Link])
labels[:3]
Output: array([[0., 1., 0., 0.],
[0., 1., 0., 0.],
[0., 1., 0., 0.]])

We need to scale our data, because unscaled input data can result in a slow or unstable learning process. We
will use the function scale from sklearn/preprocessing . It standardizes a dataset along any axis.
It centers to the mean and component wise scale to unit variance.

from sklearn import preprocessing

data = [Link](data)
data[:5]
[Link]
[Link]

180
Output: (795, 4)

from sklearn.model_selection import train_test_split

res = train_test_split(data, labels,

train_size=0.8,
test_size=0.2,
random_state=42)
train_data, test_data, train_labels, test_labels = res
train_labels[:10]
Output: array([[0., 0., 1., 0.],
[0., 0., 1., 0.],
[0., 0., 0., 1.],
[0., 0., 1., 0.],
[0., 0., 0., 1.],
[0., 0., 1., 0.],
[0., 1., 0., 0.],
[0., 1., 0., 0.],
[0., 0., 0., 1.],
[0., 0., 1., 0.]])

from neural_networks2 import NeuralNetwork

simple_network = NeuralNetwork(no_of_in_nodes=4,
no_of_out_nodes=4,
no_of_hidden_nodes=20,
learning_rate=0.3)

for i in range(len(train_data)):
simple_network.train(train_data[i], train_labels[i])

simple_network.evaluate(train_data, train_labels)
Output: (492, 144)

In [ ]:

181
SOFTMAX AS ACTIVATION FUNCTION

SOFTMAX
The previous implementations of neural networks in our tutorial
returned float values in the open interval (0, 1). To make a final
decision we had to interprete the results of the output neurons.
The one with the highest value is a likely candidate but we also
have to see it in relation to the other results. It should be obvious
that in a two classes case (c 1 and c 2) a result (0.013, 0.95) is a
clear vote for the class c 2 but (0.73, 0.89) on the other hand is a
different thing. We could say in this situation 'c 2 is more likely
than c 1, but c 1 has still a high likelihood'. Talking about
likelihoods: The return values are not probabilities. It would be
a lot better to have a normalized output with a probability
function. Here comes the softmax function into the picture. The
softmax function, also known as softargmax or normalized
exponential function, is a function that takes as input a vector of
n real numbers, and normalizes it into a probability distribution
consisting of n probabilities proportional to the exponentials of
the input vector. A probability distribution implies that the result
vector sums up to 1. Needless to say, if some components of the
input vector are negative or greater than one, they will be in the
range (0, 1) after applying Softmax . The Softmax function is
often used in neural networks, to map the results of the output
layer, which is non-normalized, to a probability distribution over
predicted output classes.

The softmax function σ is defined by the following formula:

eoi
σ(o i) =
∑n eoj
j=1

where the index i is in (0, ..., n-1) and o is the output vector of the network

o = (o 0, o 1, …, o n − 1)

We can implement the softmax function like this:

import numpy as np

182
 def softmax(x):
""" applies softmax to an input x"""
e_x = [Link](x)
return e_x / e_x.sum()

x = [Link]([1, 0, 3, 5])
y = softmax(x)
y, x / [Link]()
Output: (array([0.01578405, 0.00580663, 0.11662925, 0.86178007]),
array([0.11111111, 0. , 0.33333333, 0.55555556]))

Avoiding underflow or overflow errors due to floating point instability:

import numpy as np

def softmax(x):
""" applies softmax to an input x"""
e_x = [Link](x - [Link](x))
return e_x / e_x.sum()

softmax(x)
Output: array([0.01578405, 0.00580663, 0.11662925, 0.86178007])

x = [Link]([0.3, 0.4, 0.00005], np.float64) print(softmax(x)) print(x / [Link]())

DERIVATE OF SOFTMAX FUNCTION

The softmax function can be written as

[][]
o1 s1
o2 s2
S(o) : ?
⋯ ⋯
on sn

Per element it looks like this:

183
 eoj
s j(o) = n , ∀k = 1, ⋯, n
o
∑e k
k=1

The derivative of softmax can be calculated like this:

[ ]
∂s 1 ∂s 1
∂o 1
⋯ ∂o n
∂S
= ⋯
∂O
∂s n ∂s n
∂o 1
⋯ ∂o n

The partial derivatives can be solved for every i and j:

eoi
∂
∂s i ∑n eok
k=1
=
∂o j ∂o j

We will use the quotien rule, i.e.

the derivative of

g(x)
f(x) =
h(x)

is

g ′ (x) ⋅ h(x) − g(x) ⋅ h ′ (x)

f ′ (x) =
(h(x) 2

We can set g(x) to e o i and h(x) to ∑ n eok

k=1

The derivative of g(x) is

g ′ (x) =
{ e o i,
0,
if i = j
otherwise

and the derivative of h(x) is

184
 h ′ (x) = e o j, ∀k = 1, ⋯, n

Let's apply the quotient rule by case differentiation now:

1. case: i = j:

eoi ⋅ ∑n eok − eoi ⋅ eoj

k=1

( ∑n e o k) 2
k=1

We can rewrite this expression as

∑n eok − eoj
eoi k=1
⋅
∑n eok ∑n eok
k=1 k=1

Now we can reduce the second quotient:

eoi eoj
⋅ (1 − )
∑n eok ∑n eok
k=1 k=1

If we compare this expression with the Definition of s i, we can rewrite it to:

s i ⋅ (1 − s j)

which is the same as

s i ⋅ (1 − s i)

because i = j.

1. case: i ≠ j:

0 ⋅ ∑n eok − eoi ⋅ eoj

k=1

( ∑n e o k) 2
k=1

this can be rewritten as:

eoi eoj
− ⋅
∑n eok ∑n eok
k=1 k=1

this gives us finally:

185
 − si ⋅ sj

We can summarize these two cases and write the derivative as:

g ′ (x) =
{ s i ⋅ (1 − s i),
− s i ⋅ s j,
if i = j
otherwise

If we use the Kronecker delta function1, we can get rid of the case differentiation, i.e. we "let the Kronecker
delta do this work":

∂s i
= s i(δ ij − s j)
∂o j

Finally we can calculate the derivative of softmax:

[ ]
s 1(δ 11 − s 1) s 1(δ 12 − s 2) ⋯ s 1(δ 1n − s n)

∂S s 2(δ 21 − s 1) s 2(δ 22 − s 2) ⋯ s 2(δ 2n − s n)

=
∂O ⋯
s n(δ n1 − s 1) s n(δ n2 − s 2) ⋯ s n(δ nn − s n)

import numpy as np

def softmax(x):
e_x = [Link](x)
return e_x / e_x.sum()

s = softmax([Link]([0, 4, 5]))

si_sj = - s * [Link](3, 1)
print(s)
print(si_sj)
s_der = [Link](s) + si_sj
s_der

186
[0.00490169 0.26762315 0.72747516]
[[-2.40265555e-05 -1.31180548e-03 -3.56585701e-03]
[-1.31180548e-03 -7.16221526e-02 -1.94689196e-01]
[-3.56585701e-03 -1.94689196e-01 -5.29220104e-01]]
Output: array([[ 0.00487766, -0.00131181, -0.00356586],
[-0.00131181, 0.196001 , -0.1946892 ],
[-0.00356586, -0.1946892 , 0.19825505]])

import numpy as np
from [Link] import truncnorm

def truncated_normal(mean=0, sd=1, low=0, upp=10):

return truncnorm(
(low - mean) / sd, (upp - mean) / sd, loc=mean, scale=sd)

@[Link]
def sigmoid(x):
return 1 / (1 + np.e ** -x)

def softmax(x):
e_x = [Link](x)
return e_x / e_x.sum()

class NeuralNetwork:

def __init__(self,
no_of_in_nodes,
no_of_out_nodes,
no_of_hidden_nodes,
learning_rate,
softmax=True):
self.no_of_in_nodes = no_of_in_nodes
self.no_of_out_nodes = no_of_out_nodes
self.no_of_hidden_nodes = no_of_hidden_nodes
self.learning_rate = learning_rate
[Link] = softmax
self.create_weight_matrices()

def create_weight_matrices(self):
""" A method to initialize the weight matrices of the neur
al network"""
rad = 1 / [Link](self.no_of_in_nodes)
X = truncated_normal(mean=0, sd=1, low=-rad, upp=rad)

187
 self.weights_in_hidden = [Link]((self.no_of_hidden_nodes,
self.no_of_in_nodes))
rad = 1 / [Link](self.no_of_hidden_nodes)
X = truncated_normal(mean=0, sd=1, low=-rad, upp=rad)
self.weights_hidden_out = [Link]((self.no_of_out_nodes,
self.no_of_hidden_nodes))

def train(self, input_vector, target_vector):

"""
input_vector and target_vector can be tuples, lists or nda
rrays
"""
# make sure that the vectors have the right shape
input_vector = [Link](input_vector)
input_vector = input_vector.reshape(input_vector.size, 1)
target_vector = [Link](target_vector).reshape(target_vec
[Link], 1)

output_vector_hidden = sigmoid(self.weights_in_hidden @ in
put_vector)
if [Link]:
output_vector_network = softmax(self.weights_hidden_ou
t @ output_vector_hidden)
else:
output_vector_network = sigmoid(self.weights_hidden_ou
t @ output_vector_hidden)

output_error = target_vector - output_vector_network

if [Link]:
ovn = output_vector_network.reshape(output_vector_netw
[Link],)
si_sj = - ovn * [Link](self.no_of_out_nodes, 1)
s_der = [Link](ovn) + si_sj
tmp = s_der @ output_error
self.weights_hidden_out += self.learning_rate * (tmp
@ output_vector_hidden.T)
else:
tmp = output_error * output_vector_network * (1.0 - ou
tput_vector_network)
self.weights_hidden_out += self.learning_rate * (tmp
@ output_vector_hidden.T)

188
 # calculate hidden errors:
hidden_errors = self.weights_hidden_out.T @ output_error
# update the weights:
tmp = hidden_errors * output_vector_hidden * (1.0 - outpu
t_vector_hidden)
self.weights_in_hidden += self.learning_rate * (tmp @ inpu
t_vector.T)

def run(self, input_vector):

"""
running the network with an input vector 'input_vector'.
'input_vector' can be tuple, list or ndarray
"""
# make sure that input_vector is a column vector:
input_vector = [Link](input_vector)
input_vector = input_vector.reshape(input_vector.size, 1)
input4hidden = sigmoid(self.weights_in_hidden @ input_vect
or)
if [Link]:
output_vector_network = softmax(self.weights_hidden_ou
t @ input4hidden)
else:
output_vector_network = sigmoid(self.weights_hidden_ou
t @ input4hidden)

return output_vector_network

def evaluate(self, data, labels):

corrects, wrongs = 0, 0
for i in range(len(data)):
res = [Link](data[i])
res_max = [Link]()
if res_max == labels[i]:
corrects += 1
else:
wrongs += 1
return corrects, wrongs

from [Link] import make_blobs

n_samples = 300
samples, labels = make_blobs(n_samples=n_samples,
centers=([2, 6], [6, 2]),
random_state=0)

189
import [Link] as plt

colours = ('green', 'red', 'blue', 'magenta', 'yellow', 'cyan')

fig, ax = [Link]()

for n_class in range(2):

[Link](samples[labels==n_class][:, 0], samples[labels==n_c
lass][:, 1],
c=colours[n_class], s=40, label=str(n_class))

size_of_learn_sample = int(n_samples * 0.8)

learn_data = samples[:size_of_learn_sample]
test_data = samples[-size_of_learn_sample:]

from neural_networks_softmax import NeuralNetwork

simple_network = NeuralNetwork(no_of_in_nodes=2,
no_of_out_nodes=2,
no_of_hidden_nodes=5,
learning_rate=0.3,
softmax=True)

for x in [(1, 4), (2, 6), (3, 3), (6, 2)]:

y = simple_network.run(x)
print(x, y, [Link]())
(1, 4) [[0.53325729]
[0.46674271]] 1.0
(2, 6) [[0.50669849]
[0.49330151]] 1.0
(3, 3) [[0.53050147]
[0.46949853]] 1.0
(6, 2) [[0.52530293]
[0.47469707]] 1.0

labels_one_hot = ([Link](2) == [Link]([Link], 1))

labels_one_hot = labels_one_hot.astype([Link])

for i in range(size_of_learn_sample):
#print(learn_data[i], labels[i], labels_one_hot[i])
simple_network.train(learn_data[i],
labels_one_hot[i])

from collections import Counter

190
evaluation = Counter()
simple_network.evaluate(learn_data, labels)
Output: (236, 4)

FOOTNOTES
1
Kronecker delta:

δ ij =
{ 1,
0,
if i = j
if i ≠ j

191
CONFUSION MATRIX

In the previous chapters of our Machine

Learning tutorial (Neural Networks with
Python and Numpy and Neural Networks
from Scratch ) we implemented various
algorithms, but we didn't properly
measure the quality of the output. The
main reason was that we used very simple
and small datasets to learn and test. In the
chapter Neural Network: Testing with
MNIST, we will work with large datasets
and ten classes, so we need proper
evaluations tools. We will introduce in
this chapter the concepts of the confusion
matrix:

A confusion matrix is a matrix (table) that can be used to measure the performance of an machine learning
algorithm, usually a supervised learning one. Each row of the confusion matrix represents the instances of an
actual class and each column represents the instances of a predicted class. This is the way we keep it in this
chapter of our tutorial, but it can be the other way around as well, i.e. rows for predicted classes and columns
for actual classes. The name confusion matrix reflects the fact that it makes it easy for us to see what kind of
confusions occur in our classification algorithms. For example the algorithms should have predicted a sample
as c i because the actual class is c i, but the algorithm came out with c j. In this case of mislabelling the element
cm[i, j] will be incremented by one, when the confusion matrix is constructed.

We will define methods to calculate the confusion matrix, precision and recall in the following class.

2-CLASS CASE
In a 2-class case, i.e. "negative" and "positive", the confusion matrix may look like this:

predicted

actual negative positive

negative 11 0

positive 1 12

192
The fields of the matrix mean the following:

predicted

actual negative positive

negative TN FP
True positive False Positive

positive FN TP
False negative True positive

We can define now some important performance measures used in machine learning:

Accuracy:

TN + TP
AC =
TN + FP + FN + TP

The accuracy is not always an adequate performance measure. Let us assume we have 1000 samples. 995 of
these are negative and 5 are positive cases. Let us further assume we have a classifier, which classifies
whatever it will be presented as negative. The accuracy will be a surprising 99.5%, even though the classifier
could not recognize any positive samples.

Recall aka. True Positive Rate:

TP
recall =
FN + TP

True Negative Rate:

FP
TNR =
TN + FP

Precision:

TP
precision :
FP + TP

193
MULTI-CLASS CASE
To measure the results of machine learning algorithms, the previous confusion matrix will not be sufficient.
We will need a generalization for the multi-class case.

Let us assume that we have a sample of 25 animals, e.g. 7 cats, 8 dogs, and 10 snakes, most probably Python
snakes. The confusion matrix of our recognition algorithm may look like the following table:

predicted

actual dog cat snake

dog 6 2 0

cat 1 6 0

snake 1 1 8

In this confusion matrix, the system correctly predicted six of the eight actual dogs, but in two cases it took a
dog for a cat. The seven acutal cats were correctly recognized in six cases but in one case a cat was taken to be
a dog. Usually, it is hard to take a snake for a dog or a cat, but this is what happened to our classifier in two
cases. Yet, eight out of ten snakes had been correctly recognized. (Most probably this machine learning
algorithm was not written in a Python program, because Python should properly recognize its own species :-) )

You can see that all correct predictions are located in the diagonal of the table, so prediction errors can be
easily found in the table, as they will be represented by values outside the diagonal.

We can generalize this to the multi-class case. To do this we summarize over the rows and columns of the
confusion matrix. Given that the matrix is oriented as above, i.e., that a given row of the matrix corresponds to
specific value for the "truth", we have:

M ii
Precision i =
∑ jM ji

M ii
Recall i =
∑ jM ij

This means, precision is the fraction of cases where the algorithm correctly predicted class i out of all
instances where the algorithm predicted i (correctly and incorrectly). recall on the other hand is the fraction of
cases where the algorithm correctly predicted i out of all of the cases which are labelled as i.

Let us apply this to our example:

194
The precision for our animals can be calculated as

precision dogs = 6 / (6 + 1 + 1) = 3 / 4 = 0.75

precision cats = 6 / (2 + 6 + 1) = 6 / 9 = 0.67

precision snakes = 8 / (0 + 0 + 8) = 1

The recall is calculated like this:

recall dogs = 6 / (6 + 2 + 0) = 3 / 4 = 0.75

recall cats = 6 / (1 + 6 + 0) = 6 / 7 = 0.86

recall snakes = 8 / (1 + 1 + 8) = 4 / 5 = 0.8

EXAMPLE
We are ready now to code this into Python. The following code shows a confusion matrix for a multi-class
machine learning problem with ten labels, so for example an algorithms for recognizing the ten digits from
handwritten characters.

If you are not familiar with Numpy and Numpy arrays, we recommend our tutorial on Numpy.

import numpy as np

cm = [Link](
[[5825, 1, 49, 23, 7, 46, 30, 12, 21, 26],
[ 1, 6654, 48, 25, 10, 32, 19, 62, 111, 10],
[ 2, 20, 5561, 69, 13, 10, 2, 45, 18, 2],
[ 6, 26, 99, 5786, 5, 111, 1, 41, 110, 79],
[ 4, 10, 43, 6, 5533, 32, 11, 53, 34, 79],
[ 3, 1, 2, 56, 0, 4954, 23, 0, 12, 5],
[ 31, 4, 42, 22, 45, 103, 5806, 3, 34, 3],
[ 0, 4, 30, 29, 5, 6, 0, 5817, 2, 28],
[ 35, 6, 63, 58, 8, 59, 26, 13, 5394, 24],
[ 16, 16, 21, 57, 216, 68, 0, 219, 115, 5693]])

The functions 'precision' and 'recall' calculate values for a label, whereas the function
'precision_macro_average' the precision for the whole classification problem calculates.

def precision(label, confusion_matrix):

col = confusion_matrix[:, label]
return confusion_matrix[label, label] / [Link]()

195
def recall(label, confusion_matrix):
row = confusion_matrix[label, :]
return confusion_matrix[label, label] / [Link]()

def precision_macro_average(confusion_matrix):
rows, columns = confusion_matrix.shape
sum_of_precisions = 0
for label in range(rows):
sum_of_precisions += precision(label, confusion_matrix)
return sum_of_precisions / rows

def recall_macro_average(confusion_matrix):
rows, columns = confusion_matrix.shape
sum_of_recalls = 0
for label in range(columns):
sum_of_recalls += recall(label, confusion_matrix)
return sum_of_recalls / columns

print("label precision recall")

for label in range(10):
print(f"{label:5d} {precision(label, cm):9.3f} {recall(label,
cm):6.3f}")
label precision recall
0 0.983 0.964
1 0.987 0.954
2 0.933 0.968
3 0.944 0.924
4 0.947 0.953
5 0.914 0.980
6 0.981 0.953
7 0.928 0.982
8 0.922 0.949
9 0.957 0.887

print("precision total:", precision_macro_average(cm))

print("recall total:", recall_macro_average(cm))

precision total: 0.949688556405
recall total: 0.951453154788

def accuracy(confusion_matrix):
diagonal_sum = confusion_matrix.trace()
sum_of_all_elements = confusion_matrix.sum()

196
 return diagonal_sum / sum_of_all_elements

accuracy(cm)
Output: 0.95038333333333336

197
NEURAL NETWORK

USING MNIST

The MNIST database (Modified National Institute of

Standards and Technology database) of handwritten
digits consists of a training set of 60,000 examples,
and a test set of 10,000 examples. It is a subset of a
larger set available from NIST. Additionally, the
black and white images from NIST were size-
normalized and centered to fit into a 28x28 pixel
bounding box and anti-aliased, which introduced
grayscale levels.

This database is well liked for training and testing in

the field of machine learning and image processing.
It is a remixed subset of the original NIST datasets.
One half of the 60,000 training images consist of
images from NIST's testing dataset and the other half
from Nist's training set. The 10,000 images from the
testing set are similarly assembled.

The MNIST dataset is used by researchers to test and

compare their research results with others. The
lowest error rates in literature are as low as 0.21
percent.1

READING THE MNIST DATA SET

The images from the data set have the size 28 x 28. They are saved in the csv data files mnist_train.csv and
mnist_test.csv.

Every line of these files consists of an image, i.e. 785 numbers between 0 and 255.

The first number of each line is the label, i.e. the digit which is depicted in the image. The following 784
numbers are the pixels of the 28 x 28 image.

import numpy as np

198
 import [Link] as plt

image_size = 28 # width and length

no_of_different_labels = 10 # i.e. 0, 1, 2, 3, ..., 9
image_pixels = image_size * image_size
data_path = "data/mnist/"
train_data = [Link](data_path + "mnist_train.csv",
delimiter=",")
test_data = [Link](data_path + "mnist_test.csv",
delimiter=",")
test_data[:10]
Output: array([[7., 0., 0., ..., 0., 0., 0.],
[2., 0., 0., ..., 0., 0., 0.],
[1., 0., 0., ..., 0., 0., 0.],
...,
[9., 0., 0., ..., 0., 0., 0.],
[5., 0., 0., ..., 0., 0., 0.],
[9., 0., 0., ..., 0., 0., 0.]])

test_data[test_data==255]
test_data.shape
Output: (10000, 785)

The images of the MNIST dataset are greyscale and the pixels range between 0 and 255 including both
bounding values. We will map these values into an interval from [0.01, 1] by multiplying each pixel by 0.99 /
255 and adding 0.01 to the result. This way, we avoid 0 values as inputs, which are capable of preventing
weight updates, as we we seen in the introductory chapter.

fac = 0.99 / 255

train_imgs = [Link](train_data[:, 1:]) * fac + 0.01
test_imgs = [Link](test_data[:, 1:]) * fac + 0.01

train_labels = [Link](train_data[:, :1])

test_labels = [Link](test_data[:, :1])

We need the labels in our calculations in a one-hot representation. We have 10 digits from 0 to 9, i.e. lr =
[Link](10).

Turning a label into one-hot representation can be achieved with the command: (lr==label).astype([Link])

We demonstrate this in the following:

import numpy as np

199
lr = [Link](10)

for label in range(10):

one_hot = (lr==label).astype([Link])
print("label: ", label, " in one-hot representation: ", one_ho
t)
label: 0 in one-hot representation: [1 0 0 0 0 0 0 0 0 0]
label: 1 in one-hot representation: [0 1 0 0 0 0 0 0 0 0]
label: 2 in one-hot representation: [0 0 1 0 0 0 0 0 0 0]
label: 3 in one-hot representation: [0 0 0 1 0 0 0 0 0 0]
label: 4 in one-hot representation: [0 0 0 0 1 0 0 0 0 0]
label: 5 in one-hot representation: [0 0 0 0 0 1 0 0 0 0]
label: 6 in one-hot representation: [0 0 0 0 0 0 1 0 0 0]
label: 7 in one-hot representation: [0 0 0 0 0 0 0 1 0 0]
label: 8 in one-hot representation: [0 0 0 0 0 0 0 0 1 0]
label: 9 in one-hot representation: [0 0 0 0 0 0 0 0 0 1]

We are ready now to turn our labelled images into one-hot representations. Instead of zeroes and one, we
create 0.01 and 0.99, which will be better for our calculations:

lr = [Link](no_of_different_labels)

# transform labels into one hot representation

train_labels_one_hot = (lr==train_labels).astype([Link])
test_labels_one_hot = (lr==test_labels).astype([Link])

# we don't want zeroes and ones in the labels neither:

train_labels_one_hot[train_labels_one_hot==0] = 0.01
train_labels_one_hot[train_labels_one_hot==1] = 0.99
test_labels_one_hot[test_labels_one_hot==0] = 0.01
test_labels_one_hot[test_labels_one_hot==1] = 0.99

Before we start using the MNIST data sets with our neural network, we will have a look at some images:

for i in range(10):
img = train_imgs[i].reshape((28,28))
[Link](img, cmap="Greys")
[Link]()

200
201
202
203
DUMPING THE DATA FOR FASTER RELOAD
You may have noticed that it is quite slow to read in the data from the csv files.

We will save the data in binary format with the dump function from the pickle module:

import pickle

with open("data/mnist/pickled_mnist.pkl", "bw") as fh:

data = (train_imgs,
test_imgs,
train_labels,
test_labels,
train_labels_one_hot,
test_labels_one_hot)
[Link](data, fh)

We are able now to read in the data by using [Link]. This is a lot faster than using loadtxt on the csv files:

import pickle

with open("data/mnist/pickled_mnist.pkl", "br") as fh:

data = [Link](fh)

train_imgs = data[0]

204
 test_imgs = data[1]
train_labels = data[2]
test_labels = data[3]
train_labels_one_hot = data[4]
test_labels_one_hot = data[5]

image_size = 28 # width and length

no_of_different_labels = 10 # i.e. 0, 1, 2, 3, ..., 9
image_pixels = image_size * image_size

CLASSIFYING THE DATA

We will use the following neuronal network class for our first classification:

import numpy as np

@[Link]
def sigmoid(x):
return 1 / (1 + np.e ** -x)
activation_function = sigmoid

from [Link] import truncnorm

def truncated_normal(mean=0, sd=1, low=0, upp=10):

return truncnorm((low - mean) / sd,
(upp - mean) / sd,
loc=mean,
scale=sd)

class NeuralNetwork:

def __init__(self,
no_of_in_nodes,
no_of_out_nodes,
no_of_hidden_nodes,
learning_rate):
self.no_of_in_nodes = no_of_in_nodes
self.no_of_out_nodes = no_of_out_nodes
self.no_of_hidden_nodes = no_of_hidden_nodes
self.learning_rate = learning_rate

205
 self.create_weight_matrices()

def create_weight_matrices(self):
"""
A method to initialize the weight
matrices of the neural network
"""
rad = 1 / [Link](self.no_of_in_nodes)
X = truncated_normal(mean=0,
sd=1,
low=-rad,
upp=rad)
[Link] = [Link]((self.no_of_hidden_nodes,
self.no_of_in_nodes))
rad = 1 / [Link](self.no_of_hidden_nodes)
X = truncated_normal(mean=0, sd=1, low=-rad, upp=rad)
[Link] = [Link]((self.no_of_out_nodes,
self.no_of_hidden_nodes))

def train(self, input_vector, target_vector):

"""
input_vector and target_vector can
be tuple, list or ndarray
"""

input_vector = [Link](input_vector, ndmin=2).T

target_vector = [Link](target_vector, ndmin=2).T

output_vector1 = [Link]([Link],
input_vector)
output_hidden = activation_function(output_vector1)

output_vector2 = [Link]([Link],
output_hidden)
output_network = activation_function(output_vector2)

output_errors = target_vector - output_network

# update the weights:
tmp = output_errors * output_network \
* (1.0 - output_network)
tmp = self.learning_rate * [Link](tmp,
output_hidden.T)
[Link] += tmp

206
 # calculate hidden errors:
hidden_errors = [Link]([Link].T,
output_errors)
# update the weights:
tmp = hidden_errors * output_hidden * \
(1.0 - output_hidden)
[Link] += self.learning_rate \
* [Link](tmp, input_vector.T)

def run(self, input_vector):

# input_vector can be tuple, list or ndarray
input_vector = [Link](input_vector, ndmin=2).T

output_vector = [Link]([Link],
input_vector)
output_vector = activation_function(output_vector)

output_vector = [Link]([Link],
output_vector)
output_vector = activation_function(output_vector)

return output_vector

def confusion_matrix(self, data_array, labels):

cm = [Link]((10, 10), int)
for i in range(len(data_array)):
res = [Link](data_array[i])
res_max = [Link]()
target = labels[i][0]
cm[res_max, int(target)] += 1
return cm

def precision(self, label, confusion_matrix):

col = confusion_matrix[:, label]
return confusion_matrix[label, label] / [Link]()

def recall(self, label, confusion_matrix):

row = confusion_matrix[label, :]
return confusion_matrix[label, label] / [Link]()

207
 def evaluate(self, data, labels):
corrects, wrongs = 0, 0
for i in range(len(data)):
res = [Link](data[i])
res_max = [Link]()
if res_max == labels[i]:
corrects += 1
else:
wrongs += 1
return corrects, wrongs

ANN = NeuralNetwork(no_of_in_nodes = image_pixels,

no_of_out_nodes = 10,
no_of_hidden_nodes = 100,
learning_rate = 0.1)

for i in range(len(train_imgs)):
[Link](train_imgs[i], train_labels_one_hot[i])

for i in range(20):
res = [Link](test_imgs[i])
print(test_labels[i], [Link](res), [Link](res))

[7.] 7 0.9829245583409039
[2.] 2 0.7372766887508578
[1.] 1 0.9881823673106839
[0.] 0 0.9873289971465894
[4.] 4 0.9456335245615916
[1.] 1 0.9880120617106172
[4.] 4 0.976550583573903
[9.] 9 0.964909168118122
[5.] 6 0.36615932726182665
[9.] 9 0.9848677489827125
[0.] 0 0.9204097234781773
[6.] 6 0.8897871402453337
[9.] 9 0.9936811621891628
[0.] 0 0.9832119513084644
[1.] 1 0.988750833073612
[5.] 5 0.9156741221523511
[9.] 9 0.9812577974620423
[7.] 7 0.9888560485875889
[3.] 3 0.8772868556722897
[4.] 4 0.9900030761222965

208
corrects, wrongs = [Link](train_imgs, train_labels)
print("accuracy train: ", corrects / ( corrects + wrongs))
corrects, wrongs = [Link](test_imgs, test_labels)
print("accuracy: test", corrects / ( corrects + wrongs))

cm = ANN.confusion_matrix(train_imgs, train_labels)
print(cm)

for i in range(10):
print("digit: ", i, "precision: ", [Link](i, cm), "reca
ll: ", [Link](i, cm))
accuracy train: 0.9469166666666666
accuracy: test 0.9459
[[5802 0 53 21 9 42 35 8 14 20]
[ 1 6620 45 22 6 29 14 50 75 7]
[ 5 22 5486 51 10 11 5 53 11 3]
[ 6 36 114 5788 2 114 1 35 76 72]
[ 8 16 54 8 5439 41 10 52 25 90]
[ 5 2 3 44 0 4922 20 3 5 11]
[ 37 4 54 19 71 72 5789 3 41 4]
[ 0 5 31 38 7 4 0 5762 1 32]
[ 52 20 103 83 9 102 43 21 5535 38]
[ 7 17 15 57 289 84 1 278 68 5672]]
digit: 0 precision: 0.9795711632618606 recall: 0.96635576282478
35
digit: 1 precision: 0.9819044793829724 recall: 0.96375018197699
81
digit: 2 precision: 0.9207787848271232 recall: 0.96977196393848
33
digit: 3 precision: 0.9440548034578372 recall: 0.92696989109545
16
digit: 4 precision: 0.9310167750770284 recall: 0.94706599338324
91
digit: 5 precision: 0.9079505626268216 recall: 0.98145563310069
79
digit: 6 precision: 0.978202095302467 recall: 0.949950771250410
3
digit: 7 precision: 0.9197126895450918 recall: 0.97993197278911
57
digit: 8 precision: 0.945992138096052 recall: 0.921578421578421
6
digit: 9 precision: 0.953437552529837 recall: 0.87422934648582

209
MULTIPLE RUNS

We can repeat the training multiple times. Each run is called an "epoch".

epochs = 3

NN = NeuralNetwork(no_of_in_nodes = image_pixels,
no_of_out_nodes = 10,
no_of_hidden_nodes = 100,
learning_rate = 0.1)

for epoch in range(epochs):

print("epoch: ", epoch)
for i in range(len(train_imgs)):
[Link](train_imgs[i],
train_labels_one_hot[i])

corrects, wrongs = [Link](train_imgs, train_labels)

print("accuracy train: ", corrects / ( corrects + wrongs))
corrects, wrongs = [Link](test_imgs, test_labels)
print("accuracy: test", corrects / ( corrects + wrongs))
epoch: 0
accruracy train: 0.94515
accruracy: test 0.9459
epoch: 1
accruracy train: 0.9626833333333333
accruracy: test 0.9582
epoch: 2
accruracy train: 0.96995
accruracy: test 0.9626

We want to do the multiple training of the training set inside of our network. To this purpose we rewrite the
method train and add a method train_single. train_single is more or less what we called 'train' before. Whereas
the new 'train' method is doing the epoch counting. For testing purposes, we save the weight matrices after
each epoch in
the list intermediate_weights. This list is returned as the output of train:

import numpy as np

@[Link]
def sigmoid(x):

210
 return 1 / (1 + np.e ** -x)
activation_function = sigmoid

from [Link] import truncnorm

def truncated_normal(mean=0, sd=1, low=0, upp=10):

return truncnorm((low - mean) / sd,
(upp - mean) / sd,
loc=mean,
scale=sd)

class NeuralNetwork:

def __init__(self,
no_of_in_nodes,
no_of_out_nodes,
no_of_hidden_nodes,
learning_rate):
self.no_of_in_nodes = no_of_in_nodes
self.no_of_out_nodes = no_of_out_nodes
self.no_of_hidden_nodes = no_of_hidden_nodes
self.learning_rate = learning_rate
self.create_weight_matrices()

def create_weight_matrices(self):
""" A method to initialize the weight matrices of the neur
al network"""
rad = 1 / [Link](self.no_of_in_nodes)
X = truncated_normal(mean=0,
sd=1,
low=-rad,
upp=rad)
[Link] = [Link]((self.no_of_hidden_nodes,
self.no_of_in_nodes))
rad = 1 / [Link](self.no_of_hidden_nodes)
X = truncated_normal(mean=0,
sd=1,
low=-rad,
upp=rad)
[Link] = [Link]((self.no_of_out_nodes,
self.no_of_hidden_nodes))

def train_single(self, input_vector, target_vector):

211
 """
input_vector and target_vector can be tuple,
list or ndarray
"""

output_vectors = []
input_vector = [Link](input_vector, ndmin=2).T
target_vector = [Link](target_vector, ndmin=2).T

output_vector1 = [Link]([Link],
input_vector)
output_hidden = activation_function(output_vector1)

output_vector2 = [Link]([Link],
output_hidden)
output_network = activation_function(output_vector2)

output_errors = target_vector - output_network

# update the weights:
tmp = output_errors * output_network * \
(1.0 - output_network)
tmp = self.learning_rate * [Link](tmp,
output_hidden.T)
[Link] += tmp

# calculate hidden errors:

hidden_errors = [Link]([Link].T,
output_errors)
# update the weights:
tmp = hidden_errors * output_hidden * (1.0 - output_hidde
n)
[Link] += self.learning_rate * [Link](tmp, input_vecto
r.T)

def train(self, data_array,

labels_one_hot_array,
epochs=1,
intermediate_results=False):
intermediate_weights = []
for epoch in range(epochs):
print("*", end="")
for i in range(len(data_array)):

212
 self.train_single(data_array[i],
labels_one_hot_array[i])
if intermediate_results:
intermediate_weights.append(([Link](),
[Link]()))
return intermediate_weights

def confusion_matrix(self, data_array, labels):

cm = {}
for i in range(len(data_array)):
res = [Link](data_array[i])
res_max = [Link]()
target = labels[i][0]
if (target, res_max) in cm:
cm[(target, res_max)] += 1
else:
cm[(target, res_max)] = 1
return cm

def run(self, input_vector):

""" input_vector can be tuple, list or ndarray """

input_vector = [Link](input_vector, ndmin=2).T

output_vector = [Link]([Link],
input_vector)
output_vector = activation_function(output_vector)

output_vector = [Link]([Link],
output_vector)
output_vector = activation_function(output_vector)

return output_vector

def evaluate(self, data, labels):

corrects, wrongs = 0, 0
for i in range(len(data)):
res = [Link](data[i])
res_max = [Link]()
if res_max == labels[i]:
corrects += 1
else:
wrongs += 1
return corrects, wrongs

213
epochs = 10

ANN = NeuralNetwork(no_of_in_nodes = image_pixels,

no_of_out_nodes = 10,
no_of_hidden_nodes = 100,
learning_rate = 0.15)

weights = [Link](train_imgs,
train_labels_one_hot,
epochs=epochs,
intermediate_results=True)

**********

cm = ANN.confusion_matrix(train_imgs, train_labels)

print([Link](train_imgs[i]))
[[2.60149245e-03]
[2.52542556e-03]
[6.57990628e-03]
[1.32663729e-03]
[1.34985384e-03]
[2.63840265e-04]
[2.18329159e-04]
[1.32693720e-04]
[9.84326084e-01]
[4.34559417e-02]]

cm = list([Link]())
print(sorted(cm))

214
 [((0.0, 0), 5853), ((0.0, 1), 1), ((0.0, 2), 3), ((0.0, 4), 8),
((0.0, 5), 2), ((0.0, 6), 12), ((0.0, 7), 7), ((0.0, 8), 27),
((0.0, 9), 10), ((1.0, 0), 1), ((1.0, 1), 6674), ((1.0, 2), 17),
((1.0, 3), 5), ((1.0, 4), 14), ((1.0, 5), 2), ((1.0, 6), 1),
((1.0, 7), 6), ((1.0, 8), 15), ((1.0, 9), 7), ((2.0, 0), 37),
((2.0, 1), 14), ((2.0, 2), 5791), ((2.0, 3), 17), ((2.0, 4), 11),
((2.0, 5), 2), ((2.0, 6), 10), ((2.0, 7), 15), ((2.0, 8), 51),
((2.0, 9), 10), ((3.0, 0), 16), ((3.0, 1), 5), ((3.0, 2), 34),
((3.0, 3), 5869), ((3.0, 4), 8), ((3.0, 5), 57), ((3.0, 6), 4),
((3.0, 7), 20), ((3.0, 8), 58), ((3.0, 9), 60), ((4.0, 0), 14),
((4.0, 1), 6), ((4.0, 2), 8), ((4.0, 3), 1), ((4.0, 4), 5678),
((4.0, 5), 1), ((4.0, 6), 14), ((4.0, 7), 5), ((4.0, 8), 11),
((4.0, 9), 104), ((5.0, 0), 7), ((5.0, 1), 2), ((5.0, 2), 6),
((5.0, 3), 27), ((5.0, 4), 5), ((5.0, 5), 5312), ((5.0, 6), 12),
((5.0, 7), 5), ((5.0, 8), 20), ((5.0, 9), 25), ((6.0, 0), 32),
((6.0, 1), 5), ((6.0, 2), 1), ((6.0, 4), 10), ((6.0, 5), 52),
((6.0, 6), 5791), ((6.0, 8), 26), ((6.0, 9), 1), ((7.0, 0), 5),
((7.0, 1), 11), ((7.0, 2), 22), ((7.0, 3), 2), ((7.0, 4), 17),
((7.0, 5), 3), ((7.0, 6), 2), ((7.0, 7), 6074), ((7.0, 8), 26),
((7.0, 9), 103), ((8.0, 0), 20), ((8.0, 1), 18), ((8.0, 2), 9),
((8.0, 3), 14), ((8.0, 4), 27), ((8.0, 5), 24), ((8.0, 6), 9),
((8.0, 7), 8), ((8.0, 8), 5668), ((8.0, 9), 54), ((9.0, 0), 26),
((9.0, 1), 2), ((9.0, 2), 2), ((9.0, 3), 16), ((9.0, 4), 69),
((9.0, 5), 14), ((9.0, 6), 7), ((9.0, 7), 19), ((9.0, 8), 15),
((9.0, 9), 5779)]
In [ ]:
for i in range(epochs):
print("epoch: ", i)
[Link] = weights[i][0]
[Link] = weights[i][1]

corrects, wrongs = [Link](train_imgs, train_labels)

print("accuracy train: ", corrects / ( corrects + wrongs))
corrects, wrongs = [Link](test_imgs, test_labels)
print("accuracy: test", corrects / ( corrects + wrongs))

215
WITH BIAS NODES

import numpy as np

@[Link]
def sigmoid(x):
return 1 / (1 + np.e ** -x)
activation_function = sigmoid

from [Link] import truncnorm

def truncated_normal(mean=0, sd=1, low=0, upp=10):

return truncnorm((low - mean) / sd,
(upp - mean) / sd,
loc=mean,
scale=sd)

class NeuralNetwork:

def __init__(self,
no_of_in_nodes,
no_of_out_nodes,
no_of_hidden_nodes,
learning_rate,
bias=None
):

self.no_of_in_nodes = no_of_in_nodes
self.no_of_out_nodes = no_of_out_nodes
self.no_of_hidden_nodes = no_of_hidden_nodes
self.learning_rate = learning_rate
[Link] = bias
self.create_weight_matrices()

def create_weight_matrices(self):
"""
A method to initialize the weight
matrices of the neural network with
optional bias nodes

216
 """

bias_node = 1 if [Link] else 0

rad = 1 / [Link](self.no_of_in_nodes + bias_node)

X = truncated_normal(mean=0,
sd=1,
low=-rad,
upp=rad)
[Link] = [Link]((self.no_of_hidden_nodes,
self.no_of_in_nodes + bias_node))

rad = 1 / [Link](self.no_of_hidden_nodes + bias_node)

X = truncated_normal(mean=0, sd=1, low=-rad, upp=rad)
[Link] = [Link]((self.no_of_out_nodes,
self.no_of_hidden_nodes + bias_node))

def train(self, input_vector, target_vector):

"""
input_vector and target_vector can
be tuple, list or ndarray
"""

bias_node = 1 if [Link] else 0

if [Link]:
# adding bias node to the end of the inpuy_vector
input_vector = [Link]((input_vector,
[[Link]]) )

input_vector = [Link](input_vector, ndmin=2).T

target_vector = [Link](target_vector, ndmin=2).T

output_vector1 = [Link]([Link],
input_vector)
output_hidden = activation_function(output_vector1)

if [Link]:
output_hidden = [Link]((output_hidden,
[[[Link]]]) )

217
 output_vector2 = [Link]([Link],
output_hidden)
output_network = activation_function(output_vector2)

output_errors = target_vector - output_network

# update the weights:
tmp = output_errors * output_network * (1.0 - output_netwo
rk)
tmp = self.learning_rate * [Link](tmp, output_hidden.T)
[Link] += tmp

# calculate hidden errors:

hidden_errors = [Link]([Link].T,
output_errors)
# update the weights:
tmp = hidden_errors * output_hidden * (1.0 - output_hidde
n)
if [Link]:
x = [Link](tmp, input_vector.T)[:-1,:]
else:
x = [Link](tmp, input_vector.T)
[Link] += self.learning_rate * x

def run(self, input_vector):

"""
input_vector can be tuple, list or ndarray
"""

if [Link]:
# adding bias node to the end of the inpuy_vector
input_vector = [Link]((input_vector, [1]) )
input_vector = [Link](input_vector, ndmin=2).T

output_vector = [Link]([Link],
input_vector)
output_vector = activation_function(output_vector)

if [Link]:
output_vector = [Link]( (output_vector,
[[1]]) )

218
 output_vector = [Link]([Link],
output_vector)
output_vector = activation_function(output_vector)
return output_vector

def evaluate(self, data, labels):

corrects, wrongs = 0, 0
for i in range(len(data)):
res = [Link](data[i])
res_max = [Link]()
if res_max == labels[i]:
corrects += 1
else:
wrongs += 1
return corrects, wrongs

ANN = NeuralNetwork(no_of_in_nodes=image_pixels,
no_of_out_nodes=10,
no_of_hidden_nodes=200,
learning_rate=0.1,
bias=None)

for i in range(len(train_imgs)):
[Link](train_imgs[i], train_labels_one_hot[i])
for i in range(20):
res = [Link](test_imgs[i])
print(test_labels[i], [Link](res), [Link](res))

219
[7.] 7 0.9951478957895473
[2.] 2 0.9167137305226186
[1.] 1 0.9930670538508068
[0.] 0 0.9729093609525741
[4.] 4 0.9475097483176407
[1.] 1 0.9919906877733081
[4.] 4 0.9390079959736829
[9.] 9 0.9815469745110644
[5.] 5 0.23871278844097427
[9.] 9 0.9863859218561386
[0.] 0 0.9667234471027278
[6.] 6 0.8856024953669486
[9.] 9 0.9928943830319253
[0.] 0 0.96922568081586
[1.] 1 0.9899747475376088
[5.] 5 0.9595147911735664
[9.] 9 0.9958119066147573
[7.] 7 0.9883146384365381
[3.] 3 0.8706223167904136
[4.] 4 0.9912284156702522

corrects, wrongs = [Link](train_imgs, train_labels)

print("accuracy train: ", corrects / ( corrects + wrongs))
corrects, wrongs = [Link](test_imgs, test_labels)
print("accuracy: test", corrects / ( corrects + wrongs))
accruracy train: 0.9555666666666667
accruracy: test 0.9544

VERSION WITH BIAS AND EPOCHS:

import numpy as np

@[Link]
def sigmoid(x):
return 1 / (1 + np.e ** -x)
activation_function = sigmoid

from [Link] import truncnorm

def truncated_normal(mean=0, sd=1, low=0, upp=10):

return truncnorm((low - mean) / sd,

220
 (upp - mean) / sd,
loc=mean,
scale=sd)

class NeuralNetwork:

def __init__(self,
no_of_in_nodes,
no_of_out_nodes,
no_of_hidden_nodes,
learning_rate,
bias=None
):

self.no_of_in_nodes = no_of_in_nodes
self.no_of_out_nodes = no_of_out_nodes

self.no_of_hidden_nodes = no_of_hidden_nodes

self.learning_rate = learning_rate
[Link] = bias
self.create_weight_matrices()

def create_weight_matrices(self):
"""
A method to initialize the weight matrices
of the neural network with optional
bias nodes"""

bias_node = 1 if [Link] else 0

rad = 1 / [Link](self.no_of_in_nodes + bias_node)

X = truncated_normal(mean=0, sd=1, low=-rad, upp=rad)
[Link] = [Link]((self.no_of_hidden_nodes,
self.no_of_in_nodes + bias_node))

rad = 1 / [Link](self.no_of_hidden_nodes + bias_node)

X = truncated_normal(mean=0,
sd=1,
low=-rad,
upp=rad)
[Link] = [Link]((self.no_of_out_nodes,

221
 self.no_of_hidden_nodes + bias_node))

def train_single(self, input_vector, target_vector):

"""
input_vector and target_vector can be tuple,
list or ndarray
"""

bias_node = 1 if [Link] else 0

if [Link]:
# adding bias node to the end of the inpuy_vector
input_vector = [Link]( (input_vector,
[[Link]]) )

output_vectors = []
input_vector = [Link](input_vector, ndmin=2).T
target_vector = [Link](target_vector, ndmin=2).T

output_vector1 = [Link]([Link],
input_vector)
output_hidden = activation_function(output_vector1)

if [Link]:
output_hidden = [Link]((output_hidden,
[[[Link]]]) )

output_vector2 = [Link]([Link],
output_hidden)
output_network = activation_function(output_vector2)

output_errors = target_vector - output_network

# update the weights:
tmp = output_errors * output_network * (1.0 - output_netwo
rk)
tmp = self.learning_rate * [Link](tmp,
output_hidden.T)
[Link] += tmp

# calculate hidden errors:

hidden_errors = [Link]([Link].T,
output_errors)

222
 # update the weights:
tmp = hidden_errors * output_hidden * (1.0 - output_hidde
n)
if [Link]:
x = [Link](tmp, input_vector.T)[:-1,:]
else:
x = [Link](tmp, input_vector.T)
[Link] += self.learning_rate * x

def train(self, data_array,

labels_one_hot_array,
epochs=1,
intermediate_results=False):
intermediate_weights = []
for epoch in range(epochs):
for i in range(len(data_array)):
self.train_single(data_array[i],
labels_one_hot_array[i])
if intermediate_results:
intermediate_weights.append(([Link](),
[Link]()))
return intermediate_weights

def run(self, input_vector):

# input_vector can be tuple, list or ndarray

if [Link]:
# adding bias node to the end of the inpuy_vector
input_vector = [Link]( (input_vector,
[[Link]]) )
input_vector = [Link](input_vector, ndmin=2).T

output_vector = [Link]([Link],
input_vector)
output_vector = activation_function(output_vector)

if [Link]:
output_vector = [Link]( (output_vector,
[[[Link]]]) )

223
 output_vector = [Link]([Link],
output_vector)
output_vector = activation_function(output_vector)

return output_vector

def evaluate(self, data, labels):

corrects, wrongs = 0, 0
for i in range(len(data)):
res = [Link](data[i])
res_max = [Link]()
if res_max == labels[i]:
corrects += 1
else:
wrongs += 1
return corrects, wrongs

epochs = 12

network = NeuralNetwork(no_of_in_nodes=image_pixels,
no_of_out_nodes=10,
no_of_hidden_nodes=100,
learning_rate=0.1,
bias=None)

weights = [Link](train_imgs,
train_labels_one_hot,
epochs=epochs,
intermediate_results=True)
for epoch in range(epochs):
print("epoch: ", epoch)
[Link] = weights[epoch][0]
[Link] = weights[epoch][1]
corrects, wrongs = [Link](train_imgs,
train_labels)
print("accuracy train: ", corrects / ( corrects + wrong
s))
corrects, wrongs = [Link](test_imgs,
test_labels)
print("accuracy test: ", corrects / ( corrects + wrongs))

224
 epoch: 0
accruracy train: 0.9428166666666666
accruracy test: 0.9415
epoch: 1
accruracy train: 0.9596666666666667
accruracy test: 0.9548
epoch: 2
accruracy train: 0.9673166666666667
accruracy test: 0.9599
epoch: 3
accruracy train: 0.9693
accruracy test: 0.9601
epoch: 4
accruracy train: 0.97195
accruracy test: 0.9631
epoch: 5
accruracy train: 0.9750666666666666
accruracy test: 0.9659
epoch: 6
accruracy train: 0.97705
accruracy test: 0.9662
epoch: 7
accruracy train: 0.9767666666666667
accruracy test: 0.9644
epoch: 8
accruracy train: 0.9765666666666667
accruracy test: 0.9643
epoch: 9
accruracy train: 0.9771
accruracy test: 0.9643
epoch: 10
accruracy train: 0.9780333333333333
accruracy test: 0.9627
epoch: 11
accruracy train: 0.97875
accruracy test: 0.9638
In [ ]:
epochs = 12

with open("nist_tests.csv", "w") as fh_out:

for hidden_nodes in [20, 50, 100, 120, 150]:
for learning_rate in [0.01, 0.05, 0.1, 0.2]:
for bias in [None, 0.5]:
network = NeuralNetwork(no_of_in_nodes=image_pixel

225
s,
no_of_out_nodes=10,
no_of_hidden_nodes=hidden_n
odes,
learning_rate=learning_rat
e,
bias=bias)
weights = [Link](train_imgs,
train_labels_one_hot,
epochs=epochs,
intermediate_results=True)
for epoch in range(epochs):
print("*", end="")
[Link] = weights[epoch][0]
[Link] = weights[epoch][1]
train_corrects, train_wrongs = [Link]
e(train_imgs,

train_labels)

test_corrects, test_wrongs = [Link]

e(test_imgs,

test_labels)
outstr = str(hidden_nodes) + " " + str(learnin
g_rate) + " " + str(bias)
outstr += " " + str(epoch) + " "
outstr += str(train_corrects / (train_correct
s + train_wrongs)) + " "
outstr += str(train_wrongs / (train_corrects
+ train_wrongs)) + " "
outstr += str(test_corrects / (test_corrects
+ test_wrongs)) + " "
outstr += str(test_wrongs / (test_corrects + t
est_wrongs))

fh_out.write(outstr + "\n" )
fh_out.flush()
***************************************************************************

The file nist_tests_20_50_100_120_150.csv contains the results from a run of the previous program.

226
NETWORKS WITH MULTIPLE HIDDEN
LAYERS

We will write a new neural network class, in which we can define an arbitrary number of hidden layers. The
code is also improved, because the weight matrices are now build inside of a loop instead redundant code:

In [ ]:
import numpy as np
from [Link] import expit as activation_function
from [Link] import truncnorm

def truncated_normal(mean=0, sd=1, low=0, upp=10):

return truncnorm((low - mean) / sd,
(upp - mean) / sd,
loc=mean,
scale=sd)

class NeuralNetwork:

def __init__(self,
network_structure, # ie. [input_nodes, hidden1_no
des, ... , hidden_n_nodes, output_nodes]
learning_rate,
bias=None
):

[Link] = network_structure
self.learning_rate = learning_rate
[Link] = bias
self.create_weight_matrices()

def create_weight_matrices(self):

bias_node = 1 if [Link] else 0

self.weights_matrices = []

layer_index = 1
no_of_layers = len([Link])
while layer_index < no_of_layers:
nodes_in = [Link][layer_index-1]

227
 nodes_out = [Link][layer_index]
n = (nodes_in + bias_node) * nodes_out
rad = 1 / [Link](nodes_in)
X = truncated_normal(mean=2,
sd=1,
low=-rad,
upp=rad)
wm = [Link](n).reshape((nodes_out, nodes_in + bias_nod
e))
self.weights_matrices.append(wm)
layer_index += 1

def train(self, input_vector, target_vector):

"""
input_vector and target_vector can be tuple,
list or ndarray
"""

no_of_layers = len([Link])
input_vector = [Link](input_vector, ndmin=2).T
layer_index = 0
# The output/input vectors of the various layers:
res_vectors = [input_vector]
while layer_index < no_of_layers - 1:
in_vector = res_vectors[-1]
if [Link]:
# adding bias node to the end of the 'input'_vecto
r
in_vector = [Link]( (in_vector,
[[[Link]]]) )
res_vectors[-1] = in_vector
x = [Link](self.weights_matrices[layer_index],
in_vector)
out_vector = activation_function(x)
# the output of one layer is the input of the next on
e:
res_vectors.append(out_vector)
layer_index += 1

layer_index = no_of_layers - 1
target_vector = [Link](target_vector, ndmin=2).T
# The input vectors to the various layers
output_errors = target_vector - out_vector

228
 while layer_index > 0:
out_vector = res_vectors[layer_index]
in_vector = res_vectors[layer_index-1]

if [Link] and not layer_index==(no_of_layers-1):

out_vector = out_vector[:-1,:].copy()

tmp = output_errors * out_vector * (1.0 - out_vecto

r)
tmp = [Link](tmp, in_vector.T)

#if [Link]:
# tmp = tmp[:-1,:]

self.weights_matrices[layer_index-1] += [Link]
g_rate * tmp

output_errors = [Link](self.weights_matrices[layer_ind
ex-1].T,
output_errors)
if [Link]:
output_errors = output_errors[:-1,:]
layer_index -= 1

def run(self, input_vector):

# input_vector can be tuple, list or ndarray

no_of_layers = len([Link])
if [Link]:
# adding bias node to the end of the inpuy_vector
input_vector = [Link]( (input_vector,
[[Link]]) )
in_vector = [Link](input_vector, ndmin=2).T

layer_index = 1
# The input vectors to the various layers
while layer_index < no_of_layers:
x = [Link](self.weights_matrices[layer_index-1],
in_vector)
out_vector = activation_function(x)

# input vector for next layer

229
 in_vector = out_vector
if [Link]:
in_vector = [Link]( (in_vector,
[[[Link]]])
)

layer_index += 1

return out_vector

def evaluate(self, data, labels):

corrects, wrongs = 0, 0
for i in range(len(data)):
res = [Link](data[i])
res_max = [Link]()
if res_max == labels[i]:
corrects += 1
else:
wrongs += 1
return corrects, wrongs

In [ ]:
ANN = NeuralNetwork(network_structure=[image_pixels, 50, 50, 10],
learning_rate=0.1,
bias=None)

for i in range(len(train_imgs)):
[Link](train_imgs[i], train_labels_one_hot[i])
In [ ]:
corrects, wrongs = [Link](train_imgs, train_labels)
print("accuracy train: ", corrects / ( corrects + wrongs))
corrects, wrongs = [Link](test_imgs, test_labels)
print("accuracy: test", corrects / ( corrects + wrongs))

230
NETWORKS WITH MULTIPLE HIDDEN
LAYERS AND EPOCHS

In [ ]:
import numpy as np
from [Link] import expit as activation_function
from [Link] import truncnorm

def truncated_normal(mean=0, sd=1, low=0, upp=10):

return truncnorm((low - mean) / sd,
(upp - mean) / sd,
loc=mean,
scale=sd)

class NeuralNetwork:

def __init__(self,
network_structure, # ie. [input_nodes, hidden1_no
des, ... , hidden_n_nodes, output_nodes]
learning_rate,
bias=None
):

[Link] = network_structure
self.learning_rate = learning_rate
[Link] = bias
self.create_weight_matrices()

def create_weight_matrices(self):
X = truncated_normal(mean=2, sd=1, low=-0.5, upp=0.5)

bias_node = 1 if [Link] else 0

self.weights_matrices = []
layer_index = 1
no_of_layers = len([Link])
while layer_index < no_of_layers:
nodes_in = [Link][layer_index-1]
nodes_out = [Link][layer_index]

231
 n = (nodes_in + bias_node) * nodes_out
rad = 1 / [Link](nodes_in)
X = truncated_normal(mean=2, sd=1, low=-rad, upp=rad)
wm = [Link](n).reshape((nodes_out, nodes_in + bias_nod
e))
self.weights_matrices.append(wm)
layer_index += 1

def train_single(self, input_vector, target_vector):

# input_vector and target_vector can be tuple, list or nda
rray

no_of_layers = len([Link])
input_vector = [Link](input_vector, ndmin=2).T

layer_index = 0
# The output/input vectors of the various layers:
res_vectors = [input_vector]
while layer_index < no_of_layers - 1:
in_vector = res_vectors[-1]
if [Link]:
# adding bias node to the end of the 'input'_vecto
r
in_vector = [Link]( (in_vector,
[[[Link]]]) )
res_vectors[-1] = in_vector
x = [Link](self.weights_matrices[layer_index], in_vect
or)
out_vector = activation_function(x)
res_vectors.append(out_vector)
layer_index += 1

layer_index = no_of_layers - 1
target_vector = [Link](target_vector, ndmin=2).T
# The input vectors to the various layers
output_errors = target_vector - out_vector
while layer_index > 0:
out_vector = res_vectors[layer_index]
in_vector = res_vectors[layer_index-1]

if [Link] and not layer_index==(no_of_layers-1):

out_vector = out_vector[:-1,:].copy()

232
 tmp = output_errors * out_vector * (1.0 - out_vecto
r)
tmp = [Link](tmp, in_vector.T)

#if [Link]:
# tmp = tmp[:-1,:]

self.weights_matrices[layer_index-1] += [Link]
g_rate * tmp

output_errors = [Link](self.weights_matrices[layer_ind
ex-1].T,
output_errors)
if [Link]:
output_errors = output_errors[:-1,:]
layer_index -= 1

def train(self, data_array,

labels_one_hot_array,
epochs=1,
intermediate_results=False):
intermediate_weights = []
for epoch in range(epochs):
for i in range(len(data_array)):
self.train_single(data_array[i], labels_one_hot_ar
ray[i])
if intermediate_results:
intermediate_weights.append(([Link](),
[Link]()))
return intermediate_weights

def run(self, input_vector):

# input_vector can be tuple, list or ndarray

no_of_layers = len([Link])
if [Link]:
# adding bias node to the end of the inpuy_vector
input_vector = [Link]( (input_vector, [[Link]
as]) )

233
 in_vector = [Link](input_vector, ndmin=2).T

layer_index = 1
# The input vectors to the various layers
while layer_index < no_of_layers:
x = [Link](self.weights_matrices[layer_index-1],
in_vector)
out_vector = activation_function(x)

# input vector for next layer

in_vector = out_vector
if [Link]:
in_vector = [Link]( (in_vector,
[[[Link]]])
)

layer_index += 1

return out_vector

def evaluate(self, data, labels):

corrects, wrongs = 0, 0
for i in range(len(data)):
res = [Link](data[i])
res_max = [Link]()
if res_max == labels[i]:
corrects += 1
else:
wrongs += 1
return corrects, wrongs

In [ ]:
epochs = 3

ANN = NeuralNetwork(network_structure=[image_pixels, 80, 80, 10],

learning_rate=0.01,
bias=None)

[Link](train_imgs, train_labels_one_hot, epochs=epochs)

In [ ]:

234
corrects, wrongs = [Link](train_imgs, train_labels)
print("accuracy train: ", corrects / ( corrects + wrongs))
corrects, wrongs = [Link](test_imgs, test_labels)
print("accuracy: test", corrects / ( corrects + wrongs))

FOOTNOTES

1
Wan, Li; Matthew Zeiler; Sixin Zhang; Yann LeCun; Rob Fergus (2013). Regularization of Neural Network
using DropConnect. International Conference on Machine Learning(ICML).

235
DROPOUT NEURAL NETWORKS

INTRODUCTION
The term "dropout" is used for a technique which
drops out some nodes of the network. Dropping out
can be seen as temporarily deactivating or ignoring
neurons of the network. This technique is applied in
the training phase to reduce overfitting effects.
Overfitting is an error which occurs when a network
is too closely fit to a limited set of input samples.

The basic idea behind dropout neural networks is to

dropout nodes so that the network can concentrate on
other features. Think about it like this. You watch
lots of films from your favourite actor. At some point
you listen to the radio and here somebody in an
interview. You don't recognize your favourite actor,
because you have seen only movies and your are a
visual type. Now, imagine that you can only listen to
the audio tracks of the films. In this case you will
have to learn to differentiate the voices of the
actresses and actors. So by dropping out the visual part you are forced tp focus on the sound features!

This technique has been first proposed in a paper "Dropout: A Simple Way to Prevent Neural Networks from
Overfitting" by Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever and Ruslan
Salakhutdinov in 2014

We will implement in our tutorial on machine learning in Python a Python class which is capable of dropout.

MODIFYING THE WEIGHT ARRAYS

If we deactivate a node, we have to modify the weight arrays accordingly. To demonstrate how this can be
accomplished, we will use a network with three input nodes, four hidden and two output nodes:

236
At first, we will have a look at the weight array between the input and the hidden layer. We called this array
'wih' (weights between input and hidden layer).

Let's deactivate (drop out) the node i 2. We can see in the following diagram what's happening:

237
This means that we have to take out every second product of the summation, which means that we have to
delete the whole second column of the matrix. The second element from the input vector has to be deleted as
well.

Now we will examine what happens if we take out a hidden node. We take out the first hidden node, i.e. h 1.

In this case, we can remove the complete first line of our weight matrix:

Taking out a hidden node affects the next weight matrix as well. Let's have a look at what is happening in the
network graph:

238
It is easy to see that the first column of the who weight matrix has to be removed again:

So far we have arbitrarily chosen one node to deactivate. The dropout approach means that we randomly
choose a certain number of nodes from the input and the hidden layers, which remain active and turn off the
other nodes of these layers. After this we can train a part of our learn set with this network. The next step
consists in activating all the nodes again and randomly chose other nodes. It is also possible to train the whole
training set with the randomly created dropout networks.

We present three possible randomly chosen dropout networks in the following three diagrams:

239
Now it is time to think about a possible Python implementation.

We will start with the weight matrix between input and hidden layer. We will randomly create a weight matrix
for 10 input nodes and 5 hidden nodes. We fill our matrix with random numbers between -10 and 10, which
are not proper weight values, but this way we can see better what is going on:

import numpy as np
import random

input_nodes = 10
hidden_nodes = 5
output_nodes = 7

wih = [Link](-10, 10, (hidden_nodes, input_nodes))

wih

240
Output: array([[ -6, -8, -3, -7, 2, -9, -3, -5, -6, 4],
[ 5, 3, 7, -4, 4, 8, -2, -4, 7, 7],
[ 9, -7, 4, 0, 4, 0, -3, -6, -2, 7],
[ -8, -9, -4, -5, -9, 8, -8, -8, -2, -3],
[ 3, -10, 0, -3, 4, 0, 0, 2, -7, -9]])

We will choose now the active nodes for the input layer. We calculate random indices for the active nodes:

active_input_percentage = 0.7
active_input_nodes = int(input_nodes * active_input_percentage)
active_input_indices = sorted([Link](range(0, input_node
s),
active_input_nodes))
active_input_indices
Output: [0, 1, 2, 5, 7, 8, 9]

We learned above that we have to remove the column j, if the node i j is removed. We can easily accomplish
this for all deactived nodes by using the slicing operator with the active nodes:

wih_old = [Link]()
wih = wih[:, active_input_indices]
wih
Output: array([[ -6, -8, -3, -9, -5, -6, 4],
[ 5, 3, 7, 8, -4, 7, 7],
[ 9, -7, 4, 0, -6, -2, 7],
[ -8, -9, -4, 8, -8, -2, -3],
[ 3, -10, 0, 0, 2, -7, -9]])

As we have mentioned before, we will have to modify both the 'wih' and the 'who' matrix:

who = [Link](-10, 10, (output_nodes, hidden_nodes))

print(who)
active_hidden_percentage = 0.7
active_hidden_nodes = int(hidden_nodes * active_hidden_percentage)
active_hidden_indices = sorted([Link](range(0, hidden_node
s),
active_hidden_nodes))
print(active_hidden_indices)

who_old = [Link]()
who = who[:, active_hidden_indices]

241
print(who)
[[ 3 6 -3 -9 4]
[-10 1 2 5 7]
[ -8 1 -3 6 3]
[ -3 -3 6 -5 -3]
[ -4 -9 8 -3 5]
[ 8 4 -8 2 7]
[ -2 2 3 -8 -5]]
[0, 2, 3]
[[ 3 -3 -9]
[-10 2 5]
[ -8 -3 6]
[ -3 6 -5]
[ -4 8 -3]
[ 8 -8 2]
[ -2 3 -8]]

We have to change wih accordingly:

wih = wih[active_hidden_indices]
wih
Output: array([[-6, -8, -3, -9, -5, -6, 4],
[ 9, -7, 4, 0, -6, -2, 7],
[-8, -9, -4, 8, -8, -2, -3]])

The following Python code summarizes the sniplets from above:

import numpy as np
import random

input_nodes = 10
hidden_nodes = 5
output_nodes = 7

wih = [Link](-10, 10, (hidden_nodes, input_nodes))

print("wih: \n", wih)
who = [Link](-10, 10, (output_nodes, hidden_nodes))
print("who:\n", who)

active_input_percentage = 0.7
active_hidden_percentage = 0.7

active_input_nodes = int(input_nodes * active_input_percentage)

242
active_input_indices = sorted([Link](range(0, input_node
s),
active_input_nodes))
print("\nactive input indices: ", active_input_indices)
active_hidden_nodes = int(hidden_nodes * active_hidden_percentage)
active_hidden_indices = sorted([Link](range(0, hidden_node
s),
active_hidden_nodes))
print("active hidden indices: ", active_hidden_indices)

wih_old = [Link]()
wih = wih[:, active_input_indices]
print("\nwih after deactivating input nodes:\n", wih)
wih = wih[active_hidden_indices]
print("\nwih after deactivating hidden nodes:\n", wih)

who_old = [Link]()
who = who[:, active_hidden_indices]
print("\nwih after deactivating hidden nodes:\n", who)

243
wih:
[[ -4 9 3 5 -9 5 -3 0 9 1]
[ 4 7 -7 3 -4 7 4 -5 6 2]
[ 5 8 1 -10 -8 -6 7 -4 -6 8]
[ 6 -3 7 4 -7 -4 0 8 9 1]
[ 6 -1 4 -3 5 -5 -5 5 4 -7]]
who:
[[ -6 2 -2 4 0]
[ -5 -3 3 -4 -10]
[ 4 6 -7 -7 -1]
[ -4 -1 -10 0 -8]
[ 8 -2 9 -8 -9]
[ -6 0 -2 1 -8]
[ 1 -4 -2 -6 -5]]

active input indices: [1, 3, 4, 5, 7, 8, 9]

active hidden indices: [0, 1, 2]

wih after deactivating input nodes:

[[ 9 5 -9 5 0 9 1]
[ 7 3 -4 7 -5 6 2]
[ 8 -10 -8 -6 -4 -6 8]
[ -3 4 -7 -4 8 9 1]
[ -1 -3 5 -5 5 4 -7]]

wih after deactivating hidden nodes:

[[ 9 5 -9 5 0 9 1]
[ 7 3 -4 7 -5 6 2]
[ 8 -10 -8 -6 -4 -6 8]]

wih after deactivating hidden nodes:

[[ -6 2 -2]
[ -5 -3 3]
[ 4 6 -7]
[ -4 -1 -10]
[ 8 -2 9]
[ -6 0 -2]
[ 1 -4 -2]]

import numpy as np
import random
from [Link] import expit as activation_function
from [Link] import truncnorm

def truncated_normal(mean=0, sd=1, low=0, upp=10):

return truncnorm(

244
 (low - mean) / sd, (upp - mean) / sd, loc=mean, scale=sd)

class NeuralNetwork:

def __init__(self,
no_of_in_nodes,
no_of_out_nodes,
no_of_hidden_nodes,
learning_rate,
bias=None
):

self.no_of_in_nodes = no_of_in_nodes
self.no_of_out_nodes = no_of_out_nodes
self.no_of_hidden_nodes = no_of_hidden_nodes
self.learning_rate = learning_rate
[Link] = bias
self.create_weight_matrices()

def create_weight_matrices(self):
X = truncated_normal(mean=2, sd=1, low=-0.5, upp=0.5)

bias_node = 1 if [Link] else 0

n = (self.no_of_in_nodes + bias_node) * self.no_of_hidde

n_nodes
X = truncated_normal(mean=2, sd=1, low=-0.5, upp=0.5)
[Link] = [Link](n).reshape((self.no_of_hidden_nodes,
self.no_of_in_n
odes + bias_node))

n = (self.no_of_hidden_nodes + bias_node) * self.no_of_ou

t_nodes
X = truncated_normal(mean=2, sd=1, low=-0.5, upp=0.5)
[Link] = [Link](n).reshape((self.no_of_out_nodes,
(self.no_of_hi
dden_nodes + bias_node)))

def dropout_weight_matrices(self,
active_input_percentage=0.70,
active_hidden_percentage=0.70):
# restore wih array, if it had been used for dropout
self.wih_orig = [Link]()
self.no_of_in_nodes_orig = self.no_of_in_nodes

245
 self.no_of_hidden_nodes_orig = self.no_of_hidden_nodes
self.who_orig = [Link]()

active_input_nodes = int(self.no_of_in_nodes * active_inpu

t_percentage)
active_input_indices = sorted([Link](range(0, sel
f.no_of_in_nodes),
active_input_nodes))
active_hidden_nodes = int(self.no_of_hidden_nodes * activ
e_hidden_percentage)
active_hidden_indices = sorted([Link](range(0, sel
f.no_of_hidden_nodes),
active_hidden_nodes))

[Link] = [Link][:, active_input_indices][active_hidde

n_indices]
[Link] = [Link][:, active_hidden_indices]

self.no_of_hidden_nodes = active_hidden_nodes
self.no_of_in_nodes = active_input_nodes
return active_input_indices, active_hidden_indices

def weight_matrices_reset(self,
active_input_indices,
active_hidden_indices):

"""
[Link] and [Link] contain the newly adapted values fro
m the active nodes.
We have to reconstruct the original weight matrices by ass
igning the new values
from the active nodes
"""

temp = self.wih_orig.copy()[:,active_input_indices]
temp[active_hidden_indices] = [Link]
self.wih_orig[:, active_input_indices] = temp
[Link] = self.wih_orig.copy()

self.who_orig[:, active_hidden_indices] = [Link]

[Link] = self.who_orig.copy()
self.no_of_in_nodes = self.no_of_in_nodes_orig
self.no_of_hidden_nodes = self.no_of_hidden_nodes_orig

246
 def train_single(self, input_vector, target_vector):
"""
input_vector and target_vector can be tuple, list or ndarr
ay
"""

if [Link]:
# adding bias node to the end of the input_vector
input_vector = [Link]( (input_vector, [[Link]
as]) )

input_vector = [Link](input_vector, ndmin=2).T

target_vector = [Link](target_vector, ndmin=2).T

output_vector1 = [Link]([Link], input_vector)

output_vector_hidden = activation_function(output_vector1)

if [Link]:
output_vector_hidden = [Link]( (output_vecto
r_hidden, [[[Link]]]) )

output_vector2 = [Link]([Link], output_vector_hidden)

output_vector_network = activation_function(output_vector
2)

output_errors = target_vector - output_vector_network

# update the weights:
tmp = output_errors * output_vector_network * (1.0 - outpu
t_vector_network)
tmp = self.learning_rate * [Link](tmp, output_vector_hidd
en.T)
[Link] += tmp

# calculate hidden errors:

hidden_errors = [Link]([Link].T, output_errors)
# update the weights:
tmp = hidden_errors * output_vector_hidden * (1.0 - outpu
t_vector_hidden)
if [Link]:
x = [Link](tmp, input_vector.T)[:-1,:]
else:
x = [Link](tmp, input_vector.T)

247
 [Link] += self.learning_rate * x

def train(self, data_array,

labels_one_hot_array,
epochs=1,
active_input_percentage=0.70,
active_hidden_percentage=0.70,
no_of_dropout_tests = 10):

partition_length = int(len(data_array) / no_of_dropout_tes

ts)

for epoch in range(epochs):

print("epoch: ", epoch)
for start in range(0, len(data_array), partition_lengt
h):
active_in_indices, active_hidden_indices = \
self.dropout_weight_matrices(active_inp
ut_percentage,
active_hid
den_percentage)
for i in range(start, start + partition_length):
self.train_single(data_array[i][active_in_indi
ces],
labels_one_hot_array[i])

self.weight_matrices_reset(active_in_indices, acti
ve_hidden_indices)

def confusion_matrix(self, data_array, labels):

cm = {}
for i in range(len(data_array)):
res = [Link](data_array[i])
res_max = [Link]()
target = labels[i][0]
if (target, res_max) in cm:
cm[(target, res_max)] += 1
else:
cm[(target, res_max)] = 1
return cm

248
 def run(self, input_vector):
# input_vector can be tuple, list or ndarray

if [Link]:
# adding bias node to the end of the input_vector
input_vector = [Link]( (input_vector, [[Link]
as]) )
input_vector = [Link](input_vector, ndmin=2).T

output_vector = [Link]([Link], input_vector)

output_vector = activation_function(output_vector)

if [Link]:
output_vector = [Link]( (output_vector, [[sel
[Link]]]) )

output_vector = [Link]([Link], output_vector)

output_vector = activation_function(output_vector)

return output_vector

def evaluate(self, data, labels):

corrects, wrongs = 0, 0
for i in range(len(data)):
res = [Link](data[i])
res_max = [Link]()
if res_max == labels[i]:
corrects += 1
else:
wrongs += 1
return corrects, wrongs

import pickle

with open("data/mnist/pickled_mnist.pkl", "br") as fh:

data = [Link](fh)

train_imgs = data[0]
test_imgs = data[1]
train_labels = data[2]
test_labels = data[3]

249
train_labels_one_hot = data[4]
test_labels_one_hot = data[5]

image_size = 28 # width and length

no_of_different_labels = 10 # i.e. 0, 1, 2, 3, ..., 9
image_pixels = image_size * image_size

parts = 10
partition_length = int(len(train_imgs) / parts)
print(partition_length)

start = 0
for start in range(0, len(train_imgs), partition_length):
print(start, start + partition_length)
6000
0 6000
6000 12000
12000 18000
18000 24000
24000 30000
30000 36000
36000 42000
42000 48000
48000 54000
54000 60000

epochs = 3

simple_network = NeuralNetwork(no_of_in_nodes = image_pixels,

no_of_out_nodes = 10,
no_of_hidden_nodes = 100,
learning_rate = 0.1)

simple_network.train(train_imgs,
train_labels_one_hot,
active_input_percentage=1,
active_hidden_percentage=1,
no_of_dropout_tests = 100,
epochs=epochs)
epoch: 0
epoch: 1
epoch: 2

250
corrects, wrongs = simple_network.evaluate(train_imgs, train_label
s)
print("accuracy train: ", corrects / ( corrects + wrongs))
corrects, wrongs = simple_network.evaluate(test_imgs, test_labels)
print("accuracy: test", corrects / ( corrects + wrongs))
accruracy train: 0.9317833333333333
accruracy: test 0.9296

251