TensorFlow Developer Certificate

Preparation

Henokh Lugo Hariyanto

This work is licensed under Creative Commons Attribution-NonCommercial 4.0 International

Version 1.01 (August 2024)
[Link].h@[Link]
1
Preface
With the increase of technology in computing power, AI has its position in our society as a common
and ubiquitous part of our techniques dealing with data. The huge number of datasets, especially
images, needs more sophisticated tools to process, understand, and interpret them. One of the
tools to do that is machine learning.

The high culminating point when machines/algorithms can beat humans at image classification is in
the competition “ImageNet Large Scale Visual Recognition Challenge” (ILSVRC) 2015. The team
from Microsoft Research created a ResNet (Residual Neural Networks) architecture that achieves
3.57% error on the ImageNet test set compared to manually labeling the test set by two expert
annotators only achieves 5.1%. From this breakthrough, many architectures are invented each year
to process images, audio, video, or other forms of datasets.

This module is a short and steep introduction on how to use TensorFlow packages to solve
regression, classification, and prediction problems using a real world dataset. Most of the content in
this article followed the flow from Coursera’ course: [Link] TensorFlow Developer, with
many modifications to simplify Python's block of functions. We also add more succinct explanations
on the architecture but not implement them manually. We hope that the article instead of providing a
basic reference to take TensorFlow Developer Certification, it also gives the readers a glimpse of
deep learning.

In chapter 1, we start by installing PyCharm as the prerequisite to take TensorFlow Developer

Certification. We don’t explain Python in this chapter, only showed how to use NumPy and
Matplotlib. In chapter 2, we start to show the TensorFlow package in practice, using the Fashion
MNIST dataset and Emoji Computer Generated dataset, and some tastes of Convolution Neural
Networks (CNN). We fully explain CNN in chapter 3.

In chapter 3, we explore how to do image augmentation and dropout layers with datasets: Cats and
Dogs, Hand Sign Language, and Rock Paper Scissor game images. In chapter 4, we introduce
Recurrent Neural Networks for a dataset that has a sequential pattern. Then we jump to bi-
directional Long Short Term Memory layers. We apply this architecture to the datasets: News
Headlines from many websites, BBC News with categories, IMDb reviews. The last section of this
chapter, we will create a poem generator with an English or Bahasa Indonesia training dataset. The
last chapter is an exploration of the monthly mean total sunspots number dataset with two kinds of
approaches to do prediction: a simple deep neural network and bi-directional LSTM.

In addition, I want to acknowledge the many helpful contributions of the Deep Nusa AI members R.
Arif Firdaus Lazuardi, Akhmad Muzanni Safi'i, Anugrah Arif. I would like to thank Doni Rubiagatra
for the second instructor. I wish to express my gratitude to the PENS - PTVP ML TensorFlow
committee who held the workshop and all the participants. And, finally, infinite thanks to Yohana
Silviana and my parents who always support me in this difficult time.

Henokh Lugo Hariyanto

Bandung, August 2021

2
3
Table of Contents
Preface 2

Table of Contents 4

Python introduction with PyCharm 6

PyCharm installation 6
Installation on Windows 6
Installation on macOs 7
Installation on Linux 7
Fast Intro using PyCharm 8
Fast Introduction to NumPy and Matplotlib 11
Basics of NumPy 11
Shape manipulation of NumPy arrays 16
Copies of NumPy arrays 18
Broadcasting rule of NumPy arrays operations 19
Advanced Indexing of NumPy arrays 21
A formal way to use Matplotlib 23

Introduction to TensorFlow 25
“Hello World!” version of TensorFlow 25
Classifying Fashion MNIST 27
Classifying Fashion with CNN 31
Classifying Emotion with CNN 34

Convolutional Neural Network in TensorFlow 38

Classifying Cats and Dogs 38
Improving Cats and Dogs Classification 43
Transfer Learning based on VGG Net 47
Classifying Images of Sign Languages 50
Classifying Images of Rock, Paper, Scissors Game 55

Natural Language Processing in TensorFlow 58

Detecting sarcasm in News Headlines with LSTM and CNN 61
Exploring BBC News Data 66
Classifying BBC News into topics 68
Classifying IMDb Reviews Data 71
Poem Generation with Bi-Directional LSTM 74

Sequence, Time Series and Prediction 77

Searching a periodic pattern in the sunspot dataset 77
Using DNN as a model predictor 82
Using bi-directional LSTM as a model predictor 90

Preparation and Taking the TensorFlow Developer Certification Exam 97

Preparation 97
Purchasing the certification ticket 97
Taking the exam 97

Useful troubleshooting and Python’s commands 99

References 101

4
5
6
1. Python introduction with PyCharm

a. PyCharm installation

This installation will also include all Python 3.8+ and standard packages. In this tutorial, we choose
standalone installation. For recommended installation using JetBrains Toolbox App, see the
complete description in [Link]
Please note that the PyCharm version that you should install is build 211.* or older to support
TensorFlow Developer Certificate plugin version 0.1.12.

i. Installation on Windows
Download the installer .exe in [Link]
Choose the Community version. The file size is around 366 MB.

Then the next window will appear. Click “Download and verify the file’s SHA-255 checksum”. Keep
the number to compare with the next procedure

Verify your downloaded file of pycharm by running in command prompt

$ Certutil -hashfile <path to file> SHA256

After you have verified the installer, you can proceed to run the installer. Mind the following options
in the installation wizard
● 64-bit launcher: Adds a launching icon to the Desktop.
● Open Folder as Project: Adds an option to the folder context menu that will allow opening the
selected directory as a PyCharm project.
● .py: Establishes an association with Python files to open them in PyCharm.

7
● Add launchers dir to the PATH: Allows running this PyCharm instance from the Console
without specifying the path to it.

ii. Installation on macOs

Download the disk image in [Link] Choose
the Community version. The file size is around 450 MB.

Then the next window will appear. Click “Download and verify the file’s SHA-255 checksum”. Keep
the number to compare with the next procedure

Verify your downloaded file of pycharm by running in command prompt

$ shasum -a 256 <path to file>

After you have verified the installer, you can mount the image and drag the PyCharm app to the
Applications folder.

iii. Installation on Linux

We will use the installer through snap packages. Open the terminal and run the following command:

$ sudo snap install pycharm-community --classic.

For specific version, sometimes snap would have updated automatically, we can use the following
command

$ sudo snap install pycharm-community -channel=2021.1/stable --classic.

8
b. Fast Intro using PyCharm
When we open PyCharm for the first time, we will have a window like below

You can select “New Project” and then “New Project” window will appear

Several things that we need to set up

● Location: The location of folder for the project that we want to create
● New environment using: There are three available environments that we can use; Virtualenv,
Pipenv, and Conda. My personal option is to use the Conda environment. (For TensorFlow
Developer Certification we MUST use Virtualenv)
● Under the New environment section, we can also set the path for the environment in Location.

9
● Base interpreter: The path for your Python interpreter. For our purpose of TensorFlow Developer
Certificate, we need to use Python 3.8

Click “Create” after finishing all the above things. Then the main window of PyCharm IDE will show
up.

Then go to File >> Settings….

On the left box of the “Settings” window, expand Project:<your_project_name> and select Python
Interpreter. On the right box, click “+” to add a new Python’s packages.

Then the Available Packages window will appear. In the search toolbars search the following
packages:
● tensorflow==2.5.0
● tensorflow-datasets==4.3.0
● pillow==8.2.0

10
 ● pandas==1.2.4
● numpy==1.19.5
● scipy=1.7.0
● urllib3==1.26.6

The below image shows the “Available Packages” window for “numpy”. To select the package’s
version, you can set “Specify version”

For full description see

[Link]

After we have installed all the required packages, we can create a new Python file under the
directory of the new project that we have created by right clicking the project’s name and selecting
“New” >> “Python File”. Put a meaningful name to that Python file that represents the content of the
code.

Then, we are ready to start for the next step.

11
c. Fast Introduction to NumPy and Matplotlib
This is a short introduction on how to use Python with NumPy and Matplotlib. For complete
description go to official documentation of Python, NumPy, and Matplotlib.

i. Basics of NumPy
NumPy is one of the most useful Python’s packages to handle arrays in any dimension and is full of
methods to manipulate them. The image below shows the fundamental term that we have to
understand when we use NumPy

In NumPy, the nth-dimension is called nth-axis. Numpy’s class in Python is called ndarray.
Several important attributes in ndarray are
● [Link]: store the number of axes
● [Link]: store size of the array in each dimension
● [Link]: store total number of elements of the array
● [Link]: store the data type of the elements.

An example

In [1]: import numpy as np

In [2]: a = [Link](15).reshape(3, 5)

In [3]: a
Out[3]:
array([[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14]])

In [4]: [Link]
Out[4]: (3, 5)

In [5]: [Link]
Out[5]: 2

In [6]: [Link]
Out[6]: dtype('int64')

In [7]: [Link]
Out[7]: 15

In [8]: type(a)
Out[8]: [Link]

12
 In [9]: b = [Link]([6, 7, 8])

In [10]: b
Out[10]: array([6, 7, 8])

In [11]: type(b)
Out[11]: [Link]

Array creation

There are several ways to create arrays:

● [Link](): create an array by providing it with a list or tuple.
● [Link](), [Link](), or [Link](): Initialize an array when we only know the size of
the array before computation
● [Link](): create a one-dimensional array if we know the lower and upper values and also
the increments.
● [Link](): the most useful function to create N elements of a one-dimensional array if
we know the lower and upper values of the array.

In [1]: import numpy as np

In [4]: b = [Link]([1.2, 4.3, 9.1, 10.])

In [5]: [Link]
Out[5]: dtype('float64')

In [6]: c = [Link]([[1, 4], [9, 6]], dtype=np.float64)

In [7]: c
Out[7]:
array([[1., 4.],
[9., 6.]])

In [8]: [Link]((3, 5))

Out[8]:
array([[0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0.]])

In [9]: [Link]((2, 2, 5), dtype=np.int64)

Out[9]:
array([[[1, 1, 1, 1, 1],
[1, 1, 1, 1, 1]],

[[1, 1, 1, 1, 1],
[1, 1, 1, 1, 1]]])

In [10]: [Link]((2, 4))

Out[10]:
array([[4.66086321e-310, 0.00000000e+000, 4.66086282e-310,
3.80323684e+177],
[6.93809786e-310, 4.66086282e-310, 2.03508930e+281,
6.93810211e-310]])

13
 In [11]: [Link](2, 9, 2)
Out[11]: array([2, 4, 6, 8])

In [12]: [Link](0, 5, 11)

Out[12]: array([0. , 0.5, 1. , 1.5, 2. , 2.5, 3. , 3.5, 4. , 4.5, 5. ])

Reshaping array

When we train a machine learning model with different sizes of input, we have to make sure that the
shape of the input array matches the shape of the input array of the model. NumPy provides us with
a handy method to reshape it.
We can invoke the method by .reshape(axis1, axis2, ...) or [Link](arr, axis1,
axis2, ...)

In [1]: import numpy as np

# generate an integer array with 10 elements start from 0

In [2]: arr1D = [Link](10)

In [3]: print(arr1D)
[0 1 2 3 4 5 6 7 8 9]

In [4]: arr2D = [Link](20).reshape(2, 10)

In [5]: print(arr2D)
[[ 0 1 2 3 4 5 6 7 8 9]
[10 11 12 13 14 15 16 17 18 19]]

In [6]: arr3D = [Link](40).reshape(2, 2, 10)

In [7]: print(arr3D)
[[[ 0 1 2 3 4 5 6 7 8 9]
[10 11 12 13 14 15 16 17 18 19]]

[[20 21 22 23 24 25 26 27 28 29]
[30 31 32 33 34 35 36 37 38 39]]]

# axis2 inferred automatically from axis1

In [8]: arr2D = [Link](arr1D, (2, -1))

In [9]: print(arr2D)
[[0 1 2 3 4]
[5 6 7 8 9]]

Basic operations

All arithmetic operation applied element-wise means that it operates for each element of the array.

In [23]: a = [Link]([20, 30, 40, 50])

14
 In [24]: b = [Link](4)

In [25]: b
Out[25]: array([0, 1, 2, 3])

In [26]: c = a - b

In [27]: c
Out[27]: array([20, 29, 38, 47])

In [28]: b**2
Out[28]: array([0, 1, 4, 9])

In [29]: 10 * [Link](a)
Out[29]: array([ 9.12945251, -9.88031624, 7.4511316 , -2.62374854])

In [30]: a < 35
Out[30]: array([ True, True, False, False])

Python multiplication operator “*” will be interpreted as the operation of element wise when we
multiply two matrices. To do the usual matrix product, we can use the “@” operator or method
“.dot()”.

In [16]: A = [Link]([[1, 1], [0, 1]])

In [18]: B = [Link]([[2, 0], [3, 4]])

# element-wise product
In [20]: A * B
Out[20]:
array([[2, 0],
[0, 4]])

# matrix product
In [21]: A @ B
Out[21]:
array([[5, 4],
[3, 4]])

# matrix product using .dot() method

In [22]: [Link](B)
Out[22]:
array([[5, 4],
[3, 4]])

Indexing, slicing and iterating

Indexing in NumPy means we select a specific element in an array based on its index.
Slicing is a general way of indexing on how we collectively select more than one element with
several indices. We can put complicated rules to slice an array that we will discuss in the later
section.

15
Iterating is a short-hand notation to iterate over all element of an array when we use an array with
for-loop

In [31]: a = [Link](10, dtype=int)**3

In [32]: a
Out[32]: array([ 0, 1, 8, 27, 64, 125, 216, 343, 512, 729])

# get the element at index = 2

In [33]: a[2]
Out[33]: 8

# get the elements from index 2 to (5 - 1)

In [34]: a[2:5]
Out[34]: array([ 8, 27, 64])

# get the elements from index 0 to (6 - 1) with increment of 2

In [35]: a[:6:2]
Out[35]: array([ 0, 8, 64])

# a nice way to reverse an array

In [36]: a[::-1]
Out[36]: array([729, 512, 343, 216, 125, 64, 27, 8, 1, 0])

In [38]: for element in a:

...: print(element / 2.0)
...:
0.0
0.5
4.0
13.5
32.0
62.5
108.0
171.5
256.0
364.5

ii. Shape manipulation of NumPy arrays

In the Basics section, we have introduced a simple way to manipulate the shape of an array by
using .reshape(). The other useful NumPy functions to manipulate an array are:
● [Link](): This function is similar to [Link]() but it will change the array in-place
instead of returning a modified array.
● [Link](): This will stack two arrays along the first axis (vertical)
● [Link](): This will stack two arrays along the second axis (horizontal)
● np.row_stack(): Similar to [Link]()
● np.column_stack(): Similar to [Link]() only for 2D arrays.
● [Link]: Create a new axis.
● [Link](): Split 2D arrays along first axis (vertical)
● [Link](): Split 2D arrays along second axis (horizontal)

16
In [39]: import numpy as np

# create a generator object

In [40]: rng = [Link].default_rng()

In [41]: a = [Link](10 * [Link]((3, 4)))

In [42]: a
Out[42]:
array([[6., 1., 6., 3.],
[4., 4., 0., 1.],
[4., 5., 0., 9.]])

In [43]: [Link]
Out[43]: (3, 4)

In [44]: [Link]((2, 6))

In [45]: a
Out[45]:
array([[6., 1., 6., 3., 4., 4.],
[0., 1., 4., 5., 0., 9.]])

In [46]: a = [Link](10 * [Link]((2, 2)))

In [47]: a
Out[47]:
array([[9., 0.],
[0., 2.]])

In [48]: b = [Link](10 * [Link]((2, 2)))

In [49]: b
Out[49]:
array([[8., 5.],
[7., 4.]])

In [50]: [Link]((a, b))

Out[50]:
array([[9., 0.],
[0., 2.],
[8., 5.],
[7., 4.]])

In [51]: [Link]((a, b))

Out[51]:
array([[9., 0., 8., 5.],
[0., 2., 7., 4.]])

In [52]: np.row_stack((a, b))

Out[52]:
array([[9., 0.],
[0., 2.],
[8., 5.],
[7., 4.]])

In [53]: np.column_stack((a, b))

Out[53]:
array([[9., 0., 8., 5.],

17
 [0., 2., 7., 4.]])

In [54]: a = [Link]([3., 5.])

In [55]: a
Out[55]: array([3., 5.])

In [57]: [Link]
Out[57]: (2,)

# create a new axis for 1D array

In [59]: a[:, [Link]].shape
Out[59]: (2, 1)

In [60]: a = [Link](10 * [Link]((2, 12)))

In [61]: a
Out[61]:
array([[2., 5., 9., 4., 0., 4., 9., 6., 8., 8., 3., 7.],
[7., 5., 9., 9., 1., 5., 4., 3., 8., 1., 1., 8.]])

In [63]: [Link](a, 3)
Out[63]:
[array([[2., 5., 9., 4.],
[7., 5., 9., 9.]]),
array([[0., 4., 9., 6.],
[1., 5., 4., 3.]]),
array([[8., 8., 3., 7.],
[8., 1., 1., 8.]])]

# horizontal split before third column and after (3-1)-th column

In [65]: [Link](a, (3, 3))
Out[65]:
[array([[2., 5., 9.],
[7., 5., 9.]]),
array([], shape=(2, 0), dtype=float64),
array([[4., 0., 4., 9., 6., 8., 8., 3., 7.],
[9., 1., 5., 4., 3., 8., 1., 1., 8.]])]

In [66]: a = [Link](-1, 2)

In [67]: a
Out[67]:
array([[2., 5.],
[9., 4.],
[0., 4.],
[9., 6.],
[8., 8.],
[3., 7.],
[7., 5.],
[9., 9.],
[1., 5.],
[4., 3.],
[8., 1.],
[1., 8.]])

In [68]: [Link](a, 3)
Out[68]:
[array([[2., 5.],

18
 [9., 4.],
[0., 4.],
[9., 6.]]),
array([[8., 8.],
[3., 7.],
[7., 5.],
[9., 9.]]),
array([[1., 5.],
[4., 3.],
[8., 1.],
[1., 8.]])]

# vertical split before third row and after (3-1)-th row

In [69]: [Link](a, (3, 3))
Out[69]:
[array([[2., 5.],
[9., 4.],
[0., 4.]]),
array([], shape=(0, 2), dtype=float64),
array([[9., 6.],
[8., 8.],
[3., 7.],
[7., 5.],
[9., 9.],
[1., 5.],
[4., 3.],
[8., 1.],
[1., 8.]])]

iii. Copies of NumPy arrays

For someone without prior knowledge on how the assignment of NumPy array to variable is, should
consider very carefully.
When we declare an array and assign it into another variable, we have to fully understand that what
NumPy passed is not the values but its reference to the values.
To copy a NumPy array to another variable, we should use deep copy through method .copy().
The following example will make the explanation above clear

In [2]: import numpy as np

In [3]: a = [Link]([[ 0, 1, 2, 3],

...: [ 4, 5, 6, 7],
...: [ 8, 9, 10, 11]])

In [4]: b = a

In [5]: b[0, 0] = 100

In [6]: a
Out[6]:
array([[100, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]])

19
 In [7]: a = [Link]([[ 0, 1, 2, 3],
...: [ 4, 5, 6, 7],
...: [ 8, 9, 10, 11]])

In [8]: b = [Link]()

In [9]: b[0, 0] = 100

In [10]: a
Out[10]:
array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]])

iv. Broadcasting rule of NumPy arrays operations

This is one of the powerful features in NumPy to eliminate for-loop and make the code easy to read
and maintain.
Broadcasting rule is a process of stretching an array into a bigger one such that it will have the
same shape to the other array under binary operation.
The following illustration describes that process.

The addition operator above can be replaced by any binary operator or even a function.

In [11]: a = [Link]([1.0, 2.0, 3.0])

20
 In [12]: b = 2.0

In [13]: a * b
Out[13]: array([2., 4., 6.])

# ---------------------------------------------
In [14]: a = [Link]([[ 0.0, 0.0, 0.0],
...: [10.0, 10.0, 10.0],
...: [20.0, 20.0, 20.0],
...: [30.0, 30.0, 30.0]])

In [15]: b = [Link]([1.0, 2.0, 3.0])

In [16]: a + b
Out[16]:
array([[ 1., 2., 3.],
[11., 12., 13.],
[21., 22., 23.],
[31., 32., 33.]])

# ---------------------------------------------
In [17]: a = [Link]([0.0, 10.0, 20.0, 30.0])

In [18]: b = [Link]([1.0, 2.0, 3.0])

In [19]: a[:, [Link]] + b

Out[19]:
array([[ 1., 2., 3.],
[11., 12., 13.],
[21., 22., 23.],
[31., 32., 33.]])

v. Advanced Indexing of NumPy arrays

In the Basics of NumPy, we have introduced how to do indexing to NumPy’s array. In this section,
we will understand how the concepts of indexing worked in 2D arrays and using some boolean
indexing

Indexing with Arrays of Indices

If we have a 1D array, we can index the elements by 1D array or 2D array. The result will follow the
shape of the indexing array.

# the first 12 square numbers

In [20]: a = [Link](12)**2

In [21]: a
Out[21]: array([ 0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100, 121])

# a 1D array of indices
In [22]: i_idx = [Link]([1, 1, 3, 8, 6])

# get the elements of `a` at the position `i_idx`

In [23]: a[i_idx]
Out[23]: array([ 1, 1, 9, 64, 36])

21
 # a 2D array of indices
In [24]: j_idx = [Link]([[3, 4], [9, 7]])

# get the elements of `a` at the position `j_idx`

# each index in `j_idx` will be interpreted as a single index
In [25]: a[j_idx]
Out[25]:
array([[ 9, 16],
[81, 49]])

The following example will show us when working with 2D arrays and doing indexing using 1D or 2D
array of indices.

a = [Link](12).reshape(3, 4)

In [27]: a
Out[27]:
array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]])

# 2D array of indices for the first axis of `a`

In [28]: i_idx = [Link]([[0, 1],
...: [1, 2]])

# 2D array of indices for the second axis of `a`

In [29]: j_idx = [Link]([[2, 1],
...: [3, 3]])

# i_idx and j_idx must have equal shape

In [31]: a[i_idx, j_idx]
Out[31]:
array([[ 2, 5],
[ 7, 11]])

The above example can be explained through this simple diagram

22
The 2D array of indices will be created by taking for each element from i_idx, j_idx in the same
position of index. Then indexing using two 2D arrays of indices will return the same size of the 2D
array of indices.

Indexing with Boolean Arrays

This is a method to select elements of an array based on whether those elements satisfy some
condition or not.

In [32]: a = [Link](12).reshape(3, 4)

In [33]: b = a > 4

In [34]: b
Out[34]:
array([[False, False, False, False],
[False, True, True, True],
[ True, True, True, True]])

# only select `True` indices

In [35]: a[b]
Out[35]: array([ 5, 6, 7, 8, 9, 10, 11])

# all elements of `a` higher than 4 become 0

In [36]: a[b] = 0

In [37]: a
Out[37]:
array([[0, 1, 2, 3],
[4, 0, 0, 0],
[0, 0, 0, 0]])

23
vi. A formal way to use Matplotlib
Matplotlib is a Python package to create a visualization in 2D (mostly). There is a standard way to
code a plotting program using Matplotlib. This is not my style, but it is coming from the tutorial in the
Matplotlib documentation.

import [Link] as plt

import numpy as np

# create data
rng = [Link].default_rng()
xdata = [Link](10)
ydata = [Link](10 * [Link](10))

# define figure and axes objects

fig, ax = [Link](figsize=(8, 6))

# plot the data to specific visualization

# e.g.: .plot(), .scatter, .imshow()
[Link](xdata, ydata, linestyle='-', color='blue', label="line1")

# setting the axes attributes

ax.set_xlabel('x')
ax.set_ylabel('y')
[Link](loc="best")
ax.set_title('random line plot')

# show the figure

[Link]()

The above script will produce the following figure

24
2. Introduction to TensorFlow
In Coursera courses, the course intends us to use Jupyter Notebook, but now, we try to be familiar
as soon as possible with PyCharm IDE and how to use it. Then, all the codes below will be typed in
Python script inside PyCharm IDE.

a. “Hello World!” version of TensorFlow

First, we start to write a “Hello World!” version of TensorFlow. This program will do a regression
linear with simple data: y=2 x−1 .

Listing 2.1. simple_regression.py

1 import tensorflow as tf
2 import numpy as np
3 import [Link] as plt
4
5 from tensorflow import keras
6
7
8 def plot_data(x_data, y_data):
9 fig, ax = [Link]()
10
11 [Link](x_data, y_data, 'ro')
12
13 [Link](1)
14
15 return None
16
17
18 if __name__ == "__main__":
19 # Define and compile the neural networks
20 model = [Link](
21 [[Link](units=1, input_shape=[1])])
22 [Link](optimizer="sgd", loss="mean_squared_error")
23
24 # Providing the data
25 xs = [Link]([-1.0, 0.0, 1.0, 2.0, 3.0, 4.0])
26 ys = [Link]([-3.0, -1.0, 1.0, 3.0, 5.0, 7.0]) # y = 2x - 1
27
28 # Plotting data will help us to understand faster
29 # visually about the data
30 plot_data(xs, ys)
31
32 # Train the neural network
33 [Link](xs, ys, epochs=500)
34
35 # Predict using the model
36 print([Link](([10.0]))) # this result should be close to 19.

Lines 1-3 imports necessary packages: 1) tensorflow provides a machine learning model,
optimizer, metric and fitting subroutine; 2) numpy provides an efficient way to handle an array; and
3) matplotlib for plotting the data.

In line 5, we called a specific model’s layer (.Dense layer) through keras library.
25
Lines 8-15 is a function declaration for a simple plotting procedure.

Line 18-36 is the main section. In the later scripts, we will move several statements into functions
above this main section. Now, we stick in this format where we put all the computational procedures
under the line if __name__ == “__main__”:.

In lines 20-22, we initialize the model using a regular densely-connected neural network layer. This
is implemented by invoking [Link](). We also use the format that all layers will
be contained by [Link](). By using this container we can include as
many layers as we want sequentially. The keyword argument units=1 in .Dense() layer specified
the dimensionality of the output. The output that we want is to predict a single number for a given
single number. Then the input should have dimension or shape 1, and this is specified by keyword
argument input_shape=[1]. Please note that input shape should be defined by a list even
though there is only one dimension.

Lines 25-26 defines the data sets for x and y using NumPy’s arrays as column vectors. Line 30 is
visualizing the data sets.

Line 33 uses method .fit(), to fit the model into data sets. We also set the epochs. This keyword
argument indicates how many iterations the model trains to the data sets xs and ys.

In the last line, we try to predict y for the input x = 10. The program returns the predicted value
approximately closed to the exact value.

The theoretical background for the above program can be understood as a single fully connected
layer. What does it mean by the fully connected layer? In line 21, we wrote
[Link](). This command will create for us a linear function

y pred :=Wx+b ,

where W is a weight matrix with dimension 1x6. The number of columns represents the length of row
vector x (in the script this corresponds to the variable xs) and b is a biased term. The term “fully
connected” arises from the fact that we multiply all (“connected” and “fully”) element of the input x
with all the elements of the weight matrix W . Then this simple model of machine learning will try to
find the best value of W and b such that it will give the minimum loss. The loss function (or objective
function if you a hardcore optimization person) in the above script is defined as a mean square
error:

N
1
MSE ( y , y pred ):= ∑ ( y i− y pred ,i )2,
N i=1

where y is the provided data which x corresponds to (in the script this corresponds to the variable
ys). The optimization procedure is performed by using a stochastic gradient descent (in the script
we set keyword argument optimizer=’sgd’ to the method [Link]()). A short
description of stochastic gradient descent, this optimizer will compute gradients in a batch of data
using backpropagation (a trick to compute gradients using computational graph and chain rules).
Then will update the initial value of the weight matrix W by the amount of learning rate times

26
negative of the computed gradient. The detail of the mechanism of this optimizer, including
backpropagation, is beyond the scope of this module.

The previous description can be simplified by the following figure.

b. Classifying Fashion MNIST

Now, we want to build a model to predict what category is for a given image. For this purpose, we
use the Fashion MNIST data. This data has the following specification:
● 70,000 images
● 10 categories. All categories are saved as numerical labels starting from 0 to 9.
● Images size are 28 x 28px

The following image is one of the sample images (the first entry) from Fashion MNIST whose
category is “ankle boot” but it is saved as a numerical integer label 9.

All the Fashion MNIST data can be accessed through the TensorFlow module.
27
In Listing 2.2, we will use two dense layers to get a weight matrix as a model of the classifier that we
want to build.

Listing 2.2. fashion_classifier.py

1 import numpy as np
2 import [Link] as plt
3 import matplotlib
4 import tensorflow as tf
5
6
7 if __name__ == "__main__":
8
9 # Load the dataset
10 mnist = [Link].fashion_mnist
11 (training_images, training_labels), (test_images, test_labels) =
12 mnist.load_data()
13
14 # Plot an image from the dataset
15 [Link](training_images[0], cmap="Greys")
16 [Link](1) # increase the pause time to give time for plotting
17
18 print(training_labels[0])
19 print(training_images[0])
20
21 # Normalizing image intensities to [0, 1]
22 training_images = training_images / 255.0
23 test_images = test_images / 255.0
24
25 # Create a model
26 model = [Link]([
27 [Link](),
28 [Link](128, activation=[Link]),
29 [Link](10, activation=[Link])])
30
31 # Compile the model and train it to the dataset
32 [Link](optimizer=[Link](),
33 loss="sparse_categorical_crossentropy",
34 metrics=["accuracy"])
35
36 [Link](training_images, training_labels, epochs=5)
37
38 # Evaluate the model to the test data
39 [Link](test_images, test_labels)

Lines 1-4 imports required packages. We load the Fashion MNIST data in lines 10-12 and separate
between train and test data. From the TensorFlow API this will separate 70,000 images into 60,000
images for train data and 10,000 for test data. Then in lines 14-19, we try to see whether we load
the correct data or not, by picking a single image then plotting it and printing the content.

We also performed data preprocessing by scaling the grayscale value of the image into a range [0,
1] in lines 22-23. This will help the optimizer to only handle not too big numbers. Then, we create a
model with two fully connected layers in lines 25-29. First we flatten the shape of the image from a
matrix of 28x28 into a long row vector 28x28=784 entries. The first fully connected layer categorizes
these 784 entries into 128 hidden variables and activates using ReLU (Rectifier Linear Unit)

28
activation function to induce nonlinearity. Without this activation function, this two-layer model would
be redundant and similar to a one-layer model. In general ReLu is defined as

ReLU ( hi ) :=max(0 , hi ),

where h will be the a row vector result of linear classifier h :=Wx+b . This output from ReLU will be
the input of the second fully connected layer where instead of using ReLU as the activation, this
layer uses softmax activation function. The softmax activation function is defined as

exp (hi )
softmax(h ¿¿ i):= C
¿
,
∑ exp (h ❑
j )
j=1

where C is the number of classes/categories.

This softmax activation function is similar like computing the probability of each output hi but with
adding a twist of exponential function to scaling-up small value in each output hi .

In lines 31-34, we compile all the model’s layers and add optimizer, loss function, and metric. This
Adam optimizer is a fancier optimizer compared to stochastic gradient descent where we do not
update directly negative gradients to the current weight matrix, but with some complicated pre-
steps. In general, this optimizer incorporates AdaGrad and RMSProp optimizers. For a complete
description of Adam optimizer, see the paper of (Kingma and Ba, 2015). Next, we use lost function
sparse_categorical_entropy. This lost function will compute the following value

N C
−1
CatEntropy (s):= ∑ ∑ t (i) log (s(i)k ),
N i=1 k=1 k

(i)
where N is the total image, C is the number of categories/classes, and t k is indicator function which
(i )
is zero if the true category j of image i is not equal and one if they are equal. sk in that categorical
entropy is an output score from the last layer. As a future reminder, if the provided label of the
category is in one-hot encoding format, we should use categorical_entropy.

The last argument that we set in [Link]() is metric. Here, we use the simple metric
accuracy that counts how many images that the model would correctly predict for computed
weight in each epoch (or iteration) divided by the total number of images. We need to remember
that this accuracy is not included in optimizing the weight matrix.

In line 36, we fit the model with training data with a small number of epochs. For this simple data
set, we can achieve high accuracy, greater than 85% in the training data. But we have to know that
this high accuracy does not imply how good the classifier is.

Finally in line 39, we test our classifier to the test data. If you typed correctly the script above, you
will achieve a similar accuracy above 85%. In the above script, you can add more units in the first
fully connected layer or set different epochs. This is what we call hyperparameters. Sometimes you
should find this hyperparameter manually or you can use the hyperparameter optimization
technique that we will discuss in the later chapter.

Like in the previous script, we can simplify all above descriptions into the following figure.
29
Now we introduce a handy class to interrupt the training process if some conditions are satisfied.
This can be done by creating a callback class and overriding the existing method from
[Link]..

Listing 2.3. fashion_classifier_with_callback.py

1 import tensorflow as tf
2
3
4 class MyCallback([Link]):
5 def on_epoch_end(self, epoch, logs=None):
6 if [Link]("accuracy") > 0.6:
7 print("\nReached 60% accuracy so cancelling training!")
8 [Link].stop_training = True
9
10
11 if __name__ == "__main__":
12 mnist = [Link].fashion_mnist
13
14 (x_train, y_train), (x_test, y_test) = mnist.load_data()
15 x_train, x_test = x_train / 255.0, x_test / 255.0
16
17 callbacks = MyCallback()
18
19 model = [Link]([
20 [Link](input_shape=(28, 28)),
21 [Link](512, activation=[Link]),
22 [Link](10, activation=[Link])
23 ])
24
25 [Link](optimizer=[Link](),
26 loss="sparse_categorical_crossentropy",
27 metrics=["accuracy"])
28
29 # will reach 60% accuracy in 2 epochs!
30 [Link](x_train, y_train, epochs=10, callbacks=[callbacks])

30
All the rest codes are the same as Listing 2.2. In lines 4-8, we add a MyCallback class where all
the methods and attributes are inherited from the parent class [Link].
In that parent class, there is a method .on_epoch_end() where we can override that parent’s
class method by redefining that method with the description as in lines 6-8.

c. Classifying Fashion with CNN

We can also implement a model using CNN to classify the fashion category for a given image.
Let us briefly introduce what CNN is. First we need to understand what a convolution layer is. The
following figure explains the basic principle of how to take a convolution operation.

The simple interpretation of the convolution operation is to preserve the feature of the image
spatially and locally. Some simple examples of features are edges, ridges, or corners. This
operation also somehow stores in the encoded way what is the best feature for each filtered area
applied to the image. Every single move and operation of the convolution layer will result in a single
number where we can think of that value like an activated neuron. All the weights of the filtered
matrix will be learned by the model through the training process. In practice this convolution layer is
subsequently followed by a max pooling layer. This layer aggregates the most activated feature in
that image. The principle of max pooling layer is the same as convolution layer but instead of taking
a dot product for the superimposed value on the image with the filtered matrix, this max pooling
layer takes the maximum value in the superimposed value by the max pooling filtered matrix. The
following figure explains how max pooling operates on the result of the convolution layer.

31
In the above figure, we take a 2x2 max pool with stride 2. This stride means that we move the 2x2
max pool every two steps of the element in the input matrix in row direction and column direction. In
practice, the number of strides is the same as the size of the max pooling filter. This max pooling
operation has an interpretation to select the most activated entry in the image to be accounted for
by the computation of the weight matrix in the model. Now, let us apply that convolution and max
pooling layer to the Fashion MNIST data set.

Listing 2.4. fashion_classifier_with_cnn.py

1 import tensorflow as tf
2 import [Link] as plt
3 import numpy as np
4
5
6 def create_cnn_model():
7 mnist = [Link].fashion_mnist
8
9 (training_images, training_labels), (test_images, test_labels) = mnist.load_data()
10
11 print([Link](training_images))
12 training_images = training_images.reshape(60_000, 28, 28, 1)
13 training_images = training_images / 255.
14 test_images = test_images.reshape(10_000, 28, 28, 1)
15 test_images = test_images / 255.
16
17
18 model = [Link]([
19 [Link].Conv2D(64, (3, 3), activation="relu", input_shape=(28, 28, 1)),
20 [Link].MaxPooling2D(2, 2),
21 [Link].Conv2D(64, (3, 3), activation="relu"),
22 [Link].MaxPooling2D(2, 2),
23 [Link](),
24 [Link](128, activation="relu"),
25 [Link](10, activation="softmax")
26 ])
27
28 [Link](optimizer="adam",
29 loss="sparse_categorical_crossentropy",
30 metrics=["accuracy"])
31 [Link]()
32
33 [Link](training_images, training_labels, epochs=10)
34 test_loss = [Link](test_images, test_labels)
35
36 return [model, test_loss]
37
38
39 def visualizing_conv_and_max_pool(model):
40 mnist = [Link].fashion_mnist
41 (_, _), (test_images, test_labels) = mnist.load_data()
42
43 # from this print out, we can decide the indices for
44 # first_image, second_image, third_image
45 print(test_labels[:100].reshape(10, 10))
46

32
 47 f, ax_arr = [Link](3, 4, figsize=(10, 10))
48 first_image = 0
49 second_image = 28
50 third_image = 23
51 convolution_number = 2 # starting from 0 to 63
52
53 layer_outputs = [[Link] for layer in [Link]]
54 activation_model = [Link](inputs=[Link],
55 outputs=layer_outputs)
56
57 for x in range(0, 4):
58 f1 = activation_model.predict(test_images[first_image].reshape(1, 28, 28, 1))[x]
59 ax_arr[0, x].imshow(f1[0, :, :, convolution_number], cmap="inferno")
60 ax_arr[0, x].grid(False)
61
62 f2 = activation_model.predict(test_images[second_image].reshape(1, 28, 28, 1))[x]
63 ax_arr[1, x].imshow(f2[0, :, :, convolution_number], cmap="inferno")
64 ax_arr[1, x].grid(False)
65
66 f3 = activation_model.predict(test_images[third_image].reshape(1, 28, 28, 1))[x]
67 ax_arr[2, x].imshow(f3[0, :, :, convolution_number], cmap="inferno")
68 ax_arr[2, x].grid(False)
69
70 [Link](2)

if __name__ == "__main__":
cnn_model, cnn_test_loss = create_cnn_model()
cnn_model.save("model-saved/category2.h5")

visualizing_conv_and_max_pool(cnn_model)

Similar to Listing 2.3, now we add 4 layers [Conv2D+ReLU, MaxPooling, Conv2D+ReLU,

MaxPooling] before the two-fully connected layers. We also add a function
visualizing_conv_and_max_pool() to understand what is the influence of those four new
layers. In the figure below, we can see the result of that function to the images category 9 (“ankle
boot”). In the first application, we can see the layer Conv2D+ReLU tries to fire up the edges of the
images. And the max pooling tries to make that edges become more significant to be accounted for
the training process.

33
d. Classifying Emotion with CNN
In this section, we want to apply the CNN model to classify emotion from the following data sets.
The dataset is prepared by Laurence Moroney (AI Lead at Google). You can download it from this
link: [Link]
This data set contains two kinds of images of expression: happy or sad. Each kind of expression
contains 40 images. All the images are generated from 3D emoji characters. In this example we
want to show how to use a built-in preprocessing class in TensorFlow, the ImageGenerator
class.. This class will be explored more in the next chapter. We also employ loading data set by a
directory path instead of loading all image data to the program.

For the testing data, you can search any image with a happy or sad expression or download four
sample images in here [Link]

The following listing will show that with the small number of images and different unseen images as
the testing data, we would highly expect the bad result on the prediction. And also increasing the
number of layers will not solve the problem and severely make the prediction. The reason is the
model overfitting to the data. This is the common problem that we should be aware of when we use
machine learning as a classifier.

We also provide two functions that may come very handy to see the training data (
plot_training_images()) and predict a category for a given image by transforming first to
follow the input shape requirement of the model (classify_images())

Listing 2.5. face_expression_classifier_with_cnn.py

1 import os
2 import zipfile
3 import numpy as np
4 import tensorflow as tf
5 import [Link] as plt
6 import [Link] as mpimg

34
 7
8 from [Link] import Adam
9 from [Link] import ImageDataGenerator
10 from [Link] import image as keras_image
11
12
13 class MyCallback([Link]):
14 def __init__(self, desired_accuracy):
15 super(MyCallback, self).__init__()
16 self.DESIRED_ACCURACY = desired_accuracy
17
18 def on_epoch_end(self, epoch, logs=None):
19 if [Link]("accuracy") > self.DESIRED_ACCURACY:
20 print(f"\nReached {self.DESIRED_ACCURACY * 100:.2f}"
21 + " accuracy so cancelling training!")
22 [Link].stop_training = True
23
24
25 def load_dataset(zip_file_path, extracted_zip_file_path, train_happy_dir, train_sad_dir):
26 zip_ref = [Link](zip_file_path, "r")
27 zip_ref.extractall(extracted_zip_file_path)
28 zip_ref.close()
29
30 train_happy_names = [Link](train_happy_dir)
31 train_sad_names = [Link](train_sad_dir)
32
33 print(train_happy_names[:10])
34 print(train_sad_names[:10])
35 print(f"total training happy images {len(train_happy_names)}")
36 print(f"total training sad images {len(train_sad_names)}")
37
38 return train_happy_names, train_sad_names
39
40
41 def do_data_preprocessing(dataset_dir):
42 train_datagen = ImageDataGenerator(rescale=1. / 255)
43
44 train_generator = train_datagen.flow_from_directory(
45 dataset_dir,
46 target_size=(150, 150),
47 batch_size=8, # reduce this from 128 (our dataset is small)
48 class_mode="binary"
49 )
50
51 return train_generator
52
53
54 def create_cnn_model():
55 model = [Link]([
56 [Link].Conv2D(16, (3, 3), activation="relu", input_shape=(150, 150, 3)),
57 [Link].MaxPooling2D(2, 2),
58 [Link].Conv2D(32, (3, 3), activation="relu"),
59 [Link].MaxPooling2D(2, 2),
60 [Link].Conv2D(64, (3, 3), activation="relu"),
61 [Link].MaxPooling2D(2, 2),
62 [Link](),
63 [Link](512, activation="relu"),
64 [Link](1, activation="sigmoid")
65 ])
66
67 [Link](loss="binary_crossentropy",
68 optimizer=Adam(learning_rate=0.001),
69 metrics=["accuracy"])
70
71 [Link]()
72
73 return model
74
75
76 def plot_training_images(train_happy_dir, train_sad_dir,

35
 77 train_happy_names, train_sad_names):
78 # Parameter for our graph; we'll output images in a 4x4 configuration
79 n_rows = 4
80 n_cols = 4
81
82 # Index for iterating over images
83 img_index = 0
84
85 # Set up matplotlib fig, and size it to fit 4x4 pics
86 fig = [Link]()
87 fig.set_size_inches(n_cols * 4, n_rows * 4)
88
89 img_index += 8
90 next_happy_img = [[Link](train_happy_dir, fname)
91 for fname in train_happy_names[img_index - 8:img_index]]
92 next_sad_img = [[Link](train_sad_dir, fname)
93 for fname in train_sad_names[img_index - 8:img_index]]
94
95 for i, img_path in enumerate(next_happy_img + next_sad_img):
96 # Set up subplot; subplot indices start at 1
97 sp = [Link](n_rows, n_cols, i + 1)
98 [Link]("off") # don't show axes (or gridlines)
99
100 img = [Link](img_path)
101 [Link](img)
102 [Link](1.0)
103
104
105 def classify_images(fn_arr, model):
106 for fn in fn_arr:
107 path = "datasets/" + fn
108 img = keras_image.load_img(path, target_size=(150, 150))
109 x = keras_image.img_to_array(img)
110 x = np.expand_dims(x, axis=0)
111
112 image_i = [Link]([x])
113 classes = [Link](image_i, batch_size=10)
114 print(classes[0])
115 if classes[0] > 0.5:
116 print(fn + " is happy")
117 else:
118 print(fn + " is sad")
119
120
121 if __name__ == "__main__":
122 zip_file_path = "datasets/[Link]"
123 extracted_zip_file_path = "datasets/happy-or-sad"
124
125 # Directory with our training happy images
126 train_happy_dir = [Link]("datasets/happy-or-sad/happy")
127
128 # Directory with our training sad images
129 train_sad_dir = [Link]("datasets/happy-or-sad/sad")
130
131 # Extract zip and get path to the data sets
132 train_happy_names, train_sad_names = load_dataset(zip_file_path, extracted_zip_file_path,
133 train_happy_dir, train_sad_dir)
134
135 # Plot the data sets
136 plot_training_images(train_happy_dir, train_sad_dir, train_happy_names, train_sad_names)
137
138 # Data preprocessing
139 train_generator = do_data_preprocessing(extracted_zip_file_path)
140
141 # Building a small CNN model
142 cnn_model = create_cnn_model()
143
144 # Training the model to the training data
145 DESIRED_ACCURACY = 0.99
146 callbacks = MyCallback(DESIRED_ACCURACY)

36
147
148 history = cnn_model.fit(
149 train_generator,
150 steps_per_epoch=8,
151 epochs=50,
152 verbose=1,
153 callbacks=[callbacks]
154 )
155
156 # Predict some images
157 fn_arr = ["beauty-1132617_640.jpg", "girl-2961959_640.jpg",
158 "woman-2126727_640.jpg", "beautiful-18279_640.jpg"]
159 classify_images(fn_arr, cnn_model)

37
3. Convolutional Neural Network in TensorFlow
In the previous chapter, we have briefly introduced the convolution layer which is the building block
of the convolutional neural network. Please review the previous chapter, if you have not read it yet.

a. Classifying Cats and Dogs

In this section, we will use images of cats and dogs from Kaggle competition
([Link] But for the practical purpose, we use 1/10 of the total
images as we continue building better classifiers for cats and dogs images. You can download the
smaller version of the cats and dogs data set in (thanks again to Lawrence Moroney)
[Link] For the testing images,
you can use these 2 images of cats and 2 images of dogs from [Link]

Listing 3.1. cats_and_dogs_classifier_with_cnn.py

1 import os
2 import zipfile
3 import numpy as np
4 import random
5 import [Link] as plt
6 import [Link] as mpimg
7 import tensorflow as tf
8
9 from [Link] import Adam
10 from [Link] import ImageDataGenerator, img_to_array, load_img
11 from [Link] import image as keras_image
12
13
14 def load_dataset(zip_file_path, extracted_zip_file_path):
15 zip_ref = [Link](zip_file_path, "r")
16 zip_ref.extractall([Link](extracted_zip_file_path)[0])
17 zip_ref.close()
18
19 train_dir = [Link](extracted_zip_file_path, "train")
20 validation_dir = [Link](extracted_zip_file_path, "validation")
21
22 train_cats_dir = [Link](train_dir, "cats")
23 train_dogs_dir = [Link](train_dir, "dogs")
24
25 validation_cats_dir = [Link](validation_dir, "cats")
26 validation_dogs_dir = [Link](validation_dir, "dogs")
27
28 train_cat_fnames = [Link](train_cats_dir)
29 train_dog_fnames = [Link](train_dogs_dir)
30
31 validation_cat_fnames = [Link](validation_cats_dir)
32 validation_dog_fnames = [Link](validation_dogs_dir)
33
34 print(train_cat_fnames[:10])
35 print(train_dog_fnames[:10])
36
37 print("total training cat images :", len(train_cat_fnames))
38 print("total training dog images :", len(train_dog_fnames))
39
40 print("total validation cat images:", len(validation_cat_fnames))
41 print("total validation dog images:", len(validation_dog_fnames))
42
43 return train_dir, validation_dir, train_cats_dir, train_dogs_dir, \
44 train_cat_fnames, train_dog_fnames
45
46
47 def do_data_preprocessing(train_dir, validation_dir):

38
 48 train_datagen = ImageDataGenerator(rescale=1. / 255)
49 validation_datagen = ImageDataGenerator(rescale=1. / 255)
50
51 train_generator = train_datagen.flow_from_directory(
52 train_dir,
53 batch_size=20,
54 class_mode="binary",
55 target_size=(150, 150)
56 )
57
58 validation_generator = validation_datagen.flow_from_directory(
59 validation_dir,
60 batch_size=20,
61 class_mode="binary",
62 target_size=(150, 150)
63 )
64
65 return train_generator, validation_generator
66
67
68 def create_cnn_model():
69 # Building a small model from scratch: validation accuracy ~ 72 %
70
71 model = [Link]([
72 [Link].Conv2D(16, (3, 3), activation="relu", input_shape=(150, 150, 3)),
73 [Link].MaxPooling2D(2, 2),
74 [Link].Conv2D(32, (3, 3), activation="relu"),
75 [Link].MaxPooling2D(2, 2),
76 [Link].Conv2D(64, (3, 3), activation="relu"),
77 [Link].MaxPooling2D(2, 2),
78 [Link](),
79 [Link](512, activation="relu"),
80 [Link](1, activation="sigmoid")
81 ])
82
83 [Link]()
84
85 [Link](optimizer=Adam(learning_rate=0.001),
86 loss="binary_crossentropy",
87 metrics=["accuracy"])
88 return model
89
90
91 def plot_cats_and_dogs(train_cats_dir, train_dogs_dir,
92 train_cat_fnames, train_dog_fnames):
93 # Parameter for our graph; we'll output images in a 4x4 configuration
94 nrows = 4
95 ncols = 4
96
97 # Index for iterating over images
98 pic_index = 0
99
100 # Set up matplotlib fig, and size it to fit 4x4 pics
101 fig = [Link]()
102 fig.set_size_inches(ncols * 4, nrows * 4)
103
104 pic_index += 8
105 next_cat_pix = [[Link](train_cats_dir, fname)
106 for fname in train_cat_fnames[pic_index - 8:pic_index]]
107 next_dog_pix = [[Link](train_dogs_dir, fname)
108 for fname in train_dog_fnames[pic_index - 8:pic_index]]
109
110 for i, img_path in enumerate(next_cat_pix + next_dog_pix):
111 # Set up subplot; subplot indices start at 1
112 sp = [Link](nrows, ncols, i + 1)
113 [Link]("off") # Don't show axes (or gridlines)
114
115 img = [Link](img_path)
116 [Link](img)
117

39
118 [Link](1.0)
119
120
121 def classify_images(fn_arr, model):
122 for fn in fn_arr:
123 path = "datasets/" + fn
124 img = keras_image.load_img(path, target_size=(150, 150))
125 x = keras_image.img_to_array(img)
126 x = np.expand_dims(x, axis=0)
127
128 image_i = [Link]([x])
129 classes = [Link](image_i, batch_size=10)
130 print(classes[0])
131 if classes[0] > 0.5:
132 print(fn + " is a dog")
133 else:
134 print(fn + " is a cat")
135
136
137 def plot_intermediate_repr(model, train_cats_dir, train_dogs_dir,
138 train_cat_fnames, train_dog_fnames):
139 # Let's define a new Model that will take an image as input, and will output
140 # intermediate representations for all layers in the previous model after the first.
141 successive_outputs = [[Link] for layer in [Link][0:]]
142 visualization_model = [Link](inputs=[Link],
143 outputs=successive_outputs)
144
145 # Let's prepare a random input image from the training set.
146 cat_img_files = [[Link](train_cats_dir, f) for f in train_cat_fnames]
147 dog_img_files = [[Link](train_dogs_dir, f) for f in train_dog_fnames]
148 img_path = [Link](cat_img_files + dog_img_files)
149
150 img = load_img(img_path, target_size=(150, 150)) # this is a PIL image
151 x = img_to_array(img) # numpy array with shape (150, 150, 3)
152 print(f"[Link] : {[Link]}")
153 x = [Link]((1,) + [Link])
154 x = x / 255.
155
156 # Let's run our image through our network, thus obtaining all
157 # intermediate representations for this image.
158 successive_feature_maps = visualization_model.predict(x)
159
160 # These are the names of the layers, so can have them as part of our plot
161 layer_names = [[Link] for layer in [Link]]
162
163 # Now let's display our representations
164 for layer_name, feature_map in zip(layer_names, successive_feature_maps):
165 # Just do this for the conv / maxpool layers, not the fully connected layers
166 if len(feature_map.shape) == 4:
167 n_features = feature_map.shape[-1] # number of features (filters) in feature map
168
169 # The feature map has shape (1, size, size, n_features)
170 size = feature_map.shape[1]
171
172 # We will tile our images in this matrix
173 display_grid = [Link]((size, size * n_features))
174 for i in range(n_features):
175 # Post-process the feature to make it visually palatable
176 x = feature_map[0, :, :, i]
177 x -= [Link]()
178 # print('[Link]()', [Link]())
179 x = x / [Link]() if [Link]() > 1e-14 else x
180 x *= 64
181 x += 128
182 x = [Link](x, 0, 255).astype("uint8")
183
184 # We will tile each filter into this big horizontal grid
185 display_grid[:, i * size: (i + 1) * size] = x
186
187 # Display the grid

40
 188 [Link](figsize=(20, 2))
189 [Link](layer_name)
190 [Link](False)
191 [Link](display_grid, aspect="auto", cmap="viridis")
192 plt.subplots_adjust(left=0.03, right=0.99)
193
194 [Link](1.0)
195
196
197 def plot_history(train, val, title):
198 epochs = range(len(train))
199 [Link]()
200 [Link](epochs, train, label="train")
201 [Link](epochs, val, label="val")
202 [Link](title)
203 [Link](loc="best")
204 [Link](1.0)
205
206
207 if __name__ == "__main__":
208 zip_file_path = "datasets/cats_and_dogs_filtered.zip"
209 extracted_zip_file_path = "datasets/cats_and_dogs_filtered"
210
211 # Extract zip and get path to the data sets
212 train_dir, validation_dir, train_cats_dir, train_dogs_dir, \
213 train_cat_fnames, train_dog_fnames \
214 = load_dataset(zip_file_path, extracted_zip_file_path)
215
216 # Plot the data sets
217 plot_cats_and_dogs(train_cats_dir, train_dogs_dir,
218 train_cat_fnames, train_dog_fnames)
219
220 # Data preprocessing
221 train_generator, validation_generator \
222 = do_data_preprocessing(train_dir, validation_dir)
223
224 # Building a small CNN model
225 cnn_model = create_cnn_model()
226
227 history = cnn_model.fit(
228 train_generator,
229 validation_data=validation_generator,
230 steps_per_epoch=100,
231 epochs=15,
232 validation_steps=50,
233 verbose=1
234 )
235
236 fn_arr = ["cat-2083492_only_head.jpg", "cat-1146504_640.jpg",
237 "dog-3846767_640.jpg", "dog-3388069_640.jpg"]
238 classify_images(fn_arr, cnn_model)
239
240 # Visualizing intermediate representations
241 plot_intermediate_repr(cnn_model, train_cats_dir, train_dogs_dir,
242 train_cat_fnames, train_dog_fnames)
243
244 # Evaluating accuracy and loss for the model
245 acc = [Link]["accuracy"]
246 val_acc = [Link]["val_accuracy"]
247 loss = [Link]["loss"]
248 val_loss = [Link]["val_loss"]
249
250 plot_history(acc, val_acc, "Training and validation accuracy")
251 plot_history(loss, val_loss, "Training and validation loss")

In the above listing, we add two more functions: plot_intermediate_repr() and

plot_history(). The first function is to create several plots for what happened to the input

41
images along the operation of convolutional layers and max pooling layers. The figure below shows
the first effect of the first convolutional layer.

We see that the parameters or weight matrix of convolutional layers, which mainly control the
learning process, try to find primitive features like edges, blobs, or corners. This is pretty amazing
how the process is to be done automatically. In the below is how the tensor input image (multi-
dimensional arrays) transforms its dimension after each operation in each layer.

The second function has a purpose to create a plot of loss vs. epoch and accuracy vs. epoch. This
plot is very crucial as we tweak the hyperparameters such that it will give the optimal weights matrix
of the model. If we run the program above, the two last functions plot_history(), we give the
following figures.

42
We clearly see that the model is overfitting to the training data as it indicates a large gap between
the curve of training and validation data in the later epochs. In the next section, we will address this
issue

b. Improving Cats and Dogs Classification

If we run the previous listing program (Listing 3.1), we will have accuracy for the validation data
stalling at the level 0.72 even though the accuracy of the training data is close to 1. This is the
overfitting which always caught many deep learning models. There are many ways to improve the
classifier. For our case, we use two methods which are by image augmentation and dropout of
some layer’s connection.

Before we improve our code, let us discuss briefly what is image augmentation and dropout layer.
From its name, we can guess that image augmentation is a way to augment the existing image with
some image transformations like zoom in/out (scaling+cropping/filling), rotating, shearing, flipping
(mirroring), shifting, etc.

Fortunately, TensorFlow provides a handy class to do all such transformations above which is
ImageDataGenerator(). We have already introduced that class in Listing 2.5. In the upcoming
code, we will use several other attributes and methods from that class.

Then for the dropout layer, it serves as a masking layer which skips some entries in its input.
In practice, a dropout layer is put before fully-connected layers and after all combination of
convolutional and max pooling layers. We put it after all combination of convolutional and max
pooling layers to make the learning process of the model to account for all the possibilities first.
Then hopefully by dropping out some specific neuron, the model only considers the most important
features and withdraws all the redundant features. Inside computation process of dropout layer is
show in figure below
43
Fully equipped with the ImageDataGenerator() and dropt out layers, we are ready to improve
the previous CNN model in Listing 2.7

Listing 3.2. cats_and_dogs_classifier_with_imagedatagen_and_dropout.py

1 import os
2 import numpy as np
3 import [Link] as plt
4 import tensorflow as tf
5
6 from [Link] import Adam
7 from [Link] import ImageDataGenerator
8 from [Link] import image as keras_image
9
10
11 def plot_history(train, val, title):
12 epochs = range(len(train))
13 [Link]()
14 [Link](epochs, train, label="train")
15 [Link](epochs, val, label="val")
16 [Link](title)
17 [Link](loc="best")
18 [Link](1.0)
19
20
21 def do_data_preprocessing(train_dir, validation_dir, aug=False):
22 if aug:
23 train_datagen = ImageDataGenerator(
24 rescale=1./255,
25 rotation_range=40,
26 width_shift_range=0.2,
27 height_shift_range=0.2,
28 shear_range=0.2,
29 zoom_range=0.2,
30 horizontal_flip=True,
31 fill_mode="nearest"
32 )
33 else:
34 train_datagen = ImageDataGenerator(rescale=1./255)

44
 35
36 validation_datagen = ImageDataGenerator(rescale=1./255)
37
38 train_generator = train_datagen.flow_from_directory(
39 train_dir,
40 target_size=(150, 150),
41 batch_size=20,
42 class_mode="binary"
43 )
44
45 validation_generator = validation_datagen.flow_from_directory(
46 validation_dir,
47 target_size=(150, 150),
48 batch_size=20,
49 class_mode="binary"
50 )
51
52 return train_generator, validation_generator
53
54
55 def create_cnn_model():
56 model = [Link]([
57 [Link].Conv2D(32, (3, 3), activation="relu", input_shape=(150, 150, 3)),
58 [Link].MaxPooling2D(2, 2),
59 [Link].Conv2D(64, (3, 3), activation="relu"),
60 [Link].MaxPooling2D(2, 2),
61 [Link].Conv2D(128, (3, 3), activation="relu"),
62 [Link].MaxPooling2D(2, 2),
63 [Link].Conv2D(128, (3, 3), activation="relu"),
64 [Link].MaxPooling2D(2, 2),
65 [Link](0.5), # 3rd options (with image augmentation)
66 [Link](),
67 [Link](512, activation="relu"),
68 [Link](1, activation="sigmoid")
69 ])
70
71 [Link](loss="binary_crossentropy",
72 optimizer=Adam(lr=1e-4),
73 metrics=["accuracy"])
74
75 [Link]()
76
77 return model
78
79
80 def classify_images(fn_arr, model):
81 for fn in fn_arr:
82 path = "datasets/" + fn
83 img = keras_image.load_img(path, target_size=(150, 150))
84 x = keras_image.img_to_array(img)
85 x = np.expand_dims(x, axis=0)
86
87 image_i = [Link]([x])
88 classes = [Link](image_i, batch_size=10)
89 print(classes[0])
90 if classes[0] > 0.5:
91 print(fn + " is a dog")
92 else:
93 print(fn + " is a cat")
94
95
96 if __name__ == "__main__":
97 base_dir = "datasets/cats_and_dogs_filtered"
98 train_dir = [Link](base_dir, "train")
99 validation_dir = [Link](base_dir, "validation")
100
101 train_cats_dir = [Link](train_dir, "cats")
102 train_dogs_dir = [Link](train_dir, "dogs")
103
104 validation_cats_dir = [Link](train_dir, "dogs")

45
 105 validation_dogs_dir = [Link](validation_dir, "cats")
106
107 print(len([Link](train_cats_dir)))
108 print(len([Link](train_dogs_dir)))
109 print(len([Link](validation_cats_dir)))
110 print(len([Link](validation_dogs_dir)))
111
112 # 1st options without image augmentation and dropout layer
113 # 2nd options with image augmentation without dropout layer
114 # 3rd options with image augmentation and dropout layer
115 train_generator, validation_generator \
116 = do_data_preprocessing(train_dir, validation_dir, aug=True)
117
118 # Build a CNN model
119 cnn_model = create_cnn_model()
120
121 history = cnn_model.fit(
122 train_generator,
123 # steps_per_epoch=100, # 2000 images = batch_size * steps
124 epochs=100,
125 validation_data=validation_generator,
126 # validation_steps=50,
127 verbose=1
128 )
129
130 acc = [Link]["accuracy"]
131 val_acc = [Link]["val_accuracy"]
132 loss = [Link]["loss"]
133 val_loss = [Link]["val_loss"]
134
135 plot_history(acc, val_acc, "Training and validation accuracy")
136 plot_history(loss, val_loss, "Training and validation loss")
137
138 # test the model
139 fn_arr = ["cat-2083492_only_head.jpg", "cat-1146504_640.jpg",
140 "dog-3846767_640.jpg", "dog-3388069_640.jpg"]
141 classify_images(fn_arr, cnn_model)

In lines 23-32, we add several image transformations: rotation, shifting, shearing, zoom in/out, and
flipping. Due to those transformations, there would be an empty pixel area that should be filled. In
line 31, we put the keyword argument fill_mode=”nearest” to fill that empty pixel area with the
nearest image value. All the rest of the lines are similar to the Listing 3.1.

If we run the above program, we will get the following results for accuracy and loss. Comparing
these results to the previous result of Listing 3.1, will give so much difference on how validation
curves tend to be close to the training curve. This indicates that improvement is working and might
give a better result. If you try to apply this model to the four sample test images of dog and cats, see
lines 139-141, we will get slightly better results, unless it might give one incorrect result due to the
very differently unseen data.

46
c. Transfer Learning based on VGG Net
Now, we will use the most useful technique in machine learning in which we do not need to train our
model to the huge amount of data if we only have a small amount of data to be trained. This is
called transfer learning. The idea is we use a pre-trained model and unfreeze some layers near the
end layers. After that we train that pre-trained model using our small data. If our problem is similar
to the problem that the pre-trained model has solved, we happily get the fastest training and
accurate classifier.

To show the usefulness of transfer learning, we demonstrate it using VGG Net. For the detailed
description about this CNN architecture, you can read the following paper by (Simonyan and
Zisserman, 2014). The following figure shows a simple abstraction of VGG Net.

adapted from (Cord, 2016)

VGG Net has been trained for 1.3M images with 1000 classes which is part of the ILSVRC-2012
dataset. The pre-trained weight of VGG Net on that dataset can be downloaded in
47
[Link]
vgg16_weights_tf_dim_ordering_tf_kernels_notop.h5. This pre-trained weight is a weight without
the weight of fully-connected layers because we will change these fully-connected layers with our
need for a cat and dog classifier. Now, let us look at the listing program for transfer learning.

Listing 3.3. cats_and_dogs_classifier_with_transfer_learning.py

1 import os
2 import numpy as np
3 import [Link] as plt
4 import tensorflow as tf
5
6 from [Link] import layers
7 from [Link] import Model
8 from [Link].vgg16 import VGG16
9 from [Link] import Adam
10 from [Link] import ImageDataGenerator
11 from [Link] import image as keras_image
12
13
14 def plot_history(train, val, title):
15 epochs = range(len(train))
16 [Link]()
17 [Link](epochs, train, label="train")
18 [Link](epochs, val, label="val")
19 [Link](title)
20 [Link](loc="best")
21 [Link]()
22 [Link](1.0)
23
24
25 def do_data_preprocessing(train_dir, validation_dir, aug=False):
26 if aug:
27 train_datagen = ImageDataGenerator(
28 rescale=1./255,
29 rotation_range=40,
30 width_shift_range=0.2,
31 height_shift_range=0.2,
32 shear_range=0.2,
33 zoom_range=0.2,
34 horizontal_flip=True,
35 fill_mode="nearest"
36 )
37 else:
38 train_datagen = ImageDataGenerator(rescale=1./255)
39
40 validation_datagen = ImageDataGenerator(rescale=1. / 255)
41
42 train_generator = train_datagen.flow_from_directory(
43 train_dir,
44 batch_size=20,
45 class_mode="binary",
46 target_size=(150, 150)
47 )
48
49 validation_generator = validation_datagen.flow_from_directory(
50 validation_dir,
51 batch_size=20,
52 class_mode="binary",
53 target_size=(150, 150)
54 )
55
56 return train_generator, validation_generator
57
58
59 def create_cnn_model(local_weights_file):
60 pre_trained_model = VGG16(input_shape=(150, 150, 3),
61 include_top=False,
62 weights=None)

48
 63
64 pre_trained_model.load_weights(local_weights_file)
65
66 for layer in pre_trained_model.layers:
67 [Link] = False
68
69 pre_trained_model.summary()
70
71 last_layer = pre_trained_model.get_layer("block5_pool")
72 print("last layer output shape: ", last_layer.output_shape)
73 last_output = last_layer.output
74
75 x = [Link](0.2)(last_output)
76 x = [Link]()(x)
77 x = [Link](1024, activation="relu")(x)
78 x = [Link](1, activation="sigmoid")(x)
79
80
81 model = Model(pre_trained_model.input, x)
82
83 [Link](optimizer=Adam(learning_rate=1e-4),
84 loss="binary_crossentropy",
85 metrics=["accuracy"])
86
87 return model
88
89
90 def classify_images(fn_arr, model):
91 for fn in fn_arr:
92 path = "datasets/" + fn
93 img = keras_image.load_img(path, target_size=(150, 150))
94 x = keras_image.img_to_array(img)
95 x = np.expand_dims(x, axis=0)
96
97 image_i = [Link]([x])
98 classes = [Link](image_i, batch_size=10)
99 print(classes[0])
100 if classes[0] > 0.5:
101 print(fn + " is a dog")
102 else:
103 print(fn + " is a cat")
104
105
106 if __name__ == "__main__":
107 local_weights_file = "pre-trained/vgg16_weights_tf_dim_ordering_tf_kernels_notop.h5"
108
109 base_dir = "datasets/cats_and_dogs_filtered"
110 train_dir = [Link](base_dir, "train")
111 validation_dir = [Link](base_dir, "validation")
112
113 train_cats_dir = [Link](train_dir, "cats")
114 train_dogs_dir = [Link](train_dir, "dogs")
115
116 validation_cats_dir = [Link](validation_dir, "cats")
117 validation_dogs_dir = [Link](validation_dir, "dogs")
118
119 train_cat_fnames = [Link](train_cats_dir)
120 train_dog_fnames = [Link](train_dogs_dir)
121
122 print(len(train_cat_fnames))
123 print(len(train_dog_fnames))
124
125 train_generator, validation_generator \
126 = do_data_preprocessing(train_dir, validation_dir, aug=True)
127
128 # Build a CNN model with pre-trained VGG16 net
129 cnn_model = create_cnn_model(local_weights_file)
130
131 history = cnn_model.fit(
132 train_generator,

49
 133 #steps_per_epoch=100,
134 epochs=20,
135 validation_data=validation_generator,
136 #validation_steps=50,
137 verbose=1
138 )
139
140 acc = [Link]["accuracy"]
141 val_acc = [Link]["val_accuracy"]
142 loss = [Link]["loss"]
143 val_loss = [Link]["val_loss"]
144
145 plot_history(acc, val_acc, "Training and validation accuracy")
146 plot_history(loss, val_loss, "Training and validation loss")
147
148 # test the model
149 fn_arr = ["cat-2083492_only_head.jpg", "cat-1146504_640.jpg",
150 "dog-3846767_640.jpg", "dog-3388069_640.jpg"]
151 classify_images(fn_arr, cnn_model)

In the listing above specifically in line 59, we change the input shape of the image into 150x150
instead using the standard VGG Net 224x224. Then we also provide with pre-download the weight
of pre-trained VGG Net on ILSVRC-2012 dataset. Then we use the TensorFlow Model class and
also VGG16 architecture from [Link].vgg16. Of course, if you are
willing to build the VGG Net architecture by yourself, you certainly can do that. But, we prefer to
load that architecture from the TensorFlow library. In line 68, we print the summary of VGG Net to
know the last layer’s name where we used it as an input in line 70. We attach two fully-connected
layers with a dropout layer before them. This dropout layer serves as we don’t want to inherit so
many features from VGG Net pre-trained weight. If we run the listing above, for using only 20
epochs, we arrive with the similar shape of training and validation curves as we got in the previous
section.

d. Classifying Images of Sign Languages

We end this chapter by performing classification of sign language dataset. First you can download
the datasets and four sample images from the following links:

● training data: [Link]

● validation data: [Link]
● four sample images: [Link]

50
Those datasets are acquired from its original source from Kaggle competition “Sign Language
MNIST” ([Link] We put in those links such that
you don’t need to sign-in to download that data. The datasets are in .csv format where each column
represents the pixel values with range from 0 to 255. In total there are 28x28 pixels. Each row
represents the one image of hand-sign. The following is a table of the pairs of images and its
corresponding alphabet.

If you plot one of the images from the .csv file, you will get the grayscale version and much smaller
resolution. Looking carefully on the above table, you can notice that there is no representation of
hand-sign for letters “J” and “Z”. These two letters are represented by the finger's motion so we
can’t include them.

The CNN model that we used is similar to the CNN model that we used in cat and dog classifiers
but shallower layers. We also add a function get_data() to convert data from the .csv format into
a NumPy array. Without further ado, let us look at the listing program for this classifier.

Listing 3.4. hand_sign_language_classifier.py

1 import os
2 import sys
3 import numpy as np
4 import [Link] as plt
5
6 from [Link] import Adam
7 from [Link] import ImageDataGenerator
8 from [Link] import image as keras_image
9
10 import tensorflow as tf
11
12
13 def get_data(filename):
14 with open(filename) as training_file:
15 _ = training_file.readline() # skip first line
16 data = training_file.readlines()
17
18 labels = []
19 images = []

51
20 num_of_data = len(data)
21 for i, row in enumerate(data):
22 row = [Link]("\n").split(",")
23 [Link](row[0])
24 [Link](np.array_split(row[1:785], 28))
25
26 [Link](f"\rprocessing: {(i + 1) / float(num_of_data) * 100:.2f} %")
27 [Link]()
28
29 print("")
30 labels = [Link](labels).astype(float)
31 images = [Link](images).astype(float)
32
33 return images, labels
34
35
36 def plot_one_image(image_data, image_label):
37 fig, ax = [Link]()
38
39 [Link](image_data, cmap="gray", vmin=0, vmax=255)
40
41 num_to_alphabet = ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I',
42 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S',
43 'T', 'U', 'V', 'W', 'X', 'Y']
44 ax.set_title("image_label = {:g} ({:s})".format(image_label,
45 num_to_alphabet[int(image_label)]))
46
47 [Link]()
48 [Link](1.0)
49
50
51 def do_data_preprocessing(training_images, training_labels,
52 validation_images, validation_labels):
53 train_datagen = ImageDataGenerator(
54 rescale=1. / 255,
55 rotation_range=40,
56 width_shift_range=0.2,
57 height_shift_range=0.2,
58 shear_range=0.2,
59 zoom_range=0.2,
60 fill_mode="nearest"
61 )
62
63 validation_datagen = ImageDataGenerator(rescale=1. / 255)
64
65 training_generator = train_datagen.flow(
66 training_images,
67 training_labels,
68 batch_size=20,
69 )
70
71 validation_generator = validation_datagen.flow(
72 validation_images,
73 validation_labels,
74 batch_size=20
75 )
76
77 return training_generator, validation_generator
78
79
80 def create_cnn_model():
81 # image size is 28x28, we don't need third conv2d
82 # This will make the image size 1x1px!
83 # second conv2D will make the image size 5x5px
84
85 model = [Link]([
86 [Link].Conv2D(64, (3, 3), activation="relu", input_shape=(28, 28, 1)),
87 [Link].MaxPooling2D(2, 2),
88 # [Link].Conv2D(64, (3, 3), activation="relu"),
89 # [Link].MaxPooling2D(2, 2),

52
 90 # [Link].Conv2D(128, (3, 3), activation="relu"),
91 # [Link].MaxPooling2D(2, 2),
92 [Link](0.1),
93 [Link](),
94 [Link](1024, activation="relu"),
95 [Link](26, activation="softmax") # labels have value 0 - 24
96 ])
97
98 [Link](
99 loss="sparse_categorical_crossentropy",
100 optimizer=Adam(learning_rate=0.001),
101 metrics=["accuracy"]
102 )
103
104 [Link]()
105
106 return model
107
108
109 def plot_history(train, val, title):
110 epochs = range(len(train))
111 [Link]()
112 [Link](epochs, train, label="train")
113 [Link](epochs, val, label="val")
114 [Link](title)
115 [Link](loc="best")
116 [Link]()
117 [Link](1.0)
118
119
120 def classify_images(fn_arr, model):
121 for fn in fn_arr:
122 path = "datasets/" + fn
123
124 # because we train using grayscale image, we need to convert
125 # the sample image using color_mode="grayscale"
126 img = keras_image.load_img(path, target_size=(28, 28),
127 color_mode="grayscale")
128 x = keras_image.img_to_array(img)
129 x = np.expand_dims(x, axis=0)
130
131 num_to_alphabet = ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I',
132 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R',
133 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z']
134 num_to_alphabet = [Link](num_to_alphabet)
135
136 image_i = [Link]([x])
137 classes = [Link](image_i, batch_size=10)
138 print(classes[0])
139 class_label = num_to_alphabet[[Link](classes[0])]
140 print(fn + " is a letter {:s}".format(class_label))
141
142
143 if __name__ == "__main__":
144 # Load the dataset
145 training_images, training_labels \
146 = get_data([Link]() + "/datasets/sign_mnist_train.csv")
147 validation_images, validation_labels \
148 = get_data([Link]() + "/datasets/sign_mnist_test.csv")
149
150 print(training_images.shape)
151 print(training_labels.shape)
152 print(validation_images.shape)
153 print(validation_labels.shape)
154
155 training_images = np.expand_dims(training_images, axis=-1)
156 validation_images = np.expand_dims(validation_images, axis=-1)
157
158 print(training_images.shape)
159 print(validation_images.shape)

53
 160
161 print([Link](training_labels), [Link](training_labels))
162 print([Link](validation_labels), [Link](validation_labels))
163
164 # Plot one of the images and its label
165 image_num = 2
166 plot_one_image(training_images[image_num, :, :, 0],
167 training_labels[image_num])
168
169 # Data pre-preprocessing with ImageDataGenerator
170 training_generator, validation_generator \
171 = do_data_preprocessing(training_images, training_labels,
172 validation_images, validation_labels)
173
174 # Build a CNN model
175 cnn_model = create_cnn_model()
176
177 history = cnn_model.fit(
178 training_generator,
179 # steps_per_epoch=len(training_images)/20,
180 epochs=50,
181 validation_data=validation_generator,
182 # validation_steps=len(validation_images)/20
183 )
184
185 # Evaluate the model on the validation dataset
186 cnn_model.evaluate(validation_generator)
187
188 acc = [Link]["accuracy"]
189 val_acc = [Link]["val_accuracy"]
190 loss = [Link]["loss"]
191 val_loss = [Link]["val_loss"]
192
193 plot_history(acc, val_acc, "Training and validation accuracy")
194 plot_history(loss, val_loss, "Training and validation loss")
195
196 # Test on sample images
197 fn_arr = ["alphabet-letter-C-1298289_640.png",
198 "alphabet-letter-D-1298315_640.png",
199 "alphabet-letter-Y-1298311_640.png",
200 "sign-language-letter-A-28717_640.png"]
201 classify_images(fn_arr, cnn_model)

If we run the listing program above, we will get the following result for accuracy and loss curve for
training and validation datasets. We can see clearly that the curve of the validation dataset is
slightly higher in accuracy and slightly lower in loss. This indicates that our model is underfitting.
The model is not complex enough to capture all the features in the images. For TensorFlow
Developer Certification, having a result like these figures, is more than enough. But, you are free to
explore how to build a complex model to resolve this issue.

54
e. Classifying Images of Rock, Paper, Scissors Game
For this section, all codes inside the following listing are the same as Listing 3.4. The dataset is the
images of hand signs of rock, paper, scissors game. You can download the dataset in here:

● [training dataset]
[Link]
● [validation dataset]
[Link]
● [testing dataset]
[Link]

Unzip/extract that downloaded dataset inside the “dataset” directory in your PyCharm project. The
flow of the program is like before. We load the dataset, do preprocessing, build a CNN model, fit the
model to the datasets, plot accuracy and loss for training and validation datasets, and testing to
predict images of rocks, papers, or scissors.

Listing 3.5. rock_paper_scissors_classifier.py

1 import os
2 import [Link] as plt
3 import [Link] as mpimg
4 import numpy as np
5 import tensorflow as tf
6
7 from [Link] import RMSprop
8 from [Link] import image as keras_image
9 from [Link] import ImageDataGenerator
10
11
12 def do_data_preprocessing():
13 rock_dir = [Link]("datasets/rps/rock")
14 paper_dir = [Link]("datasets/rps/paper")
15 scissors_dir = [Link]("datasets/rps/scissors")
16
17 print("Total training rock images", len([Link](rock_dir)))
18 print("Total training paper images", len([Link](paper_dir)))
19 print("Total training scissors images", len([Link](scissors_dir)))
20
21 rock_files = [Link](rock_dir)
22 paper_files = [Link](paper_dir)
23 scissors_files = [Link](scissors_dir)
24

55
25 print(rock_files[:10])
26 print(paper_files[:10])
27 print(scissors_files[:10])
28
29 TRAINING_DIR = "datasets/rps"
30 training_datagen = ImageDataGenerator(
31 rescale=1./255,
32 rotation_range=40,
33 width_shift_range=0.2,
34 height_shift_range=0.2,
35 shear_range=0.2,
36 zoom_range=0.2,
37 horizontal_flip=True,
38 fill_mode="nearest"
39 )
40
41 VALIDATION_DIR = "datasets/rps-test-set"
42 validation_datagen = ImageDataGenerator(rescale=1./255)
43
44 train_generator = training_datagen.flow_from_directory(
45 TRAINING_DIR,
46 target_size=(150, 150),
47 class_mode="categorical",
48 batch_size=32
49 )
50
51 validation_generator = validation_datagen.flow_from_directory(
52 VALIDATION_DIR,
53 target_size=(150, 150),
54 class_mode="categorical",
55 batch_size=32
56 )
57
58 return train_generator, validation_generator
59
60
61 def create_cnn_model():
62 model = [Link]([
63 [Link].Conv2D(64, (3, 3), activation="relu", input_shape=(150, 150, 3)),
64 [Link].MaxPooling2D(2, 2),
65 [Link].Conv2D(64, (3, 3), activation="relu"),
66 [Link].MaxPooling2D(2, 2),
67 [Link].Conv2D(128, (3, 3), activation="relu"),
68 [Link].MaxPooling2D(2, 2),
69 [Link].Conv2D(128, (3, 3), activation="relu"),
70 [Link].MaxPooling2D(2, 2),
71 [Link](),
72 [Link](0.5),
73 [Link](512, activation="relu"),
74 [Link](3, activation="softmax")
75 ])
76
77 [Link]()
78
79 [Link](loss="categorical_crossentropy",
80 optimizer=RMSprop(learning_rate=0.001),
81 metrics=["accuracy"])
82
83 return model
84
85
86 def plot_sample_images(rock_dir, rock_files, paper_dir, paper_files,
87 scissors_dir, scissors_files):
88
89 # Parameter for our graph; we'll output images in a 4x4 configuration
90 nrows = 6
91 ncols = 4
92
93 # Index for iterating over images
94 pic_index = 0

56
 95
96 # Set up matplotlib fig, and size it to fit 4x4 pics
97 fig = [Link]()
98 fig.set_size_inches(ncols * 4, nrows * 4)
99
100 pic_index += 8
101 next_rock_pix = [[Link](rock_dir, fname)
102 for fname in rock_files[pic_index-8:pic_index]]
103 next_paper_pix = [[Link](paper_dir, fname)
104 for fname in paper_dir[pic_index-8:pic_index]]
105 next_scissors_pix = [[Link](scissors_dir, fname)
106 for fname in scissors_files[pic_index-8:pic_index]]
107
108 for i, img_path in enumerate(next_rock_pix + next_paper_pix + next_scissors_pix):
109 # Set up subplot; subplot indices start at 1
110 sp = [Link](nrows, ncols, i + 1)
111 [Link]("off") # Don't show axes (or gridlines)
112
113 img = [Link](img_path)
114 [Link](img)
115
116 [Link]()
117 [Link](1.0)
118
119
120 def plot_history(train, val, title):
121 epochs = range(len(train))
122 [Link]()
123 [Link](epochs, train, label="train")
124 [Link](epochs, val, label="val")
125 [Link](title)
126 [Link](loc="best")
127 [Link]()
128 [Link](1.0)
129
130
131 def predicting_images(fn_arr, model):
132
133 for fn in fn_arr:
134 path = "datasets/" + fn
135 img = keras_image.load_img(path, target_size=(150, 150))
136 x = keras_image.img_to_array(img)
137 x = np.expand_dims(x, axis=0)
138
139 image_i = [Link]([x])
140 classes = [Link](image_i, batch_size=10)
141 classes_name = [Link](["paper", "rock", "scissors"])
142 print(fn, "is", classes_name[[Link]([int(round(p)) for p in classes[0]]) == 1])
143 print(classes)
144
145
146 if __name__ == "__main__":
147
148 train_generator, validation_generator = do_data_preprocessing()
149
150 cnn_model = create_cnn_model()
151
152 history = cnn_model.fit(
153 train_generator,
154 epochs=25,
155 #steps_per_epoch=20,
156 validation_data=validation_generator,
157 #validation_steps=3)
158 )
159
160 #[Link]("model-saved/rps.h5")
161 cnn_model.save("model-saved/categ3.h5")
162
163 acc = [Link]["accuracy"]
164 val_acc = [Link]["val_accuracy"]

57
 165 loss = [Link]["loss"]
166 val_loss = [Link]["val_loss"]
167
168 plot_history(acc, val_acc, "Training and validation accuracy")
169 plot_history(loss, val_loss, "Training and validation loss")
170
171 fn_arr = [Link]("datasets/rps-predict")
172 fn_arr = [[Link]("rps-predict", fname) for fname in fn_arr]
173 print(fn_arr[0])
174
175 predicting_images(fn_arr, cnn_model)

4. Natural Language Processing in TensorFlow

Before we jump into the program, first we need to understand what LSTM (Long Short-Term
Memory) architecture is. Unfortunately, before we explain LSTM, we need to know the Recurrent
Neural Network or RNN. Now, let us start with the RNN. This kind of network is like a generalization
of computational graphs where we have recurrent relations inside the architecture as we unravel the
input or output sequence. In the following figure, we summarize all types of base recurrent neural
network architectures.

From the above figure, the horizontal arrow shows the recurrence relation. These horizontal arrows
show that there are values (hidden values) which have been passing through the weight matrices
along the horizontal arrows. In recurrent neural networks, you can feed the model with a single input
or sequence of inputs. And at the end of the layers, you can obtain a single output or sequence of
outputs. In the outgoing arrows of the weight matrix above, we didn’t show the hidden layers. Most
of the time until the end of this short book, we only use sequence to sequence of recurrent neural
networks. The following figure is the comparison between CNN and RNN on how they handle input
data and compute hidden output.

58
In RNN, we concatenate matrix input x t +1 and hidden matrix ht to compute ht +1. Explicitly, the basic
RNN is defined as the formula in the right part of the above figure. De-reconstruct this formula into
computational graphs give us the following figure

The RNN architecture got its name because we need to compute recurrence relation of h as it is
shown in the equation of the figure above. Next, we move to LSTM.

The LSTM architecture is an extension of RNN but with several twists that can handle issues in
basic RNN. The reason why people move from a basic RNN to LSTM, because RNN has a risk of
exploding and vanishing gradient when we compute the gradient, ∂ L/∂ h0 , through backpropagation
from the output layer to the first hidden layer. Here Lis a loss of the training data. In the
backpropagation procedure, the computation of the gradient will compound up because of several
times multiplication by weight matrixW . To resolve this exploding/vanishing gradient, they built RNN
architecture such that it has a memory matrix to “remember” what was learned by the weight matrix
W in a single unit of LSTM cell. The following figure is a computational graph representation of a
single LSTM cell.

59
At first glance, it is a little bit daunting how many elements in the above computational graph
interacted. The big difference between the above computational graph of LSTM to the simple RNN
is in how they used activation function. The core computational process is handled by how we used
matrix f, i, d, and o. They have special names and have the following interpretation

● Matrix f is a forget gate.

It governs whether to erase the memory matrix or not.

● Matrix i is an input gate.

It governs whether to write a hidden value to the memory matrix or not.

● Matrix d is a density gate.

It governs how much the density of information to store in the memory matrix.

● Matrix o is an output gate.

It governs whether to recall the memory or not.

In the next sections and later chapter we will use the fancier version of LSTM which bi-directional
LSTM using the TensorFlow module. This kind of LSTM will try to learn the input sequence forward
and backward. So we will have two layers of weight matrix each for forward and backward. The
output of those forward and backward learning. For completeness, we show in the figure below bi-
directional LSTM.

60
a. Detecting sarcasm in News Headlines with LSTM and CNN
First, download the dataset in the following link [Link]
[Link]/[Link]. The dataset has

● three keys: article_link (string), headline (string), and is_sarcastic (boolean in

integer, 0 or 1).
● 26,709 items
● All headlines are in English.

The following figure is the simple schematic of how the input is processed inside the architecture
that we will build.

61
In the first tensor block (1x120x1) represents a single sentence (in here a single headline news).
The first axis represents the batch size. In total there are 32 batches, but we only showed a single
batch. The second axis represents the timesteps or along the words in a single sentence. The last
axis is the feature. Here the feature is only a single scalar where the range is in the size of
vocabulary size. The vocabulary size is the number of unique words.

After that we stack an embedding layer that will convert for each word in the sentence to be a 16-
dimensional vector. We can see in the above figure, the embedding layer will learn learning
variables such that can create a semantic meaning to each word based on their neighbours in their
host sentence. A full description how this embedding layer works and how to construct such a
lookup table can be read in (Mikolov, et. al., 2013).

Because we set 16 embedding dimensions, now the words in each sentence will have a
representation of 16 features. Then the bi-directional LSTM will learn these sentences in a forward
and backward direction. The hidden output of bi-directional LSTM will be concatenated along the
last axis which are their features. The next two layers are a classifier using 2 fully-connected layers.

Listing 4.1a. headline_news_sarcasm_classifier.py

1 import json
2 import numpy as np
3 import [Link] as plt
4
5 import tensorflow as tf
6
7 from [Link] import Adam
8 from [Link] import Tokenizer
9 from [Link] import pad_sequences
10
11
12 def do_data_preprocessing(dataset_path, vocab_size, max_length,
13 padding_type, trunc_type):
14 oov_tok = "<OOV>" # out of vocabulary token
15 training_size = 20_000 # in total there are 26,709 items
16
17 with open(dataset_path, "r") as f:
18 datastore = [Link](f)
19
20 sentences = []
21 labels = []
22
23 for item in datastore:
24 [Link](item["headline"])
25 [Link](item["is_sarcastic"])
26
27 training_sentences = sentences[:training_size]
28 validation_sentences = sentences[training_size:]
29 training_labels = labels[:training_size]
30 validation_labels = labels[training_size:]
31
32 tokenizer = Tokenizer(num_words=vocab_size, oov_token=oov_tok)
33 tokenizer.fit_on_texts(training_sentences)
34
35 word_index = tokenizer.word_index
36 print(word_index)
37
38 training_sequences = tokenizer.texts_to_sequences(training_sentences)
39 training_padded = pad_sequences(training_sequences,
40 maxlen=max_length,
41 padding=padding_type,
42 truncating=trunc_type)
43 training_padded = [Link](training_padded)
44 training_labels = [Link](training_labels)
45

62
 46 validation_sequences = tokenizer.texts_to_sequences(validation_sentences)
47 validation_padded = pad_sequences(validation_sequences,
48 maxlen=max_length,
49 padding=padding_type,
50 truncating=trunc_type)
51 validation_padded = [Link](validation_padded)
52 validation_labels = [Link](validation_labels)
53
54
55 return training_padded, training_labels, \
56 validation_padded, validation_labels, tokenizer
57
58
59 def create_lstm_model(vocab_size, embedding_dim, max_length):
60 model = [Link]([
61 [Link](vocab_size, embedding_dim, input_length=max_length),
62 [Link]([Link](32)),
63 [Link](24, activation="relu"),
64 [Link](1, activation="sigmoid")
65 ])
66
67 [Link](loss="binary_crossentropy",
68 optimizer=Adam(learning_rate=0.001),
69 metrics=["accuracy"])
70
71 [Link]()
72
73 return model
74
75
76 def plot_history(train, val, title):
77 epochs = range(len(train))
78 [Link]()
79 [Link](epochs, train, label="train")
80 [Link](epochs, val, label="val")
81 [Link](title)
82 [Link](loc="best")
83 [Link](1.0)
84
85
86 def classify_headlines(headline_arr, model, tokenizer, max_length,
87 padding_type, trunc_type):
88 for headline in headline_arr:
89 test_sequence = tokenizer.texts_to_sequences(headline)
90 test_padded = pad_sequences(test_sequence,
91 maxlen=max_length,
92 padding=padding_type,
93 truncating=trunc_type)
94 test_padded = [Link](test_padded)
95
96 classes = [Link](test_padded)
97 print(classes[0])
98 if classes[0] > 0.5:
99 print(headline + ": a sarcasm")
100 else:
101 print(headline + ": not a sarcasm")
102
103
104 if __name__ == "__main__":
105 print(tf.__version__)
106
107 dataset_path = "datasets/[Link]"
108 vocab_size = 1_000
109 embedding_dim = 16
110 max_length = 120
111 trunc_type = "post"
112 padding_type = "post"
113
114 training_padded, training_labels, validation_padded, validation_labels, \
115 tokenizer = do_data_preprocessing(dataset_path, vocab_size, max_length,

63
 116 trunc_type, padding_type)
117
118 lstm_model = create_lstm_model(vocab_size, embedding_dim, max_length)
119
120 history = lstm_model.fit(
121 training_padded,
122 training_labels,
123 epochs=100,
124 validation_data=(validation_padded, validation_labels),
125 verbose=1
126 )
127
128 lstm_model.save("model-saved/categ4.h5")
129
130 acc = [Link]["accuracy"]
131 val_acc = [Link]["val_accuracy"]
132 loss = [Link]["loss"]
133 val_loss = [Link]["val_loss"]
134
135 plot_history(acc, val_acc, "Training and validation accuracy")
136 plot_history(loss, val_loss, "Training and validation loss")
137
138 # Test sample headlines
139 headline_arr = ["Tokyo's COVID-19 cases hit all-time high despite state of emergency",
140 "China: No need for 2nd WHO probe on virus origin",
141 "Turkey builds border wall to block Afghan migrants",
142 "Line accounts of Taiwan officials hacked"]
143
144 classify_headlines(headline_arr, lstm_model, tokenizer, max_length,
145 padding_type, trunc_type)

The structure of the listing above is similar to what we have done for CNN. We start from
do_data_processing in lines 12-56. We want to explain more how to represent the sentences
into a sequence of integers. This kind of process is so-called tokenizing. Tokenizing is one of the
numerous ways to encode text into a sequence of numbers.

The dataset is in JSON (JavaScript Object Notation) format, then we need to import the json
module in line 1. We load the data using a standard syntax: with open(<path_to_file>,
<flag>) as <filename>. Then we read each line in the datastore variable the headlines and
their corresponding sarcastic status. In lines 27-30, we divide our dataset into training and validation
dataset as well as their labels. The training_sentences (and validation_sentences) variable
is a list of strings where each string is the headline.

Starting from line 32, we do tokenizing. To do that, we define an instance of the Tokenizer class
with arguments num_words and oov_token. If we set num_of_words=n, then it dictates how
many first n-1 words, which have been sorted from the most highest occurring to the least
occurring words, should be used in the method .texts_to_sequences() of the tokenizer object.
oov_token serves as a string token to identify a word which is not associated with the vocabulary
(out of vocabulary) that we will build using the tokenizer object.

In line 33, we build the vocabulary, the collection of all unique words, by fitting this tokenizer object
to all the words in training sentences. This .fit_on_texts() method automatically creates a
dictionary which is sorted from the most occurring words in training sentences to the least occurring
words where the key is a word and the value is an index starting from 2 to the number of unique
words in all sentences of the training dataset. Indices 0 and 1 are reserved for padding index and
oov_token index 1. Now we are ready to map each word in a sentence headline into a sequence of

1 [Link]
64
integers in training and validation dataset. This can be done in a vectorized way by calling the
method .texts_to_sequences in line 38. Because the resulting sequence for each sentence
does not have the same length, we need to pad and truncate such that all the sequence from all the
sentences have the same length where it is set by argument maxlen in function pad_sequence.
We also set the padding and truncating process at the end of the sequence. We can see that we
have set this type of padding and truncating by variables trunc_type and padding_type in lines
111-112. Finally, we turn all the training sentences and labels into a NumPy array for optimal
operations. We do the same thing for the validation dataset. The following figure may help to
understand quickly the explanation above

In lines 59-73, we construct a bidirectional LSTM model with the initial layer as an embedding layer.
This layer serves as a converter to the training sequences of integers into smaller lengths of its
sequence. This embedding also has a training parameter which accounts for the similarity of two
sentences where each sentence is represented by a vector with smaller dimension than the length
of training sequences. For a complete description how this embedding layer works, see (Rong,
2016).

Running the above listing, we will achieve high accuracy and low loss for the training dataset, but
unfortunately, we get a poor result in validation dataset as indicated by the following figure

65
Another version of this classifier is to use CNN instead of LSTM. To do that, we can change the
architecture’s layers create_lstm_model() in lines 59-73 by the following
create_cnn_model() function and also the corresponding lines which invoke that function.

Listing 4.1b. headline_news_sarcasm_cnn_classifier.py (not complete code)

1 def create_cnn_model(vocab_size, embedding_dim, max_length):
2 model = [Link]([
3 [Link](vocab_size, embedding_dim, input_length=max_length),
4 [Link].Conv1D(128, 5, activation="relu"),
5 [Link].GlobalAveragePooling1D(),
6 [Link](24, activation="relu"),
7 [Link](1, activation="sigmoid")
8 ])
9
10 [Link](loss="binary_crossentropy",
11 optimizer=Adam(learning_rate=0.001),
12 metrics=["accuracy"])
13
14 [Link]()
15
16 return model

b. Exploring BBC News Data

Now, we explore a different format dataset. For this case, we use the BBC News dataset. You can
download the dataset, a first sample of the news, and stopwords in the following link:
[Link] .

Here is the specification of the BBC News Data:

● It contains two columns. The first column is the category of the news and the second column
is the text containing the news. There is also a header (categ,text) in the first row.
● Total news is 2,225 where the category span from 5 topics (tech, entertainment,
sport, politics, and business)

In this section, we will create a function to read this dataset such that the dataset is ready for
training the model. We will use the same technique that has been described in the previous section
(you can review Listing 4.1) which is using a Tokenizer class from TensorFlow. We also introduce
how to use stopwords.

66
Listing 4.2. data_preprocessing_bbc_news.py
1 import csv
2
3 from [Link] import Tokenizer
4 from [Link] import pad_sequences
5
6
7 def read_js_stopword(filename):
8 with open(filename, "r") as f:
9 data = [Link]()
10
11 stopwords = [Link]("\n")[-3]
12 stopwords = [word[1:-1] for word in stopwords[10:-3].split(", ")]
13 # print(stopwords)
14 return stopwords
15
16
17 def read_test_text(file_path):
18 with open(file_path, "r") as f:
19 test_text = [Link]()
20
21 return test_text
22
23
24 if __name__ == "__main__":
25 stopwords = read_js_stopword("datasets/[Link]")
26
27 # Load text
28 sentences = []
29 label = []
30
31 with open("datasets/[Link]", "r") as csvfile:
32 datastore = [Link](csvfile)
33 datastore.__next__() # skip the first line
34 for item in datastore:
35 text = [word for word in item[1].split() if word not in stopwords]
36 [Link](" ".join(text))
37 [Link](item[0])
38
39 print(f"len(sentences): {len(sentences)}")
40 test_sent = read_test_text("datasets/[Link]")
41
42 print(f"sentences[0] == test_sent => {sentences[0] == test_sent}")
43 print(f"label[:10] = {label[:10]}")
44
45 # Fit sentences to create tokens
46 sent_tokenizer = Tokenizer(oov_token="<OOV>")
47 sent_tokenizer.fit_on_texts(sentences)
48 word_index = sent_tokenizer.word_index
49
50 print(f"len(word_index): {len(word_index)}")
51
52 # Tokenizing the sentences
53 sequences = sent_tokenizer.texts_to_sequences(sentences)
54 padded = pad_sequences(sequences, padding="post")
55 print(f"padded[0]: {padded[0]}")
56 print(f"[Link]: {[Link]}")
57
58 # Fit Tokenizer to sentences and create a sequence of token for labels
59 label_tokenizer = Tokenizer()
60 label_tokenizer.fit_on_texts(label)
61 label_word_index = label_tokenizer.word_index
62 label_seq = label_tokenizer.texts_to_sequences(label)
63
64 print(f"label_seq[:10]: {label_seq[:10]}")
65 print(f"label_word_index: {label_word_index}")

67
In lines 7-14, we create a function read_js_stopword() to read [Link]. These
stopwords are a list of English words: “a”, “about”, “above”, “after”, …, “yourselves”. In lines 31-37,
we open the dataset [Link] and read each line then store it into sentences and label
variables. As a sanity check, we verify that the resulting reading process is correct by comparing
manually with the first sample of the news in line 40. We also do the same procedure like in the
Listing 4.1 that we tokenize the text of news and its category in lines 46-62.

In the later section, we will classify this news into several topics using bi-directional LSTM.

c. Classifying BBC News into topics

In this section, we extend Listing 4.2 by adding a classifier. This classifier consists of:

● an embedding layer that turns a sequence with max_length=500 into a vector with length
embedding_dim=8.
● a bi-directional LSTM layer with 4 channels (filters or depths).
● a drop-out layer to capture ensemble models and avoid overfitting.
● two-fully connected layers with activation functions: ReLU and softmax.

Listing 4.3. bbc_news_classifier.py

1 import csv
2 import numpy as np
3 import [Link] as plt
4 import tensorflow as tf
5
6 from [Link] import Adam
7 from [Link] import Tokenizer
8 from [Link] import pad_sequences
9
10
11 def read_js_stopword(filename):
12 with open(filename, "r") as f:
13 data = [Link]()
14
15 stopwords = [Link]("\n")[-3]
16 stopwords = [word[1:-1] for word in stopwords[10:-3].split(", ")]
17 return stopwords
18
19
20 def read_test_text(file_path):
21 with open(file_path, "r") as f:
22 test_text = [Link]()
23
24 return test_text
25
26
27 def read_sample_text(file_path):
28 with open(file_path, "r") as f:
29 data = [Link]()
30
31 test_text = []
32 test_label = []
33 for x in data:
34 idx = [Link]("];")
35 test_text.append(x[idx+2:])
36 test_label.append(x[1:idx])
37
38 return test_text, test_label
39

68
 40
41 def do_data_preprocessing(sentences, label, training_portion, vocab_size,
42 oov_tok, padding_type, trunc_type, max_length):
43
44 train_size = int(len(sentences) * training_portion)
45 train_sentences = sentences[:train_size]
46 train_labels = label[:train_size]
47
48 validation_sentences = sentences[train_size:]
49 validation_labels = label[train_size:]
50
51 # Fit sentences to create tokens
52 tokenizer = Tokenizer(num_words=vocab_size, oov_token=oov_tok)
53 tokenizer.fit_on_texts(train_sentences)
54 word_index = tokenizer.word_index
55 print(f"len(word_index): {len(word_index)}")
56
57 # Tokenizing the sentences
58 train_sequences = tokenizer.texts_to_sequences(train_sentences)
59 train_padded = pad_sequences(train_sequences,
60 padding=padding_type,
61 truncating=trunc_type,
62 maxlen=max_length)
63
64 validation_sequences = tokenizer.texts_to_sequences(validation_sentences)
65 validation_padded = pad_sequences(validation_sequences,
66 padding=padding_type,
67 truncating=trunc_type,
68 maxlen=max_length)
69
70 # Fit Tokenizer to sentences and create a sequence of token for labels
71 label_tokenizer = Tokenizer()
72 label_tokenizer.fit_on_texts(label)
73
74 training_label_seq = [Link](label_tokenizer.texts_to_sequences(train_labels))
75 validation_label_seq = [Link](label_tokenizer.texts_to_sequences(validation_labels))
76
77 return train_padded, training_label_seq, \
78 validation_padded, validation_label_seq, \
79 tokenizer, label_tokenizer
80
81
82 def create_language_model(vocab_size, embedding_dim, max_length):
83 model = [Link]([
84 [Link](vocab_size, embedding_dim, input_length=max_length),
85 [Link]([Link](4)),
86 [Link](0.5),
87 [Link](32, activation="relu"),
88 [Link](6, activation="softmax") # zero index is for padding
89 ])
90
91 [Link](loss="sparse_categorical_crossentropy",
92 optimizer=Adam(learning_rate=0.001),
93 metrics=["accuracy"])
94
95 [Link]()
96
97 return model
98
99
100 def plot_history(train, val, title):
101 epochs = range(len(train))
102 [Link]()
103 [Link](epochs, train, label="train")
104 [Link](epochs, val, label="val")
105 [Link](title)
106 [Link](loc="best")
107 [Link](1.0)
108
109

69
110 def classify_news(news_arr, true_label_arr, model, tokenizer, label_tokenizer, max_length,
111 padding_type, trunc_type):
112
113 reverse_label_index = dict([(v, k) for (k, v)
114 in label_tokenizer.word_index.items()])
115 print(f"reverse_label_index: {reverse_label_index}")
116
117 for i, news in enumerate(news_arr):
118 test_sequence = tokenizer.texts_to_sequences([news])
119 test_padded = pad_sequences(test_sequence, padding=padding_type,
120 maxlen=max_length, truncating=trunc_type)
121 test_padded_numpy = [Link](test_padded)
122
123 classes = [Link](test_padded_numpy)
124 get_class_key = [Link](classes[0])
125 class_label = reverse_label_index[get_class_key]
126 print(f"{news[:200]}: {class_label} [true label: {true_label_arr[i]}]")
127
128
129 if __name__ == "__main__":
130 vocab_size = 10_000
131 embedding_dim = 8
132 max_length = 500
133 trunc_type = "post"
134 padding_type = "post"
135 oov_tok = "<OOV>"
136 training_portion = 0.8
137
138 stopwords = read_js_stopword("datasets/[Link]")
139
140 # Load text
141 sentences = []
142 label = []
143 with open("datasets/[Link]", "r") as csvfile:
144 datastore = [Link](csvfile)
145 datastore.__next__() # skip the first line
146 for row in datastore:
147 [Link](row[0])
148 sentence = row[1]
149 for word in stopwords:
150 token = " " + word + " "
151 sentence = [Link](token, " ")
152
153 [Link](sentence)
154
155 # Data pre-processing
156 train_padded, training_label_seq, \
157 validation_padded, validation_label_seq, \
158 tokenizer, label_tokenizer \
159 = do_data_preprocessing(sentences, label, training_portion, vocab_size,
160 oov_tok, padding_type, trunc_type, max_length)
161
162 # Build model
163 lstm_model = create_language_model(vocab_size, embedding_dim, max_length)
164
165 history = lstm_model.fit(
166 train_padded,
167 training_label_seq,
168 epochs=50,
169 validation_data=(validation_padded, validation_label_seq),
170 verbose=1
171 )
172
173 acc = [Link]["accuracy"]
174 val_acc = [Link]["val_accuracy"]
175 loss = [Link]["loss"]
176 val_loss = [Link]["val_loss"]
177
178 plot_history(acc, val_acc, "Training and validation accuracy")
179 plot_history(loss, val_loss, "Training and validation loss")

70
 180
181 # Test sample headlines
182 news_arr, true_label_arr = read_sample_text("datasets/[Link]")
183 print(f"len(sample_sentences): {len(news_arr)}")
184 classify_news(news_arr, true_label_arr, lstm_model, tokenizer, label_tokenizer,
185 max_length, padding_type, trunc_type)

Running the above program, we will achieve a good classifier as indicated by the figure below. In
lines 186, we test this model to 15 news which are 3 news for each category. Sometimes it could
classify correctly, sometimes it does not. For this short introduction to NLP, we do not pursue the
further setting on how to improve this classifier to be a perfect classifier for the BBC News dataset.
In TensorFlow Developer Certification, we only need to achieve highest accuracy in training and/or
validation dataset.

d. Classifying IMDb Reviews Data

Now, we try to classify IMDb (Internet Movie Database) reviews data. This dataset is already pre-
built inside TensorFlow. So we don’t need to download, but we have to install the
tensorflow_datasets package. You have to make sure that you have installed the correct
version where it depends on the version of the other packages. Here is the short listed description
of IMDb reviews data

● It contains 25,000 items of training dataset and 25,000 items of testing dataset.
● Total number of vocabulary words is around 8k. In Listing 4.3, we use
imdb_reviews/subwords8k. This means that we do not separate vocabulary by a unique
word as one unit word, but use a subword. We will explain later.
● There are two labels for each review which are negative review (labeled with integer = 0)
and positive (labeled with integer = 1)

The classifier that we want to build is a classifier to know the sentiment for a given review whether it
is a positive or negative review. We also use 4 weighted layers: embedding, bi-directional LSTM,
and two fully-connected layers. The first layer is to learn the similarity between any two reviews. The
second layer will learn the pattern in the sequence representation of sentences. The third and fourth
layers are the nonlinear classifier function to map from a high dimensional input vector into a single
output of number.

71
Listing 4.4. imdb_reviews_subwords8k_classifier.py
1 import numpy as np
2 import [Link] as plt
3 import tensorflow_datasets as tfds
4 import tensorflow as tf
5
6 from [Link] import Adam
7
8
9 def do_data_preprocessing(train_dataset, validation_dataset, info,
10 BUFFER_SIZE, BATCH_SIZE):
11
12 tokenizer = [Link]["text"].encoder
13
14 train_dataset = train_dataset.shuffle(BUFFER_SIZE)
15 train_dataset_padded = train_dataset.padded_batch(
16 BATCH_SIZE,
17 [Link].get_output_shapes(train_dataset)
18 )
19 validation_dataset_padded = validation_dataset.padded_batch(
20 BATCH_SIZE,
21 [Link].get_output_shapes(validation_dataset)
22 )
23
24 return train_dataset_padded, validation_dataset_padded, tokenizer
25
26
27 def create_lstm_model(tokenizer):
28 model = [Link]([
29 [Link](tokenizer.vocab_size, 64),
30 [Link]([Link](64)),
31 [Link](64, activation="relu"),
32 [Link](1, activation="sigmoid")
33 ])
34
35 [Link]()
36
37 [Link](
38 loss="binary_crossentropy",
39 optimizer=Adam(learning_rate=0.001),
40 metrics=["accuracy"]
41 )
42
43 return model
44
45
46 def plot_history(train, val, title):
47 epochs = range(len(train))
48 [Link]()
49 [Link](epochs, train, label="train")
50 [Link](epochs, val, label="val")
51 [Link](title)
52 [Link](loc="best")
53 [Link](1.0)
54
55
56 def classify_reviews(test_reviews_path, model):
57 with open(test_reviews_path, "r") as f:
58 sample_reviews = [Link]()
59
60 sample_reviews = sample_reviews.split("\n")
61
62 for review in sample_reviews:
63 review_padded = [Link](review)
64 review_padded = [Link].from_tensors(review_padded)
65 review_padded = review_padded.padded_batch(1)
66
67 classes = [Link](review_padded)

72
 68 print(f"classes[0]: {classes[0]}")
69 if classes[0] > 0.5:
70 print(f"[positive]\n{review[:100]}...")
71 else:
72 print(f"[negative]\n{review[:100]}...")
73
74
75 if __name__ == "__main__":
76 print(tf.__version__)
77
78 # Get the data
79 dataset, info = [Link]("imdb_reviews/subwords8k", with_info=True, as_supervised=True)
80 train_dataset, validation_dataset = dataset["train"], dataset["test"]
81
82 BUFFER_SIZE = 10_000
83 BATCH_SIZE = 64
84
85 # Pre-process the dataset
86 train_dataset_padded, validation_dataset_padded, tokenizer\
87 = do_data_preprocessing(train_dataset, validation_dataset,
88 info, BUFFER_SIZE, BATCH_SIZE)
89
90 # Build a LSTM model
91 lstm_model = create_lstm_model(tokenizer)
92
93 # Train the model to the training dataset
94 history = lstm_model.fit(
95 train_dataset_padded,
96 epochs=10,
97 validation_data=validation_dataset_padded
98 )
99
100 acc = [Link]["accuracy"]
101 val_acc = [Link]["val_accuracy"]
102 loss = [Link]["loss"]
103 val_loss = [Link]["val_loss"]
104
105 plot_history(acc, val_acc, "Training and validation accuracy")
106 plot_history(loss, val_loss, "Training and validation loss")
107
108 # Test sample reviews
109 classify_reviews("datasets/[Link]", lstm_model)

In lines 9-24, we define a function for doing data preprocessing. First, we get a
SubwordTextEncoder class in line 12. This will be helpful to encode a text in the later section of
the code. In line 14, we shuffle our training dataset to make sure all the positive and negative
reviews are distributed uniformly while training the dataset. To do that we use the .shuffle()
method. This shuffling uses a buffer for each sampled review that we take. And replaced the taken
review from the buffer by the next (BUFFER_SIZE+1)-th review from the training dataset.
In lines 15-17, we use the .padded_batch method to give a zero padded integers for each batch
in the dataset. The length for each sequence in each batch will be determined by the longest
sequence in each batch. For the case of listing above, in total there would be
[Link](25,000/BATCH_SIZE)=391 batches. We do the same procedure of padding the
sequence of validation dataset in lines 19-22.

For the rest of the lines, we do again the similar procedures: build model, train model, plot the
accuracy and loss, then finally test the model to the sample reviews. Running the listing above we
will obtain the following figure

73
Because of the huge amount of dataset in IMDb review data, the LSTM model tends to have a long
computational time for the training process. We only set small epochs and one layer bi-directional
LSTM. The reader is suggested to try with different hyperparameters such as number of filters,
learning rate, adding another LSTM layer, and adding a pooling layer.

e. Poem Generation with Bi-Directional LSTM

Now, we are in the last section of this chapter. This section is the most fun application of the
machine learning model to text generation, especially to create a poem which can fool lay people
and obviously show the simple example of the Turing test. To do that we will use 154
Shakespeare’s sonnets which are concatenated to each other to build a dataset containing 2,158
rows (154 sonnets x 14 rows/sonnets + 2 rows).

The machine learning model that we employ incorporates several layers which are:

● an embedding layer with an output vector is 100 in length.

● a bi-directional LSTM layer with 150 channels.
● a dropout layer where the dropout ratio is 0.2.
● another LSTM layer with 100 channels.
● two fully-connected layers with the former layer using kernel regularizer L2 norm.

We train the model for each line in Shakespeare’s sonnets. From that we will give a feed of
sentences then try to generate n words that will come from the feed of the input. You can download
the dataset in this link:
[Link] We also provide
poems by one of the greatest Indonesian poets, Chairil Anwar. You can download his poem in this
link: [Link]

Listing 4.5. [Link]

1 import numpy as np
2 import [Link] as plt
3
4 import tensorflow as tf
5 import [Link] as keras_utils
6
7 from [Link] import Adam
8 from [Link] import pad_sequences
9 from [Link] import Tokenizer

74
10 from [Link] import regularizers
11
12
13 def do_data_preprocessing(data_path):
14 with open(data_path, "r") as f:
15 data = [Link]()
16
17 corpus = [Link]().split("\n")
18
19 tokenizer = Tokenizer()
20 tokenizer.fit_on_texts(corpus)
21 total_words = len(tokenizer.word_index) + 1
22
23 # create input sequence using list of tokens
24 input_sequences = []
25 for line in corpus:
26 token_list = tokenizer.texts_to_sequences([line])[0]
27 for i in range(1, len(token_list)):
28 n_gram_sequence = token_list[:i+1]
29 input_sequences.append(n_gram_sequence)
30
31 # pad sequences
32 max_sequence_len = max([len(x) for x in input_sequences])
33 input_sequences = [Link](pad_sequences(input_sequences,
34 maxlen=max_sequence_len,
35 padding="pre"))
36
37 # create predictors and label
38 predictors, labels = input_sequences[:, :-1], input_sequences[:, -1]
39
40 label_categ = keras_utils.to_categorical(labels, num_classes=total_words)
41
42 return predictors, label_categ, total_words, max_sequence_len, tokenizer
43
44
45 def create_language_model(total_words, ):
46 model = [Link]([
47 [Link](total_words, 100, input_length=max_sequence_len-1),
48 [Link]([Link](150, return_sequences=True)),
49 [Link](0.2),
50 [Link](100),
51 [Link](total_words/2, activation="relu",
52 kernel_regularizer=regularizers.l2(0.01)),
53 [Link](total_words, activation="softmax")
54 ])
55
56 [Link](loss="categorical_crossentropy",
57 optimizer=Adam(learning_rate=0.001),
58 metrics=["accuracy"])
59
60 [Link]()
61
62 return model
63
64
65 def plot_history(train, title):
66 epochs = range(len(train))
67 [Link]()
68 [Link](epochs, train)
69 [Link](title)
70 [Link](1.0)
71
72
73 def generate_text(sample_text, next_words, tokenizer, max_sequence_len, model):
74 for _ in range(next_words):
75 token_list = tokenizer.texts_to_sequences([sample_text])[0]
76 token_list = pad_sequences([token_list], maxlen=max_sequence_len-1, padding="pre")
77 predicted = [Link]([Link](token_list, verbose=1), axis=-1)
78 output_word = ""
79 for word, index in tokenizer.word_index.items():

75
 80 if index == predicted:
81 output_word = word
82 break
83 sample_text += " " + output_word
84 print(sample_text)
85
86
87 if __name__ == "__main__":
88
89 # data_path = "datasets/[Link]"
90 data_path = "datasets/[Link]"
91 predictors, label_categ, total_words, \
92 max_sequence_len, tokenizer = do_data_preprocessing(data_path)
93
94 # Build a language model
95 lstm_model = create_language_model(total_words)
96
97 history = lstm_model.fit(predictors, label_categ, epochs=100, verbose=1)
98
99 acc = [Link]["accuracy"]
100 loss = [Link]["loss"]
101
102 plot_history(acc, "Training accuracy")
103 plot_history(loss, "Training loss")
104
105 sample_text = "Help me Obi Wan Kenobi, you're my only hope"
106 # sample_text = "saat kenyataan hanyalah bayangan dan semu" # indonesian words
107 next_words = 100
108
109 generate_text(sample_text, next_words, tokenizer, max_sequence_len, lstm_model)

76
5. Sequence, Time Series and Prediction
For this chapter, we only focus on a single dataset which is a sunspot dataset. This dataset is
collected by SIDC (Solar Influences Data Analysis Center). The simplified version of this dataset
can be downloaded from Kaggle ([Link] but you can
download in here [Link]

The specification of this dataset is listed in the following:

● First row of this dataset contains a header with three columns. The first column is an empty
label, the second column is “Date”, and the third column is “Monthly Mean Total Sunspot
Number”.
● First column contains numbers starting from 0 to 3,234. This shows that there are 3235
records of monthly mean total sunspot numbers.
● Second column contains the date of measurements starting from 1749-01-31 until 2018-07-
31
● Third column is the monthly mean total sunspot number. The values of this column have the
range from 0.0 to 398.2

The prediction based on this dataset has a great application to spot the sun’s behaviour such that
the dangerous magnetic storm can be avoided and save many electrical installations around the
world. There is a stunning video explaining the importance of this prediction in Kurzgesagt channel:
[Link]

Another far-fetched application of this chapter is to predict the behaviour of the market using
machine learning. This has been practised by numerous quants. If you want to be rich in a clever
way and in disguise, you should go deeper to understand these techniques and apply to any data
that has relation to the market movement. At this moment, we do not pursue and discuss this
application.

a. Searching a periodic pattern in the sunspot dataset

In this section, we will employ scipy and pandas (Python Data Analysis Library)packages. Please
install those packages before starting this section. The following code snippet imports necessary
packages.

Snippet 5.1
1 import pandas as pd
2 import [Link] as plt
3 from [Link] import autocorrelation_plot
4 from scipy import signal

The first package is useful for manipulating data in a table form, formally a DataFrame. If you are
familiar with SQL, this pandas package is a similar tool to do data analysis. The first part of this
section we want to extract a hidden pattern inside the sunspot dataset. To do that, we need a
powerful function in signal processing, autocorrelation. For a given discrete time series or signal Y t ,
autocorrelation_plot2, will compute firstly, the following value

2 You can verify this formula is what pandas implemented in its source code. See [Link]
dev/pandas/blob/master/pandas/plotting/_matplotlib/[Link]#L445-L475
77
 N−h
1
N
∑ ( Y t−Y ) ( Y t+ h−Y ) ,
t=1
r (h)=
❑

where h is a lag between two signals. The first signal is obtained by windowing the signal original
signal from Y 1to Y N−h and the second signal is obtained by windowing signal from Y h +1to Y N . Those
signals have the same length in time. The numerator of the above equation describes how likely (or
correlated) those two signals are. The denominator is a normalization factor to get the output value
for r (h) between -1 and 1. The last module that is imported is [Link] where
.find_peaks() function is in there. We use this function to get the peak of the output of
autocorrelation.

The first thing every data scientist does is to print or plot the data. The following snippet helps us to
do that. In Line 1, we call a function read_csv() from pandas. In the first argument of this
function, we only need to provide the path to the .csv file that we want to read. In the second
argument, we set a keyword argument parse_date to select which column to be parsed as a date.
The third argument is to set which column to be an index column. The index column is a column
that will serve as a counter for each row in the DataFrame. We save that DataFrame object into a
variable df, then we set the frequency or the smallest unit for each step between rows in DataFrame
in .asfreq(“1M”). “1M” here means that we set the frequency at a month. This setting follows the
fact that the sunspot dataset is measured in each month.

Instead of printing all the rows, we can use methods .head() and .tail() to print only a few beginning
rows and last rows. From the output, we have an understanding that we correctly read the data and
are ready for the next step which is plotting.

Snippet 5.2
1 df = pd.read_csv("datasets/[Link]", parse_dates=["Date"], index_col="Date")
2 series = df["Monthly Mean Total Sunspot Number"].asfreq("1M")
3 print(f"[Link]:\n{[Link]()}")
4 print(f"[Link]:\n{[Link]()}")

[Link]:
Date
1749-01-31 96.7
1749-02-28 104.3
1749-03-31 116.7
1749-04-30 92.8
1749-05-31 141.7
Freq: M, Name: Monthly Mean Total Sunspot Number, dtype: float64
[Link]:
Date
2018-03-31 2.5
2018-04-30 8.9
2018-05-31 13.2
2018-06-30 15.9
2018-07-31 1.6
Freq: M, Name: Monthly Mean Total Sunspot Number, dtype: float64

Fortunately, pandas also has a plotting method which uses some of its plotting codes inherited from
matplotlib packages. Then we can integrate some matplotlib code inside the pandas plotting
method. In the following snippet, we plot the sunspot dataset.

78
Snippet 5.3
1 fig, ax = [Link](figsize=(18, 5))
2 [Link](ax=ax)
3 ax.set_ylabel("monthly mean \ntotal sunspot number")
4 [Link](True)
5 ax.set_title("[Link]")
6 plt.subplots_adjust(top=0.88, bottom=0.14, left=0.075, right=0.955)
7 [Link](1.0)

And it produces the following plot

We can notice that there is a periodic pattern which is hindered by some noises. Let’s take a closer
look by plotting the dataset from “1996-07-31” to the end of the dataset. You only need to change
Line 2 in the above snippet from [Link](ax=ax) into series[“1996-07-
31”:].plot(ax=ax). We can do this because we have set the index column as the date column.
By providing the right format of a date, we can do slicing or querying by putting any possible date
indices inside the dataset. The figure below is the result

We clearly see on this dataset, there are two oscillations between the date from 1996-07-31 to
2018-07-31. In astronomy terms, it is called “11-year solar cycles”. Now we apply
autocorrelation_plot() to this dataset. It will produce a periodic smooth version of these
sunspots datasets can be achieved by using autocorrelation.

Snippet 5.4
1 def r_autocorr(n, h, series, series_mean, c0):
2 return ((series[: n-h] - series_mean) * (series[h:] - series_mean)).sum() / (n
3 * c0)
4
5 def autocorr(series):
6 n = len(series)
7 series = [Link](series)

79
 8
9 series_mean = [Link](series)
10 c0 = [Link]((series - series_mean)**2) / n
11
12 x = [Link](n) + 1
13 y = [Link]([r_autocorr(n, h, series, series_mean, c0) for h in x])
14
15 return x, ya
16
17
18 fig, ax = [Link](figsize=(11, 7))
19 autocorrelation_plot(series, ax=ax)
20
21 x_lag_manual, y_autocorr_manual = autocorr(series)
22 x_peaks, _ = signal.find_peaks(y_autocorr_manual, distance=100)
23 [Link](x_lag_manual[x_peaks], y_autocorr_manual[x_peaks],
24 marker='o', color="white", edgecolor="red", linewidths=2)
25
26 ax.set_xlabel("h, lag")
27 ax.set_ylabel("r(h)")
28 [Link](True)
29 ax.set_title("[Link]")
30 plt.tight_layout()
31 [Link]()
32 [Link](1.0)
33
34 solar_cycle = [Link]([Link](x_lag_manual[x_peaks]))
35 print(f"solar_cycle: {solar_cycle:.2f} months = {solar_cycle/12:.2f} years")

solar_cycle: 125.67 months = 10.47 years

The figure below is the result of the above snippet.

The peak positions of the above autocorrelation curve indicated by red circles can be used to
calculate solar cycle. We have done that in the above snippet. There are two horizontal lines in the
above x-axis and two others below x-axis. Those lines indicate 95% confidence intervals (black solid
line) and 99% confidence interval (black dashed line). These confidence intervals show that for the

80
new outcome of the dataset of two signals not correlated only happens in the small interval (less
than 0.1 and greater than 0.1). The complete listing is given in the below

Listing 5.1. sunspots_dataset_exploration.py

1 import numpy as np
2 import pandas as pd
3 import [Link] as plt
4 from [Link] import autocorrelation_plot
5 from scipy import signal
6
7
8 def r_autocorr(n, h, series, series_mean, c0):
9 return ((series[: n-h] - series_mean) * (series[h:] - series_mean)).sum() / (n * c0)
10
11
12 def autocorr(series):
13 n = len(series)
14 series = [Link](series)
15
16 series_mean = [Link](series)
17 c0 = [Link]((series - series_mean)**2) / n
18
19 x = [Link](n) + 1
20 y = [Link]([r_autocorr(n, h, series, series_mean, c0) for h in x])
21
22 return x, y
23
24
25 if __name__ == "__main__":
26
27 df = pd.read_csv("datasets/[Link]",
28 parse_dates=["Date"],
29 index_col="Date")
30 series = df["Monthly Mean Total Sunspot Number"].asfreq("1M")
31 print(f"[Link]:\n{[Link]()}")
32 print(f"[Link]:\n{[Link]()}")
33
34 # -- dataset plot
35 fig, ax = [Link](figsize=(12, 3))
36 [Link](ax=ax)
37 ax.set_ylabel("monthly mean \ntotal sunspot number")
38 [Link](True)
39 ax.set_title("[Link]")
40 plt.subplots_adjust(top=0.88, bottom=0.185, left=0.080, right=0.955)
41 [Link](1.0)
42
43 # -- dataset zoom-in plot
44 fig, ax = [Link](figsize=(12, 3))
45 series["1996-07-31":].plot(ax=ax)
46 ax.set_ylabel("monthly mean \ntotal sunspot number")
47 [Link](True)
48 ax.set_title("[Link]")
49 plt.subplots_adjust(top=0.88, bottom=0.185, left=0.080, right=0.955)
50 [Link](1.0)
51 # [Link]()
52
53 # -- get auto-correlation plot
54 fig, ax = [Link](figsize=(11, 7))
55 autocorrelation_plot(series, ax=ax)
56
57 x_lag_manual, y_autocorr_manual = autocorr(series)
58 x_peaks, _ = signal.find_peaks(y_autocorr_manual, distance=100)
59 [Link](x_lag_manual[x_peaks], y_autocorr_manual[x_peaks],
60 marker='o', color="white", edgecolor="red", linewidths=2)
61
62 ax.set_xlabel("h, lag")
63 ax.set_ylabel("r(h)")
64 [Link](True)

81
 65 ax.set_title("[Link]")
66 plt.tight_layout()
67 [Link](1.0)
68
69 # -- compute solar cycle
70 solar_cycle = [Link]([Link](x_lag_manual[x_peaks]))
71 print(f"solar_cycle: {solar_cycle:.2f} months = {solar_cycle/12:.2f} years")

b. Using DNN as a model predictor

Now, we want to create a predictor for the sunspots datasets. Before we start to write the full listing,
we start from importing necessary packages and function definitions layout. In the below snippet,
we construct a function to read the dataset and make a plot of the time series of sunspots dataset.

Snippet 5.5
1 import sys
2 import csv
3 import numpy as np
4 import [Link] as plt
5 import tensorflow as tf
6
7
8 def read_sunspots_csv(filename):
9 time_step = []
10 sunspots = []
11 with open(filename) as csvfile:
12 reader = [Link](csvfile, delimiter=",")
13 next(reader)
14 for row in reader:
15 [Link](float(row[2]))
16 time_step.append(int(row[0]))
17
18 return [Link](time_step), [Link](sunspots)
19
20
21 def plot_series(time, series, format="-", start=0, end=None, label=None,
22 figsize=(10, 6), xy_axis_labels=["x", "y"]):
23 [Link](figsize=figsize)
24 [Link](time[start:end], series[start:end], format, label=label)
25 [Link](xy_axis_labels[0])
26 [Link](xy_axis_labels[1])
27 [Link]()
28 plt.tight_layout()
29 [Link](1.0)
30
31
32 if __name__ == "__main__":
33 time, series = read_sunspots_csv("datasets/[Link]")
34
35 plot_series(time, series,
36 xy_axis_labels=["month-th in integer",
37 "month mean \ntotal sunspot number"])

In lines 1-2, we import csv (for importing the .csv file format of the [Link]), then we also
need the sys package to print out the process prediction in the later snippet. And the rest of three
packages are mandatory if you want to explore the dataset efficiently in an array, to plot it, or to
make a prediction model.
82
Lines 8-18, we define a function to read the .csv file of the dataset. First we prepared two empty
lists: time_step and sunspots. Then we use the standard syntax in Python to open a file: with
open(filename) as csvfile. We use the variable csvfile as the input of the csvfile reader
object where it will automatically return a list of tuples with triplet elements from each column in the
dataset: index, Date, Monthly Mean Total Sunspot Number. In line 13, we skip the
reader then we iterate through each tuple in the reader and append the corresponding variables:
index and sunspot’s numbers to the variable time_step and sunspots. Finally this function will
return two numpy arrays of time step and sunspot’s numbers.

In the next lines 21-28, we define a plot_series() function to make a plot the sunspots dataset. In
spite of providing the standard input data for x axis and y axis, we can also set the format of the line
plot in argument format, slicing where the data that we want to plot using arguments start and
end. And the last argument is for setting the size of the plot figure. Lines 23-28 are explaining
themselves what the syntaxes are.

Running the above snippet, we obtain the following figure. In this figure, the x-ticks labels are not
printed in the date format but in the integer of nth-month from zero.

The next snippet is a function to define an input for our prediction model. In line 2, we create a
Dataset object from the series then we generate many datasets from this Dataset object with
window_size + 1, shift=1, and drop_remainder=True. We need to add 1 to windows_size to
set the last index as a label. The following illustration describes this syntax with a series containing
5 elements.

83
Because our model only processes input of one dataset, we need to convert for each Datasets
above into a single dataset. This can be done by syntax .flat_map as indicated in line 7. The
following illustration explains that process.

Next, we separate each Tensor object above into input and label by slicing for input in window[:-
1] and its label window[:-1]. We also shuffle first before slicing them. The following illustration
describes the shuffling and separation.

84
Lines 6 and 10 are some avoidances of [Link] issue in TensorFlow3. Finally we return the
dataset in batches and use prefetch to avoid out of memory. This line we pick 32 batches and
process each batch (.prefetch(1)) in the training process. If we set .prefetch(2), we prefetch 2
batches.

After line 16: if __name__ == “__main__”:, we specify how long our training and validation
dataset. In this case, we set 3,000 months and the rest of 234 months are validation dataset. In
lines 30-31, we invoke the windowed_datasets() function with those specified settings.

Snippet 5.6
1 def windowed_dataset(series, window_size, batch_size=32, shuffle_buffer=32):
2 dataset = [Link].from_tensor_slices(series)
3 dataset = [Link](window_size + 1, shift=1, drop_remainder=True)
4
5 window_batch = lambda window: [Link](window_size + 1)
6 window_batch = [Link].do_not_convert(window_batch)
7 dataset = dataset.flat_map(window_batch)
8
9 separation = lambda window: (window[:-1], window[-1:])
10 separation = [Link].do_not_convert(separation)
11 dataset = [Link](shuffle_buffer).map(separation)
12
13 return [Link](batch_size).prefetch(1)
14
15
16 if __name__ == "__main__":
17 # <some code omitted; see previous snippets>
18
19 split_time = 3_000
20 time_train = time[:split_time]
21 x_train = series[:split_time]
22 time_validation = time[split_time:]
23 x_validation = series[split_time:]
24
25 window_size = 60
26 batch_size = 32
27 shuffle_buffer_size = 1_000
28
29 # data pre-processing
30 dataset = windowed_dataset(x_train, window_size, batch_size=batch_size,
31 shuffle_buffer=shuffle_buffer_size)

Next step is building the model and training it. We use a simple prediction model consisting of three
fully-connected layers. And we do not specify the activation function of the last layer. The reason is
because we are interested in how the output value is not in the categorical label as if we use
sigmoid or softmax activation function. Please refer to the chapter 02 of linear classifiers if you need
more review about this simple model.

For this model, we use a stochastic gradient descent optimizer [Link] with
learning rate 1e-7 (quite small to the default value 1e-3) and momentum 0.9. These two values are
hyperparameters. If you want to optimize these values, you need to add another pipeline (or in the
form of a callback function). We do not pursue this technique at this moment. The loss function is
the mean absolute error in which we compute the average over all items (a windowed dataset with
length equal to 60) the difference between predicted number of sunspots with their true values.

3 [Link]
85
The last two lines in the Snippet 5.7 are to create the DNN model and train this model with
epochs=100.

Snippet 5.7
1 def create_dnn_model():
2
3 model = [Link]([
4 [Link](20, input_shape=[window_size], activation="relu"),
5 [Link](10, activation="relu"),
6 [Link](1)
7 ])
8
9 [Link]()
10
11 [Link](loss="mae",
12 optimizer=[Link](lr=1e-7, momentum=0.9))
13
14 return model
15
16
17 if __name__ == "__main__":
18 # <some code omitted; see previous snippets>
19
20 # Train using DNN
21 dnn_model = create_dnn_model()
22 dnn_model.fit(dataset, epochs=100)

Model: "sequential
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense (Dense) (None, 20) 1220
_________________________________________________________________
dense_1 (Dense) (None, 10) 210
_________________________________________________________________
dense_2 (Dense) (None, 1) 11
=================================================================
Total params: 1,441
Trainable params: 1,441
Non-trainable params: 0
_________________________________________________________________
Epoch 1/100
92/92 [==============================] - 1s 2ms/step - loss: 54.0309
Epoch 2/100
92/92 [==============================] - 0s 3ms/step - loss: 51.0969
Epoch 3/100
92/92 [==============================] - 0s 3ms/step - loss: 48.365
...
<some epochs omitted>
...
Epoch 100/100
92/92 [==============================] - 0s 3ms/step - loss: 23.5977

In the last epoch, we got MAE around 23-24 sunspots which is really small compared to the range
of the measurements between 0 to 400 sunspots. Now we move to the final step which is doing the
prediction. The following snippet does that prediction

86
In line 5, we create an empty list to store the predicted data. The subsequent lines 6-13 can be
explained easily through the following diagram with an example 16 items

For the calculation of MAE, we only compute the validation dataset.

Snippet 5.7
1 if __name__ == "__main__":
2 # <some code omitted; see previous snippets>
3
4 # Make a prediction
5 forecast = []
6 time_end = len(series) - window_size
7 for time in range(time_end):
8 [Link](dnn_model.predict(series[time:time + window_size][[Link]]))
9 [Link](f"\rpreprocessing: {(time + 1)/float(time_end) * 100:5.2f}%")
10 [Link]()
11
12 forecast = forecast[split_time-window_size:]
13 results = [Link](forecast)[:, 0, 0]
14
15 plot_series(time_validation, x_validation, label="actual",
16 figsize=(13, 5),
17 xy_axis_labels=["month-th in integer", "month mean \ntotal sunspot number"])
18 [Link](time_validation, results, label="forecast")

87
 19 [Link](loc="upper left", bbox_to_anchor=(1.01, 1.), borderaxespad=0)
20 plt.tight_layout()
21 [Link](1.0)
22
23 print("")
24 predicted_MAE = [Link].mean_absolute_error(x_validation, results)
25 print(f"predicted MAE: {predicted_MAE.numpy()}")

...
<a Matplotlib figure will pop up>
...
predicted MAE: 18.48443031311035

Running the above snippet returns the following figure and predicted MAE.

The above snippets are summarized in the following listing.

Listing 5.2. sunspots_dnn_predictor.py

1 import sys
2 import csv
3 import numpy as np
4 import [Link] as plt
5 import tensorflow as tf
6
7
8 def plot_series(time, series, format="-", start=0, end=None, label=None,
9 figsize=(10, 6), xy_axis_labels=["x", "y"]):
10 [Link](figsize=figsize)
11 [Link](time[start:end], series[start:end], format, label=label)
12 [Link](xy_axis_labels[0])
13 [Link](xy_axis_labels[1])
14 [Link]()
15 [Link](1.0)
16 plt.tight_layout()
17
18
19 def read_sunspots_csv(filename):
20 time_step = []
21 sunspots = []
22 with open(filename) as csvfile:
23 reader = [Link](csvfile, delimiter=",")
24 next(reader)
25 for row in reader:
26 [Link](float(row[2]))
27 time_step.append(int(row[0]))
28
29 return [Link](time_step), [Link](sunspots)

88
30
31
32 def windowed_dataset(series, window_size, batch_size=32, shuffle_buffer=32):
33 dataset = [Link].from_tensor_slices(series)
34 dataset = [Link](window_size + 1, shift=1, drop_remainder=True)
35
36 window_batch = lambda window: [Link](window_size + 1)
37 window_batch = [Link].do_not_convert(window_batch)
38 dataset = dataset.flat_map(window_batch)
39
40 separation = lambda window: (window[:-1], window[-1:])
41 separation = [Link].do_not_convert(separation)
42 dataset = [Link](shuffle_buffer).map(separation)
43
44 return [Link](batch_size).prefetch(1)
45
46
47 def create_dnn_model():
48
49 model = [Link]([
50 [Link](20, input_shape=[window_size], activation="relu"),
51 [Link](10, activation="relu"),
52 [Link](1)
53 ])
54
55 [Link]()
56
57 [Link](loss="mae",
58 optimizer=[Link](lr=1e-7, momentum=0.9))
59
60 return model
61
62
63 if __name__ == "__main__":
64
65 time, series = read_sunspots_csv("datasets/[Link]")
66
67 plot_series(time, series,
68 xy_axis_labels=["month-th in integer", "month mean \ntotal sunspot number"])
69
70 split_time = 3_000
71 time_train = time[:split_time]
72 x_train = series[:split_time]
73 time_validation = time[split_time:]
74 x_validation = series[split_time:]
75
76 window_size = 60
77 batch_size = 32
78 shuffle_buffer_size = 1_000
79
80 # data pre-processing
81 dataset = windowed_dataset(x_train, window_size, batch_size=batch_size,
82 shuffle_buffer=shuffle_buffer_size)
83
84 # Train using DNN
85 dnn_model = create_dnn_model()
86 dnn_model.fit(dataset, epochs=100)
87
88 # Make a prediction
89 forecast = []
90 time_end = len(series) - window_size
91 for time in range(time_end):
92 [Link](dnn_model.predict(series[time:time + window_size][[Link]]))
93 [Link](f"\rpreprocessing: {(time + 1)/float(time_end) * 100:5.2f}%")
94 [Link]()
95
96 forecast = forecast[split_time-window_size:]
97 results = [Link](forecast)[:, 0, 0]
98
99 plot_series(time_validation, x_validation, label="actual",

89
 100 figsize=(13, 5),
101 xy_axis_labels=["month-th in integer", "month mean \ntotal sunspot number"])
102 [Link](time_validation, results, label="forecast")
103 [Link](loc="upper left", bbox_to_anchor=(1.01, 1.), borderaxespad=0)
104 plt.tight_layout()
105 [Link](1.0)
106
107 # possible MAE = [15.359105, 14.897159, 15.164404]
108 print("")
109 predicted_MAE = [Link].mean_absolute_error(x_validation, results)
110 print(f"predicted MAE: {predicted_MAE.numpy()}")

c. Using bi-directional LSTM as a model predictor

In this section we apply bi-directional LSTM layer as our model predictor. This section is completely
similar to the previous section, but we only change the function create_dnn_model() into
create_lstm_model(). We also introduce a callback function for learning rate such that it will
increase the learning after some epochs. We will build an architecture with 1 convolutional layer, 1
bidirectional LSTM layer, and 2 fully-connected layers. The reason why we use a convolutional layer
as the first layer to preserve the causal pattern of the time series. This forces us to set the padding
argument in the first layer equal to ”causal”. This padding setting will put zeros in the beginning of
the sequence to preserve the output with the same size along the timestep/window to the input
sequence. This architecture is quite similar to the previous chapter when we processed the
sequence of text. The figure below shows that architecture.

All the dimensions of the tensors above are in the format (batch x timestep/window x
feature/units). In the above figure, we only show the process of one item in the dataset. The
following snippet is the correspondence function definition of the above architecture

Snippet 5.8
1 def create_lstm_model(learning_rate=1e-8):
2 model = [Link]([
3 [Link].Conv1D(filters=32, kernel_size=5, strides=1,
4 padding="causal", activation="relu",
5 input_shape=[None, 1]),
6 [Link]([Link](64, return_sequences=True)),
7 [Link](30, activation="relu"),
8 [Link](1)
9 ])

90
 10
11 [Link]()
12
13 [Link](loss="mae",
14 optimizer=[Link](lr=learning_rate, momentum=0.9))
15 return model

Our loss function is the mean absolute error which is not very sensitive to the outliers since our
dataset mostly has the difference of the numbers of sunspots between each month quite small (you
can make a plot of this difference between months by using [Link](1)). We also put the
learning rate as the argument because we want to have flexibility to change the learning of our
model. In the optimizer argument we set momentum to be 0.9. You can interpret superficially this
momentum argument as a “physical momentum” to the optimizing process such that when the
weight matrix / learning variables falls into the local minimum, it will be “kicked out” from this local
minimum to find another minimum.

Before we do training on the dataset, we create a LearningRateScheduler object from

[Link]. We can set this scheduler using the lambda function where the input
argument is an epoch. So we have our fitting process like the following snippet

Snippet 5.9
1 if __name__ == "__main__":
2 #<some statements are omitted>
3
4 lr_schedule = [Link](
5 lambda epoch: 1e-8 * 10 ** (epoch / 20))
6 history = lstm_model.fit(dataset, epochs=100, callbacks=[lr_schedule])
7
8 # plot the loss as a function of learning rate
9 [Link](figsize=(13, 5))
10 [Link]([Link]["lr"], [Link]["loss"])
11
12 [Link]("learning rate")
13 [Link]("loss")
14 plt.tight_layout()
15 [Link](1.0)

The scheduler means that every 20 epochs we reduce the learning by a factor of 10 starting from
initial learning rate 1e-8. So the final learning rate will be 1e-3. In lines 9-15, we plot the loss as a
function of learning rate in the semilog plot along the x-axis. If we run our model with this scheduler,
we have the following curve. We could gain a best choice to our learning rate by picking at the
steepest gradient which is around 7e-5.

91
Setting a new learning rate to be 7e-5, we can achieve a faster convergence of the loss function. To
do a re-training, we have to call [Link].clear_session(). Then we re-create the
lstm_model and re-train it. In the following snippet, we modify plot_history that we have
created in the previous chapters to plot only one curve.

Snippet 5.10
1 def plot_history(loss, start=0, end=None):
2 epochs = range(len(loss))
3 [Link](figsize=(13,5))
4 [Link](epochs[start:end], loss[start:end])
5 [Link]("Epochs")
6 [Link]("MAE")
7 plt.tight_layout()
8 [Link](1.0)
9
10
11 if __name__ == "__main__":
12 #<some statements are omitted>
13
14 loss = [Link]["loss"]
15 plot_history(loss)

The output of the above snippet is the figure below. We can infer from the figure that around
epoch=20, we have already arrived with no more significant reduction in MAE.

The final snippet is creating a plot of the prediction. First we need to define a function to take a
validation dataset from the series variable and change it into a [Link] object such that it
matches with the input of our model. This function is similar to the function windowed_dataset().

Snippet 5.11
1 def model_forecast(model, series, window_size, batch_size=32):
2 ds = [Link].from_tensor_slices(series)
3 ds = [Link](window_size, shift=1, drop_remainder=True)
4
5 window_batch = lambda window: [Link](window_size)
6 window_batch = [Link].do_not_convert(window_batch)
7 ds = ds.flat_map(window_batch)
8 ds = [Link](batch_size).prefetch(1)
9 return [Link](ds)
10
11
12 if __name__ == "__main__":

92
 13 #<some statements are omitted>
14
15 # Make a prediction
16 rnn_forecast = model_forecast(lstm_model, series[..., [Link]], window_size)
17 rnn_forecast = rnn_forecast[split_time - window_size:-1, -1, 0]
18
19 plot_series(time_validation, x_validation, label="actual")
20 [Link](time_validation, rnn_forecast, label="forecast")
21 [Link](loc="upper left", bbox_to_anchor=(1.01, 1.), borderaxespad=0)
22 [Link](1.0)
23
24 print("")
25 print([Link].mean_absolute_error(x_validation, rnn_forecast).numpy())

predicted MAE: 22.115623474121094

We can see that MAE is higher than using DNN (around 18). You can beat DNN if you stack
another bi-directional LSTM alongside with the current LSTM and also another Dense layer after
Dense layer with 30 units and you will achieve MAE around 14. But this also looks counter-intuitive
since we can also do the same thing in our DNN architecture by adding more Dense layers. What
we have learned from this section is that more complex architectures do not guarantee that we will
get better results.

The complete summary of the above snippets with re-training section is available in the below
listing.

Listing 5.3. sunspots_lstm_predictor.py

1 import csv
2 import numpy as np
3 import [Link] as plt
4 import tensorflow as tf
5
6
7 def plot_series(time, series, format="-", start=0, end=None, label=None,
8 figsize=(10, 6), xy_axis_labels=["x", "y"]):
9 [Link](figsize=figsize)
10 [Link](time[start:end], series[start:end], format, label=label)
11 [Link](xy_axis_labels[0])
12 [Link](xy_axis_labels[1])
13 [Link]()
14 [Link](1.0)
15
16
17 def read_sunspots_csv(filename):
18 time_step = []
19 sunspots = []

93
20 with open(filename) as csvfile:
21 reader = [Link](csvfile, delimiter=",")
22 next(reader)
23 for row in reader:
24 [Link](float(row[2]))
25 time_step.append(int(row[0]))
26
27 return [Link](time_step), [Link](sunspots)
28
29
30 def windowed_dataset(series, window_size, batch_size=32, shuffle_buffer=32):
31 series = tf.expand_dims(series, axis=-1)
32 dataset = [Link].from_tensor_slices(series)
33 dataset = [Link](window_size + 1, shift=1, drop_remainder=True)
34
35 window_batch = lambda window: [Link](window_size + 1)
36 window_batch = [Link].do_not_convert(window_batch)
37 dataset = dataset.flat_map(window_batch)
38
39 separation = lambda window: (window[:-1], window[1:]) # because we use RNN
40 separation = [Link].do_not_convert(separation)
41 dataset = [Link](shuffle_buffer).map(separation)
42
43 return [Link](batch_size).prefetch(1)
44
45
46 def create_lstm_model(learning_rate=1e-8):
47 # comment out LSTM layer and Dense layer to achieve MAE around 14
48
49 model = [Link]([
50 [Link].Conv1D(filters=32, kernel_size=5, strides=1,
51 padding="causal", activation="relu",
52 input_shape=[None, 1]),
53 [Link]([Link](64, return_sequences=True)),
54 #[Link]([Link](64, return_sequences=True)),
55 [Link](30, activation="relu"),
56 #[Link](10, activation="relu"),
57 [Link](1)
58 ])
59
60 [Link]()
61
62 [Link](loss="mae",
63 optimizer=[Link](lr=learning_rate, momentum=0.9))
64 return model
65
66
67 def plot_history(loss, start=0, end=None):
68 epochs = range(len(loss))
69 [Link](figsize=(13,5))
70 [Link](epochs[start:end], loss[start:end])
71 [Link]("Epochs")
72 [Link]("MAE")
73 plt.tight_layout()
74 [Link](1.0)
75
76
77 def model_forecast(model, series, window_size, batch_size=32):
78 ds = [Link].from_tensor_slices(series)
79 ds = [Link](window_size, shift=1, drop_remainder=True)
80
81 window_batch = lambda window: [Link](window_size)
82 window_batch = [Link].do_not_convert(window_batch)
83 ds = ds.flat_map(window_batch)
84 ds = [Link](batch_size).prefetch(1)
85 return [Link](ds)
86
87
88 if __name__ == "__main__":
89

94
 90 time, series = read_sunspots_csv("datasets/[Link]")
91
92 # normalize series (tensorflow certification)
93 #min_series = [Link](series)
94 #max_series = [Link](series)
95 #series -= min_series
96 #series /= max_series
97
98 plot_series(time, series)
99
100 split_time = 3_000
101 time_train = time[:split_time]
102 x_train = series[:split_time]
103 time_validation = time[split_time:]
104 x_validation = series[split_time:]
105
106 window_size = 60
107 batch_size = 32
108 shuffle_buffer_size = 1_000
109
110 # Train dataset to the model (1D CNN + 1 layer bidirectional LSTM + DNN
111 #[Link].clear_session()
112 #[Link].set_seed(51)
113 #[Link](51)
114
115 dataset = windowed_dataset(x_train, window_size, batch_size=batch_size,
116 shuffle_buffer=shuffle_buffer_size)
117
118 lstm_model = create_lstm_model(learning_rate=1e-8)
119 lr_schedule = [Link](
120 lambda epoch: 1e-8 * 10 ** (epoch / 20))
121 history = lstm_model.fit(dataset, epochs=100, callbacks=[lr_schedule])
122
123 [Link](figsize=(13, 5))
124 [Link]([Link]["lr"], [Link]["loss"])
125 [Link]("learning rate")
126 [Link]("loss")
127 [Link](True)
128 plt.tight_layout()
129 [Link](1.0)
130
131 # Re-train
132 [Link].clear_session()
133 #[Link].set_seed(51)
134 #[Link](51)
135
136 #dataset = windowed_dataset(x_train, window_size, batch_size=batch_size,
137 # shuffle_buffer=shuffle_buffer_size)
138
139 lstm_model = create_lstm_model(learning_rate=5e-5)
140
141 history = lstm_model.fit(dataset, epochs=100)
142
143 lstm_model.save("model-saved/categ5.h5")
144
145 loss = [Link]["loss"]
146 plot_history(loss)
147
148 # Make a prediction
149 rnn_forecast = model_forecast(lstm_model, series[..., [Link]], window_size)
150 rnn_forecast = rnn_forecast[split_time - window_size:-1, -1, 0]
151
152 plot_series(time_validation, x_validation, label="actual", figsize=(13, 5),
153 xy_axis_labels=["month-th in integer",
154 "monthly mean\ntotal sunspot number"])
155 [Link](time_validation, rnn_forecast, label="forecast")
156 [Link](loc="upper left", bbox_to_anchor=(1.01, 1.), borderaxespad=0)
157 plt.tight_layout()
158 [Link](1.0)
159

95
160 # possible MAE = [14.636255, 14.354073, 14.655728]
167 # Second and third run is running without computing learning rate
168 # (use learning rate from the first run)
169 print("")
170 predicted_MAE = [Link].mean_absolute_error(x_validation, rnn_forecast)
171 print(f"predicted MAE: {predicted_MAE}")

96
6. Preparation and Taking the TensorFlow Developer
Certification Exam
A complete description of how to and guidelines in Bahasa is available in this link:
[Link]
edit?usp=sharing.

a. Preparation
Selected Listing program that has correspondence or similar to the problems that are questioned in
TensorFlow Developer Certification in August 2020
- Chapter 01: Listing 2.1 (Simple Regression)
- Chapter 02: Listing 2.4 (CNN - Fashion)
- Chapter 03: Listing 3.5 (Cats and Dogs with Aug)
- Chapter 04: Listing 4.1a (NLP - Sarcasm)
- Chapter 05: Listing 5.3 (Time Series - Sunspots with LSTM / CNN)

b. Purchasing the certification ticket

All the procedures follow the guidelines from TensorFlow Developer Certification Handbook. A
complete description in Bahasa is also available in the above link.

c. Taking the exam

In the top right of the PyCharm main windows, there is a button “Start Exam”.

Then, follow all the procedures in the pop-up dialog after clicking that button. This process requires
you to login into your Gmail account with a TensorFlow Developer Certification voucher that has
been bought using that Gmail account.

97
After all the preparation above, PyCharm will create a new environment for you with five directories.
Every directory represents each problem that will be tested.

98
Useful troubleshooting and Python’s commands
It is recommended to use Python 3.8

Required Python package

- numpy
- tensorflow
- tensorflow-gpu
- matplotlib
- scipy
----------------------------------//----------------------------------

Adding the following commands after importing TensorFlow packages if you want to use GPU
instead of CPU
physical_devices = [Link].list_physical_devices("GPU")
[Link].set_memory_growth(physical_devices[0], enable=True)

----------------------------------//----------------------------------

Always check the memory usage of your NVidia card through the command if you use TensorFlow
with GPU
$ nvidia-smi -l 1

----------------------------------//----------------------------------

If you use PyCharm in Linux, please install the stable version that has been released in the
beginning of the year when you installed it. The most recent version sometimes will not be
compatible with the TensorFlow version. Use the following command
$ sudo snap install pycharm-community -channel=2021.1/stable --classic.

----------------------------------//----------------------------------

If you have the following error message while running LSTM model:

NotImplementedError: Cannot convert a symbolic Tensor

(bidirectional_1/forward_lstm_1/strided_slice:0) to a numpy array. This
error may indicate that you're trying to pass a Tensor to a NumPy call,
which is not supported

Please downgrade the numpy into version 1.19.5.

----------------------------------//----------------------------------

Determine which backend is being used by matplotlib

>>> import matplotlib
>>> matplotlib.get_backend()

99
If the result is “agg”, this backend does not support for the command [Link]()

Install matplotlib backend tkinter

$ sudo apt install python3-tk

Use tkinter backend in matplotlib

>>> import matplotlib
>>> [Link](“TkAgg”)

Qt5Agg backend can be installed through Python’s package: PyQt5

To be able non-blocking figure plotting, we have to use [Link]() instead of [Link]

[Link](x_arr, y_arr)
[Link](1)

100
References
(Cord, 2016) - Deep CNN and Weak Supervision Learning for visual recognition.

(Mikolov, [Link]., 2013) - Efficient Estimation of Word Representations in Vector Space.

(Moroney, [Link]., 2020) - Coursera: [Link] TensorFlow Developer.

(Kingma and Ba, 2015) - Adam: A Method for Stochastic Optimization.

(Simonyan and Zisserman, 2014) - Very Deep Convolutional Networks for Large-Scale Image
Recognition.

(Rong, 2016) - word2vec Parameter Learning Explained

101