EELE 0531: Machine Learning

Lecture 8
Instructor: Waleed Ejaz
wejaz@[Link]

Slides adopted from Prof. Andrew Ng Founder of [Link], Co-founder Coursera, Adjunct Professor at Stanford University 1
Handwritten digit recognition using Forward Propagation

▪ Binary Classification problem: Is this the digit

‘0’ or ‘1’

Probability of being a handwritten ‘1’

2
Handwritten digit recognition using Forward Propagation

▪ Digit images 0 and 1

Probability of
being a
handwritten ‘1’

3
TensorFlow Implementation

4
NumPy
▪ Powerful library for numerical computations in Python.
▪ Provides support for large, multi-dimensional arrays and matrices.
▪ Offers a wide range of mathematical functions for array manipulation.
▪ Efficiently implemented in C and Fortran for optimized performance.
▪ Commonly used for tasks like:
• Linear algebra
• Fourier transforms
• Random number generation
• Image processing

5
TensorFlow
▪ Developed by Google for deep learning and machine learning tasks.
▪ Supports CPUs, GPUs, and TPUs for accelerated computations.
▪ Offers a flexible architecture for building various types of neural
networks.
▪ Provides tools for:
• Automatic differentiation
• Model optimization
• Deployment across different platforms
▪ Widely used in research and industry for applications like:
• Image recognition
• Natural language processing
• Time series analysis

6
PyTorch
▪ Developed by Facebook's AI Research lab (FAIR).
▪ Known for its dynamic computation graph and ease of use.
▪ Provides a more intuitive and Pythonic API compared to TensorFlow.
▪ Strong support for research and experimentation.
▪ Growing popularity in both academia and industry.

7
Key differences: Community and Ecosystem
▪ NumPy: Mature and stable library with a large and active
community.
▪ TensorFlow: Extensive ecosystem with numerous tools and
resources, including TensorFlow Hub, TensorFlow Serving, and
TensorFlow Lite.
▪ PyTorch: Growing community with strong support from Facebook
and a vibrant research community. PyTorch Hub provides pre-trained
models and tools.

8
Key Differences: Performance
▪ NumPy: Optimized for CPU computations.
▪ TensorFlow: Designed for both CPU and GPU acceleration.
▪ TensorFlow can significantly outperform NumPy on GPUs, especially
for large-scale deep learning models.

Introduction: 1-9
Coffee Roasting
▪ Problem: Optimizing coffee roasting
process.
▪ Parameters: Temperature and duration.
▪ Goal: Predict coffee quality based on these
parameters.
▪ Positive examples (good coffee) within a
specific temperature and duration range
(cross).
▪ Undercooked, overcooked, or poorly
roasted beans are negative examples
(circle).

10
Forward Propagation in Action
▪ Step 1: Define input feature vector x (temperature, duration).
• x=[Link]([[200.0, 17.0]])
▪ Step 2: Create hidden layer 1
• layer_1 = Dense(units=3, activation='sigmoid’)
• a1 = layer_1(x)
▪ Step 3: Compute activations:
• layer_2 = Dense(units=1, activation='sigmoid’)
• a2 = layer_2(a1)
▪ Step 4: Threshold output:
• if a2 >= 0.5:
y_hat = 1
else:
y_hat = 0

11
Handwritten digit recognition using Forward Propagation

▪ Binary Classification problem: Is this the digit

‘0’ or ‘1’

Probability of being a handwritten ‘1’

12
TensorFlow and NumPy Arrays
▪ Building neural networks requires a clear understanding of how data
is structured.
▪ NumPy and TensorFlow have slightly different conventions for data
representation.
▪ Understanding these differences is crucial for writing correct and
efficient code.

13
Feature vectors

▪ Why [[…]] (double brackets)???

14
NumPy-Matices and Vectors
▪ Matrices: 2D arrays of numbers, organized in rows and columns.
• Example: A 2x3 matrix has 2 rows and 3 columns.
• Code: x = [Link]([[1, 2, 3], [4, 5, 6]])
▪ Vectors: Special cases of matrices with one dimension equal to 1.
• Row Vector: A matrix with a single row.
• Column Vector: A matrix with a single column.

15
NumPy-1D Arrays
▪ 1D arrays are linear lists of numbers with no inherent row or column
structure.

1D vector
▪ Often used in simpler models like linear regression.
▪ TensorFlow favors matrices for efficiency with large datasets.

16
TensorFlow’s Data Structure: The Tensor
▪ Tensor: A generalization of matrices, optimized for efficient
computation.
▪ For this course, consider tensors as matrices.
▪ TensorFlow converts NumPy arrays to tensors internally for
performance.

• [Link]([[0.2, 0.7, 0.3]]) - A 1x3 tensor.

17
Activation vector

18
Bridging the Gap
▪ TensorFlow and NumPy have different internal representations of
matrices.
▪ This is due to their independent development histories.
▪ Conversion between tensors and NumPy arrays is possible.
▪ Code: [Link]() - Converts a TensorFlow tensor (a1) to a NumPy
array.

19
Neural Network Output
▪ Input features are represented as matrices in TensorFlow.
▪ Activations from each layer are also matrices (tensors).
▪ Even a single output value is technically a 1x1 matrix in TensorFlow.

20
Building Neural Network Architecture
▪ Previously

▪ String layers together to form a neural network.

▪ TensorFlow handles forward propagation automatically.

21
Building Neural Network Architecture
▪ Prepare data:

▪ Compile the model: [Link](...)

▪ Fit the model to the data: [Link](X, Y)
▪ Use [Link](X_new) for forward propagation on new data.
▪ Obtain predictions directly.

22
Simplified Model Definition
▪ Define layers directly within the Sequential function.
▪ More compact and readable code.

23
Digital Classification Example
▪ Sequential model for classifying handwritten digits.
▪ Multiple layers (e.g., input, hidden, output).

24
Understanding the Underlying Mechanisms
▪ While TensorFlow simplifies development, it's crucial to understand
how forward propagation works.
▪ You can implement forward propagation from scratch in Python.
▪ Gain deeper insights into the algorithms.

25
 Train a Neural Network in TensorFlow
▪ Given set of (x,y) examples
▪ How do you build and train this in code?

1: Specify the model

2: Compile model

3: Train model
26
TensorFlow Code
▪ TensorFlow code to define the network layers (as described in the
transcript)
▪ TensorFlow code to compile the model with binary_crossentropy
loss.
• Loss function measures the error between predictions and true labels.
• binary_crossentropy is suitable for binary classification.
▪ TensorFlow code to train the model using the fit function.
• fit function adjusts network parameters to minimize loss.
• epochs: Number of times the training algorithm sees the entire dataset.

27
Recalling Logistic Regression
f(x) = w. b

▪ Step 1: Define Model

• Logistic Regression Model
1
𝑓𝑤,𝑏 𝑥Ԧ = 𝑔 𝑤 ∙ 𝑥Ԧ + 𝑏 =
1+𝑒 −(𝑤∙𝑥+𝑏)

▪ Step 2: Define low and cost functions

𝑚
1 𝑖 𝑖
𝐽(𝑤, 𝑏) = ෍𝑦 − log 𝑓𝑤,𝑏 𝑥Ԧ − (1 − 𝑦 𝑖 ) log 1 − 𝑓𝑤,𝑏 𝑥Ԧ 𝑖
𝑚
𝑖=1
▪ Step 3: Train data to minimize cost function
• Gradient descent algorithm 𝑚
𝜕 1 𝑖 (𝑖)
𝑤𝑗 = 𝑤𝑗 − 𝛼 𝐽 𝑤, 𝑏 ෍(𝑓𝑤,𝑏 𝑥Ԧ − 𝑦 (𝑖) ) 𝑥𝑗
𝜕𝑤𝑗 𝑚
𝑖=1
Repeat until convergence 𝑚
1
𝜕 ෍(𝑓𝑤,𝑏 𝑥Ԧ 𝑖 − 𝑦 (𝑖) )
𝑏 =𝑏−𝛼 𝐽(𝑤, 𝑏) 𝑚
𝜕𝑏 𝑖=1

Simultaneous updates 28
Three Steps to Train the Neural Network
▪ Step 1: Specify the network architecture and output computation (forward
propagation).

▪ Step 2: Compile the model and define the loss function (binary cross-entropy).

▪ Step 3: Train the model using [Link]() (gradient descent or similar optimizer).

29
Step 1: Define the model
▪ This code defines the structure of the neural network, including the
number of layers, units in each layer, and the activation function
used.

30
Step 2: Define the Loss and Cost Functions
▪ Formula: Binary cross-entropy loss function:
𝑖 𝑖
𝐿(𝑓(𝑥),
Ԧ 𝑦) = 𝑦 − log 𝑓𝑤,𝑏 𝑥Ԧ − (1 − 𝑦 𝑖 ) log 1 − 𝑓𝑤,𝑏 𝑥Ԧ 𝑖

▪ This is same loss function as in logistic regression which compares

predicted value versus target value
▪ It is also known as binary cross entropy (in statistics this function is
called cross-entropy loss function).
▪ The word binary highlights that this is for a binary classification
problem.
[Link](loss='binary_crossentropy')

31
Alternative Loss Functions
▪ Example: Mean squared error (MSE) for regression problems:

[Link](loss='mean_squared_error’)
▪ Explanation: TensorFlow offers various loss functions for different
types of problems.

32
Step 3: Training The Model
▪ Gradient descent update rule:

▪ TensorFlow uses backpropagation to compute gradients.

▪ [Link]() handles the training process, including optimization.
[Link](x, y, epochs=100)

33
Epoch versus iterations
▪ Epoch: A complete cycle
• One full pass through the entire training dataset.
• Like going through a deck of cards from start to finish.
• The model learns from all training examples in one go.
▪ Iteration: A small step
• One update of the model's parameters.
• Uses a mini-batch of data for training.
• Like taking a small handful of cards and adjusting the model.
▪ Epoch and Iterations
• One epoch usually consists of multiple iterations.
• Number of iterations per epoch = Dataset size / Batch size
• Example: 1000 training examples / batch size of 100 = 10 iterations per epoch.

34
The Power of Libraries
▪ Libraries provide optimized and readily available implementations of
complex algorithms.
▪ Benefits: Efficiency, ease of use, community support.

35