UNIT -I

Neural networks are a computational model that shares some properties with the animal brain in
which many simple units are working in parallel with no centralized control unit. The weights
between the units are the primary means of long-term information storage in neural networks.
Updating the weights is the primary way the neural network learns new information.

Modeling sets of equations in the form of the equation Ax = b. In the context of neural networks,
the A matrix is still the input data and the b column vector is still the labels or outcomes for each
row in the A matrix. The weights on the neural network connections becomes x (the parameter
vector)

Neural network architecture

Now that we understand the differences between the artificial neuron and the perception, we can
better understand the structure of the full multilayer feed-forward neural network. With
multilayer feed-forward neural networks, we have artificial neurons arranged into groups called
layers. Building on the layer concept, we see that the multilayer neural network has the
following:

• A single input layer

• One or many hidden layers, fully connected

• A single output layer

Input layer. This layer is how we get input data (vectors) fed into our network

Hidden layer. There are one or more hidden layers in a feed-forward neural network.

Output layer. We get the answer or prediction from our model from the output layer.
Training Neural Networks

A well-trained artificial neural network has weights that amplify the signal and dampen the
noise. A bigger weight signifies a tighter correlation between a signal and the network’s
outcome. Inputs paired with large weights will affect the network’s interpretation of the data
more than inputs paired with smaller weights.

The loss functions in optimization algorithms, such as stochastic gradient descent (SGD), reward
the network for good guesses and penalize it for bad ones.

Back propagation Learning

Back propagation is an important part of reducing error in a neural network model. To explain
back propagation, we’ll return to our discussion about how information circulates within a feed-
forward neural network

A closer look at back propagation

However, for the topic of back propagation and to better understand a core fundamental concept
that much of the book is based on, these concepts down to the notation level.

Understanding back propagation pseudo code

Network: a multilayer feed-forward neural network

• Training records: a set of training vectors with associated outputs

• Learning rate: the learning rate (sometimes denoted by the Greek alpha symbol)
Activation Functions

We use activation functions to propagate the output of one layer’s nodes forward to the next
layer (up to and including the output layer). Activation functions are a scalar-to-scalar function,
yielding the neuron’s activation. We use activation functions for hidden neurons in a neural
network to introduce nonlinearity into the network’s modeling capabilities. Many activation
functions belong to a logistic class of transforms that (when graphed) resemble an S. This class
of function is called sigmoid. The sigmoid family of functions contains several variations, one of
which is known as the sigmoid function.

Linear

A linear transform is basically the identity function, and f(x) = Wx, where the dependent variable
has a direct, proportional relationship with the independent variable. In practical terms, it means
the function passes the signal through unchanged

Sigmoid

Like all logistic transforms, sigmoid can reduce extreme values or outliers in data without
removing them. The vertical line in is the decision boundary. A sigmoid function is a machine
that converts independent variables of near infinite range into simple probabilities between 0 and
1, and most of its output will be very close to 0 or 1

Tanh

Pronounced “tanch,” tanh is a hyperbolic trigonometric function. Just as the tangent represents a
ratio between the opposite and adjacent sides of a right triangle, tanh represents the ratio of the
hyperbolic sine to the hyperbolic cosine: tanh(x) = sinh(x) / cosh(x). Unlike the Sigmoid
function, the normalized range of tanh is –1 to 1.

Hard Tanh

Similar to tanh, hard tanh simply applies hard caps to the normalized range. Any thing more than
1 is made into 1, and anything less than –1 is made into –1. This allows for a more robust
activation function that allows for a limited decision boundary

Rectified

Linear Rectified linear is a more interesting transform that activates a node only if the input is
above a certain quantity. While the input is below zero, the output is zero, but when the input
rises above a certain threshold, it has a linear relationship with the dependent variable f(x) =
max(0, x)

Loss Functions

Loss functions quantify how close a given neural network is to the ideal toward which it is
training. The idea is simple. We calculate a metric based on the error we observe in the
network’s predictions. We then aggregate these errors over the entire dataset and average them
and now we have a single number representative of how close the neural network is to its ideal

Loss Function Notation

Consider the dataset gathered to train a neural net. Let “N” denote the number of samples (set of
inputs with corresponding outcomes) that have been gathered.

Consider the nature of the input and output collected. Each data point records some set of unique
input features and output features. Let “P” denote the number of input features gathered and “M”
denote the number of output features that have been observed. We will use (X,Y) to denote the
input and output data we collected. Note that there will be N such pairs where the input is a
collection of P values and the output Y is a collection of M values. We will denote the it pair in
the dataset.

Loss Functions for Regression

Mean squared error loss .When working on a regression model that requires a real valued output,
we use the squared loss function, such like the case of ordinary least squares in linear regression.
Consider the case in which we have to predict only one output feature (M = 1).

Other loss functions for regression

Although the MSE is used widely, it is quite sensitive to outliers, and this is something that you
should consider when picking a loss function.

Hyper parameters

In machine learning, we have both model parameters and parameters we tune to make networks
train better and faster. These tuning parameters are called hyper parameters, and they deal with
controlling optimization functions and model selection during training with our learning
algorithm.

Learning Rate

The learning rate affects the amount by which you adjust parameters during optimization in order
to minimize the error of neural network’s guesses.

Regularization
Regularization helps with the effects of out-of-control parameters by using different methods to
minimize parameter size over time.

Momentum

Momentum helps the learning algorithm get out of spots in the search space where it would
otherwise become stuck. In the error scrape, it helps the updater find the galleys that lead toward
the minima.

Sparsity

The sparsity hyper parameter recognizes that for some inputs only a few features are relevant.
For example, let’s assume that a network can classify a million images. Any one of those images
will be indicated by a limited number of features.
 UNIT II

Defining Deep Learning

Our definition of deep learning, here we define the four major architectures of deep networks

• Unsupervised Pretained Networks

• Convolution Neural Networks

• Recurrent Neural Networks

• Recursive Neural Networks

Unsupervised Pretrained Networks

• Autoencoders

• Deep Belief Networks (DBNs)

• Generative Adversarial Networks (GANs)

Auto encoders

We use auto encoders to learn compressed representations of datasets. Typically, we use them to
reduce a dataset’s dimensionality. The output of the auto encoder network is a reconstruction of
the input data in the most efficient form.

Deep Belief Networks

DBNs are composed of layers of Restricted Boltzmann Machines (RBMs) for the retrain phase
and then a feed-forward network for the fine-tune phase.

Feature Extraction with RBM Layers

We use RBMs to extract higher-level features from the raw input vectors.

Learning higher-order features automatically. Learning these features in an unsupervised fashion

is considered the retrain phase of DBNs.

Initializing the feed-forward network. We then use these layers of features as the initial
weights in a traditional back propagation driven feed-forward neural network.

Fine-tuning a DBN with a feed-forward multilayer neural network. In the fine-tune phase of a
DBN we use normal back propagation with a lower learning rate to do “gentle” back propagation

Gentle back propagation. The retrain phase with RBM learns higher-order features from the
data, which we use as good initial starting values for our feed-forward network.
The output layer. The normal goal of a deep network is to learn a set of features. The first layer
of a deep network learns how to reconstruct the original dataset

Generative Adversarial Networks

A network of note is the GAN.1 GANs have been shown to be quite adept at synthesizing novel
new images based on other training images.

We can extend this concept to model other domains such as the following:

• Sound

• Video2

• Generating images from text descriptions

Training generative models, unsupervised learning, and GANs .If we had a large corpus of
training images (such as the Image Net dataset), we could build a generative neural network that
outputs images (as opposed to classifications)

Training generative models, unsupervised learning, and GANs

If we had a large corpus of training images (such as the ImageNet dataset), we could build a
generative neural network that outputs images

The discriminator network. When modeling images, the discriminator network is typically a
standard CNN

The generative network. The generative network in GANs generates data (or images) with a
special kind of layer called a deconvolutional layer

Conditional GANs Conditional GANs5 also can use class label information, allowing them to
condition ally generate data of a specific class

Comparing GANs and variational autoencoders GANs focus on trying to classify training
records as being from the model distribution or the real distribution

Convolution Neural Networks (CNNs)

The goal of a CNN is to learn higher-order features in the data via convolutions. They are well
suited to object recognition with images and consistently top image classification competitions

Intuition Feed-forward multilayer neural networks take input as a single one-dimensional vector
and transform the data with one or more hidden layers (fully connected).

CNN Architecture Overview CNNs transform the input data from the input layer through all
connected layers into a set of class scores given by the output layer.
 1. Input layer
2. Feature-extraction (learning) layers
3. Classification layer

Convolution layer We express the Rectified Linear Unit (ReLU) activation function as a
layer in the diagram here to match up to other literature. 2. Pooling layer these layers find a
number of features in the images and progressively construct higher-order features

Convolution layer hyper parameters

Following are the hyper parameters that dictate the spatial arrangement and size of the output
volume from a convolution layer are:

• Filter (or kernel) size (field size)

• Output depth
• Stride
• Zero-padding
Filter size. More on Sizing Convolution Layers .In this section, we explain how these
hyper parameters work. We lay out the tuning mechanics for CNN layers. Every filter is
small spatially with respect to the width and height of the filter size.
Output Depth. We can manually pick the depth of the output volume. The depth hyper
parameter controls the neuron count in the convolution layer that is connected to the same
region of the input volume.
Stride. Stride configures how far our sliding filter window will move per application of
the filter function.
Zero-padding. The last hyper parameter is zero-padding, with which we can control the
spatial size of the output volumes. Batch normalization and layers to accelerate training
in CNNs we can normalize the activations of the previous layer at each batch.
Pooling Layers
Pooling layers are commonly inserted between successive convolution layers. We want to
follow convolution layers with pooling layers to progressively reduce the spatial size
(width and height) of the data representation. Fully Connected Layers We uses this layer
to compute class scores that we’ll use as output of the network (e.g., the output layer at
the end of the network).

Recurrent Neural Networks

Recurrent Neural Networks are in the family of feed-forward neural networks. They are
different from other feed-forward networks in their ability to send information over time-
steps. Modeling the Time Dimension Recurrent Neural Networks are considered Turing
complete and can simulate arbitrary programs (with weights)Lost in time many
classification tools (support vector machines, logistic regression, and regular feed-
forward networks) have been applied successfully without modeling the time dimension,
assuming independence. Temporal feedback and loops in connections Recurrent Neural
Networks can have loops in the connections. This allows them to model temporal
behavior gain accuracy in domains such as time-series, language, audio, and text.
Sequences and time-series data we find sequential data in many problem domains in
industry for which our model needs to output a sequence of vectors
• Image captioning
• Speech synthesis
• Music generation
• Playing video games
• Language modeling
• Character-level text generation model.
Understanding model input and output Traditional machine learning operates on the
concept of a single fixed-sized input vector
3D Volumetric Input into Recurrent Neural Networks involves more dimensions than
standard machine learning modeling input
Uneven time-series and masking we previously described how with Recurrent Neural
Network input we have the concept of time-steps in addition to features in our input
vector.
Recursive Neural Networks
Recursive Neural Networks, like Recurrent Neural Networks, can deal with variable
length input. The primary difference is that Recurrent Neural Networks have the ability to
model the hierarchical structures in the training dataset.
Network Architecture Recursive Neural Network architecture is composed of a shared-
weight matrix and a binary tree structure that allows the recursive network to learn
varying sequences of words or parts of an image.
Varieties of Recursive Neural Networks Recursive Neural Networks come in a few
varieties. One is the recursive auto encoder. Just like its feed-forward cousin, recursive
auto encoders learn how to reconstruct the input.
Applications of Recursive Neural Networks Both Recursive and Recurrent Neural
Networks share many of the same use cases. Recurrent Neural Networks are traditionally
used in NLP because of their ties to binary trees, contexts, and natural-language-based
parsers.
Building Deep Networks
Matching Deep Networks to the Right Problem.
Deep learning is about designing the network architecture to match the problem as
opposed to hand-engineering features in the input data.
Modeling columnar data
• Modeling image data
• Modeling sequence/time-series data
• Natural Language Processing applications
Columnar Data and Multilayer Perceptrons Common columnar data has a static structure
to it, and is best modeled in DL4J by a classic multilayer perceptron neural network.
Images and Convolutional Neural Networks Convolutional Neural Networks (CNNs)
have proven to be adept at finding structure in raw image data
Using Hybrid Networks When we see data that is the combination of both time and
image (e.g., video), we use a special hybrid network of Long Short-Term Memory
(LSTM) and convolutional layers.
The DL4J Suite of Tools DL4J is a group of deep learning tools that is packaged together
as a suite to perform functions such as the following: • Integration • Vectorization •
Modeling • Evaluation.
Vectorization and DataVec Because neural networks can train only on vectors,
vectorizing data is a necessary preprocessing step.
Runtimes and ND4J ND4J is a library for scientific computing on the JVM. Its syntax
emulates that of Numpy and MATLAB.

UNIT –III
A tensor is an array of data that can be processed by Tensor Flow. A tensor can be represented
as a matrix or a vector. In simple terms, we can think of it as a collection of numbers arranged
into a particular shape. Mathematically, a tensor can be used to create n-dimensional datasets.

A zero-dimensional tensor is a scalar, which contains a single value and has no axes. A one-
dimensional tensor is a vector, which contains a list of values and has one axis. A two-
dimensional tensor is a matrix that contains values stored across two axes.

What Is Tensor Flow Used for?

Tensor Flow is used in a variety of applications, ranging from natural language processing (NLP)
and image recognition to predictive analytics and autonomous vehicle control. It can be used to
train deep neural networks for object detection and classification, generate recommendations,
classify images, and build voice-powered applications

How Does TensorFlow Work?

At the core of Tensor Flow is a dataflow graph, which describes how data moves through a series
of operations or transformations. The basic idea behind the dataflow graph is that operations are
expressed as nodes, with each node performing a single operation on its inputs. The inputs and
outputs of the operations are passed through edges (tensors). This makes it possible to break
down complex computations into smaller, more manageable chunks

Creating a variable
The Variable () constructor expects an initial value for the variable, which can be any kind or
shape of Tensor. The type and form of the variable are defined by its initial value
[Link] () constructor is used to create a variable in Tensor Flow.
Syntax

[Link](initial_value=None,trainable=None,validate_shape=True, caching_device=None,
name=None, variable_def=None, dtype=None

Tensorflow Operation
 Add

Subtract

Multiply

Divide

Square

Reshape

add two tensors using [Link](tensorB):

const tensorA = [Link]([[1, 2], [3, 4], [5, 6]]);

const tensorB = [Link]([[1,-1], [2,-2], [3,-3]]);

const tensorNew = [Link](tensorB);

subtract two tensors using [Link](tensorB):

const tensorA = [Link]([[1, 2], [3, 4], [5, 6]]);

const tensorB = [Link]([[1,-1], [2,-2], [3,-3]]);

// Tensor Subtraction
const tensorNew = [Link](tensorB);

multiply two tensors using [Link](tensorB):

const tensorA=[Link]([1, 2, 3, 4]);

const tensorB=[Link]([4, 4, 2, 2]);

const tensorNew = [Link](tensorB);

divide two tensors using [Link](tensorB):

const tensorA=[Link]([2, 4, 6, 8]);

const tensorB=[Link]([1, 2, 2, 2]);

const tensorNew = [Link](tensorB);

square a tensor using [Link]():

 const tensorA=[Link]([1, 2, 3, 4]);

const tensorNew = [Link]();

reshape a tensor using [Link]():

const tensorA=[Link]([[1, 2],[3, 4]]);

const tensorB = [Link]([4, 1]);

placeholder

A placeholder is a variable in Tensorflow to which data will be assigned sometime later on. It
enables us to create processes or operations without the requirement for data. Data is fed into
the placeholder as the session starts, and the session is run. We can feed in data into tensorflow
graphs using placeholders.

Syntax: [Link](dtype, shape=None, name=None)

 dtype: the datatype of the elements in the tensor that will be fed.
 shape : by default None. The tensor's shape that will be fed , it is an optional parameter.
One can feed a tensor of any shape if the shape isn't specified.
 name: by default None. The operation's name , optional parameter.

Session
In deep learning frameworks like Tensor Flow 1.x, a "session" serves as the execution
environment for a computational graph. It encapsulates the runtime where operations
defined in the graph are executed and tensor values are computed.

 Execution Environment:
A session provides the necessary environment to run the operations (ops) and evaluate the
tensors within a TensorFlow graph.
 Resource Management:
Sessions manage resources like memory and computational devices (CPU, GPU). They
allocate and deallocate these resources as needed during the execution of the graph.
 Graph Execution:
The [Link]() method is used to execute specific parts of the graph within the session. You can
pass tensors or operations to [Link]() to trigger their computation.
Example:
 importtensorflowastf
a=[Link](5)
b=[Link](3)
c=[Link](a,b)
[Link]()assess:
result=[Link](c)
print(result)

Sharing variable

Sharing variables in TensorFlow, particularly in TensorFlow 1.x with its graph-based execution
and variable scopes, is primarily achieved through the use of tf.variable_scope and
its reuse argument. In TensorFlow 2.x, with its emphasis on eager execution and Keras layers,
variable sharing is more implicitly handled through object-oriented programming and Keras's
built-in mechanisms.

TensorFlow 1.x (Graph Mode):

 tf.variable_scope with reuse=True: This is the most common method. You define your variables
within a tf.variable_scope and then, in another part of your code where you want to reuse those
same variables, you open the same tf.variable_scope but set reuse=True. This tells TensorFlow to
look for existing variables with the same names within that scope rather than creating new ones.
Passing Variables as Arguments: You can explicitly create [Link] objects and then pass
them as arguments to functions or classes that need to use them. This gives you direct control
over which variables are shared.
Example

importtensorflowastf

dense_layer=[Link](units=10)
output1=dense_layer([Link]([1,5]))
output2 = dense_layer([Link]([1,5]))

Graphs

TensorFlow utilizes the concept of computational graphs to represent mathematical operations

and their dependencies. These graphs are fundamental to how TensorFlow executes
computations.

 Nodes and Edges:

 A graph consists of nodes representing [Link] objects (units of computation, like addition
or matrix multiplication) and edges representing [Link] objects (the units of data that flow
between operations).
 Portability: Graphs are data structures that can be saved, loaded, and run in environments
without a Python interpreter (e.g., mobile devices, embedded systems, backend servers).
 Optimization: TensorFlow's optimization system, Grappler, can perform various
transformations on graphs to improve performance, such as constant folding, common
subexpression elimination, and parallelization.
 Execution Efficiency: Once a graph is built, it can be executed repeatedly with different
inputs, often leading to faster computations compared to executing operations individually in
eager mode.

Visualization

It supports the display of various data types, including images, text, and audio, which can be
useful for debugging input pipelines or visualizing model outputs

 Model Graph Visualization:

TensorBoard allows users to visualize the computational graph of a TensorFlow model,
showing the flow of data and operations between layers. This helps in understanding the
model's architecture and identifying potential issues with connectivity or complexity.
 Tracking Training Metrics:
Users can log and visualize scalar metrics like loss, accuracy, and custom metrics over time
during training. This provides insights into the model's performance and convergence.
 Histograms and Distributions:
TensorBoard can display histograms and distributions of weights, biases, and activations
within the network, helping to identify vanishing or exploding gradients and other training
instabilities.
 Embedding Projector:
For models that involve embeddings, TensorBoard's Embedding Projector allows for
interactive visualization of high-dimensional embeddings in lower-dimensional space,
facilitating the understanding of relationships between embedded entities.
 Image, Audio, and Text Data Visualization:
TensorBoard supports the display of various data types, enabling visual inspection of input
data, intermediate outputs, and model predictions in formats such as images, audio waveforms,
or text.