📌 TensorFlow Ranks and Tensors

In TensorFlow, data is represented as tensors, which are multi-dimensional arrays. The

rank of a tensor refers to the number of dimensions (axes) it has.

Tensor Ranks:

Rank Tensor Type Example Description

Shape
0 Scalar () Single value (e.g., 5, 3.14)
1 Vector (4,) One-dimensional array (e.g., [1, 2, 3, 4])
2 Matrix (3, 3) Two-dimensional matrix (e.g., 3x3 matrix)
3 3D Tensor (3, 3, 3) Three-dimensional tensor (e.g., RGB image
with height, width, channels)
4+ Higher Dimensional (2, 4, 6, 8) Used in deep learning for batch processing,
Tensor video processing, etc.

Example in TensorFlow

import tensorflow as tf

# Scalar (Rank 0)
scalar = [Link](5)
print([Link]) # Output: ()

# Vector (Rank 1)
vector = [Link]([1, 2, 3])
print([Link]) # Output: (3,)

# Matrix (Rank 2)
matrix = [Link]([[1, 2], [3, 4]])
print([Link]) # Output: (2, 2)

# 3D Tensor (Rank 3)
tensor_3d = [Link]([[[1, 2], [3, 4]], [[5, 6], [7, 8]]])
print(tensor_3d.shape) # Output: (2, 2, 2)

📌 TensorFlow’s Computation Graphs

TensorFlow operates using computation graphs, which define the sequence of operations
before running them.

Types of Computation Graphs

1. Static Computation Graph (TF 1.x)

• Defined first and executed later.

• Required [Link]() to compute values.

2. Dynamic Computation Graph (TF 2.x, Eager Execution)

• Operations are executed immediately.

• No need to predefine the entire graph.

• More intuitive and Pythonic.

Example: Computation Graph in TensorFlow 2.x

# TensorFlow automatically builds computation graphs in the background

x = [Link](2.0)
y = [Link](3.0)
z = x * y + 2 # Computation graph: (x * y) + 2

print(z) # Output: 8.0

👉 Under the hood, TensorFlow builds a graph of dependencies automatically!

Graph Nodes: Operations (e.g., +, *)

Graph Edges: Tensors flowing through computations

📌 Variables in TensorFlow

In TensorFlow, variables are used to store mutable state (weights, biases, etc.), whereas
tensors are immutable.

Why Use Variables?

• Variables allow modifying values during training (e.g., updating weights in a neural
network).

• Unlike [Link], [Link] can be reassigned.

Creating TensorFlow Variables

# Create a variable
var = [Link](10)

# Access the value

print([Link]()) # Output: 10

# Modify the variable

[Link](20)
print([Link]()) # Output: 20

Using Variables in Computation

w = [Link](2.0)
b = [Link](1.0)

x = [Link](3.0)
y = w * x + b # y = 2 * 3 + 1 = 7

print([Link]()) # Output: 7.0

Key Differences Between [Link] and [Link]

 Feature [Link] [Link]
Mutability Immutable Mutable
Use Case Fixed values (e.g., input data) Weights, biases, trainable parameters
Assignmen ❌ Not allowed ✅ Allowed (using .assign())
t

📌 Summary

Concept Description
Ranks & Tensors TensorFlow tensors have different ranks (0: Scalars, 1: Vectors, 2:
Matrices, etc.)
Computation TensorFlow dynamically builds computation graphs for operations
Graphs
Variables Unlike tensors, variables can be updated and are used for trainable
parameters

📌 TensorFlow Optimizers Explained

In TensorFlow, optimizers are algorithms used to update model weights during training by
minimizing the loss function. Optimizers adjust the model’s parameters (weights & biases)
based on the gradients computed during backpropagation.

🔹 Types of TensorFlow Optimizers

1️. Gradient Descent Optimizer (SGD)

• Basic optimizer that updates weights using the gradient.

• Formula:

W =W −η⋅ ∇ L (W )

Where:

W = weight

η = learning rate

∇ L ( W ) = gradient of the loss function

• Usage in TensorFlow:

optimizer = [Link](learning_rate=0.01)

• Pros:

✅ Simple and widely used

✅ Works well for convex optimization problems

• Cons:

❌ Can be slow

❌ May get stuck in local minima

2️. Stochastic Gradient Descent with Momentum (SGD + Momentum)

• Momentum helps accelerate SGD by using an exponential moving average of past

gradients.

• Formula:

v t=β v { t−1} +η ∇ L ( W )

W =W −v t

vt = velocity update (stores past gradients)

β = momentum factor (e.g., 0.9)

• Usage in TensorFlow:

optimizer = [Link](learning_rate=0.01, momentum=0.9)

• Pros:

✅ Faster convergence than vanilla SGD

✅ Helps escape local minima

• Cons:

❌ Still sensitive to learning rate

3️. Adaptive Gradient Algorithm (Adagrad)

• Adapts learning rate for each parameter, reducing it over time.

• Formula:

{η}
W =W −
√ ¿¿

Where Gt is the sum of squared gradients.

• Usage in TensorFlow:

optimizer = [Link](learning_rate=0.01)

• Pros:

✅ Adapts learning rates per parameter

✅ Good for sparse data

• Cons:

❌ Learning rate shrinks too much over time (may stop learning)

4️. Root Mean Square Propagation (RMSprop)

• Modifies Adagrad by keeping a moving average of squared gradients.

• Formula:

E [ g ] t= β E [ g ] {t −1 }+ (1−β ) gt
2 2 2

{η }
W =W − ⋅∇ L ( W )
√ ¿+ϵ }

β = decay factor (e.g., 0.9)

• Usage in TensorFlow:

optimizer = [Link](learning_rate=0.001)

• Pros:

✅ Works well for non-stationary loss functions

✅ Good for training deep neural networks

• Cons:

❌ Sensitive to learning rate selection

5️. Adaptive Moment Estimation (Adam)

• Combines momentum (SGD) and adaptive learning rate (RMSprop).

• Formula:
 mt =β 1 m{t −1}+ ( 1−β 1 ) ∇ L ( W )

2
v t=β 2 v {t −1} + ( 1−β 2 ) ( ∇ L ( W ) )

{η }
W =W − mt
{ √ {v }+ ϵ }
t

β 1 (default = 0.9) → Controls momentum

β 2 (default = 0.999) → Controls adaptive learning rate

• Usage in TensorFlow:

optimizer = [Link](learning_rate=0.001)

• Pros:

✅ Fast convergence

✅ Works well for most deep learning problems

• Cons:

❌ Uses more memory

6️. Adaptive Gradient Algorithm with Momentum (Adamax)

• Variant of Adam using the infinity norm for better stability.

• Usage in TensorFlow:

optimizer = [Link](learning_rate=0.002)

• Pros:

✅ More stable updates than Adam

• Cons:

❌ Not widely used

🔥 Choosing the Right Optimizer

Optimizer Best For Pros Cons

SGD Simple datasets Easy to implement Can be slow
SGD + Deep networks Faster than SGD Still needs tuning
Momentum
Adagrad Sparse data Adapts learning rates Learning rate shrinks
 too much
RMSprop RNNs, deep Works well on deep Requires learning rate
learning networks tuning
Adam General deep Combines best of RMSprop Higher memory usage
learning & Momentum
Adamax Stable learning More stable updates Not commonly used

✅ Final Thoughts

🔹 For most deep learning tasks → Use Adam (best default choice)

🔹 For simple cases → Use SGD (less memory & computation)

🔹 For recurrent neural networks (RNNs) → Use RMSprop

📌 The Vanishing Gradient Problem in Deep Learning

🔹 What is the Vanishing Gradient Problem?

The vanishing gradient problem occurs in deep neural networks when gradients become
extremely small during backpropagation, causing earlier layers to stop learning.

This happens when:

• The weight updates become negligible.

• The model stops improving due to minimal gradient flow.

🔹 Why Does This Happen?

During backpropagation, the gradient is computed using the chain rule:

{ ∂ L } { ∂ L } {∂ a n } {∂ a2 } { ∂ a1 }
= ⋅ ⋅… ⋅ ⋅
{ ∂ W } { ∂ an } {∂ a{n−1 } } {∂ a1 } { ∂W }
Since gradients are multiplicative, if activation function derivatives are small (<1), then
multiplying them shrinks the gradients exponentially.

For deep networks, this results in:

{ Very Small Gradients } ⇒ { Minimal Weight Updates } ⇒ { Network Stops Learning }

🔹 Causes of Vanishing Gradients

1️. Sigmoid Activation Function

• The sigmoid function squashes values into the range (0,1).

• Its derivative is small when inputs are large or small.

• This causes gradient values to shrink exponentially.

Sigmoid Activation & Derivative:

1
f ( x )= −x
1+e

f ¿ ( x )=f ( x ) ( 1−f ( x )) ¿
'

Gradient range: 0 < f'(x) < 0.25 (very small!)

Solution: Use ReLU, Leaky ReLU, or Batch Normalization instead.

2️. Deep Networks (Many Layers)

• In deep networks, backpropagation multiplies many small derivatives.

• Gradients become exponentially smaller in earlier layers.

• First layers barely learn, while the last layers learn normally.

Solution: Use skip connections (ResNets), Batch Normalization, or better activations.

3️. Poor Weight Initialization

• If weights start too small, activations shrink further at each layer.

• If weights start too large, activations get stuck at extremes (e.g., sigmoid → 0 or 1).

Solution: Use proper weight initialization like:

• Xavier Initialization (Glorot) → for sigmoid, tanh

• He Initialization → for ReLU, Leaky ReLU

🔹 Solutions to Vanishing Gradient Problem

✅ 1. Use ReLU Instead of Sigmoid/Tanh

• ReLU (max(0, x)) does not saturate for positive values.

• Its derivative is 1 (for x > 0), preventing gradient shrinkage.

• Avoids vanishing gradient in deep networks.

from [Link] import ReLU

layer = ReLU() # Instead of Sigmoid or Tanh

✅ 2. Use Leaky ReLU or Parametric ReLU (PReLU)

• ReLU still has a dead neuron problem for negative values.

• Leaky ReLU allows small gradients for negative inputs:

f(x) = {x, x>0 or a x, x<0

from [Link] import LeakyReLU

layer = LeakyReLU(alpha=0.01) # Small slope for negative values

✅ 3. Use Batch Normalization

• BN normalizes activations at each layer, reducing gradient shrinkage.

• Works well with any activation function.

from [Link] import BatchNormalization

[Link](BatchNormalization()) # Normalizes activations

✅ 4. Use Skip Connections (ResNets)

• ResNets add identity mappings (shortcut connections).

• Bypasses vanishing gradients by keeping some information intact.

from [Link] import Add

def residual_block(x):
shortcut = x # Skip connection
x = Dense(64, activation="relu")(x)
x = Dense(64)(x)
x = Add()([x, shortcut]) # Skip connection adds input back
return x

✅ 5. Use Proper Weight Initialization

• Xavier/Glorot Initialization (For Sigmoid, Tanh):

from [Link] import GlorotUniform

Dense(64, activation="tanh", kernel_initializer=GlorotUniform())

• He Initialization (For ReLU, Leaky ReLU):

from [Link] import HeNormal

Dense(64, activation="relu", kernel_initializer=HeNormal())

🔥 Summary

Cause Solution
Sigmoid/Tanh activation → Small ✅ Use ReLU, Leaky ReLU, PReLU
derivatives
Deep networks (many layers) → ✅ Use Skip Connections (ResNets), Batch
Gradients vanish Normalization
Poor weight initialization → ✅ Use He or Xavier Initialization
Activations shrink

👉 ReLU + He Initialization + Batch Normalization + ResNets = No Vanishing Gradient!

🎯