Scribd
Upload
15 views6 pages

Deep Learning: Linear Algebra & Basics

The document outlines the foundational concepts of deep learning, focusing on linear algebra, probability distributions, and their applications in data representation and model optimization. It explains the significance of scalars, vectors, matrices, and tensors in deep learning, as well as the characteristics and functions of discrete and continuous random variables. Additionally, it covers essential topics such as loss functions, metrics, and probability mass functions, providing a comprehensive overview of the mathematical principles underlying deep learning techniques.

Uploaded by

flytokiruba_46872986
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
15 views6 pages

Deep Learning: Linear Algebra & Basics

The document outlines the foundational concepts of deep learning, focusing on linear algebra, probability distributions, and their applications in data representation and model optimization. It explains the significance of scalars, vectors, matrices, and tensors in deep learning, as well as the characteristics and functions of discrete and continuous random variables. Additionally, it covers essential topics such as loss functions, metrics, and probability mass functions, providing a comprehensive overview of the mathematical principles underlying deep learning techniques.

Uploaded by

flytokiruba_46872986
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Approved by AICTE, New Delhi, Affiliated to Anna University, Chennai, Accredited by NAAC with A++ Grade

DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND DATA SCIENCE

AD3501 - DEEP LEARNING


UNIT 1: DEEP NETWORK BASICS
Linear Algebra: Scalars -- Vectors -- Matrices and tensors; Probability Distributions --
Gradientbased Optimization – Machine Learning Basics: Capacity -- Overfitting and
underfitting -- Hyperparameters and validation sets -- Estimators -- Bias and variance --S
tochastic gradient descent -- Challenges motivating deep learning; Deep Networks: Deep fe
edforward networks; Regularization – Optimization

1. Linear Algebra

Linear algebra plays a crucial role in deep learning, as it provides the mathematical
foundation for

1. Representing and Manipulating Data

2. Model Parameters

3. Computation within Neural Network

Some of the key applications

1. Representation of Data - Multi Dimensional Data Representation - Operations such as


Matrix multiplication, Reshaping etc....

2. Weighted Sum of Inputs

3. Activation Functions

4. Loss Functions

5. Gradient Computation

6. Optimization

7. Singular Value Decomposition (SVD) - dimensionality reduction, matrix approximation ,


understanding underlying structure of data

8. Eigenvalues and Eigenvectors - matrix factorization techniques and principal component


analysis (PCA) for feature extraction and reduction.

Scalar
 A scalar is a single value that represents a quantity, and it has only magnitude but no
direction. Examples of scalars include:
Temperature: It is a scalar quantity since it only requires a single value (e.g., 25 degre e
s Celsius) to define its magnitude.
 Time: Time is a scalar quantity as it can be expressed with a single value (e.g., 10
seconds).

 Speed: The speed of an object, like 50 kilometers per hour, is a scalar quantity since
it represents the magnitude of the velocity without specifying its direction.

Use of Scalar in Linear Algebra

Scalars are frequently used to represent and handle various aspects of the training process
and model evaluation.

Some Common Use Cases

 Loss Function - Measures the discrepancy between the predicted outputs of a deep
learning model and the true labels. The value of the loss function is a scalar that
quantifies how well the model is performing on a specific task.

 Metrics - accuracy, precision, recall, and F1 score, are used to evaluate the
performance of a model.

 Learning Rate - The learning rate is a scalar hyperparameter that determines the
step size of the optimization algorithm during training. It controls how much the
model's parameters are updated after each iteration, impacting the convergence and
stability of the training process.

 Regularization Strength: Regularization is used to prevent overfitting in deep


learning models.

 Batch Size: The batch size is a scalar hyperparameter that determines the number
of data samples used in each iteration of the training process. It impacts the
efficiency and memory requirements during training

 Number of Epochs - specifies how many times the entire dataset is used during
training.
 Model Parameters: The weights and biases of a deep learning model are scalar
values that are iteratively updated during training to learn from the data and make
predictions.
 Thresholds: In binary classification tasks, decision thresholds are scalar values
used to convert the model's continuous output probabilities into discrete class
predictions.
Vectors
 A Vector is a mathematical object that represents both magnitude (size) and
direction.
 For example, in two-dimensional space, a vector can be represented as (x, y), where
x is the horizontal component (magnitude in the x-direction), and y is the vertical
component (magnitude in the y-direction).

 In the context of machine learning and deep learning, vectors are used to represent
data points, model parameters, and feature representations.

Representation of Data as Vectors

The process of converting data into numerical vectors is known as "Vectorization."

Some Common Techniques to represent different data as vectors

1. Numerical Data:

 For one-dimensional numerical data, a simple approach is to represent each data


point as a scalar value in the vector.

 For multi-dimensional numerical data, such as time-series or sensor readings,


each data point can be represented as a vector with multiple components.

2. Categorical Data:

 One-hot encoding: In this method, each category is represented as a binary vector,


where all elements are zero except for the element corresponding to the category,
which is set to one.

 Embeddings: For categorical data with high cardinality, embeddings can be used to
map each category to a dense vector representation of lower dimensions.

[Link] Data:

 Bag-of-words: Each document is represented as a vector that counts the frequency


of each word present in the document. The length of the vector is equal to the size of
the vocabulary.
 Word embeddings: Pre-trained word embeddings like Word2Vec, GloVe, or
FastText can be used to convert words into dense vector representations

[Link] Data:

 Pixel values: For grayscale images, each pixel's intensity can be represented as a
scalar value in the vector. For color images, a vector can contain multiple
components representing pixel intensities for different color channels.
 Convolutional Neural Network (CNN) features: Extracting features from CNN
layers can provide meaningful vector representations for images.

5. Audio Data:

 Mel-frequency cepstral coefficients (MFCCs): These are commonly used features


to represent audio data as vectors for tasks like speech recognition.

6. Graph Data:
 Graph embeddings: Techniques like node2vec, GraphSAGE, or GNNs (Graph Neural
Networks) can be used to represent nodes in a graph as vectors.

Matrix

 A matrix is a two-dimensional array of numbers arranged in rows and columns.

 It is an example of 2nd-order tensors.

Tensors

 Tensors are multi-dimensional arrays that generalize scalars, vectors, and matrices
to higher dimensions.

 Here are some examples of tensors in different dimensions:

o Scalar (0-dimensional tensor)

o Vector (1-dimensional tensor)

o Matrix (2-dimensional tensor)

o 3D Tensor

o n-D Tensor

2. Probability Distribution

Probability

It is a mathematical concept that predicts how likely events are to occur. The probability
values are expressed between 0 and 1. The definition of probability is the degree to which
something is likely to occur.

Probability Distributions?

 A probability distribution is a description of how likely a random variable or a set of


random variables is to take on each of its possible states.

 The way we describe probability distribution depends on whether the variables are
discrete or continuous.
Discrete Random Variable

 It represents a numerical quantity associated with the outcomes of a random


experiment, where the possible values are countable and distinct.

 In other words, a discrete random variable can only take on specific, isolated values,
and there are gaps between these values.

Key characteristics of a discrete random variable:

 Countable and distinct outcomes: The set of possible values for a discrete random
variable is countable, meaning you can list them one by one, and they are distinct
(no two values are the same).

 Probability mass function (PMF): A discrete random variable is fully described by its
probability mass function (PMF), which assigns probabilities to each possible value
of the random variable. The PMF gives the probability of each outcome occurring

 Probabilities are non-negative: The probabilities assigned by the PMF for each value
must be non-negative (greater than or equal to 0).

 4Sum of probabilities is 1: The sum of all probabilities in the PMF must equal 1. Since
one of the possible outcomes must occur, the sum of probabilities across all outcomes is
100%.

Probability Mass Function


It maps each possible value of the random variable to the probability of that value
occurring. For a discrete random variable X, the PMF is denoted by P(X = x) or simply P(x),
where "x" represents a specific value that X can take.
The PMF satisfies the following properties:
 Non-negativity: P(x) ≥ 0 for all possible values of "x."
 Sum of probabilities: The sum of all probabilities in the PMF is equal to 1. In
mathematical terms, Σ P(x) = 1, where the sum is taken over all possible values of X.
Mathematically, the PMF of a discrete random variable X is defined as: P(X = x) = Pr(X = x)
where "Pr" denotes the probability of an event.
Let's take an example of a simple discrete random variable to demonstrate the PMF. Consider a
random variable X representing the outcome of rolling a fair six-sided die. The possible values of
X are {1, 2, 3, 4, 5, 6}, and each value has an equal probability of 1/6 since the die is fair.
The PMF for this random variable X is as follows:
P(X = 1) = 1/6
P(X = 2) = 1/6
lOMoARcPSD|42273479

P(X = 3) = 1/6
P(X = 4) = 1/6
P(X = 5) = 1/6
P(X = 6) = 1/6
This PMF tells us that the probability of getting a 1, 2, 3, 4, 5, or 6 is 1/6 each, as we would
expect from a fair die.
Continuous Random Variables
Unlike a discrete random variable, a continuous random variable can take on any val ue
within a specific range, and there are no gaps between possible values.
These values are typically associated with measurements, such as height, weight, time,
temperature, etc., which can be expressed as real numbers.
Key characteristics of a continuous random variable:
 Uncountable and infinite outcomes: The set of possible values for a continuous
random variable is uncountably infinite, meaning there are infinitely many possible
values within the specified range.
 Probability density function (PDF): A continuous random variable is fully described
by its probability density function (PDF). The PDF represents the likelihood of the
random variable taking on a particular value, but it does not give the probability of
any specific value occurring since the probability of any individual point in a
continuous distribution is zero. Instead, the probability of the random variable
falling within a certain range is calculated by finding the area under the PDF curve
over that range.
Examples of continuous random variables include:
 The height of a person, which can take on any non-negative real value within a
certain range.
 The time it takes for a computer program to execute, which can take on any non -
negative real value (assuming it does not take forever to run).
 The temperature of a given location, which can take on any real value within a
range.
Probability Density Function
PDF of a continuous random variable provides information about the likelihood of the
variable falling within certain intervals or ranges of values.
the probability that "X" falls within a specific interval [a, b] is given by the integral of the
PDF over that interval: P(a ≤ X ≤ b) = ∫[a,b] f(x) dx.

Common questions

Powered by AI

PMFs are used for discrete random variables and outline probabilities of exact numerical outcomes, summing up to 1 across all outcomes. PDFs apply to continuous random variables, expressing probabilities over a range, where the area under the curve of the PDF between two values represents the probability, not the probability of individual points. Understanding these functions is critical in machine learning for handling different data types and distributions, ensuring models can properly interpret probabilistic information and make accurate predictions .

Vectorization is crucial because it enables efficient computation and processing in machine learning by representing data as vectors. Numerical data can be straightforwardly converted to vectors; categorical data may use techniques like one-hot encoding or embeddings; text data might be vectorized via bag-of-words or embeddings; image data can be represented by pixel intensity vectors or CNN-derived features, and for audio, MFCCs are used. This diversity allows machine learning models to interpret different data formats effectively .

Overfitting occurs when a machine learning model captures noise instead of the underlying data pattern. Regularization techniques such as L1 and L2 regularization, dropout, and early stopping are used to prevent overfitting. These methods limit model complexity or stop training when the model begins to overfit. This is crucial in deep learning because models typically have high capacity, and overfitting can lead to poor generalization on unseen data .

Gradient-based optimization helps adjust the model parameters to minimize the loss function in deep learning. Techniques such as stochastic gradient descent and its variants like Adam and RMSProp are used to find the optimal parameters by iteratively adjusting them in the direction of the gradient derived from the loss function's output. This process is crucial because it enables deep networks to learn patterns from the data by reducing errors incrementally .

Graph embeddings convert nodes of graph structures into a numerical vector space, preserving relationships between nodes. This representation enhances structured data processing in deep learning by making graph data amenable to algorithms designed for vector inputs. Techniques like node2vec, GraphSAGE, and GNNs allow for capturing complex patterns and dependencies within graphs, supporting tasks such as node classification, link prediction, and recommendation systems .

Linear algebra is essential in deep learning for representing and manipulating data, model parameters, and computations within neural networks. Matrix operations such as multiplication and reshaping are critical in data representation. Scalars, vectors, tensors, and matrices are used to represent model parameters and inputs. Operations like singular value decomposition and eigenvalue/eigenvector methods facilitate dimensionality reduction and feature extraction, which are fundamental in optimizing and structuring deep learning tasks .

SVD is a linear algebra technique used in deep learning to reduce the dimensionality of data while retaining its essential structure. It decomposes a matrix into three other matrices, capturing patterns along different axes, which can be useful for tasks like noise reduction and feature extraction. SVD is significant because it simplifies complex datasets, enhances computational efficiency, and can improve the performance of neural networks by reducing overfitting risks on high-dimensional data .

Bias-variance tradeoff is central to model evaluation, balancing simplicity and complexity to minimize total error. High bias can underfit data leading to simplistic models, whereas high variance can overfit causing models to memorize noise. Techniques such as cross-validation and regularization adjust model parameters to navigate this tradeoff. Understanding and addressing bias and variance is crucial for improving predictions and model generalization in machine learning .

Hyperparameters such as learning rate, batch size, regularization strength, and the number of epochs influence the model's training process and performance. They determine how quickly a model learns, the stability and convergence of learning, and the generalization ability of the trained model. Hyperparameter tuning techniques like grid search or random search, possibly automated with tools like Bayesian optimization, are used to find optimal values, balancing model accuracy and computational efficiency .

Initialization can significantly impact the convergence speed and final performance of deep learning models. Poor initialization can lead to vanishing or exploding gradients, hindering learning. Common strategies include Xavier/Glorot initialization for sigmoid activation functions, and He initialization for ReLU, designed to maintain variance across layers during input transformations. Proper initialization is fundamental for achieving efficient and stable training .

You might also like