MODULE – 2

IMPORTANT QUESTIONS
[Link] the concept of Eigenvalue Decomposition and its applications in
data science, such as Principal Component Analysis (PCA). (imp)

2. Describe the method of solving overdetermined equations using the

pseudoinverse. (imp)

3. Explain the null space and rank space of a matrix, and why is it
important in understanding the solutions to systems of linear equations in
data science?(imp)

4. Describe the geometric interpretation of vector projections and their

significance in machine learning algorithms like linear regression. (imp)

5. Explain the concept of projections in linear algebra. (-)

6. Describe the product of a matrix and a vector and its applications. (-)

7. How do you determine the rank of a matrix? What does the rank tell you
about the linear independence of vectors in data science contexts? (imp)

8. Compare and contrast L1 and L2 norms. How are they used in

regularization techniques like Lasso and Ridge regression? (-)

( Least squares )
[Link] the concept of Eigenvalue Decomposition and its applications in
data science, such as Principal Component Analysis (PCA).

1. What is Eigenvalue Decomposition?

• Eigenvalue Decomposition (EVD) is a technique used in linear algebra to decompose a
square matrix into its eigenvalues and eigenvectors.
• It is applicable only to square matrices that are diagonalizable.
• Mathematically, for a matrix A, if there exists a matrix V (eigenvectors) and a diagonal
matrix Λ (eigenvalues), such that:
A=VΛV−1
then A can be decomposed using EVD.

2. Meaning of Eigenvalues and Eigenvectors

• An eigenvector of a matrix A is a non-zero vector v such that:
Av=λv
where λ (lambda) is the corresponding eigenvalue.
• Eigenvectors represent the directions along which a linear transformation acts by stretching
or compressing, and the eigenvalues tell how much.

3. Steps in Eigenvalue Decomposition

1. Start with a square matrix A.
2. Solve the characteristic equation:
det(A−λI)=0
to find eigenvalues λ.
3. For each eigenvalue, solve:
(A−λI)v=0
to find the corresponding eigenvector v.
4. Construct the diagonal matrix Λ and eigenvector matrix V.
5. Express the original matrix as:
A=VΛV−1
4. Applications in Data Science

a. Principal Component Analysis (PCA)

• PCA is a dimensionality reduction technique that uses eigenvalue decomposition of the
covariance matrix of a dataset.
• Steps in PCA using EVD:
1. Standardize the dataset.
2. Compute the covariance matrix.
3. Apply eigenvalue decomposition on the covariance matrix.
4. Sort eigenvectors by decreasing eigenvalues (i.e., principal components).
5. Project data onto the top k eigenvectors to reduce dimensions.

b. Feature Selection and Noise Reduction

• By selecting the top eigenvectors, PCA retains features that capture the most variance,
discarding noisy, less informative parts.
• This helps improve model performance and reduces computational cost.

c. Data Visualization
• PCA using EVD can reduce high-dimensional data to 2D or 3D for easier visualization,
making it easier to understand patterns or clusters.

d. Face Recognition (e.g., Eigenfaces)

• Eigenvalue decomposition is used in face recognition systems.
• Images are treated as matrices and decomposed to find patterns (eigenfaces) that represent
different faces.
2. Describe the method of solving overdetermined equations using the
pseudoinverse.

Solving Overdetermined Equations Using the Pseudoinverse

1. What is an Overdetermined System?

• An overdetermined system is a system of linear equations where the number of equations
(rows) is greater than the number of unknowns (columns).
• It is represented as:
Ax=b
where:
• A is an m×n matrix (with m>n),
• x is an n×1 vector of unknowns,
• b is an m×1 vector.

2. Problem with Direct Solution

• Since there are more equations than unknowns, it’s usually not possible to find an exact
solution that satisfies all equations.
• Therefore, we look for a solution that minimizes the error, typically using the least
squares approach.

3. Least Squares Solution

• The goal is to find x such that the difference between Ax and b is minimized:
xmin∥Ax−b∥2
• This is known as the least squares approximation.

4. Introduction to the Pseudoinverse

• The Moore–Penrose pseudoinverse, denoted as A+, is a generalization of the matrix
inverse used when A is not square.
• It provides the best least-squares solution to the overdetermined system.
5. Formula for Least Squares Solution Using Pseudoinverse
• The optimal solution x is given by:
x=A+b
• If A has full column rank, then:
A+=(ATA)−1AT

6. Steps to Solve Using Pseudoinverse

1. Start with the overdetermined equation Ax=b.
2. Compute the transpose of matrix A, i.e., AT.
3. Compute ATA and check if it is invertible.
4. Compute the pseudoinverse:
A+=(ATA)−1AT
5. Multiply the pseudoinverse with b to get the solution:
x=A+b
6. This x minimizes the square of the residual error ∥Ax−b∥2.

7. Applications in Data Science

• Used in linear regression to compute best-fit line when data is noisy.
• Helps in dimensionality reduction and solving systems with no exact solution.
• Used in machine learning when training data has more examples (rows) than features
(columns).
3. Explain the null space and rank space of a matrix, and why is it
important in understanding the solutions to systems of linear equations in
data science?
Null Space and Rank Space of a Matrix
1. What is a Matrix?
• A matrix is a rectangular array of numbers, often used to represent linear transformations
or systems of linear equations.

2. Definition of Null Space (Kernel)

• The null space of a matrix A is the set of all vectors x such that:
Ax=0
• It represents all the solutions to the homogeneous system of equations.
• Notation: Null(A) or N(A)

3. Properties of Null Space

• The null space is a subspace of Rn, where n is the number of columns of A.
• If A is an m×n matrix, then:
space Null space ⊆Rn
• The dimension of the null space is called the nullity of the matrix.

4. Definition of Rank (Column Space)

• The rank of a matrix A is the dimension of the column space of A.
• The column space (also known as rank space) is the set of all linear combinations of the
columns of A.
• Notation: Rank(A)

5. Rank-Nullity Theorem
• A key relationship:
Rank(A)+Nullity(A)=n
where n is the number of columns of A.
• This helps understand the structure of solutions to a system Ax=b.
6. Importance in Solving Linear Equations
• If b lies in the column space of A, then Ax=b has a solution.
• The null space gives information about:
• Uniqueness of the solution
• Infinite solutions (if null space is non-zero)
• Example:
• If null space contains only the zero vector, the solution is unique.
• If not, solutions exist in the form:
x=x0+xnwhere xn∈Null(A)

7. Importance in Data Science

• Helps in understanding redundant or linearly dependent features.
• If a feature (column) is in the null space, it doesn't contribute to the output—can be
removed.
• In dimensionality reduction (e.g., PCA), rank tells how many meaningful components
exist.
• Identifies loss of information or underlying structure in high-dimensional data.

8. Application Example
• In linear regression, we solve Ax=b. Knowing the rank helps:
• Ensure the problem is well-posed.
• Avoid multicollinearity (where features are linearly dependent).
• Decide whether to use pseudoinverse for overdetermined systems.

9. Geometric Intuition
• Null space → the “shadow” cast to zero by the transformation A
• Column space → the direction into which the input vector can be transformed

10. Summary Table

Concept Description Relevance
Null Space Vectors mapped to 0 by matrix A Determines uniqueness of solution
Column Space Set of all possible outputs Ax Ensures solution exists
Rank Dim. of column space Indicates number of independent features
 Concept Description Relevance
Nullity Dim. of null space Indicates redundancy

4. Describe the geometric interpretation of vector projections and their

significance in machine learning algorithms like linear regression.

1. What is Vector Projection?

• Vector projection is the operation of projecting one vector onto another.
• It gives the shadow or component of a vector in the direction of another vector.

2. Mathematical Formula of Projection

• Given vectors a and b, the projection of a onto b is:
projba=∣∣b∣∣2a⋅bb
• This results in a new vector in the direction of b, representing how much of a is aligned with
b.

3. Geometric Meaning
• Imagine a light source above a vector a casting a shadow on vector b.
• That shadow is the projection of a onto b.
• It's the closest point on the line spanned by b to the tip of a.

4. Orthogonal Decomposition
• Any vector a can be split as:
a=projba+e
where e is the error vector perpendicular (orthogonal) to b.
• This is fundamental in minimizing errors in machine learning.

5. Connection to Linear Regression

• In linear regression, we fit a line (or plane) to data to minimize error.
• Geometrically, this is like projecting the output vector y onto the column space of the
input matrix X.
• The predicted values y^ are the projections of y onto the space spanned by features.
6. Least Squares Solution Uses Projections
• The best-fit line minimizes the squared distance between actual y and predicted y^.
• The solution:
y^=Xβ^=projCol(X)y
• Hence, linear regression is based on projection of the target vector onto the feature space.

7. Importance in Machine Learning

• Error minimization: Projection minimizes the perpendicular error.
• Feature importance: Vectors that align closely with output indicate stronger influence.
• Model accuracy: Better projections lead to better predictions.
• Dimensionality reduction: PCA uses orthogonal projections to reduce feature space.

8. Visual Example
• Suppose we have:
• Input data along a plane (feature space)
• Output vector y floating in space
• Projection brings y down onto the plane – that’s the prediction.

9. Summary Table
Concept Description Relevance
Component of a vector in the direction of
Vector Projection Used in error minimization
another
Represents residuals in
Orthogonal Error Difference between actual and projection
regression
Linear
Projects output y onto feature space of X Builds best-fit prediction line
Regression
Use projections to reduce and interpret feature
PCA & Others Dimensionality reduction
spaces

10. Key Takeaways

• Projections help in understanding how much influence one vector (feature) has on another
(output).
• In linear regression, accurate projections result in better predictions.
• This geometric view is essential in building and interpreting models in data science.
5. Explain the concept of projections in linear algebra.
1. Definition of Projection
• In linear algebra, a projection is a transformation of a vector onto a subspace.
• It results in another vector that lies within the subspace, representing the closest
approximation to the original vector within that subspace.

2. Mathematical Formula
• The projection of a vector a onto a vector b is given by:
projba=∣∣b∣∣2a⋅bb
• Here, ⋅ denotes the dot product and ∣∣b∣∣ is the magnitude of b.

3. Projection Matrix
• For projecting a vector onto a subspace spanned by a matrix A, the projection is:
P=A(ATA)−1AT
• The matrix P is called the projection matrix.
• It satisfies the property: P2=P (idempotent).

4. Geometric Interpretation
• A projection can be visualized as dropping a perpendicular from a vector onto a line or
plane.
• The result is the "shadow" of the original vector on the subspace.

5. Orthogonal Projections
• A projection is orthogonal if the error (difference between original and projected vector) is
perpendicular to the subspace.
• Important in minimizing error, such as in least squares regression.

6. Projection onto a Line

• When projecting onto a line (1D subspace), the result is a scaled version of the line vector.
• Useful for understanding how much a vector aligns with a given direction.
7. Projection onto a Plane
• In 3D, projecting onto a plane gives a vector that lies on the plane.
• This is used in computer graphics, physics, and geometry applications.

8. Use in Linear Systems

• Projection helps find approximate solutions to systems of equations that are inconsistent
(like overdetermined systems).
• It allows us to find the best approximation within a subspace.

9. Application in Data Science

• Core to linear regression, where we project the output vector onto the column space of the
input matrix.
• Helps reduce dimensionality and noise in data using techniques like PCA (Principal
Component Analysis).
6. Describe the product of a matrix and a vector and its applications.
Product of a Matrix and a Vector and its Applications

1. Basic Definition
• The product of a matrix and a vector refers to multiplying a matrix A (of size m×n) with a
vector x (of size n×1).
• The result is a new vector y (of size m×1), calculated as:
y=Ax

2. How It Works
• Each element of the resulting vector y is a linear combination of the columns of A,
weighted by the corresponding entries of vector x.
• This operation effectively transforms the input vector into a new space.

3. Visual Interpretation
• Think of the matrix as an operator that scales, rotates, or projects the vector into a different
direction or dimension.
• For example, in 2D or 3D graphics, this helps rotate or scale objects.

4. Mathematical Properties
• Matrix-vector multiplication is distributive and associative, which allows for efficient
computation.
• Used to model linear transformations.

5. Example
If
A=[2435],x=[12]
Then
Ax=[2∗1+3∗24∗1+5∗2]=[814]
6. Applications in Data Science
a) Linear Systems
• Matrix-vector multiplication is used to represent and solve systems of linear equations:
Ax=b

b) Linear Regression
• In machine learning, predictions are made using:
y^=Xw
where X is the feature matrix and w is the weight vector.

c) Dimensionality Reduction
• Used in algorithms like Principal Component Analysis (PCA) to project data into lower
dimensions.

d) Neural Networks
• Each layer in a neural network uses matrix-vector multiplication to compute outputs from
inputs.

e) Image Processing
• Images are represented as matrices, and operations like blurring or edge detection use
matrix-vector (or matrix-matrix) products.

7. Efficiency
• Matrix-vector products are computationally efficient and can be parallelized for large
datasets (e.g., using GPUs).

8. Data Transformation
• Applying transformation matrices on data points (vectors) is essential for standardization,
normalization, or rotating coordinate systems.

9. Geometric Insight
• Helps understand how data moves or changes across spaces.
• For example, in PCA, data is projected onto principal axes using matrix-vector
multiplication.
7. How do you determine the rank of a matrix? What does the rank tell you
about the linear independence of vectors in data science contexts?
How to Determine the Rank of a Matrix and Its Importance in Data
Science

1. What is the Rank of a Matrix?

• The rank of a matrix is the maximum number of linearly independent rows or columns
in the matrix.
• It indicates the dimension of the row space or column space of the matrix.

2. Determining the Rank of a Matrix

a) Row Echelon Form (REF) or Reduced Row Echelon Form (RREF)
• Convert the matrix into REF or RREF using Gaussian elimination.
• Count the number of non-zero rows in the resulting matrix.
• That count is the rank of the matrix.

b) Determinants (for square submatrices)

• For small matrices, find the largest square submatrix with a non-zero determinant.
• The order of that submatrix is the rank.

c) Using SVD (Singular Value Decomposition)

• In numerical computing, rank is often calculated from non-zero singular values.
• The number of non-zero singular values = rank.

3. Example
Let
A=123246369
Performing row operations reveals that only one row is independent, so:
Rank(A) = 1

4. What Rank Tells About Linear Independence

• If rank = number of columns, then all columns are linearly independent.
 • If rank < number of columns, then there is linear dependence — some columns can be
written as combinations of others.

5. Significance in Data Science

a) Feature Redundancy Detection
• In datasets, if columns (features) are linearly dependent, they don't add new information.
• Rank helps in feature selection or dimensionality reduction.

b) PCA (Principal Component Analysis)

• The rank of the data matrix determines the maximum number of principal components
you can extract.
• PCA works by finding a low-rank approximation.

c) Solving Linear Systems

• A full-rank coefficient matrix ensures a unique solution exists.
• Rank deficiency may lead to infinite or no solutions.

d) Overfitting Detection
• Rank can be used to verify if a model is over-parametrized (more parameters than
necessary).
8. Compare and contrast L1 and L2 norms. How are they used in
regularization techniques like Lasso and Ridge regression?

Point L1 Norm (Lasso Regression) L2 Norm (Ridge Regression)

Sum of absolute values of coefficients:
1. Definition w_i
(|w|_1 = \sum
2. Regularization L1 Regularization (Lasso) – adds L2 Regularization (Ridge) – adds
Type absolute value penalty to loss square value penalty to loss
Yes, it can make some coefficients
3. Feature No, it only shrinks coefficients but
exactly zero (automatic feature
Selection keeps all features
selection)
Produces sparse models (few non-zero Produces dense models (all features
4. Sparsity
features) with small weights)
Best when only a few features are Best when many features contribute
5. Use Case
important slightly
Easier to interpret due to fewer selected Harder to interpret if many features
6. Interpretation
features remain
7. Effect on Forces some coefficients to zero Shrinks all coefficients toward zero
Coefficients (eliminates them) but none are eliminated
Constraint region is diamond-shaped – Constraint region is circular –
8. Visual Shape encourages solutions on axes (zeroing smooth penalty, all coefficients
out) reduced uniformly
9. Stability Less stable with correlated features More stable with correlated features
Used alone in Lasso, but can be
Used alone in Ridge, or combined
10. Hybrid Option combined with L2 in Elastic Net for
with L1 in Elastic Net
balance