Lecture Notes: Compiled by Maqsood Ahmad (A.P. Maths.
) for
students of CUI, Lahore. (FA19-BCE-A & B, SP19-BSE-A).
Chapter 5: Inner Product Spaces
Courtesy:- Material for this lecture is selected from Kolman’s book, and
handouts of Virtual University, Lahore, Virtual COMSATS and Imperial
College London.
Objective of this Lecture:-
(A) The prime objective of this chapter is to convert a set of linearly independent vectors
(basis) into a set of orthonormal vectors (basis).
(B) To achieve this goal, first we need to define several bits and pieces like “dot product
(inner product), length (norm or magnitude) of vectors, unit vector, distance between
vectors, projection of vectors, orthogonal vectors and orthonormal vectors”.
After studying this lecture, you are desired to do
Home Work: Do Questions 6, 8, 18 and 24 of Exercise 5.1, Questions
8,9,12,13,29,30 of Exercise 5.3, and Questions 1-4, 9-20 of Exercise 5.4.
Section 5.1:
Definition 1: (Dot or inner or scalar or projection product of vectors in terms of
vector components)
⎡ ⎤ ⎡ ⎤
⎢ . ⎥ ⎢ . ⎥
Let , ∈ then = ⎢ . ⎥ and = ⎢ . ⎥, consider these vectors as ×1
⎢ ⎥ ⎢ ⎥
⎢ . ⎥ ⎢ . ⎥
⎣ ⎦ ⎣ ⎦
matrices. Clearly =[ . . . ].
Now the dot or inner product is a scalar quantity dented by or ( , ) and is
defined as matrix product
⎡ ⎤
⎢ . ⎥
= ( , )= =[ . . . ]⎢ . ⎥ = + +⋯+
⎢ ⎥
⎢ . ⎥
⎣ ⎦
Note: Although notation is more familiar but we will use this natation
( , ) for dot/inner product to maintain the consistency with Kolman’s book.
Example 1: Let = and = − then
( , ) = ( ) + (− ) + ( ) = = ( , ) (Inner product is commutative)
Definition 2: (Length or norm or magnitude of vector)
⎡ ⎤
⎢. ⎥
Let ∈ then = ⎢ . ⎥, then length of this vector is denoted by ‖ ‖ (scalar
⎢ ⎥
⎢. ⎥
⎣ ⎦
quantity) and defined as ‖ ‖ = + +. . . + . When = 2 3 i.e.,
our vector is from (2D) or (3D) then
‖ ‖= + And ‖ ‖ = + + , respectively.
Observation: A nice relation between dot product and norm of a vector.
( , )= + +⋯+ = + +. . . + =‖ ‖
Example 2: Let be vector in , such that = then ‖ ‖ = √1 + 4 + 4=3
−
Definition 3: (Unit vector and normalization)
A vector having length 1 is called a unit vector. We can convert any arbitrary
vector in to a unit vector using formula
=
‖ ‖
This way of getting from is called normalization of and is called
normalized vector. Both and are in same direction.
Example 3: Let be vector in , such that = then ‖ ‖ = √1 + 4 + 4=3
−
⎡ ⎤
⎢ ⎥
Then =‖ ‖
= =⎢ ⎥. Now verify
− ⎢ ⎥
⎣ ⎦
1 2 −2 1 4 4
‖ ‖= + + = + + =1
3 3 3 9 9 9
Definition 1a: (Dot or inner product of vectors in terms of projection)
Here we present an alternative formula for dot/inner product in terms of projection
of vectors.
Let , ∈ then ( , ) = ‖ ‖‖ ‖ cos( ), Proof of this formula is simple using
Law of cosines of triangle.
( , ) = ‖ ‖‖ ‖ cos( ) , = ‖ ‖. scalar projection of on , Where is angle
between and . (See Figure 1)
Figure 1
Observation: The formula in Definition 1a is excellent in establishing various
facts about dot/inner product whereas formula in Definition 1 is handy for
calculations.
NOTE: Watch the video given in link below, only for students interested in
proof, for , ∈ ,
( , )= + = ‖ ‖‖ ‖ ( )
[Link]
Some facts about dot/inner product:
Fact 1: ( , ) = 0 if = i.e., vectors are orthogonal to each other when their dot
product is zero.
Fact 2: ( , ) = 0 if i.e., Zero vector is orthogonal to every vector.
Fact 3: ( , ) = ‖ ‖‖ ‖ if = i.e., vectors are pointing in same direction.
Fact 4: ( , ) = −‖ ‖‖ ‖ if = i.e., vectors are in opposite direction.
Definition 4: (Distance between two vectors)
Let , ∈ then distance between and is denoted by ( , ) and is defined
as
( , )=‖ − ‖= ( − ) +( − ) +. . . +( − ) .
Note: Next figure is only for interested readers, proof can directly be seen
geometrically from law of parallelogram of vectors.
Figure 2
Now we present some theorems having many applications in Mathematics and
proofs are beyond the scope of this course.
Do questions 6, 8, 18 and 24 of Exercise 5.1
Theorem 1: Cauchy-Schwartz Inequality:-
If and are any two vectors in , then | , | ≤ ‖ ‖‖ ‖
i.e., Absolute value of dot product is less or equal to product of magnitudes of
vectors.
Example 4:
i.e., |−5| ≤ √14 . √17 ⇒ 5 ≤ 15.43
Theorem 2: Triangle Inequality:-
If and are any two vectors in , then ‖ + ‖ ≤ ‖ ‖ + ‖ ‖
−2
Example 5: Consider again and given in example 4, then + = 4 and
−1
‖ + ‖ = √21 , Now √21 ≤ √14 + √17 ⇒ 4.6 ≤ 7.86.
Theorem 3: The Pythagorean Theorem:-
Two vectors and in are orthogonal if and only if
(‖ + ‖) = ‖ ‖ + ‖ ‖ .
Definition 5: (Orthogonal Set)
Let = , ,..., be the set of non-zero vectors in , then is said to be
orthogonal set if all vectors in are mutually orthogonal. Mathematically
, =0 ≠
Theorem 4:- Let = , ,..., be a finite orthogonal set of vectors of a
Euclidean space . Then is linearly independent.
Definition 6: (Orthogonal Basis)
Let = , ,..., be a basis for subspace in , then is said to be
orthogonal basis if set is an orthogonal set.
Definition 7: (Orthonormal Set)
Let = , ,..., be the set of non-zero vectors in , then is said to be
orthonormal set if is orthogonal set of unit vectors. Mathematically
, =0 ≠ ‖ ‖=1 ∈ .
Definition 8: (Orthonormal Basis)
Let = , ,..., be a basis for subspace in , then is said to be
orthonormal basis if set is an orthonormal set.
Example 6:
Now we have enough tools and finally arrived at the beginning of our core
problem.
Gram-Schmidt Process to obtain orthonormal basis:-
Our life is very easy while working with orthonormal bases. We observed that
to work with the natural/standard basis for the computations are
kept to a minimum.
Recall (1) if ={ , , } are standard basis of , where = , =
, = then any vector = ∈ can directly be written as linear
combination of basis i.e., = + + . (Coefficients of bases vectors are
coordinates of the vector )
Recall (2) if set = { , , } is any basis of , and vector ∈ then can
be written as = + + . (See Examle 1(LC)), We have to solve a non-
homogeneous system to obtain the coordinates of .
Following theorem shows the advantage of orthonormal basis over any basis of
vector space.
Theorem 5:- (In book Theorem 5.5)
Let = { , , . . . , } be a finite orthonormal set of finite vectors of a Euclidean
space and let be any vector in . Then
= + + +⋯+
Where = ( , ), = 1,2,3, … ,
Note:- We only need to compute few dot product in presence of orthonormal
basis.
Example 7:-
Solution:-
Theorem 6:- (In book Theorem 5.6)
Let = be a Euclidean space and an m-dimensional subspace of , then
there exists an orthonormal basis = { , , . . . , }.
OR equivalently, for subspace given any basis set say ={ , ,..., }
∗
Obtain/convert (1) set to an orthogonal basis set say ={ , ,..., }
(using Gram-Schmidt process)
(2) Finally normalize this set to get orthonormal basis ={ , ,..., }.
NOTE:-PROOF OF THIS THEOREM IS BEYOND THE SCOPE OF OUR
COURSE, HOWEVER I WILL GIVE GENERAL OUTLINE OF PROOF.
Do Example 2 on page (323) of Kolman’s book. For video support of
this please visit following links.
[Link]
[Link]