Chapter 4

Vector Spaces (II): Inner

Product Space – Combined
Algebraic and Topological
Structures

Wen-Hsien Fang Engineering Mathematics (II) 1/85

 APPLICATIONS

Classification
Datasets

Minimum distance? Linear regression

Wen-Hsien Fang Engineering Mathematics (II) 2/85
 4.0 Two Geometric Measures
In this chapter, we will introduce two geometric measures,
length (size) and angle (similarity) in a vector space.
First, consider the simplest vector space R 2 .
y
 x1 
• LENGTH  x 
 2
x x2
 x
x  x1  x2
2 2
x1

3
Wen-Hsien Fang Engineering Mathematics (II) 3/85
4.0 Two Extra Measures


• ANGLE between two vectors 
- Dot product operation (formula):x  y  x x +y y
1 2 1 2
  x1 
y 
- Dot product operation (physical meaning): x  1

 x2 
(cosine formula for the dot product).
     y 
 2
0 y
x  y  x y cos
Proof : cos  cos( x   y )  cos( x ) cos( y )  sin( x )sin( y )
 
x1 y1 x2 y2 x1 y1  x2 y2 xy
=           
x y x y x y x y
   
or, equivalently, x  y  x y cos

component form
polar form 4
Wen-Hsien Fang Engineering Mathematics (II) 4/85
4.0 Two Extra Measures

Example 4.0-1
3  1 
Determine the angle between   and   ?
 4 7 
Solution

 1 
 3, 4   
 7  1
cos  
3  4 (1)  7
2 2 2 2
2


=
4
5
Wen-Hsien Fang Engineering Mathematics (II) 5/85
4.0 Two Extra Measures

Remarks: (For Euclidean Space R 2 and R 3 )

  T   
1. If x  y (= x y )  x y cos  0
 
  =90 , x & y are orthogonal (perpendicular)
o

 
denoted by x  y
    Cauchy-Schwarz
2. Since x  y = x y cos , inequality

     
x  y  x y cos  x y

x
3. Unit vector (vector of unit length): 
x
6
Wen-Hsien Fang Engineering Mathematics (II) 6/85
4.0 Two Extra Measures

Remarks:
4. Geometric Point of View
     
 x  y  x y cos  . Assume x  y =1,

  x

 x  y large   small 
  y
 x and y are aligned and pointing in the same direct
 
 x  y small   large
 
 x and y are less aligned and
7 pointing in
different directions x
 
y
7
Wen-Hsien Fang Engineering Mathematics (II) 7/85
4.0 Two Extra Measures

5. Concept of Orthogonal Projection

  
The Projection of a onto b is P. Then

  a
  T b  b 
P = a  ( a  cos )  
b b P b


Unit length vector 1
along b

  T 
  T b b a b
P =a      2b
b b b
8
Wen-Hsien Fang Engineering Mathematics (II) 8/85
4 . 0 Tw o E x t r a M e a s u r e s

Application of Projection normal vector

3  w1 
- Plane Equation (in R )  w  =w
 2
T  w3 
Let [x0 , y0 , z0 ] be any point on the plane,  x0 
y   x
then any plane can be written as  0  y  =x
 z0   
w1 ( x  x0 )  w2 ( y  y0 )  w3 ( z  z0 )  0  z 

or w1 x  w2 y  w3 z  w1 x0  w2 y0  w3 z0
  
b

  w1 , w2 , w3  x, y , z   b  0
T


wT

 A plane equation is thus w x  b  0, T

where w is a vector orthogonal to the plane 9

Wen-Hsien Fang Engineering Mathematics (II) 9/85
4 . 0 Tw o E x t r a M e a s u r e s

- Distance from a point to the Plane wT x  b  0

T
The distance between the point [ x1 , y1 , z1 ] and the plane
is the projection of [ x1 , y1 , z1 ]  [ x0 , y0 , z0 ] =x  x 0
T T
 
x x0

along the direction w : normal vector

 w1   x1 
w w ( x  x0 ) T
 w  =w  y  =x
( x  x0 ) = T  2
 w3   1
w w  z1 

b
 x0 
w x  w x0
T T  y  =x
 0 0
w xb
T
 z0 
  wT x  b  0
w w 10
Wen-Hsien Fang Engineering Mathematics (II) 10/85
4 . 0 Tw o E x t r a M e a s u r e s

Example 4.0-2
Find the distance from [4,1,1]T to the plane x+2y+2z=1.
Solution
From the above, the normal vector w  [1, 2, 2] . T

So the distance from [4,1,1] to this plane is normal vector

4 1   4
   2 1 
[1,2,2] 1  1    
   2 
wT x  b 1  7  1 
 
w 1  3
2  x0 
  y 
 2   0
 z0  11
Wen-Hsien Fang Engineering Mathematics (II) 11/85
 4.1 Inner Product and Norms
We extend the concept of “angle” to general vector spaces:
4.1.1 Inner Product
Definition
An inner product on a vector space V is an operation on V that
assigns to each pair of vectors v and w in V a real number <v, w>
satisfying the following axioms:
P1. <v, w> is a real number for all v and w in V
P2. <v, w> = <w,v> for all v and w in V (symmetry)
P3. <v+w, u> = <v, u> + <w, u> for all u, v, w in V (additivity)
P4. <rv, w> = r<v, w> for all v and w in V and r in R (homogeneity)
P5. <v, v>  0 with equality if and only if v = 0 (positivity)
• A vector space V with an inner product is called an
inner product space.
12
Wen-Hsien Fang Engineering Mathematics (II) 12/85
4.1.1 Inner Product

Some Common Inner Products

Examples 4.1-1 (Euclidean Space R n )
The dot product
 v1   w1 
n v  w 
v, w  v  w   vi wi , where v 
R3  2
,w   2

i 1   
   
vn   wn 
is a legitimate inner product in R n .

13
Wen-Hsien Fang Engineering Mathematics (II) 13/85
4.1.1 Inner Product

Next, we show that such a defintion satisfies all axioms:

P1. It is obvious.
n n
P2. v, w   vi wi  wi vi  w , v
i 1 i 1

P3. v  w , u   v  w   u
n n n
  (vi  wi )ui   vi ui   wi ui  v, u  w, u
i 1 i 1 i 1
n n
P4. rv, w  ( rv )  w   rvi wi r  vi wi  r v, w
i 1 i 1
n
P5. v, v   v 0 if v  0;  0 if v  0
2
i
i 1
14
Wen-Hsien Fang Engineering Mathematics (II) 14/85
4.1.1 Inner Product

Remarks:
1. The inner product (dot product) in Ex.4-1-1 is referred
to as the standard inner product in R n .
n
There are other possible inner products for R as long as
they satisfy the axioms of being an inner product.

E.g. In R 3 , we could define

 x1   y1 
  
x, y  2 x1 y1  3 x2 y2  4 x3 y3 , where x  x2 , y  y2 
   
 x3   y3 
It is easy to verity that this is also an inner product.
n
In fact,  x, y   wi xi yi is a legitimate inner product
i 1

if wi  0  i 15
Wen-Hsien Fang Engineering Mathematics (II) 15/85
 2. The cosine formula for the dot product (and the
concept of projection) extends naturally to Euclidean
spaces of any dimension (not just 2D or 3D), i.e.
x, y  x  y  x y cos
or, equivalently,
xy x y
cos    (  1 based on C-S  )
x y x y
Here cos , referred to as cosine similarity,
provides a measure of the "similarity" of x and y.
The dot product helps measure a vector's projection
onto another and their directional alignment.
16
Wen-Hsien Fang Engineering Mathematics (II) 16/85
4.1.1 Inner Product

Applications 1: Information Retrieval

If there are m possible key search words and a total of n documents
in the collection, then the database can be represented by an mn
matrix A.
– Each column of A represents a document in the database.
– The entries of the j-th column correspond to the relative
frequencies of the key words in the j-th document.

Q: How to identify the column vectors in the database

matrix that most closely match a given search vector?
A: The cosine similarity between two vectors can serve as a
measure of their similarity.

17
Wen-Hsien Fang Engineering Mathematics (II) 17/85
4.1.1 Inner Product

An Example
Table 1 Frequency of Key Words
Modules
Key words M1 M2 M3 M4 M5 M6 M7 M8
determinants 0 6 3 0 1 0 1 1
eigenvalues 0 0 0 0 0 5 3 2
linear 5 4 4 5 4 0 3 3
matrices 6 5 3 3 4 4 3 2
numerical 0 0 0 0 3 0 4 3
orthogonality 0 0 0 0 4 6 0 2
spaces 0 0 5 2 3 3 0 1
systems 5 3 3 2 4 2 1 1
transformations 0 0 0 5 1 3 1 0
vector 0 4 4 3 4 1 0 3
18
Wen-Hsien Fang Engineering Mathematics (II) 18/85
4.1.1 Inner Product

The database matrix is formed by scaling each column

of the table so that all column vectors are unit vectors.
1
qj  a j j  1,,8
aj
0.000 0.594 0.327 0.000 0.100 0.000 0.147 0.154
0.000 0.000 0.000 0.000 0.000 0.500 0.442 0.309
 
0.539 0.396 0.436 0.574 0.400 0.000 0.442 0.463
0.647 0.495 0.327 0.344 0.400 0.400 0.442 0.309 
 
0.000 0.000 0.000 0.000 0.300 0.000 0.590 0.463
Q 
0.000 0.000 0.000 0.000 0.400 0.600 0.000 0.309
0.000 0.000 0.546 0.229 0.300 0.300 0.000 0.154
 
0.539 0.297 0.327 0.229 0.400 0.200 0.147 0.154
0.000 0.000 0.000 0.574 0.100 0.300 0.147 0.000
 
0.000 0.396 0.436 0.344 0.400 0.100 0.000 0.463
19
Wen-Hsien Fang Engineering Mathematics (II) 19/85
4.1.1 Inner Product

If we set y = QTx, then

yi  q x  cos  i
T
i

where i is the angle between the unit vectors x and qi.

0.000 
0.000 
 
0.000 
To do a search for the key words orthogonality, spaces, 0.000 
vector, we from a (normalized) vector x whose entries  
0.000 
Are all 0 except for the three rows corresponding to the x 
 0 . 577 
search rows.
0.577 
 
0.000 
0.000 
 
0.577 
20
Wen-Hsien Fang Engineering Mathematics (II) 20/85
4.1.1 Inner Product

• In this example:
T
0.000 0.594 0.327 0.000 0.100 0.000 0.147 0.154 0.000
0.000 0.000 0.000 0.000 0.000 0.500 0.442 0.309 0.000
0.539 0.396 0.436 0.574 0.400 0.000 0.442 0.463 0.000
0.647 0.495 0.327 0.344 0.400 0.400 0.442 0.309 0.000
   
y  QT x  0.000 0.000 0.000 0.000 0.300 0.000 0.590 0.463 0.000
0.000 0.000 0.000 0.000 0.400 0.600 0.000 0.309 0.577
0.000 0.000 0.546 0.229 0.300 0.300 0.000 0.154 0.577
0.539 0.297 0.327 0.229 0.400 0.200 0.147 0.154 0.000
0.000 0.000 0.000 0.574 0.100 0.300 0.147 0.000 0.000
0.000 0.396 0.436 0.344 0.400 0.100 0.000 0.463 0.577
   
0.000
0.229
0.567 The search vector x is closest to q5 and hence
 
  0.331 module 5 is the best one that matches our
0.635
0.577
0.000 Search criteria. Modules 1 and 7 do not have
0.535
any of the search words and consequently
y1  q1T x  0 and y7  qT7 x  0 21
Wen-Hsien Fang Engineering Mathematics (II) 21/85
4.1.1 Inner Product

Applications 2: Recommender Systems

**Example: Movie Recommendations**

**Users & Preferences (Genres: Action, Comedy, Drama)**

Alice: [1, 2, 3]
Bob: [2, 3, 4]
Charlie: [3, 4, 5]
**Movies & Genre Scores**
Gladiator: [1, 0, 3]
Deadpool: [2, 1, 1]
Forrest Gump: [0, 2, 4]
**Calculating Inner Product for Alice:**
Gladiator: (1×1) + (2×0) + (3×3) = 10
Deadpool : (1×2) + (2×1) + (3×1) = 7
Forrest Gump: (1×0) + (2×2) + (3×4) = 16
Recommended Movie for Alice: Forrest Gump 22
Wen-Hsien Fang Engineering Mathematics (II) 22/85
4.1.1 Inner Product

Applications 3: A Core Operation in ChatGPT

• Transformer Architecture (ChatGPT):
A powerful neural network architecture uses self-attention to
assign importance to input tokens based on their relevance.
• Concept of Self-Attention
1. Input Sentence: Example: "The cat sat on the mat."
2. Self-Attention Mechanism: Each word attends to every other
word in the sentence, capturing contextual relationships.
3. Attention Scores: These scores indicate the relevance of one
token to another. The attention scores are computed by taking
the inner product of the query and key vectors associated with
each token.
23
Wen-Hsien Fang Engineering Mathematics (II) 23/85
4.1.1 Inner Product

Example 4.1-2 (Function Space (I))

Given polynomials p = p ( x ), q  q ( x ) in Pn , define

1
p, q   p ( x)q ( x)dx
0

Example 4.1-3 (Function Space (II))

C  a, b : continuous function from  a , b  to R, then

f , g   f ( x) g ( x )dx is an inner product on C  a, b 

b

24
Wen-Hsien Fang Engineering Mathematics (II) 24/85
4.1.1 Inner Product

More on Geometric Point of View

f ( x)
b
 f , g   f ( x ) g ( x) dx
a
do not overlap
 g ( x)
 f , g  0, then f ( x) and g( x) 
or vary in the opposite diretions

f ( x)
f ( x)

g ( x)
g ( x)
 f, g large when f ( x ) and g ( x ) are
more "similar"

25
Wen-Hsien Fang Engineering Mathematics (II) 25/85
4.1.1 Inner Product

Properties of Inner Product

Theorem
Let < , > be an inner product on a space V . Let v, u and
w denote vectors in V. and let r denote a real number.
(1) <u, v+w> = <u, v> + <u, w>
(2) <w, rv> = r<w, v>
(3) <v, 0> = 0 = <0, v>

Proof P2 P3 P2
(1) u, v  w  v  w, u  v, u  w, u  u, v  u, w
P2 P4 P2
(2) w , rv  rv, w  r v, w  r w , v
(3)
(3) v, 0  v,0  0  0 v, 0  0, Similarly 0, v  0
26
Wen-Hsien Fang Engineering Mathematics (II) 26/85
4.1.1 Inner Product

Theorem (Preservation of Linear Combinations)

Let < , > be an inner product on a space V

(1) r1v1  r2 v 2    rn v n , w  r1 v1 , w  r2 v 2 , w 
rn v n , w
(2) v, s1w1  s2 w 2    sm w m  s1 v, w1  s2 v, w 2 
sm v, w m

for all ri , si in R and all v, w , v i ,and w i in V

27
Wen-Hsien Fang Engineering Mathematics (II) 27/85
4.1.1 Inner Product

Example 4.1-4

If u and v are vectors in an inner product space, expand

<2u-v, 3u+2v>.

Solution

2u  v,3u  2 v  2u,3u  2u,2 v   v,3u   v,2 v

 6 u, u  4 u , v  3 u, v  2 v , v

 6 u, u + u, v  2 v, v

28
Wen-Hsien Fang Engineering Mathematics (II) 28/85
 4.1.2 Norms and Distance
We can extend the concept of a vector's magnitude in Euclidean
space to an arbitrary vector space through the norm, which
measures its length or "size."
Definition:
A vector space V is said to be a normed linear space if to
each vector vV there is associated a real number ||v||
called the norm of v, satisfying
I. ||v||  0 for every vector v in V (Positivity)
II. ||v|| = 0 if and only if v=0
III. ||rv|| = |r| ||v|| (Homogeneity)
IV. ||v+w||  ||v|| + ||w|| for all v, wV (Triangle
inequality)
Wen-Hsien Fang Engineering Mathematics (II)
29
29/85
4.1.2 Norms and Distance

Recall that the Euclidean length of a vector x in either

R2 or R3 can be defined in terms of the dot product:

  x 2  x 2 if x  R 2
1 2
x  xx   x, x  = 
 1
x 2
 x2
2
 x3
2
if x  R 3

n
if the dot product is employed for the inner product for R

We can extend the above definition to more general

vector spaces.

30
Wen-Hsien Fang Engineering Mathematics (II) 30/85
4.1.2 Norms and Distance

Definition:
If V is an inner product space, then the norm of a
vector v can be defined by
v  v, v , for all v V
Note: v, v  0, so v is a real number
E.g .
 v1 
v 
The length of v   2
in R n is v   v, v   v12  v22   vn2

v 
 n 1
The length of p in Ex. 4-1-2 is p  <p, p >  
0
p 2 ( x )dx
31
Wen-Hsien Fang Engineering Mathematics (II) 31/85
4.1.2 Norms and Distance

Proof:

I & II v  v, v  0, =0 iff v  0

III rv  rv, rv  r 2 v, v = r v

IV ||u+v||2 = < u+v, u+v >

= < u, u > + 2 < u, v > + < v, v >
 ||u||2 + 2 ||u|| ||v|| + ||v||2 (Cauchy-Schwarz Inequality)
= (||u|| + ||v||)2
Thus
||u+v||  ||u|| + ||v||
32
Wen-Hsien Fang Engineering Mathematics (II) 32/85
4.1.2 Norms and Distance

Remarks (for general vector space):

• A vector v is called a unit vector, if v  1.

•  1
We can construct a unit vector by v  v
v

1 1 1
v  v , v 
2
v, v  2
v, v  1
v v v
P4+Theorem

33
Wen-Hsien Fang Engineering Mathematics (II) 33/85
4.1.2 Norms and Distance

Example 4.1-5
1
 1
Find a unit vector v that is a positive multiple of v   
2
0
 
in R 4 .
Solution

 1
v  1  ( 1)  2  6  v  v
2 2 2

6
34
Wen-Hsien Fang Engineering Mathematics (II) 34/85
4.1.2 Norms and Distance

There is an inequality which relates the inner products

and the norms.

Theorem (The Cauchy-Schwarz Inequality)

If u and v are any two vectors in an inner product space
V, then
| <u, v> |  ||u|| ||v||
Equality holds if and only if u and v are linearly
dependent.

35
Wen-Hsien Fang Engineering Mathematics (II) 35/85
4.1.2 Norms and Distance

Proof:
Let x be any scalar, then
0  xu  v, xu  v  x u, u  2 x u, v  v, v
2

Axiom P.5
 x u  x( 2 u, v )  v  (*)
2 2 2

   
a b c
2 2
Let a  u , b  2 u, v , and c  v
2 2
 b  4ac  0  4( u, v )  4 u
2 2
v 0
2 2 2
u, v  u v  u, v  u v
If equality holds in (*), then v  xu,  need xu  v , xu  v  0
36
Wen-Hsien Fang Engineering Mathematics (II) 36/85
4.1.2 Norms and Distance

Example 4.1-6 ( C-S  R)

in n

Show that
1 1 2 2
v w  v w    vn n
w  1 2
v
2
2
 v 2
   vn  1
2
w 2
 w2
2
   wn
2

Proof
 v1   w1 
v  w  2 2 2
Take v   2  , w   2  in R n
  
v, w  v w
v  w 
 n  n

37
Wen-Hsien Fang Engineering Mathematics (II) 37/85
4.1.2 Norms and Distance

Example 4.1-7 (C-S  in function space )

b 2 b b
   f ( x )dx  g ( x )dx
2 2
f ( x ) g ( x )dx
a a a

Solution

2 2 2
f,g  f g

38
Wen-Hsien Fang Engineering Mathematics (II) 38/85
4.1.2 Norms and Distance

Using the norm notation, we can also address the concept

of “distance” between two vectors as:

Definition:
The distance between v and w in an inner product space
V can be defined by d(v, w)= ||v – w||.

v vw
w
39
Wen-Hsien Fang Engineering Mathematics (II) 39/85
4.1.2 Norms and Distance

Example 4.1-8

 x1   y1 
Let x    , y    be any two vectors in R .
2

x 2  y2
If standard inner product is used, then

 x1  y1    x2  y2 
2 2
d(x, y )= x  y 

40
Wen-Hsien Fang Engineering Mathematics (II) 40/85
4.1.2 Norms and Distance

Example 4.1-9
Let the inner product of C  0,1 be defined as
1
f , g   f ( x) g ( x) dx then determine (a) f (b) f  g ,
0

where f ( x)  x, g ( x)  x 2 .
Solution
1 1 1
(a) f  f ,f    f ( x) dx   x dx 
f ( x)  2
0
x x
0 3
(b) f  g  f  g, f  g
1 1
  [ f ( x )  g ( x )][ f ( x)  g ( x)]dx 
0  30
2
x x
41
Wen-Hsien Fang Engineering Mathematics (II) 41/85
 Organization

• Definition of
What? Which ? • Properties
orthogonal set

Orthogonality

• Applications • Gram-Schmidt
Where? How?
(LS) orthogonalization

42
Wen-Hsien Fang Engineering Mathematics (II) 42/85
 4.2 Orthogonality (I) –Basic Concepts
• Recall that in R2 (R3 ), the angle between two vectors is a
right angle if their inner product (scalar product) is 0.

x
<x, y>=0..orthogonal
E.g. x, y = x y cos (perpendicular)

y
(a) The vector 0 is orthogonal to every vector in R2.
(b) The vectors [3, 2]T and [-4, 6]T are orthogonal in R2.
(c) The vectors [2, -3, 1]T and [1, 1, 1]T are orthogonal
in R3.
43
Wen-Hsien Fang Engineering Mathematics (II) 43/85
4.2 Orthogonality (I) –Basics

In general, two vectors u and v in V are said to be

orthogonal if <u, v> = 0.

Orthogonality can be thought of as a generalization of

perpendicularity to any vector space with an inner product.

We can generalize this concept to more (>2) vectors cases.

Definition:
Let {e1, e2, ..., en }be nonzero vectors in an inner product
space V. If <ei, ej> = 0 whenever i  j, then {e1, e2, ..., en}
is said to be an orthogonal set of vectors.

44
Wen-Hsien Fang Engineering Mathematics (II) 44/85
4.2 Orthogonality (I) –Basics

Example 4.2-1
 1  2   4  
      
The set  1 , 1 , 5  is an orthogonal set in R .
3
    
 1  3  1  
      
Solution

1  2  1  4 
 1 ,  1   1  2  1  1  1  (3)  0  1 ,  5  1  4  1  (5)  1  1  0
     
1  3 1  1 
2 4
  1  ,  5  2  4  1  ( 5)  (3)  1  0
   
 3  1 
45
Wen-Hsien Fang Engineering Mathematics (II) 45/85
4.2 Orthogonality (I) –Basics

Sometimes, it is preferred to consider vectors which are

not only perpendicular, but also of unit length. So we
have the following definition:
Definition :
An orthonormal set of vectors is an orthogonal set of unit
vector, i.e. {e1, e2 , , en} are orthonormal if {e1, e2 , , en}
are orthogonal and ei  1 ,  i

We can convert an orthogonal set into an orthonormal

set of vectors. For this, we have the following theorem:

46
Wen-Hsien Fang Engineering Mathematics (II) 46/85
4.2 Orthogonality (I) –Basics

Theorem:
Let {e1 , e 2 ,  , e n } be an orthogonal set of vectors.
(1) {r1e1 , r2e 2 ,  , rne n } is also orthogonal for any
choice of ri  0 in R.
1 1 1
(2) { e1 , e2 , , e n } is an orthonormal set.
e1 e2 en
Proof :
(1) <ri ei , rj e j >  ri rj <ei , e j >=0 , for i  j
{e1 , e 2 ,  , e n } orthogonal.
 0 ,i j
1 1 
(2) { ei , e j }=  1
ei ej  ei , ei  1 , i  j
 e 2
 i
47
Wen-Hsien Fang Engineering Mathematics (II) 47/85
4.2 Orthogonality (I) –Basics

Example 4.2-2  1  2   4  

     
To get an orthonormal set from the set  1 , 1 , 5   
    
 1  3  1  
      
Sol :
Solution

1 2
1 1  1 1  
 u1  v1  1 , u2  v2  1
v1 3  v2 14  
1  3
4
1 1  
u3  v3   5
v3 42  
 1  48
Wen-Hsien Fang Engineering Mathematics (II) 48/85
4.2 Orthogonality (I) –Basics

Example 4.2-3
Show that {e1 =[1, 1, 1,  1]T , e 2 =[1, 0, 1, 2]T , e3 =[  1, 0, 1, 0]T
e 4 =[  1, 3,  1, 1] } is orthogonal in R (with the dot product),
T 4

and find the corresponding normalized set.

Solution

 ei , e j  0 for all i  j

e1  12  12  12  (1) 2 =2 , e 2  12  02  12  22  6
e3  (1) 2  02  12  0 2  2, e4  (1) 2  32  (1) 2  12  2 3
1 1 1 1
 { e1 , e2 , e3 , e 4 } are the corresponding orthonormal set.
2 6 2 2 3
49
Wen-Hsien Fang Engineering Mathematics (II) 49/85
4.3 Orthogonality (II) –Properties

(The Pythagorean Law)(I)

If v and w are orthogonal vectors in an inner product
space V, then
||v+w||2 = ||v||2 + ||w||2
Proof:
||v+w||2 = <v+w, v+w>
= <v, v+w> + <w, v+w>
= <v, v> + <v, w> + <w, v> + <w, w>
= ||v||2 + 2<v, w> + ||w||2 (Note <v, w> = 0)
= ||v||2 + ||w||2
50
Wen-Hsien Fang Engineering Mathematics (II) 50/85
4.3 Orthogonality (II) –Properties

(The Pythagorean Law)(II) :

If {e1, e2 , , en} is an orthogonal set of nonzero vectors in
an inner product space V , then
|| e1  e2    en || || e1 ||  || e2 ||  || en ||
2 2 2 2

Proof:
|| e1  e2    e n |2  e1  e 2    e n , e1  e 2    e n 
 <e1 , e1 >  <e2 , e 2 >     e n , e n     ei , e j 
i j
 ei , e j  0 , for i  j

|| e1  e2    e n || || e1 ||  || e 2 ||   || e n ||
2 2 2 2

51
Wen-Hsien Fang Engineering Mathematics (II) 51/85
4.3 Orthogonality (II) –Properties

Theorem:
If {e1, e2, ..., en} is an orthogonal set of nonzero vectors
in an inner product space V, then e1, e2, ..., en are
linearly independent.

Proof:
Let r1e1 + r2e2 + . . . + rnen = 0
<ej , r1e1 + r2e2 + . . . + rnen> = 0 for all j
 r1<ej, e1> + r2<ej, e2> + ...+ rn<ej, en> = 0
 rj ||ej||2 = 0  rj = 0 for all j
 e1, e2, ..., en are linearly independent.

52
Wen-Hsien Fang Engineering Mathematics (II) 52/85
4.3 Orthogonality (II) –Properties

Remarks:
1. Orthogonality is a stronger statement than linear
independence, orthogonality means that the vectors
are not only linear independent but also orthogonal
to each other.

e.g. w
 linearly independent but not orthogonal
v

Note: orthogonal set linearly independent

53
Wen-Hsien Fang Engineering Mathematics (II) 53/85
4.3 Orthogonality (II) –Properties

2. Suppose B={e1, e2, ..., en} is a basis of an inner product space V

with dimension n.
If {e1, e2, ..., en} is also an orthogonal (orthonormal) set
 B is an orthogonal (orthonormal) basis for V
3. It is generally much easier to work with an orthonormal
basis than with an ordinary basis. In particular, the
coordinates of a given vector v with respect to an
orthonormal basis can be determined easily.

Orthogonality is Good!!!

54
Wen-Hsien Fang Engineering Mathematics (II) 54/85
4.3 Orthogonality (II) –Properties

Expansion of orthogonal/orthonormal basis

Theorem :
Let {e1 , e 2 , , e n } be an orthogonal basis of an inner
product space V . If v is any vector in V , then
 v, e1   v, e 2   v, e n 
v 2
e1  2
e2    2
en
|| e1 || || e 2 || || e n ||
is the expansion of v as a linear combination of the
basis vectors.
 v, e1   v, e n 
2
, , 2
: (generalized) Fourier coefficients
|| e1 || || en ||
of v with respect to the orthogonal basis {e1 , ,e n }
55
Wen-Hsien Fang Engineering Mathematics (II) 55/85
4.3 Orthogonality (II) –Properties

Proof:

Since {e1 , e 2 ,  , e n } is a basis,

v  r1e1  r2e2    rne n for some ri  R.
Using the orthogonality of {e1 , e 2 ,  , e n }, we have
 v, ei  r1e1    rne n , ei 
 r1  e1 , ei     ri  ei , ei     rn  e n , ei 
 0    ri || ei ||   0
2

 ri || ei ||
2

 v, ei 
 ri  2
|| ei ||
56
Wen-Hsien Fang Engineering Mathematics (II) 56/85
4.3 Orthogonality (II) –Properties

Example 4.3-1 (Ex. 4.2-3 cont.)

If R (with the dot product), expand v  [a , b, c, d ] as a
4 T

linear combination of the basis {e1 =[1, 1, 1,  1]T ,

e 2 =[1, 0, 1, 2] , e3 =[  1, 0, 1, 0] ,e 4 =[  1, 3,  1, 1] }
T T T

Solution

The task is to find r1 , r2 , r3 , and r4 such that v  r1e1  r2e 2  r3e3  r4e 4 .

 v, e1  a  b  c  d  v, e 2  a  c  2d
r1  2
 , r2  2

|| e1 || 4 || e 2 || 6
 v, e3   a  c  v, e 4  a  3b  c  d
r3  2
 , r4  2

|| e3 || 2 || e 4 || 12
57
Wen-Hsien Fang Engineering Mathematics (II) 57/85
4.4 Orthogonality (III) –Geometric
Interpretation of Fourier Series
Consider an inner product space C[0,2π] with

f ( x ), g ( x)  02 f ( x) g ( x) dx
It can be shown that
1 1 1 1 1
, cos x,, cos nx, sin x, , sin nx,
2    
is an orthonormal basis of C[0,2π].

Every function f  C  0, 2  can thus be expressed as

58
Wen-Hsien Fang Engineering Mathematics (II) 58/85
4.4 Orthogonality (III) –Geometric Interpretation of FS

1 1 1 1
f ( x ) = <f ( x ),   < f ( x) , cos x  cos x   
2 2  
1 1
< f ( x) , cos nx  cos nx   
 
1 1 1 1
< f ( x) , sin x  sin x    <f ( x ) , sin nx  sin nx  
   
1 2 1  2 1  1
= 0 f ( x) dx   0 f ( x) cos x  dx cos x   
2
  2      
1 2  1 2 f ( x ) cos xdx  cos x
0 f ( x )dx
2  0 

 2 1  1  2 1  1
  0 f ( x) sin x  dx sin x     0 f ( x) sin nxdx  sin nx  
    
  
 1 2 f ( x )sin xdx  sin x  1 2 f ( x )sin nxdx  sin nx
 0   0 
Fourier Series Expansion
59
Wen-Hsien Fang Engineering Mathematics (II) 59/85
4.4 Orthogonality (III) –Geometric Interpretation of FS

Remarks:
Importance of Fourier Analysis
1. Representation of Periodic Signals – Almost all periodic
functions can be expressed as a sum of sine and cosine functions,
forming a basis.
1. Physical Interpretation – The basis functions correspond to
harmonics with distinct frequencies.
2. Easy Computation of Expansion Coefficients– Since the basis is
orthogonal, expansion coefficients are easily determined.
3. Convenient Analysis in Linear Systems– Harmonic components
behave predictably in linear time-invariant systems (see Chap. 6).

These properties make Fourier analysis essential in signal

processing, physics, and engineering. 60
Wen-Hsien Fang Engineering Mathematics (II) 60/85
4.5 Orthogonality (IV) : Gram
-Schmidt Orthogonalization

The Gram-Schmidt orthogonalization process is a

process for systematically constructing an orthogonal
basis for an inner product space V.

Key point: projection

Let us first consider a simpler case: transform an

ordinary basis {v1, v2} into an orthogonal basis {e1, e2}.

61
Wen-Hsien Fang Engineering Mathematics (II) 61/85
4.5 Orthogonality (IV): G-S Orthogonalization

v2
 e1  v1 e1  v1
 e2
e 2  v 2   v1
 v2 , e1 
2
e1
|| e1 ||

  ?  e 2 , e1  0  v 2   e1 , e1  0



e2

 v 2 , e1   e1 , e1  0
v 2 , e1
  2
 e1 , e1 || e1 || 2

|| e1 || 62
Wen-Hsien Fang Engineering Mathematics (II) 62/85
4.5 Orthogonality (IV): G-S Orthogonalization

We can extend the above to any arbitrary number of

vectors. For this, we have the following theorem:
(The Gram-Schmidt Process)
Let {v1, v2, ..., vn} be a basis for the inner product space
V. We can construct an orthogonal basis {e1, e2, …, en}
for V recursively by
e1  v1

and for m=2, …,n

 v m , e1   v m , e2   v m , e m1 
em  v m  2
e1  2
e2    2
e m 1
e1 e2 em 1

63
Wen-Hsien Fang Engineering Mathematics (II) 63/85
4.5 Orthogonality (IV): G-S Orthogonalization

Proof:

For all 1  i  m  1
m 1  v , e 
 e m , ei    v m   m
2
j
e j , ei 
j 1 ej
m 1  vm , e j 
  v m ,ei   2
 e j ,ei 
j 1 ej
 e 2 , j  i
  v m ,ei    v m ,ei   0 
j

 0 , j  i

64
Wen-Hsien Fang Engineering Mathematics (II) 64/85
4.5 Orthogonality (IV): G-S Orthogonalization

Example 4.5-1
Given the orthogonal set 1,0, 2 ,  2,3,1 in R ,
3

find an orhgonal basis of R 3 containing these vectors.

Solution  v ,e   v ,e 
e v  3
e  2
1
e 3
2
2

e e
3 3 1 2

1 2

1 2
 (1,0,0)  (1, 0, 2)  (2,3,1)
5 14
1
  (35, 0, 0)  (7, 0, 14)  (10,15,5)
35
1 3
 (18, 15,9)  (6, 5,3)
35 35 65
Wen-Hsien Fang Engineering Mathematics (II) 65/85
4.5 Orthogonality (IV): G-S Orthogonalization

Examp1e 4.5-2
U  span  v1  (1,1, 1, 1), v 2  (3, 2,0,1), v 3  (1,0,1,0)
Find an orthogonal basis for U.
Solution
e1  v1  (1,1, 1, 1)
 v 2 , e1  4
e2  v 2  2
e1  (3, 2, 0,1)  (1,1, 1, 1)  (2,1,1, 2)
e1 4
 v3 , e1   v3 , e2  0 3 1
e3  v 3  2
e1  2
e2  v3  e1  e 2  (4, 3,7, 6)
e1 e2 4 10 10

 1

(1,1, 1, 1), (2,1,1, 2), (4, 3,7, 6) is an orthogonal basis for U
10
66
Wen-Hsien Fang Engineering Mathematics (II) 66/85
4.5 Orthogonality (IV): G-S Orthogonalization

Example 4.5-3 1
Consider V  P4 with the inner product p, q  
1
p ( x) q( x) dx.
If the Gram-Schmidt algorithm is applied to the basis 1, x , x 2 3
,x ,  
Solution
3 
show that the result is the orthogonal basis 1, x, 1 (3x 2  1), 1 (5 x3  3x)
5 
Take e1  1
 x , e1  0 the sequence of
e2  x  2
e1  x  e1  x Legendre polynomials
e1 2
2
2  x 2 , e1   x2 , e2  2 3 0 1
e3  x  e1  e2  x  1 x  (3 x 2  1)
e1
2
e2
2 2 2 3
3

1
e 4    (5 x 3  3 x )
Wen-Hsien Fang
5 Engineering Mathematics (II)
67
67/85
 Adrien-Marie Legendre
• 阿德里安-馬里·勒讓德（1752.09.18~1833.01.10）
，法國數學家。
• 他的主要貢獻在統計學、數論、抽象代數與數學分
析上。勒讓德的主要研究領域是分析學(尤其是橢圓
積分理論)、數論、初等幾何與天體力學，取得了許
多成果， 導致了㇐系列重要理論的誕生。勒讓德是
橢圓積分理論奠基人之㇐。勒讓德對數論
的主要貢獻是二次互反律，這是同餘式
論中的㇐條基本定理。他還是解析數論
的先驅者之㇐，歸納出了素數分布律，
促使許多數學家研究這個問題。其他貢
獻包括：橢圓函數論、最小二乘法、
測地線理論等。
68
Wen-Hsien Fang Engineering Mathematics (II) 68/85
4.5 Orthogonality (IV): G-S Orthogonalization

Example 4.5-4
Consider C[0,3], with f ( x ), g ( x )  03 f ( x) g ( x) dx
Use the Gram-Schmidt algorithm to get an orthogonal set e1 , e 2 , e3 
from v1,v2,v3
1 v1 v3
1

3 -1 2 3

1 v2

1
69
Wen-Hsien Fang Engineering Mathematics (II) 69/85
4.5 Orthogonality (IV): G-S Orthogonalization

Solution 1 e1 e2
2/3
e1  v1  1 3
3 -1/3
1
 v 2 , e1  01dx
e2  v 2  2
e1  v2  3 e1 e3
e1 0 1dx
1
 v 3 , e1   v3 , e2  1 2 3
e3  v 3  e1  e2
e1
2
e2
2 -1
1 2 dx  2 (  2 )dx  3 (1 ) dx
 021dx  031dx  3
0 1
3
2
3
 v3  e1  e2
1 ( 2 ) 2 dx  3 (1 ) 2 dx
3
0 1dx
0 1
3 3
1
 v3  e1  e 2
3
70
Wen-Hsien Fang Engineering Mathematics (II) 70/85
 4.6 Orthogonality (V) –
Projection Theorem

In many applications, the minimization problem can

be formulated as finding the minimum distance from
a vector to a subspace in a vector space. For this, we
have the following Projection Theorem for the
general vector space.
v
v  ProjU (v)

ProjU (v ) ~U

71
Wen-Hsien Fang Engineering Mathematics (II) 71/85
4.6 Orthogonality (V) – Projection Theorem

Theorem (Projection (Approximation) Theorem) (I):

Let U be a finite dimension subspace of an inner product
space V. If v is any vector in V , then ProjU(v) is the
vector in U that is closest to v, i.e.
|| v – ProjU(v) || ≤ || v – u || for all u  U,
where ProjU(v) is a vector that satisfies

< v – ProjU(v) , u > = 0 for all u U

72
Wen-Hsien Fang Engineering Mathematics (II) 72/85
4.6 Orthogonality (V) – Projection Theorem

Proof : v
v ProjU (v)
Let u be any vectors in U ,
ProjU (v )

2
2
v  u = (v  projU (v ))+(projU (v )  u )
     
U U
2 2 2
 v  projU (v )  projU (v )  u  v  projU (v )

The equality occurs when u = projU (v )

73
Wen-Hsien Fang Engineering Mathematics (II) 73/85
4.6 Orthogonality (V) – Projection Theorem

• Applications-Least Squares Fitting (linear regression)

Determine the selling price that would In thousands of bottles
y
maximize its profits: A perfume company (6.25, 6.03)
wished to express its weekly sales y (in
(8, 4.78)
unit of thousand bottles) as a linear y  a  bx
function of the price x (dollars).
(6.75, 5.62)

(8.75, 4.34) x
City 1 City 2 City 3 City 4
In dollars per bottle
x 6.25 6.75 8.00 8.75
y 6.03 5.62 4.78 4.34

74
Wen-Hsien Fang Engineering Mathematics (II) 74/85
4.6 Orthogonality (V) – Projection Theorem

Solution
a  6.25b  6.03 (City1) 
a  6.75b  5.62 (City2) 
y  a  bx   
 # of eqns.=# of measurements
a  8.00b  4.78 (City3) 
a  8.75b  4.34 (City4) 

It's quite possible that there is no solution for a & b

 Consider the optimal fitting in the least squares sense
Choose a, b to minimize
2 2 2
(a  6.25b  6.03)  (a  6.75b  5.62)  (a  8b  4.78)
2
(a  8.75b  4.34)
75
Wen-Hsien Fang Engineering Mathematics (II) 75/85
4.6 Orthogonality (V) – Projection Theorem

Choose a line such that sum of the squares of the vertical distance
from the points to the line y  ax  b is minimum.

We can also solve this problem using

matrix/vector notation:

1 6.25 6.03 
1 6.75 a 5.62 
     
  
1 8.00    4.78 
b
1 8.75  x  4.34 
  
A b
76
Wen-Hsien Fang Engineering Mathematics (II) 76/85
4.6 Orthogonality (V) – Projection Theorem

In general no solution when m >n

mn n1 LS m1
A xb (Determine x  Ax - b is min.)

From the projection theorem

b
(b  Ax LS )  span a1 , , a n 

 
AxLS
a  0 ~ span a1,,an
T
1
   
    (b  AxLS )    
aTn  0
 
  
AT

77
Wen-Hsien Fang Engineering Mathematics (II) 77/85
4.6 Orthogonality (V) – Projection Theorem

 A (b  AxLS )  0  A b  A AxLS
T T T

Normal equation
1
 x LS  ( A A ) A b
T T

1 6.25 6.03 
1 6.75 5.62 
A  , b 
1 8.00   4.78 
1 
8.75   
  4.34 

 a  10.2, b  0.67  y  10.2  0.67 x

78
Wen-Hsien Fang Engineering Mathematics (II) 78/85
4.6 Orthogonality (V) – Projection Theorem

約翰·卡爾·弗里德里希·高斯(1777~1855)
• 德國著名數學家、物理學家、天文學家、大地
測量學家。高斯被認為是歷史上最重要的數學
家之㇐，有「數學王子」的美譽。
• 19歲的高斯得到了㇐個數學史上極重要的結果，
就是《正十七邊形尺規作圖之理論與方法》。
• 穀神星於1801年被義大利天文學家皮亞齊發現，
但因病他耽誤了觀測，從而失去了這顆小行星的
軌跡。高斯通過以前3次的觀測數據，在最小
平方法基礎上創立的測量平差理論 的幫助下，
GAUSS 計算出了穀神星的運行軌跡。奧地利天文學家
海因里希·歐伯斯根據高斯計算出的軌道成功地
發現了穀神星。

79
Wen-Hsien Fang Engineering Mathematics (II) 79/85
4.6 Orthogonality (V) – Projection Theorem

• World Petroleum Consumption by Least Squares Fitting

(from Boyd VMLS)

80
Wen-Hsien Fang Engineering Mathematics (II) 80/85
4.6 Orthogonality (V) – Projection Theorem

Remarks:
• Find a least squares curve (a linear function, a
polynomial, a trigonometric polynomial, etc.) fits to a
set of data points in the plane.
E.g. (Polynomial fitting)
P1
y  a0  a1x  a2 x +  aP1x
2

• The curve provides an optimal approximation in the

sense that the sum of squares of errors between the y
values of the data points and the corresponding y
values of the approximating curve are minimized.

81
Wen-Hsien Fang Engineering Mathematics (II) 81/85
 4.6 Orthogonality (V) – Projection Theorem

• Polynomial Least Squares Fitting of Data (from Boyd

VMLS)

82
Wen-Hsien Fang Engineering Mathematics (II) 82/85
4.6 Orthogonality (V) – Projection Theorem

Remarks:
In general, we can only find ProjU(v) by the
property < v – ProjU(v) , u > = 0 for all u U.
However, if we have an orthogonal basis for U,
then we can easily find ProjU(v).
Theorem (Projection (Approximation) Theorem)(II):
Assume {e1, e2, …, em} is an orthogonal basis for U,
then
 v1 , e1   v1, e 2   v1 , e m 
ProjU ( v )  2
e1  2
e2    2
em
e1 e2 em

83
Wen-Hsien Fang Engineering Mathematics (II) 83/85
4.6 Orthogonality (V) – Projection Theorem

Proof :
projU (v )  U ,and e1 ,, e m  is an orthogonal basis

 projU (v )=1e1     me m
From the above, < v  (1e1     me m ),ei >=0, i =1,...,m
 v, e1 
Take i  1  1  2
e1

 v, e m 
Take i  m   m  2
em
84
Wen-Hsien Fang Engineering Mathematics (II) 84/85
 4.6 Orthogonality (V) – Projection Theorem

Examp1e 4.6-1 (Ex 4.5-2 cont.)

Determine the vector in
U  span e1  (1,1, 1, 1), e 2  (3, 2, 0,1)
that is cloaset to w =(2,0,1,5).
Solution Based on projection theorem, this vector is
 w, e1   w, e2 
ProjU (w )  2
e1  2
e2
e1 e2
4 15
 (1,1, 1, 1)  (2,1,1, 2)
4 10
1 5
 (2, , , 4)
2 2
85
Wen-Hsien Fang Engineering Mathematics (II) 85/85