Curve Fitting and Least Squares Principles
Curve Fitting and Least Squares Principles
Curve FittingandPrincipleo!
LeastSquares
9·1. Curve Fitting. Let (x., Yi) ; i = I, 2, ... , n be a given set of n pairs
of values, X being independent variable and Y the dependent variable. The
general problem in curve fitting is to find, if 'possible, ail analytic expression
of the form Y ==1 (x), for the functional relationship. suggested by the given data.
Fitting of curves to a set of numerical data is of considerable importance-
theoretical as well as practical. Theoretically it is useful in the study of correla-
tion arid (egressjon, e.g., lines of regression can be regarded as fitting of linear
curves to the given bivariate distribution (c.c. § 10·8'1). In pfactical statistics .it
enables us to represent the relationship between two var~bIes by simple al-
gebraic expressions, e.g., polynomials, exponential or logarithmic functions.
Moreover, it may be used to estimate the values of one variable which 'would
correspond to the specified values of the other variable.
9·1·1. Fitting of a straight line. Let us• consider the 'fitting of a straight
line
Y=a+bX ...(9,1)
to a set of n points (x.. Yi) ; i = 1,2, ... , n. Equation. (9·1) represents a family of
straight lines for different values of the arbitrary constants' a' and :b' . The problem
is to determine 'a' and 'b' so that the line (9(1) is the line of " best fit ".
. The term 'best, fit' is interpreted in accor~ce with Legender.'s principle
of least squares which consists in minimising the sum of the squares of the
deviations of the actual values of y
from their estimated values as given
'by the line Of best fit.
Let Pi (Xi. Yi) be any general
point in the scatter diagram (§ 10·2).
Draw Pi M 1. to x-axis meeting the
lin~, (9·1) ilJ Hi: Abscissa of Hi is
Xi and since H;. lies on (9·1), [Link]-
dinate is a + bx i. Hence the 'co-or-
dinates 'of Hi are (Xi. a + bXi).
piHi=PiM-HiM
=Yi-(a+bxi),
-O~-------M~---------~X
is Called the error of estimate .or the
residual for Yi.
According 'to the principle of,
Fundamentals of Matbematical Statistics
is minimum. From the principle [Link] and minima, the partial derivaUves of
E, with respect to (w.r.t.) a and b should vanish separately, i.e .•
aE' n BE n
-=0=-2 L (y.-a-bx,) and-=O=-2 L x.(y.-a-bx,) ... (9'2)
Oa ;=1 1 1 Bb ;=\ 1 1 1
n n .n n n 2
=> .L Yi
1=\
=na + b .L X;
I=\'
and .L xiY;
1=\
=a .L x; + b .L Xi
1=1 1=\
... (9'20)
Equations (9'2) and (9'2a) are:known as the normal equations for estimating
a and h.
'n n 2 n n
All the quantities L x; E x;, L y; and L x;Y;, can be obtained from
;=\ ;=\ ;=\ ;=(
the given set of points (x, Yi); j = .1, 2, ... , nand .the equations (9'2a) can be
solved for a and b. With the values of a and b so obtained, equation (9'1) is
the line of best fit to the given set of poin~ (Xi' Yi); i = 1, 2, ... , n.
Remark. The eqmllion of the line of best. f\t of Y on x is obtained on
eliminating a clOd b in (9' !) am) (9'2a) and can be expressed in the determinant
forlp as follows:
y X
. LY; LX; n ~O ...(9·2b)
LX;Y; LX; LX;
9-1-2. Fitting of second degree I)araboln. 'Let
Y = a + hX + e}{2 ... (9'3)
be the second degree parabola of best fit to set of n points (Xi' Yi); i = 1, 2, ... ,
n, Using the principle of l¥ast squares, we have to determine the constcints 0,
band e so that
2 ~
n
L (v-I -a-bx.1 -ex·1
E= ;=( r
is mininuun.
Equating to zero the partial derivatives of E with respect to a, h clOd c
separately, we get the normal equations for estimating Q. band c as
BE • 1 -a -bx.1 -cx1 )
fJa = 0 =..,..2 L(y.
2
DE 2
-Db = 0= -2 LX.(y. -a~bx· -ex·1 )
1 1 1
...(9,4)
0: = = 0 -2 L xi(Y; -a-bx; -cxi>
Curve Fittillg IIl1d Prilldple of lellst Squares 9·3
y X2 X
LYi LX~I LXi n
=0
LXiYi LX~I LX~I LXi ... (9·4b)
LX; Yj LX4I LX3I LX2I
is minimum. Thus the normal: equations for estimating ao, aI' .... ak are obtained
~n equating to zero the partial derivatives of f
w.r.t. ao, ai' ... , ak separately,
I.e.•
BE 2 I:
-;-- =0 =-2 L(Yi -ao -aJ Xj -a~ Xi - ... - al;xi )
vao
aE 2 I;
--=0=-2 LXj(Yj-aO-a J Xj -a2 Xj - ... -al;xj )
aaJ ... (9'6)
BE k 2 I;
-;--=0=-2LXj (Vj-aO-a J xi -a2 xj - ... -al;xj )
val;
~
8 95·5 5 25 477·5
9 ](\2·2 7 49 71~-4
10 IOS·4 9 81 975·6
Total 796·2 0 330 1016·8
Thus the normal equations are
796·2= lOa + 0 x band 1016·8 =a x 0+ 330b,
1016·S
which give a = 79·62 and b =~ = 3·0S (approx).
EXERCISE 9 (a)
1. (a) (Xi, Yi) ; i = 1,2, ... , n, give the co-ordinates of n points in- a plane.
It is proposed to fit a straight line Y =aX + b to. those points such that the sum
-of the squares of the perpendiculars from those n points to the line is a mini-
mum. Find the constants a and b. Use the above [Link] to 'fit a straight line
to the following points :
X: 0 2 3 4
Y: 1 I·S 3·3 4;5 6-3
Ans. Y=O·72+ 1·33X
(b) Fit a [Link] of the form Y =AX + B to. the following data:
X: 0 5 10 15 20 '25 30
Y: 10 14 19 25 31 36 39
2. Show that the line' of best fit to the following data is given by
Y=-0·5X + S
X: 6 77'S 8 8 9 9 10
Y: 5 5 4 5 4 3 4 3 3
3. (a) How do you define ~e ter:m "line of best fit". Give the normal
equations generally used to obtain such a line. Fit a straight line and parabolic
curve to the following data :X :1·01·52·02·53·03·54·0
Y: I-I 1·3 1·6 2·6 2·7 3·4 4·1
Ans. Y = t-()4 - O· 20X + 0- 24X 2
(b) Fit a straight line to the following data. P,Iot the observed and the ex·
("urn' Fittin\! ,IOI! I'rincillll' ,,' Ll'ast Squarl's 97
(h) FiL a ~LraighL line LO the following daw. PIOL Lhc observcd and Lhc cx-
I)CCLCd values III a graph amll~xamil)c wheLhcr the sLraighL line givcs an adequaLc
rlL
x... I 2 3 4 5 6 7 R
55
y... 46 40 38 33 30 29 30
4. An expcrimcnt is conductcd to verify the law of falling under gravity
expressed by . S = ~ gt 2
where S is the distance fallen at time t and go is a gravitational constant. The
following resulL<; arc obtained:
t(seconds): 1 2 3 4 5
S ( feet) 15 70 14<r 250 380
Taking S as thc dcpenent variable, fit a stmight line to the data by the
method of least squares in a manner thar you can estimate g. What is the estimate
of g? .
S. (a) Explain the 'method of fitting a second degree parabola by using the
principle <,>f lea<;t squares.
(b) Fit a parabola Y = a + bx + ex 2 to the following data :
X; I 2 3 4 5 6 7
Y: 2·3 5·2 9·7 16·5 29·4 35·5 544
6. Fit a second degree parabola to the following data taking X as the
independent variable :
X ': I 2 3 4 5 6 7 '8 9
Y: 2 6 7 ,8 10" 11 11 10 9
An~. Y = - 1 + 3·55X - 0·27X 2
7. In a spectroscopic method for determining the .per cent X of natural
rubber-content of vu\cani7..ates, the variable Y used is 1 + 10glO r, where r is the
ratio of transmission at two selected wavelengths. In order to establish a relation-
ship between X and Y, the following data were obtained :
X: 0 20 40 60 80 100
Y: 2·19 2·65 3-16 3·57 3·93 4·27
Using least square method, fit a parabola. Comment on your results.
8. Fit ~ second degrec curve'Y = a + bX + eX 2 to the following data relat-
ing to- profjt of a certain [Link] ..
Year : 1980 1982 1984 1986 1988
Profit in lakhs o{rupees.: 125 140 165 195 230
Estimate the profit in the year 1995.
Ans. Y=114+7·2X+3·15X 2
9. Explain the method of least squares of fitting a curve to the given mass
of data :
X:. -2 -1 I) 2
Fundamentals (If Mathematical Statistics
Y: YI Y2 Ya Y4 Ys
Fll a parabola Y = a + bX + c (X 2 - 2), by lhe ,melhod of leasl squares and
show lhal
- b=IO(-2
a=y, I 2 1 2
Y I-Y2+ Y4+ ys), C=14(2Y I-Y2+ Y4+ ys)
10. Show lhal lhe besl filling linear function for the points (x\,Y\),
(X2, Y2), ... , (XII' YII) may be expressed in lhe form
x Y 1
EXi EYi n =0, (i=1,2, .. ,n).
Ex? LX;Yi LXi
Show that lhe line passes through the mean point (x, Y).
9.·2. Most Plausible Solution of a System of L..ine~r Equations.
Method of least squares is helpful in fmding the most plausible values of the
variables satisfying a system of independent linear equations whose number is
more than the number of variables under study. Consider the follQwing set of
m equations in n variables X, Y, Z, ... , T :
a\X+b\Y+c\Z+ ... +k\T=/\ }
a2X + blY + c2Z + .. , + k2T =h ...(9·8)
a",)( + bmY + cmZ + ... + k".T= 1m
where aj, bj, ... , 'i; i =--1, 2, ... , m are constants.
If m = n, the system of equations (9·8) can 'be solved uniquely with the help
of algebra. If m > n, it is not posible to detennine a u,nique solution X, Y, Z, ... ,
T which will satisfy the system (9·8). In this case we find th& values of X, Y,
Z, ... , T which will satisfy the system (9·8) as nearly as possible.
Legender's principle of least squares Consists in minimising the-sum Of the
squares o'f the 'residuals' or the 'errors'. If
Ej = aX + bjY + c;Z + ... +- kiT -/j; i = 1,2, .. , m
is the residual for lhe ilh equation, then we have to determine X, Y, Z, ... , T so that
m m
U = L E? ~ L (aj X + bjY + 'Cj Z + ... + kj T _ /j)2
j .. I ;= I
is minimum.
Using the principle of maxima and minima in differential calculus, the par-
tial derivatives of 'U' w.r.t. X, Y, Z, ... , T should vanish separately.
Thus
Curve Fitting lind Principle (If Lellst Squares 99
8
(/ = antilog (A) and fl=--
loge
Example 9·5. Fit an exponential curve of the form Y = ab~ 10 Ihe
following data :
X: I ·2 3 4 5 6 7 8
}' : 1·0 1·2 1·8 2·5 3·6 4·7 6·6 9·1
Solution.
X y V = log Y XU X2
t 1·0 0·0000
0·0000 1
2 1·2 0·0792
0·1584 4
3 1·8 0·2553
0·7659 9
4 2·5 0·3979
1·5916 16
5 3·6 2·7815
0·5563 25
6 4·7 0·6721
4·0326 36
7 6·6 0·8195
5·7365 49
8 9·1 0·9590
7·6720 64
total 3~ 30·5 . 22·7385
, .. 3·7393 204
(~rlla) gives the nonnal equations as
3· 7393 ::= 8A + 368
and 22· 7385 =36A + 2048
Solving. we get
8 ::= 0·1408 and A =~·t662 = f.8338
f
Ei = ( Yi - axi - ~J
According to the principle of least squares. we have to detennine thy values of a
and b so that sum of the Squares of errors £, viz.,
E= E E?::= E
n n ( b
Ji-axi--. J2
;=1 ;=1 XI
is minimum.
Consequently. the normal equations are
iJE
da
=0 ::= - 2
;= 1
i Xi ( Yi - ax; - ~XI J
9·12 Fundamentals of Math<''Il1atical Statistics
dE n I ( b '\
db
- = 0 =- 2
i=lxi
1: - Yi - axi - -
x,
J
which on simplificalion give
n n
1: Xi Yi = a 1: xl + nb
i= I i= 1
and i [~)=:na
i=1 x,
+ b .i [-~)
x, i='l
Example 9·7. Three independent measurements on each of the
three angles A, B, C of a tria'lgle are as follows,'
ABC
39·5 60·3 80·1
39·3 62·2 80·3
39·6 60·1 80·4
Obtain the best estimates of the three angles taking into account the relation 'that
the sum ofthe angles is equal to 180°.
Solution. Let the Lhree observations on A be denoted by XI, X2, X3, on B
by y., Y2, Y3 and on C by ZI, Z2, Z3. ~t 9.,92 be the best estimates for A and
B respectively.
According to the principle of least squares, our problem is to estimale
9. and 92, so that
E =1: (Xi - 91)2 + 1: (yj - 92)2 + 1: (Zj - 180 + 91 + 92)2
is minimum, summation being taken over i from 1 to 3.
Equating to zerQ the ,partial derivatives of E w.r.t. 91 and 92, the. nonnal
equations are
dE
:l9.
o
=0 =- 1: (Xi - 9d + 1: (Zj - 180 + 9. + 92)
.. ,(*)
dE
a92 = 0 = -1: (yj - 92) + 1: (Zj - 180 + 9. + 92) ...(**)
From {*) and (**), we get
391 - 1: Xi + 1: Zj - 540 + 39. + 392 = () }
392 - 1: Yj + 1: Zj - 540 + 39. + 392 = 0 ... (***)
But 1: x, =39·5 + 39· 3 + 39·6 = 118·4
1: Yj '" 60·3 + 62·2 + 60·1 = 182·6
1: Zj = '80·1 + 80· 3 + 80·4 = 240·8
Substituting in (***), we get
69.+392-417·6=0 and 39.+692-481.8=0
:. ~ =Q1 = 39.27,h =Q2 = 60·66 and t = 180 - QI - G2 = 80·07
cur"" Fitting nnd Pri,nciple of lellst Squnrcs 9,13
i x
Y n(n+l)(2n+l)
Iy; 3
o an+ 1
n(n+l)(2n+l) .0 =0
IX;Yi o 3 n (m + 1 )( 2n + 1 )
r.x~ Yi n2 (n+ 1 )2 n 3
2
[Delhi Univ. B.A. (Pass), 198'1)
Hint. Use (9·4b), with
xi=a+i; i= 1,2, ... ,(2n+ I). Since x=O,
l:xi=(2n+ I)a+l:i => 0-(211+ I)a+ (2n+ 1)(2n+2)
- 2
a=-(n+ I)
,i
l:Xi 2 = l: (a + = (2n + oi + l:i 2 + 2d ~i
and so on, for l: x? and 1; Xi4 •
[ iil=m(m+l)(2m+I); i;3=[m(m+l)f
i= 1 6 i= I 2
and m
i -: t= 1
30 m(m + 1)(2m + 1)(3m2 + 3m - I)
]
(b)
.
When do we prefer logarithmic curve to ordinary curve?
9·5. Curve Ji'itting by Orthogonal Polynomials. Suppose that the
polynomial of pth degree of Y on X is
= 2
Y ao + alX + az X + ... + ap X p
'
...(9·13)
The normal equations for detennining the .constants als are 'obtained by
the principle of least squares by minimising the residual or error sum of squares
E=l:(y-ao-alx-a2X2- ... -apxP)'" ...(9·14)
summation being extended over the given set of observations. The nonnal equa-
tions are:
=J..Lpl
aof..lp+al f..lp+1 +a2Ilp+2+ ... +apll2p
Solving (9·16) for ao. a1. ...• up in terms of the moments 11, 's and J..Ljl's.
j = O. 1.2 ..... p and substituting in (9·13) we get the required curve of be~t fit.
Let
J..Lp
IIp+ I ...(9·17)
...(9·18)
The required curve of best fit is the eliminant of aj's in (9·13)iand (9·16)
and is given. by
Y 1 X
~I J..Lo J..LI
J..LII J..LI J..L2 ...(9·19)
the polynomial is independent or the other so that each of them can be calculated
[Link]. In this method. the coefficients computed earlier remain the same
and we have to compute the coefficient only for the added term.
95·1. Orthogonal Polynomials (Del). Two polynomials p)(x) and
P2(X) are said to be orthogonal to each other if
1:. Pl(X) P2(X) = o. ...(9·20)
where summation is taken over a specified set of values of x. If x were a
continuous variable in the range from a to b, the condition for orthogonality gives
b
I PI(X) P2(X) dx = 0
a ...(9·2Oa)
For example. if we take
Po = [Link](x) = X-4.P2(X) = X2 - 8x+ 12.P3(x) =x 3 - 12t2 +4lx - 36
.•.(9·20b)
then these are orthogonal to each other for a set of integral values of x from 1 to
7 as ~xplained in the following table. Other examples of orthogonal polynomials
are HCrinit~ polynomials. Gram Charlier's polynomials. Legender.·s polynomials.
etc.
ORTHOGONALITY OF POLYNOMIALS DEFINED IN (9·20b)
constants which can be assigned arbitrarily. We will take one for each polynomial
Pj (j = 0, 1, 2, ... , p) and assign it such that the coefficient of x1 in Pj is unity
i.e.,
cjj = I,} =. 0, 1, 2, "', p, ...(9.27)
In particular c oo = Po = l. The orthogonality conditions give:
~(f
x }=o
cpj .xjJxr = 0
=> f (c ~x xJ+r )
j=O PJ
= 0
~l ~2'" -~p~l'" ~P
where !!.(p)h~l" been defined in (9 1'7) and !!.(P)PI is the minor 01 the element
in the last row and (j + 1)th column in !!.(P). SubstilUling lhis value of CPI in (9·26),
wegct
1
Pp =-- ...(9·32)
!!.(p-I)
1lP-1 IIp
11:>+ 1 1l2p-1
) X x2 xP
In particular if 110 = 1, III = 0 and 112 = I, i.e.~ if x is a standardised variate
then thc orthogonal polynomials are given by
Po= 1 ... (9·33)
PI (x)
17 ~l
----:::;x ... (9·330)
110
110 III 112
III 112 113
1 X x2
Pix) =-,-Ilo--Il-l -1- =x 2 -1l3 x - I ...(9·33b)
III 1l2.
110 III 112 113
110 III 112
P3(X) = III 112 113 J.14 ...(9·.33c)
+ III 112 113
112 113 J.4 Ils
1 X x2 x3 112 113 J.4
and soon.
If we further assume that x is· a standard normal variate so that 113 = Ils
= '" =1l2r+ 1= 0, then the above orthogonal polynomials are called Hermite
Polynomials and are given by
'\ 2· 3' 4
Po=l;PI(x)=x;Pix)=x -I;P3(x)=x -3X;,P4(X)=X -6x 2 +3;
and so on,. where x is a continuous r. v. taking values from - 00 to 00. ...(9· 34)
Remark. Hermite Polynomials defin~.d in (9·34) are orthogonal w.r.t. the
weight function
922 Fundamentals of Mathematical Statistics
00
. r.,p/=r., (
x
.!
)=0
CPii] xp=.! (Cpjr.,xp+jJ
)=o-x
l!.- P 6.(P) .
= N . r. Cpj IlP +j = N . r., (P-"~) ·1lP+j [From (9·31)]
)=0 )=0 6.
~ ·~I IlP
N ~J ~2 1lP+ I
=--
IlP-J IlP
6.(p-J) ~1p-l
IlP ~p+ I ~1p
[Proceeding exactly as we obtained (9·32)
N 6.(P)
r
- 6.(P- J)
...(9·37)
P 6.") .
Similarly, 'r., y Pp = N r., 6.(P-PIJ) . ~jJ
j=JJ
~ ~I IlP
.~I
. ~2 1lP+ I
- -N-
6.(P-J)
IlP-I IlP ~1p-J
~J ~11 IlPI
N .6.(P)
= 6.(P- J)
...(9·38)
where 6.(P) and 6.(P)j arci<lefined in (9·17). Substituting in (9,36) we get
curve Filtin~ and Principle of Least Squares 923
...(9·39)
1f Lhe variable x takcs Lhe inLcgral val ues 1, 2, ... , N. Lhen Lhe firsL seven
of these orLhogonal polynomials Pj'S j = 0, 1,2,3, ... ,6 are given by :
Po(x) = 1, Pt(x) = At . ~
P2(X):;=A2{~2_N:;I }
'\
P3(X) = 11.3
{~3
., -
3N
20
2- 7 .,~}
P4(X) = A)1~4 _ 3N
2
14
- 13 ~2 +...1..
560
(N 2 _ I)(N 2 - 9)}
~yPo ~v
bo=--z =/t ; (.: Po= 1)
~Po
~yP. ... (9·40)
x .
bi=--2 ,(i= 1.2, ...• P,
~Pi
x
The values of P;'s and A.;·s are obtained Jrom. 'StatisLical Tables' by
924 Fundamentals of Math(.'fI1atical Stati~tics
R.A. Fisher for the values of N from 3 to 75. In these tabl~s the orthogonal
polynomials P/s are denoted by <I>/s. We reproduce below these tables for
N= 3toN=6.
x 0 "I 2 3 4
- 0
I
2
-2
-1
0
1
1·8
3·3
-2
-1
0
-2
-J.8
0
3 1 4·5 1 4·5
4 2 6·3 2 12·6
Total 16·9 -0 13,3
Thc values of c)ll are noted 'from .the tablcs for N = 5. From tables we also
find
Lc)l12= 10, AI = 1
. LV 16·9 Lyc)ll 13·3
Now usmg (9·40), bo == =..L == - = 3·38 ; /)1 = - - 2 = - = 1·33
N 5 Lc)ll 10
c)l1(X) = 1..\ ~ = 1 . (x - 2):;:: x - 2
Substituting in (**), the required straight line is
y = 3·38 + 1·33 (x- 2)
=> Y = 1· 33x + O· 72
Example 9·9. Fit a second' degree parapola to the following data, ,using
lhe method of orthogonal polynomials.
Ey¢22 ::-=4·43;
17t ~
b2= ~I(X)::~' -::2[4x-7]=8x-14
E~2 1M