Numerical Analysis and optimization

ENGG3700

Week 12A
Curve Fitting

CONSTITUTE NUMERICAL ALGORITHMS FOR CURVE FITTING

CLO3 PLO1
FOR LINEAR AND NON-LINEAR FUNCTIONS.

Dr. Abdoulhdi Omran

Assistant Professor
Mechanical and Mechatronic Engineering
Faculty of Engineering
 Curve Fitting

 Curve Fitting- Linear Regression.

 Multiple Linear Regression.
 Non-Linear Regression.
 Curve Fitting
General Approaches of Curve Fitting (how much error in the data)

 Data are often given for very precise discrete values along a
continuum, The basic approach is to fit a curve (or curves)
that pass directly through each of the points. The estimation
of values between well-known discrete points is called
Interpolation.

 Data exhibit a significant degree of error (noise): derive a

single curve that represents the general trend of these data.
One approach is called Least-Squares Regression.
Three attempts to fit a best curve through five data points

(a) Least-Square Regression

(b) Linear Interpolation

(c) Curvilinear Interpolation

Curve Fitting and Engineering Practice

 Determine intermediate values from tabulated data.

(Tabulated Engineering properties).

Find the thermal properties of saturated water at 33 C?

 Develop a new predictive relationships for experimental Data
(Experimental Curve Fitting)

• Trend Analysis: represents the process of using the pattern

of the experimental data to make prediction.

- High precision data: utilize Interpolating polynomials.

- Imprecise data: analyze with Least Square Regression.
- Forecast values of the dependent variable: Extrapolation

• Hypothesis Testing: an existing mathematical model is

compared with measured data (i.e. determine the unknown
coefficients of the modal to fit the collected data)- Empirical
Model.

 Derive simple function to approximate complicated functions.

 Chapter Objectives

Knowing how to compute the slope and intercept of a

best fit straight line with linear regression.
Knowing how to compute and understand the meaning
of the coefficient of determination and the standard
error of the estimate.
Understanding how to use transformations to linearize
nonlinear equations so that they can be fit with linear
regression.
Statistics Review
Measure of Location
• Arithmetic mean: the sum of the individual data points (yi) divided by
the number of points n:

y
 y i

n
• Median: the midpoint of a group of data.
• Mode: the value that occurs most frequently in a group of data.
Standard deviation:
The standard deviation is a measure of the amount of variation of the
values of a variable about its mean
St
sy 
n 1
where St is the sum of the squares of the data residuals:
St   yi  y 
2
Statistics Review
Measure of Location
Standard deviation:
The standard deviation is a measure of the amount of variation of the
values of a variable about its mean
St
sy 
n 1
• where St is the sum of the squares of the data residuals:
St   yi  y 
2

Meaning of Standard deviation:

• Small Standard Deviation

Data points are close to the mean, indicating low variability.

• Large Standard Deviation

Data points are spread out from the mean, indicating high variability.
 Least-Squares Regression
(Linear Regression)
Linear least-squares regression is a method to determine the
“Best” coefficients in a linear model for given data set.
y  a0  a1 x
“Best” for least-squares regression means minimizing the
sum of the squares of the estimate residuals. For a straight
line model, this gives:
n n
S r   ei2    yi  a0  a1 xi 
2

i 1 i 1

This method will yield a unique line for a given set of data.
To determine values for a0 and a1, Sr is differentiated with
respect to each unknown coefficient:

Setting these derivatives equal to zero will result in a minimum

Sr . If this is done, the equations can be expressed as

OR

Solve the two equation simultaneously:

and
Least-Squares Fit of a Straight Line

• Using the model:

y  a0  a1 x
the slope and intercept producing the best fit can be found using:

n xi yi   xi  yi
a1 
n x   xi 
2 2
i

a0  y  a1 x
Case Study: Wind tunnel experiment to measure how the force
of air resistance depends on velocity.

Fit a straight line to the values in Table

Example-1: Experimental data for force (N) and velocity (m/s) from wind tunnel
experiment. Fit a straight line to the values in Table:

V F
(m/s) (N)
i xi yi (xi)2 x iy i

1 10 25
2 20 70
3 30 380

4 40 550
5 50 610
6 60 1220
7 70 830
8 80 1450


Example-1: Experimental data for force (N) and velocity (m/s) from wind tunnel
experiment. Fit a straight line to the values in Table:

n xi yi   xi  yi 8312850  3605135
V F a1   19.47024
n x   x  820400  360
2 2
2
(m/s) (N) i i

i xi yi (xi)2 x iy i a0  y  a1x  641.875 19.4702445  234.2857

1 10 25 100 250
Fest  234.2857 19.47024v
2 20 70 400 1400
3 30 380 900 11400

4 40 550 1600 22000

5 50 610 2500 30500
6 60 1220 3600 73200
7 70 830 4900 58100
8 80 1450 6400 116000

 360 5135 20400 312850

 Quantification of Error
• Recall for a straight line, the sum of the squares of the estimate residuals:
n n
Sr   ei2   yi  a0  a1 xi 
2

i1 i1

 SD for the regression line:

Sr
s y/ x 
n2
Called the
Standard error of the estimate

y/x : designate that the error is for a predicted value of y corresponding to a

particular value of x
St SD for the data
sy 
n 1
 Standard Error of the Estimate
• Regression data showing (a) the spread of data around the mean of
the dependent data (Standard deviation) and (b) the spread of the
data around the best fit line (Standard error of the estimate):

St Sr
sy  s y/ x 
n 1 n2

• The reduction in spread represents the improvement due to linear

regression.
Coefficient of Determination
• The coefficient of determination r2 is the difference between the
sum of the squares of the data residuals and the sum of the
squares of the estimate residuals, normalized by the sum of the
squares of the data residuals:
St  Sr
2
r 
St
• r2 represents the percentage of the original uncertainty
explained by the model.
• For a perfect fit, Sr=0 and r2=1.
• If r2=0, there is no improvement over simply picking the mean.
• If r2<0, the model is worse than simply picking the mean!
Example-2: Estimate of errors for the linear least-squares fit

V F
(m/s) (N)
i xi yi a0+a1xi (yi- ȳ)2 (yi-a0-a1xi)2
St  Sr
1 10 25 r 
2

2 20 70 St
3 30 380
Fest  234.2857  19.47024v
4 40 550
St    yi  y   1808297
2
5 50 610
Sr    yi  a0  a1 xi   216118
2
6 60 1220

7 70 830

8 80 1450


Example-2: Estimate of errors for the linear least-squares fit

V F St  Sr
(m/s) (N) r 
2

i xi yi a0+a1xi (yi- ȳ)2 (yi-a0-a1xi)2

St
Fest  234.2857  19.47024v
1 10 25 -39.58 380535 4171
St    yi  y   1808297
2
2 20 70 155.12 327041 7245
Sr    yi  a0  a1 xi   216118
2
3 30 380 349.82 68579 911

4 40 550 544.52 8441 30 82

5 50 610 739.23 1016 16699 1808297  216118
r2   0.8805
6 60 1220 933.93 334229 81837
1808297

7 70 830 1128.63 35391 89180

88.05% of the original uncertainty
8 80 1450 1323.33 653066 16044
has been explained by the
 360 5135 1808297 216118 linear model
No.
1 1 0.5
2 2 2.5
3 3 2
4 4 4
5 5 3.5
6 6 6
7 7 5.5
No.
1 1 0.5 0.5 1 0.25
2 2 2.5 5 4 6.25
3 3 2 6 9 4
4 4 4 16 16 16
5 5 3.5 17.5 25 12.25
6 6 6 36 36 36
7 7 5.5 38.5 49 30.25
28 24 119.5 140 105
LINEARIZATION OF NONLINEAR RELATIONSHIPS
• Linear regression provides a powerful technique for fitting a best
line to data.
• However, it is predicted on the fact that the relationship between
the dependent and independent variables is linear - this is not
always the case.
• Three common examples are:

Exponential Model : y  1 e 1x

2
Power Model : y  2 x
x
Saturation - growth - rate : y   3
3  x
Linearization of Nonlinear Relationships
• One option for finding the coefficients for a nonlinear fit is to
linearize it. For the three common models, this may involve taking
logarithms or inversion:

Model Nonlinear Linearized

Exponential : y  1e 1x ln y  ln 1  1 x

Power : y   2 x 2 log y  log  2   2 log x

x 1 1 3 1
Saturation - growth - rate : y   3  
3  x y 3 3 x
Transformation Examples
EXAMPLE: Linearization of a Power Equation
Fit the power equation to the data in Table below using a logarithmic
transformation.
2
F  2 v
Linearize the power function

y  a0  a1x

y   2 x 2 log y  log  2   2 log x

Determine the function coefficients

EXAMPLE: Linearization of a Power Equation
Fit the power equation to the data in Table below using a logarithmic
2
transformation.
F  2 v
Solution: The means can be computed as:

log( F )  log( 2 )   2 log(v) y  a0  a1x

EXAMPLE: Linearization of a Power Equation
Fit the power equation to the data in Table below using a logarithmic
2
transformation.
F  2 v
Solution: The means can be computed as:
Least-squares fit of a power model to the data from Table 1.
(a) The fit of the transformed data.
(b) The power equation fit along with the data.
 Curve Fitting

Polynomial Regression
Multiple Linear Regression
Chapter Objectives

• Knowing how to implement polynomial regression.

• Knowing how to implement multiple linear regression.

• Understanding the formulation of the general linear

least-squares model.
 Polynomial Regression
The least-squares procedure
can be readily extended to fit
data to a higher-order
polynomial. Again, the idea is
to minimize the sum of the
squares of the estimate
residuals.
The figure shows the same
data fit with:
 A first order polynomial
 A second order polynomial
Process and Measures of Fit
For a second order polynomial, the best fit would mean
minimizing:
n n
Sr   e   yi  a0  a1 xi  a x
2
i
2 2
2 i 
i1 i1

In general, this would mean minimizing:

n n
Sr   e   yi  a0  a1 xi  a x  a x
2
i
2
2 i m i 
m 2

 i1 i1

The coefficient of determination r2 is still found using:

St  Sr
r 
2

St
 The Regression SD is
n= no. of data point
m= polynomial order
m+1= no. of coefficients (a’s)
To generate the least-squares fit, we take the derivative with
respect to each of the unknown coefficients of the polynomial,
as in

These equations can be set equal to zero and rearranged to develop the following
set of normal equations:
EXAMPLE : Polynomial Regression
Fit a second-order polynomial to the data in the first two columns
of Table below.
The results indicate that 99.851% of the original data has been
represented by the model. This result supports the conclusion
that the quadratic equation represents an excellent fit.
Multiple Linear Regression
• Another useful extension of linear
regression is the case where y is a
linear function of two or more
independent variables:

y  a0  a1 x1  a2 x2 am xm

• Again, the best fit is obtained by

minimizing the sum of the squares
of the estimate residuals:
n n
Sr   e   yi  a0  a1 x1,i  a2 x2,i am xm,i 
2 2
i
 i1 i1
For example, y might be a linear function of x and x , as in
1 2

the sum of the squares of the residuals

and differentiating with respect to each of the unknown

coefficients:
The coefficients yielding the minimum sum of the squares of
the residuals are obtained by setting the partial derivatives
equal to zero and expressing the result in matrix form as
EXAMPLE: Multiple Linear Regression
Problem Statement: For the following data, use multiple linear
regression to fit this data.
Solution:

Arrange in matrix form:

Solving the equations:

Solution:

Arrange in matrix form:

Solving the equations: