Well-posed and ill-posed
problems
• A well-posed problem has the following properties:
a) Existence: a solution exists
b) Uniqueness: the solution is unique
c) Stability: the solution’s behaviour changes continuously
with data.
• Problems which are not well-posed are called ill-posed
problems.
• Solution of ill-posed problems can be obtained using
regularization. Tikhonov regularization is a commonly used
method of regularization.
Tikhonov Regularization
• Suppose that for a known matrix A and vector b, we wish to find a
vector x such that
Ax = b
If the problem is ill-posed, ordinary least squares cannot be used.
min
• To give preference to a particular solution with desirable properties, a
regularization term can be included in this minimization:
min
where is called the Tikhonov matrix.
This method is called Tikhonov regularization.
• If the Tikhonov matrix is chosen as a multiple of the identity matrix
, it gives preference to solutions with smaller norms. This is
known as L2 regularization.
Regularization favours simpler solutions
Maximum Likelihood Solution
• Prior probability of model parameters:
P(m)
• Likelihood of data given model parameters:
P(d|m)
• Posterior probability of model parameters
given data:
P(m|d)
Maximum Likelihood Solution
• The Maximum Likelihood Solution is obtained
by finding the model parameters that
maximizes the likelihood of the observed
data.
mml = argmax m P(dobs|m)
Maximum Likelihood Solution
Example problem 1:
Maximum Likelihood Solution
Example problem 2:
• Obtain the maximum likelihood solution for :
The likelihood is given by:
Maximum Likelihood Solution
Example problem 3:
• Obtain the maximum likelihood solution for :