Applied Optimization

Lecture 2: History and Background

Vaibhav Kumar Upadhyay

Assistant Professor, Data Science and Engineering
IISERB

09-01-2026
3
 Suggest in context of
Optimization Data, constraints, objectives,
preferences, uncertainties
Input Formulation Output
Can be vague at this moment Optimal
Decision variables, objective decision,
function, constraints trade-offs,
sensitivity,
insights

• Optimization also has a representation.

• Solving a problem requires identifying inputs, formulation, and
output.
• A common thinking through out history by the researchers.

Finding the conditions that

give the maximum or
minimum value of a function
Brief History and Set Up
Bigger Push during World War II

British military faced the problem of allocating very scarce and limited
resources

Methods developed by the team were instrumental in the winning of

the Air Battle by Britain

2
• Ancient civilizations practiced optimization (constructions, warfare's, day to day life).
1800s
• Existing mathematical form -> since the days of Newton, Lagrange, and Cauchy.
𝑑𝑓
• Newton, Calculus: The optimal point satisfies =0
𝑑𝑥

Joseph-Louis Lagrange answered “What if optimization has constraints? (involves the

addition of unknown multipliers)“

Augustin-Louis Cauchy -> made optimization computable and provable

(Gradient descent: 1847)
• The foundations of calculus of variations, which deals with the minimization of functionals,
were laid by Bernoulli, Euler, Lagrange, and Weirstrass.
1900s (till 60s)
• The foundations of game theory were laid by von Neumann in 1928 and since then the
technique has been applied to solve several mathematical economics and military
problems.
• The development of the simplex method by Dantzig in 1947 for linear programming
• Principle of optimality in 1957 by Bellman for dynamic programming problems
• Kuhn and Tucker in 1951 inspired non linear programming fuled by Zoutendijk and Rosen
1960s
1960s
• McCormick proposed solving problems through Unconstrained optimization.
• Geometric programming was developed in the 1960s by Duffin, Zener, and
Peterson.
• Gomory pioneered integer programming, one of the most rapidly developing
areas of optimization
• Dantzig and Charnes and Cooper developed stochastic programming
techniques.
• Development of Mult objective programming methods.
• The goal programming was originally proposed for linear problems by
Charnes and Cooper in 1961
Modern Methods of Optimization
• Genetic algorithms were originally proposed by John Holland in 1975
• Particle swarm optimization algorithm was proposed by Kennedy and
Eberhart in 1995
• Ant colony optimization: Marco Dorigo in 1992
• Hopfield and Tank used neural network for optimization in 1985
• 1986: Rao-> fuzzy approaches for single and multiobjective
 Statement Of An Optimization Problem
X is an n-dimensional vector called the design
vector, f(X) is termed the objective function

gj (X) and lj (X) are known as inequality and equality

constraints

The number of variables n and the number of

constraints m and/or p need not be related in any
e.g., of constrained optimization problem way.

Design Vector
• Vector comprises of variables during the design process
• Variables in the design process and are called design or decision
variables
• Design variables cannot be chosen arbitrarily; rather, they have to satisfy
certain specified functional and other requirements

2
Statement Of An Optimization Problem
Objective function

• We need to know what we want to improve to the extreme.

• Extreme can be a maximum or minimum, depending on the identified objective.
Cost, we minimize; and if it is profit, we maximize.
In structures, if it weight, we minimize; and if it is strength we maximize, most often.

• We call it an objective function because it must depend on some variables in order to optimize.
• The objective function should be a function of optimization variables.

Objective variables

These are the variables to which we try to assign suitable values to optimize the objective function.

Formulae or methods to quantify and compute objective function and constraints

• We should have either mathematical expressions (formulae) or numerical methods to compute the
values of the objective function and constraints once we know the values of the optimization variables.

There will be subsidiary/state variables too. Normal (design) variables

are chosen by us.
• There will also be constraints that govern those subsidiary variables. State (subsidiary) variables
• Subsidiary variables are called state variables in the context of structural are derived once we choose
optimization. the design. 2
Optimum solution of cf (x) or c + f (x)
same as that of f (x). Minimum of f (x) is same as
maximum of −f (x).

2
Types of optimization problems

There are many, many types of optimization problems.

The types arise because of…

• How many objective functions you have.

• Types of objective function and constraints.
• Types of variables.
• Nature of optimization we want to do.

2
Classification based on the objective function
One or more objective functions.
Working with a single objective is easy.
• Even in life!
• Most of the theory of optimization is focused on dealing with a single objective
function.

Multi-objective optimization is hard.

• Weights can be given but then how do you give the weights when you do not know
which objective is more important for you?
• The best thing to do is to move the less important objective to a constraint.

Pareto optimum
✓ Pareto optimum concept is an important concept in multi-objective optimization.
✓ Pareto optimum is one where you cannot improve one without hurting another.
✓ Often, Pareto optimum will be a set; that is there will be many Pareto optima.
➢ Pareto optimum set can be continuous or discontinuous.
➢ Generating the entire Pareto set is difficult in practice.

2
Classification based on the nature of optimization
Global or local
Local optimum is one in a small vicinity of the optimum point
Global optimum is one that considers the entire domain
of the objective function.

Deterministic or stochastic
• Deterministic means that you have the same thing any number of times
you try it.
• Stochastic means that there is also an element of randomness in addition
to deterministic nature.
• An optimization problem can be stochastic if the variables involved are
stochastic.
• But one can use non-deterministic methods to solve a deterministic
problem.
Genetic algorithms, Simulated annealing methods, Monte Carlo search,
etc. 2
Classification based on constraints
Without or with constraints
If there are no constraints, it becomes an unconstrained optimization problem.
With constraints, it is constrained optimization problem.

Equality or inequality
• Constraints can be equalities.
• They can be inequalities

Constraints reduce the permissible values of the optimization variables

• Constraints constrain the space of optimization variables.

Feasible space
• The space of optimization variables where all constraints are satisfied is called
the feasible space.
• In constrained optimization, we need to search for the optimum of the
objective function only in the feasible space.
• Constructing feasible space is often impractical but we can certainly search
within it. 2
Constraint Surface

Constraint surfaces in a
hypothetical two-dimensional Contours of the objective function
design space

2
Classification based on objective and constraints
Linear programing
Both objective function and constraints are linear.
Quadratic programming
Objective function is quadratic and constraints are linear.
Nonlinear programming
Both objective and constraints are nonlinear.
Geometric programming
◦Objective and constraints are posynomials.
◦Posynomials are polynomials with positive coefficients.
◦Positive + Polynomial = posynomial!
◦Exponents in posynomials can be real numbers, positive or negative, in the
context of geometric programming.
Convex optimization
◦The objective function is convex and so is the feasible space.
Non-smooth optimization
◦Where either objective function or the constraints, or both, are not
differentiable.
2
 Type Typical application
Linear programming Logistics, planning
Quadratic programming Control, finance
Nonlinear programming Robotics, aerodynamics
Geometric programming VLSI, wireless
Convex optimization ML, signal processing
Non-smooth optimization Sparsity, robustness

17
Classification based on types of variables

Continuous or discrete
Discrete
• Binary
• Only 0 or 1 are allowed.
• Integer
• Discrete sets
• Bearing sizes, screw threads, etc.; you cannot have whatever you want. They will be
certain pre-specified values.
Deterministic or stochastic
• Discussed in the previous slide
Finite variables or functions themselves!
• Variables are finite if they are like x1, x2, x3, … , xn.
• What if variables are not finite?
• What if variables are functions themselves?
• This is what brings us to calculus of variations
2