Motion Control

lecture 3: Optimization

Dr. Omar Ahmed

Department of Mechatronics
Optimization
History of Optimization
Techniques
History of finding optimal designs

Design Design with Design with

experimentally simulated models optimization algorithms
Generative AI in Design

AI that generates new

What is Generative AI? content or models based
on learned data patterns.

Minimizes human bias and

Capable of producing highly
explores vast design
optimized and innovative
possibilities beyond
designs.
conventional constraints.
Generative AI: Revolutionizing Model Creation

Beyond Components: AI can design complete systems, not just improve parts, by
understanding and integrating complex requirements and constraints.

Cross-Disciplinary Innovation: AI's capacity to analyze data from various fields can
lead to novel designs that a human or traditional computer-aided design might not
conceive.

Adaptive and Evolving Designs: AI models can evolve over time, continuously
improving through feedback loops, unlike static traditional models.
Design experimentally

A car with efficient aerodynamics Wind tunnel to analyse aerodynamic forces (drag)

Bob (Engineer)

Bob wants to design a car with minimum drag He needs to build prototypes and a wind tunnel to calculate drag
Design experimentally 𝐶𝑔 = 0.85
Coefficient of drag (Cg)

𝐶𝑔 = 0.90

𝐶𝑔 = 0.50

𝐶𝑔 = 0.45

𝐶𝑔 = 0.40

𝐶𝑔 = 0.32
Design experimentally Design with simulated Design with optimization
models algorithms

• Tedious
• Expensive
• Time consuming
• Human involvement
• Not accurate
Design with simulated
models
Design with simulated
models 𝐶𝑔 = 0.85
𝐶𝑔 = 0.90
𝐶𝑔 = 0.50
𝐶𝑔 = 0.45
𝐶𝑔 = 0.40
𝐶𝑔 = 0.32
Design experimentally Design with simulated Design with
models optimization
algorithms

• Tedious • Fast and cheap

• Expensive • Slow design
• Time consuming • Medium human
• Human involvement involvement
• Not accurate • Error prone
Design experimentally Design with simulated Design with optimization
models algorithms

• Tedious • Fast and cheap

• Expensive • Slow design
• Time consuming • Medium human
• Human involvement involvement
• Not accurate • Error prone
Design with optimization algorithms

𝐶𝑔 = 0.85
𝐶𝑔 = 0.90
𝐶𝑔 = 0.50
𝐶𝑔 = 0.45
𝐶𝑔 = 0.40
𝐶𝑔 = 0.32
Design experimentally Design with simulated Design with optimization
models algorithms

• Tedious • Fast and cheap • Fast modeling

• Expensive • Slow design • Fast design process
• Time consuming • Medium human • Automated (minimum
• Human involvement involvement human involvement)
• Not accurate • Error prone • Low error
• Complex optimization
algorithm
• Difficulties of solving
real world problems
OPTIMIZER

 Optimizer—Optimization
algorithms are used to find
the optimum weight values
that minimize the error.
Global VS Local
Minima
Single-objective
Optimization
Problems
Main components of an optimization problem
Inputs (variables) Output (objective)

System
MAIN
COMPONENTS
OF AN
OPTIMIZATION
PROBLEM
Main Components Of an Optimization Problem
Inputs Constraints Output

Sys… …tem
Formulating an optimization problem
Inputs Constraints Output

Sys… …tem

𝑀𝑖𝑛𝑖𝑚𝑖𝑠𝑒: 𝒇(𝒙𝟏, 𝒙𝟐, … , 𝒙𝒏)

𝑆𝑢𝑗𝑒𝑐𝑡 𝑡𝑜: 𝑪𝒐𝒏𝒔𝒕𝒓𝒂𝒊𝒏𝒕𝒔
Optimization algorithm
Inputs Constraints Output

Sys… …tem

Optimizer
Example: designing a table

Length

Width
 Example: designing a table
Length
Inputs Constraints Output
2<length<7 2<width<7 Width

Length in [1,10]
Weight

Width in [1,10]
Sys… …tem

Weight = 1*(Length + Width)

The objective is to minimize the weight

Example: designing a table
Length

Width

𝑀𝑖𝑛𝑖𝑚𝑖𝑠𝑒: 𝒇 𝒍𝒆𝒏𝒈𝒕𝒉, 𝒘𝒊𝒅𝒕𝒉 =1*(Length + Width)

𝑆𝑢𝑗𝑒𝑐𝑡 𝑡𝑜: 𝟐 < 𝒍𝒆𝒏𝒈𝒕𝒉 < 𝟕

𝟐 < 𝒘𝒊𝒅𝒕𝒉 < 𝟕
Search landscape of the table problem
Table #1: W=10, L=10
Inputs: Table #2: W=9 , L=10
width , length 20 Table #3: W=10, L=9
Output: 15
Table #4: W=9 , L=9

Weight
10
weight
5

0
10
10
8
5 6
4
2
Length 0 0 Width
Search landscape of the table problem
Table #1: W=10, L=10
Inputs: Table #2: W=9 , L=10
width , length 20 Table #3: W=10, L=9
Output: 15
Table #4: W=9 , L=9

Weight
10
weight
5

0
10
10
8
5 6
4
2
Length 0 0 Width
Example
8 tables
Inputs:
width , length 20

Output: 15

Weight
10
weight
5

0
10
10
8
5 6
4
2
Length 0 0 Width
Example
50 tables
Inputs:
width , length 20

Output: 15

Weight
10
weight
5

0
10
10
8
5 6
4
2
Length 0 0 Width
Example
100 tables
Inputs:
width , length 20

Output: 15

Weight
10
weight
5

0
10
10
8
5 6
4
2
Length 0 0 Width
Example
Inputs:
width , length 20

Output: 15

Weight
10
weight
5

0
10
10
8
5 6
4
2
Length 0 0 Width
Example
Inputs:
width , length 20

Output: 15

Weight
10
weight
5

0
Constraints: 10
10
8
5 6
2< width <7 2
4
Length 0 0 Width
2< length <7
Search landscape
Inputs:
x,y
Output:
f(x,y)

Constraints:
𝑦 ≤ 3.2 ∨ 𝑦 ≥ 3.4
𝑥 ≤ 2.2 ∨ 𝑥 ≥ 2.3
𝑥 − 3 2 + 𝑦 − 1 2 ≥ 0.1
𝑥 + 3 2 + 𝑦 − 1 2 ≥ 0.3
𝑥2 + 𝑦2 ≥ 1
𝑥≠ 𝑦
Search landscape
Inputs:
x,y
Output:
f(x,y)

Constraints:
𝑦 ≤ 3.2 ∨ 𝑦 ≥ 3.4
𝑥 ≤ 2.2 ∨ 𝑥 ≥ 2.3
𝑥 − 3 2 + 𝑦 − 1 2 ≥ 0.1
𝑥 + 3 2 + 𝑦 − 1 2 ≥ 0.3
𝑥2 + 𝑦2 ≥ 1
𝑥≠ 𝑦
Difficulties of a real-world problem

• A large number of local solutions

• A large number of constraints
• Discrete variables
• Deceptive search space
• Multiple objectives
• Dynamically changing
• Uncertainties in inputs, outputs, or constraints
• Etc.
Stochastic Optimization
Algorithms
Classification of stochastic optimization algorithms

Individual-based algorithms Population-based algorithms

Individual-based vs. Population-based

Jennifer Bob, Anthony, and Jennifer

 Individual-based vs. Population-based

Minimum function High function

evaluation evaluation

Local optima Low stagnation in

stagnation local solutions
Population-based algorithms

High exploration

High local optima avoidance

Real-world problems
Exploration vs. Exploitation
Exploration vs. Exploitation in our analogy
Exploration vs. Exploitation in population
based algorithms
Classification of population-based algorithms
Population-based
algorithms

Evolutionary algorithms Physics-based algorithms Swarm-based algorithms

GA ES GSA CSS PSO ACO

GP DE CFO BBBC ABS GWO

Discrete Optimization
Algorithms
Formulation

𝑀𝑖𝑛𝑖𝑚𝑖𝑧𝑒: 𝑓 𝑥1, 𝑥2, … , 𝑥𝑛

Where

𝑥1, 𝑥2, … , 𝑥𝑛 ∈ 0,1

Formulation

𝑀𝑖𝑛𝑖𝑚𝑖𝑧𝑒: 𝑓 𝑥1, 𝑥2, … , 𝑥𝑛

where

𝑥1, 𝑥2, … , 𝑥𝑛 ∈ 0,1,2,3,4,5,6

Search landscape of discrete problems

𝑴𝒊𝒏𝒊𝒎𝒊𝒛𝒆: 𝒇 𝒙𝟏, 𝒙𝟐 = 𝒙𝟐 + 𝒙𝟐 𝑴𝒊𝒏𝒊𝒎𝒊𝒛𝒆: 𝒇 𝒙𝟏, 𝒙𝟐 = 𝒙𝟐 + 𝒙𝟐

𝟏 𝟐 𝟏 𝟐

where where

−𝟏𝟎 ≤ 𝒙𝟏, 𝒙𝟐 ≤ 𝟏𝟎 𝒙𝟏, 𝒙𝟐 ∈ −𝟏𝟎, −𝟗, −𝟖, … , 𝟖, 𝟗, 𝟏𝟎

Continuous Discrete
Search landscape of discrete problems

Continuous Discrete
Optimization in control
Example of PID Design using PSO (Conceptual
Overview)
Define Objective: Minimize the Integral of Squared Error (ISE) of a temperature control system.

Choose Algorithm: Particle Swarm Optimization.

Formulate Problem: Objective function based on ISE, with constraints on Kp, Ki, Kd values.

Implement PSO: Initialize a swarm of particles, each representing a set of PID parameters. Iterate by
evaluating the ISE for each particle, updating particles based on personal and global best positions.

Validate and Fine-tune: Test the best-found PID parameters in simulation and real-world scenarios,
adjusting as necessary.
MAIN
COMPONENTS
OF AN
OPTIMIZATION
PROBLEM
Optimization algorithm
Inputs Constraints Output

Sys… …tem

Optimizer
LINEAR
REGRESSION
Linear Regression
• Underlying Principle: Linear regression attempts to model the relationship between a
dependent variable (y) and one or more independent variables (x) by fitting a straight
line (or hyperplane in a multi-dimensional case) through your data points.
• The Equation: The equation for a simple linear regression model is represented as:
• y = a + b*x
o Where:
▪ y is the dependent variable we're trying to predict
▪ x is the independent variable
▪ a is the y-intercept (the value of y when x is 0)
▪ b is the slope of the line (how much y changes for a one-unit change in x)
• Finding the Best Fit: The goal of linear regression is to find the values of a and b that
minimize the error (residuals) between the predicted values and the actual data points.
VISUALIZATION
LR FLOWCHART
PROCESS
GRADIENT
DECENT
Training Initial Model: Start with an initial guess for the parameters b0 and b1. This
could be random, or based on some basic assumptions.

Processe Prediction: Use the current linear model to predict the dependent variable
(y) for each data point in your dataset.

Cost Function: Calculate the error between the predicted values and actual
values using a cost function like the Mean Squared Error (MSE).

Gradient Descent: An optimization algorithm (often gradient descent)

calculates the gradient of the cost function with respect to the model
parameters (b0 and b1). The gradient indicates how much the cost
function changes for slight changes in the parameters.

Parameter Update: The values of b0 and b1 are updated in the direction

opposite of the gradient, with the goal of reducing the cost function. The
amount of update is determined by a parameter called the learning rate.

Iteration: Steps 2-5 are repeated until the model converges (the cost
function stops decreasing significantly) or a maximum number of iterations
is reached.
THE PROGRESS
OF TRAINING
 Types
of
Gradient Descent
PROCESS
Learning Rate

 Learning rate—One of the optimizer’s input parameters that we tune.

THE EFFECT OF
LEARNING RATE