Developed By: Dr. Md.
Ashraful Babu Chapter | Optimization
Optimization
1. Introduction
Optimization is the process of finding the best solution or outcome from a set of possible
choices, given certain constraints and objectives. It involves maximizing or minimizing an
objective function by systematically choosing input values from within an allowed set and
computing the value of the function. Optimization is widely used in various fields, including
mathematics, engineering, economics, and computer science, to improve performance, reduce
costs, or achieve the most efficient and effective results.
Key components of optimization include:
i. Objective Function: The function that needs to be optimized, either maximized or
minimized.
ii. Decision Variables: The variables that can be controlled or adjusted to achieve the
best outcome.
iii. Constraints: The limitations or requirements that the solution must satisfy.
iv. Feasible Region: The set of all possible solutions that satisfy the constraints.
Optimization can be classified into different types based on the nature of the objective
function, constraints, and variables, such as linear vs. non-linear, single-variable vs. multi-
variable, constrained vs. unconstrained, and deterministic vs. stochastic optimization.
2. Types of Optimizations
2.1. Convex versus Non-convex Optimization
Convex Optimization: Convex optimization refers to a class of optimization problems where
the objective function is convex, and the feasible region defined by the constraints is a
convex set. In a convex optimization problem, any local minimum is also a global minimum.
These problems are easier to solve because convex functions have well-defined properties
that can be exploited using efficient algorithms.
Non-convex Optimization: Non-convex optimization involves optimization problems where
the objective function or the feasible region is non-convex. These problems can have multiple
local minima, making it challenging to find the global minimum. Non-convex optimization
problems are generally harder to solve due to their complexity and the potential for getting
trapped in local minima.
2.2. Deterministic Optimization versus Stochastic Optimization
Page 1 of 4
Developed By: Dr. Md. Ashraful Babu Chapter | Optimization
Deterministic Optimization: Deterministic optimization refers to optimization methods that
do not involve randomness in the process. These methods follow a predefined algorithm or
set of rules to find the optimal solution. The outcome is predictable and repeatable, given the
same initial conditions and parameters.
Stochastic Optimization: Stochastic optimization involves optimization methods that
incorporate randomness or probabilistic elements. These methods are often used to handle
uncertainty, noise, or incomplete information. Examples include genetic algorithms,
simulated annealing, and particle swarm optimization. The solutions obtained can vary
between runs due to their inherent randomness.
2.3. Continuous Optimization versus Discrete Optimization
Continuous Optimization: Continuous optimization deals with problems where the decision
variables can take any value within a continuous range. The objective function and
constraints are typically defined over continuous domains. Examples include finding the
minimum of a function defined on a real-valued domain.
Discrete Optimization: Discrete optimization involves problems where the decision
variables can only take on discrete values, often integers. These problems often arise in
combinatorial contexts, such as scheduling, routing, and allocation problems. Examples
include integer programming and combinatorial optimization problems.
2.4. Unconstrained Optimization versus Constrained Optimization
Unconstrained Optimization: Unconstrained optimization refers to optimization problems
that do not have any constraints on the decision variables. The goal is to find the optimal
solution for the objective function without any restrictions on the values that the variables can
take.
Constrained Optimization: Constrained optimization involves optimization problems where
the decision variables are subject to one or more constraints. These constraints can be
equalities or inequalities that the solution must satisfy. The presence of constraints often
makes the problem more complex, requiring specialized algorithms to find feasible and
optimal solutions.
3. Classification of Optimization Problem
Page 2 of 4
Developed By: Dr. Md. Ashraful Babu Chapter | Optimization
Figure-1: Types of Optimizations
Page 3 of 4
Developed By: Dr. Md. Ashraful Babu Chapter | Optimization
Figure-2: Types of Optimizations
Figure-3: Types of Optimizations
Page 4 of 4