Shape optimization is part of the field of optimal control theory.
The typical problem is to find
the shape which is optimal in that it minimizes a certain cost functional while satisfying
given constraints. In many cases, the functional being solved depends on the solution of a given
partial differential equation defined on the variable domain.
Topology optimization is, in addition, concerned with the number of connected
components/boundaries belonging to the domain. Such methods are needed since typically shape
optimization methods work in a subset of allowable shapes which have fixed topological properties,
such as having a fixed number of holes in them. Topological optimization techniques can then help
work around the limitations of pure shape optimization.
Source: Wiki