Demystifying the characterization of SDP matrices in mathematical

programming
Daniel Porumbel
June 18, 2022

Argument
This manuscript was written because I found no other introduction to SDP programming that targets the same
audience. This work is intended to be accessible to anybody who does not hate maths, who knows what a derivative
is and accepts (or has a proof of) results like det(AB) = det(A) det(B). If you know this, I think you can understand
most of this text without buying other books; my goal is not to remind/enumerate a list of results but to (try to)
make the reader examine (the proofs of) these results so as to get full insight into them.
A first difference compared to other existing introductions to SDP is that this work comes out of a mind that
was itself struggling to understand and not from a long established expert. This may seem to be only a weakness,
but, paradoxically, it is both a weakness and a strength. I really did not try to overpower the reader or to transform
the author–reader relationship into a formal professor–student relationship. My goal was to minimize the distance
between the author and the reader as much as possible. Another strength comes from the fact that many long-
acknowledged experts tend to forget the difficulties of beginners; I think this did not happened to me. Other experts
try to make all proofs as short as possible and to dismiss as unimportant certain key results they have seen thousands
of time in their career. I also avoided this, even if I did shorten a few proofs when I revised this manuscript two
years after it was first written. However, I also kept certain proofs that seem longer than necessary because I feel
they offer more insight; an important goal is to capture the “spirit” of each proven result instead of reducing it to a
flow of formulae.
The very first key step towards mastering SDP programming is to get full insight into the eigen-decomposition of
real symmetric matrices. Many other published introductions to SDP programming that merit our (research work)
consideration address this eigen-decomposition in a way that shows their target audience is different from mine. They
usually list the eigen-decomposition without proof, while I give two proofs to really familiarize the reader with this
key concept.1
If you can say “Ok, I still vaguely remember the eigen-decomposition (and other key SDP properties) from my
(under-)graduate studies some n ≥ 5 years ago; I don’t need a proof”, then you do not belong to my target audience.
I am highly skeptical that such approach can lead to anything but superficial learning. Anyhow, my brain functions
in the most opposite manner. I like to learn by checking all proofs by myself and I can’t stand taking things for
granted. The only unproven facts from this manuscript are the fundamental theorem of algebra and two results from
Section [Link]. But I do provide complete proofs, for instance, for the Cholesky decomposition of SDP matrices, the
strong duality theorem for linear conic programming (including SDP programming), six equivalent formulations of the
Lovász theta number, the copositive formulation of the maximum stable, a few convexification results for quadratic
programs and many others. I tried to prove everything by myself, so that certain proofs are original although this
introduction was not intended to be research work; of course I got help multiple times from the internet, articles and
books, the references being indicated as footnote citations.
1 In “Handbook of Semidefinite Programming Theory, Algorithms, and Applications” by H. Wolkowicz, R. Saigal and
L. Vandenberghe, the eigen-decomposition (called spectral theorem) is listed with no proof in Chapter 2 “Convex Analysis on
Symmetric Matrices”. The introduction of the “Handbook on Semidefinite, Conic and Polynomial Optimization” by M. Anjost
and J.B. Lasserre refers the reader to the (700 pages long) book “Matrix analysis” by Horn and Johnson. As a side remark, the
introductions of both these handbooks are rather short (14 or resp. 22 pages) and they mainly remind/enumerate different key
results pointing to other books for proofs. In “Semidefinite Programming for Combinatorial Optimization”, by C. Helmberg,
the eigen-decomposition is presented in an appendix and redirects the reader to the same “Matrix analysis” book. The
slides of the course “Programmation linaire et optimisation combinatoire” of Frédéric Roupin for the “Master Parisien de
Recherche Oprationnelle” ([Link]/~roupin/docs/[Link]) provide many results from
my manuscript but no proof is given. The MIT course “Introduction to Semidefinite Programming ” by R. Freund does not even
provide the SDP definition or the eigen-decomposition. The book “Convex Optimization” by S. Boyd and L. Vandenberghe
starts using SDP matrices from the beginning (e.g., to define ellipsoids in Section 2.2.2) without defining the concept of SDP
matrix, not even in appendix. The argument could extend to other non-trivial concepts that are taken as pre-requisite in above
works. For instance, the above “Convex Optimization” book introduces the square root of an SDP matrix (in five lines in
Appendix A.5.2), without showing the uniqueness – the proof takes half a page in Appendix B.4 of this manuscript.

1
 The most essential building blocks are presented in the first part. One should really master this first part before
jumping to the second one; the essentials from the first part may even be generally useful for reading other SDP
work. In fact, the goal of this manuscript is to give you all the tools needed to move to the next level and carry out
research work.

Contents
PART 1 THE ESSENTIAL BUILDING BLOCKS

1 Characterization of semidefinite positive (SDP) matrices 4

1.1 Real symmetric matrices, eigenvalues and the eigendecomposition . . . . . . . . . . . . . . . . 4
1.2 Equivalent SDP definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3 Schur complements, the self-duality of the SDP cone and related properties . . . . . . . . . . 8
1.4 Three easy ways to generate (semi-)definite positive matrices . . . . . . . . . . . . . . . . . . 10
1.5 Positive definite matrices: unique Cholesky factorization and Sylvester criterion . . . . . . . . 11
1.6 Cholesky decomposition of semidefinite positive matrices . . . . . . . . . . . . . . . . . . . . 13
1.7 Any A 0 has infinitely many factorizations A = V V > related by rotations and reflections . 18
1.8 Convex functions have an SDP Hessian assuming the Hessian is symmetric . . . . . . . . . . 19

2 Primal-Dual SDP programs and optimization considerations 20

2.1 Primal and dual SDP programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.2 Relations between the primal optimum and the dual optimum . . . . . . . . . . . . . . . . . . 27
2.3 Strong duality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.4 The difficulty of exactly solving (SDP) and algorithmic comments . . . . . . . . . . . . . . . 33

3 Interesting SDP programs 34

3.1 An SDP program does not always reach its min (inf) or max (sup) value . . . . . . . . . . . . 34
3.2 The lowest and greatest eigenvalue using the SDP duality . . . . . . . . . . . . . . . . . . . . 34
3.3 Change of variable in SDP programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3.4 Convex quadratic programming is a particular case of SDP programming . . . . . . . . . . . 35
3.5 An LP with equality constraints as an SDP program in the dual form . . . . . . . . . . . . . 38

PART 2 MORE ADVANCED SDP PROGRAMMING

4 Six equivalent formulations of the Lovász theta number ϑ(G) 39

4.1 A first SDP formulation of the theta number . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
4.2 A second SDP formulation (ϑ0G ) of the theta number . . . . . . . . . . . . . . . . . . . . . . . 42
4.3 A formulation ϑ00 (G) of the theta number without SDP matrices . . . . . . . . . . . . . . . . 43
4.4 A fourth formulation ϑ` (G) of the theta number . . . . . . . . . . . . . . . . . . . . . . . . . 44
4.5 Two formulations of the theta number using maximum eigenvalues . . . . . . . . . . . . . . . 45
4.6 The theta number ϑ(G) is bounded by the fractional chromatic number χ∗ (G) of G . . . . . 46

5 A taste of copositive optimization and sum of squares hierarchies 48

5.1 Introducing the completely positive and the copositive cones . . . . . . . . . . . . . . . . . . . 48
5.2 Reformulating a homogeneous quadratic program as a copositive problem . . . . . . . . . . . 49
5.3 Relaxations of the copositive formulation of the maximum stable . . . . . . . . . . . . . . . . 52
5.4 Further characterization of the completely positive and the copositive cones . . . . . . . . . . 58
5.5 A final short property: the Schur complement does not apply in C n∗ . . . . . . . . . . . . . . 60

6 SDP relaxations and convexifications of quadratic programs 60

6.1 The most general quadratic program: SDP relaxation and total Lagrangian . . . . . . . . . . 61
6.2 Partial and total Lagrangians for quadratic programs with linear equality constraints . . . . . 62
6.3 The case of 0 − 1 quadratic programs: partial and total Lagrangians . . . . . . . . . . . . . . 67

2
7 Basic elements of several other research topics: under construction 73
7.1 Approximation algorithms using SDP programming . . . . . . . . . . . . . . . . . . . . . . . 73
7.2 Strong duality in the more general context of linear conic programming . . . . . . . . . . . . 74
7.3 Polynomial Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
7.4 Algorithms for SDP optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

APPENDIX

A On ranks, determinants and space dimensions 79

A.1 The rank-nullity theorem and other interesting rank properties . . . . . . . . . . . . . . . . . 79
A.2 Results on determinants and space dimensions . . . . . . . . . . . . . . . . . . . . . . . . . . 81

B Three decompositions: eigenvalue, QR and square root 84

B.1 Preliminaries on eigen-values/vectors and similar matrices . . . . . . . . . . . . . . . . . . . . 84
B.2 The eigenvalue decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
B.3 The QR decomposition of real matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
B.4 An SDP matrix has a unique SDP square root factor . . . . . . . . . . . . . . . . . . . . . . . 90

C Useful related facts 91

C.1 Optimality conditions for linearly-constrained quadratic programs . . . . . . . . . . . . . . . 91
C.2 More insight and detail into the convexifications from Section 6 . . . . . . . . . . . . . . . . . 93
C.3 A convex function with an asymmetric Hessian . . . . . . . . . . . . . . . . . . . . . . . . . . 97
C.4 The separating hyperplane theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

3
 PART 1: THE ESSENTIAL BUILDING BLOCKS

1 Characterization of semidefinite positive (SDP) matrices

1.1 Real symmetric matrices, eigenvalues and the eigendecomposition
This light introduction aims at familiarizing the reader with the main concepts of real symmetric matrices,
eigenvalues and the eigenvalue decomposition. Experts on this topic can skip to Section 1.2 or even to further
sections. Absolute beginners should first consult Appendix A to get the notion of matrix rank, (sub-)space
dimension, (principal) minor, or to recall how det(A) = 0 ⇐⇒ ∃x s. t. Ax = 0. To familiarize with such
introductory concepts it is also useful to solve a few exercises, but the only exercise I propose is to ask the
reader to prove by himself all theorems whose proof does not exceed half a page.
Given (real symmetric) matrix A, we say λ is a (real) eigenvalue of A if there exists eigenvector v such
that Av = λv. Notice that by multiplying the eigenvector with a constant we Pnobtain another eigenvector.
An eigenvector is called unitary if it has a norm of 1, i.e., |v|2 = v> v = i=1 vi2 = 1. We say v is an
eigenvector if it satisfies λIn v − Av = 0, equivalent to (λIn − A)v = 0, which also means det(λIn − A) = 0.
We can also define the eigenvalues as the roots of the characteristic polynomial det(xIn − A). Indeed, if
det(λ1 In −A) = 0, there exists a (real) eigenvector v1 such that (λ1 In −A)v1 = 0, equivalent to Av1 = λ1 v1 .
The fact that v1 needs to exist is formally proven in Prop A.2.2; this Prop A.2.2 proves the statement in
the more general context of complex matrices because we should not yet take for granted that root λ1 is not
complex, i.e., λ1 In − A could have complex entries in principle. However, it is possible to show that a real
symmetric matrix has only real eigenvalues and real proper eigenvectors. The proof is not really completely
obvious, it takes a third of a page and it is given in appendix (Prop. B.1.2). By developing the characteristic
polynomial, one can also prove that the determinant is the product of the eigenvalues (Prop. A.2.4).
The characteristic polynomial has degree n, and so, it has n roots (eigenvalues), but some of them can have
multiplicities greater than 1, i.e., some eigenvalues can appear more than once. However, each eigenvalue is
associated to at least one eigenvector. An eigenvalue with multiplicity greater than 1 can have more than
one eigenvector. The following result is called the eigendecomposition of the real symmetric matrix A.

unitary eigenvectors of A eigenvalues of A

↓ ↓ ... ↓
   
v1,n ←
eigenvectors
v11 v21 ... vn,1 λ1 0 ... 0 v11 v12 ... transposed
←
 v21 v22 ... vn,2  0 λ2 ... 0   v12 v22 ... v2,n 
A= .
  
.. .. ..   .. .. .. ..   .. .. .. ..  ..
 .. . . .  . . . .  . . . .  .
vn,1 vn,2 ... vn,n 0 0 . . . λn v1,n v2,n ... vn,n ←
 >
v1
h i v2> 
= v1 v2 . . . vn diag(λ1 , λ2 , . . . λn )  .  = U ΛU > (1.1.1)
 
 .. 
vn>
X n
= λi vi vi> ,
i=1

where v1 , v2 , . . . vn are the unitary (column) eigenvectors of A associated to (some repeated) eigenvalues
λ1 , λ2 , . . . λn and Λ = diag(λ1 , λ2 . . . , λn ) is the diagonal matrix with Λii = λi ∀i ∈ [1..n]. The eigenvectors
are unitary and orthogonal, meaning that vi> vj = 0, ∀i, j ∈ [1..n], i 6= j and vi> vi = 1 ∀i ∈ [1..n]. This
directly leads to

[v1 v2 . . . vn ]> [v1 v2 . . . vn ] = In , equivalent to [v1 v2 . . . vn ][v1 v2 . . . vn ]> = In , (1.1.2)

using the very well known property XY = I =⇒ Y X = I (Prop. A.2.5).

4
The simple case of distinct eigenvalues
It is important to familiarize with this decomposition. For this, let us first examine a proof that works for
symmetric matrices A with distinct eigenvalues (all with multiplicity one). We will first show that the eigen-
vectors of symmetric A are orthogonal. Let v1 , v2 , . . . vn be the unitary eigenvectors of resp. λ1 , λ2 , . . . λn .
For any i, j ∈ [1..n], i 6= j, we can write vi> Avj = vi> λj vj = λj vi> vj and also vi> Avj = λi vi> vj based on
>
vi> A = (vi> A)> = (A> vi )> = (Avi )> = λi vi> . This leads to λj vi> vj = λi vi> vj , and, using λi 6= λj , we
obtain vi> vj = 0 ∀i, j ∈ [1..n], i 6= j.
We now construct the eigen-decomposition. Using λi vi = Avi ∀i ∈ [1..n], we obtain A[v1 v2 . . . vn ] =
[λ1 v1 λ2 v2 . . . λn vn ], equivalent to A[v1 v2 . . . vn ] = [v1 v2 . . . vn ]diag(λ1 , λ2 , . . . λn ). We now multiply
both sides at right by [v1 v2 . . . vn ]> which is equal to [v1 v2 . . . vn ]−1 by virtue (1.1.2); we thus obtain
A = [v1 v2 . . . vn ]diag(λ1 , λ2 , . . . λn )[v1 v2 . . . vn ]> , which is exactly (1.1.1).

Generalizing the proof for the case of eigenvalues with different multiplicities
We now prove the eigen-decomposition in the general case by extending the above construction. We need
two steps

Step. 1 We first show that the above construction for eigenvalues with unitary multiplicities can be readily
generalized to the case where there is a different eigenvector for each of the ki repetitions of each
root λi of the characteristic polynomial (i.e., for any eigenvalue λi ). In this case, there are ki linearly
independent eigenvectors associated to λi ; we say that the eigenspace dimension ki0 of λi (the geometric
multiplicity ki0 ) is equal to the algebraic multiplicity ki . This step is relatively straightforward and it
is proved in full detail (and slightly generalized) in Prop. B.2.2.
Step. 2 We need to show that the eigenspace of λi has dimension ki . To show this, we use the notion
of similar matrices: X and Y are similar if there exists U such that X = U Y U −1 ; we say that
Y is the representation of X in the basis composed by the columns of U . Two simple properties
(Prop. B.1.4 and B.1.5) show that: (i) two similar matrices have the same characteristic polynomial so
that any common eigenvalue λi is repeated ki times and (ii) similar matrices have the same eigenspace
dimension ki0 for any common eigenvalue λi . Assuming ki0 < ki for symmetric A ∈ Rn×n , we can
construct U ∈ Rn×n by putting ki0 orthogonal unitary eigenvectors of λi on the first ki0 columns and by
0 0
filling the other columns (introducing n−k
h i unitary
i vectors perpendicular on the first ki columns) such
λi Ik0 0
that U > U = In . We obtain U > AU = i . Since similar matrices have the same characteristic
0 D
polynomial and ki0
< k, λi needs to be a root of the characteristic polynomial
h i of D, and so, λi needs
λi Ik0 0
to have an eigenvector in D. This means that the eigenspace of λi in i has dimension at least
0 D
ki0 + 1, contradicting the fact that similar matrices have the same eigenspace dimension for λi . As such,
the assumption ki0 < ki was false and we obtain ki0 = ki , i.e., λi has the same algebraic and geometric
multiplicity ki . For the skeptical, this Step 2 is proven in full detail in Prop. B.2.3.

1.2 Equivalent SDP definitions

Definition 1.2.1. A symmetric matrix A ∈ Rn×n is positive semidefinite (SDP) if the following holds for
any vector x ∈ Rn
>
· >
·
x Ax = x Ax = A xx = trace(Xxx ) = >
Xn X n
aij xi xj ≥ 0,
i=1 j=1
(1.2.1)

·
where stands for the scalar product. If the above inequality is always strict, the matrix is positive definite.
If the inequality is reversed, the matrix is negative semidefinite.
Proposition 1.2.2. A symmetric matrix A ∈ Rn×n is positive semidefinite (resp. definite) if and only if all
eigenvalues λi verify λi ≥ 0 (resp. > 0).
Proof.
=⇒
We consider A is positive semidefinite (resp. definite). Assume there exist an eigenvalue λ < 0 (resp λ ≤ 0)
associated to eigenvector v. We have v> Av = v> λv = λ(v12 + v22 + . . . vn2 ). If λ < 0 (resp λ ≤ 0), then

5
v> Av < 0 (resp v> Av ≤ 0). Thus, A is not positive semidefinite (resp. definite). This is a contradiction,
and so, the assumption λ < 0 (resp λ ≤ 0) was false. We need to have λ ≥ 0 (resp λ > 0).
⇐=
Without loss of generality, we suppose
λ1 ≤ λ2 ≤ . . . λn
We consider λ1 satisfies λ1 ≥ 0 (resp λ1 > 0). We consider the eigenvalue decomposition of symmetric
matrix A, as constructed in (1.1.1) – see also (B.2.3) of Proposition B.2.1.
n
X
A= λi vi vi> , (1.2.2)
i=1

where v1 , v2 , . . . , vn are the unitary orthonormal eigenvectors of A. We consider the following minimization
problem over all unitary vectors x ∈ Rn and we will show it is non-negative (resp. strictly positive):

min x
|x|=1 · Ax (1.2.3)

·
The above formula x Ax with unitary x is actually a particular case of the Rayleigh ratio/quotient
usually written under the form R(A, x) = xx· Ax
x . ·
Lemma [Link]. The minimum value of (1.2.3) above is the smallest eigenvalue λ1 . This minimum is
attained by the unit eigenvector v1 of λ1 . This lemma actually holds for any real symmetric matrix A. This
also implies that the maximum eigenvalue can be determined by calculating the minimum eigenvalue of −A,
·
i.e., λn = −λmin (−A) = − min|x|=1 −x Ax = max|x|=1 x Ax. ·
Proof. We can write x in basis v1 , v2 , . . . vn . This is always possible because the equation [v1 v2 . . . vn ]a = x
always has the solution a = [v1 v2 . . . vn ]> x simply using (1.1.2). We can write

x = a1 v1 + a2 v2 + . . . an vn (1.2.4)
Pn
·
Pn Pn
Since |x| = 1, we have i=1 a2i vi vi + 2 i=1 j=1,j6=i ai aj vi vj = 1. Since vi · ·v j = 0 ∀i 6= j and
· Pn
vi vi = 1, we obtain 1 = i=1 a2i . We now replace this in (1.2.3) and we obtain

· ·
n
! n
!
X X
min x Ax = Pnmin2 ai vi A ai vi
|x|=1 i=1 ai =1
i=1 i=1

·
n
! n
!
X X
= Pnmin2 ai vi ai λi vi
i=1 ai =1
i=1 i=1
n
X
= Pnmin2 λi a2i
i=1 ai =1
i=1
n
X
≥ Pnmin2 λ1 a2i
i=1 ai =1
i=1
= λ1 , (1.2.5)

·
The inequality is not strict. Using x = v1 we obtain v1 Av1 = λ1 v1 v1 = λ1 . The proof has not used
the fact that λ1 ≥ 0, and so, it can be applied for any symmetric matrix A.
·
pPn
We recall that any y ∈ Rn can be written as y = αx such that |x| = 1 (technically α = |y| = 2
i=1 yi ).
> 2 > 2
·2 2
We obtain y Ay = α x Ax = α x Ax ≥ α λ1 . Observe this minimum α λ1 can always be attained by
y = αv1 . If λ1 ≥ 0 (resp. λ1 > 0), the matrix A is positive semidefinite (resp. definite).

Proposition 1.2.3. We say that matrices S and T are congruent if there is a non-singular Q such that
T = Q> SQ. Two congruent matrices have the same SDP status, i.e., S is semidefinite positive if and only
if T is semidefinite positive and S is definite positive if and only if T is definite positive:

S  0 ⇐⇒ T  0 and S  0 ⇐⇒ T  0.

6
 Proof.
=⇒
If S is SDP, then x> T x = x> Q> SQx = (Qx)> S(Qx) ≥ 0 =⇒ T  0. Using the fact that Q is non-singular,
we also obtain that Qx is zero only when x is zero, and so, S  0 =⇒ (Qx)> S(Qx) > 0∀x 6= 0 =⇒ T  0.
⇐=
>
The converse proof is identical, because we can write S = Q−1 T Q−1 and apply the above two lines
argument on swapped S and T . We simply used (Q> )−1 = (Q−1 )> , which is equivalent to Q> (Q−1 )> = In ,
which follows from transposing Q−1 Q = In .
We now use the above result2 to introduce a (very practical) remark on how certain well-known elementary
row/column operations preserve the SDP status.

Proposition 1.2.4. It is well known that the operations below preserve the determinant; we now prove that
they also preserve the SDP status: (a) the initial matrix A is SDP if only if the transformed matrix A0
is SDP and (b) A is positive definite if and only if the A0 is positive definite. Finally, A and A0 are also
congruent.

(i) add row i multiplied by zji ∈ R to row j and then column i multiplied by zji to column j
(ii) perform a sequence of row operations as above and then the corresponding (transposed) column operations
(iii) permute the rows of A and then permute in the same manner (apply the same permutation) on the
columns of A

Proof. The row operation from (i) amounts to performing (In + Zj.i ) A, where Zj.i is a matrix that
contains only one non-zero element: put zji at row j and column i. The column operation from (i) amounts
>
to multiplying at right by (In + Zj.i ) = In + Zi.j . The final matrix resulting from operation (i) is:
>
A0 = (In + Zj.i ) A (In + Zj.i )

As such, A and A0 become congruent (notice det (In + Zj.i ) = 1), finishing the proof by Prop 1.2.3 above.
The operation (ii) simply consists of applying (i) several times, leading to the following congruent matrices:

> > >
A0 = (In + Zj1 .i1 ) (In + Zj2 .i2 ) . . . In + Zjp .ip A In + Zjp .ip . . . (In + Zj2 .i2 ) (In + Zj1 .i1 )

The operation (iii) does not change the SDP status because the permutation (reordering) that transforms
· ·
A into A0 can be applied to transform x into some x0 so that A xx> = A0 x0 x0> , and this operation can
also be reversed.
Proposition 1.2.5. A symmetric matrix A ∈ Rn×n is positive semidefinite if and only if all principal minors
(recall Def A.2.1) are non-negative. This is equivalent to stating A  0 ⇐⇒ det([A]J ) ≥ 0 ∀J ⊆ [1..n],
where the operator [·]J represents the principal minor obtained by selecting rows J and columns J.
We will later see that A is positive definite if and only if all leading principal minors are strictly positive
(Sylvester criterion, see Prop. 1.5.2), which is also equivalent to det([A]J ) > 0 ∀J ⊆ [1..n] using Prop. 1.6.3.
Proof.
=⇒
Take any subset of indices J ⊆ [1..n] and consider any vector xJ ∈ Rn that contains non-zero elements only
on positions J. Let xJ be the |J|-dimensional vector obtained by extracting/keeping only the positions J of
xJ . Using the SDP definition (1.2.1), the following needs to hold:

A
·xJ x>
J ≥0 (1.2.6)

Since xJ x>
J contains non-zero elements only on lines J and columns J, we can re-write above formula as:

[A]J
· xJ x>
J ≥0 (1.2.7)
2 The same result appears in Section 6.9.1 (p. 91) of the lecture notes of Maurı́cio de Oliveira, available on-line as of 2019

at [Link]

7
Since this holds for any xJ , the principal minor [A]J is SDP. This means that the eigenvalues of [A]J are
non-negative (Prop. 1.2.2), and so, the determinant of [A]J is non-negative because it is equal to the product
of its eigenvalues (Prop. A.2.4).

⇐=
Let r be the rank of A. Based on Prop. A.1.2, A has at least a non-zero principal minor of order r. We
can reorder the rows and columns  of A to move this principal minor in the upper-left corner; we obtain
a (permuted) matrix A0 = B A B > , where A is non singular; A, A0 and A have rang r. Prop 1.2.4.(iii),

C
certifies that A0 has the same SDP status as A. Since A0 has rang r, the bottom n − r rows (i.e.[ B C ]), can
be written as a linear combination of the first r rows (i.e., [ A B > ]). We can subtract this linear combination
of the first r rows from the last n−r rows to cancel them
 (make them zero). After performing the transposed
operation on the columns, we obtain a matrix A00 = A 0 that has the same SDP status as A0 and A, using

0 0
Prop. 1.2.4.(ii). To prove A, A0 , A00  0, it is enough to solve the following (sub-)problem:

Any non-singular symmetric A ∈ Rr×r that satisfies det([A]J ) ≥ 0 ∀J ⊆ [1..r] is SDP (*)
First, notice Ar,r > 0 because Ar,r = 0 would lead to Ai,r = 0 ∀i ∈ [1..r − 1] (otherwise the 2 × 2 minor
of A selecting rows/columns i and r would be negative) and if the last column of A is zero then det(A) = 0,
contradiction. We can now subtract the last row r from each other row i ∈ [1..r − 1] in such a manner (i.e.,
Ai,r
after multiplying it by to cancel all elements on the last column above Ar,r . We then apply the same
Ar,r
)
h i
Ar−1 0
row operations but transposed (generating column operations). This leads to a matrix 0 A
with the
r,r

same SDP status as A by virtue of Prop. 1.2.4.(ii).

h iAlso, any principal minor AJ of Ar−1 is non-singular
because it corresponds to a principal minor A0J A0 whose determinant is det(AJ ) · Ar,r where Ar,r > 0.
r,r

Thus, to prove (*), it is enough to prove Ar−1  0. But since Ar−1 satisfies all conditions of (*), we have
actually obtained the same problem (*) on a dimension reduced by one. We can repeat this until we remain
with a matrix of size 1 and this proves A  0, enough to certify A  0.

1.3 Schur complements, the self-duality of the SDP cone and related properties
1 b>
 
Proposition 1.3.1. (Schur complements particular case) The (n + 1) × (n + 1) matrix b C
is SDP if
and only if C − bb> is SDP.
Proof. We will give two proofs. The first one produces a congruent matrix using row/column operations.
The second one is actually a formalization of the first, but it uses a “magical” short decomposition.
 >
Proof 1) Let us subtract the first row of b1 bC from all other rows i + 1 (∀i ∈ [1..n]) premultiplying [1 b> ]
with bi . We then perform the transposed operation on the resulting matrix. The two operations lead to

1 b> b> 0>

     
1 1 n
→ → (1.3.1)
b C 0n C − bb> 0n C − bb>
Using Prop 1.2.4, the above two operations together lead to a congruent matrix (at right) with the same
SDP status as the initial one (at left). The second matrix is SDP if and only if C − bb> is SDP (the “⇒”
implication follows from performing a scalar product with any xx> with x0 = 0 and the “⇐” implication
follows from the fact that the sum of two SDP matrices is SDP).
Proof 2) We formalize (1.3.1) using matrix multiplications. The first transformation can be realized by:
b> 1 0> 1 b>
    
1 n
= (1.3.2)
0n C − bb> −b In b C
and the second one by:
0> b> −b>
    
1 n 1 1
= (1.3.3)
0n C − bb> 0n C − bb> 0n In
Combining (1.3.2)-(1.3.3), we obtain:
0> 0> 1 b> −b>
     
1 n 1 n 1
=
0n C − bb> −b In b C 0n In

8
 0>
h i h i
1 0>
We obtain again that 1 1 b>
 
n
is congruent to , because det n = 1. This finishes the proof
0n C−bb> b C −b In
by virtue of Prop 1.2.3.
Proposition 1.3.2. (Schur complements general case) Given positive definite A ∈ Rm×m , the (n + m) ×
>
is SDP if and only if C − BA−1 B > is SDP.
 
(n + m) matrix BA B
C

Proof. As in previous Prop. 1.3.1, we want to subtract from the (bottom) n rows that cover B ∈ Rn×m
a linear combination of the top m rows so as to cancel (make zero) all terms of B. We look for a matrix
X ∈ Rn×m such that XA = −B; incidentally, X and B have the same size because the multiplication with a
square matrix conserves the size (of A). By this operation, each row i of X generates a linear combination of
the rows of A that equals the negative of row i of B. The transpose of this operation is applied on columns
to cancel B > . It is clear that X = −BA−1 . This explains the bottom-left term of matrix U below.

0m×n A B > −A−1 B >

     
A 0m×n Im Im
=
0n×m C − BA−1 B > −BA−1 In B C 0n×m In = (A −
} 1>
)
| {z } | {z
U U> B>

Notice that U > is written after applying the simplification (A−1 )> = A−1 (which follows from I =

>
>
−1  A 0m×n
(A−1 A)> = A> A−1 =⇒ A−1 = A> = A−1 ). Since det(U ) = 1, we obtain that 0n×m C−BA

−1 >
B
A B > are congruent. We finish with Prop 1.2.3 as for Prop 1.3.1 above.
 
and B C

Proposition 1.3.3. A symmetric matrix A ∈ Rn×n is SDP if and only if A B ≥ 0 for any SDP matrix
B. We say that the cone Sn+ of SDP matrices is self-dual.
·
Proof. We apply the eigen-decomposition (1.1.1) on A and B: :
n
X n
X
A= λi vi vi> , B= λ0i ui u>
i , (1.3.4)
i=1 i=1

Lemma [Link]. Given vectors u, v ∈ Rn , we have

(uu> )
· (vv> ) = (u
· v)2

· ·
Proof.
(uu> ) (vv> ) = v> (uu> )v = (v> u)(u> v) = (u> v)> (u> v) = (u v)2

=⇒
If A is SDP, then λi ≥ 0∀i ∈ [1..n]. Using substitution (1.3.4), we obtain

A
· B=
n X
X n

i=1 j=1
λi λ0j (vi vi> )
· (uj u>
j )=
n X
X n

i=1 j=1
λi λ0j (vi
· uj )2 ,

where we used Lemma [Link] for the last equality. This shows that A B ≥ 0.
⇐=
·
Let us take some i ∈ [1..n] and consider B = vi> vi , where recall that vi is a unit eigenvector of A. Since
· ·
A B ≥ 0, we deduce A vi> vi ≥ 0, or vi> Avi ≥ 0, equivalent to vi> λi vi ≥ 0. This means that λi ≥ 0. All
eigenvalues of A are non-negative, and so, A is SDP.

·
Proposition 1.3.4. If A, B are SDP, then A B ≥ 0 and A B = 0 ⇐⇒ AB = 0n×n . ·
Proof. We apply the eigen-decomposition (1.1.1) listing the only terms with non-zero eigenvalues:
0
r
X r
X
A= λi vi vi> , B= λ0i ui u>
i (1.3.5)
i=1 i=1

9
where r and r0 are the ranks of A and resp. B (the number of non-zero eigenvalues, see Prop A.1.7). We
now use Lemma [Link] to calculate

· · ·
0
n X
X r n X
X r
A B= λi λ0j (vi vi> ) (uj u>
j ) = λi λ0j (vi uj )2 ≥ 0
i=1 j=1 i=1 j=1

· ·
If A B = 0, then all terms vi uj with i ∈ [1..r] and j ∈ [1..r0 ] need to be zero (recall we only use
strictly positive eigenvalues). Now observe
0 0 0
r X
X r r X
X r r X
X r
AB = λi λj (vi vi> )(uj u>
j ) = λi λj vi (vi> uj )u>
j = λi λj vi · 0 · u>
j = 0n×n
i=1 j=1 i=1 j=1 i=1 j=1

We still need to show the P

diagonal
n
·
converse: AB = 0n×n =⇒ A B = 0. Taking any k ∈ [1..n], the k th
Pn element of AB is 0 = `=1 ak` bk` (we used the symmetry of B). Summing up for all k we obtain
Pn
0 = k=1 `=1 ak` bk` = A B. ·
1.4 Three easy ways to generate (semi-)definite positive matrices
There are at least three easy ways of generating semidefinite (or definite) positive matrices.
1. Generate A ∈ Rn×n by taking A = V > V for any V ∈ Rp×n . It is easy to verify x> V > V x =
(V x)> (V x) = |V x|2 ≥ 0 ∀x ∈ Rn . If V has rank n, then V x is non-zero for any non-zero x, and so,
|V x|2 > 0 ∀x ∈ Rn − {0}, meaning that A = V > V is positive definite. If S  0 and V has rank n, we
also have V > SV  0. As a side remark, notice rank(A) = rank(V ) using Prop. A.1.8 (based on the
rank–nullity theorem)
n
P
2. Take a diagonally dominant matrix such that Aii ≥ ri = |Aij | ∀i ∈ [1..n]. Any eigenvalue λ of such
j=1
j6=i
matrices verify |λ − Aii | ≤ ri for some i ∈ [1..n] by virtue of the (relatively easy to prove) Gershgorin
circle Theorem A.2.8.3 If Aii ≥ ri ∀i ∈ [1..n], we need to have λ ≥ 0. The matrix A + εIn is positive
definite for any ε > 0.
p
·
3. Take A = |X|In + X, where |X| = X X. This follows from the fact that the minimum eigenvalue
of X is greater than or equal to −|X|, by virtue of Prop. 1.4.1 below.
s
·
n p
n×n 2 =
P
Proposition 1.4.1. Given symmetric X ∈ R , the Frobenius norm |X| = Xij X X is
i,j=1
p
equal to λ21 + λ22 + . . . λ2n , where λ1 , λ2 , . . . λn are the eigenvalues of X. This means that the maximum
eigenvalue of X is at most |X| and the minimum eigenvalue is at least −|X|.
p p
·
Proof. Standard calculations can confirm |X| = X X = trace(XX); and more generally we have
·
trace(XY ) = X Y . We apply the eigendecomposition (1.1.1) to write symmetric X in the form X =
U diag(λ1 , λ2 , . . . λn )U > , where U > = U −1 . We obtain XX = U diag(λ21 , λ22 , . . . λ2n )U > . This means that
the eigenvalues of XX are λ21 , λ22 , . . . λ2n , see also Prop B.1.4. Since the trace is the sum of the eigenvalues
(see Prop. A.2.4), we obtain
p q
|X| = trace(XX) = λ21 + λ22 + · · · + λ2n . (1.4.1)

Is is clear X can have no eigenvalue strictly larger than |X| or strictly lower than −|X|, because this would
violate (1.4.1).4
3 I found this approach at page 4 of the Habilitation thesis (Habilitationsschrift) of Christoph Helmberg “Semidefinite

Programming for Combinatorial Optimization”, Technical University of Berlin (Technische Universität Berlin), The Zuse
Institute Berlin (Konrad-Zuse-Zentrum für Informationstechnik Berlin), ZIB-report ZR-00-34, available on-line as of 2017 at
[Link]
4 I first found this result in Section 10.1 of the lecture notes of Robert Freund “Introduction to Semidefinite Program-

ming (SDP)”, available on-line as of 2017 at [Link]

6-251j-introduction-to-mathematical-programming-fall-2009/readings/MIT6_251JF09_SDP.pdf.

10
1.5 Positive definite matrices: unique Cholesky factorization and Sylvester cri-
terion
Proposition 1.5.1. (Cholesky factorization of positive definite matrices) A real symmetric matrix A is
positive definite if and only if it can be factorized as:
  
r11 0 0 ... 0 r11 r21 r31 . . . rn1
 r21 r22 0 ... 0   0 r22 r32 . . . rn2 
 

A = RR> =  r31 r32 r33 . . . 0  0 0 r33 . . . rn3 ,
  
 .. .. .. .. . .
..   ..
  .. .. .. .. 
 . . . . . . . . 
rn1 rn2 rn3 . . . rnn 0 0 0 . . . rnn

where the diagonal terms are strictly positive. The factorization is unique.
Practical hint: It could be useful to interpret A = RR> in the sense that Aij is the product of rows i and
j of R; only the first i (resp. j) components of row i (resp. j) are non-zero.
Proof.
⇐=
Take any non-zero x ∈PRn − {0n } and observe that x> Ax = x> RR> x = (R> x)> (R> x). Writing Pny = R
>
x,
n 2 n 2
this value is equal to i=1 yi ≥ 0. This inequality is strict because the only y ∈ R such that i=1 yi = 0
is y = 0n and because y = R> x can not be 0n for any non-zero x (since det(R> ) = r11 r22 . . . rnn > 0).
=⇒
We proceed by induction. The implication is obviously true for n = 1. We suppose that there exists a unique
factorization:
  
r11 0 ... 0 r11 r21 . . . rn−1,1
 r21 r22 ... 0   0 r22 . . . rn−1,2 
[A]n−1 = [R]n−1 [R]>
n−1 =  ,
  
.. .. . . .
.   .
. .
. . . ..
 . . . .  . . . . 
rn−1,1 rn−1,2 . . . rn−1,n−1 0 0 . . . rn−1,n−1

where r11 , r22 , . . . rn−1,n−1 > 0. We will prove that this decomposition can be extended to a n × n
decomposition for matrix A. The values r1n , r2n , . . . rn−1,n are set to zero by definition to preserve the
decomposition of [A]n−1 . We can exactly determine rn,1 , rn,2 , . . . rn,n using the following calculations:

(a) rn1 =
an1
r11
, based on an1 = rn,× ·r
1,× , where ri,× is the ith line of R.

(b) rn2 =
an2 − r21 rn1
r22
, based on an2 = rn,× ·r 2,× .

(c) rni =
ani − ri1 rn1 − ri2 rn2 − · · · − ri,i−1 rn,i−1
rii ·
for any i ≤ n − 1, based on ani = rn,× ri,× .
q Pn−1 2
(d) rnn = ann − i rni , so that the value of rnn that makes the factorisation A = RR> work can be
potentially non-real, e.g., we could have rnn = i so that i2 = −1.
We only still need to show rnn is real. Since [R]n−1 is non-singular by the induction hypothesis, there
exists x ∈ Rn−1 so that x> [R]n−1 = [rn,1 rn,2 , . . . rn,n−1 ]. By bordering x with a nth component of value
x x
-1, we obtain [x> − 1]R = [0n−1 − rnn ]. This means that [x> − 1]A [ −1 ] = [x> − 1]RR> [ −1 ] =
> 2 2
[0n−1 − rnn ][0n−1 − rnn ] = rnn . We thus have rnn > 0 because A  0; recalling how rnn was determined
at point (d) above, it is clear that rnn can not be an imaginary number, i.e., we can only have rnn ∈ R+ .
Proposition 1.5.2. (Sylvester criterion) A symmetric matrix A ∈ Rn×n is positive definite if and only
all leading principal minors are strictly positive. By symmetrically permuting the rows and columns, this
is equivalent to the fact that any nested sequence of principal minors contains only strictly positive minors.
Principal minors A00 and A0 corresponding to rows/columns J 00 and resp. J 0 are nested if and only if J 00 ( J 0
and |J 00 | + 1 = |J 0 |.

11
This theorem can be tackled from many different angles. The interested reader may be able to find a proof by
himself (in less than an hour) if all material presented until here (including Appendix A) has been acquired.
We present below three proofs, so as to gain an insight from every possible angle.
Proof 1
We proceed by induction. Both implications are obviously true for n = 1. We now show how to move from
n − 1 to n. We can use the fact that the (n − 1) × (n − 1) leading principal minor [A]n−1 has a non-null
determinant. Since det[A]n−1 6= 0, there exists x ∈ Rn−1 such that x> [A]n−1 is equal to the first n − 1
positions of the last row of A. After subtracting this linear combination of the first n − 1 rows of A from the
last row followed by the transposed operation on columns, we obtain a matrix of the following form:
 
0
0 
A0 = [A]n−1

..  .
 
 . 
0 0 ... a0n,n

We finish by applying Prop 1.2.4: the above subtraction does not change the SDP status or the deter-
minant, and so, it is enough the prove the Sylvester criterion for a matrix of the form of A0 above. And
this is obvious. For the direct implication, simply check that A0  0 =⇒ a0n,n > 0 =⇒ det(A0 ) =
det([A]n−1 ) · a0n,n > 0. For the converse, we use that det(A0 ) > 0 =⇒ a0n,n > 0. This proves A0  0 because
A0 · xx> = [x1 x2 . . . xn−1 ][A]n−1 [x1 x2 . . . xn−1 ]> + a0n,n x2n , which is strictly positive for any x 6= 0 because
[A]n−1  0 and a0n,n > 0.
Proof 2
=⇒
Using the above Cholesky factorization of positive definite matrices (Prop. 1.5.1), A can be written A = RR> ,
where R is a lower triangular matrix with strictly positive diagonal elements r11 , r22 , . . . rkk > 0. One can
simply verify that [A]k = [R]k [R]> k , where the operator [·]k represents the leading principal minor of size k.
2
We obtain that det([A]k ) = det([R]k )det([R]> k ) = (r11 r22 . . . rkk ) > 0, by virtue of r11 , r22 , . . . rkk > 0.
⇐=
We proceed by induction. The implication is obviously true for n = 1. Suppose it is true for n − 1, so that
the (n − 1) × (n − 1) leading principal minor [A]n−1 is positive definite.
We need to show that A is positive definite as well. Assume the contrary: A has an eigenvalue λu ≤ 0
with unit eigenvector u. Since det(A) > 0 is the product of the eigenvalues of A, λu can not be zero, and so,
λu < 0. Using again the fact that det(A) is the product of the eigenvectors, we obtain that A needs to have
(at least) another negative eigenvalue λv < 0 with unit eigenvector v.
There exist au , av ∈ R (not both 0) such that au un + av vn = 0, i.e., if un = 0 take au = 1, av = 0 and if
un 6= 0 take au = −v un , av = 1. We define x = au u + av v, so that xn = 0. We compute
n

x> Ax = x
· Ax = (au u + av v)
· A (au u + av v)

= (au u + av v)
· (λu au u + λv av v)

= λu a2u u
= λu a2u +
· u + λv a2v v
λv a2v
· v

<0
The last inequality follows from λu , λv < 0 and from the fact that au and av are not both zero. We now
·
develop x> Ax = A xx> and notice that matrix xx> has non-zero elements only on the first n − 1 rows and
· ·
columns. As such, A xx> < 0 simplifies to [A]n−1 [x1 x2 . . . xn−1 ][x1 x2 . . . xn−1 ]> < 0, which violates
the induction hypothesis that [A]n−1 is positive definite. We obtained a contradiction on the existence of an
eigenvalue λu ≤ 0. This means that all eigenvalues of A are strictly positive. By simply applying Prop. 1.2.2,
we obtain that A is positive definite.
Proof 3
=⇒
We can use the proof of the “=⇒” implication of Prop. 1.2.5, but the inequalities (1.2.6)-(1.2.7) become
strict. This means that every principal minor of A is positive definite, and, using Prop. 1.2.2, the principal
minor has only strictly positive eigenvalues. The determinant of the minor is strictly positive, as it is the

12
product of the eigenvalues (Prop. A.2.4).
⇐=
We proceed by induction. The statement is true for n = 1. Suppose it is true for n − 1. The (n − 1) × (n − 1)
leading principal minor is positive definite, and so, its minimum eigenvalue λmin ([A]n−1 ) is strictly positive.
We then apply lemma below to show that A has at least n − 1 strictly positive eigenvalues, i.e., 0 <
λmin ([A]n−1 ) ≤ λ2 ≤ λ3 ≤ . . . λn . This ensures that λ1 λ2 . . . λn = det(A) > 0 (recall Prop. A.2.4) can only
hold because λ1 > 0, so that actually all eigenvalues need to be positive which proves A  0 (via Prop 1.2.2).
Lemma [Link]. Consider symmetric matrix A with eigenvalues λ1 ≤ λ2 ≤ · · · ≤ λn . Any principal minor
A0 of order n − 1 with eigenvalues λ01 ≤ λ02 ≤ · · · ≤ λ0n−1 verifies λ01 ≤ λ2 .

Proof. We consider A0 is obtained from A by removing row i and column i. Denote by u1 and u2 the
unit orthogonal eigenvectors of A corresponding to λ1 and resp. λ2 , recall (1.1.1). One can surely find
α1 , α2 ∈ R (not both zero) such that vector u = α1 u1 + α2 u2 satisfies ui = 0. Furthermore, using an
appropriate scaling, the values of α1 and α2 can be chosen such that α12 + α22 = 1. Notice that |u|2 =
· · · ·
(α1 u1 + α2 u2 ) (α1 u1 + α2 u2 ) = α12 u1 u1 + α22 u2 u2 + 2α1 α2 u1 u2 = α12 |u1 |2 + α22 |u2 |2 = α12 + α22 = 1,
where we used that u1 and u2 are unitary orthogonal eigenvectors.
Let us calculate u Au (a particular form of the Rayleigh ratio R(A, u) = uu· Au
· ·
· u ). We have u Au =
· · 2 2
u (α1 λ1 u1 + α2 λ2 u2 ) = (α1 u1 + α2 u2 ) (α1 λ1 u1 + α2 λ2 u2 ) = α1 λ1 + α2 λ2 ≤ λ2 , where we used again
that u1 and u2 are unitary orthogonal eigenvectors, followed by λ1 ≤ λ2 . Let u0 be u without component i
· · · ·
and we obtain u Au = A uu> = A0 u0 u0> = u0 A0 u0 , where we used the fact that row and column i of
·
uu> are zero based on ui = 0. We obtained that the unitary u0 yields u0 A0 u0 ≤ λ2 . Using Lemma [Link],
·
the smallest eigenvalue of A0 is less than or equal to u0 A0 u0 ≤ λ2 , i.e., λ01 ≤ λ2 .

The following lemma could be generally useful, although it is not used for other proofs in this document.

Lemma [Link]. Consider symmetric matrix A with eigenvalues λ1 ≤ λ2 ≤ · · · ≤ λn . Any principal minor
A0 of order n − 1 with eigenvalues λ01 ≤ λ02 ≤ · · · ≤ λ0n−1 satisfies:5

λ1 ≤ λ01 ≤ λ2 (1.5.1a)
λn−1 ≤ λ0n−1 ≤ λn (1.5.1b)

Proof. We have already proved in Lemma [Link] that λ01 ≤ λ2 . To show that λ1 ≤ λ01 , let us consider the
·
unitary eigenvector v0 of λ01 in A0 and notice v0 A0 v0 = λ01 . If A0 is obtained from A by removing row i
and column i, we can say v0 is obtained by removing position i from an v ∈ Rn such that vi = 0. We can
· ·
write λ01 = v0 A0 v0 = v Av. We recall Lemma [Link] that states that λ1 is the minimizer of the Rayleigh
·
ratio, and so, we obtain: λ1 = min x Ax ≤ v Av = λ01 .
|x|=1
·
To prove (1.5.1b) it is enough to apply (1.5.1a) on matrices −A and −A0 .

1.6 Cholesky decomposition of semidefinite positive matrices

We provide two proofs. The first one is much shorter, but the factorization arises somehow out of the
blue. The second one is longer and a bit more general, discussing along the way several properties that are
generally useful (Prop. 1.6.3 and 1.6.4), leading the reader to a deeper insight into the factorization.
5 It is possible to prove that
λ1 ≤ λ01 ≤ λ2 ≤ λ02 ≤ . . . λ0n−2 ≤ λn−1 ≤ λ0n−1 ≤ λn
The proof lies outside the scope of this document, but it could use the following argument by contradiction. Assume
λ01 , λ02 , . . . λ0k < λk and we can derive a contradiction. All vectors of the subspace S 0 generated by the first k eigenvec-
tors of A0 have a Rayleigh ratio ≤ λ0k . All vectors of the subspace S generated by the last n − k + 1 eigenvectors of A
have a Rayleigh ratio ≥ λk . But the subspaces S and S 0 need to have an intersection of at least dimension 1, because
dim(S) + dim(S 0 ) = n − k + 1 + k = n + 1. We obtained a contradiction: the Rayleigh ratio over this intersection needs
to be both ≥ λk and ≤ λ0k . An analogous reversed argument could be used to show that λ0k ≤ λk+1 . Different proofs
can be found by searching key words “Interlacing eigenvalues” on the internet, the one from the David Williamson’s course
([Link] is the most related to the above ideas. Another proof can be
found at least in [Ikramov H., Recueil de problèmes d’algébre lineaire, publisher MIR - Moscou, 1977], exercise 7.4.35.

13
1.6.1 A short proof using the square root and the QR decompositions
[Link] Square root factorisations of SDP matrices
Given matrix A  0, we are looking for matrices K such that KK = A. We apply the eigendecomposition
(1.1.1) and write A
√=√ U ΛU > , √
where

Λ = diag(λ1 , λ2 , . . . λn ) contains the non-negative eigenvalues of A.
Take D = diag λ1 , λ2 , . . . λn and consider K = U DU > . One can easily check that this K is a
square root of A: KK = U DU > U DU > = U DDU > = U ΛU > = A, where we used U > U = In from
(1.1.2).√This√K is called
√
the principal square root of A and it is SDP, because it is similar and congruent to
diag λ1 , λ2 , . . . λn . It is the only SDP square root of A (Appendix B.4).

Remark 1.6.1. There are multiple symmetric matrices K ∈ Rn×n such that A = KK that can be found
by taking K = U DU > for any symmetric
√ D
√ such that√ DD
= Λ. Examples of such symmetric matrices D
can be found by taking D = diag ± λ1 , ± λ2 , . . . ± λn .
Non-symmetric matrices K such that A = KK do exist; in fact, they can be infinitely many, because
there can hbe infinitely
i h s rmany matrices D ∈ Rn×n such that DD = diag(λ1 , λ2 , . . . λn ). To see this, notice
s r
i
that I2 = r − s
t t t
r
t
s for infinitely many Pythagorean triples (r, s, t) such that r2 + s2 = t2 . If we have
t t t −t
λ1 = λ2 , the above 2 × 2 construction can be extended to a n × n matrix in which the leading principal
minor of size 2 × 2 is given by this 2 × 2 construction.
As a side remark, we can prove that rank(A) = rank(K) if K is symmetric by
 1 applying Prop. A.1.8 on
A = K > K. This is no longer true with non-symmetric matrices, e.g., 0 = 11 −1 −1
 
−1 1 −1 has rank zero while
the square root factor has rank 1.

[Link] The Cholesky factorization proof

Proposition 1.6.2. (Cholesky factorization of positive semidefinite matrices) Any real SDP matrix A can
be factorized as A = LL> where L is a lower triangular matrix with non-negative diagonal elements. This
proposition is slightly weaker than Prop 1.6.7 where we will show (using a longer proof ) that there always
exists a factorization in which the value of Lnn depends on the ranks of A and of the leading principal minor
of size n − 1 of A.
Proof.
We know there exists a symmetric K ∈ Rn×n such that A = KK
   

  
using Remark 1.6.1 above. We now apply the QR decomposition


   
on K and write K = QR where Q ∈ Rn×p satisfies Q> Q = Ip n 

K
 
 = Q

 R 
0
    
    
and R ∈ Rp×n is upper triangular for some p ≤ n—see the proof



in Prop. B.3.1. We can develop: | {z
p
}

A = KK = K > K = (QR)> QR = R> Q> QR = R> Ip R = R> R,

which is very close to a Cholesky decomposition, because R> is lower triangular. However, R> has only
p ≤ n columns, but we can extended to a n × n matrix by adding n − p zero columns. This way, R> is
transformed into a lower triangular square matrix L such that L L> = R> R = A. The last point to address
is the fact that the QR decomposition from Prop. B.3.1 does not state that the diagonal elements of R are
non-negative. There might exist multiple i ∈ [1..p] such that Lii = Rii < 0. We can overcome this with a
simple trick. The product LL> does not change if we negate all columns i of L satisfying Lii < 0, because
Ajj 0 is the dot product of rows j and j 0 of L which does not change if both rows j and j 0 negate some
column(s) i. This leads to a factorisation in which the factors have a non-negative diagonal. 6

1.6.2 A longer Cholesky proof providing more insight into the properties of SDP matrices
Proposition 1.6.3. If SDP matrix A ∈ Rn×n has some null principal minor (i.e., ∃J ⊂ [1..n] such that
det([A]J ) = 0), then A  0 and det(A) = 0.
6 I first found this proof in an answer of user loup blanc on the on-line forum [Link]

1331451/how-to-prove-cholesky-decomposition-for-positive-semidefinite-matrices. It also appear in Corollary 7.2.9 of

the book “Matrix Analysis” by Roger Horn and Charles Johnson, second edition, Cambridge University Press, 2013.

14
Proof 1. One of the eigenvalues of [A]J need to be zero so that [A]J  0 and [A]J xJ x> ·
J = 0 for some
vector xJ with |J| components. Construct x ∈ Rn by keeping the values of xJ on the positions J of x and
·
by filling the rest with zeros. It is not hard to check that A xx> = [A]J xJ x>
and det(A) is zero as the product of the eigenvalues.
·
J = 0, so that then A  0

Proof 2. Re-order the rows and columns of A so that the minor [A]J becomes a leading principal minor; the
Sylvester criterion is thus violated, which shows A  0 .
Proposition 1.6.4. If we are given an SDP matrix A ∈ Rn×n written under the form
 
b1
b2 
A =  [A]n−1 .  ,

 
 .. 
b1 b2 ... bn

then [b1 b2 . . . bn−1 ] can be written as a linear combination of the rows of [A]n−1 . This combination uses
only rows J where J ⊂ [1..n − 1] such that [A]J (matrix obtained by selecting rows J and columns J) is a
non-null principal minor of maximum order (the rank of [A]n−1 ).
We provide two proofs. The first one is much shorter, but the second one provides more insight into the
arrangement of the matrices.
Proof 1. Assume [b1 b2 . . . bn−1 ] does not belong to the row image (set of linear combinations of the rows)
of [A]n−1 . Using the rank-nullity Theorem A.1.3, we have rank([A]n−1 ) + nullity([A]n−1 ) = n − 1. The
dimension of the image img([A]n−1 ) plus the dimension of null([A]n−1 ) is equal to n − 1. Since the two
spaces are perpendicular (any x0 ∈ null([A]n−1 ) satisfies [A]in−1 x0 = 0 for any row i of [A]n−1 ), the sums
between elements of null([A]n−1 ) and elements of img([A]n−1 ) cover the whole (transposed) Rn−1 . As such,
we can write [b1 b2 . . . bn−1 ] = b> > >
img + b0 , where bimg ∈ img([A]n−1 ) and b0 ∈ null([A]n−1 ) with b0 6= 0.
We can now calculate:
 
> −tb0
[−tb0 1]A = t2 b> > >
0 [A]n−1 b0 − 2t(bimg + b0 )b0 + bn
1
= 0 − 2tb> 2
0 b0 + bn = bn − 2t|b0 | → −∞

We obtained a contradiction, the assumption [b1 b2 . . . bn−1 ] ∈

/ img([A]n−1 ) was false.

Proof 2. If [A]n−1 is non-singular, the conclusion is obvious: x> [A]n−1 = [b1 b2 . . . bn−1 ] has solution
x> = [b1 b2 . . . bn−1 ][A]−1
n−1 .
We hereafter consider [A]n−1 has rank r < n−1. Based on Prop. A.1.2, A has at least a non-zero principal
minor of order r. Without loss of generality, we permute the rows and columns of A until this non-zero minor
is positioned in the upper-left corner – this does not change the determinant or the SDP status (Prop 1.2.4).
Let [A]r be this leading principal minor. Consider the solution x of the system x> [A]r = b> r , where br is b
reduced to positions [1..r]. This solution exists and it has value x> = b> −1
r [A]r .
We will show that [b1 b2 . . . bn−1 ] can be written as a linear combination of the rows [1..r] of [A]n−1 , the
coefficients of this combination being x> . Take any i ∈ [r + 1..n − 1]. Let us consider the minor obtained by
selecting rows and columns [1..r] ∪ {i, n} of A, see left matrix below. We subtract from the last row the linear
combination of the first r rows defined by x, so as to cancel the first r positions of the last row, followed by
the transposed operation on columns. We obtain the right matrix below.

ai,1 0
   
ai,1 b1
 ai,2 b2   ai,2 0

 [A]r ..

..  row and column operations

 [A]r ..
.
.. 
.


 . . −−−−−−−−−−−−−−−−−−−−−−→ 

,

 ai,r br 
that do not change the determinant 
 ai,r 0 
ai,1 ai,2 ... ai,r ai,i bi  ai, ai,2 ... ai,r ai,i bbi 
b1 b2 ... br bi bn 0 0 ... 0 bbi bbn

15
 P
where bbi = bi − j∈[1..r] xj aij . The determinant of this right matrix can be calculated (see the Leibniz
formula or the Laplace formula for determinants) as follows:

2
bbn · det [A][1..r]∪{i} − bbi · det ([A]r ) ,

where [A][1..r]∪{i} is the (r + 1) × (r + 1) upper left minor of above matrix. This (r + 1) × (r + 1) minor needs
to be null because [A]n−1 has rang r and [1..r] ∪ {i} ⊆ [1..n − 1]. On the other hand we have det([A]r ) > 0,
2
so that the above determinant simplifies to −bbi · det ([A]r ). But this is the determinant of a matrix obtained
from a minor of A after performing linear operations with rows and columns P of A. Since A is SDP, this
determinant needs to be non-negative, and so, we need to have bbi = 0, i.e., bi − j∈[1..r] xj aij = 0, meaning
that bi is a linear combination (defined by x) of the first r rows of column i (recall aij = aji ). Recall i was
chosen at random from [r + 1..n − 1], so that actually all elements bi of [b1 b2 . . . bn−1 ] can be written as a
linear combination (defined by x) of the first r rows.
Corollary 1.6.5. If A  0 and Ajj = 0 for some j ∈ [1..n], then the row and column j contain only zeros.
Corollary 1.6.6. If A  0 and a11 = a12 = a21 = a22 , then ai1 = ai2 ∀i ∈ [1..n].
Proposition 1.6.7. (Cholesky factorization of positive semidefinite matrices) A real symmetric matrix A
is positive semidefinite if and only if it can be factorized as:
  
r11 0 0 ... 0 r11 r21 r31 . . . rn1
 r21 r22 0 . . . 0   0 r22 r32 . . . rn2 
  
A = RR> =  r31 r32 r33 . . . 0  0 0 r33 . . . rn3 
, (1.6.1)
 
 .. .. .. . .. . .
..   ..
  .. .. .. .. 
 . . . . . . . 
rn1 rn2 rn3 ... rnn 0 0 0 ... rnn

where the diagonal terms are non-negative.

The factorization is not always unique. There exists a factorization that has rnn = 0 only if rank(A) =
rank([A]n−1 ), where [A]n−1 is the leading principal minor of size (n − 1) × (n − 1).
Proof.
⇐=
Take any non-zero x ∈ Rn − {0n } and observe that x> Ax = x> RR> x = (R> x)> (R> x) ≥ 0. This is enough
to prove that A is SDP.
=⇒
We proceed by induction. The implication is obviously true for n = 1. We suppose that there exists a
factorization:
  
r11 0 ... 0 r11 r21 . . . rn−1,1
 r21 r22 ... 0   0 r22 . . . rn−1,2 
[A]n−1 = [R]n−1 [R]>
n−1 =  ,
  
.. .. . . .
.   .
. .
. . . ..
 . . . .  . . . . 
rn−1,1 rn−1,2 ... rn−1,n−1 0 0 ... rn−1,n−1

where r11 , r22 , . . . rn−1,n−1 ≥ 0. We will prove that this decomposition can be extended to a n × n
decomposition for matrix A. The values r1n , r2n , . . . rn−1,n (the elements above the diagonal on the last
column of R) are set to zero by definition.
The main difficulty is to determine the last row r> = [ r>
n−1 rn ] of R. We write

 
[A]n−1 bn−1 
A=
b>
n−1 bn

Using Prop. 1.6.4, there exists x ∈ Rn−1 such that b> >
n−1 = x [A]n−1 . This leads to
       
[A]n−1  = In−1  [A]n−1 = In−1  [R]n−1 [R]>
n−1 =
 [R]n−1  [R]>
n−1 .
b>
n−1 x> x> x> [R]n−1

16
Using the notational shorthand r> >
n−1 = x [R]n−1 , we can write:
   
[A]n−1  = [R]n−1  [R]>
n−1 .
b>
n−1 r>
n−1

In fact, we can already fix the first n − 1 positions of the last row (of R) to r> = [ r>
n−1 rn ]. Then, let us

extend the above equality to work with n × n matrices, by adding: (i) a column with zeros to the left factor
of the product and (ii) a row with zeros and a column rn−1 to the right factor. We obtain
  >  
[R]n−1 0 [R]n−1 0 =
[A]n−1 bn−1 
= Az ,
r>
n−1 0 r>
n−1 0 b>
n−1 z

where z = r> n−1 rn−1 . The matrix at right has the same rank as the matrices at left (use Prop. A.1.8),
that is precisely the rank of [R]n−1 which is equal to the rank of [A]n−1 (by applying again Prop. A.1.8 on
>
[A]n−1 = [R]n−1 Rn−1 ). In short, hAz hasi the same rank as [A]n−1 .
A b
Let us now study the minor > of Az obtained by bordering the largest order non-zero minor A
b z
of [A]n−1 with the last column and row of Az (keeping only the positions that constitute the minor). The
determinant of this new minor of Az has to be zero because Az has the same rank as [A]n−1 . We now
compare A to Az ; let us write theh bottom-right
i term bn of A in the form bn = z + α. By replacing z with
A b


z + α, the above minor evolves to b> z+α which is a matrix of determinant α det A . Since A is SDP and
A is a non-zero minor of A, this determinant has to be non-negative, and so, we have α ≥ 0. This enables
us to finish the construction by setting
  >  
[R]n−1 0  [R]n−1 0  = [A]n−1 bn−1 
= A,
√ √
r>
n−1 α r>
n−1 α b>
n−1 z+α

Finally, if rank(A) = rank([A]n−1 ) then we can say rank(A) = rank([A]n−1 ) = rank([R]n−1 ) = rank(R),
2
and so, we need to have rnn = 0 because any rnn > 0 would make α = rnn > 0, √
leading to rank(R) >
rank([R]n−1 ). On the other hand, if rank(A) > rank([A]n−1 ), then we have rnn = α > 0. This justifies
the last statement of the proposition.
Corollary 1.6.8. If a SDP matrix A is not positive definite, the Cholesky factorization may or may no be
unique.
Proof. The following factorization is unique:
    
1 1 1 0 1 1
=
1 1 1 0 0 0

because it requires 1 = r12 and 1 = r1 r2 + r22 , imposing r1 = 1 and r2 = 0.

The following factorization is not unique:
    
1 1 2 1 0 0 1 1 2
1 1 2  = 1 0 0 0 0 0
2 2 13 2 0 3 0 0 3

This is the decomposition that the proof of the above theorem would construct by taking x> = [ 2 0 ] on
the induction basis of the previous 2 × 2 decomposition of [A]n−1 = [ 11 11 ]. Precisely, this proof calculates
[ r31 r32 ] = x> [R]n−1 = [ 2 0 ] [ 11 00 ] = [ 2 0 ]. However, more generally, the coefficients r32√and r33 are active
only in the equation of a33 , i.e., 13 = 22 + r32 2 2
+ r33 . We could take r32 = 1 and r33 = 8. Also, we could
take r32 = 3 and r33 = 0, leading to a different factorization:
    
1 1 2 1 0 0 1 1 2
1 1 2  = 1 0 0 0 0 3
2 2 13 2 3 0 0 0 0

17
Unlike the previous factorization constructed by the proof of the above theorem, this decomposition has
a null bottom-right diagonal term, although rank(A) > rank([A]2 ). However, notice that rank(R) =
rank([R]n−1 ) + 1, where [R]n−1 is the leading principal minor of size 2 × 2. Since rank(RR> ) = rank(R)
(use Prop. A.1.8), the rank of A is equal to the rank of R and the rank of [A]n−1 equals the rank of [R]n−1 .

1.7 Any A 0 has infinitely many factorizations A = V V > related by rotations

and reflections
Corollary 1.7.1. Any SDP matrix A ∈ Rn×n can be factorized in a form A = V V > . Generating such V is
·
simply equivalent to taking n vectors v1> , v2> , . . . vn> (the rows of V ) such that Aij = vi> vj = vi vj ∀i, j ∈
[1..n]. There are infinitely many such matrices V (or vectors v1 , v2 , . . . vn ) for a fixed A. We can say that
any SDP matrix A can be constructed by choosing a set of n vectors of Rn .
Proof. For a given A  0, we have actually already presented three ways of computing a factor V such that
A = V V > . We recall these three ways at points (1)-(3) below. At point (4), we show that from any such
factor V we can further generate infinitely many other factors.
(1) Use the above Cholesky decomposition of SDP matrices to write A = RR> and take V = R as needed.
>
(2) Use the eigendecomposition (1.1.1) to write
√ A = U ΛU √ λ>i ≥
√ ,√where Λ√= diag(λ1 , λ2 , . . . λn ).√ Since
0 ∀i
√ ∈ [1..n],
√ we can define real matrix
√ Λ = diag( λ 1 , λ 2 , . . . λn ) and write A = U Λ ΛU =
(U Λ)(U Λ)> = V V > with V = U Λ.
(3) Use one of the multiple square root decompositions A = KK with symmetric K from Remark 1.6.1.
This gives V = K and V > = K.
(4) We can generate infinitely many more decomposition from any V determined as above. For this,
let us consider any unitary orthogonal matrix R, i.e., a matrix such that R> R = In . There are
infinitely-many such matrices R, each one of them representing a composition of rotation and reflection
>
operators.7 Now check that Vnew Vnew = (V R)(V R)> = V RR> V > = V V > = A.

As a side remark, the factorizations mentioned at above points (3) and (4) are related. In fact, Remark
1.6.1 used at (3)
√ constructs
√ the factorisation by applying a particular case of the technique used at (4) on
A = V V > = U Λ ΛU > . More exactly, recall that symmetric K from Remark 1.6.1 was generated √ by setting
K = U DU > for any symmetric D such that DD = Λ. Such a D could be found by taking D = ΛR, where
R = diag(e1 , e2 , . . . , en ) with ei = ±1 ∀i ∈ [1..n]; this R can actually be √
seen√as a composition
√ of√reflexion
operators like matrix R at point (4) above. We can write A = V V > = U Λ ΛU > = U ΛRR> ΛU > =
U DD> U > = U DDU > = U DU > U DU > = KK.

Remark 1.7.2. Given two factorizations A = V V > = U U > , it is always possible to write V = U R, where
R satisfies RR> = In , i.e., R represents a composition of rotation and reflection operators. The rows of V
can thus be obtained from the rows of U by applying a composition of rotations and reflections.
Proof. Let r be the rank of V , U and A (recall Prop A.1.8) and J a set of rows such that bV cJ has rank
|J| = r, where the operator b·cJ represents the given matrix reduced to rows J. Any row vi of V outside
J (i.e., such that i ∈ [1..n] \ J) can be written as linear combination of rows J using coefficients x ∈ Rr :
vi = x> bV cJ
Let’s examine row i of the product A = V V > = U U > for any fixed i ∈ [1..n] \ J. Replacing above
vi = x> bV cJ in ai = vi V > , we obtain ai = x> bV cJ V > . But now notice that bV cJ V > actually represents
the rows J of matrix A; we can thus replace |V |J V > = bAcJ and obtain ai = x> bAcJ .
We will now prove that ui = x> bU cJ ; we introduce notational shortcuts ui = ui + z = x> bU cJ + z and
we will show z = 0. Let us first calculate ui U > = x> bU cJ U > = xbAcJ = ai . Using A = U U > , we also have
>
ui U > = ai which means that (ui − ui ) U > = zU > = 0> . Taking row i, we obtain zui = 0, or z (ui + z) = 0.
>
We now simply replace ui = x> bU cJ and obtain z x> bU cJ + z = 0 =⇒ zbU c> > >


J x + zz = 0. But zbU cJ
7 Using R R> = In , notice x> y = x> R R> y = (x> R)(y> R)> , i.e., the operator that maps x> → x> R preserves the
angles, and so, it needs to be a composition of rotations and reflections.

18
represents the columns J of zU > = 0> , i.e., zbU c> J = 0
>
is a row vector (of length |J| = r) that only
contains zeros. This leads to zz = 0 and this shows that z = 0, leading to ui = x> bU cJ . We actually
>

obtained that all rows [1..n] − J of the equality A = V V > = U U > represent merely linear combinations of
the rows J of this equality.
This equality is thus a (linear combination) consequence of bAcJ = bV cJ V > = bU cJ U > , where bV cJ
and bU cJ have full rank |J| = r. We now replace all rows [1..n] − J of V with n − r unit orthogonal
vectors rows that are also perpendicular to bV c> J and obtain matrix V ; this is possible because these unit
orthogonal vectors are simply a basis for the null space of bV c> J which has dimension n − r by virtue of the
>
rank-nullity theorem. One can check that the product V V is a matrix A such that aj,j 0 = aj,j 0 ∀j, j 0 ∈ J,
ai,i = 1 ∀i ∈ [1..n] − J and ai,j = aj,i = 0 ∀i ∈ [1..n] − J, j ∈ J. We perform a similar operation on U and
obtain matrix U (filling rows [1..n] − J with a different basis than for V ) and one can check that similarly
we have
> >
UU =V V =A
We can now use that that U , V and A are non-singular and invertible. This ensures that there exists R
> > −1 > −1
such that V = U R. We obtain U U = U RR> U . Multiplying at left with U and at right with U ,
we obtain that RR> = In . Now recall that V = U R contains rows J inherited from V and resp. U , so that
we also have bV cJ = bU cJ R. This can easily be extended to V = U R because each missing row (i.e., each
i ∈ [1..n] − J) is a linear combination of rows J (i.e., vi = xbV cJ = xbU cJ R = ui R).

1.8 Convex functions have an SDP Hessian assuming the Hessian is symmetric
Notice proposition below requires the Hessian matrix to be symmetric. This condition was omitted from cer-
tain texts (see Section 3.1.4 of the book “Convex Optimization” by Stephen Boyd and Lieven Vandenberghe,
p. 71, Cambridge University Press, 2004) but we address it in our work. Convex functions with asymmetric
non-SDP Hessians do exist, see Example C.3.1 in Appendix C. For such cases, the convexity condition should
actually evolve from ∇2 f (y)  0 ∀y ∈ Rn (i.e., SDP Hessian) to ∇2 f (y) + ∇2 f (y)>  0 ∀y ∈ Rn .
Proposition 1.8.1. A twice differentiable function f : Rn → R with a symmetric Hessian matrix ∇2 f (y) =
∂2f
Hy defined by terms hyji =


y for all y ∈ Rn and i, j ∈ [1..n] is convex if and only if Hy  0 ∀y ∈ Rn ,
∂xj ∂xi
i.e., if and only if the Hessian is SDP in all points.
Proof. Let us consider any y ∈ Rn . We take any direction v ∈ Rn and define g : R → R via
g(t) = f (y + tv)
8
Using the chain rule to the gradient, we obtain
n
X ∂f
g 0 (t) = ∇f (y + tv) v = (y + tv)vi (1.8.1)
| {z }
i=1
∂xi
row vector

00 ∂f
We derivate again in t to obtain g (t). For this, we observe that the derivative in t of any term ∂x i
(y+tv) can

·
 0 Pn 2
be calculated as in (1.8.1) using the chain rule, i.e., ∂xi (y + tv) = ∇ ∂xi (y + tv) v = j=1 ∂x∂j ∂x
∂f ∂f f
i
(y +
tv)vj . Summing up over all i ∈ [1..n], we obtain:

g 00 (t) =
n X
X

i=1 j=1
n
∂2f
∂xj ∂xi
(y + tv)vj vi = Hy+tv
· vv>


(1.8.2)

8 If
=⇒ g 00 (0) = Hy
·vv> = v> Hy v

you are unfamiliar with gradients, consider that ∇f (y) is the hyperplane tangent to the function graph at y. Using
notational shortcut ∇f (y) = [ ∇1 ∇2 ...∇n ], the function value (of this hyperplane) increases by ∇i when one performs a step
of length  along direction xi . Let us further study this hyperplane: its value increases by ∇i when one performs a unit step
(of length 1) along direction xi from any starting point. What happens if one moves along some other direction v? Answer:
this is equivalent to advancing a step of v1 along x1 , followed by a step of v2 along x2 , etc, leading to a total increase of
∇1 v1 + ∇2 v2 + . . . ∇n vn = [ ∇1 ∇2 ...∇n ] v. As a side remark, we can also see [ ∇1 ∇2 ...∇n ] as a gradient direction (vector).
The increase (of the hyperplane) when one moves along some direction v is given by the scalar product between v and this
gradient vector, equivalent to the projection of v along this gradient. Among all vectors of the unit sphere, the vector/direction
that makes the hyperplane value change the most (in absolute value) is the one that is collinear to the gradient direction.

19
If f is convex, then g is convex for any direction v ∈ Rn . As such, g(0) = v> Hy v ≥ 0 ∀v ∈ Rn , which
means that Hy is SDP for any chosen y ∈ Rn .
Conversely, if Hy is SDP for any y ∈ Rn , then g is convex for any v ∈ Rn because g 00 (t) ≥ 0 ∀t ∈ R
by virtue of (1.8.2). The convexity definition ensures that αf (y1 ) + βf (y2 ) ≥ f (αy1 + βy2 ) for all α, β ≥
0 such that α + β = 1, i.e., the line joining f (y1 ) and f (y2 ) is above or equal to the function value evaluated
at any point αy1 + βy2 on the segment [y1 , y2 ]. This can be instantiated as follows:

βf (y − αv) + αf (y + βv) ≥ f (y) ∀y, v ∈ Rn , α, β ≥ 0 such that α + β = 1 (1.8.3)

To prove f is convex, we need to show that for any y1 , y2 ∈ Rn , α ≥ 0 and β = 1 − α we have βf (y1 ) +
αf (y2 ) ≥ f (βy1 + αy2 ). But this reduces to (1.8.3): if we fix v = y2 − y1 and replace βy1 + αy2 = y, one
can check y1 = y − αv and y2 = y + βv, i.e., we obtain (1.8.3) for y and v defined above.

2 Primal-Dual SDP programs and optimization considerations

2.1 Primal and dual SDP programs
2.1.1 Main duality
Proposition 2.1.1. The dual of a primal SDP program is an SDP program and the following degeneracy-
related properties hold:
(a) If the primal is unbounded, the dual is infeasible.
(b) If the primal is infeasible, the dual can be unbounded, infeasible or non-degenerate.
We will later see that that if the primal is feasible and bounded (non-degenerate), there might be a duality
gap with regards to the optimal value of the dual, and, even more, the dual can even be infeasible.
Proof. Let us introduce the first SDP program in variables x ∈ Rn , using matrices of an arbitrary order.

Xn

min ci xi (2.1.1a)






 i=1
(SDP ) Xn


 s.t Ai xi  B (2.1.1b)



 i=1
x ∈ Rn

(2.1.1c)

The inequalities (2.1.1b) are often called linear matrix inequalities. Let us now relax them (or penalize their
potential violation) using Lagrangian multipliers Y  0 to obtain the Lagrangian function:

L (x, Y ) =
n
X

i=1
ci xi − Y
· (
X
Ai xi − B)

Observe that if x satisfies (2.1.1b), we get awarded in the above Lagrangian because the Lagrangian term
subtracts the product of two SDP matrices (non-negative by virtue of Prop. 1.3.4). As in linear programming,
we use the convention that OP T (SDP ) = ∞ if (2.1.1b) has no feasible solution (i.e., we have an infeasible
program) and OP T (SDP ) = −∞ if (2.1.1a)-(2.1.1c) can be indefinitely small (i.e., we have an unbounded
program). However, in all cases, minn L (x, Y ) is a relaxation of (SDP) and we can write:
x∈R

min L (x, Y ) ≤ OP T (SDP ), ∀Y  0 (2.1.2)

x∈Rn

We now develop the expression of the Lagrangian:

minn L (x, Y ) = minn Y

x∈R x∈R · B+
n
X
(ci − Y
i=1
· Ai )xi

20
 ·
If there is a single i ∈ [1..n] such that ci − Y Ai 6= 0, the above minimum is −∞ (unbounded), by using an
appropriate value of xi . To have a bounded minx∈Rn L (x, Y ), the matrix Y needs to satisfy ci − Y Ai = 0
for all i ∈ [1..n]. Notice that if we actually consider a non-negative variable xi ≥ 0 for some i ∈ [1..n], the
·
·
condition becomes ci − Y Ai ≥ 0 for such i. We are interested in finding:

max minn L (x, Y )

Y 0 x∈R
Or Ai · Y ≤ c if x ≥ 0 is added to (SDP)
i i

·
that can be written: 
 max B Y (2.1.3a)

·



(DSDP ) s.t Ai Y = ci ∀i ∈ [1..n] (2.1.3b)



Y 0 (2.1.3c)


Based on (2.1.2), we obtain:

OP T (DSDP ) ≤ OP T (SDP ). (2.1.4)
The case of degenerate programs is addressed below, proving points (a) and (b) of the conclusion.

(a) In above (2.1.4), we can say that that OP T (DSDP ) = −∞ if (DSDP ) is not feasible, because
minn L (x, Y ) = −∞ ∀Y  0 in this case. It is clear that if (DSDP ) is feasible, then (SDP ) can not
x∈R
be unbounded from below. Thus, if OP T (SDP ) = −∞ (unbounded from below), then (DSDP ) needs
to be infeasible.
(b) If (SDP) is infeasible, we can infer nothing about the dual, i.e., (DSDP ) can be unbounded, in-
feasible or non-degenerate. If you are familiar with linear programming, then it is well-known that
the dual of an infeasible LP is infeasible or unbounded. By writing such LPs in an SDP form, one
can obtain the examples (i) and (i) below. The most difficult is to find a primal SDP whose dual
is non-degenerate.
Pn Such programs can be found by exploiting a phenomenon of “clenching” in the
development of i=1 Ai xi − B  0 or Y  0, e.g., we can use Corollary 1.6.5 to force certain rows or
columns of an SDP matrix to be zero, pushing it to a certain form (or to infeasibility). For instance,
the SDP matrix in example (iii) below need to have y = 0: the zero at position (2,2) forces the second
row and the second column to contain only zeros by virtue of Corollary 1.6.5. This makes this program
infeasible. On the dual side, we have Y11 = 0 because x has a coefficient of zero in the primal objective
function and Y33 = 1 because Y12 + Y21 + Y33 = 1, which leads to a unique dual feasible solution.
min − y min y
min x min x      

     x 0 0 1 0 0 x y 0 0 0 0
x 0 1 0 x−y 0 1 0
s.t.  s.t.  s.t.  0 −x 0  0 1 0 s.t. y 0 0  0 0 0
0 −x 0 1 0 −x + y 0 1
0 0 y 0 0 0 0 0 y 0 0 1

(i) unbounded dual (ii) infeasible duals (iii) non-degenerate dual

Proposition 2.1.2. Program (DSDP) from (2.1.3a)-(2.1.3c) can be written in the primal form of (SDP)
from (2.1.1a)-(2.1.1c).

·
Proof. We first solve the system Ai Y = ci ∀i ∈ [1..n]. If this system has no solution, then the given (DSDP)
program is infeasible; in this case, any infeasible (SDP) can be considered (by convention) equivalent to the
given infeasible (DSDP) – assuming that both programs have the same optimization (min/max) direction.
The non-degenerate case is the essential one: we consider from now on that the system Ai Y = ci ∀i ∈
[1..n] has at least a feasible solution −B 0 . The set of all solutions is given by
·
k
X
Y = −B 0 + A0j x0j , (2.1.5)
j=1

21
where A01 , A02 , . . . A0k are a basis (maximum set of independent vectors) of the null space of {Ai : i ∈ [1..n]}
(see the null space definition in (A.1.2)).9 The matrices A0j with j ∈ [1..k] and Ai with i ∈ [1..n] satisfy:

Ai
· A0j = 0, ∀i ∈ [1..n], j ∈ [1..k] and − Ai
· B 0 = ci , ∀i ∈ [1..n]. (2.1.6)

The space spanned by (the linear combinations of) Ai (∀i ∈ [1..n]) and A0j (∀j ∈ [1..k]) need to cover the
whole space of symmetric matrices of the size of Y , by virtue of the rank-nullity Theorem A.1.3 that could be
applied on the vectorized versions of these matrices. One can confirm that any feasible Y can be expressed
in the form (2.1.5): just notice that Y + B 0 belongs to the null space of {Ai : i ∈ [1..n]}, and so, it can be
expressed as a linear combination of the basis A01 , A02 , . . . A0k .
Replacing (2.1.5) in (2.1.3a)-(2.1.3c), we obtain:

max −B
· B0 +
k
X

j=1
(B
· A0j )x0j

k
X
s.t. A0j x0j  B 0 (2.1.7)
j=1

x0 ∈ Rk ,

which is an SDP program in the primal form (2.1.1a)-(2.1.1c).

Proposition 2.1.3. If we have xj ≥ 0 for certain variables J ⊆ [1..n] of (2.1.1a)-(2.1.1c), we can still write
an equivalent program without explicit non-negative variables by incorporating the non-negativities in new
rows and columns of (2.1.1b). The dual can be written in the form (2.1.3a)-(2.1.3c), but there is an equivalent
·
dual in which equalities (2.1.3b) become Aj Y ≤ cj for all j ∈ J.
Proof. We define matrices A0i (∀i ∈ [1..n]) and B 0 by bordering Ai and resp. B with |J| rows and columns
that contain only zeros except at the following new positions: A0j contains 1 at a position (mj , mj ) that
correspond to the bordering row and column associated to j ∈ J. We can drop xj ≥ 0 but write an
equivalent program (2.1.1a)-(2.1.1c) with matrices A0i (∀i ∈ [1..n]) and B 0 .
The dual of this program has the form (2.1.3a)-(2.1.3c) and it contains constraints of the form A0i Y 0 = ·
ci ∀i ∈ [1..n]. For i ∈ · ·
/ J, A0i Y 0 = ci is equivalent to Ai Y = ci . For j ∈ J, A0j Y 0 = cj becomes ·
· ·
Ym0 j ,mj + Aj Y = cj , which is equivalent to Ai Y ≤ ci , because Ym0 j ,mj ≥ 0 does not play a role elsewhere
· ·
in the program. The objective function B 0 Y 0 is equivalent to B Y . The initial dual with matrices A0i
(∀i ∈ [1..n]) and B 0 can be equivalently written with matrices Ai (∀i ∈ [1..n]) and B by using inequality
·
constraints Aj Y ≤ cj ∀j ∈ J. Finally, notice that Y 0  0 =⇒ Y  0 because Y is a principal minor of
Y (use the definition from Prop. 1.2.5).
Proposition 2.1.4. (the case of multiple constraints) Suppose one needs to impose multiple constraints in
(2.1.1a)-(2.1.1c):
X n
Aji xji  B j (2.1.8)
i=1

for j ∈ [1..n0 ]. Notices that the involved matrices can have order 1 for some j ∈ [1..n0 ], i.e., (2.1.8) for
such j is a linear constraint. However, all these constraints can be incorporated in a unique constraint of the
form (2.1.1b) expressed with aggregated block-diagonal matrices with n0 blocks. An aggregated dual can be
expressed in the canonical form (2.1.3a)-(2.1.3b) using aggregated block-diagonal matrices. This aggregated
dual is equivalent to a dual in which we have n0 dual matrix variables.
9 Assuming m(m+1)
A1 , A2 , . . . An ∈ Rm×m are linearly independent, we have k = 2
− n. We used the rank-nullity Theo-
m(m+1)
rem A.1.3 with the fact the set of symmetric matrices of size m × m has dimension 2
, using the symmetry constraints.

22
Proof. We define aggregated block-diagonal matrices A0i (∀i ∈ [1..n]) and B 0 with n0 blocks defined by Aji
and resp. B j for all j ∈ [1..n0 ]:
 1   1 
B 0 ... 0 Ai 0 ... 0
 0 B2 . . . 0  0 A2i ... 0 
B 0 =  .. .. ..  and A0i =  .. .. ..  ∀i ∈ [1..n]
   
.. ..
 . . . .   . . . . 
0 0
0 0 ... Bn 0 0 ... Ani

Constraints (2.1.8) are equivalent to a unique constraint in aggregated block-diagonal matrices

n
X
A0i  B 0 .
i=1

We obtain an aggregated program (2.1.1a)-(2.1.1c) expressed with block-diagonal matrices A0i (∀i ∈ [1..n])
and B 0 .
We can now construct the dual of this aggregated primal program. We obtain an aggregated dual program
of the form (2.1.3a)-(2.1.3c) expressed in aggregated variables Y 0  0. The dual objective function is:

· ·
0
n
X
0 0
B Y = Bj Y j, (2.1.9)
j=1

where Y j is obtained from Y 0 by keeping only the rows and columns that correspond to block B j inside B 0
·
(∀j ∈ [1..n0 ]). The dual constraints are A0i Y 0 = ci (∀i ∈ [1..n]) and they can also be written as:

·
0
n
X
Aji Y j = ci ∀i ∈ [1..n]. (2.1.10)
j=1

Notice that the variables yi00 ,j 0 outside the n0 diagonal blocks of Y 0 play no role in the constraints or in the
objective function of the dual. Also, Y 0  0 implies Y j  0 ∀j ∈ [1..n0 ] because all Y j are principal minors
of Y (use the definition from Prop. 1.2.5). The aggregated dual of the form (2.1.3a)-(2.1.3b) in variables Y 0
is equivalent to a dual in variables Y j (with j ∈ [1..n]) using objective (2.1.9) and constraints (2.1.10).

2.1.2 The dual of the dual is the initial program

The remaining of Section 2.1 is devoted to a few properties that may seem a bit boring and easy to trust,
because they only ask to verify certain equivalences between the dual and the primal forms. However, the
exercise of verifying these properties may offer a good insight into the different ways of expressing the same
SDP program and into the different ways of understanding its space of feasible solutions.
We now provide two results on the dual of the dual. The first one uses a new type of Lagrangian duality,
while the second one only uses the first duality from Prop. 2.1.1.
Proposition 2.1.5. By dualizing twice the (SDP ) from (2.1.1a)-(2.1.1c) we obtain the initial (SDP ) and
the following properties hold
(a) If the (DSDP ) from (2.1.3a)-(2.1.3c) is unbounded, its dual (SDP ) is infeasible.
(b) If (DSDP ) is infeasible, (SDP ) can be unbounded, infeasible or non-degenerate.
Proof. Let us calculate the Lagrangian dual of (DSDP) from (2.1.3a)-(2.1.3c) and verify that we obtain the
(SDP) from (2.1.1a)-(2.1.1c). We relax constraints (2.1.3b) using coefficients x0 ∈ Rn :

L (Y, x ) = B
0 0
· Y +
n
X

i=1
(ci − Ai
· Y )x0i . (2.1.11)

·
For any Y that satisfies (2.1.3b), the value of above Lagrangian is B Y , i.e., the objective value (2.1.3a)
of Y in (DSDP ). If no Y  0 satisfies (2.1.3b), we say OP T (DSDP ) = −∞, adopting a similar convention

23
as in linear programming or in Prop. 2.1.1. We also say and OP T (DSDP ) = ∞ if (2.1.3a)-(2.1.3c) can be
indefinitely large (i.e., we have an unbounded program). However, in all cases, we can state:

max L 0 (Y, x0 ) ≥ OP T (DSDP ), ∀x0 ∈ Rn (2.1.12)

Y 0

We now re-write the above Lagrangian:

·
n
! n
X X
max L (Y, x ) = max B −
0 0
Ai x0i Y + ci x0i
Y 0 Y 0
i=1 i=1

n
Ai x0i  0. Notice that Y = 0 leads to
P
We will show that this expression can only be bounded if B −
   n i=1 
· ·
n
Ai x0i Y = 0. We need values x0 such that Ai x0i − B
P P
B− Y ≥ 0 ∀Y  0. Using Prop 1.3.3,
i=1 i=1
n Pn
Ai x0i − B  0. For such x0 , we have max L 0 (Y, x0 ) = 0
P
this can only hold if i=1 ci xi . We can write:
i=1 Y 0

Xn
ci x0i

min





i=1
min max L 0 (Y, x0 ) = X
x0 ∈Rn Y 0 

 s.t Ai x0i  B

x0 ∈ R n ,



which is exactly the (SDP ) from (2.1.1a)-(2.1.1c). Based on (2.1.12), we discover (2.1.4) again:

OP T (DSDP ) ≤ OP T (SDP ). (2.1.14)

We now address points (a) and (b) of the conclusion.

(a) It is clear that if (SDP ) is feasible, then (DSDP ) can not be unbounded. This means that if (DSDP )
unbounded, then (SDP ) is infeasible.
(b) If (DSDP ) is infeasible, we can infer nothing about (SDP ), i.e., (SDP ) could be unbounded, infeasible,
or non-degenerate. One can find examples of unbounded or infeasible duals by generalizing the linear
programming examples. For instance, if if (DSDP ) = max{y : y = −1, y ≥ 0}, then (SDP ) =
min{−x : x ≥ 1} is unbounded. A pair of infeasible primal-dual programs can simply be taken from
example (ii) provided at the end of the proof of Prop 2.1.1. To find an example of an infeasible (DSDP)
with a feasible (SDP), it is enough to take example (iii) at the end of the proof of Prop. 2.1.1 and to
change the right-hand side and the objective function of the (SDP).

· · ·
nh 1 0 0 i h1 0 0i h0 1 0i o
If (DSDP ) = max 0 0 0 Y : 000 Y = 0, 1 0 0 Y = −1, , then we obtain (SDP ) =
n h x1 x2 0 i0 0 0h 1 0 0 io 0 0 0 001

min −x2 : x2 0 0  000 that has solution x2 = 0 (apply Corollary 1.6.5 on the fact that
0 0 x2 000
the middle element is zero) and x1 ≥ 1 of objective value zero.
Proposition 2.1.6. Assuming the dual (DSDP ) from (2.1.3a)-(2.1.3c) is feasible, we can write it into the
primal form (2.1.7) as described by Prop. 2.1.2. If we apply the first duality from Prop. 2.1.1 on this primal
form, we obtain a dual that is equivalent to the primal (SDP) from (2.1.1a)-(2.1.1c).
Proof. The proof relies on a few arguments from the proof of Prop. 2.1.2. First, recall that the feasible dual
program (2.1.3a)-(2.1.3c) can be written in the primal form (2.1.7). We re-write (2.1.7) as follows:

·
k

·
 
X
0 0 0
−B B −  min (−B A )x
i i
i=1
 
 

 Xk .

0 0 0
 s.t
 Ai x i  B  
 i=1 
x0 ∈ R k .

24
Recall that A01 , A02 , . . . A0k and B 0 arise from solving Ai
Pk
form Y = −B 0 + i=1 A0i x0i , where
· Y = c , i.e., any solution Y can be written in the
i

A01 , . . . , A0k are a basis of the null space of A1 , . . . , An and − B 0

We now apply the first duality from Prop. 2.1.1 on the above program in (SDP) form and obtain:
· Ai = ci ∀i ∈ [1..n] (*)

·
−B B 0 − max B 0 Y

· 

 i i

Y  0.
·
 s.t A0 Y = −B A0 ∀i ∈ [1..k]

 ·
This dual could be infeasible even if the corresponding primal is feasible (we have already presented such
examples, see also Prop. 2.2.3). However, the system of linear equations A0i Y = −B A0i ∀i ∈ [1..k] has
at least the feasible solution Y = −B. The existence of this solution is sufficient to place us in the non-
· ·
degenerate case of the proof of Prop. 2.1.2, which leads to re-writing the above program in the primal form.
For this, we consider the origin −B and the basis A1 , A2 , . . . , An that is orthogonal to all A0i , ∀i ∈ [1..k].
Recall (*) that A01 , A02 , . . . , A0k are a basis for the null space of A1 , A2 , . . . An . As such, the space generated by
(the linear combinations of) A1 , A2 , . . . An and A01 , A02 , . . . A0k cover the whole space of symmetric matrices.
n
P
Any Y  0 can be written in the form xi Ai − B. The above program can thus be re-written as:
i=1

· · ·
n
!
X
0 0 0
−B B − max −B B + (B Ai )xi
i=1
n
X
s.t. Ai x i  B
i=1
x ∈ Rn .
Recall now B 0
we obtain:
· A = −c , ∀i ∈ [1..n] from (*). By replacing this! in above program and simplifying −B · B ,
i i
0

n
X
− max −ci xi
i=1
n
X
s.t. Ai x i  B
i=1
x ∈ Rn ,
which is exactly (2.1.1a)-(2.1.1c).

2.1.3 From the primal form to the dual form

The transformation from the primal form to the dual form is more difficult and it is not always possible.
Proposition 2.1.7. The (SDP) program in the primal form (2.1.1a)-(2.1.1c) can be written in the dual form
(2.1.3a)-(2.1.3c), provided that the matrices A1 , A2 , . . . An from (2.1.1b) are linearly independent.
Pn
Proof. We write Y = i=1 Ai xi − B and notice that (2.1.1b) stipulates Y  0. We need to show that the
·
set of such Y can be expressed as the solution set of a system of equations A0i Y = c0i ∀i ∈ [1..k], i.e., of the
form (2.1.3b). However, the most difficult task is to write objective function (2.1.1a) in the form B 0 Y as
in (2.1.3a); for this, we need to express x1 , x2 , . . . xn as linear combinations of variables Y . We will see that
·
x1 , x2 , . . . xn can be exactly determined from Y when A1 , A2 , . . . An are linearly independent.
Consider the following program in the form (2.1.1a)-(2.1.1c) in which A1 and A2 are not linearly inde-
pendent, since A1 = A2 = [ 01 10 ].
min 2x1 + 3x2
 
1 x1 + x2
s.t. 0
x1 + x2 1
x1 , x2 ∈ R

25
  1 x1+x2 
If we try to write Y = x1 +x , then the set of feasible symmetric matrices Y is the set of solutions of
· ·
2 1
the system Y [ 10 00 ] = 1 and Y [ 00 01 ] = 1. However, it is impossible to express the objective function value
in terms of variables Y . What is the objective value of Y = [ 10 01 ] ? It can actually be any real number or
even −∞, depending on the choice of x1 and x2 such that 0 = x1 + x2 .
We hereafter assume that A1 , A2 , . . . An are linearly independent and wePprovide an algorithm for finding
n
a system of equations whose solutions Y can all be written in the form Y = i=1 Ai xi −B. Let us work with
“vectorized” versions of the matrices: the notation X represents a column vector containing the diagonal
and the upper triangular elements of symmetric matrix X. The size of this vector is m = q(q+1) 2 , where q is
Pn Pn
the order of X. We have Y = i=1 Ai xi − B ⇐⇒ Y = i=1 Ai xi − B. Using aggregate matrix A ∈ Rm×n
such that A = [A1 A2 . . . An ], we can write:

Y + B = Ax (2.1.15)

Since A1 , A2 , . . . An are linearly independent, A has rank n; this also ensures m ≥ n. Without loss of
generality, we can consider that the first n rows of A are linearly independent and form a non-null minor
[A]n . The equations corresponding to the first n rows of (2.1.15) can be written [YP ]n + [B]n = [A]n x. As
n
such, we can deduce x = [A]−1 n ([Y ]n + [B]n ). This ensures that the objective value i=1 ci xi from (2.1.1a)
·
can be written as a linear combination of the [Y ]n values, i.e., in the form B 0 Y as in (2.1.3a). We now
replace x in (2.1.15) and obtain:
Y + B = A[A]−1
n ([Y ]n + [B]n ). (2.1.16)
The first n rows of above formula are redundant. The remaining k = m − n rows actually represent a
system of linear equations: notice that each element of the left-hand Y is expressed as a linear combination
of the free variables [Y ]n (plus a fixed coefficient). Now check than any such linear equation in variables Y
·
can be rewritten in the form A0i Y = c0i (for any i ∈ [1..k]).
·
We have just shown that any solution Y of (2.1.15) satisfies a system of equations of the form A0i Y = c0i
(with i ∈ [1..k]). We now prove the converse: any solution of this system can be written in the form (2.1.15)
for a certain x ∈ Rn ; more exactly, this x is the unique solution of [Y ]n = [A]n x − [B]n . To show this,
it is enough to see that any solution of the above system satisfies (2.1.16) by construction and that, after
replacing [Y ]n = [A]n x − [B]n , (2.1.16) becomes

Y + B = A[A]−1
n ([Y ]n + [B]n )
= A[A]−1
n ([A]n x − [B]n + [B]n )
= A[A]−1
n ([A]n x)
= Ax
By “devectorizing” Y , B, A1 , A2 , . . . An into symmetric matrices Y, B, A1 , A2 , . . . An (put
Pnthe correspond-
ing elements on the diagonal and resp. on the symmetric matrix positions), we obtain Y = i=1 Ai xi −B.
Example 2.1.8. We apply the above proof of Prop. 2.1.7 on an example showing how to rewrite a program
(2.1.1a)-(2.1.1c) in the form of (2.1.3a)-(2.1.3c). Consider

min x1 + 2x2 + x3 + 2x4

 
0 x1 + x3 x4
s.t. x1 + x3 x2 + x3 + 1 x4 + 2x2   0 (2.1.17)
x4 x4 + 2x2 x1 + x2 + 2
x1 , x 2 , x 3 , x 4 ∈ R

We write a formula corresponding to (2.1.15) but without “vectorization”:

   
y11 y12 y13 0 x1 + x3 x4
y12 y22 − 1 y23  = x1 + x3 x2 + x3 x4 + 2x2  (2.1.18)
y13 y23 y33 − 2 x4 x4 + 2x2 x1 + x2

As in paragraph below (2.1.15), we will express the four variables x1 , x2 , x3 and x4 in terms of four variables
Y . We can not choose Y11 , because the corresponding row of (2.1.15) has only zeros in the matrix A. Let us

26
choose: y12 = x1 + x3 , y22 − 1 = x2 + x3 , y33 − 2 = x1 + x2 and y13 = x4 . We obtain
y12 + y33 − y22 − 1 y22 + y33 − y12 − 3 y12 + y22 − y33 + 1
x1 = , x2 = , x3 = and x4 = y13 . (2.1.19)
2 2 2
Our objective function can be written:
y12 + y33 − y22 − 1 y22 + y33 − y12 − 3 y12 + y22 − y33 + 1 3
+2 + + 2y13 = y22 + y33 + 2y13 − . (2.1.20)
2 2 2 2
We now replace the values of x1 , x2 , x3 , x4 in the remaining equations of (2.1.18). We have y11 = 0 and
y23 = y13 + y22 + y33 − y12 − 3, or 3 = y13 + y22 + y33 − y12 − y23 . Combining these two equations with
(2.1.20), we obtain the program:
h0 0 1i
min 0 1 0 Y −
3
2·
·
1 0 1
h1 0 0i
0 0 0 Y =0
0 0 0
h 0 − 21 21
− 12 1 − 21
1
2 −2
1
1
i
· Y =3

Y 0
One can check that any feasible solution Y of above program can be written as in (2.1.17). For this, it
is enough to determine variables x1 , x2 , x3 , x4 from Y using (2.1.19) and check that all positions of Y
and the objective function from above program can be written using variables x which leads to (2.1.17). For
instance, to check that y23 can be written as x4 + 2x2 with x2 and x4 determined by (2.1.19), we write:
x4 + 2x2 = y13 + y22 + y33 − y12 − 3 = y23 , where we used the second constraint on Y for the last equality.
We used several times in this section the transformation from the dual form into the primal form. All
·
these transformations rely on solving the system Ai Y = ci , with i ∈ [1..n]. You might have noticed there
are several ways to solve this system. Just to give an example, consider that y1 + y2 = 10 can be solved in
two manners: (a) take y1 = x1 as a free variable and all solutions have the form [y1 y2 ] = [x1 x1 − 10] or
(b) take solution [5 5] and all solutions take the form [5 5] + x1 [−1 1]. We can say that the approach (a)
was used in the Example 2.1.8 above, while the approach (b) was used in Prop 2.1.2. They are both related,
because switching from (a) to (b) is equivalent to performing a change of variable in the resulting primal,
0
e.g., take (2.1.1a)-(2.1.1c) and use variables xP
i = xi + 7 instead ofPxi ∀i ∈ [1..n]. Such a change leads to a
n n
different primal form, in which the objective i=1 ci xi evolves to i=1 ci x0i − 7n.

2.2 Relations between the primal optimum and the dual optimum
Proposition 2.2.1. (Complementary Slackness) If x and Y are the optimum primal and resp. dual solutions
of SDP and resp. DSDP (in (2.1.1a)-(2.1.1c) and (2.1.3a)-(2.1.3c) resp.), the duality gap can be written:

·
n
!
X
OP T (SDP ) − OP T (DSDP ) = Y Ai x i − B (2.2.1)
i=1

There is no strict complementarity as in linear programming, i.e., the matrices of the above product might
share an eigenvector whose eigenvalue is zero in both matrices for any optimal x and Y .
Proof. It is enough to develop

OP T (SDP ) − OP T (DSDP ) =
n
X

i=1
ci xi −B
· Y

=
Xn 

i=1
Ai
· 
Y xi − B
· Y

· n
!
X
=Y Ai x i − B
i=1

When strong duality holds (i.e., when OP T (SDP ) − OP T (DSDP ) = 0), we can observe the following using
the eigen-decomposition (1.1.1):

27
 Pn
(a) the eigenvectors of ( i=1 Ai xi − B) with non-zero eigenvalues belong to the space generated by the
eigenvectors of Y with eigenvalue 0.
(b) the
Pneigenvectors of Y with non-zero eigenvalues belong to the space generated by the eigenvectors of
( i=1 Ai xi − B) with eigenvalue 0.
Pn
In intuitive terms, we can say that any eigenvector of ( i=1 Ai xi − B) with a non-zero eigenvalue
Pn can be seen
as an eigenvector of Y with eigenvalue 0 and vice-versa.
h i Notice
h we
i did not claim that rank ( i=1 i xi − B) =
A
100 000
nullity(Y ). Take for instance the matrices 0 0 0 and 0 0 0 . They satisfy above conditions (a) and (b),
000 001
but the first one has rank 1 and the second one has nullity
Pn 2. However, if there exist multiple primal and
dual optimal solutions, do some of them satisfy rank ( i=1 Ai xi − B) = nullity(Y
Pn )?
This is called the strict complementarity property: any eigenvector of ( i=1 Ai xi − B) with a zero
eigenvalue is an eigenvector of Y with non-zero eigenvalue. In linear programming, a theorem of Goldman
and Tucker10 states that there always exist primal-dual solutions x and y that are strictly complementary,
i.e., every zero of Ax − b corresponds
Pnto non-zero of y. In SDP programming, this strict complementarity
property no longer holds. Matrices ( i=1 Ai xi ) and Y can share an eigenvector whose eigenvalue is zero in
both matrices. Consider the following primal-dual programs, both expressed in the form of (2.1.1a)-(2.1.1c);
the dual was transformed as described in Prop. 2.1.2 (see below).

min x3 max 2y44

   
x1 x2 0 0 0 0 0 0
x2 0 0 0  0
0 a2 + ∆2 0 a
s.t. 
0 s.t. Y =  
0 x2 0  0 0 0 0
0 0 0 x3 − 2 0 a 0 1
x1 , x2 , x3 ∈ R Y 0

The optimum primal solution x satisfies x2 = 0 (row  x2 and column 2 need to be zero, because position
11 0 0 0
n

P
(2,2) is zero) and x3 = 2. The matrix Ai xi − B is 00 00 00 00 which has at maximum rank 1.
i=1 0 000
Any feasible Z satisfies z11 = 0 because the coefficient of x1 is zero in the primal objective function. This
forces row 1 and column 1 of Z to have only zeros. The dual constraint corresponding to x2 is 2z12 + z33 = 0;
since z12 = 0, we have z13 = 0. The dual constraint corresponding to x3 imposes z4 = 1. There is no
constraint on z24 = a; z22 needs to be greater than or equal to a2 so as to have a non-negative principal
minor corresponding to rows/columns 2 and 4; we can write z22 = a2 + ∆2 .
On the dual side, any feasible Y satisfies y11 = 0 because the coefficient of x1 is zero in the primal objective
function. This forces row 1 and column 1 of Y to have only zeros. The dual constraint corresponding to x2
is 2y12 + y33 = 0; since y12 = 0, we have y13 = 0. The dual constraint corresponding to x3 imposes y4 = 1.
There is no constraint on y24 = a; y22 needs to be greater than or equal to a2 so as to have a non-negative
principal minor corresponding to rows/columns 2 and 4; we can write y22 = a2 + ∆2 . The nullity of any
>
feasible Y is at least 2. Both matrices have eigenvector [ 0 0 1 0 ] with eigenvalue 0.
Proposition 2.2.2. (Non-zero duality gap) If x and Y are the optimum primal and resp. dual solutions
of SDP and resp. DSDP (in (2.1.1a)-(2.1.1c) and (2.1.3a)-(2.1.3c) resp.), the duality gap OP T (SDP ) −
OP T (DSDP ) is not necessarily 0.

Proof. Consider the following primal-dual programs, where the dual is actually written in the primal form,
as obtained after applying the transformation from Prop. 2.1.2.

10 I first learned of this theorem from the article “Semidefinite programming” by M. Overton and H. Wolkowicz, a foreword

for a special issue on SDP of Mathematical Programming (Volume 77, Issue 1, 1997).

28
 min x3 max − 2y34
   
x1 x2 0 0 0 0 0 0
x2 0 0 0  0
0 a2 + ∆2 0 a
s.t.  s.t. Y =  
0 0 x2 1 − x3  0 0 0 0
0 0 1 − x3 x3 0 a 0 1
x1 , x2 , x3 ∈ R Y 0

The optimum primal solution satisfies x2 = 0 (row 2 and column 2 need to be zero using Corollary 1.6.5,
because position (2,2) is zero) and x3 = 1 (row 3 and column 3 need to be zero because position (3,3) is
x2 = 0). The primal optimum solution is 1.
On the dual side, y11 needs to be zero because the objective coefficient of x1 is zero. As such, all elements
on row 1 and column 1 of Y need to be zero. The constraint corresponding to x2 stipulates 2y12 + y33 = 0.
Since y12 = 0, we need to have y33 = 0. This means that the row 3 and column 3 contain only zeros, and
so, y34 = 0, i.e., the dual optimum objective value is 0. Finally, let us fill the remaining elements of the
optimal Y and check its feasibility. The dual constraint corresponding to x3 imposes y4 = 1. There is no
constraint on y24 = a; y22 needs to be greater than or equal to a2 so as to have a non-negative principal
minor corresponding to rows/columns 2 and 4.
Proposition 2.2.3. The dual DSDP (2.1.3a)-(2.1.3c) of a feasible SDP (2.1.1a)-(2.1.1c) is not necessarily
feasible.
Proof. We have actually already shown an example that shows this at point (b) in the proof of Prop 2.1.5.
Let us modify a bit this example for the sake of diversity, by adding a variable x3 . Consider the following
primal program that has at least the feasible solution x1 = x2 = x3 = 0.
min x3 − x2
 
x1 x2 x3
s.t. x2 0 00
x3 0 x2
x1 , x2 , x3 ∈ R
The dual of this program is infeasible. Since x1 has a null objective function coefficient, y11 needs to be
zero. This means (Corollary 1.6.5) that all elements on row 1 and column 1 of Y need to be zero. We thus
obtain y13 = 0. On the other hand, the dual constraint corresponding to x3 stipulates that y13 = 21 , which
is a contradiction. Finally, notice −x2 is not necessary in the primal objective value.

2.3 Strong duality

Several results from this section (including the final proof of the strong duality) are taken from a course of
Anupam Gupta, also using arguments from the lecture notes of László Lovász.11

2.3.1 Basic facts on the cone of SDP matrices and the cone of definite positive matrices
Proposition 2.3.1. The SDP matrices of size n×n form a closed convex cone Sn+ (see also Def. 7.2.1). The
set of positive definite matrices form an open cone interior(Sn+ ), see also the interior Definition 7.2.3.
The closure of interior(Sn+ ) is Sn+ .
Proof. Most statements follow by applying the definitions. First, it is easy to prove that any X  0 for
which there is a non-zero v ∈ Rn such that v> Xv = 0 does not belong to the interior of Sn+ . We only
need to show there is no open ball centered at X which is completely contained in Sn+ (see also the interior
Definition 7.2.3). This follows from the fact that Sn+ contains p
no element from the set {X − εI : ε > 0}
because v> (X − εI)v = 0 − ε|v|2 < 0, where |v| is the 2-norm v12 + v22 + · · · + vn2 of v.
On the other hand, a matrix X  0 does belong to the interior interior(Sn+ ) as it does contain an
open ball centered at X. It is not hard to show there exists a sufficiently small ε > 0 such that X + εY
remains positive definite for any symmetric matrix of bounded 2-norm. Take any v ∈ Rn such that |v| = 1.
11 As of 2017, they are available, respectively at [Link]

notes/[Link] and [Link]

29
Let λ1 > 0 be the minimum eigenvalue of X so that v> Xv ≥ λ1 using Lemma [Link]. We now develop
v> (X + εY ) v = v> Xv + εv> Y v ≥ λ1 + εv> Y v. Since both v and Y have bounded norm, v> Y v is
bounded; this way, there exists a sufficiently small ε > 0 that makes εv> Y v ridiculously small compared to
λ1 . This shows that v> (X + εY ) v > 0 for any v of norm 1, equivalent to X + εY  0.
It is easy to verify that both Sn+ and interior(Sn+ ) are convex cones. If X  0 (resp. X  0), for any
α > 0 we have αX  0 (resp. αX  0) because v> (αX)v = αv> Xv ≥ 0 (resp. > 0) for any v ∈ Rn − {0}.
One can also confirm the convexity: for any α ∈ (0, 1) and X, Y  0 (resp. X, Y  0), we have that
Zα = αX + (1 − α)Y verifies Zα  0 (resp. Zα  0), because v> Zα v = αv> Xv + (1 − α)v> Y v ≥ 0
(resp. > 0) for any v ∈ Rn − {0}.
To prove that Sn+ is closed, we need to show Sn+ contains all its limit points. Assume the contrary: there
is a sequence {Xi } with Xi  0 ∀i ∈ N∗ such that lim Xi = Z  0. This means there exists a non-zero rank
i→∞
1 matrix V = vv> such that Z · V = −a < 0. As such, lim X · V = Z · V = −a < 0. The convergence
i

definition states that for any ε > 0 there exists an m ∈ N such that X · V ∈ [−a − ε, −a + ε] for any i ≥ m.
i→∞
i
Taking an ε < a, we obtain that X · V < 0 which means X  0, contradiction. All limit points lim X
m m i
i→∞
have to belong to Sn+ .
We still need to prove that the closure of interior(Sn+ ) is Sn+ . Since all limit points of Sn+ belong to Sn+
(above paragraph), all limit points of interior(Sn+ ) ⊂ Sn+ have to belong to Sn+ as well. We only need to
prove that any X  0 is a limit point of a sequence {Xi } such that Xi  0 ∀i ∈ N∗ . It is enough to take
Xi = X+ 1i In and one can check that Xi = X+ 1i In  0 because v> (X+ 1i In )v = v> Xv+ 1i v> Iv ≥ 1i |v|2 > 0
1
for all v ∈ Rn − {0}. The convergence to X is easy to prove using lim In = 0.
i→∞ i

2.3.2 The proof of the strong duality

We need the following proposition.
n
xi Ai − B for any x ∈ Rn , where all matrices B and Ai (with i ∈ [1..n])
P
Proposition 2.3.2. Let F (x) =
i=1
are symmetric.
F (x)  0 ∀x ∈ Rn ⇐⇒

∃Y  0, Y 6= 0 such that Ai
· Y = 0 ∀i ∈ [1..n] and F (x)
· Y = −B
· Y ≤ 0, ∀x ∈ Rn .

In other words, if the sub-space generated by A1 , A2 , . . . An with basis −B does not touch interior(Sn+ ),
(2.3.1)

· ·
then this sub-space belongs a hyperplane {X : X Y = −B Y, X symmetric}.
Proof.
⇐=
· ·
If F (x) Y ≤ 0, F (x) can not be positive definite because any Z  0 verifies Z Y > 0 for non-zero Y  0.
To check this, use the eigenvalue decomposition (i.e., (B.2.3) of Proposition B.2.1) to write Y as a sum of
·
non-zero rank 1 matrices of the form vv> . We have Z vv> > 0 by Definition 1.2.1.
=⇒
+
The interior interior(Sm ) of the SDP cone does not intersect the image of F . We can apply the hyperplane
separation Theorem C.4.1: there exists a non-zero symmetric Y ∈ Rm×m and a c ∈ R such that

F (x)
· Y ≤c≤X
· Y ∀x ∈ Rn , ∀X ∈ interior(Sm
+
) (2.3.2)

·
It is clear that we can not have c > 0 because X Y can be arbitrarily close to 0 by choosing X = εIm
·
for an arbitrarily small ε > 0. We now prove X Y ≥ 0 ∀X ∈ interior(Sm +
). Let us assume the contrary:
+
· 0
∃X ∈ interior(Sm ) such that X Y = c < 0. By the cone property of interior(Sm +
), we have tX ∈
+
· 0
interior(Sm ) ∀t > 0. The value (tX) Y = tc can be arbitrarily low by choosing an arbitrarily large t, and
· ·
so, (tX) Y can be easily less than c, contradiction. This means that X Y ≥ 0 for all X ∈ interior(Sm
Based on (2.3.2) and on the fact that c ≤ 0, we obtain:
+
).

F (x)
· Y ≤0≤X
· Y ∀x ∈ Rn , ∀X ∈ interior(Sm
+
)

30
We prove that Y  0. For this, we show that X Y ≥ 0 ∀X ∈ Sm · +
. Assume the contrary: there is
+
·
some X ∈ Sm such that X Y < 0. For any ε > 0, we have X + εIm ∈ interior(Sm +
). For a small
·
enough ε, (X + εIm ) Y remains strictly negative, which contradicts X + εIm ∈ interior(Sm+
). We obtain
·
X Y ≥ 0 ∀X  0. Using the fact that the SDP cone is self-dual (Prop 1.3.3), we obtain Y  0. We have
just found an SDP matrix Y such that

F (x)
· Y ≤ 0 ∀x ∈ Rn . (2.3.3)

We still need to show that F (x) is constant for every x ∈ R, which is equivalent to Ai Y = 0 ∀i ∈ [1..n].
This is not difficult. If there is a single x ∈ Rn such that F (x) = F (0) + ∆ with ∆ 6= 0, notice F (tx) can
·
become arbitrarily large using an appropriate value of t (i.e., use t → ∞ for ∆ > 0 or t → −∞ otherwise),
which is impossible. We thus need to have:

F (x)
·
where we used (2.3.3) for the last inequality.
Y = F (0)
· Y = −B
· Y ≤ 0 ∀x ∈ Rn ,

For the reader’s convenience, we repeat the definitions of the SDP program (2.1.1a)-(2.1.1c) and resp. of
its dual (2.1.3a)-(2.1.3c):
( n n
)
X X
n
(SDP ) min ci xi : Ai xi  B, x ∈ R , (2.3.4)
i=1 i=1

and n
(DSDP ) max B
· · Y : Ai
o
Y = ci ∀i ∈ [1..n], Y  0 .

Theorem 2.3.3. If the primal (SDP ) from (2.3.4) is bounded and has a strictly feasible solution (Slater’s
(2.3.5)

interiority condition), then the primal and the dual optimal values are the same and the dual (DSDP )
from (2.3.5) reaches this optimum value. Recall (Prop. 2.1.1) that if (SDP ) is unbounded, then (DSDP ) is
infeasible.
n
P Pn
Proof. Let p be the optimal primal value. The system ci xi < p and i=1 Ai xi  B has no solution. We
i=1
define
0> 0>
   
−ci −p
A0i = n
∀i ∈ [1..n] and B = 0 n
0n Ai 0n B
n n
A0i xi − B 0  0 ∀x ∈ Rn (we can not say A0i xi − B 0  0, as the optimal solution x can
P P
and observe that
i=1 i=1
cancel the top-left term of the expression). We can thus apply Prop. 2.3.2 (implication “=⇒”) and conclude

· ·
there is some non-zero Y 0  0 such that A0i Y 0 = 0 ∀i ∈ [1..n] and −B 0 Y 0 ≤ 0. Writing Y 0 = ..
t ...

.Y
, we
obtain:
·
tci = Ai Y, ∀i ∈ [1..n] (2.3.6)
and
−B
· Y ≤ −tp.

·
We now prove t > 0 by contradiction. Supposing t = 0, we obtain Ai Y = 0 ∀i ∈ [1..n] and −B Y ≤ 0. ·
(2.3.7)

n
Ai xi − B  0 ∀x ∈ Rn , which contradicts
P
Applying again Prop. 2.3.2 (implication “⇐=”), we conclude
i=1
the fact the primal (2.3.4) is strictly feasible. We need to have t > 0.
·
Taking SDP matrix Y = 1t Y , (2.3.6)-(2.3.7) become: ci = Ai Y ∀i ∈ [1..n] and B Y ≥ p. In other ·
words, Y is a feasible solution in the dual (2.3.5) and it has an objective value B Y ≥ p. Using the weak ·
· ·
duality (2.1.4), B Y ≤ p, and so, B Y = p, i.e., the dual achieves the optimum primal value.
Theorem 2.3.4. If the dual (DSDP ) from (2.3.5) is bounded and has a strictly feasible solution, then the
primal and the dual optimal values are the same and the primal (SDP ) from (2.3.4) reaches this optimum
value. Recall (Prop. 2.1.5) that if (DSDP ) is unbounded, then (SDP ) is infeasible.

31
Proof. Apply Theorem 2.3.3 and Prop. 2.1.6. The main idea is to write the dual (DSDP ) from (2.3.5) in
the primal form (this is possible using Prop. 2.1.2). Theorem 2.3.3 states that the dual of this primal form
reaches the optimum solution. But the dual of this primal form is exactly equivalent to the primal (SDP )
from (2.3.4) by virtue of Prop. 2.1.6.
Theorem 2.3.5. If both (SDP ) and (DSDP ) are strictly feasible, then OP T (SDP ) = OP T (DSDP ) and
this value is reached by both programs.
Proof. Using Prop. 2.1.1, if (DSDP ) has a feasible solution, then (SDP ) is bounded. Using Prop. 2.1.5, if
(SDP ) has a feasible solution, (DSDP ) is bounded. We can now apply Theorems 2.3.3 and 2.3.4 to obtain
the desired result.

2.3.3 Further properties on the intersection of the SDP cone with a sub-space
n
P
Proposition 2.3.6. Let F (x) = xi Ai − B. If the image of F (i.e., the space spanned by Ai with i ∈ [1..n]
i=1
+
and basis −B) does not intersect the SDP cone Sm
n
0 ∀x ∈ R .
, then there exists Y  0 such that F (x) Y = −B · ·Y <
Proof.
⇐=
· ·
We know that Y  0 satisfies F (x) Y < 0 for any x ∈ Rn . Since all X  0 verify X Y ≥ 0, F (x) can not
belong to the SDP cone.
=⇒ Pn
There exists a sufficiently small ε such that (the image of) Fε (x) = i=1 xi Ai − B + εIm still does not
+
intersect the SDP cone. This means that this image does not intersect interior(Sm ) either and we can
· ·
apply Prop 2.3.2 on Fε . There exists a non-zero Y  0 such that Fε (x) Y = (−B + εIm ) Y ≤ 0 ∀x ∈ Rn .
·
It is now enough to check that for any x ∈ Rn the following value is constant: F (x) Y = Fε (x) Y − ·
· ·
εIm Y < 0; we used Im Y > 0 which follows from trace(Y ) > 0 (or apply Prop. 1.3.4).
n
xi Ai − B such that ∃x0 ∈ Rn such that F (x0 ) = 0. If the image of F
P
Proposition 2.3.7. Let F (x) =
i=1
+
intersects the SDP cone Sm ·
only in the origin 0, then there exists Y  0 such that F (x) Y = 0 ∀x ∈ Rn .
Proof.
⇐=
·
We know that Y  0 satisfies F (x) Y = 0 for any x ∈ Rn . It is not hard to prove that all non-zero X ∈ Sm +

· ·
satisfy X Y > 0, because Prop. 1.3.4 states that X Y = 0 =⇒ XY = 0, which leads to X = 0Y
(because Y  0 is non-singular). Thus F (x) can not cover any non-zero SDP matrix.
−1
=0

=⇒
∗ + ∗
Let Sm = {X ∈ Sm : trace(X) = 1}. It is not hard to check that Sm is convex, closed and bounded. The
image img(F ) of F is a closed convex set.
∗ ∗
We show that Sm − img(F ) = {X a − X b : X a ∈ Sm , X b ∈ img(F )} is closed. We take any convergent
a b ∗
sequence {Xi − Xi } and we will show the limit point belongs to Sm − img(F ). It is not hard to see that any
a ∗ a
X ∈ Sm is bounded in the sense that it satisfies Xij ≤ 1 ∀i, j ∈ [1..m], because the 2×2 minor corresponding
to rows/columns i and j has to be non-negative. The sequence {Xib } (with i → ∞) needs to be bounded,
because otherwise {Xia − Xib } would be unbounded, and so, non-convergent. Using the Bolzano-Weierstrass
theorem (Theorem C.4.9), the bounded sequence {Xia } has a convergent sub-sequence {Xnai }. Using the
Bolzano-Weierstrass theorem again, the sub-sequence {Xnb i } contains a convergent sub-sub-sequence {Xm b
i
},
∗ a a ∗ b b
with {mi } ⊆ {ni }. Since Sm and img(F ) are closed, lim Xmi = X ∈ Sm and lim Xmi = X ∈ img(F ),
i→∞ i→∞
a b ∗ ∗
and so, lim Xm i
− Xm i
= X a − X b ∈ Sm − img(F ), i.e., Sm − img(F ) contains all its limit points.
i→∞
∗
Since 0 ∈
/ Sm − img(F ), the simple separation Theorem C.4.5 states there is an Y such that (X a −
b
· a
X ) Y > 0, ∀X ∈ Sm ∗
, X b ∈ img(F ). This is equivalent to

Xa
· Y > Xb
· ∗
Y, ∀X a ∈ Sm , X b ∈ img(F ). (2.3.8)

· ∗
Since 0 ∈ img(F ), we have X a Y > 0 ∀X a ∈ Sm . This is equivalent to X a · Y > 0 ∀X ∈ S +
m \ {0}, and so,
· > n
Y (vv ) > 0 ∀v ∈ R − {0}, i.e., Y  0.

32
 ·
We still need to show that F (x) Y = 0 ∀x ∈ Rn . Assume there exists x ∈ Rn such that F (x) Y =
∆ 6= 0. Recall from hypothesis that there exists x0 ∈ Rn such that F (x0 ) = 0. We can write x = x0 + x1
·
·
and we obtain F (x0 + x1 ) = d. For any α ∈ R, we have F (x0 + αx1 ) Y = α∆, and so, F (x0 + αx1 ) Y
can be arbitrarily large, violating (2.3.8).
·
2.4 The difficulty of exactly solving (SDP) and algorithmic comments
√
We first notice that max y12 : y121 y212  0 is 2. This means it is rather unlikely that a purely numerical
  

algorithm based on binary encodings can exactly optimize any SDP program. It is however possible to find
the optimum value of any SDP program up to any specified additive error (precision) using ellipsoid, interior
point, spectral or conic bundle methods.
We now show that exactly solving SDP is at least as hard as the square-root sum problem whose exact
complexity is still an open problem. Consider the following feasibility program in variables x2 , x3 , . . . xn .12
 
1 k − (x2 + x3 + · · · + xn )
k − (x2 + x3 + · · · + xn ) a1 
 

 1 x 2



 x 2 a2



 1 x 3   0.


 x 3 a3



 . .. 

 
 1 xn 
xn an
Pn √
This program is feasible if and Pnonly if k ≤ i=1 ai . This is the square-root sum problem: given k and
√
a1 , a2 , . . . an , decide if k ≤ i=1 ai . It is still an open question if this problem is polynomial or not,
using at least the on-line forum [Link]
sum. However, the same link reads that the SDP optimum can be determined (approximated) with any
specified additive accuracy (error) ε using the interior point method or the ellipsoid algorithm in time
polynomial in the size of the instance and log 1/ε.
To show the difficulty of exactly solving SDP programs, we consider the following minimization problem.
min x
 
x 1 3
1 x+2 0 0
3 0 x+1

The determinant of the whole matrix is f (x) = x3 +3x2 −8x−19. We first present an intuitive figure (graph)
of f on the right13 and then we will formally determine the
values of x ≥ 0 such that f (x) ≥ 0. We are not interested
in values x < 0 because the leading principal minor of size 1
0 x∗
would be x < 0. The optimum of above program is clearly
greater than 0.
By calculating the second derivative f 00 (x) = 6x + 6, it
−19
is clear that f is strictly convex over [0, ∞). Since f 0 (0) =
−8, the function first decreases when going from 0 to ∞
and, after a point, it increases. This means f has a unique
root x∗ ∈ [0, ∞) and the optimum of above program is at Figure 1: The graph of f (x) = x3 +3x2 −8x−19
least x∗ . Before determining this root, notice the figure
intuitively shows that x∗ > 2.6. We can also formally check that f (2.6) = −1.944, and, by convexity, we
surely have x∗ > 2.6. Other principal minors do not pose any problem because they are surely non-negative
for any x∗ > 2.6. The most problematic one is the one associated to rows and columns 1 and 3. Since
2.6 ∗ 3.6 = 9.36 > 9, this principal minor is positive for x = 2.6 and it remains so by increasing x.
12 See the article “Semidefinite programming and combinatorial optimization” by Michel Goemans, in the International
Congress of Mathematicians, Volume III, Documenta Mathematica, Extra Vol. ICM III, 1998, 657-666, page 2. The article is
available on-line as of 2017 at [Link]
13 Obtained using [Link]

33
 This means that the optimum value of above program is equal to the largest root x∗ of the cubic equation
f (x) = 0. To determine this root, let us write f (x) = (x + 1)3 − 11(x + 1) − 9. Using Cardano’s formula,14
we obtain: s s
r r
∗ 3 9 3137 3 9 3137
x = + i+ − i − 1 ≈ 2.6679286.
2 108 2 108
It is highly unlikely that an SDP algorithm could exactly determine such values in polynomial time, solving
cubic equations like f (x) = 0 or possibly other higher degree equations. On the other hand, the numerical
methods mentioned above can determine the optimum value up to any specified additive error.

3 Interesting SDP programs

3.1 An SDP program does not always reach its min (inf ) or max (sup) value
The value    
t 1
min t : 0
1 t0
is zero but no feasible solution reaches this optimum objective value of zero. However, we could use “min”
to actually mean “inf”. Since the program is bounded and strictly feasible (e.g., for t = t0 = 2, we have
[ 21 12 ]  0) we can use the strong duality (Theorem 2.3.3) to conclude that the dual does achieve the optimum
0. One can easily check that the dual has only one feasible point [ 10 00 ] with objective value 0.
An example of a program in the dual form (2.1.3a)-(2.1.3c) can be found by re-formulating the above
primal program using the method from Section 2.1.3; aiming at a maximization form, one may find:

·
   
0 1
max −Y11 : Y = 2, Y  0 .
1 0

3.2 The lowest and greatest eigenvalue using the SDP duality
Consider symmetric matrix X ∈ Rm×m with eigenvalues λ1 ≤ λ2 ≤ · · · ≤ λn . The following program reports
the smallest eigenvalue.
max t


λ1 = s.t X − tIm  0

t∈R
Observe this program is bounded and strictly feasible (take t → −∞), and so, the strong duality Theo-
rem 2.3.3 states that the dual reaches the optimum value λ1 . By dualizing after transforming “max t” into
·
“− min −t”, we obtain an objective “− max −X Y ”, that is equivalent to “min X Y ”. ·
λ1 =

min X Y


 s.t trace(Y ) = 1
·

Y 0


We have obtained a more general version of Lemma [Link] in which the rank of Y is not necessarily 1 (as in
 of −X is −λn . The first above program for −X can be
Lemma [Link]). Furthermore, the lowest eigenvalue
written, after replacing t with −t, as −λn = max − t : tIm − X  0, t ∈ R . As such, λn can be obtained
by the following primal-dual programs.

λn =

 min
s.t
t
tIm − X  0 (3.2.1a) λn =

 max
s.t
·
X Y
trace(Y ) = 1 (3.2.1b)
t∈R Y 0
 

14 See the wikipedia article [Link]

34
3.3 Change of variable in SDP programs
Let us focus on the dual SDP (DSDP ) form from (2.1.3a)-(2.1.3c); recall its constraints have the form
·
Ai Y = ci ∀i ∈ [1..n], where Y  0 are the variables. We will now change variables Y into variables
Z = QY Q> for a non-singular matrix Q. Recalling Prop. 1.2.3, such Z and Y are congruent and they satisfy
Z  0 ⇐⇒ Y  0. Notice that if Y = yy> , then the variable change maps yy> to Qyy> Q> = (Qy)(Qy)> ,
i.e., we can also say vector y is mapped to z = Qy.
>
One could try to express the dual program (DSDP ) in variables Z by simply replacing Y with Q−1 ZQ−1 .
· · >
But developing Ai Y = Ai (Q−1 ZQ−1 ) by brute force could be quite painful and lead to a mess. We
can obtain a more elegant reformulation using the lemma below.
Lemma [Link]. Given symmetric matrices A, B ∈ Rm×m and non-singular matrix R, the following holds:

·
A B = R−1 AR−1
>

RBR>

Pm
· Pm b
Proof. Using the eigendecomposition (1.1.1) can write A = i=1 λai ai a>
i and B =
>
i=1 λi
bi bi . Given
>
that the scalar product is distributive, to prove the lemma, it is enough to show ai ai bj b>
j


= ·
·
 
>
R−1 ai a> −1
Rbj b> >


i R j R ∀i, j ∈ [1..n]. We can calculate

· ·
    > 
> > >
R−1 aa> R−1 Rbb> R> = R−1 a R−1 a (Rb)(Rb)>

=


= a> R−1 Rb


>
R−1 a
2

(Rb)
· 2
(we applied Lemma [Link])

2
= a> b
= aa>


·
bb> .


(we applied Lemma [Link])

which finishes the proof.

· · ·
   
> >
Y = Q−1 BQ−1 QY Q> = Q−1 Y Q−1


Based on this lemma, we have B Z. By applying
the same calculations on Ai · X ∀i ∈ [1..n], we obtain that the dual (DSDP ) program is equivalent to:



 >
max Q−1 BQ−1

· Z

·

  
(DSDPZ ) −1 > −1
 s.t Q A i Q Z = ci ∀i ∈ [1..n]



Z0 (recall Z  0 ⇐⇒ Y  0 since Y and Z are congruent)

3.4 Convex quadratic programming is a particular case of SDP programming

We consider a convex quadratic program with n variables x1 , x2 , . . . xn ∈ R:

min x> A0 x + b>

0x (3.4.1a)
>
s.t. x Ai x + b>
i x ≤ ci ∀i ∈ [1..p] (3.4.1b)

Particularizing Prop. 1.8.1, a quadratic function x> Ai x (∀i ∈ [0..p]) is convex if and only if the Hessian 2Ai
is SDP, i.e., we need to have Ai  0 ∀i ∈ [0..p].

3.4.1 Reformulation using the (Cholesky) factorization of SDP matrices

Each Ai (with i ∈ [0..p]) can be factorized as Ai = Ri Ri> using any of the presented decompositions of SDP
matrices (e.g., Cholesky, eigenvalue or square root, see Corollary 1.7.1). Let us move the objective function

35
into the constraint set by introducing a real variable c0 that has to be minimized such that x> A0 x+b>
0 x ≤ c0 .
After simple algebraic manipulations, the above program (3.4.1a)-(3.4.1b) can be written as:
min c0
s.t. ci − b> > >
i x − x Ri Ri x ≥ 0 ∀i ∈ [0..p]

Using the Schur complements Property 1.3.2, the above program is further equivalent to program below.
Indeed, by applying the Schur complement, one can easily check that a feasible solution of above program
is feasible in program below and vice-versa.
min c0
Ri> x
 
In
s.t.  0 ∀i ∈ [0..p],
x> Ri ci − b>i x

which is a program in variables x1 , x2 , . . . xn and c0 . This is an SDP program with p + 1 constraints (that
could be expressed in an aggregated form (2.1.1a)-(2.1.1c) with a unique constraint, see also Prop. 2.1.4).

3.4.2 Reformulation by relaxing xx> into X  xx>

We will show that (3.4.1a)-(3.4.1b) is equivalent to the following SDP program:

min A0
· X + b>
0x (3.4.2a)

s.t. Ai

1
· X + b>
x >

i x ≤ ci ∀i ∈ [1..p] (3.4.2b)

0 (3.4.2c)
x X

First, it is clear that any feasible solution x of (3.4.1a)-(3.4.1b) can be associated to a feasible solution
X = xx> of (3.4.2a)-(3.4.2c) that has the same objective value. Conversely, it is possible to show that
a feasible solution (3.4.2a)-(3.4.2c) corresponds to feasible solution of (3.4.1a)-(3.4.1b), but we still need a
few lines. Any X and x that satisfy (3.4.2c) also satisfy X  xx> by virtue of the Schur complements
Prop. 1.3.1. Thus, X can be written in the form X = xx> + S for some S  0. Since Ai  0 ∀i ∈ [0..p],
·
each product Ai S is non-negative. This ensures the fact
that xx> satisfy all constraints (3.4.1b) and,
·
similarly, the objective value (3.4.2a) satisfies A0 xx> + S + b> · > >
0 x ≥ A0 xx + b0 x. In other words, x
is a feasible solution of (3.4.1a)-(3.4.1b) and its objective value is no worse than that of X and x in (3.4.2a).
The above (3.4.2a)-(3.4.2c) is actually an SDP program that can be easily written in the standard
form (2.1.1a)-(2.1.1c). Both (3.4.2b) and (3.4.2c) are linear matrix inequalities like (2.1.1b) in variables
x1 , x2 , . . . , xn and X11 , X12 , . . . Xnn ; (3.4.2b) uses “1 × 1 matrices” and (3.4.2c) uses (n + 1) × (n + 1)
matrices. We thus have two inequalities of the form (2.1.1b) that can be aggregated into a unique constraint
(2.1.1b) that uses block-diagonal i matrices (Prop. 2.1.4). As a final side remark, similar programs may even
−y 21 b>
·
h  >
have constraints like 1 b A 0 1 x
x X
≤ 0 that contain variables in both factors, because of variable y in
2 0 0
the left-hand term. However, after developing the scalar product, the variable y simply arises as a term in
a final sum as in (2.1.1b).

3.4.3 Unconstrained quadratic programming reduces to SDP programming

We consider an unconstrained quadratic program :

min x> Ax + b> x (3.4.3)

If A is not SDP, we can show that this program is unbounded from below. For this, let us take any eigenvector
v such that Av = −αv with α > 0. Consider the function f (t) = (tv> )A(tv) + b> v = −αt2 v> v + tb> v.
This is clearly a strictly concave function in t and it goes to −∞ when t goes to ∞.
To solve (3.4.3) by SDP programming, one first has to check if A is SDP or not. The necessity of this
step makes the approach from Section 3.4.1 unusable here, because we here need a method that handles at
the same time both the case A  0 and A  0.
Let us now show how to address the case A  0 by SDP programming. We first need the following result
which should not be taken from granted (although it is omitted from certain lecture notes).

36
Proposition 3.4.1. If an unconstrained quadratic program x> Ax + b> x (i.e., a polynomial of degree 2)
is bounded from below, there is a solution x that does reach the minimum value. The polynomial is convex
(A  0) and the gradient in x is 0.

This does not hold for polynomials of any degree or for convex functions. Indeed, p(x1 , x2 ) =
1
(1 − x1 x2 )2 + x21 does not reach its minimum (infimum) 0 = lim p( m , m). Function h(x) = ex is infinitely
m→∞
differentiable, convex and bounded from below, but there is no x with h0 (x) = 0 that does reach the minimum
value. On the other hand, it is actually possible to prove than a convex polynomial (of any degree) always
reaches its minimum value, but the proof of such result is outside the scope of this document.15
Proof. We showed above that the polynomial is bounded only if A  0.
n
> >

x ∈ R that cancels the gradient, the program
Let us show that if there is no is unbounded. The gradient
can be written ∇ x Ax + b x = 2Ax + b, and so, there is no x ∈ Rn such that 2Ax = −b. This means
that −b (or b) does not belong to the image (set of linear combinations of the columns) of 2A. Using a
similar (transposed) argument as in the first paragraph of the first proof of Prop. 1.6.4, −b needs to have
the form −b = −bimg − b0 , where −bimg ∈ img(2A) and −b0 ∈ null(2A) with b0 6= 0, where img(2A) is
the image of 2A and null(2A) is the null space of A (see the null space definition in (A.1.2)).
Taking x = tb0 , we can now define h(t) = (tb0 )> A(tb0 ) + (bimg + b0 )> (tb0 ) = 0 + tb> 2
0 b0 = t|b0 | ,
where we used b0 ∈ null(A) =⇒ Ab0 = 0. Since b0 6= 0, we have |b0 | 6= 0, and so, h(t) is a non-zero
linear function that is clearly unbounded from below. Since x = tb0 is a perfectly feasible solution of our
unconstrained program, this program is unbounded from below. We proved that if there is no stationary point
x ∈ Rn that cancels the gradient, the program is unbounded. This means that a bounded convex quadratic
program needs to have some stationary point x ∈ Rn such that ∇ x> Ax + b> x x = 2Ax + b = 0. For a

convex function, the stationary point needs to be the global minimum, and so, x reaches the minimum value
of x> Ax + b> x.
We hereafter consider (3.4.3) is convex (A  0).
Proposition 3.4.2. The optimum value of unconstrained convex quadratic program (3.4.3) is equal to:

t∗ = max t (3.4.4a)
1 >
 
−t 2b
s.t. 1  0, (3.4.4b)
2b A

where we use the convention t∗ = −∞ if (3.4.4b) is infeasible for any t ∈ R, this case being equivalent to the
fact that (3.4.3) is unbounded from below.
Proof. It is enough to prove that: (i) if (3.4.3) is unbounded, then (3.4.4a)-(3.4.4b) is infeasible and (ii) if
(3.4.3) is bounded then its optimum value is equal to t∗ in (3.4.4a)-(3.4.4b).
Based on Prop 3.4.1, the above case (i) can only arise if either A is not SDP so that (3.4.4b) is clearly
infeasible, or if (3.4.3) has no stationary point. If there is no stationary point, then there is no x such that
2Ax + b = 0. This means that 12 b> can not be written as a linear combination of (the rows of) A. Using
Prop. 1.6.4, we obtain that (3.4.4b) is infeasible
In the non-degenerate case (ii), Prop. 3.4.1 guarantees that the quadratic function has a stationary point
x so that 2Ax = −b. This stationary point minimizes (3.4.3) so that the optimum value is x> Ax + b> x =
−x> 21 b + b> x = 12 x> b. We will prove that t∗ = 12 x> b.
−t 1 b>
h i
Using Prop. 1.2.4, the SDP status of 1 b 2 A does not change if we add to the first row a linear
2
combination of the other rows. If we add x>h[ 21 b A ] to thei first row followed by the same (transposed)
−t+ 21 x> b 0
operation on the columns, we obtain the matrix . This latter matrix is SDP only if −t+ 12 x> b ≥
0 A
0, which means that the optimum t∗ value is t∗ = 21 x> b, which is also the optimum value of the unconstrained
quadratic function.
15 For a proof, see the response of R. Israel on my question asked on the on-line forum [Link]

[Link]/questions/2341163/does-a-convex-polynomial-always-reaches-its-minimum-value/. The example

(1 − x1 x2 )2 + x21 is taken from a response of J.P. McCarthy on [Link]
polynomial-px-y-with-inf-mathbbr2-p-0-but-without-any-point-where.

37
 Finally, it is interesting that
 the1 linear
 matrix inequality (3.4.4b) is equivalent to the following system
n −t 2 b>
of 2 linear inequalities: det 1 b A J
≥ 0 ∀J ⊆ [1..n], where bJ and resp. AJ are obtained from b and
2 J J
resp. A by selecting rows or columns J. When A  0, (3.4.4a)-(3.4.4b) can actually be written as a linear
program.

3.5 An LP with equality constraints as an SDP program in the dual form

Consider the Linear Program (LP):

max b1 x1 + b2 x2 + . . . bn xn
s.t. cj0 + cj1 x1 + cj2 x2 + . . . cjn xn = 0 ∀j ∈ [1..m]

There are two ways of converting this LP into an SDP. The direct method consists of replacing each equality
with two inequalities and of applying Prop. 2.1.4 on the resulting system of inequalities. We obtain an
aggregated SDP programs expressed with aggregated diagonal matrices.
  1     2
· 1
·

j j> > j
The second method uses the property c c [1 x ] = c proved in Lemma [Link],
x x
where cj = [cj0 cj1 cj2 . . . cjn ]> ∀j ∈ [1..m]. We can thus write:

· · x>
   
1 > 1
 
cj = 0 ⇐⇒ cj cj = 0. (3.5.1)
x x xx>

Thus, any feasible solution x of the above LP can be directly converted to a feasible solution of the SDP
program below by simply setting X = xx> and performing a sum over all j ∈ [1..m].
max b1 x1 + b2 x2 + . . . bn xn

·
 
m
1 x>
 
X >
j j 
s.t.  c c =0
x X
j=1

1 x>
 
0
x X
 >
The last point to prove is that any feasible solution or x1 xX of the above SDP program can be associated
to a feasible solution x of the LP. First, we notice that

· ·
 
m
1 x> 1 x>
    
> >
X 
 cj cj  = 0 =⇒ cj cj = 0 ∀j ∈ [1..m],
x X x X
j=1

 for each j ∈ [1..m]

because we have a non-negative product of two SDP matrices in the left-hand side sum,
i.e., cj cj
>
· 1 x> ≥ 0 ∀j ∈ [1..m]. We can now use that 1 x>  0 is equivalent to X  xx> so as to
x X
 
x X


obtain

·  
1 x>
·   1 x> 
·
  0  
 >
 > 0 j j>
cj cj = 0 =⇒ cj cj + c c =0
x X 0 X − xx> x xx>

=⇒ cj cj
>
·
  1 x> 
x xx>
= 0,

where we used the fact that all involved matrices are SDP and that their scalar products are non-negative.
We can now apply (3.5.1) to obtain the implication below, which confirms x is feasible in the initial LP.

·
  1 x> 
·
 
 > 1
cj cj = 0 =⇒ c j
= 0 ∀j ∈ [1..m].
x xx> x

38
 PART 2: MORE ADVANCED SDP PROGRAMMING

4 Six equivalent formulations of the Lovász theta number ϑ(G)

4.1 A first SDP formulation of the theta number
4.1.1 The primal form (ϑG )
We consider a graph G = ([1..n], E). We introduce the Lovász theta number using the following program
based on SDP matrix Z ∈ Rn×n .

min t (4.1.1a)



 s.t. z ii = t − 1 ∀i ∈ [1..n]

(4.1.1b)
(ϑG )

 z ij = −1 ∀{i, j} ∈ /E (4.1.1c)


Z  0. (4.1.1d)

We use the notational convention ϑ(G) = OP T (ϑG ). We consider above (ϑG ) as a primal SDP program
of the form (2.1.1a)-(2.1.1c) in which the variables are t ∈ Rn and z ij ∈ Rn ∀{i, j} ∈ E. The matrix B from
(2.1.1a)-(2.1.1c) contains ones on the diagonal and on all positions (i, j) corresponding to {i, j} ∈
/ E. As a
side remark, using (3.2.1a), we observe that (ϑG ) returns the maximum eigenvalue of the matrix B described
above.
Theorem 4.1.1. (ϑ(G) ≥ α(G)) The Lovász theta number ϑ(G) is greater than or equal to the maximum
stable α(G) of G.
Proof. Let us consider the maximum stable J ∈ [1..n] with |J| = α(G). If we restrict Z from (4.1.1a)-(4.1.1d)
to its minor corresponding to rows J and columns J, we obtain tIα(G) − 1  0, where 1 is a matrix in which
all elements are equal to 1. The fact that t ≥ α(G) follows from the next lemma (that needs to hold for
n = α(G)).
Lemma [Link]. The lowest z ∈ R for which the n × n matrix below is SDP is z = n − 1.
 
z −1 −1 . . . −1
−1 z −1 . . . −1
 
An (z) = −1 −1 z . . . −1 0

 .. .. .. .. .. 
 . . . . . 
−1 −1 −1 ... z
| {z }
n

We provide four proofs so as to master multiple proof techniques and explore a variety of SDP tools presented
in this manuscript.
Proof 1. We can write An (z) = (z + 1) − 1, where 1 is a matrix filled with n × n ones. Using (3.2.1a),
the minimum value (z + 1) such that (z + 1) − 1  0 is the maximum eigenvalue of 1. But what are the
eigenvalues and eigenvectors of 1? Since all positions of 1v must be equal for any eigenvector v ∈ Rn , we
obtain that either (i) the eigenvalue λv of v is zero or (ii) all elements of v are equal so that λv = n. The
sought value z + 1 is thus n, leading to z = n − 1. s .
n
P p
Proof 2. It is easy to see that the Frobenius norm of An (0) is |An (0)| = An (0)ij = (n − 1)2 = n − 1.
i,j=1
Using Prop. 1.4.1, the minimum eigenvalue of An (0) is at least −|An (0)| = −(n − 1). This is enough to
guarantee that An (n − 1) = (n − 1)In + An (0)  0. We show An (z)  0 for any z < n − 1 by noticing
[1 1 1 . . . 1]An (z)[1 1 1 . . . 1]> < 0 for any z < n − 1. This proves that z = n − 1 is the lowest z such that
An (z)  0.

39
Proof 3. We apply the Gershgorin circle Theorem A.2.8 to show An (n − 1)  0. The theorem states that any
n
P
eigenvalue λ of An (n − 1) needs to satisfy |λ − An (n − 1)ii | ≤ |An (n − 1)ij | for some i ∈ [1..n]. However,
j=1
j6=i
for any i ∈ [1..n] this reduces to |λ − (n − 1)| ≤ n − 1, which means λ ≥ 0, leading to An (n − 1)  0. To
show An (z)  0 for any z < n − 1, use the method from the last two sentences of above Proof 2.

Proof 4. We proceed by induction. The lemma is clearly true for n = 1. For any n ≥ 2, we need to
have z > 0 because z = 0 would make negative any 2 × 2 minor (use Prop 1.2.5). We apply the Schur
A B>
complement from Prop 1.3.2. Using the notations A, B and C from Prop 1.3.2, we write An (z) = B C
>
with A = z (which satisfies condition A  0), B > = −1n−1 = [−1 − 1 · · · − 1], and C = An−1 (z), i.e.,
| {z }
n−1 positions
An−1 (z) is the (n − 1) × (n − 1) bottom-right minor. Using the Schur complement, we obtain An (z)  0 ⇐⇒
>
C − BA−1 B >  0 ⇐⇒ An−1 (z) − z1 1n−1 1n−1  0. This last matrix inequality boils down to:
 (z+1)(z−1) 
z − z1 −1 − z1 . . . −1 − z1 − z+1 − z+1
 
z z ... z
−1 − 1 (z+1)(z−1)
z − z1 . . . −1 − z1   − z+1 ... − z+1
 
z z z z 
..   0 ⇐⇒  0
 
 .. .. ..  .. .. .. ..
 . . . .   . . . .


1 1 1
−1 − z −1 − z . . . z−z z+1
− z z+1
− z ... (z+1)(z−1)
| {z } z
| {z }
n−1 n−1
 
z − 1 −1 . . . −1
 −1 z − 1 . . . −1 
⇐⇒  ..   0
 
.. .. . .
 . . . . 
−1 −1 . . . z − 1
| {z }
n−1

We have obtained that An (z)  0 ⇐⇒ An−1 (z − 1)  0. We can use the induction hypothesis: the lowest
z − 1 such that An−1 (z − 1)  0 is z − 1 = n − 2, and so, z = n − 1 is the lowest z such that An (z)  0.
Theorem 4.1.2. (ϑ(G) ≤ χ(G)) The Lovász theta number ϑ(G) is less than or equal to the clique cover
number of G. This clique cover number is the chromatic number of the complementary graph G.
Proof. Consider a partition (C1 , C2 , . . . Ck ) of the vertex set [1..n] such that each Ci (with i ∈ [1..k]) is a
clique. Applying Lemma [Link] above, we have
 
k − 1 −1 −1 . . . −1
 −1 k − 1 −1 . . . −1 
 
 −1 −1 k − 1 . . . −1 
 0
 .. .. .. . . .. 
 . . . . . 
−1 −1 −1 . . . k − 1
| {z }
k

This means (use the Cholesky factorisation from Prop 1.6.7 or other decompositions from Corollary 1.7.1)
· ·
there exist k-dimensional vectors v1 , v2 , . . . vk such that vi vi = k − 1 and vi vj = −1 ∀i, j ∈ [1..k], i 6= j.
Since {C1 , C2 , . . . Ck } cover the vertex set [1..n], all vertices u ∈ [1..n] can be associated to vector
vu = v`(u) , where `(u) is the clique that contains u (we have u ∈ C`(u) ). We actually associate the same
vector to all vertices of a clique.
·
We now define the matrix Z by setting z ij = vi vj . This is a Gram matrix that is clearly SDP (use
Prop. A.1.8). Notice that: (i) z ii = k − 1 ∀i ∈ [1..n] and (ii) if i, j ∈ [1..n] do not belong to the same
clique, then z ij = −1, and so, z ij = −1 ∀{i, j} ∈ / E. Properties (i) and (ii) are enough to ensure that Z can
be written in the form (4.1.1b)-(4.1.1c) with t = k. We have constructed a feasible solution of (ϑG ) from
(4.1.1a)-(4.1.1d) with objective value k. Taking k = χ(G), we have OP T (ϑG ) ≤ χ(G).
This proof is inspired from a related result of a lecture note of Anupam Gupta.16
16 See (11.3) of the document available, as of 2017, at [Link]

www/notes/[Link].

40
 Using α(G) = ω(G) (where ω denotes the maximum clique size), we also obtain the “sandwich” property
as a direct consequence of Theorem 4.1.1 and Theorem 4.1.2.

ω(G) ≤ ϑ(G) ≤ χ(G) (4.1.2)

The following corollary of Lemma [Link] can be generally useful, but we do not use it in this document.
Corollary 4.1.3. When n ≥ 2 unitary vectors are as spread out as much as possible, the dot product of any
−1
pair of them is n−1 .
·
Proof. By “spread out as much as possible”, we want t = max{vi vj : i, j ∈ [1..n], i 6= j} to be as low
as possible. Lemma [Link] states that nIn − 1  0, where 1 is a matrix with all elements equal to 1. By
1 −1
dividing all terms by n−1 , we obtain a matrix T  0 with 1 on the diagonal and with n−1 on all non-diagonal
−1
positions. This means that t ≤ n−1 . This t value is optimal. If we decrease any non-diagonal term(s) of T ,
we obtain a matrix T 0  0, since [1 1 . . . 1]T 0 [1 1 . . . 1]> < 0.

4.1.2 The dual form of (ϑG )

We now introduce the dual program (DϑG ) of (ϑG ) from (4.1.1a)-(4.1.1d). Let Dϑ(G) = OP T (DϑG ).

·

X n X

max Y + Yij = 1 Y (4.1.3a)

ii





 i=1 {i,j}∈E
/
(DϑG ) s.t. trace(Y ) = 1 (4.1.3b)


Y = 0 ∀{i, j} ∈ E (4.1.3c)



 ij

Y  0, (4.1.3d)


·
where (4.1.3a) simplifies to 1 Y because Yij = 0 ∀{i, j} ∈ E by virtue of (4.1.3c). Notice that both (ϑG )
and (DϑG ) are strictly feasible (take a sufficiently large t in (ϑG ) and Y = n1 In in (DϑG )), and so, we can
apply the strong duality Theorem 2.3.5 to state that the optimum value is reached by both programs.
Proposition 4.1.4. The “sandwich” property α(G) ≤ ϑ(G) ≤ χ(G) obtained in (4.1.2) in the previous
Section 4.1.1 can also be proved using the dual (DϑG ).
Proof of α(G) ≤ ϑ(G)
1
Take the largest stable S of G. Construct matrix Y S such that YijS = |S| ∀i, j ∈ S and YijS = 0 otherwise.
S
It is not hard to check that Y is a feasible solution of (DϑG ) in (4.1.3a)-(4.1.3d) with objective value |S|.
This is enough to state α(G) ≤ Dϑ(G) = ϑ(G).
Proof of ϑ(G) ≤ χ(G)
Consider a clique cover {C1 , C2 , . . . Ck } of G. Without loss of generality, we can re-order [1..n]  such
that all C
 i (with i ∈ [1..k])
 represent segments of [1..n], i.e., C 1 contains the first |C1 | elements 1..|C 1| ,
C2 = |C1 | + 1..|C1 | + |C2 | , etc. We can decompose Y of (Dϑ(G)) from (4.1.3a)-(4.1.3d) into k 2 blocks:
 
Y11 Y12 . . . Y1k
 Y21 Y22 . . . Y2k 
Y = . ..  ,
 
.. ..
 .. . . . 
Yk1 Yk2 . . . Ykk
where block Yij has size |Ci | × |Cj | for any i, j ∈ [1..k]. Notice that Yii is diagonal because Ci is a clique
Pk
·
(∀i ∈ [1..k]). This means i=1 1 Yii = 1, based on trace(Y ) = 1. We will show that

1 · Y =
k X
X k

i=1 j=1
1 · Yij ≤ k. (4.1.4)

The iprincipal minor of Y associated to rows and columns h Ci and i Cj needs to be SDP. We thus obtain
·
h
Yii Yij > Yii Yij
Yji Yjj  0. Take x = [1 . . . 1} −1 − 1 · · · − 1]. Using Yji Yjj
| 1{z xx> ≥ 0, we obtain:
| {z }
|Ci | |Cj |

1 · Yij ≤ ·
1 Yi + 1 Yj
2
· , ∀i, j ∈ [1..k]

41
Notice this holds (with equality) for terms i = j ∈ [1..k]. We can simply now obtain (4.1.4) by applying
k X
k

· k X k
· ·
1 Yi + 1 Yj 2k ki=1 1 Yi ·
X X P
1 Yij ≤ = = k.
i=1 j=1 i=1 j=1
2 2

4.2 A second SDP formulation (ϑ0G ) of the theta number

We start from the following well-known linear program for the maximum stable α(G).
n

 X
max yi



i=1
(αG )
 s.t. yi + yj ≤ 1 ∀{i, j} ∈ E



y ∈ {0, 1}n


We now introduce a first SDP relaxation of above (αG ).

 n
 X
max y0i (4.2.2a)






 i=1


(ϑ0G ) s.t. y00 = 1 (4.2.2b)


 yij = 0 ∀{i, j} ∈ E (4.2.2c)

yii = y0i ∀i ∈ [1..n]



 (4.2.2d)

Y 0

(4.2.2e)
0
 1  hard>to check that (ϑG ) is a0 relaxation of (αG ). Take an optimal solution y of (αG ) and construct
It is not
Y = y [1 y ] that is feasible for (ϑG ) and has the same objective value as in (αG ).
Theorem 4.2.1. The optimum value ϑ0 (G) of (ϑ0G ) from (4.2.2a)-(4.2.2e) is equal to the Lovász theta number
i.e., to the optimum value ϑ(G) of (ϑG ) from (4.1.1a)-(4.1.1d).
Proof. We first show that (ϑ0G ) is equivalent to the following relaxation of (αG ).

 X n
2y0i − yii



 max (4.2.3a)

i=1


(ϑe0 G ) s.t. y00 = 1 (4.2.3b)


yij = 0 ∀{i, j} ∈ E (4.2.3c)





Y 0 (4.2.3d)


We take an optimal solution Y of (ϑe0 G ) and we will show it satisfies yii = y0i ∀i ∈ [1..n]. Consider any
i ∈ [1..n]. If yii = 0, then y0i needs to be zero as well. If yii 6= 0, we can multiply row i and column i of Y by
y0i e0
yii to obtain a matrix that is still feasible for (ϑ G ) and whose objective value is greater than or equal to that
y2 yii >0
of Y . This simply follows from 2y0i yy0iii
− yii y0i 2 2
2 ≥ 2y0i − yii ⇐⇒ y0i ≥ 2y0i yii − yii ⇐⇒ (y0i − yii )
2
≥ 0.
ii

This proves OP T (ϑe0 G ) = OP T (ϑ0G ). We will show that OP T (ϑe0 G ) = OP T (ϑG ) = ϑ(G).
Let us write the dual of (ϑe0 G ).

 min t (4.2.4a)

  



 t −1 −1 −1 ... −1


 −1 1 z12 z13 ... zn1 

  >
  −1 z21 1 z23 ...

zn2 
(Dϑe0 G )

s.t.
t − 1 = −1
 
0 (4.2.4b)
− 1 Z  z31 z32 1 ... zn3 

 .. .. .. .. .. 


 ..




 . . . . . . 
−1 zn1 zn2 zn3 ... znn





zij = 0 ∀{i, j} ∈
/E (4.2.4c)


42
Programs (ϑe0 G ) in (4.2.3a)-(4.2.3d) and (Dhϑe0 G ) in (4.2.4a)-(4.2.4c) are strictly feasible. For (ϑe0 G ), it is
> i
enough to take Y = I. For (Dϑe0 G ), we have n+1 −1  0, as certified by applying the Sylvester criterion
1
− In
(Prop. 1.5.2) in reversed order (from bottom-right to top-left). The determinant of the whole matrix is 1: if
we add the last n rows to the first one, we obtain a first row with only one non-zero element of value 1 in
the upper-left corner. As such, using the strong duality Theorem 2.3.5, we have OP T (ϑe0 G ) = OP T (Dϑe0 G )
and this value is effectively reached by both programs.
We now reformulate (Dϑe0 G ) and let us focus on (4.2.4b). Using the Schur complement Prop. 1.3.2, we
have
>
tZ − 1
 
t −1 1
 0 ⇐⇒ Z − 1  0 ⇐⇒  0 ⇐⇒ tZ − 1  0,
−1 Z t t
where we used several times t > 0 (notice t = 0 would render (Dϑe0 G ) infeasible because of the upper-left 2×
2 minor of the matrix in (4.2.4b)). Writing Z = tZ − 1, we notice z ii = t − 1 ∀i ∈ [1..n] (because zii = 1 in
(4.2.4b)) and z ij = −1 ∀{i, j} ∈
/ E (because zij = 0 in (4.2.4c)). If {i, j} ∈ E, we have z ij = tzij − 1, i.e., z ij
can actually take any real value, independent from the other terms of the matrix. Replacing this in (Dϑe0 G ),
we obtain that (Dϑe0 G ) is equivalent to following program which is exactly (ϑG ) from (4.1.1a)-(4.1.1d).
min t



 s.t. z ii = t − 1 ∀i ∈ [1..n]

(ϑG )

 z ij = −1 ∀{i, j} ∈ /E


Z  0.
All programs presented during the proof have the same optimum, i.e., we proved that OP T (ϑ0G ) =
OP T (ϑe0 G ) = OP T (Dϑe0 G ) = OP T (ϑG ) = ϑ(G). The arguments from this proof are based on a reversed
version of the proof from Section 3.1. of the PhD thesis of of Nebojša Gvozdenović.17

4.3 A formulation ϑ00 (G) of the theta number without SDP matrices
We associate an unit vector ui for each vertex i ∈ [1..n]. We say that the unit vectors {u1 , u2 , . . . un }
constitute an orthonormal representation of G = ([1..n], E) if and only if ui uj = 0 ∀{i, j} ∈
introduce the following function:
· / E, i 6= j. We

 
1
ϑ00 (G) = min max 1 2 n
·
i∈[1..n] (c ui )2
: |c| = 1, {u , u , . . . u } is an orthonormal representation of G (4.3.1)

Given any c and {u1 , u2 , . . . un } that yield an optimum in above (4.3.1), we can apply a rotation R that
maps c into unitary vector Rc = [1 0 0 0 . . . 0]> and {u1 , u2 , . . . un } into {u01 , u02 , . . . u0n }. A rotation is
always represented by a matrix R that is unitary and orthonormal,18 and so, does not change scalar products
or norms. Indeed, if R> R = I, for any two vectors v and v0 we have v> v0 = v> R> Rv0 = (Rv)> (Rv0 ), and
so, the mapping v → Rv does not change scalar products or norms. This means that [1 0 0 0 . . . 0]> and
{u01 , u02 , . . . u0n } is also an optimal solution for (4.3.1). As such, we can simplify (4.3.1) to:

(  2 )
00 1 1 2 n
ϑ (G) = min max : {u , u , . . . u } is an orthonormal representation of G (4.3.2)
i∈[1..n] ui1
17 As of 2019, it is available on-line at [Link]
18 We actually still need to prove there exists a rotation of the space that maps the unitary vector c to c = [1 0 0 0 . . . 0]> ∈ Rk .
This task would be very simple if c had the form c = [c1 c2 0 0 0 . . . 0]> , i.e., it would be enough to apply a rotation on the
(space of) the first two coordinates and keep all remaining k − 2 coordinates fixed. We will construct a basis so as to make the
representation of c in this basis always have the above form. Let us take an orthonormal basis v1 , v2 ∈ Rk (with v1 = c) of the
sub-space spanned by c and c and complete it to a full orthonormal basis by adding v3 , v4 . . . vk . Writing V = [v1 v2 v3 . . . vk ],
any vector v ∈ Rk can be represented in the new basis as v = V v0 , i.e., v0 = V −1 v is the expression of v in the new basis.
Notice c and c can be written as linear combinations of v1 and v2 , and so, their representations c0 and c0 in the new basis
only use
 the
R2 = −cos
first two
α sinα

rotates
·
coordinates. After checking that c0 and c0 are unitary, we can define α = arccos c0 c0 . The matrix
by α any 2-dimensional point, and so, it maps c to c0 in the space of the first two coordinates (or
sin α cos α
0
c to c, case in which we replace α by −α). We extend R2 to Rk ∈ Rk×k by putting an one on each new diagonal position
(i, i) with i > 2 and zeros on all other positions (i, j) with i > 2 or j > 2. Rk rotates the first two coordinates of any vector
v0 ∈ Rk and leaves untouched the rest of values v30 , v30 . . . vk0 . Let us calculate V Rk V > c = V Rk V > V c0 = V Rk c0 = V c0 = c.
Since V Rk V > is orthonormal, it is a rotation matrix that performs the desired rotation.

43
Theorem 4.3.1. ϑ00 (G) = ϑ(G)
Proof. We will first show ϑ(G) ≤ ϑ00 (G) and then ϑ00 (G) ≤ ϑ(G).
ϑ(G) ≤ ϑ00 (G)
We will start from an optimal orthonormal representation of G and we will construct a feasible solution of
(Dϑe0 G ) from (4.2.4a)-(4.2.4c) that we proved (Theorem 4.2.1) to reach the optimum ϑ(G). Without loss of
 2
generality, we can consider |u11 | ≤ |ui1 | ∀i ∈ [1..n], and so, (4.3.2) states that ϑ00 (G) = u11 . Let us construct
1
u11 i
vectors v0 , v1 , v2 , . . . vn such that v0 = [− u11 0 0 0 . . . 0]> and for all i ∈ [1..n] we set vi = ui1
u. We
·
1
0 0 i j 0
construct the Gram matrix Z of these vectors such that = v v ∀i, j ∈ [0..n]. We have Z  0 using
Zij

· ·
 2
0 u1 u1
Prop. A.1.8, Z00 = u11 = ϑ00 (G), and Zij
0
/ E, using vi vj = u1i u1j ui uj and the fact that
= 0 ∀{i, j} ∈
1 1 1
1 2 n 0
u , u , . . . u are a orthonormal representation of G. We also have = −1, because v1i = u11 ∀i ∈ [1..n].
Z0i
·
 1 2
u
The diagonal elements are Zii0 = ui1 ui ui ≤ 1 ∀i ∈ [1..n], where we used |u11 | ≤ |ui1 | and the fact that
1
u1 , u2 , u3 , . . . un are unitary. We transform Z 0 into a matrix Z by only increasing the diagonal elements
up to Zii = 1 ∀i ∈ [1..n]; Z remains SDP as the sum of Z and a positive diagonal matrix. This matrix
 2
Z is a feasible solution in (Dϑe0 G ) in (4.2.4a)-(4.2.4c) with objective value Z 0 = 11 = ϑ00 (G). Since
00 u1
OP T (Dϑe0 G ) = ϑ(G) as shown in the proof of Theorem 4.2.1, we have ϑ(G) ≤ ϑ00 (G).

ϑ00 (G) ≤ ϑ(G)

We start from (ϑG ) in (4.1.1a)-(4.1.1d) and we will construct an orthogonal representation of value ϑ(G).
Notice the optimal solution Z of (4.1.1a)-(4.1.1d) verifies Z ii = ϑ(G) − 1 and Z ij = −1 ∀{i, j} ∈ / E.
Using the Cholesky factorisation from Prop. 1.6.7 (or the eigenvalue or the square root factorization as
in Corollary 1.7.1),
1
 ·
there exist v1 , v2 , . . . vn such that Z ij = vi vj . We construct the representation
ui = √ 1 ∀i ∈ [1..n]. We verify the following:
ϑ(G) vi

– |ui |2 = 1
ϑ(G) ·v ) =
(1 + vi i
ϑ(G)
ϑ(G) = 1 ∀i ∈ [1..n], i.e., the representation is unitary;

/ E, we have u · u
– for {i, j} ∈ i j = 1
ϑ(G) (1 + vi · v ) = 0, i.e., the representation is orthonormal;
j

 2
1
– the value of the representation in (4.3.2) is √ 1 = ϑ(G) because ui1 = √ 1 ∀i ∈ [1..n].
ϑ(G)
ϑ(G)

This shows ϑ00 (G) ≤ ϑ(G), which finishes the proof.

The proof of ϑ(G) ≤ ϑ00 (G) is personal. The proof of ϑ00 (G) ≤ ϑ(G) is adapted from the proof of
“ϑ1 (G, w) ≤ ϑ2 (G, w)” from Section 6 of the survey article “The sandwich theorem” of Donald Knuth.19

4.4 A fourth formulation ϑ` (G) of the theta number

We consider an orthonormal representation u1 , u2 , . . . un of G, i.e., we have ui uj = 0 ∀{i, j} ∈ E. The ·
·
n n
(e1 ui )2 = . . . 0}]> . One can think of this
(ui1 )2 , where e1 = [1 0| 0{z
P P
leaning of this representation is
i=1 i=1
n−1
leaning term in the sense that any vector is leaning somehow on (casts its shadow on) the first dimension.
We introduce:

·
( n )
X
` i 2 1 2 n
ϑ (G) = max (e1 u ) : {u , u , . . . u } is an orthonormal representation of G (4.4.1)
i=1

Proposition 4.4.1. ϑ` (G) = ϑ(G)

19 Published in the Electronic Journal of Combinatorics in 1994 (1), available at [Link]

php/eljc/article/view/v1i1a1/pdf.

44
Proof. We will show that generating an orthonormal representation of G in (4.4.1) is equivalent to generating
a feasible solution of (ϑ0G ) in (4.2.2a)-(4.2.2e) with the same objective value. We start by noticing that gen-
erating a feasible solution Y ∈ R(n+1)×(n+1) of (ϑ0G ) is equivalent to generating n + 1 vectors c, v1 , v2 , . . . vn
· ·
such that Yij = vi vj ∀i, j ∈ [1..n] and Y0i = Yi0 = c vi ∀i ∈ [1..n]. Given constraints (4.2.2b)-(4.2.2d),
· · · ·
these vectors need to satisfy: c c = Y00 = 1, vi vj = 0 ∀{i, j} ∈ E, and c vi = vi vi = d2i , ∀i ∈ [1..n].
n
The objective value of Y in (ϑ0G ) is d2i .
P
i=1
Since there exists an orthonormal rotation matrix R that maps c to Rc = e1 and leaves unchanged scalar
products (see Footnote 18, p. 43), generating vectors c, v1 , v2 , . . . vn with above properties is equivalent to
generating vectors Rc = e1 , Rv1 = v1 , Rv2 = v2 , . . . Rvn = vn with the same pairwise scalar products, i.e.,
· · ·
such that vi vj = 0 ∀{i, j} ∈ E and e vi = vi vi = d2i , ∀i ∈ [1..n] because RR> = I. Remark v1i = d2i .
Generating vectors e1 , v1 , v2 , . . . vn with above properties is equivalent to generating vectors e1 , u1 , u2 , . . . un ,
1 i i
i i i
 0
where u = di v ∀i ∈ [1..n] (if di = 0, we have v = 0 and we set u = 1 = [ 1 ] and extend all other
v

· ·
uj ∀j ∈ [1..n] − {i} with a 0 element). One can check that |ui |2 = ui ui = d12 vi vi = 1 ∀i ∈ [1..n] and
· ·
i
1 1 i
ui uj = vj = 0 ∀{i, j} ∈ E. This means that u1 , u2 , . . . un is an orthonormal representation of G
di dj v
n
such that ui1 = di ∀i ∈ [1..n]. The leaning of u1 , u2 , . . . un in (4.4.1) is d2i , i.e., the same as the objective
P
i=1
value of feasible matrix Y in (ϑ0G ) in (4.2.2a)-(4.2.2e). This confirms ϑ` (G) = OP T (ϑ0G ) = ϑ(G).

4.5 Two formulations of the theta number using maximum eigenvalues

4.5.1 A formulation only using the maximum eigenvalue
Let us introduce:

ϑλmax (G) = max λmax (Z) : Z  0, Zii = 1 ∀i ∈ [1..n], zij = 0 ∀{i, j} ∈ E ,


(4.5.1)

where λmax (Z) is the maximum eigenvalue of Z.

Theorem 4.5.1. ϑλmax (G) = ϑ(G)
Proof. Using Lemma [Link], the maximum eigenvalue of Z can be computed as λmax (Z) = max Z (xx> ).
|x|=1
·
As such, (4.5.1) is equivalent to:

ϑλmax (G) = max Z


· (xx> ) : Z  0, Zii = 1 ∀i ∈ [1..n], Zij = 0 ∀{i, j} ∈ E, |x| = 1 (4.5.2)

We show that any feasible solution (Z, x) of above program can be mapped to a feasible solution Y of (DϑG )
from (4.1.3a)-(4.1.3d). We showed in Section 4.1.2 that ϑ(G) = OP T (DϑG ). For the reader’s convenience,
we recall below the definition of this program.

Dϑ(G) = max Y
n
·
1 : Y  0, Yij = 0 ∀{i, j} ∈ E, trace(Y ) = 1
o
(4.5.3)

Consider matrix Y obtained by multiplying each row and column i of Z with xi . In other Pwords, Yij =
Zij xi xj . It is not hard to check that Y reaches in (4.5.3) the same objective value Y 1 = ·
Zij xi xj =
i,j∈[1..n]

·
n
>
x2i = |x|2 = 1.
P
Z (xx ). Furthermore, Y satisfies all constraints of (4.5.3), i.e., trace(Y ) =
i=1
By reversing the above transformation, it is quite straightforward to prove the converse: any √ feasible
solution Y of (4.5.3) can be mapped to a feasible solution (Z, x) of (4.5.2). For this, we take xi = Yii ∀i ∈
[1..n]. We obtain Z by dividing each row and column i of Y with xi for all i such that xi > 0. More exactly,
Y
we obtain Zij = xiijxj ∀i, j ∈ [1..n], xi > 0, xj > 0. If xi = 0, we set Zii = 1 and Zij = 0 ∀j ∈ [1..n] − {i}. It
2
is not hard to check that (Z, x) satisfies the constraints P instance, we have |x|> = trace(Y ) = 1
Pof (4.5.2). For
Yii
and Zii = √Y 2 = 1. One can also check Y 1 =
ii
· Yij =
i,j∈[1..n]
·
Zij xi xj = Z (xx ), where we used
i,j∈[1..n]
Yij = Zij xi xj that holds even if xi = 0 or xj = 0.
This proves (4.5.2) and (4.5.3) are equivalent, which means ϑλmax (G) = Dϑ(G) = ϑ(G).

45
4.5.2 A formulation using the maximum and the minimum eigenvalue
Let us note λmin (X) and λmax (X) the minimum and resp. the maximum eigenvalue of X. We introduce

λmax (X)
ϑλ (G) = max 1 −

: Xii = 0 ∀i ∈ [1..n], Xij = 0 ∀{i, j} ∈ E , (4.5.4)
λmin (X)

where we use the convention λλmax (X)

min (X)
= 0 if λmax (X) = λmin (X) = 0. Since the only SDP matrix with
zeros on the diagonal is 0, any feasible X 6= 0 in above (4.5.4) verifies X  0 and X  0, and so,
λmax (X) > 0 > λmin (X). Any X 6= 0 yields a value 1 − λλmax (X)
min (X)
> 1 in (4.5.4). If X = 0 is the optimal
solution of (4.5.4), then G must be a clique, so that constraints Xij = 0 ∀{i, j} ∈ E force every non-diagonal
element of X to be zero. In this case, we have α(G) = ϑ(G) = χ(G) = ϑλ (G) = 1.
Theorem 4.5.2. ϑλ (G) = ϑ(G)
Proof. We take the optimal X in (4.5.4) and we construct a feasible solution Z of (4.5.1) with the same
objective value. As stated above, X = 0 leads to ϑλ (G) = ϑ(G) = 1 and any X 6= 0 satisfies λmin (X) < 0.
(X 0 )
Considering λmin (X) < 0, we can construct X 0 = |λminX(X)| and notice λλmax (X)
min (X)
= λλmax 0
min (X )
= −λmax (X 0 ).
0 0
This means X is also optimal in (4.5.4) and has λmin (X) = −1, more exactly X lead (4.5.4) to objective
value 1 + λmax (X 0 ). Consider now Z = X 0 + I and notice λmin (Z) = 0, and so, Z  0. One can easily check
Z is feasible in (4.5.1) and has objective value λmax (X 0 + I) = λmax (X 0 ) + 1, i.e., the same as the objective
value of X or X 0 in (4.5.4). This shows ϑλ (G) ≤ ϑλmax (G) = ϑ(G).
We still have to prove ϑλmax (G) ≤ ϑλ (G). We now attempt to reverse the above process. Let Z be the
optimal solution of (4.5.1) and take X = Z − I. Notice X is feasible in (4.5.4) and that

λmax (X) λmax (X)

ϑλmax (G) = λmax (Z − I + I) = 1 + λmax (X) = 1 + ≤1− , (4.5.5)
1 λmin (X)
where we used 0 > λmin (X) ≥ −1 that we prove now. In fact, we already left aside the particular case
λmin (X) = 0 because that would imply X  0 which would further lead (by applying diag(X) = 0) to
X = 0, which means the optimal Z is Z = X + I = I and the graph is a clique. We can not have
λmin (X) > 0 because that would imply X  0, impossible when diag(X) = 0. We now notice Z = X + I 
0 =⇒ λmin (X + I) ≥ 0 =⇒ λmin (X) ≥ −1. This shows 0 > λmin (X) ≥ −1, confirming the last inequality
of (4.5.5). This finishes the proof because (4.5.5) simplifies to ϑλmax (G) ≤ 1 − λλmax (X)
min (X)
≤ ϑλ (G).

4.6 The theta number ϑ(G) is bounded by the fractional chromatic number
χ∗ (G) of G
4.6.1 The fractional chromatic number
Let C be the set of cliques of G. In the primal-dual programs below we introduce the most standard
formulation of the fractional chromatic number of G. It is not hard to see that the standard chromatic number
χ(G) is an upper bound for these programs, by taking λCi = 1 ∀i ∈ [1..χ(G)], where {C1 , C2 , . . . Cχ(G) } is
the optimal coloring of G (clique covering of G).

n
 P 
min λC P
 max xi
 
C∈C

 

i=1
 P
(χ̂∗G ) s.t. λC ≥ 1 ∀i ∈ [1..n] ∗
(Dχ̂G ) ∀C ∈ C
P
 C∈C  s.t. xi ≤ 1

 C3i 
 i∈C
λC ≥ 0 ∀C ∈ C xi ≥ 0 ∀i ∈ [1..n]
 

These primal-dual programs can be modified as follows. Let us drop the non-negativity constraint x ≥ 0
∗
in the dual (Dχ̂G ). We prove by contradiction that the resulting program has only non-negative optimal
solutions. Assume there is an optimal solution x such that xj < 0 for some j ∈ [1..n]. We show that
feasible. For this, take any clique C 3 j; since C − {j} is naturally a
by increasing xj to 0, x remains P
clique and x is feasible, we have i∈C−{j} xi ≤ 1. If we now increase xj to 0, the new solution satisfies

46
P P
i∈C xi = i∈C−{j} xi ≤ 1. This new solution is still feasible and has a higher objective value, and so, we
obtained a contradiction. The assumption xj < 0 was false. The above programs are thus equivalent to the
∗
next ones, i.e., OP T (χ̂G ) = OP T (Dχ̂∗G ) = OP T (χ∗G ) = OP T (DχG
∗
) = χ∗ (G)

n
 P 
min λC P
 max xi
 
C∈C

 

i=1
 P
∗ s.t. λC = 1 ∀i ∈ [1..n] ∗
(χG ) (4.6.1a) (DχG ) ∀C ∈ C (4.6.1b)
P
 C∈C  s.t. xi ≤ 1

 C3i 
 i∈C
λC ≥ 0 ∀C ∈ C xi ∈ R ∀i ∈ [1..n]
 

4.6.2 A hierarchy of SDP programs sandwiched between ϑ(G) and χ∗ (G)

We will use the (Dϑe0 G ) formulation of ϑ(G) from (4.2.4a)-(4.2.4c). For the reader’s convenience, we repeat
the definition of this program, multiplying by −1 the first row and column of the SDP matrix (this does not
change its SDP status).

min t


(4.6.2a)
>

t 1


s.t. 0 (4.6.2b)
 

0
Dϑe G 1 Z

zii = 1 ∀i ∈ [1..n] (4.6.2c)





zij = 0 ∀{i, j} ∈
/E (4.6.2d)

∗
Notice that (χG ) in (4.6.1a) is defined using variables indexed by a subset of the power set P([1..n]) of the
vertex set [1..n]. This gives us some intuitions that we might need programs with variables indexed by
certain subsets of [1..n]. Let us introduce Pr ([1..n]) = {S ⊆ [1..n] : |S| ≤ r} ∀r ∈ [1..n]. Given a vector y
indexed by all I ∈ P2r ([1..n]), we can write y = (yI )I∈P2r ([1..n]) . Let us now introduce a matrix Mr (y) with
rows and columns indexed by Pr ([1..n]) such that Mr (y)I,J = yI∪J . We can also compactly write:

Mr (y) = (yI∪J )I,J∈Pr ([1..n]) . (4.6.3)

We now introduce the following program:

ψ r (G) =

min y∅ : Mr (y)  0, y{i} = 1 ∀i ∈ [1..n], y{i,j} = 0 ∀{i, j} ∈
/E , (4.6.4)

where one could notice that Mr (y)I,I = Mr (y)∅,I ∀I ∈ Pr ([1..n]), in particular Mr (y){i},{i} = Mr (y)∅,{i} =
y{i} = 1 ∀i ∈ [1..n].

Theorem 4.6.1. ϑ(G) = ψ 1 (G) ≤ ψ 2 (G) ≤ · · · ≤ ψ ω(G) (G) ≤ χ∗ (G).

 
Proof. We first show that (4.6.4) with r = 1 is equivalent to Dϑe0 G in (4.6.2a)-(4.6.2d). It is actually enough
to carefully “decode” all notations to see that the two programs are simply identical up to a notational
translation. With this goal, we can replace M1 (y)∅,∅ = y∅ with t, M1 (y){i},{j} = y{i,j} = 0 with zij =
0 ∀{i, j} ∈
/ E, and M1 (y){i},{j} = y{i,j} with zij ∀{i, j} ∈ E; M∅,{i} = y{i} = 1 for i ∈ [1..n] is simply
>
translated into the vectors 1 and 1 that border Z in (4.6.2b). One can check that we have just mapped
one program into another, which guarantees that ϑ(G) = OP T Dϑe0 G = ψ 1 (G).
We now show that ψ r (G) ≤ ψ r+1 (G) ∀r ∈ [1..n]. Notice it is not possible to border a solution Mr (y)
of ψ r (G) with zeros to obtain a solution Mr+1 (y0 ) of ψ r+1 (G), because certain elements of Mr+1 (y0 ) are
inherited from Mr (y0 ), e.g., a proper bordering imposes Mr+1 (y0 )∅,[1..r+1] = Mr (y){1},[1..r] . Take the solution
y0 that achieves the optimum value ψ r+1 (G). The key is to notice Mr (y0 ) is a principal minor of Mr+1 (y0 ),
and so, Mr (y0 )  0. By removing from y0 all indices S ∈ P2r+2 [1..n] − P2r [1..n], we obtain a vector
y indexed by P2r [1..n] that generates a feasible solution Mr (y) of ψ r (G). Indeed, Mr (y)  0 because
Mr (y) = Mr (y0 ) is a principal minor of Mr+1 (y0 ); all other constraints in (4.6.4) concern the values of y on
sets of P2 ([1..n]) that are inherited from y0 . The objective value is the same, i.e., y∅ = y∅0 . This is enough to
state ψ r (G) ≤ ψ r+1 (G), because (4.6.4) has a minimizing objective. The value ψ r (G) could be even strictly
less that y∅ because ψ r (G) is obtained with a program (4.6.4) with fewer constraints than ψ r+1 (G), i.e., it

47
involves smaller SDP matrices Mr (y). In other words, there might exist some y indexed by P2r ([1..n]) that
achieves a lower value ψ r (G) and that can not be extended to some feasible y0 indexed by P2i+2 ([1..n]).
We finally show ψ ω(G) (G) ≤ χ∗ (G). We take the optimum solution λ of (χ∗G ) from (4.6.1a) and construct
a feasible solution with the same objective value in (4.6.4) with r = ω(G) = Pω(G) ([1..n]). Take any clique
C ∈ C ⊆ Pω(G) ([1..n]) and construct yC indexed by P2ω(G) ([1..n]) such that
(
C 1 if S ⊆ C
yS = (4.6.5)
0 if S * C
>
One only needs to decode notations to see that Mω(G) (yC ) = yC yC ∈ {0, 1}Pω(G) ([1..n])×Pω(G) ([1..n]) , where
yC is a reduced version of yC that contains only Pω(G) ([1..n]) elements ySC = ySC with S ∈ Pω(G) ([1..n]).
Indeed, if Mω(G) (yC )I,J = 0, then I ∪ J * C using (4.6.5), and so, we have I * C or J * C, which means
that yIC = 0 or yJC = 0. Also, if Mω(G) (yC )I,J = 1, then we have I ∪ J ⊆ C by virtue of (4.6.5), which means
that I, J ⊆ C, and so, yIC = yJC = 1. This confirms that Mω(G) (yC ) = y C C
 y  0. 


λC yC we obtain Mω(G) (y) = Mω(G) λC y C = λC Mω(G) yC .
P P P
By applying (4.6.3) on y =
C∈C C∈C C∈C
As a sum of PSD matrices Mω(G) (yC ) resp. multiplied by positive scalars λC , the
Pmatrix Mω(G) (y)
P is SDP.
C
We now check the two non-SDP constraints of (4.6.4). First, we have y{i} = λC y{i} = λC =
C∈C C∈C ,C3i
1 ∀i ∈ [1..n], using (4.6.1a). Secondly, y{i,j} = 0 ∀{i, j} ∈/ E also holds because the non-edge {i, j} belongs
to no P
clique. We have just constructed a feasible solution y in (4.6.4) for r = ω(G) with objective value
λC y∅C = λC = χ∗ (G). This is enough to conclude ψ ω(G) (G) ≤ χ∗ (G).
P
y∅ =
C∈C C∈C
Parts of this proof are a simplification of the proof of Theorem 3.1. from the article “The operator Ψ for
the chromatic number of a graph” of Nebojša Gvozdenović and Monique Laurent.20

5 A taste of copositive optimization and sum of squares hierar-

chies
5.1 Introducing the completely positive and the copositive cones
Let us try to produce better relaxations and reformulations by replacing the cone Sn+ of SDP matrices with
a smaller cone. For this, we introduce the cone of completely positive matrices:
( k
)
X
C n∗ = X ∈ Sn : X = yi yi> with yi ≥ 0 ∀i ∈ [1..k] (5.1.1)
i=1
= conv yy> : y ≥ 0

, (5.1.2)
where Sn ( Rn×n is the set of real symmetric matrices and the operator conv(...) produces all convex
combinations of the elements from the set given as argument. It is clear that any X from (5.1.1) can be
written as a convex combination of rank-1 matrices as in (5.1.2). For this, it is enough to write X =
Pk >
Pk 1 √  √ >
i=1 y i y i = i=1 k kyi kyi . Any X ∈ C n∗ can be written X = Y Y > with Y = [y1 y2 . . . yk ],
and so, it is clear that X is also SDP using Prop. A.1.8. The difference between a completely positive matrix
X and an SDP one is that above Y needs to be non-negative. Any SDP matrix X e  0 can be written as
>
X = Y Y using various decompositions (e.g., Cholesky, eigenvalue or square root, see Corollary 1.7.1), but
e e e
Ye is not necessarily non-negative. For any n ≥ 2, we have C n∗ ( Sn+ .
We can define the dual of a cone C (with regards to the scalar product) using the formula C ∗ =
·
{Y : X Y ≥ 0 ∀X ∈ C}. We already proved in Prop. 1.3.3 that the SDP cone is self-dual, i.e., (Sn+ )∗ = Sn+ .
Since C n∗ is smaller than Sn∗ , its dual cone might be larger than Sn+ . Indeed, we introduce the cone of copos-
itive matrices n
·
C n = X ∈ Sn : X yy> ≥ 0 ∀y ≥ 0
o
(5.1.3)
20 Published in SIAM Journal on Optimization in 2008, vol 19(2), pp. 572–591, available on-line as of 2017 at http:

//[Link]/~monique/files/[Link].

48
such that C n = (C n∗ )∗ .
Let N n ( Sn be the set of non-negative symmetric matrices. The following hierarchy of inclusions

C n∗ ⊂ Sn+ ∩ N n ⊂ Sn+ ⊂ Sn+ + N n ⊂ C n (5.1.4)

∗
follows from two short arguments. First, any X ∈ C n satisfies X  0 (see above) and X ≥ 0 (see the
·
definition (5.1.1)). Secondly, any S  0 and N ≥ 0 verify S yy> ≥ 0 and N yy> ≥ 0 for any y ≥ 0. ·
5.2 Reformulating a homogeneous quadratic program as a copositive problem
We consider the following problem with a homogeneous objective function and non-negative variables:

min Q xx


(QP+ )
>

>
 s.t a x = b
· (5.2.1a)
(5.2.1b)
x ∈ Rn+ ,

(5.2.1c)


where a > 0 is a strictly positive parameter. One should keep in mind that x is non-negative.

5.2.1 Solving (QP+ ) is NP-hard

Not surprisingly, solving this program is NP hard. This follows from the fact that it contains the maximum
stable problem as a particular case,21 as a consequence of the following result.
Proposition 5.2.1. Consider a graph G with adjacency matrix AG . The maximum stable α(G) can be
determined by solving the following program, which is a particular case of (QP+ ) from above (5.2.1a)-(5.2.1c).

1
α(G)
n
= min AG + In


· xx> : e>
o
nx = 1 , (5.2.2)

where e>
n = [1 1 1 . . . 1].
| {z }
n positions

Proof. We use the technique described next. Consider any feasible solution x of above program (5.2.2). We
say that an edge {i, j} is supported by x if xi , xj > 0. We will show that if x has more than zero supported
edges, then it be transformed into a no-worse solution x0 that has strictly fewer supported edges. For this,
let us start by evaluating the contribution of xi and xj to the objective value of edge {i, j}:
X X
val(xi , xj ) = x2i + 2xi xj + x2j + 2 AG
ki xk xi + 2 AG
kj xk xj (5.2.3)
k∈[1..n]−{i,j} k∈[1..n]−{i,j}

= (xi + xj )2 + zi xi + zj xj , (5.2.4)

where the values zi and zj are determined by the above sums and do not depend on xi or xj . Consider now
the function f : [0, xi + xj ] → R defined by f (t) = val(t, xi + xj − t), i.e., we replace xi with t and xj with
xi + xj − t in above (5.2.3)-(5.2.4). This way f (t) can be written

f (t) = (xi + xj )2 + zi t + zj (xi + xj − t) = (zi − zj )t + (xi + xj )2 + zj (xi + xj )

The only-non constant term is (zi − zj )t. We obtain that f is a linear function that reaches its minimum
value at one of the

two bounds of [0, x
i + xj ], i.e., a value t that minimizes f is either t = 0 or t = xi + xj .
We can say f t = val t, xi + xj − t is at least as good as f (xi ) = val(xi , xj ).
If we define x0i = t, x0j = xi + xj − t and x0k = xk ∀k ∈ [1..n] − {i, j}, we obtain that x0 is at least as
good as x. Notice that the edge {i, j} is no longer supported in x0 because x0i or x0j is equal to zero. Any
other supported edge in x0 is also supported in x, because x0` > 0 =⇒ x` > 0 ∀` ∈ [1..n]. This means that
21 If we allow the objective to be non-homogeneous, we obtain a particular case of unconstrained quadratic programming in

non-negative variables. This problem is NP-hard because it is at least as hard as the bi-partition problem, i.e., one can solve
1 P

2
the (bi-)partition problem for elements a1 , a2 , . . . an by solving minxi ,x0 ≥0 i (xi + x0i − 1)2 + xi x0i +
P P
i i xi ai − 2 i ai .
More generally, unconstrained quadratic programming is not NP-hard because it reduces to SDP programming (Section 3.4.3).

49
x0 has at least one supported edge less than x. By repeating above operation iteratively for all supported
edges, we will eventually find some x0 that is at least as good as x and has no supported edge.
This means that the non-zero elements of x0 generate a stable S of G. The objective value of x0 can

e>
·
be written I|S| x x> , where x is a vector containing only the |S| non-zero values of x0 . We can still say
0
|S| x = 1, i.e., the sum of the elements of x (or x ) is one. We now show that all elements of x are equal if
0
x is optimal. Assume the contrary: there is i, j ∈ [1..|S|] such that xi 6= xj . The contribution of xi and xj
 2
x +x
to the objective function is val(xi , xj ) = x2i + x2j = 12 2x2i + 2x2j > 12 x2i + x2j + 2xi xj = 2 i 2 j . By



 
x +x
replacing xi and xj with i 2 j we obtain a better objective value, while still respecting the constraint (the
sum of the elements remains the same). This means that x0 is not optimal, which is a contradiction. As such,
P|S| 1 |S|
the optimal x0 has the same value |S| 1
at all positions i ∈ S and its objective value is i=1 |S| 1
2 = |S|2 = |S| .

We started from an arbitrary solution x and we constructed a solution x0 that is no worse than x and
that has no supported edge; this means there is always an optimal solution with no supported edges. We
then showed that if x0 is optimal, it has to contain the value |S| 1
on all positions of some stable S of G; the
1
objective value of this solution is |S| . But this value can only be optimal if S is a maximum stable of G with
1
|S| = α(G). We obtained that the optimum of the program from (5.2.2) is indeed α(G) . It is achieved by
1
taking xi = |S| for all i ∈ S and xi = 0 if i ∈ / S for a maximum stable S.

According to the article “Copositive Optimization” by Immanuel Bomze, Mirjam Dür and Chung-Piaw
Teo,22 the above result dates back to the 1960s. However, the proof is personal.

5.2.2 The reformulation of (QP+ ) as a copositive program

Let us consider the following program associated to (QP+ ) from (5.2.1a)-(5.2.1c).

min Q X
· (5.2.5a)

· ·



C(QP+ ) s.t A X = (aa> ) X = b2 (5.2.5b)


X ∈ C n∗ ,

(5.2.5c)


where recall A = aa> respects A ≥ 0 and Aii > 0 ∀i ∈ [1..n] because (5.2.1a)-(5.2.1c) imposes ai > 0∀i ∈
[1..n], i.e., a is a strictly positive parameter. Since the above program minimizes a linear function, its
optimum might only be achieved by an extreme point (or along an extreme ray) of the feasible area defined
by (5.2.5b)-(5.2.5c).
Proposition 5.2.2. We consider a symmetric matrix A ≥ 0 with a strictly positive diagonal (Aii > 0 ∀i ∈
[1..n]) and some b ∈ R. The extreme solutions (vertices) of the set below are the rank-1 matrices of the form
X = uu> with u ≥ 0. This set has no extreme rays.

[CAn∗
n
] = X ∈ C n∗ : A X = b2
· o
(5.2.6)

Proof. We will prove three facts:

(i) A rank-1 completely positive matrix X such that AX = b2 is an extreme solution of [CA
n∗
].
n∗
(ii) A completely positive matrix X of rank higher than one can not be an extreme solution of [CA ].
n∗
(iii) The set [CA ] has no extreme ray.
(i)
We first prove that a rank-1 matrix X = yy> (with y ≥ 0) such that AX = b2 is an extreme solution of
n∗
this [CA ] set. Assume the contrary: there is symmetric non-zero M ∈ Rn×n such that X − M, X + M ∈
[CA ]. Based on (5.1.1), we can write X − M = Ya Ya> and
X + M = Yb Yb> ; combining the two, we
n∗

obtain X = 12 [Ya Yb ][Ya Yb ]> . But since rank(X) = rank yy> = 1, we can apply Prop. A.1.8 to conclude
22 Published in the Optima 89 newsletter in august 2012, pp. 2-8, available on line at [Link]

Optima-Issues/[Link].

50
rank(X) = rank [Ya Yb ][Ya Yb ]> = rank([Ya Yb ]), and so, we have rank([Ya Yb ]) = 1. The columns of Ya and

Yb need to be multiples of y. The matrices X and X − M are multiples of yy> , and so, M is a multiple of
n∗
X, i.e., M = tX with t 6= 0. This contradicts X − M ∈ [CA ] because A (X − M ) = b2 − tb2 6= b2 . The
n∗
assumption X − M, X + M ∈ [CA ] for some M 6= 0 was false, and so, X is an extreme solution.
(ii)
We now show that any matrix X ∈ C n∗ of rank higher than 1 can not be an extreme solution of [CA n∗
].
Assume there exists k non-zero vectors y1 , y2 , . . . yk ≥ 0 such that rank[y1 y2 . . . yk ] > 1 (i.e., they are not
Pk
all multiples of the same vector) and X = i=1 yi yi> , recall definition (5.1.1). Without loss of generality,
we suppose y1 and y2 are linearly independent. We can write:

A
· ·
X=A
|
y1 y1>
{z
t1
}
+A
| ·{z
t2
y2 y2>
}
+A
· k
X

i=3
yi yi> . (5.2.7)

· ·
Notice we have t1 , t2 > 0, because A y1 y1> ≥ diag(A) diag(y1 y1> ) > 0 follows from the fact that A and
y1 are non-negative, the diagonal of A is strictly positive and y1 is non-null. An analogous argument proves
t2 > 0. We now introduce the following family of matrices depending on a parameter α.
 
1 1
Xα = X + α y1 y1> − y2 y2> (5.2.8)
t1 t2
| {z }
6=0 because y1 and
y2 are independent

Notice Xα remains completely positive for a sufficiently small positive or negative α — more exactly the
n∗
limits of α are α ∈ [−t1 , t2 ]. Let us check if Xα ∈ [CA ] by calculating the scalar product with A:

· ·
   
1 > 1 > t1 t2
AXα = AX + α A y1 yi − A y2 y2 = AX + α − = AX
t1 t2 t1 t2
 
We can move from X back and forward along non-zero t11 y1 yi> − t12 y2 y2> until we reach (the above) limits
n∗
of α. Thus, such X can not be an extreme solution of [CA ].
(iii)
n∗
We finally show (by contradiction) that [CA ] can not contain an extreme ray of the form X + tZ with t > 0
and Z 6= 0n×n . Assume such Z exists. We can easily notice Z ≥ 0 because C n∗ ⊂ Rn+ . Since A Z has to
be zero and the diagonal of A is strictly positive, we also obtain diag(Z) = 0, which means Z is not SDP
·
·
by applying Corollary 1.6.5. There exists a vector u such that Z uu> = −z < 0. But now notice that
· ·
(X + tZ) uu> = X uu> − tz can become negative for a sufficiently large t, which means X + tZ is not
SDP. This is a contradiction because C n∗ ⊂ Sn+ in (5.1.4).
The above Prop. 5.2.2 leads to the fact that the optimal solution of (C(QP+ )) from (5.2.5a)-(5.2.5c) has
the form X = yy> with y ≥ 0. This means y is an optimal solution of (QP+ ) from (5.2.1a)-(5.2.1c). Indeed,
if (QP+ ) would have a solution y of better quality, X = y y> would also be a solution of better quality than
X in (C(QP+ )). This means (C(QP+ )) is an exact reformulation of (QP+ ).
The difficulty of (C(QP+ )) is hidden in the cone constraint. Indeed, checking membership in C n∗ is
NP-hard. In particular, if we try to factorise X ∈ C n∗ into X = Y Y > using any of the decompositions
presented for SDP matrices (e.g., Cholesky, eigenvalue or square root, see Corollary 1.7.1), the factor Y
is not necessarily non-negative. It is still an open question whether checking C n∗ membership is in NP
(making the problem NP-complete, because it is NP-Hard) or not. Checking membership in the dual cone
C n is even co-NP complete.23 Under a legitimate well-accepted (but still open) assumption co-NP6= NP, a
co-NP complete problem can not belong to NP – if that were the case, all co-NP problems would belong to
NP, which is unlikely. This way, it is very likely that checking C n membership is not even in NP. For more
details on such aspects, we refer the reader to the article “On the computational complexity of membership
problems for the completely positive cone and its dual” by Peter Dickinson and Luuk Gijben.24
23 Recall a decision problem is NP (reps. co-NP) if and only if there is a polynomial-time algorithm that can verify “yes”
(resp. “no”) instances.
24 Published in Computational optimization and applications in 2014, vol 57(2), pp. 403–415, available on-line at http:

//[Link]/DB_FILE/2011/05/[Link].

51
5.2.3 Comparing with the SDP relaxation of (QP+ )
We now investigate the reasons why replacing C n∗ with Sn+ does not lead to such a strong result (reformula-
n∗
tion). In other words, we replace the completely positive (A-)constrained set [CA ] from (5.2.6) with an SDP
set [SDPA ] using the same constraint defined by A. More exactly, we investigate why the characterization
n∗
of extreme solutions from Prop. 5.2.2 of [CA ] does not hold in the same way for
n
[SDPA ] = X  0 : A X = b2 .
· o
(5.2.9)

·
First, we can still say that the rank-1 matrices X  0 such that A X = b2 are extreme solutions of
[SDPA ]. It is enough to check that the arguments for point (i) from the proof of Prop. 5.2.2 still hold for
[SDPA ]. However, one should bear in mind that a rank-1 SDP matrix X = yy> might not verify y ≥ 0,
and so, y might not be a feasible solution of the initial program [QP+ ] from (5.2.5a)-(5.2.5c).
Secondly, the SDP set might have extreme rays and the proof of (iii) from Prop. 5.2.2 fails in the SDP
case. This follows from the fact that even if A ≥ 0, we can still find SDP matrices Z such that A Z = 0. ·
·
This means there might well be matrices X + tZ ∈ [SDPA ], ∀t > 0, based on A (A + tZ) = A X = b2 . If ·
· ·
∃Z  0 such that A Z = 0 and Q Z < 0, the SDP relaxation of [QP+ ] from (5.2.5a)-(5.2.5c) is unbounded.
·
We will assume that this relaxation is not unbounded, i.e., A Z = 0 =⇒ Q Z ≥ 0. ·
Finally, we tackle the point (iii) of the proof of Prop. 5.2.2. We can still prove there is no extreme solution
of [SDPA ] with rank higher than 1. For this, we can still write (5.2.7) with independent y1 and y2 . However,
we can no longer state t1 , t2 > 0. If we have t1 = 0, then y1 y1> is a ray of [SDPA ] and it is clear X is not an
Pk
extreme point because we can add or subtract multiples of y1 y1> from the description X = i=1 yi yi> and
remain in [SDPA ]. The same happens if t2 = 0. We can hereafter assume t1 6= 0 and t2 6= 0. This way, we
can still construct the family Xα of matrices from (5.2.8). One can check that Xα remains in the [SDPA ]
for sufficiently small values of α, i.e., check that if α ∈ [−ε, +ε] with ε < min(|t1 |, |t2 |), then the coefficients
of y1 y1> and y2 y2> in the sum composing Xα remain strictly positive. This is enough to guarantee that X
is not an extreme solution.
To summarize, we have found two differences between the completely positive (re-)formulation and the
SDP relaxation. First, the feasible area in the SDP case can have extreme rays and the objective is unbounded
· ·
if there is Z  0 such that A Z = 0 and Q Z < 0. Secondly, it the objective is not unbounded, the optimal
solution has rank 1 like in the completely positive case, but it is not necessarily non-negative.

5.3 Relaxations of the copositive formulation of the maximum stable

5.3.1 A second completely positive formulation of the maximum stable
We have already introduced the α(G) formulation (5.2.2) and proven it in Prop. 5.2.1. We will show that

where A G
n
α(G) = max en e> n X :
· A G
+ I n

is the adjacency matrix of graph G and recall

X

e>
n
= 1,
·X ∈ C n∗
o
,

= [1 1 1 . . . 1].
(5.3.1)

| {z }
n positions
First, notice this program is very similar to C(QP+ ) from (5.2.5a)- (5.2.5c). In particular, the matrix
AG + In satisfies exactly all conditions imposed on A in Prop. 5.2.2. As such, an optimal solution of (5.3.1)
can be achieved by an extreme solution of the feasible area that has the form X = yy> with y ≥ 0 by virtue
of Prop. 5.2.2. It is enough to prove the following:
n
α(G) = max en e> n ·
yy >
: A G
+ In


yy >
=
·
1
o
, (5.3.2)
√
√2


G
·
Consider a feasible solution x of (5.2.2) with objective value A + In xx> = 1t . Let us define y = tx

√


·

√2·
and one can calculate AG + In yy> = t AG + In xx> = t 1t = 1, i.e., y is feasible in (5.3.2). Based
on e> >
n x = 1, we obtain en y = t, and so, en e>
n ·
yy> = t = t. From a feasible solution x of (5.2.2) with
value 1t , we constructed a feasible solution in (5.3.2) with objective value t. The converse is also possible.
Consider any feasible solution y of (5.3.2) of objective value (e> 2
n y) = t (notice any feasible y is non-zero,
 2
y > > >
and so, t > 0). Let us define x = √t . From (en y) = t, we have (en x) = √t en y = √1 2 t = 1, i.e., x is
2 2 1
t


·
feasible in (5.2.2). The objective value of x is AG + In xx> = √1 2 AG + In yy> = 1t .
t


·
52
 From a feasible solution of (5.2.2) with objective value 1t we can construct a feasible solution of (5.3.2)
with objective value t. Conversely, from a feasible solution of (5.3.2) with objective value t we can construct
a feasible solution of (5.2.2) with objective value 1t . This is enough to guarantee that the optimum of (5.3.2)
is 1 divided by the optimum of (5.2.2), i.e., it is 11 = α(G).
α(G)

5.3.2 Sum-of-squares relaxations of the copositive formulation of α(G)

[Link] The copositive formulation of α(G)
The dual of (5.3.1) can be calculated as in the SDP case, see also the description of primal-dual conic
programs from Section 7.2.2. We can apply the technique used in the proof of Prop. 2.1.5, but we need the
dual cone of C n∗ , i.e., C n . The dual of (5.3.1) can thus be written as follows:

α(G) = min t : t AG + In − en e> n



n ∈C . (5.3.3)
It is possible to show that both (5.3.1) and (5.3.3) are strictly feasible. Let us start with (5.3.1) and


·
notice that if Y ∈ interior (C n∗ ), then diag(Y ) 6= 0 and AG + In Y ≥ diag(Y ) diag(In ) > 0. As
such, we can define X = (AG +IYn ) Y that also belongs to interior (C n∗ ) and is feasible in (5.3.1). We
·
·
only need to show that C n∗ has a non-empty interior, i.e., we have to generate some strictly feasible matrix
of C n∗ . For this, consider the set A(0,1) = {nIn + A ∈ Sn : 0 < Aij < 1}. For any nIn + A ∈ A(0,1) and
M ∈ Sn , there is a sufficiently small ε > 0 such that nIn + A − εM, nIn + A + εM ∈ A(0,1) . Because of this
last property, to prove A(0,1) ( interior (C n∗ ) it is now enough to show A(0,1) ( C n∗ . We can write
 
X X X
A(0,1) 3 nIn + A = Aij E ij + n + Aii − Aij  E ii , (5.3.4)
i<j i∈[1..n] j6=i

where E ij is a matrix full of zeros except at positions (i, i), (i, j), (j, i) and (j, j) where it has ones. All
> ij
terms in above (5.3.4) can be written under the form yy  for a non-negative  y. For Aij E , it is enough to
take a y full of zeros except for yi = yj = Aij . For n + Aii − j6=i Aij E ii , we take a y full of zeros,
p P
q P p
except for yi = n + Aii − j6=i Aij > n + Aii − (n − 1) > 0. This allows one to write nIn + A in the
form required by the C n∗ definition (5.1.1).
We now show
(5.3.3) has
strictly feasible solutions. It is enough to show there is a sufficiently large t such
that t AG
+ In − en e> n yy >
>· 0 for any non-negative y ≥ 0. Using t A G
+ I n


− en e>
n


yy >
·
≥
>
tIn − en en · >
yy , it suffices to show that this last scalar product is strictly positive if t is large enough.
But this simply follows from the fact that the matrix tIn − en e> n becomes diagonally dominant and can have
arbitrarily large eigenvalues when t → ∞, i.e., we can easily have tIn − en e> n  0.
Since both (5.3.1) and (5.3.3) are strictly feasible, we can apply the strong duality Theorem 7.2.8 for
linear conic programming that states that both programs have the same optimum value and they effectively
reach this value.

[Link] A “natural” strengthening bounded by the Lovász number

There are several cone hierarchies and relaxations used to approximate the copositive cone C n . Let us
first recall hierarchy (5.1.4) and define a “natural” strengthening (more constrained restricted version) that
replaces C n with Sn+ + N n , where N n ( Sn is the set of non-negative symmetric matrices. The strengthening
can simply be written by modifying (5.3.3):

α0 (G) = min t : t AG + In − en e> + n



n ∈ Sn + N [≥ α(G)] . (5.3.5)
Since the dual cone of Sn+ +N is n
Sn+ n 25
∩ N , the dual can be written as follows:

25 To +
see this, simply notice Sn ( Sn +
n
α0 (G) = max en e>
n X : A G
·

+
+
+ N n , and so, Sn
I n

+ Nn
X
∗
= 1, X
+ ∗
( (Sn
∈ S
+
) = Sn
·
n
+
∩ N n
o
.

+
. Similarly, Sn + Nn
∗
(5.3.6)

( (N n )∗ = N n .
 ∗
+ + + +
As such, Sn + Nn ⊂ Sn ∩ N n . One can check that X ∈ Sn ∩ N n and Y = Y1 + Y2 with Y1 ∈ Sn and Y2 ∈ N n yield
·
X (Y1 + Y2 ) = X ·Y 1 +X ·Y 2 ≥ 0, using the fact that X is both SDP and non-negative.

53
 The first program (5.3.5) is strictly feasible by taking a sufficiently large t, so that t AG + In − en e>


n =
> G > > G
G


tA and t A + In − en en ± εM = tIn − en en ± εM + |{z}
tIn − en en + |{z} tA using a sufficiently small ε > 0
| {z } | {z }
0 ≥0 0 ≥0
tIn +en e>
for any M ∈ Sn . The dual (5.3.6) is strictly feasible taking X = for a sufficiently large
n
·
(AG +In ) (tIn +en e>
n)
t. We can apply the strong duality Theorem 7.2.8 for linear conic programming to state that both programs
have the same optimum value α0 (G) and they effectively reach this value.
We now prove the following:
n X o
α0 (G) = min t : tIn + tij Eij − en e> +
n ∈ Sn + N
n
, (5.3.7)
{i,j}∈E:i<j

where tij are decision variables and Eij is a matrix that contains a value of one at positions (i, j) and (j, i)
and zeros everywhere else. This is a relaxation of (5.3.5) because it lifts the constraints tij = t ∀{i, j} ∈ E.
However, we can show that any feasible solution of (5.3.7) with objective value t can be transformed into a
feasible solution of (5.3.5) with the same objective value t.
Take any edge {i, j} ∈ E such that tij 6= t. If tij < t, one can simply increase tij to t and remain feasible
because the resulting matrix is the old matrix plus a non-negative increase that belongs to N n . If tij > t,
let us decrease tij to the minimum value t that keeps the resulting matrix in Sn+ + N n . Let us focus on
the 2 × 2 minor corresponding to i and j. If t > t, this minor is not SDP, and so, a decrease from t down
towards t would only represent a decrease of the N n component the matrix in Sn+ + N n . The only possible
case that can forbid any decrease below t is t = t.
As a relaxation of (5.3.5), the new program (5.3.7) remains strictly feasible. This means that the following
dual does achieve the optimum value α0 (G).

n
α0 (G) = max en e>
n · ·
X: AG + I n


X = 1, Xij = 0 ∀{i, j} ∈ E, X ∈ Sn+ ∩ N n
o
(5.3.8)
n
= max en e>
n · ·X : In
o
X = 1, Xij = 0 ∀{i, j} ∈ E, X  0, X ≥ 0 . (5.3.9)

It is clear that above (5.3.9) without the non-negativity constraint X ≥ 0 is equivalent to the (DϑG ) for-
mulation from (4.1.3a)-(4.1.3d). As such, α0 (G) ≤ OP T (DϑG ) = ϑ(G). This “natural” strengthening of
the copositive formulation of the maximum stable is bounded by the Lovász theta number. Since we have
α(G) ≤ α0 (G) in (5.3.5), we can write α(G) ≤ α0 (G) ≤ ϑ(G).

[Link] The sum of squares (SOS) hierarchy

This is the only subsection where I will use a few results without an explicit proof. In all cases, I will leave
a comment on the margin of the document.
This subsection is devoted to the SOS approach due to Parilo, De Klerk and Pasechnik as cited in the
“Copositive Optimization” article of the Optima 89 newsletter (see Footnote 22, p. 50) or in the article
“Semidefinite Bounds for the Stability Number of a Graph via Sums of Squares of Polynomials” by Nebojša
Gvozdenović and Monique Laurent.26

The general SOS setting

P 2 2 n
·
First, notice that M ∈ C n (recall Def. (5.1.3)) ⇐⇒ pM (x) = M [x21 x22 . . . x2n ]> [x21 x22 . . . x2n ] =
n
i,j=1 Mij xi xj ≥ 0 ∀x ∈ R . This means that the polynomial pM (x) takes only non-negative values
over the reals. We can start using results related to non-negative polynomials. There are at least two condi-
tions that guarantee that a polynomial is always non-negative: (i) all coefficients are positive and all powers
of the variables are even, (ii) it can be written as a sum of squares (SOS). One of the earliest work on these
aspects dates back to Hilbert who studied the classes of non-negative polynomials that can always be writ-
ten as an SOS. He proved, for instance, that non-negative univariate polynomials (n = 1) and non-negative
polynomials of degree 2 can always be decomposed into a SOS, see also Hilbert’s 17th problem. However,
even if a non-negative polynomial does not respect any of above conditions (i) or (ii), we could multiply it
by another non-negative polynomial and obtain a product that does respect (i) or (ii).
26 Published in Mathematical Programming in 2007, vol 110(1), pp 145–173, available on-line as of 2017 at [Link]

nl/oai/asset/11672/[Link].

54
 Applying this last argument, we notice that pM (x) is non-negative if and only if

n
!r
(r)
X
pM = x2i pM (x) (5.3.10)
i=1

is non-negative. Pólya proved in the 1970s the following theorem.

Theorem 5.3.1. If pM (x) is a homogeneous polynomial (with all terms of the same degree) that is strictly
(r)
positive over Rn − {0}, then pM has only non-negative coefficients for a sufficiently large r.

·
Missing:
(r)
Considering our pM above of the form pM (x) = M [x21 x22 . . . x2n ]> [x21 x22 . . . x2n ], we notice that pM a proof for
(r) the result of
accepts an obvious SOS decomposition for a sufficiently large r, i.e., all terms of pM are squares multiplied Pólya (Theo-
rem 5.3.1).
by a positive value. We now can define the cone:

Kn(r) = {M ∈ Sn : p(r)
M can be written as an SOS}. (5.3.11)

Using Theorem 5.3.1, we have interior(C n ) = Kn(r) . We also have Kn(i) ⊆ Kn(i+1) because switching
S
r≥1
Pn
from i to i + 1 is equivalent to multiplying an SOS form with i=1 x2i , which develops into a sum of squared
(r)
(and thus SOS) terms. Replacing C n with Kn in (5.3.3), we obtain the following strengthening (restriction)
of (5.3.3): n o
α(i) (G) = min t : t AG + In − en e> (i)


n ∈ Kn (5.3.12)

and we naturally have

α(0) (G) ≥ α(1) (G) ≥ α(2) (G) · · · ≥ α(r) (G) → α(G),
(i) (i+1) (r)
using Kn ⊆ Kn , with the convergence following from interior(C n ) = lim Kn . As such, we easily
 (r)  r→∞
have α(G) = α (G) for a sufficiently large r. A sufficient value of r is r = α(G)2 , as proved in Theorem
4.1. of the article “Approximation of the stability number of a graph via copositive programming” by Etienne Missing: the
de Klerk and Dmitri Pasechnik.27 It is conjectured that α(r) (G) = α(G) for r ≥ α(G)−1, see also Conjecture proof for the
sufficiency of
1 in the article indicated at Footnote 26 (p. 54). r = α(G)2 .
(i)
We will show that optimizing over some Kn might be easier than optimizing over C n , particularly
for bounded values of i. While checking membership in C n is NP-hard and probably not NP (assuming
(i)
NP6=co-NP,
paragraph of Section 5.2.2), checking membership in Kn requires solving an SDP with
see last
n+i+1

n+i+1
i+2 × i+2 variables, as we will see later in Remark 5.3.3. More generally, we will show one can deter-
mine an SOS decomposition of any polynomial of degree 2d by solving and SDP with at most n+d × n+d



d d
variables, using results from the Phd thesis “Structured Semidefinite Programs and Semialgebraic Geometry
Methods in Robustness and Optimization” by Pablo Parrilo.28
(0)
Characterizing Kn
(0)
Let us start with Kn and we will show that α(0) (G) is actually equal to the natural strengthening α0 (G)
from (5.3.5) which is bounded by the Lovász theta number ϑ(G) as described in Subsection [Link]. However,
let us first present a small example.
(0)
Example 5.3.2. Consider pM (x) = pM (x) = x41 + x42 + 3(x1 x2 )2 that accepts multiple SOS decompositions,
e.g., (x21 + x22 )2 + (x1 x2 )2 or (x21 )2 + (x22 )2 + (x1 x2 )2 . Such decompositions can be determined by solving an
SDP with a null objective function (SDP feasibility problem).
Proof. We will later show (see Remark 5.3.3) that for such polynomial, the only terms that can appear in the
squared expressions are x21 , x22 and x1 x2 . Let us define x> = [x21 , x22 , x1 x2 ] and write pM (x) = M x x> ,
with M ∈ S3 .
·
27 Published in SIAM journal on optimization in 2002, vol 12(4), pp. 875–892, available on-line as of 2017 at [Link]

[Link]/handle/10220/6790.
28 Defended in 2000 at California Institute of Technology, available on-line as of 2017 at [Link] parrilo/
~
pubs/files/[Link].

55
 ·
Step 1 This last equality pM (x) = M x x> does not allow one to uniquely identify M . This comes from
the fact that (x1 x2 )2 can be written both as a square of x1 x2 (involving a non-zero M 33 in this last equality)


and as a product of x21 and x22 (involving M 12 ). We can write pM (x) = pM (x) + λ12 −x21 x22 + (x1 x2 )2 .
This way, M can take the form
3−λ12
 
1 2 0
M λ12 =  3−λ2
12
1 0 .
0 0 λ12
Pk Pk
·
Step 2 Assume there is an SOS decomposition pM (x) = i=1 (vi x)2 = i=1 vi vi> x x> = V V > x x> .
This can only happen if M = V V > is SDP (see also Prop. A.1.8). The above decomposition (x21 + x22 )2 +
· ·
(x1 x2 )2 corresponds to λ12 = 1. We can find a decomposition by solving an SDP program M λ12  0 with
null objective function, i.e., we only want to know if there exists feasible M λ12  0.
Both steps can generate infinitely-many SOS decompositions. At Step 1, there is a feasible matrix M λ12
for each λ12 ∈ R. At Step 2, any SDP matrix M λ12 accepts infinitely many factorizations M = V V > , as
stated in Corollary 1.7.1.

Let us return to general case of the polynomial pM (x) of (homogeneous) degree 2d = 4, i.e., all terms
have degree d = 4. The squared expressions contain no free terms because pM (x) has no free term. They
can neither contain terms of degree d − 1 = 1 like xi because there is no way other squared expression
cancel some x2i . We define x to contain all monomials of degree d = 2 and we attempt to write pM (x) =
· Pk
M x x> . As already stated in the proof of above example, an SOS decomposition pM (x) = i=1 (vi x)2 = ·
Pk >
i=1 vi vi · ·
x x> = V V > x x> exists if and only there is some SDP M = V V > such that pM (x) =
· ·
M x x . All expressions of the form pM (x) = M x x> can be found by writing
>

X
λij −(x2i )(x2j ) + (xi xj )2


pM (x) = pM (x) +
i<j
X
µijj 0 (xi xj )(xi xj 0 ) − (x2i )(xj xj 0 )


+
j<j 0
i6=j,i6=j 0
X
+ µii0 jj 0 ((xi xj )(xi0 xj 0 ) − (xi xi0 )(xj xj 0 )) ,
i0 6=j 0 ,j6=i0 ,j6=j 0
i<j,i<i0 ,i<j 0

where “i < j, i < i0 , i < j 0 ” in the last sum comes from the fact that (i) permuting i and j is equivalent to
permuting i0 and j 0 , (ii) permuting i and i0 is equivalent to permuting j and j 0 , and (iii) permuting i and j 0
is equivalent to permuting i0 and j. A feasible M can take the following form:
 
M11 M12 − λ12 . . . M1n − λ1n ∗ ∗ ... ∗
 M12 − λ12 M22 . . . M2n − λ2n ∗ ∗ ... ∗ 
.. .. . . . .
 
 . . . . . . . . 

 . . . . . . . . 

Mn1 − λn1 Mn2 − λn2 . . . Mnn ∗ ∗ ... ∗ 
M λ,µ =  , (5.3.13)

 ∗ ∗ ... ∗ 2λ12 ∗ ... ∗ 


 ∗ ∗ ... ∗ ∗ 2λ13 . . . ∗ 

 .. .. .. .
.. .
.. .
.. .. .
..

 . . . . 
∗ ∗ ... ∗ ∗ ∗ ... 2λ(n−1),n

where the asterisks stand for null terms or linear combinations of the µ variables. However, if M λ,µ  0
for some µ 6= 0, then M λ,0  0, because any minor of M λ,0 (replace above asterisks with zeros) can be
seen as a diagonal of blocks that also appear in the minors of M λ,µ if µ 6= 0. All these blocks need to have
a non-negative determinant in both matrices. As such, we can find an SOS decomposition if and only if
M λ,0  0.
We now investigate the relation between the above M λ,0 and the initial matrix M that defines pM (x) =
·
M [x21 x22 . . . x2n ]> [x21 x22 . . . x2n ]. We obtain relatively easily that if M λ,0  0, then M can be written as an
SDP matrix (the leading principal minor of size n × n of M λ,0  0) plus a non-negative matrix filled with
(0)
all the λij values. In other words, M ∈ Sn+ + N n , and so, Kn ⊆ Sn+ + N n .

56
 (0) 0
We can also show Sn+ + N n ⊆ Kn . This follows from the fact that if Sn+ + N n 3 M = [M ] + [λ]0 with
0
[M ]  0 and [λ]0 ≥ 0, we can also write M as the sum of an SDP matrix and a non-negative matrix [λ]
0
such that diag([λ]) = 0. For this, let us (re-)write M = [M ] + diag([λ]0 ) + [λ]0 − diag([λ]0 ) = [M ] + [λ]
| {z } | {z }
[M ]0 [λ]≥0
such that [M ]  0, [λ] ≥ 0 and [λ] has zeros on the diagonal. We can thus construct a matrix M λ,0  0 as
(0)
above from these two matrices [M ] and [λ]. This means pM (x) is an SOS, so that M ∈ Kn . We have just
(0) (0)
proved that Sn+ + N n ⊆ Kn so that Kn = Sn+ + N n .
We obtained that α(0) (G) and α0 (G) have the same feasible area. This means that α(0) (G) is the natural
strengthening α0 (G) from (5.3.5) bounded by the theta number ϑ(G) as described in Subsection [Link].
(i) (i)
Studying Kn and finding the SOS decomposition of pM
(i) (i) (i)
Recall the definitions (5.3.10) and (5.3.11) of pM and respectively Kn . We simply obtain that pM is a
polynomial of degree 2d = 2(i + 2). We can find an SOS decomposition of a polynomial of degree 2d
using the following approach already exemplified above. For most general polynomials of degree 2d, we
(i)
define a column vector x containing all monomials of degree less than or equal to d. We have that pM is
(i)
·
an SOS if and only if we can write it pM (x) = M x x> with M  0. We can write j=1 (vj x)2 =
P k
·
Pk
j=1 vj vj
>
· · (i)
x x> = V V > x x> if and only if there exists M = V V >  0 such that pM = M x x> . The ·
number of (independent) squared expressions is equal to the rank of V which is equal to the rank of M (see
Prop. A.1.8).
(i)
Checking membership in Kn reduces to checking SDP membership for a matrix M of the form (5.3.13)
but of much larger size and with more variables like λ or µ. For instance, a term like x2i · Mij (xi )2 (xj )2 =
(i)
Mij x4i xj of pM (x) leads to a constraint Mij = ab=x4 x2 M a,b , where M a,b indicates the M term corre-
P
i j
(i)
sponding to monomials a and b in M . An optimization problem over Kn can be transformed into an SDP
(i)
problem by replacing an initial constraint of the form M ∈ Kn into some M  0, but where the size of M
can be much larger than n × n, recall x contains all monomials of degree less than or equal to d.

[Link] The size of the SDP programs used for computing SOS decompositions
In the most general setting, the length of x is given by the number of monomials of degree less than or equal
to d. A classical combinatorics method known as “stars and bars” can determine this number as n+d


d ; it
relies on encoding any monomial using a string of stars and bars that arises quite out of the blue (e.g., x21 x3 x4
is ? ? || ? |?). However, I prefer to show it using a different argument that I personally find more natural.
Consider the following set {x1 , x2 , . . . xn , c1 , c2 , . . . cd } of cardinal n + d. Choosing some xi1 , xi2 , . . . xik
amounts to building an initial expression e = xi1 xi2 . . . xik (or e = 1 if no xi term is chosen). Then, choosing
some cj amounts to the following: (i) if the j th factor of e exists, copy (insert) it at position j + 1 or (ii) if
the j th factor of e does not exist (e is shorter), do not modify e. Under this interpretation, any monomial of
degree less than or equal to d corresponds to choosing d elements from {x1 , x2 , . . . xn , c1 , c2 , . . . cd }.
• x1 corresponds to choosing {x1 , c2 , c3 , . . . cd }. We start with expression e = x1 and the copying
elements c2 , c3 , . . . cd do not modify e because e has no factors at positions 2, 3, . . . d.
• x21 x2 corresponds to choosing {x1 , x2 , c1 , c4 , c5 , . . . cd }. Indeed, we first have the expression e = x1 x2 ;
then, c1 will insert a copy of x1 at position 2 to obtain e = x1 x1 x2 . The c4 element does not change e,
because e has no factor at position 4. Same applies to c5 , c6 . . . cd , and so, the final e is e = x1 x1 x2 .
• x5 x37 x9 corresponds to choosing {x5 , x7 , x9 , c2 , c3 , c6 , c7 , . . . cd }. We start with e = x5 x7 x9 and c2
inserts at position 3 a copy of the second element x7 , leading to e = x5 x7 x7 x9 . Then, c3 duplicates
the third element x7 , generating e = x5 x7 x7 x7 x9 . Elements c6 , c7 , . . . cd will perform no modification
on e because e has no factor at positions 6, 7, . . . d.
• xd7 corresponds to choosing {x7 , c1 , c2 , . . . cd−1 }. Indeed, we start with e = x7 and c1 duplicates
x7 leading to x7 x7 . Then c2 duplicates the second term, leading to x7 x7 x7 . Applying this for all
c1 , c2 , . . . cd−1 , we obtain that x7 is duplicated d − 1 times, and so, the final expression is x7 x7 . . . x7 .
| {z }
d times

57
 • x27 xd−2
9 corresponds to choosing {x7 , x9 , c1 , c3 , c4 , . . . cd−1 }. We start with e = x7 x9 and c1 duplicates
x7 leading to e = x7 x7 x9 . Since c2 is not chosen, x7 is not duplicated again. On the other hand, x9 is
duplicated in cascade d − 3 times and we obtain e = x7 x7 x9 x9 x9 . . . x9 .
| {z }
d−2 times

• 1 corresponds to choosing c1 , c2 , . . . cd .
(i)
Remark 5.3.3. The polynomial pM of degree 2d = 2(i + 2) also has the property that it contains only
monomials of degree 2d. In this case, we do not need to construct x using all monomials of degree less than
or equal to d, but we only need monomials of degree exactly d. The number of such monomials is
   
n+d−1 n+i+1
= .
d i+2
(i)
Proof. First, since pM has no free member, we can not use a squared expression with a free term, because no
other squared expression can cancel a free term. We can repeat this argument by induction. Let us take any
d0 < d; the induction basis is that there is no term of degree strictly smaller than d0 in any squared expression.
Now notice we can not have a squared expression of the form (e + x)2 with degree(x)=d0 , because we can
2
not cancel the monomial x2 . Indeed, such monomial could only be canceled by some (e + x0 − x00 ) with
x2 = x0 x00 and the induction hypothesis states there are no terms such as x0 or x00 of degree strictly smaller
than d0 in any squared expression. We proved by induction that there are no terms of degree d0 < d in our
squared expressions, i.e., x is constructed using only monomials of degree d.
Investigating in detail the combinatorial argument above this proposition, we notice that expressions of
degree exactly d are only constructed when cd is not chosen (hint: let c` be the first chosen among the ci ’s
and notice there are at least ` elements chosen among the xi ’s
because none of c1 , c2 , . . . c`−1 , cd is chosen).
The number of monomials of degree exactly d is thus n+d−1 d .
A final remark sometimes partially overlooked is that we took somehow for granted that we can not have
an SOS decomposition with some terms of degree d > d. However, this follows from the following argument.
Take the maximum degree P of a term in a squared expression and assume this value is d > d. Assume there is
2
an SOS decomposition (ei + fi ) , where the polynomials fi0 s contain all terms of degree
P 2 d. The monomials
of degree 2d are those resulting from (summing up) the terms fi2 . We need to have fi = 0, which means
any fi yields fi2 = 0, and so, fi = 0.

5.4 Further characterization of the completely positive and the copositive cones
We first recall below the following basic hierarchy from (5.1.4) of Section 5.1.

C n∗ ⊂ Sn+ ∩ N n ⊂ Sn+ ⊂ Sn+ + N n ⊂ C n , (5.4.1)

∗
We showed in Footnote 25 (p. 53) that (Sn+ + N n ) = Sn+ ∩ N n . We will now prove the converse:
∗
(Sn+ ∩ N n ) = Sn+ + N n . I could not easily find a direct proof using the properties of these matrices.
However, it is direct consequence of the following general proposition combined with the fact that Sn+ + N n
is closed (which follows from Prop. 2.3.1 and the fact that N n is closed). More exactly, property below
∗∗ ∗ ∗
enables us to derive the following implication: (Sn+ + N n ) = (Sn+ ∩ N n ) =⇒ Sn+ + N n = (Sn+ ∩ N n ) .

Proposition 5.4.1. Given any convex cone C, we have closure(C) = C ∗ ∗ .

Proof. We first show closure(C) ⊆ C ∗ ∗ . Assume the opposite: there is a sequence (ci ) with all ci ∈ C and
·
/ C ∗ ∗ . This means there is c∗ ∈ C ∗ such that c c∗ = z < 0. But since lim ci = c, there exists
lim ci = c ∈
·
i→∞ i→∞
some ci sufficiently close to c such that ci c∗ ∈ [z − ε, z + ε], for any ε > 0. For a sufficiently small ε,
·
this means ci c∗ < 0 which is a contradiction of c∗ ∈ C ∗ and ci ∈ C. As such, lim ci = c ∈
i→∞
/ C ∗ ∗ is false
=⇒ closure(C) ⊆ C ∗ ∗ .
We now show b ∈ / closure(C) =⇒ b ∈ / C ∗ ∗ . We apply the simple separation Theorem C.4.5 on
· ·
closure(C) and {b} to obtain there is a strictly separating inequality c c∗ > b c∗ ∀c ∈ closure(C).
· · ·
Since 0 ∈ C, we have 0 c∗ > b c∗ =⇒ b c∗ < 0. We now show c∗ ∈ C ∗ , i.e., c c∗ ≥ 0 ∀c ∈ C. ·
·
Assume ∃c ∈ C such that c c∗ < 0. Using the cone property, tc ∈ C for any arbitrarily large t. This means

58
 · ·
tc c∗ can be arbitrarily low, easily less than b c∗ , which is a contradiction. The assumption ∃c ∈ C such
· · ·
that c c∗ < 0 was false. All c ∈ C satisfy c c∗ ≥ 0. We produced c∗ ∈ C ∗ such that b c∗ < 0 for any b
outside closure(C).
Using the results from Section [Link], the above hierarchy (5.4.1) is refined to the hierarchy below.
(∞) (i)
Recalling Pólya’s Theorem 5.3.1, we have interior(C n ) = Kn = lim Kn .
i→∞

C n∗ ⊂ Sn+ ∩ N n ⊂ Sn+ ⊂ Sn+ + N n = Kn(0) ⊂ Kn(1) ⊂ Kn(2) ⊂ . . . Kn(∞) ⊂ C n , (5.4.2)

n Pn o
We can also build an outer approximation hierarchy for C n . Consider Nnr = z ∈ Nn : j=1 zi ≤ r
(r) 
(r)
· (r)
and define Pn = X ∈ Sn : X zz> ≥ 0 ∀z ∈ Nnr .29 Since Pn contains less constraints than C n , we
(r+1) (r)
simply obtain C n ⊂ Pn ∀r ∈ N. Notice also Nnr ( Nnr+1 , and so C n ⊂ Pn ⊂ Pn ∀r ∈ N. Writing
(∞) (r)
Pn = lim Pn , we will prove that
r→∞

C n = Pn(∞) ⊂ · · · ⊂ Pn(3) ⊂ Pn(2) ⊂ Pn(1)

We still have to show the first equality; the other inclusions were proved in above paragraph. We know
(∞) (∞) (∞)
C n ⊆ Pn , because C n contains more constraints than Pn . We only need to show that X ∈ Pn =⇒
(∞)
X ∈ C . Assume the opposite: ∃X ∈ Pn such that X ∈
n
/ C . This means there exists y ∈ Rn+ such
n

· ·
that X yy> < 0. Now we can construct a rational yq such that X yq yq> < 0. Such a yq can simply be
·
constructed by taking sufficiently many decimals of of y, enough to make X yq yq> as close as necessary to
·
X yy> . By multiplying rational zq with the least common denominator, we obtain an integer zi such that
·
X zi z> i < 0, which contradicts X ∈ Pn
The final remark
(∞)
. The assumption X ∈
(∞)
/ C n was false, and so, C n = Pn .
in the basic hierarchy (5.4.1) are strict. Indeed, Sn+ ∩ N n ( Sn+
 1 −1isthat +all inclusions
simply because −1 1 ∈ S2 − S2 ∩ N . It is also very easy to check [ 01 10 ] belongs to S2+ + N 2 , but not to
+ 2

S2+ . We can derive C n∗ ( Sn+ ∩ N n from the fact that C n∗ and Sn+ ∩ N n have disjoint dual cones C n and
resp. Sn+ + N n . Indeed, we can show Sn+ + N n ( C n using the so-called Horn-matrix.30
 
1 −1 1 1 −1
 −1 1 −1 1 1 
 
H=  1 −1 1 −1 1 

 1 1 −1 1 −1 
−1 1 1 −1 1

5.4.1 The Horn matrix is copositive

·
We will show this without decomposing H xx> and calculating. We start from some x ≥ 0 and we
·
investigate the evolution of function H xx> as we change variables x in a particular way described next.
·
The key is to notice that if we can decrease xi and xj by any ε ∈ (0, min(xi , xj )], the value H xx> decreases,
four times −1
 
 i, j ∈ [1..5] with
for1 any i 6= j. Indeed, take any two lines i and j and notice they contain 1 or
−1 and one times [ 1 ·
1 ]. The contribution to H xx> of a sub-matrix −1 or

1
  1
−1

is a multiple
Since we decrease both variables xi and xj by ε, the value xi − xj does not change. On the other hand, the
of x i − x j.

contribution of a sub-matrix [ 11 ] is x` (xi + xj ) for some ` ∈

/ {i, j} that has to verify x` ≥ 0. Decreasing both
xi and xj can only decrease x` (xi + xj ). An analogous (transposed) phenomenon happens on the columns i
and j. This way, decreasing xi and xj by the same ε > 0 can only decrease H xx> . ·
We can sequentially decrease pairs of non-zero variables xi and xj until we end up with a final x that
·
contains only one non-zero variable and whose value H xx> is less than or equal to the initial one. This
finishes the proof, because this final value is a multiple of a diagonal element of H that is surely non-negative.
29 Some
Pn
papers define Nn
r using j=1 zi = r. However, as far as I can see, this does no longer ensures the property
h 7 −8 0 i
(r+1) (r) (3) (2)
Pn ⊂ Pn that we need afterwords. We would have X = −8 7 0 ∈ P3 but X ∈ / P3 .
0 0 2
30 For an exact reference, check the comments on equation (7) from the “Copositive Optimization” article of the Optima 89

newsletter, see the http link from Footnote 22, p. 50.

59
5.4.2 The Horn matrix does not belong to S5+ + N 5
The easiest way to see H is not SDP is by noticing it does not respect the structure from Prop. 1.6.4. Indeed,
the first four elements of the last row can not be written as a linear combination of the rows of [H][1..4] ,
because of columns 2 and 3, where [H]I is the principal minor of A that selects rows I and columns I. We
still need to show H − N is not SDP for any N ≥ 0. Assume the contrary: there is N ≥ 0 such that
H − N  0 and we sequentially conclude:
(i) The operation H
− N can not decrease any diagonal element Hii , because that would lead to
det [H − N ]{i,j} < 0 for j = i + 1 mod 5. Indeed, the product of the elements on the main di-
agonal of such [H − N ]{i,j} would become less than 1 if Nii > 0 or Njj > 0, which is strictly less that
the product of the elements of the second diagonal (remark (H − N )ij ≤ Hij = −1). We thus need to
have diag(N ) = 0.
(ii) The operation H
− N can not decrease any Hij for j = i + 1 mod 5, because this would lead to
det [H − N ]{i,j} < 0. Indeed, this would increase the product of the elements of the second diagonal
of [H − N ]{i,j} while the main diagonal does not change by virtue of diag(N ) = 0 from above point
(i). We obtain that all negative elements of H can not be decreased by the operation H − N .
(iii) Notice we have H 3 = [H][1..3] = [H][2..4] = [H][3..5] . Since we need such matrices H 3 to remain SDP
3 3
after subtracting N , the extremal elements of the second diagonal (i.e., H13 and H31 ) need to remain
unchanged. This follows from applying Prop. 1.6.4 and the fact that the other elements of any H 3 are
not changed by the H − N operation by virtue of (i) and (ii) above. We obtain N13 = N24 = N25 = 0
and the same applies to the transposed elements.
(iv) The only elements of N that can still be non-zero are N14 , N25 and their transposed, so that |H −
N |{1,3,5} = H{1,3,5} . Since the determinant of this 3 × 3 matrix is −4, H − N can not be SDP.

/ S5+ + N 5 .
We obtained there is no N ≥ 0 that can lead to H − N  0, and thus, H ∈

5.5 A final short property: the Schur complement does not apply in C n∗
I finish with a final remark not directly related to other property from this chapter. In certain proofs, I was
tempted to use a “Schur complement property” with completely positive matrices but this is not possible.
Proposition 5.5.1. The Schur
h > icomplement property from Prop. 1.3.2 does not hold for completely positive
matrices. In particular, if y1 yY ∈ C (n+1)∗ , we do not necessarily have Y − yy> ∈ C n∗ .

Proof. It is enough to give an example. We will exhibit a 2 × 2 matrix Y with some zeros and a vector y
> >
with no zero. It is enough to take the sum [1 0 a] [1 0 a] + [1 a 0] [1 a 0] and divide it by 2 to obtain the
completely positive matrix below.  
1 a/2 a/2
a/2 a2 /2 0 
2
a/2 0 a /2
The non-diagonal terms of Y are zero. If we try to apply the Schur complement, we obtain negative entries
on the non-diagonal terms of Y − yy> , so that this matrix can not be completely positive.

6 SDP relaxations and convexifications of quadratic programs

Let us give a short warning: an important difficulty in this section comes from parsing a number of (La-
grangian) notations. It may be useful to print this section (twice) to more easily jump from one formula or
notation to another.

60
6.1 The most general quadratic program: SDP relaxation and total Lagrangian
We presented SDP reformulations and relaxations of convex quadratic programs in Section 3.4. We now
introduce a quadratic program in its most general form, not necessarily convex. Notice this form could
include 0 − 1 quadratic programs by adding x2i = xi ∀i ∈ [1..n] to the set of quadratic constraints (6.1.1b).

>
min Q xx + c x

·
>
(6.1.1a)

·

(QP ) s.t Bi xx> + d> ∀i ∈ [1..p]

 i x = (≤)ei (6.1.1b)

x ∈ Rn (6.1.1c)
The basic SDP relaxation of above (QP ) program is the following:

 min Q X + c> x
· (6.1.2a)

·




s.t Bi X + d>

SDP (QP ) i x = (≤)ei ∀i ∈ [1..p] (6.1.2b)
>
  

 1 x

 0 (6.1.2c)
 x X

Let us start with a rather negative result: the SDP relaxation of min{−x2 : x = 0} = 0 is min{−X11 : x =
0, X  xx> = 0} = −∞. This means that the above basic SDP relaxation can be arbitrarily bad in the
worst case. However, we will overcome this with different strengthening methods in the next subsections.
Let us introduce the Total Lagrangian (TL) in variables x of (QP ) from (6.1.1a)-(6.1.1c). We can write:

T Lx (QP ) = max L T Lx (QP ) (µ)

 
(6.1.3a)
µ

·
p p
! !
L T Lx (QP ) (µ) = x∈R
X X
min n
Q+ µi Bi xx> + c> + µi d>
i x − µ> e (6.1.3b)
i=1 i=1

Notice we have µi ≥ 0 if we have an inequality constraint Bi ·
xx >

+ d>
i x ≤ ei for some i ∈ [1..p].

Theorem 6.1.1. The optimum total Lagrangian T Lx (QP ) of a quadratic program (QP ) is equal to the
optimum value of the dual of the basic SDP relaxation (SDP (QP )) of (QP ) from (6.1.2a)-(6.1.2c).
Proof. To compute L T Lx (QP ) (µ) for a fixed µ, we can use the results for unconstrained quadratic programs
from Section 3.4.3, more exactly Proposition 3.4.2 that can be applied on L T Lx (QP ) (µ) as follows:
L T Lx (QP ) (µ) = max
t
t − µe
Pp
1
c> +P i=1 µi d>


−t i
s.t. 1
Pp
2
p  0.
2 (c + i=1 µi di ) Q + i=1 µi Bi
Replacing this in (6.1.3a), we obtain:
p
T Lx (QP ) = max t −
  X
µi ei (6.1.4a)
t,µ
i=1
Pp
1
c> +P i=1 µi d>


−t i
s.t. 1
Pp
2
p  0, (6.1.4b)
2 (c + i=1 µi di ) Q + i=1 µi Bi

which is exactly the dual of SDP (QP ) from (6.1.2a)-(6.1.2c). Recall that an inequality constraint Bi xx> +
d>
·
i x ≤ ei leads to non-negative µi ≥ 0 in the total Lagrangian. When dualizing such µi of above (6.1.4a)-
(6.1.4b), we obtain

a inequality constraint in SDP (QP ), see also Prop. 2.1.3. However, we can certainly
write T Lx (QP ) = OP T (DU AL(SDP (QP ))). One should bear in mind that there might be a duality
gap between (DU AL(SDP (QP
))) and (SDP (QP )).
The fact that T Lx (QP ) ≤ OP T (SDP (QP )) can also be shown by noticing

T Lx (QP ) = T LX (QP ) ,
   
(6.1.5)

61
 T LX (QP ) is the total Lagrangian of (SDP (QP )), i.e., it is defined using the same formulas
 
where
(6.1.3a)-(6.1.3b) but we replace the term xx> from (6.1.3b) with X and add constraint X  xx> to
(6.1.3b). To show L X (µ) = L T Lx (QP ) (µ), it is enough to show there is an optimal solution
TL (QP )
(X, x) of L X
Pp
such that X = xx> . We consider two cases: (a) Q + i=1 µi Bi  0 and (b)
TL (QP )
Pp Pp
·
Q + i=1 µi Bi  0. In the first case (a), we take an x ∈ Rn such that (Q + i=1 µi Bi ) xx> < 0. By
replacing x with tx we obtain a quadratic concave function in t that converges to −∞; by taking X = xx>
L X (µ) converges to −∞ as well. In the case (b), there is no use to consider any X = Y + xx>
TL (QP )
with some non-zero Y  0: the existence of such Y can only increase the value of L X (µ) by
TL (QP )

·
(Q + i=1 µi Bi ) Y ≥ 0. This confirms that T Lx (QP ) = T LX (QP ) ≤ OP T (SDP (QP )).
Pp
 

The dual of (SDP (QP )) is actually constructed by relaxing all constraints from (6.1.2a)-(6.1.2c), i.e.,
the proof of the general dualization property (Proposition 2.1.5) starts by computing in (2.1.11) the total
SDP Lagrangian using coefficients x0 . These coefficients x0 may include µ and also the dual variable (t) of
the constraint the puts a value of 1 in the upper right corner of the matrix from (6.1.2c). This explains the
very close relationship between DU AL(SDP (QP )) and T LX (QP ) . The above argument could actually
 

extend to a second proof of

 Theorem 6.1.1. Many convexification ideas presented next have certain roots in
the fact that T LX (QP ) is very related to DU AL(SDP (QP )).


6.2 Partial and total Lagrangians for quadratic programs with linear equality
constraints
We consider a version of the general quadratic program (QP ) from (6.1.1a)-(6.1.1c) in which we explicitly
separate linear equality constraints Ax = b. Furthermore, we consider adding p redundant constraints
· >
B j xx> + dj x = ej ∀j ∈ [1..p] that are always satisfied for all x ∈ Rn such that Ax = b. An example of
such redundant constraint is A> · > 2
j Aj xx = bj , where Aj is row j of A. By summing up over all j ∈ [1..p]
p
·
we can also obtain A> A xx> =
P
b2j . Many other examples can be found, e.g., we will later see the
j=1
redundant constraint sets from Example 6.3.4 and Example 6.3.5. However, for now, it is enough to say
that such constraints have no impact on the initial quadratic program, but they can be useful to convexify
it (i.e., so that the factor of the quadratic term becomes SDP) or to strengthen its SDP relaxation.

·
We formulate 


min Q xx> + c> x (6.2.1a)



 s.t Ax = b
 (6.2.1b)

·


> >
(QP= ) B j xx + dj x = ej ∀j ∈ [1..p] (6.2.1c)

·



> >
Bi xx + di x = (≤)ei ∀i ∈ [1..p] (6.2.1d)






n
x∈R (6.2.1e)


The formulation of the SDP relaxation leads to a program SDP (QP= ) defined by writting the SDP form
(6.1.2a)-(6.1.2c) corresponding to above (QP= ). More exactly, we obtain the following SDP relaxation of
(QP= ). 






·
min Q X + c> x
s.t Ax = b
(6.2.2a)
(6.2.2b)

·




 >
B j X + dj x = ej ∀j ∈ [1..p] (6.2.2c)


·
SDP (QP= )
Bi X + d>

i x = (≤)ei ∀i ∈ [1..p] (6.2.2d)





1 x>

  


 0 (6.2.2e)
 x X

62
6.2.1 The partial Lagrangians of (QP= ) and SDP (QP= )
We introduce below the Partial Lagrangian (PL) in variables x of (QP= ).

(P Lx (QP= )) = max L P Lx (QP= ) (µ) (6.2.3a)

µ


L P Lx (QP= ) (µ) =  s.t. Ax = b

Pp
· Pp
 min (Q + i=1 µi Bi ) xx> + c> + i=1 µi d>

>
i x−µ e
(6.2.3b)
x ∈ Rn

This partial Lagrangian does not dualize the linear equality constraints Ax = b from (6.2.1b) and ignores
the redundant constraints (6.2.1c). In other words, we obtain the partial Lagrangian of the initial program
without redundant constraints. Notice that adding redundant constraints to above L P Lx (QP= ) (µ) would
not change its value, because L P Lx (QP= ) (µ) already imposes Ax = b, and so, the redundant constraints
are satisfied.
The above partial Lagrangian is equal to the following augmented partial Lagrangian that also dualizes
the redundant constraints (6.2.1c).

(P Lx (QP= )) = max L P Lx (QP= ) (µ, µ) (6.2.4a)

µ,µ

·
  Pp Pp 


 min Q + i=1 µi Bi + j=1 µj B j xx> +
>
  Pp Pp 
c> + i=1 µi d>

i + j=1 µj dj x

L P Lx (QP= ) (µ, µ) = 

> > (6.2.4b)
 −µ e − µ e
s.t. Ax = b




x ∈ Rn


Notice we slightly abused notations, because we used L P Lx (QP= ) both as a function with one argument
µ in (6.2.3b) or as an augmented function with two arguments µ and µ in (6.2.4b) just above. However, it
is not hard to check that the value of this partial Lagrangian only depends on the first argument, i.e., since
the µ terms are multiplied with (redundant) expressions that are equal to zero, we can write:

L P Lx (QP =)
(µ, µ) = L P Lx (QP= ) (µ) ∀µ, µ,

Let us now introduce the augmented partial Lagrangian in variables X (such that X  xx> ) of the SDP
relaxation (SDP (QP= )) from (6.2.2a)-(6.2.2e).

P LX (QP= ) = max L X
 
(µ, µ) (6.2.5a)
µ,µ PL (QP= )

·
  Pp Pp 


 min Q + i=1 µi Bi + j=1 µj B j X+
>
  Pp Pp 
> >

c + i=1 µi di + j=1 µj dj x



L P LX (QP ) (µ, µ) = 

−µ> e − µ> e (6.2.5b)
= 

 s.t. Ax
 = b> 
1 x


0


x X


The redundant constraints are dualized in objective function terms that are no longer redundant. In this
· >
relaxation, we have no guarantee that the term of µj is zero, i.e., we no longer have B j X + dj x − ej =
0 ∀j ∈ [1..p], as the only constraint on X is X  xx> . It is not hard to see that the above program is: (i)
a Lagrangian relaxation of (SDP (QP= )) and (ii) an SDP relaxation of L P Lx (QP= ) (µ, µ). Combining (i)
and (ii), we directly obtain:
P LX (QP= ) ≤ OP T (SDP (QP= )),
 
(6.2.6a)

P LX (QP= ) ≤ P Lx (QP= ) .
   
(6.2.6b)

63
6.2.2 The total Lagrangian using equality constraints
The total Lagrangian can be expressed as in (6.1.3a)-(6.1.3b). However, technically, we now separate the dual
variables β associated to the linear equality constraints. We also separate the dual variables µ associated
with the redundant constraints. Thus, the total Lagrangian from (6.1.3a)-(6.1.3b) evolves to:

T Lx (QP= ) = max L T Lx (QP

 
=)
(µ, µ, β) (6.2.7a)
µ,µ,β

·
  Pp Pp 
 min
 x∈Rn
 Q + i=1 µi Bi + j=1 µj B j xx>
L T Lx (QP= ) (µ, µ, β) =  >
 
Pp
+ c> + i=1 µi d>
Pp
µj dj + β > A x (6.2.7b)
i + j=1

−µ> e − µ> e − β > b


We can define the SDP version of the above total Lagrangian by replacing xx> with X in (6.2.7b)
and by adding constraint X  xx> . We thus construct the total Lagrangian L X (µ, µ, β)
TL (QP )
in variables X (and x). For any fixed µ and µ, the value max L (µ, µ, β) can be seen as
β T LX (QP )

L P LX (QP (µ, µ, ). As such, the optimum SDP total Lagrangian T LX (QP= ) =

 
a Lagrangian of
=)

max
µ,µ,β
L T LX (QP (µ, µ, β), satisfies:
=)

T LX (QP= ) ≤ P LX (QP= ) .
   
(6.2.8)

T Lx
 
Proposition 6.2.1. The total Lagrangian ∗
µ ,µ∗ (QP= ) for fixed µ∗ and µ∗ (formally defined as the
 
maximum of (6.2.11.a) below) is equal to OP T L X ∗ ∗
(µ , µ ) from (6.2.5b). This is enough to
PL (QP= )
ensure:
T Lx (QP= ) = T LX (QP= ) = P LX (QP= ) .
     
(6.2.9)
Pp Pp
Proof. Let us introduce notational shortcuts Qµ,µ = Q + i=1 µi Bi + j=1 µj B j and c> µ,µ = c +
>
 
>
L
Pp >
Pp ∗ ∗
i=1 µi di + µ d
j=1 j j . Starting from the right term of (6.2.9), notice that X (µ , µ )
PL (QP= )
from (6.2.5b) can be compactly written:


 min ·
Qµ∗ ,µ∗ X + c>
µ∗ ,µ∗ x − µ
∗>
e − µ∗ > e
L P LX (QP ) (µ∗ , µ∗ ) =  s.t. Ax
 = b> 

(6.2.10)
= 1 x
 0
x X


The total Lagrangian (6.2.7a)-(6.2.7b) with fixed µ∗ and µ∗ , can be written in compact notations using only
variables β associated to the linear equality constraint Ax = b.

T Lx
 
µ∗ ,µ∗ (QP= ) =

max L T Lx (QP= ) (µ∗ , µ∗ , β)


 (6.2.11a)
 β

·
(  
= >
minn Qµ∗ ,µ∗ xx> + c> µ∗ ,µ∗ + β A x

 s.t. L T Lx (QP= )
(µ∗
, µ∗
, β) = x∈R (6.2.11b)
−µ∗ > e − µ∗ > e − β > b


max t − µ∗ > e − µ∗ > e − β > b


 (6.2.11c)
 t,β "

> > #
= cµ∗ ,µ∗ +β A
−t

 s.t. >
cµ∗ ,µ∗ +A β
2 0 (6.2.11d)
Q

2 µ∗ ,µ∗

64
where we applied Proposition 3.4.2 from Section 3.4.3, similarly to what we did for the total Lagrangian
formulation (6.1.4a)-(6.1.4b). We obtained in (6.2.11c)-(6.2.11d) the dual of L X (µ∗ , µ∗ ) from
PL (QP= )
(6.2.10). Since this last program is strictly feasible (it has no constraint on X other than X  xx> ), we can
apply the strong duality Theorem 2.3.4 to conclude:
     
OP T DU AL L X = T Lµ∗ ,µ∗ (QP= ) = OP T L X
x

∗ ∗ ∗ ∗
(µ , µ ) (µ , µ )
PL (QP= ) PL (QP= )

 
We can also notice that L P LX (QP (µ∗ , µ∗ ) could be either bounded or unbounded (−∞); recall
=)
these two cases are also separated in the strong duality Theorem 2.3.4. If it is unbounded, then (6.2.11d) is
surely infeasible meaning that the total Lagrangian (6.2.11a)–(6.2.11b) has to be unbounded (−∞).
Till here, we worked with an arbitrary fixed solution (µ∗ , µ∗ ). From ∗ ∗
 now on, let us consider that (µ , µ )
is the optimal solution of the SDP partial Lagrangian P LX (QP= ) from (6.2.5a); we proved above that
P LX (QP= ) = OP T (L X (µ∗ , µ∗ )) is effectively reached by the total Lagrangian (T Lx (QP= ))
 
PL (QP= )

in a point defined by above µ∗ and µ∗ and some β ∗ . Combining this with T Lx (QP= ) = T LX (QP= ) ≤

 

P LX (QP= ) from (6.1.5) and (6.2.8), we obtain the sought inequality (6.2.9) that we recall below for the
 

reader’s convenience.

T Lx (QP= ) = T LX (QP= ) = P LX (QP= )

     
(6.2.12)

Now recall Theorem 6.1.1 on total Lagrangians that states OP T (DU AL(SDP (P= ))) = T Lx (QP= ) .

Combining this with above (6.2.12) and (6.2.6a)-(6.2.6b), we obtain the following fundamental hierarchies
(inequalities):

OP T (DU AL(SDP (P= ))) = T Lx (QP= ) = T LX (QP= ) = P LX (QP= ) ≤ P Lx (QP= )

       
(6.2.13a)

OP T (DU AL(SDP (P= ))) = T Lx (QP= ) = T LX (QP= ) = P LX (QP= ) ≤ OP T (SDP (P= )) (6.2.13b)
     

The values of P Lx (QP= ) and OP T (SDP (P= )) can not be ordered in the most general setting. We will

provide Example 6.2.3 in which P Lx (QP= ) < OP T (SDP (P= )) and Example 6.3.3 in which P Lx (QP= ) >

OP T (SDP (P= )). As a side remark, recall from equations (6.2.6a)-(6.2.6b) that (P LX (QP= )) is a relaxation
of both (P Lx (QP= )) and OP T (SDP (P= )).

Using convexifications to obtain P LX (QP= ) = P Lx (QP= )

 

6.2.3

We here show how (P Lx (QP= )) becomes equal to (P LX (QP= )) by exploiting the relaxation of the redundant
constraints. The following proposition is a (quite deeply) modified and adapted version of a theorem from a
lecture note of Frédéric Roupin,31 but the proof is personal.
Proposition 6.2.2. If P Lx (QP= ) = −∞ it is clear that all programs from the hierarchy (6.2.13a) are

unbounded. If P Lx (QP= ) 6= −∞, we can take an optimal solution µ∗ of P Lx (QP= ) in (6.2.3a)-(6.2.3b).

If there exists µ∗ that convexifies the augmented formulation of P Lx (QP= ) from (6.2.4a)-(6.2.4b), then

P LX (QP= ) = P Lx (QP= ) . This means that the first fundamental hierarchy (6.2.13a) collapses.
 

We have P LX (QP= ) ≤ P Lx (QP= ) from (6.2.13a).

 

Proof. Since we say (slightly abus-
ing notations) that P Lx (QP= ) = L P Lx (QP= ) (µ∗ ) = L P Lx (QP= ) (µ∗ , µ∗ ), we can write

31 See slide 57 of [Link]

65
L P LX (QP ) (µ∗ , µ∗ ) ≤ P LX (QP ) ≤ P Lx (QP ) = L P Lx (QP ) (µ∗ , µ∗ ). It is thus enough to
 

= =
=
=

show that L X (µ∗ , µ∗ ) = L P Lx (QP ) (µ∗ , µ∗ ).

=
PL (QP ) =

Since the Hessian of L X (µ∗ , µ∗ ) is equal to the Hessian of L P Lx (QP ) (µ∗ , µ∗ ), both programs
=
PL (QP ) =

are convex and have an SDP Hessian. Writing the optimal solution of L X (µ∗ , µ∗ ) as X = xx> +Y ,
PL (QP ) =

we notice Y  0 can only increase the objective value of L X (µ∗ , µ∗ ), and so, an optimal Y can
PL (QP ) =

be Y = 0. An optimal solution of L X (µ∗ , µ∗ ) satisfies X = xx> , i.e., L X (µ∗ , µ∗ ) =

PL (QP ) = PL (QP ) =

L P Lx (QP ) (µ∗ , µ∗ ) = P Lx (QP ) .

=
=

We now provide an example in which the hierarchy (6.2.13a) collapses, but the last inequality in (6.2.13b)
is strict, i.e., there o (DU AL(SDP (P= ))) and (SDP (P= )). The example relies on the
isha duality gapibetween
n 0 x12 0
fact that min x12 : x12 x22 0  0 = 0, while the dual of this program has optimum value -1.
0 0 1+x12

Example 6.2.3. We present the (QP= ) example on the left and its partial Lagrangian P Lx (QP= ) on the

right. After that, we will compare (SDP (QP= )) and (DU AL(SDP (QP= )).
The left program has only two feasible solutions x> = [0 0 1 0] and

·
 " 1 #
 0 2 0 0
x> = [0 0 − 1 0] both of value 0.32 We now formulate the (augmented)
xx>
 1


 min 2 0 0 0
partial Lagrangian.

 0 0 0 0

 0 0 0 1

P Lx (QP= ) = max min4
 
s.t. x =0

4



·

µ,µ x∈R

 0 0 0 0

µ : 0000 xx> = 0 x4 =0


 44 00 00 00 01


·
µ12
  1

µ11 2− 2 µ13 +µ11 0

·

 1 µ12
2− 2 xx> − µ12

 0 0 0 0 0 µ23 +µ11 0
xx> = 0
  
 23 00 01 10 00
µ : µ13 +µ11 µ23 +µ11 µ12 0


0 0 0 1+µ44

·
0 0 0 0
(QP= )

·
0 0 1 0
= [µ] xx> − µ12

µ13 : 01 00 00 00 xx> = 0




 0 0 0 0

·


The above (inner) minimization problem can be unbounded only if

 1 0 1 0
µ11 : 01 01 10 00 xx> = 0


[µ]3×3  0, where [µ]3×3 is the leading principal minor of size 3 × 3


0 0 0 0


of [µ]. The variable µ44 is redundant and does  not appear in the
·

0 − 12 0 0

 " #
non-augmented formulation (6.2.3a)-(6.2.3b) of P Lx (QP= ) . How-

xx> = 1
 1
 µ12 : − 2 0 0 0



 0 0 1 0

 0 0 0 0 ever, notice that Prop 6.2.2 does hold, i.e., the augmented partial
L P Lx (QP= ) (µ∗ , µ∗44 ) from the inner minimization problem is convex

x ∈ R4


for any optimal solution (µ∗ , µ∗44 ) of P Lx (QP= ) . Since [µ∗ ]3×3  0
 
The first above constraint is a linear
equality constraint. The next one is and [µ∗ ]22 = 0, we need to have [µ∗ ]12 = 0, i.e., µ∗12 = 1. The rest of
a redundant constraint, i.e., we have the elements can be, for instance, µ∗11 = µ∗13 = µ∗23 = µ∗44 = 0. However,
p = 1 with regards to the canon- for any choice of these latter elements, the objective value of the min-
ical formulation (6.2.1a)-(6.2.1e) of imization problem will be −1 (notice x = 0 leads to an objective value
(QP= ). The other constraints are clas- of −µ∗12 = −1), and so, P Lx (QP= ) = −1.
sical quadratic constraints.

  0 X12 0 0 
The SDP relaxation SDP (QP= ) can be written: min X12 : X = X012 X22 0 0
0 1+X12 0
 xx>  0 = 0. We
0 0 0 0
have OP T (SDP (QP= )) = 0 and this solution is achieved by X12 = 0, X22 = 0, X33
= 1 + X12 = 1 and
x = [0 0 0 0]> (or x = [0 0 1 0]> ); notice X12 can not be strictly less than 0 because X11 = 0. One could
calculate that OP T (DU AL(SDP (QP= ))) = −1. All values in the hierarchies (6.2.13a)-(6.2.13b) are equal
to −1 except OP T (SDP (QP= )) = 0.
32 Constraints µ 2
23 and µ13 enforce x2 x3 = x1 x3 = 0; combining this with constraint µ11 that states x1 + 2x2 x3 + 2x1 x3 = 0,
we obtain x1 = 0. Constraint µ12 enforces −x1 x2 + x23 = 1 =⇒ x23 = 1. Finally, x2 = 0 follows from constraint µ23 .

66
6.3 The case of 0 − 1 quadratic programs: partial and total Lagrangians
A part of this section aims at proving results related to the QCR method of Alain Billionnet, Sourour
Elloumi and Marie-Christine Plateau (see link in Footnote 38, p. 96) and to the article of Alain Faye and
Frédéric Roupin indicated in Footnote 33, p. 71. However, I think the presentation style (using longer and
more detailed arguments) and the order of the theorems from this manuscript is completely different.

6.3.1 Main characterization

We interpret a 0 − 1 quadratic program exactly as a particular case of (QP= ) from (6.2.1a)-(6.2.1e) in which
the first n constraints (6.2.1d) are x2i = xi . We can thus have p ≥ n non-redundant quadratic constraints.
All results from the previous Subsection 6.2 do hold in this new 0 − 1 context. In fact, the 0 − 1 programs
are in some sense simpler because we can prove OP T (DU AL(SDP (QP= ))) = OP T(SDP (QP= )), i.e., the
hierarchy (6.2.13b) collapses and we can no longer have a duality gap P Lx (QP= ) < OP T (SDP (QP= ))
as in Example 6.2.3.
Theorem 6.3.1. The following fundamental hierarchies hold for 0 − 1 quadratic programs.

OP T (DU AL(SDP (P= ))) = T Lx (QP= ) = T LX (QP= ) = P LX (QP= ) = OP T (SDP (P= )) (6.3.1a)
     

≤ P Lx (QP= )
 
(6.3.1b)

Proof. The first equality (6.3.1a) follows from the fact that we can show that the hierarchy (6.2.13b) col-
lapses, i.e., all inequalities in (6.2.13b) are equalities in a 0 − 1 context. To show this, it is enough to
prove there is no duality gap between (DU AL(SDP (P= ))) and (SDP (P= )). This is a consequence of the
strong duality Theorem 2.3.3, considering that (DU AL(SDP (P= ))) is strictly feasible and bounded – the
unbounded case OP T (DU AL(SDP (P= ))) = ∞ would lead to to (OP T (SDP (P= ))) = ∞ by virtue of
OP T (DU AL(SDP (P= ))) ≤ OP T (SDP (P= )). However, the dual is always strictly feasible because the SDP
constraint (6.1.4b) can be written under the form:

− 12 µ1 − 12 µ2 . . . − 12 µn
 
−t
 − 1 µ1 µ1 0 ... 0 
 12 
 − µ2 0 µ2 ... 0 
Y + Yµ = Y +  2   0.
 .. .. .. .. 
 . . . . 0 
− 12 µn 0 0 ... µn

The above matrix can be strictly feasible (positive definite) by taking µ1 = µ2 · · · = µn = M for a sufficiently
large M , so that the bottom-right n × n minor [Y + diag(µ)]n×n become positive definite. A sufficient M
value can be 1 minus the lowest eigenvalue of Y . Using the Sylvester criterion (Prop. 1.5.2) in reversed order,
we can prove that the above matrix Y + Yµ is positive definite by showing that the whole determinant is
positive. This can always be the case by taking a sufficiently large value of −t. By developing the Leibniz
formula for the determinant of the matrix Y +Yµ , there will be a term −t det ([Y + diag(µ)]n×n ) that can be
arbitrarily large, so as to make the determinant as high as possible. Using the strong duality as mentioned
above, we obtain OP T (DU AL(SDP (P= ))) = OP T (SDP (P= )), and so, the hierarchy (6.2.13b) collapses into
(6.3.1a). Finally, (6.3.1b) follows from (6.2.13a).

The above proof does show that OP T (DU AL(SDP (P= ))) = OP T (SDP (P= )), but the strong duality
theorem only guarantees that (SDP (P= )) does reach the optimum value. The program (DU AL(SDP (P= )))
does not necessarily effectively reach its optimum value. This will become clear in the following example.

6.3.2 Two examples: OP T (DU AL(SDP (P= ))) may not reach its own optimum value and
OP T (SDP (P= )) may be strictly lower than P Lx (QP= )

Example 6.3.2. We modify Example 6.2.3 as follows:

67
 – The constraint associated to dual variable µ12 becomes
0 − 12
 
0 0
− 1
 2
 0
0
0 0 0
0 −1 0
0 0 0

· xx> = 0

– We add four constraints x2i = xi with i ∈ [1..4] whose dual values are µ1 , µ2 , µ3 and µ4 . These
constraints imply that all variables are binary.
The SDP (P= ) relaxation of (QP= ) has the solution X = 04×4 of objective value 0. We will show that
(DU AL(SDP (P= ))) converges to 0, even if there is no feasible solution with value 0 in this dual. For this,
we first write (DU AL(SDP (P= ))) as follows:


 max t

  −µ1 −µ2 −µ3 −µ4


 −t 2 2 2 2

(DU AL(SDP (P= )))  −µ2 1 µ11 +µ1 12 − µ212 µ13 +µ11 0 
 −µ2 1 µ12
s.t.   0,


  2 2− 2 µ2 µ23 +µ11 0
 −µ 3 µ +µ µ +µ −µ +µ 0
  13 11 23 11 12 3


 2
−µ4
0 0 0 1+µ +µ

2 44 4

A solution of value zero of the above program would clearly set t = 0. This would imply that Row 1 contains
only zeros, so that µ1 = µ2 = µ3 = µ4 = 0. Furthermore, because µ2 stands alone on the diagonal at position
(3,3), row 3 has to contain only zeros as well. This means µ12 = 1 which leads to a negative value at position
(4,4) on the diagonal. The resulting matrix can not be SDP.
We can prove the optimal solution converges to zero. Let us take µ1 = 0, µ2 = ε, µ3 = 0, µ4 = 0,
µ11 = M , µ12 = 0, µ13 = −µ11 = −M , µ23 = −µ11 = −M , µ44 = 0. The variables ε and M stand, resp. ,
for a very small and a very large positive value. The above program simplifies to:
max t
 −t 0 −ε/2 0 0 
0 M 1/2 0 0
s.t. [µM,ε ] =  −ε/2 1/2 ε 0 0
  0,
0 0 0 0 0
0 0 0 0 1
1
We take a sufficiently large value of M (> 4ε ), so as to be able to have det([µM,ε ]3×3 ) > 0 for small values
of −t, where [µM,ε ]3×3 is the leading principal minor of size 3 × 3. However, the optimum value of above
program is − 4ε , associated to −t = 4ε . Recall M can be as large as necessary to ensure det([µM,ε ]3×3 ) > 0.
The objective value of this solution is − 4ε and its limit is limε→0 − 4ε = 0.
We continue

with an example in which the fundamental inequality (6.3.1b) is strict, i.e., OP T (SDP (P= )) <
P Lx (QP= ) .
Example 6.3.3. We introduce a 0 − 1 program on the left, the partial Lagrangian on the right and then we
will analyse the SDP relaxation.

The only feasible solutions of (QP= ) are [x1 x2 ] = [1 0] and [x1 x2 ] =

·

min
 0 −1 
xx> [0 1], both of value OP T (QP= ) = 0. We now formulate the partial
−1 0

Lagrangian.




 s.t. x1 + x2 = 1

·


·

P Lx (QP= ) = max min2
    µ1 −1 
(QP= ) µ1 : [ 10 00 ] xx> − x1 = 0 −1 µ2 xx> − µ1 x1 − µ2 x2
µ1 ,µ2

·
 x∈R

 x1 +x2 =1
µ2 : [ 00 01 ] xx> − x2 = 0






x ∈ R2

We know by the Lagrangian definition that P Lx (QP= )


≤
Notice that the first constraint in the OP T (QP= ). We will show that for [µ1 µ2 ] = [−1 − 1], the Lagrangian
above program is a linear equality con- reaches OP T (QP= ) = 0. Indeed, replacing these [µ1 µ2 ] values in above
straint. We have no redundant con- formula, we obtain
straints, i.e., we have p = 0 with
regards to the canonical formulation
(6.2.1a)-(6.2.1e) of (QP= ).
min2
x∈R
x1 +x2 =1
 −1 −1 
−1 −1 · xx> + x1 + x2 = −(x1 + x2 )2 + (x1 + x2 ) = 0

68
For now, we have P Lx (QP= ) = OP T (QP= ) = 0 and let us turn towards (SDP (QP= )). We notice that


0.5 0.25 ] = xx> + 0.25I = [ 0.25 0.25 ] + [ 0.25 0 ] is a feasible solution
[x1 x2 ] = [0.5 0.5] combined with X = [ 0.25 0.5 2 0.25 0.25 0 0.25
of (SDP (QP= )) with objective value −0.5. This leads to

OP T (SDP (QP= )) ≤ −0.5 < 0 = P Lx (QP= ) ,

 

and so, the inequality (6.3.1b) can be strict.

The limit of the strongest convexification is P Lx (QP= )

6.3.3

[Link] The strongest convexifications can lead to P LX (QP= ) = P Lx (QP= )

 

We recall Proposition 6.2.2.
It states that if µ∗ is an optimal solution of the non-augmented (6.2.3a)-(6.2.3b)
formulation of P Lx (QP= ) and there there exists µ∗ that convexifies the augmented (6.2.4a)-(6.2.4b)
formulation of P Lx (QP= ) , then P LX (QP= ) = P Lx (QP= ) . In other words, if the convexification is

 

successful (for the optimal µ∗ chosen above), both hierarchies (6.3.1a)-(6.3.1b) collapse (in the binary case).
Let Qµ∗ be the factor of the quadratic term xx>
associated to the optimal solution µ∗ of the non-
augmented formulation (6.2.3a)-(6.2.3b) of P Lx (QP= ) . The factor of xx> in the augmented formulation
(6.2.4a)-(6.2.4b) of P Lx (QP= ) has the form Qµ∗ ,µ = Qµ∗ + j=1 µj B j where the B j s with j ∈ [1..p] are

Pp 0

the quadratic factors of xx> in the redundant constraints (6.2.1c) of the (QP= ) definition.
Let us first investigate the case in which if Qµ∗ is not positive over {y ∈ Rn : Ay

= 0}, i.e., over the null
space null(A) of A (see also (A.1.2)). In this case, we can show that P Lx (QP= ) = −∞. Indeed, if there
·
is some y ∈ Rn such that Ay = 0 and Qµ∗ yy> = −z < 0, we can take any feasible solution x ∈ Rn
of L P Lx (QP= ) (µ∗ ) from the non-augmented P Lx (QP= ) formulation (6.2.3a)-(6.2.3b). This x satisfies

Ax = b and notice that x + ty satisfies A(x + ty) = b as well ∀t > 0. If we write the objective value of
x + ty as a function of t, we obtain a polynomial of degree 2 with the leading term −zt2 . This is a concave
polynomialthat is unbounded from below regardless of its non-quadratic terms. In this case it is not difficult
to achieve P L (QP= ) = P Lx (QP= ) = −∞.
X


We hereafter focus on the contrary case: Qµ∗ is positive over null(A). We are looking for redundant
constraints that can convexify any quadratic factor Qµ that is non-negative (positive) over null(A). In
fact, it may be enough to convexify the matrix Qµ∗ associated to the optimal solution µ∗ of P Lx (QP= ) ,

but let us keep in mind a broader objective. However, if it is possible to convexify Qµ∗ into Qµ∗ ,µ∗  0,
we can apply Proposition 6.2.2, and so, both the fundamental hierarchies(6.3.1a)-(6.3.1b)  collapse – also
recall Theorem 6.3.1. In such a case, we obtain that OP T (SDP (Q= )) = P LX (QP= ) reaches the limit
P Lx (QP= ) .

However, if the redundant constraints are not strong enough to convexify Qµ∗ for the above chosen
optimal µ∗ , they
 might be able to convexify some Qµ for some other µ. If this happens, we are certain that
P L (QP= ) is bounded, although its value may be L P Lx (QP= ) (µ) < L P Lx (QP= ) (µ∗ ) = P Lx (QP= ) .
X


Finally, keep in mind that only P Lx (QP= ) does not depend on the chosen redundant constraints in the

fundamental hierarchy (6.3.1a)-(6.3.1b), i.e., one can consider that all other terms in this hierarchy are
actually indexed by a set of chosen redundant constraints.

[Link] Examples of redundant constraints of different strengths

Example 6.3.4. (A unique redundant constraint) A redundant constraint can be constructed by observing
that (Ai x − bi )2 = 0 for any i ∈ [1..p], where Ai is the row i of A. We obtain (Ai x)2 − 2bi Ai x + b2i = 0,
equivalent to
A>i Ai ·
xx> − 2bi Ai x + b2i = 0,
by virtue of Lemma [Link]. By summing over all i ∈ [1..p], we obtain
(6.3.2)

A> A
· xx> − 2b> Ax + b> b = 0 (6.3.3)

69
As long as we consider a partial and not a total Lagrangian, we have 2bi Ai x = 2b2i because Ax = b is active;
thus, the constraint (6.3.2) reduces to A> · > 2
i Ai xx = bi and (6.3.3) reduces to

A> A
· xx> = b> b (6.3.4)

Example 6.3.5. (A set of redundant constraints) We first use constraints Ax = b. For each i ∈ [1..p], we
generate a set of constraints xj Ai x = xj bi for all j ∈ [1..n]. There exists a linear combination of these
constraints that generate the constraint from previous Example 6.3.4.
Proof. For each i ∈ [1..p], we perform the following. We ! constraint xj Ai x = xj bi , ∀j ∈
! multiply by Aij the
Pn n
P
[1..n]. By summing up over all j, we obtain Aij xj Ai x = Aij xj bi that can be written Ai xAi x =
j=1 j=1
Ai xbi , or furthermore A> · >
i Ai xx −bi Ai x = 0. We now multiply −Ai x+bi = 0 by bi to obtain −bi Ai x+bi =
2

0 and we add this to the previous equality to obtain A>

i A i xx ·
>
−2b A
i i x+b2
i = 0, which is exactly (6.3.2).

The above constraint sets are not equivalent in the general non-binary case

Remark 6.3.6. In the general non-binary case, the first redundant constraint set (Example 6.3.4) might
be weaker. For instance, consider Qµ∗ = [ 01 10 ] and a unique linear constraint Ax = b with A = [0 1] and
·
b = 0. The constraint generated by the first Example 6.3.4 is (6.3.3), i.e., A> A xx> = 0, equivalent to
·
[ 00 01 ] xx> = 0. This A> A = [ 00 01 ] matrix can not convexify Qµ∗ . On the other hand, the second redundant
· ·
constraint set (Example 6.3.5) generates x1 Ax = x1 b = 0 (or x1 x2 = 0, i.e., A0 xx> = [ 01 10 ] xx> = 0)
· ·
and x2 Ax = x2 b = 0 (or x2 x2 = 0, i.e., A00 xx> = [ 00 01 ] xx> = 0). The matrices A0 = [ 01 10 ] and
A00 = [ 00 01 ] can easily convexify Qµ∗ .
Remark 6.3.7. The second redundant constraint set (Example 6.3.5) can convexify any matrix Qµ∗ that is
non-negative over null(A). This way, we can always apply Prop. 6.2.2 and collapse the hierarchy (6.2.13a),
i.e., the convexification is optimal and it reaches the limit value P Lx (QP= ) . The proof of this convexifi-

cation is given in Appendix C.2.1, see more exactly Prop C.2.2.

If Qµ∗ is strictly positive over null(A) (i.e., u> Qu > 0 ∀u ∈ null(A)−{0}), Qµ∗ can be convexified more
easily. We can use any unique redundant constraint of the form x> A> SAx = b> Sb for some S  0, e.g., for
instance (6.3.4) corresponds to choosing S = In . In other words, we can always construct Qµ∗ + λA> SA  0
for a sufficiently large λ, see the proof in Prop. C.2.1.

The above constraint sets are equivalent in the binary case

For the binary case, we will prove below (Prop. 6.3.8) that the SDP programs integrating the SDP versions
of above redundant constraint sets (Examples 6.3.4 and 6.3.5) are equivalent. Recall that in the 0-1 case
the convexified total and partial Lagrangians reach OP T (SDP (P= )) as stated in (6.3.1a). Using above two
statements, the two redundant constraint sets make the convexified total and partial Lagrangians reach the
same value OP T (SDP (P= )), i.e., they are equivalent.
Proposition 6.3.8. The SDP constraints associated to the redundant constraints from Example 6.3.4 and
Example 6.3.5 are equivalent. This means that the best convexifications (of the total or partial Lagrangians)
achieved by the two redundant constraint sets have the same value in the 0 − 1 case, i.e., that of the SDP
bound OP T (SDP (P= )) expressed using either set of redundant constraints.
Proof. Notice using Examples 6.3.4 and 6.3.5 that the two SDP constraint sets are respectively:

A> A
n
X
· X − 2b> Ax + b> b = 0 (6.3.5a)

Aik Xjk = xj bi , ∀j ∈ [1..n], i ∈ [1..p] (6.3.5b)

k=1

The implication (6.3.5b) =⇒ (6.3.5a) can be constructed by applying on constraints (6.3.5b) the linear
combination presented in the proof of Example 6.3.5. This linear combination leads to (6.3.5a).

70
 It is more difficult to show the converse (6.3.5a) =⇒ (6.3.5b). We write (6.3.5a) as 0 = A> A (X − ·
· ·
xx ) + A> A xx> − 2b> Ax + b> b = A> A (X − xx> ) + 0, where we used (6.3.3) which is a consequence of
>

Ax = b (this constraint does appear in both SDP formulations even if it is not necessary as it can be inferred
·
from (6.3.5a) or (6.3.5b) respectively). We thus obtain A> A (X − xx> ) = 0 and Prop. 1.3.4 implies that
A> A(X − xx> ) = 0. Taking row r ∈ [1..n] and column c ∈ [1..n] of this product, we obtain:
n
X
0= (A> A)rk (X − xx> )kc
k=1
p
n
!
X X
= Air Aik (Xkc − xk xc )
k=1 i=1
Xn Xp
= Air (Aik Xkc − Aik xk xc )
k=1 i=1
Xn X p p
X
= Air Aik Xkc − Air bi xc (we used Ai x = bi )
k=1 i=1 i=1
p n
!
X X
= Air Aik Xck − xc bi
i=1 k=1

Since the P last formula holds for all r ∈ [1..n], we can reformulate it in terms of the rows Ai of A, obtaining
P p n
i=1 A i ( k=1 Aik Xck − xc bi ) = 0. This is a linear combination of the rows Ai of A. Assuming a legitimate
condition rank(A) P= p (i.e., the constraints Ax = b are linearly independent), this linear combination can
n
lead to 0 only if k=1 Aik Xck − xc bi = 0 ∀i ∈ [1..p]. Since this holds for any c ∈ [1..n], we have obtained
(6.3.5b). This proof is taken from Prop. 5 of the paper “Partial Lagrangian relaxation for General Quadratic
Programming” by Alain Faye and Frédéric Roupin.33

6.3.4 Collapsing both hierarchies by convexification and an associated Branch-and-bound

We here focus on solving binary equality-constrained quadratic programming. The main idea is that
we can use the best convexification constructed in Section [Link] to determine fast lower bounds for a
Branch-and-bound (Section [Link]) that solves the initial binary equality-constrained quadratic problem.
As a side remark, certain convexification ideas below can well apply to the non-binary problem as well.

[Link] Determining the best convexification coefficients µ∗ and µ∗ by solving DU AL(SDP (P= ))
According to (6.3.1a), in the binary case, the value of a convexified total Lagrangian reaches OP T (SDP (P= )).
However, the quality of OP T (SDP (P= )) is dependent on the redundant constraints it integrates. Let us
write the total Lagrangian from (6.2.7a)-(6.2.7b) in a more compact form:

T Lx (QP= ) = max L T Lx (QP= ) (µ, µ, β)

 
(6.3.6a)
µ,µ,β

·
(  
>
minn Qµ,µ xx> + c> µ,µ + β A x
L T Lx (QP= ) (µ, µ, β) = x∈R (6.3.6b)
−µ> e − µ> e − β > b,
Pp Pp Pp Pp >
where Qµ,µ = Q + i=1 µi Bi + j=1 µj B j and c> µ,µ = c> + i=1 µi d> i + j=1 µj dj .
Let us first show that T Lx (QP= ) is bounded. For this, we can use the fact that in the 0 − 1 case
 

the objective function contains n terms of the form µi x2i − µi xi (for all i ∈ [1..n]) that could always strictly
convexify any matrix. In other words, there always exist µo and µo such that Qµo ,µo  0, i.e., take a
sufficiently large µoi in the terms µoi x2i − µoi xi (for all i ∈ [1..n]). We have Qµo ,µo invertible by virtue
of det (Qµo ,µo ) > 0. The (transposed) gradient of the objective function of L T Lx (QP= ) (µo , µo , β) from
>
(6.3.6b) is ∇> > >
µo ,µo (x) = 2x Qµo ,µ + cµ,µ + β A. Since Qµo ,µ is invertible, there exists a stationary point
o o

33 Published in A Quarterly Journal of Operations in 2007, vol 5(1), pp. 75-88.

71
xo in which ∇> o
µo ,µo (x ) = 0. The stationary point of a (strictly) convex function is its minimizer, and so,
x

we are sure that OP T (DU AL(SDP (P= ))) = T L (QP= ) = OP T (SDP (P= )) is bounded.
Let us study what happens if we determine µ∗ and µ∗ from the optimal values of DU AL(SDP (P= ))
from (6.1.4a)-(6.1.4b), i.e., solve (6.1.4a)-(6.1.4b) and retrieve the optimal value of µ (think of µ and µ as
merged into a unique µ). He hereafter consider that the coefficients µ∗ and µ∗ are fixed. We will show that
using these coefficients in

the augmented partial Lagrangian allows it to reach the same value as the total
Lagrangian T Lx (QP= ) . We recall the definition of the augmented partial Lagrangian from (6.2.4b) using
more compact notations for fixed µ∗ and µ∗ :


L P Lx (QP= ) (µ , µ ) =  s.t. Ax = b
∗ ∗
·
 min Qµ∗ ,µ∗ xx> + c> µ∗ ,µ∗ x − µ
∗>
e − µ∗ > e
(6.3.7)
x ∈ Rn

For the fixed µ∗ and µ∗ determined above, we have Qµ∗ ,µ∗  0 because of the bottom-right term of (6.1.4b).
As such, the above program has the same optimum value as its SDP version obtained by replacing xx> with
X in (6.3.7) and adding X  xx> . This SDP partial Lagrangian L X (µ∗ , µ∗ ) is formally defined
PL (QP= )
by (6.2.5b).
One can easily check that T Lx (QP= ) = L T Lx (QP= ) (µ∗ , µ∗ , β ∗ ) ≤ L P Lx (QP= ) (µ∗ , µ∗ ) =
 

L P LX (QP ) (µ∗ , µ∗ ) ≤ P LX (QP= ) . Both inequalities follow from Lagrangian relaxation relations.
 
=

But since we know OP T (DU AL(SDP (P= ))) = T Lx (QP= ) = P LX (QP= ) = OP T (SDP (P= )) from
   

the fundamental hierarchy (6.3.1a), we can (re-)write:

L P Lx (QP x (QP= ) = OP T (DU AL(SDP (QP= ))) = OP T (SDP (QP= )).

   
∗ ∗
OP T =)
(µ , µ ) = T L (6.3.8)

Remark 6.3.9. In the 0–1 case, the coefficients µ∗ and µ∗ determined by solving DU AL(SDP (P= )) as
explained above can be used in the partial Lagrangian program L P Lx (QP= ) (µ , µ ) and make this pro-

∗ ∗

gram reach the value OP T (SDP (QP= )). If (QP= ) integrates the redundant constraints from Example 6.3.5,
the hierarchy (6.2.13a) collapses as stated in Remark 6.3.7, i.e., we can write L P Lx (QP= ) (µ∗ , µ∗ ) =
 

T Lx (QP= ) = P Lx (QP= ) . Applying (6.3.8), we obtain L P Lx (QP= ) (µ∗ , µ∗ ) = OP T (SDP (QP= )) =

 
 

P Lx (QP= ) , i.e., both (6.3.1a)-(6.3.1b) collapse; the convexification is optimal, reaching its limit value

P Lx (QP= ) from (6.3.1b). In fact, since we are in the 0 − 1 case, we can also use the redundant constraint

from Example 6.3.4 and obtain the same value OP T (SDP (Q= )) = P Lx (QP= ) using Prop. 6.3.8.

Finally, we showed in Example 6.3.2 that the optimal solution of DU AL(SDP (QP= )) might only converge
towards OP T (SDP (QP= )) without effectivelly reaching this value. In this case, we can have a sequence of
solutions (µ∗i , µ∗i ) whose objective values converge towards the optimum. We can apply the same calculations
from any µ∗i , µ∗i that reach an objective value arbitrarily close to the optimum OP T (SDP (QP= )).

[Link] Using the optimal convexification coefficients in a convex quadratic Branch-and-bound

Let us here consider that the only quadratic constraints of (QP= ) are of the form x2i = xi ∀i ∈ [1..n] (i.e.,
integrality constraints). However, (QP= ) can integrate various redundant constraint sets, e.g., like those
from Example 6.3.4 or Example 6.3.5. In fact, these two constraint sets produce the same SDP relaxation
value OP T (SDP (QP= )) in the 0 − 1 case,
by virtue of Prop. 6.3.8. If we use any of these constraint sets, we
obtain OP T (SDP (QP= )) = P Lx (QP= ) as stated in Remark 6.3.9. This means that the convexification
is optimal, because it reaches its limit P Lx (QP= ) .

Once the optimal coefficients µ∗ and µ∗ have been found, we focus on the fact that the augmented partial
Lagrangian L P Lx (QP= ) (µ∗ , µ∗ ) from (6.3.7) is convex with linear constraints and has the optimum value
OP T (SDP (QP= )) by virtue of (6.3.8). From now on, we can apply a convex quadratic solver to optimize
by Branch-and-bound the binary version of L P Lx (QP= ) (µ∗ , µ∗ ) from (6.3.7). One can verify that the
objective value of x in this program is equal to the value of x in (QP= ) whenever we have Ax = b and x is

72
binary (i.e., all quadratic constraints are satisfied). By solving the binary version of L P Lx (QP= ) (µ∗ , µ∗ )
we actually solve (QP= ). Optimizing a convex quadratic program takes usually less time than solving an
SDP. We solved a single SDP program (i.e., (SDP (QP= )) and its dual equal to the total Lagrangian) to
obtain the best convexification coefficients µ∗ , µ∗ ; we use the quality of the SDP bound at the root of the
Branch-and-bound tree. More importantly, at each Brand-and-bound node some of the x variables are fixed
to binary values and we can still solve a reduced partial Lagrangian only using the remaining variables. The
value of this reduced partial Lagrangian can be used to prune the node.
Appendix C.2.2 briefly discusses a more refined approach that further restricts the feasible area of
L P Lx (QP= ) (µ∗ , µ∗ ) by imposing the additional constraint xi ∈ [0, 1], equivalent to x2i ≤ xi ∀i ∈ [1..n].

7 Basic elements of several other research topics: under construc-

tion
7.1 Approximation algorithms using SDP programming
We here only present a (famous) SDP 0.8785–approximation algorithm for Max–Cut, but one should keep
in mind there are also other SDP approximation algorithms that exploit a similar approach.
Proposition 7.1.1. (Goemans-Williamson SDP approximation algorithm) Consider a weighted graph G =
([1..n], E) with weights wij ≥ 0 {i, j} ∈ E and wij = 0 ∀{i, j} ∈
/ E. The Max-Cut problem requires splitting
[1..n] in two sub-sets so as to maximize the total weighted sum of the edges with end vertices in different
subsets. The optimum value OP T (M Cw ) satisfies
0.8785 · OP T (SDPw ) < OP T (M Cw ) ≤ OP T (SDPw ),
where (SDPw ) is the SDP program:
n n

 X X 1
max

 wij (1 − Xij ) (7.1.1a)
2

i=1 j=i+1
(SDPw )



 s.t diag(X) = In (7.1.1b)

X0 (7.1.1c)
Proof. One can formulate the Max-Cut problem using variables zi ∈ {−1, 1} such that zi 6= zj implies
( and so, edge {i, j} has a contribution wij to the
that vertices i and j belong to different subsets, ) objective
n n
1
P P
function. We can formulate Max-Cut as max 2 wij (1 − zi zj ) : zi ∈ {−1, 1} ∀i ∈ [1..n] . We now
i=1 i=j+1
apply the following relaxation: we transform zi into a vector yi (with i ∈ [1..n]) of the unit sphere. The
·
product zi zj is generalized to Xij = yi yj , and so, X is a Gram matrix that needs to be SDP (use
·
Prop. A.1.8) and that satisfies Xii = yi yi = 1 (because yi belongs to the unit sphere for all i ∈ [1..n]).
We thus obtain that the above (SDPw ) from (7.1.1a)-(7.1.1c) is a relaxation of the Max-Cut problem, and
so, OP T (M Cw ) ≤ OP T (SDPw ).
We now prove 0.8785OP T (SDPw ) < OP T (M Cw ). From a feasible solution of (SDPw ) from (7.1.1a)-
(7.1.1c) we can generate a feasible Max-Cut solution as follows. Take any vector v ∈ Rn and set zi =
·
−1 if yi v ≤ 0 and zi = 1 otherwise. For different vectors v we obtain different Max-Cut solutions.
But the probability of separating yi and yj (with i, j ∈ [1..n]) so as to give rise to an objective function
contribution of wij is exactly
π
·
arccos(yi yj )
. The expected value resulting from taking different v vectors
Pn Pn
is i=1 j=i+1 wij
π
·
arccos(yi yj )
. This expected value needs to be less than or equal to OP T (M Cw ).
We will show that the expected value is greater than 0.8785 · OP T (SDPw ). For this, notice each term
1
2 ·
wij (1−yi yj ) from (7.1.1a) corresponds to wij
arccos(yi yj )
π
·
in the above sum (representing the expected
value). We calculate the minimum of

π
·
arccos(yi yj )
·
2 arccos(yi yj ) 2 α
1
2 (1 − ·
yi yj )
=
·
π 1 − yi yj
=
π 1 − cos(α)
= f (α).

73
For α = 0 we have yi ·y j = 1 and the correspondence
1
2 ·
wij (1 − yi yj ) → wij
π
·
arccos(yi yj )
is equivalent
2 α
to 0 → 0, so we can ignore such terms. It is enough to prove f (α) = > 0.8785 for any α ∈ (0, π].
π 1 − cos(α)
2 (1 − α sin (α) − cos (α))
The derivative in α is f 0 (α) = 2 . The denominator of f 0 (α) is thus positive over
π (cos (α) − 1)
the wholehinterval of interest α ∈ (0, π]. The h πderivative of the numerator is −2α cos(α), which is negative
π i
over α ∈ 0, and non-negative over α ∈ , π . As such, the numerator is 0 in α = 0 and it decreases
2 2
π π
as α increases up to and starts increasing after . This numerator continue to increase even after the
2 2
point α where f 0 (α) = 0. Consequently, the numerator is negative for α ≤ α (and so is f 0 (α)) and positive
for α ≥ α (and so is f 0 (α)). Thus, the stationary point α reaches the minimum of f . We obtain that for
α = 2.33112237 the derivative is slightly negative and for α = 2.33112238 it is slightly positive. For both
these values of α, f is greater than 0.878567. The figure below34 plots in blue the value of f close to 2.331.

Using numerical arguments (see figure above), the value of f is always greater than 0.8785.
As a side remark, it was proved that the above ratio of the Goemans-Williamson algorithm is essentially
optimal if the Unique Games Conjecture holds.35

7.2 Strong duality in the more general context of linear conic programming
All SDP and linear programs presented in this work are actually particular cases of more general linear conic
programs. We here only present how the SDP strong duality actually holds in linear conic programming. The
line of reasoning is a relatively direct generalization of analogous results from the SDP case. As mentioned
in the first paragraph of Section 2.3, the initial ideas are taken from a course of Anupam Gupta, also using
arguments from the lecture notes of László Lovász (see Footnote 11, p. 29).

7.2.1 A preliminary conic separation result

We need Prop. 7.2.4 below that generalizes Prop. 2.3.2. We first recall three basic definitions to establish a
rigorous framework.
Definition 7.2.1. A convex cone C ⊂ Rm is a set closed under linear combinations with positive coefficients.
In particular, if X, Y ∈ C , then tX ∈ C ∀t > 0 and X + Y ∈ C .
If the cone is not convex, we only have X ∈ C =⇒ tX ∈ C ∀t > 0.
Definition 7.2.2. The dual (convex) cone C ∗ of cone C is defined by
n
·
C ∗ = Y ∈ Rm : X Y ≥ 0 ∀X ∈ C .
o

Definition 7.2.3. The interior of cone C is the cone

interior(C ) = {Z ∈ C : ∃ε > 0 s.t. |Z 0 − Z| < ε =⇒ Z 0 ∈ C }.

In other words, the set interior(C ) contains an open ball around each of its elements.
The interior is a cone because if C contains a ball around Z, it also contains a ball scaled by a factor of t
(shrinked or enlarged by t) around tZ ∀t > 0.
34 The figure and the derivatives are obtained using [Link]
35 In the article “Optimal inapproximability results for MAX-CUT and other 2-variable CSPs?” by Subhash Khot, Guy
Kindler, Elchanan Mossel and Ryan O’Donnell, published in SIAM Journal on Computing in 2007, vol 37(1), pp 319–357, a
preliminary version is available at [Link]

74
 n
xi Ai − B for any x ∈ Rn , where B ∈ Rm and Ai ∈ Rm ∀i ∈ [1..n]. We
P
Proposition 7.2.4. Let F (x) =
i=1
consider a closed convex cone C ∈ R m
and its dual C ∗ . The following needs to hold.

/ interior(C ) ∀x ∈ Rn ⇐⇒
F (x) ∈

∃Y ∈ C ∗ , Y 6= 0 such that F (x)

We say that F (x) belongs to the hyperplane {X :

·· · ·
X
Y = −B

Y = −B
Y ≤ 0, ∀x ∈ Rn . (7.2.1)

Y }. Above equation (7.2.1) implies

·
Ai Y = 0 ∀i ∈ [1..n].
Proof.
⇐=
·
If F (x) Y ≤ 0 for some non-null Y ∈ C ∗ , F (x) can not belong to the interior of C because any Z ∈
·
interior(C ) satisfies Z Y > 0. Indeed, if Z ∈ interior(C ), then C contains a ball around Z, and so, there
· ·
exists ε > 0 such that Z − εY ∈ C . If we assume Z Y = 0, then (Z − εY ) Y = −ε|Y |2 < 0, since Y 6= 0.
This is a contradiction on Y ∈ C ∗ and Z − εY ∈ C . As such, any Z ∈ interior(C ) satisfies Z Y > 0, and
/ interior(C ).
so, F (x) ∈
·
=⇒
We know that interior(C ) does not intersect the image of F . We can apply the hyperplane separation
Theorem C.4.1: there exists a non-zero Y ∈ Rm and a c ∈ R such that

F (x)
· Y ≤c≤X
· Y ∀x ∈ Rn , ∀X ∈ interior(C ) (7.2.2)

·
It is clear that we can not have c > 0 because X Y can be arbitrarily close to 0 by choosing X =
εZ ∈ interior(C ) for an arbitrarily small ε > 0 and some Z ∈ interior(C )—recall εZ ∈ interior(C ),
·
because interior(C ) is a cone (Def. 7.2.3). We now prove X Y ≥ 0 ∀X ∈ interior(C ). Let us assume the
·
contrary: ∃X ∈ interior(C ) such that X Y = c0 < 0. Since interior(C ) is a cone (Def 7.2.3), we have
·
tX ∈ interior(C ) ∀t > 0. The value (tX) Y = tc0 can be arbitrarily low by choosing an arbitrarily large
· ·
t, and so, (tX) Y can be easily less than c, contradiction. This means that X Y ≥ 0 ∀X ∈ interior(C ).
Using c ≤ 0, (7.2.2) simplifies to

F (x)
· Y ≤0≤X
· Y ∀x ∈ Rn , ∀X ∈ interior(C )

·
It is not hard to prove that Y ∈ C ∗ . For this, we will show that X Y ≥ 0 ∀X ∈ C , relying on the
·
above proved fact X Y ≥ 0 ∀X ∈ interior(C ). Assume the contrary: there is some X ∈ C such that
·
X Y < 0. For any ε > 0, we have X + εZ ∈ interior(Sm +
) for any Z ∈ interior(C )—because of the
cone property and of the fact that if C contains a ball centered at Z, then C contains this ball shrinked
·
by ε centered at X + εZ. For a small enough ε, (X + εZ) Y remains strictly negative, which contradicts
·
X + εZ ∈ interior(C ). We obtain X Y ≥ 0 ∀X ∈ C . This means that Y ∈ C ∗ and Y satisfies

F (x)
· Y ≤ 0 ∀x ∈ Rn . (7.2.3)

· ·
We write d = F (0) Y = −B Y . Assume for the
sake of contradiction there is an x ∈ Rn such that
F (x) = d + d0 with d0 6= 0. Let us calculate F −d+1
d0 x Y = d + −d+1
(7.2.3). The assumption d0 6= 0 was false, and so, we obtain
·
0
d0 d = d − d + 1 = 1, which contradicts

F (x)
· Y = F (0)
· Y = −B
· Y ≤ 0, ∀x ∈ Rn

where we used (7.2.3) for the last inequality.

75
7.2.2 Recalling the weak duality for conic programming
Let us introduce the first conic linear program in variables x ∈ Rn based on closed convex cone C, generalizing
(SDP ) from (2.1.1a)-(2.1.1c). 
X n

min ci x i (7.2.4a)






 i=1
(ConicLP) X n


 s.t Ai x i − B ∈ C (7.2.4b)



 i=1
x ∈ Rn

(7.2.4c)
By relaxing the conic constraint (7.2.4b) with multipliers Y ∈ C ∗ , we obtain the following Lagrangian:

L (x, Y ) =
n
X

i=1
ci xi − Y
· (
X
Ai xi − B)

Notice that if x satisfies

when Y ∈ C ∗ and
P (7.2.4b), we get awarded in the above Lagrangian because Y
Ai xi − B ∈ C. Thus, minn L (x, Y ) is a relaxation of (ConicLP):
·(P A x − B) ≥ 0
i i

x∈R

min L (x, Y ) ≤ OP T (ConicLP), (7.2.5)

x∈Rn

where OP T (ConicLP) can also be unbounded (−∞).

Let us write:
minn L (x, Y ) = minn Y
x∈R x∈R · B+
n
X
(ci − Y
i=1
· Ai )xi

·
If there is a single i ∈ [1..n] such that ci − Y Ai 6= 0, the above minimum is −∞ (unbounded), by using
an appropriate value of xi . To have a bounded minx∈Rn L (x, Y ), Y needs to verify ci − Y Ai = 0 for all
i ∈ [1..n]. We are interested in finding:
·
max∗ minn L (x, Y )
Y ∈C x∈R

·
that can be written: 
max B
 Y (7.2.6a)

·

(DConicLP) s.t Ai Y = ci ∀i ∈ [1..n] (7.2.6b)



Y ∈ C∗ (7.2.6c)
Based on (7.2.5), we obtain the weak duality:

OP T (DConicLP) ≤ OP T (ConicLP). (7.2.7)

In particular, if OP T (ConicLP) = −∞, then (DConicLP) needs to be infeasible, because otherwise the above
(7.2.7) would limit OP T (ConicLP) from going to −∞.

7.2.3 Strong duality for the primal form

The following theorem generalizes Theorem 2.3.3 of the SDP case.
Theorem 7.2.5. If the primal (ConicLP) from (7.2.4a)-(7.2.4c) is bounded and has a strictly feasible solution
(Slater’s interiority condition), then the primal and the dual optimal values are the same and the dual
(DConicLP) from (7.2.6a)-(7.2.6c) reaches this optimum value. Recall (last phrase of Section 7.2.2) that if
(ConicLP) is unbounded, then (DConicLP) is infeasible.
n
P Pn
Proof. Let p be the optimal primal value. The system ci xi < p and i=1 Ai xi − B ∈ C has no solution.
i=1
We define    
−ci −p
Ai = ∀i ∈ [1..n] and B = ,
Ai B

76
and the closed convex cone   
x
C= : x ≥ 0, X ∈ C
X
n n
/ interior(C) ∀x ∈ Rn (we can not say
P P
Notice that Ai x i − B ∈ Ai x i − B ∈
/ C, as the optimal solution
i=1 i=1
x can cancel the value −p on the first position of the expression). We can thus apply Prop. 7.2.4 (implication
∗
·
“=⇒”) and conclude there is some non-zero Y ∈ C such that Ai Y = 0 and −B Y ≤ 0. Writing Y = [ Yt ]
∗
with t ≥ 0 and Y ∈ C ∗ (necessary and sufficient conditions to have Y ∈ C ), we obtain:
·
tci = Ai
· Y, ∀i ∈ [1..n] (7.2.8)

and

Assume t = 0. We obtain Ai
−B
· Y ≤ −tp.

· Y = 0 ∀i ∈ [1..n] andP−B · Y ≤ 0 with Y ∈ C ∗

(7.2.9)

as discussed above. Applying

n
again Prop. 7.2.4 (“⇐=” implication), we conclude / interior(C) ∀x ∈ Rn , which contradicts
Ai x i − B ∈
i=1
the fact the primal (ConicLP) from (7.2.4a)-(7.2.4c) is strictly feasible. We thus need to have t > 0.
Taking Yb = 1t Y ∈ C ∗ (recall Def. 7.2.1 and Def. 7.2.2), the above (7.2.8)-(7.2.9) become: ci = Ai Yb ∀i ∈ ·
·
[1..n] and B Yb ≥ p. In other words, Yb is a feasible solution in the dual (DConicLP) from (7.2.6a)-(7.2.6c)
·
and it has an objective value B Yb ≥ p. Using the weak duality (7.2.7), the feasible solution Yb of (DConicLP)
· ·
needs to satisfy B Yb ≤ p, and so, B Yb = p, i.e., the dual achieves the optimum primal value in Yb .

7.2.4 Strong duality for dual forms

We start from the conic dual. For the reader’s convenience, we repeat the definitions of (DConicLP)
from (7.2.6a)-(7.2.6c):

max B

· Y (7.2.10a)

·

(DConicLP) s.t. Ai Y = ci ∀i ∈ [1..n] (7.2.10b)



Y ∈ C∗ (7.2.10c)
The following is a generalization of Prop. 2.1.2.

Proposition 7.2.6. Program (DConicLP) from (7.2.10a)-(7.2.10c) is equivalent to (ConicLP0 ) from (7.2.13a)-
(7.2.13c) below which is a program in the primal form (7.2.4a)-(7.2.4c), provided that the system of linear
·
equations Ai Y = ci ∀i ∈ [1..n] from (7.2.10b) has at least a feasible a solution.
Proof. We first solve the system Ai
The set of all solutions is given by
·Y = c i ∀i ∈ [1..n]. This system has at least a feasible solution −B 0 .

k
X
Y = −B 0 + A0j x0j with x0j ∈ R ∀j ∈ [1..k], (7.2.11)
j=1

where A01 , A02 , . . . A0k are a basis of the null space of {Ai : i ∈ [1..n]} satisfying:

Ai
· A0j = 0, ∀i ∈ [1..n], j ∈ [1..k]. (7.2.12)

The space spanned by (the linear combinations of) Ai (∀i ∈ [1..n]) and A0j (∀j ∈ [1..k]) need to cover the
whole space Rm , by virtue of the rank-nullity Theorem A.1.3. Replacing (7.2.11) in the objective function
· ·
 Pk
of (DConicLP) from (7.2.10a), we obtain objective function max −B 0 + j=1 A0j x0j B = max −B B 0 +

77
Pk
j=1 (B · A )x
0
j
0
j = − min B ·B + P
0 k
j=1 (−B · A )x . The dual (DConicLP) can be written:
0
j
0
j

· ·

 Xk
0
− min B B + (−B A0j )x0j (7.2.13a)







 j=1
0 k
(ConicLP ) X


 s.t. A0j x0j − B 0 ∈ C ∗ (7.2.13b)

j=1




x0 ∈ R k ,

(7.2.13c)
which is a linear conic program in the primal form (7.2.4a)-(7.2.4c).
Applying the duality (ConicLP) → (DConicLP) from Section 7.2.2 on above (ConicLP0 ), we obtain
(DConicLP0 ) below. Notice we proved in Prop. 5.4.1 that closure(C) = C ∗ ∗ . Since we said from the
beginning (first paragraph of Section 7.2.2) that C is closed, we can use C = C ∗ ∗ .

− max B B + B
 0
· ·0
Y0 (7.2.14a)

· ·

(DConicLP0 ) s.t. A0j Y 0 = −B A0j ∀j ∈ [1..k] (7.2.14b)


Y 0 ∈ C ∗∗ = C

(7.2.14c)
The system of equations (7.2.14b) has at least the solution Y 0 = −B and this allows us to express (DConicLP0 )
in the primal form using (the approach from) Prop 7.2.6. Any solution of this system could be written as
−B plus a linear combination of vectors from the null space of {A0j : j ∈ [1..k]}. But we said in Prop 7.2.6
(see (7.2.12) and above) that {A0j : j ∈ [1..k]} are a basis of the null space of {Ai : i ∈ [1..n]}. Vectors A0j
(with j ∈ [1..k]) and Ai (with i ∈ [1..n]) span the whole space Rm , by virtue of the rank-nullity Theorem
A.1.3. The null space of {A0j : j ∈ [1..k]} can thus be generated by the linear combinations of (a maximum
set of independent vectors of) {Ai : i ∈ [1..n]}. This is enough to allow us to say that all solutions Y 0 of
(7.2.14b) verify:
Xn
Y 0 = −B + Ai xi with xi ∈ Rn ∀i ∈ [1..n].
i=1
Replacing this in the objective function (7.2.14a) of (DConicLP0 ) above, we obtain the objective function in
variables x1 , x2 , . . . xi :

· · ·
n
! n n n
X X X X
0
− max B B + −B + Ai x i B 0 = − max (Ai B 0 )xi = − max −ci xi = min ci xi ,
i=1 i=1 i=1 i=1

· 0
where we used Ai −B = ci from the first line of the proof of Prop. 7.2.6. This way, replacing the value of
Y 0 , program (DConicLP0 ) from (7.2.14a)-(7.2.14c) is completely equivalent to:
 n
 X
min ci x i (7.2.15a)





 i=1

n
(ConicLP) X


 s.t Ai x i − B ∈ C (7.2.15b)


 i=1

x ∈ Rn ,

(7.2.15c)
which is exactly the initial program from (7.2.4a)-(7.2.4c).
Theorem 7.2.7. If the dual (DConicLP) from (7.2.10a)-(7.2.10c) (or (7.2.6a)-(7.2.6c)) is bounded and has a
strictly feasible solution, then the primal and the dual optimal values are the same and the primal (ConicLP)
from (7.2.15a)-(7.2.15c) (or (7.2.4a)-(7.2.4c)) reaches this optimum value.
Proof. Since (DConicLP) is feasible, we can apply Prop. 7.2.6 and obtain that (DConicLP) can be reformulated
as (ConicLP0 ) from (7.2.13a)-(7.2.13c). This (ConicLP0 ) needs to be bounded and to have a strictly feasible
solution as well, and so, we can apply Theorem 7.2.5 that states that the dual of (ConicLP0 ) reaches the
optimum value of (ConicLP0 ). But the dual of (ConicLP0 ) is (DConicLP0 ) from (7.2.14a)-(7.2.14c), which
is completely equivalent to (ConicLP) from (7.2.15a)-(7.2.15c). We obtained that (ConicLP) reaches the
optimum value of (DConicLP).

78
Theorem 7.2.8. If both the primal (ConicLP) from (7.2.15a)-(7.2.15c) (or (7.2.4a)-(7.2.4c)) and the dual
(DConicLP) from (7.2.10a)-(7.2.10c) (or (7.2.6a)-(7.2.6c)) are strictly feasible, then they have the same opti-
mum value and they both reach it.
Proof. Using weak duality (7.2.7), both programs need to be bounded. The conclusion then simply follows
from combining Theorem 7.2.5 and Theorem 7.2.7.

7.3 Polynomial Optimization

7.4 Algorithms for SDP optimization

A On ranks, determinants and space dimensions

This is the most elementary section from this document. If you are an absolute beginner, you could read
first App. A.2 before App. A.1.

A.1 The rank-nullity theorem and other interesting rank properties

Definition A.1.1. Given matrix A ∈ Rn×m , the rank rank(A) has two equivalent definitions:
(a) the largest order of any non-zero minor of A (see Def A.2.1)

(b) the maximum number of independent rows (or columns) of A, i.e., the dimension of the space generated
by the rows (or columns) of A using linear combinations (see also Def. A.2.7 for the notion of dimension).
Proof.
(a) =⇒ (b)
We show that the existence of a non-zero minor of order r. Without loss of generality, we permute rows
and columns until the non-zero minor is positioned in the upper-left corner, and, so we can consider that
det([A]r ) 6= 0, where [A]r is the leading principal minor of size r × r, where r = rank(A). Since the rows
and columns of [A]r are independent, the corresponding rows and columns in the full matrix are linearly
independent (in A). We have at least r independent rows and columns.
We will show that any row ai with i > r is dependent of rows a1 , a2 , . . . ar . Since det([A]r ) 6= 0, there
exist x1 , x2 , . . . xr ∈ R such that

[ai1 ai2 . . . air ] = [x1 x2 . . . xr ][A]r (A.1.1)

Take any column j ∈ [r + 1..m] and consider the matrix obtained by bordering [A]r with row i and column
j.  
a1j
 a2j 
[A]
 
 r .. 

 . 

 arj 
ai1 ai2 . . . air aij
We subtract from the last row the first r rows respectively multiplied by x1 , x2 , . . . xr . By virtue of (A.1.1),
the above matrix becomes:  
a1j
 a2j 
[A]r
 
 .
.. 
 
 
 a
Prrj

0 0 . . . 0 aij − k=1 xk akj
Since [A]r is the largest non zero minor, both above Prbordered matrices need to have a zero determinant. But
the determinant
Pr of the last above matrix is (aij − k=1 xk akj ) det([A]r ). Since det([A]r ) 6= 0, we obtain that
aij = k=1 xk akj . Since this needs to hold for all j ∈ [r + 1..m], we obtain ai = x1 a1 + x2 a2 + . . . xr ar , and
so, row i is dependent on the first r rows. The number of independent rows is thus exactly r. An analogous

79
argument can be used to obtain that the number of independent columns is r. This is the dimension of the
space generated by the rows (resp. columns).
(b) =⇒ (a)
The determinant of any minor that contains rows r + 1 rows (resp. columns) is zero, because one of these
rows (resp. columns) can be written as a linear combination of the other r. The order of the largest non-zero
minor is thus at most r. It is easy to see that this largest order can not be r − 1. If that were the case, we
could apply the above (a) =⇒ (b) proof to show that all rows are dependent on some r − 1 rows.
Proposition A.1.2. Given symmetric matrix A ∈ Rn×n of rank r, A has at least one non-zero principal
minor of order r.
Proof. The rank definition ensures the existence of a set of rows J (with |J| = r) such that any other row
i ∈ [1..n] − J can be written as a linear combination of rows J. This means that the matrix A reduced to
rows J has full rank r. By symmetry, each column of this latter matrix with r rows can be written as a linear
combination of the columns J. This means that the matrix A reduced to rows J and (then) to columns J
has full rank r, i.e., the principal minor corresponding to rows J and columns J is non-zero.
The above proposition is sometimes referred to as the “principal minor theorem” and it also holds if A
is skew-symmetric, i.e., if A> = −A.
Theorem A.1.3. (Rank-nullity theorem) Given A ∈ Rn×m , the null space (kernel) of A is given by

null(A) = {x ∈ Rm : Ax = 0n } (A.1.2)
We denote by nullity(A) the dimension (maximum number of linearly independent vectors—see also Def. A.2.7)
of null(A). The following equation holds:

m = rank(A) + nullity(A) (A.1.3)

Proof. Without loss of generality, we permute rows and columns until a non-zero principal minor of size
r = rank(A) is positioned in the upper-left corner, so that the leading principal minor [A]r has a non-null
determinant. We want to study the set of solutions of

Ax = 0n (A.1.4)
Using Def A.1.1, the last n − r rows of A can be written as a linear combination of the first r rows. It is
enough to investigate only the first r equations in above system (A.1.4). We can say that x is a solution of
above system if and only if it is a solution of the following:
 
a1,r+1 a1,r+2 . . . a1,m
a2,r+1 a2,r+2 . . . a2,m 
[A]r

..  x = 0r
 
 .. .. ..
 . . . . 
ar,r+1 ar,r+2 ... ar,m
Using simple notational shortcuts, we write the above as follows:
i 
xr
h
[A]r [A]m−r = 0r , (A.1.5)
xm−r
where [A]m−r is the A minor obtained by selecting the first r rows and last m − r columns; xr selects the
first r components of x and xm−r selects the last m − r. We can further re-write (A.1.5) above as:
[A]r xr + [A]m−r xm−r = 0r
We can now write xr as a function of xm−r , more exactly:
xr = −[A]−1
r [A]m−r xm−r

We can say that each of the first r components of x (i.e., xr ) can be written as a linear combination of the
last m − r components (i.e., xm−r ). The dimension of the space generated by all above solutions x of (A.1.4)
reduces to the dimension of the space generated by the last m − r components xm−r , i.e., this dimension is
m − r, which confirms (A.1.3).

80
Proposition A.1.4. Given A ∈ Rn×m and B ∈ Rm×p , we have


rank(AB) ≤ min rank(A), rank(B) (A.1.6)

Proof. Take any x ∈ Rn in the null space of A> , i.e., A> x = 0m or x> A = 0> >
m . We observe that x AB =
> >
0m B = 0p . This means that x also belongs to the null space of (AB) . This means that
the null space
of (AB)> is greater than or equal to the null space of A> . We can write nullity (AB)> ≥ nullity(A).
theorem (Th A.1.3), we have
rank(A> ) + nullity(A> ) = n and rank (AB)> +


Using the rank-nullity
nullity (AB)> = n. This means that rank (AB)> ≤ rank(A> ), equivalent to rank(AB) ≤ rank(A).

Analogously, we can obtain rank(AB) ≤ rank(B), which leads to (A.1.6).

Proposition A.1.5. Given any A ∈ Rn×n and invertible U , the following holds:

rank(AU ) = rank(A)

Proof. Using (A.1.6), we have rank(A) ≥ rank(AU ) ≥ rank(AU U −1 ) = rank(A), which means that all
inequalities are actually equalities.
Proposition A.1.6. Any similar matrices A, B ∈ Rn×n (i.e., such that B = U −1 AU for some U ∈ Rn×n )
have the same rank.
Proof. This simply follows from applying Proposition A.1.5 twice, once for U and once for U −1 .

Proposition A.1.7. The rank of symmetric A ∈ Rn×n is equal to n minus the multiplicity of the eigenvalue
0. We can write:
n = rank(A) + eigenmult(0)
Proof. We use the eigendecomposition (as proved in Prop B.2.1):

A = U ΛU > ,

where U is invertible (U −1 = U > ) and Λ is a diagonal matrix with the eigenvalues of A on the diagonal.
Using Prop A.1.6, A and Λ have the same rank. Since Λ is diagonal, its rank is equal to the number of non-
zero elements on the diagonal, i.e., n minus the multiplicity of eigenvalue 0. Additionally, remark that we
proved in Prop. B.2.3 and Prop. B.2.5 that in symmetric matrices the algebraic multiplicity of an eigenvalue
is equal to its geometric multiplicity, hence we use the term multiplicity to refer to both.
Proposition A.1.8. Given V ∈ Rm×n , the matrix A = V > V ∈ Rn×n is SDP. We say A is the Gram
matrix of the column vectors of V . Furthermore, rank(V ) = rank(V > V ) = rank(V > SV ) for any positive
definite S ∈ Rm×m .
Proof.
1) To see V > V is SDP, notice that for any x ∈ Rn , we have Px
>
(V > V )x = (x> V > )(V x) = (V x)> (V x) =
n
|V x|2 . Also, by writing y = V x, this value simply becomes i=1 yi2 ≥ 0.
2) rank(V > V ) = rank(V ) follows from rank(V > SV ) = rank(V ) with S = Im . We now prove the equality
for an arbitrary S  0. We will show V > SV and V have the same null space so as to apply the rank-nullity
Theorem A.1.3. It is clear that any x ∈ Rn in the null space of V belongs to the null space of V > SV , because
V x = 0m =⇒ V > SV x = 0n . We now prove the converse: V > SV x = 0n =⇒ V x = 0m . We multiply both
sides of V > SV x = 0n by x> and we obtain x> V > SV x = 0, equivalent to (V x)> S(V x) = 0. Using S  0,
this value can only be zero if V x = 0m . We obtained that the null space of V is equal to the null space of
V > SV . Using the rank-nullity Theorem A.1.3, the two matrices need to have the same rank.

A.2 Results on determinants and space dimensions

Definition A.2.1. Given matrix A of size n × m, a minor of A is a sub-matrix obtained by selecting only
some rows J1 ⊆ [1..n] and some columns J2 ⊆ [1..m] of A. A principal minor [A]J is a minor obtained by
selecting the same rows and columns J = J1 = J2 . A principal minor [A]J is a leading principal minor if
J = [1..p] for some p ≤ n, m. We say [A]J is null (or zero) if det([A]J ) = 0. The order of [AJ ] is |J|.

81
A.2.1 Very elementary results on matrices and determinants
Proposition A.2.2. Given complex matrix A ∈ Cn×n , det(A) = 0 ⇐⇒ ∃u ∈ Cn − {0} such that Au = 0.
Proof.
⇐=
Take u 6= 0 such that Au = 0. Without loss of generality, we assume u1 6= 0. We can write:
         
a11 a21 a31 an1 0
 a12  u  a22  u  a32  u  an2  0
2  3  n 
 ..  +  . +  .  + ...... +  .  = .
      
 .  u1  ..  u1  ..  u1  ..   .. 
a1n a2n a3n ann 0

We use the fact that adding a multiple of a column to another column does not change the determinant. By
performing all additions of column multiples from above formula, we obtain only zeros in the first column.
The determinant of the resulting matrix can only be zero.
=⇒
We proceed by induction. For n = 1, the implication is obviously true. Assume that it holds for n − 1. The
implication is also obvious if A = 0n×n . We assume that A has some non-zero elements and without loss of
generality we can use a11 6= 0 (it is enough to permute lines/columns to obtain this).
We want to find u ∈ Cn − {0} such that Au = 0. A can be changed to a form in which a11 becomes 1:
it is enough to divide first line by the initial a11 to obtain an equivalent system of equations. To simplify
notations, we can continue assuming a11 = 1. We perform Gaussian elimination on first column using pivot
a11 = 1. We want to solve:
    
1 a12 a13 ... a1n u1 0
0 a22 − a12 a23 − a13 . . . a2n − a1n   u2  0
    
0 a32 − a12 a33 − a13 . . . a3n − a1n   u3  0
   =  
 ..   ..   .. 
.   .  .
0 an2 − a12 an3 − a13 . . . ann − a1n un 0

Using compact notations, this can also be written:

b> u1
    
1 0
= (A.2.1)
0n−1 B v 0n−1

Since above
row multiplications and addition did not change the determinant of the matrix, we obtain
1 b>
det 0n−1 = 0, which means det(B) = 0. We can use the induction hypothesis: there exists non-zero
B
n−1
v∈C such that Bv = 0n−1 . The last n-1 equations of above system are satisfied. We
 still need to find
u1 such that u1 + b> v = 0 =⇒ u1 = −b> v. We have just found a solution u = uv1 = 0 for (A.2.1). It
is easy to check that u verifies Au = 0, i.e., it is enough to reverse the Gaussian elimination since the first
equation/line is verified by u.
Corollary A.2.3. If we replace C by R in the proof of above Proposition A.2.2, everything remains correct.
The solution of the system will be real, because u1 = −b> v is real using the induction hypothesis that
v ∈ Rn−1 .
Proposition A.2.4. The determinant of a matrix A is the product of its eigenvalues. The trace of A is
equal to the sum of the eigenvalues.
Proof. Consider the characteristic polynomial det(xI −A). The eigenvalues of A are the roots of det(xI −A),
and so,
det(xI − A) = (x − λ1 )(x − λ2 ) . . . (x − λn ). (A.2.2)
1) It is enough to evaluate this equation in x = 0 and we obtain det(−A) = (−1)n λ1 λ2 . . . λn . We also
have det(−A) = (−1)n det(A), because any term in the Leibniz formula for determinants is a product of n
elements of the matrix. This simply leads to (−1)n det(−A) = (−1)n λ1 λ2 . . . λn , and so, det(A) = λ1 λ2 . . . λn

82
2) We evaluate
Pn the term corresponding to xn−1 of both sides of (A.2.2). PIn the right side we obtain the
n
term − i=1 λi xn−1 . We will show that in the lefthand side we obtain − i=1 aii xn−1 . Using the Leibniz
formula for determinants, we observe that the only determinant terms that make x arise n − 1 times in
det(xI − A) are those that use n − 1 diagonal terms of the form x − aii with i ∈ [1..n]. Such a determinant
term needs to also use the nth diagonal value as well. To find Pthe term of xn−1 in det(xI − A) we need to
n−1 n
develop (x − a11 )(x − a22 ) . . . (x − ann ). The x term is − i=1 aii xn−1 .
Proposition A.2.5. Given A, B ∈ Cn , if AB = In then BA = In . We want a proof for the lazy, without
calculating some left A−1 or some right B −1 .
Proof. We first show that the columns of B are linearly independent. This follows from α ∈ Cn − {0} =⇒
(AB)α 6= 0. It is clear we can not have Bα = 0 because this would make ABα = 0. The columns of B
need to be linearly independent, and so, their linear combinations cover a space of dimension n, i.e., Cn (see
also Prop. A.2.7). As such, for any column xi of In = [x1 x2 . . . xn ], there exist a linear combination of the
columns of B that is equal to xi , i.e., ∃yi s. t. xi = Byi . Joining toghether all columns xi of In , there exists
Y = [y1 y2 . . . yn ] s. t. In = BY . We finish by Y = (AB)Y = A(BY ) = A which proves BA = In .

A.2.2 The dimension of a (sub-)space

Definition
Pk A.2.6. ThePvectors x1 , x2 , . . . xk ∈ Rn are affinely indepedent if there is no λ1 , λ2 , . . . λk with
k
i=1 λi = 0 such that i=1 λi xi = 0n . This is equivalent to the fact that x1 − xk , x2 − xk , . . . xk−1 − xk
are linearly independent.
Proof. It is enough to show that

x1 , x2 , . . . xk affinely dependent ⇐⇒ x1 − xk , x2 − xk , . . . xk−1 − xk linearly dependent

=⇒ Pk Pk
Given λ1 , λ2 , . . . λk with i=1 λi = 0 such that i=1 λi xi = 0n , we obtain
k−1
X k
X k
X k
X
λi (xi − xk ) = λi (xi − xk ) = λi xi − λi xk = 0n − 0xk = 0n
i=1 i=1 i=1 i=1
| {z }
we added +λk (xk − xk )

⇐= Pk−1 Pk−1 Pk−1

We consider there is λ1 , λ2 , . . . λk−1 such that i=1 λi (xi −xk ) = 0, equivalent to i=1 λi xi − i=1 λi xk =
Pk−1 Pk−1
0. By taking λk = − i=1 λi , this reduces to i=1 λi xi + λk xk = 0.
Definition A.2.7. A subspace S ⊆ Rn has dimension k if k + 1 is the maximum number of affinely
independent vectors x0 , x1 , x2 . . . xk ∈ S. This is equivalent to the existence of maximum k linearly in-
dependent vectors x1 − x0 , x2 − x0 , . . . xk − x0 by virtue of Def A.2.6. We say that x0 is an orgin and
x1 − x0 , x2 − x0 , . . . xk − x0 are a basis of S.
If S contains 0n , we can take x0 = 0n and the definition is equivalent to the existence of maximum k
linearly independent vectors x1 , x2 , . . . xk .

A.2.3 Every eigenvalue belongs to a Gershgorin disc

Theorem A.2.8. (Gershgorin circle theorem) Given complex matrix A ∈ Cn×n , every eigenvalue λ belongs
to a Gershgorin disk of the following form for some i ∈ [1..n].
 
 

 Xn 

z ∈ C : |z − Aii | ≤ |Aij |
 

 j=1 

j6=i

Proof. Consider eigenvalue λ with an eigenvector v ∈ Cn − {0}. Take an index i ∈ [1..n] such that |vi | ≥
vj
|vj | ∀j ∈ [1..n]. This means that ≤ 1 ∀j ∈ [1..n]. We now develop position i of Av = λv and we obtain
vi

83
 n
P n
P vj
λvi = Aij vj . Dividing this by vi leads to λ = Aii + Aij . We can further develop:
j=1 j=1 vi
j6=i

X vj
|λ − Aii | = Aij
j=1
vi
j6=i
X vj
≤ Aij (we used the triangle inequality)
j=1
vi
j6=i
n
X vj
≤ |Aij | , (we used ≤ 1)
j=1
vi
j6=i

which proves that λ belongs to the Gershgorin disk associated to i.

B Three decompositions: eigenvalue, QR and square root

B.1 Preliminaries on eigen-values/vectors and similar matrices
Proposition B.1.1. Given matrix A ∈ Rn×n , one can always find some λ ∈ C and u ∈ Cn such that
Au = λu, i.e., there is at least one eigenvalue associated to an eigenvector.

Proof. We consider the characteristic polynomial det(xI −A) = 0. Using the fundamental theorem of algebra,
this polynomial has at least one complex root λ. We have det(A − λI) = 0. Using Proposition A.2.2, there
exists some u ∈ Cn such that (A − λI)u = 0 =⇒ Au = λu.
Proposition B.1.2. All eigenvalues and proper eigenvectors of a real symmetric matrix are real (non
complex).

Proof. Take symmetric matrix A ∈ Rn×n , as well as λ and u such that λu = Au. Let u∗ be the conjugate
transpose of u, i.e., transpose u and negate all terms that contain i. We have:

u∗ λu = u∗ Au

We take the conjugate transpose of both sides (transpose and then negate imaginary terms) and obtain:

u∗ λ∗ u = u∗ Au.

We used the fact that the conjugate of an expression can be obtained by conjugating each of the expression’s
∗ ∗
terms,36 as well as (u∗ ) = u and (A∗ ) = A. The two above expressions have the same right-hand side and
need to be Pequal: u λu = u λ u =⇒ λ(u∗ u) = λ∗ (u∗ u). Since (a + bi)(a − bi) = a2 + b2 , it is easy to
∗ ∗ ∗
n
check that i u∗i ui is a strictly positive real (unless u = 0). As such, λ = λ∗ , i.e., λ is real.
We now show that a proper eigenvector u is real. Suppose u = ua + ub i, with ua , ub ∈ Rn . Since
(A − λI)(ua + ub i) = 0, we obtain: (A − λI)ua = 0 and (A − λI)ub = 0. This means that u is merely a
combination of other eigenvectors ua and ub of λ. We consider that the proper eigenvectors are ua and ub .
One can always multiply them by complex numbers and combine them to obtain eigenvectors like u.
Definition B.1.3. (algebraic and geometric multiplicity) The algebraic multiplicity of an eigenvalue λ of
matrix A ∈ Cn×n is the multiplicity of root λ in the characteristic polynomial det(xI − A). The geometric
multiplicity of λ is the dimension of the eigen space {u ∈ Cn : Au = λu}, i.e., the maximum number of
linearly independent eigenvectors of λ (this is how we calculate the dimension of any space that includes 0,
recall Def. A.2.7). The two multiplicities are not necessarily equal.
36 If you are unfamiliar with complex numbers, take the product of two complex numbers: a a − b b + (a b + a b )i =
1 2 1 2 1 1 2 2
(a1 + b1 i)(a2 + b2 i). By conjugating each term in the right-hand side, we obtain the conjugate of the left-hand side, i.e.,
a1 a2 − b1 b2 − (a1 b1 + a2 b2 )i. For additions, the property is even easier to verify.

84
  
Proof. We give an example in which the two multiplicities are not equal. Take A = 10 21 . The characteristic
polynomial is (x − 1)2 = 0, and so, the multiplicity of eigenvalue λ = 1 is 2. To compute the geometric
multiplicity, we determine the solutions of u1 = u1 + 2u2 and u2 = 0u1 + u2 . We obtain u2 = 0 and u1 can
take any value. The eigen space of λ contains all vectors u with u2 = 0; the dimension of this space is 1.

Proposition B.1.4. Any similar matrices A, B ∈ Cn×n (i.e., such that B = U −1 AU for some non-singular
U ∈ C n×n ) have the same characteristic polynomial. Consequently, A and B have the same eigenvalues with
the same algebraic multiplicities. We also say that B is the representation of A in the basis determined by
the columns of U ; the change of basis matrix from this basis to the canonical basis is exactly U .
Proof.
det(xI − B) = det(xI − U −1 AU )
= det(xU −1 U − U −1 AU )
= det(U −1 (xI − A)U )
= det(U −1 ) · det(xI − A) · det(U )
= det(xI − A)

Proposition B.1.5. Any two similar matrices A, B ∈ Cn×n have the same geometric multiplicity for any
common eigenvalue λ.
Proof. Take any fixed λ and uB ∈ Cn such that BuB = λuB . We can write λuB = U −1 AU uB , which is
equivalent to λU uB = AU uB . As such, A has eigenvector uA = U uB with eigenvalue λ. There is a bijection
between the eigenvectors of B and the eigenvectors of A, given by above transformation uB → uA = U ub . To
check the bijectivity, notice the injectivity follows from U uB = U u0B =⇒ U (uB − u0B ) = 0 =⇒ uB = u0B
based on the non-singularity of U . The surjectivity follows from ∀uA ∈ Cn , ∃uB = U −1 uA ∈ Cn such that
uA = U uB . This bijection shows that the geometric multiplicities of λ are the same.

B.2 The eigenvalue decomposition

Proposition B.2.1. (Eigendecomposition) Any symmetric matrix A ∈ Rn×n can be decomposed as follows:

A = U ΛU > (B.2.1)
Xn
= λi u×,i u>
×,i , (B.2.2)
i=1

where Λ = diag(λ1 , λ2 , . . . λn ) contains the eigenvalues of A and U contains columns u×,1 , u×,2 , . . . u×,n
that represent the orthonormal unit eigenvectors of A.
Proof. We provide two proofs (Appendix B.2.1 and Appendix B.2.2) for showing the key fact (B.2.1):
– The first proof was actually given in Section 1.1 and we here only repeat it in greater detail and a bit
generalized. It relies on the equality between the geometric and algebraic multiplicities of each eigenvalue.
– Use the Schur decomposition of complex matrices and above (B.2.1) becomes a simple re-writing of
(B.2.9) in Prop. B.2.5. Reading this second proof is useful to develop a general culture (on complex or
asymmetric matrices).

Both above proofs also show that U contains n orthonormal unit eigenvectors of A and that this way we have
U > = U −1 . After showing (B.2.1) with either proof, (B.2.2) simply follows from developing A = U ΛU > :
 > 
λ1 u>
 
u×,1 ×,1
 u>  λ2 u> n
 ×,2  ×,2 
  X
A = [u×,1 , u×,2 , . . . , u×,n ]Λ  .  = [u×,1 , u×,2 , . . . , u×,n ]  .  = λi u×,i u>

×,i
 ..   .. 
i=1
u>
×,n λn u>
×,n

85
Writing eigenvector vi = u×,i ∈ Rn , the above formula can be expressed in a very compact form:
n
X
A= λi vi vi> , (B.2.3)
i=1

where v1 , v2 , . . . vn are orthonormal eigenvectors (pairwise orthogonal and of unitary norm).

B.2.1 Proof using the equality of the geometric and algebraic multiplicities
Proposition B.2.2. Consider (possibly non-symmetric) matrix A ∈ Rn×n such that each eigenvalue λi has
the same geometric and algebraic multiplicity ki . This means root λi arises ki times in the characteristic
polynomial and there are ki linearly independent eigenvectors of λi . Matrix A has the following eigen-
decomposition (diagonalization):
A = U diag(λ1 , λ2 , . . . λn )U −1 (B.2.4)
where U contains n orthonormal eigenvectors of A and diag(λ1 , λ2 , . . . λn ) is a diagonal matrix with the
(possibly complex) eigenvalues on the diagonal.
If A is symmetric, we have λ1 , λ2 , . . . λn ∈ R (Prop. B.1.2). We also prove U −1 = U > , and so, (B.2.4)
becomes:
A = U diag(λ1 , λ2 , . . . λn )U > (B.2.5)
Proof. Since each eigenvalue λi (with i ∈ [1..n]) has the same geometric and algebraic multiplicity ki , we can
say each repetition of λi as root of the characteristic polynomial can be associated to a different eigenvector.
The eigenspace of λi has dimension ki and we can surely find an orthonormal basis of this space to represent
the ki eigenvectors associated to the ki repetitions of root λi . The sums of the algebraic multiplicities is n
because the characteristic polynomial has degree n, and so, we can fill an n×n matrix U with the eigenvectors
of λ1 , λ2 , . . . λn .
Since by multiplying A with any eigenvector (column) of U we obtain the eigenvector multiplied by its
eigenvalue, we can write U diag(λ1 , λ2 , . . . λn ) = AU . By multiplying at right with U −1 , we obtain (B.2.4).
If A is symmetric, we can show U −1 = U > using the fact that the eigenvectors are orthonormal. We
already said above that the eigenvectors corresponding to the same eigenvalue can be chosen to be or-
thonormal. Any eigenvectors vi and vj corresponding to distinct eigenvalues λi 6= λj need to be or-
thogonal. To see this, notice vi> Avj = vi> λj vj = λj vi> vj and also vi> Avj = λi vi> vj based on vi> A =
>
(vi> A)> = (A> vi )> = (Avi )> = λi vi> . This leads to λj vi> vj = λi vi> vj , and, using λi 6= λj , we obtain
vi> vj = 0 ∀i, j ∈ [1..n], i 6= j. This shows U > U = In , and so, U −1 = U > . By simply replacing U −1 with
U > in (B.2.4), we obtain (B.2.5) as needed.
Proposition B.2.3. Any eigenvalue λ of real symmetric matrix A has the same algebraic and geometric
multiplicity.
Proof. We assume the characteristic polynomial of A has a factor (x − λ)s , i.e., the algebraic multiplicity
of λ is s. Let t be the dimension of the eigenspace of λ, i.e., the geometric multiplicity is t. We will show
s = t. Consider t orthonormal eigenvectors of λ, generated by taking an orthonormal basis of the eigenspace
of λ. We construct an unitary matrix V by putting these eigenvectors on the first t columns and by filling
the remaining n − t columns with other vectors that generate an orthonormal basis of Rn together with
v1 , v2 , . . . vt . We thus have V −1 = V > and we can compute:
 
λIt C
V > AV = V > [λv1 λv2 . . . λvt B] = ,
0n−t,t D

where the zeros on the first t columns are due to the fact that each vi with i ∈ [1..t] is orthogonal to
all the other column vectors of V . By transposing above formula, we obtain the same matrix because

> h i h i>
λIt C λIt C
V > AV = V > A> V = V > AV . This means that 0n−t,t D = 0n−t,t D , and so, C must be zero and D
must be diagonal. We can write:
 
λIt 0t,n−t
V −1 AV = V > AV = .
0n−t,t D

86
 h i
λIt 0t,n−t
This means that matrices A and 0n−t,t D are similar, and so, they have the same characteristic
polynomial by virtue of Prop. B.1.4. The characteristic polynomial of the second matrix is (x−λ)t det(xI−D).
This directly shows that we can not have t > s. This would be equivalent to the existence of a term (x − λ)t
with t > s in the characteristic polynomial of A, which is impossible because the algebraic multiplicity of λ
is s.
We now prove by contradiction that t = s. Assuming the contrary, the only remaining case is t < s. This
means that det(xI − D) has to contain a term (x − λ)s−t because det(xI − A) contains a term (x − λ)s . This
n−t
way, λ is an eigenvalue of D for which there i an eigenvector d ∈ R . We will show that λ has
h exists at least
λIt 0t,n−t
a geometric multiplicity higher than t in 0n−t,t D , which contradicts Prop. B.1.5, i.e., similar matrices
must have the same eigenvalue multiplicitiesh for a commoni eigenvalue λ. It is not hard to check that λ has
λIt 0t,n−t
at least the following t + 1 eigenvectors in 0n−t,t D :

0 
   
 
1 
 
0
 
0 0 
 ..  t-1  .. 

0 1 0  .  . t
      
0  0 1  
0
 
 0 
        
0  0  0  
1 0 .
  , 
 ,   ...   and
0 n-1
 
0 0 0
 
n-2
  
      n-3
 .. 
  ..   ..   
.
 
 .   .   .   ..  n-t d
   

 
 

0 0 0
   
0


The last
h vector is ian eigenvector, because it is enough to check that the top first t positions of the product
λIt 0t,n−t
with 0n−t,t D are λIt 0t + 0t,n−t d = 0t and the bottom n − t positions are 0n−t,t 0t + Dd = λd since
h i
It 0t,n−t
d is an eigenvector of D. We obtained that λ has geometric multiplicity at least t + 1 in 0n−t,t D and
t in A, contradicting Prop. B.1.5 as stated above. This ensures the only possible case is t = s.

B.2.2 Proof using the Schur triangulation of general complex square matrices
Theorem B.2.4. (Schur decomposition) Given any A ∈ Cn×n , there exists unitary matrix U ∈ Cn×n (i.e.,
such that the conjugate transpose satisfies U ∗ U = I) for which:

U ∗ AU = T, (B.2.6)

where T is an upper triangular matrix. The diagonal elements of T are the eigenvalues of A. The number
of times eigenvalue λ appears on the diagonal of T is the algebraic multiplicity of λ in A.
Proof. We proceed by induction. For n = 1, the theorem is clearly true. Assume it holds for n − 1 and we
prove it also holds for n.
Consider an eigenvalue λ and an eigenvector u (they exists as proved by Proposition B.1.1). Without
loss of generality we assume u is unitary, i.e., u∗ u = 1 (from a non-unitary eigenvector we can easily obtain
an unitary one by dividing each of its term by a non-negative real number).
We construct an unitary matrix U ∈ Cn×n with u on the first column. We write U = [u, V ] with
V ∈ C n×(n−1) . The construction of V can be done column by column as follows. The first unitary column
v×,1 of V is chosen by solving u∗ v×,1 = 0 in variables v11 , v21 , . . . vn1 . The existence of a solution for this
equation can follow from the more general Proposition A.2.2; and once a solution is found, we easily make
v×,1 unitary by dividing all terms by the initial norm |v×,1 |. The ith unitary column v×,i (for i ≤ n − 1)
is chosen by solving the following system in variables v1i , v2i , . . . vni : (a) u∗ v×,i = 0 and (b) v×,j
∗
v×,i = 0
for all j ≤ i − 1. There are at most n − 1 equations for n variables, and so, a solution has to exist (for the
skeptical, the coefficients of the n − 1 equations can be put in a n × n matrix filled with zeros on the last
row, so as to obtain a null determinant and apply Proposition A.2.2). We obtain:
 ∗  ∗
u∗ AV u∗ AV
   
∗ u u λ λ
U AU = A[u, V ] = [λu, AV ] = =
V∗ V∗ λV ∗ u V ∗ AV 0n−1 V ∗ AV
As a side remark, remark that if A is hermitian (A = A∗ ) or real symmetric, the top-right term is also
zero: u∗ AV = (Au)∗ V = λu∗ V = 0> n−1 . In fact, it is possible to particularize the above line of proof to

87
directly prove that real symmetric matrices are diagonalizable (and produce an eigen-decomposition), but it
may be useful now to stay a bit more on the general case.
Let us use more compact notations for above equation:

b>
 
∗ λ
U AU = (B.2.7)
0n−1 B

Using the induction hypothesis, there is some unitary matrix UB ∈ C(n−1)×(n−1) such that UB∗ BUB = TB
 1 0> 
is upper triangular. We construct unitary matrix U b=
0 U
n−1
and we obtain:
n−1 B

0> b> 0> 0> b> UB

      
b ∗ U ∗ AU U 1 n−1 λ 1 n−1 1 n−1 λ
U b= =
0n−1 UB∗ 0n−1 B 0n−1 UB 0n−1 UB∗ 0n−1 BUB
>
b> UB
   
λ b UB λ
= = =T (B.2.8)
0n−1 UB∗ BUB 0n−1 TB

Since TB is upper triangular by the induction hypothesis, T is also upper triangular. Noting U = U U b , we
obtain (B.2.6). It is not hard to check that U is unitary: U ∗ U = U b ∗U ∗U U b ∗U
b =U b = I.
We still need to prove that the diagonal elements of T are the eigenvalues of A. Applying Proposition
B.1.4, similar matrices A and T have the same characteristic polynomial. But the characteristic polynomial
of upper triangular matrix T is (x − t11 )(x − t22 ) . . . (x − tnn ). The diagonal elements of T coincide thus
with the roots of the characteristic polynomial of A and T .
U does not necessarily contain the eigenvectors of A
 as columns, even if the construction starts from an
eigenvector of A. Consider, for instance, the matrix 11 −1
 2
−1 . The characteristic polynomial is x
 so that
√1
2
root λ = 0 has algebraic multiplicity 2. However, the matrix has only one eigenvector u = √1 , because
2
it has rank 1; the geometric multiplicity of eigenvalue λ = 0 is 1. The Schur decomposition constructs U
(equal to U because UB = 1 in the proof) by putting u on thefirst column and by filling the other columns
√1 1
√
of U so as to make U orthonormal. We obtain U = √12 √ 2
−1 and U ∗ = U > = U . The decomposition is
2 2
 1 1   1 1   
√
2
√
2 1 −1 √
2
√
2 0 2
1 −1 1 −1 = . U does not contain an eigenvector on the second column.
√
2
√
2 1 −1 √
2
√
2 0 0

Proposition B.2.5. If A ∈ Rn×n is a symmetric matrix, the Schur decomposition computes a diagonaliza-
tion of A. There exists an unitary (orthonormal) matrix U ∈ Rn×n such that

T = U ∗ AU = U > AU (B.2.9)

is a diagonal matrix with the eigenvalues of A on the diagonal. The columns of U are the eigenvectors of A
which needs to be real (Prop. B.1.2), and U ∈ Rn×n so that U ∗ = U > .
Proof. Theorem B.2.4 shows that there is a decomposition (B.2.9) that generates an upper triangular matrix
T . We apply the conjugate transpose on both sides of (B.2.9):

T ∗ = (U ∗ AU )∗ = U ∗ A∗ (U ∗ )∗ = U ∗ AU = T

The equality T ∗ = T can only hold if T is a diagonal matrix with real elements on the diagonal. Recalling that
Theorem B.2.4 shows that the diagonal elements of T are the eigenvalues of A (each taken with its algebraic
multiplicity), we can write T = diag(λ1 , λ2 , . . . λn ). Multiplying (B.2.9) by U at left (and using that U is
unitary, i.e., U U ∗ = U ∗ U = In , as stated by Theorem B.2.4), we obtain: U diag(λ1 , λ2 , . . . λn ) = AU. The
column i (with i ∈ [1..n]) on both sides can be written: λi u×,i = Au×,i . This shows that column i of U is
an eigenvector associated to λi . Using Proposition B.1.2, this column u×,i contains only real elements, and
so, U is real.

88
B.3 The QR decomposition of real matrices
Proposition B.3.1. Any matrix A ∈ Rn×n can be decomposed as
   

  


   
A = QR,

n  A
 
 = Q

 R 
0
   

   
n×p

where Q ∈ R is an orthonormal matrix (its columns are orthogonal


unit vectors meaning Q> Q = Ip ) and R ∈ Rp×n is upper triangular.

| {z }
p

Proof. In preamble, let us first (try to) capture the “spirit” of the factorization we want to prove. Q has
to contain a number of p unit vectors that actually represent an orthonormal basis of the space spanned by
them. Then, A = QR means that each column ak of A can be written as a linear combination of the first
k elements of this basis (the first k columns of Q). It is easy to obtain this linear combination when ak
belongs to the space spanned by these first k elements of the basis. We need to construct an increasingly
larger orthonormal basis that first covers only a1 , then a1 and a2 , then a1 , a2 and a3 , etc. Geometrically,
we can first simply take the unit vector e1 collinear with a1 ; at step 2, we take a vector e2 perpendicular to
a1 or e1 such that e1 and e2 determine the same 2D hyperplane as a1 and a2 ; at the third step we take a
vector e3 perpendicular on this 2D hyperplane so as to determine the same 3D (sub-)space as a1 , a2 , a3 .
This is the goal of the Gram-Schmidt process presented next.
We now formally present the Gram-Schmidt process on the columns of A = [a1 , a2 , . . . an ]. This process
takes the column vectors a1 , a2 , . . . an and generates an orthogonal set of vectors that spans the same
u
subspace as a1 , a2 , . . . an . Let us define the normalized vector e ∈ Rn of u ∈ Rn as e = |u| if u 6= 0, or
e = 0 if u = 0 (degenerate normalized vector). Notice a non-degenerate normalized vector e has norm 1
·
because e e = u|u| ·2u = 1. We define the projection operator proj by setting:

· · ·
 
u u u a
proje a = (e a)e = proju a =
|u|
a
|u|
=
u u ·
u, (B.3.1)

where e is the normalized vector of u. If u = e = 0, we define proj0 a = 0. This represents the projection
of a on e or u. The Gram-Schmidt process constructs the following sequence:
u1 = a1
u2 = a2 − proju1 a2
u3 = a3 − proju1 a3 − proju2 a3
..
.
n−1
X
un = an − projui an
i=1

Geometrically, this construction work as follows: to compute uk , it projects ak orthogonally onto the
subspace U generated by u1 , u2 , . . . uk−1 , which is the same as the subspace generated by a1 , a2 , . . . ak−1 .
The vector uk is then defined to be the difference between ak and this projection, guaranteed to be orthogonal
to all of the vectors in the subspace U .
We now show formally that uk is orthogonal to all u1 , u2 , . . . uk−1 . Assume by induction uk0 uj = ·
·
0 ∀j, k 0 ∈ [1..n], j < k 0 ≤ k − 1. To show uk uj = 0 ∀j < k, we first observe uk uj = 0 if uj = 0. If
uj 6= 0, we calculate:
·

· ·
k−1
!
X
uk uj = ak − projui ak uj
i=1

= ak
· uj −
X

i∈[1..k−1]
ui 6=0
ui ak
ui ui
·
·ui
· uj (we developped projui ak using (B.3.1) for ui 6= 0)

= ak
· uj −
uj
uj
· ua u
·
k
j
· uj (we used ui ·u j = 0 for i 6= j and i, j ∈ [1..k − 1])

· ·
j

= ak uj − uj ak = 0

89
Notice we can have uk = 0 for certain k ∈ [1..n]. However, we can also calculate

· · · · · ·
k−1
!
X X ui ak
uk uk = ak −
i=1
projui ak uk = ak uk −
ui ui
ui
i∈[1..k−1]
ui 6=0
· uk = ak uk ,

meaning that uk = projuk uk = projuk ak . This allows us to write the equation at step k of above Gram-
Schmidt process as:
k−1
X k−1
X
ak = uk + projui ak = projuk ak + projui ak
i=1 i=1
k
X k
X
= projui ak = projei ak (see (B.3.1))
i=1 i=1

=
k
X

i=1
(ei
· ak )ei ,

where ei is the normalized vector of ui . Since this holds for any k ∈ [1..n], we can write it in matrix form:

·
e1 a1 e1 a2 · ·
e1 a3 . . . e1 an· 
 0 e2 a2 · · ·
e2 a3 . . . e2 an 

 0
A = [a1 a2 . . . an ] = [e1 e2 . . . en ] 
 ..
0
..
·
.. ..
·
e3 a3 . . . e3 an 
.. 

=QR
 . . . . . 
0 0 0 ·
. . . en an
Notice that some columns of Q = [e1 e2 . . . en ] can be zero. We can transform the n×n matrix Q into a n×p
matrix Q by removing n − p zero columns. At the same time, we need to remove the corresponding rows of
R and we obtain matrix R ∈ Rp×n that remains upper triangular, i.e., the elements below the diagonal are
zero, leading to A = QR as needed.

B.4 An SDP matrix has a unique SDP square root factor

We introduced the square root decompositions in Section [Link]. Let’s examine the square root matrices K ∈
Rn×n such that KK = A  0. Using the eigendecomposition (1.1.1), we have KK = A = U ΛU > where Λ is
diagonal and U satisfies U U > = In . We can thus write KK = KU U > K = U ΛU > =⇒ U > KU U > KU = Λ.
Using notation D = U > KU , we obtain DD = Λ. In other words, K need to have the form K = U DU > for
some D ∈ Rn×n such that DD = Λ.
Proposition B.4.1. Given SDP matrix A ∈ Rn×n , there exists a unique matrix K ∈ Rn×n such that
KK = A and K  0. This K is called the principal square root of A.
Proof. As described in above paragraph, any K such that KK = A satisfies K = U DU > ; the columns of U
are the unitary orthonormal eigenvectors of A, so that U U > = I. We notice x> Dx = x> U > U DU > U x =
(U x)> K(U x) ≥ 0 ∀x ∈ Rn , and so, D is also SDP. To prove the unicity of SDP matrix K, it is enough to
show there is a unique SDP matrix D ∈ Rn×n such that DD = diag(λ1 , λ2 , . . . , λn ), where λ1 , λ2 , . . . , λn
are the eigenvalues of A. We apply the eigendecomposition on SDP matrix D: we obtain D = V EV > , where
E = diag(e1 , e2 , . . . en ) ≥ 0 and V V > = V > V = In .
Using DD = Λ, we have V EV > V EV > = Λ, and so, V E 2 V > = Λ, or V E 2 = ΛV . Taking any i, j ∈ [1..n],
we have (V E 2 )ij = (ΛV )ij , equivalent to Vij e2j = λi Vij . We observe the following property:

e2j 6= λi =⇒ Vij = 0 (*)

To prove that K = U DU > is unique, it is√enough to √ show that D √ is unique.
√ We will√ exactly determine
the value of D by showing D = V EV > = Λ, where Λ = diag λ1 , λ2 , . . . , λn . For this, it is


√
enough to prove V E = ΛV , which is somehow a consequence of above V E 2 √ = ΛV . More exactly, we need
to show that we find the same√value at position√ (i, j) of both sides of V E = ΛV for any i, j ∈ [1..n]. For
√
this, we have to show Vij ej = λi Vij . If ej = λi , this is clearly true. If ej 6= λi , we have e2j 6= λi (recall
both ej √and λi are non-negative eigenvalues of SDP matrices), and so, (∗) states that Vij = 0, showing
Vij ej = λi Vij = 0.

90
C Useful related facts
We provide two classical results, a proposition related to the completely positive cone, finishing with an
example of a convex function with an asymmetric non-SDP Hessian.

C.1 Optimality conditions for linearly-constrained quadratic programs

Proposition C.1.1. Consider the following linearly-constrained quadratic optimization problem, based on
(not necessarily SDP) symmetric matrix Q ∈ Rn×n , full-rank matrix A ∈ Rp×n with p ≤ n and b ∈ Rp .

> >
min p(x) = x Qx + c x
 (C.1.1a)
(QP= ) s.t. Ax = b (C.1.1b)

x ∈ Rn

(C.1.1c)

The solution x∗ is optimal if and only if the following conditions are satisfied:

2Qx∗ + c = A> µ for some µ ∈ Rp (C.1.2a)

z> Qz ≥ 0 ∀z ∈ null(A) = {z ∈ Rn : Az = 0} (C.1.2b)

It can be a useful exercise to give three proofs using different techniques.

Proof 1. We solve by force the system Ax = b. Since A has full rank p, the null space null(A) of A has
dimension n − p by virtue of the rank-nullity Theorem A.1.3. Let z1 , z2 , . . . zn−p be a basis of the null space.
Pn−p
Any solution x of the system considered above has the form x = v + i=1 zi yi where Av = b. Constructing
matrix Z = [z1 z2 . . . zn−p ], we can write the solutions of the system under the form v + Zy with y ∈ Rn−p .
We can write (QP= ) from (C.1.1a)-(C.1.1c) under the form:
(
min (v + Zy)> Q(v + Zy) + c> (v + Zy) (C.1.3a)
(QP= ) n−p
s.t. y ∈ R (C.1.3b)

We re-write above (C.1.3a) as:

y> Z > QZy + 2v> QZy + v> Qv + c> v + c> Zy,

where we used v> QZy = (v> QZy)> = yZ > Qv. Using Prop. 3.4.1, this unconstrained quadratic program
is bounded from below if and only if it is convex and the gradient vanishes at the optimal solution y∗ . The
necessary and sufficient conditions for the optimality of x∗ = v + y∗ are:
(a) Z > QZ  0, which is equivalent to the fact that Q is positive over null(A), i.e., we obtain the second
order condition (C.1.2b).
(b) The optimal solution y∗ needs to cancel the (column vector) gradient 2Z > QZy∗ + 2Z > Qv + Z > c =
Z > (2Q(v + Zy∗ ) + c) = Z > (2Qx∗ + c). Since the rows of Z > are a transposed basis for null(A),
the above gradient can only cancel if 2Qx∗ + c belongs to the transposed row image of A, i.e., there
is some µ ∈ Rp such that 2Qx∗ + c = A> µ, which is exactly the first order condition (C.1.2a).
Finally, there is a degenerate case p = n in which above Z and y have dimension 0 and they disappear. In
this case, (C.1.2a) holds because A> µ with µ ∈ Rn can cover the whole space Rn since A is a square full
rank matrix. The second order condition (C.1.2b) holds because it reduces to nothing using null(A) = {0}.
The system has thus only one feasible solution x∗ = A−1 b that satisfies both conditions above.

Proof 2.
=⇒ The first order condition (C.1.2a) follows by applying the method of Lagrange multipliers, obtaining a
particular case of the KKT conditions.37 However, in our case the argument reduces to the following. The
37 See also my document “Trying to demystify the Karush-Kuhn-Tucker conditions”, available on-line at [Link]

[Link]/~porumbed/papers/[Link].

91
gradient ∇p(x∗ ) is perpendicular in x∗ to a surface (level set) on which p takes the constant value p(x∗ ).
Why does not p increase or decrease by moving backward or forward from x∗ along some z ∈ null(A) ?
Because the gradient in x∗ is perpendicular to z. Indeed, using the chain rule, the derivative in t = 0 of
·
f (t) = p(x∗ + tz) is equal to f 0 (0) = ∇p(x∗ ) z = 0. If this were not zero, one could move backward or
forward from x∗ along z by some ε > 0 to decrease p. We obtain that the gradient ∇p(x∗ ) is perpendicular
to null(A), and so, it belongs to the transposed row image of A, i.e., ∇p(x∗ ) = A> µ for some µ ∈ Rp ,
which is exactly (C.1.2a).
We prove z> Qz ≥ 0∀z ∈ null(A) by assuming the opposite: ∃z ∈ null(A) such that z> Qz = −ε < 0.
All x + tz are feasible since they satisfy A(x + tz) = Ax = b. The function f (t) = p(x + tz) has degree 2 and
the quadratic factor is z> Qzt2 = −εt2 . This is a concave function that goes to −∞ in both directions. This
means x∗ is not minimal. We obtained a contradiction from z> Qz = −ε < 0. The second order condition
(C.1.2b) needs to hold.
⇐= We suppose both conditions (C.1.2a)-(C.1.2b) are satisfied by some x∗ such that Ax∗ = b. We will prove
that p(x) ≥ p(x∗ ) for any feasible x = x∗ + z, with z ∈ null(A). Consider the function f (t) = p(x∗ + tz).
·
Using the chain rule and (C.1.2a), we obtain f 0 (0) = ∇p(x∗ ) z = ∇p(x∗ )> z. Replacing the gradient with
the right hand side of (C.1.2a), this is further equal to µ> Az which is equal to 0 because z ∈ null(A). We
thus obtained f 0 (0) = 0. We will show f is convex. The only quadratic factor in t of p(x∗ + tz) is z> Qzt2
and its second derivative is 2z> Qz which is non-negative by virtue of the second order condition (C.1.2b).
This means that f is convex and reaches its minimum at t = 0, i.e., p(x∗ ) ≤ p(x) for all feasible x.

Proof 3.
=⇒ We show the second order condition (C.1.2b) using the second proof above. We can prove the first order
condition (C.1.2a) using convexification
P results. We first apply Prop. C.2.2 that states there exists λ ∈ Rn×p
such that pλ (x) = p(x) + λji (xj Ai x − xj bi ) is convex, where Ai is the row i of A. Notice pλ (x) = p(x)
whenever Ax = b. We can write the following program

p(x∗ ) = min {pλ (x) : Ax = b} (C.1.4)

x

as an SDP
 program because it is convex. Using the results from Section 6.2.2, this SDP program takes the
X

form of P L (QP= ) from (6.2.5a)-(6.2.5b) with µ = 0 and µ = 0. We can write p(x∗ ) = OP T (C.1.4) =
P Lx (QP= ) = P LX (QP= ) because the Hessian of all these programs except the first one is SDP;

 

this means that the first essential hierarchy (6.2.13a) collapses (like in Remark 6.3.7). As such, the total
Lagrangian of (C.1.4) reaches p(x∗ ). The total Lagrangian for optimal Lagrangian multipliers β ∈ Rp is

min pλ (x) + β > Ax − β > b. (C.1.5)

x∈Rn

Evaluating this total Lagrangian in x∗ , we obtain the value p(x∗ ). Since the total Lagrangian reaches p(x∗ ),
the minimum of (C.1.5) needs to be p(x∗ ). Using Prop. 3.4.1, this can only be the case if the gradient in x∗
of (C.1.5) is zero.
Let us calculate the gradient of p(x) + λji (xj Ai x − xj bi ) + β > Ax in x∗ . The gradient of each term
P

P as follows. For any k 6= j, the partial derivative on xk is xj Aik . The partial deriva-
xj Ai x−xj bi is computed
tive on xj is 2Aij xj + k6=j Aik xk −bi = xj Aij +Ai x−bi . The second term vanishes in feasible x∗ . We obtain
 
>
that the gradient of xj Ai x − xj bi in x∗ is x∗j A>
P
i . We obtain ∇ p(x) + λji (x j A i x − x j bi ) + β Ax =
x∗
∗ ∗ > >
P
∇p(x ) + λji xj Ai + A β. The last two terms belong to the transposed row image of A. Since the
gradient needs to be zero, the first term ∇p(x∗ ) has to belong to the transposed row image of A as well, i.e.,
∇p(x∗ ) = A> µ for some µ ∈ Rp , which is exactly (C.1.2a).
⇐= Using the fact that Q is non-negative over null(A), we can use a relatively similar (reversed) argument
as in the above “=⇒”P proof. First, as above, we apply Prop. C.2.2 that states there exists λ ∈ Rn×p such
that pλ (x) = p(x) + λji (xj Ai x − xj bi ) is convex. Notice pλ (x) = p(x) whenever Ax = b. The total
Lagrangian of min {pλ (x) : Ax = b} is pλ (x) + β > Ax − β > b which is a convex polynomial for any value of
β ∈ Rp . The value of this total Lagrangian in x∗ is p(x∗ ). It is enough to show that the gradient of the total
Lagrangian in x∗ is zero forP an appropriate β ∈ Rp . UsingPthe calculations from the above “=⇒” proof, the
gradient in x is ∇p(x ) + λji x∗j A>
∗ ∗ >
i +A β = A µ+
>
λji x∗j A> >
i + A β. The first two terms belong to
the transposed row image of A, and so, they can be canceled by an appropriate β ∈ Rp . We obtain that the

92
total Lagrangian reaches the value p(x∗ ), and so, x∗ needs to be a minimizer of p (the Lagrangian is always
less than or equal to the constrained optimum of p or pλ ).

C.2 More insight and detail into the convexifications from Section 6
C.2.1 Constraints that can be used to convexify any matrix non-negative over null(A)
We consider a full-rank matrix A ∈ Rp×n associated to linear constraint Ax = b. Based on these constraints,
one can generate various redundant quadratic constraints that are surely satisfied when Ax = b, see Sec-
tion [Link] for examples. We showed in Section [Link] that a partial Lagrangian subject to Ax = b can
reach the optimum value only when the Lagrangian multipliers construct a matrix Q that is non-negative
over the null space null(A) of A (see (A.1.2) for the null space definition); if this does not happen, the
partial Lagrangian converges to −∞. We here discuss convexifications that can make Q non-negative over
the whole Rn (i.e., SDP) using the Lagrangian multipliers associated to the redundant quadratic constraints.
The first paragraph of Section 2.3. from “Partial Lagrangian relaxation for General Quadratic Program-
ming” (see Footnote 33, p. 71) states the following result as already known in the literature. If Q is strictly
positive over null(A), there exists V ∈ Rn×n such that Q + A> V A  0. We prove below a generalization
of this result. This proof is not taken from existing work and we think it is original; it uses the Bolzano-
Weierstrass theorem C.4.9 (any bounded sequence contains a convergent sub-sequence).
Proposition C.2.1. Consider any Q ∈ Rn×n strictly positive over null(A), i.e., u> Qu > 0 ∀u ∈ null(A)−
{0}. If B  0 satisfies
B
· uu> = 0 ⇐⇒ u ∈ null(A),
then there exists λ > 0 such that Q + λB  0. In other words, we can convexify Q using the Lagrangian
(C.2.1)

multiplier of a (redundant) quadratic constraint with quadratic factor B. The matrix B can be for instance
B = A> SA for any S  0, generalizing the redundant constraint from Example 6.3.4.

·
e = {x ∈ Rn : |x| = 1, Q xx> < 0}. We need to make the
Proof. We define a set of particular interest: X
>
·
elements x of this set verify (Q + λB) xx ≥ 0. Consider the function f : X
e → R defined by

·
B xx>
f (x) =
·
|Q xx> |

Based on (C.2.1), B  0 is strictly positive over all x outside the null space of A. Since Xe ⊆ Rn − null(A),
we easily obtain f (x) > 0 x ∈ X.
e
We will prove by contradiction that inf f (x) > 0. Assuming the contrary (i.e., inf f (x) = 0), there exists
a sequence (xi ) with xi ∈ X
e such that lim f (xi ) = 0. Using the Bolzano-Weierstrass Theorem C.4.9, there
i→∞
exists a sub-sequence {xni } such that lim xni = x
e. We will show we can obtain a contradiction for each of
i→∞
e ∈ Rn :
the following three cases that cover x
e∈X
(i) x e

e ∈ null(A)
(ii) x

e ∈ Rn − null(A) − X
(iii) x e

For case (i), it is enough to notice that the sequence f (xni ) can be arbitrarily close to f (e
x) > 0, which
contradicts that f (xi ) converges to zero.
For case (ii), we obtain by hypothesis that Q x
that |f
xε − x
ex ·
e> > 0. This leads to Q x fε x
>
·
fε > 0 for any x fε such
e| < ε for a sufficiently small ε. But this would mean that f is undefined in a sufficiently small
ball around xe, and so, we can not have lim xni = x e with xni ∈ X e ∀i ∈ N, contradiction.
i→∞
e∈
For case (iii), we can use (C.2.1) and x / null(A) to obtain B x
ex ·
e> = z > 0. When i → ∞, the value
· > >
·
B xni xni can become arbitrarily close to z and Q xni xni can become arbitrarily close to 0. This latter
·
fact (limi→∞ Q xni x> ·
ni = 0) follows from Q xni xni < 0 ∀i ∈ N (because xni ∈ X) and Q x
e e> ≥ 0
ex ·
(because xe∈/ X). >
· >
·
e Combining the above convergence properties of B xn xn and Q xn xn , we obtain that
·
B xni x>
ni
i i i i

f (xni ) = can become arbitrarily large, contradicting that f (xi ) converges to zero.
·
−Q xni x>
n i

93
 This means there exists a (possibly large) λ > 0 such that f (x) = B · xx
>
> 1
∀x ∈ X.
e We obtain
|Q · xx
>| λ

· · ·
λB xx> + Q xx> > 0 for all x such that Q xx> < 0 (we developed the X e definition, forgetting that
all its elements are unitary). For the remaining cases, i.e., for all x ∈ R such that Q xx> ≥ 0, we
n
·
·
also obtain (Q + λB) xx> ≥ 0 simply because B  0 and λ > 0. Combining both cases above, we get
·
(Q + λB) xx> ≥ 0 ∀x ∈ Rn .
The next proofs are modified versions of the proofs from (Section 2.3 of) “Partial Lagrangian relaxation
for General Quadratic Programming” by Alain Faye and Frédéric Roupin (see Footnote 33, p. 71).
Proposition C.2.2. We are given a full rank matrix A ∈ Rp×n associated to constraints Ax = b. Consider
any Q ∈ Rn×n non-negative over null(A), i.e., u> Qu ≥ 0 ∀u ∈ null(A) − {0}. There exists a linear
combination of the redundant constraints xj Ai x − xj bi = 0 (where Ai is the row i of A, with j ∈ [1..n]
·
and i ∈ [1..p]) from Example 6.3.5 that can be added to Q xx> to transform Q into an SDP matrix.
Equivalently, if Aj.i is the n × n matrix with only a non-zero row
P  j> that contains
 Ai , then there always exist
λji ∈ R (for all j ∈ [1..n] and i ∈ [1..p]) such that Q + j,i λji Aj.i + Aj.i  0.
>
P show in Prop. C.2.3 there exists W ∈ R
Proof. We will n×p
such that Q + A> W + W A  0. We can
write W = Wji Eji where the Eji ’s represent the canonical base indexed by j ∈ [1..n] and i ∈ [1..p], i.e.,
Eji ∈ Rn×p has a value of one at position (j, i) and only zeros at all other positions. We have
> X
A> W + W A = Wji A> Eji >


+ Eji A
j,i
X  > 
= Wji Aj.i + Aj.i ,
j,i

which concludes the proof, with the values λji = Wji .

Proposition C.2.3. We are given a full rank matrix A ∈ Rp×n associated to constraints Ax = b. Consider
any Q ∈ Rn×n non-negative over null(A), i.e., u> Qu ≥ 0 ∀u ∈ null(A) − {0}. There exists W ∈ Rn×p
>
such that Q + A> W + W A  0.
Proof. We recall the QR decomposition from Prop. B.3.1 and the Gram-Schmidt orthogonalization process
described in the proof of this Prop. B.3.1. We apply this process up to the last column p of A> , so as
to factorize A> = U > R, where U has the size of A, U U > = Ip and R ∈ Rp×p is upper triangular. One
can see this as a QR decomposition restricted to the first p columns, i.e., it can be extended to a full
QR decomposition by adding n − p zero columns to A> and R. However, we can equivalently factorize
A = R> U . Notice that the (first i ≤ p) rows of U span the same subspace as the (first i ≤ p) rows of A.
Using Prop. A.1.4, we have p = rank(A) ≤ rank(R> ), rank(U ), and so, U and R> are full rank and R> is
invertible. If we find W ∈ Rn×p such that S + U > W > + W U  0, we can use equation below and finish the
proof:

−1 >
−1 >
W U = W R> R U = W R> A and U > W > = A> W . (C.2.2)
| {z }
W

It is enough to show there exists W ∈ R such that Q + U > W > + W U  0, where the (first i ≤ p) rows
n×p

of U are an orthonormal basis spanning the same subspace as the (first i ≤ p) rows of A. Let B be a matrix
whose columns are an orthonormal basis of null(A) = null(U ). Using the rank-nullity Theorem A.1.3, we
have B ∈ Rn,n−p . The fact that Q is non-negative over null(A) is equivalent to B > QB  0; this follows
from

x> B > QBx ≥ 0 ∀x ∈ Rn−p ⇐⇒ (Bx)> Q(Bx) ≥ 0 ∀x ∈ Rn−p ⇐⇒ y> Qy ≥ 0 ∀y ∈ null(A) (C.2.3)

Let Li be the sub-space spanned by ui+1 , ui+2 , . . . up and (the columns of) B, where ui is the row i of
U written as a column vector. This sub-space has dimension n − i and is perpendicular on u1 , u2 , . . . ui
(recall U B = 0). We can also write:

Li = img (ui+1 , ui+2 , . . . up , B) = {y ∈ Rn : uj y = 0 ∀j ∈ [1..i]} , (C.2.4)

94
where img(...) is the sub-space generated by the column vectors of the matrices given as arguments. In
particular, we have L0 = Rn and Lp = img(B) = null(A). We will construct a matrix Qi ∈ Rn×n that is
non-negative on Li , by induction on i from i = p (with Qp = Q) down to i = 0 (when i = 0, Q0 non-negative
over Rn is equivalent to Q0  0). Using an argument as the one from (C.2.3), a matrix Qi ∈ Rn×n is non-
>
negative over Li if and only if M i = [ui+1 , ui+2 , . . . up , B] Qi [ui+1 , ui+2 , . . . up , B]  0. We will prove we
can use Qi associated to M i  0 to construct Qi−1 such that M i−1  0. We will thus iteratively construct
p
uj wj> + wj u>
P
Qi = Q + j with i from p down to 0. At the last iteration we will obtain:
j=i+1

p
X
Q0 = Q + uj wj> + wj u> > >
j = Q + U W + WU (C.2.5)
j=1

as needed, where W = [w1 w2 w3 . . . wp ]. We will use several times the following property:

u> > >

a uj wj + wj uj ub = 0 ∀ua , ub ∈ {u1 , u2 , . . . , uj−1 , uj+1 , uj+2 , . . . , up , B} , (C.2.6)

where one can read B as an enumeration of column vectors (slightly abusing notations). This simply follows
from u> >
a uj = uj ub = 0 with ua and ub from the above set.
Now we present the induction step. We can assume Qi is already constructed and we need to deter-
mine Qi−1 = Qi + ui wi> + wi u> n
i . More exactly, the goal is to find some wi ∈ R such that M
i−1
=
>
[ui , ui+1 , . . . up , B] Qi−1 [ui , ui+1 , . . . up , B]  0, as argued above. At each transition i → i − 1, we can
use the induction hypothesis that Qi is associated to M i  0 (recall this is surely true for Qp = Q and
M p = B > QB by hypothesis). Let us develop the formula of M i−1 that we will construct to be SDP.

u>
 
i
u> 
 i+1 
! " i−1 i−1
#
p M1,1 M1,[2..n−i+1]
i−1  ..  P > >
M =  .  Q+ uj wj + wj uj [ui , ui+1 , . . . up , B] = i−1 i−1
  j=i M[2..n−i+1],1 M[2..n−i+1],[2..n−i+1]
 u>
p

B>
 ! ! 
p p
u> uj wj> wj u> u> uj wj> wj u>
P P
 i Q+ + uij i Q+ + j [ui+1 , . . . up , B] 
 j=i j=i 
  >  
 ui+1 
=  .. 
!  , (C.2.7)
p
Mi
 
 .  P > >
 Q+ uj wj + wj uj ui
 
  
  u>
p
 j=i 
B>

where we used (C.2.6) with j = i and ua , ub ∈ {ui+1 , ui+2 , . . . , up , B} to remain with the M i term in the
bottom-right cell. Let us develop the first line of M i−1 :
!
p
i−1
= u> uj wj> + wj u> ui = u> > >
P

– the first position is M1,1 i Q+ j i Q + ui wi + wi ui ui by virtue of
j=i
i−1
(C.2.6). We can further develop this into M1,1 = u> > >
i Qui + wi ui + ui wi .

– on the remaining n − i positions, we have:

 
Xp
i−1
M1,[2..n−i+1] = u>i
Q + uj wj> + wj u>
j
 [ui+1 , . . . up , B]
j=i

= u>
i Q [ui+1 , . . . up , B] + u> >
i ui wi [ui+1 , . . . up , B] (we used (C.2.6))
 
p
X
+ u>
i
 wj u>
j
 [ui+1 , . . . up , B]
j=i

= u>
i Q [ui+1 , . . . up , B] + wi> [ui+1 , . . . up , B]
+ u>
i [wi+1 , . . . wp , 0] (we used (C.2.6))

95
 Using the orthonormality properties of ui+1 , . . . up and B, the last term can be written
>
u> >
i [wi+1 , . . . wp , 0] = ui [wi+1 , . . . wp , 0] [ui+1 , . . . up , B] [ui+1 , . . . up , B]
| {z }
Pi

= u>
i Pi [ui+1 , . . . up , B] ,

where Pi ∈ Rn×n does not depend on the vector wi we need to determine. We will need the following:

Pi ui = 0n×1 and u> >

i Pi = 01×n (C.2.8)

Finally, by simplifying above formulas, the first row of M i−1 can be written:

M1i−1 = u> >

u> > >


i Qui + 2ui wi , i Q + wi + ui Pi [ui+1 , . . . up , B]

We need to determine wi such that M i−1  0, given that we can rely on the induction hypothesis
M  0. The simplest way to construct an SDP matrix M i−1 is to generate only zeros on the first row
i

M1i−1 . First, we would like wi> to cancel the terms u> >
i Q + ui Pi in above M1
i−1
formula. As such, wi
> >

>
integrates a first term − ui Q + ui Pi . A second term of wi is zui ; this second term does not change the
i−1
canceled positions of the first row (see point (b) below), but it can make M1,1 ≥ 0 for a sufficiently large
>


z. Thus, we set wi = − Q + Pi ui + zui , where the value of z will be determined at point (a) below.
However, this wi vector leads to the following values on the first row of M i−1 .
 
i−1
= u> >
− Q + Pi> ui + zui = u> > > > > >


(a) M1,1 i Qui + 2ui i Qui − 2ui Qui − 2ui Pi ui + 2ui zui = −ui Qui +
2zu> > > 1 >


i ui = −ui Qui + 2z, where we used (C.2.8) to cancel the Pi term. We set z = 2 ui Qui to
i−1
make M1,1 = 0, but larger values can also be chosen.

> 
i−1
= u> > >
[ui+1 , . . . up , B] = zu>


(b) M1,[2..n−i+1] i Q + − Q + P i ui + zui + u i P i i [ui+1 , . . . up , B] =
01×(n−i) .

The resulting first row of M i−1 is filled

 with zeros, and so needs to be the first column by symmetry.
Recalling (C.2.7), we have M i−1 = 00 M0 i . Using the induction hypothesis M i  0, we obtain M i−1  0,
which finishes the induction step. At the last iteration, we obtain M 0  0, which is enough to guarantee
Q0  0 using arguments discussed above. Recalling (C.2.5) and (C.2.2), this finishes the proof.

C.2.2 Refining the Branch-and-bound for equality-constrained binary quadratic programming

from Section [Link]
An approach like in Section [Link] can be found in the article “Improving the performance of standard solvers
for quadratic 0-1 programs by a tight convex reformulation: The QCR method” by Alain Billionnet, Sourour
Elloumi and Marie-Christine Plateau.38 They use the
redundant constraints from Example 6.3.5 which yield
the optimum value OP T (SDP (QP= )) = P Lx (QP= ) as stated in Remark 6.3.9. However, instead of using
L P Lx (QP (µ∗ , µ∗ ), they work with a restricted version L P Lx (QP= ) (µ∗ , µ∗ ) of this program in which
[0−1]
=)
they also impose xi ∈ [0, 1], equivalent to x2i ≤ xi ∀i ∈ [1..n]. However, they show that this program has
the same objective value OP T (SDP (QP= )) using an argument based on the Slater’s interiority condition,
stating that the total dual Lagrangian of L P Lx (QP= ) (µ∗ , µ∗ ) has no duality gap—see the implication
[0−1]

(D1) → (D2). However, we can here use a different argument.

L P Lx (QP ) (µ∗ , µ∗ ), L P LX (QP ) (µ∗ , µ∗ ) and SDP [0−1] (P )
[0−1] [0−1]
If we construct =
= from
=

resp. L P Lx (QP = )
(µ∗ , µ∗ ), L X
PL (QP )
(µ∗ , µ∗ ) and SDP (P ) by adding constraints x2i ≤
= xi (or
=

38 Published in Discrete Applied Mathematics in 2009, vol 157 (6), pp. 1185-1197, a draft is available at [Link]

[Link]/fichiers/[Link].

96
resp. Xii ≤ xi ) ∀i ∈ [1..n] in resp. (6.3.7), (6.2.5b) and (6.2.2a)-(6.2.2e), then the following hierarchy holds
and it naturally collapses:

OP T (SDP (P= )) = OP T L P Lx (QP= ) (µ∗ , µ∗ )

 
(C.2.9a)
 
≤ OP T L P Lx (QP= ) (µ∗ , µ∗ )
[0−1]
(C.2.9b)
 
= OP T L X
[0−1]
(µ∗ , µ∗ ) (C.2.9c)
PL (QP= )
 
≤ OP T SDP [0−1] (P= ) (C.2.9d)
= OP T (SDP (P= )) (C.2.9e)

Proof. The first equality (C.2.9a) is taken from (6.3.8). The inequality (C.2.9b) follows from the fact that
the “[0 − 1]” version of the partial Lagrangian (a minimization program) is more constrained. The equality
(C.2.9c) is due to the fact that the quadratic factor Qµ∗ ,µ∗ of both programs (in variables X or x) is SDP
and that X = xx> respects  all constraints of theSDP program in variables X. The inequality (C.2.9d)
L P LX (QP
[0−1]
follows from the fact that (µ∗ , µ∗ ) is a Lagrangian of (SDP [0−1] (P= )). Finally, (C.2.9e)
=)
holds because (SDP (P= )) already integrates binary constraints Xii = xi ∀i ∈ [1..n].

More advanced convexifications can be found in the work of A. Billionnet, S. Elloumi and A. Lambert,
e.g., see papers “Extending the QCR method to general mixed integer programs.” and “Exact quadratic
convex reformulations of mixed-integer quadratically constrained problems”.39 However, for the moment,
such methods lie outside the scope of this non-research document; further progress is unessential for now.

C.3 A convex function with an asymmetric Hessian

We next provide an example of a convex function with an asymmetric Hessian. This shows that a statement
like “A twice differentiable function is convex if and only if its Hessian is SDP” is technically not complete,
because a convex function can have an asymmetric non-SDP Hessian. This case is omitted from certain
textbooks (see a reference in the first paragraph of Section 1.8) but we addressed it in our work by requiring
the Hessian to be symmetric in Prop. 1.8.1.
Example C.3.1. The following function f is convex for any µ ≥ 9 and has a non-symmetric Hessian in 0,
i.e., ∇2 f (0) is not symmetric.
 3
 x y + µx2 + µy 2 if (x, y) 6= (0, 0)
f (x, y) = x + y 2
2 (C.3.1)
0 if (x, y) = 0


∂f
Proof. The gradient of f at (0, 0) can not be computed algebraically. We can however obtain (0, 0) =
∂x
f (ε, 0) − f (0) µε2 ∂f
lim = lim = 0 and similarly (0, 0) = 0. The gradient of f is thus:
ε→0 ε ε→0 ε ∂y


!
2 2 2 3 2 2
 x y x + 3y + 2µx, x x − y + 2µy

if (x, y) 6= (0, 0)
2 2
∇f (x, y) = (x2 + y 2 ) (x2 + y 2 )

(0, 0) if (x, y) = 0


Let us now calculate the Hessian ∇2 f (x). As above, we apply the derivative formula to calculate
39 Both published in Mathematical Programming, resp .in 2012 (vol. 131(1), pp. 381-401) and 2016 (vol 158(1), pp 235-266).

97
∇2 f (0, 0):
∂f   ∂f  
2
∂ f   ε, 0 − 0, 0 2µε
0, 0 = lim ∂x ∂x = lim = 2µ
∂x∂x ε→0 ε ε→0 ε
∂f   ∂f  
∂2f   0, ε − 0, 0 0
0, 0 = lim ∂x ∂x = lim = 0
∂y∂x ε→0 ε ε→0 ε

∂f   ∂f   ε3 (ε2 − 02 )
ε, 0 − 0, 0
∂2f   ∂y ∂y (ε2 + 02 )2 ε
0, 0 = lim = lim = lim =1
∂x∂y ε→0 ε ε→0 ε ε→0 ε
∂f   ∂f  
0, ε − 0, 0
∂2f   ∂y ∂y 2µε
0, 0 = lim = lim = 2µ
∂y∂y ε→0 ε ε→0 ε

The Hessian of f is thus:




 2xy 3 3y 2 − x2 x2 x4 + 6y 2 x2 − 3y 4

 3 + 2µ 3
(x2 + y 2 ) (x2 + y 2 )


 
 if (x, y) 6= (0, 0)

 

 2 4
2
  x x + 6y 2 x2 − 3y 4 2x y y 2 − 3x2
3
+ 2µ

∇ f (x, y) = 2 2 3 3

" (x
# +y ) (x2 + y 2 )


 2µ 1
if (x, y) = 0



 0 2µ

p1 (x, y)
All fractions in the Hessian have the form , where p1 and p are both homogeneous polynomials
p(x, y)
p1 (tx, ty) t6 p1 (x, y) p1 (x, y)
of degree 6. We easily obtain = 6 = . The graph of such fraction can be seen
p(tx, ty) t p(x, y) p(x, y)
as a surface that has the same height (value) on each ray starting from (but not touching) the origin. The
p1 (x, y) p1 (x, y)
image of over R2 − {0} is equal to the image of over unit circle x2 + y 2 = 1, which needs
p(x, y) p(x, y)
to be bounded.
p1 (x, y)
We now calculate the bounds of the fractions. First, notice = p1 (x, y) over the unit circle, since
p(x, y)
2 2 6
p(x, y) = (x + y ) . Each monomial of degree 6 of p1 (x, y) belongs to the interval [−1, 1]. Using this,
∂2f 




we obtain, for instance, that − 2µ ∈ min 2xy 3 3y 2 − x2 , max 2xy 3 3y 2 − x2 ⊂ [−8, 8] when
∂x∂x
2 2
x + y = 1. By applying the same approach on all fractions, we obtain:
     
8 10 2 2µ 0 8 10
− ≤ ∇ f (x, y) − ≤
10 8 0 2µ 10 8

By taking any µ ≥ 9, we obtain ∇2 f (x, y)  0 for (x, y) 6= (0, 0). On the other hand, we can never state
∇ (0, 0)  0 because ∇2 (0, 0) is not symmetric. We thus need to use ∇2 (0, 0) + ∇2 (0, 0)>  0.
2

The idea is taken from the last pages of the article “On second derivatives of convex functions” by Richard
Dudley,40 but we are the first to calculate an explicit minimum value of µ. The derivatives were calculated
at [Link]
40 Published in Mathematica Scandinavica in 1978, vol 41, pp 159–174, available on-line as of 2017 at [Link]

dk/article/download/11710/9726, see also the discussion on the on-line math forum [Link]
questions/1181713/convex-function-with-non-symmetric-hessian.

98
C.4 The separating hyperplane theorem
C.4.1 General theorems and their reduction to a particular case
Theorem C.4.1. (Hyperplane separation theorem) Given two disjoint convex sets X, Y ⊂ Rn , there exist a
non-zero v ∈ Rn and a real number c such that

v
· x≥c≥v
· y, (C.4.1)

·
for any x ∈ X and y ∈ Y . The hyperplane {u ∈ Rn : v u = c} separates X and Y .
Proof. We will show that the general theorem reduces to a simpler theorem version in which Y = {0}, i.e.,
the hyperplane 0-separation Theorem C.4.2.
Consider the set Z = X − Y = {x − y : x ∈ X, y ∈ Y }. The set Z is convex: take za = xa − ya ,
zb = xb −yb and any α ∈ [0, 1] and observe αza +(1−α)zb = α(xa −ya )+(1−α)(xb −yb ) = αxa +(1−α)xb −
(αya + (1 − α)yb ). Since X and Y are convex, xα = αxa + (1 − α)xb ∈ X and yα = αya + (1 − α)yb ∈ Y ,
and so, za + (1 − α)zb = xα − yα ∈ Z, i.e., Z is convex.
We apply the hyperplane 0-separation Theorem C.4.2 on Z and 0 (observe 0 ∈ / Z because X and Y are
·
disjoint) and obtain there is a non-zero v ∈ Rn such that v z ≥ 0 ∀z ∈ Z. This means that

v
· · x≥v y, ∀x ∈ X and ∀y ∈ Y. (C.4.2)

· ·
We obtain inf x∈X v x ≥ supy∈Y v y, because otherwise (C.4.2) would be violated by some x and y

inf x∈X v · x+sup

· y∈Y v
· ·
such that v x is close enough to inf x∈X v x and v y is close enough to supy∈Y v y. Taking c =
· y , (C.4.2) can be written in the form (C.4.1).
·
2

Theorem C.4.2. (Hyperplane 0-separation theorem) Given convex set X ⊂ Rn that does not contain 0,
there exist a non-zero v ∈ Rn such that

for any x ∈ X.
v x ≥ 0,
·
We give two proofs. The first one is based on induction and it takes a bit more than one page. It is a
personal proof; I doubt it can also be found in classical textbooks. The second proof takes 2.5 pages and
it follows well-established arguments (some of them using convergent sequences) that I could found on the
Internet.41 The first proof essentially relies on Theorem C.4.4 from Appendix C.4.2; the second one relies
on Theorem C.4.5 and Theorem C.4.6 from Appendix C.4.3.

Proof 1 For any u ∈ Rn , let f (u) be the largest t such that tu ∈ X, or −∞ if no tu belongs to X. If
f (u) > 0 for all non-zero u ∈ Rn , then 0 ∈ X: it is enough to take u and −u and observe that the segment
joining f (u)u and f (−u) − u contains 0. Since X does not contain 0, there exist some non-zero u ∈ Rn
such that f (u) ≤ 0. The theorem then follows from applying Theorem C.4.4.

Proof 2 We first prove that the closure X of X (i.e., the set X along with all its limit points) is convex. Take
any x, y ∈ X and consider two sequences {xi } and {yi } in X that converge to x and resp. y. Such sequences
always exist because, by definition, X is the set of the limit points of all sequences of X. Take any α ∈ [0, 1];
it is enough to show αx + (1 − α)y ∈ X to prove that X is convex. We have zi = αxi + (1 − α)yi ∈ X,
because X is convex. We next observe that z = lim zi ∈ X, because X contains all limit points of X. But
i→∞
αx + (1 − α)y = α lim xi + (1 − α) lim yi = lim zi = z ∈ X, and so, X is convex.
i→∞ i→∞ i→∞
If 0 does not belong to the closure X, then the conclusion follows from the Simple Separation Theo-
rem C.4.5 applied on 0 and X. If 0 belongs to the closure of X, the conclusion follows from the Simple
Supporting Hyperplane Theorem C.4.6 applied on X and 0 as a boundary point of X (0 does not belong to
the interior, because it does not belong to X).

The following variant can be generally useful, but we do not need it in this document.
41 Iused the Wikipedia article [Link]/wiki/Hyperplane_separation_theorem and the course of Peter Norman
[Link]/~normanp/[Link].

99
Theorem C.4.3. (Separation theorem for open set X) Given two disjoint convex sets X, Y ⊂ Rn such that
X is open, there exist a non-zero v ∈ Rn and a real number c such that

v
· x>c≥v
· y, ∀x ∈ X, y ∈ Y (C.4.3)

The closure X of X satisfies

v
· x≥c≥v
· y, ∀x ∈ X, y ∈ Y

Proof. Using the standard hyperplane separation Theorem C.4.1, there is a non-zero v ∈ Rn and some c ∈ R
(C.4.4)

such that:
· ·
v x ≥ c ≥ v y, ∀x ∈ X, y ∈ Y

·
For the sake of contradiction, assume there exists some x ∈ X such that v x = c. Since X is open,
(C.4.5)

X contains an open ball around x, and so, for a sufficiently small  > 0, we have x − v ∈ X. But
· ·
v (x − v) = c − |v|2 < c which contradicts (C.4.5). The assumption v x = c was false, and so, (C.4.5)
becomes (C.4.3).
·
We still have to prove (C.4.4). Assume there is some x in X such that v x < c. Since x has to be the
limit point of some sequence {xi } with elements xi ∈ X ∀i ∈ N∗ , we deduce that lim v xi = v x. For · ·
· · ·
i→∞
∗
any  > 0 there exists some m ∈ N such that |v xi − v x| <  ∀i ≥ m. Taking any  < c − v x, we have
· ·
v xm < c, which contradicts (C.4.3). The assumption v x < c was false, which proves (C.4.4).

C.4.2 Proving the theorem using personal arguments

Theorem C.4.4. (Hyperplane 0-separation theorem in presence of open rays) Consider convex set X ∈ Rn
(that may contain 0 or not) and let f (u) be the largest t such that tu ∈ X for any u ∈ Rn . If f (u) ≤ 0, we
say that ray u is open, because there is no  > 0 such that u ∈ X. If there is at least an open ray, then
there exist some non-zero v ∈ Rn such that

v
· x ≥ 0, ∀x ∈ X.

Proof. We proceed by induction. We first prove it for n = 2 using the notion of angle.
Lemma C.4.4.1. The theorem holds for n = 2.
Proof. Without loss of generality, we consider the open ray u ∈ R2 is unitary. We associate u with an angle
θ = 0. Using a slight notation abuse, let f (θ) = f (uθ ), where uθ is an unitary vector of R2 that makes an
·
angle of θ with u measured clockwise. Technically, uθ satisfies u uθ = cos(θ) and it is the first unitary
vector with this property found by moving clockwise from u. Let Θ be the set of angles θ for which f (θ) > 0.
Θ belongs to segment (0, 2π) because it does not contain θ = 0 and it might be open. However, Θ needs to
have an infimum inf(Θ) and a supremum sup(Θ), see also Prop. C.4.7.
Assume for the sake of contradiction that sup(Θ) − inf(Θ) > π. This means there are two angles
θM , θm ∈ Θ close enough to sup(Θ) and resp. inf(Θ) so that θM − θm > π. By convexity, the segment that
joins f (θM )uθM and f (θm )uθm is included in X and it also intersects the segment [0, u] in some point u
with  > 0. This contradicts the fact that u is an open ray.
We can now consider sup(Θ) − inf(Θ) ≤ π. The line tu∗ : t ∈ R, u∗ ∈ R2 − {0} that goes through 0


and makes an angle of inf(Θ) with u (clockwise) has the whole X on one side; all points in X make an angle
in [inf(Θ), sup(Θ)] with u (clockwise). We can take v one of the two vectors perpendicular to u∗ in 0 and
·
obtain v x ≥ 0 for all x ∈ X.

Now consider n > 2. Take a 2-dimensional sub-space S2 that contains the unitary vector u, i.e., S2 =
·
{tu + t0 u0 : t, t0 ∈ R} for some unitary u0 such that u u0 = 0. The intersection of two convex sets is convex,
and so, X2 = S2 ∩ X is convex. Using above lemma for n = 2, there exists some unitary v2 ∈ S2 such that
·
v2 x2 ≥ 0, ∀x2 ∈ X2 . Notice that f (−v2 ) ≤ 0 because all −tv2 with t > 0 do not belong to X or X2 ,
·
since v2 (−tv2 ) < 0.
Take any unitary u2 ∈ S2 such that

·
v2 u2 = 0. (C.4.6)

100
Consider the (n − 1)-dimensional sub-space Sn−1 ( Rn perpendicular on u2 , i.e., Sn−1 = {x ∈ X : u2 x =
0}. Observe v2 and −v2 belong to Sn−1 , using (C.4.6). We now project the whole space X on Sn−1 , i.e.,
·
·
we obtain the set Xn−1 = {x − u2 (u2 x) : x ∈ X}. One can easily check that all elements of Xn−1 satisfy
· · · · ·
u2 (x − u2 (u2 x)) = u2 x − u2 u2 (u2 x) = u2 x − u2 x = 0. · ·
We now define function fn−1 : Sn−1 → R ∪ {−∞} in the same style as f , i.e., fn−1 (sn−1 ) is the smallest
t for which tsn−1 ∈ Xn−1 , for any sn−1 ∈ Sn−1 . We showed above that f (−v2 ) ≤ 0. We can also prove
·
fn−1 (−v2 ) ≤ 0. Recall we have v2 x2 ≥ 0, ∀x2 ∈ X2 . Consider now the projection xn−1 = x2 −u2 (u2 x2 ) ·
· · · · · ·
and notice that v2 xn−1 = v2 (x2 − u2 (u2 x2 )) = v2 x2 − v2 u2 (u2 x2 ) = v2 x2 ≥ 0 (we used ·
(C.4.6) for the last equality). The elements −tv2 with t > 0 can not belong to the projection of X2 , because
· ·
v2 (−tv2 ) < 0 and the elements xn−1 of the projection verify v2 xn−1 ≥ 0. Finally, remark we do not
lose generality by restricting the argument to the projections of X2 : all elements of X that could project on
−tv2 could only belong to X2 , i.e., the space generated by v2 and u2 .
We can easily check that Xn−1 is convex. Consider xn−1 ∈ Xn−1 as the projection of xn−1 + au2 ∈ X
and yn−1 ∈ Xn−1 as the projection of yn−1 + bu2 ∈ X. Using the convexity of X, the following holds for
any α ∈ [0, 1]: α(xn−1 + au2 ) + (1 − α)(yn−1 + bu2 ) ∈ X. We can re-write this as: αxn−1 + (1 − α)yn−1 +
(αa + (1 − α)b)u2 ∈ X, and so, αxn−1 + (1 − α)yn−1 ∈ Xn−1 (one can easily verify that the scalar product
of this with u2 is zero).
We now apply the induction hypothesis on set Xn−1 with fn−1 (−v2 ) ≤ 0 in the sub-space Sn−1 (notice
this is a full (n − 1)-dimensional space where all elements can be written as a linear combination of a
·
canonical basis perpendicular to u2 ). We obtain there is non-zero vn−1 ∈ Sn−1 such that vn−1 xn−1 ≥ 0
·
for all xn−1 ∈ Xn−1 . This means that vn−1 (xn−1 + au2 ) ≥ 0 for any xn−1 ∈ Xn−1 and any a ∈ R, because
vn−1 is perpendicular on u2 . It is easy to check that X ⊆ {xn−1 + au2 : xn−1 ∈ Xn−1 , a ∈ R}. This is
enough to conclude that vn−1 x ≥ 0 for any x ∈ X.

C.4.3 Proving the theorem using well-established textbook arguments

Theorem C.4.5. (Simple separation theorem) Given convex closed set X ⊂ Rn and some y ∈ Rn such that
/ X, there exist a non-zero v ∈ Rn such that
y∈

Proof. We need the following lemma.

v
· · x>v y, ∀x ∈ X. (C.4.7)

Lemma C.4.5.1. Given closed convex set p X, there exist a unique x ∈ X such that |x| = inf{|x0 | : x0 ∈ X},
where | · | is the norm (length), e.g., |x| = x x. ·
Proof. The Wikipedia proof is kind of magical for my taste, with a few tricks arising rather out of the blue.
I provide a more natural and even simpler (without Cauchy sequences) proof.
Let δ = inf{|x0 | : x0 ∈ X}. We first show X contains a sequence {xi } such that lim |xi | = δ. For
i→∞
instance, we can consider 1 = 1, 2 = 21 , 3 = 13 , . . . . For each i (i ∈ N∗ ), X needs to contain some xi
such that |xi | < δ + i , because otherwise we would have δ + i ≤ inf{|x0 | : x0 ∈ X}, impossible since
δ = inf{|x0 | : x0 ∈ X}. The sequence {xi } constructed this way satisfies lim |xi | = δ.
i→∞
This sequence {xi } of elements of Rn needs to contain a convergent subsequence using the Bolzano-
Weierstrass Theorem C.4.9, i.e., there exists a sub-sequence {xni } such that lim xni = x. Since X is closed,
i→∞
it contains all limit points, and so, x ∈ X. It is not hard now to check that the sub-sequence {|xni |} converges
to δ. Since for any  there exists m ∈ N∗ such that |xj | < δ +  ∀j ≥ m, there must be some nm ≥ m (the
sub-sequence is infinite) such that |xnj | < δ +  ∀nj ≥ nm . This confirms δ = lim |xni | = |x|.
i→∞
We still need to show that x is the unique element of minimum norm. Suppose there exists y ∈ X − {x}
x · x+y · y+2x · y
= δ +x2 · y . We will
2
such that |y| = δ. By convexity, x+y x+y 2
2 ∈ X. We can calculate | 2 | =
· 4
· · ·
show x y < δ 2 . For this, it is enough to observe that 0 < |x − y|2 = x x + y y − 2x y = 2δ 2 − 2x y, ·
·
i.e., x y < δ 2 . We obtained that | x+y 2 2
2 | < δ , contradiction. There is no y 6= x in X such that |y| = δ.

We will first prove the theorem for y = 0 and then we will use a simple translation argument to extend
it for an arbitrary y. Using the lemma, let us take v ∈ X of minimum norm δ > 0 (because y = 0 ∈ / X).
We will prove the following:

·
v x > 0, ∀x ∈ X (C.4.8)

101
 Consider any x ∈ X and write ∆ = x − v. Keeping in mind the goal of showing v ∆ ≥ 0, consider ·
·
a function f : [0, 1] → R defined by f (t) = (v + t∆) (v + t∆). By convexity, we simply have v + t∆ ∈
X ∀t ∈ [0, 1], and, using the above lemma, we also obtain f (t) > f (0) = γ ∀t > 0. This means the derivative
· ·
in 0 can not be negative; we need to have f 0 (t) ≥ 0. Since f 0 (t) = 2v ∆ + t2 ∆ ∆, this means v ∆ ≥ 0, ·
· ·
enough to show v x = v (v + ∆) = γ + v ∆ ≥ γ > 0. ·
Finally, if y 6= 0, it is enough to consider set X 0 = {x − y : x ∈ X} which is convex and does not contain
·
0. We can apply the theorem for 0 and X 0 and we obtain there is v ∈ Rn such that v (x − y) > 0, ∀x ∈ X,
which is equivalent to (C.4.7) as needed.
Theorem C.4.6. (Simple supporting hyperplane theorem) Given convex closed set X ⊂ Rn such that 0 is a
boundary point of X, there exist a non-zero v ∈ Rn such that

v
· x ≥ 0, ∀x ∈ X. (C.4.9)

/ X, ∀i ∈ N∗ )
Proof. Since 0 is a boundary point, there exists a sequence {xi } of exterior points (i.e., xi ∈
that converges to 0. Using the simple separation Theorem C.4.5, for each xi there exists a non-zero unitary
vi ∈ Rn such that

· ·
vi x > vi xi ∀x ∈ X. (C.4.10)

Without loss of generality, we can consider all vi are unitary, i.e., |vi | = 1 ∀i ∈ N∗ . As such, the sequence
{vi } is bounded, and so, we can apply the Bolzano-Weierstrass Theorem C.4.9 to conclude that {vi } contains
a convergent sub-sequence {vni } such that lim vni = v.
·
i→∞
Assume there is some x ∈ X such that v x = −a < 0. We derive a contradiction using a limiting
· ·
argument. We take an  < |a|; since lim vni x = v x = −a, there exits some m ∈ N∗ such that
·
i→∞
v ni x ∈ [−a − , −a + ] for all i ≥ m. Applying (C.4.10), we obtain that all these i ≥ m satisfy
v ni · xni < −a +  < 0. This contradicts the fact that lim xni = 0. Indeed, if vni can become arbitrarily
·x
i→∞
close to v while at the same time xni becomes arbitrarily close to 0, then the product vni ni can also
become arbitrarily close to 0.

C.4.3.1 Convergence theorems on sequences

We need several convergence results on sequences for the classical proof of the hyperplane separation theorem,
i.e., in particular for Theorems C.4.5 and C.4.6.

Proposition C.4.7. Any bounded set S ( R has a unique finite least upper bound (or supremum) sup(S).
Equivalently, a unique inf(S) also exists and is finite.
Proof. To avoid unessential complication, we will assume that S contains at least a positive number. Any
set S 0 can be transformed to this form by applying a simple translation S = {s0 − s00 : s ∈ S 0 } for some
fixed s00 ∈ S 0 . If sup(S) exists, then sup(S 0 ) = sup(S) + s00 .
We will determine sup(S) as a real number a written in the decimal expansion as a = a0 .a1 a2 a3 . . . with
potentially infinite number of digits. We can suppose that a0 .a1 a2 . . . an 99 . . . 9 with an 6= 9 is equal to
a0 .a1 a2 . . . an−1 (an + 1). We choose the decimals as follows.
– a0 is the greatest integer that is not a strict upper bound for S (i.e., that is not strictly greater than
all elements of S). A finite a0 value must exists because S is bounded.

– a1 is the greatest digit such that a0 .a1 is not a strict upper bound of S
– a2 is the greatest digit such that a0 .a1 a2 is not a strict upper bound of S
..
.
– an is the greatest digit such that a0 .a1 a2 . . . an is not a strict upper bound of S
..
.

102
We now prove that a = a0 .a1 a2 a3 . . . is an upper bound of S. Assume there exists an s ∈ S such that
s > a. This means there exists an index n such that s can be written a0 .a1 a2 . . . an−1 sn sn+1 sn+2 . . . where
sn > an . Since an is the greatest digit such that a0 .a1 a2 . . . an is not a strict upper bound for S, we obtain
that a0 .a1 a2 . . . an−1 sn is a strict upper bound for S. This implies that s ≥ a0 .a1 a2 . . . an−1 sn is also a strict
upper bound of S, which contradicts s ∈ S. There can be no s ∈ S such that s > a.
To prove the uniqueness, we still have to show that a is the minimum upper bound, i.e., there is no other
upper bound a0 < a. For the sake of contradiction, assume there exists such an upper bound a0 < a; it can
be written a0 = a0 .a1 a2 . . . an−1 a0n a0n+1 a0n+2 . . . such that a0n < an for some n and a0n+1 , a0n+2 . . . are not all
9—such a number would reduce to a0 .a1 a2 . . . an−1 (a0n + 1). We thus obtain that a0 < a0 .a1 a2 . . . an−1 an .
Since a0 is an upper bound, a0 .a1 a2 . . . an−1 an needs to be a strict upper bound. This contradicts the choice
of an as the greatest digit such that a0 .a1 a2 . . . an−1 an is not a strict upper bound of S.
By combining the two above paragraphs, we obtain that a = a0 .a1 a2 a3 . . . is the least upper bound
sup(S). Recall we can suppose that a0 .a1 a2 . . . an 99 . . . 9 with an 6= 9 is equal to a0 .a1 a2 . . . an−1 (an + 1).
Proposition C.4.8. Any bounded monotone sequence {ai } of real numbers is convergent.
Proof. Since the sequence is monotone, we can consider it is non-decreasing, the non-increasing case being
completely analogous. Let A = {ai : i ∈ {1, 2, . . . ∞}}. Since A is bounded, the least upper bound property
(Prop. C.4.7) states that a = sup(A) exists and is finite. For any  > 0, there needs to exist some positive
integer n such that an > a − , because otherwise a −  would be an upper bound lower than sup(A),
impossible. Since {ai } is non-decreasing, all integers n0 > n verify an0 ≥ an > a − . This is exactly the
definition of the fact that lim ai = a.
i→∞

Theorem C.4.9. (Bolzano–Weierstrass theorem) Any bounded sequence {xi } of Rn contains a convergent
subsequence.
Proof. We first show the theorem for n = 1.
Lemma C.4.9.1. Any bounded sequence {xi } of R contains a convergent subsequence.
Proof. We consider the set of maxima N = {i ∈ {1, 2, . . . } : xj < xi , ∀j > i}. We distinguish three cases
depending on the cardinal of N :
1. If N = ∅, then for any index ni , there exists a position ni+1 such that xni+1 ≥ xni . The subsequence
xn1 , xn2 , xn3 , . . . is monotone non-decreasing, and so, convergent using Prop. C.4.8.
2. If N is not finite, then N contains an infinite sequence of indices n1 < n2 < n3 < . . . such that
xn1 > xn2 > xn3 . . . . The subsequence xn1 , xn2 , xn3 . . . is monotone decreasing, and using Prop. C.4.8,
it is convergent.
3. If |N | = t with t ∈ N − {0}, then N contains a finite sequence of indices n1 < n2 < n3 < . . . nt such
that xn1 > xn2 > xn3 · · · > xnt . The set N>nt = {i ∈ {nt + 1, nt + 2, nt + 3, . . . } : xj < xi , ∀j > i} is
empty. We can thus apply the argument of case 1 and obtain that the infinite sequence xnt +1 , xnt +2 ,
xnt +3 , . . . contains a convergent sub-sequence.
We now generalize the result for any n > 1. Considering the first position of the sequence {xi }, the above
lemma shows that {xi } contains a sub-sequence {x1i } whose first position converges to some y1 . We now
consider the second position of {x1i }. Using the lemma again, we obtain that {x1i } contains a sub-sequence
{x2i } whose second position converges to some y2 . We observe that the first position of {x2i } converges to
y1 and the second to y2 . The argument can be repeated to find sub-sequences {x1i }, {x2i }, . . . , {xni } such
that {xni } converges to [y1 y2 . . . yn ]> .

103
References
References are provided throughout the document as footnote citations. This is because I wanted to make
each reference readily available to the reader. I acknowledge again that I mentioned throughout the docu-
ment the work of the following people (lecture notes and papers only, excluding web-sites and responses on
mathematical on-line forums), in the order of apparition: Maurı́cio de Oliveira, Christoph Helmberg, Robert
Freund, David Williamson, Roger Horn, Charles Johnson H. Ikramov, Stephen Boyd, Lieven Vandenberghe,
Anupam Gupta, László Lovász, Michael Overton, Henry Wolkowicz, Michel Goemans, Nebojša Gvozden-
ović, Donald Knuth, Monique Laurent, Immanuel Bomze, Mirjam Dür, Chung-Piaw Teo, Peter Dickinson,
Luuk Gijben, Etienne de Klerk, Dmitri Pasechnik, Pablo Parrilo, Frédéric Roupin, Alain Billionnet, Sourour
Elloumi, Marie-Christine Plateau, Alain Faye, Amélie Lambert, Peter Norman, Subhash Khot, Guy Kindler,
Elchanan Mossel, Ryan O’Donnell and Richard Dudley.

104