Introduction to MIMO Wireless

Communication
MIMO Wireless Communication
(ELL8415)

Anupama Rajoriya
Assistant Professor
Department of Electrical Engineering
Motivation for MIMO communication
Multiple transmit and/or receive antennas
o Diversity gain: Transmit same signal over multiple channels→ boosts the SNR
o Spatial multiplexing: Provides transmission of multiple parallel streams
o Beamforming- serving users in a particular direction→ Interference reduction

Course Objectives

Basics of Information Understanding MIMO Combining and

theory wireless channel beamforming designs
Tools to derive capacity Mathematical modelling To maximize the capacity

5G and 6G MIMO Channel estimation

systems techniques
Introducing state-of-the-art Challenges, solutions and
MIMO technologies some open problems
Evaluation plan
Evaluation Plan
35

30
30 % 30 %

25

20
20 % 20 %

15

10

0
Assignments/Quizes Midsem Term project Endsem
A Wireless communication link
• Additive white Gaussian noise (AWGN)

Transmitter Receiver
𝑥 𝑦
o Received signal:
𝑦 =𝑥+𝑛

o Here, 𝑥 ∈ − 𝑃, 𝑃 , and 𝑛 ∼ 𝐶𝑁(0, 𝜎 2 )

− 𝑃 + 𝑛, 𝑖𝑓 𝑥 = − 𝑃
o𝑦= ൝
𝑃 + 𝑛, 𝑖𝑓 𝑥 = 𝑃

o If 𝑦 ≥ 0, the 𝑥ො = 𝐴, and if 𝑦 < 0, the 𝑥ො = −𝐴.

Motivation for MIMO communication?
• Additive white Gaussian noise (AWGN)

Transmitter Receiver
𝑥 𝑦

o Capacity: maximum number of bits per second that can be transmitted reliably over a wireless channel
𝑃
𝐶 = log 2 1 + 2
𝜎
𝑃
o Here is commonly known as SNR - Signal to Noise Ratio
𝜎2

𝑃 𝑃 𝑃 Property of Logarithm
o When 𝜎2 ≪ 1 , 𝐶 = log 2 1 + ≈ log 2 𝑒→ Low SNR regime
𝜎2 𝜎2
o
𝑃
When 𝜎2 ≫ 1 , 𝐶 = log 2 1 +
𝑃
≈
𝑃
log 2 𝜎2 → High SNR regime 𝑥 log 2 𝑒, if 𝑥 ≪ 1
𝜎2 log 2 1 + 𝑥 = ቊ
log 2 𝑥, if 𝑥 ≫ 1.

At high SNR, capacity increases only logarithmically!

Motivation for MIMO communication?
• Additive white Gaussian noise (AWGN)

Transmitter Receiver
𝑥 𝑦

o Capacity: maximum number of bits per second that can be transmitted reliably over a wireless channel
𝑃
𝐶 = log 2 1 + 2
𝜎
𝑃
o Here is commonly known as SNR - Signal to Noise Ratio
𝜎2

𝑃 𝑃 𝑃 Property of Logarithm
o When 𝜎2 ≪ 1 , 𝐶 = log 2 1 + ≈ log 2 𝑒→ Low SNR regime
𝜎2 𝜎2
o
𝑃
When 𝜎2 ≫ 1 , 𝐶 = log 2 1 +
𝑃
≈
𝑃
log 2 𝜎2 → High SNR regime 𝑥 log 2 𝑒, if 𝑥 ≪ 1
𝜎2 log 2 1 + 𝑥 = ቊ
log 2 𝑥, if 𝑥 ≫ 1.

At high SNR, capacity increases only logarithmically!

Motivation for MIMO communication
Multiple transmit/ receive antennas
o Transmit same signal over multiple channels (Diversity gain)→ boosts the SNR
o Provide transmission of multiple parallel streams (Spatial multiplexing)
o Beamforming- serving users in a particular direction→ Interference reduction

Single input and multiple output (SIMO) Multiple input and single output (MISO)
o One transmit antenna, two receive antennas o Two transmit antennas, one receive antenna

𝑦1 𝑥1
Transmitter Receiver Transmitter
𝑥 Receiver
𝑦2 𝑥2 𝑦
Single input and multiple output (SIMO)
• Additive white Gaussian noise (AWGN)

𝑦1
o Received signal at antenna 1:
𝑦1 = 𝑥 + 𝑛1 Transmitter Receiver
o Received signal at antenna 2: 𝑥
𝑦2 = 𝑥 + 𝑛2 𝑦2
o 𝑛1 and 𝑛2 are independently random variables with distribution CN(0, 𝜎 2 )
o Added received signal
𝑦 = 𝑦1 + 𝑦2 = 2𝑥 + 𝑛1 + 𝑛2
2𝑃
o SNR calculation : 𝜎2
o Capacity
2𝑃
𝐶 = log 2 1 + 2
𝜎
2𝑃 2𝑃
o At low SNR, 𝐶 = log 2 1 + 𝜎2 ≈ 𝜎2 log 2 𝑒→ low SNR regime

At low SNR, capacity increases linearly with number of receive antennas→ Boosting the SNR
Multiple input and single output (MISO)
• Additive white Gaussian noise (AWGN)
o One symbol transmitted over two antennas 𝑥1
o Signal transmitted over antenna 1:
Transmitter
𝑥1 = 𝑥/ 2 Receiver
o Signal transmitted over antenna 2: 𝑥2 𝑦
𝑥2 = 𝑥/ 2
P P
o Total transmit power: 2 + 2 = 𝑃 (same as single antenna case)
o Received signal:
𝑥 𝑥
𝑦 = 𝑥1 + 𝑥2 + 𝑛 = + + 𝑛 = 2𝑥 + 𝑛
2 2
2𝑃
o SNR calculation : 𝜎2
o Capacity
2𝑃
𝐶 = log 2 1 + 2
𝜎
2𝑃 2𝑃
o At low SNR, 𝐶 = log 2 1 + 𝜎2 ≈ 𝜎2 log 2 𝑒→ Low SNR regime
At low SNR, capacity increases linearly with number of receive antennas→ Boosting the SNR
Motivation for MIMO communication?
• Fading channel

Channel
Transmitter Receiver
𝑥 𝑦

o Modeled as complex multiplicative noise

o Received signal:
𝑦 = ℎ𝑥 + 𝑛
o Signal power: ℎ 2 𝑃
ℎ 2𝑃
o Capacity: 𝐶 = log 2 1 + 𝜎2
Single input and multiple output (SIMO)
• Fading channel
ℎ1
𝑦1
o Received signal at antenna 1:
𝑦1 = ℎ1 𝑥 + 𝑛1 ℎ2
Transmitter Receiver
o Received signal at antenna 2: 𝑥
𝑦2 = ℎ2 𝑥 + 𝑛2 𝑦2
o 𝑛1 and 𝑛2 are independently random variables with distribution CN(0, 𝜎 2 )
o Weigh and Add the two signals to strengthen the SNR
o Lets use Matched filtering
𝑦 = ℎ1∗ 𝑦1 + ℎ2∗ 𝑦2 = ℎ1 2 + ℎ2 2 𝑥 + ℎ1∗ 𝑛1 + ℎ2∗ 𝑛2

o SNR calculation :
ℎ1 2 + ℎ2 2 2 𝑃 ℎ1 2 + ℎ2 2 𝑃
= Example
ℎ1 2 𝜎 2 + ℎ2 2 𝜎 2 𝜎2
o Capacity For ℎ1 = 1, ℎ2 = −0.5,
2 2
ℎ1 + ℎ2 𝑃 1.25𝑃
𝐶 = log 2 1 + SNR = 𝜎2
𝜎2
Matched filter used here maximizes the SNR
Single input and multiple output (SIMO)
• Fading channel
ℎ1
𝑦1
o Received signal at antenna 1:
𝑦1 = ℎ1 𝑥 + 𝑛1 ℎ2
Transmitter Receiver
o Received signal at antenna 2: 𝑥
𝑦2 = ℎ2 𝑥 + 𝑛2 𝑦2
o 𝑛1 and 𝑛2 are independently random variables with distribution CN(0, 𝜎 2 )
o Weigh and Add the two signals to strengthen the SNR Receive beamforming/combining
o Lets use Matched filtering
𝑦 = ℎ1∗ 𝑦1 + ℎ2∗ 𝑦2 = ℎ1 2 + ℎ2 2 𝑥 + ℎ1∗ 𝑛1 + ℎ2∗ 𝑛2

o SNR calculation :
ℎ1 2 + ℎ2 2 2 𝑃 ℎ1 2 + ℎ2 2 𝑃
= Example
ℎ1 2 𝜎 2 + ℎ2 2 𝜎 2 𝜎2
o Capacity For ℎ1 = 1, ℎ2 = −0.5,
2 2
ℎ1 + ℎ2 𝑃 1.25𝑃
𝐶 = log 2 1 + SNR = 𝜎2
𝜎2
Matched filter used here maximizes the SNR
Multiple input and single output (MISO)
• Fading channel ℎ1
o One symbol transmitted over two antennas 𝑥1 ℎ2
o Channel is assumed to be unknown at the transmitter
o Signal transmitted over antenna 1: Transmitter
Receiver
𝑥1 = 𝑥/ 2 𝑥2 𝑦
o Signal transmitted over antenna 2:
𝑥2 = 𝑥/ 2
o Received signal:
𝑥 𝑥 𝑥
𝑦 = ℎ1 𝑥1 + ℎ2 𝑥2 + 𝑛 = ℎ1 + ℎ2 + 𝑛 = ℎ1 + ℎ2 +𝑛
2 2 2
o SNR calculation :
| ℎ1 + ℎ2 |2 𝑃
Example
2𝜎 2
For ℎ1 = 1, ℎ2 = −0.5,
0.125𝑃
SNR = 𝜎2

SNR reduced by a factor of 10!

Can we do any better?
Multiple input and single output (MISO)
• Fading channel ℎ1
o Assume the channels to be known at the transmitter 𝑥1 ℎ2
o Signal transmitted over antenna 1:
ℎ1∗ 𝑥 Transmitter
Receiver
𝑥1 =
𝑐 𝑥2 𝑦
o Signal transmitted over antenna 2:
ℎ2∗ 𝑥
𝑥2 =
𝑐
ℎ1 2 + ℎ2 2 𝑃
o Transmit power (should be P): =𝑃
𝑐
⟹𝑐 = ℎ1 2 + ℎ2 2

o Received signal:
2
𝑥 2
𝑥 2 2
𝑥
𝑦 = ℎ1 𝑥1 + ℎ2 𝑥2 + 𝑛 = ℎ1 + ℎ2 +𝑛= ℎ1 + ℎ2 +𝑛
𝑐 𝑐 𝑐
Example
o SNR calculation :
ℎ1 2 + ℎ2 2 𝑃
For ℎ1 = 1, ℎ2 = −0.5,
𝜎2 1.25𝑃
SNR = 𝜎2
Multiple input and single output (MISO)
• Fading channel ℎ1
o Assume the channels to be known at the transmitter 𝑥1 ℎ2
o Signal transmitted over antenna 1:
ℎ1∗ 𝑥 Transmitter
Receiver
𝑥1 =
𝑐 𝑥2 𝑦
o Signal transmitted over antenna 2:
ℎ2∗ 𝑥
Transmit beamforming 𝑥2 = 𝑐
ℎ1 2 + ℎ2 2 𝑃
o Transmit power (should be P): =𝑃
𝑐
⟹𝑐 = ℎ1 2 + ℎ2 2

o Received signal:
2
𝑥 2
𝑥 2 2
𝑥
𝑦 = ℎ1 𝑥1 + ℎ2 𝑥2 + 𝑛 = ℎ1 + ℎ2 +𝑛= ℎ1 + ℎ2 +𝑛
𝑐 𝑐 𝑐
Example
o SNR calculation :
ℎ1 2 + ℎ2 2 𝑃
For ℎ1 = 1, ℎ2 = −0.5,
𝜎2 1.25𝑃
SNR = 𝜎2
Multiple input and single output (MISO)
• Fading channel ℎ1
o Assume the channels to be known at the transmitter 𝑥1 ℎ2
o Signal transmitted over antenna 1:
ℎ1∗ 𝑥 Transmitter
Receiver
𝑥1 =
𝑐 𝑥2 𝑦
o Signal transmitted over antenna 2:
ℎ2∗ 𝑥
Transmit beamforming 𝑥2 = 𝑐
ℎ1 2 + ℎ2 2 𝑃
o Transmit power (should be P): =𝑃 Multiple receive antennas Multiple transmit antennas
𝑐
⟹𝑐 = ℎ1 2 + ℎ2 2
Combining Beamforming
o Received signal:
2
𝑥 2
𝑥 2 2
𝑥
𝑦 = ℎ1 𝑥1 + ℎ2 𝑥2 + 𝑛 = ℎ1 + ℎ2 +𝑛= ℎ1 + ℎ2 +𝑛
𝑐 𝑐 𝑐
Example
o SNR calculation :
ℎ1 2 + ℎ2 2 𝑃
For ℎ1 = 1, ℎ2 = −0.5,
𝜎2 1.25𝑃
SNR = 𝜎2
Multiple input and multiple output (MIMO)
ℎ11
• Additive white Gaussian noise (AWGN) ℎ12
o Two symbols transmitted over two antennas ℎ21 𝑦1
𝑥1
o Signals transmitted: 𝑥1 and 𝑥2 ℎ22
o Received signal at antenna 1: Transmitter Receiver
𝑦1 = ℎ11 𝑥1 + ℎ12 𝑥2 + 𝑛1 𝑥2 𝑦2
o Received signal at antenna 2:
𝑦2 = ℎ21 𝑥1 + ℎ22 𝑥2 + 𝑛2

• Matrix representation of MIMO system

o Complete received signal, transmit signal and noise as vectors: o Transmitted signals can be recovered
𝑦1 𝑥1 𝑛1 as follows:
𝒚 = 𝑦 , 𝒙 = 𝑥 , and 𝒏 = 𝑛 .
2 2 2 ෝ = 𝑯−𝟏 𝒚
𝒙
o Complete channel as a matrix
ℎ ℎ12
𝑯 = 11
ℎ21 ℎ22
o Input output relationship of MIMO system becomes

𝒚 = 𝑯𝒙 + 𝒏
2×1
2×1 2×2
Motivation for MIMO communication
Multiple transmit and/or receive antennas
o Diversity gain: Transmit same signal over multiple channels→ boosts the SNR
A signal transmitted over two independently fading channels is more likely to be correctly detected than the
same signal transmitted over a single fading channel
Motivation for MIMO communication
Multiple transmit and/or receive antennas
o Diversity gain: Transmit same signal over multiple channels→ boosts the SNR
A signal transmitted over two independently fading channels is more likely to be correctly detected than the
same signal transmitted over a single fading channel
o Spatial multiplexing: Provides transmission of multiple parallel streams

ℎ11
ℎ12
ℎ21 𝑦1
𝑥1
ℎ22
Transmitter Receiver
𝑥2 𝑦2
Motivation for MIMO communication
Multiple transmit and/or receive antennas
o Diversity gain: Transmit same signal over multiple channels→ boosts the SNR
A signal transmitted over two independently fading channels is more likely to be correctly detected than the
same signal transmitted over a single fading channel
o Spatial multiplexing: Provides transmission of multiple parallel streams

ℎ11
ℎ12
ℎ21 𝑦1
𝑥1
ℎ22
Transmitter Receiver
𝑥2 𝑦2
 Lecture 2

MIMO Wireless Communication

(ELL8415)
Anupama Rajoriya
Basics of Information Theory

Reference:
Elements of Information Theory
Thomas M. Cover, Joy A. Thomas
Information theory
o Information theory- compressibility, storage of data and reliable communication
o Every channel has a characterizing quantity (capacity), such that, for the transmission rates below it,
the error probability could be made arbitrarily small.
o In other words, it is the maximum achievable rate with reliability (very low error probability)
o Information required to describe a quantity ≡ uncertainty
o If a quantity is certain, no information
o Appropriate to model information and data communication using probability theory
o Familiarity with probability theory is expected

o Some basic definitions:

Entropy,
Joint and conditional entropies,
Relative entropy,
Mutual information
Entropy
o Let 𝑋 ∈ 𝒳be a discrete random variable
o With probability mass function (pmf) :
𝑝 𝑥 = Pr 𝑋 = 𝑥 , 𝑋 ∈ 𝒳

Entropy: self-information/uncertainty of a random variable

𝐻 𝑋 = − ෍ 𝑝 𝑥 log 2 𝑝 𝑥 = −𝔼 log 2 𝑝 𝑋
𝑥∈𝒳
o Non-negative: 𝐻 𝑋 ≥ 0
o larger entropy ⇒ 𝑋 is more unpredictable
o Zero entropy- 𝑋 is deterministic (no uncertainty)
o Information (no of bits) required to describe the quantity
o Function of probabilities p(x), not of X

Example
1, with probabilit𝑦 p
o Entropy of a Bernoulli random variable 𝑋 = ቊ
0, with probabilit𝑦 (1 − p)
𝐻 𝑋 = −𝑝 log 2 𝑝 − (1 − 𝑝) log 2(1 − 𝑝)
o X is output of a fair coin toss: 𝑝 = 1/2
1 1 1 1
o Entropy: 𝐻 𝑋 = − log 2 − log 2 = 1 bit
2 2 2 2
Entropy
o Let 𝑋 ∈ 𝒳be a discrete random variable
o With probability mass function (pmf) :
𝑝 𝑥 = Pr 𝑋 = 𝑥 , 𝑋 ∈ 𝒳

Entropy: self-information/uncertainty of a random variable

𝐻 𝑋 = − ෍ 𝑝 𝑥 log 2 𝑝 𝑥 = −𝔼 log 2 𝑝 𝑋

𝐻 𝑋
𝑥∈𝒳
o Non-negative: 𝐻 𝑋 ≥ 0
o larger entropy ⇒ 𝑋 is more unpredictable
o Zero entropy- 𝑋 is deterministic (no uncertainty)
o Information (no of bits) required to describe the quantity
o Function of probabilities p(x), not of X

Example
1, with probabilit𝑦 p
o Entropy of a Bernoulli random variable 𝑋 = ቊ
0, with probabilit𝑦 (1 − p)
𝐻 𝑋 = −𝑝 log 2 𝑝 − (1 − 𝑝) log 2(1 − 𝑝)
o X is output of a fair coin toss: 𝑝 = 1/2
1 1 1 1
o Entropy: 𝐻 𝑋 = − log 2 − log 2 = 1 bit
2 2 2 2
Joint entropy and conditional entropy
o For a pair of random variables (𝑋, 𝑌) with the joint distribution 𝑝(𝑥, 𝑦), joint entropy is defined as
𝐻 𝑋, 𝑌 = − ෍ ෍ 𝑝 𝑥, 𝑦 log 2 𝑝 𝑥, 𝑦 = −𝔼 log 2 𝑝(𝑥, 𝑦)
𝑥∈𝒳 𝑦∈𝒴
o Conditional entropy of random variable 𝑌 given 𝑋 is defined as
𝐻 𝑌|𝑋 = − ෍ 𝑝 𝑥 𝐻(𝑌|𝑋 = 𝑥) = −𝔼𝑝 𝑥 𝐻(𝑌|𝑋 = 𝑥)
𝑥∈𝒳

= ෍ 𝑝 𝑥 ෍ 𝑝 𝑦|𝑥 log 2 𝑝 𝑦|𝑥

𝑥∈𝒳 𝑦∈𝒴

= ෍ ෍ 𝑝 𝑥, 𝑦 log 2 𝑝 𝑦|𝑥 expected value of entropies of

conditional distributions H(Y|X),
𝑥∈𝒳 𝑦∈𝒴
= −𝔼𝑝 log 2 𝑝 𝑦|𝑥 averaged over the conditioning
𝑥,𝑦
random variable X
Properties
o Conditional entropy of 𝑌 given 𝑋 is defined as
𝐻 𝑌|𝑋 = − ෍ 𝑝 𝑥 𝐻(𝑌|𝑋 = 𝑥) = ෍ ෍ 𝑝 𝑥, 𝑦 log 2 𝑝 𝑦|𝑥
𝑥∈𝒳 𝑥∈𝒳 𝑦∈𝒴
o Conditional entropy of 𝑋 given 𝑌 is defined as
𝐻 𝑋|𝑌 = − ෍ 𝑝 𝑦 𝐻(𝑋|𝑌 = 𝑦) = ෍ ෍ 𝑝 𝑥, 𝑦 log 2 𝑝 𝑥|𝑦
𝑦∈𝒴 𝑥∈𝒳 𝑦∈𝒴
o Conditional entropy is asymmetric, i.e.,
𝐻 𝑌|𝑋 ≠ 𝐻 𝑋|𝑌
o Chain rule: entropy of a pair of random variables is the entropy of one plus the conditional entropy of
the other 𝐻 𝑋, 𝑌 = 𝐻 𝑋 + 𝐻 𝑌|𝑋
𝐻 𝑋, 𝑌 = 𝐻 𝑌 + 𝐻 𝑋|𝑌
Proof:
𝐻 𝑋, 𝑌 = − ෍ ෍ 𝑝 𝑥, 𝑦 log 2 𝑝 𝑥, 𝑦 = − ෍ ෍ 𝑝 𝑥, 𝑦 log 2 𝑝 𝑥 𝑝(𝑦|𝑥)
𝑥∈𝒳 𝑦∈𝒴 𝑥∈𝒳 𝑦∈𝒴

= − ෍ ෍ 𝑝 𝑥, 𝑦 log 2 𝑝 𝑥 − ෍ ෍ 𝑝 𝑥, 𝑦 log 2 𝑝(𝑦|𝑥)

𝑥∈𝒳 𝑦∈𝒴 𝑥∈𝒳 𝑦∈𝒴

= − ෍ 𝑝 𝑥 log 2 𝑝 𝑥 − ෍ ෍ 𝑝 𝑥, 𝑦 log 2 𝑝 𝑦 𝑥 = 𝐻 𝑋 + 𝐻 𝑌|𝑋

𝑥∈𝒳 𝑥∈𝒳 𝑦∈𝒴
Relative Entropy
o Distance between two distributions (PMFs). For two PMFs 𝑝 𝑥 and q 𝑥 , it is defined as

𝑝 𝑥 𝑝 𝑥
𝐷(𝑝||𝑞) = ෍ 𝑝 𝑥 log = 𝔼𝑝 log .
𝑞 𝑥 𝑞 𝑥
𝑥∈𝒳

0 𝑝
o Convention used: 0 log 𝑞 = 0 and p log 0 = ∞
o measure of the inefficiency of assuming that the distribution is q when the true distribution is p
o if true distribution 𝑝 𝑥 is known, code can be constructed with average description length H(p)
o if a distribution 𝑞 𝑥 is used, we would need H p + 𝐷(𝑝||𝑞) bits on the average
o Always non-negative and 𝐷(𝑝| 𝑞 = 0, if and only 𝑝 = 𝑞.
o Not symmetric: 𝐷(𝑝| 𝑞 ≠ 𝐷(𝑞||𝑝)
Example
o Consider two distributions 𝑝 and 𝑞 on a random variable 𝑋 ∈ {0,1}
o Let 𝑝(0) = 1 − 𝑟, 𝑝 1 = 𝑟 and 𝑞 0 = 1 − 𝑠, 𝑞 1 = 𝑠
o Relative entropy:
𝑝 0 𝑝 1 1−𝑟 𝑟
D(p| q = 𝑝 0 log 2 𝑞 + 𝑝 1 log 2 𝑞 = 1 − 𝑟 log 2 + 𝑟 log 2 𝑠
0 1 1−𝑠
1−𝑠 𝑠
D(q| p = 1 − 𝑠 log 2 + 𝑠 log 2
1−𝑟 𝑟
Relative Entropy
Example
o Consider two distributions 𝑝 and 𝑞 on a random variable 𝑋 ∈ {0,1}
o Let 𝑝(0) = 1 − 𝑟, 𝑝 1 = 𝑟 and 𝑞 0 = 1 − 𝑠, 𝑞 1 = 𝑠
o Relative entropy:
𝑝 0 𝑝 1 1−𝑟 𝑟
D(p| q = 𝑝 0 log 2 𝑞 + 𝑝 1 log 2 𝑞 = 1 − 𝑟 log 2 + 𝑟 log 2 𝑠
0 1 1−𝑠
1−𝑠 𝑠
D(q| p = 1 − 𝑠 log 2 + 𝑠 log 2
1−𝑟 𝑟

o If 𝑟 = 𝑠, ⇒ D(p| q = D(q| p = 0

1 1
o Let 𝑟 = 2 and s = 4
1 1
1 2 1 2
D(p| q = log 2 3 + log 2 1 = 0.2075 bits
2 2
4 4

3 1
3 4 1 4
D(q| p = log 2 1 + log 2 1 = 0.187 bits
4 4
2 2
Mutual information
o measure of the amount of information that one random variable contains about another random variable
o For X and Y, the mutual information 𝐼 𝑋; 𝑌 is defined as
𝑝 𝑥, 𝑦
𝐼 𝑋; 𝑌 = ෍ ෍ 𝑝 𝑥, 𝑦 log
𝑝 𝑥 𝑝(𝑦)
𝑥∈𝒳 𝑦∈𝒴
= 𝐷(𝑝(𝑥, 𝑦)||𝑝 𝑥 𝑝(𝑦))
𝑝 𝑥, 𝑦
= 𝔼𝑝 𝑥,𝑦 log
𝑝 𝑥 𝑝(𝑦)
o Quantifies the reduction in the uncertainty of one random variable due to the knowledge of the other
o Mutual information is symmetric: 𝐼(𝑋; 𝑌) = 𝐼(𝑌; 𝑋)
Mutual information and entropy
o Rewriting the mutual information expression
𝑝 𝑥, 𝑦
𝐼 𝑋; 𝑌 = ෍ ෍ 𝑝 𝑥, 𝑦 log
𝑝 𝑥 𝑝(𝑦)
𝑥∈𝒳 𝑦∈𝒴
𝑝 𝑥|𝑦
= ෍ ෍ 𝑝 𝑥, 𝑦 log
𝑝 𝑥
𝑥∈𝒳 𝑦∈𝒴

= − ෍ ෍ 𝑝 𝑥, 𝑦 log 𝑝 𝑥 + ෍ ෍ 𝑝 𝑥, 𝑦 log 𝑝 𝑥|𝑦

𝑥∈𝒳 𝑦∈𝒴 𝑥∈𝒳 𝑦∈𝒴

= − ෍ 𝑝 𝑥 log 𝑝 𝑥 − − ෍ ෍ 𝑝 𝑥, 𝑦 log 𝑝 𝑥|𝑦

𝑥∈𝒳 𝑥∈𝒳 𝑦∈𝒴
= 𝐻 𝑋 − 𝐻(𝑋|𝑌)
o reduction in the uncertainty of 𝑋 due to the knowledge of the 𝑌
o Similarly, we have 𝐼(𝑌; 𝑋) = 𝐻 𝑌 − 𝐻(𝑌|𝑋)
o Thus, X says as much about Y as Y says about X
o Also, 𝐼(𝑋; 𝑋) = 𝐻(𝑋)→ Entropy is sometimes referred to as self-information
o As we have 𝐻 𝑋, 𝑌 = 𝐻 𝑋 + 𝐻 𝑌 𝑋 = 𝐻 𝑌 + 𝐻 𝑋 𝑌 , we deduce
𝐼 𝑋; 𝑌 = 𝐻 𝑋 + 𝐻 𝑌 − 𝐻(𝑋, 𝑌)
Summary
o 𝐼(𝑋; 𝑌) = 𝐻 𝑋 − 𝐻(𝑋|𝑌)
o 𝐼(𝑌; 𝑋) = 𝐻 𝑌 − 𝐻(𝑌|𝑋)
o 𝐼 𝑋; 𝑌 = 𝐻 𝑋 + 𝐻 𝑌 − 𝐻(𝑋, 𝑌)
o 𝐼(𝑋; 𝑌) = 𝐼(𝑌; 𝑋)
o 𝐼(𝑋; 𝑋) = 𝐻(𝑋)
intersection of the information in
𝑋 with the information in Y
 Jensen’s inequality
o Going to prove some properties of entropy and information
o Recalling convex functions
o Convex functions: A function 𝑓(𝑥) is said to be convex over an interval (𝑎, 𝑏) if for every 𝑥1 , 𝑥2 ∈ (𝑎, 𝑏) and 0 ≤ 𝜆 ≤ 1,

𝑓 𝜆𝑥1 + 1 − 𝜆 𝑥2 ≤ 𝜆𝑓(𝑥1 ) + 1 − 𝜆 𝑓(𝑥2 ).

o A function 𝑓 is said to be strictly convex if equality holds only if 𝜆 = 0 or 𝜆 = 1

o Jensen’s inequality for convex functions: If 𝑓 is a convex function and 𝑋 is a random variable, then
𝑓 𝔼 𝑋 ≤ 𝔼 𝑓(𝑋)
o Equality holds when 𝑋 is constant
Proof: For a convex function 𝑓, from its definition we have
𝑓 𝑝1 𝑥1 + 𝑝2 𝑥2 ≤ 𝑝1 𝑓(𝑥1 ) + 𝑝2 𝑓(𝑥2 )
where 𝑝1 + 𝑝2 = 1. It holds true for a random variable 𝑋 ∈ {𝑥1 , 𝑥2 }, with 𝑝 𝑋 = 𝑥1 = 𝑝1 and 𝑝 𝑋 = 𝑥2 = 𝑝2 .
o By induction, the inequality holds for a distribution on an 𝑋 with (𝑘 − 1) mass points, i.e., if 𝑝 𝑋 = 𝑥𝑖 = 𝑝𝑖 for all 𝑖 =
1, . . , (𝑘 − 1), then we have
𝑘−1 𝑘−1

𝑓 ෍ 𝑝𝑖 𝑥𝑖 ≤ ෍ 𝑝𝑖 𝑓 𝑥𝑖
𝑖=1 𝑖=1
 Jensen’s inequality
𝑘−1 𝑘−1

𝑓 ෍ 𝑝𝑖 𝑥𝑖 ≤ ෍ 𝑝𝑖 𝑓 𝑥𝑖 (1)
𝑖=1 𝑖=1
o For distribution with k mass points and probabilities 𝑞1 , 𝑞2 , . . 𝑞𝑘
𝑘 𝑘−1

෍ 𝑞𝑖 𝑓 𝑥𝑖 = 𝑞𝑘 𝑓 𝑥𝑘 + ෍ 𝑞𝑖 𝑓(𝑥𝑖 )
𝑖=1 𝑖=1
𝑞𝑖 𝑞𝑖
o Lets define 𝑝𝑖 = σ𝑘−1 = . Replace 𝑞𝑖 with 𝑝𝑖 for 𝑖 = 1, . . (𝑘 − 1) in the RHS to get
𝑗=1 𝑞𝑗 (1−𝑞𝑘 )
𝑘 𝑘−1

෍ 𝑞𝑖 𝑓 𝑥𝑖 = 𝑞𝑘 𝑓 𝑥𝑘 + (1 − 𝑞𝑘 ) ෍ 𝑝𝑖 𝑓(𝑥𝑖 )
𝑖=1 𝑖=1
𝑘−1
By using Eq (1)
≥ 𝑞𝑘 𝑓 𝑥𝑘 + (1 − 𝑞𝑘 )𝑓 ෍ 𝑝𝑖 𝑥𝑖
𝑖=1
𝑘−1 𝑘

≥ 𝑓 𝑞𝑘 𝑥𝑘 + 1 − 𝑞𝑘 ෍ 𝑝𝑖 𝑥𝑖 = 𝑓 ෍ 𝑞𝑖 𝑥𝑖 Convexity definition
𝑖=1 𝑖=1
𝑘 𝑘

⇒ ෍ 𝑞𝑖 𝑓 𝑥𝑖 ≥ 𝑓 ෍ 𝑞𝑖 𝑥𝑖 ⇒𝑓 𝔼𝑋 ≤ 𝔼[𝑓(𝑋)]
𝑖=1 𝑖=1
Information inequality
• Relative entropy

o For two PMFs 𝑝 𝑥 and 𝑞 𝑥 , it is defined as

𝐷(𝑝| 𝑞 ≥ 0
o Equality holds if and only if 𝑝 𝑥 = 𝑞(𝑥) for all 𝑥.

Proof: Let 𝐴 = {𝑥: 𝑝 𝑥 > 0} be the support of 𝑝(𝑥), then

𝑝 𝑥 𝑞 𝑥 𝑞 𝑥 𝑞 𝑥
−𝐷(𝑝| 𝑞 = − ෍ 𝑝 𝑥 log = ෍ 𝑝 𝑥 log = 𝔼𝑝 log ≤ log 𝔼
𝑞 𝑥 𝑝 𝑥 𝑝 𝑥 𝑝 𝑥
𝑥∈A 𝑥∈A
𝑞 𝑥
= log ෍ 𝑝 𝑥 = log ෍ 𝑞 𝑥 =0
𝑝 𝑥
𝑥∈A 𝑥∈A
⇒ 𝐷(𝑝| 𝑞 ≥ 0
𝑞 𝑥
o Equality holds when 𝑝 is constant ⇒ 𝑞 𝑥 = 𝑝(𝑥)
𝑥
More inequalities
o Non-negativity of mutual information
o For any two random variables 𝑋 and 𝑌,
𝐼(𝑋; 𝑌) ≥ 0
o Equality holds if and only if 𝑋 and 𝑌are independent
Proof:
o Recall 𝐼 𝑋; 𝑌 = 𝐷(𝑝(𝑥, 𝑦)||𝑝 𝑥 𝑝(𝑦))
o As proved in the previous slide 𝐷(𝑝(𝑥, 𝑦)| 𝑝 𝑥 𝑝 𝑦 ≥ 0
o Equality holds if and only if 𝑝 𝑥, 𝑦 = 𝑝 𝑥 𝑝 𝑦 ⇒ 𝑋 and 𝑌 are independent

o Conditioning reduces entropy:

𝐻 𝑋𝑌 ≤𝐻 𝑋
Proof:
𝐼 𝑋; 𝑌 ≥ 0
𝐻 𝑋 −𝐻 𝑋 𝑌 ≥0
⇒ 𝐻 𝑋 ≥ 𝐻(𝑋|𝑌)
o Equality holds if and only if 𝑋 and 𝑌 are independent
More inequalities
o Uniform distribution has maximum entropy:
o Let 𝑋 be a random variable taking values from 𝒳, and 𝒳 denotes the size of 𝒳. Then we have
𝐻 𝑋 ≤ log |𝒳|
1
o Equality holds if and only if 𝑋 is uniformly distributed over 𝒳, i.e., 𝑝 𝑥 = 𝒳 for all 𝑥 ∈ 𝒳.
Proof:
1
o Let 𝑞 𝑥 = be a uniform pmf over 𝑋. Let 𝑝(𝑥) be some other pmf over 𝑋.
𝒳
o Relative entropy
𝑝 𝑥
𝐷(𝑝| 𝑢 = ෍ 𝑝 𝑥 log
𝑞 𝑥
𝑥∈𝒳

= ෍ 𝑝 𝑥 log 𝑝(𝑥) − ෍ 𝑝 𝑥 log 𝑞(𝑥)

𝑥∈𝒳 𝑥∈𝒳
1
= −𝐻 𝑋 − ෍ 𝑝 𝑥 log
𝒳
𝑥∈𝒳

= −𝐻 𝑋 + log 𝒳 ෍ 𝑝 𝑥
𝑥∈𝒳
𝐷(𝑝| 𝑢 = −𝐻 𝑋 + log 𝒳 ≥ 0
⇒ 𝐻 𝑋 ≤ log |𝒳|
 Lecture 3

MIMO Wireless Communication

(ELL8415)
Anupama Rajoriya
Basics of Information Theory

Reference:
Elements of Information Theory
Thomas M. Cover, Joy A. Thomas
Recap
o Some basic definitions for discrete random variables:
▪ Entropy,
▪ Joint and conditional entropies,
▪ Relative entropy,
▪ Mutual information

In today’s class
o For continuous random variables:
▪ Differential entropy
▪ Joint and conditional entropies,
▪ Relative entropy
▪ Mutual information
o Entropy of Gaussian random variables and random vectors
More on Relative entropy
o Kullback-Leibler divergence (KL divergence)
o Not a mathematical distance metric (no symmetry, does not satisfy triangular inequality)
o Signifies the “information loss”
o if true distribution 𝑝 𝑥 is known: average description length 𝐻(𝑝)
o if a distribution 𝑞 𝑥 is used, we would need H p + 𝐷(𝑝||𝑞) bits on the average
o Simple difference between distributions will not have similar meaning

o Example: Lets assume the case where 𝑞 𝑋 = 𝑥 = 0 when the true distribution 𝑝 𝑋 = 𝑥 > 0
Relative entropy/KL divergence becomes ∞→ signifying complete failure of the assumption
o Simple difference will give just a number, which does not properly represent the catastrophic failure

o Example (how subtraction fails):

Let a random variable 𝑋 ∈ {0,1,2}, ⇒ 𝑝 𝑥 − 𝑞 𝑥 = 0 but they are not same!
o KL divergence is zero if and only if 𝑝 𝑥 = 𝑞 𝑥 .
3
3
4

𝑝(𝑋 = 𝑥)
4

𝑞(𝑋 = 𝑥)
1 1
1 2 1 2
4 4

0 1 2 𝑥 0 1 2 𝑥
Differential entropy
o Let 𝑋 be a continuous random variable with cumulative distribution function (CDF)
𝐹 𝑥 = Pr(𝑋 ≤ 𝑥)
o Its probability distribution function (pdf):
𝑓 𝑥 = 𝐹′(𝑥)
o Set where 𝑓 𝑥 > 0 is called the support set of 𝑋

o Differential Entropy: for a random variable 𝑋 with a pdf 𝑓 𝑥 , it is defined as

ℎ 𝑋 = − න 𝑓 𝑥 log 𝑓 𝑥 𝑑𝑥 ,
𝑆
o 𝑆: support set of 𝑋
o Depends only on the pdf 𝑓(𝑥), not on 𝑋

Example
When a<1, ℎ 𝑋 < 0
o Differential entropy of a uniformly distributed random variable 𝑋 ∈ [0, 𝑎]
1
⇒ Differential entropy can be negative
if 0 ≤ 𝑥 ≤ 𝑎,
o Pdf: 𝑓 𝑥 = ൝𝑎
0 otherwise.
𝑎
𝑎 1 1
ℎ 𝑋 = −∫0 𝑓 𝑥 log 𝑓 𝑥 𝑑𝑥 = − න log 𝑑𝑥 = log 𝑎
0 𝑎 𝑎
Joint and conditional differential entropy
o For pair of random variables 𝑋 and 𝑌 with joint pdf 𝑓(𝑥, 𝑦), joint differential entropy is defined as
ℎ 𝑋, 𝑌 = − න න 𝑓 𝑥, 𝑦 log 𝑓 𝑥, 𝑦 𝑑𝑦 𝑑𝑥
𝑆𝑋 𝑆𝑌
o 𝑆𝑋 : support set of 𝑋 and 𝑆𝑌 : support set of 𝑌
o The conditional differential entropy is defined as
ℎ 𝑋 𝑌 = − න න 𝑓 𝑥, 𝑦 log 𝑓 𝑥 𝑦 𝑑𝑥 𝑑𝑦
𝑆𝑋 𝑆𝑌
𝑓(𝑥, 𝑦)
= − න න 𝑓 𝑥, 𝑦 log 𝑑𝑥 𝑑𝑦
𝑆𝑋 𝑆𝑌 𝑓 𝑦
= − න න 𝑓 𝑥, 𝑦 log 𝑓(𝑥, 𝑦) 𝑑𝑥 𝑑𝑦 + න න 𝑓 𝑥, 𝑦 log 𝑓 𝑦 𝑑𝑥 𝑑𝑦
𝑆𝑋 𝑆𝑌 𝑆𝑋 𝑆𝑌

= ℎ 𝑋, 𝑌 + න 𝑓 𝑦 log 𝑓 𝑦 𝑑𝑦
𝑆𝑌
= ℎ 𝑋, 𝑌 − ℎ(𝑌)
Relative entropy and Mutual information
o Relative Entropy: KL-divergence between two densities 𝑓 𝑥 and 𝑔(𝑥) on a random variable 𝑋
𝑓 𝑥
𝐷(𝑓| 𝑔 = න 𝑓 𝑥 log 𝑑𝑥
𝑆 𝑔 𝑥
o Mutual Information:
𝐼 𝑋; 𝑌 = 𝐷(𝑓(𝑥, 𝑦)||𝑓 𝑥 𝑓(𝑦))
𝑓(𝑥, 𝑦)
= න න 𝑓 𝑥, 𝑦 log 𝑑𝑥 𝑑𝑦
𝑆 𝑋 𝑆𝑌 𝑓(𝑥)𝑓 𝑦
𝑓 𝑥 𝑓(𝑦|𝑥)
= න න 𝑓 𝑥, 𝑦 log 𝑑𝑥 𝑑𝑦
𝑆 𝑋 𝑆𝑌 𝑓(𝑥)𝑓 𝑦
= − න න 𝑓 𝑥, 𝑦 log 𝑓(𝑦) 𝑑𝑥 𝑑𝑦 + න න 𝑓 𝑥, 𝑦 log 𝑓 𝑦|𝑥 𝑑𝑥 𝑑𝑦
𝑆𝑋 𝑆𝑌 𝑆𝑋 𝑆𝑌

= − න 𝑓 𝑦 log 𝑓 𝑦 𝑑𝑦 − ℎ(𝑌|𝑋)
𝑆𝑌
= ℎ 𝑌 − ℎ(𝑌|𝑋)
= ℎ 𝑋 − ℎ(𝑋|𝑌)
= ℎ 𝑋 + ℎ(𝑌) − ℎ(𝑋, 𝑌) As ℎ 𝑌 𝑋 = ℎ 𝑋, 𝑌 − ℎ(𝑌)
Properties
o Relative Entropy: 𝐷(𝑓| 𝑔 ≥ 0
o Proof:
𝑓 𝑥 𝑔 𝑥
−𝐷(𝑓| 𝑔 = − න 𝑓 𝑥 log 𝑑𝑥 = න 𝑓 𝑥 log 𝑑𝑥
𝑆 𝑔 𝑥 𝑆 𝑓 𝑥
𝑔 𝑥 𝑔 𝑥
= 𝔼 log ≤ log 𝔼
𝑓 𝑥 𝑓 𝑥
𝑔 𝑥
= log න 𝑓 𝑥 𝑑𝑥 = log 1 = 0
𝑆 𝑓 𝑥
⇒ 𝐷(𝑓| 𝑔 ≥ 0
o Mutual Information: 𝐼 𝑋; 𝑌 ≥ 0. Equality holds iff X and Y are independent
o Conditioning reduces entropy: ℎ 𝑋 𝑌 ≤ ℎ(𝑋). Equality holds iff X and Y are independent
For Gaussian random variables
Differential entropy of a Gaussian random variable:
o Let 𝑋 be a zero mean Gaussian random variable with variance 𝜎 2 : 𝑋 ∼ 𝑁(0, 𝜎 2 )
o Its pdf
1 𝑥2
− 2
𝑓 𝑥 = e 2𝜎
2𝜋𝜎 2
o Differential entropy:
∞
ℎ 𝑋 = − න 𝑓 𝑥 log 𝑓 𝑥 𝑑𝑥
−∞
∞
1 𝑥2
−
= − න 𝑓 𝑥 log e 2𝜎2 𝑑𝑥
−∞ 2𝜋𝜎 2
∞ 2∞
𝑥
=න 𝑓 𝑥 log 2𝜋𝜎 2 𝑑𝑥 + න 2 𝑓 𝑥 log 𝑒 𝑑𝑥
−∞ −∞ 2𝜎
∞
2
log 𝑒 ∞ 2
= log 2𝜋𝜎 න 𝑓 𝑥 𝑑𝑥 + 2 න 𝑥 𝑓 𝑥 𝑑𝑥
−∞ 2𝜎 −∞
1 log 𝑒
= log 2𝜋𝜎 2 × 1 + 2 𝔼 𝑋2
2 2𝜎
1 log 𝑒
= log 2𝜋𝜎 2 × 1 + 2 × 𝜎2
2 2𝜎
1 1 1
= log 2𝜋𝜎 + log 𝑒 = log 2𝜋𝑒𝜎 2
2
2 2 2
For Gaussian random variables
Differential entropy of a Gaussian random vector:
o Let 𝑿 = [𝑋1 , 𝑋2 , … , 𝑋𝑛 ] ∈ ℝ𝑛×1be a random vector with pdf
1 1
−2 𝒙−𝝁 𝑇 𝚺 −1 (𝒙−𝝁)
𝑓 𝒙 = 1e
𝑛
2𝜋 𝚺 2
o Here 𝝁 ∈ ℝ𝑛×1 : mean of the vector 𝑿 and 𝚺 ∈ ℝ𝑛×𝑛 : covariance, defined as 𝚺 = 𝔼 (𝒙 − 𝝁) 𝒙 − 𝝁 𝑇

o Joint differential entropy:

∞
ℎ 𝑿 = − න 𝑓 𝒙 log 𝑓(𝒙) 𝑑𝒙
−∞
∞
1 1
−2 𝒙−𝝁 𝑇 𝚺 −1 (𝒙−𝝁)
= − න 𝑓 𝒙 log 1 e 𝑑𝒙
𝑛
−∞ 2𝜋 𝚺 2

∞ ∞
𝑛 1 1
=න 𝑓 𝒙 log 2𝜋 𝚺 2 𝒙 − 𝝁 𝑇 𝚺 −1 (𝒙 − 𝝁)𝑓 𝒙 log 𝑒 𝑑𝒙
𝑑𝒙 + න
−∞ −∞ 2
∞
𝑛 1 log 𝑒 ∞
= log 2𝜋 𝚺 2 න 𝑓 𝒙 𝑑𝒙 + න 𝒙 − 𝝁 𝑇 𝚺 −1 (𝒙 − 𝝁)𝑓 𝒙 𝑑𝒙
−∞ 2 −∞
1 𝑛
log 𝑒
= log 2𝜋 𝚺 × 1 + 𝔼 𝒙 − 𝝁 𝑇 𝚺 −1 (𝒙 − 𝝁)
2 2
For Gaussian random variables
Differential entropy of a Gaussian random vector:

o Lets simplify the term 𝔼 𝒙 − 𝝁 𝑇 𝚺 −1 (𝒙 − 𝝁)

𝔼 𝒙 − 𝝁 𝑇 𝚺 −1 (𝒙 − 𝝁) = Tr 𝔼 𝒙 − 𝝁 𝑇 𝚺 −1 (𝒙 − 𝝁) Trace of scalar is same as the scalar
= 𝔼 Tr 𝒙 − 𝝁 𝑇 𝚺 −1 (𝒙 − 𝝁) Both are linear operators
= 𝔼 Tr 𝚺 −1 (𝒙 − 𝝁) 𝒙 − 𝝁 𝑇 Tr 𝑨𝑩 = Tr(𝑩𝑨)
= Tr 𝔼 𝚺 −1 (𝒙 − 𝝁) 𝒙 − 𝝁 𝑇
= Tr 𝚺 −1 𝔼 (𝒙 − 𝝁) 𝒙 − 𝝁 𝑇
= Tr 𝚺 −1 𝚺 = Tr 𝑰𝑛×𝑛 = 𝑛
o Joint differential entropy:
1 𝑛
log 𝑒
ℎ 𝑿 = log 2𝜋 𝚺 × 1 + 𝔼 𝒙 − 𝝁 𝑇 𝚺 −1 (𝒙 − 𝝁)
2 2
1 𝑛
log 𝑒
= log 2𝜋 𝚺 × 1 + ×𝑛
2 2
1
= log 2𝜋𝑒 𝑛 𝚺
2
For Gaussian random variables
Mutual information between correlated Gaussian random variables:
o Lets 𝑋 and 𝑌 be zero mean Gaussian random variables with covariance matrix
𝜎 2 𝜌𝜎 2
𝚺= , −1 < 𝜌 < 1
𝜌𝜎 2 𝜎 2
o Marginal distributions of 𝑋 and 𝑌 are: 𝑓 𝑥 = 𝑁 0, 𝜎 2 , 𝑓 𝑦 = 𝑁(0, 𝜎 2 )
o Mutual information:
𝐼 𝑋; 𝑌 = ℎ 𝑋 + ℎ 𝑌 − ℎ(𝑋, 𝑌)
o Differential entropies of 𝑋 and 𝑌:
1 1
ℎ 𝑋 = 2 log 2𝜋𝑒𝜎 2 and ℎ 𝑌 = 2 log 2𝜋𝑒𝜎 2
o Joint entropy
1 1
ℎ 𝑋, 𝑌 = log 2𝜋e 2 |𝚺| = log 2𝜋e 2 𝜎 4 (1 − 𝜌2 )
2 2
o Mutual information:
𝐼 𝑋; 𝑌 = ℎ 𝑋 + ℎ 𝑌 − ℎ 𝑋, 𝑌
1 1 1
= log 2𝜋𝑒𝜎 2 + log 2𝜋𝑒𝜎 2 − log 2𝜋e 2 𝜎 4 1 − 𝜌2
2 2 2
2 2
1 2𝜋𝑒𝜎 1
= log 2 4 2 = − log(1 − 𝜌2 )
2 2𝜋e 𝜎 1 − 𝜌 2
For Gaussian random variables
Mutual information between correlated Gaussian random variables:
o Lets 𝑋 and 𝑌 be zero mean Gaussian random variables with covariance matrix
𝜎 2 𝜌𝜎 2
𝚺= , −1 < 𝜌 < 1
𝜌𝜎 2 𝜎 2
o Marginal distributions of 𝑋 and 𝑌 are: 𝑓 𝑥 = 𝑁 0, 𝜎 2 , 𝑓 𝑦 = 𝑁(0, 𝜎 2 )
o Mutual information:
𝐼 𝑋; 𝑌 = ℎ 𝑋 + ℎ 𝑌 − ℎ(𝑋, 𝑌)
o Differential entropies of 𝑋 and 𝑌:
1 1
ℎ 𝑋 = 2 log 2𝜋𝑒𝜎 2 and ℎ 𝑌 = 2 log 2𝜋𝑒𝜎 2
o Joint entropy
1 1
ℎ 𝑋, 𝑌 = log 2𝜋e 2 |𝚺| = log 2𝜋e 2 𝜎 4 (1 − 𝜌2 )o When 𝜌 = 0:
2 2
o Mutual information: 𝑋 and 𝑌 are uncorrelated
𝐼 𝑋; 𝑌 = ℎ 𝑋 + ℎ 𝑌 − ℎ 𝑋, 𝑌 ⇒ 𝐼 𝑋; 𝑌 = 0
1 1 1 o When 𝜌 = ±1:
= log 2𝜋𝑒𝜎 2 + log 2𝜋𝑒𝜎 2 − log 2𝜋e 2 𝜎 4 1 − 𝜌2
2 2 2 𝑋 and 𝑌 are perfectly correlated
2 2
1 2𝜋𝑒𝜎 1 ⇒ 𝐼 𝑋; 𝑌 = ∞
= log 2 4 2 = − log(1 − 𝜌2 )
2 2𝜋e 𝜎 1 − 𝜌 2
Summary
o Differential entropy of Gaussian random variable 𝑋 ∼ 𝑁(0, 𝜎 2 )
1
ℎ 𝑋 = log 2𝜋𝑒𝜎 2
2
o Differential entropy of an 𝑛-dimensional Gaussian random vector 𝑿 ∼ 𝑁(𝝁, 𝚺)
1
ℎ 𝑿 = log 2𝜋𝑒 𝑛 𝚺
2
o Note that entropy is independent of the mean.
o Translation does not change entropy! Exercise
⇒ ℎ 𝑋 + 𝑐 = ℎ(𝑋)
Gaussian random vector has maximum entropy
Let the random vector 𝑿 ∈ ℝ𝑛×1 has zero mean and covariance matrix 𝚺 = 𝔼 𝑿𝑿𝑇 ∈ ℝ𝑛×𝑛 . Then
1
• ℎ 𝑿 ≤ 2 log 2𝜋𝑒 𝑛 |𝚺|
• Equality holds if and only if 𝑿 ∼ 𝑁(𝟎, 𝚺)

Proof:
o Let 𝑔(𝒙) be any pdf on 𝑿 such that 𝔼𝑔 𝑿 = 𝟎 and 𝔼𝑔 𝑿𝑿𝑇 = 𝚺
o Let 𝑓 𝒙 be the pdf when 𝑿 ∼ 𝑁(𝟎, 𝚺), i.e., 𝑓 𝒙 = 𝑁(𝟎, 𝚺) or
1 1
−2𝒙𝑇 𝚺 −1 𝒙
𝑓 𝒙 = 1e
𝑛
2𝜋 𝚺 2
o Relative entropy:
𝑔 𝒙
𝐷(𝑔| 𝑓 = න 𝑔 𝒙 log 𝑑𝒙
𝑓 𝒙
= න 𝑔 𝒙 log 𝑔 𝒙 𝑑𝒙 − න𝑔 𝒙 log 𝑓 𝒙 𝑑𝒙
1 1
−2𝒙𝑇 𝚺 −1 𝒙
= −ℎ 𝑿 − න 𝑔 𝒙 log 1e 𝑑𝒙
𝑛
2𝜋 𝚺 2
Gaussian random vector has maximum entropy
Let the random vector 𝑿 ∈ ℝ𝑛×1 has zero mean and covariance matrix 𝚺 = 𝔼 𝑿𝑿𝑇 ∈ ℝ𝑛×𝑛 . Then
1
• ℎ 𝑿 ≤ 2 log 2𝜋𝑒 𝑛 |𝚺|
• Equality holds if and only if 𝑿 ∼ 𝑁(𝟎, 𝚺)

𝑛 1 1 𝑇 −1
𝐷(𝑔| 𝑓 = −ℎ 𝑿 + log 2𝜋 𝚺 2 න𝑔 𝒙 𝑑𝒙 + න 𝒙 𝚺 𝒙 𝑔 𝒙 log 𝑒 𝑑𝒙
2
𝑛 1 1
= −ℎ 𝑿 + log 2𝜋 𝚺 2 × 1 + log 𝑒 𝔼𝑔 𝒙𝑇 𝚺 −1 𝒙
2
1 1
= −ℎ 𝑿 + log 2𝜋 𝑛 𝚺 + log 𝑒 Tr 𝔼𝑔 𝒙𝑇 𝚺 −1 𝒙
2 2
1 1
= −ℎ 𝑿 + log 2𝜋 𝑛 𝚺 + log 𝑒 𝔼𝑔 Tr 𝒙𝑇 𝚺 −1 𝒙
2 2
1 1
= −ℎ 𝑿 + log 2𝜋 𝑛 𝚺 + log 𝑒 Tr 𝚺 −1 𝔼𝑔 𝒙𝒙𝑇
2 2
1 𝑛
= −ℎ 𝑿 + log 2𝜋 𝑛 𝚺 + log 𝑒
2 2
1
= −ℎ 𝑿 + log 2𝜋𝑒 𝑛 𝚺 ≥0 Equality holds when 𝑓 𝒙 = 𝑔(𝒙)
2
1 ⇒ 𝑔 𝒙 = 𝑁(𝟎, 𝚺)
⇒ ℎ 𝑿 ≤ log 2𝜋𝑒 𝑛 𝚺
2
Motivation for MIMO communication
Multiple transmit and/or receive antennas
o Diversity gain: Transmit same signal over multiple channels→ boosts the SNR
A signal transmitted over two independently fading channels is more likely to be correctly detected than the
same signal transmitted over a single fading channel
o Spatial multiplexing: Provides transmission of multiple parallel streams
o Beamforming- serving users in a particular direction→ Interference reduction

ℎ11
ℎ12
ℎ21 𝑦1
𝑥1
ℎ22
Transmitter Receiver
𝑥2 𝑦2
 Lecture 4

MIMO Wireless Communication

(ELL8415)
Anupama Rajoriya
Capacity

Reference:
Elements of Information Theory
Thomas M. Cover, Joy A. Thomas
Recap
o For continuous random variables:
▪ Differential entropy
▪ Joint and conditional entropies,
▪ Relative entropy
▪ Mutual information
o Entropy of Gaussian random variables and random vectors

In today’s class
o Definition of capacity
o Examples: Channel capacity
Capacity
Input 𝑋 Output 𝑌
Channel

o Capacity of a channel is the maximum of the mutual information between the input and output over all
distributions on the input that satisfy the power constraint:
𝐶 = max𝑝 𝑥 𝐼 𝑋; 𝑌
o Such that 𝔼 𝑋 2 ≤ 𝑃

o Capacity for Different kinds of channel:

o Noiseless binary channel
o Binary symmetric channel
o Gaussian channel
Examples
Noiseless binary channel
Input Output
Channel
𝑋 ∈ {0,1} 𝑌 ∈ {0,1}

o Noiseless ⇒ 𝑌 = 𝑋
o When 0 is transmitted 0 is received and vice versa
o Mutual information: 𝐼 𝑋; 𝑌 = 𝐻 𝑋 − 𝐻 𝑋 𝑌
o Lets calculate 𝑝 𝑥 𝑦
𝑝 𝑋=1𝑌 =1 =1
𝑝 𝑋=1𝑌 =0 =0
𝑝 𝑋=0𝑌 =0 =1
𝑝 𝑋=0𝑌 =1 =0
1 if 𝑥 = 𝑦,
𝑝 𝑋=𝑥𝑌=𝑦 =ቊ
0 if 𝑥 ≠ 𝑦
o Conditional entropy:
𝐻 𝑋 𝑌 = − ෍ ෍ 𝑝 𝑥 𝑦 log 𝑝 𝑥 𝑦 = −1 × log 1 = 0
𝑥 𝑦
Examples
Noiseless binary channel
Input Output
Channel
𝑋 ∈ {0,1} 𝑌 ∈ {0,1}

o Mutual information:
𝐼 𝑋; 𝑌 = 𝐻 𝑋 − 𝐻 𝑋 𝑌 = 𝐻 𝑋
Examples
Noiseless binary channel
Input Output
Channel
𝑋 ∈ {0,1} 𝑌 ∈ {0,1}

o Mutual information:
𝐼 𝑋; 𝑌 = 𝐻 𝑋 − 𝐻 𝑋 𝑌 = 𝐻 𝑋
Examples
Noiseless binary channel
Input Output
Channel
𝑋 ∈ {0,1} 𝑌 ∈ {0,1}

o Mutual information:
𝐼 𝑋; 𝑌 = 𝐻 𝑋 − 𝐻 𝑋 𝑌 = 𝐻 𝑋

𝐼 𝑋; 𝑌 = −𝑝 log 2 𝑝 − (1 − 𝑝) log 2 (1 − 𝑝)
Examples
Noiseless binary channel
Input Output
Channel
𝑋 ∈ {0,1} 𝑌 ∈ {0,1}

o Mutual information:
𝐼 𝑋; 𝑌 = 𝐻 𝑋 − 𝐻 𝑋 𝑌 = 𝐻 𝑋

𝐼 𝑋; 𝑌 = −𝑝 log 2 𝑝 − (1 − 𝑝) log 2 (1 − 𝑝)
o Capacity
𝐶 = max 𝐼 𝑋; 𝑌 = max 𝐻 𝑋
Examples
Noiseless binary channel
Input Output
Channel
𝑋 ∈ {0,1} 𝑌 ∈ {0,1}

o Mutual information:
𝐼 𝑋; 𝑌 = 𝐻 𝑋 − 𝐻 𝑋 𝑌 = 𝐻 𝑋

𝐼 𝑋; 𝑌 = −𝑝 log 2 𝑝 − (1 − 𝑝) log 2 (1 − 𝑝)
o Capacity
𝐶 = max 𝐼 𝑋; 𝑌 = max 𝐻 𝑋

o We proved that maximum entropy is achieved when X is uniformly distributed

1 1
o Equiprobable points, i.e., 𝑝 = 2 , 1 − 𝑝 = 2
o Capacity, thus, becomes
𝐶 = max 𝐻 𝑋 = 1
Examples
Noiseless binary channel
Input Output
Channel
𝑋 ∈ {0,1} 𝑌 ∈ {0,1}

o Mutual information:
𝐼 𝑋; 𝑌 = 𝐻 𝑋 − 𝐻 𝑋 𝑌 = 𝐻 𝑋

𝐼 𝑋; 𝑌 = −𝑝 log 2 𝑝 − (1 − 𝑝) log 2 (1 − 𝑝)
o Capacity
𝐶 = max 𝐼 𝑋; 𝑌 = max 𝐻 𝑋

o We proved that maximum entropy is achieved when X is uniformly distributed

1 1
o Equiprobable points, i.e., 𝑝 = 2 , 1 − 𝑝 = 2
o Capacity, thus, becomes Capacity of a noiseless binary channel:
𝐶 = max 𝐻 𝑋 = 1 C = 1 bit
Examples
Noiseless binary channel
Input Output
Channel
𝑋 ∈ {0,1} 𝑌 ∈ {0,1}

o Mutual information:
𝐼 𝑋; 𝑌 = 𝐻 𝑋 − 𝐻 𝑋 𝑌 = 𝐻 𝑋

𝐼 𝑋; 𝑌 = −𝑝 log 2 𝑝 − (1 − 𝑝) log 2 (1 − 𝑝)
o Capacity
𝐶 = max 𝐼 𝑋; 𝑌 = max 𝐻 𝑋

o We proved that maximum entropy is achieved when X is uniformly distributed

1 1
o Equiprobable points, i.e., 𝑝 = 2 , 1 − 𝑝 = 2
o Capacity, thus, becomes Capacity of a noiseless binary channel:
𝐶 = max 𝐻 𝑋 = 1 C = 1 bit

we can send 1 bit per transmission over this channel with no errors
Examples
Noisy channel: Binary symmetric channel 1−𝑝
𝑥=0 𝑦=0
Input Output
Channel
𝑋 ∈ {0,1} 𝑌 ∈ {0,1}
𝑝
o Mutual information: 𝐼 𝑋; 𝑌 = 𝐻 𝑌 − 𝐻 𝑌 𝑋
o Lets calculate 𝐻 𝑌 𝑋 = 𝑥 first
𝑝
𝐻 𝑌 𝑋 = 𝑥 = − ෍ 𝑝 𝑦 𝑋 = 𝑥 log 𝑝 𝑦 𝑋 = 𝑥 1−𝑝
𝑦 𝑥=1 𝑦=1

𝐻 𝑌 𝑋 = 0 = − ෍ 𝑝 𝑦 𝑋 = 0 log 𝑝(𝑦|𝑋 = 0)
𝑦
= −𝑝 log 𝑝 − 1 − 𝑝 log 1 − 𝑝 = 𝐻(𝑝)
o Similarly: 𝐻 𝑌 𝑋 = 1 = 𝐻(𝑝)
o Conditional entropy: 𝐻 𝑌 𝑋 = σ𝑥 𝑝 𝑥 𝐻 𝑌 𝑋 = 𝑥
o Mutual information:
𝐼 𝑋; 𝑌 = 𝐻 𝑌 − 𝐻 𝑌 𝑋
= 𝐻 𝑌 − ෍𝑝 𝑥 𝐻 𝑌 𝑋 = 𝑥
𝑥
=𝐻 𝑌 −𝐻 𝑝
≤ 1 − 𝐻(𝑝) As 𝐻 𝑌 = 1 is the maximum value of entropy
Examples
Noisy channel: Binary symmetric channel 1−𝑝
𝑥=0 𝑦=0
Input Output
Channel
𝑋 ∈ {0,1} 𝑌 ∈ {0,1}
𝑝
o Mutual information: 𝐼 𝑋; 𝑌 = 𝐻 𝑌 − 𝐻 𝑌 𝑋
o Lets calculate 𝐻 𝑌 𝑋 = 𝑥 first
𝑝
𝐻 𝑌 𝑋 = 𝑥 = − ෍ 𝑝 𝑦 𝑋 = 𝑥 log 𝑝 𝑦 𝑋 = 𝑥 1−𝑝
𝑦 𝑥=1 𝑦=1

𝐻 𝑌 𝑋 = 0 = − ෍ 𝑝 𝑦 𝑋 = 0 log 𝑝(𝑦|𝑋 = 0)
𝑦
= −𝑝 log 𝑝 − 1 − 𝑝 log 1 − 𝑝 = 𝐻(𝑝) Capacity of a binary symmetric channel:
o Similarly: 𝐻 𝑌 𝑋 = 1 = 𝐻(𝑝) C = 1 − 𝐻(𝑝)
o Conditional entropy: 𝐻 𝑌 𝑋 = σ𝑥 𝑝 𝑥 𝐻 𝑌 𝑋 = 𝑥
o Mutual information: It is achieved when 𝑋 is uniformly
𝐼 𝑋; 𝑌 = 𝐻 𝑌 − 𝐻 𝑌 𝑋 distributed, i.e., 𝑝 = 1/2
= 𝐻 𝑌 − ෍𝑝 𝑥 𝐻 𝑌 𝑋 = 𝑥
𝑥
=𝐻 𝑌 −𝐻 𝑝
≤ 1 − 𝐻(𝑝) As 𝐻 𝑌 = 1 is the maximum value of entropy
 𝑍: noise
Examples
Noisy Gaussian channel:
Input Output
Channel
𝑋 𝑌
o Input output relation: 𝑋 𝑌
𝑌 =𝑋+𝑍
o The noise follows 𝑍 ∼ 𝑁(0, 𝜎 2 )
o Noise 𝑍 is independent of 𝑋
o Capacity with specified power constraint
max 𝐼(𝑋; 𝑌)
𝑝 𝑥
o Such that 𝔼 𝑋2≤𝑃
o Mutual information calculation
𝐼 𝑋; 𝑌 =ℎ 𝑌 −ℎ 𝑌 𝑋
=ℎ 𝑌 −ℎ 𝑋+𝑍 𝑋
=ℎ 𝑌 − ℎ(𝑍|𝑋) Using the result ℎ 𝑋 + 𝑍 𝑋 = ℎ(𝑍|𝑋)
=ℎ 𝑌 − ℎ(𝑍) Since 𝑍 is independent of 𝑋
o Since 𝑍 ∼ 𝑁(0, 𝜎 2 ), its entropy becomes:
1
ℎ 𝑍 = log 2𝜋𝑒𝜎 2
2
 𝑍: noise
Examples
Noisy Gaussian channel:
Input Output
Channel
𝑋 𝑌
1 𝑋 𝑌
o Also ℎ 𝑌 ≤ 2 log 2𝜋𝑒𝔼[𝑌 2 ]: Gaussian distribution has maximum enetropy
o Lets calculate 𝔼[𝑌 2 ]
𝔼 𝑌2 = 𝔼 𝑋 + 𝑍 2
= 𝔼 𝑋 2 + 𝑍 2 + 2𝑋𝑍
= 𝑃 + 𝜎2 + 0
1 1
o We now have ℎ 𝑌 ≤ 2 log 2𝜋𝑒(𝑃 + 𝜎 2 ) and ℎ 𝑍 = 2 log 2𝜋𝑒𝜎 2
o This gives us
𝐼 𝑋; 𝑌 = ℎ 𝑌 − ℎ(𝑍)
1 1
≤ log 2𝜋𝑒 𝑃 + 𝜎 − log 2𝜋𝑒𝜎 2
2
2 2
1 𝑃
= log 1 + 2
2 𝜎
o Equality holds/maximum is achieved when 𝑌 is Gaussian
o As 𝑍 is Gaussian, if 𝑋 ∼ 𝑁(0, 𝑃), 𝑌 becomes Gaussian
 𝑍: noise
Examples
Noisy Gaussian channel:
Input Output
Channel
𝑋 𝑌
1 𝑋 𝑌
o Also ℎ 𝑌 ≤ 2 log 2𝜋𝑒𝔼[𝑌 2 ]: Gaussian distribution has maximum enetropy
o Lets calculate 𝔼[𝑌 2 ]
𝔼 𝑌2 = 𝔼 𝑋 + 𝑍 2
= 𝔼 𝑋 2 + 𝑍 2 + 2𝑋𝑍
= 𝑃 + 𝜎2 + 0
1 1
o We now have ℎ 𝑌 ≤ 2 log 2𝜋𝑒(𝑃 + 𝜎 2 ) and ℎ 𝑍 = 2 log 2𝜋𝑒𝜎 2
o This gives us
𝐼 𝑋; 𝑌 = ℎ 𝑌 − ℎ(𝑍)
1 1
≤ log 2𝜋𝑒 𝑃 + 𝜎 − log 2𝜋𝑒𝜎 2
2
2 2
1 𝑃 Capacity of an AWGN channel:
= log 1 + 2 1 𝑃
2 𝜎 C = log 1 + 2
o Equality holds/maximum is achieved when 𝑌 is Gaussian 2 𝜎
o As 𝑍 is Gaussian, if 𝑋 ∼ 𝑁(0, 𝑃), 𝑌 becomes Gaussian
It is achieved when 𝑋 is Gaussian
Channel capacity

Input 𝑋 Output 𝑌
Channel

o Objective of data transmission: receiver should be able to decode the transmitted data with small
probability of error
o In practice, we cannot always identify a subset of the inputs to send information without error
o The idea is to send a sequence of data over multiple such channels (use codes with long block length)
o This allows us to send information at a rate C bits per transmission with an arbitrarily low probability of
error→ channel capacity theorem
Some definitions and notations
o Discrete channel: consists of input alphabet 𝒳, output alphabet 𝒴 and pmf 𝑝(𝑦|𝑥)
o Example: binary symmetric channel

o We will denote this channel as (𝒳, 𝑝 𝑦 𝑥 , 𝒴)

o Note that 𝑝(𝑦|𝑥) tells about observing the output 𝑦 given that we sent the symbol 𝑥
o If we use the discrete channel 𝑛 times, we denote it as (𝒳 𝑛 , 𝑝 𝑦 𝑛 𝑥 𝑛 , 𝒴 𝑛 )
o Discrete memoryless channel: Output depends only on input at a particular time
o and it is conditionally independent of previous channel inputs or outputs
𝑝 𝑦𝑘 𝑥 𝑘 , 𝑦 𝑘−1 = 𝑝 𝑦𝑘 𝑥𝑘
o For 𝑘 = 1, 2, … , 𝑛
o Here 𝑥 𝑘 denotes {𝑥1 , 𝑥2 , … 𝑥𝑘 } and 𝑥𝑘 denotes input at time 𝑘
System model for discrete memoryless channel

o An 𝑀, 𝑛 code for the channel 𝒳, 𝑝 𝑦 𝑥 , 𝒴 consists of

▪ An index set 1, 2,· · · , 𝑀 - finite alphabet
▪ An encoding function 𝑋 𝑛 : 1, 2,· · · , 𝑀 → 𝒳 𝑛 , yielding codewords 𝑋 𝑛 1 , 𝑋 𝑛 2 , … , 𝑋 𝑛 𝑀 .
▪ The set of codewords is called the codebook.
▪ Decoder 𝑔: 𝒴 𝑛 → {1,2, … , 𝑀} is a deterministic rule that assigns a guess to each received vector.
o Two different sequences may give rise to same output sequence—confusable inputs
o Can we choose a “non-confusable” subset of inputs, so that we can reconstruct the input sequences from output
with negligible probability of error?
System model for discrete memoryless channel

o An 𝑀, 𝑛 code for the channel 𝒳, 𝑝 𝑦 𝑥 , 𝒴

o Conditional probability of error is given as
𝜆𝑖 = Pr(𝑔 𝑌 𝑛 ≠ 𝑖|𝑋 𝑛 = 𝑋 𝑛 (𝑖))
o Maximal probability of error is defined as
𝜆(𝑛) = max 𝜆𝑖
𝑖∈ 1,2,…,𝑀
log 𝑀
o Rate of an 𝑀, 𝑛 code is: 𝑅 = 𝑛 bits per transmission
o Rate 𝑅 is achievable if there exist a code (𝑀, 𝑛) such that 𝜆(𝑛) tends to 0 as 𝑛 → ∞, where 𝑀 = 2𝑛𝑅
Channel Coding Theorem
For a discrete memoryless channel, all rates below capacity 𝐶 are achievable. Specifically, for
every rate 𝑅 < 𝐶, there exists a sequence of 𝑀, 𝑛 codes with maximum probability of error 𝜆(𝑛) → 0.

Conversely, any sequence of 𝑀, 𝑛 codes with 𝜆(𝑛) → 0 must have 𝑅 < 𝐶.

Theorem 8.7.1 in
“Elements of Information Theory”,
Thoman M cover, Joy A Thomas

Rate R is said to be achievable for a Gaussian channel with a power constraint P if there exists a
sequence of 𝑀, 𝑛 codes with codewords satisfying the power constraint such that the maximal
probability of error 𝜆(𝑛) → 0.
Gaussian channel
o We derived the capacity of a single random variable→ considered single transmission

Capacity of an AWGN channel:

1 𝑃
C = 2 log 1 + 𝜎2 bits per transmission

o For band-limited channel with bandwidth 𝑊 the capacity becomes

𝑊 𝑃
C= log 1 + bits /sec
2 𝑊𝜎 2

o Channel capacity with unity bandwidth 𝑊:

1 𝑃
C = log 1 + 2 Bits/sec/Hz
2 𝜎
o Capacity per complex dimensions

𝑃
C = log 1 + 2 Bits/sec/Hz
𝜎
 Lecture 5

MIMO Wireless Communication

(ELL8415)
Anupama Rajoriya
Capacity of Wireless Channels

Reference:
Chapter 5
Fundamentals of Wireless communication
David Tse, Pramod Vishwanath
Recap
o Definition of capacity
o Examples: Channel capacity

In today’s class
o Capacity of AWGN channel
• SISO system
• SIMO system
• MISO system
Gaussian channel
o Capacity per complex dimensions
C = log 1 + 𝑆𝑁𝑅 Bits/sec/Hz

o Maximum achievable spectral efficiency through AWGN channel as a function of SNR

o SNR = 𝑃/𝑊𝜎 2 , where 𝑊 is the channel bandwidth
o Given the basic resources (Power and bandwidth) how the communication system will perform→
performance benchmark
Linear time-invariant (LTI) Gaussian channels
o Channel is not changing with time and is known at transmitter and receiver
o A SISO LTI channel model looks like:
𝑦 = ℎ𝑥 + 𝑛
o Complex channel ℎ is constant and 𝑛 ∼ 𝐶𝑁(0, 𝜎 2 )
o Power constraint 𝔼 𝑥𝑥 𝐻 ≤ 𝑃

o Capacity
ℎ 2𝑃
𝐶 = log 2 1+ 2
𝜎
o Will talk about fading channels subsequently
 SIMO Gaussian channel ℎ1
𝑦1
o Consider single-antenna transmitter and 𝐿 −antenna receiver .
o Received signal at the 𝑙-th antenna : Transmitter .
ℎ𝐿 . Receiver
𝑦𝑙 = ℎ𝑙 𝑥 + 𝑛𝑙 𝑥
o For all 𝑙 = 1, 2, … , 𝐿
o Here ℎ𝑙 is constant channel gain from transmitter to 𝑙-th antennas at the receiver 𝑦𝐿
o 𝑛𝑙 are independently random variables with distribution CN(0, 𝜎 2 )
o Vector representation of the channel model:

𝒚 = 𝒉𝑥 + 𝒏

o Here 𝒚 = 𝑦1 , 𝑦2 , … , 𝑦𝐿 𝑇 , 𝒉 = ℎ1 , ℎ2 , … , ℎ𝐿 𝑇 and 𝒏 = 𝑛1 , 𝑛2 , … , 𝑛𝐿 𝑇

o Also, since 𝑛𝑙 ∼ 𝐶𝑁(0, 𝜎 2 ),

𝔼 𝒏𝒏𝐻 = 𝜎 2 𝑰𝐿

Optimally combine the 𝐿 signals 𝑦1 , 𝑦2 , … 𝑦𝐿

 SIMO Gaussian channel ℎ1
𝑦1
Optimally combine the 𝐿 signals 𝑦1 , 𝑦2 , … 𝑦𝐿 .
Transmitter .
ℎ𝐿 .
o 𝐿 receive signals 𝑥 Receiver
𝒚 = 𝒉𝑥 + 𝒏
𝑦𝐿
o Let the optimal combiner combines the 𝐿 receive signals as
𝑦෤ = 𝒘𝐻 𝒚 = 𝒘𝐻 𝒉𝑥 + 𝒏
= 𝒘𝐻 𝒉𝑥 + 𝒘 ถ 𝐻𝒏

desired signal noise

Problem: Design 𝒘 such that the SNR is maximized

o This will maximize the rate too

o Lets calculate the SNR
o Signal power:
𝔼 𝒘𝐻 𝒉𝑥 𝒘𝐻 𝒉𝑥 𝐻
= 𝔼 𝒘𝐻 𝒉𝑥𝑥 𝐻 𝒉𝐻 𝒘 = 𝒘𝐻 𝒉 2 𝔼 𝑥𝑥 𝐻 = 𝒘𝐻 𝒉 2 𝑃
o Noise power:
𝔼 𝒘𝐻 𝒏 𝒘𝐻 𝒏 𝐻
= 𝔼 𝒘𝐻 𝒏𝒏𝐻 𝒘 = 𝒘𝐻 𝔼 𝒏𝒏𝐻 𝒘 = 𝜎 2 𝒘 𝟐
 SIMO Gaussian channel ℎ1
Problem: Design 𝒘 such that the SNR is maximized 𝑦1
.
Transmitter .
ℎ𝐿 .
o Signal power: 𝑥
𝐻 𝐻 𝐻 Receiver
𝔼 𝒘 𝒉𝑥 𝒘 𝒉𝑥
= 𝔼 𝒘𝐻 𝒉𝑥𝑥 𝐻 𝒉𝐻 𝒘 = 𝒘𝐻 𝒉 2 𝔼 𝑥𝑥 𝐻 = 𝒘𝐻 𝒉 2 𝑃 𝑦𝐿
o Noise power:
𝔼 𝒘𝐻 𝒏 𝒘𝐻 𝒏 𝐻
= 𝔼 𝒘𝐻 𝒏𝒏𝐻 𝒘 = 𝒘𝐻 𝔼 𝒏𝒏𝐻 𝒘 = 𝜎 2 𝒘 𝟐

o SNR:
𝒘𝐻 𝒉 2 𝑃
SNR = 2
𝜎 𝒘 𝟐
o By applying Cauchy-Schwartz inequality ( 𝒖𝑇 𝒗 ≤ ‖𝒖‖‖𝒗‖) in the numerator, we get
𝒘𝐻 𝒉 2 𝑃 𝒉 𝟐𝑃
SNR = 2 2 ≤
𝜎 𝑤 𝜎2
o Equality is met when 𝒘 = 𝑐𝒉, for some constant 𝑐
o 𝒘 = 𝒉→ This receive beamforming is called maximal ratio combining
 SIMO Gaussian channel ℎ1
𝑦1
.
Transmitter .
ℎ𝐿 .
o Overall system after combining becomes 𝑥 Receiver
෨ + 𝑛෤
𝑦෤ = ℎ𝑥 𝑦𝐿
o Here 𝑦෤ = 𝒉𝐻 𝒚, ℎ෨ = 𝒉 𝟐 and 𝑛෤ = 𝒘𝐻 𝒏
o SNR becomes
𝒉 𝟐𝑃
SNR =
𝜎2
o And the capacity becomes
‖𝒉‖2 𝑃
𝐶 = log 2 1 +
𝜎2
o 𝐿 receive antennas provide beamforming/power gain
 ℎ1
MISO Gaussian channel 𝑥෤1
.
.
. Receiver
Transmitter ℎ𝐿
o Consider a system with 𝐿-antenna transmitter 𝑦
o And single-antenna receiver
o Transmit single symbol 𝑥 over multiple antennas 𝑥෤𝐿
o Multiply the symbol 𝑥 with an 𝐿-length vector 𝒘→ 𝒙 ෥ = 𝒘𝑥
o Here 𝒘 = 𝑤1 , 𝑤2 , … , 𝑤𝐿 𝑇 and 𝒙
෥ = 𝑥෤1 , 𝑥෤2 , … , 𝑥෤𝐿 𝑇 = 𝑤1 𝑥, 𝑤2 𝑥, … , 𝑤𝐿 𝑥 𝑇

Problem: Design 𝒘 such that the SNR is maximized OR capacity is achieved

o First antenna transmits 𝑤1 𝑥

o Second antenna transmits 𝑤2 𝑥 and so on…
o The receive signal at the receiver
𝑦 = ℎ1 𝑥෤1 + ℎ2 𝑥෤2 + ⋯ + ℎ𝐿 𝑥෤𝐿 + 𝑛
o In vector form:
𝑦 = 𝒉𝑇 𝒙 ෥+𝑛
o Here 𝒉𝑇 = ℎ1 , ℎ2 , … , ℎ𝐿 ∈ ℂ1×𝐿
MISO Gaussian channel
Problem: Design 𝒘 such that the SNR is maximized OR capacity is achieved

o Received signal
𝑦 = 𝒉𝑇 𝒙
෥+𝑛
= 𝒉𝑇 𝒘𝑥 + ณ 𝑛
desired signal noise
o Noise 𝑛 ∼ 𝐶𝑁(0, 𝜎 2 )
o Channels gains are constant and known at both transmitter and receiver
o Calculate the SNR and maximize it
o Signal power 𝔼 𝒉𝑇 𝒘𝑥 2 = 𝒉𝑇 𝒘 2 𝑃
o Noise power 𝔼 𝑛 2 = 𝜎 2 ℎ1
o Apply Cauchy Schwartz inequality to get
𝑇 2 𝑥෤1
𝒉 𝒘 𝑃 .
SNR = .
𝜎2 . Receiver
ℎ 2 𝑤 2𝑃 Transmitter ℎ𝐿
≤ 𝑦
𝜎2
o Equality holds when 𝒘 = 𝑐𝒉
1 𝑥෤𝐿
o To make the transmit power constrained to 𝑃, 𝑐 = h
MISO Gaussian channel
Problem: Design 𝒘 such that the SNR is maximized OR capacity is achieved

ℎ
𝑤=
ℎ
o This transmit beamforming is called maximal ratio transmission
o Capacity
‖𝒉‖2 𝑃
𝐶 = log 2 1 +
𝜎2
o 𝐿 transmit antennas also provide beamforming/power gain

ℎ1
𝑥෤1
.
.
. Receiver
Transmitter ℎ𝐿 𝑦

𝑥෤𝐿
 Lecture 6

MIMO Wireless Communication

(ELL8415)
Anupama Rajoriya
Capacity of Wireless Channels

Reference:
Chapter 5
Fundamentals of Wireless communication
David Tse, Pramod Vishwanath
Recap
o Capacity of AWGN channel
• SISO system
• SIMO system
• MISO system

In today’s class
o Fading channels
• Slow fading
• Fast fading
o Outage capacity for fading channels
• Slow fading SISO channels
SIMO Gaussian channel MISO Gaussian channel

ℎ1
ℎ1
𝑦1
. 𝑥෤1
. .
Transmitter ℎ𝐿 .
. Receiver
𝑥 Receiver Transmitter .
ℎ𝐿 𝑦
𝑦𝐿
𝑥෤𝐿

𝒉
o 𝒘= → maximal ratio transmission
o 𝒘 = 𝒉→ maximal ratio combining 𝒉
o Capacity of SIMO Gaussian channel o Capacity of MISO Gaussian channel
‖𝒉‖2 𝑃 ‖𝒉‖2 𝑃
𝐶 = log 2 1 + 𝐶 = log 2 1 +
𝜎2 𝜎2
o 𝐿 receive antennas provide beamforming/power gain o 𝐿 transmit antennas also provide
beamforming/power gain
Fading channels
o Channel varies over time
o Linear time variant channel
o SISO system model at instant 𝑚:
𝑦 𝑚 =ℎ 𝑚 𝑥 𝑚 +𝑛 𝑚
o Two kinds of fading:
o Slow fading
o fast fading
o Defined based on the relation between coherence time and symbol duration
o Coherence time is the time duration over which the channel can be considered approximately constant
o If the symbol duration is larger than the coherence time, the symbol will see the changing channel over its
transmission- fading of the channel is fast
o Coherence time ≫ symbol duration → Slow fading
o Coherence time ≪ symbol duration → Fast fading

o Fast fading is seen for high-speed vehicles (coherence time reduces)

o If delay requirements are tighter (low latency applications) symbol duration is small—slow fading
o Channel is modeled as a random quantity
o Only the receiver knows the channel
Slow fading SISO channel
o Channel is modeled as a random quantity
o Channel is constant for the symbol duration
o Input output relation at time instant 𝑚
𝑦 𝑚 = ℎ𝑥 𝑚 + 𝑛 𝑚
o As ℎ 𝑚 = ℎ
o Lets try to answer: Can we transmit reliably at a rate 𝑅?
o Recall if the channel is constant, the capacity (maximum rate that can be achieved) is
ℎ 2𝑃
𝐶 = log 1 + 2
𝜎
o Coding theorem says that a rate 𝑅 < 𝐶 can be achieved with low probability
o But ℎ is not known to the transmitter
o Transmitter can send the data at a rate 𝑅 which can be
o 𝑅 < 𝐶: reliable transmission happens
o 𝑅 > 𝐶: error probability is not small enough→ Outage event
Slow fading SISO channel
o Transmitter can send the data at a rate 𝑅 < 𝐶
o If the channel is strong→ communication can happen
o If the channel is weak → Outage event occurs
o The overall probability of error can be accounted by just looking at the probability of occurrence of outage
o Outage probability when operating at a rate 𝑅
𝑃out 𝑅 = Pr 𝑅 > 𝐶
ℎ 2𝑃
= Pr 𝑅 > log 1 + 2
𝜎
o Here ℎ is the random quantity
Slow fading SISO channel –Rayleigh fading
o Lets calculate outage probability for Rayleigh faded channel ℎ ∼ 𝐶𝑁(0,1)
𝑃out 𝑅 = Pr 𝑅 > log 1 + ℎ 2 SNR
2
2𝑅 − 1
= Pr ℎ <
SNR
o For Rayleigh fading channel ℎ, ℎ 2 follows an exponential distribution from zero to infinity
o The pdf takes the form 𝑓 𝑥 = 𝑒 −𝑥
2
2𝑅 − 1
𝑃out 𝑅 = Pr ℎ <
SNR
2𝑅−1
SNR 2𝑅 −1
− SNR
=න 𝑒 −𝑥 𝑑𝑥 = 1− 𝑒
0
o At high SNR
2𝑅 −1 2𝑅 − 1 2𝑅 − 1
𝑃out 𝑅 = 1 − 𝑒 − SNR
≈1− 1− = 𝑒 −𝑥 = 1 − 𝑥 for 𝑥 ≪ 1
SNR SNR
o Cannot guarantee 𝑃out 𝑅 = 0
o Define 𝜖-outage capacity
Outage capacity for slow fading SISO channel
o 𝜖-outage capacity 𝐶𝜖 : Largest transmission rate such that any rate 𝑅 < 𝐶𝜖 follows 𝑃out 𝑅 < 𝜖
𝑃out 𝐶𝜖 = 𝜖

𝜖 = 𝑃out 𝐶𝜖 = Pr 𝐶𝜖 > log 1 + ℎ 2 SNR

2
2𝐶𝜖 − 1
= Pr ℎ <
SNR
2𝐶𝜖 − 1
= 1 − Pr ℎ > 2 Complimentary CDF
SNR
𝐹 𝑥 = Pr ℎ 2 > 𝑥
2𝐶𝜖 − 1
=1−𝐹
SNR
2𝐶𝜖 − 1
⇒1−𝜖 =𝐹
SNR
⇒ 𝐶𝜖 = log 1 + SNR × 𝐹 −1 1 − 𝜖
 Outage capacity for slow fading SISO channel
o 𝜖-outage capacity 𝐶𝜖 : 𝐶𝜖 = log 1 + SNR × 𝐹 −1 1 − 𝜖

o High SNR regime:

𝐶𝜖 = log 1 + SNR × 𝐹 −1 1 − 𝜖 ≈ log SNR 𝐹 −1 1 − 𝜖
= log SNR) + log 𝐹 −1 1 − 𝜖
o We have
1
𝐶𝜖 ≈ 𝐶AWGN − log −1
𝐹 1−𝜖
o Here 𝐶AWGN = log 1 + SNR
o A constant difference irrespective of the SNR
o Low SNR regime:
𝐶𝜖 = log 1 + SNR × 𝐹 −1 1 − 𝜖 ≈ 𝐹 −1 1 − 𝜖 SNR log 𝑒
𝐶𝜖 ≈ 𝐹 −1 1 − 𝜖 𝐶AWGN ≈ 𝜖𝐶AWGN For small 𝜖, 𝐹 −1 1 − 𝜖 = 𝜖
o A fraction of 𝐶AWGN (smaller than the difference)
o Capacity 𝐶𝜖 should be kept low to ensure a low 𝜖
o At an outage probability of 0.01, the outage capacity is only 1% of the AWGN capacity!
Slow fading SIMO channel
o Lets consider 𝐿 receive antennas and single transmit antenna system
o The outage probability for 𝒉 ∈ ℂ1×𝐿 :
𝑃out 𝑅 = Pr 𝑅 > log 1 + ‖ℎ‖2 SNR
2
2𝑅 − 1
= Pr ‖ℎ‖ <
SNR
o For Rayleigh fading channel 𝒉,‖𝒉‖ becomes sum of squares of 2𝐿 independent Gaussian random variables
2

o It follows a Chi-square distribution with degree 2𝐿

o Its pdf takes the following form for 𝑥 > 0
1
𝑓 𝑥 = 𝑥 𝐿−1 𝑒 −𝑥
𝐿−1 !
o Approximate 𝑒 ≈ 1 for small 𝑥 (recall 𝑥 = 𝒉 )
−𝑥 2
1
𝑓 𝑥 ≈ 𝑥 𝐿−1
𝐿−1 !
Outage probability for slow fading SIMO channel
o At high SNR, outage probability:
2
2𝑅 − 1
𝑃out 𝑅 = Pr ‖𝒉‖ <
SNR
2𝑅−1 𝐿
SNR 1 𝐿−1
1 2𝑅 − 1
=න 𝑥 𝑑𝑥 =
0 𝐿−1 ! 𝐿! SNR

o Recall, for SISO channel we derived

2𝑅 − 1
𝑃out 𝑅 =
SNR
1 1
o For SISO 𝑃out 𝑅 ∝ SNR and for SIMO 𝑃out 𝑅 ∝ (SNR)𝐿→ Diversity gain of 𝐿
1
o Outage probability now decays as (SNR)𝐿
Capacity of SISO and SIMO channels

o 𝐿 receive antennas yield 𝐿-fold power gain as well as a diversity gain of 𝐿

o Power of SNR term in outage probability –Diversity gain
o Recall for low SNR the outage capacity is
1 1
𝐶𝜖 ≈ 𝐹 −1 1 − 𝜖 SNR log 𝑒 ≈ 𝐿! 𝐿 𝜖 𝐿 𝐶AWGN
o Here, 𝐶AWGN = log(1 + 𝐿SNR)
1
o Loss is lesser (𝜖 rather than 𝜖)
𝐿

o At 𝜖 = 0.01, 𝐶𝜖 is 14% of 𝐶AWGN , unlike the SISO case where it was just 1%.
Why are more antennas helping?
o Plot shows the PDF of 𝒉 2 for different values of 𝐿
o Notice that as 𝐿 increases the pdf around 𝒉 2 = 0 decreases
o It is less probable to have all the channels bad when 𝐿 is high
 Lecture 7

MIMO Wireless Communication

(ELL8415)
Anupama Rajoriya
Capacity of Wireless Channels

Reference:
Chapter 5
Fundamentals of Wireless communication
David Tse, Pramod Vishwanath
 Recap
o Fading channels
• Slow fading
• Fast fading
o Outage capacity for fading channels
• Slow fading SISO channels
• Slow fading SIMO channels

In today’s class
o Outage capacity for fading channels
• Slow fading MISO channels
o Capacity of fast fading SISO, SIMO and MISO channels
Slow fading SISO and SIMO channels
o Transmitter can send the data at a rate 𝑅 > 𝐶
o If the channel is strong→ communication can happen
o If the channel is weak → Outage event occurs
o Outage probability when operating at a rate 𝑅
𝑃out 𝑅 = Pr 𝑅 > 𝐶
o Outage capacity: Largest transmission rate such that any rate 𝑅 < 𝐶𝜖 follows 𝑃out 𝑅 < 𝜖
𝑃out 𝐶𝜖 = 𝜖
o Here ℎ is the random quantity, not known at the transmitter

o For SISO: Outage capacity 𝐶𝜖 = log 1 + SNR × 𝐹 −1 1 − 𝜖

1
o At high SNR: 𝐶𝜖 ≈ 𝐶AWGN − log • Capacity 𝐶 is a
𝐹 −1 1−𝜖
o At low SNR and low 𝜖: 𝐶𝜖 ≈ 𝜖𝐶AWGN function the fading
channel ℎ
o For SIMO: • Outage capacity 𝐶𝜖 is
1 1 not a function ℎ
o Outage capacity at low SNR: 𝐶𝜖 ≈ 𝐿! 𝐿 𝜖 𝐿 𝐶AWGN
Slow fading MISO channel
o Lets consider 𝐿 transmit antennas and single receive antenna system
𝑦 = 𝒉𝑇 𝒙
෥+𝑛
o Here 𝒙
෥ = 𝒘𝑥
o Does the definition of outage probability: 𝑃out 𝑅 = Pr 𝑅 > log 1 + ‖𝒉‖2 SNR still hold?
o No. As the transmitter does not know the channel 𝒉, it can not perform maximal ratio transmission (𝒘 = 𝒉/‖𝒉‖)
o Capacity (maximum achievable rate) to be compared with is no longer log 1 + ‖𝒉‖2 SNR
o Transmitter uses a transmission strategy which does not depend on 𝒉
Alamouti codes for 2x1 MISO channel
o Alamouti scheme: allows us to have the diversity gain without having channel knowledge
o Space time coding: symbols are spread in both space (over multiple antennas) and time (over multiple time
slots) domains
o Helps in reducing outages due to fading
o Consider a MISO channel with two transmit and one receive antenna
o At the first time instant we transmit 𝑥1 from the first antenna and 𝑥2 from second, to get
𝑦 1 = ℎ1 𝑥1 + ℎ2 𝑥2 + 𝑛[1]
∗
o At second time instant we transmit −𝑥2 from the first antenna and 𝑥1∗ from second, to get
𝑦 2 = −ℎ1 𝑥2∗ + ℎ2 𝑥1∗ + 𝑛[2]

o Here 𝑥1 and 𝑥2 have power constraint 𝑃/2 for total power constraint to be 𝑃
o Channel is assumed to be constant over two symbol durations
Alamouti codes for 2x1 MISO channel
o System model:
𝑥1 −𝑥2∗
𝑦1 𝑦 2 = ℎ1 ℎ2 + 𝑛1 𝑛2
𝑥2 𝑥1∗
o It can rewritten as
𝑦1 ℎ1 ℎ2 𝑥1 𝑛[1]
= ∗ ∗ ต +
𝑦2 ∗ ℎ2 − ℎ1 𝑥2 𝑛2∗
𝒚 𝑯 𝒙 𝒏
⇒ 𝒚 = 𝑯𝒙 + 𝒏

o The channel here is unitary, i.e., 𝑯𝐻 𝑯 = ( ℎ1 2 + ℎ2 2 )𝑰

o This allows us to receive the signal 𝒚 as :
𝑯𝐻 𝒚 = 𝑯𝐻 𝑯𝒙 + 𝑯𝐻 𝒏
෥ = ( ℎ1 2 + ℎ2 2 )𝒙 + 𝒏
𝒚 ෥
o This gives us
𝑦෤1 2 2
𝑥1 ℎ1∗ 𝑛 1 + ℎ2 𝑛 2 ∗
= ( ℎ1 + ℎ2 ) 𝑥 + ∗ ∗
𝑦෤2 2 ℎ2 𝑛 1 + ℎ1 𝑛 2
Alamouti codes for 2x1 MISO channel
o Lets calculate the SNR for this system model
𝑦෤1 2 2
𝑥1 ℎ1∗ 𝑛 1 + ℎ2 𝑛 2 ∗
= ( ℎ1 + ℎ2 ) 𝑥 + ∗ ∗
𝑦෤2 2 ℎ2 𝑛 1 + ℎ1 𝑛 2

ℎ1 2 + ℎ2 2 𝟐 𝑃 ℎ1 2
+ ℎ2 2
𝑃 𝒉 2 𝑆𝑁𝑅
SNR = = =
ℎ1 2 + ℎ2 2 2𝜎 2 2𝜎 2 2

𝒉 2
o Maximum achievable rate or the capacity thus becomes: log 1 + SNR
2
o Outage probability for Alamouti scheme:
‖𝒉‖2
𝑃out 𝑅 = Pr log 1 + SNR < 𝑅
2
o When the transmitter has the channel knowledge, the outage probability:
𝑃out 𝑅 = Pr log 1 + ‖𝒉‖2 SNR < 𝑅
o Power loss of factor 2, but same diversity gain
o Alamouti scheme radiates energy in isotropic manner: all the signals have same energy
o Outage probability for 𝐿 transmit antennas:
‖𝒉‖2
𝑃out 𝑅 = Pr log 1 + SNR < 𝑅
𝐿
Fast fading channels
o Coherence time ≪ symbol duration → Fast fading
o SISO system model at instant 𝑚:
𝑦 𝑚 =ℎ 𝑚 𝑥 𝑚 +𝑛 𝑚

o Channel changes within one symbol transmission

o It is mathematically modelled using a block-fading model
o Whole symbol duration 𝑇𝑆 is partitioned into 𝐿 coherence periods
o Within 𝑙-th coherence period, the channel remains same
o This means that we have system model of the following form
𝑦𝑙 𝑚 = ℎ𝑙 𝑥𝑙 𝑚 + 𝑛𝑙 𝑚
o Here 𝑙 = 1, 2, … , 𝐿 and 𝑚 = 1, … , 𝑇𝑆 denotes symbol index
o If 𝑇𝑆 ≫ 1, these 𝐿 channels can be assumed independently distributed
o Over a block, slow fading is being considered
o This gives us 𝐿 slow fading channels
o Over each block of time, power is constrained to 𝑃
Fast fading channels
o We have 𝐿 slow fading channels of the following form
𝑦𝑙 𝑚 = ℎ𝑙 𝑥𝑙 𝑚 + 𝑛𝑙 𝑚
𝑃
o With input power constraint 𝑃, SNR = 2
𝜎
o Capacity (maximum achievable rate) of each channel
log(1 + ℎ𝑙 2 SNR)
o Total rate of 𝐿 such channels
𝐿

෍ log 1 + ℎ𝑙 2 SNR
𝑙=1
o Average rate
𝐿
1
෍ log 1 + ℎ𝑙 2 SNR
𝐿
𝑙=1
o Outage probability
𝐿
1
𝑃out 𝑅 = Pr ෍ log 1 + ℎ𝑙 2 SNR < 𝑅
𝐿
𝑙=1
Fast fading channels
o What if the number of blocks in the block fading model goes to infinity 𝐿 → ∞
o The capacity, using law of large number reduces to
𝐿
1
෍ log 1 + ℎ𝑙 2 SNR → 𝔼 log 1 + ℎ 2 SNR
𝐿
𝑙=1
o The randomness due to channel has been removed by taking the expectation
o Capacity for a SISO fast fading channel
𝐶𝐹𝐹 = 𝔼 log 1 + ℎ 2 SNR
o Transmitter needs to know the statistics of the channel, and not the exact channel!

o When can this be achieved?

▪ Coding over a large number of coherence time intervals 𝐿
▪ Code observes multiple independent fades

o Similarly the capacity of fast fading SIMO channel (L receive antennas) and MISO channel (L transmit antennas)
𝐶𝐹𝐹 = 𝔼 log 1 + ‖𝒉‖2 SNR
o Whole codeword needs to be of length 𝐿𝑇𝑆 where 𝐿 and 𝑇𝑆 both are large
Fast fading channels
o AWGN capacity for SISO channel 𝐶AWGN = log(1 + SNR)

o Relation between 𝐶FF and 𝐶AWGN :

𝐶FF ≤ 𝐶AWGN
o Proof: Apply Jensen’s inequality on the log (concave) function
𝔼 log 1 + ℎ 2 SNR ≤ log 𝔼 1 + ℎ 2 SNR = log(1 + SNR)

o Here we assume 𝔼 ℎ 2
=1

o At low SNR:
𝔼 log 1 + ℎ 2 SNR ≈𝔼 ℎ 2
SNR log 𝑒 = 𝐶AWGN

o At high SNR:
𝔼 log 1 + ℎ 2 SNR ≈ 𝔼 log ℎ 2 SNR = 𝐶AWGN + 𝔼[log ℎ 2 ]
o Constant difference term can be further bounded as 𝔼[log ℎ 2 ] ≤ log 𝔼 ℎ 2 = 0
Fast fading channels

o For Rayleigh channel

o AWGN capacity for SISO channel
𝐶AWGN = log(1 + SNR)
o Full CSI capacity
1 𝐿
σ log 1 + ℎ𝑙 2 SNR
𝐿 𝑙=1
o CSIR capacity
𝔼 log 1 + ℎ 2 SNR
o Difference is not large as opposed
to slow fading case
 Lecture 8

MIMO Wireless Communication

(ELL8415)
Anupama Rajoriya
MIMO Wireless Channels

Reference:
Chapter 7
Fundamentals of Wireless communication
David Tse, Pramod Vishwanath
Recap
o Fading channels
• Slow fading
• Fast fading
o Outage capacity for slow fading SISO, SIMO and MISO channels
o Capacity of fast fading SISO, SIMO and MISO channels

In today’s class
o MIMO channel representation
o MIMO channel characteristics
o Capacity MIMO channels
▪ Deterministic channel
▪ Full CSI
▪ CSI at receiver only
Summary: capacity of wireless channels
Deterministic channel with channel information at transmitter and receiver

SISO 𝑦 = ℎ𝑥 + 𝑛 𝐶 = log(1 + ℎ 2 SNR)

Summary: capacity of wireless channels
Deterministic channel with channel information at transmitter and receiver

SISO 𝑦 = ℎ𝑥 + 𝑛 𝐶 = log(1 + ℎ 2 SNR)

SIMO 𝒚 = 𝒉𝑥 + 𝒏 𝐶 = log(1 + ‖𝒉‖2 SNR)

Summary: capacity of wireless channels
Deterministic channel with channel information at transmitter and receiver

SISO 𝑦 = ℎ𝑥 + 𝑛 𝐶 = log(1 + ℎ 2 SNR)

SIMO 𝒚 = 𝒉𝑥 + 𝒏 𝐶 = log(1 + ‖𝒉‖2 SNR)

MISO 𝑦 = 𝒉𝑇 𝒙
෥+𝑛 𝐶 = log(1 + ‖𝒉‖2 SNR)
Summary: capacity of wireless channels
Slow fading channel without channel information at transmitter

System System model Outage probability Outage capacity

SISO 𝑦 = ℎ𝑥 + 𝑛 Pr 𝑅 > log 1 + ℎ 2 SNR 𝐶𝜖 = log 1 + SNR × 𝐹 −1 1 − 𝜖

Summary: capacity of wireless channels
Slow fading channel without channel information at transmitter

System System model Outage probability Outage capacity

SISO 𝑦 = ℎ𝑥 + 𝑛 Pr 𝑅 > log 1 + ℎ 2 SNR 𝐶𝜖 = log 1 + SNR × 𝐹 −1 1 − 𝜖

SIMO 𝒚 = 𝒉𝑥 + 𝒏 Pr 𝑅 > log 1 + ‖𝒉‖2 SNR 𝐶𝜖 = log 1 + SNR × 𝐹 −1 1 − 𝜖

Summary: capacity of wireless channels
Slow fading channel without channel information at transmitter

System System model Outage probability Outage capacity

SISO 𝑦 = ℎ𝑥 + 𝑛 Pr 𝑅 > log 1 + ℎ 2 SNR 𝐶𝜖 = log 1 + SNR × 𝐹 −1 1 − 𝜖

SIMO 𝒚 = 𝒉𝑥 + 𝒏 Pr 𝑅 > log 1 + ‖𝒉‖2 SNR 𝐶𝜖 = log 1 + SNR × 𝐹 −1 1 − 𝜖

MISO 𝑦 = 𝒉𝑇 𝒙
෥+𝑛 ‖𝒉‖2 𝐶𝜖
Pr 𝑅 > log 1 + SNR SNR
2 = log 1 + × 𝐹 −1 1 − 𝜖
2
Summary: capacity of wireless channels
Fast fading channel without channel information at transmitter

System System model Capacity

SISO 𝑦 = ℎ𝑥 + 𝑛 𝐶 = 𝔼ℎ log(1 + ℎ 2 SNR)

Summary: capacity of wireless channels
Fast fading channel without channel information at transmitter

System System model Capacity

SISO 𝑦 = ℎ𝑥 + 𝑛 𝐶 = 𝔼ℎ log(1 + ℎ 2 SNR)

SIMO 𝒚 = 𝒉𝑥 + 𝒏 𝐶 = 𝔼𝒉 log(1 + ‖𝒉‖2 SNR)

Summary: capacity of wireless channels
Fast fading channel without channel information at transmitter

System System model Capacity

SISO 𝑦 = ℎ𝑥 + 𝑛 𝐶 = 𝔼ℎ log(1 + ℎ 2 SNR)

SIMO 𝒚 = 𝒉𝑥 + 𝒏 𝐶 = 𝔼𝒉 log(1 + ‖𝒉‖2 SNR)

MISO 𝑦 = 𝒉𝑇 𝒙
෥+𝑛 𝐶 = 𝔼𝒉 log(1 + ‖𝒉‖2 SNR)
MIMO channels
o Multiple antennas at the Tx or Rx give power and diversity gain
o Can we increase the capacity linearly with number of antennas?
o Multiple Tx and Rx antennas help us achieve that
MIMO channels
ℎ11
o Consider a system with 𝑛𝑡 -antenna transmiiter and ℎ𝑛𝑟 1
𝑦1
𝑛𝑟 -antenna receiver 𝑥෤1
o Let the data transmitted over 𝑛𝑡 antennas is given . .
by: 𝒙෥ = 𝑾𝑡 𝒙 . .
ℎ1𝑛𝑡
. Receiver
o Here, 𝒙 ∈ ℂ𝑛𝑡 ×1 data to be transmitted and 𝑾𝑡 ∈ Transmitter .
ℎ𝑛𝑟 𝑛𝑡
ℂ𝑛𝑡×𝑛𝑡 is the beamforming matrix
o Transmit signal:
𝑇 𝑥෤𝑛𝑡 𝑦𝑛𝑟
෥ = 𝑥෤1 , 𝑥෤2 , … , 𝑥෤𝑛𝑡
𝒙
𝑛𝑡
o Here each 𝑥෤𝑖 = σ𝑗=1 𝑤𝑖𝑗 𝑥𝑗
o Receive signal at antenna 1:
𝑦1 = ℎ11 𝑥෤1 + ℎ12 𝑥෤2 + ⋯ + ℎ1𝑛𝑡 𝑥෤𝑛𝑡 + 𝑛1
o Receive signal at antenna 2:
𝑦2 = ℎ21 𝑥෤1 + ℎ22 𝑥෤2 + ⋯ + ℎ2𝑛𝑡 𝑥෤𝑛𝑡 + 𝑛2
o Receive signal at antenna 𝑛𝑟 :
𝑦𝑛𝑟 = ℎ𝑛𝑟 1 𝑥෤1 + ℎ𝑛𝑟 2 𝑥෤2 + ⋯ + ℎ𝑛𝑟 𝑛𝑡 𝑥෤𝑛𝑡 + 𝑛𝑛𝑟
MIMO channels
o Complete receive signal in vector representation
𝑦1 ℎ11 𝑥෤1 + ℎ12 𝑥෤2 + ⋯ + ℎ1𝑛𝑡 𝑥෤𝑛𝑡 𝑛1
𝑦2 ℎ21 𝑥෤1 + ℎ22 𝑥෤2 + ⋯ + ℎ2𝑛𝑡 𝑥෤𝑛𝑡 𝑛2
… = …
+ …
𝑦𝑛𝑟 ℎ𝑛𝑟 1 𝑥෤1 + ℎ𝑛𝑟 2 𝑥෤2 + ⋯ + ℎ𝑛𝑟 𝑛𝑡 𝑥෤𝑛𝑡 𝑛𝑛𝑟
o Can be re-written as
𝑦1 ℎ11 ℎ12 ⋯ ℎ1𝑛𝑡 𝑥෤1 𝑛1
𝑦2 ℎ21 ℎ22 ⋯ ℎ2𝑛𝑡 𝑥෤2 𝑛2
… = ⋮ ⋮ ⋱ ⋮ … + …
𝑦𝑛𝑟 ℎ𝑛 1 ℎ𝑛 2 ⋯ ℎ𝑛 𝑛 𝑥෤𝑛𝑡 𝑛𝑛𝑡
𝑟 𝑟 𝑟 𝑡

𝒚 ෥
𝒙 𝒏
𝑯
⇒ 𝒚 = 𝑯෥
𝒙+𝒏
o Here, 𝒚, 𝒏 ∈ ℂ𝑛𝑟 ×1 , 𝒙
෥ ∈ ℂ𝑛𝑡×1 and 𝑯 ∈ ℂ𝑛𝑟 ×𝑛𝑡
o Noise 𝒏 ∼ 𝒞𝒩(𝟎, 𝚺), where 𝚺 = 𝔼 𝒏𝒏𝐻 = 𝜎 2 𝑰 ∈ ℂ𝑛𝑟 ×𝑛𝑟 is the noise covariance matrix
o At the receiver end, combining is also performed using 𝑾𝒓 ∈ ℂ𝑛𝑟 ×𝑛𝑟

Design of 𝑾𝒓 and 𝑾𝒕 is crucial to maximize the rate

MIMO channels
o Complete MIMO channel
ℎ11 ℎ12 ⋯ ℎ1𝑛𝑡
ℎ21 ℎ22 ⋯ ℎ2𝑛𝑡
𝑯=
⋮ ⋮ ⋱ ⋮
ℎ𝑛𝑟 1 ℎ𝑛𝑟 2 ⋯ ℎ𝑛𝑟 𝑛𝑡
o For now, we will assume that all the entries of 𝑯 are independent and identically distributed
o Some channel characteristics:
o SVD decomposition of the channel 𝑯
𝑯 = 𝑼𝚲𝐕 H
o It is defined for any matrix
o Matrices 𝑼 ∈ ℂ𝑛𝑟 ×𝑛𝑟 and 𝑽 ∈ ℂ𝑛𝑡 ×𝑛𝑡 are unitary matrices, i.e., 𝑼𝐻 𝑼 = 𝑼𝑼𝐻 = 𝑰𝑛𝑟 and 𝑽𝐻 𝑽 = 𝑽𝑽𝐻 = 𝑰𝑛𝑡
𝑛 ×𝑛
o Matrix 𝚲 ∈ ℝ+𝑟 𝑡 is a diagonal matrix, with ordered singular values of 𝑯 in its diagonals
o 𝑯 has 𝑛min = min 𝑛𝑟 , 𝑛𝑡 positive singular values
o For example, if 𝑛𝑡 > 𝑛𝑟 , 𝑛min = 𝑛𝑟 and
𝜆1 ⋯ 0 0 ⋯ 0
𝚲= ⋮ ⋱ ⋮ ⋮ ⋱ ⋮
0 ⋯ 𝜆𝑛𝑟 0 ⋯ 0

𝑛𝑟 𝑛𝑟
o Also𝜆1 ≥ 𝜆2 ≥ ⋯ ≥ 𝜆𝑛min × 𝑛𝑟 × (𝑛𝑡 −𝑛𝑟 )
MIMO channels
o For the channel
ℎ11 ℎ12 ⋯ ℎ1𝑛𝑡
ℎ21 ℎ22 ⋯ ℎ2𝑛𝑡
𝑯=
⋮ ⋮ ⋱ ⋮
ℎ𝑛𝑟 1 ℎ𝑛𝑟 2 ⋯ ℎ𝑛𝑟 𝑛𝑡
o SVD decomposition: 𝑯 = 𝑼𝚲𝐕 H
o very useful tool if the channel is known at both transmitter and receiver
MIMO channels
o For the channel
ℎ11 ℎ12 ⋯ ℎ1𝑛𝑡
ℎ21 ℎ22 ⋯ ℎ2𝑛𝑡
𝑯=
⋮ ⋮ ⋱ ⋮
ℎ𝑛𝑟 1 ℎ𝑛𝑟 2 ⋯ ℎ𝑛𝑟 𝑛𝑡
o SVD decomposition: 𝑯 = 𝑼𝚲𝐕 H
o very useful tool if the channel is known at both transmitter and receiver

𝑦1
𝑥෤1
. . .
. . .
. Receiver
Transmitter . .

𝑥෤𝑛𝑡 𝑦𝑛𝑟
MIMO channels
o For the channel
ℎ11 ℎ12 ⋯ ℎ1𝑛𝑡
ℎ21 ℎ22 ⋯ ℎ2𝑛𝑡
𝑯=
⋮ ⋮ ⋱ ⋮
ℎ𝑛𝑟 1 ℎ𝑛𝑟 2 ⋯ ℎ𝑛𝑟 𝑛𝑡
o SVD decomposition: 𝑯 = 𝑼𝚲𝐕 H
o very useful tool if the channel is known at both transmitter and receiver

Decouples channel matrix 𝑯

into 𝑛min parallel channels

𝑦1
𝑥෤1
. . .
. . .
. Receiver
Transmitter . .

𝑥෤𝑛𝑡 𝑦𝑛𝑟
MIMO Gaussian channels
o We consider the channel deterministic in this case
o Assume it to be known at both transmitter and receiver (full CSI)
o As there are only at max 𝑛min distinct paths, we send the data with 𝑛min non-zeros
o The data vector 𝒙 = 𝑥1 , 𝑥2 , … , 𝑥𝑛𝑚𝑖𝑛 , 0, … , 0 ∈ ℂ𝑛𝑡 ×1 ⇒ (𝑛𝑡 − 𝑛min ) zeros
o Recall the data transmitted over 𝑛𝑡 antennas is given by: 𝒙 ෥ = 𝑾𝑡 𝒙
o Here, 𝑾𝑡 ∈ ℂ 𝑡 𝑡 is the beamforming/precoding matrix
𝑛 ×𝑛

o Received signal:
𝒚 = 𝑯෥ 𝒙+𝒏
o Use the SVD decomposition of 𝑯
𝒚 = 𝑼𝚲𝐕 H 𝒙෥ + 𝒏 = 𝑼𝚲𝐕 H 𝑾𝑡 𝒙 + 𝒏

o At the receiver, the received signals are combined using 𝑾𝑟 to get

𝑾𝐻 𝐻 H 𝐻
𝑟 𝒚 = 𝑾𝑟 𝑼𝚲𝐕 𝑾𝑡 𝒙 + 𝑾𝑟 𝒏
MIMO Gaussian channels
o What should be the optimal beamforming/precoding and combining matrices 𝑾𝑡 and 𝑾𝑟 , respectively?
o Lets take
𝑾𝑡 = 𝑽 and 𝑾𝒓 = 𝑼

o Combined received signal

𝑾𝐻 𝐻 H 𝐻
𝑟 𝒚 = 𝑾𝑟 𝑼𝚲𝐕 𝑾𝑡 𝒙 + 𝑾𝑟 𝒏
⇒= 𝑼𝐻 𝑼𝚲𝐕 H 𝐕𝒙 + 𝑼𝐻 𝒏
⇒𝑼 𝐻 𝒚 = 𝚲𝒙 + 𝑼
ถ𝐻𝒏
ถ
෥
𝒚 ෥
𝒏
o Lets look at the equivalent noise term 𝒏
෥= 𝑼𝐻 𝒏
o Its mean 𝔼[𝑼 𝒏] = 𝟎 and covariance𝔼[෥
𝐻 ෥ 𝐻 ] = 𝔼[𝑼𝐻 𝒏𝒏𝐻 𝑼] = 𝑼𝐻 𝔼 𝒏𝒏𝐻 𝑼 = 𝑼𝐻 𝜎 2 𝑰 𝑼 = 𝜎 2 𝑰
𝒏𝒏
o ෥ has same distribution as 𝒏, as 𝑼 is a unitary matrix
𝒏
o Similarly, we have 𝔼 𝒙 ෥ 𝟐 = 𝔼 𝒙 𝟐 , where 𝒙 ෥ = 𝑽𝒙
MIMO Gaussian channels
o Combined received signal
෥ = 𝚲𝒙 + 𝒏
𝒚 ෥
o Interestingly, this can be written as multiple (𝑛min to be precise) SISO channels
𝑦෤𝑖 = 𝜆𝑖 𝑥𝑖 + 𝑛෤ 𝑖
o Here the index 𝑖 = 1, 2, … , 𝑛min
o Capacity of each of these channels is
𝜆2𝑖 𝑃𝑖
𝐶𝑖 = log 1 + 2
𝜎
o We used the fact that 𝔼 𝑥𝑖 = 𝑃𝑖
2
𝑛
o For constrained transmit power, we need to have σ𝑖 min 𝑃𝑖 ≤ 𝑃
o Recall the capacity C is achieved when 𝑥𝑖 is Gaussian distributed
o Total capacity: Sum of capacities of all these channels
𝑛min
𝜆2𝑖 𝑃𝑖
𝐶sum = ෍ log 1 + 2
𝜎
𝑖
o Still there is a scope to maximize the rate: optimal power allocation
o Optimal power allocation: Find 𝑃𝑖 for all 𝑖 such that 𝐶sum is maximized
 Lecture 9

MIMO Wireless Communication

(ELL8415)
Anupama Rajoriya
MIMO Wireless Channels

Reference:
Chapter 7
Fundamentals of Wireless communication
David Tse, Pramod Vishwanath
Recap
o MIMO channel representation
o MIMO channel characteristics

In today’s class
o Capacity MIMO channels
▪ Deterministic channel
▪ Full CSI
▪ CSI at receiver only
 ℎ11

MIMO channels 𝑥෤1

ℎ𝑛 𝑟 1
𝑦1

. .
o Consider a system with 𝑛𝑡 -antenna transmiiter and . ℎ1𝑛𝑡 .
𝑛𝑟 -antenna receiver . Receiver
Transmitter .
ℎ𝑛𝑟𝑛𝑡
o Let the data transmitted over 𝑛𝑡 antennas is given
by: 𝒙෥ = 𝑾𝑡 𝒙
o Here, 𝒙 ∈ ℂ𝑛𝑡 ×1 data to be transmitted and 𝑾𝑡 ∈ 𝑥෤𝑛𝑡 𝑦𝑛𝑟
ℂ𝑛𝑡×𝑛𝑡 is the beamforming matrix
o Complete receive signal in vector representation
𝑦1 ℎ11 ℎ12 ⋯ ℎ1𝑛𝑡 𝑥෤1 𝑛1
𝑦2 ℎ21 ℎ22 ⋯ ℎ2𝑛𝑡 𝑥෤2 𝑛2
… = ⋮ ⋮ ⋱ ⋮ … + …
𝑦𝑛𝑟 ℎ𝑛 𝑟 1 ℎ𝑛 𝑟 2 ⋯ ℎ𝑛 𝑟 𝑛 𝑡 𝑥෤𝑛𝑡 𝑛𝑛 𝑡

𝒚 ෥
𝒙 𝒏
𝑯
⇒ 𝒚 = 𝑯෥
𝒙+𝒏
o Here, 𝒚, 𝒏 ∈ ℂ ,𝒙
𝑛𝑟 ×1
෥∈ℂ and 𝑯 ∈ ℂ
𝑛𝑡 ×1 𝑛𝑟 ×𝑛𝑡

o Noise 𝒏 ∼ 𝒞𝒩(𝟎, 𝚺), where 𝚺 = 𝔼 𝒏𝒏𝐻 = 𝜎 2 𝑰 ∈ ℂ𝑛𝑟 ×𝑛𝑟 is the noise covariance matrix
o Received signals are combined using 𝑾𝑟 ∈ ℂ𝑛𝑟×𝑛𝑟 to get
𝑾𝐻 𝐻
𝑟 𝒚 = 𝑾𝑟 𝑯𝑾𝑡 𝒙 + 𝑾𝑟 𝒏
𝐻
MIMO channels
o SVD decomposition of the channel 𝑯
𝑯 = 𝑼𝚲𝐕 H
o It is defined for any matrix
o Matrices 𝑼 ∈ ℂ𝑛𝑟×𝑛𝑟 and 𝑽 ∈ ℂ𝑛𝑡 ×𝑛𝑡 are unitary matrices, i.e., 𝑼𝐻 𝑼 = 𝑼𝑼𝐻 = 𝑰𝑛𝑟 and 𝑽𝐻 𝑽 = 𝑽𝑽𝐻 = 𝑰𝑛𝑡
𝑛 ×𝑛
o Matrix 𝚲 ∈ ℝ+𝑟 𝑡 is a diagonal matrix, with ordered singular values of 𝑯 in its diagonals
o 𝑯 has 𝑛min = min 𝑛𝑟 , 𝑛𝑡 positive singular values

o Singular values are square root of eigenvalues of 𝑯𝐻 𝑯

o Eigenvectors of 𝑯𝐻 𝑯 constitute 𝑽
𝟏
o Each column of 𝑼 follows 𝒖𝑖 = 𝑯𝒗𝑖
𝜆𝑖
MIMO Gaussian channels
o We consider the channel is deterministic in this case
o Assume it to be known at both transmitter and receiver (full CSI)
o At the receiver, the received signals
𝑾𝐻 𝐻
𝑟 𝒚 = 𝑾𝑟 𝑯𝑾𝑡 𝒙 + 𝑾𝑟 𝒏
𝐻

⇒𝒚 ෥ = 𝑾𝐻 H
𝑟 𝑼𝚲𝐕 𝑾𝑡 𝒙 + 𝒏 ෥
o By taking 𝑾𝑡 = 𝑽 and 𝑾𝒓 = 𝑼, we get
෥ = 𝑼𝐻 𝑼𝚲𝐕 H 𝐕𝒙 + 𝒏
𝒚 ෥
⇒𝒚 ෥ = 𝚲𝒙 + 𝒏 ෥
o We already saw that 𝔼 𝒙 ෥ 𝟐 = 𝔼 𝒙 𝟐 and 𝔼[෥ ෥ 𝐻 ] = 𝔼[𝒏𝒏𝐻 ] = 𝜎 2 𝑰
𝒏𝒏
MIMO Gaussian channels
o Combined received signal
෥ = 𝚲𝒙 + 𝒏
𝒚 ෥
o Interestingly, this can be written as multiple (𝑛min to be precise) SISO channels
𝑦෤𝑖 = 𝜆𝑖 𝑥𝑖 + 𝑛෤ 𝑖
o Here the index 𝑖 = 1, 2, … , 𝑛min
o Capacity of each of these channels is
𝜆2𝑖 𝑃𝑖
𝐶𝑖 = log 1 + 2
𝜎
o We used the fact that 𝔼 𝑥𝑖 = 𝑃𝑖
2
𝑛
o For constrained transmit power, we need to have σ𝑖 min 𝑃𝑖 ≤ 𝑃
o Recall the capacity C is achieved when 𝑥𝑖 is Gaussian distributed
o Total capacity: Sum of capacities of all these channels
𝑛min
𝜆2𝑖 𝑃𝑖
𝐶sum = ෍ log 1 + 2
𝜎
𝑖
o Still there is a scope to maximize the rate: optimal power allocation
o Optimal power allocation: Find 𝑃𝑖 for all 𝑖 such that 𝐶sum is maximized
MIMO Gaussian channels
o Optimal power allocation: Find 𝑃𝑖 for all 𝑖 such that 𝐶sum is maximized
max 𝐶sum
𝑃𝑖
𝑛min
𝜆2𝑖 𝑃𝑖
⇒ max ෍ log 1 + 2
𝑃1 ,𝑃2 ,…,𝑃𝑛𝑚𝑖𝑛 𝜎
𝑖
𝑛
Subject to σ𝑖 min 𝑃𝑖 = 𝑃

o Lets solve this optimization problem

o Maximization of a concave function (log) with linear constraints
o Closed form solution can be obtained using Langrangian multiplier method
o Langrangian cost function becomes
𝑛min 𝑛min
𝜆2𝑖 𝑃𝑖
ℒ 𝑃1 , … , 𝑃𝑛min , 𝜇 = ෍ log 1 + 2 + 𝜇 𝑃 − ෍ 𝑃𝑖
𝜎
𝑖 𝑖

o Differentiating ℒ 𝑃1 , … , 𝑃𝑛min , 𝜇 with respect to 𝑃𝑖 and setting it to zero

MIMO Gaussian channels
o Optimal power allocation:
o Differentiating ℒ 𝑃1 , … , 𝑃𝑛min , 𝜇 with respect to 𝑃𝑖 and setting it to zero
𝑑
ℒ 𝑃1 , … , 𝑃𝑛min , 𝜇 = 0
𝑑𝑃𝑖
𝑛min
𝑑 𝜆2𝑖 𝑃𝑖
⇒ log 1 + 2 + 𝜇 𝑃 − ෍ 𝑃𝑖 =0
𝑑𝑃𝑖 𝜎
𝑖
1 𝜆2𝑖
⇒ × −𝜇 =0
𝜆2𝑖 𝑃𝑖 𝜎2
1+ 2
𝜎
𝜆2𝑖 𝜆2𝑖 𝑃𝑖
⇒ 2 =𝜇 1+ 2
𝜎 𝜎
2
1 𝜎
⇒ 𝑃𝑖 = −
𝜇 𝜆2𝑖
+
1 𝜎2
⇒ 𝑃𝑖 = − To ensure non-negativity of power
𝜇 𝜆2𝑖
MIMO Gaussian channels
o Optimal power allocation:
+
1 𝜎2
𝑃𝑖 = −
𝜇 𝜆2𝑖
𝑛
o Further, the Lagrangian multiplier 𝜇 can be obtained by using σ𝑖 min 𝑃𝑖 = 𝑃
𝑛min +
1 𝜎2
෍ − =𝑃
𝜇 𝜆2𝑖
𝑖=1
o This power allocation technique is popularly known as Waterfilling
o It is a greedy allocation→ more power to good channel
o Less or no power to poor channel
MIMO Gaussian channels
o Example: SVD-based precoder and combiner, and Waterfilling power allocation:
o Consider a MIMO system with total transmit power 𝑃 = 0.75 W and 𝜎 2 = 2 W
o Consider a channel matrix
2 −6 0
𝑯= 3 4 0
0 0 2
o Its SVD decomposition can be obtained as

−3 2
1 0 52 0 0 0 1 0
𝑯= 2 3 0 0 13 0 1 0 0
13 0 0 13 0 2 0 0 1
0
o Singular values are square root of eigenvalues of 𝑯𝐻 𝑯, its eigenvectors constitute 𝑽 and each column of 𝑼 follows
𝟏
𝒖𝑖 = 𝑯𝒗𝑖
𝜆𝑖
𝜆2𝑖 𝑃𝑖
o Capacity: 𝐶sum = σ𝑛𝑖 min log 1+
𝜎2
𝜆12 𝑃1 𝜆22 𝑃2 𝜆23 𝑃3
𝐶sum = log 1 + 2 + log 1 + 2 + log 1 + 2
𝜎 𝜎 𝜎
52𝑃1 13𝑃2 4𝑃3
= log 1 + + log 1 + + log 1 +
2 2 2
o Such that 𝑃1 + 𝑃2 + 𝑃3 = 0.75
MIMO Gaussian channels
o Example: SVD-based precoder and combiner, and Waterfilling power allocation:
o Lets determine 𝜇
𝑛min
1 𝜎2
෍ − =𝑃
𝜇 𝜆2𝑖
𝑖=1
1 2 1 2 1 2
⇒ − + − + − = 0.75
𝜇 52 𝜇 13 𝜇 4
1 25
⇒ = = 0.4808
𝜇 52
o First allocate the power to the weakest channel– channel 3
1 2 1
𝑃3 = − =− <0
𝜇 4 52
⇒ 𝑃3 = 0
o No power allocated to channel 3
o Recalculate the value of 𝜇
1 2 1 2
− + − = 0.75
𝜇 52 𝜇 13
1 49
⇒ = = 0.4712
𝜇 104
MIMO Gaussian channels
o Example: SVD-based precoder and combiner, and Waterfilling power allocation:
o Lets now allocate the power to the current weakest channel– channel 2
1 2 33
𝑃2 = − = = 0.3174 > 0
𝜇 13 104
⇒ 𝑃2 = 0.3174
o Similarly, the power of the strongest channel- channel 1 becomes
1 2 45
𝑃1 = − = = 0.4327 > 0
𝜇 52 104
⇒ 𝑃1 = 0.4327
MIMO Gaussian channels
o Example: SVD-based precoder and combiner, and Waterfilling power allocation:
MIMO Gaussian channels
o Example: SVD-based precoder and combiner, and Waterfilling power allocation:

+
1 𝜎2
−
𝜇 𝜆12

𝜎2
𝜆12
Waterfilling algorithm Unfair to weaker channels
o Set 𝑘 = 𝑛min
1. Calculate 𝜇 according to the following equation
𝑘
1 𝜎2
෍ − 2 =𝑃
𝜇 𝜆𝑖
𝑖=1

2. Start with the channel with lowest SNR (i.e., lowest singular value)

3. Allocate power to the 𝑘-th channel using the value of 𝜇 obtained as follows
+
1 𝜎2
𝑃𝑘 = −
𝜇 𝜆2𝑘

4. If 𝑃𝑘 = 0, no power allocated to channel 𝑘. Set 𝑘 = 𝑘 − 1 and go to step 1

5. If 𝑃𝑘 > 0, set 𝑘 = 𝑘 − 1 and go to step 3

Power allocation with QoS constraint
o Optimal power allocation: Find 𝑃𝑖 for all 𝑖 such that 𝐶sum is maximized
max 𝐶sum
𝑃𝑖
𝑛min
𝜆2𝑖 𝑃𝑖
⇒ max ෍ log 1 + 2
𝑃1 ,𝑃2 ,…,𝑃𝑛𝑚𝑖𝑛 𝜎
𝑖
𝑛
Subject to σ𝑖 min 𝑃𝑖 = 𝑃
Power allocation with QoS constraint
o Optimal power allocation with QoS: Find 𝑃𝑖 for all 𝑖 such that 𝐶sum is maximized
max 𝐶sum
𝑃𝑖
𝑛min
𝜆2𝑖 𝑃𝑖
⇒ max ෍ log 1 + 2
𝑃1 ,𝑃2 ,…,𝑃𝑛𝑚𝑖𝑛 𝜎 Again an optimization
𝑖
𝑛 problem to be solved!
Subject to σ𝑖 min 𝑃𝑖 = 𝑃,
𝜆2𝑖 𝑃𝑖
log 1 + ≥ 𝑅𝑖 for 1 = 1, 2, … , 𝑛min
𝜎2

o Last constraint ensures a positive rate for each channel

o Relevant when each channel caters to a different application
o For example, channel-1 for Youtube video, channel-2 for voice etc
o Both these applications have different rate requirements
Capacity of deterministic MIMO channel
o Capacity of deterministic MIMO channel
𝑛min
𝜆2𝑖 𝑃𝑖
𝐶sum = ෍ log 1 + 2
𝜎
𝑖
o When the transmitter and receiver both know the channel
o At high SNR:
𝑛min
𝜆2𝑖 𝑃𝑖
𝐶sum = ෍ log
𝜎2
𝑖
o For equal power allocation
𝑛min
𝜆2𝑖 SNR
𝐶sum = ෍ log
𝑛min
𝑖
𝑛min
SNR
= 𝑛min log + ෍ log 𝜆2𝑖
𝑛min
𝑖

Degrees of freedom
MIMO channels
o Capacity linearly increases with number of antennas→ degrees of freedom