PART - I

Information Theory and Source Coding

Chapter 1 Source Coding
Chapter 2 Channel Capacity and Coding
 CHAPTER ONE

Source Coding

Not everything that can be counted counts, and not everything that counts can be counted.
Albert Einstein (1879-1955)

learning objectives

After studying this chapter the students will be able to:

LO 1 Understanding the concept of information.

LO 2
LO 3 ! "
# $ % #
LO 4 Underline the concept of the rate distortion function and the design of the optimum

LO 5 & ' (
areas of source coding.

1.1 Introduction to Information Theory

Today we live in the information age. Internet has
This chapter comes with a video
become an integral part of our lives, making this third overview by the author. Scan here to
planet from the sun, a global village. People are seen know more or
talking over the cellular phones, sometimes even in Visit [Link]
cinema halls. Movies can be rented in the form of a
DVD disk or streamed directly to our smart phones. Email and web addresses are commonly visible on
business cards. Most of the people today prefer to send emails and e-cards to their friends rather than the

captured and uploaded on social media sites with just a click of a button.
12 ° Information Theory, Coding and Cryptography

!
world it is the " # $ % &
(the omnipresent bits) that hold the information by the mere way they sit next to one another.
' %*+/
Information Theory—a theory that was initiated by one man, the
American Electrical Engineer Claude E. Shannon, whose ideas appeared in the article “The Mathematical
Theory of Communication” in the Bell System Technical Journal %*+/" !
interpreted to include the messages occurring in any of the standard communications media, such as telephone,
radio, or television, and the signals involved in electronic computers, electromechanical systems, and other
data-processing devices. The theory is even applicable to the signals appearing in the nerve networks of
humans and other animals.
The chief concern of information theory is to discover mathematical laws governing systems designed to
communicate or manipulate information. It sets up quantitative measures of information and of the capacity
of various systems to transmit, store, and otherwise process information. Some of the problems treated are
0 0
for separating the wanted information, or signal, from the extraneous information, or noise. Another problem
is the setting of upper bounds on what it is possible to achieve with a given information-carrying medium
" 1 2 !
0 3
!
DNA regions (genotypes) responsible for particular traits (phenotypes).
The boundaries of information theory are quite fuzzy. The theory overlaps heavily with
communication theory but is more oriented toward the fundamental limitations on the
processing and communication of information and less oriented toward the detailed operation
of the devices employed.

In this chapter
We will start with an intuitive understanding of information and link uncertainty to information.
) ' ( ( ( ( * + (
( #, 1.
- / ' & / (
0 / ) ' ( / #, 2.
) '
! " # $ % #
#, 3.
- / ' ' ( ( (
) ' ' ( ) '
1 & #, +
! ' ' 0 ( ' & '
( ( 2
34"5 ' 0 #, 5.
 Source Coding ° 13

1.2 Uncertainty and Information

Any information source produces an output that is random in nature. If

the source output had no randomness, i.e., the output were known exactly, LO 1
there would be no need to transmit it! There exist both analog and discrete
Understanding the concept
information sources. Actually, we live in an analog world, and most sources of information.
! $ ! ! 2 ! 3
discrete sources are man made sources, for example, a source (say, a man)
"
Before we go on to develop a mathematical measure of information, let us develop an intuitive feel for it.
Read the following sentences:
(A) Tomorrow, the sun will rise from the East.
(B) The phone will ring in the next one hour.
(C) It will snow in Delhi this winter.

3 !
any information. It is a sure-shot thing. Everybody knows that the sun rises from the East and the
probability of this happening again is almost unity (“Making predictions is risky, especially when it
involves the future.” - N. Bohr). Sentence (B) appears to carry more information than sentence (A).
3 ! 3 $
(unless the maintenance people are at work again). The last sentence probably made you read it over twice.
This is because it has never snowed in Delhi, and the probability of a snowfall is very low. It is interesting to
note that the amount of information carried by the sentences listed above have something to do with the
probability of occurrence of the events stated in the sentences. And we observe an inverse relationship.
Sentence (A), which talks about an event which has a probability of occurrence very close to 1 carries almost
no information. Sentence (C), which has a very low probability of occurrence, appears to carry a lot of
information (made us read it twice to be sure we got the information right!). The other interesting thing to
note is that the length of the sentence has nothing to do with the amount of information it conveys. In fact,
sentence (A) is the longest of the three sentences but carries the minimum information.
We will now develop a mathematical measure of information.

Consider a discrete random variable X with possible outcomes xi , i = 1, 2, …, n. The Self

Information of the event X = xi

Ê 1 ˆ
I (xi) = log Á = – log P (xi) (1.1)
Ë P ( xi ) ˜¯

We note that a high probability event conveys less information than a low probability event. For an event
with P (xi) = 1, I (xi" 9 & vice versa), a
random variable with a higher degree of uncertainty contains more information. We will use this correlation
between uncertainty and information for physical interpretations throughout this chapter.
14 ° Information Theory, Coding and Cryptography

The units of I(xi) are determined by the base of the logarithm, which is usually selected as 2 or e.
When the base is 2, the units are in bits and when the base is e, the units are in nats (natural
" & £ P(xi) £ 1, I(xi) ≥ &! i.e., self-information is non-negative. The following two
examples illustrate why a logarithmic measure of information is appropriate.

Example 1.1 Consider a binary source which tosses a fair coin and produces an output equal to
% & ; ! P(1) = P &" 9 & < 3
of each output from the source is
I (xi) = – log2 P (xi) = – log2 & <" 9 % % >"
Indeed, we have to use only one bit to represent the output from this binary source (say, we use a 1
? & 3" @ ! 0
statistically independent, i.e., the source is memoryless. Consider a block of m binary digits. There are
2m possible m-bit blocks, each of which is equally probable with probability 2 –m. The self information
of an m-bit block is
I(xi) = – log2P(xi) = – log22 –m = m bits (1.3)
Again, we observe that we indeed need m bits to represent the possible m-bit blocks. Thus, this
logarithmic measure of information possesses the desired additive property when a number of source
outputs are considered as a block.

Example 1.2 Consider a discrete, memoryless source (source C) that generates two bits at a
time. This source comprises two binary sources (sources A and B) as mentioned in Example 1.1, each
source contributing one bit. The two binary sources within the source C are independent. Intuitively, the
information content of the aggregate source (source C) should be the sum of the information contained
in the outputs of the two independent sources that constitute this source C. Let us look at the infor-
mation content of the outputs of source C 3 G&&! &%! %&! %%H!
a probability P(C) = P(A)P(B" 9 & <" & <" 9 & ><! A and B are independent. The
information content of each output from the source C is
I(C) = – log2 P(xi)
= – log2 & ><" 9 > % +"
We have to use two bits to represent the output from this combined binary source. Thus, the logarithmic
measure of information possesses the desired additive property for independent events.

Next, consider two discrete random variables X and Y with possible outcomes xi, i = 1, 2, …, n. and yj,
j = 1, 2, …, m respectively. Suppose we observe some outcome Y = yj and we want to determine the amount
of information this event provides about the event X = xi, i = 1, 2, …, n, i.e., we want to mathematically
represent the mutual information. We note the two extreme cases:
(i) X and Y are independent, in which case the occurrence of Y = yj provides no information about
X = xi.
(ii) X and Y are fully dependent events, in which case the occurrence of Y = yj determines the
occurrence of the event X = xi.
#
P (X = xi | Y = yj) = P(xi | yj) (1.5)
divided by the probability P(X = xi) = P(xi). (1.6)
 Source Coding ° 15

The Mutual Information I(xi; yj) between xi and yj

Ê P (xi | y j ) ˆ
I(xi; yj) = log Á (1.7)
Ë P (xi ) ˜¯
As before, the units of I(xi ; yj) are determined by the base of the logarithm, which is usually selected
as 2 or e. When the base is 2 the units are in bits. Note that
P (xi | yj ) P ( xi | yj ) P ( y j ) P (xi , yj ) P ( yj | xi )
= = = % /"
P (xi ) P (xi )P ( y j ) P (xi )P ( y j ) P ( yj )

Ê P (xi | y j ) ˆ Ê P ( y j | xi ) ˆ
Therefore, I(xi ; yj) = log Á ˜ = log Á ˜ = I (yj ; xi" % *"
Ë P (xi ) ¯ Ë P( yj ) ¯

The physical interpretation of I(xi; yj) = I(yj ; xi) is as follows:

The information provided by the occurrence of the event Y = yj about the event X = xi is identical to the
information provided by the occurrence of the event X = xi about the event Y = yj.
Let us now verify the two extreme cases:
(i) When the random variables X and Y are statistically independent, P(xi | yj) = P(xi), which leads to
I(xi ; yj" 9 &
(ii) When the occurrence of Y = yj uniquely determines the occurrence of the event X = xi, P(xi | yj) = 1,
and the mutual information becomes
Ê 1 ˆ
I (xi; yj) = log Á = – log P(xi" % %&"
Ë P ( xi ) ˜¯
This is the self information of the event X = xi.
3 !

Example 1.3 Consider a binary symmetric channel (BSC) as shown in Fig. 1.1. A binary channel
0 Q % & 1–p
0 p 0
from the transmitter (Tx) to the receiver (Rx). It
makes an error occasionally, with probability p.
Tx Rx
#U X 2 % & vice-versa with
equal probability. Let X and Y be binary random
variables that represent the input and output of 1–p p
1 1
this BSC. Let the input symbols be equally likely, Channel
and the output symbols depend upon the input
Fig. 1.1 A binary symmetric channel.
according to the channel transition probabilities
are as follows:
P (Y 9 & Y X 9 &" 9 % Z p.
P (Y 9 & Y X = 1) = p.
P (Y = 1 | X = 1) = 1 – p.
P (Y = 1 | X 9 &" 9 p.
2 " 0 U X
is p. From the channel transition probabilities we have
16 ° Information Theory, Coding and Cryptography

P(Y 9 &" 9 P(X 9 &".P(Y 9 & Y X 9 &" [ P(X = 1).P(Y 9 & Y X = 1)

9 & < % Z p" [ & < p" 9 & <
and P(Y = 1) = P(X 9 &".P(Y = 1 | X 9 &" [ P(X = 1).P(Y = 1 | X = 1)
9 & < p" [ & < % Z p" 9 & <
Let us suppose that we are at the receiver end and we want to comment about what was transmitted at
the transmitter, having observed what is received at the receiver end. The mutual information about the
occurrence of the event X 9 & 0 Y9& 0
Ê P (Y = 0| X = 0) ˆ Ê1 - pˆ
I(x&; y&)= I &\ &" 9 log 2 Á = log 2 Á = log22(1 – p).
Ë P (Y = 0) ¯ ˜ Ë 0.5 ˜¯
Ê P (Y = 0| X = 1) ˆ Ê pˆ
Similarly, I(x1; y&) = I %\ &" 9 log 2 Á = log 2 Á = log22p.
Ë P (Y = 0) ˜¯ Ë 0.5 ˜¯
] ! p 9 &! i.e., it is an ideal channel (noiseless). In that case
I(x&; y&) = I &\ &" 9 22(1 – p) = 1 bit.
Hence having observed the output we can determine what was transmitted with certainty. Recall that
the self-information about the event X = x& was 1 bit. However, if p 9 & <!
I(x&; y&) = I &\ &" 9 22(1 – p) = log2> & <" 9 &

This implies that having observed the output, we have no information about what was
transmitted. Thus, it is a useless channel. For such a channel, there is no point in observing
the received symbol and trying to make a guess as to what was sent. Instead, we can also
toss a fair coin at the receiver in order to estimate what was sent!
Suppose we have a channel where p 9 & % ; !
I(x&; y&) = I &\ &" 9 22(1 – p) = log2> & *" 9 & /+/

Example 1.4 Let X and Y be binary random variables that represent the input and output of
a binary channel shown in Fig. 1.2. Can you think 1 – p0
0 p1 0
of a real-life channel that has unequal crossover
% &_]
be equally likely, and the output symbols depend Tx Rx
upon the input according to the channel transition
probabilities: p
1 – p1 0
1 1
P(Y 9 & Y X 9 &" 9 % Z p& Channel
P(Y 9 & Y X = 1) = p1.
Fig. 1.2 A binary channel with asymmetric
P(Y = 1 | X = 1) = 1 – p1. probabilities.
P(Y = 1 | X 9 &" 9 p&
From the channel transition probabilities we have
P(Y 9 &" 9 P(X 9 &".P(Y 9 & Y X 9 &" [ P(X = 1).P(Y 9 & Y X = 1)
9 & < % Z p&" [ & < p1" 9 & < % Z p& [ p1)
and P(Y = 1) = P(X 9 &".P(Y = 1 | X 9 &" [ P(X = 1).P(Y = 1 | X = 1)
9 & < p&" [ & < % Z p1" 9 & < % Z p1 [ p&).
 Source Coding ° 17

Suppose that we are at the receiver end and we want to comment about what was transmitted at the
transmitter, having observed what is received at the receiver. The mutual information about the occur-
rence of the event X 9 & 0 Y9& 0
Ê P (Y = 0| X = 0) ˆ Ê 1 - p0 ˆ Ê 2(1 - p0 ) ˆ
I(x&; y&) = I &\ &" 9 log 2 Á = log 2 Á = log 2 Á
Ë P (Y = 0) ˜¯ Ë 0.5(1 - p0 + p1) ˜¯ Ë 1 - p0 + p1 ˜¯
Ê P (Y = 0| X = 1) ˆ Ê 2 p1 ˆ
Similarly, I(x1; y&) = I %\ &" 9 log 2 Á ˜ = log 2 Á
Ë P (Y = 0) ¯ Ë 1 - p0 + p1 ˜¯

The Conditional Self-Information of the event X = xi given Y = yj

Ê 1 ˆ
I(xi | yj) = log Á ˜ = – log P(xi | yj) (1.11)
Ë P ( xi | yj ) ¯
Thus, we may write I(xi; yj) = I(xi) – I(xi | yj) (1.12)

The conditional self-information can be interpreted as the self-information about the event
X = xi having observed the event Y = yj. Recall that both I(xi) ≥ 0 and I(xi | yj) ≥ 0. Therefore,
I(xi; yj) < 0 when I(xi) < I(xi | yj) and I(xi; yj) > 0 when I(xi) > I(xi | yj). Hence, mutual
information can be positive, negative or zero.

Example 1.5 Consider the BSC discussed in Example 1.3. The plot of the mutual information
I(x ; y ) versus the probability of error, p is given in Fig. 1.3.
It can be seen from Fig. 1.3 that I(x ; y ) is negative for p
follows. A negative mutual information implies that having observed Y = y , we must avoid choosing
X = x as the transmitted bit.
For p I(x ; y1) = I 22(p) = log2 ! "
This shows that the mutual information between the events X = x and Y = y1 is negative for p
For the extreme case of p = 1, we have
I(x ; y1) = I 22(p) = log22(1) = 1 bit

Fig. 1.3 The plot of the mutual information I(x0; y0) versus the probability of error, p.
18 ° Information Theory, Coding and Cryptography

# $ $ p = 1). This implies that if y1 is observed

at the receiver, it can be concluded that x was actually transmitted. This is actually a useful channel with
%" & ' *+ - $ "

1.3 Average Mutual Information and Entropy LO 1

So far we have studied the mutual information associated with a pair of events xi and yj which are the possible
outcomes of the two random variables X and Y ' ## + $ /+ + 4 /
between the two random variables. This can be obtained simply by weighting I(xi; yj) by the probability of
occurrence of the joint event and summing over all possible joint events.

The Average Mutual Information between two random variables X and Y is given by
n m n m P (xi , y j )
I(X; Y) = Â Â P (xi , yj ) I (xi ; yj ) = Â Â P (xi , yj ) log P (x ) P ( y )
i =1 j =1 i =1 j =1 i j
n m P ( y j | xi )
= Â Â P ( xi ) P ( yj | xi ) log P( yj )
i =1 j =1
m n P ( xi | yj )
= Â Â P ( yj ) P ( xi | yj ) log (1.13)
j =1 i =1 P ( xi )
For the case when X and Y are statistically independent, I(X; Y i.e., there is no average mutual
information between X and Y. An important property of the average mutual information is that I(X; Y) ≥
with equality if and only if X and Y are statistically independent.

The Average Self Information of a random variable X

n n
H(X) = Â P (xi ) I (xi ) = -Â P (xi ) log P (xi ) 6
i =1 i =1

When X represents the alphabet of possible output letters from a source, H(X) represents the average
information per source letter. In this case H(X) is called the entropy. The entropy of X can be interpreted
Ê 1 ˆ
as the expected value of log Á . The term entropy has been borrowed from statistical mechanics,
Ë P ( X ) ˜¯
where it is used to denote the level of disorder in a system. It is interesting to see that the Chinese character
for entropy looks like !
Ê 1 ˆ
We observe that since 0 £ P (xi ) £ 1 , log Á ≥ 8 H(X) ≥ .
Ë P( xi ) ˜¯

Example 1.6 Consider a discrete binary source that emits a sequence of statistically independent
/" + + # " " p or 1 with a probability 1 – p. The entropy of this binary
source is
 Source Coding ° 19

1
H(X) = - Â P (xi ) log P (xi )
i =0
= – p log2(p) – (1 – p) log2(1 – p) (1.15)
The plot of the binary entropy function versus p
$ 9 6
' " $ 4 / + $ + 4
binary entropy function reaches its maximum value
for p i.e. # " :+ <
In general, it can be shown that the entropy of a
discrete source is maximum when the letters from
Fig. 1.4 The entropy function, H(X) = – p log2(p)
the source are equally probable.
– (1 – p) log2(1 – p).

Example 1.7 > ? + # " @D G H JK L4 $

occurred with the same probability and was independent from the other letters, then the entropy per
letter would be log2 M 6 N " + + " + 8 # $ # < #
appear with equal probability. If we take into consideration the probabilities of occurrences of different
alphabets (normalized letter frequency), the entropy per letter, HL, would be
HL £ H(X) ª 6 6 " M
This is still an over estimate because the letters are not independent. For example Q is always followed
by U and the bigram TH is likely to occur frequently. Thus a better estimate of the entropy per letter
could be possibly obtained by looking at the bigrams. If X2 denotes the random variable of bigrams in
the English language, the upperbound on HL "
H (X 2)
HL £ ª 3.56 bits (1.17)
2
This logic can be extended to n-grams + 4 + "
n
H (X )
H L = lim V
n n
Even though the exact value of HL / " 4 + / $ #
for the English language
1 £ HL £ " X
So each letter in the English text gives at most 1.5 bits of information. Let us assume the value of
HL ª 1.25 bits. Thus the redundancy of the English language is
HL 1.25
REng = 1 - =1- ª 0.75
log 2 26 4.70
This suggests that, theoretically, the English text can be compressed without loss to one-fourth of its
size. It is interesting that most natural languages in the world have similar redundancies.
20 ° Information Theory, Coding and Cryptography

speaks 60 words per minute and the average number of letters per word is 6. The average
number of letters spoken per second in this case is 6 letters per second. Assuming each letter
carries 1.25 bits of information, the information rate of an average speaker is 7.5 bits/second.
If each letter is represented by 5 bits, the bit rate of an average speaker is 30 bits/s. However, the typical data
rate requirement for speech is 32 kilobits/s!

INDUSTRIAL RELEVANCE
# '0 0 ( 6
0 ' 7 &08 ' ( ( ' ( 0
6 '
( 0 0 7 &08 ' ( 0
/ *9"#4+ # '0
: 12# 6; <==> ? 7 &08 1"#4 - 6, 6 - 57>@A 1"#4
?7== A?== B== 08 26: 7 &08 0 1"#489"#4

The Average Conditional Self Information called the Conditional Entropy

as n m
1
H(X|Y) = Â Â P (xi , yj )log P (x | y ) (1.21)
i =1 j =1 i j

4 4 # H(X|Y) is the information (or uncertainty) in

X after Y " $ G 4 H(X|Y) and H(X|Y) we can write
I(X; Y) = H(X) – H(X|Y) = H(Y) – H(Y|X) (1.22)

We make the following observations.

(i) Since I(X; Y) ≥ / H(X) ≥ H(X|Y).
(ii) The case I(X; Y / H(X) = H(X|Y), which is possible if and only if X and Y
are statistically independent.
(iii) Since H(X|Y) is the average amount of uncertainty (information) in X after we observe
Y and H(X) is the average amount of uncertainty (self information) of X, I(X; Y) is the
average amount of uncertainty (mutual information) about X having observed Y.
(iv) Since H(X) ≥ H(X|Y), the observation of Y does not increase the entropy (uncertainty). It
can only decrease the entropy. That is, observing Y cannot reduce the information about
X, it can only add to the information. Thus, conditioning does not increase entropy.

The Joint Entropy of a pair of discrete random variables (X, Y) with a joint distribution
p(x, y n m
H (X, Y) = - Â Â p ( xi , y j ) log p ( xi , yj ) (1.23)
i =1 j =1
 Source Coding ° 21

G + / / 4 H(X), H(X, Y) and H(X|Y) we obtain the following chain rule:

H(X, Y) = H(X ] H(Y|X) = H(Y ] H(X|Y 6
9 / 6 # "
I(X; Y) = H(X ] H(Y) – H(X, Y) (1.25)
Finally, we note that
I(X; X) = H(X) – H(X|X) = H(X) (1.26)
^ # _ $ 4` 4 /
between entropy and the mutual information can be expressed neatly using a Venn diagram, as depicted in
Fig. 1.5.
H (X, Y )

H (X ) H (X |Y ) I (X; Y ) H (Y |X ) H (Y )

Fig. 1.5 The relationship between entropy and the mutual information.

Example 1.8 > G}> + ?^ / ! ~ + /" "  €#

probability q  €# " " q as shown in Fig. 1.6. Please note that we are dealing with
two sets of probabilities:
(i) The source symbol probabilities (input probabilities to the channel): {q, 1 – qK
(ii) The BSC crossover probabilities (channel transition probabilities): {p, 1 – pK
These two sets of probabilities are independent.
The entropy of this binary source is
1
H(X) = - Â P (xi ) log P (xi ) = – q log2(q) – (1 – q) log2(1 – q).
i =0
The conditional entropy is given by
n m
1
H(X|Y) = Â Â P (xi , yj ) log P (x | y ) (1.27)
i =1 j =1 i j

In order to calculate the values of H(X|Y), we can make use of the following equalities:
P (xi , yj) = P (xi | yj) P( yj) = P ( yj | xi) P(xi V
22 ° Information Theory, Coding and Cryptography

Probability 1–p
q 0 0
p

Tx Rx

1–p p
1–q 1 1
Channel

Fig. 1.6 A binary symmetric channel (BSC) with input symbols probabilities equal to q and 1 – q.
The plot of H(X|Y) versus q is given in Fig. 1.7 with p as the parameter.

H(X/Y) I(X;Y)
1 1
p = 0.5 p = 0, 1
0.8 0.8

p = 0.2, 0.8 0.6 p = 0.1, 0.9

0.6
H (X ) H (X )

0.4 p = 0.1, 0.9 0.4

0.2 p = 0.3, 0.7

0.2
q 0 q
0 0 0.2 0.4 0.6 0.8 1
0 0.2 0.4 0.6 0.8 1

Fig. 1.7 The plot of the conditional entropy Fig. 1.8 The plot of the average mutual
H(X|Y) versus q. information I(X; Y) versus q.

The average mutual information I(X; Y

increase the parameter p I(X; Y) decreases. Physically it implies that, as we make the
channel less reliable (increase the value of p £
X (at the transmitter end) and the random variable Y (receiver end) decreases.

1.4 Information Measures for Continuous

Random Variables LO 1
! " #
random variables. Let X and Y be random variables with joint probability density function (pdf) p(x, y) and
marginal pdfs p(x) and p(y). The average mutual information between X and Y !

The Average Mutual Information between two continuous random variables X and Y
!
p ( y | x) p ( x)
I(X;Y) = Ú Ú p ( x) p ( y | x) log
p ( x) p ( y )
dxdy $ %&

!
from discrete random variables to continuous random variables, but the concept and physical
interpretation cannot. The reason is that the information content in a continuous random variable
 Source Coding ° 23

" ! ' ! "

" ! ! ' "
the differential entropy.

The Differential Entropy of a continuous random variable X !

h(X) = - Ú p ( x) log p ( x)dx $ *

Again, it should be understood that there is no physical meaning attached to the above quantity. We
" # !

The Average Conditional Entropy of a continuous random variable X given Y is

!
h(X | Y) = – Ú Ú p ( x, y ) log p ( x | y ) dxdy (1.31)

The average mutual information can be expressed as

I(X; Y) = h(X) – h(X|Y) = h(Y) – h(Y|X) (1.32)

Following is the list of some properties of differential entropy:

n
1. h (X1, X2, …, Xn) = Â h( X i | X1, X 2 ,º, X i-1). This is the chain rule for differential entropy.
i =1
2. h(X + c) = h(X), i.e., translation does not alter the differential entropy.
3. h(aX) = h(X + ,a|.
- X and Y are independent, then h(X + Y) ≥ h(X). This is because h(X + Y) ≥ h(X + Y | Y) =
h(X | Y) = h(X).

1.5 Relative Entropy LO 1

. ' / $ " 01 "
is used as a measure of distance between two distributions.

The Relative Entropy or Kullback Leibler (KL) Distance between two

probability mass functions p(x) and q(x !
Ê p ( x) ˆ
D( p || q ) =  p ( x) log ÁË q ( x) ˜¯ (1.33)
xŒX

Ê p( x) ˆ
It can be interpreted as the expected value of log Á .
Ë q ( x) ˜¯

Example 1.9 Consider a Gaussian distribution p(x) with mean and variance given by (m1, s12),
and another Gaussian distribution q(x) with mean and variance given by (m2, s22). Using (1.33), we can
! 23 4
24 ° Information Theory, Coding and Cryptography

1 È s 2 Ê m - m1 ˆ Ês2ˆ˘
2

D(p|| q) = Í 12 + Á 2 ˜ - 1 - log 2 Á 12 ˜ ˙ $ *-
2 Ís 2 Ë s 2 ¯ Ë s 2 ¯ ˙˚
Î
The distance becomes zero when the two distributions are identical, i.e., m1 = m2 and s12 = s22. It is
interesting to note that when m1 π m2 , the distance is minimum for s12 = s22. This minimum distance is
given by 2
1 Ê m 2 - m1 ˆ
Dmin (p || q) = Á
2 Ë s 2 ˜¯
. 23 ! s12 Æ or s22 Æ , that is, if either of the distributions tends
to the Dirac delta.

The average mutual information can be seen as the relative entropy between the joint distribution, p(x, y),
and the product distribution, p(x)p(y), i.e.,
I (X; Y) = D (p(x, y) || p(x) p(y)) (1.35)
We note that, in general, D (p || q) π D (q || p). Thus, even though the relative entropy is a distance measure,
it does not follow the symmetry property of distances. To overcome this, another measure, called the Jensen
5 ! "

The Jensen Shannon Distance between two probability mass functions p(x) and q(x)
!
1 1
JSD (p || q) = D ( p || m) + D (q || m) (1.36)
2 2
1
where m = ( p + q ).
2

£ JSD (p || q) £ 1. Note that Jensen Shannon distance is

sometimes referred to as Jensen Shannon divergence or Information Radius in literature.

A function f is said to be convex if

f (lx1 l) x2) £ lf (x1 l) f (x2) (1.37)
£ l £ 1 and all x1 and x2 contained in the convex domain of f. On the other hand, for a function to
be concave, it must satisfy
f (lx1 l) x2) ≥ lf (x1 l) f (x2)
Replacing ¢ £ ¢ by ¢ < ¢ in (1.37) and ¢ ≥ ¢ by ¢ > ¢ ! strictly convex
and strictly concave respectively. For any convex function f, we have the Jensen’s inequality given by
[ f (X )] ≥ f ( [X ]) where [.] is the expectation operator.

One can show that the relative entropy D(p || q) is convex in the pair (p, q), while the entropy function
H(p) is a concave function of p. The concavity of H(p) is also evident from observing the binary entropy
! " # $ % !
H (lp1 l) p2) ≥ lH(p1) ( l) H(p2) &
" & ' * " "*
is always larger. We also know from thermodynamics that mixing of two ideal gases of equal entropy results
in a gas with higher entropy due to the irreversible processes of expansion of the two gases! Again, consider
 Source Coding ° 25

X and Y to be two independent real-valued random variables. Let Z / 0 *

Z = X Y (that is, we add results of experiments X and Y together). One can show that H(Z) ≥ H(X) and
H(Z) ≥ H(Y). Thus addition of independent random variables adds uncertainty, which is intuitive.

Learning Review
YOU ARE NOW READY TO ATTEMPT THE FOLLOWING QUESTIONS LINKED TO LO 1:
4 589 ! " : ; ; ? # ! S
entropy, H(X).
2. A source, X 0 * " " * !! ! / * S
P(xi) = 2-i, i = 1, 2, 3, …. What is the average self information, H(X ! @
3. Consider a geometrically distributed random variable X with P(xi) = p(1 – p)i-1, i = 1, 2, M
3, … What is the average self information, H(X ! @
$ 4 N identical unfair coins, with the probability of getting a head P(H) = p. S
Suppose these N unfair coins are tossed simultaneously and we are interested in the event
of getting at least one head.
B ! ! / * * / @
(ii) What happens when N Æ in (i). Explain what it physically means.
5. The following table shows the joint probability distribution of two random variables X M
and Y with respective values xi and yi. Calculate H(X), H(Y), H(X,Y), H(X | Y) and H(Y | X).
x1 x2 x3
y1 1/2 H H
y2 H H

6. Calculate the differential entropy, H(X), of the uniformly distributed random variable X S
with the pdf:
ÏÔa -1 (0 £ x £ a )
p(x) = Ì
ÔÓ 0 (otherwise)
Plot the differential entropy, H(X), versus the parameter a MaM 4
result.

If you have successfully solved the above problems,

! " # $

The answers are available here. Scan to check

Or
Visit [Link]

# % % & #
S Simple: Level 1 and Level 2 Category
M Medium: Q / Q / $4 *
D % & # ' Level 5 and Level 6 Category
26 ° Information Theory, Coding and Cryptography

1.6 Source Coding Theorem

! '" 0! " 0!
LO 2 coding) of symbols generated by a source. The primary motivation is the
! " 0! " * 9 ""
State and prove the source a discrete memoryless source (DMS) outputs a symbol every t seconds.
coding theorem and discuss X ! * ! 0 * xi, i = 1, 2, …, L,
occurring with probabilities P(xi), i = 1, 2, …, L. The entropy of this DMS
in bits per source symbols is
L
H(X) = - Â P (xi ) log 2 P (xi ) £ log2L $
j =1
The equality holds when the symbols are equally likely. It implies that the average number of bits per
source symbol is H(X) and the source rate is H(X)/t bits/s.
Now let us suppose that we wish to represent the 26 letters in the English alphabets using bits. We observe
that 25 = 32 > 26. Hence, each of the letters can be uniquely represented using 5 bits. This is an example of a
Fixed Length Code (FLC). Each letter has a corresponding 5 bit long codeword.

A code is a set of vectors called codewords.

9 "" ! * ! 589 " * ! 0 *

xi, i = 1, 2, …, L. The number of binary digits (bits) R required for unique coding when L is a power of 2 is
R = log2 L $
and, when L is not a power of 2, it is
R = ÍÎlog 2 L ˙˚ + 1 $
As we saw earlier, to encode the letters of the English alphabet, we need R = ÍÎlog 2 26˙˚ + 1 = 5 bits.
% 0' length code for the English alphabet suggests that each of the letters in the alphabet is
equally important (probable) and hence each one requires 5 bits for representation. However, we
know that some of the letters are less common (x, q, z etc.) while some others are more frequently used
(s, t, e etc.). It appears that allotting equal number of bits to both the frequently used letters as well as not so
commonly used letters is not 0! * " ! / * "
more frequently occurring letters by fewer numbers of bits and represent the less frequently occurring letters
by larger number of bits. In this manner, if we have to encode a whole page of written text, we might end
" / \ ! * ^ *" 0!
method is to use a Variable Length Code (VLC).

Example 1.10 9 "" / * 0 X " _`B

/ ! * % 0' !
Fixed length code
Letter Codeword Letter Codeword
A E
B F
C G
D H 111
A variable length code for the same set of letters can be
 Source Coding ° 27

Variable length code 1

Letter Codeword Letter Codeword

A E
B F
C G
D H 1111

9 "" / ! ~ _ €_5 4_€ % 0' /

representation of the pseudo sentence would be

Fixed Length Code Total bits = 21

Variable Length Code % „

Note that the variable length code uses fewer numbers of bits simply because the letters appearing
more frequently in the pseudo sentence are represented with fewer numbers of bits.
\ † * / ! 0 X " ~
Variable length code 2

Letter Codeword Letter Codeword

A E
B 1 F 11
C G
D H 111

% ! / ! "" 0! "

Variable Length Code 1 % „

Variable Length Code 2 % „&

However there is a problem with VLC2. Consider the sequence of bits 0 1001 0001 which
is used to represent A BAD CAB in the second variable length coding scheme. We could
regroup the bits in a different manner to have [0] [10][0][1] [0][0][01] which translates
"# $# "# "# $# "# "# "# $#
for A BAAB AAAB ! Obviously there is a problem with the unique decoding of the code. We
have no clue where the codeword of one letter (symbol) ends and where the next one begins, since the
lengths of the codewords are variable. However, this problem does not exist with the VLC1. It can be seen
% & '* % + . So, as soon
as a sequence of bits corresponding to any one of the possible codewords is detected, we can declare that
symbol decoded. Such codes are called . There is no decoding delay in these codes.
If decoding delays are permitted, we may use in which the encoded string
could be generated by only one possible input string. However, one may have to wait until the entire
% ' & 4 * & 8 9 :? @
decodable code, hence not a code of any practical utility. The VLC1 is uniquely decodable, though less
economical in terms of bits per symbol.
28 ° Information Theory, Coding and Cryptography

A* +! ! ! " 0' * !
Such codes are also called Instantaneous Codes.

Our objective now is to devise a systematic procedure for constructing uniquely decodable, variable length
! 0! / " ! Q ! " *
0 * xi, i = 1, 2, …, L, occurring with probabilities P(xi), i = 1, 2, …, L. The average
" ! 0
L
R = Â ni P (xi ) $
i =1
where ni is the length of the codeword (in terms of number of bits) for the symbol xi .

. / % 0 1 # ' _ ! * 0! ! ' ! *!
with codewords having lengths n1 £ n2 £… nL * " 0' !

L
 2- n k
£ $$
k =1

# " / 0! ! 4 * " n = nL. This tree has

2n "! # & Q ! *! n1 0 ! c1. Since
! " 0' * ! " 0' ! ! ! n–n1 terminal

codes. This process continues until the last codeword is assigned at the terminal node n = nL. Consider the
node of order j < L. The fraction of number of terminal nodes eliminated is
j L
 2- nk <  2- n k
£ $;
k =1 k =1

% / ! ! " 0' !
embedded in the full tree of nL nodes. The nodes that are elimi-
c1
nated are depicted by the dotted arrow lines leading on to them
0 0
We now prove the necessary condition. We observe that in
0
the code tree of the order n = nL, the number of terminal nodes
eliminated from the total number of 2n terminal nodes is 1 c2
0
L
 2n- n
0 c3
k
£ 2n $‡ 1
k =1 1 c4
This leads to n
L
 2- n k
£ $ˆ Fig. 1.9 A binary tree of order nL.
k =1

\ ! * ' " " 0' ! / " ‰ M. For the proof we will have to
consider an M-ary tree instead of a binary tree. The inequality in this case would become
L
 M -n k
£ $
k =1
 Source Coding ° 29

Example 1.11 4 ! ! " 0' ! *

We start from the mother node and proceed toward
* # Q
Š ‹ ! / n00
Š ‹ % * !
from each node (binary tree). Let us label the upper n0 0
0
! Š ‹ ! Š ‹ !
1 n010
have also been mutually exchanged). First we follow
0
the upper branch from the mother node. We obtain our
n01 0 n0110
0 ! c1 „ n . Since we 1
n011
! ! " 0' ! ! 1 n0111
" 0' * ! !
Fig. 1.10
the daughter nodes generated as a result of the node using a binary tree.
labeled c1. We now proceed on the lower branch from
the mother node and reach the node n \ " ! "" ! 0 ! n .
We label this as the codeword c2 „ ! " /
the mother node). Following the lower branch from the node n , we ultimately reach the terminal nodes
n and n , which correspond to the codewords c3 „ c$ = 111 respectively. Thus, the binary
/ " 0' ! ~: ? €* ! ! " 0' ! #
this code
L
 2- n k
= 2–1 –2 –3 –3 „ ; ; ; ;„
k =1
B ! Π^ * 0

We now state and prove the noiseless Source Coding theorem, which applies to the codes that satisfy the
" 0' !
. 2 & ! ". ' Let X 589 0 "*
H(X) and the output symbols xk, k = 1, 2, …, L, occurring with probabilities P(xk), k = 1, 2, …, L. It is possible
! ! ! 0 " 0' ! / R 0 ^ *

H(X) £ R < H(X $&

First consider the lower bound of the inequality. For codewords that have length nk,
1 £ k £ L, the difference H(X) – R can be expressed as
L
1 L L
2- nk
H(X) –R = Â P (xk ) log 2 - Â P ( xk ) nk = Â P (xk ) log 2 P(x )
k =1 P(xk ) k =1 k =1 k

We now make use of the inequality ln x £ x – 1 to get

L Ê 2- nk ˆ
H(X) – R £ (log2 e) Â P (xk ) Á - 1˜
k =1 Ë P (xk ) ¯

Ê L ˆ
£ (log2 e) Á Â 2- nk - 1˜ £
Ë k =1 ¯
30 ° Information Theory, Coding and Cryptography

P (xk) = 2–nk for 1£ k £ L.

Thus, the lower bound is proved.
We now prove the upper bound. Let us select the codeword lengths nk such that 2–nk £ P(xk) < 2–nk . First
L L
consider 2 –nk £ P(xk). Summing both sides over 1 £ k £ L gives us  2- n £  P (xk ) =
k

k =1 k =1
! " P (xk) < 2 –nk . Take
logarithm on both sides to get
log2 P(xk) < – nk
or nk < 1 – log2 P(xk)
On multiplying both sides by P(xk) and summing over 1 £ k £ L we obtain
L L Ê L ˆ
 P (xk ) nk <  P (xk ) Á -  P (xk )log 2 P ( xk )˜
Ë k =1 ¯
k =1 k =1

or R < H(X#
Thus, the upper bound is proved.
! $
the minimum number of bits required to represent the source symbols on an average must be at least equal
% & ! ! R = H(X) for a certain source X, we
must abandon further search because we cannot do any better. The theorem also tells us that a source with
higher entropy (uncertainty) requires, on an average, more number of bits to represent the source symbols in
!

The 3% & & ! !

h = H (X ) ' * #
R

% ! ! h£ !
of symbols leads to compression of data. Source coding is primarily used for compression of data (speech,
text, facsimile, image, video etc.).

Example 1.12 Consider a source X which generates four symbols with probabilities P(x1# + *
P(x2) + - P(x3) + P(x/) +
4
H(X) = - Â P (xk ) log 2 P (xk ) + 04* $
k =1
5 ! 6 7 &
codeword length, R is given by
4
R = Â nk P (xk ) + ' *# 8' -# -' # -' #+ 9 $
k =1

Thus, we have H(X) £ R £ H(X#

! h + ' 04*: 9 # + ;; 8 > $ $ $ $
P(xk) = 2– nk i.e., P(x1) = 2 –1 + * P(x2) = 2 –2 + 8* P(x3) = 2–3 + 8* P(x/) = 2–3 + 8*
average codeword length, R + 9* $ + H(X). In this case, h = 1.
 Source Coding ° 31

Learning Review
YOU ARE NOW READY TO ATTEMPT THE FOLLOWING QUESTIONS LINKED TO LO 2:
? $ ! S
6 8 8 -7
2. Suppose we have to assign binary codewords to N symbols, with each codeword satisfying S
! % $ & 6 8 - @ N7A
3. We would like to encode a sequence of symbols that come from an alphabet with d - M
symbols. We want to encode symbols a1, a2, and a3 using codewords that are three bits
long. We want to encode symbols a/, a5, …, ad - using codewords that are eight bits
long. What is the maximum value of d for which this will be possible, if the code must be
$ A
/ 5 L of an instantaneous D-ary code for a given random D
variable X can be bounded by L ≥ HD(X), where HD(X) is the source entropy. When will
A
If you have successfully solved the above problems,
! " # $

The answers are available here. Scan to check

Or
Visit [Link]

1.7 Huffman Coding

E !
for a DMS with source symbols that are not equally probable. A variable LO 3
$ > ;*8 $
source symbol probabilities P(xi), i = 1, 2, …, L. The algorithm is optimal
in the sense that the average number of bits required to represent the source
$ & ! !
the Huffman coding algorithm are as follows: "
# $
(i) Arrange the source symbols in decreasing order of their probabilities. % #
(ii) Take the bottom two symbols and tie them together as shown in encoding.
Fig. 1.11. Add the probabilities of the two symbols and write it on
$ N $ $ P Q
P Q R
(iii) Treat this sum of probabilities as a new probability associated
with a new symbol. Again pick the two smallest probabilities, tie
them together to form a new probability. Each time we perform
the combination of two symbols we reduce the total number of Fig. 1.11 Combining probabilities in
symbols by one. Whenever we tie together two probabilities Huffman coding.
' # $ $ P Q P Q
32 ° Information Theory, Coding and Cryptography

(iv) Continue the procedure until only one probability is left (and it should be 1 if your addition is right!).
This completes the construction of the Huffman tree.

symbol. While tracing back the route, read out the labels on the branches. This is the codeword for the
symbol.
The algorithm can be easily understood using the following example.

Example 1.13 Consider a DMS with seven possible symbols xi, i = 1, 2, …, 7 and the corre-
sponding probabilities P(x1 "# P(x2 "" P(x3 $% P(x& # P(x5 & P(x6) = '
and P(x7 $ * + /// / / + 0
tree (Fig. 1.12).
Symbol Probability Self Codeword
Information
x1 "# $ &"&&
x2 "" $ 8998 $
x3 $% ' %&"9 $$
x& # " :"%8 $$$
x5 & & %&"9 $$$$
x6 ' 8 %&"9 $$$$$
x7 $ % %&"9 111111
/ ;
symbol. For the sake of illustration we show the route for the symbol x& / // # /
/ * / $$$
The entropy of the source is found out to be
7
H(X) = - Â P (xk ) log 2 P (xk ) = 2.1152 bits
k =1

Fig. 1.12
and the average number of binary digits per symbol is calculated to be
7
R = Â nk P (xk )
k =1
$ "# < ' "" < " $% < & # <8 & <% ' <% $ ' $# /
/ / / h ' $$8'>' $# 9#&#
 Source Coding ° 33

Example 1.14 This example shows that Huffman coding is not unique. Consider a DMS with
seven possible symbols xi, i = 1, 2, …, 7 and the corresponding probabilities P(x1 &% P(x2 "
P(x3 $' P(x& % P(x5 " P(x6 ' and P(x7 $

Symbol Probability Self Codeword Codeword

Information Length
x1 &% $ $' " 1 1
x2 " $ #"# 2
x3 $' " 8:9 $ 3
x& % & 8:9 $$ &
x5 " 8 8:9 $$$ 5
x6 ' 8 %&"9 $$$$ 6
x7 $ % %&"9 $$$$$ 6

The entropy of the source is found out to be

7
H(X) = - Â P (xk ) log 2 P (xk ) $ 9#:$ /
k =1

Fig. 1.13

and the average number of binary digits per symbol is calculated to be

7
R = Â nk P (xk )
k =1

$ &% < ' " < " $' < & % <8 " <% ' <% $ $ 99 /
/ / / h $ 9#:$>$ 99 99&
We shall now see that Huffman coding is not unique. Consider the combination of the two
smallest probabilities (symbols x6 and x7 / / ? " / / ?
next higher probability corresponding to the symbol x5. So, for the second step, we may
choose to put this combined probability (belonging to, say, symbol x6¢) higher than, or lower than, the
34 ° Information Theory, Coding and Cryptography

symbol x5 @ / // * + /
the combination of x6¢ and x5 / // % / / ? x&. We again
have a choice whether to put the combined probability higher than, or lower than, the symbol x&. Each
/ / A/ / / +/ + B
G/+ $ $& / / /// ?
probability of the combined symbols at a higher level.

Fig. 1.14 ! !

Symbol Probability Self Codeword Codeword

Information Length

x1 &% $ $' " 1 1

x2 " $ #"# 2

x3 $' " 8:9 $$ 3

x& % & 8:9 $ $ &

x5 " 8 8:9 $ $ 5

x6 ' 8 %&"9 $ 6

x7 $ % %&"9 $ $ 6

The entropy of the source is found out to be

7
H(X) = - Â P (xk ) log 2 P (xk ) $ 9#:$ /
k =1
and the average number of binary digits per symbol is calculated to be
7
R = Â nk P (xk )
k =1

$ &% < ' " < " $' < & % <8 " <% ' <% $ $ 99 /
/ / / h $ 9#:$>$ 99 99& ? /
This also tells us that Huffman codes are not unique for a given set of probabilities.
Suppose we change the probabilities slightly and the new probabilities are given by P(x1 &'
P(x2 " P(x3 $8 P(x& # P(x5 " P(x6 ' and P(x7 $
 Source Coding ° 35

7
The entropy of this source is H(X) = - Â P (xk ) log 2 P (xk ) ' 8%9 / 0 / +/
k =1

Symbol Probability Self Codeword Codeword

Information Length

x1 &' 1.2515 1 1

x2 " $ #"# 2

x3 $8 ' #"# $ 3

x& # " :"%8 $$ &

x5 " 8 8:9 $$$ 5

x6 ' 8 %&"9 $$$$ 6

x7 $ % %&"9 $$$$$ 6

We observe that in spite of the change in the probability values, the Huffman code is the same as
7
generated by Fig. 1.13. The average codeword length R = Â nk P (xk ) ' : / /
k =1

this code is h ' 8%9>' : 9::9. Intuitively, we can say that the codeword lengths, which must
/ + + A + / /// / /
the input signals.
Let us now carry out a similar exercise for another set of symbol probabilities as shown in the given
table.
Symbol Probability Self Codeword Codeword
Information Length

x1 1/2 1 1 1

x2 1 22 2 2

x3 1 23 3 $ 3

x& 1 24 & $$ &

x5 1 25 5 $$$ 5

x6 1 26 6 $$$$ 6

x7 1 26 6 $$$$$ 6
36 ° Information Theory, Coding and Cryptography

A probability distribution is called -adic with respect to D if each of the probabilities is

equal to D-n for some integer n N // / / / / / D-adic.
7
The entropy of this source is H(X) = - Â P (xk ) log 2 P (xk ) $ 9%:: / +
7 k =1
of binary digits per symbol is R = Â nk P (xk ) $ 9%:: / / h = 1. The
k =1
Huffman coding scheme has been able to match the codeword lengths exactly to the probabilities (and
thus the self information) of the symbols. For example, if the self information of x5 is 5 bits, the Huffman
code has actually allocated a codeword of length 5 to this symbol. It is clear that to achieve optimality
(h = 1) the self information of the symbols must be integers, which in turn, implies that the probabilities
+ / ' / / / O P /
the codeword lengths to the symbol probabilities is done using Arithmetic code, which we will study
later in this chapter.

B / +/ P / /
blocks of B symbols at a time. In this case the bounds of the source coding theorem becomes
BH(X) £ RB < BH(X < $ $ 8$
since the entropy of a B-symbol block is simply BH(X), and RB is the average number of bits per B-symbol
block. We can rewrite the bound as
RB 1
H(X) £ < H(X < (1.52)
B B
RB
where ∫ R is the average number of bits per source symbol.
B

Thus, R can be made arbitrarily close to H(X) by selecting a large enough block B.

Example 1.15 Consider the source symbols and their respective probabilities listed in the given
table.
Symbol Probability Self Information Codeword
x1 & $ "'$9 1
x2 "8 $ 8$&%
x3 '8 ' $

For this code, the entropy of the source is

3
H(X) = - Â P (xk ) log 2 P (xk ) $ 88:9 /
k =1
 Source Coding ° 37

The average number of binary digits per symbol is

3
R = Â nk P (xk ) $ & <' "8 < ' '8 $% /
k =1
/ / / h $ 88:9>$ % 9#&"
We now group together the symbols, two at a time, and again apply the Huffman encoding algorithm.
The probabilities of the symbol pairs, in decreasing order, are listed in the table.
Symbol Pairs Probability Self Information Codeword
x1x1 $% ' %&"9 $
x1x2 $& ' :"%8 $
x2x1 $& ' :"%8 $
x2x2 $''8 " '9$ $$
x1x3 $ " "'$9 111
x3x1 $ " "'$9
x2x3 :#8 " 8$&% $
x3x2 :#8 " 8$&% $$
x3x3 %'8 & $$ $

For this code, the entropy is

9
2H(X) = - Â P (xk ) log 2 P (xk ) = 3.1177 bits
k =1
fi H(X $ 88:9 /
Note that the source entropy has not changed! The average number of binary digits per block (symbol
pair) is 9
RB = Â nk P (xk )
k =1
' $% <" $& <" $& < " $''8 < " $ <
& $ <& :#8 < & :#8 < & %'8
= 3.1775 bits per symbol pair
fi R " $##8>' $ 8::: /
/ / / h $ 88:9>$ 8::: 9:$' + / +
/ / + /

Example 1.16 Consider the source symbols and their respective probabilities listed in the table.

Symbol Probability Self Codeword

Information
x1 8 $ 1
x2 " $ #"#
x3 ' ' "'$9 $
38 ° Information Theory, Coding and Cryptography

For this code, the entropy of the source is

3
H(X) = - Â P (xk ) log 2 P (xk ) $ &:88 /
k =1
The average number of binary digits per symbol is
3
R= Â nk P (xk ) $ 8 <' " <' ' $8 /
k =1
/ / / h $ &:88 >$ 8 99 "
We now group together the symbols, two at a time, and again apply the Huffman encoding algorithm.
The probabilities of the symbol pairs, in decreasing order, are listed in the table.

Symbol Probability Self Codeword

Pairs Information

x1x1 '8 '

x1x2 $8 ' #"# $

x2x1 $8 ' #"# $$

x1x3 $ " "'$9 $

x3x1 $ " "'$9 $$

x2x2 9 " &#"9 $ $

x2x3 % & 8:9 $ $$

x3x2 % & 8:9 $$$

x3x3 & & %&"9 1111

For this code, the entropy is

9
2H(X) = - Â P (xk ) log 2 P (xk ) ' 9#$ /
k =1
fi H(X $ &:88 /

The average number of binary digits per block (symbol pair) is

9
RB = Â nk P (xk )
k =1

' '8 < " $8 < " $8 < " $ <" $ <& 9 <& % <
& % <& & " / /
fi R " >' $8 /
/ // h2 $ &:88 >$ 8 99 " B / + / + +
/ / / V0 / + " /
/ 0 / + / / h3 99"' X + / +
four letters at a time we see a further improvement (h& 99&%
 Source Coding ° 39

1.8 Shannon-Fano-Elias Coding LO 3

È 1 ˘
Codes that use the codeword lengths of l(x) = Ílog are called Shannon Codes. Shannon codeword
Í P ( x) ˙˙
+ / Y / ? / /? . In this
section we will study another simple method for constructing uniquely decodable codes based on Shannon-
Fano-Elias encoding technique. It uses the Cumulative Distribution Function to allocate the codewords.
/ / / / / /
F(x) = Â P( z) (1.53)
z£ x

where P(z) are the probability of occurrence of the z. The F(x)

cumulative distribution function consists of steps of size P(x),
/ G/+ $ $8 Z / /
distribution function as P(x)
1
F ( x) = Â P ( z ) + P ( x) $ 8&
z< x 2
where F ( x) represents the sum of the probabilities of all x
symbols less than x plus half the probability of the symbols x. 0 1 2
The value of the function F ( x) is the midpoint of the step
Fig. 1.15 The cumulative distribution
corresponding to x of the cumulative distribution function.
function.
Since probabilities are positive, F(x) π F(y) if x π y. Thus it is
possible to determine x given F ( x) merely by looking at the graph of the cumulative distribution function.
Therefore, the value of F ( x) can be used to code x.
In general, F ( x) / / ? / / / / F ( x) ,
/ / / @ F ( x) l(x) bits, denoted
by ÍÎ F ( x) ˙˚ N // / +
l (x)
1
F ( x) - ÍÎ F ( x) ˙˚l ( x ) £ l ( x ) (1.55)
2
È 1 ˘
If l(x) = Ílog + 1 , then,
Í P ( x) ˙˙
1 P ( x)
l ( x)
< = F ( x) – F(x – 1). (1.56)
2 2

This implies that ÍÎ F ( x) ˙˚ l (x) lies within the step corresponding to x, and l(x / / / x.
–l(x)
The interval corresponding to any codeword is of length 2 . From (1.55) we see that this interval is less
È 1 ˘
than half the height of the step corresponding to x. Since we use l(x) = Ílog + 1 bits to represent x, the
Í P ( x) ˙˙
expected length of this code is
ÊÈ 1 ˘ ˆ
R = Â P ( x) l ( x) = Â P ( x) Á Ílog ˙ + 1 < H (X ) + 2
P ( x) ˙ ˜¯
(1.57)
x x ËÍ
Thus the Shannon-Fano-Elias coding scheme achieves codeword length within two bits of the entropy.
40 ° Information Theory, Coding and Cryptography

Example 1.17 Consider the D-adic distribution given in the following table.

È 1 ˘
Symbol Probability F(x) F ( x) F ( x) (binary) l(x) = Ílog ˙ +1 Codeword
Í P ( x) ˙

x1 1/2 2

x2 1 22 3

x3 1 23

x 1 23 1 1111

The entropy for this distribution is 1.75 bits. However, the average codeword length for the Shannon-
Fano-Elias coding scheme is 2.75 bits. It is easy to observe that if the last bit from all the codewords is
deleted, we get the optimal code (Huffman code). It is worthwhile to note that unlike in Huffman coding
! " #$
probabilities and redo the exercise.

È 1 ˘
Symbol Probability F(x) F ( x) F ( x) (binary) l(x) = Ílog ˙ +1 Codeword
Í P ( x) ˙

x1 1 22 3
x2 1/2 2
3
x3 12

x 1 23 1 1111

We observe that the codewords obtained from the Shannon-Fano-Elias coding procedure is not
unique. The average codeword length is again 2.75 bits. However, this time we cannot get the optimal
code simply by deleting the last bit from every codeword. If we do so, the code no longer remains a
!% & #' *+ *, - . #!
algorithm for encoding and decoding called Arithmetic Coding.

1.9 Arithmetic Coding LO 3

/ # - ,% - 0 ##- . - # # .-
# & !% 4 & . 4

(a) 5 !% . - # # - # .-
are integers. The length matching may ascribe a codeword either longer than the self information
or shorter. The exact match is possible if and only if the self information is in integral number
of bits.
(b) 6# !% .
tree branches always take one bit. This is regardless of the fact whether the probabilities for
7 7 6 . 4 .
 Source Coding ° 41

(– log28 99 ! 8: 28 99
- 4 8 ¥ ; ¥ 3.32). The Huffman code still
needs one bit for each decision.
/ - 6 4 . ! .
# - ' .- -
accordance with the probability of that symbol. The more likely symbols reduce the range by less, and thus
add fewer bits to the message.

Example 1.18 Let our alphabet consists of only three symbols A, B and C with probabilities of
occurrence P(A9 < P(B9 < P(C9 < = ! > 9
proportional to their probabilities, as depicted in step 1 of Fig 1.16. Thus, the variable A corresponds
> 9 B > 9 C > 9 ?
that the lengths of these intervals are proportional to their probabilities. Next, suppose the input symbol
stream is B A C A @ = ! B. This is nothing but choosing the corresponding interval, i.e.,
> 9 ?
of occurrence. So, for the second step (see Fig. 1.16), the variable A > 9
variable B > 9 C > 9 ' %
symbol to arrive after B is A / > 9 &
. > 9# A > 9 # B, and the
> 9 # C. Now we look at the next symbol to encode, which is C. This corresponds
> 9 A # A, we are left with the interval
> 9 & - # B A C A is any number that lies within this interval. To
complete this example, we can say that the arithmetic code for the sequence B A C A
Next consider the decoding at the receiver. The receiver needs to know a-priori, the probabilities of
A, B and C. So, it will also have an identical number line, partitioned into three segments, proportional
to the probabilities of A, B and C " . + 4
- A . B B - B. So the 1st decoded
symbol is B. Now we split the segment corresponding to B into three sub-segments proportional to the
probabilities of A, B and C (exactly as we did at the encoder side). Again we map the received number
! # A. The decoding is instantaneous. Mechanically, we proceed
% .- C 4 D& #
to have a stopping criterion, or a pre-decided protocol to do so.

Fig. 1.16 ! B A C A.
42 ° Information Theory, Coding and Cryptography

1.10 The Lempel-Ziv Algorithm LO 3

Huffman coding requires symbol probabilities. But most real life scenarios do not provide the symbol
probabilities in advance (i.e., the statistics of the source is unknown). In principle, it is possible to observe
the output of the source for a long enough time period and estimate the symbol probabilities. However,
this is impractical for real-time application. Also, Huffman coding is optimal for a DMS source where the
occurrence of one symbol does not alter the probabilities of the subsequent symbols. Huffman coding is not
the best choice for a source with memory. For example, consider the problem of compression of written text.
= 4 - . 4 EF* G E * G E * * G 6 - - #!
use the statistical inter-dependence of the letters in the alphabet along with their individual probabilities of
' - ." - H & -
6 * *!% -
of universal source coding algorithms.
The logic behind Lempel-Ziv universal coding is as follows. The compression of an arbitrary sequence
# . # G G - previous 8 !% 9
& # - . . !% -
!% # # & 4 phrases. The phrases are
listed in a dictionary which stores the existing phrases and their locations. In encoding a new phrase, we
specify the location of the existing phrase in the dictionary and append the new letter. We can derive a better
understanding of how the Lempel-Ziv algorithm works by the following example.

Example 1.19 ' I = .

parsing it into comma-separated phrases that represent strings that can be represented by a previous
!%
& ! !% % #I

& - # # G % - # . % !%I

' # . E G E G ? % . %
. 0 # & # . - # !%
I

Continuing in this way we eventually parse the whole string as follows:

? # !
# # - 8 # %
- 9 ? % - # - # !%
phrase plus the new bit needed to create the new phrase. We will use parentheses and commas to separate
! J # & .I
8 98 98 98 98 98 98 98 9
 Source Coding ° 43

6 I8 9 8 9 8
9 8 9 8 9 8 9@
& # I & .
for this example is given in Table 1.1. In this case we have not obtained any compression, our coded
string is actually longer! However, the larger the initial string, the more saving we get as we move along,
!% F - - - 6 # H
# - - # ! - - -
by the information content of the document.

Table 1.1 Dictionary for the Lempel-Ziv algorithm.

Dictionary Dictionary Fixed Length

Location Content Codeword

11

111

The next question is what should be the length of the table. In practical application, regardless of the
# . $ & - . *
size of the dictionary. The encoder and decoder can update their dictionaries by periodically substituting the
less used phrases from their dictionaries by more frequently used ones.

INDUSTRIAL RELEVANCE
# $ ' 6 (
:-2C ( 6
( 0 ' ( A? 0 A / 0 '
: # $ ("
/ 0 0 >> ' 0
7<
44 ° Information Theory, Coding and Cryptography

The Graphics Interchange Format (GIF) is also based on Lempel-Ziv coding. In GIF, the image
! . - * .* & " - *H
coding is then applied to this string. However, the 2-D structure of the image is not used as an
& 5 ? 4V 85?V9 # - J *W # - .!
subtracting the pixel values of a row from the one above it, and then carrying out Lempel-Ziv coding on the
residual.
In the following section we will study another type of source coding scheme, particularly useful for
facsimile transmission and image compression.

1.11 Run Length Encoding LO 3

Run-length Encoding, or RLE is a technique used to reduce the size of a repeating string of characters.
This repeating string is called a run. Typically RLE encodes a run of symbols into two bytes, a count and a
symbol. RLE can compress any type of data regardless of its information content, but the content of data to
be compressed affects the compression ratio. RLE cannot achieve high compression ratios compared to other
compression methods, but it is easy to implement and is quick to execute. Run-length encoding is supported
.- - ! # - &6++ [5V C\5 5A] # %-

Example 1.20 Consider the following bit stream:

S < 8 9
& I !# G G # G i.e 8 9 8 9 8 9 '
- %- - - # -
8 9 8 9 8 9 & - I < I

R % E G & % %' & + K &

(black and white) are predominantly white. If we spatially sample these images for conversion
8 ' & Q U % "\ ^
Furthermore, if a given pixel is black or white, the chances are very good that the next pixel
will match. The code for fax machines is actually a combination of a run-length code and a
Huffman code. A run-length code maps run lengths into codewords, and the codebook is partitioned into two
+ ' & % % & % _`{ &
% % & " _} * & % _` & &
For example, a run of 205 pixels would be sent using the codeword for a run of length 192 (3 ¥ _`^
codeword for a run of length 13. In this way the number of bits needed to represent the run is decreased
' 4 8 %
into codewords of short length, further reducing the number of bits that need to be transmitted. Using this
% 8 & % % & & ` $ € $ :
to higher modem speeds, these compressions reduce the transmission time of a single page to less than a
minute.
 Source Coding ° 45

INDUSTRIAL RELEVANCE
% ( ( 49C ( 6 49C (
' ( 49 4 0 ( (' (
0 ( ' ;, 6 49C (
umbrella name for several possible image compression methods and a means to identify which
0 6 49C 0 0 (
52! 34"5 4-5

= # 5A] - . -
and not those parts that store the color palette and image information such as number of lines, pixels per line,
! -
The basic scheme is as follows. If a string of pixels are identical in color value, encode them as a special
4 " byte which contains the count followed by a byte with the value of the repeated pixel. If the pixel is not
repeated, simply encode it as the byte itself. Such simple schemes can often become more complicated in
practice. Consider that in the above scheme, if all 256 colors in a palette are used in an image, then, we need
all 256 values of a byte to represent those colors. Hence if we are going to use just bytes as our basic code
G . . $ 7 . }
# . # . % -# $ 7 - -
size of pathological images instead of compressing them.
& - - 5A] # - # # - . *
drawings (which was the primary intended output of their software) would not use all 256
' . -J - - # # .
Images with more colors will also probably get good compression, just not quite as good, with this scheme.

Example 1.21 5A] - # 8 %

# # - 9 - . . . # % .
numerical value. Consider the Table 1.2.

Table 1.2 " #

Pixel color Hex Code Binary Code

Value

1
2
3
: : :
BE
BF
46 ° Information Theory, Coding and Cryptography

+ 8 9 F .
- ! 8\'C 9 = #. $
. 6# \'C F . . $ & -
$ 7 . . - # 8# - 9 &
byte is then followed by the byte which represents the color. In fact, if we have a run of pixels of one of
.
reserved in this second, color code byte of a run coding byte pair.
6# # % % - . # ! %
then code additional runs of that pixel until we exhaust all pixels in the run. The next question is: how
- .# - D=
. - . & - # # -
appear as single pixels in the image and not part of runs, we expand the data by a factor of two. Luckily
this rarely happens!

6 % . # € . ! -
of bits to accurately represent an analog source. Anything fewer will only be an approximate representation.
We can choose to use fewer and fewer bits for representation at the cost of a poorer approximation of the
original signal (rule of thumb: there is no free lunch!). Thus, quantization of the amplitudes of the sampled
signals results in data compression. We would like to study the distortion introduced when the samples from
the information source are quantized.

Learning Review
YOU ARE NOW READY TO ATTEMPT THE FOLLOWING QUESTIONS LINKED TO LO 3:
A W\' ƒ „
(i) Determine the Huffman code for this source. S

(ii) Determine the average length R of the codewords.

= #! .h # D
/ W\' ƒ „ M
(i) Determine the ternary Huffman code for this source using symbols 1, 2 and 3.
(ii) Determine the average length R of the codewords and compare it with the binary case.
V ' *+ *, # . ƒ S
„ A - 0 ##-
W - " - H # # - M
€ F # - -

If you have successfully solved the above problems,

! " # $

The answers are available here. Scan to check

Or
Visit [Link]
 Source Coding ° 47

1.12 Rate Distortion Function

Although we live in an analog world, most of the communication takes
place in the digital form. Since most natural sources (e.g. speech, video etc.) LO 4
. ! - F J 0
# . - F ! - Underline the concept of the
# & ! # - rate distortion function and
never be perfect. the design of the optimum
Consider an analog message waveform x(t) which is a sample waveform
of a stochastic process X(t). Assuming X(t) is a bandlimited, stationary
process, it can be represented by a sequence of uniform samples taken at the Nyquist rate. These samples
are quantized in amplitude and encoded as a sequence of binary digits. A simple encoding strategy can be to
! L levels and encode every sample using
R = log2L bits if L is a power of 2, or
R = ÍÎlog 2 L ˙˚ ; #L # 8 9
6# F . - . . # - #! 6
order to represent the analog waveform more accurately, we need more number of levels, which would
- .- - # - & . ! - # .
‡ J # - - # 6G
a form of lossy data compression. Distortion implies some measure of difference between the actual source
samples {xk„ F J ƒ x�k „

The Squared-Error Distortion !

d (xk , x�k ) = (xk – x�k )2 8 9
p
In general a distortion measure may be represented as d (xk , x�k ) = xk - x�k .
The Hamming Distortion !
0 if xk = x�k
d ( xk , x�k ) = ÏÌ (1.61)
Ó1 if xk π x�k
Consider a sequence of n samples, Xn , and the corresponding n quantized values, X� n . Let d ( xk , x�k ) be
the distortion measure per sample (letter). Then the distortion measure between the original sequence and the
sequence of quantized values will simply be the average over the n source output samples, i.e.,
1 n
d (X n , X� n ) = Â d (xk , x�k ) (1.62)
n k =1
We observe that the source is a random process, hence Xn and consequently d (X n , X� n ) are random
= ! #

The Distortion between a sequence of n samples, Xn, and their corresponding n

quantized values, X� n !
1 n
D = [ d (X n , X� n ) ] = Â E[d (xk , x�k )] = [ d (xk , x�k ) ] (1.63)
n k =1
48 ° Information Theory, Coding and Cryptography

It has been assumed here that the random process is stationary. Next, let a memoryless source have
a continuous output X and the quantized output alphabet X� . Let the probability density function of this
continuous amplitude be p(x) and per letter distortion measure be d ( x, x� ) , where x Œ and x� Œ X� . We next
introduce the rate distortion function which gives us the minimum number of bits per sample required to
.- !

The minimum rate (in bits/source output) required to represent the output X of the
memoryless source with a distortion less than or equal to D is called the Rate Distortion Function R(D),
!
R(D) = min I ( X ; X� ) 8 9
p ( x� | x ): E [ d (X , X� )] £ D

where I ( X ; X� ) is the average mutual information between X and X� . R(D) is also known as
the Information Rate 5 &

Some interesting properties of the rate distortion function are as follow:

(i) R(D) is non-increasing in D
(ii) R(D) is convex
(iii) R8 9 £ H(X)
(iv) R(D9 < # D ≥ Dmax
We will now state (without proof) two theorems related to the rate distortion function.
. The minimum information rate necessary to represent the output of a discrete time, continuous
amplitude memoryless Gaussian source with variance s x2 , based on a mean square-error distortion measure
per symbol, is
Ï1 2 2
Ô log 2 (s x /D) 0 £ D £ s x
Rg (D) = Ì 2 (1.65)
Ô0 D > s 2
Ó x

Consider the two cases:

2
(i) D ≥ s x : For this case there is no need to transfer any information. For the reconstruction of
the samples (with distortion greater than or equal to the variance) one can use statistically
2
independent, zero mean Gaussian noise samples with variance D – s x .
2
(ii) D < s x : For this case the number of bits per output symbol decreases monotonically as D
increases. The plot of the rate distortion function is given in Fig. 1.17.
4

Rg(D) 2

0 D/s 2x
0 0.2 0.2 0.6 0.8 1
2
Fig. 1.17 " ! Rg(D) versus D/s x .
 Source Coding ° 49

. There exists an encoding scheme that maps the source output into codewords such that for
any given distortion D, the minimum rate R(D9 - #!
with an average distortion that is arbitrarily close to D.
Thus, the distortion function for any source gives the lower bound on the source rate that is possible for a
given level of distortion.

The Distortion Rate Function for a discrete time, memoryless Gaussian source is
!
2
Dg(R) = 2 –2R s x (1.66)

Example 1.22 For a discrete time, memoryless Gaussian source, the distortion (in dB) as a
function of its variance can be expressed as
Dg(R) = – 6R ; s x2 (1.67)
Thus, the mean square distortion decreases at a rate of 6 dB/bit. In other words, each bit reduces the
% . # #
The rate distortion function of a discrete time, memoryless continuous amplitude source with zero mean
! s x2 with respect to the mean square error distortion measure D is upper bounded as
1
R(D) £ log 2 (s x2 /D) £ D £ s x2 8 9
2
This upper bound can be intuitively understood as follows. We have seen earlier that for a given variance,
the zero mean Gaussian random variable exhibits the maximum differential entropy attainable by any random
variable. Hence, for a given distortion, the minimum number of bits per sample required is upper bounded
by the Gaussian random variable.
& % F I= # F J D6 .
F J - -J - . D= !
questions in the next section.

1.13 Optimum Quantizer Design LO 4

In this section, we look at optimum quantizers design. Consider a continuous amplitude signal whose
amplitude is not uniformly distributed, but varies according to a certain probability density function, p(x). We
wish to design the optimum scalar quantizer that minimizes some function of the quantization error q = x� - x ,
where x� is the quantized value of x. The distortion resulting due to the quantization can be expressed as
D = Ú f ( x� - x) p ( x) dx 8 9

where f ( x� - x) is the desired function of the error. An optimum quantizer is one that minimizes
D by optimally selecting the output levels and the corresponding input range of each output
level. The resulting optimum quantizer is called the Lloyd-Max quantizer. For an L-level
quantizer the distortion is given by
L xk
D = ÂÚ f ( x�k - x) p ( x)dx 8 9
xk -1
k =1
50 ° Information Theory, Coding and Cryptography

The necessary conditions for minimum distortion are obtained by differentiating D with respect to {xk„
and { x�k „ / # ## # . - # F
f ( x�k - xk ) = f ( x�k +1 - xk ), k = 1, 2, …, L – 1
xk (1.71)
Ú f ¢ ( x�k +1 - x) p ( x) dx , k = 1, 2, …, L
xk -1
For f (x) = x2, i.e., the mean square value of the distortion, the above equations simplify to
1
xk = ( x�k + x�k +1) , k = 1, 2, …, L – 1 (1.72)
2
xk
Úx k -1
( x�k - x) p ( x) dx < k = 1, 2, …, L

The non uniform quantizers are optimized with respect to the distortion. However, each quantized sample
is represented by equal number of bits (say, R 7 - 9 6 - #!
length coding. The discrete source outputs that result from quantization can be characterized by a set of
probabilities pk & #! 8 9
6 - # - # ## # -F J ! !% D, and
then compare the average number of bits required per sample.

Example 1.23 Consider an eight level quantizer for a Gaussian random variable. This problem

mean square error minimization, the values xk and x�k are listed in Table 1.3.
Table 1.3 Optimum quantization and Huffman coding.

x�k Huffman
Level, k xk P (xk)
Code
1 " #$% – 2.152 $
2 " & " '$$ #
3 " & " #& *
$ " *$&
5 & *$& 11
6 & #& *
7 #$% '$$ #
% 2.152 $

For these values, D + '$& " $ * -

0 4 8 0 8 %9 R = 3. On performing Huffman
coding, the average number of bits per sample required is RH + * %% 4 8
H(X: + * %* 4 8

1.14 Entropy Rate of a Stochastic Process LO 4

; < 0 0 0 0 > > > 0
8 0 < >
 Source Coding ° 51

in the selection of symbols. It would, therefore, be interesting to calculate the average information content
of a symbol in a long message. Suppose, the source X uses M symbols s1, s2, …., sM with associated proba-
bilities of occurrences p1, p2, …., pM. In a long message of length N, the symbol si will occur piN times. The
Ê 1ˆ
information content of the ith symbol will be log 2 Á ˜ bits. Thus, the piN occurrences of si will contribute
Ë pi ¯
Ê 1ˆ
an information content of pi N log 2 Á ˜ bits. Therefore, for all the M symbols, the total contribution will be
Ë pi ¯
M
Ê 1ˆ
ITot = Â pi N log 2 Á ˜ bits (1.73)
i =1 Ë pi ¯
The average information per symbol for the source X is obtained by
ITot
M
Ê 1ˆ
H(X) = = Â pi log 2 Á ˜ @ #$:
N i =1 Ë pi ¯
0 8 > >
A simple extension of the source coding theorem tells us that nH(X: 0 ; > ;
describe n independent and identically distributed random variable, each with entropy H(X). But, in the real
world, we do encounter random variables that are dependent. What if the random variables form a stationary
8 C

A stochastic process is said to be stationary if the joint distribution of any subset of the
sequence of random variables is invariant with respect to shifts in time index, i.e.,
P(X1 = x1, X2 = x2,… Xn = xn) = P(X Dm = x1, X* D m = x2,… XnDm = xn) (1.75)
for every shift m and for all x1, x2,… xn ŒX.

A discrete stochastic process X1, X2 , … is said to be a Markov Chain or a

Markov Process if, for n = 1, 2 …
P(Xn D = xnD | Xn = xn, Xn –1 = xn –1,…, X1 = x1) = P(Xn D = xn D | Xn = xn) (1.76)
for all x1, x2,… xn, xn D Œ X.
The probability density function of a Markov process can be written as
p(x1, x2,… xn) = p(x1) p(x2 | x1) p(x3 | x2)… p(xn | xn –1) (1.77)

Example 1.24 Suppose random variables X, Y and Z form a Markov chain and we denote it by
X Æ Y Æ Z. We observe from (1.77) that p(x, y, z) = p(x) p(y | x) p(z | y) = p(x, y) p(z |y). We can write

p(x, z | y) = p ( x, y, z ) = p ( x, y ) p ( z | y ) = p ( x | y ) p ( z | y ) @ #%:
p( y) p( y )
Thus, X Æ Y Æ Z implies X and Z are conditionally independent given Y. Also note that
XÆYÆZ ¤ZÆYÆX @ # :
52 ° Information Theory, Coding and Cryptography

Interestingly, if Z = f(Y) then X Æ Y Æ Z. Using the chain rule we can write

I(X; Y, Z) = I(X; Z: D I(X; Y|Z) = I(X; Y: D I(X; Z|Y: @ % :
But, X and Z are conditionally independent given Y and therefore I(X; Z|Y: + 0
I(X; Y) = I(X; Z: D I(X; Y|Z: @ % :

Since, I(X; Y|Z) ≥ ; X Æ Y Æ Z implies I(X; Y) ≥ I(X; Z). This shows that processing of data (recall
Z = f(Y)) cannot improve the inferences that can be drawn from it. Manipulating the data (X) can only
increase our uncertainty about X. This is also known as the Data Processing Inequality.
We now revert back to the entropy rate of a stochastic process. If we have a sequence of n random
variables, it is interesting to explore how the entropy of the sequence grows with n. Entropy rate is used to
0>

The Entropy Rate of a stochastic process X is given by

1
H(X) = lim H (X1 , X 2 , � X n ) @ %*:
n n
provided the limit exists.

Example 1.25 Let X1, X2, X3, … be independent and identically distributed (i.i.d.) random variables.
1 nH (X1)
In this case the entropy rate would be H(X) = lim H (X1 , X 2 , � X n ) = lim = H (X1) . Thus, the
n n n n
entropy rate of an i.i.d. source is the entropy any of its single symbols.
However, if X1, X2, X3, … form a sequence of independent but not identically distributed random
n
1 n
variables, then H (X1,X2,… Xn) = Â H (X i ) . Thus, the entropy rate is lim
n
 H (X i ) . If H (Xi) are
n i =1
i =1
unequal, it is possible to choose a sequence of distributions on X1, X2, X3, … such that the limit of
1 n
 H (X i ) does not exist.
n i =1

For Stationary Markov Chain, the entropy rate is given by

H (X) = lim H ( X n | X n -1 , � , X1) = lim H (X n | X n -1) = H (X 2 | X1) @ %':

n n

Example 1.26 Consider a two-state Markov chain with a probability transition matrix
È1 - p1 p1 ˘
P= Í ˙ @ %$:
Î p2 1 - p2 ˚
8 > %
 Source Coding ° 53

Fig. 1.18 The state transition graph of a two-state Markov chain.

For stationary distribution, the net probability distribution across any cut set in the state transition
graph should be zero. Let a and b be the stationary probabilities of the two states. Thus, the stationary
distribution is given by
p2 p1
a = and b = @ %&:
p1 + p 2 p1 + p 2

Note that a D b = 1. The entropy of the state Xn at time n will be

Ê p2 p1 ˆ
H (Xn) = H Á , = H (a , b ) @ % :
Ë p1 + p 2 p1 + p 2 ˜¯

The entropy rate of this two-state Markov chain is given by

p2 p1
H (X) = H (X2 | X1) = H ( p1) + H ( p 2) @ %#:
p1 + p 2 p1 + p 2

Learning Review
YOU ARE NOW READY TO ATTEMPT THE FOLLOWING QUESTIONS LINKED TO LO 4:
K P > 0 D
R (D) = min I (X , X� ) for Bernoulli distributed X for a given p.
2. Let a source S{XkQ > 8 Xk, such that H(Xk) = k for S
k ≥ 1. Find the entropy rate of this source.
3. Plot the rate distortion function for a Gaussian source. Verify that R(D) is non-increasing M
in D, R(D) is convex, R@ : £ H(X) and R(D: + 0 D ≥ Dmax.

If you have successfully solved the above problems,

! " # $

The answers are available here. Scan to check

Or
Visit [Link]
54 ° Information Theory, Coding and Cryptography

1.15 Introduction to Image Compression

Earlier in this chapter we discussed the coding of data sets for compression.
LO 5 By applying these techniques we can store or transmit all of the information
content of a string of data with fewer bits than are in the source data. The
Apply the knowledge to minimum number of bits that we must use to convey all the information in
the source data is determined by the entropy measure of the source. Good
one of the important
application areas of source compression ratios can be obtained via entropy encoders and universal
coding. 0 0 > Y
Z > 8 0
data stream can represent the original data to an extent that is satisfactory
to the most discerning eye. Since we can represent a picture by something between a thousand and a million
bytes of data, we should be able to apply the techniques studied earlier directly to the task of compressing
that data for storage and transmission. First, we consider the following points:

1. High quality images are represented by very large data sets. A photographic quality image
$ 0 8 >
for extremely high compression ratios to make storage and transmission (particularly of
movies) practical.
2. Applications that involve imagery seem to be inherently linked to immediate human consumption,
and so need to be fast in execution on computers and in transmission. Television, movies, computer
graphical user interfaces, and the World Wide Web are examples of applications in which imagery
must be moved from storage or across some kind of distribution network very quickly for immediate
human intake.
3. Imagery has the quality of higher redundancy than we can generally expect in arbitrary data. For
example, a pair of adjacent horizontal lines in an image are nearly identical (typically), while, two
adjacent lines in a book have essentially no commonality.
8 88 > 0 8
technology available for the movement and storage of image data. The third factor indicates that compression
ratios will usually be quite high. The third factor also says that some special compression techniques may be
possible that will take advantage of the structure and properties of image data. The close relationship between
neighboring pixels in an image can be exploited to improve the compression ratios. This is very important for
the task of coding and decoding image data for real-time applications.
Another interesting point to note is that the human eye is very tolerant to approximation error in an image.
Thus, it may be possible to compress the image data in a manner in which the less important information
(to the human eye) can be dropped. That is, by trading off some of the quality of the image we might obtain
> lossy compression, as opposed to the lossless
compression techniques discussed earlier. This sentiment, however, can never be expressed with regards to,
; \] 8 88 >
0 > 0
 Source Coding ° 55

1.16 The JPEG Standard for Lossless Compression LO 5

INDUSTRIAL RELEVANCE
# ' ( 34"5
6 34"5 ( ?@ (
( ) E2 0 (
'
0 ( ' ) 0 F
here two methods that use entropy coding.

The two lossless JPEG compression options differ only in the form of the entropy code that is applied to
the innovations data. The user can choose to use either a Huffman code or an Arithmetic code.

We have seen earlier that Arithmetic code, like Huffman code, achieves compression by using
the probabilistic nature of the data to render the information with fewer bits than used in the
original data stream. Its primary advantage over the Huffman code is that it can come closer to
the Shannon entropy limit of compression for data streams that involve a relatively small alphabet. The
reason is that Huffman codes work best (highest compression ratios) when the probabilities of the symbols
can be expressed as fractions of powers of two. The Arithmetic code construction is not closely tied to these
particular values, as is the Huffman code. The computation of coding and decoding Arithmetic codes is more
0 P 00 8 & _ 88 0
Arithmetic codes over that obtained with Huffman coding.
Some compression can be achieved if we can predict the next pixel using the previous pixels. In this way
` 8 0 @ 00 : 0 8
The predictive process that is used in the lossless JPEG coding schemes to form the innovations data is also
P ; ; 8 } ; ;0 >
a line-by-line basis. The choice is made according to that prediction method that yields the best prediction
overall for the entire line.
There are eight prediction methods available in the JPEG coding standards. One of the eight (which is the
no prediction option) is not used for the lossless coding option that we are examining here. The other seven
may be divided into the following categories:
1. Predict the next pixel on the line as having the same value as the last one.
2. Predict the next pixel on the line as having the same value as the pixel in this position on the previous
line (that is, above it).
3. Predict the next pixel on the line as having a value related to a combination of the previous, above and
previous to the above pixel values. One such combination is simply the average of the other three.
The differential encoding used in the JPEG standard consists of the differences between the
actual image pixel values and the predicted values. As a result of the smoothness and general
redundancy of most pictures, these differences consists of a series of relatively small positive
56 ° Information Theory, Coding and Cryptography

and negative numbers that represent the small typical error in the prediction. Hence, the probabilities
associated with these values are large for the small innovation values and quite small for large ones. This is
exactly the kind of data stream that compresses well with an entropy code.
The typical lossless compression for natural images is 2:1. While this is substantial, it does not in general
solve the problem of storing or moving large sequences of images as encountered in high quality video.

1.17 The JPEG Standard for Lossy Compression LO 5

The JPEG standard includes a set of sophisticated lossy compression options which resulted from much
experimentation by the creators of JPEG with regard to human acceptance of types of image distortion. The
JPEG standard was the result of years of effort by the JPEG which was formed as a joint effort by two large,
standing, standards organizations, the CCITT (The European telecommunications standards organization)
and the ISO (International Standards Organization).
~€ 8 > 0 > 8 > ;
> 8 0 ; 0 8 8 8
> P 00 ‚ >
> 8 8; > ; 8
of an operation known as the Discrete Cosine Transform @„† :; 0
N -1 M -1
Ê pk ˆ Ê pl ˆ
Y (k, l) =   4 y (i, j ) cos ÁË 2 N (2i + 1)˜¯ cos ÁË 2M (2 j + 1)˜¯ @ %%:
i =0 j =0

where the input image is N pixels by M pixels, y(i, j) is the intensity of the pixel in row i and
column j, Y(k, l: „† 0 k and column l of the DCT matrix. All DCT
multiplications are real. This lowers the number of required multiplications, as compared to the
discrete Fourier transform. For most images, much of the signal energy lies at low frequencies, which appear
in the upper left corner of the DCT. The lower right values represent higher frequencies, and are often small
(usually small enough to be neglected with little visible distortion). The DCT is, unfortunately, computationally
very expensive and its complexity increases as O(N 2: 0 ; > 8 > „†
divided into blocks.
Z ~€ > 8 ; „† 88 % %8 Y 0 > P ; 0
> *& *& 8 ; Y '* '* Y 0% %8
8 $8 Y 0 „† 0 $
$ ;Y „† 0 ;0 0 8 > >
„† 0 8 8 0 0 > 9 Y 88 0 0
DCT matrix has low frequency components and the lower right-corner the high frequency components (see
> : 8 0 0 !& % & . Its value is proportional to the average value
0 % % Y 08 6! & % & .
So far we have not obtained any reduction simply by taking the DCT. However, due to the nature of most
natural images, maximum energy (information) lies in low frequency as opposed to high frequency. We can
represent the high frequency components coarsely, or drop them altogether, without strongly affecting the
quality of the resulting image reconstruction. This leads to a lot of compression (lossy). The JPEG lossy
compression algorithm does the following operations:
1. First the lowest weights are trimmed by setting them to zero.
2. The remaining weights are quantized (that is, rounded off to the nearest of some number of discrete
code represented values), some more coarsely than others according to observed levels of sensitivity
of viewers to these degradations.
 Source Coding ° 57

Fig. 1.19 $ %& $' * + 3 4" 56 7 7 +

are used.

Now several lossless compression steps are applied to the weight data that results from the above DCT
8 ;0 > Y Š „† 0 ; 8
> > ; 0 Y 0%¥%8 P ; 8
0 0 > Y Y Š ` „† 0 00
„† 0 0 Y 00
Š Y ‚† 0 Š ; 0 0 > 0
0 Š zig-zag > > * 8 8 0 >9 > >
is that we gradually move from the low frequency to high frequency, avoiding abrupt jumps in the values.
‹ >9 > > > 0 }; 0 Œ]€ 0 P 00 ‚
coding.

Fig. 1.20 8 9 9 :9

The typically quoted performance for JPEG is that photographic quality images of natural
8 8 0 8 *  *& K
@ ;0 8 8 : 0 8 > 0*  8
*' 

1.18 Video Compression Standards LO 5

† 8 0 0 #* ¥ 576 pixels and a refresh rate of 25 frames per
@08 : ] 8 0 8 >% 8 8
>% 8 8 ; 0 8
B = #* ¥ 576 ¥ 25 ¥ + 4 @ % :
58 ° Information Theory, Coding and Cryptography

P> „ @P„ Ž: 0 0 * ¥ % 8 0 0
08 ; 8 >
B= * ¥ % ¥ ¥ + 4 @ :
Clearly, uncompressed video produces an enormous amount of data. Fortunately, digital video contains
a great deal of inherent redundancy, and it is suitable for compression. There are several lossy compression
techniques available for video compression, but there is a trade-off between computational time and quality
of the output. Some of the popular video compression standards and the corresponding application areas are
> $

Table 1.4 " *

H.261 Video conferencing over ISDN

MPEG-1 Video on digital storage media (CD-ROM)
MPEG-2 Digital Television
H.263 Video telephony over PSTN
€9$ Object-based coding, synthetic content, interactivity
P* $ Improved video compression

P < P* € >

INDUSTRIAL RELEVANCE
H.261 is an ITU-T video compression standard ' & ( B7 <G7 &0 2
0 ?B/ ( ( ( 26: 6 Video Coding Experts Group
*H9"5+ 6 0 ( ?BA I 0 &J ( 0 & '
0 & * G¥G + 6 ( 92! *<>? ¥ ?GG / +
K92! *ALB ¥ A77 / + H 7 ? = M 9 90 ( 6
0 ( 6
' G ¥ G ;96 ( ' 0 4
( 34"5 6
0 ( * + ' ?; 6
G¥G0 & 0 & " 0 & 0
I J * + I J *'
+ ?BA ' A@@= ' ' / '
' (' ?BA 0 0
?B< ' N F /0 14"5 ( Moving
Picture Coding Exports Group and encompasses a whole family of international standards for
the compression of audio and video digital data.
 Source Coding ° 59

6 14"5
1. Reduction of the resolution: The human eye has a lower sensitivity to color than to intensity
@ C: 0 ; 0 Œ-9 98
KŽ9 98 0 8 0 0
than intensity information.

2. Motion compensation in order to reduce temporal redundancy: An MPEG video can be

interpreted as a sequence of frames. Because of the inherent inertia in the world that we live in, two
successive frames of a video sequence often have small differences (except in major scene changes).
The MPEG-standard offers a smart way of reducing this temporal redundancy by using three types of
frames: I-frames (intra), P-frames (predicted) and B-frames (bidirectional). The I-frames are ‘intra-
0 }; 0 0 8 90
can be predicted from an earlier I-frame or another P-frame. P-frames cannot be reconstructed without
their referencing frame, but they are more compressed than the I-frames, because only the differences
are stored. The B-frames are two-directional versions of the P-frame, referring to both directions (one
forward frame and one backward frame). The distance between two I-frames can be seen as a measure
for the quality of an MPEG-video. Recall that each frame is divided into blocks. To decrease the
amount of temporal redundancy in a video, only blocks that change are updated. The MPEG encoder
compares the current frame with adjacent parts of the video from the previous I- or P-frame. Only the
direction and distance (i.e., the motion vector) is encoded into the inter-frame. This is called motion
estimation. The reverse of this process, for reconstruction, is called motion compensation.

3. Discrete Cosine Transformation (DCT): € 0 %¥% Y € %¥%

Y 88 > 0 > @ %%: ‚
previous section, The upper left corner of the DCT matrix has low frequency components and the
lower right-corner contains the high frequency components. After the DCT operation, the ground is
set for compression using quantization.

4. Quantization: We have seen in the case of JPEG compression that quantization is the primary
source of data loss. The human eye is more reactive to low frequencies than to high ones (again, is the
> 8 C:  ; > 0 8 8
a zero entry thereby setting up the stage for run-length coding. Again, the quantized AC-terms are
stored in a zig-zag-manner with increasing frequency values.

The entropy
coding has two steps: Run Length Encoding (RLE) and Huffman coding. These have been discussed
earlier in the context of JPEG compression.
60 ° Information Theory, Coding and Cryptography

Learning Review
YOU ARE NOW READY TO ATTEMPT THE FOLLOWING QUESTIONS LINKED TO LO 5:
È0 0 0 1 1˘
Í0 1 1 0 1 ˙˙
Í
1. Consider an image represented by A = Í1 0 1 1 0˙ . Find its DCT. M
Í ˙
Í0 1 0 0 0˙
ÍÎ0 0 1 1 0˙˚
È6.77 4.31 2.19 1.56 0.21˘
Í4.02 2.44 1.83 0.31 0.01˙
Í ˙
2. Carry out quantization for the matrix given by A = Í2.26 2.09 0.43 0.02 0.002˙ . S
Í ˙
Í1.97 0.15 0.02 0.01 0 ˙
ÍÎ 0.03 0.04 0.02 0.01 0 ˙˚
Next perform the zig-zag encoding.
3. Perform run length coding on the sequence obtained after zig-zag coding in the previous S
question. NG
RATULATIONS
CO

If you have successfully solved the above problems,

! " # $
LEVEL
5 DONE

The answers are available here. Scan to check

Or
Visit [Link]

1.19 Concluding Remarks

8 0 8 Z $%; ” 8 Y •‚ -
matical Theory of Communication”. He begins this pioneering paper on information theory by observing
that the fundamental problem of communication is that of reproducing at one point either exactly or approxi-
mately a message selected at another point. He then proceeds to so thoroughly establish the foundations of
0 0 Y > ” }
> > } Z 0
‚> ” 8 8 < 0 8 P
} ; •‚ ” ‚ 0Œ ” >† —; - >
8 > 0 > Y > >
fundamental to the operation of modern computers and telecommunications systems.
 Source Coding ° 61

Shannon was renowned for his eclectic interests and capabilities. A favorite story describes him juggling
while riding a unicycle down the halls of Bell Labs. He designed and built chess-playing, maze-solving,
` >> > 9 > ” }
by curiosity than usefulness. In his words “I just wondered how things were put together.”
™ 0 0 8 0 ;
0 ™ 09 P 00 ‚ „ ‚ P 00
&* P 00 > ~€ € ‚ >;
is an extension of the Shannon-Fano-Elias coding technique, was developed by Rissanen and Pasco and
generalized by Rissanen and Langdon.
‚ 0 0 > > Y 8
8 # Z ‚ ] 8 ~ ‹ 8 8
0 KšZ› 8 > 0 > Z 0 @ 8
Interchange Format), developed by CompuServe, involves simply an application of the Lempel-Ziv-Welch
(LZW) universal coding algorithm to the image data. A more recent approach is to use quantum image
processing.
‚ ; 8 ” ; 0 0Z 0 ;8
*$; * € 8 0 8 New York Times:

Claude Shannon, Mathematician, Dies at 84

By GEORGE JOHNSON
Dr. Claude Elwood Shannon, the American mathematician and computer scientist whose theories laid
the groundwork for the electronic communications networks that now lace the earth, died on Saturday
in Medford, Mass…

LEARNING OUTCOMES

Ê 1 ˆ
!Y The self information of the event X = xi is given by I(xi) = log Á = – log P(xi).
Ë P ( xi ) ˜¯

Ê P (xi | y j ) ˆ
Y The mutual information I(xi; yj) between xi and yj is given by I(xi; yj) = log Á .
Ë P (x ) ˜¯
i

Y The conditional self information of the event X = xi given Y = yj

Ê 1 ˆ
I(xi | yj) = log Á ˜ = – log P(xi | yj)
Ë P ( xi | y j ) ¯
Y The average mutual information between two random variables X and Y is given by I(X; Y) =
n m n m P (xi , yj )
  P (xi , yj ) I (xi , yj ) =   P (xi , yj ) log P (x ) P (y ) . For the case when X and Y are statistically
i =1 j =1 i =1 j =1 i j
62 ° Information Theory, Coding and Cryptography

independent, I(X; Y: + . The average mutual information I(X; Y) ≥ ; 0 0X

and Y are statistically independent.
Y The average self information of a random variable X is given by
n n
H(X) = Â P (xi ) I (xi ) = - Â P (xi ) log P (xi ) . H(X) is called the entropy.
i =1 i =1
Y The average conditional self information called the conditional entropy is given by
n m
1
H(X|Y) = Â Â P (xi , yj )log P (x | y ) .
i =1 j =1 i j

!Y!I(xi; yj) = I(xi) – I(xi | yj) and I(X; Y) = H(X) – H(X|Y) = H(Y) – H(Y|X). Since I(X; Y) ≥ ; 8
H(X) ≥ H(X|Y).
!Y The joint entropy of a pair of discrete random variables (X, Y) with a joint distribution p(x, y:
n m
as H (X, Y) = - Â Â p ( xi , y j ) log p ( xi , y j ) .
i =1 j =1

Y The chain rule relating H(X), H(X, Y) and H(X|Y) is H(X, Y) = H(X: D H(Y|X).
!Y The relationship between entropy and mutual information can be expressed as
I(X; Y) = H(X) – H(X|Y) = H(Y) – H(Y|X).
Y The relationship between mutual information and joint entropy can be expressed as
I(X; Y) = H(X: D H(Y) – H(X, Y).
!Y!The average mutual information between two continuous random variables X and Y is given by
p ( y | x) p ( x)
I(X;Y) = Ú Ú p ( x) p ( y | x) log
p ( x) p ( y )
dxdy .

!Y The differential entropy of a continuous random variables X is given by H(X) = - Ú p ( x) log p ( x) dx.

!Y The average conditional entropy of a continuous random variables X given Y is given by

h(X |Y ) = Ú Ú p ( x, y ) log p ( x | y ) dxdy .

!Y Œ € 8 ™ Y] @™]: 8 0 p(x)
Ê p ( x) ˆ
and q(x: D ( p || q) =  p( x) log ÁË q( x) ˜¯ .
x ŒX

!Y The Jensen Shannon distance between two probability mass functions p(x) and q(x:
1 1 1
JSD ( p || q ) = D ( p || m) + D (q || m) , where m = ( p + q ) .
2 2 2
 Source Coding ° 63

Y ‚ 0 0 0 > >
L
n1 £ n2 £… nL 0 8 Â 2- n k
£ 0 0 8 >
k =1
H (X ) .
by h =
R
Y Let X 0 0 „ ” 8 H(X). The source coding theorem
8 8 ; > >
R H(X) £ R < H(X: D €0 8 0
compression of data.
!Y The expected codeword length of Huffman code is bounded as R £ H ( X ) + 1 .
!Y The expected codeword length of Shannon-Fano-Elias code is bounded as R < H ( X ) + 2 .
!Y 8 0” 9 9€ > 8 0 > 0
> > ‚ † > Y 8 >
0
Y In contrast to the Huffman coding scheme, the Lempel-Ziv technique is independent of the source
] 8 9‹ > > ; P 00
variable length code.
Y Run-length Encoding, or RLE is a technique used to reduce the size of a repeating string of characters.
8 > > Π9 > > 88 8 0
Z ;-  †›
Y Distortion implies some measure of difference between the actual source samples {xkQ
corresponding quantized value { x�k Q 9 is given by d ( xk , x�k ) = (xk – x�k )2.
p
In general a distortion measure may be represented as d ( xk , x�k ) = xk - x�k .
Ï0 if xk = x�k
!Y P > d ( xk , x�k ) = Ì .
Ó1 if xk π x�k
Y The minimum rate (in bits/source output) required to represent the output X of the memoryless
source with a distortion less than or equal to D is called the rate distortion function R(D:;
R(D) = min I ( X , X� ) , where I ( X , X� ) is the average mutual information between X and X� .
p ( x� | x ): E [d (X , X� )] £ D

!Y Some properties of rate distortion function: R(D) is non-increasing in D, R(D) is convex, R@ : £ H(X)
and R(D: + 0 D ≥ Dmax.
Y The distortion resulting due to the quantization can be expressed as D = Ú f ( x� - x) p ( x)dx , where
f ( x� - x) is the desired function of the error. An optimum quantizer is one that that minimizes D by
optimally selecting the output levels and the corresponding input range of each output level. The
resulting optimum quantizer is called the Lloyd-Max quantizer.
!Y A discrete stochastic process X1, X2 , … is said to be a Markov chain or a Markov process if, for n = 1,
2…
P(Xn D = xn D | Xn = xn, Xn –1 = xn –1,…, X1 = x1) = P(Xn D = xn D | Xn = xn)
for all x1, x2,… xn, xnD Œ X .
64 ° Information Theory, Coding and Cryptography

1
!Y The entropy rate of a stochastic process X is given by H(X) = lim H ( X1 , X 2 , � , X n ), provided the
n n
limit exists.
!Y For stationary Markov chain, the entropy rate is given by

H(X) = lim H (X n | X n -1 , � , X1 ) = lim H ( X n | X n -1) = H (X 2 | X 1 ) .

n n
!Y Quantization and source coding techniques (Huffman coding, arithmetic coding and run-length coding)
are used in the JPEG standard for image compression.
!Y MPEG stands for Moving Picture Coding Experts Group and encompasses a whole family of
international standards for the compression of audio and video digital data.
!Y The important steps in MPEG compression are reduction of the resolution, motion estimation/
compensation, Discrete Cosine Transformation (DCT), quantization and entropy coding.

Obstacles are those frightful things you see when you

take your eyes off your goal.
Henry Ford (1863-1947)

MULTIPLE CHOICE QUESTIONS

1.1 Pick the correct choice

(a) I(X; Y) = H(X) – H(X|Y) (b) I(X; Y) = H(Y) – H(Y|X)
(c) Both a and b (d) None
* Š > 0 > 8 C
(a) Information Rate (b) Noiseless Channel
@ : † † > @ : ™ 0 Z
' 0 0 8 >
R H (X )
(a) h = (b) h =
H (X ) R
H (X ) + 1
(c) h = (d) None of the above
R
$ ™ Y] @™]: 8 8 0
(a) Non-negative (b) Symmetry property
(c) Triangle inequality (d) None of the above
 Source Coding ° 65

1.5 The relationship between mutual information and joint entropy can be expressed as
(a) I(X; Y) = H(X) – H(Y: D H(X, Y) (b) I(X; Y) = H(X: D H(Y) – H(X, Y)
(c) I(X; Y) = H(X: D H(Y: D H(X, Y) (d) None of the above
1.6 The expected codeword length of Huffman code is bounded as
(a) R £ H ( X ) + 1 (b) R £ H ( X ) - 1
(c) R < H ( X ) + 1 (d) None of the above
# Š > 0 > 8 4 0 > C
(a) JPEG (b) H.263
(c) MPEG (d) All of the above
% Š > @: > C
(a) Lempel Ziv (b) Huffman
(c) Both a and b (d) None of the above
~€ 0 > 8
(a) Huffman coding (b) Arithmetic coding
(c) Run-length coding (d) All of the above
€9 8 >
(a) I-frames, P-frames and B-frames (b) Discrete Cosine Transformation (DCT)
(c) Quantization (d) Entropy coding

For interactive quiz with answers, scan the QR code given here
Or
Visit [Link]

SHORT-ANSWER TYPE QUESTIONS

† 00 8 > C€ 8
* Š 0 8 8 0 C
1.3 If X and Y are independent, is it true that H(X,Y) = H(X: D H(Y: C
$ ] X, Y, and Z be discrete random variables with I(X; Y: + I(X; Z: + Z I(X; Y, Z: + C
& „ > 8 C Z H(X|Y) £ H(X: C Z h(X|Y) £ h(X: C
1.6 When is H(X|Y) = H(X: C
1.7 Is the function log(x: C
% Z 0 xlog(x: C
Š 0 I(X; X:C
† 0 > C
Z ¡ ; ; ; Q 8 CŠ ¡ ; ; ; QC
* „ 0 8 C
1.13 By what factor does one bit reduce the expected distortion.
$  * > 0 0 @ > :
You have a beam balance. What is the minimum number of weighs needed to identify the counterfeit
CP 0 C
66 ° Information Theory, Coding and Cryptography

& Š „† 0 ~€ 8 C

For answers, scan the QR code given here

Or
Visit [Link]

PROBLEMS

M 1.1 Prove that the entropy for a discrete source is a maximum when the output symbols are equally
probable.
S 1.2 Prove the inequality ln x £ x – 1. Plot the curves y1 = ln x and y2 = x – 1 to demonstrate the validity
of this inequality.
M 1.3 Show that I(X; Y) ≥ K C
S $ ] X 0 0 88
(i) Find the entropy, Hf(X) if the coin is fair.
(ii) Next assume the coin to be unfair with p being the probability of getting a tail. Find the
entropy, Hu(X).
S & † 0 š 0 0
@: Z CŠ C
@ : † > > 0 0 > > 0
P 00 0 C€ 8
M † ™ Y] 8 0 p(x) and q(x),
Ê p ( x) ˆ
given by D ( p || q ) =  p( x) log ÁË q( x) ˜¯ .
x ŒX

(i) Show that D ( p || q ) is non negative.

@ : ” ™ Y] 0 8 8 0 ;
i.e., symmetry and the triangle inequality.
(iii) Show that I(X; Y) = D ( p ( x, y )|| p ( x) p ( y )) . This would elegantly prove that mutual infor-
mation is non negative.
D 1.7 Suppose a source produces independent symbols from the alphabet {a1, a2, a3Q; 8
p1 + $ ; p2 + $ ; p3 + *
(i) Compute the entropy, H(X), of this source.
(ii) Find an optimal code for this source, and compute its expected codeword length.
(iii) Find an optimal code for the second extension of this source (i.e., for blocks of two symbols),
and compute its expected codeword length, and the expected codeword length divided by two.
@ :  8 _ 0 8 > Y 0 N from
this source, N & š 0 @ $&: > 88
bound.
 Source Coding ° 67

1
S % † > ; X, given by P(X = n) = , where
An log 2 n
1
A=Â 2
and n = 2, 3, … , . Find the entropy, H(X).
n=2 n log n
S ] 88 0 p(xi) @ $: 8 p(x)i Dxi . Show that the expression
0 8 0 > @ ' : 8

H(X) = - Ú p ( x) log p ( x) - lim log (D x)

Dx Æ 0

M 00 8 0 Z = X1 D X2, where X1 and X2 are independent

Gaussian random variables with means ‚i and variances s i2 , i = 1, 2.
S † „ ” 8 ¡ * ; * ; &; &; ; ; &; &Q
@: „ 0 > 0
(ii) Determine the Huffman code for this source.
(iii) Compare the two codes and comment.
M * ‚„ ” 8 8 ¡ &; $; Q
@: „ P 00 0 0 h.
(ii) Determine the Huffman code for this source taking two
0 h.
(iii) Determine the Huffman code for this source taking three
0 h.
S 1.13 For a source with entropy H(X), prove that the entropy of a B-symbol block is BH(X).
Ê 1 ˆ
M $ ] 8 0> > 0 p = Á � ˜ and the probability of
ÁË x x ˜¯
Ê ˆ x
Á 1 ˜
getting a tail be q = .
Á y y� ˜
Ëy ¯
(i) Show that the entropy of this biased coin is log2(xy).
(ii) Give the plot of x versus p.
M 1.15 Let X and Y be random variables that take on values x1, x2, …, xr and y1, y2, …, ys respectively. Let
Z = X D Y.
(i) Show that H(Z|X) = H(Y|X).
(ii) If X and Y are independent, then argue that H(Y) £ H(Z) and H(X) £ H(Z). Comment on this
observation.
(iii) Find an example of X and Y with H(X) > H(Z) and H(Y) > H(Z).
(iv) Under what condition will H(Z) = H(X) D H(Y:C
S 1.16 Let X be a discrete random variable and let Y = f(X) be a deterministic function of X.
(i) Show that H(X) = H(Y : D H(X|Y ).
(ii) Under what condition is H(X) = H(Y :C
S 1.17 (i) Show that the entropy of a random variable X is zero if and only if X has only one value with
non-zero probability.
(ii) Show that if the conditional entropy H(X|Y) is zero, then the value of X is completely
determined by the value of Y, with probability 1.
68 ° Information Theory, Coding and Cryptography

S % † P 00 > *; *; *; *; *; *; *; *; '; ' 'C

Explain.
S † > 1 +¡ ; ; Q Z CZ
CŠ > 2 + ¡ ; ; ; ; QC Z C
Z C
M * Z ;Z ;* > ;' ;
$> ;& > ;Z
8 Y 8 0 > 0 ” >>
(a) binary code and (b) ternary code for doing so. What is the expected codeword length in each
C
M 1.21 We know that in Huffman coding, the most probable symbol gets assigned the shortest codeword
length. Practically, the shortest codeword length possible is 1. Show that, for the most probable
symbol x1, if the associated probability p1 ¤ $; 0 >
D ** † ] 8 ‹ >0 @  ; ; *; ':
@: € 0 >  '' * * ' ** ***'' Š
8 ; 8 0
after encoding divided by the number of bits before encoding.
@ : š ; 8 @ ¥ ; ¥ ;*¥ ;
'¥ : š 8 0 ] 8 ‹ > 8
ratio.
(iii) Compare the results of (i) and (ii). Comment.
D *' † >> ‚ 8 0 0¦ }
bit stream is given by
ÏÔ 2- r for 1 £ r < n
pr = Ì - ( n -1) @ % :
ÔÓ2 for r=n
where r is the length of the run and n is the maximum run length possible. Note that the probabilities
8 > ; 8 0 } 0 } }
@: „ > > 8 8
(ii) Suppose you want to encode the runs using Huffman code. Generate the code for this bit
stream.
Š 8 8 C
M *$ Š 0 > > Š 8 9 > >
process again and again until no more compression is possible. Using this procedure, if we have to
encode a run of, say n } ; 8 @ : 8 C
M 1.25 Consider a source X uniformly distributed on the set {1, 2, …, mQ 0

0 P > d ( x, x� ) = Ï0, x = x� .
Ì
Ó1, x π x�
 Source Coding ° 69

COMPUTER PROBLEMS
1.1 Write a program that performs Huffman coding, given the source probabilities. It should generate the
> > 0 Z 9 8
1.2 Modify the above program so that it can group together n source symbols and then generate the
P 00  > 0 h versus n for the following source symbol probabilities:
¡ &&; *&; * Q 0n 0 CŒ 8
0 0 > 8 ¡ $&; *&; &; ; &Q
1.3 Write a program that executes the Lempel-Ziv algorithm. The input to the program can be the English
alphabets. It should convert the alphabets to their ASCII code and then perform the compression
Z 8 8 K > 8 > ; 8
achieved for the following strings of letters:
@: ] 8 9‹ > 8 € > 0 8
(ii) The cat can not sit on the canopy of the car.
$ Š 8 > 8 0 Œ ] > € > @Œ]€: 0 >
output along with the compression ratio. What is the output of the program if the following sequence
is fed into it:

Now feed back the encoded output to the program, i.e., perform the RLE two times on the original
0 Š C†
1.5 Suppose for a certain type of communication we only use three letters A, B and C. The probability
0 0 ¡ & '; *Q P ; 0 8
8 0 0 > @‚-: + *&; @-†: + &; @†‚: + '&;
@-‚: + &; @†-: + &; @‚†: + & ] P 00 Y
the bigrams as Pseudo Markov Huffman (PMH) code.
(i) Design a PMH code for this communication scenario which takes into consideration the bigrams
as well. Write a computer program that can take these probabilities as input and generate the PMH
code.
(ii) Write a program that performs arithmetic coding for this case and compare with the result of the
PMH code.
1.6 Write a program that takes in a 2n level gray scale image (n bits per pixel) and performs the following
operations:
@: - Y 8 % %8 Y
(ii) Performs DCT on each 0 % % Y
@ : § „† 0 > m ”> - @ ”-:;
m £ n.
(iv) Performs the zig-zag coding followed by run length coding.
(v) Performs Huffman coding on the bit stream obtained above (think of a reasonable way of calcu-
lating the symbol probabilities).
(vi) Calculates the compression ratio.
(vii) Performs the decompression (i.e., the inverse operation of the steps (v) back to (i)).
Perform image compression using this program for different values of m. Up to what value of m is
8 8 00 > > 8 > C
70 ° Information Theory, Coding and Cryptography

PROJECT IDEAS

Superinformation: Š ¦ 8 0 8 };
superinformation.
@:  ¦ > >} 0 0 8
of occurrences are design parameters.
(ii) Divide the string into sub-blocks of size B.
(iii) For i th 9 Y; 8 ; Hi.
(iv) Find the histogram of Hi. Normalize this histogram to obtain a pdf.
(v) Find the entropy for this pdf. This is the superinformation (entropy of entropy) of the sequence.
(vi) Use the concept of superinformation on DNA sequences and see if you can segregate exons (coding
regions) from introns (non-coding regions).
3% & & % 7 ! ' The Morse Code for the English alphabet is given in Fig. 1.21.
@: „ > > 0 † 0 0
code.
@ : Z C€ 8
(iii) Compare Morse code with the Huffman code for the English alphabet.

Fig. 1.21

3 8 9; 0 " * "' Consider an energy-limited wireless device that periodically

captures images and uploads it over a wireless channel. Energy is expended in the device for wireless
transmission of bits ( t, mJ/bit) and computation ( c, mJ/bit). We would like to optimize the total energy
per bit. We have three options (i) We upload the uncompressed image (i.e., we have to transmit a larger
number of bits). (ii) We compress the entire image and then send (we will save energy by transmitting
a shorted bit stream but spend energy doing compression). (iii) We can compress part of the image
(based on entropy-based partitioning) and send partly compressed and party uncompressed (raw bits).
 > Y 8 „ > 8
using entropy-based image partitioning in order to minimize energy per bit. Also carry out simulations
using realistic numbers for t @ ; 0 Z€€€ % * : c (say, for the ARM processor).
 Source Coding ° 71

< " 3% & & = ! " (HEVC): Consider the HEVC standard, which is a successor to the
‚ Ž † > @P * $4 €9$: P€Ž† 9 8
coding (CABAC) algorithm. CABAC has multiple probability modes for different contexts. It works
> 9 ;0 ;
an appropriate probability model to use. It then uses information from nearby elements to optimize
8 ; > 8 „ 8 8
HEVC scheme using CABAC and perform encoding on simple binary images. Comment on your
results.

REFERENCES FOR FURTHER READING

1. Cover, Thomas M. and Joy, A. Thomas; & %4 % & + ;~ Š ¨” ;* *

2. Robert, Ash B. Information Theory „ ”8  ;* #
 CHAPTER TWO

Channel Capacity
and Coding
Experimentalists think that it is a mathematical theorem while the mathematicians believe
it to be an experimental fact.
[On the Gaussian curve, remarked to Poincaré:]
Lippman, Gabriel (1845-1921)

learning objectives

After studying this chapter the students will be able to:

LO 1
LO 2
LO 3

LO 4 !
LO 5 !

2.1 Introduction
In the previous chapter we saw that most of the natural
This chapter comes with a video
sources have inherent redundancies and it is possible
overview by the author. Scan here
to compress data by removing these redundancies.
to know more or
Compression is possible by different source coding Visit [Link]

symbols by the minimum possible number of bits, we

etc.). These bits may be transmitted as they are (for baseband communications), or after modulation (for
passband communications). Unfortunately, all real-life channels are noisy (this is not so unfortunate for those
who make a living out of designing communication systems for noisy channels!). The term noise is used
to designate unwanted waves that tend to disturb the transmission and processing of the wanted signals in
 Channel Capacity and Coding ° 73

communication systems. The sources of noise may be external to the system (e.g., atmospheric noise, man
generated noise etc.), or internal to the system (e.g., thermal noise, shot noise etc.). In effect, the bit stream
obtained at the receiver end is likely to be different from the bit stream that is transmitted. For the case
of passband communication, the demodulator processes the channel-corrupted waveform and reduces each
waveform to a scalar or a vector that represents an estimate of the transmitted data symbols. The detector,
which follows the demodulator, may decide on whether the transmitted bit is a 0 or a 1. This is called a hard
decision decoding. This decision process at the decoder is like a binary quantization with two levels. If there
are more than 2 levels of quantization, the detector is said to perform a soft decision decoding. In the extreme
case, no quantization is performed for soft decision decoding.
The use of hard decision decoding causes an irreversible loss of information at the receiver. Suppose the
modulator sends only binary symbols but the demodulator has an alphabet with Q symbols. Assuming the
use of the quantizer as depicted in Fig. 2.1 (a), we have Q = 8. Such a channel is called a Binary input Q-ary
output Discrete Memoryless Channel. The corresponding channel is shown in Fig. 2.1 (b). The decoder
performance depends on the location of the representation levels of the quantizers, which in turn depends on
the signal level and the noise power. Accordingly, the demodulator must incorporate automatic gain control
in order to realize an effective multilevel quantizer. It is clear that the construction of such a decoder is

improvement in the performance over hard decision decoding.

Fig. 2.1 (a) Transfer characteristic of multilevel quantizer. (b) binary input Q-ary output discrete memoryless
channel.

There are three balls that a digital communication engineer must juggle: (i) the transmitted signal power
(ii) the channel bandwidth and (iii) the reliability of the communication system (in terms of the bit error rate).
Channel coding allows us to trade-off one of these commodities (signal power, bandwidth or reliability) with
respect to others. In this chapter, we will study how to achieve reliable communication in the presence of
noise. We shall ask ourselves questions like how many bits per second can be sent reliably over a channel of
a given bandwidth and for a given signal to noise ratio (SNR)? For that, we begin by studying a few channel
74 ° Information Theory, Coding and Cryptography

In this chapter
" # # ! $
%# & % & %
' % ' ' ( % ) %'
* $ % +( 1.
%# # ! #
" # , ! $ %$
" , % +( 2.
" # ,
" # # - - * +( 3.
%# # % - &
- " # , # . %
' ' ( '
* % +( 4.
/ % +( 5, # # ! )

2.2 Channel Models

We have already come across the simplest of the channel models, the Binary
LO 1 Symmetric Channel (BSC), in the previous chapter. If the modulator
employs binary waveforms and the detector makes hard decisions, then the
channel may be viewed as one in which a binary bit stream enters at the
transmitting end and another bit stream comes out at the receiving end. A
BSC is shown in Fig. 2.2 (a). Figure 2.2(b) shows one possible output for a
binary image sent through a given BSC with p = 0.1.
1–p 0.9
0 0 0 0
p 0.1
p 0.1
1 1 1 1
1–p 0.9
(a) (b)

Fig. 2.2 (a) A binary symmetric channel (BSC). (b) How an image might look after transmission through a
BSC.

This binary Discrete-input, Discrete-output channel is characterized by the set X = {0,1} of possible
inputs, the set Y = {0,1} of possible outputs and a set of conditional probabilities that relate the possible
outputs to the possible inputs. Let the noise in the channel cause independent errors in the transmitted binary
sequence with average probability of error p.
Then,
P(Y = 0| X = 1) = P(Y = 1| X = 0) = p
P(Y = 1| X = 1) = P(Y = 0| X = 0) = 1 – p (2.1)
 Channel Capacity and Coding ° 75

Channel Binary Demodulator/ Channel

Channel
Encoder Modulator Detector Decoder

Composite Channel

Fig. 2.3 A composite discrete-input, discrete-output channel formed by including the modulator and
demodulator/detector.

The BSC is a special case of a composite Discrete-input,

Discrete-output channel depicted in Fig. 2.3. Let the input to
the channel be q-ary symbols, i.e., X = {x0, x1, …, xq–1} and the
output of the detector at the receiving end of the channel consist
of Q-ary symbols, i.e., Y = {y0, y1, …, yQ–1}. We assume that
the channel and the modulation are both memoryless. The inputs
and the outputs can then be related by a set of qQ conditional
probabilities
P(Y = yi | X = xj) = P(yi | xj) (2.2)
where i = 0, 1, … Q – 1 and j = 0, 1, … q – 1. This channel
Fig. 2.4 A discrete memoryless channel
is known as a Discrete Memoryless Channel (DMC) and is (DMC) with q-ary input and
depicted in Fig. 2.4. Q-ary output.

The conditional probability P( yi | x j ) Channel Transition

Probability and is denoted by pji.

The conditional probabilities {P ( yi | x j )} that characterize a DMC can be arranged in the

matrix form P = [pji] . P is called the Probability Transition Matrix for the channel. Note: Sometimes,
P is also referred to as the Transition Probability Matrix or the Channel Transition Matrix or the
Channel Transition Probability Matrix.

So far we have discussed a single channel with discrete inputs and discrete outputs. We can decouple
the modulator and the demodulator from the physical channel (as depicted in Fig. 2.3). Thus, the input to
the channel will be a waveform and the output of the channel will also be a waveform. These are called
Waveform Channels. Such channels are typically associated with a given bandwidth, W. Suppose, x(t) is a
bandlimited input to a waveform channel and y(t) is the output of the channel. Then,
y(t) = x(t) + n(t) (2.3)
where, n(t) is the additive noise.
It is also possible to realize multiple channels between the transmitter and receiver. Such channels can
be readily realized in wireless communication scenarios by using multiple antennas at the transmitter and
receiver. The four obvious combinations are as follows:
(i) Single Input Single Output
antenna both at the transmitter and receiver.
76 ° Information Theory, Coding and Cryptography

(ii) Single Input Multiple Output

transmitter and multiple antennas at the receiver.
(iii) Multiple Input Single Output
transmitter but only a single antenna at the receiver.
(iv) Multiple Input Multiple Output
antennas both at the transmitter and receiver.
Consider an MIMO system with MT transmits antennas and MR receive antennas. Let us denote the impulse
response between the jth ( j = 1, 2, … MT ) transmit antenna and the ith (i = 1, 2, … MR) receive antenna by
hi,j(t, t). Note that here we are considering the waveform channel and the modulator/demodulator is not a part
of the channel. The MIMO channel can be represented using a MR ¥ MT matrix as follow.

È h11, (t , t ) h1,2 (t , t ) h1,M T (t , t ) ˘

Í ˙
Í h2,1 (t , t ) h2,2 (t , t ) … h2,M T (t , t ) ˙
H(t, t) = Í ˙ (2.4)
Í ˙
ÍhM ,1 (t , t ) hM ,2 (t , t ) hM R ,M T (t , t ) ˙˚
Î R R

A wireless channel is time varying and the variable t is used to capture the time-varying nature of the
channel. If a signal sj(t) is transmitted from the jth transmit antenna, the signal received at the ith receive
antenna is given by
MT
yi (t ) = Â hi , j (t , t ) s j (t ) , i = 1, 2, … MR (2.5)
j =1

The input-output relation of a MIMO channel can be expressed succinctly in matrix notation as
y(t) = H(t, t) s(t) (2.6)
where, s(t) = [ s1 (t ) s2 (t ) sM T (t )]T and , y(t) = [ y1 (t ) y2 (t ) yM R (t )]T

Each link between a pair of transmitter and receiver can be independently represented as a
discrete memoryless channel.

We can also have Relay Channels where there is a source, a destination and intermediate relay
nodes. These relay nodes facilitate the communication between the source and the destination,
as shown in Fig. 2.5(a). There could be several ways to facilitate the transfer of information from the source
to the destination by hopping over the intermediate nodes. One possibility is that each relay node simply
! "
power. This protocol is known as Amplify-and-Forward #$ %# !" !
decode the received signal and then re-encodes the signal before forwarding it to the next relay node. The
processing of the signal at each relay node requires making a hard decision. This protocol is known as
Decode-and-Forward (DF) scheme.
So far we have considered only a single transmitter and a single receiver (though, each one of the trans-
mitters or receivers can have multiple antennas!). In real-life, there can be multiple senders and multiple
receivers (with each one having the possibility of multiple antennas!). Suppose M transmitters (say, mobile
phone users) want to communicate with a single receiver (say, the base station) over a common channel,
as depicted in Fig. 2.5 (b). This scenario is known as a Multiple Access Channel. We can also reverse
the scenario. Suppose a single transmitter (say, a low earth orbit satellite) wants to communicate with M
 Channel Capacity and Coding ° 77

receivers (say, the home dish antennas) over a common channel, as shown in Fig. 2.5 (c). This is an example
of a Broadcast Channel.

Fig. 2.5 (a) Relay Channel. (b) Multiple Access Channel. (c) Broadcast Channel.

Learning Review
YOU ARE NOW READY TO ATTEMPT THE FOLLOWING QUESTIONS LINKED TO LO 1:
1. We would like to represent the wireless local area home network consisting of one access S
point and three laptops simultaneously connected to it. What types of channel model
should we use?
2. Graphically represent a binary channel with S
P(Y = 0| X = 1) = p, P(Y = 1| X = 0) = q, P(Y = 1|
X = 1) = 1 – p, P(Y = 0| X = 0) = 1 – q.
3. Consider the binary channel shown in Fig. 2.6. M
Let the a priori probabilities of sending the
Fig. 2.6
binary symbol be p0 and p1, where p0 + p1 = 1.
Find the a posteriori probabilities
P(X = 0|Y = 0) and P(X = 1|Y = 1).
4. Show that a cascade of n BSCs can be effectively represented as a single BSC. M

If you have successfully solved the above problems,

The answers are available here. Scan to check

Or
Visit [Link]

S Simple: Level 1 and Level 2 Category

M Medium: Level 3 and Level 4 Category
D Level 5 and Level 6 Category
78 ° Information Theory, Coding and Cryptography

2.3 Channel Capacity

Consider a DMC having an input alphabet X = {x0, x1, ..., xq –1}and an output
LO 2 alphabet Y = {y0, y1, ..., yr-1}. Let us denote the set of channel transition
probabilities by P(yi|xj). The average mutual information provided by the
output Y about the input X is given by (see Chapter 1, Section 1.3)

q -1 r -1 P( yi | x j )
I ( X ;Y ) = Â Â P( x j ) P( yi | x j ) log P( yi )
(2.7)
j =0 i =0

The channel transition probabilities P(yi|xj) are determined by the channel characteristics (primarily the
noise in the channel). However, the input symbol probabilities P(xj) are within the control of the discrete
channel encoder. The value of the average mutual information, I(X; Y), maximized over the set of input
symbol probabilities P(xj) is a quantity that depends only on the channel transition probabilities P(yi|xj)
(hence only on the characteristics of the channel). This quantity is called the

! The Capacity ! ' *

as the maximum average mutual information in any single use of the channel, where the
maximization is over all possible input probabilities. That is,

C = max I ( X ; Y )
P( x j )

q -1 r -1 P( yi | x j )
= max   P( x j ) P( yi | x j ) log (2.8)
P( x j )
j =0 i =0 P( yi )

The maximization of I(X; Y) is performed under the constraints

q -1
P(xj) ≥ 0 and  P( x j ) = 1
j =0

The units of channel capacity is bits per channel use (provided the base of the logarithm is 2).
If the base of the logarithm is e, the units of capacity is in nats per channel use (coming from
the use of natural logarithm).
 Channel Capacity and Coding ° 79

Example 2.1 Consider a BSC with channel transition probabilities

P(0|1) = p = P(1|0)

È1 - p p ˘
Thus the transition probability matrix is given by P = Í , and p is referred to as
the crossover probability. Î p 1 - p ˙˚

By symmetry, the capacity, C = max I ( X ; Y ) ), is achieved for input probabilities

P( x j )

P(0) = 0.5 = P(1). From (2.8) we obtain the capacity of a BSC as

C = 1 + plog2p + (1 – p)log2(1 – p)
9 entropy function
H(p) = – plog2p – (1 – p)log2(1 – p)
Hence, we can rewrite the capacity of a binary symmetric channel as
C = 1 – H(p)
1

0.8

0.6

Capacity

0.4

0.2

0
0 0.2 0.4 0.6 0.8 1
Probability of error

Fig. 2.7 The capacity of a BSC.

The plot of the capacity versus p is given in Fig. 2.7. From the plot we make the following observations:
80 ° Information Theory, Coding and Cryptography

(i) For p = 0 (i.e., noise free channel), the capacity is 1 bit/use, as expected. Each time we use the
channel, we can successfully transmit 1 bit of information.
(ii) For p = 0.5, the channel capacity is 0, i.e., observing the output gives no information about the
input. It is equivalent to the case when the channel is broken. We can, might as well, discard the
channel and toss a fair coin in order to estimate what was transmitted.
(iii) For 0.5 < p < 1, the capacity increases with increasing p. In this case, we simply reverse the
positions of 1 and 0 at the output of the BSC.
(iv) For p > @ % %" !J K J! " ! @J P " %
" !K J %
(v) Since p is a monotonically decreasing function of SNR, the capacity of a BSC is a monotonically
increasing function of SNR.

Example 2.2 Consider a Binary Erasure Channel p

1–e
0 0
(BEC), shown in Fig. 2.8 with transition probability matrix e
È1 - e e 0 ˘
given by P = Í . We will calculate the capacity
e ˙˚
e
Î 0 e 1 -
of this BEC. e
1–p 1 1
C = max I ( X ; Y ) = max( H (Y ) - H (Y | X )) = max( H (Y ) - H (e )) 1–e
P( x j ) P( x j ) P( x j )
Fig. 2.8 The binary erasure channel
Note that given the input X, there are two possible outcomes (BEC).
(Y) with probabilities {1 – e , e}. Hence, H (Y | X ) = H (e ). For
the output Y, we have three cases with PY(0) = p(1 – e), PY(e) = e and PY(1) = (1 – p)(1 – e). Therefore,
H(Y) = H(p(1 – e), e , (1 – p)(1 – e)) = H(e) + (1 – e) H(p)
Hence,
C = max( H (Y ) - H (e )) = max((1- e ) H ( p ) + H (e ) - H (e )) = max((1- e ) H ( p ))
P( x j ) p p
We know that the maximum value of H(p) = 1 for p = 0.5. Therefore, the capacity of a BEC is
CBEC = 1 – e
This result is very intuitive. Suppose we use the channel n times, i.e., we attempt to send n bits
through the channel. Of these, ne bits are ‘lost’(erased) in the channel. So, we can recover, at
most, (1 – e)n bits. Hence, per channel use, we can successfully send (1 – e) bits, leading to
CBEC = 1 – e.

" A discrete memoryless channel is said to be Symmetric if the rows of the channel
transition probability matrix are permutations of each other and the columns are permutations of each
other.
 Channel Capacity and Coding ° 81

# A discrete memoryless channel is said to be Weakly Symmetric if the rows of the channel
transition probability matrix are permutations of each other and the column sums  p ( y | xi ) are equal.
For weakly symmetric channels, the capacity is given by xi

C = log|Y| – H(row of transition matrix) (2.9)

and this is obtained for uniform distribution on the input alphabet. Here |Y| represents the cardinality of Y.

È0.5 0.3 0.2˘

Example 2.3 Consider the transition probability matrix given by P1 = ÍÍ0.2 0.5 0.3˙˙ .
ÍÎ0.3 0.2 0.5˙˚
of the channel transition
probability matrix are permutations of each other, and so are the columns. On the other hand, consider
È1 1 1˘
Í 2 6 3˙
P2 = Í ˙ . This is a weakly symmetric channel, but not symmetric. While the rows of the channel
Í1 1 1˙
ÍÎ 6 2 3 ˙˚
transition probability matrix are permutations of each other, the columns are not. However, each column
adds up to 2/3. From (2.9), the capacity of this weakly symmetric channel is readily calculated to be

Ê 1 1 1ˆ
C = log(3) - H Á , , ˜ = 0.1258 bits/use
Ë 2 6 3¯

Properties
We now list some properties of channel capacity.
1. C ≥ 0, since I(X; Y) ≥ 0.
2. C £ log|X|, since C = max I(X; Y) £ max H(X) = log|X|.
3. C £ log|Y|, since C = max I(X; Y) £ max H(Y) = log|Y|.

Having developed the notion of capacity of a channel, we shall now try to relate it to reliable communi-
cation over the channel. So far, we have only talked about bits that can be sent over a channel each time it is
used (bits/use). But, what is the number of bits that can be sent per second (bits/sec)? To answer this question
we introduce the concept of Channel Coding.
82 ° Information Theory, Coding and Cryptography

Learning Review
YOU ARE NOW READY TO ATTEMPT THE FOLLOWING QUESTIONS LINKED TO LO 2:
1. Find the capacity of the non-symmetric binary erasure channel shown in Fig. 2.9, where M
p0 and p1 are the a priori probabilities.
2. Find the capacity of the channel shown in Fig. 2.10. M

Fig. 2.9 Fig. 2.10

È1 - p 1- p p p ˘
Í 2 2 2 2 ˙
3. Determine the capacity of the channel given by P = Í ˙. S
Í p p 1- p 1- p ˙
ÍÎ 2 2 2 2 ˙˚

If you have successfully solved the above problems,

The answers are available here. Scan to check

Or
Visit [Link]

2.4 Channel Coding

All real-life channels are affected by noise. Noise causes

LO 3 discrepancies (errors) between the input and the output data
sequences of a digital communication system. For a typical
noisy channel, the probability of bit error may be as high as 10-2. This
means that, on an average, 1 bit out of every 100 bits that are transmitted
level of reliability
is far from adequate. Different applications require different levels of
reliability (which is a component of the quality of service). Table 2.1 lists the typical acceptable bit error rates
for various applications.
 Channel Capacity and Coding ° 83

Table 2.1 Acceptable bit error rates for various applications

Application Probability of Error
Speech telephony 10–4
Voice band data 10–6
Electronic mail, Electronic newspaper 10–6
Internet access 10–6
Streaming services 10–7
Video telephony, High speed computing 10–7

In order to achieve such high levels of reliability we have to resort to the use of Channel Coding. The
basic objective of channel coding is to increase the resistance of the digital communication system to channel
noise. This is done by adding redundancies in the transmitted data stream in a controlled manner.
In channel coding, we map the incoming data sequence to a channel input sequence. This encoding
procedure is done by the Channel Encoder. The encoded sequence is then transmitted over the noisy
channel. The channel output sequence at the receiver is inverse mapped to an output data sequence. This is
called the decoding procedure, and is carried out by the Channel Decoder. Both the encoder and the decoder
are under the designer’s control.
As already mentioned, the encoder introduces redundancy in a prescribed manner. The decoder exploits
this redundancy so as to reconstruct the original source sequence as accurately as possible. Thus, channel
coding makes it possible to carry out reliable communication over unreliable (noisy) channels. Channel
coding is also referred to as Error Control Coding. It is interesting to note here that the source coder reduces

improve reliability.

$ An Error Control Code for a channel, represented by the channel transition probability
matrix p(y|x), consists of:
(i) A message set {1, 2, … , M}.
(ii) An encoding function, X , which maps each message to a unique codeword, i.e., 1 Æ X (1) ,
n n

2 Æ X n ( 2), …, M Æ X n ( M ) . The set of codewords is called a codebook.

(iii) A decoding function, D Æ {1, 2, … , M}, which makes a guess based on a decoding strategy in
order to map back the received vector to one of the possible messages.

! block codes. In this class of codes, the incoming message

" # " ! k bits. Thus the cardinality of the message
set M = 2k. Each k–bit long information block is mapped into an n–bit block by the channel coder, where
n > k. This means that for every k bits of information, (n – k) redundant bits are added. The ratio
k log M
r= = (2.10)
n n
is called the Code Rate. It represents the information bits per transmission. Code rate of any coding scheme
is, naturally, less than unity. A small code rate implies that more and more bits per block are the redundant
bits corresponding to a higher coding overhead. This may reduce the effect of noise, but will also reduce the
communication rate as we will end up transmitting more of the redundant bits and fewer information bits.
The question before us is whether there exists a coding scheme such that the probability that the message bit
$
provided by Shannon in his second theorem regarding the channel capacity. We will study this shortly.
84 ° Information Theory, Coding and Cryptography

% A rate r is said to be achievable if there exists a coding scheme (n, k) such that the
maximal probability of error tends to 0 as n Æ . The (n, k) code may also be expressed as a (2nr, n) code
or a (M, n) code, where M = 2k = 2nr.

Let us now introduce the concept of time in our discussion. We wish to look at questions like
how many bits per second can we send over a given noisy channel with arbitrarily low bit error
rates? Suppose the discrete memoryless source has the source alphabet X and entropy H(X) bits
per source symbol. Let the source generate a symbol every Ts seconds. Then the average information rate of
H (X )
the source is bits per second. Let us assume that the channel can be used once every Tc seconds and
Ts
C
the capacity of the channel is C bits per channel use. Then the channel capacity per unit time is bits per
Tc
second. We now state Shannon’s second theorem known as the Noisy Channel Coding Theorem or simply
the Channel Coding Theorem.

& ' &

(i) Let a DMS with an alphabet X have entropy H(X) and produce symbols every Ts seconds. Let a discrete
memoryless channel have capacity C and be used once every Tc seconds. Then if
H (X ) C
£ (2.11)
Ts Tc
exists a coding scheme for which the source output can be transmitted over the noisy channel and be
reconstructed with an arbitrarily low probability of error.
(ii) Conversely if
H (X ) C
> (2.12)
Ts Tc
It is not possible to transmit information over the channel and reconstruct it with an arbitrarily small
probability of error.
C
The parameter is called the Critical Rate. Thus, all rates below capacity, C, are achievable.
Tc

The channel coding theorem is a very important result in information theory. The theorem

communication can be carried out over an unreliable (noisy) DMS channel. It should be noted
that the channel coding theorem tells us about the existence of some codes that can achieve
reliable communications in a noisy environment. Unfortunately, it does not give us the recipe to construct
these codes. Therefore, channel coding is still an active area of research as the search for better and better

Example 2.4 Consider a DMS source that emits equally likely binary symbols (p = 0.5) once
every Ts seconds. This entropy for this binary source is
H(p) = – p log2 p – (1 – p)log2(1 – p) = 1 bit
 Channel Capacity and Coding ° 85

The information rate of this source is

H (X ) 1
= bits/second
Ts Ts
Suppose we wish to transmit the source symbols over a noisy channel. The source sequence is applied
to a channel coder with code rate r. This channel coder uses the channel once every Tc seconds to send the
coded sequence. We want to have reliable communication (the probability of error as small as desired).
From the channel coding theorem, if
1 C
£ (2.13)
Ts Tc
we can make the probability of error as small as desired by a suitable choice of a channel coding scheme,
and hence have reliable communication. We note that the ratio is equal to the code rate of the coder.
Therefore,
Tc
r= (2.14)
Ts
Hence the condition for reliable communication can be rewritten as
r £C (2.15)

%&' r £ C, which will ensure

reliable communication regardless of how noisy the channel is!
In general, all rates below capacity, C, are achievable. That is, for every rate r £ C there exists
a (2nr, n) code with arbitrarily low probability of error. When we say the probability is ‘as
small as desired’ or ‘arbitrarily low’ we mean it literally! If we want the residual error rate
(i.e., the error rate after error correction) to be 10–20 we can (in principle) achieve it as long
as r £ C. Shall we become more ambitious and desire that the residual error rate be 10–35?
No problem! We need to just ensure that r £ C. Of course, we can state that at least one such code exists, but
+ <
itself by limiting the channel capacity, and hence the code rate.

Example 2.5 Consider a binary symmetric channel (BSC) with a transition probability p = 10–2.
Such error rates are typical of wireless channels. We saw in Example 2.1 that for a BSC, the capacity is
given by
C = 1 + p log2 p + (1 – p)log2(1 – p)
By plugging in the value of p = 10–2 we obtain the channel capacity C = 0.919. From the previous
example we can conclude that there exists at least one coding scheme with the code rate r £ 0.919 which
will guarantee us a (non-zero) probability of error that is ‘as small as desired’. We mean it literally!

Example 2.6 Consider the repetition code in which each message bit is simply repeated n times,
where n is an odd integer. For example, for n = 3, we have the mapping scheme
0 Æ 000; 1 Æ 111
Similarly, for n = 5 we have the mapping scheme
0 Æ 00000; 1 Æ 11111
86 ° Information Theory, Coding and Cryptography

Note that the code rate of the repetition code with blocklength n is
1
r = (2.16)
n
The decoding strategy is as follows: If in a block of n received bits the number of 0’s exceeds the
number of 1’s, decide in favor of 0 and vice versa. This is also known as Majority Decoding. This also
answers the question why should n be an odd integer for repetition codes.
Let n = 2m + 1, where m is a positive integer. This decoding strategy will make an error if more than
m bits are in error, because in that case even if a 0 is encoded and sent, there will be more number of 1’s
in the received word. Let us assume that the a priori probabilities of 1 and 0 are equal. Then, the average
probability of error is given by
n
Ê nˆ
Pe = Â Á ˜ p i (1 - p ) n - i (2.17)
i = m +1 Ë i ¯

where p is the channel transition probability. The average probability of error for repetition codes for
different code rates is given in Table 2.2.

Table 2.2 Average probability of error for repetition codes

Code Rate, r Average Probability of Error, Pe

1 10–2

1/3 3 ¥ 10–4

1/5 10–6

1/7 4 ¥ 10–7

1/9 10–8

1/11 5 ¥ 10–10

probability of error. The decrease in the Pe is much more rapid than the decrease in the
code rate, ! "
and smaller Pe. Thus the repetition code exchanges code rate for message reliability. But
" arbitrarily
low probability of error. The theorem merely requires the code rate r to be less than the channel capacity,
# $ % r ª &'
# * + '
$ +& '& +&& '&&
information bits, …) and give us as small a Pe $ +&-20%/
1 # 3 4 5 3
6
 Channel Capacity and Coding ° 87

Learning Review
YOU ARE NOW READY TO ATTEMPT THE FOLLOWING QUESTIONS LINKED TO LO 3:
1. Consider the binary channel shown in Fig. 2.11.
S
Let the a priori probabilities of sending the
binary symbol be p0 and p1, where p0 = p1 =
0.5. What is the maximum rate at which trans-
mission can be carried out over this channel
with arbitrarily low probability of error?
2. Consider a satellite hop channel shown in Fig. Fig. 2.11
M
2.12. The uplink can be modeled as a BSC with
crossover probability p while the downlink can be modeled as a BSC with crossover
probability q. Assuming the uplink channel and downlink channel are independent, draw
the capacity region for this satellite hop channel. Hint: Choose the two orthogonal axes as
r1 and r2 and mark a 2-D region where reliable communication is achievable.

Fig. 2.12

If you have successfully solved the above problems,

!

The answers are available here. Scan to check

Or
Visit [Link]

2.5 Information Capacity Theorem

So far we have studied limits on the maximum rate at which information
can be sent over a channel reliably in terms of the channel capacity. In LO 4
this section we will formulate the information capacity theorem for
band-limited, power-limited Gaussian channels. An important and useful &
J
!
88 ° Information Theory, Coding and Cryptography

* A Gaussian Channel Zk
is a time discrete channel with output Yk,
at time k, which is the result of the sum
of the input Xk and the Gaussian noise Zk.
This noise is drawn from a Gaussian distribution with
mean zero and variance s 2. Thus, Yk = Xk + Zk, where Xk Yk
Zk ~ N(0, s 2). The noise Zk is independent of the input
Xk. The Gaussian Channel is shown in Fig. 2.13. Fig. 2.13 Gaussian channel.

7 3
!
that these random effects are independent and identically distributed and manifest
themselves as creating electrical disturbances in the communication circuit. Because of the

sum of these independent and identically distributed random effects is Gaussian from the central limit
theorem.
Consider a zero mean, stationary random process X(t) that is band limited to W hertz. Let Xk, k = 1, 2,..., K,
denote the continuous random variables obtained by uniform sampling of the process X(t) at the Nyquist rate
of 2W samples per second. These symbols are transmitted over a noisy channel which is also band-limited
to W Hertz. The channel output is corrupted by AWGN of zero mean and power spectral density (psd) N0/2.
Because of the channel, the noise is band limited to W Hertz. Let Yk, k = 1, 2,..., K, denote the samples of the
received signal. Therefore,
Yk = Xk + Nk, k = 1, 2, ..., K (2.18)
where Nk is the noise sample with zero mean and variance s2 =N0W. It is assumed that Yk, k = 1, 2,..., K, are
statistically independent. Since the transmitter is usually power-limited, let us put a constraint on the average
power in Xk:
E[ X k2 ] = P , k = 1, 2,..., K (2.19)
The information capacity of this band-limited, power-limited channel is the maximum of the mutual infor-
mation between the channel input Xk and the channel output Yk. The maximization has to be done over
all distributions on the input Xk that satisfy the power constraint of equation (2.19). Thus, the information
capacity of the channel (same as the channel capacity) is given by
C = max {I ( X ; Y ) | E[ X k2 ] = P} (2.20)
f Xk ( x)

where f xk ( x) is the probability density function of Xk.

Now, from Chapter 1, equation (1.32), we have,
I ( X k ; Yk ) = h(Yk ) - h(Yk | X k ) (2.21)
Note that Xk and Nk are independent random variables. Therefore, the conditional differential entropy of Yk
given Xk is equal to the differential entropy of Nk. Intuitively, this is because given Xk the uncertainty arising
in Yk is purely due to Nk. That is,
h(Yk | X k ) = h( N k ) (2.22)
 Channel Capacity and Coding ° 89

Hence we can write equation (2.21) as

I ( X k ; Yk ) = h(Yk ) - h( N k ) (2.23)
Since h(Nk) is independent of Xk, maximizing I ( X k ; Yk ) translates to maximizing h(Yk). It can be shown
that in order for h(Yk) to be maximum, Yk has to be a Gaussian random variable. If we assume Yk to be
Gaussian, and Nk J Xk is also Gaussian. This is because the sum (or difference)
of two Gaussian random variables is also Gaussian. Thus, in order to maximize the mutual information
between the channel input Xk and the channel output Yk, the transmitted signal should be Gaussian. Therefore,
we can rewrite (2.20) as
C = I ( X ; Y ) | E[ X k2 ] = P and Xk is Gaussian (2.24)
We know that if two independent Gaussian random variables are added, the variance of the resulting
Gaussian random variable is the sum of the variances. Therefore, the variance of the received sample Yk
equals P + N0W. It can be shown that the differential entropy of a Gaussian random variable with variance
1 2
s is log 2 (2p es ) . Therefore,
2
1
h(Yk) = log 2 [ 2p e( P + N 0W ) ] (2.25)
2
1
and h(Nk) = log 2 [ 2p e( N 0W ) ] (2.26)
2
Substituting these values of differential entropy for Yk and Nk we get

1 Ê P ˆ
C= log 2 Á1 + bits per channel use (2.27)
2 Ë N 0W ˜¯
We are transmitting 2W samples per second, i.e., the channel is being used 2W times in one second.
Therefore, the information capacity can be expressed as

Ê P ˆ
C = W log 2 Á1+ bits per second (2.28)
Ë N 0W ˜¯
This basic formula for the capacity of the band-limited, AWGN waveform channel with a band-limited
# & KPQR ! & U
P
theorem, the Information Capacity Theorem. Since N 0W is the average noise power, SNR = .
Therefore, (2.28) can be succinctly expressed as C = W log (1 + SNR). N 0W
2

Example 2.7 Consider telephone channel with bandwidth 4 kHz and SNR ª 25 dB. Let us model
it as a band-limited and power-limited Gaussian channel.
SNR in dB is calculated as 10 log10(SNR). Hence, 25 dB SNR = 102.5 = 316.22.
The capacity of this channel is C = W log2(1 + SNR) = 4 × 103 log2(1 + 316.22) = 33,238 b/s.
If we increase the SNR by 1 dB (i.e., SNR ª 26 dB), the capacity increases to 34,563 b/s.
So, a 1 dB increase in SNR bought us roughly an additional 1325 b/s.
90 ° Information Theory, Coding and Cryptography

The ITU V.90 modem, that operates on the telephone line, provides a data rate of 33,600 b/s. Thus,
the ITU V.90 modem operates very close to the capacity of the channel! If the channel SNR is poor, a
lower data rate is used by the modem.

& . & The information capacity of a continuous channel of

bandwidth W Hertz, perturbed by additive white Gaussian noise of power spectral density N0/2 and limited
in bandwidth to W, is given by
Ê P ˆ
C = W log 2 Á1+ ˜ bits per second
Ë N 0W ¯

where P is the average transmitted power. This theorem is also called the &
The Information Capacity Theorem is one of the important results in information theory. In a
single formula one can see the trade off between the channel bandwidth, the average transmitted
power and the noise power spectral density. Given the channel bandwidth and the SNR the
channel capacity (bits/second) can be computed. This channel capacity is the fundamental limit
on the rate of reliable communication for a power-limited, band-limited Gaussian channel. It should be kept
in mind that in order to approach this limit, the transmitted signal must have statistical properties that are
Gaussian in nature. Note that the terms channel capacity and information capacity have been used
interchangeably.
Let us now derive the same result in a more intuitive manner. Suppose we have a coding scheme that
results in an acceptably low probability of error. Let this coding scheme take k information bits and encode
them into n bit long codewords. The total number of codewords is M = 2k. Let the average power per bit be
P. Thus the average power required to transmit an entire codeword is nP. Let these codewords be transmitted
2
over a Gaussian channel with the noise variance equal to s . The received vector of n bits is also Gaussian
2
with the mean equal to the transmitted codeword and the variance equal to ns . Since the code is a good one
2
(acceptable error rate), the vector lies inside a sphere of radius ns , centered on the transmitted codeword.
2 2
This sphere itself is contained in a larger sphere of radius n( P + s ) , where n( P + s ) is the average power
of the received vector.
This concept may be visualized as depicted in Fig. 2.14. There
is a large sphere of radius n( P + s 2 ) which contains M smaller
2
spheres of radius ns . Here M = 2k is the total number of
codewords. Each of these small spheres is centered on a codeword.
These are called the Decoding Spheres. Any received word lying
within a sphere is decoded as the codeword on which the sphere is
centered. Suppose a codeword is transmitted over a noisy channel.
Then there is a high probability that received vector will lie inside
the correct decoding sphere (since it is a reasonably good code). The
question arises: How many non-intersecting spheres can be packed
inside the large sphere? The more number of spheres one can pack,
Fig. 2.14 Visualization of the sphere
packing problem.
known as the Sphere Packing Problem.
 Channel Capacity and Coding ° 91

The volume of an n-dimensional sphere of radius r can be expressed as

n
V = An r (2.29)
where An is a scaling factor. Therefore, the volume of the large sphere (sphere of all possible received vectors)
can be written as:
( )
n/2
Vall = An ÈÎn P + s ˘˚
2
(2.30)

and the volume of the decoding sphere can be written as:

n/2
Vds = An ÈÎns ˘˚
2
(2.31)

The maximum number of non intersecting decoding spheres that can be packed inside the large sphere of
all possible received vectors is

( )
n/2
An Èn P + s 2 ˘ Ê Pˆ ( n / 2 )log 2 (1+ P / s 2 )
Î ˚ = Á1 + 2 ˜ = 2
M = (2.32)
n/2 Ë s ¯
An ÈÎns 2 ˘˚

On taking logarithm (base 2) on both sides of the equation we get

n Ê Pˆ
log2M = log 2 Á1 + 2 ˜ (2.33)
2 Ë s ¯
Observing that k = log2M, we have
k 1 Ê Pˆ
= log 2 Á1 + 2 ˜ (2.34)
n 2 Ë s ¯
k
Note that each time we use the channel, we effectively transmit bits. Thus, the maximum
n
number of bits that can be transmitted per channel use, with a low probability of error, is
1 Ê Pˆ
log 2 Á1 + 2 ˜ , as seen previously in equation (2.27). Note that s 2 represents the noise power
2 Ë s ¯
N
and is equal to N 0W for AWGN with power spectral density 0 and limited in bandwidth to W.
2

Example 2.8 Consider a communi-

cation system, as depicted in Fig. 2.15 with
two Gaussian noise sources Z1 and Z2 that are
independent, have zero mean and variances
s12 and s 22 respectively. Assume that X is
Fig. 2.15 Two noise sources.
constrained to have power E[X 2] = P.
Since the two noise sources Z1 and Z2 are independent, the effective noise can be modeled as Gaussian
2 2
with zero mean and variance s1 + s 2 . Therefore, the capacity of this channel with two noise sources is
simply
92 ° Information Theory, Coding and Cryptography

1 Ê P ˆ
C= log 2 Á1 + 2 ˜ bits per channel use.
2 Ë s1 + s 22 ¯
Now, suppose Z2 is known to the receiver. In this case, the noise Z2 can be subtracted at the receiver,
thereby yielding the capacity
1 Ê Pˆ
C= log 2 Á1 + 2 ˜ bits per channel use.
2 Ë s1 ¯

Example 2.9 Consider two parallel Gaussian

channels as shown in Fig 2.16. The input X is transmitted
over these Gaussian channels. X is constrained to have
power E[X2] = P. The noises Z1 and Z2 are independent
2 2
and have variances s1 and s 2 .
At the receiver we obtain Y1 and Y2 <
us combine the outputs to produce
Y = Y + Y = X + X + (Z + Z ) = 2X + Z,
where Z = Z1 + Z2. This is a Gaussian channel with
received power = E[(2X)2] = 4P. Fig. 2.16 Parallel gaussian channels.
Since Z1 and Z2 are independent, the total noise power
is the sum of the variances
s 2 = s12 + s 22 .
The capacity of the X to Y channel is the Shannon capacity given by

1 Ê 4P ˆ
C= log 2 Á1 + 2 ˜ bits per channel use
2 Ë s1 + s 22 ¯

81
received signals
Y = aY1 + (1 – a)Y2
where 0 £ a £ 1. Thus we have
Y = a(X + Z1) + (1 – a)(X + Z2) = X + Z,
where Z = aZ1 + (1 – a)Z2. This is a Gaussian channel with received power = E[(X)2] = P. The total noise
power is given by
s 2 = a 2s12 + (1 - a ) s 22
2

ds 2
In order to maximize capacity, we must minimize the total noise power. Setting = 0 we obtain
da
s 22
a opt =
s12 + s 22
 Channel Capacity and Coding ° 93

2 s12s 22
and the minimum total noise power s min = . Using this, we obtain the maximum capacity as
s12 + s 22
1 Ê P Pˆ
C¢ = log 2 Á1 + 2 + 2 ˜ bits per channel use
2 Ë s1 s 2 ¯
2 2
One can show that C ¢ ≥ C , with the condition for equality, s1 = s 2 . This implies that if the two
2 2
channels are not equally good ( s1 π s 2 ), one must use weighted combining in order to maximize
capacity.

In the previous example, equal power is transmitted over the two independent channels. In the case when
the channels are not equally good (or bad!), we do optimal combining. To do so, we need the Channel State
Information (CSI). Can we use the CSI at the transmitter to increase the capacity? We explore that exciting
option in the next section.

2.6 Parallel Gaussian Channels LO 4

Consider k independent parallel Gaussian channels, as depicted Z1
in Fig. 2.17. Let us assume that there exists a constraint on the
total transmit power. The aim is to distribute the total available
X1 Y1
power among the k channels so as to maximize the capacity.
The output of channel j is given by Z2

Yj = Xj + Zj (2.35)
2 X2 Y2
with Zi ~ N(0, s j ). The noise in each channel is assumed to be
independent. The total power constraint can be expressed as
k
E Â X 2j £ P (2.36) Zk
j =1

The information capacity of these k independent parallel Xk Yk

Gaussian channels can be written as

I ( X1 , X 2 ,… , X k ; Y1 , Y2 ,… , Yk )
Fig. 2.17 Parallel independent
C= max (2.37)
f ( x1 , x2 ,..., xk ), E [ X i2 ] £ P gaussian channels.

We can simplify the right hand side as follows:

I ( X1 , X 2 ,… , X k ; Y1 , Y2 ,… , Yk )
= h (Y1 , Y2 ,… , Yk ) - h (Y1 , Y2 ,… , Yk | X1 , X 2 ,… , X k ) (2.38)
94 ° Information Theory, Coding and Cryptography

Since the uncertainty in the outputs {Y1 , Y2 ,… , Yk } , given the inputs { X1 , X 2 ,… , X k } , depends only on the
noise {Z1 , Z 2 ,… , Z k } , we can write

I ( X1 , X 2 ,… , X k ; Y1 , Y2 ,… , Yk )
= h (Y1 , Y2 ,… , Yk ) - h ( Z1 , Z 2 ,… , Z k | X1 , X 2 ,… , X k )
k
= h (Y1 , Y2 ,… , Yk ) - h ( Z1 , Z 2 ,… , Z k ) = h (Y1 , Y2 ,… , Yk ) - Â h ( Z i ) (2.39)
i =1

The last step follows from the assumption that the noise is independent of the input in the channel, and the
k
noise in each channel is independent. Using the fact that h (Y1 , Y2 ,… , Yk ) £ Â Yi , we have
i =1

I ( X1 , X 2 ,… , X k ; Y1 , Y2 ,… , Yk )
k k
£ Â h (Yi ) - Â h ( Zi )
i =1 i =1
k
1 Ê P ˆ
£ Â log Á1 + i2 ˜ (2.40)
i =1 2 Ë si ¯
k
2
where Pi = E[ X i ] and  Pi = P . For equality in the last step, Xi ~ N(0, Pi) for i = 1, 2, …, k. Maximizing
i =1
the power allocation in each channel in order to maximize the capacity, we obtain

Pi = (p - s i2 ) + such that
k
 (p - s i2 )+ = P (2.41)
i =1
+
Here, ( x)
Ï x if x ≥ 0
( x)+ = Ì (2.42)
Ó0 if x < 0

From (2.41), the optimal power allocation strategy can be visualized as follows. Channel
2
j can be assumed to be a solid block of height s j . All blocks (channels) are placed inside
a container, as shown in Fig. 2.18(a). We now pour water (power) into the container such
that the water level in the container rises to p, as depicted in Fig. 2.18(b). The water level
above each block (channel) indicates how much power is to be allocated to that channel. For
2 2
example, channel j with noise power s j gets (p – s j ) power allocated to it. Those blocks (channels) that
are above the water level p do not get any power allocated to them. Because of this physical analogy, this
algorithm is also known as the Water Filling or 0 1 2
 Channel Capacity and Coding ° 95

(a) (b)
Channels Unused Power
channels allocation =p – s 2j

Level p
sk2 s k2
s 22 s 22
s12 2
sk–1 s12 2
sk–1
s j2 s 2j

Fig. 2.18
allocations.

2.7 The Shannon Limit LO 4

Consider a Gaussian channel that is limited both in power and bandwidth. We wish to explore the limits of a

rate Rb which is equal to the capacity, C, of the channel, i.e., Rb = C. Suppose the energy per bit is Eb. Then
the average transmitted power is
P = EbRb = EbC (2.43)
Therefore, the channel capacity theorem for this ideal system can be written as:

C Ê E Cˆ
= log 2 Á1+ b ˜ (2.44)
W Ë N0 W ¯
This equation can be re-written in the following form:

Eb 2C W - 1
= (2.45)
N0 CW

Rb E
versus b is called the 3 4 5 , and
W N0
is given in Fig. 2.19. The ideal system is represented by the line Rb = C.

Eb
tends to the limiting value
N0
Eb
= ln 2 = 0.693 = – 1.6 dB (2.46)
N0 W
96 ° Information Theory, Coding and Cryptography

101

Rb/W
Shannon
Limit
100

10–1 Eb/N0
0 10 20 30 40 dB

Fig. 2.19

This value is called the Shannon Limit. It is interesting to note that the Shannon limit is a fraction. This
implies that for very large bandwidths, reliable communication is possible even for the case when the signal
power is less than the noise power! The channel capacity corresponding to this limiting value is
P
CW = log2e (2.47)
N0
"
(ii) The curve for the critical rate Rb = C is known as the Capacity Boundary. For the case Rb > C, reliable
communication is not guaranteed. However, for Rb < C, there exist some coding scheme which can
provide an arbitrarily low probability of error.
Rb Eb
# $ , and the
W N0
probability of error, Pe. Note that for designing any communication system the basic design parameters
are the bandwidth available, the SNR and the bit error rate (BER). The BER is determined by the
application and the quality of service (QoS) desired. The bandwidth and the power can be traded one
for the other to provide the desired BER.
% &
% " ' '*"
The information capacity theorem predicts the maximum amount of information that can be
transmitted through a given bandwidth for a given SNR. We see from Fig. 2.19 that acceptable
capacity can be achieved even for low SNRs, provided adequate bandwidth is available. We also
note that the capacity is a linear function of bandwidth and a logarithmic function of SNR. Thus the
optimum usage of a given bandwidth is obtained when the signals are noise-like (i.e., of large bandwidth) and
a minimal SNR is maintained at the receiver. This principle lies in the heart of any spread spectrum
communication system, like Code Division Multiple Access (CDMA). In the light of this theorem one can
justify why all the Third Generation (3G) mobile systems are CDMA based.
 Channel Capacity and Coding ° 97

2.8 Channel Capacity for MIMO Systems LO 4

Let us model the MIMO channel described by (2.4) using the matrix H of dimension MR ¥ MT. Assuming the
average transmit symbol energy is Es, the sampled signal model can be expressed as

Es
y[k] = Hs[k] + n[k] (2.48)
MT
where y[k] is the received signal vector with dimension MR ¥ 1, s[k] is the transmit signal vector with
dimension MT ¥ 1 and n[k] is the MR ¥ 1 spatio-temporal zero mean, complex Gaussian white noise vector
with variance N0 in each dimension. This signal model is true for Frequency Flat Fading
channels. We can drop the time index k for clarity and write the above equation as

Es
y= Hs + n (2.49)
MT
The covariance matrix of s is given by
Rss = E{ssH} (2.50)
The superscript ‘H’ denotes the Hermitian operation. We make the assumptions that the channel, H,
is deterministic and known to the receiver. The Channel State Information (CSI) can be obtained at the
receiver using a pilot or training signals. The capacity of the MIMO channel is given by

Ê Es ˆ
C= max W log 2 det Á I M R + HRss H H ˜ bits per second (2.51)
Tr ( RSS )= M T Ë M N
T 0 ¯

where W is the bandwidth and I M R denotes the identity matrix of size MR. The condition Tr(Rss) = MT
constrains the total average energy transmitted over a symbol period.
If the channel is unknown to the transmitter, the vector s may be chosen such that Rss = I M T . This simply
means that the signals at the transmit antennas are independent and of equal power. The capacity of the
MIMO channel in this case is given by

r
Ê Es ˆ
C = W Â log 2 Á1 + li ˜ bits per second (2.52)
i =1 Ë M T N0 ¯

where r is the rank of the channel and li (i = 1, 2, …, r) are the positive eigenvalues of HHH. It is interesting
to observe that the capacity of the MIMO channel unknown to the transmitter comes out as the sum of r SISO
E
channels, each having power gain li (i = 1, 2, …, r) and equal transmit power, s . It can be interpreted as
MT
follows. The use of multiple transmit and receive antennas have opened multiple parallel data pipes between
the transmitter and receiver. The number of these scalar data pipes depends on the rank of H.
Next, consider a full-rank MIMO channel with MT = MR = M, so that r = M. The maximum capacity is
achieved when H is an orthogonal matrix, i.e., HHH = HHH. The capacity of this MIMO channel is given by
98 ° Information Theory, Coding and Cryptography

Ê E ˆ
C = WM log 2 Á1 + s ˜ bits per second (2.53)
Ë N0 ¯

The capacity of an orthogonal MIMO channel is simply M times the scalar channel capacity.

If the channel is known to the transmitter, the different scalar data pipes may be accessed
individually through processing at the transmitter and receiver. The basic idea is to allocate
variable energy across the different data pipes in order to maximize the mutual information. The
optimal energy can be found iteratively using the water pouring algorithm. The capacity of a
MIMO channel when the channel is known to the transmitter is necessarily greater than or equal to the
capacity when the channel is unknown to the transmitter.

INDUSTRIAL RELEVANCE
+ 0+ 1* 8 ¥ 2 ' '( # ,
4 ¥ 3 ' '( , + ' '(% -
! & 5

2.9 Capacity Region for Multiple Access Channels LO 4

Multiple access channels are commonly found in the domain of cellular communications where several
mobile phone users try to access the same base station over a common channel. Similarly, a satellite receiver
with several independent ground weather stations constitutes a multiple access channel. We note that, in
addition to noise, there is interference generated by the multiple users (one user’s signal is someone else’s
interference!). In a classroom scenario, several students asking questions to the instructor at the same time
also tantamount to a multiple access channel. Clearly, if two students ask two different questions simultane-

of clever coding.

6 A discrete memoryless Multiple Access Channel with two senders and one receiver
can be characterized by three alphabets, X1, X2 and Y and a corresponding probability transition matrix
p(y| x1, x2).
 Channel Capacity and Coding ° 99

Suppose, the rates of the two transmitters are r1 and r2 respectively. Let the two transmitters accessing the
common channel use two encoding functions, E1 and E2 in order to encode message sets M1 = 1, 2,… , 2 1{ nr
}
{ nr
}
and M2 = 1, 2,… , 2 2 into X1 and X2, i.e.,
E1: M1 Æ X1 and E2: M2 Æ X2 (2.54)
where n is the length of the encoded sequence Xi, i = 1, 2. The receiver needs to have a decoding function D
such that
D: Y Æ X1 ¥ X2 (2.55)

7 For a multiple access channel, a rate pair (r1, r2) is achievable if there exists codes
((2nr1 , 2nr2 ), n) such that the average probability of error Pe Æ 0.

For a multiple access channel, the Capacity Region is the closure of the set of achievable
rate pairs (r1, r2).

Example 2.10 Consider a multiple access channel with two independent senders and one receiver.
Suppose sender one has a binary symmetric channel with crossover probability p1, and sender two has
a binary symmetric channel with crossover probability p2. If the two channels are indeed independent,
then they experience no interference from each other (they could be using orthogonal frequencies). Thus,
sender one can send information at rate C1 = 1 – H(p1) and similarly, sender two can send information at
rate C2 = 1 – H(p2). The capacity region is shown in Fig. 2.20.

r1

1 – H( p1)

r2
1 – H( p1)

Fig. 2.20 The capacity region for independent binary symmetric channels.
100 ° Information Theory, Coding and Cryptography

Learning Review
YOU ARE NOW READY TO ATTEMPT THE FOLLOWING QUESTIONS LINKED TO LO 4:
1. A telephone channel has a bandwidth of 3000 Hz and the SNR = 20 dB. S
(i) Determine the channel capacity.
(ii) If the SNR is increased to 25 dB, determine the increased capacity.
2. Suppose a TV displays 30 frames/second. There are approximately 2 ¥ 105 pixels per M
frame, each pixel requiring 16 bits for color display. Assuming an SNR of 25 dB calculate
the bandwidth required to support the transmission of the TV video signal.
s2, the Gaussian random variable has the largest M
differential entropy attainable by any random variable.
1 2
(ii) Show that this entropy is given by log 2 (2p es ) .
2
4. Consider a channel consisting of two parallel AWGN channels with inputs X1, X2 and D
outputs
Y1 = X1 + Z1
Y2 = X2 + Z2
The noises Z1 and Z2 are independent and have variances s12 and s22 with s12 < s22.
However, we are constrained to use the same symbol on both channels, i.e. X1 = X2 = X,
where X is constrained to have power E[X 2] = P. Suppose at the receiver, we combine
the outputs to produce Y = Y1 + Y2 . What is the capacity C1 of channel with input X and
output Y ?
5. Consider two parallel channels with independent Gaussian noise Z1 and Z2 with variances D
s12 = 1 and s 22 = 2. The signal at the receiver is given by
Y1 = X1 + Z1
Y2 = X2 + Z2
2 2
The transmitter is subject to the power constraint E[ X1 + X 2 ] £ P.
Find and sketch the capacity C(P) of this channel as a function of P. What happens when
P becomes very small?
If you have successfully solved the above problems,
"

The answers are available here. Scan to check

Or
Visit [Link]

2.10 Random Selection of Codes

Consider a set of M coded signal waveforms constructed from a set of
LO 5 n-dimensional binary codewords. Let us represent these codewords as
follows:
! Ci = [ci1ci 2 ...cin ] , i = 1,2, ...M (2.56)
 Channel Capacity and Coding ° 101

Since we are considering binary codes, cij is either a 0 or a 1. Let each bit of the codeword be mapped on
to a BPSK waveform so that the codeword may be represented as
n
si (t ) = Â sij p j (t ) , i = 1, 2, ...M (2.57)
i =1
where
ÏÔ E for cij = 1
sij = Ì (2.58)
ÔÓ - E for cij = 0
and E is the energy per code bit. The waveform si (t ) can then be represented as the n-dimensional vector
si = [ si1 si 2 ... sin ] , i = 1, 2, ...M (2.59)
We observe that this corresponds to a hypercube in the n-dimensional space. Let us now encode k bits of
information into an n bit long codeword, and map this codeword to one of the M waveforms. Note that there
are a total of 2k possible waveforms corresponding to the M = 2k different codewords.
Let the information rate into the encoder be R bits/sec. The encoder takes in k bits at a time and maps the
k-bit block to one of the M waveforms. Thus k = RT and M = 2k
D as follows:
n
D = dimensions/sec (2.60)
T
n = DT is the dimensionality of the space. The hypercube mentioned above has 2n = 2DT vertices. Of these,
we must choose M = 2RT to transmit the information. Under the constraint D > R, the fraction of vertices that
can be used as signal points is
2k 2 RT
F= = = 2- ( D - R )T (2.61)
2n 2 DT
For D > R, F Æ 0 as T . Since, n = DT, it implies that F Æ 0 as n . Designing a good coding
scheme translates to choosing M vertices out of the 2n vertices of the hypercube in such a manner that the
probability of error tends to zero as we increase n. We saw that the fraction F tends to zero as we choose
larger and larger n. This implies that it is possible to increase the minimum distance between these M signal
points as n . Increasing the minimum distance between the signal points would give us the probability
of error, Pe Æ 0.
There are (2n ) M distinct ways of choosing M out of the total 2n vertices. Each of these choices
corresponds to a coding scheme. For each set of M waveforms, it is possible to design a
communication system consisting of a modulator and a demodulator. Thus there are 2nM
communication systems, one for each choice of the M coded waveforms. Each of these
communication systems is characterized by its probability of error. Of course, many of these communication
systems will perform poorly in terms of the probability of error.
Let us pick one of the codes at random from the possible 2nM sets of codes. The random selection of this
mth codeword occurs with the probability
1
P({si }m ) = nM
(2.62)
2
Let the corresponding probability of error for this choice of code be Pe ({si }m ). Then the average proba-
bility of error over the ensemble of codes is
102 ° Information Theory, Coding and Cryptography

2nM 2nM
1
Pe = Â Pe ({si }m )P({si }m ) = 2nM Â Pe ({si }m ) (2.63)
m =1 m =1

We will next try to upper bound this average probability of error. If we have an upper bound on Pe , then
we can conclude that there exists at least one code for which this upper bound will also hold. Furthermore, if
Pe Æ 0 as n , we can surmise that Pe ({si }m ) Æ 0 as n .
Consider the transmission of a k-bit message X k = [ x1 x2 ...xk ] where xj is binary for j = 1,2,..., k. The
conditional probability of error averaged over all possible codes is
Pe ( X k ) = Â Pe ( X k ,{si }m )P({si }m ) (2.64)
all codes

where Pe ( X k ,{si }m ) is the conditional probability of error for a given k-bit message X k = [ x1 x2 ...xk ] , which
is transmitted using the code {si }m . For the mth code,
M
Pe ( X k ,{si }m ) £ Â P2 m ( sl , sk ) (2.65)
l =1
l πk

where, P2 m ( sl , sk ) is the probability of error for the binary communication system using signal vectors sl and
sk. to transmit one of two equally likely k-bit messages. Hence,
M
Pe ( X k ) £ Â Pe ({si }m )Â P2 m ( sl , sk ) (2.66)
all codes l =1
l πk
On changing the order of summation we obtain
M È ˘ M
Pe ( X k ) £ Â Í Â Pe ({si }m )P2 m ( sl , sk ) ˙ £ Â P2 ( sl , sk ) (2.67)
l =1 Î all codes ˚ l =1
l πk l πk

where P2 ( sl , sk ) represents the ensemble average of P2 m ( sl , sk ) over the 2nM codes. For additive white
Gaussian noise channel,
Ê d2 ˆ
P2 m ( sl , sk ) = Q Á lk
˜ (2.68)
ÁË 2 N 0 ˜¯

1 2

Ú e - z / 2 dz . If sl and sk differ in d coordinates,

2
where dlk2 = sl - sk and Q( x) =
2p x
n
= Â ( slj - skj ) 2 = d (2 E ) 2 = 4dE
2
dlk2 = sl - sk (2.69)
j =1
Therefore,
Ê 2dE ˆ
P2 m ( sl , sk ) = Q Á ˜ (2.70)
Ë N0 ¯
 Channel Capacity and Coding ° 103

Under the assumption that all codes are equally probable, it is equally likely that the vector sl is any of the
2n vertices of the hypercube. Further, sl and sk are statistically independent. Hence, the probability that sl and
sk differ in exactly d places is
n
Ê 1 ˆ Ê nˆ
P(d ) = Á ˜ Á ˜ (2.71)
Ë 2¯ Ë d ¯
The expected value of P2 m ( sl , sk ) over the ensemble of codes is then given by
n Ê 2dE ˆ Ê 1 ˆ n Ê n ˆ Ê 2dE ˆ
P2 ( sl , sk ) =  P ( d )Q Á N ˜ = ÁË n ˜¯  ÁË d ˜¯ Q Á N ˜
Ë
(2.72)
d =0 0 ¯ 2 d =0 Ë 0 ¯

Using the following upper bound

- dE
Ê 2dE ˆ N0
QÁ ˜ <e (2.73)
Ë N0 ¯
we obtain
n
- dE È Ê -E ˆ ˘
Ê 1ˆ
n
Ê nˆ 1
P2 ( sl , sk ) £ Á n ˜ Â Á ˜ e N0
£ Á1 + e N0 ˜ ˙
Í (2.74)
Ë 2 ¯ d =0 Ë d ¯ Í2 Á ˜¯ ˙
Î Ë ˚
From (2.67) and (2.74) we obtain
n n
M È Ê -E ˆ ˘ È Ê -E ˆ ˘
1 Í 1 Á1 + e N0 ˜ ˙
Pe ( X k ) £ Â P2 ( sl , sk ) = ( M - 1) Á1 + e N0 ˜ ˙ < M
Í (2.75)
Í2 Á ˜¯ ˙ Í2 Á ˜¯ ˙
Î Ë Î Ë
l =1
˚ ˚
lπk

Recall that we need an upper bound on Pe , the average error probability. To obtain Pe we average Pe ( X k )
over all possible k-bit information sequences. Thus,
n n
È Ê -E ˆ ˘ È Ê -E ˆ ˘
1 1
Pe = Â Pe ( X k )P ( X k ) < M Í Á 1 + e N0 ˜ ˙ Â k
P ( X ) < M Í Á 1 + e N0 ˙
˜ (2.76)
Í2 Á ˜¯ ˙ Í 2 Á ˜¯ ˙
k
Î Ë ˚ k
Î Ë ˚

The Cutoff Rate R7

Ê -E ˆ
2
R0 = log 2 -E
= 1 - log 2 Á1 + e N0 ˜ (2.77)
ÁË ˜¯
N0
1+ e

The cutoff rate has the units of bits/dimension. Observe that 0 £ R0 £ 1 . The plot of R0 with respect to
SNR per dimension is given in Fig. 2.21.
104 ° Information Theory, Coding and Cryptography

The equation (2.76) can now be written succinctly as

Pe < M 2- nR0 = 2- RT 2- nR0 (2.78)
Substituting n = DT we obtain
Pe < 2-T ( DR0 - R ) (2.79)
If we substitute T = n/D, we obtain
Pe < 2- n ( R0 - R / D ) (2.80)

0.9

0.8

0.7

0.6

R0 0.5

0.4

0.3

0.2

0.1

0
–10 –5 0 5 10
E/N0 (dB)
Fig. 2.21 Cutoff rate, R0, versus the SNR (in dB) per dimension.

Observe that
R R RT k
= = = = Rc (2.81)
D n /T n n
Here Rc represents the code rate. Hence the average error probability can be written in the following
instructive form
Pe < 2- n ( R0 - Rc ) (2.82)
From the above equation, we can conclude the following:
(i) For Rc < R0 the average probability of error Pe Æ 0 as n . Since by choosing large
values of n, Pe can be made arbitrarily small, there exist good codes in the ensemble (which
have the probability of error less than Pe ).
(ii) We observe the fact that Pe is the ensemble average. Therefore, if a code is selected at random, the
probability that its error Pe > a Pe is less than 1/a. This implies that there are no more than 10% of the
codes that have an error probability that exceeds 10 Pe . Thus, there are many good codes.
 Channel Capacity and Coding ° 105

(iii) The codes whose probability of error exceed Pe are not always bad codes. The probability of error of
these codes may be reduced by increasing the dimensionality, n.
E
For binary coded signals, the cutoff rate, R0, saturates at 1 bit/dimension for large values of , say
N0
E
> 10 . Thus, to achieve lower probabilities of error one must reduce the code rate, Rc. Alternately, very
N0
large block lengths have to be used.
at high SNRs. For high SNR scenarios, non-binary coded signal sets should be used to achieve an increase
in the number of bits per dimension. Multiple-amplitude coded signal sets can be easily constructed from
non-binary codes by mapping each code element into one of the possible amplitude levels (e.g. PAM). For
random codes using M-ary multi-amplitude signals, it was shown by Shannon (in 1959) that

È 2˘ È Ê 2 ˆ˘
1Í E Ê Eˆ ˙ 1 Í 1Á Ê Eˆ ˙
R0* = 1+ - 1+ Á log 2 e + log 2 1+ 1+ Á ˜ (2.83)
2 Í N0 Ë N 0 ˜¯ ˙ 2 Í2 Á Ë N 0 ˜¯ ˜ ˙
ÍÎ ˙˚ ÍÎ Ë ¯ ˙˚
*
Let us now relate the cutoff rate R0 to the capacity of the AWGN channel, which is given by

Ê P ˆ
C = W log 2 Á1+ bits per second (2.84)
Ë N 0W ˜¯
where P is the average power and W is the channel bandwidth. The energy per code bit is equal to
PT
E= (2.85)
n
Recall that from the sampling theorem, a signal of bandwidth W may be represented by samples taken
at a rate 2W. Thus, in the time interval of length T there are n = 2WT samples. Therefore we may write
n
D = = 2W . Hence,
T
nE
P= = DE (2.86)
T
C C
Cn = = and substitute for W and P in (2.84) to obtain
2W D
Ê 1ˆ Ê Eˆ
Cn = Á ˜ log 2 Á1 + 2 ˜
Ë 2¯ Ë N0 ¯
Ê 1ˆ
= Á ˜ log 2 (1 + 2 Rcg b ) (2.87)
Ë 2¯
*
where g b is the SNR per bit and Rc is the code rate. The normalized capacity, Cn and cutoff rate, R0 , are
!

* E
(i) R0 < Cn for all values of . This is expected because Cn is the ultimate limit on the
N0
transmission rate R/D.
106 ° Information Theory, Coding and Cryptography

E *
(ii) For smaller values of , the difference between Cn and R0 is approximately 3 dB. This means that
N0 *
randomly selected, average power limited, multi-amplitude signals yield R0 within 3 dB of channel
capacity.
2.5

1.5

R0* and C n
1 Cn

R *0
0.5

0
–10 –5 0 5 10
E/N0 (dB)

*
Fig. 2.22 The normalized capacity, Cn and cutoff rate, R0 , for an AWGN channel.

Learning Review
YOU ARE NOW READY TO ATTEMPT THE FOLLOWING QUESTIONS LINKED TO LO 5:

( )
1. For a BSC, the cut-off rate is given by R0 = 1 - log 2 1 + 4 p (1 - p ) where p is the
S
crossover probability of the BSC.
(i) For what value of p is the cut-off rate minimum? What does it physically mean?
(ii) Plot the cut-off rate as a function of p.
2. Consider a communication system using antipodal signaling. The SNR is 20 dB. D
(i) Find the cutoff rate, R0.
(ii) We want to design a code which results in an average probability of error, Pe < 10-6.
What is the best code rate I can achieve?
(iii) What will be the dimensionality, n, of this code?
(iv) Repeat the earlier parts (i), (ii) and (iii) for an SNR = 5dB. Compare the results.

If you have successfully solved the above problems,

#

The answers are available here. Scan to check

Or
Visit [Link]
 Channel Capacity and Coding ° 107

2.11 Concluding Remarks

Pioneering work in the area of channel capacity was done by Shannon in 1948. Shannon’s second theorem
was indeed a surprising result at the time of its publication. It claimed that the probability of error for a
binary symmetric channel could be made as small as desired provided the code rate was less than the channel
capacity. This theorem paved way for a systematic study of reliable communication over unreliable (noisy)
channels. Shannon’s third theorem, the information capacity theorem, is one of the most remarkable results
in information theory. It gives a relation between the channel bandwidth, the signal to noise ratio and the
channel capacity. Additional work was carried out in the 1950s and 1960s by Gilbert, Gallager, Wyner,
Forney and Viterbi to name some of the prominent contributors. The work related to capacity of MIMO
channels was pioneered by Foschini and Telatar in the second half of 1990. The concept of cutoff rate was
also developed by Shannon, but was later used by Wozencraft, Jacobs and Kennedy as a design parameter
for communication systems. Jordan used the concept of cutoff rate to design coded waveforms for M-ary
orthogonal signals with coherent and non-coherent detection. Cutoff rates have been widely used as a design
criterion for various different channels, including fading channels encountered in wireless communications.

LEARNING OUTCOMES

!Y The conditional probability P (yi | xj) is called the channel transition probability and is denoted by pji.
The conditional probabilities {P (yi | xj)} that characterize a DMC can be arranged in the matrix form
P = [pji]. P is known as the probability transition matrix for the channel.
!Y
information in any single use of the channel, where the maximization is overall possible input
probabilities. That is,
q -1 r -1 P( yi | x j )
C = max I ( X ; Y ) = max   P( x j ) P( yi | x j ) log
P( x j ) P( x j )
j =0 i =0 P( yi )
Y The basic objective of channel coding is to increase the resistance of the digital communication system
to channel noise. This is done by adding redundancies in the transmitted data stream in a controlled
manner. Channel coding is also referred to as error control coding.
k
!Y The ratio, r = , is called the code rate. Code rate of any coding scheme is always less than unity.
n
Y A rate r is said to be achievable if there exists a coding scheme (n, k) such that the maximal probability
of error tends to 0 as n Æ . The (n, k) code may also be expressed as a (2nr, n) code or a (M, n) code,
where M = 2k = 2nr. All rates below capacity, C, are achievable.
!Y A Gaussian Channel is a time discrete channel with output Yk, at time k, which is the result of the sum
of the input Xk and the Gaussian noise Zk. This noise is drawn from a Gaussian distribution with mean
zero and variance s2.
!Y Let a DMS with an alphabet X have entropy H (X ) and produce symbols every Ts seconds. Let a
H (X ) C
DMC have capacity C and be used once every Tc seconds. Then, if £ , there exists a coding
Ts Tc
scheme for which the source output can be transmitted over the noisy channel and be reconstructed
108 ° Information Theory, Coding and Cryptography

with an arbitrarily low probability of error. This is the Channel Coding Theorem or the Noisy Coding
Theorem.
H (X ) C
Y For > , it is not possible to transmit information over the channel and reconstruct it with an
Ts Tc
arbitrarily small probability of error. The parameter is called the Critical Rate.
Ê P ˆ
!Y The information capacity of a SISO channel can be expressed as C = W log 2 Á 1+ bits per
Ë N 0W ˜¯
second. This is the basic formula for the capacity of the band-limited, AWGN waveform channel with
a band-limited and average power-limited input. This is the crux of the Information Capacity Theorem.
This theorem is also called the Channel Capacity Theorem.
!Y ! " #
Ê Es ˆ
C= max W log 2 det Á I M R + HRss H H ˜ bits per second
Tr(RSS )= M T Ë M T N0 ¯

where W is the bandwidth. The condition Tr(Rss) = MT constrains the total average energy transmitted
over a symbol period. Here the assumption is that the channel is known to the receiver.
Y If the channel is unknown to the transmitter, the capacity of the MIMO channel is given by
r
Ê Es ˆ
C = W Â log 2 Á1 + li ˜ bits per second
i =1 Ë M T N0 ¯

where r is the rank of the channel and li (i = 1, 2, …, r) are the positive eigen values of HHH.
!Y If the channel is known to the transmitter, the different scalar data pipes may be accessed individually
through processing at the transmitter and receiver. The optimal energy for each data pipe is found
iteratively using the water pouring algorithm.
Ê -E ˆ
2
!Y The cutoff rate R0 is given by R0 = log 2
-E
= 1 - log 2 Á1 + e N0 ˜ . The cutoff rate has the units of
ÁË ˜¯
N0
1+ e
bits/dimension. Note that 0 £ R0 £ 1. The average error probability in terms of the cutoff rate can be
- n ( R0 - Rc )
written as Pe < 2 . For Rc < R0 the average probability of error Pe Æ 0 as n .

Mathematics is like checkers in being suitable for the young,

not too difficult, amusing, and without peril to the state.
Plato (ca 429-347 BC)
 Channel Capacity and Coding ° 109

MULTIPLE CHOICE QUESTIONS

2.1 The random and unpredictable electric signals from natural causes, both internal and external to the
system is ___________.
(a) Interference (b) Attenuation (c) Distortion (d) Noise
2.2 In an ideal channel, the crossover probability, p, is
(a) 1 (b) 0 (c) 0.5 (d) None of these
2.3 Relay Channels can use
(a) Amplify-and-Forward (AF) scheme
(b) Decode-and-Forward (DF) scheme
(c) Hybrid of AF and DF
(d) All of these
2.4 The Direct-to-Home (DTH) satellite channel is an example of
(a) Multiple access channel (b) Broadcast channel
(c) MIMO channel (d) None of these
2.5 The units of channel capacity is ‘bits per channel use’ provided the base of the logarithm is
(a) 10 (b) 2 (c) e (d) None of these
2.6 For a BSC with 0.5 < p < 1, the capacity
(a) increases with increasing p (b) decreases with increasing p
(c) increases with decreasing p (d) None of these
2.7 Channel capacity is a measure of
(a) Entropy
(b) Lower-bound on the maximum rate of information transfer
(c) The maximum rate at which information can be reliably transmitted over a channel.
(d) None of these
2.8 The capacity of a binary symmetric channel, given H(p) is the binary entropy function, is
(a) 1 – H(p) (b) H(p) – 1 (c) 1 – H(p)2 (d) H(p)
2.9 For M equally likely messages, the average amount of information H (in bits) is
(a) log10M (b) log2M (c) log10M 2 (d) M log10M
2.10 The capacity of Gaussian channel is
(a) C = 2W log2 (1 + SNR) bits/s (b) C = W log10(1 + SNR) bits/s
(c) C = W log2 (1 + SNR) bits/s (d) C = W(1 + SNR) bits/s

For interactive quiz with answers, scan the QR code given here
Or
Visit [Link]
110 ° Information Theory, Coding and Cryptography

SHORT-ANSWER TYPE QUESTIONS

2.1 Explain the difference between hard decision decoding and soft decision decoding.
2.2 Explain what is meant by a composite Discrete-input, Discrete-output channel.
2.3 What is a Multiple Input Multiple Output (MIMO) channel?
2.4 Why is the capacity of a BSC equal to zero for p = 0.5?
2.5 Suppose I have two parallel independent BSCs with crossover probabilities p and q. If I choose to send
two bits each time over these parallel channels, what will be my capacity?
2.6 List three properties of channel capacity.
2.7 What is the need for channel coding?
2.8 What is meant by achievability of a rate?
$%& ' %
2.10 What are the assumptions made in the Information Capacity Theorem?
2.11 Explain intuitively the Water Filling algorithm.
$%:$ @ Q U X
2.13 Explain why should the capacity of a MIMO channel, with the channel is known to the transmitter, be
necessarily greater than or equal to the capacity when the channel is unknown to the transmitter?
2.14 Explain what is meant by the Capacity Region for a multiple access channel?
2.15 What is the physical interpretation of the Cutoff Rate?

For answers, scan the QR code given here

Or
Visit [Link]

PROBLEMS

M 2.1 Consider the channels A, B and the cascaded channel AB shown in Fig. 2.23.
(i) Find CA the capacity of channel A.
(ii) Find CB the capacity of channel B.
(iii) Next, cascade the two channels and determine the combined capacity CAB.
(iv) Explain the relation between CA, CB and CAB.

Fig. 2.23
 Channel Capacity and Coding ° 111

S 2.2 Consider a miniature keyboard with small keys. Whenever we intend to press any key, the two
adjacent keys also have equal probability of getting pressed. This is like a noisy keyboard. The
equivalent channel is depicted in Fig. 2.24. Find the capacity of this noisy channel with overlapping
outputs.

Fig. 2.24

M 2.3 Determine the channel capacity of the channel shown in Fig. 2.25.

Fig. 2.25 Fig. 2.26

M 2.4 Consider the Z channel shown Fig. 2.26.

(i) Find the input probabilities that result in capacity.
(ii) If N such channels are cascaded, show that the combined channel can be represented by an
equivalent Z channel with the channel transition probability (1 – p)N.
(iii) What is the capacity of the combined channel as N ?
M 2.5 Consider a wireless network with n hops with each node transmitting using an orthogonal channel
(Fig. 2.27). For i = 0, 1, . . . , n Y :Z i transmits coded messages at rate Ri to node i + 1 over
2
an AWGN channel with noise variance s i . Node i decodes messages of node i Y: \
to node i +1. Node i transmits at power Pi and the multihop system is subject to the constraint
n -1
 Pi = P .
i =0
Z0 Z1 Zn–1

Node X0 Y0 Node X1 Y1 Node X2

...
Yn–2 Node Xn–1 Yn–1 Node
0 1 2 n–1 n

Fig. 2.27
112 ° Information Theory, Coding and Cryptography

(i) For a given set of powers P0, . . . , Pn, what is the capacity of C of the multihop communication
system from node 0 to node n? Express your answer in terms of Ci .
(ii) Explain your answer in terms of the end-to-end data rate R.
D 2.6 Consider two binary symmetric channels with crossover probability p £ 1/2 operating in parallel
as shown in the Fig. 2.28. Here, for i Π{1, 2}, the i th channel has input Xi, noise Zi and output
Yi = Xi ŠZi where Xi, Yi, Zi Π{0, 1} and where Šdenotes modulo-two addition. The noise in the
two channels is not independent and the noise pair (Z1, Z2) has the joint distribution PZ , Z ("1, "2)
1 2
given according to the following table, in which a is a parameter satisfying 0 £ a £ p.

X1 Y1 = X1 ≈ Z1
z2 = 0 z2 = 1
p–a Z1
z1 = 0 1 + a – 2p
z1 = 1 p–a a Z2

X2 Y2 = X2 ≈ Z2

Fig. 2.28

(i) Let p # % ^ \ a does the channel behave as two independent binary

symmetric channels in parallel? What is the corresponding channel capacity?
(ii) Determine the channel capacity as a function of a and p.
(iii) Let p # %^ \ a is the channel capacity minimized? For what value(s) of a
is the channel capacity maximized? Comment on this result.
D 2.7 Consider a discrete-time additive white Gaussian channel subject to a periodic Gaussian jammer,
who alternates between periods of transmission (on) and periods of non-transmission (off). The
duty cycle of the jammer is 1/3, i.e., the jammer follows the pattern on, off, off, on, off, off,
… When the jammer is off, additive noise in the channel is zero-mean Gaussian with variance
s 02 > 0. When the jammer is on, the additive noise is zero-mean Gaussian with variance s12 > s 02 .
You have an average power constraint P, i.e., if X1, X2, … represents a sequence of channel inputs,
1 n
then lim  E [ X i2 ] £ P .
n n i =1
(i) Determine the maximum achievable rate when the transmitter is turned off when the jammer
is on, i.e, X1 = X4 = X7 = · · · = 0.
(ii) Determine the optimum transmission strategy and the corresponding maximum achievable
rate of reliable information transmission.
M 2.8 Consider a diversity channel shown in Fig. 2.29 in which the binary input X is transmitted
simultaneously over two independent binary symmetric channels (BSC). The BSC has a crossover
probability of p.

BSC Y1

BSC Y2

Fig. 2.29
 Channel Capacity and Coding ° 113

(i) First consider only a single BSC. Let the input probability PX(0) be equal to q. Determine the
capacity of this single BSC.
(ii) Next consider diversity, with two independent BSCs. Assuming PX ~  ~%€Z I (X; Y1, Y2).
M 2.9 Consider the discrete memoryless channel (Fig. 2.30) with source X = {0, 1} and an independent
on-off jammer, Z, such that P(Z = 0) = P(Z = a) = 0.5. Find the capacity of this channel.

Fig. 2.30 Fig. 2.31

D 2.10 Consider the multiplier channel, shown in Fig. 2.31, with Y = XZ, where the source X = {0, 1}. The
independent binary noise Z = {0, 1} is Bernoulli distributed, i.e., P(Z = 0) = 1 – a, and P(Z = 1) = a.
Find the capacity of this channel.
M 2.11 Next, reconsider the multiplier channel of Prob. 2.10 with the added advantage that the receiver
can observe Z, as depicted in Fig. 2.32. Find the capacity of this channel.
1–p
0 0
p
p
1 1
1–p

1
2 2

Fig. 2.32 Fig. 2.33

D 2.12 Consider a miltary scenario, where we have two independent BSCs at our disposal, with capacities
C1 and C2. In order to have an element of surprise for the enemy, sometimes BSC1 is used, and
sometimes BSC2 is used, but not both simultaneously. Find the capacity of this ‘surprise channel’.
Generalize the result if we had the luxury of K independent channels.
S 2.13 Calculate the capacity of the ternary channel given in Fig. 2.33.
S 2.14 Consider the fading channel, shown in Fig. 2.34, where V is a random variable representing
fading and Z is the additive noise. Both V and Z are independent. Suppose we have the luxury of
channel estimation, i.e., we know V. Show that the capacity with channel state information (CSI)
is necessarily greater than without CSI, i.e., I(X; Y | V) ≥ I(X; Y).

Fig. 2.34

S 2.15 Consider a binary erasure multiple access channel with two senders transmitting either a 1 or a 0.
The receiver computes Y = X1 + X2. Draw the capacity region for the binary erasure multiple access
channel.
114 ° Information Theory, Coding and Cryptography

S 2.16 Consider a binary orthogonal broadcast channel with two independent receivers. The channel
between the transmitter and receivers one and two can be modeled as a BSCs with crossover
probabilities p1 and p2 respectively. Draw the capacity region for this orthogonal broadcast channel.

COMPUTER PROBLEMS

2.1 Write a computer program that takes in the channel transition probability matrix and computes the
capacity of the channel.
$%$  # \ M-PSK, M = 2, 4, 8, 16 and 32, and
the probabilities of error: (a) Pe = 10-6 and (b) Pe = 10-8.
2.3 Write a program that implements the binary repetition code of rate 1/n, where n is an odd integer.
Develop a decoder for the repetition code. Test the performance of this coding scheme over a BSC
with the channel transition probability, p. Generalize the program for a repetition code of rate 1/n over
GF (q). Plot the residual Bit Error Rate (BER) versus p and q (make a 3-D mesh plot).
2.4 Write a program that can generate Capacity versus SNR plots for MT transmit antennas and MR receive
antennas.
(i) Generate plots for the combinations (MT, MR) = (1, 1), (1, 2), (1, 3), (2, 1), (2, 2), (3, 1).
(ii) Compare the capacity of a MISO and a SIMO channel. Comment.
(iii) If we have a total number of 4 antennas, which is the best solution: (1, 3), (2, 2) or (3, 1)? Justify.

PROJECT IDEAS

2.1 Cascade of Channels: Consider a system in which information is transmitted through a series of noisy
channels, with no processing allowed between successive channels. We wish to analyze a cascade of
channels.
(i) Given a set of n channels, how should the channels be ordered so as to give maximum overall
capacity?
(ii) How big a difference does ordering make? That is, what is the maximum difference between best
and worst or best and average performance?
(iii) What channels will have higher capacity when cascaded with themselves n times?
(iv) Express the channel transition probability matrix as P = QLQ-1, where Q is the matrix of eigen-
vectors of P and L is the diagonal matrix of eigenvalues. Now, the n-cascade will have the transition
n
probability matrix Pn with eigenvalues li . Relate the channel capacity of the n-cascade with |li|.
2.2 Image Transmission over Binary Channels: We wish to study how a degraded image looks after
passing through binary channels. Start with a binary image (black and white image) of your choice.
(i) Assume that this image is transmitted through a BSC with p = 10-2. Compare it with the original
image. Repeat with p = 10-1.
(ii) Assume that this image is transmitted through a Z-channel with p = 10-2. Compare it with the
original image. Repeat with p = 10-1.
 Channel Capacity and Coding ° 115

(iii) Use the repetition code (n = 3) for error correction and then show the improvement in the above
two cases.
(iv) Repeat (i) and (ii) for a cascade of two channels.
2.3 Time Varying Binary Channels
(i) Suppose the crossover probability, p, of a BSC is time varying, and is distributed uniformly over
(p1, p2), i.e., p ~ U(p1, p2), p1 < p2. Find the capacity of this time varying BSC.
(ii) Use simulations to plot the capacity versus p1 for p2 = 0.5. Can you intuitively explain this plot?
(iii) Next, consider a Z channel with p ~ U(p1, p2), p1 < p2. Find the capacity of this time varying Z
channel.
2.4 Channel Model for Long Term Evolution (LTE)
When a wireless signal travels from a transmitter to a receiver it follows multiple paths. When these
copies of the same signal arrive at the receiver they are delayed and attenuated. A well-known technique
\ ^!ˆ ^ ! ˆ % \
channel thus performs the convolution operation on the transmitted signal. Suppose, the pedestrian
channel in 4G-LTE standard can be modeled using the following tap-delay line model.

Tap 1 2 3 4 5 6 7

Delay (ns) 0 30 70 90 110 190 410

Relative power (dB) 0.0 –1.0 –2.0 –3.0 –8.0 –17.2 –20.8

Q U ‰ %
(ii) What will the received signal look like if we transmit a pulse with pulsewidth (a) 10 ns, (b) 50 ns.
(iii) Assuming QPSK modulation, plot the BER vs. SNR curve for this channel. What role will the
symbol-rate play?
@ # %

REFERENCES FOR FURTHER READING

1. Cover, Thomas M., and Joy A., Thomas. Elements of information theory, John Wiley & Sons, 2012.
2. Robert, Ash B., Information Theory. Dover Special Priced Titles, 2007.
 PART - II
Error Control Coding (Channel Coding)
Chapter 3 Linear Block Codes for Error Correction
Chapter 4 Cyclic Codes
Chapter 5 Bose–Chaudhuri Hocquenghem (BCH) Codes
Chapter 6 Space–Time Codes
 CHAPTER THREE

Linear Block Codes

for Error Correction
Mathematics is an interesting intellectual sport but it should not be allowed to stand in
the way of obtaining sensible information about physical processes.
Richard W. Hamming

learning objectives

After studying this chapter the students will be able to:

LO 1 Understand the basics of error control coding and the process of encoding linear block
codes.
LO 2 Learn how to decode linear block codes.
LO 3 Identify some of the known good linear block codes.
LO 4 Discuss the wonderful world of space time block codes.

3.1 Introduction to Error Correcting Codes

In this age of information, there is an increasing impor-
This chapter comes with a video
tance not only of speed, but also in accuracy of storage, overview by the author. Scan here to
retrieval, and transmission of data. The channels over know more or
which messages are transmitted are often imperfect. Visit [Link]
Machines and their non-human mistakes can turn

that will remain standing even through an earthquake, their computer counterparts have come up with
sophisticated techniques capable of counteracting the digital manifestations of Murphy’s Law (‘If anything
can go wrong, it will go’). Error correcting codes are a kind of safety net – the mathematical insurance
against the vagaries of an imperfect material world.
Error correcting codes, as the name suggests, are used for correcting errors when messages are trans-
mitted over a noisy channel or stored data is retrieved. The physical medium through which the messages
are transmitted is called a channel (e.g., a telephone line, a satellite link, a wireless channel used for mobile
120 ° Information Theory, Coding and Cryptography

communications etc.). Different kinds of channels are prone to different kinds of noise, which corrupt the data
being transmitted. The noise could be caused by lightning, human errors, equipment malfunctioning, voltage
surges, random motions of electrons in the receiver hardware, and so on. Because these error correcting codes
try to overcome the detrimental effects of noise in the channel, the encoding procedure is also called channel
coding. Error control codes are also used for accurate storage of information, for example storing data and
reading it from a compact disc (CD). In this case, the error could be due to a scratch on the surface of the
CD. The error correcting coding scheme will try to recover the original data from the corrupted one. The
motivation for the study of error control coding comes from the fact that nowadays a large volume of data is
being communicated daily and there exists an ever increasing need to communicate, store and retrieve data
with greater reliability.
The basic idea behind error correcting codes is to add a certain amount of redundancy to the message
prior to its transmission through the noisy channel. This redundancy, which basically consists of some extra
symbols, is added in a known manner. The encoded message, when transmitted through the channel, might
get corrupted due to noise in the channel. At the receiver, the original message can be recovered from the
corrupted one if the number of errors is within the limit for which the coding strategy has been designed. By
adding redundancy intelligently we can succeed in ‘fooling’ the random noise to some extent and diluting
its effect. The block diagram of a digital communication system is illustrated in Fig. 3.1. Note that the most

Example 3.1 Let us see how redundancy can help combat the effects of noise. The normal
language that we use to communicate (say, English) has a lot of redundancy built in it already. Consider
the following sentence (which may be thought of as being corrupted by noise – maybe the moths have
eaten portions of a written text!):
"# $%& '* +/ 0" +"$'/$"# /24 $
As we can see, there are lots of errors in this sentence. However, due to the familiarity with the language
we may probably guess that the original text would have looked something like:
+"# $%& '* +/ 0" +"$&'&/$+"# /24 & $
We have just now used an error correcting strategy that makes use of the in-built redundancy in the
English language to construct the original message from the corrupted one.

Information Channel
Modulator
Source Encoder
C
H
A
N Noise
N
E
L
User of Channel
Demodulator
Information Decoder

Fig. 3.1 Block diagram (and the principle) of a digital communication system. Here the source
coder/decoder block has not been shown.
 Linear Block Codes for Error Correction ° 121

The objectives of a good error control coding scheme are as follows:

(i) Error correcting capability in terms of the number of errors that it can rectify,
5 6 7 8
5 6 7
(iv) Maximum transfer of information per unit time (i.e., fewer overheads in terms of the redundancy).
$ 9 8 + 8
scheme one must introduce more redundancies. However, increased redundancy leads to a slower rate of
transfer of the actual information. Thus the objectives (i) and (iv) are not necessarily compatible. Also, as the
coding strategies become more complicated for correcting larger number of errors, the objectives (ii) and (iii)

In this chapter

! " # " # #
! $ % 1.
& $ ' # ( # # !
% 2.
( ) # !
# * !! + #, # +, !
#! $ ! ! -+( ! % 3.
/ # ! ( #
! ! % 4.

+ = 8
used here as well as in the later chapters. LO 1
Understand the basics of
A word is a sequence of symbols. error control coding and the
process of encoding linear
block codes.
A code is a set of vectors called

The Hamming weight of a codeword (or any vector) is equal to the number of
non-zero elements in the codeword. The Hamming weight of a codeword c is denoted by w(c). The
Hamming distance between two codewords is the number of places by which the codewords differ.
The Hamming distance between two codewords c1 and c2 is denoted by d(c1, c2). It is easy to see that
d(c1, c2) = w (c1 – c2).
122 ° Information Theory, Coding and Cryptography

It is easy to show that the Hamming distance follows the three properties of a distance metric which are
as follows:
(i) d(c1, c2) ≥ 0 with equality if and only if c = c
(ii) d(c1, c2) = d(c2, c1).
(iii) d(c1, c2) ≥ d(c1, c3) + d(c2, c3). This is the triangle inequality.

Example 3.2 Consider a code C = {0100, 1111} which consists of two codewords i.e., 0100
and 1111. The Hamming weight w(0100) = 1 and w(1111) is 4. The Hamming distance between the two
codewords is 3 because they differ at the 1st, 3rd and 4th places. Observe that w(0100 – 1111) = w(1011)
= 3 = d(0100, 1111).

Example 3.3 For the code C = {01234, 43210}, the Hamming weight of each codeword is 4
and the Hamming distance between the codewords is 4 (because only the 3rd component of the two
codewords are identical while they differ at 4 places).

A block code
codewords is called the block length and is typically denoted by n. Thus, a code of block length n consists
of a set of codewords having n components.

A block code of size M q symbols is a set of M q-ary sequences, each of

length n. In the special case that q = 2, the symbols are called bits (short for binary digits) and the code is
said to be a binary code. Usually, M = qk for some integer k, and we call such a code an (n, k) code. Thus, an
(n, k) block code over an alphabet q is a set of qk codewords of block length n.

Example 3.4 The code C = {00000, 10100, 11110, 11001} is a block code of block length equal
to 5. This code can be used to represent two-bit binary numbers which are as follows:

Uncoded bits Codewords

00 00000
01 10100
10 11110
11 11001

Here M = 4, k = 2 and n = 5. Suppose we have to transmit a sequence of 1’s and 0’s using the above
coding scheme. Let us say that the sequence to be encoded is 1 0 0 1 0 1 0 0 1 1.…

time). So we partition the sequence as following:

10 01 01 00 11…
 Linear Block Codes for Error Correction ° 123

Next, replace each block by its corresponding codeword.

11110 10100 10100 00000 11001…
Thus, 5 bits (coded) are sent for every 2 bits of uncoded message. It should be observed that for every
2 bits of information we are sending 3 extra bits (redundancy).

The code rate, r, of an (n, k k/n

the codeword that consists of the information symbols.

Code rate is always less than unity (code rate equal to one implies no coding at all!). The smaller the code
rate, the greater is the redundancy within the code, i.e., number of redundant symbols present per information
symbol is more in a codeword. A code with greater redundancy has the potential to detect and correct greater
number of symbols in error, as we shall soon see. However, a smaller code rate reduces the actual rate of
transmission of information.

INDUSTRIAL RELEVANCE
0 1 23% & % & !! 455 467
8 769 1 : # ;

The minimum distance of a code is the minimum Hamming distance between any two
codewords. If the code C consists of the set of codewords {ci, i = 0, 1 ... M – 1} then the minimum distance
of the code is given by d* = min d(ci, cj), i π j. An (n, k) code with minimum distance d* is sometimes
denoted by (n, k, d*).

The minimum weight of a code is the smallest weight of any non-zero codeword, and
is denoted by w*.

A linear code has the following properties:

(i) The sum of two codewords belonging to the code is also a codeword belonging to the code.
(ii) The all-zero codeword is always a codeword.
(iii) The minimum Hamming distance between two codewords of a linear code is equal to the minimum
weight of any non-zero codeword, i.e., d* = w*.

Note that if the sum of two codewords is another codeword, the difference of two codewords will also yield
a valid codeword. For example, if c1, c2 and c3 are valid codewords such that c + c = c then c – c = c .
Hence, it is obvious that the all-zero codeword must always be a valid codeword for a linear block code (self-
subtraction of a codeword).
124 ° Information Theory, Coding and Cryptography

For a linear code the minimum distance is equal to the minimum weight of the code, i.e.,
d* = w*.
Intuitive Proof: The distance dij between any two codewords ci and cj is simply the weight of the codeword
formed by ci – cj. Since the code is linear, the difference of two codewords results in another valid codeword.
" " " #$ " " "

Example 3.5 The code C = {0000, 1010, 0101, 1111} is a linear block code of block length
n = 4. This is a (4, 2) code. Observe that all the ten possible sums of the codewords
0000 + 0000 = 0000,0000 + 1010 = 1010,
0000 + 0101 = 0101,
0000 + 1111 = 1111,1010 + 1010 = 0000,
1010 + 0101 = 1111,
1010 + 1111 = 0101,0101 + 0101 = 0000,
0101 + 1111 = 1010 and
1111 + 1111 = 0000
are in C and the all-zero codeword is in C. The minimum distance of this code is d* = 2. In order to verify
the minimum distance of this linear code we can determine the distance between all pairs of codewords
Ê 4ˆ
(which is Á ˜ = 6 in number):
Ë 2¯
d(0000, 1010) = 2, d(0000, 0101) = 2, d(0000, 1111) = 4
d(1010, 0101) = 4, d(1010, 1111) = 2, d(0101, 1111) = 2
We observe that the minimum distance of this code is indeed 2.
Note that the code given in Example 3.4 is not linear because 1010 + 1111 = 0101 is not a valid codeword.
Even though the all-zero word is a valid codeword, it does not guarantee linearity. The presence of an

Example 3.6 Consider the simple parity-check codes. Given k information bits, they add one
additional parity bit so that the weight of the codeword is even. Clearly, the rate of the parity-check codes
Ê k ˆ Ê n - 1ˆ
is r = Á
Ë k + 1˜¯ ÁË n ˜¯
= , which tends to unity as n becomes very large (i.e., for large blocklengths). For

k = 4, we have
Information word Codeword
0000 00000
0001 00011
0010 00101
0011 00110
… …
… …

1111 11110
 Linear Block Codes for Error Correction ° 125

The minimum weight w*= 2 (see for example the 2nd codeword). Hence d* = 2. It can detect one
error and cannot correct a single error. Thus, simple parity-check codes have good rates, but poor error
correcting capability.

In order to make the error correcting codes easier to use, understand and analyse, it is helpful to impose
some basic algebraic structure on them. As we shall soon see, it will be useful to have an alphabet in which
it is easy to carry out the basic mathematical operations such as addition, subtraction, multiplication and
division.

% F is a set of elements with two operations + (addition) and . (multiplication)

satisfying the following properties:
(i) F is closed under + and ., i.e., a + b and a.b are in F if a and b are in F.
For all a, b and c in F, the following hold:
(ii) Commutative laws: a + b = b + a, a.b = b.a
(iii) Associative laws: (a + b) + c = a + (b + c), a.(b.c) = (a.b).c
(iv) Distributive law: a.(b + c) = a.b + a.c.
Further, identity elements 0 and 1 must exist in F satisfying:
(v) a+0=a
(vi) a.1 = a
(vii) For any a in F, there exists an additive inverse (–a) such that a + (–a) = 0
(viii) For any a in F other than 0, there exists an multiplicative inverse (a–1) such that a (a–1) = 1.
" %
number of elements (say, q) is called a Galois Field (pronounced Galva Field) and is denoted by GF(q).
& ' ring.

In a layman’s language we can loosely *

(i) Abelian group: A set of mathematical elements that can be ‘added’ and ‘subtracted’.
(ii) Ring: A set of mathematical elements that can be ‘added’, ‘subtracted’ and ‘multiplied’.
(iii) Field: A set of mathematical elements that can be ‘added’, ‘subtracted’, ‘multiplied’ and ‘divided’.
Some interesting facts about Galois Fields are listed as following:
(i) For any Galois Field, the number of elements must be a power of prime.
(ii) If p is a prime number and m is an integer, then there exists a Galois Field with pm elements.
(iii) If q is a prime power and m is an integer, then GF(q GF(qm) and GF(qm) is called
GF(q).
(iv) Every Galois Field has at least one element a, called a primitive element, such that all other elements
(except 0) can be expressed as a power of a.
(v) Two Galois Fields with the same number of elements are isomorphic.

Example 3.7 Consider GF(4) with 4 elements {0, 1, 2, 3}. The addition and multiplication
tables for GF(4) are
126 ° Information Theory, Coding and Cryptography

+ 0 · 0
0 0 1 2 3 0 0 0 0 0
1 0 3 2 0 1 2 3
2 3 0 1 0 2 3 1
3 2 1 0 0 3 1 2

Each element has an additive inverse and a multiplicative inverse other than 0. It should be noted here
that the addition in GF(4) is not modulo 4 addition.

Example 3.8 Consider the following ring with 6 elements.

+ 0
0 0 1 2 3 4 5
1 2 3 4 5 0
2 3 4 5 0 1
3 4 5 0 1 2
4 5 0 1 2 3
5 0 1 2 3 4

· 0
0 0 0 0 0 0 0
0 1 2 3 4 5
0 2 4 0 2 4
0 3 0 3 0 3
0 4 2 0 4 2
0 5 4 3 2 1

See, for example, elements 2, 3 or 4 do not have multiplicative inverse. Hence, it is not a Field.

/ GF(qn), which is a set of n-tuples of elements from GF(q). Linear block codes
can be looked upon as a set of n-tuples (vectors of length n) over GF(q) such that the sum of two codewords
' ' "
code is a subspace of GF(qn).
Let S be a set of vectors of length n " GF(q). The set of all linear
combinations of the vectors of S is called the linear span of S and is denoted by <S>. The linear span is thus
a subspace of GF(qn), generated by S.
Given any subset S of GF(qn), it is possible to obtain a linear code C = <S> generated by S, consisting of
precisely the following codewords:
(i) All-zero word,
(ii) All words in S,
(iii) All linear combinations of two or more words in S.
 Linear Block Codes for Error Correction ° 127

Example 3.9 Let S = {1100, 0100, 0011}. All possible linear combinations of S are
1100 + 0100 = 1000, 1100 + 0011 = 1111, 0100 + 0011 = 0111, 1100 + 0100 + 0011 = 1011.
Therefore, C = < S > = {0000, 1100, 0100, 0011, 1000, 1111, 0111, 1011}. The minimum distance of
this code is w(0100) = 1.

Example 3.10 Let S 7 89; ;9< GF(3). The addition and multiplication tables of
GF(3) = {0, 1, 2} are given by:

+ 0 1 2 0 1 2
0 0 1 2 0 0 0 0
1 1 2 0 1 0 1 2
2 2 0 1 2 0 2 1

All possible linear combinations of 12 and 21 are as follows:

12 + 21 = 00, 12 + 2(21) = 21, 2(12) + 21 = 12.
Therefore, C = < S > = {00, 12, 21, 00, 21, 12} = {00, 12, 21}.

3.3 Matrix Description of Linear Block LO 1

As we have observed earlier, any code C is a subspace of GF(qn). Any set of basis vectors can be used to
> " G, the rows of which form the basis
vectors of the subspace. The rows of G will be linearly independent. Thus, a linear combination of the rows
can be used to generate the codewords of C. The generator matrix will be a k ¥ n matrix with rank k. Since the
choice of the basis vectors is not unique, the generator matrix is not unique for a given linear code.
The generator matrix converts (encodes) a vector of length k to a vector of length n. Let the input vector
(uncoded symbols) be represented by i. The coded symbols will be given by
c = iG (3.1)
where c is called the codeword and i is called the information word.
" ' n¥k
matrix can generate qk codewords. Thus, instead of having a large look-up table of qk codewords, one can
simply have a generator matrix. This provides an enormous saving in storage space for large codes. For
example, for the binary (46, 24) code the total number of codewords are 224 = 1,777,216 and the size of the
lookup table of codewords will be n ¥ 2k = 771,751,936 bits. On the other hand if we use a generator matrix,
the total storage requirement would be n ¥ k = 46 ¥ 24 = 1104 bits.

Example 3.11 Consider a generator matrix

È1 0 1 ˘
G= Í ˙
Î0 1 0 ˚
128 ° Information Theory, Coding and Cryptography

È1 0 1 ˘ È1 0 1 ˘
c = [0 0] Í ˙ = [0 0 0], c = [0 1] Í ˙ = [0 1 0]
Î0 1 0 ˚ Î0 1 0 ˚
È1 0 1 ˘ È1 0 1 ˘
c = [1 0] Í ˙ = [1 0 1], c = [1 1] Í ˙ = [1 1 1]
Î0 1 0 ˚ Î0 1 0 ˚
Therefore, this generator matrix generates the code C = {000, 010, 101, 111}. We observe that this is a
(3, 2) code from the fact that the dimension of the generator matrix is 3 ¥ 2. The code rate r = 2/3.

3.4 Equivalent Codes LO 1

A permutation of a set S = {x1,x2, ...,xn} is a one-to-one mapping from S to itself. A

permutation can be denoted as follows:
x1 x2 ...... xn
Ø Ø ...... Ø (3.2)
f ( x1 ) f ( x2 ) ...... f ( xn )

Two q-ary codes are called equivalent if one can be obtained from the other by one or
both operations listed in the following:
(i) Permutation of the components.
(ii) Permutation of the position of the codeword.

Suppose a code containing M codewords are displayed in the form of an M ¥ n matrix, where the rows
represent the codewords, then operation (i) corresponds to the re-labelling of the symbols appearing in a
given column, and operation (ii) represents the rearrangements of the columns of the matrix.

Example 3.12 Consider the ternary code (a code whose components Π{0, 1, 2}) of blocklength 3

Ï2 0 1
Ô
C = Ì1 2 0
Ô0 1 2
Ó
If we apply the permutation 0 Æ 2 , 2 Æ 1, 1 Æ 0 to column 2 and 1 Æ 2, 0 Æ 1, 2 Æ 0 to column
3 we obtain
Ï2 2 2
Ô
C = Ì1 1 1
Ô0 0 0
Ó
The code C is equivalent to a repetition code of length 3. Note that the original code is not linear, but
is equivalent to a linear code.
 Linear Block Codes for Error Correction ° 129

Two linear q-ary codes are called equivalent if one can be obtained from the other by
one or both operations listed below:
(i) multiplication of the components by a non-zero scalar
(ii) permutation of the position of the codeword.
J K M 99 '

Two k ¥ n matrices generate equivalent linear (n, k) codes over GF(q) if one matrix can be
obtained from the other by a sequence of the following operations:
(i) Permutation of rows
(ii) Multiplication of a row by a non-scalar
(iii) Addition of a scalar multiple of one row to another
(iv) Permutation of columns
(v) Multiplication of any column by a non-zero scalar.
Proof: R
rows of the generator matrix. The operations merely modify the basis. The last two operations (which are
column operations) convert the matrix to one which will produce an equivalent code.

A generator matrix can be reduced to its Systematic Form (also called the standard form of
the generator matrix) of the type G = [ I | P] where I is a k ¥ k identity matrix and P is a k ¥ (n – k) matrix,
called the Parity Matrix.
Proof: The k rows of any generator matrix (of size k ¥ n) are linearly independent. Hence, by performing
elementary row operations and column permutations it is possible to obtain an equivalent generator matrix in
a row echelon form. This matrix will be of the form [I | P].

Example 3.13 Consider the generator matrix of a (4, 3) code over GF(3):

È0 1 2 1 ˘
Í ˙
G = Í1 0 1 0 ˙
ÍÎ1 2 2 1 ˙˚
Let us represent the ith row by ri and the jth column by rj. Upon replacing r3 by r3 – r1 – r2 we get (note
that in GF(3), –1 = 2 and –2 = 1 because 1 + 2 = 0, see table in Example 3.10)

È0 1 2 1 ˘
Í ˙
G = Í1 0 1 0 ˙
ÍÎ0 1 2 0˙˚
Next we replace r1 by r1 – r3 to obtain

È0 0 0 1 ˘
Í ˙
G = Í1 0 1 0 ˙
ÍÎ0 1 2 0˙˚
130 ° Information Theory, Coding and Cryptography

Finally, shifting c4 Æ c1, c1 Æ c2, c2 Æ c3 and c3 Æ c4 we obtain the standard form of the generator
matrix

È1 0 0 0 ˘ È1 0 0 0 ˘
Í ˙ Í ˙
G = Í0 1 0 1 ˙ = Í0 1 0 1 ˙ = [I | P]
ÍÎ0 0 1 2˙˚ ÍÎ0 0 1 2˙˚

3.5 Parity Check Matrix LO 1

V R " -
X "
Codewords are obtained simply by multiplying the input vector (uncoded word) by the generator matrix.
Is it possible to detect a valid codeword using a similar concept? The answer is yes, and such a matrix is
called the parity check matrix, H, for the given code. For a parity check matrix,
cHT = 0 (3.3)
where c is a valid codeword. Since c = iG, therefore, iGHT = 0. For this to hold true for all valid codewords,
we must have
GHT = 0 (3.4)
The size of the parity check matrix is (n – k) ¥ n. A parity check matrix provides a simple method of
detecting whether an error has occurred or not. If the multiplication of the received word (at the receiver) with
the transpose of H yields a non-zero vector, it implies that an error has occurred. This methodology, however,
will fail if the errors in the transmitted codeword exceeds the number of errors for which the coding scheme
> #$ cHT might help us not only to detect but also
to correct the errors under some conditions.
Suppose the generator matrix is represented in its systematic form G = [ I | P]. The matrix P is also called
the ' " *
H = [ – PT | I ] (3.5)
where PT represents the transpose of matrix P. This is because
È- P ˘
GHT = [ I | P] Í ˙ = 0 (3.6)
Î I ˚
Since the choice of a generator matrix is not unique for a code, the parity check matrix will not be unique.
Given a generator matrix G, we can determine the corresponding parity check matrix and vice-versa. Thus
the parity check matrix H can be used to specify the code completely.

Let us make the following interesting observation. Since a linear code, C, is a subspace of
GF(qn), it must have an orthogonal complement C ^. Since this orthogonal complement is a
subspace of GF(qn), it can also be used as a code. We call C ^ the dual of the code C. The
orthogonal complement C ^ has dimension n – k and these n – k basis vectors form the rows of
H. The orthogonality is obvious from (3.4). A code that is equal to its dual code is called a self-dual code. It
can also be observed from (3.6) that for a self-dual code, P must be a square matrix with PPT = I. It is also
obvious that (C ^) ^ = C.
 Linear Block Codes for Error Correction ° 131

From (3.3) we observe that the vector c must have 1’s in such positions that the corresponding rows of HT
add up to the zero vector, 0. Now, we know that the number of 1’s in a codeword pertains to its Hamming
weight. Hence, the minimum distance d* of a linear block code is given by the minimum number of rows of
HT (or, the columns of H) whose sum is equal to the zero vector. This is true for binary linear block codes. For
the non-binary case, the columns of H, weighted by the codewords have to be considered. Thus it is easy to
determine the minimum weight of a code from the H matrix: The minimum weight of a code is w* if every
set of w*– 1 columns of H are linearly independent.

Example 3.14 For a (7, 4) linear block code the generator matrix is given by
È1 0 0 0 1 0 1˘
Í0 1 0 0 1 1 1 ˙˙
G=Í
Í0 0 1 0 0 1 0˙
Í ˙
Î0 0 0 1 0 1 0˚
È1 0 1˘
Í1 È1 1 0 0 ˘
1 1 ˙˙ Í ˙
the matrix P is given by Í and PT is given by Í0 1 1 1 ˙ . Observing the fact that – 1 = 1 for
Í0 1 0˙
Í ˙ ÍÎ1 1 0 0˙˚
Î0 1 0˚
the case of binary, we can write the parity check matrix as
H = [ – PT | I ]
È1 1 0 0 1 0 0 ˘
Í ˙
= Í0 1 1 1 0 1 0 ˙
ÍÎ1 1 0 0 0 0 1 ˙˚

Note that the columns 1, 5 and 7 of the parity check matrix, H, add up to the zero vector. Hence, for this
code, d* = 3.

The code C contains a non-zero codeword of Hamming weight w or less if and only if a
linearly dependent set of w columns of H exist.
Proof: Consider a codeword c ΠC. Let the weight of c be w which implies that there are w non-zero compo-
nents and (n – w) zero components in c. If we throw away the w zero components, then from the relation
CHT = 0 we can conclude that w columns of H are linearly dependent.
Conversely, if H has w linearly dependent columns, then a linear combination of most w columns is zero.
These w #$ w CHT = 0.

An (n, k) systematic code k symbols of the codeword of block

length n are the information symbols themselves (i.e., the uncoded vector). The remainder of the (n – k)
symbols are called the parity symbols.
132 ° Information Theory, Coding and Cryptography

Example 3.15 The following is a (5, 2) systematic code over GF(3) with code rate r = 2/5.

Information symbols Codewords

S.N. (k = 2) (n = 5)
1. 00 00 000
2. 01 01 121
3. 02 02 220
4. 10 10 012
5. 11 11 221
6. 12 12 210
7. 20 20 020
8. 21 21 100
9. 22 22 212

Note that the total number of codewords is 3k = 32 = 9. Each codeword begins with the information
symbols, and has three parity symbols at the end. The parity symbols for the information word 01 are 121
in the above table. A generator matrix in the systematic form (standard form) will generate a systematic
code.

The minimum distance (minimum weight) of an (n, k) linear code is bounded as follows.
d* £ n – k + 1 (3.7)

This is known as the singleton bound. This holds for both binary and non-binary linear block
codes.

Proof: We can reduce all linear block codes to their equivalent systematic forms. A systematic code can have
one information symbol and (n – k) parity symbols. At most all the parity symbols can be non-zero, resulting
in the total weight of the codeword to be (n – k) + 1. Thus the weight of no codeword can exceed n – k + 1.
It should be pointed out that (3.7) gives only a bound on d*. It should not be used to determine the actual
" " " " " "

A maximum distance code d* = n – k + 1.

Having familiarised ourselves with the concept of minimum distance of a linear code, we shall now
explore how this minimum distance is related to the total number of errors the code can detect and possibly
correct. So we move over to the receiver’s end and take a look at the methods of decoding a linear block code.
 Linear Block Codes for Error Correction ° 133

Learning Review
YOU ARE NOW READY TO ATTEMPT THE FOLLOWING QUESTIONS LINKED TO LO1:

È1 1 1 1 0 0 0 ˘
Í ˙
1. Consider the generator matrix G = Í0 1 1 0 1 0 1 ˙ . S

(i) Determine n and k. ÍÎ1 0 1 0 0 1 1 ˙˚

(ii) List out all the codewords.
; \ " " 8^9 9< _ ' S
show that it is isomorphic to GF(2)?
3. Give the addition and multiplication table for GF(5). Verify that all elements have an S
additive inverse and all non-zero elements have a multiplicative inverse.

È1 1 0 1 0 0 0˘
Í0 1 1 0 1 0 0˙˙
4. Using the (7, 4) Hamming code given by G = Í encode the input S
Í0 0 1 1 0 1 0˙
message vectors m1 = 1111 and m2 = 1010. Í0 ˙
Î 0 0 1 1 0 1˚

È1 1 0 1 0 0 0˘ È1 1 0 1 0 0 0˘
Í0 1 1 0 1 0 0˙˙ Í0 1 1 0 1 0 0˙˙
5. Consider G = Í and G = Í . Are the two M
Í0 0 1 1 0 1 0˙ Í1 1 1 0 0 1 0˙
Í ˙ Í ˙
Î0 0 0 1 1 0 1˚ Î1 0 1 0 0 0 1˚
generator matrices G and G equivalent?

If you have successfully solved the above problems,

! " # $% & ' "- .

The answers are available here. Scan to check

Or
Visit [Link]

$ # - "- !
S Simple: Level 1 and Level 2 Category
M Medium: Level 3 and Level 4 Category
D "- 1 Level 5 and Level 6 Category
134 ° Information Theory, Coding and Cryptography

3.6 Decoding of a Linear Block Code

The basic objective of channel coding is to detect and correct errors when
LO 2 messages are transmitted over a noisy channel. The noise in the channel
randomly transforms some of the symbols of the transmitted codeword
Learn how to decode linear into some other symbols. If the noise, for example, changes just one of the
block codes. symbols in the transmitted codeword, the erroneous codeword will be at a
Hamming distance of one from the original codeword. If the noise trans-
forms t symbols (that is, t symbols in the codeword are in error), the Hamming distance of the received word
will be at a Hamming distance of t from the originally transmitted codeword. Given a code, how many errors
" ' ~/ "
An error will be detected as long as, due to noise, one codeword does not transform into another valid
codeword. If the minimum distance between the codewords is d *, the weight of the error pattern must be d *
or more to cause a transformation from one codeword to another. Therefore, an (n, k, d *) code will detect at
least all non-zero error patterns of weight less than or equal to (d * – 1). Moreover, there is at least one error
pattern of weight d * which will not be detected. This corresponds to the two codewords that are the closest.
It may be possible that some error patterns of weight d * or more are detected, but all error patterns of weight
d * will not be detected.

Example 3.16 For the code C = {000, 111} the minimum distance is 3. Therefore, error patterns
of weight 2 or 1 can be detected. This means that any error pattern belonging to the set {011, 101, 110,
001, 010, 100} will be detected by this code.
Next consider the code C = {001, 110, 101} with d * = 1. Nothing can be said regarding how many errors
this code can detect because d * – 1 = 0. However, the error pattern 010 of weight 1 can be detected by this
code. But it cannot detect all error patterns with weight one, e.g., the error vector 100 cannot be detected.

Next, let us look at the problem of error correction. The objective is to make the best possible guess
regarding the originally transmitted codeword after observing the received word. What could be a smart
decoding strategy? Since only one of the valid codewords must have been transmitted it is logical to conclude
that a valid codeword nearest (in terms of Hamming distance) to the received word must have been actually
transmitted. In other words, the codeword which resembles the received word most is assumed to be the
one that was sent. This strategy is called the Nearest Neighbour decoding, as we are picking the codeword
nearest to the received word in terms of the Hamming distance.
It may be possible that more than one codeword is at the same Hamming distance from the received word.
In that case the receiver can do one of the following:
(i) It can pick one of the equally distant neighbours randomly, or
(ii) Request the transmitter to re-transmit.
To ensure that the received word (that has at most t errors) is closest to the original codeword, and farther
from all other codewords, we must put the following condition on the minimum distance of the code
d * ≥ 2t + 1 (3.8)
Graphically, the condition for correcting t errors or less can be visualised from Fig. 3.2. Consider the
space of all q-ary n-tuples. Every q-ary vector of length n can be represented as a point in this space. Every
codeword can thus be depicted as a point in this space, and all words at a Hamming distance of t or less would
 Linear Block Codes for Error Correction ° 135

lie within the sphere centred at the codeword and with a radius of t. If the minimum distance of the code is
d *, and the condition d * ≥ 2t + 1 holds good, then none of these spheres would intersect. Any received vector
R
any other codeword. We will call the spheres associated with each codeword its Decoding Sphere. Hence it
is possible to decode the received vector using the ‘nearest neighbour’ method without ambiguity.

Fig. 3.2 Words within the sphere of radius t and centred at c1 will be decoded as c1. For unambiguous
decoding, d * ≥ 2t + 1.

The condition d * ≥ 2t + 1 takes care of the worst case scenario. It may be possible, however, that the above
condition is not met but still it is feasible to correct t errors as is illustrated in the following example.

Example 3.17 Let consider the code C = {00000, 01010, 10101, 11111}. The minimum distance
d * = 2. Suppose the codeword 11111 was transmitted and the received word is 11110, i.e., t = 1 (one
" J
d(11110, 00000) = 4, d(11110, 01010) = 2
d(11110, 10101) = 3, d(11110, 11111) = 1
Using the nearest neighbour decoding we can conclude that 11111 was transmitted. Even though a single
error correction (t = 1) was done in this case, d * < 2t + 1 = 3. So, for certain scenarios, it is possible
to correct errors even when d* < 2t + 1. However, in many cases a single error correction may not be
possible with this code. For example, if 00000 was sent and 01000 was received,
d(01000, 00000) = 1, d(01000, 01010) = 1
d(01000, 10101) = 4, d(01000, 11111) = 4
& # 

An incomplete decoder decodes only those received codewords that are clearly
closest to one of the codewords. In case of ambiguity, the decoder declares that the received word is
unrecognisable. The receiver is then requested to re-transmit. A complete decoder decodes every received
word, i.e., it tries to map every received word to some codeword, even if it has to make a guess.

Example 3.17 was that of a complete decoder. Such decoders may be used when it is better to have a good
guess rather than to have no guess at all. Many of the real life decoders are incomplete decoders. Usually they
send a message back to the transmitter requesting them to re-transmit.

Example 3.18 The International Standard Book Number (ISBN) is a linear block code of block
length n = 10 over GF(11). The symbols used are 0, 1, 2, …, 9, X. Instead of using the symbol ‘10’ ,
136 ° Information Theory, Coding and Cryptography

‘X’ is used in order to avoid confusion between ‘10’ and the case when ‘1’ is followed by a ‘0’, The ISBN
*
9
 (10 - i)ci = 0 calculated (mod 11) (3.9)
i =0
For example, consider the ISBN 0-07-048297-7. If we perform the check on this ISBN we get:
10 ¥ 0 + 9 ¥ 0 + 8 ¥ 7 + 7 ¥ 0 + 6 ¥ 4 + 5 ¥ 8 + 4 ¥ 2 + 3 ¥ 9 + 2 ¥ 7 + 1 ¥ 7 = 176 = 0 (mod 11)
Thus 0-07-048297-7 is a valid ISBN.
Since ISBN is a linear block code, the all-zero codeword is a valid codeword. Also, 1000000001 is a
valid ISBN. Thus the minimum weight of the code is d * = 2. Theoretically, it cannot correct any errors
and can detect a single error. However, suppose one of the digits of the ISBN gets smudged and we have
0-07-048e97-7. We can recover the erroneous digit, e, by solving (10 ¥ 0 + 9 ¥ 0 + 8 ¥ 7 + 7 ¥ 0 + 6 ¥
4 + 5 ¥ 8 + 4 ¥ e + 3 ¥ 9 + 2 ¥ 7 + 1 ¥ 7) mod 11 = 0. The solution of this equation with one unknown
yields e = 2. Thus, we have been able to correct a single error simply because we know the location of
the error digit.

INDUSTRIAL RELEVANCE
1( & ) ! # #

A receiver declares that an erasure has occurred (i.e., a received symbol has been erased)
when the symbol is received ambiguously, or the presence of an interference is detected during reception.

Example 3.19 Consider a binary pulse amplitude modulation (PAM) scheme where one is
represented by 5 volts and zero is represented by 0 volts. The noise margin is 1 volt, which implies that
at the receiver:
if the received voltage is between 4 volts and 5 volts fi the bit sent is 1,
if the received voltage is between 0 volt and 1 volts fi the bit sent is 0,
if the received voltage is between 1 volt and 4 volts fi an erasure has occurred.
Thus if the receiver received 2.9 volts during a bit interval, it will declare that an erasure has occurred.
A channel can be prone both to errors and erasures. If in such a channel t errors and r erasures occur, the
error correcting scheme should be able to compensate for the erasures and correct the errors as well. If r
erasures occur, the minimum distance of the code will become d * – r in the worst case. This is because, the
erased symbols have to be simply discarded, and if they were contributing to the minimum distance, this
distance will reduce. A simple example will illustrate the point. Consider the repetition code in which
0 Æ 00000
1 Æ 11111
*
Here d = 5. If r 7 ; '
0 Æ ??000
1 Æ ??111
 Linear Block Codes for Error Correction ° 137

Now, the effective minimum distance d1* = d * – r = 3.

Therefore, for a channel with t errors and r erasures, d * – r ≥ 2t + 1. Or,
d * ≥ 2t + r + 1 (3.10)
For a channel which has no errors (t = 0), only r erasures,
d * ≥ r + 1. (3.11)
Decoding in the presence of binary erasures can be done as follows. Suppose we receive a vector v with
r erasures. We follow the given steps:
(i) Replace all the r erasures by ‘0’ and carry out the regular decoding to obtain c0. Find the Hamming
distance d0 = d(c0, v).
(ii) Next, replace all the r erasures by ‘1’ and carry out the regular decoding to obtain c1. Find the Hamming
distance d1 = d(c1, v).
(iii) Choose the decoded word ci corresponding to the minimum distance di.
Next let us give a little more formal treatment to the decoding procedure. Can we construct some mathe-
matical tools to simplify the nearest neighbour decoding? Suppose the codeword c = c1c2 ... cn is transmitted
over a noisy channel. The noise in the channel changes some or all if the symbols of the codeword. Let the
received vector be denoted by v = v1v2 ... vn. K error vector as
e = v – c = v1v2 ... vn – c1c2 ... cn = e1e2 ... en (3.12)
The decoder has to decide from the received vector, v, which codeword was transmitted, or equivalently,
it must determine the error vector, e.

Let C be an (n, k) code over GF(q) and a be any vector of length n. Then the set
a + C = {a + x | x ΠC} (3.13)
is called a coset (or translate) of C. a and b are said to be in the same coset if (a Рb) ΠC.

Suppose C is an (n, k) code over GF(q). Then

(i) every vector b of length n is in some coset of C.
(ii) each coset contains exactly qk vectors
(iii) two cosets are either disjoint or coincide (partial overlap is not possible)
(iv) if a + C is a coset of C and b Πa + C, we have b + C = a + C.
Proofs:
(i) b = b + 0 Πb + C
(ii) Observe that the mapping C Æ a + C ' x Œ a + x, for all x Œ C is a one-to-one mapping. Thus
the cardinality of a + C is the same as that of C, which is equal to qk.
(iii) Suppose the cosets a + C and b + C overlap, i.e., they have at least one vector in common. Let
v Œ (a + C) « (b + C). Thus, for some x, y Œ C,
v=a+x=b+y
Or, b = a + x Рy = a + z, where z ΠC (because, the difference of two codewords is also a codeword).
Thus, b + C = a + C + z or (b + C) Ã (a + C).
Similarly, it can be shown that (a + C) Ã (b + C ). From these two we can conclude that (b + C)
= (a + C).
138 ° Information Theory, Coding and Cryptography

(iv) Since b Πa + C, it implies that b = a + x, for some x ΠC. Next, if b + y Πb + C then,

b + y = (a + x) + y = a + (x + y) Πa + C.
Hence, b + C Õ a + C. On the other hand, if a + z Œ a + C, then,
a + z = (b Рx) + z = b + (z Рx) Πb + C
Hence, a + C Õ b + C, and so b + C = a + C.

The vector having the minimum weight in a coset is called the coset leader. If there
are more than one vector with the minimum weight, one of them is chosen at random and is declared the
coset leader.

Example 3.20 Let C be the binary (3, 2) code with the generator matrix given by
È1 0 1 ˘
G= Í ˙
Î0 1 0 ˚
i.e., C = {000, 010, 101, 111}. The cosets of C are
000 + C = 000, 010, 101, 111,
001 + C = 001, 011, 100, 110.
Note that all the eight vectors have been covered by these two cosets. As we have already seen (in the
above theorem), if a + C is a coset of C and b Πa + C, we have b + C = a + C. Hence all cosets have
been listed. For the sake of illustration we write down the following
010 + C = 010, 000, 111, 101,
011 + C = 011, 001, 110, 101,
100 + C = 100, 110, 001, 011,
101 + C = 101, 111, 000, 101,
110 + C = 110, 100, 011, 001,
111 + C = 111, 101, 010, 000.
It can be seen that all these sets are already covered.
Since two cosets are either disjoint or coincide (from Theorem 3.6), the set of all vectors, GF(q)n can
be written as
GF(q)n = C » (a1 + C ) » (a2 + C ) » » (at + C )
Where t = qn-k –1.

A standard array for an (n, k) code C is a qn-k ¥ qk array of all vectors in GF(q n) in
C (with 0 on the extreme left), and the other rows are the cosets
ai + C, each arranged in corresponding order, with the coset leader on the left.

Steps for constructing a standard array:

& #$
 Linear Block Codes for Error Correction ° 139

(ii) Choose a vector a1 > a1 + C as the second row such

that a1 + x is written under x ΠC.
(iii) Next choose another vector a2 " " "
the coset a2 + C as the third row such that a2 + x is written under x ΠC.
(iv) Continue the process until all the cosets are listed and every vector in GF(qn) appears exactly once.

Example 3.21 Consider the code C = {0000, 1011, 0101, 1110}. The corresponding standard
array is
Codewords Æ 0000 1011 0101 1110
1000 0011 1101 0110
0100 1111 0001 1010
0010 1001 0111 1100
≠
coset leader
Note that each entry is the sum of the codeword and its coset leader.

Let us now look at the concept of decoding (obtaining the information symbols from the received
codewords) using the standard array. Since the standard array comprises all possible words belonging to
GF(qn ' " ' &
received word is a valid codeword, it is concluded that no errors have occurred (this conclusion may be wrong
' ' "
to noise!). In the case when the received word, v, does not belong to the set of valid codewords, we surmise
that an error has occurred. The decoder then declares that the coset leader is the error vector, e, and decodes
the codeword as v – e. This is the codeword at the top of the column containing v. Thus, mechanically, we
decode the codeword as the one on the top of the column containing the received word.

Example 3.22 Suppose the code in the previous example C = {0000, 1011, 0101, 1110} is used
and the received word is v = 1101. Since it is not one of the valid codewords, we deduce that an error has
occurred. Next we try to estimate which one of the four possible codewords was actually transmitted. If
" ' " 99„9 Mrd column. The
topmost entry of this column is 0101. Hence the estimated codeword is 0101. Observe that:
d(1101, 0000) = 3, d(1101, 1011) = 2,
d(1101, 0101) = 1, d(1101, 1110) = 2
and the error vector e = 1000, the coset leader.

Codes with larger blocklengths are desirable (though not always; see the concluding remarks on this
chapter) because the code rates of larger codes perform closer to the Shannon limit. As we go to larger codes
(with larger values of k and n) the method of standard array will become less practical because the size of the
standard array (qn-k ¥ qk) will become unmanageably large. One of the basic objectives of coding theory is
& # "
scheme should be realisable both in terms of memory required as well as the computational load. Is it possible
to reduce the standard array? The answer lies in the concept of syndrome decoding, which we are going to
discuss next.
140 ° Information Theory, Coding and Cryptography

3.7 Syndrome Decoding LO 2

' " ' " " " "

if needed. To do so, we introduce the concept of the syndrome of the error pattern.

Suppose H is a parity check matrix of an (n, k) code. Then for any vector v ΠGF(q)n,
the vector
s = vHT (3.14)
is called the syndrome of v. The syndrome of v is sometimes explicitly written as s(v). It is called a
syndrome because it gives us the symptoms of the error, thereby helping us to diagnose the error. The size
of the s vector is 1 ¥ (n – k).

Two vectors x and y are in the same coset of C if and only if they have the same syndrome.
Proof: The vectors x and y belong to the same coset ¤ x + C = y + C
¤ x–yŒC
¤ (x – y)HT = 0
¤ xHT = yHT
¤ s(x) = s(y)
Thus there is a one-to-one correspondence between cosets and syndromes.
We can reduce the size of the standard array by simply listing the syndromes and the corresponding coset
leaders.

Example 3.23 We now extend the standard array listed in Example 3.21 by adding the syndrome
column.
The code is C = {0000, 1011, 0101, 1110}. The corresponding standard array is
Syndrome
Codewords – – – Æ 0000 1011 0101 1111 00
1000 0011 1101 0110 11
0100 1111 0001 1010 01
0010 1001 0111 1100 10
≠
Coset leader

The steps for syndrome decoding are as follows:

(i) Determine the syndrome (s = vHT ) of the received word, v.
(ii) Locate the syndrome in the ‘syndrome column’.
(iii) Determine the corresponding coset leader. This is the error vector, e.
(iv) Subtract this error vector from the received word to get the codeword y = v – e.
‡ " ' '" ' "
how much advantage does coding actually provide.
 Linear Block Codes for Error Correction ° 141

3.8 Error Probability after Coding

(Probability of Error Correction) LO 2

The probability of error (or, the word error rate) Perr for any decoding scheme is the
probability that the decoder output is a wrong codeword. It is also called the Residual Error Rate.

Suppose there are M codewords (of length n) which are used with equal probability. Let the decoding be
done using a standard array. Let the number of coset leaders with weight i be denoted by ai. We assume that
the channel is a binary symmetric channel (BSC) with symbol error probability p. A decoding error occurs if
the error vector e is not a coset leader. Therefore, the probability of correct decoding will be
n
Pcor = Â a i p i (1 - p ) n -1 (3.15)
i =0
Hence, the probability of error will be
n
Perr = 1 - Â a i p i (1 - p ) n -1 (3.16)
i =0

Let us now consider m errors within the block length of n bits. We have seen that for a linear block code
Í1 * ˙
( )
with minimum distance d*, any number of errors up to Í d - 1 ˙ are always correctable. The probability of
Î 2 ˚
m errors in a block of n bits is
Ê nˆ
Pm, n = Á ˜ p m (1 - p ) n - m (3.17)
Ë m¯
where p is the probability of one bit in error. The codeword will be in error if more than t errors occur. Thus,
the probability of a codeword error is upper-bounded by
n
PM £ Â Pm, n (3.18)
m = t +1

The equality holds for the case of 2 & , which we shall study shortly. We also note that PM
cannot be less than the probability of erroneously decoding a transmitted codeword as another codeword
which is at a distance d *. That is
d* Ê d*ˆ m
PM ≥ Â Á ˜ p (1 - p )
d* -m
(3.19)
m = Íd * / 2˙ +1 Ë m ¯
Î ˚

We can obtain an upper-bound if we consider all the M – 1 other codewords. All these codewords are at a
distance of at least d *. Therefore, the union bound can be expressed as
d* Ê d*ˆ m
PM £ ( M - 1) Â Á ˜ p (1 - p )
d* -m
(3.20)
m = Íd * / 2˙ +1 Ë m ¯
Î ˚
142 ° Information Theory, Coding and Cryptography

Example 3.24 Consider the standard array in Example 3.21. The coset leaders are 0000, 1000,
0100 and 0010. Therefore a0 = 1 (only one coset leader with weight equal to zero), a1 = 3 (the remaining
three are of weight one) and all other ai = 0.
Therefore,
Perr = 1 – [(1 – p)4 + 3p(1 – p)3]
Recall that this code has four codewords, and can be used to send 2 bits at a time. If we did not perform
coding, the probability of error of the 2-bit message being received incorrectly would be
Perr = 1 – Pcor = 1 – (1 – p)2
Note that for p = 0.01, the word error rate (upon coding) is Perr = 0.0103, while for the uncoded case
Perr = 0.0199. So coding has almost halved the word error rate. The comparison of Perr for messages with
and without coding is plotted in Fig. 3.3. It can be seen that coding outperforms the uncoded case only for
p < 0.5 (guess why?). Note that the improvement due to coding comes at the cost of information transfer
rate. In this example, the rate of information transfer has been cut down by half as we are sending two
parity bits for every two information bits.
1

0.8

0.6
Without coding
Perr

0.4
With coding

0.2

0
0 0.2 0.4 0.6 0.8 1
p

Fig. 3.3 Comparison of Perr for coded and uncoded 2 bit messages.

Example 3.25 This example will help us visualise the power of coding. Consider a binary
symmetric channel with the probability of symbol error p = 10-7. Suppose 10 bit long words are being
transmitted without coding. Let the bit rate of the transmitter be 107 b/s, which implies that 106 words/s
are being sent. The probability that a word is received incorrectly is
Ê10ˆ 9 Ê10ˆ 8 2 Ê 10ˆ 7 3 Ê10ˆ 9
ÁË 1 ˜¯ (1 - p ) p + ÁË 2 ˜¯ (1 - p ) p + ÁË 3 ˜¯ (1 - p ) p + ... ª ÁË 1 ˜¯ (1 - p ) p ª10 words/s
-6

Therefore, in one second, 10-6 ¥ 106 = 1 word will be in error! The implication is that every second a
word will be in error and it will not be detected.
Next, let us add a bit of parity to the uncoded words so as to make them 11 bits long. The parity makes
all the codewords of even and thus ensures that a single bit in error will be detected. The only way that
 Linear Block Codes for Error Correction ° 143

"
be computed as 1 – probability that less than two bits are in error. Therefore, the probability of word
error will be
Ê11ˆ
1 - (1 - p )11 - Á ˜ (1 - p )10 p ª 1 – (1 – 11p) – 11(1 – 10p)p = 110p2 = 11 ¥ 10–13
Ë 1¯
The new word rate will be 107/11 words/s because now 11 bits constitute one word and the bit rate is the
same as before. Thus in one second, (107/11) ¥ (11 ¥ 10–13) = 10–6 words will be in error. This implies
that, after coding, one word will be received incorrectly without detection every 106 seconds = 11.5 days!
So just by increasing the word length from 10 bits (coded) to 11 bits (with coding), we have been able to
obtain a dramatic decrease in the word error rate. For the second case, each time a word is detected to be
in error, we can request the transmitter to re-transmit the word. This strategy for retransmission is called
the Automatic Repeat Request (ARQ).

INDUSTRIAL RELEVANCE
&0 % ! ( 0&0< 58== 0>? 1 *
/ ) # */ > + !!

If we have a linear block (n, k, d * ) code we can either use it for error correction or for error detection
(typically combined with ARQ). Suppose we have a single error correcting code (d * = 3) and two errors occur
in the received word, then the syndrome in this case will be non-zero, indicating the presence of 1 or 2 errors.
However, it cannot tell whether it is a single error or a double error. At this stage, two mutually exclusive
actions can be taken. One strategy is to use the code as an error detecting tool and request for retransmission.
In this case we do not attempt to correct the error. The other (mutually exclusive) possibility is to assume that

to correct a single error might introduce a third error. Thus, the word after error correction may actually
contain three errors. If we try to calculate the syndrome for this wrongly ‘corrected’ word, the syndrome may
turn out to be zero! This is because the code can only detect up to two errors.
It is important to ask the following question: How much energy is saved because of channel coding?
The probability of error (Pe) curve versus the
100
Eb/N0 of a typical communication system is plotted in

the same value of Pe can be achieved at a lower 10–2

Eb/N0 than without coding. Thus the coding
Pe
scheme is able to provide a & ' 3 . Coding
gain is measured in dB. It should be observed that 10–4 Coded Uncoded
the coding gain can be negative if the Pe is not low
enough. This implies that channel coding provides
10–6 Coding gain
improvement if the channel is not too bad to begin at 10–5
with. Also, coding gain typically increases as Pe Eb/N0
0 5 10 15 20 25 30
decreases. The limiting value, as Pe Æ 0, is called
the 4 ! 5 & '3 . Fig. 3.4 Illustration of coding gain.
144 ° Information Theory, Coding and Cryptography

It should be pointed out that after deploying an error control code, the energy per bit goes down
by a factor of the code rate. If Eb is the energy per bit for an uncoded system, the effective energy
per bit after error control coding will be Ec = (code rate) ¥ Eb. This is true for all LBCs.

Learning Review
YOU ARE NOW READY TO ATTEMPT THE FOLLOWING QUESTIONS LINKED TO LO2:
È1 1 0 1 0 0 0˘
Í0 1 1 0 1 0 0˙˙
1. Find the parity check matrix for the generator matrix G = Í . S
Í1 1 1 0 0 1 0˙
Give two realisations of the parity check matrix. Í ˙
Î1 0 1 0 0 0 1˚
2. Consider a (15, 11) LBC with S

È1 0 1 0 1 0 1 0 1 0 1 0 1 0 1˘
Í0 1 1 0 0 1 1 0 0 1 1 0 0 1 1˙˙
H= Í
Í0 0 0 1 1 1 1 0 0 0 0 1 1 1 1˙
Í ˙
Î0 0 0 0 0 0 0 1 1 1 1 1 1 1 1˚
(i) How many errors can it detect?
(ii) How many errors can it correct?
(iii) How many errors can it detect in the presence of r erasures?
3. Consider a (7, 1) repetition code. Let the raw error rate be p = 0.01. What is the probability M
of error after coding, i.e., the residual error rate? Can we apply this repetition code for a
video transfer application over wireless requiring a BER = 10-7 ?
È1 1 0 1 0 0 0 ˘
Í0 1 1 0 1 0 0 ˙
4. Consider a (7, 4) code with G = Í ˙ S
Í0 0 1 1 0 1 0 ˙
Í ˙
Î0 0 0 1 1 0 1 ˚
Determine whether the following received words are valid codewords:
(i) 0001101
(ii) 0110100
(iii) 1110000
(iv) 1111111
If you have successfully solved the above problems,
! " # $% & ' "- .

The answers are available here. Scan to check

Or
Visit [Link]
 Linear Block Codes for Error Correction ° 145

3.9 Perfect Codes

For any vector u in GF(qn) and any integer r ≥ 0, the LO 3

sphere of radius r and centre u, denoted by S(u, r), is the set {v ΠGF(qn)
| d(u, v) £ r}. Identify some of the known
good linear block codes.
"
Consider a code C with minimum distance d *(C) ≥ 2t + 1. The spheres
of radius t centred at the codewords {c1, c2, .... cM} of C will then be disjoint. Now consider the decoding
problem. Any received vector can be represented as a point in this space. If this point lies within a sphere,
then by nearest neighbour decoding it will be decoded as the centre of the sphere. If t or fewer errors occur,

correctly decoded. If, however, larger than t errors occur, it will escape the sphere, thus resulting in an
incorrect decoding.

c2
t cM
t
c1 c3
t t

Fig. 3.5 The concept of spheres in GF(qn). The codewords of the code with d*(C) ≥ 2t + 1 are the centres of
these non-overlapping spheres.

A sphere of radius r (0 £ r £ n) contains exactly

Ê nˆ Ê nˆ Ê nˆ 2 Ê nˆ r
ÁË 0˜¯ + ÁË 1˜¯ (q - 1) + ÁË 2˜¯ (q - 1) + ... + ÁË r ˜¯ (q - 1) (3.21)
vectors.
Proof: Consider a vector u in GF(q n) and another vector v which is at a distance m from u. This implies that
the vectors u and v differ at exactly m places. The total number of ways in which m position can be chosen
Ê nˆ
from n positions is Á ˜ . Now, each of these m places can be replaced by (q – 1) possible symbols. This is
Ë m¯
because the total size of the alphabet is q, out of which one is currently being used in that particular position
in u. Hence, the number of vectors at a distance exactly m from u is

Ê nˆ Ê nˆ Ê nˆ 2 Ê nˆ r
ÁË 0˜¯ + ÁË 1˜¯ (q - 1) + ÁË 2˜¯ (q - 1) + ... + ÁË r ˜¯ (q - 1) (3.22)
146 ° Information Theory, Coding and Cryptography

Example 3.26 Consider a binary code (i.e., q = 2) and blocklength n = 4. The number of vectors
at a distance 2 or less from any codeword will be
Ê 4ˆ Ê 4ˆ Ê 4ˆ 2
ÁË 0˜¯ + ÁË 1˜¯ (1) + ÁË 2˜¯ (1) = 1 + 4 + 6 = 11

u = 0000. The vectors of distance 2 or

less are
vectors at a distance 2: 0011, 1001, 1010, 1100, 0110, 0101,
vectors at a distance 1: 0001, 0010, 0100, 1000,
vectors at a distance 0: 0000.
Thus the total number of such vectors is 11.

A q-ary (n, k) code with M codewords and minimum distance (2t

ÏÊ nˆ Ê nˆ Ê nˆ Ê nˆ ¸
M ÌÁ ˜ + Á ˜ (q - 1) + Á ˜ (q - 1) 2 + ... + Á ˜ (q - 1)t ˝ £ q n (3.23)
ÔÓË 0¯ Ë 1¯ Ë 2¯ Ë t¯ Ô˛

Proof: Suppose C is a q-ary (n, k) code. Consider spheres of radius t centred on the M codewords. Each
sphere of radius t has

Ê nˆ Ê nˆ Ê nˆ 2 Ê nˆ t
ÁË 0˜¯ + ÁË 1˜¯ (q - 1) + ÁË 2˜¯ (q - 1) + ... + ÁË t ˜¯ (q - 1)

vectors (Theorem 3.8). Since none of the spheres intersect, the total number of vectors for the M disjoint
ÏÊ nˆ Ê nˆ Ê nˆ 2 Ê nˆ t¸
spheres is M ÌÁ ˜ + Á ˜ (q - 1) + Á ˜ (q - 1) + ... + Á ˜ (q - 1) ˝ which is upper bounded by qn, the total
ÓÔË 0 ¯ Ë 1 ¯ Ë 2¯ Ë t ¯ ˛Ô
number of vectors of length n in GF(q)n.
This bound is called the Hamming Bound or the Sphere Packing Bound and it holds good for nonlinear
codes as well. For binary codes, the Hamming bound will become

ÏÊ nˆ Ê nˆ Ê nˆ Ê nˆ ¸
M ÌÁ ˜ + Á ˜ + Á ˜ + ... + Á ˜ ˝ £ 2n (3.24)
ÔÓË 0¯ Ë 1¯ Ë 2¯ Ë t ¯ Ô˛

It should be noted here that just because a set of integers n, M and t

!" n = 5, M = 5 and t #
!
Observe that for the case when M = qk, the Hamming bound may be alternately written as:

ÏÊ nˆ Ê nˆ Ê nˆ Ê nˆ ¸
log q ÌÁ ˜ + Á ˜ (q - 1) + Á ˜ (q - 1) 2 + ... + Á ˜ (q - 1)t ˝ £ n - k (3.25)
ÓÔË 0¯ Ë 1¯ Ë 2¯ Ë t¯ ˛Ô
 Linear Block Codes for Error Correction ° 147

A perfect code is one which achieves the Hamming bound, i.e.,

ÏÊ nˆ Ê nˆ Ê nˆ Ê nˆ ¸
M ÌÁ ˜ + Á ˜ (q - 1) + Á ˜ (q - 1) 2 + ... + Á ˜ (q - 1)t ˝ = q n (3.26)
ÓÔË 0 ¯ Ë 1 ¯ Ë 2¯ Ë t ¯ ˛Ô
for a perfect code.

For a perfect code there are equal radius disjoint spheres centred at the codewords which
!$
code! A t& !

Example 3.27 Consider the binary repetition code

Ï00...0
C=Ì
Ó11...1
of block length n, where n is odd. In this case M = 2 and t = (n – 1)/2. Upon substituting these values in
the left hand side of the inequality for Hamming bound we get
ÏÊ nˆ Ê nˆ Ê nˆ Ê n ˆ¸ n -1 n
LHS = 2 ÌÁ ˜ + Á ˜ + Á ˜ + ... + Á ˝ = 2.2 = 2 = RHS
ÔÓË 0¯ Ë 1¯ Ë 2¯ Ë (n - 1) 2˜¯ Ô˛
Thus the repetition code is a perfect code. It is actually called a trivial perfect code. In the next
chapter we shall see some examples of non-trivial perfect codes.

One of the ways to search for perfect codes is to obtain the integer solutions for the parameters n, q, M
and t in the equation for Hamming bound. Some of the solutions found by exhaustive computer searches are
listed here.
S.N. n q M T
1 23 2 212 3
2 90 2 278 2
3 11 3 36 2

3.10 Hamming Codes LO 3

There are both binary and non-binary Hamming codes. Here, we shall limit our discussion to binary Hamming
codes. The binary Hamming codes have the property that

(n, k) = (2m – 1, 2m – 1 – m) (3.27)

where m is any positive integer. For example, for m = 3 we have a (7, 4) Hamming code. The parity check
matrix, H, of a Hamming code is a very interesting matrix. Recall that the parity check matrix of an (n, k)
code has n – k rows and n columns. For the binary (n, k) Hamming code, the n = 2m – 1 columns consist of
all possible binary vectors with n – k = m elements, except the all zero vector.
148 ° Information Theory, Coding and Cryptography

Example 3.28 The generator matrix for the binary (7, 4) Hamming code is given by

È1 1 0 1 0 0 0˘
Í0 1 1 0 1 0 0˙˙
G= Í
Í0 0 1 1 0 1 0˙
Í ˙
Î0 0 0 1 1 0 1˚
The corresponding parity check matrix is
È1 0 1 1 1 0 0 ˘
Í ˙
H = Í0 1 0 1 1 1 0 ˙
ÍÎ0 0 1 0 1 1 1 ˙˚

Observe that the columns of the parity check matrix consist of (100), (010), (101), (110), (111), (011)
and (001). These seven are all the possible non-zero binary vectors of length three. It is quite easy to
generate a systematic Hamming code. The parity check matrix H can be arranged in the systematic form
as follows:
È1 1 1 0 1 0 0 ˘
Í ˙
H = Í0 1 1 1 0 1 0˙ = [ – PT | I ]
ÍÎ1 1 0 1 0 0 1 ˙˚

Thus, the generator matrix in the systematic form for the binary Hamming code is

È1 0 0 0 1 0 1˘
Í0 1 0 0 1 1 1˙˙
G = [ I | P] = Í
Í0 0 1 0 1 1 0˙
Í ˙
Î0 0 0 1 0 1 1˚

From the above example we observe that no two columns of H are linearly dependent (otherwise they
would be identical). However, for m > 1, it is possible to identify three columns of H that would add up to
zero. Thus, the minimum distance, d *, of an (n, k) Hamming code is equal to 3, which implies that it is a
single-error correcting code. Hamming codes are perfect codes.
By adding an overall parity bit, an (n, k) 'n+1, k) code with
d * = 4. On the other hand, an (n, k) Hamming code can be shortened to an (n – l, k – l) code by removing
l rows of its generator matrix G or, equivalently, by removing l columns of its parity check matrix, H. We can
; !

Let n = (qk – 1)/(q – 1). The (n, k) Hamming code over GF(q) is a code for which the
parity check matrix has columns that are pairwise linearly independent (over GF(q)), i.e., the columns are
a maximal set of pairwise linearly independent vectors.
 Linear Block Codes for Error Correction ° 149

Example 3.29 Consider the (8, 4) extended Hamming code obtained from the binary (7, 4)
È1 1 1 1 1 1 1 1˘
Í0 0 0 0 1 1 1 1˙˙
Hamming code He = Í
Í0 0 1 1 0 0 1 1˙
Í ˙
Î0 1 0 1 0 1 0 1˚
Note that we have added a row of 1’s as the 1st row and introduced a column [1 0 0 … 0]T as the 1st
column in the original H matrix. The (8, 4) extended Hamming code has d * = 4. We further note that He
is also a generator matrix. Thus, the (8, 4) extended Hamming code is a self-dual code.

In general, any binary (n, k) code with minimum distance d *can be expanded to a (n + 1, k) code with
minimum distance d * + 1 by appending the sum of all the codeword components to make the overall parity
check. If an original codeword has an odd weight, the parity bit must be 1. Thus all codewords with weight
w*= d * become codewords of weight d * + 1. Hence, the minimum weight of the code goes up by one! The
È1 1 � 1 ˘
Í0 ˙
original parity check matrix, H, in its expanded avatar, would become Í ˙.
Í� H ˙
Í ˙
Î0 ˚

INDUSTRIAL RELEVANCE
( * !! " 1@@@ A84 6 (

3.11 Low Density Parity Check (LDPC) Codes LO 3

An (r, s) Gallager code is a linear code with a check matrix H, satisfying the condition
that every column has r ones and every row has s ones.

A Gallager code with small values of r and s is a $ !2 ! & 6 7$ 2&8

code. Thus, an LDPC code has a sparse parity check matrix, with very few ones in each row and column.
Typically, r £ log2n, where n is the block length. The code is succinctly written as a (n, r, s) LDPC code.

Example 3.30 Consider the following parity check matrix.

È1 0 0 0 1 1 1 0 0 0˘
Í0 1 0 0 1 0 0 1 1 0˙˙
Í
H = Í0 0 1 0 0 1 0 1 0 1˙ (3.28)
Í ˙
Í0 0 0 1 0 0 1 0 1 1˙
ÍÎ1 1 1 1 0 0 0 0 0 0 ˙˚
150 ° Information Theory, Coding and Cryptography

The matrix has r = 2 ones in each column and s = 4 ones in each row. We cannot have any more
independent columns in this 5 ¥ 10 matrix because all the possible combinations of choosing two ones
Ê 5ˆ
out of t ! ! Á ˜ have been exhausted. Since the dimension
Ë 2¯
of the parity check matrix is (n – k) ¥ n, we have n = 10 and k = 5. We also observe that the last three
columns, i.e., 8 through 10 add up to the zero vector. Therefore, the minimum distance of this code is 3.
This is a (10, 2, 4) LDPC code.

$ <?@; r and s is called a regular LDPC code. If the number of

ones in the columns and the number of ones in the rows are approximately r and s, it is called an irregular
LDPC code.

LDPC codes can be constructed by using random constructions, algebraic constructions or a combination
of the two. A simple algorithm for random construction of LDPC codes is given below.
9 5 1 Set i = 1.
nr
9 5 1 Generate a random binary vector of length and Hamming weight r. This is the ith column of H.
s
9 5 1 If the weight of each row of H at this point is £ s, and the scalar product of each pair of columns is
£ 1, set i = i + 1. Else, go to step 2.
9 5 1 If i = n, stop. Else, go to step 2.
The random algorithm does not guarantee s ones in each row. Hence, we may generate irregular LDPC
codes by this method. To make the codes regular, suitable constraints have to be put in the algorithm.
An algorithm for algebraic construction of LDPC codes is given below.
r -1
9 5 1 Choose p > .
s -1
9 5 1 Construct a p ¥ p matrix from the identity matrix Ip by cyclically shifting its rows by one position to
the right.
È0 1 0 0 � 0 ˘
Í0 0 1 0 � 0 ˙
Í ˙
J = Í0 0 0 1 � 0 ˙ (3.29)
Í ˙
Í0 0 0 0 � 1 ˙
ÍÎ1 0 0 0 � 0˙˚

9 5 1 The parity check matrix is obtained as

ÈJ 0 J0 J0 � J0 ˘
Í 0 1 2 ( s -1)
˙
ÍJ J J � J ˙
Í ( 2 ( s -1)) ˙
H = ÍJ 0 J2 J3 � J ˙ (3.30)
Í ˙
Í � ˙
ÍJ 0 ( r -1) ( 2 ( r -1))
� J (( r -1)( s -1)) ˙˚
Î J J
 Linear Block Codes for Error Correction ° 151

where J0 = Ip and the lth power of J is obtained from Ip by cyclically shifting its rows by l mod p positions
to the right. This matrix has exactly r ones in each column and s ones in each row. Thus it is a regular LDPC
code, by design.

In a regular LDPC code, each received codeword is checked by precisely r equations and every
check equation adds up precisely s code symbols. All linear codes, including LDPC codes, can
be represented using a 3 5

A Tanner graph is a graphical representation of the linear code based on the set of
check equations. It is a bipartite graph, which means it has two kinds of nodes: symbol nodes and check
nodes. Each symbol nodes are connected only to check nodes and each check node is connected only to
symbol nodes.

Each component of the codeword is represented in the Tanner graph by a symbol node. Hence, there are n
symbol nodes. Each check equation of the code is represented in the Tanner graph by a check node. Hence,
there are n – k check nodes. Each check node is connected by an edge to those symbol nodes that it checks.
An LDPC code can be decoded using a simple iterative ; ? 55 ' -' . This suboptimum
algorithm can be summarised in the following steps:
1. Perform hard decision decoding of the received symbols from the channel to form a binary vector v. If
we have the received word to start with, this step need not be performed. The vector v is n bit long.
2. Compute the syndrome s = vH T. Note that each component of v affects only s components of the
syndrome, s. If only a single bit is in error, only s syndrome components will be equal to 1.
J! ; K K n bits of v.
4. Flip those bits of v K!
5. Go back to step number 3. Keep iterating until either:
! $ K
b. A predetermined number of iterations is reached.
Q V !
However, for large block lengths, this sub-optimum algorithm works remarkably well.

Example 3.31 Let us consider the linear rate 1/3 code given by the following parity check matrix.
È1 1 1 0 0 0˘
Í1 0 0 1 1 0˙˙
H= Í (3.31)
Í0 1 0 1 0 1˙
Í ˙
Î0 0 1 0 1 1˚
Here n = 6, r = 3 and s = 2. This is not exactly an LDPC code since n is not large enough to yield a
sparse H matrix. The Tanner graph for this code is shown in Fig 3.6.
Suppose the received word is v = [0 0 1 0 0 0]. By performing syndrome calculation we obtain s = vHT
= [1 0 0 1], which is non-zero. Hence this is not a valid codeword. From the syndrome vector we note
that the parity checks that have failed are 1 and 4. This implies that there is an error among the symbols
connected to the check nodes 1 and 4 of the Tanner graph. We make the following observations:
152 ° Information Theory, Coding and Cryptography

(a) Bit 4 of the received vector corresponds to no s0

failed checks because it is connected to check
nodes 2 and 3, both of which are zeros in the c0
s1
syndrome vector.
(b) Bits 1 and 2 of the received vector correspond to
c1
one failed check because they are connected to s2
check node 1.
(c) Bits 5 and 6 of the received vector correspond to c2
s3
one failed check because they are connected to
check node 4.
c3
(d) Bit 3 of the received vector corresponds to two s4
failed checks because it is connected to check
nodes 1 and 4, both of which are ones in the
syndrome vector. s5

$ V V Fig. 3.6 The tanner graph corresponding to

3rd bit of the received vector to obtain [0 0 0 0 0 0]. the rate 1/3 linear code discussed in
Example 3.31.
K
meets all the checks. Hence, the corrected vector is the
all-zero vector in this case.

INDUSTRIAL RELEVANCE
1@@@ A84 B= - $ 1 # - " 0 !!
+,

3.12 Optimal Linear Codes LO 3

For an (n, k, d * ) optimal code, no (n – 1, k, d * ), (n + 1, k + 1, d * ) or (n + 1, k, d * + 1)

code exists.

Optimal linear codes give the best distance property under the constraint of the block length. Most of the
optimal codes have been found by long computer searches. It may be possible to have more than one optimal
code for a given set of parameters n, k and d *. For instance there exist two different binary (25, 5, 10) optimal
codes
 Linear Block Codes for Error Correction ° 153

Example 3.32 ; 'YZ Y \ !^

(i) a (23, 12, 8) code does not exist (only the (23, 12, 7) code exists),
(ii) a (25, 13, 8) code does not exist,
(iii) a (25, 12, 9) code does not exist.
Thus the binary (24, 12, 8) code is an optimal code.

3.13 Maximum Distance Separable (MDS) Codes LO 3

d * possible for a given redun-

dancy, r.

An (n, n – r, d * d * £ r + 1.

Proof: From the Singleton bound we have

d* £ n – k + 1
Substitute k = n – r to get
d* £ r + 1

An (n, n – r, r + 1) code is called a Maximum Distance Separable (MDS) code. An

MDS code is a linear code of redundancy r, whose minimum distance is equal to r + 1. An MDS code
meets the singleton bound.

Except the trivial repetition code, there are no binary MDS codes. However, we do have non-binary MDS
codes, for example the Reed Solomon codes, which we will study in Chapter 5. Some interesting properties
of MDS codes are:
q-ary (n, k) linear code is an MDS code if, and only if, the minimum non-zero weight of any
codeword is n – k + 1.
q-ary (n, k) linear code is an MDS code if, and only if, every set of n – k columns of a parity check
matrix is linearly independent.

3.14 Bounds on Minimum Distance LO 3

A simple upper bound on the minimum distance of an (n, k) binary or non-binary linear block code was given
as the Singleton bound in (3.7) as d * £ n – k + 1. The normalised form of this bound can be written as
154 ° Information Theory, Coding and Cryptography

d* 1
£ (1 - r ) + (3.32)
n n

where r is the code rate. For large values of block length, the factor 1/n can be neglected. Another upper
bound can be obtained directly from the Hamming bound expressed in (3.26). By dividing both sides of the
inequality by n and setting q = 2 for the binary case, we obtain

1 Ê t Ê nˆ ˆ
log 2 Á Â Á ˜ ˜ £ 1 - r (3.33)
n Ë i =0 Ë i ¯ ¯

Since the minimum distance, d *, and the error correcting capability, t, are related, the above relation is an
upper bound on d *. For any n, let t0 be the largest integer t for which (3.33) holds. Then, it can be shown that
as n , the ratio t/n cannot exceed t0/n, where t0/n

Êt ˆ
1- r = H Á 0 ˜ (3.34)
Ë n¯
The generalised Hamming bound can be written as

1 Ê t Ê nˆ iˆ
log q Á Â Á ˜ ( q - 1) ˜ £ 1 - r (3.35)
n Ë i =0 Ë i ¯ ¯

Another bound on d* given by Plotkin Bound is

1 Ê qd * - 1 ˆ
Á - 1 - log q d * ˜ £ 1 - r (3.36)
n Ë q -1 ¯

A tighter upper bound on d* is due to Elias, and is given below in its asymptotic form.

d*
£ 2 A(1 - A) (3.37)
n
where
r = 1 + A log 2 A + (1 - A) log 2 (1 - A) , 0 £ A £ ½ (3.38)

Lower bounds also exist for d*. A lower bound on d*, based on a theorem proposed by Gilbert and
Varshamov Bound is given below.

d*
≥a (3.39)
n
where
r = 1 + a log 2 a + (1 - a ) log 2 (1 - a ) = 1 - H (a ), 0 £ a £ ½ (3.40)
 Linear Block Codes for Error Correction ° 155

Learning Review
YOU ARE NOW READY TO ATTEMPT THE FOLLOWING QUESTIONS LINKED TO LO3:
1. Show that the binary (23, 12, 7) code is a perfect code. S

2. Show that the ternary (11, 6, 7) code does not exist. S

3. Consider a system without using error control coding, and allocating Eb energy per bit. S
What is the available energy per bit after using the LBCs given below? Also, what will be
the energy loss per bit (in dB) for each of the following cases?
(i) (7, 4) Hamming code.
(ii) (15, 11) Hamming code.
(iii) (3, 1) Repetition code.

(iv) Plot the performance of the (7, 4) Hamming code over AWGN channel keeping in
consideration the energy loss per bit due to coding.
4. Give the parity check matrix, H, for a (15, 11) Hamming code. From the parity check M
matrix determine how many errors can it correct?
5. The proposed spacecraft to Mars, Mangalyan X, would be sending colour photographs D
over a binary symmetric satellite channel that has a reliability of 0.999 and is subject
to randomly scattered noises. The spacecraft creates photographs using pixels of 128
different colours. Thus each colour is a codeword. The space mission would like the
probability of a pixel in the received image being assigned an incorrect colour to be less
than 0.0001. Determine the parameters (n, k, d *
could be used by the spacecraft.
6. Consider a linear code with distance d * = 3 and length n = 2x where x is a positive integer. M
Use the Hamming bound to show that an upper bound U on the number of codewords in
such a linear code is U = 22– x – 1.

If you have successfully solved the above problems,

! " # $% & ' "- .

The answers are available here. Scan to check

Or
Visit [Link]

3.15 Space Time Block Codes

Wireless communication involves multipath propagation, which leads to
signal fading. A simple way to counter the ill-effects of fading is to use LO 4
antenna diversity. However, it is not practical to deploy multiple antennas at
Discuss the wonderful world
the receiver because of size constraints. On the other hand, the base station
of space time block codes.
can easily accommodate multiple antennas, thus making multiple-antenna
156 ° Information Theory, Coding and Cryptography

transmissions possible. Space-time codes are coding techniques designed for multiple-antenna transmissions.
In particular, coding is performed by adding properly designed redundancy in both spatial and temporal
!" # $ $ % #
system, the symbol error rate (SER) for the system can be approximated by
c
Pe ª (3.41)
(Gc S )Gd
where S is the SNR and c !
channel. Gc (Gc ≥ 1) denotes the & '3 G
and d is the Diversity Order of the system. The diversity
gain/order determines the slope of an error rate curve plotted as a function of SNR, while the coding gain
determines the horizontal shift of the uncoded system error rate curve to the space-time coded error rate
curve obtained for the same diversity order. There are two main kinds of space-time coding schemes, namely,
space-time trellis codes (STTC) and space-time block codes (STBC). While STTC are designed to provide
both maximum coding and diversity gain, STBC provide only full diversity gain and zero or minimal coding
gain. Here we will study the STBC and return to STTC in Chapter 8.
The Alamouti scheme &''; $ !$ # <
transmit diversity for systems with two transmit antennas. Tarokh later generalised Alamouti’s transmit
diversity scheme to an arbitrary number of transmit antennas and presented more complex STBCs based on
orthogonal matrices. STBCs achieve Maximum Likelihood (ML) decoding through linear processing at
the receiver. Figure 3.7 shows the block diagram of the Alamouti space-time encoder.
# $ # >

Information
Source
h1
n1, n2 Channel
2b bits Tx 1 h
h1 Estimator 2
Signal {s1, s2} r1, r2 ML
Space-Time + h1 h2 ~ Detector
Constellation r1
Encoder
with 2b symbols h2
Tx 2 Combiner ~
r2
�s s�
C = � 1* *2� time
�–s2 s1�
space

Fig. 3.7 Alamouti’s two antenna transmit diversity scheme.

To transmit b bits/cycle, we use a modulation scheme that maps one symbol from a constellation with 2b
symbols. The output of the mapper, { x1 , x2 }, is input to a space-time coding block. The encoding takes in
two time slots (the time axis) over two antenna elements (the space axis). In each encoding operation, the
encoder takes two symbols s1 and s2 and transmits according to the following coding strategy, depicted by
the matrix
È x1 x2 ˘
X =Í * ˙ (3.42)
Î- x2 x1* ˚
*
where x1 is the complex conjugate of x1 . During symbol period 1, antenna 1 and antenna 2 transmit x1 and
x2 respectively. For example, if symbols s1 and s2 are selected, we map x1 Æ s1 and x2 Æ s2 and obtain
È s1 s2 ˘
the codeword matrix as C = Í * ˙.
Î- s2 s1* ˚
 Linear Block Codes for Error Correction ° 157

During symbol period 1, antenna 1 and antenna 2 transmit s1 and s2 respectively. During symbol period 2,
* *
antenna 1 and antenna 2 transmit -s2 and s1 respectively.
Note that

X H X = x1 + x2 ( 2 2
)I 2 (3.43)

where I2 is a 2 ¥ @ ! > h1 and h2 are constant

across two consecutive symbol periods.
The received signals over two consecutive symbol periods, denoted by r1 and r2, can be expressed as:
r1 = h1s1 + h2 s2 + n1 (3.44)
r2 = - h1s2* + h2 x1* + n2 (3.45)
where n1 and n2 are AWGN samples for the two symbol periods respectively.
The combiner combines the received signal as follows

r�1 = h1* r1 + h2 r2*

(3.46)
r�2 = h2* r1 - h1r2*
Substituting (3.44) and (3.45) in (3.46) we obtain

(
r�1 = h1 + h2
2 2
)s + h n + h n
1
*
1 1
*
2 2
(3.47)

r� = ( h )s - h n + h n
2 2
2 1 + h2 2
*
1 2
*
2 1

Note that the receiver must have an estimate of h1 and h2 in order to implement the combining scheme.
The beauty of (3.47) is that r�1 depends only on s1, and not on s2. Similarly, r�2 depends only on s2, and not on
s1 " > #

(
x�1 = arg min r�1 - h1 + h2
x ŒS
2 2
)s (3.48)
= arg min r� - ( h )s
2 2
x�2 2 1 + h2
x ŒS

J ! $ "J% 9 @& - ' - or 9 @&

with complex signals. A Real Orthogonal Design of size N is an N ¥ N matrix G = G (x1, x2,… xN), with
entries consisting of real elements drawn from ±x1, ±x2,…, ±xN such that

Ê N ˆ (3.49)
G T G = Á Â xi2 ˜ I N
Ë i =1 ¯

A real orthogonal design exists if and only if N = 2, 4 and 8. A Generalised Real Orthogonal Design is a
T ¥ N matrix with entries consisting of real elements drawn from ±x1, ±x2,…, ±xK such that
158 ° Information Theory, Coding and Cryptography

Ê K ˆ
G T G = Á Â xi2 ˜ I N (3.50)
Ë i =1 ¯
where IN is a N ¥ N identity matrix and K is a constant that represents the number of distinct symbols being
transmitted. A & 5- % ' - ' of size N is an N ¥ N matrix G with entries consisting of
* * *
complex elements drawn from ±x1, ±x2,…, ±xN and their conjugates ± x1 , ± x2 ,� , ± xN , or multiples of these
by j = -1 such that

Ê N 2ˆ
G H G = Á Â xi ˜ I N (3.51)
Ë i =1 ¯
where IN is a N ¥ N identity matrix. A complex orthogonal design exists if and only if N = 2 (the
Alamouti code). A Generalised & 5- % ' - ' is a T ¥ N matrix G with complex
* * *
elements drawn from 0, ± x1 , ± x1 , ± x2 , ± x2 ,� , ± xK , ± xK , or multiples of these by
j = -1 such that

Ê K 2ˆ
G H G = k Á Â xi ˜ I N (3.52)
Ë i =1 ¯
where IN is a N ¥ N identity matrix and k is a constant.
Some examples of real orthogonal design are

È x1 - x2 ˘
X2 = Í ˙ (3.53)
ÍÎ x2 x1 ˙˚
for N = 2 transmit antennas and

È x1 - x2 - x3 - x4 ˘
Íx x1 x4 - x3 ˙˙
X4 = Í
2
(3.54)
Í x3 - x4 x1 x2 ˙
Í ˙
Î x4 x3 - x2 x1 ˚
for N = 4 transmit antennas.
Like in Alamouti scheme, ML decoding is accomplished in orthogonal STBCs through linear processing
at the receiver because of the orthogonality of the transmission matrices. STBCs will be covered in greater
detail in Chapter 6.

INDUSTRIAL RELEVANCE
0 ! " ! !!
1@@@ A84 B= - $ @30 "
 Linear Block Codes for Error Correction ° 159

Learning Review
YOU ARE NOW READY TO ATTEMPT THE FOLLOWING QUESTIONS LINKED TO LO4:

È x1 x2 ˘
1. Show that the generator matrix, G2 = Í ˙ , corresponds to a real orthogonal S
ÍÎ- x2 x1 ˙˚
design. What is interesting about this generator matrix?
È x1 x2 x3 x4 ˘
Í- x x1 - x4 x3 ˙˙
2. Does the generator matrix, G4 = Í 2 , correspond to a real M
Í - x3 x4 x1 - x2 ˙
orthogonal? Why/why not? Í ˙
Î- x4 - x3 x2 x1 ˚
3. What kind of design does the following generator matrix represent: M

È x1 x2 x3 ˘
Í ˙
Í- x2 x1 - x4 ˙
G=Í ˙ ? What is the rate of the code?
Í - x3 x4 x1 ˙
Í- x - x x2 ˙˚
Î 4 3

4. Is the following generator matrix a valid generalised complex orthogonal design: M

È x1 - x2* - x3* 0 ˘
Í ˙
Íx x1* 0 x3* ˙
G=Í 2 ˙ ? What is interesting about this STBC scheme?
Í x3 0 x1* - x2* ˙
Í0 - x3 x2 x1 ˙˚
Î
If you have successfully solved the above problems,
! " # $% & ' "- .

The answers are available here. Scan to check

Or
Visit [Link]

3.16 Concluding Remarks

The classic paper by Claude Elwood Shannon in the Bell System Technical Journal in 1948 gave birth to two
important areas: (i) information theory and (ii) coding theory. At that time Shannon was only 32 years old.
According to Shannon’s channel coding theorem, “the error rate of data transmitted over a band-limited
noisy channel can be reduced to an arbitrarily small amount if the information rate is less than the channel
160 ° Information Theory, Coding and Cryptography

capacity”. Shannon predicted the existence of good channel codes, but did not tell how to construct them.
Since then the search for good codes is on. Shannon’s seminal paper can be accessed from the site#.
!
work on linear codes was extended by Golay (whose codes will be studied in the following chapter). Golay
also introduced the concept of perfect codes. Non-binary Hamming codes were developed by Golay and
" $ % & ' + 7 +
some of the best known codes are the ones discovered by sheer genius, rather than exhaustive searches.
According to Shannon’s theorem, if C(p) represents the capacity (see Chapter 1 for further details) of
a binary symmetric channel with probability of bit error equal to p, then for arbitrarily low probability of
symbol error we must have the code rate R < C(p). Even though the channel capacity provides an upperbound
on the achievable code rate (R = k/n), evaluating a code purely against channel capacity may be misleading.
The block length of the code, which translates directly into the delay, is also an important parameter. Even
if a code performs far from capacity, it may be possible that it is the best code possible of the same rate and
length. It has been observed that as we increase the block length of the codes, the bounds on the code rate
are closer to the channel capacity as opposed to the codes with smaller blocklengths. However, longer block-
lengths imply longer delays for decoding. This is because we cannot begin the decoding of a codeword until
we have received the entire codeword. The maximum delay allowable is limited by practical constraints. For
example, in mobile radio communications, packets of data are restricted to fewer than 200 bits. In these cases
codewords with very large blocklengths cannot be used.
Low Density Parity Check (LDPC) codes were introduced by Gallager in his doctoral thesis in 1963. They
were rediscovered after three decades in 1990s and have become part of several wireless standards today. The
use of multiple antennas is also rapidly becoming popular in wireless communications. Transmit diversity
techniques are being currently integrated with 4G wireless standards. Space time coding was introduced by
! $ ; " 7 $ >> ! @ ' > '
block code to provide full transmit diversity for systems with two transmit antennas.

LEARNING OUTCOMES

!Y A word is a sequence of symbols. A code is a set of vectors called codewords.

!Y The Hamming weight of a codeword (or any vector) is equal to the number of non-zero elements in the
codeword. The Hamming weight of a codeword c is denoted by w(c).
!Y @ 7 $ & & J ! J &
called the block length and is typically denoted by n. A block coding scheme converts a block of k
information symbols to n coded symbols. Such a code is denoted by (n, k).
!Y The code rate of an (n, kK Ok/nK Q & &
that consists of the information symbols.
!Y The minimum distance of a code is the minimum Hamming distance between any two codewords.
An (n, k) code with minimum distance d* is sometimes denoted by (n, k, d*). The minimum weight of
a code is the smallest weight of any non-zero codeword, and is denoted by w*. For a linear code the
minimum distance is equal to the minimum weight of the code, i.e., d* = w*.

# [Link]
 Linear Block Codes for Error Correction ° 161

Y A linear code has the following properties:

(i) The sum of two codewords belonging to the code is also a codeword belonging to the code.
(ii) The all-zero codeword is always a codeword.
(iii) The minimum Hamming distance between two codewords of a linear code is equal to the
minimum weight of any non-zero codeword, i.e., d* = w*.
Y The generator matrix converts (encodes) a vector of length k to a vector of length n. Let the input
vector (uncoded symbols) be represented by i. The coded symbols will be given by c = iG.
Y Two q-ary codes are called equivalent if one can be obtained from the other by one or both operations
listed as following:
(i) Permutation of the components.
(ii) Permutation of the position of the code.
Y An (n, kK k symbols of the codeword of block length n are the
information symbols themselves. A generator matrix of the form G = [ I | P] is called the systematic
form or the standard form of the generator matrix, where I is a k ¥ k identity matrix and P is a k ¥ (n–k)
matrix.
Y The parity check matrix, H & + cHT = 0, where c is a valid codeword. Since
T
c = iG, therefore, iGH = 0. The parity check matrix is not unique for a given code.
Y @ J d* = n – k + 1.
Y For a code to be able to correct up to t errors, we must have d *≥ 2t + 1, where d * is minimum distance
of the code.
Y Let C be an (n, k) code over GF(q) and a be any vector of length n. Then the set a + C = {a + x |
x ΠC} is called a coset (or translate) of C. a and b are said to be in the same coset if (a Рb) ΠC.
Y Suppose H is a parity check matrix of an (n, k) code. Then for any vector v ΠGF(q n), the vector
s = vHT is called the syndrome of v. It is called a syndrome because it gives us the symptoms of the
error, thereby helping us to diagnose the error.
Y A perfect code achieves the Hamming bound, i.e.,
ÏÊ nˆ Ê nˆ Ê nˆ Ê nˆ ¸
M ÌÁ ˜ + Á ˜ (q - 1) + Á ˜ (q - 1) 2 + ... + Á ˜ (q - 1)t ˝ = q n .
ÔÓË 0¯ Ë 1¯ Ë 2¯ Ë t¯ Ô˛
Y The binary Hamming codes have the property that (n, k) = (2m – 1, 2m – 1 – m), where m is any positive
integer. Hamming codes are perfect codes.
Y An (r, s) Gallager code is a linear code with a check matrix H, satisfying the condition that every
column has r ones and every row has s ones. Gallager code with small values of r and s is a Low
Density Parity Check (LDPC) code.
Y A Tanner graph is a graphical representation of the linear code based on the set of check equations. It is
a bipartite graph, which means it has two kinds of nodes: symbol nodes and check nodes. Each symbol
node is connected only to a check node and each check node is connected only to a symbol node.
Y For an (n, k, d *) optimal code, no (n – 1, k, d *), (n + 1, k + 1, d *) or (n + 1, k, d * + 1) code exists.
Y An (n, n – r, r + 1) code is called a maximum distance separable (MDS) code. An MDS code is a linear
code of redundancy r, whose minimum distance is equal to r + 1.
Y Space-time codes are coding techniques designed for multiple-antenna transmissions. In particular,
coding is performed by adding properly designed redundancy in both spatial and temporal domains.
162 ° Information Theory, Coding and Cryptography

È x1
x2 ˘
Y The orthogonal code matrix of the Alamouti scheme is given by X = Í * ˙.
x1* ˚
Î- x2
Y A real orthogonal design of size N is an N ¥ N matrix G = G (x1, x2,… xN), with entries consisting of
Ê N ˆ
real elements drawn from ±x1, ±x2,…, ±xN such that GT G = Á Â xi2 ˜ I N where IN is a N ¥ N identity
matrix. Ë i =1 ¯

Y A generalised real orthogonal design is a T ¥ N matrix with entries consisting of real elements drawn
Êk ˆ
from ±x1, ±x2,…, ±xk such that G T G = Á Â xi2 ˜ I N where IN is a N ¥ N identity matrix and k is a
Ë i =1 ¯
constant that represents the number of distinct symbols being transmitted.
Y A complex orthogonal design of size N is an N ¥ N matrix G with entries consisting of complex
* * *
elements drawn from ±x1, ±x2,…, ±xN and their conjugates ± x1 , ± x2 ,� , ± xN , or multiples of these by
Ê N 2ˆ
j = -1 such that G H G = Á Â xi ˜ I N where IN is a N ¥ N identity matrix.
Ë i =1 ¯
Y A generalised complex orthogonal design is a T ¥ N matrix G with complex elements drawn
from 0, ± x1 , ± x1* , ± x2 , ± x2* ,� , ± xk , ± xk* , or multiples of these by j = -1 such that
Êk 2ˆ
G H G = k Á Â xi ˜ I N where IN is a N ¥ N identity matrix and k is a constant.
Ë i =1 ¯

The greatest unsolved theorem in mathematics is why some

people are better at it than others.
Adrian Mathesis

MULTIPLE CHOICE QUESTIONS

3.1 The code rate of an (n, k) code is given by

(a) k/n (b) n/k
(c) k/H(X) (d) None of the above
3.2 The d *for a linear block code is the
(a) minimum Hamming distance between any two codewords
(b) minimum Hamming weight of a non-zero codeword
(c) both (a) and (b)
(d) None of the above
3.3 An (n, k) systematic code is one in which
(a) the last k symbols are the information symbols
 Linear Block Codes for Error Correction ° 163

(b) k symbols are the information symbols

(c) k symbols are the parity symbols
(d) None of the above
UV @ J
(a) d * = n + k + 1 (b) d * = n + k – 1
*
(c) d = n – k + 1 (d) None of the above
3.5 For a code to be able to correct up to t errors, we must have
(a) d * ≥ t + 1 (b) d * ≥ 2t + 1
(c) d ≥ 2t – 1
* (d) None of the above
3.6 If H is a parity check matrix of an (n, k) code, then the syndrome of the received vector, v, is given by
(a) s = vHT (b) s = vH
(c) s = vTH (d) None of the above
3.7 The code rate of repetition code is given by
(a) 1/n (b) n/k
(c) 1/H(X) (d) None of the above
3.8 Hamming codes
(a) are perfect codes
(b) achieve the Hamming bound
(c) have (n, k) = (2m – 1, 2m – 1 – m), where m is any positive integer
(d) All of the above
3.9 For an (n, k, d * ) optimal code
(a) no (n – 1, k, d * ) code exists
(b) no (n + 1, k + 1, d * ) code exists
(c) no (n + 1, k, d * + 1 code exists
(d) All of the above
3.10 The code matrix of the Alamouti scheme is given by
È x1 x2 ˘ È x1 x2 ˘
(a) X = Í * *˙
(b) X = Í ˙
Î- x2 x1 ˚ Î- x2 x1 ˚
È x1 x2 ˘ È x1 - x2 ˘
(c) X = Í * ˙ (d) X = Í * ˙
Î x2 x1* ˚ Î- x2 x1* ˚

For interactive quiz with answers, scan the QR code given here
Or
Visit [Link]

SHORT-ANSWER TYPE QUESTIONS

3.1 Explain the difference between hard decision decoding and soft decision decoding.
3.2 How can n, k and d * be linked?
3.3 The code {11111, 22222, 33333} is a (5, 1) LBC over GF(3). True or false?
3.4 Find the Hamming weight of the vector [®0 0 ‡ ß © 0 ™ 0 ‡ ©]
3.5 Find the Hamming distance between V1 = [® ß © 0 ™ ©] and V2 = [0 ß ß 0 ™ ©]
3.6 Consider the addition and multiplication tables for GF(2). What circuit elements can be used to
implement them in practice?
164 ° Information Theory, Coding and Cryptography

3.7 Find a set of 16 binary words of length 7 such that each pair of words is at least at a Hamming distance
of 3.
3.8 For the binary repetition code of length 5, determine the G and the H matrix.
3.9 Which of the following is not a valid Hamming code: (255, 247), (2047, 2035) and (4095, 4083)?
3.10 What happens if two errors occur in a received word using (7, 4) Hamming code?
3.11 What does a zero syndrome vector signify?
U Y Z \Z; O^ VK \Z; _
3.13 Are all odd-length binary repetition codes perfect codes?
È1 0 0 1 1 ˘
Í ˙
3.14 For the code given by G = Í0 1 0 1 2˙ over GF(4), what is the minimum distance, d *? Is this
ÍÎ0 0 1 1 3˙˚
code a perfect code?
3.15 Is the binary Hamming (15, 11) code perfect?

For answers, scan the QR code given here

Or
Visit [Link]

PROBLEMS
S 3.1 Show that C = {0000, 1100, 0011, 1111} is a linear code. What is its minimum distance?
M 3.2 Construct, if possible, binary (n, k, d *) codes with the following parameters:
(i) (6, 1, 6)
(ii) (3, 3, 1)
(iii) (4, 3, 2)
M 3.3 Consider the following generator matrix over GF(2).
È1 0 1 0 0 ˘
Í ˙
G = Í1 0 0 1 1 ˙
ÍÎ0 1 0 1 0˙˚
(i)
Generate all possible codewords using this matrix.
(ii)
Find the parity check matrix, H.
(iii)
Find the generator matrix of an equivalent systematic code.
(iv)Construct the standard array for this code.
(v)What is the minimum distance of this code?
(vi)How many errors can this code detect?
(vii)Write down the set of error patterns this code can detect.
(viii)How many errors can this code correct?
(ix)What is the probability of symbol error if we use this encoding scheme? Compare
it with the uncoded probability of error.
(x) Is this a linear code?
 Linear Block Codes for Error Correction ° 165

M 3.4 Let u = [u1 , u2 , · · · , un ], v = [v1 , v2 , · · · , vn ] and w = [w1 , w2 , · · · , wn ] be binary n-tuples.

Show the following:
(i) d(u, v) = w(u + v).
(ii) If u and v have even weight, u + v has even weight.
(iii) d(u, vK ` d(u, w) + d(w, v).
(iv) w(u + vK ~ w(uK  w(v).
S 3.5 For the code C = {00000, 10101, 01010, 11111} construct the generator matrix. Since this G is not
unique, suggest another generator matrix that can also generate this set of codewords.
M 3.6 Let G1, G2 be the generator matrices for two linear codes (n1, k, d1) and (n2, k, d2) respectively.
(i) What are the three parameters (n, k, d) for the code with G = [G1 |G2 ]?
ÈG1 0 ˘
(ii) What are the three parameters (n, k, d) for the code with G = Í ˙?
Î 0 G2 ˚
S 3.7 Show that if there is a binary (n, k, d) code with d * even, then there exists a binary (n, k, d *) code
in which all codewords have even weight.
S 3.8 Show that if C is a binary linear code, then the code obtained by adding an overall parity check bit
to C is also linear.
S 3.9 Find the the parity check matrix, H, for an (n, 8, 4) binary linear code with minimum n.
S 3.10 Let us denote vectors u = u1 u2 u3 …. un and v = v1 v2 v3 …. vn . Denote (u|v) = u1 u2 u3 …. un v1
v2 v3 …. vn as a vector of length 2n. Next, suppose C1 is a binary (n, k1, d1) code and C2 is a binary
(n, k2, d2) code with codewords u ΠC1 and v ΠC2, where d1 and d2 represent the minimum
distances of the codes respectively. Form a new codeword c3 as (u | u + v).
(i) Show that C3 is a (2n, k1 + k2) code.
(ii) What is the minimum distance of C3?
S 3.11 For each of the following sets S, list the code <S>.
(i) S = {0101, 1010, 1100}.
(ii) S = {1000, 0100, 0010, 0001}.
(iii) S = {11000, 01111, 11110, 01010}.
S 3.12 Consider the (23, 12, 7) binary code. Show that if its used over a binary symmetric channel (BSC)
with probability of bit error p = 0.01, the word error will be approximately 0.00008.
M 3.13 Suppose C is a binary code with parity check matrix, H. Show that the extended code C , obtained
from C by adding an overall parity bit, has the parity check matrix.
È | 0˘
Í 0 ˙˙
Í H |
Í ˙
H = Í ˙
Í | 0˙
Í_ _ _ _ _ _˙
Í ˙
ÍÎ 1 1 .... 1 | 1 ˙˚
D 3.14 For a (5, 3) code over GF(4), the generator matrix is given by
È1 0 0 1 1 ˘
Í ˙
G = Í0 1 0 1 2 ˙
ÍÎ0 0 1 1 3 ˙˚
166 ° Information Theory, Coding and Cryptography

(i) Find the parity check matrix.

(ii) How many errors can this code detect?
(iii) How many errors can this code correct?
(iv) How many erasures can this code correct?
(v) Is this a perfect code?
S 3.15 Consider a linear block code over GF(11) with blocklength n = 10, satisfying the following two
9 9
constraints:  ici = 0 calculated (mod 11) and  (10 - i)ci = 0 calculated (mod 11). Find the
i= 0 i =0
minimum distance of this code.

È2 4 2 2 4 4˘
Í0 0 3 0 1 1 ˙˙
D 3.16 Consider the generator matrix over GF(5), G = Í . Find the parity check
Í3 1 4 0 4 0˙
Í ˙
Î2 3 4 1 1 1˚
matrix, H.

M 3.17 Let C be a binary perfect code of length n with minimum distance 7. Show that n = 7 or n = 23.
S 3.18 Consider a linear binary code C and a vector u œ C. Show that C » (u + C) is also a linear code.
S 3.19 Let rH denote the code rate for the binary Hamming code. Determine lim rH .
k
S 3.20 Show that a (15, 8, 5) code does not exist.
S 3.21 Does a ternary double-error correcting perfect code exist? Explain.
M 3.22 Check whether the following STBC code is orthogonal:

È x1 - x2* - x3* 0 ˘
Í ˙
G = Í x2 x1* 0 - x3* ˙
Í ˙
ÍÎ x3 0 x1* x2* ˙˚

M 3.23 Check whether the following STBC code is orthogonal:

È x1 x2 x3 x4 ˘
Í- x x1 - x4 x3 ˙˙
Í 2
Í - x3 x4 x1 - x2 ˙
Í ˙
Í- x4 - x3 x2 x1 ˙
G=Í *
x x2* x3* x4* ˙
Í 1 ˙
Í - x* x1* - x4* x3* ˙
Í 2 ˙
Í - x3* x4* x1* - x2* ˙
Í * ˙
ÍÎ- x4 - x3* x2* x1* ˙˚
 Linear Block Codes for Error Correction ° 167

COMPUTER PROBLEMS
U ' ' & 7 $ + GF(2), given the
generator matrix for the code.
UY ‚ 7 + ' & 7 $ + GF(q).
3.3 Show by simulations that if a binary (15, 11) Hamming code is used on a channel that makes two
errors then the output of the decoder is always wrong. What can you say about (2m – 1, 2m – 1 – m)
Hamming code in general?
3.4 Write a computer program to exhaustively search for all the perfect code parameters n, q, M and t in
the equation for the Hamming bound. Search for 1 ≥ n ≥ 200, 2 ≥ q ≥ 11.
2m - 1
3.5 Write a computer program for a universal binary Hamming encoder with rate m . The program
2 -1- m
should take as input the value of m and a bit-stream to be encoded. It should then generate an encoded
bit-stream. Develop a program for the decoder also. Now, perform the following tasks.
(i) Write an error generator module that takes in a bit stream and outputs another bit-stream after
inverting every bit with probability p, i.e., the probability of a bit error is p.
(ii) For m = 3, pass the Hamming encoded bit-stream through the above-mentioned module and then
decode the received words using the decoder block.
(iii) Plot the residual error probability (the probability of error after decoding) as a function of p.
Note that if you are working in the range of BER = 10-r, you must transmit on the order of
10r+2 bits (why?).
(iv) Repeat your simulations for m = 5, 8 and 15. What happens as m .
3.6 Write a computer program to generate a (20, 3, 4) LDPC code using:
(i) the random construction method,
(ii) the algebraic construction method.
Compare the two.
3.7 Write a computer program to generate a table of Hamming distances between all pairs of codewords
for a given Hamming code. The table would be of size (2k + 1) ¥ (2k + 1). The 1st row and 1st column
should list out all the valid codewords. All the other entries in this table will be Hamming distances.
;' J & O^ VK
3.8 Generate another table of size (2k + 1) ¥ (2k + 1) which shows that sum of each pair of codeword also
+ @ 7 7 ;'
this exercise for the (15, 11) Hamming code.
3.9 You are given 12 gold coins, of which 1 is a counterfeit (weighs either more or less than a genuine gold
coin). You have a beam balance at your disposal. Write a program that uses the parity-check matrix of
a (12, 9) LBC over GF(3) in order to detect the counterfeit coin with three weighings.
3.10 Write a computer program to search for orthogonal STBCs with N > 2.

PROJECT IDEAS
3.1 4 " 9! $@&1 Simulate a wireless communication system consisting of a
Channel Encoder, a Fading Channel and a Decoder. The aim of the project is to evaluate the performance
of an (2m – 1, 2m – 1 – mK " + & ;' ' +
the following for a range of SNRs:
168 ° Information Theory, Coding and Cryptography

(i) probability of word error,

(ii) probability of bit error,
(iii) probability of undetected codeword error.
3.2 & & 1 We would like to study the behavior of concatenated codes, and whether the
sequence of coding makes a difference. Simulate a system consisting of a concatenated encoder, an
AWGN Channel and a concatenated decoder. The concatenated encoder consists of two or more LBCs
with parameters (ni, ki), i = 1, 2, … p. Similarly, the concatenated decoder consists of a series of
decoders. Determine the BER performance of the concatenated codes. Does the sequence of the block
codes make a difference? Explain.
3.3 & " C 5- 1 We would like to implement a general Hamming code using basic circuit
elements. Use shift-registers and xor gates to implement a (2m – 1, 2m – 1 – m) Hamming Code. How
will you implement non-binary Hamming codes using these circuit elements?
3.4 J K ' & 1 Use an over-all parity check bit on the (15, 11) Hamming code to convert
it into a (16, 11) code that can (i) detect all double errors and (ii) correct all single errors. Will adding
2 parity bits buy you anything? Does the (16, 11) code attain the Plotkin bound? Also, derive an
expression for the probability of decoding error when this (16, 11) code is used over a BSC with
crossover probability, p.
3.5 Weight Distribution: For an (n, k) code C + GF(q), let Ai be the number of codewords
with weight i Z '
2 n
A(z) = A0 + A1 z + A2 z + � + An z
which essentially gives the weight distribution. If B(z) is the weight enumerator polynomial for the
dual code C^, then show that
Ê 1- z ˆ
B(z) = q - k (1 + (q - 1) z ) n A Á
Ë 1 - (q - 1) z ˜¯
Write a computer program that gives the weight distribution of any (n, k) code.
3.6 2 - & 1 Introduced by E. Arikan in 2009, polar codes are linear codes which provably achieve
the capacity of symmetric binary DMC (B-DMC). Channel polarisation is an operation which produces
N channels from N independent copies of a B-DMC, such that the new parallel channels are polarised
in the sense that their mutual information is either close to 0 (completely noisy channels) or close to 1
(perfectly noiseless channels). The polar encoder chooses a set of NR rows of the matrix Gn to form a
NR ¥ N matrix, which is then used as the generator matrix in the encoding procedure. Write a program
to implement polar encoder and decoder. Determine its performance over BSC with different values
of p.

REFERENCES FOR FURTHER READING

1. Roth, R. M., Introduction to Coding Theory, Cambridge University Press, 2006.
Y % ; " Z ˆ Error Control Coding (2nd Edition), Prentice-Hall, 2004.
3. Blahut, R. E., Algebraic Codes for Data Transmission, Cambridge University Press, 2002.
 CHAPTER FOUR

Cyclic Codes

We arrive at truth, not by reason only, but also by the heart.

Blaise Pascal (1623–1662)

learning objectives

After studying this chapter the students will be able to:

LO 1 Recall the knowledge of polynomials, which will come handy in studying cyclic codes.
LO 2 Know how to generate cyclic codes and to represent cyclic codes using generator
polynomials and generator matrices.
LO 3 Describe some popular cyclic codes.
LO 4 Explain how to implement cyclic codes using circuits.

4.1 Introduction to Cyclic Codes

In the previous chapter, while dealing with Linear This chapter comes with a video
Block Codes (LBCs), certain linearity constraints overview by the author. Scan here to
were imposed on the structure of the block codes. know more or
These structural properties help us to search for good Visit [Link]
LBCs which are fast and easy to encode and decode.
However, LBCs of very large block lengths can be prohibitive both in terms of memory and computational
costs. Also, most of the good LBCs have been found by the sheer ingenuity of the inventors, and often carry
the name of the inventor of the coding scheme. We have not studied any mechanism to construct or design
the generator matrix or the parity check matrix.
In this chapter we shall explore a subclass of LBCs which has another constraint on the structure of the
codes. The additional constraint may be caused by any cyclic shift of a codeword resulting in another valid
codeword. This condition allows the simple implementation of these cyclic codes by using shift registers.
 170 ° Information Theory, Coding and Cryptography

!
low-complexity encoding and decoding algorithms.

In this chapter

!" 1.
Next, we will learn how to generate cyclic codes using operations on polynomials. Since cyclic
codes form a sub-class of linear block codes, we will study the matrix description of cyclic codes.
# $ $ %
!" 2.
& ' ( ) -
( *()(+ !" 3.
&
/ !" 4.

4.2 Polynomials
" # $ #
LO 1 %
Recall the knowledge of
polynomials, which will
A code C is if
come handy in studying (i) C is a linear code, and,
cyclic codes. (ii) any cyclic shift of a codeword is also a codeword, i.e., if the
codeword a0a1…an –1 is in C then an –1a0…an–2 is also in C.

Example 4.1 The binary code C = {0000, 0101, 1010, 1111} is a cyclic code.
However C2 = {0000, 0110, 1001, 1111} is not a cyclic code, but is equivalent & -
changing the third and the fourth components of C2 yield C .
The (5, 2) linear code C = {00000, 01101, 11010, 10111} is also not cyclic. The second codeword
when cyclic-shifted to the left gives the third codeword. However, another left-shift does not yield a
valid codeword.

A is a mathematical expression
f(x) = f0 + f1x + … + fmxm (4.1)
where the symbol x f0, f1, … fm are the elements of GF(q).
! fm & fm π 0, then m is called the degree of the polynomial,
and is denoted by deg f(x).
 Cyclic Codes ° 171

' f(x) = f0 + f1x + f2x2

xm
+… + fm corresponds to the vector [f0, f1, f2 … fm]. We observe a one-to-one relation between a polynomial
and a vector (word). Thus, a codeword, c, of length n can be expressed as a polynomial c(x* %
c = [c0 c1 c2 … cn–1] ´ c(x) = c0 + c1x + c2x2 +… + cn–1xn–1
Hence, the binary word [1 0 0 1 1] ´ 1 + 0x + 0x2 + 1x3 + 1x4 = 1 + x3 + x4.

A polynomial is called

Example 4.2 f(x) = 3 + 7x + x3 + 5x4 + x6 is a monic polynomial over GF(8). The degree of this
/ & x6 is 1.
3 4 6
g(x) = 1 + x + x + x + 3x is not a monic polynomial over GF9:* x6 is not
unity.

Polynomials play an important role in the study of cyclic codes, the subject of this chapter. Let F[x] be the
set of polynomials in x GF(q). Different polynomials in F[x] can be added, subtracted
and multiplied in the usual manner. F[x] is an example of an algebraic structure called a ring
' 9 ; <> ? <* F[x@
because polynomials of degree greater than zero do not have a multiplicative inverse. It can be seen that if
f(x), g(x) ΠF[x], then deg (f(x)g(x)) = deg f(x) + deg g(x). However, deg (f(x) + g(x)) is not necessarily
max{deg f(x), deg g(x*E ' f(x) and g(x), over GF(2) such that
f(x) = 1 + x2 and g(x) = 1 + x + x2. Then, deg (f(x) + g(x)) = deg (x) = 1. This is because, in GF(2), 1 + 1 = 0,
and x2 + x2 = (1 + 1) x2 = 0.

Example 4.3 Consider the polynomials f(x) = 2 + x + x2 + 2x4 and g(x) = 1 + 2x2 + 2x4 + x5 over
GF(3). Then,
f(x) + g(x) = (2 + 1) + x + (1 + 2)x2 + (2 + 2)x4 + x5 = x + x 4 + x5.
f(x) ◊ g(x) = (2 + x + x2 + 2x4)(1 + 2x2 + 2x4 + x5)
= 2 + x + (1 + 2.2)x2 + 2x3 + (2 + 2 + 2.2)x4 + (2 + 2)x5
+ (1 + 2 + 1)x6 + x7 + 2.2x8 + 2x9
= 2 + x + (1 + 1)x + 2x + (2 + 2 + 1)x4 + (2 + 2)x5
2 3

+ (1 + 2 + 1)x6 + x7 + x8 + 2x9
= 2 + x + 2x2 + 2x3 + 2x4 + x5 + x6 + x7 + x8 + 2x9
G GF(3).

Example 4.4 Consider the polynomial f(x) = 1 + x over GF(2).

(f(x))2 = 1 + (1 + 1)x + x2 = 1 + x2
Again consider f(x) = 1 + x over GF(3).
(f(x))2 = 1 + (1 + 1)x + x2 = 1 + 2x + x2
172 ° Information Theory, Coding and Cryptography

4.3 The Division Algorithm for Polynomials LO 1

The division algorithm states that, for every pair of polynomial a(x) and b(x) π 0 in F[x], there exists a
$ q(x* $ r(x), the remainder, such that a(x) = q(x) b(x) + r(x), where
deg r(x) < deg b(x). This is also called the .
The remainder is sometimes also called the , and is denoted by Rb(x)[a(x)] = r(x).
! %
(i) Rf(x)[a(x) + b(x)] = Rf(x)[a(x)] + Rf(x)[b(x)], and (4.2)
(ii) Rf(x)[a(x) ◊ b(x)] = Rf(x){Rf(x)[a(x)] ◊ Rf(x)[b(x)]} (4.3)
Where a(x), b(x) and f(x) are polynomials over GF(q).

Example 4.5 Let the polynomials, a(x) = x3 + x + 1 and b(x) = x2 + x K M GF(2).

We can carry out the long division of a(x) by b(x* %

x+1 q(x)
b(x) x2 + x + 1 x3 + x + 1 a(x)
x3 + x2 + x
x2 + 1
x2 + x + 1
x r(x)

Thus, a(x) = (x + 1) b(x) + x. Hence, we may write a(x) = q(x) b(x) + r(x), where q(x) = x + 1 and
r(x) = x. Note that deg r(x) < deg b(x).

Let f(x* ' F[x]. Two polynomials, g(x) and h(x) in F[x] are said to
be f(x), depicted by
g(x) ∫ h(x) (mod f(x))
if g(x) – h(x) is divisible by f(x).

Example 4.6 Let the polynomials g(x) = x9 + x2 + 1, h(x) = x5 + x2 + 1 and f(x) = x4 K M

over GF9>* ; g(x) – h(x) = x5f(x), we can write g(x) ∫ h(x) (mod f(x)).

Next, let us denote F[x]/f(x) as the set of polynomials in F[x] of degree less than deg f(x), with addition
and multiplication carried out modulo f(x* %
(i) If a(x) and b(x) belong to F[x]/f(x), then the sum a(x) + b(x) in F[x]/f(x) is the same as in F[x]. This is
because deg a(x) < deg f(x), deg b(x) < deg f(x) and therefore deg (a(x) + b(x)) < deg f(x).
(ii) The product a(x)b(x* $ f(x) to which a(x)b(x)
(multiplication being carried out in F[x]) is congruent modulo f(x).
F[x]/f(x) is called the ring of polynomials (over F[x]) modulo f(x* Q <R
'
 Cyclic Codes ° 173

Example 4.7 Consider the product (x + 1)2 in F[x]/(x2 + x K M* GF(2).

(x + 1)2 = x2 +x+x+1= x2 + 1 = x (mod x2 + x + 1)
The product (x + 1)2 in F[x]/(x2 K M* GF(2) can be expressed as
(x + 1)2 = x2 + x + x + 1 = x2 + 1 = 0 (mod x2 + x + 1)
The product (x + 1)2 in F[x]/(x2 + x K M* GF(3) can be expressed as
(x + 1)2 = x2 +x+x+1= x2 + 2x + 1 = x (mod x2 + x + 1).
If f(x) has degree n, then the ring F[x]/f(x) over GF(q) consists of polynomials of degree £ n – 1. The
size of ring will be qn because each of the n q elements
in GF(q).

Example 4.8 Consider the ring F[x]/(x2 + x K M* GF(2). This ring will have
polynomials with highest degree = 1. This ring contains qn = 22 = 4 elements (each element is a
polynomial). The elements of the ring will be 0, 1, x and x + 1. The addition and multiplication tables
can be written as follows.
Table 4.1 F [x]/(x2 + x + 1)

+ 0 1 x x+1 0 1 x x+1
0 0 1 x x+1 0 0 0 0 0
1 1 0 x+1 x 1 0 1 x x+1
x x x+1 0 1 x 0 x x+1 1
x+1 x+1 x 1 0 x+1 0 x+1 1 x
Next, consider F[x]/(x2 K M* GF(2). The elements of the ring will be 0, 1, x and x + 1. The
addition and multiplication tables can be written as follows.
Table 4.2 Addition and multiplication tables for the ring F[x]/(x2 + 1)
+ 0 1 x x+1 0 1 x x+1
0 0 1 x x+1 0 0 0 0 0
1 1 0 x+1 x 1 0 1 x x+1
x x x+1 0 1 x 0 x 1 x+1
x+1 x+1 x 1 0 x+1 0 x+1 x+1 0

It is interesting to note that F[x]/(x2 + x K M*

non-zero elements also exist. On the other hand, F[x]/(x2 + 1) is not
inverse of element x + 1 does not exist.
It is worthwhile exploring the properties of f(x) which makes F[x]/f(x*
the polynomial f(x) must be irreducible (non-factorisable).

A polynomial f(x) in F[x] is said to be if f(x) = a(x) b(x), where a(x), b(x) are
elements of F(x) and deg a(x) and deg b(x) are both smaller than deg f(x). If f(x) is not reducible, it is called
. A monic irreducible polynomial of degree at least one is called a .
174 ° Information Theory, Coding and Cryptography

It is helpful to compare a reducible polynomial with a positive integer that can be factorised into a product
of prime numbers. Any monic polynomial in f(x* $
monic polynomials (prime polynomials). One way to verify a prime polynomial is by trial and error, testing
! $ U '

(i) A polynomial f(x) has a linear factor (x – a) if and only if f(a) = 0 where a
(ii) A polynomial f(x) in F[x] of degree 2 or 3 over GF(q) is irreducible if, and only if f(a) π 0 for all a in
GF(q).
9 * V xn – 1 = (x – 1)(xn–1 + xn–2 + … + x +1). The second factor may be further reducible.
Proof:
(i) If f(x) = (x – a)g(x), then obviously f(a) = 0. On the other hand if f(a) = 0, by division algorithm,
f(x) = q(x)(x – a) + r(x), where deg r(x) < deg (x – a) = 1. This implies that r(x) is a constant. But,
since f(a) = 0, r(x) must be zero, and therefore, f(x) = q(x)(x – a).
(ii) A polynomial of degree 2 or 3 over GF(q) will be reducible if and only if it has at least one linear factor.
The result (ii) then directly follows from (i). This result does not necessarily hold for polynomials of
degree more than 3. This is because it might be possible to factorise a polynomial of degree 4 or higher
into a product of polynomials, none of which are linear, i.e., of the type (x – a).
* 9 * 9x – 1) is a factor of (xn –1). By carrying out long division of (xn –1) by (x –1) we obtain
(xn – 1 + xn – 2 + … + x + 1).

Example 4.9 Consider f(x) = x3 –1 over GF(2). Using (iii) of Theorem 4.1 we can write
x3 –1 = (x – 1)(x2 + x K M* ! G Y
term, p(x) = (x2 + x + 1).
p(0) = 0 + 0 + 1 = 1, over GF(2)
p(1) = 1 + 1 + 1 = 1, over GF(2)
Therefore, p(x) cannot be factorised further (from (ii)).
Thus, over GF(2), x3 – 1 = (x – 1)(x2 + x + 1).
Next, consider f(x) = x3 – 1 over GF(3).
x3 –1 = (x – 1)(x2 + x + 1). Again, let p(x) = (x2 + x + 1)
p(0) = 0 + 0 + 1 = 1, over GF(3)
p(1) = 1 + 1 + 1 = 0, over GF(3)
p(2) = 2.2 + 2 + 1 = 1 + 2 + 1 = 1, over GF(3)
; p(1) = 0, from (i) we have (x – 1) as a factor of p(x).
Thus, over GF(3), x3 –1 = (x – 1)(x – 1) (x – 1).

The ring F[x]/f(x* f(x) is a prime polynomial in F[x].

Proof: ! Z\
inverse. Let s(x) be a non-zero element of the ring.
We have, deg s(x) < deg f(x), because s(x) is contained in the ring F[x]/f(x).
& 9 ?Q* f(x) and s(x) can be expressed
as
?Q9f(x), s(x)) = a(x) f(x) + b(x) s(x)
where a(x) and b(x) are polynomials over GF(q* ; f(x) is irreducible in F[x], we have
?Q9f(x), s(x)) = 1 = a(x) f(x) + b(x) s(x).
 Cyclic Codes ° 175

Now, 1 = Rf(x)[1] = Rf(x)[a(x) f(x) + b(x) s(x)]

= Rf(x)[a(x) f(x)] + Rf(x)[b(x) s(x)] (property (i) of residues)
= 0 + Rf(x)[b(x) s(x)]
= Rf(x){Rf(x)[b(x)]◊Rf(x)[s(x)]} (property (ii) of residues)
= Rf(x){Rf(x)[b(x)] ◊s(x)}
Hence, Rf(x)[b(x)] is the multiplicative inverse of s(x).
Next, let us prove the only if part of the theorem. Let us suppose f(x) has a degree of at least 2, and is not a
prime polynomial (a polynomial of degree one is always irreducible). Therefore, we can write f(x) = r(x) s(x)
for some polynomials r(x) and s(x) with a degree of at least one. If the ring F[x]/f(x*
a multiplicative index of r(x), r–1(x* '
multiplicative inverses. Hence,
s(x) = Rf(x){s(x)} = Rf(x){r(x)r –1(x)s(x)} = Rf(x){r –1(x)r(x)s(x)} = Rf(x){r –1(x)f(x)} = 0. However, we had
assumed s(x) π ^ !
G &
f(x* ! % _! F[x]/f(x) is a
f(x) is irreducible in F[x@Y
; `&
of degree n over GF(q* qn ;
! GF(q) and consist of all polynomials of
degree less than n. It can be seen that there will be qn

Example 4.10 Consider the polynomial p(x) = x3 + x + 1 over GF9>* ; p(0) π 0 and p(1) π 0,
the polynomial is irreducible in GF9>* ; p(x) is a prime polynomial. Here we have
n = 3, so we can use p(x* >3 = 8 elements, i.e., GF9:* !
will be 0, 1, x, x + 1, x2, x2 + 1, x2 + x, x2 + x + 1, which are all possible polynomials of degree less than
nq< & 9 ' *

Having developed the necessary mathematical tools, we now resume our study of cyclic codes. We now
' f(x) = xn – 1 for the remainder of the chapter. We also denote F[x]/f(x) by Rn. Before we proceed, we make
%
(i) xn = 1 (mod xn – 1). Hence, any polynomial, modulo xn – 1, can be reduced simply by
replacing xn by 1, xn+1 by x and so on.
(ii) A codeword can uniquely $
of elements. We can use a polynomial to represent the locations and the values of all the
' c0c1c2…cn-1 can be represented by the
polynomial c(x) = c0 + c1x + c2x2 + … cn–1xn–1. Thus,
c = c0c1c2…cn–1 ´ c0 + c1x + c2x2 + … cn–1xn–1 = c(x)
As another example, the codeword over GF(8), c = 207735 can be represented by the polynomial
c(x) = 2 + 7x2 + 7x3 + 3x4 + 5x5.
(iii) Multiplying any polynomial by x corresponds to a single cyclic right-shift of the codeword elements.
More explicitly, in Rn, by multiplying c(x) by x we get
x◊c(x) = c0x + c1 x2 + c2x3 + … cn–1xn = cn-1 + c0x + c1x2 + c2x3 + … cn–2xn–1
176 ° Information Theory, Coding and Cryptography

A code C in Rn is a cyclic code if and only if C %

(i) a(x), b(x) ΠC fi a(x) + b(x) ΠC (4.4)
(ii) a(x) ΠC and r(x) ΠRn fi a(x)r(x) ΠC (4.5)
Proof:
9* ; C is a cyclic code in Rn ; ~"?
(ii) Let r(x) = r0 + r1x + r2x2 + … rnxn. Multiplication by x corresponds to a cyclic right-shift. But,
! x.a(x) ΠC,
x.(xa(x)) ΠC, and so on. Hence,
r(x)a(x) = r0a(x) + r1xa(x) + r2x2a(x) + … rnxna(x) is also in C since each summand is also in C.
Next, we need to prove the only if ; 9* 9 * ! # r(x) to be a scalar.
Then (i) implies that C is linear. Take r(x) = x in (ii), which shows that any cyclic shift also leads to a
codeword. Hence (i) and (ii) imply that C is a cyclic code.
In the next section, we shall use the mathematical tools developed so far to construct cyclic codes.

Learning Review
YOU ARE NOW READY TO ATTEMPT THE FOLLOWING QUESTIONS LINKED TO LO 1:
GF(3). S

! S
follow the addition and multiplication rules for the ring.
(i) (2x + 1) + (4x + 5) + (3x + 3)
(ii) (2x + 1)2
(iii) (4x + 3)(x + 5)
" # x15 – 1 over GF(2). S

4. Divide x7 + 1 by x3 + x + 1 over GF(2). S

5. Divide x12 + x7 + x4 + x3 + 1 by x3 + x2 + 1 over GF$ % & M

!" " #
$% & '

The answers are available here. Scan to check

Or
Visit [Link]

$ " "
S ( ) Level 1 and Level 2 Category
M * ) Level 3 and Level 4 Category
D " ) Level 5 and Level 6 Category
 Cyclic Codes ° 177

4.4 A Method for Generating Cyclic Codes

' * ,
LO 2
(i) Take a polynomial f(x) in Rn.
(ii) Obtain a set of polynomials by multiplying f(x) by all possible Know how to generate cyclic
polynomials in Rn. codes and to represent
(iii) The set of polynomials obtained above corresponds to the set of cyclic codes using generator
polynomials and generator
codewords belonging to a cyclic code. The block length of the code matrices.
is n.

Example 4.11 Consider the polynomial f(x) = 1 + x2 in R3 ! GF(2). In general, a

polynomial in R3 (= F[x]/(x3
– 1)) can be represented as r(x) = r0 + r1x + r2 x2 !
8 9 $ ! GF(2)). Thus, there can be a total of 2 ¥ 2 ¥ 2 = 8 polynomials
in R3 ! GF(2), which are 0, 1, x, x2, 1 + x, 1 + x2, x + x2, 1 + x + x2. To generate the cyclic
code, we multiply f(x) with these 8 possible elements of R3 and then reduce the results modulo (x3 ; %,
(1 + x2)◊0 = 0, (1 + x2) ◊1 = (1 + x2), (1 + x2)◊ x = 1 + x, (1 + x2) ◊ x2 = x + x2
(1 + x2) ◊ (1 + x) = x + x2, (1 + x2) ◊ (1 + x2) = 1 + x, (1 + x2)◊ (x + x2) = (1 + x2),
(1 + x2)◊ (1 + x + x2) = 0
' , <9 > x, 1 + x2, x + x2} which correspond to {000, 011,
101, 110}.

# * which can be
used to construct the cyclic code.

Let C be a (n, k) non-zero cyclic code in Rn. Then

$% & g(x) of the smallest degree in C,
(ii) the cyclic code C consists of all multiples of the generator polynomial g(x) by polynomials of degree
k – 1 or less.
(iii) g(x) is a factor of xn – 1.

Proof:
$% * g(x) and h(x) are monic polynomials in C of smallest degree. Then g(x) – h(x) is also in
C, and has a smaller degree. If g(x) π h(x), then a suitable scalar multiplier of g(x) – h(x) is monic, and
is in C, and is of smaller degree than g(x). This gives a contradiction.
(ii) Let a(x) ΠC. Then, by division algorithm, a(x) = q(x)g(x) + r(x), where deg r(x) < deg g(x). But
r(x) = a(x) Рq(x)g(x) ΠC * * &
However, the degree of g(x) must be the minimum among all codewords. This can only be possible
if r(x) = 0 and a(x) = q(x)g(x). Thus, a codeword is obtained by multiplying the generator polynomial
178 ° Information Theory, Coding and Cryptography

g(x) with the polynomial q(x% # ! GF(q), here are qk distinct codewords possible.
These codewords correspond to multiplying g(x) with the qk distinct polynomials, q(x), where
deg q(x) £ (k – 1).
(iii) By division algorithm, xn – 1 = q(x)g(x) + r(x), where deg r(x) < deg g(x) or r(x) = {(xn – 1) – q(x)g(x)}
modulo (xn Р1) = Рq(x)g(x). But Рq(x)g(x) ΠC because we are multiplying the generator polynomial
by another polynomial, – q(x). Thus, we have a codeword r(x) whose degree is less than that of g(x).
This violates the minimality of the degree of g(x), unless r(x) = 0, which implies xn – 1 = q(x)g(x), i.e.,
g(x) is a factor of xn – 1.
The last part of the theorem gives us the recipe to obtain the generator polynomial for a cyclic code. All we
have to do is to factorise xn ; * E ! *
codewords of blocklength n simply by factorising xn – 1.

+ ) A cyclic code C may contain polynomials other than the generator polynomial which
also generates C. But the polynomial with the minimum degree is called the generator polynomial.
+ ) The degree of g(x) is n – k (this will be shown later).

Example 4.12 T ! * * 8 " ! x3 – 1. Note that

for GF(2), 1 = – 1, since 1 + 1 = 0. Hence, x3 – 1 = x3 + 1 = (x + 1)(x2 + x + 1). Thus, we can construct
Table 4.3.

Table 4.3 All possible binary cyclic codes of block length 3

Generator Polynomial Code (polynomial) Code (binary)

1 {R3} {000, 001, 010, 011,

100, 101, 110, 111}

(x + 1) {0, x + 1, x2 + x, x2 + 1} {000, 011, 110, 101}

(x2 + x + 1) {0, x2 + x + 1} {000, 111}

(x3 + 1) = 0 {0} {000}

A simple encoding rule to generate the codewords from the generator polynomial is
c(x) = i(x)g(x) (4.6)
where i(x) is the information polynomial, c(x) is the codeword polynomial and g(x) is the generator polynomial.
We have seen already that there is a one-to-one correspondence between a word (vector) and a polynomial.
The error vector can also be represented as the error polynomial, e(x). Thus, the received word at the receiver,
after passing through a noisy channel can be expressed as
v(x) = c(x) + e(x) (4.7)
 Cyclic Codes ° 179

E ! ( , , s(x) as the remainder of v(x) under division by g(x), i.e.,

s(x) = Rg(x)[v(x)] = Rg(x)[c(x) + e(x)] = Rg(x)[c(x)] + Rg(x)[e(x)] = Rg(x)[e(x)] (4.8)
because Rg(x)[c(x)] = 0 from (4.6).

Example 4.13 Consider the generator polynomial g(x) = x3 + x + 1 for a binary cyclic code
(i.e., over GF(2)) of block length n = 7. The codewords and the corresponding codeword polynomials
are listed in Table 4.4. It is clear from the table that the minimum weight of this code is 3. This is, in
fact, the (7, 4) Hamming code. Observe how every codeword is a cyclic shift of some other codeword.

Table 4.4 Binary cyclic code constructed using g(x) = x3 + x + 1

c c(x)
0000000 0
0001011 x3 + x + 1
0010110 x4 + x2 + x
0011101 x4 + x3 + x2 + 1
0100111 x5 + x2 + x + 1
0101100 x5 + x3 + x2
0110001 x5 + x4 + 1
0111010 x5 + x4 +x3 + 1
1000101 x6 + x2 + 1
1001110 x6 + x3 + x2 + x
1010011 x6 + x4 + x + 1
1011000 x6 + x4 + x3
1100010 x6 + x5 + x
1101001 x6 + x5 + x3 + 1
1110100 x6 +x5 + x4 + x2
1111111 x6 +x5 + x4 + x3 + x2 + x + 1

Example 4.14 Consider the generator polynomial g(x) = x2 + 1 for ternary cyclic codes (i.e., over
GF(3)) of block length n = 4. Here we are dealing with cyclic codes, therefore, the highest power of
g(x) is n – k n = 4, k * $F % '
will be a total of qk = 32 = 9 code words. The information polynomials and the corresponding code word
polynomials are listed in Table 4.5.
180 ° Information Theory, Coding and Cryptography

Table 4.5 Ternary cyclic code constructed using g(x) = x2 + 1

I i(x) c(x) = i(x)g(x) C

00 0 0 0000
01 1 x2 +1 0101
02 2 2x2 +2 0202
10 x x3 +x 1010
11 x+1 x3 + x2 + x + 1 1111
12 x+2 x3 + 2x2 + x + 2 1212
20 2x 2x3 + 2x 2020
21 2x + 1 2x3 + x2 + 2x + 1 2121
22 2x + 2 2x3 + 2x2 + 2x + 2 2222

It can be seen that the cyclic shift of any code word results in another valid code word. By observing
! $ JK
with the minimum Hamming weight = 2). Therefore, this code is capable of detecting one error and
correcting zero errors.
Observing the fact that the codeword polynomial is divisible by the generator polynomial, we can
detect * *
with cyclic codes that are a subset of LBCs, we can use the all-zero codeword to illustrate this point
without loss of generality.
Assume that g(x) = x2 + 1 and the transmitted codeword is the all-zero codeword. Therefore, the
received word is the error polynomial, i.e.,
v(x) = c(x) + e(x) = e(x) (4.9)

M P * g(x) fails to divide the received word v(x) = e(x).

Now, g(x% e(x) has odd number of terms, i.e., if the number of errors are
* * Q# e(x) = x3 + x + 1 by g(x), we will
always get a remainder. In the case of the (4, 2) cyclic code with g(x) = x2 + 1, the d * = 2, suggesting
that it can detect d * ; S U * * !
number of errors £ n. In this case it can detect 1 error or 3 errors, but not 2 errors.

4.5 Matrix Description of Cyclic Codes LO 2

C is a cyclic code with generator polynomial g(x) = g0 + g1x + …+ grxr of degree r.
Then the generator matrix of C is given by
 Cyclic Codes ° 181

È g0 g1 ... gr 0 0 0 ... 0˘
Í0 g0 g1 ... gr 0 0 ... 0 ˙˙
Í
G = Í0 0 g0 g1 ... gr 0 ... 0 ˙ k = (n – r) rows (4.10)
Í ˙
Í : : : ˙
ÍÎ 0 0 0 0 0 g0 g1 ... g r ˙˚

n columns

Proof: The (n - r) rows of the matrix are obviously linearly independent because of the echelon form of
the matrix. These (n – r) rows represent the codewords g(x), xg(x), x2g(x), …, xn–r–1g(x). Thus, the matrix can
generate these codewords. Now, to prove that the matrix can generate all the possible codewords, we must
show that every possible codeword can be represented as linear combinations of the codewords g(x), xg(x),
x2g(x), …, xn–r–1g(x).
We know that if c(x) is a codeword, it can be represented as
c(x)= q(x)◊g(x)
for some polynomial q(x c(x) < n (because the length of the codeword is n), it follows
that the degree of q(x) < n – r. Hence,

q(x)◊g(x) = (q0 + q1x + … + qn–r–1xn–r–1)g(x) = q0g(x)+ q1xg(x) + …+ qn–r–1xn–r–1g(x)

Thus, any codeword can be represented as a linear combination of g(x), xg(x), x2g(x), …, xn–r–1g(x). This
proves that the matrix G is indeed the generator matrix.

We also know that the dimensions of the generator matrix is k ¥ n. Therefore, r = n – k, i.e., the
degree of g(x) is n – k. Note that the generator matrix has an interesting structure because it is
constant along the diagonal. Such a matrix is called a Toeplitz matrix.

Example 4.15 T GF(3)) of

blocklength n x4 – 1.

x4 – 1 = (x – 1)(x3 + x2 + x + 1) = (x – 1) (x + 1)(x2 + 1)
Thus all the divisors of x4 x – 1), (x + 1), (x2 + 1), (x – 1)◊(x + 1) = (x2 – 1), (x – 1) ◊(x2 + 1)
= (x – x + x +1), (x + 1) ◊(x + 1) = (x + x2 + x +1) and (x4 – 1). We know that all these factors of x4 – 1
3 2 2 3

are capable of generating a cyclic code.

The corresponding generator matrices are listed in Table 4.6. Note that – 1 = 2 for GF(3).
It can be seen from Table 4.6 that none of the (4, 2) ternary cyclic codes are single error correcting
codes (since their minimum distance is less than 3). An interesting observation is that we do not have any
182 ° Information Theory, Coding and Cryptography

! " #$ % "
with n = (qr – 1)/(q –1) and k = (qr –1)/(q –1) – r, where r is an integer ≥ 2. Therefore, a (4, 2) ternary
Hamming code exists, but it is not a cyclic code.

Table 4.6 Cyclic codes of blocklength n = 4 over GF(3).

g(x) (n, k) d* G

1 (4, 4) 1 [I4]

(x – 1) (4, 3) 2 È-1 1 0 0˘
Í 0 -1 1 0˙
Í ˙
ÍÎ 0 0 -1 1 ˙˚

(x + 1) (4, 3) 2 È1 1 0 0 ˘
Í0 1 1 0 ˙
Í ˙
ÍÎ0 0 1 1˙˚

(x2 + 1) (4, 2) 2 È1 0 1 0 ˘
Í0 1 0 1 ˙
Î ˚

È-1 0 1 0˘
(x2 – 1) (4, 2) 2 Í 0 -1 0 1 ˙
Î ˚

(x3 – x2 + x – 1) (4, 1) 4 [–1 1 –1 1]

(x3 + x2 + x + 1) (4, 1) 4 [1 1 1 1]

(x4 – 1) (4, 1) 0 [0 0 0 0]

& ' * '* * -* * 5

polynomial, g(x). We already know that g(x) is a factor of xn – 1. Hence we can write
xn – 1 = h(x)g(x) (4.11)
where h(x * 7 % 5 % * % % 85
g(x) is monic, h(x % % 5 85
5
g(x) is n – k, the degree of h(x) must be k.
 Cyclic Codes ° 183

5** C is a cyclic code in Rn with the generator polynomial g(x $ F[x]/f(x)

by Rn, where f(x) = xn – 1. In Rn, h(x)g(x) = xn – 1 = 0. Then, any codeword belonging to C can be written as
c(x) = a(x)g(x), where the polynomial a(x) ΠRn. Therefore in Rn,
c(x)h(x) = a(x)g(x)h(x) = c(x)◊0 = 0 (4.12)
Thus, h(x) behaves like a Parity Check Polynomial. Any valid codeword when multiplied by
the parity check polynomial yields the zero polynomial. This concept is parallel to that of the
parity check matrix introduced in the previous chapter. Since we are still in the domain of

The parity check polynomial can be used to generate another cyclic code since it is a factor of
xn – 1 and is called the dual code of C.

Example 4.16 Consider the generator polynomial g(x) = x3 + 1 for the (9, 6) binary cyclic codes.
The parity check polynomial can be found by simply dividing x9 – 1 by g(x).
Thus h(x) = (x9 – 1) / (x3 + 1) = x6 + x3 + 1.
Therefore, the dual code of g(x) = x3 + 1 is h(x) = x6 + x3 + 1.

5** C is a cyclic code with the parity check polynomial h(x) = h0 + h1x + …+ hkxk, then the parity
check matrix of C is given by

Èhk hk -1 ... h0 0 0 0 ... 0˘

Í0 hk hk -1 ... h0 0 0 ... 0 ˙˙
Í
H=Í0 0 hk hk -1 ... h0 0 ... 0 ˙ (n – k) rows (4.13)
Í ˙
Í: : : ˙
ÍÎ 0 0 0 0 0 hk hk -1 ... h0 ˙˚

n columns
We observe that H & * ; ' $ cHT = 0. Therefore, iGHT = 0 for any information
T T
vector, i. Hence, GH = 0. We further have s = vH where s is the syndrome vector and v is the received word.

Example 4.17 7or binary codes of blocklength, n = 7, we have

x7 – 1 = (x – 1)(x3 + x + 1)(x3 + x2 + 1)
Consider g(x) = (x3 + x + 1). g(x) is a factor of x7 – 1, there is a cyclic code that can be generated
by it. The generator matrix corresponding to g(x) is

È1 1 0 1 0 0 0˘
Í0 1 1 0 1 0 0˙˙
G= Í
Í0 0 1 1 0 1 0˙
Í ˙
Î0 0 0 1 1 0 1˚

The parity check polynomial h(x) is (x – 1)(x3 + x2 + 1) = (x4 + x2 + x +1). And the corresponding
parity check matrix is
184 ° Information Theory, Coding and Cryptography

È1 0 1 1 1 0 0 ˘
Í ˙
H = Í0 1 0 1 1 1 0 ˙
ÍÎ0 0 1 0 1 1 1 ˙˚

The minimum distance of this code is 3, and this happens to be the (7, 4) Hamming code. Thus, the
binary (7, 4) Hamming code is also a cyclic code with a generator polynomial g(x) = (x3 + x + 1).
5** * i(x) = (x2 + x + 1). This corresponds to the
information word [1 1 1 0]. The corresponding codeword polynomial will be simply
c(x) = i(x)g(x) = (x2 + x + 1) (x3 + x + 1) = x5 + x4 + 1
This corresponds to the codeword [1 0 0 0 1 1 0].
We can verify the validity of the codeword polynomial by computing
s(x) = c(x)h(x) = (x5 + x4 + 1) (x4 + x2 + x +1) mod (x7 – 1)
= x9 + x8 + x7 + x2 + x + 1 mod (x7 – 1)
= (x2 + x + 1) (x7 – 1) mod (x7 – 1) = 0
B % 5 D–E % * % DFE - %
codes, i.e., over GF(2).

4.6 Quasi-Cyclic Codes and Shortened Cyclic Codes LO 2

In this section we will study codes that are closely related to cyclic codes.

. An (n, k) / 1 is a linear block code such that, for some m coprime with
n, the polynomial xmc(x) calculated mod xn – 1 is a codeword polynomial if c(x) is a valid codeword
polynomial.

Example 4.18 C ! 85 G % '

È1 1 1 0 1 1 0 0 0 0 0 1˘
Í0 0 1 1 1 1 0 1 1 0 0 0 ˙˙
G= Í
Í0 0 0 0 0 1 1 1 1 0 1 1˙
Í ˙
Î0 1 1 0 0 0 0 0 1 1 1 1˚

Here m = 3 because the cyclic shift of any row of G by three places produces another row of the
matrix.

Any cyclic code can be , 85 G % ** mth symbol where m

is a factor of n & 85 G % ** % - %
& ** % % % ;

3 An (n, k) linear code is called a proper if it is obtained by deleting

m consecutive places from an (n + m, k + m) cyclic code.
 Cyclic Codes ° 185

The unused symbols in a shortened code are always set to zero and are not transmitted. The receiver
re-inserts the zeroes prior to decoding. Thus, even though the code is no longer cyclic, the same encoder and
decoder can be used after the zeroes have been taken care of.

Example 4.19 Consider the (7, 4) Hamming code, which is also a cyclic code. We form a
shortened (5, 3) code as follows. The two leading bits in the codeword are set to zero, followed by two
information bits and the same three parity bits. The minimum distance of the shortened code is also 3,
making it a single error correcting code.

4.7 Burst Error Correction LO 2

K %5 5 %5 7 ' * %
communications channel, fading results in burst errors. When errors occur at a stretch, as
opposed to random errors, we term them as 4 .

Example 4.20 L
be
c=0100011101010000101101

! " #
b=0000011011011110000000
# " "
channel. Here the length of the burst is 10 bits, however, not all the ten locations are in error.

5 A of length t is a vector whose non-zero components are among

t $ %&

If we are constructing codes for channels more prone to burst errors of length t (as opposed to an arbitrary
pattern of t ' " " $ )
e(x) = xi.b(x) (4.14)
where e(x) is a polynomial of degree £ t – 1 and b(x) is the burst pattern. xi marks the starting location of the
burst pattern within the codeword being transmitted.
A code designed for correcting a burst of length t + +
s(x) = Rg(x)[e(x)] (4.15)
is different for each polynomial representing a burst of length t.

Example 4.21 /or a binary code of block length n = 15, consider the generator polynomial
g(x) = x6 + x3 + x2 + x + 1
This code is capable of correcting bursts of length 3 or less. To prove this we must show that all the
syndromes corresponding to the different burst errors are distinct. The different burst errors are
186 ° Information Theory, Coding and Cryptography

(i) Bursts of length 1

e(x) = xi for i = 0, 1, …, 14
(ii) Bursts of length 2
e(x) = xi◊ (1 + x) for i = 0, 1, …, 13, and e(x) = xi◊(1 + x2) for i = 0, 1, …, 13
(iii) Bursts of length 3
e(x) = xi◊(1 + x + x2) for i = 0, 1, …, 12
It can be shown that the syndrome of all these 56 (15 + 14 + 14 + 13) error patterns are distinct. A
table can be made for each pattern and the corresponding syndrome which can be used for correcting a
" 8 9 " $ + "
$ # " : ;' +
code with code rate = k/n = 0.6 and minimum distance d * = 3. This code can correct only one random
: ?' @ "
burst error of length 1.

" C
+ % " D EF
bursts of length t or less must have at least 2t + + G # 6 4 .
In the next three sections, we will study three different sub-classes of cyclic codes. Each sub-class has a
$ K

Learning Review
YOU ARE NOW READY TO ATTEMPT THE FOLLOWING QUESTIONS LINKED TO LO 2:
/ " + " + " S
GF(2).
M / " :N 8' + GF(2). List all the codewords. S

8 / + + + GF(2), with block length S

n = 7 and g(x) = x4 + x3 + x2 + 1. Also, write the corresponding H matrix.
4. Determine all binary cyclic codes of block length n = 9. M

5. Consider the generator polynomial g(x) = (x3 + x + 1) for a (7, 3) cyclic code over GF(2). D
Let the received vector r(x) = (x + x2 + x4+ x5+ x6). Calculate the syndrome polynomial
for r(x) and all its cyclic shifts.

!" " #
$% & '

The answers are available here. Scan to check

Or
Visit [Link]
 Cyclic Codes ° 187

4.8 Fire Codes

7 AF is a cyclic burst error correcting code over LO 3

GF(q) with the generator polynomial
Describe some popular cyclic
g(x) = (x2t–1–1)p(x) (4.16)
codes.
where p(x) is a prime polynomial over GF(q) whose degree m is not
smaller than t and p(x) does not divide x2t–1–1. The block length of the
/ " n such that g(x) divides xnS F /
can correct all burst errors of length t or less.

Example 4.22 C / t = m = 3. A prime polynomial over GF(2) of degree

3 is p(x) = x3 + x + 1, which does not divide (x5 S ' # " + /
g(x) = (x5 – 1)p(x) = (x5 – 1)(x3 + x + 1)
x8
= + + – x6
x–1 x5 x3 –
8 6 5 3
=x +x +x +x +x+1

1
0.9

0.8

0.7

0.6

0.5
0.4 t
2 3 4 5 6 7 8 9 10

Fig. 4.1

The degree of g(x) = n – k = 8. The block length is the smallest integer n such that g(x) divides
xn – 1. After trial and error we get n V 8 # / :8 MN'
g(x) = x8 + x6 + x5 + x3 + x + 1. This code can correct up to bursts of length 3. The code rate of this code
NN $ " + g(x) = x6 + x3 + x2 + x + 1 which has a code rate
of only 0.6.
/ $ t # +/ : m = t)
for different values of t /" W
188 ° Information Theory, Coding and Cryptography

Example 4.23 T +/ "

(n, k) t
(9, 4) 2
(35, 27) 3
(105, 94) 4
(279, 265) 5
(693, 676) 6
(1651, 1631) 7
(3825, 3802) 8

4.9 Golay Codes LO 3

The Binary Golay Code
9 8; :M8 M' d* V N X
perfect code,
ÏÊ nˆ Ê nˆ Ê nˆ Ê nˆ ¸ (4.17)
M ÌÁ ˜ + Á ˜ (q - 1) + Á ˜ (q - 1) 2 + ... + Á ˜ (q - 1)t ˝ = q n
ÔÓË 0¯ Ë 1¯ Ë 2¯ Ë t¯ Ô˛
$ D n = 23, k = 12, M = 2k = 212, q = 2 and t = (d* – 1)/2 = 3. Thus (23, 12)
+Y + ) + )
the factorisation of (x23 – 1).
(x23 – 1) = (x – 1)(x11 + x10 + x6 + x5 + x4 + x2 + 1)(x11 + x9 + x7 + x6 + x5 + x + 1)
= (x – 1)g1(x)g2(x) (4.18)
The degree of g1(x) = n – k = 11, hence k = 12, which implies that there exists a (23, 12) cyclic code. In
order to prove that it is a perfect code, we must show that the minimum distance of this (23, 12) cyclic code
is 7. One way is to write out the parity check matrix, H, and show that no six columns are linearly dependent.
Another way is to prove it analytically, which is a long and drawn-out proof. The easiest way is to write a
computer program to list out all the 212 $ " :
?' # M " \ +
block length of this perfect code makes it impractical for most real life applications.

INDUSTRIAL RELEVANCE
The Automatic Link Establishment *0!1+ 2& 3040
5 *67 86+ '
 Cyclic Codes ° 189

The Ternary Golay Code

! " # $ # $ x11 – 1) over GF(3).

(x11 – 1) = (x – 1)(x5 + x4 – x3 + x2 – 1) (x5 – x3 – x2 – x – 1)
= (x – 1)g1(x)g2(x) (4.19)
The degree of g1(x) = n – k = 5, hence k = 6, which implies that there exists a (11, 6) cyclic code. In order to
prove that it is a perfect code, we must show that the minimum distance of this (11, 6) cyclic code is 5. Again,
" !
It can be shown that (xp – 1) has a factorisation of the form (x – 1)g1(x)g2(x) over GF(2), whenever p is a
prime number of the form 8m ± 1 (m is a positive integer). In such cases, g1(x) and g2(x $ &
codes. If the minimum distance of the code generated by g1(x) is odd ! /
–
d * ≥ ÷n (4.20)
Note that n denotes the block length.

4.10 Cyclic Redundancy Check (CRC) Codes LO 3

' # $ -/- 7 k-bit block of bits the (n, k -/-
generates (n – k $# ; & 7-< > ! # $
T = n-bit frame to be transmitted
D = k-bit message block (information bits)
F = (n – k 7-< n – k) bits of T
P = the predetermined divisor, a pattern of (n – k + 1) bits.
< " ? P, should be able to divide the codeword T. Thus, T/P has no remainder.
Now, D is the k-bit message block. Therefore, 2n–kD amounts to shifting the k bits to the left by (n – k) bits, and
" $ @ # # # & & " $
" & T, can then be represented as
T = 2n–k D + F (4.21)
Adding F & # D and F. If we divide 2n–kD by P, we obtain
2n - k D R
=Q+ (4.22)
P P
where, Q & R/P < "" R 7-<
T= 2n–k D +R (4.23)
In this case, upon dividing T by P we obtain
T 2n - k D + R 2n - k D R
= = +
P P P P
R R R+R
=Q+ + =Q+ =Q (4.24)
P P P
Thus there is no remainder, i.e., T is exactly divisible by P $ 7-< "
2n–k D by P and use the (n – k C 7-<
190 ° Information Theory, Coding and Cryptography

Let an error E occur when T is transmitted over a noisy channel. The received word is given by
V=T+E (4.25)
-/- # # V is completely divisible by P. This translates to the
case when E is completely divisible by P (because T is divisible by P).

Example 4.24 Let the message D = 1010001101, i.e., k = 10 and the pattern, P = 110101. The
# 7-< # n 7-<
7 $ " D5 (left shift by 5 and pad with 5 zeroes). This yields
25D = 101000110100000
Next divide the resulting number by P = 110101. By long division, we obtain Q = 1101010110 and
R = 01110. The remainder is added to 25 D to obtain
T = 101000110101110
T is the transmitted codeword. If no errors occur in the channel, the received word, when divided by
P, will yield 0 as the remainder.

-/- ! $ " " > $ " D(x)

and the predetermined divisor be P(x). Therefore,
x n - k D( x) R( x)
= Q( x) +
P( x) P( x)
T ( x) = x n - k D( x) + R( x) (4.26)
At the receiver end, the received word is divided by P(x < ""
V(x) = T(x) + E(x) (4.27)
where E(x) is the error polynomial. Then [T(x) + E(x)]/P(x) = E(x)/P(x) because T(x)/P(x) = 0. Those errors
that happen to correspond to polynomials containing P(x) as a factor will slip by, and the others will be caught
# -/- " P(x $ " # -/-
-/- ; .

Example 4.25 Let P(x) = x16 + x15 + x2 + 1. This P(x) acts as the generator polynomial. Let the
message to encode be D(x) = x13 + x + 1.
< n – k = degree of P(x ! " D(x) by x16.
x16D(x) = x19 + x17 + x16
Divide x16 D(x) by P(x) to obtain
R(x) = x15 + x5 + x4 + x3 + 1
The codeword will be
T(x) = x16 D(x) + R(x)
= (x19 + x17 + x16) + (x15 + x5 + x4 + x3 + 1)
= x19 + x17 + x16 + x15 + x5 + x4 + x3 + 1
 Cyclic Codes ° 191

Example 4.26 <uppose the transmitted codeword undergoes a single-bit error. The error
polynomial E(x) can be represented by E(x) = xi, where i determines the location of the single-bit error.
If P(x) contains two or more terms, E(x)/P(x) can never be zero. Thus all the single errors will be caught
-/-

Example 4.27 <uppose two isolated errors occur, i.e., E(x) = xi + x j, i > j. Alternately,
E(x) = x j(xi–j + 1). If we assume that P(x) is not divisible by x #! # $
double errors is that P(x) does not divide xk + 1 for any k up to the maximum value of i – j (i.e., the frame
$ 7 " x15 + x14 + 1 will not divide xk + 1 for any value of k below 32,768.

Example 4.28 <uppose the error polynomial has an odd number of terms (corresponding to an
odd number of errors). An interesting fact is that there is no polynomial with an odd number of terms
that has x + 1 as a factor if we are performing binary arithmetic (modulo 2 operations). By making
(x + 1) as a factor of P(x), we can catch all errors consisting of odd number of bits (i.e., we can detect
# # " J

Another interesting feature of CRC codes is its ability to detect burst errors. A burst error of
length k can be represented by xi(xk–1 + xk–2 + … + 1), where i determines how far from the
right end of the received frame the burst is located. If P(x) has an x0 term, it will not have an xi
as a factor. So, if the degree of (xk–1 + xk–2 + … + 1) is less than the degree of P(x), the
remainder can never be zero. Therefore, a polynomial code with r check bits can detect all
burst errors of length £ r. If the burst length is r + 1, the remainder of the division by P(x) will be zero if
and only if the burst is identical to P(x). Now, the 1st ! " #
intermediate bits can be 1 or 0. Therefore, the exact matching of the burst error with the polynomial P(x)
depends on the r – 1 intermediate bits. Assuming all combinations are equally likely, the probability of a miss
1
is r -1 . One can show that when error bursts of length greater than r + 1 occur, or several shorter bursts
2 1
occur, the probability of a bad frame slipping through is r .
2

Example 4.29 <ome of the popular of P(x K

x4 x3
-/-?MK P(x) = + + + + 1 x2 x1
-/-?OK P(x) = x7 + x6 + x4 + 1
-/-?PK P(x) = x8 + x7 + x6 + x4 + x2 + 1
-/-? DK P(x) = x12 + x11 + x3 + x2 + x1 + 1
-/-? K P(x) = x16 + x15 + x2 + 1
-/-?--Q K P(x) = x16 + x12 + x5 + 1
-/-?RDK P(x) = x32 + x26 + x23 + x22 + x16 + x12 + x11 + x10 + x8 + x7 + x5 + x4 + x2 + x1 + 1
-/-?MS <UK P(x) = x40 + x26 + x23 + x17 + x3 + 1
192 ° Information Theory, Coding and Cryptography

-/-? M Q<'K P(x) = x64 + x4 + x3 + x1 + 1

-/-? M V-UW? PDK P(x) = x64 + x62 + x57 + x55 + x54 + x53 + x52 + x47 + x46 + x45 + x40 +
x39+ x38 + x37 + x35 + x33 + x32 + x31 + x29 + x27 + x24 + x23 +
x22 + x21 + x19 + x17 + x13 + x12 + x10 + x9 + x7 + x4 + x1 + 1 (4.28)
W -/- xX # < "" P(x) has a factor (x + 1). Then,
" # J
-/-? D # # # ? $ D? 7-< Y -/-?
-/-?--Q " " # P? 7-< # $
and double errors, all errors with odd number of bits, all burst errors of length 16 or less, 99.997% of
17-bit bursts and 99.998% of18-bit and longer bursts.

INDUSTRIAL RELEVANCE
()($96 $ $ Synchronous Transmission Standards.
()($7: '4/ ()($;7 <4" High-Level Data Link
Control *2=!(+ $ $
< " > 4 > *<4"+ ()($;7 1(/0$8?6
part of the European Computer Manufacturers Association *1(/0+

Learning Review
YOU ARE NOW READY TO ATTEMPT THE FOLLOWING QUESTIONS LINKED TO LO 3:
- -/- P(x) = x16 + x15 + x2 X 7 " S
determine how many odd number of errors it can detect? Also comment on the detection
capability for double errors.
D - -/- P(x) = x16 + x15 + x2 + 1. Let the message polynomial be M
D(x) = x + x + x + x + x8 + x5 + x2 + x X 7
14 13 11 10 " $ codeword
polynomial, T(x - -/- O ^
R - -/-? P(x) = x16 + x15 + x2 + 1. What can be the largest size of the M
codeword (in bytes) that can detect double errors ?
M - DM D `7 jW<W # W > ; M
Establishment with g(x) = x11 + x9 + x7 + x6 + x5 + x X " $$
matrix, G24 and the table of weights of different codewords.

!" " #
$% & '

The answers are available here. Scan to check

Or
Visit [Link]
 Cyclic Codes ° 193

4.11 Circuit Implementation of Cyclic Codes

< # $
V $ $ # & " LO 4
by polynomials. The shift property of shift registers is ideally suited for such
Explain how to implement
" < # $ ; # " #
cyclic codes using circuits.
shifting the contents of one unit to the next at every clock pulse. Here we
will focus on circuit implementation for codes over GF(2m). Beside the
# $ ; # # $ K
W | " " ! !
(ii) An adder, which takes in two inputs and adds them together. A simple circuit realisation of an adder is
}exclusive-or’ }xor’ gate.
W " }and’ gate.
" 7$ MD

Fig. 4.2 Circuit elements used to construct encoders and decoders for cyclic codes.

W! # GF D " " $ 7 GF(2m & m bits to

" 7 " # GF(8) can be represented as the elements of the set
~SSS SS S S S SS S S  7 " ;" #
element from one stage of the shift register to the next. The effective shift register for GF P 7$
MR W # ! " ax2 + bx + c, where a, b, c are binary, and the
power of the indeterminate x " 7 " S x2 + 1.

One stage of the effective shift register

Fig. 4.3 The effective shift register for GF(8).

194 ° Information Theory, Coding and Cryptography

Example 4.30 W " # !

element over GF P / # GF(8) from GF(2) using the prime polynomial
p(x) = x3 + x X # ! S x, x + 1, x2, x2 + 1, x2 + x, x2 + x + 1. We want to
" # " # ! ax2 + bx + c) by
another element, say, x2 + x. We have
(ax2 + bx + c)(x2 + x) = ax4 + (a + b)x3 + (b + c) x2 + cx (modulo p(x))
= (a + b + c)x2 + (b + c)x + (a + b)
' " 7$ MM

a b c

+ +

Fig. 4.4 ax2 + bx + c) by the element (x2 + x) over GF(8).

We next focus on the multiplication of an arbitrary polynomial a(x) by g(x). Let the polynomial g(x) be
represented as
g(x) = gLxL +… + g1x + g0 (4.29)
the polynomial a(x) be represented as
a(x) = akxk +… + a1x + a0 (4.30)
the resultant polynomial b(x) = a(x)g(x) be represented as

b(x) = bk+Lxk+L +… + b1x + b0 (4.31)

The circuit realisation of b(x

a8 ! 6 8!69

+ b(x)
+ + + +

gL gL – 1 gL–2 gL–3 g1 g0

a(x)

Fig. 4.5 b(x) = a(x)g(x).

 Cyclic Codes ° 195

a(x) and g(x) are convolved by the shift register.

! ! " !# ! # "
mechanism of encoding a cyclic code by multiplying the information polynomial by the generator polynomial.

Example 4.31 The encoder circuit for the generator polynomial

g(x) = x8 + x6 + x5 + x3 + x + 1
$
! # " t = m = 3. It is easy to interpret the circuit. The
8 memory units shift the input, one unit at a time. The shifted outputs are summed at the proper locations.
"" ! % !
x x2 x3 x4 x5 x6 x7 x8
+ + + + +

Fig. 4.6 !
g(x) = x8 + x6 + x5 + x3 + x + 1.

Example 4.32 Consider the systematic encoding process for the cyclic (7, 3) code using
g(x) = x4 + x3 + x2 &' * -/
encoding is done in two steps.
Switch 1

+ + + Information bits

B A

Switch 2

Encoded bits

Fig. 4.7 A circuit for systematic encoding using g(x) = x4 + x3 + x2 + 1.

( ): ' ! ! ; : < >?@ " /

during the 1st three shifts. This being a systematic encoder, the 1st three codeword bits will be the
information bits.
196 ° Information Theory, Coding and Cryptography

( ): ' ! ; : < >A@ /

next four shifts.

We can also use a shift register circuit for dividing an arbitrary polynomial, a(x /# % ! # "
g(x). We assume here that the divisor is a monic polynomial. We already know how to factor out a scalar in
order to convert any polynomial to a monic polynomial. The division process can be expressed as a pair of
B D Q(r)(x) and R(r)(x / B ! # " " ! # "
rth recursion step, with the initial conditions Q(0)(x) = 0 and R(0)(x) = a(x B
be written as
Q(r)(x) = Q(r–1)(x) + R(n-r)xk–r
R(r)(x) = R(r–1)(x) – R(n-r)xk–rg(x) (4.32)
where R(n–r) ! " ! # " Fr G '
arbitrary polynomial a(x /# % ! # " g(x H ? n shifts,
B ! "
the shift register implementation of a decoder is very simple. The contents of the shift register can be checked
for all entries to be zero after the division of the received polynomial by the generator polynomial. If even a
single memory unit of the shift register is non-zero, an error is detected.

g0 g1 gn–k–2 gn–k–1 gn–k–1

+ + + +

Fig. 4.8 A shift register circuit for dividing by g(x).

Example 4.33 The circuit for dividing by g(x) = x3 + x & ' J

+ +

Fig. 4.9 Circuit realisation for dividing by g(x) = x3 + x + 1.

 Cyclic Codes ° 197

Example 4.34 The circuit for dividing by g(x) = x4 + x2 & ' 'K

+ +
Fig. 4.10 Circuit realisation for dividing by g(x) = x4 + x2 + 1.

Example 4.35 The shift register circuit for dividing by g(x) = x8 + x6 + x5 + x3 + x + 1 is given in
''
x x2 x3 x4 x5 x6 x7 x8
+ + + + +

Fig. 4.11 A shift register circuit for dividing by g(x) = x8 + x6 + x5 + x3 + x + 1.

!
a buffer. It is subjected to divide-by-g(x) operation. As we have seen, the division can be carried out very
# /# " "! ! /
(pre-computed) syndromes. This set of syndromes corresponds to the set of correctable error patterns. If a
syndrome match is found, the error is subtracted out form the received word. The corrected version of the
received word is then passed on to the next stage of the receiver unit for further processing. This kind of a
decoder is known as the * N '<

Divide by g(x) feedback

Received word
+ n – k stage shift register

Compare with all test syndromes

Error word –1

+ Corrected word

n stage shift register

Fig. 4.12 " # !! $

198 ° Information Theory, Coding and Cryptography

Learning Review
YOU ARE NOW READY TO ATTEMPT THE FOLLOWING QUESTIONS LINKED TO LO 4:
1. Design a circuit for error detection for the binary cyclic (7, 3) code using M

g(x) = x4 + x3 + x2 + 1
2. P " ! # " 'Q M
! !' : ' ! ! : < >?@
in order to read in the information bits during the 1st !< : '
! : < >A@ /
next three shifts.
Switch 1

+ + Information bits
B A

Switch 2

Encoded bits
Fig. 4.13

!" " #
$% & '

The answers are available here. Scan to check

Or
Visit [Link]

4.12 Concluding Remarks

structure formation in the code and (ii) the existence of elegant mathematical tools based on polynomials
!"#
The work on cyclic codes was further developed by Peterson and Kasami. Pioneering work on the minimum
$ % !&' *
+ $ , $ + , - /
chapter. It was soon discovered that almost all the LBCs discovered earlier could be converted to cyclic.
0 * !"! 1
2 2 + +
1949.
 Cyclic Codes ° 199

5 0 + 6
!&' 7 8 +6 : +6 %

LEARNING OUTCOMES

Y A polynomial is a mathematical expression f(x) = f0 + f1x + … + fmxm, where the symbol x is called the
f0, f1, … fm are the elements of GF(q fm is called the
7 fm π 0, then m is called the degree of the polynomial, and is denoted by deg f(x) ◊
*
Y The division algorithm states that, for every pair of polynomial a(x) and b(x) π 0 in F[x], there exists a
- q(x + - + r(x), the remainder, such that a(x) = q(x) b(x) + r(x),
where deg r(x) < deg b(x). The remainder is sometimes also called the residue, and is denoted by
Rb(x)[a(x)] = r(x).
Y
(i) Rf(x)[a(x) + b(x)] = Rf(x)[a(x)] + Rf(x)[b(x)], and
(ii) Rf(x)[a(x) ◊ b(x)] = Rf(x){Rf(x)[a(x)] ◊Rf(x)[b(x)]}
where a(x), b(x) and f(x) are polynomials over GF(q).
Y A polynomial f(x) in F[x] is said to be reducible if f(x) = a(x) b(x), where a(x), b(x) are elements of f(x)
and deg a(x) and deg b(x) are both smaller than deg f(x). If f(x) is not reducible, it is called irreducible.
A monic irreducible polynomial of degree at least one is called a prime polynomial.
!Y The ring F[x]/f(x f(x) is a prime polynomial in F[x].
!Y A code C in Rn is a cyclic code if and only if C
(i) a(x), b(x) ΠC fi a(x) + b(x) ΠC,
(ii) a(x) ΠC and r(x) ΠRn fi a(x)r(x) ΠC.
!Y 1
(i) Take a polynomial f(x) in Rn.
(ii) Obtain a set of polynomials by multiplying f(x) by all possible polynomials in Rn.
(iii) The set of polynomials obtained as above corresponds to the set of codewords belonging to a
cyclic code. The block length of the code is n.
!Y Let C be a (n, k) non-zero cyclic code in Rn. Then,
/ - g(x) of the smallest degree in C,
(ii) the cyclic code C consists of all multiples of the generator polynomial g(x) by polynomials of
degree k – 1 or less.
(iii) g(x) is a factor of xn – 1.
(iv) The degree of g(x) is n – k.
!Y 1 + C, with generator polynomial g(x) = g0 + g1x + …+ grxr of degree r, the generator
matrix of given by
200 ° Information Theory, Coding and Cryptography

È g0 g1 ... gr 0 0 0 ... 0˘
Í0 g0 g1 ... gr 0 0 ... 0 ˙˙
Í
G= Í0 0 g0 g1 ... gr 0 ... 0 ˙ k = (n – r) rows
Í ˙
Í : : : ˙
ÍÎ 0 0 0 0 0 g0 g1 ... g r ˙˚

n columns

!Y 1 + C, with the parity check polynomial h(x) = h0 + h1x + …+ hkxk, then the parity check
matrix is given by

Èhk hk -1 ... h0 0 0 0 ... 0˘

Í0 hk hk -1 ... h0 0 0 ... 0 ˙˙
Í
H=Í0 0 hk hk -1 ... h0 0 ... 0 ˙ (n – k) rows
Í ˙
Í: : : ˙
ÍÎ 0 0 0 0 0 hk hk -1 ... h0 ˙˚

n columns

!Y xn – 1 = h(x)g(x), where, g(x) is the generator polynomial and h(x) is the parity check polynomial.
!Y An (n, k - F 0 m, co-prime with n, the polynomial
xmc(x) calculated mod xn – 1 is a codeword polynomial if c(x) is a valid codeword polynomial.
!Y An (n, k) linear code is called a proper shortened cyclic code if it is obtained by deleting m consecutive
places from an (n + m, k + m) cyclic code.
!Y * 1 G GF(q) with the generator polynomial
g(x) = (x2t–1 – 1)p(x), where p(x) is a prime polynomial over GF(q) whose degree m is not smaller than
t and p(x) does not divide x2t–1 J 0 1 n such that
g(x) divides xn J * 1 t or less.
!Y The generator polynomial of the 2
g1(x) = (x11 + x10 + x6 + x5 + x4 + x2 + 1), or
g2(x) = (x11 + x9 + x7 + x6 + x5 + x + 1).
!Y The generator polynomial of the 2
g1(x) = (x5 + x4 – x3+ x2 – 1), or
g2(x) = (x5 – x3 – x2– x – 1).
!Y S % 1 k-bit block of bits the (n, k %
encoder generates (n – k 0 - 1 5
!Y 5 U
- G
suited for such operations.
 Cyclic Codes ° 201

Everything should be made as simple as possible, but not simpler

Albert Einstein (1879–1955)

MULTIPLE CHOICE QUESTIONS

V 1 +
(a) xn+1 + 1 (b) xn – 1 (c) xn /2 (d) x2n/3
VW 1 + X
(a) Rf(x)[a(x) + b(x)] = Rf(x)[a(x)] + Rf(x)[b(x)], and
(b) Rf(x)[a(x) ◊b(x)] = Rf(x){Rf(x)[a(x)] ◊ Rf(x)[b(x)]}
(c) Both (a) and (b)
(d) None of the above
4.3 Which of the following is true?
(a) A ring F[x]/f(x f(x) is a monic polynomial in F[x]
(b) A ring F[x]/f(x f(x) is a prime polynomial in F[x]
(c) A ring F[x]/(f(x ^ f(x) is a prime polynomial in F[x]
(d) None of the above
4.4 A code C in Rn is a cyclic code if and only if C
(a) a(x), b(x) ΠC fi a(x) + b(x) ΠC
(b) a(x) ΠC and r(x) ΠRn fi a(x)r(x) ΠC
(c) Both (a) and (b)
(d) None of the above
4.5 The degree of g(x) is
(a) n – k + 1 (b) n + k + 1 (c) n – k – 1 (d) n – k
4.6 n
If x – 1 = h(x)g(x), where, g(x) is the generator polynomial, then the parity check polynomial would
be
(a) h(x) (b) g(x)h(x) (c) xn (d) x – 1
4.7 An (n, k) linear code is called a proper shortened cyclic code if it is obtained by deleting m consecutive
places from an
(a) (n – m, k + m) cyclic code (b) (n + m, k – m) cyclic code
(c) (n + m, k + m) cyclic code (d) None of the above
4.8 Which of the following is capable of correcting any combination of three or fewer random errors in a
block of 23 bits?
, 5 ,
% 2
202 ° Information Theory, Coding and Cryptography

4.9 Which is the generator polynomial of the 2

(a) g1(x) = (x11 + x10 + x6 + x5 + x4 + x2 + 1), or
(b) g2(x) = (x11 + x9 + x7 + x6 + x5 + x + 1)
(c) Both (a) and (b)
(d) None of the above
4.10 Which is the generator polynomial of the 2
(a) g1(x) = (x5 + x4 – x3+ x2 – 1), or
(b) g2(x) = (x5 – x3 – x2– x – 1)
(c) Both (a) and (b)
(d) None of the above

For interactive quiz with answers, scan the QR code given here
Or
Visit [Link]

SHORT-ANSWER TYPE QUESTIONS

4.1 No element of ring can have a multiplicative inverse. True or false?

4.2 Is the (n, 1) repetition code a cyclic code? Is the (7, 4) binary Hamming Code cyclic?
4.3 The generator matrix of a cyclic code can be represented as a Toeplitz matrix. True or false?
4.4 The parity check polynomial of a cyclic code can be used to generate another cyclic code. True or
false?
V" 2G V "+ ,
V& 5 g(x) = g0 + g1x + … + grxr. Can g0 be 0?
4.7 If g(x) has a factor (x + 1) then no codeword has odd weight. True or false?
Vj 1 "+ + g(x) = x4 + x ^ + 0 h(x).
V! 1 + g(x) = xn-1 + xn-2 + … + x ^ + 0 h(x).
4.10 g
If g(x) is a generator polynomial of an (n, k), then g(x ) is a generator polynomial of an (g n, gk) code.
True or false?
4.11 The minimum weights of the cyclic code generated by g(x) and by the cyclic code generated by g(xg)
are the same. True or false?
4.12 What is the code rate of the cyclic code of n = 7 with g(x) = x + 1?
4.13 How many errors can the cyclic code given by g(x) = x8 + x7 + x6 + x4 + 1 correct?
È0 1 1 0 1 ˘
4.14 Is the code generated by G = Í ˙ cyclic?
Î1 1 0 1 0 ˚
V " 1 !+ & G g(x) = x3 + 1.

For answers, scan the QR code given here

Or
Visit [Link]
 Cyclic Codes ° 203

PROBLEMS
S V : + - G X
(i) {0000, 0110, 1100, 0011, 1001} over GF(2).
(ii) {00000, 10110, 01101, 11011} over GF(2).
(iii) {00000, 10110, 01101, 11011} over GF(3).
(iv) {0000, 1122, 2211} over GF(3).
(v) The q-ary repetition code of length n.
M 4.2 Construct the addition and multiplication table for
(i) F[x]/(x2 ^ G GF(2).
(ii) F[x]/(x2 ^ G GF(3).
: G X
S 4.3 Answer the following for polynomials over GF &
(i) How many distinct second-degree monic polynomials of the form x2 + ax + b (b π 0) exist?
(ii) How many distinct polynomials of the form (x q a)(x q b) exist?
S 4.4 List out all the irreducible polynomials over
(i) GF(2) of degrees 1 to 5.
(ii) GF(3) of degrees 1 to 3.
S V" 1 0 " 1
S V& 5 xn – 1 is a product of r distinct irreducible polynomials over GF(q). How many cyclic
codes of block length n over GF(q) exist? What can you comment on the minimum distance of
these codes?
S V# 1 x8 – 1 over GF(3).
(ii) How many ternary cyclic codes of length 8 exist?
, - j / X
M 4.8 Let the polynomial
g(x) = x10 + x8 + x5 + x4 + x2 + x + 1
be the generator polynomial of a cyclic code over GF(2) with block length 15.
1 G.
1 0 / H.
(iii) How many errors can this code detect?
(iv) How many errors can this code correct?
(v) Write the generator matrix in the systematic form.
D 4.9 Consider the polynomial
g(x) = x6 + 3x5 + x4 + x3 + 2x2 + 2x + 1
(i) Is this a valid generator polynomial for a cyclic code over GF(4) with block length 15?
1 0 / H.
(iii) What is the minimum distance of this code?
(iv) What is the code rate of this code?
(v) Is the received word v(x) = x8 + x5 + 3x4 + x3 + 3x + 1 a valid codeword?
204 ° Information Theory, Coding and Cryptography

S 4.10 Is g(x) = x10 + x7 + x6 + x4 + x2 + 1 a valid generator polynomial for a binary cyclic code with
n = 21? Why/why not?
M 4.11 Consider a binary cyclic code with n = 15 and a generator polynomial given by g(x) = x8 + x7 + x6
+ x4 ^ 1 h(x) and the parity check matrix.
D 4.12 An error vector of the form xi + xi+1 in Rn | 5
generated by the generator polynomial g1(x) = (x – 1)gH(x) is capable of correcting all double
adjacent errors, where gH(x) is the generator polynomial of the binary Hamming code.
S 4.13 Determine the burst error correction length for the cyclic code represented by the following
generator polynomial
g(x) = x13 + x10 + x7 + x6 + x5 + x4 + x2 + 1
S V V 5 0 1 n = LCM(2t – 1, qm – 1).
S V " 5 g(x) = (x11 + 1)(x6 + x ^ G 1 :
burst error correction length for this code?
S V & 5 G g(x) = x14 + x11 + x9 + x5 + x2 + 1 is capable of correcting a burst
error of length 5 or less.
S V # 5 n – k can detect a
fraction 1 – 2 – (n – k) + 1 of all burst patterns of length (n – k) + 1.
M 4.18 Design the shift register encoder and the Meggitt decoder for the code generated in Problem 4.9.
M 4.19 The code with the generator polynomial g(x) = (x23 + 1)(x17 + x3 + 1) is used for error-detection and
256
(i) How many random errors can this code correct?
(ii) How many burst errors can this code correct?
M V W' W+ W 2 G g(x) = x11 + x9 + x7 + x6 + x5 + x + 1.
row of the generator matrix for this code will be [1 0 1 0 1 1 1 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0]. The
WF 1
M 4.21 Consider a binary cyclic code with the generator polynomial g(x) and block length n.
(i) If (x + 1) is a factor of g(x), show that all codewords must be of even weight.
(ii) If (x + 1) is not a factor of g(x) and n is odd, show that the code must contain the all-one
codeword 111….1.

COMPUTER PROBLEMS

V : G GF(q), given the

generator polynomial (or the generator matrix) for the code.
4.2 Write a computer program to divide a polynomial a(x) by another polynomial b(x) over GF(q). The
-
4.3 Write a computer program to factorise xn – 1 into prime polynomials over GF(2). Can you generalise
this over GF(q X
 Cyclic Codes ° 205

4.4 Consider g(x) = (x + 1)(xa + xb + 1). Let the codeword length, n, be a design parameter. Write a
‚a, b} such that g(x) has good double error detection capability
for large n.
V" : "+ W# 1 7
correct bursts of length 3 or less. What happens when you try to decode a received word with a burst
VX : !V#+ !V'j 1
What is the burst error correction length for this code?
4.6 Write a computer program to search for cyclic Hamming codes over GF(5). Is there a cyclic Hamming
code of block length n „ "& G 2 1 X
V# 7UUU j'W & %
g(x) = x32 + x26 + x23 + x22 + x16 + x12 + x11 + x10 + x8 + x7 + x5 + x4 + x2 + x + 1.
(i) What error patterns can this code detect?
(ii) What is the burst error correction length for this code?

PROJECT IDEAS
4.1 : & ( ; < & )5
2 + 1 2 | G
W+ W 2 +& 2 G F
% 5 + G 5†%
(i) probability of word error,
(ii) probability of bit error,
(iii) probability of undetected codeword error.
4.2 6 ) Consider a cyclic code with the generator polynomial g(x ‡ g(x)
as g ( x) = x g (1 x ).
* n-k

1 , G g(x) = x3 + x + 1.
*
5 g ( x) is also cyclic.
*
5 g(x) and g ( x) are the same.
G % G
g1(x) = (x11 + x10 + x6 + x5 + x4 + x2 + 1) and
g2(x) = (x11 + x9 + x7 + x6 + x5 + x + 1)
4.3 * ) Design a Meggitt Decoder that can handle both binary and ternary cyclic codes.
Comment on the computational complexity of this decoder. Assuming realistic values for each circuit
operation (addition, multiplication, comparison, shift register operation etc.) in mJ per operation, give
the average energy estimate for decoding (in mJ/bit).
4.4 = "& & ) Make a table of the weights of different codewords and how
/
$ W+ W 2
+& 2
$ "+ W# 1
G $ !V#+ !V'j 1
206 ° Information Theory, Coding and Cryptography

REFERENCES FOR FURTHER READING

% + % 6 + Introduction to Coding Theory, Cambridge University Press, 2006.
Error Control Coding Fundamentals and Applications (2nd edition),
Prentice-Hall, 2004.
Algebraic Codes for Data Transmission, Cambridge University Press, 2002.
4. Moon, T.K., Error Correction Coding: Mathematical Methods and Algorithms, Wiley, 2005.