Scribd
Upload
66 views3 pages

Understanding Entropy in Information Theory

Entropy is a measure of uncertainty or average information in messages from a source. Channel redundancy refers to extra or repeated bits in transmitted data. Source coding techniques include Huffman coding and Shannon-Fano coding. The properties of entropy include it being zero for sure or impossible events, and the upper bound being the log of the number of messages. The channel coding theorem states that if the transmission rate is below the channel capacity, errors can be made arbitrarily small using error-correcting codes. Memoryless sources produce symbols independently of previous symbols, like characters from a keyboard. Conditional entropy measures uncertainty remaining in the transmitted variable given the received variable, representing information lost in a noisy channel.

Uploaded by

vanchip28
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOC, PDF, TXT or read online on Scribd
66 views3 pages

Understanding Entropy in Information Theory

Entropy is a measure of uncertainty or average information in messages from a source. Channel redundancy refers to extra or repeated bits in transmitted data. Source coding techniques include Huffman coding and Shannon-Fano coding. The properties of entropy include it being zero for sure or impossible events, and the upper bound being the log of the number of messages. The channel coding theorem states that if the transmission rate is below the channel capacity, errors can be made arbitrarily small using error-correcting codes. Memoryless sources produce symbols independently of previous symbols, like characters from a keyboard. Conditional entropy measures uncertainty remaining in the transmitted variable given the received variable, representing information lost in a noisy channel.

Uploaded by

vanchip28
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOC, PDF, TXT or read online on Scribd

UNIT-V INFORMATION THEORY

2 Marks Questions and Answers


[Link] is entropy?
Entropy is also called average information per message. It is the ratio of total
information to number of messages. i.e.,
Entropy, H =(Total information)/ Number of messages .

2. What is channel redundancy?


Redundancy () = 1 code efficiency
Redundancy should be as low as possible.

3. Name the two source coding techniques.


The source coding techniques are, a) prefix coding
b) Shannon-fano coding c) Huffman coding

4. State any four properties of entropy.


a) For sure event or impossible event entropy is zero.
b) For M number of equally likely symbols, entropy is log 2 M
c) Upper bound on entropy is H max = log 2 M
d) Entropy is lower bound on average number of bits per symbol.

5. State the channel coding theorem for a discrete memory less channel.
Statement of the theorem:
Given a source of M equally likely messages, with M >>1, which is generating information at a rate.
Given channel with capacity C. Then if,R C
There exits a coding technique such that the output of the source may be transmitted over the channel
with a probability of error in the received message which may be made arbitrarily small.

[Link]
7. What is memory less source? Give an example.
The alphabets emitted by memory less source do not depend upon previous alphabets. Every alphabet is
independent. For example a character generated by keyboard represents memory less source.

8. Explain the significance of the entropy H(X/Y) of a communication system where X is the transmitter
and Y is the receiver.
a) H(X/Y) is called conditional entropy. It represents uncertainty of X, on average,
when Y is known.
b) In other words H(X/Y) is an average measure of uncertainty in X after Y is received. c) H(X/Y)
represents the information lost in the noisy channel.

9. What is prefix code?


In prefix code, no codeword is the prefix of any other codeword. It is variable length code. The binary
digits (codewords) are assigned to the messages as per their probabilities of occurrence.

10. What is information theory?


Information theory deals with the mathematical modeling and analysis of a communication system rather
than with physical sources and physical channels

11. Explain Shannon-Fano coding.


An efficient code can be obtained by the following simple procedure, known as Shannon
Fano algorithm.
Step 1: List the source symbols in order of decreasing probability.
Step 2: Partition the set into two sets that are as close to equiprobable as possible, and sign 0 to the upper
set and 1 to the lower set.
Step: Continue this process, each time partitioning the sets with as nearly equal probabilities as possible
until further partitioning is not possible.

12. Define bandwidth efficiency.


The ratio of channel capacity to bandwidth is called bandwidth efficiency. i.e,
Bandwidth
efficiency = Channel capacity (C ) /Bandwidth (B)

13. Define channel capacity of the discrete memory less channel.


The channel capacity of the discrete memory less channel is given as maximum
average mutual information.
16 MARK QUESTIONS
1. Discuss source coding theorem, give the advantage and disadvantages of channel coding in detail, and
discuss the data compaction.
2. Explain in detail Huffman coding algorithm and compare this with the other types of coding.
3. Explain the properties of entropy and with suitable example, explain the entropy of binary memory less
source.
4. Define mutual information. Find the relation between the mutual information and the Joint entropy of
the channel input and channel output. Explain the important properties of mutual information.
5. Encode the source symbols with following set of probabilities using Huffman coding.

[Link]
m = {0.4 0.2 0.12 0.08 0.08 0.08 0.04}
6. Derive the channel capacity theorem.
7. Write a note on rate distortion theory.
8. Explain the information capacity theorem.

[Link]

Common questions

Powered by AI

Bandwidth efficiency is the ratio of channel capacity to bandwidth, indicating how effectively the available bandwidth is utilized. It's important as it reflects the system's ability to convey maximum information within a given bandwidth, crucial for optimizing communication system performance .

Huffman coding is a type of prefix coding where codewords are assigned based on the probabilities of the messages, aiming to minimize the average code length. Compared to Shannon-Fano coding, Huffman coding is generally more efficient as it always produces a prefix-free code with the shortest possible average length when codes are generated optimally .

Entropy in a communication system measures the average information per message. The key properties of entropy include that it is zero for a sure or impossible event, for M equally likely symbols entropy is log2 M, the upper bound is log2 M, and it is the lower bound on the average number of bits per symbol .

Shannon-Fano coding involves listing source symbols by decreasing probability, then partitioning them into two sets with near-equal probabilities, assigning a '0' to the upper set and '1' to the lower, repeating until sets can't be split. This approach creates efficient codes by reflecting symbol probabilities in code lengths .

A memoryless source emits symbols independently, with each output not depending on previous outputs. This concept is significant in simplifying the analysis and modeling of information sources, allowing each symbol to be treated separately in probability calculations .

Source coding reduces data redundancy, optimizing the data representation, while channel coding adds redundancy intentionally to counteract errors during transmission. Together, they ensure efficient and reliable data transmission, balancing efficiency with reliability across communication mediums .

Channel redundancy, defined as 1 minus code efficiency, should be minimized to maximize code efficiency. Reducing redundancy ensures that data compression is effective by requiring fewer bits to represent information without loss, thus improving efficiency .

Mutual information measures the reduction in uncertainty of one random variable due to the knowledge of another. It is related to joint entropy by the relation I(X;Y) = H(X) + H(Y) - H(X, Y), where I is mutual information, and H represents entropy. Mutual information is crucial for assessing the amount of information shared between channel inputs and outputs .

The channel coding theorem states that for a source outputting M equally likely messages, if the information generation rate R is less than or equal to the channel capacity C, there exists a coding technique to transmit the information with arbitrarily small error probability. This theorem underlines the possibility of reliable data transmission at rates below the channel capacity .

The conditional entropy H(X/Y) represents the average uncertainty in the transmitter output X when the receiver output Y is known. It indicates the information lost in a noisy channel, as it quantifies the residual uncertainty in X after taking into account Y .

You might also like