Information theory
• Information theory is the scientific study of
the quantification, storage, and
communication of information.
Information
• Information is an idea/ message in
information theory.
• The message can be:
– Electrical signal
– Speech/voice
– Picture
– Video
– text
• In communication system, information is
transmitted from source to destination
• INFORMATION SOURCE->ENCODER->
Transmitter->Transmission channel-> Receiver-
>DECODER->RECEIVED INFORMATION
• A key measure in information theory is
entropy.
• Entropy quantifies the amount of uncertainty
involved in the value of a random variable or
the outcome of a random process.
• For example, identifying the outcome of a fair
coin flip (with two equally likely outcomes)
provides less information (lower entropy) than
specifying the outcome from a roll of a die
(with six equally likely outcomes).
• The amount of Information – inversely
proportional to probability and directly
proportional to uncertainty.
• If Probability of event is high, then the amount of
information obtained is less and vice versa.
• If the uncertainty is high, then the amount of
information obtained is high and viceversa.
• If the probabiity of event is high, then the
uncertainty is less and vice versa.
Uncertainty
• Eg.
• Message X={X0,X1,….XN}
• Probability of different messages
• P={P0,P1,….,PN}
• Total Probability= (i=0 to N) Pi
• Send message X0 with probability P0 is zero (i.e)
this event will never happen which means no
uncertainty.
• Send message X0 with probability P0 is one (i.e)
this event will happen 100% which means no
uncertainty.
• Send message X0 with probability P0 is 0.5
which means event has uncertainty.
• As probability of event decreases then
uncertainty increases.
Measure of information
• It is the information content present in a
message.
• Consider if the source is emitting independent
messages m={m1,m2,…..mn} with the
probability of occurrence is P={P1,P2,….Pn}
• Here P1+P2+…+Pn=1
• Amount of information is given by
Ik = log2 (1/Pk) wher k=1 to
n.
• Information is measured in bits.
• Uncertainty is more , then information is
more.
• Eg. P1=1/4 -- Uncertainty-U1
• P2=1/2 -- Uncertainty-U2
• Uncertainty-U1 > U2
• I1=log2(1/P1)
• =log2(1/1/4)=log2(4)=log2(22)=2log2(2)=2 bits
• I2=log2(1/P2)
• =log2(1/1/2)=log2(2)=log2(2)=log2(2)=1 bit
• I1 >I2