Chapter 7

Multimedia Networking

A note on the use of these ppt slides:

We’re making these slides freely available to all (faculty, students, readers). Computer
They’re in PowerPoint form so you see the animations; and can add, modify,
and delete slides (including this one) and slide content to suit your needs. Networking: A Top
They obviously represent a lot of work on our part. In return for use, we only
ask the following: Down Approach
❖ If you use these slides (e.g., in a class) that you mention their source
(after all, we’d like people to use our book!)
6th edition
❖ If you post any slides on a www site, that you note that they are adapted Jim Kurose, Keith Ross
from (or perhaps identical to) our slides, and note our copyright of this
material.
Addison-Wesley
March 2012
Thanks and enjoy! JFK/KWR

All material copyright 1996-2012

J.F Kurose and K.W. Ross, All Rights Reserved

Multmedia Networking 7-1

Multimedia networking: outline
7.1 multimedia networking applications
7.2 streaming stored video
7.3 voice-over-IP
7.4 protocols for real-time conversational
applications
7.5 network support for multimedia

Multmedia Networking 7-2

Multimedia networking: outline
7.1 multimedia networking applications
7.2 streaming stored video
7.3 voice-over-IP
7.4 protocols for real-time conversational
applications
7.5 network support for multimedia

Multmedia Networking 7-3

Multimedia: audio
❖ analog audio signal
sampled at constant rate
quantization
▪ telephone: 8,000
quantized
error value of
samples/sec analog value

audio signal amplitude

▪ CD music: 44,100 analog

samples/sec signal

❖ each sample quantized, i.e.,

rounded
▪ e.g., 28=256 possible time
quantized values
sampling rate
▪ each quantized value (N sample/sec)

represented by bits,
e.g., 8 bits for 256
values
Multmedia Networking 7-4
Multimedia: audio
❖ example: 8,000 samples/sec,
256 quantized values: 64,000
bps quantization
error
quantized
value of
❖ receiver converts bits back analog value

audio signal amplitude

to analog signal: analog

▪ some quality reduction signal

example rates
❖ CD: 1.411 Mbps time

❖ MP3: 96, 128, 160 kbps sampling rate

(N sample/sec)
❖ Internet telephony: 5.3 kbps
and up

Multmedia Networking 7-5

Multimedia: audio

Multmedia Networking 7-6

Multimedia: video
spatial coding example: instead
of sending N values of same
color (all purple), send only two
values: color value (purple) and
number of repeated values (N)
❖ video: sequence of images
displayed at constant rate ……………………...…
……………………...…
▪ e.g. 24 images/sec
❖ digital image: array of pixels
▪ each pixel represented
by bits
❖ coding: use redundancy
within and between images frame i
to decrease # bits used to
encode image
▪ spatial (within image) temporal coding example:
instead of sending
▪ temporal (from one complete frame at i+1,
image to next) send only differences from
frame i

frame i+1
Multmedia Networking 7-7
 Multimedia: video
spatial coding example: instead
of sending N values of same
color (all purple), send only two
values: color value (purple) and

❖ CBR: (constant bit rate): video number of repeated values (N)

encoding rate fixed ……………………...…

……………………...…
❖ VBR: (variable bit rate): video
encoding rate changes as
amount of spatial, temporal
coding changes
❖ examples:
▪ MPEG 1 (CD-ROM) 1.5
Mbps frame i
▪ MPEG2 (DVD) 3-6 Mbps
▪ MPEG4 (often used in temporal coding example:
Internet, < 1 Mbps) instead of sending
complete frame at i+1,
send only differences from
frame i

frame i+1
Multmedia Networking 7-8
Multimedia networking: 3 application types

❖ streaming, stored audio, video

▪ streaming: can begin playout before downloading entire
file
▪ stored (at server): can transmit faster than audio/video
will be rendered (implies storing/buffering at client)
▪ e.g., YouTube, Netflix, Hulu
❖ conversational voice/video over IP
▪ interactive nature of human-to-human conversation
limits delay tolerance
▪ e.g., Skype
❖ streaming live audio, video
▪ e.g., live sporting event (futbol)

Multmedia Networking 7-9

Multimedia networking: outline
7.1 multimedia networking applications
7.2 streaming stored video
7.3 voice-over-IP
7.4 protocols for real-time conversational
applications
7.5 network support for multimedia

Multmedia Networking 7-10

Streaming stored video:

2. video
sent
1. video 3. video received,
recorded network delay played out at client
(e.g., 30 (fixed in this (30 frames/sec) time
frames/sec) example)

streaming: at this time, client

playing out early part of video,
while server still sending later
part of video

Multmedia Networking 7-11

Streaming stored video: challenges
❖ continuous playout constraint: once client playout
begins, playback must match original timing
▪ … but network delays are variable (jitter), so
will need client-side buffer to match playout
requirements
❖ other challenges:
▪ client interactivity: pause, fast-forward,
rewind, jump through video
▪ video packets may be lost, retransmitted

Multmedia Networking 7-12

Streaming stored video: revisted
constant bit
rate video client video constant bit
transmission reception rate video
playout at client
variable
network

buffered
video
delay

client playout time

delay

❖ client-side buffering and playout delay: compensate

for network-added delay, delay jitter

Multmedia Networking 7-13

 Client-side buffering, playout
buffer fill level,
Q(t)
variable fill playout rate,
rate, x(t) e.g., CBR r

client application
video server buffer, size B

client

Multmedia Networking 7-14

 Client-side buffering, playout
buffer fill level,
Q(t)
variable fill playout rate,
rate, x(t) e.g., CBR r

client application
video server buffer, size B

client

1. Initial fill of buffer until playout begins at tp

2. playout begins at tp,
3. buffer fill level varies over time as fill rate x(t) varies
and playout rate r is constant
Multmedia Networking 7-15
 Client-side buffering, playout
buffer fill level,
Q(t)
variable fill playout rate,
rate, x(t) e.g., CBR r

client application
video server buffer, size B

playout buffering: average fill rate (x), playout rate (r):

❖ x < r: buffer eventually empties (causing freezing of video
playout until buffer again fills)
❖ x > r: buffer will not empty, provided initial playout delay is
large enough to absorb variability in x(t)
▪ initial playout delay tradeoff: buffer starvation less likely
with larger delay, but larger delay until user begins
watching
Multmedia Networking 7-16
Streaming multimedia: UDP
❖ server sends at rate appropriate for client
▪ often: send rate = encoding rate = constant
rate
▪ transmission rate can be oblivious to
congestion levels
❖ short playout delay (2-5 seconds) to remove
network jitter
❖ error recovery: application-level, timeipermitting
❖ RTP [RFC 2326]: multimedia payload types
❖ UDP may not go through firewalls

Multmedia Networking 7-17

Streaming multimedia: UDP
❖ Drawbacks:

▪ May fail to provide constant playout due to due to

varying amount of bandwidth. Uses small client side
buffer.

▪ Required media control server, such as RTSP server.

▪ Firewalls are configured to block UDP traffic.

Multmedia Networking 7-18

Streaming multimedia: HTTP
❖ multimedia file retrieved via HTTP GET
❖ send at maximum possible rate under TCP

variable
rate, x(t)

video TCP send TCP receive application

file buffer buffer playout buffer
server client

❖ fill rate fluctuates due to TCP congestion control,

retransmissions (in-order delivery)
❖ larger playout delay: smooth TCP delivery rate
❖ HTTP/TCP passes more easily through firewalls
Multmedia Networking 7-19
Streaming multimedia: HTTP
❖ Prefetching video: TCP results in minimal
starvation and low buffering delay. playback.
❖ Application and TCP buffer: back pressure !
❖ Early termination and reposition: uses byte
range header in HTTP GET message. Video
content may be prefetched but not watched!

Multmedia Networking 7-20

Multimedia networking: outline
7.1 multimedia networking applications
7.2 streaming stored video
7.3 voice-over-IP
7.4 protocols for real-time conversational
applications
7.5 network support for multimedia

Multmedia Networking 7-21

Voice-over-IP (VoIP)
❖ VoIP end-end-delay requirement: needed to maintain
“conversational” aspect
▪ higher delays noticeable, impair interactivity
▪ < 150 msec: good
▪ > 400 msec bad
▪ includes application-level (packetization,playout),
network delays
❖ session initialization: how does callee advertise IP
address, port number, encoding algorithms?
❖ value-added services: call forwarding, screening,
recording
❖ emergency services: 911
Multmedia Networking 7-22
VoIP characteristics
❖ speaker’s audio: alternating talk spurts, silent
periods.
▪ 64 kbps during talk spurt
▪ pkts generated only during talk spurts
▪ 20 msec chunks at 8 Kbytes/sec: 160 bytes of data
❖ application-layer header added to each chunk
❖ chunk+header encapsulated into UDP or TCP
segment
❖ application sends segment into socket every 20
msec during talkspurt

Multmedia Networking 7-23

VoIP Talk spurt

Multmedia Networking 7-24

VoIP: packet loss, delay
❖ network loss: IP datagram lost due to network
congestion (router buffer overflow)
❖ delay loss: IP datagram arrives too late for playout
at receiver
▪ delays: processing, queueing in network; end-system
(sender, receiver) delays
▪ typical maximum tolerable delay: 400 ms
❖ loss tolerance: depending on voice encoding, loss
concealment, packet loss rates between 1% and
10% can be tolerated

Multmedia Networking 7-25

Delay jitter
constant bit
rate client constant bit
transmission reception rate playout
at client
variable
network

buffered
data
delay
(jitter)

client playout time

delay

❖ end-to-end delays of two consecutive packets:

difference can be more or less than 20 msec
(transmission time difference)
Multmedia Networking 7-26
VoIP: fixed playout delay
❖ receiver attempts to playout each chunk exactly q
msecs after chunk was generated.
▪ chunk has time stamp t: play out chunk at t+q
▪ chunk arrives after t+q: data arrives too late
for playout: data “lost”
❖ tradeoff in choosing q:
▪ large q: less packet loss
▪ small q: better interactive experience

Multmedia Networking 7-27

VoIP: fixed playout delay
▪ sender generates packets every 20 msec during talk spurt.
▪ first packet received at time r
▪ first playout schedule: begins at p
▪ second playout schedule: begins at p’
packets

packets loss
generated
packets
playout schedule
received
p' - r

playout schedule
p-r

time

r
Multmedia Networking 5-28
p p'
Adaptive playout delay (1)
❖ goal: low playout delay, low late loss rate
❖ approach: adaptive playout delay adjustment:
▪ estimate network delay, adjust playout delay at
beginning of each talk spurt
▪ silent periods compressed and elongated
▪ chunks still played out every 20 msec during talk spurt
❖ adaptively estimate packet delay: (EWMA -
exponentially weighted moving average, recall TCP RTT
estimate):
di = (1-a)di-1 + a (ri – ti)

delay estimate small constant, time received - time sent

after ith packet e.g. 0.1 (timestamp)
measured delay of ith packet
Multmedia Networking 7-29
Adaptive playout delay (2)
❖ also useful to estimate average deviation of delay, vi :
vi = (1-b)vi-1 + b |ri – ti – di|
❖ estimates di, vi calculated for every received
packet, but used only at start of talk spurt

❖ for first packet in talk spurt, playout time is:

playout-timei = ti + di + Kvi
remaining packets in talkspurt are played out
periodically

Multmedia Networking 5-30

Adaptive playout delay (2)
❖Firstcompute the difference between first
packet layout and sent time:
q i = p i – ti

❖Then compute the playout time for other

packets as an offset from first one: For the jth
packet :
pi = tj + qi

Multmedia Networking 5-31

Adaptive playout delay (3)
Q: How does receiver determine whether packet is
first in a talkspurt?
❖ if no loss, receiver looks at successive timestamps
▪ difference of successive stamps > 20 msec -->talk spurt
begins.
❖ with loss possible, receiver must look at both time
stamps and sequence numbers
▪ difference of successive stamps > 20 msec and sequence
numbers without gaps --> talk spurt begins.

Multmedia Networking 7-32

VoiP: recovery from packet loss (1)
Challenge: recover from packet loss given small
tolerable delay between original transmission and
playout
❖ each ACK/NAK takes ~ one RTT
❖ alternative: Forward Error Correction (FEC)
▪ send enough bits to allow recovery without
retransmission (recall two-dimensional parity in Ch. 5)

simple FEC
❖ for every group of n chunks, create redundant chunk by
exclusive OR-ing n original chunks
❖ send n+1 chunks, increasing bandwidth by factor 1/n
❖ can reconstruct original n chunks if at most one lost chunk
from n+1 chunks, with playout delay
Multmedia Networking 7-33
VoiP: recovery from packet loss (2)
another FEC scheme:
❖ “piggyback lower
quality stream”
❖ send lower resolution
audio stream as
redundant information
❖ e.g., nominal
stream PCM at 64 kbps
and redundant stream
GSM at 13 kbps
❖ non-consecutive loss: receiver can conceal loss
❖ generalization: can also append (n-1)st and (n-2)nd low-bit rate
chunk
Multmedia Networking 7-34
 VoiP: recovery from packet loss (3)

interleaving to conceal loss:

❖ audio chunks divided into ❖ if packet lost, still have most
smaller units, e.g. four 5 of every original chunk
msec units per 20 msec ❖ no redundancy overhead,
audio chunk but increases playout delay
❖ packet contains small units
from different chunks
Multmedia Networking 7-35
Voice-over-IP: Skype
Skype clients (SC)
❖ proprietary application-
layer protocol (inferred
via reverse engineering)
▪ encrypted msgs
Skype
❖ P2P components: login server supernode (SN)
▪ clients: skype peers
connect directly to supernode
overlay
each other for VoIP call network
▪ super nodes (SN):
skype peers with
special functions
▪ overlay network: among
SNs to locate SCs
▪ login server
Application Layer 2-36
P2P voice-over-IP: skype
skype client operation:
1. joins skype network by
contacting SN (IP address
cached) using TCP Skype
2. logs-in (usename, login server

password) to centralized
skype login server
3. obtains IP address for
callee from SN, SN
overlay
▪ or client buddy list
4. initiate call directly to
callee

Application Layer 2-37

Skype: peers as relays
❖ problem: both Alice, Bob
are behind “NATs”
▪ NAT prevents outside peer
from initiating connection to
insider peer
▪ inside peer can initiate
connection to outside
❖ relay solution:Alice, Bob maintain
open connection
to their SNs
▪ Alice signals her SN to connect
to Bob
▪ Alice’s SN connects to Bob’s
SN
▪ Bob’s SN connects to Bob over
open connection Bob initially
initiated to his SN
Application Layer 2-38
Multimedia networking: outline
7.1 multimedia networking applications
7.2 streaming stored video
7.3 voice-over-IP
7.4 protocols for real-time conversational
applications: RTP, SIP
7.5 network support for multimedia

Multmedia Networking 7-39

Real-Time Protocol (RTP)

❖ RTP specifies packet ❖ RTP runs in end

structure for packets systems
carrying audio, video ❖ RTP packets
data encapsulated in UDP
❖ RFC 3550 segments
❖ RTP packet provides ❖ interoperability: if two
▪ payload type VoIP applications run
identification RTP, they may be able
▪ packet sequence to work together
numbering
▪ time stamping
▪ Common formats,
PCM, MP3,etc.
Multmedia Networking 7-40
RTP runs on top of UDP
RTP libraries provide transport-layer interface
that extends UDP:
• port numbers, IP addresses
• payload type identification
• packet sequence numbering
• time-stamping

Multmedia Networking 5-41

RTP example
example: sending 64 kbps ❖ RTP header indicates
PCM-encoded voice over type of audio encoding
RTP in each packet
❖ application collects ▪ sender can change
encoded data in chunks, encoding during
e.g., every 20 msec = conference
160 bytes in a chunk ❖ RTP header also
❖ audio chunk + RTP
contains sequence
header form RTP numbers, timestamps
packet, which is ❖ 12 bytes length
encapsulated in UDP
segment

Multmedia Networking 7-42

RTP and QoS
❖ RTP does not provide any mechanism to ensure
timely data delivery or other QoS guarantees
❖ RTP encapsulation only seen at end systems (not
by intermediate routers)
▪ routers provide best-effort service, making no
special effort to ensure that RTP packets arrive
at destination in timely matter

Multmedia Networking 7-43

RTP header
payload sequence Synchronization Miscellaneous
time stamp
type number Source ID fields
type

payload type (7 bits): indicates type of encoding currently being

used. If sender changes encoding during call, sender
informs receiver via payload type field
Payload type 0: PCM mu-law, 64 kbps
Payload type 3: GSM, 13 kbps
Payload type 7: LPC, 2.4 kbps
Payload type 26: Motion JPEG
Payload type 31: H.261
Payload type 33: MPEG2 video

sequence # (16 bits): increment by one for each RTP packet sent
❖ detect packet loss, restore packet sequence

Multmedia Networking 5-44

RTP header
payload sequence Synchronization Miscellaneous
time stamp
type number Source ID fields
type

❖ timestamp field (32 bits long): sampling instant of first

byte in this RTP data packet
▪ for audio, timestamp clock increments by one for each
sampling period (e.g., each 125 usecs for 8 KHz sampling
clock)
▪ if application generates chunks of 160 encoded samples,
timestamp increases by 160 for each RTP packet when
source is active. Timestamp clock continues to increase
at constant rate when source is inactive.

❖ SSRC field (32 bits long): identifies source of RTP

stream. Each stream in RTP session has distinct SSRC
Multmedia Networking 7-45
SIP: Session Initiation Protocol [RFC 3261]
long-term vision:
❖ Defined in RFC 3261, RFC 5411.
❖ all telephone calls, video conference calls take
place over Internet
❖ people identified by names or e-mail addresses,
rather than by phone numbers
❖ can reach callee (if callee so desires), no matter
where callee roams, no matter what IP device
callee is currently using

Multmedia Networking 7-46

SIP services
❖ SIP provides ❖ determine current IP
mechanisms for call address of callee:
setup: ▪ maps mnemonic
▪ for caller to let identifier to current IP
address
callee know she
wants to establish a ❖ call management:
call ▪ add new media
streams during call
▪ so caller, callee can
▪ change encoding
agree on media type, during call
encoding ▪ invite others
▪ to end call ▪ transfer, hold calls

Multmedia Networking 7-47

 Example: setting up call to known IP address
Bob
Alice

❖ Alice’s SIP invite message

[Link] [Link] indicates her port number, IP
INVITE bob
c=IN IP4 16
@193.64.21
[Link]
0.89 address, encoding she prefers
m=audio 38
060 RTP/AV
4
P0 to receive (PCM mlaw). RTP
port 5060 Bob's
terminal rings used !
200 OK
10.89
c=IN IP4 193.64.2
P/AVP 3
port 5060
m=audio 4875 3 RT
❖ Bob’s 200 OK message
ACK
indicates his port number, IP
port 5060
address, preferred encoding
m Law audio (GSM)
port 38060

❖ SIP messages can be sent

GSM
port 48753
over TCP or UDP; here sent
over RTP/UDP

time time
❖ default SIP port number is
5060 Multmedia Networking 5-48
Setting up a call (more)
❖ codec negotiation: ❖ rejecting a call
▪ suppose Bob doesn’t ▪ Bob can reject with
have PCM mlaw encoder replies “busy,” “gone,”
▪ Bob will instead reply “payment required,”
with 606 Not “forbidden”
Acceptable Reply, listing ❖ media can be sent
his encoders. Alice can over RTP or some
then send new INVITE other protocol
message, advertising
different encoder

Multmedia Networking 7-49

Example of SIP message
INVITE sip:bob@[Link] SIP/2.0
Via: SIP/2.0/UDP [Link] ❖ Here we don’t know
From: sip:alice@[Link] Bob’s IP address
To: sip:bob@[Link] ▪ intermediate SIP
Call-ID: a2e3a@[Link]
Content-Type: application/sdp
servers needed
Content-Length: 885 ❖ Alice sends, receives
SIP messages using SIP
c=IN IP4 [Link] default port 506
m=audio 38060 RTP/AVP 0
❖ Alice specifies in
header that SIP client
Notes:
❖ HTTP message syntax
sends, receives SIP
❖ sdp = session description protocol messages over UDP
❖ Call-ID is unique for every call

Multmedia Networking 7-50

Name translation, user location
❖ caller wants to call ❖ result can be based on:
callee, but only has ▪ time of day (work,
callee’s name or e-mail home)
address. ▪ caller (don’t want boss
to call you at home)
❖ INVITE message is sent
to SIP proxy to ▪ status of callee (calls sent
to voicemail when callee
determine receiver IP is already talking to
❖ need to get IP address of someone)
callee’s current host:
▪ user moves around
▪ DHCP protocol
▪ user has different IP
devices (PC, smartphone,
car device) Multmedia Networking 7-51
SIP registrar
❖ one function of SIP server: registrar
❖ when Bob starts SIP client, client sends SIP REGISTER
message to Bob’s registrar server

register message:
REGISTER sip:[Link] SIP/2.0
Via: SIP/2.0/UDP [Link]
From: sip:bob@[Link]
To: sip:bob@[Link]
Expires: 3600

Multmedia Networking 7-52

SIP proxy
❖ another function of SIP server: proxy
❖ Alice sends invite message to her proxy server
▪ contains address sip:bob@[Link]
▪ proxy responsible for routing SIP messages to callee,
possibly through multiple proxies
❖ Bob sends response back through same set of SIP
proxies
❖ proxy returns Bob’s SIP response message to Alice
▪ contains Bob’s IP address
❖ SIP proxy analogous to local DNS server plus TCP
setup

Multmedia Networking 7-53

SIP example: jim@[Link] calls keith@[Link]
Poly SIP
2. UMass proxy forwards request registrar
to Poly registrar server
2 3 3. Poly server returns redirect response,
indicating that it should try keith@[Link]

UMass 4. Umass proxy forwards request Eurecom SIP

SIP proxy to Eurecom registrar server 4 registrar
7
1. Jim sends INVITE 6-8. SIP response returned to Jim 5. eurecom
message to UMass 8 5 registrar
6 forwards INVITE
SIP proxy. 1
to [Link],
which is running
keith’s SIP
9 client
[Link]
9. Data flows between clients
[Link]
Multmedia Networking 7-54
Comparison with H.323
❖ H.323: another signaling ❖ H.323 comes from the
protocol for real-time, ITU (telephony)
interactive multimedia ❖ SIP comes from IETF:
❖ H.323: complete, borrows much of its
vertically integrated suite concepts from HTTP
of protocols for ▪ SIP has Web flavor;
multimedia conferencing: H.323 has telephony
signaling, registration, flavor
admission control, ❖ SIP uses KISS principle:
transport, codecs Keep It Simple Stupid
❖ SIP: single component.
Works with RTP, but
does not mandate it. Can
be combined with other
protocols, services
Multmedia Networking 7-55