A Guide to MPEG Fundamentals and

Protocol Analysis
Updated with QoS/QoE Diagnostics and Troubleshooting

Primer
Primer

B [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

Table of Contents
Section 1 – Introduction to MPEG.................................. 1 Section 4 – The Standards............................................ 29
1.1 Convergence...........................................................1 4.1 What is MPEG.......................................................29
1.2 Why Compression Is Needed..................................1 4.2 Who are ITU..........................................................29
1.3 Principles of Compression.......................................1 4.3 Evolution of the Standards....................................29
1.4 Compression in Television Applications...................2 4.3.1 Video Performance ...................................30
1.5 Introduction to Digital Video Compression ..............3 4.3.2 Licensing ..................................................31
1.6 Introduction to Audio Compression.........................5 4.4 MPEG-1................................................................31
1.7 MPEG Streams.......................................................6 4.5 MPEG-2................................................................31
1.8 Need for Monitoring and Analysis............................7 4.5.1 Profiles and Levels in MPEG-2 . ...............32
1.9 Pitfalls of Compression............................................7 4.6 MPEG-4................................................................34
4.6.1 MPEG-4 Standards Documents................35
Section 2 – Compression in Video.................................. 8
4.6.2 Object Coding...........................................35
2.1 Spatial or Temporal Coding?...................................8
4.6.3 Video and Audio Coding............................36
2.2 Spatial Coding.........................................................8
4.6.4 Scalability..................................................36
2.3 Weighting..............................................................10
4.6.5 Variable Macroblock Size...........................37
2.4 Scanning...............................................................11
4.6.6 Profiles and Levels in MPEG-4 part 10
2.5 Entropy Coding.....................................................12 (AVC/H.264)..............................................37
2.6 A Spatial Coder.....................................................12 4.6.7 MP4 Container Format..............................38
2.7 Temporal Coding...................................................13 4.7 SMPTE VC-1.........................................................38
2.8 Motion Compensation...........................................14 4.7.1 Variable Block Size Transform....................39
2.9 Bidirectional Coding..............................................16 4.7.2 Motion Compensation...............................39
2.10 I-, P- and B-pictures..............................................16 4.7.3 Deblocking and Smoothing........................39
2.11 An MPEG Compressor..........................................18 4.7.4 Fading Compensation................................40
2.12 Preprocessing.......................................................21 4.7.5 Profiles and Levels.....................................40
2.13 Wavelets...............................................................22 4.8 MPEG-7 and MXF.................................................40
Section 3 – Audio Compression................................... 23 4.9 MPEG-21..............................................................40
3.1 The Hearing Mechanism........................................23 Section 5 – Elementary Streams................................... 41
3.2 Subband Coding...................................................24 5.1 Video Elementary Stream Syntax...........................41
3.3 MPEG Layer 1.......................................................25 5.2 Audio Elementary Streams....................................42
3.4 MPEG Layer 2.......................................................26
3.5 Transform Coding..................................................26 Section 6 – Packetized Elementary Streams (PES)..... 43
3.6 MPEG Layer 3.......................................................27 6.1 PES Packets.........................................................43
3.7 MPEG-2 Audio......................................................27 6.2 Time Stamps.........................................................43
3.8 MPEG-4 Audio......................................................27 6.3 PTS/DTS...............................................................44
3.9 AC-3.....................................................................28

[Link]/video_audio i
Primer

Section 7 – Program Streams....................................... 45 Section 11 – Data Broadcast........................................ 68

7.1 Recording vs. Transmission...................................45 11.1 Applications..........................................................68
7.2 Introduction to Program Streams...........................45 11.1.1 Program Related Data...............................68
11.1.2 Opportunistic Data.....................................68
Section 8 – Transport Streams...................................... 46
11.1.3 Network Data............................................69
8.1 The Job of a Transport Stream..............................46
11.1.4 Enhanced TV.............................................69
8.2 Packets.................................................................47
11.1.5 Interactive TV.............................................69
8.3 Program Clock Reference (PCR)...........................47
11.2 Content Encapsulation..........................................69
8.4 Packet Identification (PID)......................................48
11.2.1 MPEG Data Encapsulation .......................69
8.5 Program Specific Information (PSI)........................48
11.2.2 DVB Data Encapsulation............................73
Section 9 – Digital Modulation...................................... 50 11.2.3 ATSC A/90 Data Encapsulation.................73
9.1 Principles of Modulation........................................50 11.2.4 ARIB Data Encapsulation...........................73
9.2 Analog Modulation................................................50 11.3 Broadcasting Data Content.....................................74
9.3 Quadrature Modulation..........................................50 11.3.1 How Object Carousels Are Broadcast........74
9.4 Simple Digital Modulation Systems........................51 11.3.2 MPEG-2 Data Synchronization .................74
9.5 Phase Shift Keying................................................52 11.3.3 Signaling Data Contents............................75
9.5.1 Turbo 8PSK...............................................52 11.4 Content Presentation............................................75
9.5.2 DVB S2.....................................................52 11.4.1 Set Top Box Middleware............................75
9.6 Quadrature Amplitude Modulation - QAM..............53 11.4.2 History of Open Middleware Standards......76
9.7 Vestigial Sideband Modulation – VSB....................54 11.4.3 The DVB Multimedia Home Platform
9.8 Coded Orthogonal Frequency Division (MHP) . .....................................................77
Multiplex – COFDM...............................................55 11.4.4 ATSC Advanced Common Application
9.9 China Digital Multimedia Broadcast- Platform (ACAP)........................................77
Terrestrial/Handheld - DMB-T/H............................56
Section 12 – MPEG QoS Testing.................................. 78
9.10 Integrated Services Data Broadcasting (ISDB).......56
12.1 Testing Requirements - QoS..................................78
9.10.1 ISDB-S Satellite System.............................56
12.2 Analyzing a Transport Stream................................78
9.10.2 ISDB-C Cable System...............................56
12.3 Hierarchic View.....................................................79
9.10.3 ISDB-T Terrestrial Modulation.....................56
12.4 Standards Compliance Checking..........................81
9.10.4 ISDB-Tss Mobile Modulation......................57
12.4.1 DVB Measurement Guidelines,
9.10.5 ISDB in Summary......................................57 TR 101 290...............................................81
9.11 Mobile Broadcast Systems....................................57 12.4.2 ATSC Transport Stream Verification,
9.11.1 DVB-H.......................................................58 A/78..........................................................82
9.11.2 T-DMB.......................................................58 12.5 Packet View..........................................................83
9.11.3 DAB IP......................................................58 12.6 Syntax and CRC Analysis......................................83
9.11.4 FLO..........................................................60 12.7 Timing Analysis.....................................................84
9.11.5 ISDB Single Segment................................60 12.8 Elementary Stream Testing....................................86
12.9 Elementary Stream Analysis..................................86
Section 10 – Introduction to DVB & ATSC.................... 61
12.10 Audio Visual Synchronization.................................88
10.1 An Overall View.....................................................61
12.11 Creating a Transport Stream..................................88
10.2 Remultiplexing.......................................................62
12.12 PCR Inaccuracy Generation..................................89
10.3 Service Information (SI)..........................................62
12.13 Data Broadcast Carousel Analysis.........................90
10.4 Error Correction.....................................................63
10.4.1 Turbo Coding............................................64 Section 13 – MPEG QoE Testing................................... 92
10.5 Channel Coding; Energy Dispersion......................65 13.1 Testing Requirements - QoE..................................92
10.6 Inner Coding.........................................................66 13.2 How are QoS and QoE interrelated? ....................94
10.7 Transmitting Digits.................................................67 13.3 Using QoS and QoE to identify “real” problems.....95
13.4 Summary..............................................................99

Glossary......................................................................... 99

References................................................................... 103
ii [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

Section 1 – Introduction to MPEG 1.2 Why Compression Is Needed

MPEG is one of the most popular audio/video compression The initial success of digital video was in post-production
techniques because it is not just a single standard. Instead, applications, where the high cost of digital video was offset
it is a range of standards suitable for different applications by its limitless layering and effects capability. However,
but based on similar principles. MPEG is an acronym for the production-standard digital video generates over 200
Moving Picture Experts Group, part of the Joint Technical megabits per second of data, and this bit rate requires
Committee, JTC1, established by the ISO (International extensive capacity for storage and wide bandwidth for
Standards Organization) and IEC (International Electrotechnical transmission. Digital video could only be used in wider
Commission). JTC1 is responsible for Information Technology; applications if the storage and bandwidth requirements could
within JTC1, Sub Group SG29 is responsible for “Coding of be eased; this is the purpose of compression.
Audio, Picture, and Multimedia and Hypermedia Information.” Compression is a way of expressing digital audio and video by
There are a number of working groups within SG29, including using less data. Compression has the following advantages:
JPEG (Joint Photographic Experts Group), and Working Group
A smaller amount of storage is needed for a given amount
11 for compression of moving pictures. ISO/IEC JTC1/SG29/
of source material.
WG11 is MPEG.
When working in real time, compression reduces the
MPEG can be described as the interaction of acronyms. As
bandwidth needed.
ETSI stated, “The CAT is a pointer to enable the IRD to find
Additionally, compression allows faster-than-real-time
the EMMs associated with the CA system(s) that it uses.” If
transfer between media, for example, between tape and
you can understand that sentence you don’t need this book!
disk.
1.1 Convergence A compressed recording format can use a lower recording
Digital techniques have made rapid progress in audio and density and this can make the recorder less sensitive to
video for a number of reasons. Digital information is more environmental factors and maintenance.
robust and can be coded to substantially eliminate error.
This means that generation-losses in recording and losses in
1.3 Principles of Compression
transmission may be eliminated. The compact disc (CD) was There are two fundamentally different techniques that may
the first consumer product to demonstrate this. be used to reduce the quantity of data used to convey
information content. In practical compression systems, these
While the CD has an improved sound quality with respect to
are usually combined, often in very complex ways.
its vinyl predecessor, comparison of quality alone misses the
point. The real point is that digital recording and transmission The first technique is to improve coding efficiency. There are
techniques allow content manipulation to a degree that is many ways of coding any given information, and most simple
impossible with analog. Once audio or video is digitized, the data representations of video and audio contain a substantial
content is in the form of data. Such data can be handled in the amount of redundancy. The concept of entropy is discussed
same way as any other kind of data; therefore, digital video below.
and audio become the province of computer technology.
Many coding tricks can be used to reduce or eliminate
The convergence of computers and audio/video is an redundancy; examples include run-length coding and variable-
inevitable consequence of the key inventions of computing length coding systems such as Huffman codes. When
and pulse code modulation (PCM). Digital media can store any properly used, these techniques are completely reversible so
type of information, so it is easy to use a computer storage that after decompression the data is identical to that at the
device for digital video. The nonlinear workstation was the first input of the system. This type of compression is known as
example of an application of convergent technology that did lossless. Archiving computer programs such as PKZip employ
not have an analog forerunner. Another example, multimedia, lossless compression.
combines the storage of audio, video, graphics, text and data
on the same medium. Multimedia has no equivalent in the
analog domain.

[Link]/video_audio 1
Primer

Obviously, lossless compression is ideal, but unfortunately it The first part of the solution was to transform the signals
does not usually provide the degree of data reduction needed from GBR to a brightness signal (normally designated Y) plus
for video and audio applications. However, because it is two color difference signals, U and V, or I and Q. Generation
lossless, it can be applied at any point in the system and is of a brightness signal went a long way towards solving the
often used on the data output of lossy compressors. problem of compatibility with monochrome receivers, but the
important step for bandwidth minimization came from the
If the elimination of redundancy does not reduce the data as
color difference signals.
much as needed, some information will have to be discarded.
Lossy compression systems achieve data reduction by It turns out that the human visual system uses sensors that
removing information that is irrelevant, or of lesser relevance. are sensitive to brightness, and that can “see” a very high-
These are not general techniques that can be applied to any resolution image. Other sensors capture color information,
data stream; the assessment of relevance can only be made but at much lower resolution. The net result is that, within
in the context of the application, understanding what the data certain limits, a sharp monochrome image representing
represents and how it will be used. In the case of television, scene brightness overlaid with fuzzy (low-bandwidth) color
the application is the presentation of images and sound to the information will appear as a sharp color picture. It is not
human visual and hearing systems, and the human factors possible to take advantage of this when dealing with GBR
must be well understood to design an effective compression signals, as each signal contains both brightness and color
system. information. However, in YUV space, most of the brightness
information is carried in the Y signal, and very little in the
Some information in video signals cannot be perceived by the
color difference signals. So, it is possible to filter the color
human visual system and is, therefore, truly irrelevant in this
difference signals and drastically reduce the information to be
context. A compression system that discards only irrelevant
transmitted.
image information is known as visually lossless.
This is an example of eliminating (mostly) irrelevant information.
1.4 Compression in Television Applications Under the design viewing conditions, the human visual
Television signals, analog or digital, have always represented a system does not respond significantly to the high frequency
great deal of information, and bandwidth reduction techniques information in the color difference signals, so it may be
have been used from a very early stage. Probably the earliest discarded. NTSC television transmissions carry only about
example is interlace. For a given number of lines, and a 500 kHz in each color difference signal, but the pictures are
given rate of picture refresh, interlace offers a 2:1 reduction adequately sharp for many applications.
in bandwidth requirement. The process is lossy; interlace The final step in the bandwidth reduction process of NTSC
generates artifacts caused by interference between vertical and PAL was to “hide” the color difference signals in unused
and temporal information, and reduces the usable vertical parts of the spectrum of the monochrome signal. Although
resolution of the image. Nevertheless, most of what is given up the process is not strictly lossless, this can be thought of as
is largely irrelevant, so interlace represented a simple and very increasing the coding efficiency of the signal.
valuable trade-off in its time. Unfortunately interlace and the
Some of the techniques in the digital world are quite different,
artifacts it generates are very disruptive to more sophisticated
but similar principles apply. For example, MPEG transforms
digital compression schemes. Much of the complexity of
signals into a different domain to permit the isolation of
MPEG-2 results from the need to handle interlaced signals,
irrelevant information. The transform to color-difference space
and there is still a significant loss in coding efficiency when
is still employed, but digital techniques permit filtering of the
compared to progressive signals.
color difference signal to reduce vertical resolution for further
The next major steps came with the advent of color. Color savings.
cameras produce GBR signals, so nominally there is three
times the information of a monochrome signal – but there was
a requirement to transmit color signals in the same channels
used for monochrome.

2 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

Figure 1-1. Component video translations.

Figure 1-1a shows that in traditional television systems, the 1.5 Introduction to Digital Video Compression
GBR camera signal is converted to Y, Pb, Pr components
In all real program material, there are two types of components
for production and encoded into analog composite for
of the signal: those that are novel and unpredictable and
transmission. Figure 1-1b shows the modern equivalent. The
those that can be anticipated. The novel component is called
Y, Pb, Pr signals are digitized and carried as Y, Cb, Cr signals
entropy and is the true information in the signal. The remainder
in SDI form through the production process prior to being
is called redundancy because it is not essential. Redundancy
encoded with MPEG for transmission. Clearly, MPEG can be
may be spatial, as it is in large plain areas of picture where
considered by the broadcaster as a more efficient replacement
adjacent pixels have almost the same value. Redundancy can
for composite video. In addition, MPEG has greater flexibility
also be temporal as it is where similarities between successive
because the bit rate required can be adjusted to suit the
pictures are used. All compression systems work by
application. At lower bit rates and resolutions, MPEG can be
separating entropy from redundancy in the encoder. Only the
used for video conferencing and video telephones.
entropy is recorded or transmitted and the decoder computes
Digital Video Broadcasting (DVB) and Advanced Television the redundancy from the transmitted signal. Figure 1-2a (see
Systems Committee (ATSC) (the European- and American- next page) shows this concept.
originated digital-television broadcasting standards) would
An ideal encoder would extract all the entropy and only this
not be viable without compression because the bandwidth
will be transmitted to the decoder. An ideal decoder would
required would be too great. Compression extends the playing
then reproduce the original signal. In practice, this ideal cannot
time of DVD (digital video/versatile disk) allowing full-length
be reached. An ideal coder would be complex and cause a
movies on a single disk. Compression also reduces the cost
very long delay in order to use temporal redundancy. In certain
of ENG and other contributions to television production. DVB,
applications, such as recording or broadcasting, some delay is
ATSC and digital video disc (DVD) are all based on MPEG-2
acceptable, but in videoconferencing it is not. In some cases,
compression.
a very complex coder would be too expensive. It follows that
In tape recording, mild compression eases tolerances and there is no one ideal compression system.
adds reliability in Digital Betacam and Digital-S, whereas in
SX, DVC, DVCPRO and DVCAM, the goal is miniaturization. In
disk-based video servers, compression lowers storage cost.
Compression also lowers bandwidth, which allows more users
to access a given server. This characteristic is also important
for VOD (video on demand) applications.

[Link]/video_audio 3
Primer

Figure 1-2. Trade-offs between quality, complexity, and latency.

In practice, a range of coders is needed which have a range Unfortunately, the number of variables involved makes it
of processing delays and complexities. The power of MPEG very difficult to determine a suitable compression factor.
is that it is not a single compression format, but a range of Figure 1-2a shows that for an ideal coder, if all of the entropy
standardized coding tools that can be combined flexibly to suit is sent, the quality is good. However, if the compression
a range of applications. The way in which coding has been factor is increased in order to reduce the bit rate, not all of the
performed is included in the compressed data so that the entropy is sent and the quality falls. Note that in a compressed
decoder can automatically handle whatever the coder decided system when the quality loss occurs, it is steep (Figure 1-2b).
to do. If the available bit rate is inadequate, it is better to avoid this
area by reducing the entropy of the input picture. This can be
In MPEG-2 and MPEG-4 coding is divided into several
done by filtering. The loss of resolution caused by the filtering
profiles that have different complexity, and each profile can be
is subjectively more acceptable than the compression artifacts.
implemented at a different level depending on the resolution
of the input picture. Section 4 considers profiles and levels in To identify the entropy perfectly, an ideal compressor would
detail. have to be extremely complex. A practical compressor may be
less complex for economic reasons and must send more data
There are many different digital video formats and each has a
to be sure of carrying all of the entropy. Figure 1-2b shows the
different bit rate. For example a high definition system might
relationship between coder complexity and performance. The
have six times the bit rate of a standard definition system.
higher the compression factor required, the more complex the
Consequently, just knowing the bit rate out of the coder is not
encoder has to be.
very useful. What matters is the compression factor, which is
the ratio of the input bit rate to the compressed bit rate, for
example 2:1, 5:1 and so on.

4 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

The entropy in video signals varies. A recording of an MPEG can handle both interlaced and non-interlaced images.
announcer delivering the news has much redundancy and is An image at some point on the time axis is called a “picture,”
easy to compress. In contrast, it is more difficult to compress whether it is a field or a frame. Interlace is not ideal as a source
a recording with leaves blowing in the wind or one of a for digital compression because it is in itself a compression
football crowd that is constantly moving and therefore has technique. Temporal coding is made more complex because
less redundancy (more information or entropy). In either case, pixels in one field are in a different position to those in the next.
if all the entropy is not sent, there will be quality loss. Thus,
Motion compensation minimizes but does not eliminate
we may choose between a constant bit-rate channel with
the differences between successive pictures. The picture
variable quality or a constant quality channel with variable bit
difference is itself a spatial image and can be compressed
rate. Telecommunications network operators tend to prefer a
using transform-based intra-coding as previously described.
constant bit rate for practical purposes, but a buffer memory
Motion compensation simply reduces the amount of data in
can be used to average out entropy variations if the resulting
the difference image.
increase in delay is acceptable. In recording, a variable bit
rate may be easier to handle and DVD uses variable bit rate, The efficiency of a temporal coder rises with the time span
using buffering so that the average bit rate remains within the over which it can act. Figure 1-2c shows that if a high
capabilities of the disk system. compression factor is required, a longer time span in the input
must be considered and thus a longer coding delay will be
Intra-coding (intra = within) is a technique that exploits
experienced. Clearly, temporally coded signals are difficult to
spatial redundancy, or redundancy within the picture; inter-
edit because the content of a given output picture may be
coding (inter = between) is a technique that exploits temporal
based on image data which was transmitted some time earlier.
redundancy. Intra-coding may be used alone, as in the JPEG
Production systems will have to limit the degree of temporal
standard for still pictures, or combined with inter-coding as in
coding to allow editing and this limitation will in turn limit the
MPEG.
available compression factor.
Intra-coding relies on two characteristics of typical images.
First, not all spatial frequencies are simultaneously present, 1.6 Introduction to Audio Compression
and second, the higher the spatial frequency, the lower the The bit rate of a PCM digital audio channel is only about 1.5
amplitude is likely to be. Intra-coding requires analysis of the megabits per second, which is about 0.5% of 4:2:2 digital
spatial frequencies in an image. This analysis is the purpose video. With mild video compression schemes, such as Digital
of transforms such as wavelets and DCT (discrete cosine Betacam, audio compression is unnecessary. But, as the
transform). Transforms produce coefficients that describe video compression factor is raised, it becomes important to
the magnitude of each spatial frequency. Typically, many compress the audio as well.
coefficients will be zero, or nearly zero, and these coefficients
Audio compression takes advantage of two facts. First, in
can be omitted, resulting in a reduction in bit rate.
typical audio signals, not all frequencies are simultaneously
Inter-coding relies on finding similarities between successive present. Second, because of the phenomenon of masking,
pictures. If a given picture is available at the decoder, the next human hearing cannot discern every detail of an audio signal.
picture can be created by sending only the picture differences. Audio compression splits the audio spectrum into bands by
The picture differences will be increased when objects filtering or transforms, and includes less data when describing
move, but this magnification can be offset by using motion bands in which the level is low. Where masking prevents or
compensation, since a moving object does not generally reduces audibility of a particular band, even less data needs to
change its appearance very much from one picture to the be sent.
next. If the motion can be measured, a closer approximation
to the current picture can be created by shifting part of the
previous picture to a new location. The shifting process is
controlled by a pair of horizontal and vertical displacement
values (collectively known as the motion vector) that is
transmitted to the decoder. The motion vector transmission
requires less data than sending the picture-difference data.

[Link]/video_audio 5
Primer

Figure 1-3. Encoding and encapsulation steps in Program and Transport Stream creation.

Audio compression is not as easy to achieve as video For transmission and digital broadcasting, several programs
compression because of the acuity of hearing. Masking only and their associated PES can be multiplexed into a single
works properly when the masking and the masked sounds transport stream. A transport stream differs from a program
coincide spatially. Spatial coincidence is always the case in stream in that the PES packets are further subdivided into
mono recordings but not in stereo recordings, where low- short fixed-size packets and in that multiple programs
level signals can still be heard if they are in a different part encoded with different clocks can be carried. This is possible
of the sound stage. Consequently, in stereo and surround because a transport stream has a program clock reference
sound systems, a lower compression factor is allowable for a (PCR) mechanism that allows transmission of multiple clocks,
given quality. Another factor complicating audio compression one of which is selected and regenerated at the decoder. A
is that delayed resonances in poor loudspeakers actually single program transport stream (SPTS) is also possible and
mask compression artifacts. Testing a compressor with poor this may be found between a coder and a multiplexer. Since
speakers gives a false result, and signals that are apparently a transport stream can genlock the decoder clock to the
satisfactory may be disappointing when heard on good encoder clock, the SPTS is more common than the Program
equipment. Stream.

1.7 MPEG Streams A transport stream is more than just a multiplex of audio
and video PES. In addition to the compressed audio, video
The output of a single MPEG audio or video coder is called and data, a transport stream includes metadata describing
an elementary stream. An elementary stream is an endless the bit stream. This includes the program association table
near real-time signal. For convenience, the elementary stream (PAT) that lists every program in the transport stream. Each
may be broken into data blocks of manageable size, forming a entry in the PAT points to a program map table (PMT) that
packetized elementary stream (PES). These data blocks need lists the elementary streams making up each program. Some
header information to identify the start of the packets and programs will be open, but some programs may be subject
must include time stamps because packetizing disrupts the to conditional access (encryption) and this information is also
time axis. carried in the metadata.
Figure 1-3 shows that one video PES and a number of audio The transport stream consists of fixed-size data packets,
PES can be combined to form a program stream, provided each containing 188 bytes. Each packet carries a program
that all of the coders are locked to a common clock. Time identifier code (PID). Packets in the same elementary stream
stamps in each PES can be used to ensure lip-sync between all have the same PID, so that the decoder (or a demultiplexer)
the video and audio. Program streams have variable-length can select the elementary stream(s) it wants and reject the
packets with headers. They find use in data transfers to and remainder. Packet continuity counts ensure that every packet
from optical and hard disks, which are essentially error free, that is needed to decode a stream is received. An effective
and in which files of arbitrary sizes are expected. DVD uses synchronization system is needed so that decoders can
program streams. correctly identify the beginning of each packet and deserialize
the bit stream into words.

6 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

1.8 Need for Monitoring and Analysis 1.9 Pitfalls of Compression

The MPEG transport stream is an extremely complex structure MPEG compression is lossy in that what is decoded is not
using interlinked tables and coded identifiers to separate the identical to the original. The entropy of the source varies, and
programs and the elementary streams within the programs. when entropy is high, the compression system may leave
Within each elementary stream, there is a complex structure, visible artifacts when decoded. In temporal compression,
allowing a decoder to distinguish between, for example, redundancy between successive pictures is assumed. When
vectors, coefficients and quantization tables. this is not the case, the system may fail. An example is video
from a press conference where flashguns are firing. Individual
Failures can be divided into two broad categories. In the first
pictures containing the flash are totally different from their
category, the transport system correctly delivers information
neighbors, and coding artifacts may become obvious.
from an encoder/multiplexer to a decoder with no bit errors or
added jitter, but the encoder/multiplexer or the decoder has Irregular motion or several independently moving objects on
a fault. In the second category, the encoder/multiplexer and screen require a lot of vector bandwidth and this requirement
decoder are fine, but the transport of data from one to the may only be met by reducing the bandwidth available for
other is defective. It is very important to know whether the fault picture-data. Again, visible artifacts may occur whose level
lies in the encoder/ multiplexer, the transport or the decoder if varies and depends on the motion. This problem often occurs
a prompt solution is to be found. in sports-coverage video.
Synchronizing problems, such as loss or corruption of sync Coarse quantizing results in luminance contouring and
patterns, may prevent reception of the entire transport stream. posterized color. These can be seen as blotchy shadows
Transport stream protocol defects may prevent the decoder and blocking on large areas of plain color. Subjectively,
from finding all of the data for a program, perhaps delivering compression artifacts are more annoying than the relatively
picture but not sound. Correct delivery of the data but with constant impairments resulting from analog television
excessive jitter can cause decoder timing problems. transmission systems.
If a system using an MPEG transport stream fails, the fault The only solution to these problems is to reduce the
could be in the encoder, the multiplexer or in the decoder. compression factor. Consequently, the compression user has
How can this fault be isolated? First, verify that a transport to make a value judgment between the economy of a high
stream is compliant with the MPEG-coding standards. If the compression factor and the level of artifacts.
stream is not compliant, a decoder can hardly be blamed for
In addition to extending the encoding and decoding delay,
having difficulty. If the stream is compliant, the decoder may
temporal coding also causes difficulty in editing. In fact, an
need attention.
MPEG bit stream cannot be arbitrarily edited. This restriction
Traditional video testing tools, the signal generator, the occurs because, in temporal coding, the decoding of one
waveform monitor and vectorscope, are not appropriate in picture may require the contents of an earlier picture and
analyzing MPEG systems, except to ensure that the video the contents may not be available following an edit. The fact
signals entering and leaving an MPEG system are of suitable that pictures may be sent out of sequence also complicates
quality. Instead, a reliable source of valid MPEG test signals is editing.
essential for testing receiving equipment and decoders. With a
If suitable coding has been used, edits can take place, but
suitable analyzer, the performance of encoders, transmission
only at splice points that are relatively widely spaced. If
systems, multiplexers and remultiplexers can be assessed
arbitrary editing is required, the MPEG stream must undergo a
with a high degree of confidence. As a long standing supplier
decode-modify-recode process, which will result in generation
of high grade test equipment to the video industry, Tektronix
loss.
continues to provide test and measurement solutions as the
technology evolves, giving the MPEG user the confidence that
complex compressed systems are correctly functioning and
allowing rapid diagnosis when they are not.

[Link]/video_audio 7
Primer

Figure 2-1. Related and unrelated waveforms.

Section 2 – Compression in Video 2.2 Spatial Coding

This section shows how video compression is based on the The first step in spatial coding is to perform an analysis of
perception of the eye. Important enabling techniques, such as spatial frequencies using a transform. A transform is simply
transforms and motion compensation, are considered as an a way of expressing a waveform in a different domain, in
introduction to the structure of an MPEG coder. this case, the frequency domain. The output of a transform
is a set of coefficients that describe how much of a given
2.1 Spatial or Temporal Coding? frequency is present. An inverse transform reproduces the
As was seen in Section 1, video compression can take original waveform. If the coefficients are handled with sufficient
advantage of both spatial and temporal redundancy. In MPEG, accuracy, the output of the inverse transform is identical to the
temporal redundancy is reduced first by using similarities original waveform.
between successive pictures. As much as possible of the The most well known transform is the Fourier transform. This
current picture is created or “predicted” by using information transform finds each frequency in the input signal. It finds each
from pictures already sent. When this technique is used, it is frequency by multiplying the input waveform by a sample of a
only necessary to send a difference picture, which eliminates target frequency, called a basis function, and integrating the
the differences between the actual picture and the prediction. product. Figure 2-1 shows that when the input waveform does
The difference picture is then subject to spatial compression. not contain the target frequency, the integral will be zero, but
As a practical matter it is easier to explain spatial compression when it does, the integral will be a coefficient describing the
prior to explaining temporal compression. amplitude of that component frequency.
Spatial compression relies on similarities between adjacent The results will be as described if the frequency component
pixels in plain areas of picture and on dominant spatial is in phase with the basis function. However if the frequency
frequencies in areas of patterning. The JPEG system uses component is in quadrature with the basis function, the
spatial compression only, since it is designed to transmit integral will still be zero. Therefore, it is necessary to perform
individual still pictures. However, JPEG may be used to code two searches for each frequency, with the basis functions in
a succession of individual pictures for video. In the so-called quadrature with one another so that every phase of the input
“Motion JPEG” application, the compression factor will not be will be detected.
as good as if temporal coding was used, but the bit stream
will be freely editable on a picture-by-picture basis.

8 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

Figure 2-2. Relationship of sine and cosine components.

The Fourier transform has the disadvantage of requiring

coefficients for both sine and cosine components of each
frequency. In the cosine transform, the input waveform is
time-mirrored with itself prior to multiplication by the basis
functions. Figure 2-2 shows that this mirroring cancels out all
sine components and doubles all of the cosine components.
The sine basis function is unnecessary and only one coefficient
is needed for each frequency.
The discrete cosine transform (DCT) is the sampled
version of the cosine transform and is used extensively in
two-dimensional form in MPEG. A block of 8x8 pixels is Figure 2-3. 64 unique coefficients of a DCT.
transformed to become a block of 8x8 coefficients. Since
the transform requires multiplication by fractions, there is
wordlength extension, resulting in coefficients that have longer Figure 2-3 also shows 8 coefficients as one-dimensional
wordlength than the pixel values. Typically an 8-bit pixel block horizontal waveforms. Combining these waveforms with
results in an 11-bit coefficient block. Thus, a DCT does not various amplitudes and either polarity can reproduce any
result in any compression; in fact it results in the opposite. combination of 8 pixels. Thus combining the 64 coefficients of
However, the DCT converts the source pixels into a form the 2-D DCT will result in the original 8x8 pixel block. Clearly
where compression is easier. for color pictures, the color difference samples will also need
Figure 2-3 shows the results of an inverse transform of each to be handled. Y, Cb and Cr data are assembled into separate
of the individual coefficients of an 8x8 DCT. In the case of 8x8 arrays and are transformed individually.
the luminance signal, the top-left coefficient is the average In much real program material, many of the coefficients will
brightness or DC component of the whole block. Moving have zero or near-zero values and, therefore, will not be
across the top row, horizontal spatial frequency increases. transmitted. This fact results in significant compression that
Moving down the left column, vertical spatial frequency is virtually lossless. If a higher compression factor is needed,
increases. In real pictures, different vertical and horizontal then the wordlength of the non-zero coefficients must
spatial frequencies may occur simultaneously and a coefficient be reduced. This reduction will reduce accuracy of these
at some point within the block will represent all possible coefficients and will introduce losses into the process. With
horizontal and vertical combinations. care, the losses can be introduced in a way that is least visible
to the viewer.

[Link]/video_audio 9
Primer

2.3 Weighting
Figure 2-4 shows that the human perception of noise
in pictures is not uniform but is a function of the spatial
frequency. More noise can be tolerated at high spatial
frequencies. Also, video noise is effectively masked by fine
detail in the picture, whereas in plain areas it is highly visible.
The reader will be aware that traditional noise measurements
are frequently weighted so that technical measurements relate
more closely to the subjective result.
Compression reduces the accuracy of coefficients and
has a similar effect to using shorter wordlength samples
in PCM; that is, the noise level rises. In PCM, the result of
shortening the word-length is that the noise level rises equally
at all frequencies. As the DCT splits the signal into different Figure 2-4. Human perception of noise to spatial frequency.

frequencies, it becomes possible to control the spectrum of

the noise. Effectively, low-frequency coefficients are rendered
In the decoder, low-order zeros will be added to return the
more accurately than high-frequency coefficients by a process
weighted coefficients to their correct magnitude. They will
of weighting.
then be multiplied by inverse weighting factors. Clearly, at
Figure 2-5 shows that, in the weighting process, the high frequencies the multiplication factors will be larger, so the
coefficients from the DCT are divided by constants that are requantizing noise will be greater. Following inverse weighting,
a function of two-dimensional frequency. Low-frequency the coefficients will have their original DCT output values, plus
coefficients will be divided by small numbers, and high- requantizing error, which will be greater at high frequency than
frequency coefficients will be divided by large numbers. at low frequency.
Following the division, the result is truncated to the nearest
As an alternative to truncation, weighted coefficients may
integer. This truncation is a form of requantizing. In the
be nonlinearly requantized so that the quantizing step size
absence of weighting, this requantizing would have the effect
increases with the magnitude of the coefficient. This technique
of uniformly increasing the size of the quantizing step, but with
allows higher compression factors but worse levels of artifacts.
weighting, it increases the step size according to the division
factor. Clearly, the degree of compression obtained and, in turn,
the output bit rate obtained, is a function of the severity
As a result, coefficients representing low spatial frequencies
of the requantizing process. Different bit rates will require
are requantized with relatively small steps and suffer little
different weighting tables. In MPEG, it is possible to use
increased noise. Coefficients representing higher spatial
various different weighting tables and the table in use can
frequencies are requantized with large steps and suffer more
be transmitted to the decoder, so that correct decoding is
noise. However, fewer steps means that fewer bits are needed
ensured.
to identify the step and compression is obtained.

10 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

Figure 2-5. DCT coefficients are weighted by Quant Matrix and Quant Scale.

2.4 Scanning In Figure 2-6b, an alternative scan pattern is shown that

may be used for interlaced sources. In an interlaced picture,
In typical program material, the most significant DCT
an 8x8 DCT block from one field extends over twice the
coefficients are generally found in or near the top-left corner
vertical screen area, so that for a given picture detail, vertical
of the matrix. After weighting, low-value coefficients might be
frequencies will appear to be twice as great as horizontal
truncated to zero. More efficient transmission can be obtained
frequencies. Thus, the ideal scan for an interlaced picture will
if all of the non-zero coefficients are sent first, followed by a
be on a diagonal that is twice as steep. Figure 2-6b shows
code indicating that the remainder are all zero. Scanning is
that a given vertical spatial frequency is scanned before
a technique that increases the probability of achieving this
scanning the same horizontal spatial frequency.
result, because it sends coefficients in descending order of
magnitude probability. Figure 2-6a (see next page) shows
that in a non-interlaced system, the probability of a coefficient
having a high value is highest in the top-left corner and lowest
in the bottom-right corner. A 45 degree diagonal zigzag scan
is the best sequence to use here.

[Link]/video_audio 11
Primer

Figure 2-6. Two methods for serializing the DCT table. The alternate method is often used with interlaced video.

Figure 2-7. Rate Control and Quantization determine the output rate and quality.

2.5 Entropy Coding 2.6 A Spatial Coder

In real video, not all spatial frequencies are present Figure 2-7 ties together all of the preceding spatial coding
simultaneously; therefore, the DCT coefficient matrix will have concepts. The input signal is assumed to be 4:2:2 SDI (Serial
zero terms in it. Requantization will increase the number of Digital Interface), which may have 8- or 10-bit wordlength.
zeros by eliminating small values. Despite the use of scanning, MPEG uses only 8-bit resolution; therefore, a rounding stage
zero coefficients will still appear between the significant values. will be needed when the SDI signal contains 10-bit words.
Run length coding (RLC) allows these coefficients to be Most MPEG profiles operate with 4:2:0 sampling; therefore, a
handled more efficiently. Where repeating values, such as a vertical low-
string of zeros, are present, RLC simply transmits the number
pass filter/interpolation stage will be needed. Rounding and
of zeros rather than each individual bit.
color subsampling introduces a small irreversible loss of
The probability of occurrence of particular coefficient values in information and a proportional reduction in bit rate. The raster
real video can be studied. In practice, some values occur very scanned input format will need to be stored so that it can be
often; others occur less often. This statistical information can converted to 8x8 pixel blocks.
be used to achieve further compression using variable length
coding (VLC). Frequently occurring values are converted to
short code words, and infrequent values are converted to long
code words. To aid decoding, no code word can be the prefix
of another.

12 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

Figure 2-8. Luma/Chroma sample tables.

Figure 2-9. Picture differencing.

The DCT stage transforms the picture information to the scan, and inverse weighting is applied to recreate the block
frequency domain. The DCT itself does not achieve any of DCT coefficients. Following an inverse transform, the 8x8
compression. Following DCT, the coefficients are weighted pixel block is recreated. To obtain a raster-scanned output,
and truncated, providing the first significant compression. The the blocks are stored in RAM, which is read a line at a time. To
coefficients are then zigzag scanned to increase the probability obtain a 4:2:2 output from 4:2:0 data, a vertical interpolation
that the significant coefficients occur early in the scan. After process will be needed as shown in Figure 2-8.
the last non-zero coefficient, an EOB (end of block) code is
The chroma samples in 4:2:0 are positioned half way between
generated.
luminance samples in the vertical axis so that they are evenly
Coefficient data are further compressed by run-length and spaced when an interlaced source is used.
variable-length coding. In a variable bit-rate system, the
quantizing may be fixed, but in a fixed bit-rate system, a 2.7 Temporal Coding
buffer memory is used to absorb variations in coding difficulty. Temporal redundancy can be exploited by inter-coding or
Highly detailed pictures will tend to fill the buffer, whereas transmitting only the differences between pictures. Figure 2-9
plain pictures will allow it to empty. If the buffer is in danger of shows that a one-picture delay combined with a subtracter
overflowing, the requantizing steps will have to be made larger, can compute the picture differences. The picture difference is
so that the compression factor is raised. an image in its own right and can be further compressed by
In the decoder, the bit stream is deserialized and the entropy the spatial coder as was previously described. The decoder
coding is reversed to reproduce the weighted coefficients. The reverses the spatial coding and adds the difference picture to
coefficients are placed in the matrix according to the zigzag the previous picture to obtain the next picture.

[Link]/video_audio 13
Primer

Figure 2-10. Bandwidth reduction achieved by sending Difference-frames between Intra-frames.

Figure 2-11. Picture N+1 created by sending Motion Vectors rather than DCT values.

There are some disadvantages to this simple system. First, as 2.8 Motion Compensation
only differences are sent, it is impossible to begin decoding
Motion reduces the similarities between pictures and increases
after the start of the transmission. This limitation makes it
the data needed to create the difference picture. Motion
difficult for a decoder to provide pictures following a switch
compensation is used to increase the similarity. Figure 2-11
from one bit stream to another (as occurs when the viewer
shows the principle. When an object moves across the TV
changes channels). Second, if any part of the difference data
screen, it may appear in a different place in each picture, but
is incorrect, the error in the picture will propagate indefinitely.
it does not change in appearance very much. The picture
The solution to these problems is to use a system that is not difference can be reduced by measuring the motion at the
completely differential. Figure 2-10 shows that periodically encoder. This is sent to the decoder as a vector. The decoder
complete pictures are sent. These are called Intra-coded uses the vector to shift part of the previous picture to a more
pictures (or I-pictures), and they are obtained by spatial appropriate place in the new picture.
compression only. If an error or a channel switch occurs, it will
be possible to resume correct decoding at the next I-picture.

14 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

Figure 2-12. 4:2:0 sample size vs. 4:2:2 sample size.

One vector controls the shifting of an entire area of the picture The motion estimator works by comparing the luminance data
that is known as a macroblock. The size of the macroblock from two successive pictures. A macroblock in the first picture
is determined by the DCT coding and the color subsampling is used as a reference. The correlation between the reference
structure. Figure 2-12a shows that, with a 4:2:0 system, the and the next picture is measured at all possible displacements
vertical and horizontal spacing of color samples is exactly with a resolution of half a pixel over the entire search range.
twice the spacing of luminance. A single 8x8 DCT block When the greatest correlation is found, this correlation is
of color samples extends over the same area as four 8x8 assumed to represent the correct motion.
luminance blocks; therefore this is the minimum picture area
The motion vector has a vertical and horizontal component.
that can be shifted by a vector. One 4:2:0 macroblock contains
In typical program material, a moving object may extend over
four luminance blocks: one Cb block and one Cr block.
a number of macroblocks. A greater compression factor is
In the 4:2:2 profile, color is only subsampled in the horizontal obtained if the vectors are transmitted differentially. When a
axis. Figure 2-12b shows that in 4:2:2, a single 8x8 DCT block large object moves, adjacent macroblocks have the same
of color samples extends over two luminance blocks. A 4:2:2 vectors and the vector differential becomes zero.
macroblock contains four luminance blocks: two Cb blocks
and two Cr blocks.

[Link]/video_audio 15
Primer

Motion vectors are associated with macroblocks, not with

real objects in the image and there will be occasions where
part of the macroblock moves and part of it does not. In
this case, it is impossible to compensate properly. If the
motion of the moving part is compensated by transmitting
a vector, the stationary part will be incorrectly shifted, and it
will need difference data to be corrected. If no vector is sent,
the stationary part will be correct, but difference data will be
needed to correct the moving part. A practical compressor
might attempt both strategies and select the one that required
the least data.

2.9 Bidirectional Coding

When an object moves, it conceals the background at its
leading edge and reveals the background at its trailing edge.
The revealed background requires new data to be transmitted
because the area of background was previously concealed
Figure 2-13. Area revealed as a result of moving a block.
and no information can be obtained from a previous picture.
A similar problem occurs if the camera pans; new areas come
into view and nothing is known about them. MPEG helps to P-pictures are forward predicted from an earlier picture, which
minimize this problem by using bidirectional coding, which could be an I-picture or a P-picture. P-picture data consists of
allows information to be taken from pictures before and after vectors describing where, in the previous picture, each macro-
the current picture. If a background is being revealed, it will be block should be taken from, and transform coefficients that
present in a later picture, and the information can be moved describe the correction or difference data that must be added
backwards in time to create part of an earlier picture. to that macroblock. Where no suitable match for a macroblock
Figure 2-13 shows the concept of bidirectional coding. could be found by the motion compensation search, intra data
On an individual macroblock basis, a bidirectionally-coded is sent to code that macroblock. P-pictures require roughly
picture can obtain motion-compensated data from an earlier half the data of an I-picture.
or later picture, or even use an average of earlier and later B-pictures are bidirectionally predicted from earlier and/
data. Bidirectional coding significantly reduces the amount of or later I- or P-pictures. B-picture data consists of vectors
difference data needed by improving the degree of prediction describing where in earlier or later pictures data should be
possible. MPEG does not specify how an encoder should be taken from. It also contains the intracoded data that provide
built, only what constitutes a compliant bit stream. However, necessary corrections. Again, when no suitable match for a
an intelligent compressor could try all three coding strategies macroblock is found by the motion compensation search, intra
and select the one that results in the least data to be data is sent to code that macroblock. Bidirectional prediction
transmitted. is quite effective, so most macroblocks in a B-picture will be
coded largely by motion vectors. Also, a B-picture is never
2.10 I-, P- and B-pictures
used as a reference for coding other pictures, so there is no
In MPEG, three different types of pictures are needed to possibility of error propagation. This permits encoders to use
support differential and bidirectional coding while minimizing more aggressive requantization for correction data. A typical
error propagation: B-picture requires about one quarter the data of an I-picture.
I-pictures are intra-coded pictures that need no additional Note that a B-picture does not have to use both directions
information for decoding. They require a lot of data compared of prediction; in some circumstances only one direction is
to other picture types, and therefore they are not transmitted employed. This option may be used when constructing closed
any more frequently than necessary. They consist primarily groups of pictures (GOP).
of transform coefficients and have no vectors. I-pictures are
decoded without reference to any other pictures, so they
allow the viewer to switch channels, and they arrest error
propagation.

16 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

Figure 2-14. Input frame sequence sent out of order to accommodate revealed areas.

Figure 2-14 introduces the concept of the GOP. The GOP

represents the structure of I-, P-, and B-pictures in the
sequence. Generally the GOP structure repeats through the
sequence, but the GOP length and structure may be changed
at any time. There are no formal limits on the length of a GOP,
but for transmission purposes a typical length is 12 or 15
pictures.
The nature of MPEG’s temporal compression means that the
transmission order of pictures is not the same as the display
order. A P-picture naturally follows the I- or P-picture from
which it is predicted, so there are no special requirements. A
bidirectionally-coded B-picture, however, cannot be decoded
until both of its reference pictures have been received and
decoded. Figure 2-14 shows the pictures of a GOP in display
Figure 2-15. Bandwidth and quality decreases as GOP increases.
order at the top, and in transmission order below. Note that,
in transmission order, the B-pictures always follow the two
reference pictures from which they are predicted.
The GOP structure may be altered by the encoder when
There are two types of GOP, open and closed. A closed there are scene changes. Predictions across a scene change
GOP requires no reference outside itself. In display order, will usually fail, since there will be large entropy between the
it may begin with an I-picture and end with a P-picture. In two pictures either side of the scene change. An encoder
transmission order there will usually be B-pictures following may choose to detect the scene change, use a closed GOP
the last P-picture, but these are pictures that will be displayed leading up to the scene change, and start a new GOP (open
before that last P-picture. or closed) with an I-picture representing the first picture of the
It is possible to start and or end a closed GOP with B-pictures new scene.
(in display order), but in this case the starting and ending Sending picture data out of sequence requires additional
B-pictures must be coded using only a single direction of memory at the encoder and decoder and also causes delay.
prediction. B-pictures at the start of a closed GOP must use The number of bidirectionally-coded pictures between intra- or
backward prediction only. B-pictures at the end of a closed forward-predicted pictures must be restricted to reduce cost,
GOP may use forward prediction only – similar to a P-picture, and to minimize delay if this is an issue.
but B-picture rules would be used for requantization etc.
Figure 2-15 shows the trade-off that must be made between
An open GOP does not have these restrictions on prediction compression factor and coding delay. For a given quality,
vectors. For example, B-pictures at the end of the GOP can sending only I-pictures requires more than twice the bit rate of
use forward prediction from the last P-picture and backward an IBBP sequence.
prediction from the first I-picture of the next GOP. This
structure is slightly more efficient, but predictions can cross
any picture boundary. It is much more difficult to splice bit
streams, and events such as channel changes are more likely
to cause picture errors.

[Link]/video_audio 17
Primer

Figure 2-16a. MPEG-2 video encoder flowchart (I-frames highlighted).

2.11 An MPEG Compressor motion estimator. To create an I-picture, the end of the input
delay is selected and the subtracter is turned off so that
Figures 2-16a, b, and c show a typical bidirectional motion
the data pass straight through to be spatially coded (see
compensator structure. Pre-processed input video enters a
Figure 2-16a). Subtracter output data also pass to a frame
series of frame stores that can be bypassed to change the
store that can hold several pictures. The I-picture is held in
picture order. The data then enter the subtracter and the
the store.

18 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

Figure 2-16b. MPEG-2 video encoder flowchart (P-frames highlighted).

To encode a P-picture, the B-pictures in the input buffer are P-picture is subtracted from the actual P-picture to produce
bypassed, so that the future picture is selected (see Figure the prediction error, which is spatially coded and sent along
2-16b). The motion estimator compares the I-picture in the with the vectors. The prediction error is also added to the
output store with the P-picture in the input store to create predicted P-picture to create a locally decoded P-picture that
forward motion vectors. The I-picture macroblocks are shifted also enters the output store.
by these vectors to make a predicted P-picture. The predicted

[Link]/video_audio 19
Primer

Figure 2-16c. MPEG-2 video encoder flowchart (B-frames highlighted).

The output store then contains an I-picture and a P-picture. current B-picture. On a macroblock- by-macroblock basis,
A B-picture from the input buffer can now be selected. The the forward or backward data are selected according to which
motion compensator will compare the B-picture with the represent the smallest differences. The picture differences are
I-picture that precedes it and the P-picture that follows it to then spatially coded and sent with the vectors.
obtain bidirectional vectors (see Figure 2-16c). Forward and
When all of the intermediate B-pictures are coded, the
backward motion compensation is performed to produce
input memory will once more be bypassed to create a new
two predicted B-pictures. These are subtracted from the
P-picture based on the previous P-picture.

20 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

Figure 2-17. Rate controller diagram.

Figure 2-17 shows an MPEG coder. The motion compensator Any practice that causes unwanted motion is to be avoided.
output is spatially coded and the vectors are added in a Unstable camera mountings, in addition to giving a shaky
multiplexer. Syntactical data is also added which identifies picture, increase picture differences and vector transmission
the type of picture (I, P, or B) and provides other information requirements. This will also happen with telecine material if
to help a decoder (see Section 5 – Elementary Streams). sprocket hole damage results in film weave or hop. In general,
The output data are buffered to allow temporary variations video that is to be compressed must be of the highest quality
in bit rate. If the mean bit rate is too high, the buffer will tend possible. If high quality cannot be achieved, then noise
to fill up. To prevent overflow, quantization will have to be reduction and other stabilization techniques will be desirable.
made more severe. Equally, should the buffer show signs of
If a high compression factor is required, the level of artifacts
underflow, the quantization will be relaxed to maintain the
can increase, especially if input quality is poor. In this case, it
average bit rate.
may be better to reduce the entropy presented to the coder
2.12 Preprocessing by using pre-filtering. The video signal is subject to two-
dimensional, low-pass filtering, which reduces the number
A compressor attempts to eliminate redundancy within of coefficients needed and reduces the level of artifacts. The
the picture and between pictures. Anything that reduces picture will be less sharp, but less sharpness is preferable to a
that apparent redundancy, that is not picture content, is high level of artifacts.
undesirable. Noise and film grain are particularly problematic
because they generally occur over the entire picture. After the In most MPEG-2 applications, 4:2:0 sampling is used,
DCT process, noise results in more non-zero coefficients, and which requires a chroma downsampling process if the
the coder cannot distinguish this information from genuine source is 4:2:2. In MPEG-1,the luminance and chroma are
picture data. Heavier quantizing will be required to encode all further downsampled to produce an input picture or CIF
of the coefficients, reducing picture quality. Noise also reduces (common image format) that is only 352-pixels wide. This
similarities between successive pictures, increasing the technique reduces the entropy by a further factor. For very
difference data needed. high compression, the QCIF (quarter common image format)
picture, which is 176-pixels wide, is used. Downsampling
Residual subcarrier in video decoded from composite is a process that combines a spatial low-pass filter with an
video is a serious problem because it results in high, spatial interpolator. Downsampling interlaced signals is problematic
frequencies that are normally at a low level in component because vertical detail is spread over two fields that may de-
programs. Subcarrier also alternates in phase from picture to correlate due to motion.
picture causing an increase in difference data. Naturally, any
composite decoding artifact that is visible in the input to the
MPEG coder is likely to be reproduced at the decoder.

[Link]/video_audio 21
Primer

When the source material is telecine, the video signal has

different characteristics than normal video. In 50-Hz video,
pairs of fields represent the same film frame, and there is
no motion between them. Thus, the motion between fields
alternates between zero and the motion between frames. In
60-Hz video, 3:2 pulldown is used to obtain 60 Hz from 24 Hz
film. One frame is made into two fields; the next is made into
three fields, and so on.
Consequently, one field in five is completely redundant. MPEG
handles film material best by discarding the third field in 3:2
systems. A 24-Hz code in the transmission alerts the decoder
to recreate the 3:2 sequence by re-reading a field store. In 50-
and 60-Hz telecine, pairs of fields are deinterlaced to create
frames, and then motion is measured between frames. The
decoder can recreate interlace by reading alternate lines in the
frame store.
A cut is a difficult event for a compressor to handle because it
often results in an almost complete prediction failure, requiring
a large amount of correction data. If a coding delay can be
Figure 2-18. MPEG audio use fixed-size FFT while wavelet uses variable size.
tolerated, a coder may detect cuts in advance and modify
the GOP structure dynamically, so that an I-picture is inserted
to coincide with the cut. In this case, the cut is handled with
The wavelet transform is not affected by this problem because
very little extra data. The last B-pictures before the I frame
its frequency resolution is a fixed fraction of an octave and
will almost certainly need to use forward prediction. In some
therefore has a logarithmic characteristic. This is done by
applications that are not real-time, such as DVD mastering, a
changing the block length as a function of frequency. As
coder could take two passes at the input video: one pass to
frequency goes down, the block becomes longer. Thus,
identify the difficult or high entropy areas and create a coding
a characteristic of the wavelet transform is that the basis
strategy, and a second pass to actually compress the input
functions all contain the same number of cycles, and these
video.
cycles are simply scaled along the time axis to search for
2.13 Wavelets different frequencies. Figure 2-18 contrasts the fixed block size
of the DFT/DCT with the variable size of the wavelet.
All transforms suffer from uncertainty because the more
accurately the frequency domain is known, the less Wavelets are especially useful for audio coding because
accurately the time domain is known (and vice versa). In they automatically adapt to the conflicting requirements of
most transforms such as discreet Fourier transport (DFT) and the accurate location of transients in time and the accurate
discreet cosine transform (DCT), the block length is fixed, assessment of pitch in steady tones.
so the time and frequency resolution is fixed. The frequency For video coding, wavelets have the advantage of producing
coefficients represent evenly spaced values on a linear scale. resolution-scalable signals with almost no extra effort. In
Unfortunately, because human senses are logarithmic, the moving video, the advantages of wavelets are offset by the
even scale of the DFT and DCT gives inadequate frequency difficulty of assigning motion vectors to a variable size block,
resolution at one end and excess resolution at the other. but in still-picture or I-picture coding this difficulty is not an
issue. Wavelet coding has shown particular benefits for very-
low bit rate applications. The artifacts generated by excessive
quantization of wavelet coefficients generally appear as
“smearing,” and these are much less objectionable than the
“blockiness” that results from excessive quantization of DCT
coefficients.

22 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

Section 3 – Audio Compression

Lossy audio compression is based entirely on the
characteristics of human hearing, which must be considered
before any description of compression is possible. Surprisingly,
human hearing, particularly in stereo, is actually more critically
discriminating than human vision, and consequently audio
compression should be undertaken with care. As with
video compression, audio compression requires a number
of different levels of complexity according to the required
compression factor.
Figure 3-1. Components of the human ear.
3.1 The Hearing Mechanism
Hearing comprises physical processes in the ear and nervous/
mental processes that combine to give us an impression of The resonant behavior of the basilar membrane is an exact
sound. The impression we receive is not identical to the actual parallel with the behavior of a transform analyzer. According
acoustic waveform present in the ear canal because some to the uncertainty theory of transforms, the more accurately
entropy is lost. Audio compression systems that lose only that the frequency domain of a signal is known, the less accurately
part of the entropy that will be lost in the hearing mechanism the time domain is known. Consequently, the more able a
will produce good results. transform is to discriminate between two frequencies, the
less able it is to discriminate between the time of two events.
The physical hearing mechanism consists of the outer, middle,
Human hearing has evolved with a certain compromise that
and inner ears. The outer ear comprises the ear canal and
balances time-uncertainty discrimination and frequency
the eardrum. The eardrum converts the incident sound into a
discrimination; in the balance, neither ability is perfect.
vibration, in much the same way as a microphone diaphragm.
The inner ear works by sensing vibrations transmitted through The imperfect frequency discrimination results in the inability to
a fluid. The impedance of fluid is much higher than that separate closely spaced frequencies. This inability is known as
of air and the middle ear acts as an impedance-matching auditory masking, defined as the reduced sensitivity to sound
transformer that improves power transfer. in the presence of another.

Figure 3-1 shows that vibrations are transferred to the inner Figure 3-2a (see next page) shows that the threshold of
ear by the stirrup bone, which acts on the oval window. hearing is a function of frequency. The greatest sensitivity is,
Vibrations in the fluid in the ear travel up the cochlea, a spiral not surprisingly, in the speech range. In the presence of a
cavity in the skull (shown unrolled in Figure 3-1 for clarity). single tone, the threshold is modified as in Figure 3-2b. Note
The basilar membrane is stretched across the cochlea. This that the threshold is raised for tones at higher frequency
membrane varies in mass and stiffness along its length. At and to some extent at lower frequency. In the presence of
the end near the oval window, the membrane is stiff and a complex input spectrum, such as music, the threshold is
light, so its resonant frequency is high. At the distant end, the raised at nearly all frequencies. One consequence of this
membrane is heavy and soft and resonates at low frequency. behavior is that the hiss from an analog audio cassette is only
The range of resonant frequencies available determines the audible during quiet passages in music. Companding makes
frequency range of human hearing, which in most people is use of this principle by amplifying low-level audio signals
from 20 Hz to about 15 kHz. prior to recording or transmission and returning them to their
correct level afterwards.
Different frequencies in the input sound cause different areas
of the membrane to vibrate. Each area has different nerve
endings to allow pitch discrimination. The basilar membrane
also has tiny muscles controlled by the nerves that together
act as a kind of positive-feedback system that improves the
Q-factor of the resonance.

[Link]/video_audio 23
Primer

Figure 3-2a. Frequency range of the human ear. Figure 3-2b. MPEG masking threshold, based upon the ear being more sensitive to the
human voice range.

Figure 3-3. Temporal masking applied due to human ear recovery time as amplitude Figure 3-4. Audio compression uses different gain levels on different frequency bands.
changes.

The imperfect time discrimination of the ear is due to its 3.2 Subband Coding
resonant response. The Q-factor is such that a given sound
Figure 3-4 shows a band-splitting compander. The band-
has to be present for at least about 1 millisecond before it
splitting filter is a set of narrow-band, linear-phase filters that
becomes audible. Because of this slow response, masking
overlap and all have the same bandwidth. The output in each
can still take place even when the two signals involved are not
band consists of samples representing a waveform. In each
simultaneous. Forward and backward masking occur when
frequency band, the audio input is amplified up to maximum
the masking sound continues to mask sounds at lower levels
level prior to transmission. Afterwards, each level is returned
before and after the masking sound’s actual duration. Figure
to its correct value. Noise picked up in the transmission is
3-3 shows this concept.
reduced in each band. If the noise reduction is compared
Masking raises the threshold of hearing, and compressors with the threshold of hearing, it can be seen that greater
take advantage of this effect by raising the noise floor, which noise can be tolerated in some bands because of masking.
allows the audio waveform to be expressed with fewer bits. Consequently, in each band, after companding it is possible
The noise floor can only be raised at frequencies at which to reduce the wordlength of samples. This technique achieves
there is effective masking. To maximize effective masking, compression because the quantization noise introduced by
it is necessary to split the audio spectrum into different the loss of resolution is masked.
frequency bands to allow introduction of different amounts of
companding and noise in each band.

24 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

Figure 3-5. MPEG-1 using 32-subband encoder.

Figure 3-5 shows a simple band-splitting coder as is used in

MPEG layer 1. The digital audio input is fed to a band-splitting
filter that divides the spectrum of the signal into a number of
bands. In MPEG, this number is 32. The time axis is divided
into blocks of equal length. In MPEG layer 1, there are 384
input samples, so there are 12 samples in each of 32 bands in
the output of the filter. Within each band, the level is amplified
by multiplication to bring the level up to maximum. The gain
required is constant for the duration of a block, and a single
scale factor is transmitted with each block for each band in
order to allow the process to be reversed at the decoder.
The filter bank output for MPEG layer 1 is analyzed using a
512-point FFT to determine the spectrum of the input signal. Figure 3-6. MPEG-1 frame headers and payload.

This analysis drives a masking model that determines the

degree of masking that can be expected in each band. The
more masking available, the less accurate the samples in 3.3 MPEG Layer 1
each band need to be. The sample accuracy is reduced by Figure 3-6 shows an MPEG layer 1 audio bit stream. Following
requantizing to reduce wordlength. This reduction is also the synchronizing pattern and the header, there are 32-bit
constant for every word in a band, but different bands can use allocation codes of four bits each. These codes describe the
different wordlengths. The wordlength needs to be transmitted wordlength of samples in each subband. Next come the 32
as a bit allocation code for each band to allow the decoder to scale factors used in the companding of each band. These
deserialize the bit stream properly. scale factors determine the gain needed in the decoder to
return the audio to the correct level. The scale factors are
followed, in turn, by the audio data in each band.

[Link]/video_audio 25
Primer

Figure 3-7. MPEG 32-subband decoder.

Figure 3-7 shows the layer 1 decoder. The synchronization

pattern is detected by the timing generator, which deserializes
the bit allocation and scale factor data. The bit allocation data
then allows deserialization of the variable length samples. The
requantizing is reversed and the compression is reversed by
the scale factor data to put each band back to the correct
level. These 32 separate bands are then combined in a
combiner filter that produces the audio output.

3.4 MPEG Layer 2

Figure 3-8 shows that when the band-splitting filter is used
to drive the masking model, the spectral analysis is not very
Figure 3-8. 32-subband filtering creates abiguity and lower quality compared to the
accurate, since there are only 32 bands and the energy 1024-subband filtering in MPEG-2.
could be anywhere in the band. The noise floor cannot be
raised very much because, in the worst case shown, the
masking may not operate. A more accurate spectral analysis 3.5 Transform Coding
would allow a higher compression factor. In MPEG layer 2, Layers 1 and 2 are based on band-splitting filters in which
the spectral analysis is performed by a separate process. the signal is still represented as a waveform. However, layer
A 1024-point FFT (Fast Fourier Transform) working directly 3 adopts transform coding similar to that used in video
from the input is used to drive the masking model instead. coding. As was mentioned above, the ear performs a kind
To resolve frequencies more accurately, the time span of the of frequency transform on the incident sound and, because
transform has to be increased, which is done by raising the of the Q-factor of the basilar membrane, the response
block size to 1152 samples. cannot increase or reduce rapidly. Consequently, if an audio
While the block-companding scheme is the same as in layer 1, waveform is transformed into the frequency domain, the
not all of the scale factors are transmitted, since they contain coefficients do not need to be sent very often. This principle
a degree of redundancy on real program material. The scale is the basis of transform coding. For higher compression
factor of successive blocks in the same band differs by 2 factors, the coefficients can be requantized, making them less
dB less than 10 percent of the time, and advantage is taken accurate. This process produces noise that will be placed at
of this characteristic by analyzing sets of three successive frequencies where the masking is the greatest. A by-product
scale factors. When the sub- band content is approximately of a transform coder is that the input spectrum is accurately
constant (known as stationary or quasi-stationary program), known, so a precise masking model can be created.
only one scale factor out of three is sent. As transient content
increases in a given subband, two or three scale factors will
be sent. A scale factor select code is also sent to allow the
decoder to determine what has been sent in each subband.
This technique effectively halves the scale factor bit rate.

26 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

3.6 MPEG Layer 3

This complex level of coding is really only required when the
highest compression factor is needed. It has a degree of
commonality with layer 2. A discrete cosine transform is used
having 384 output coefficients per block. This output can
be obtained by direct processing of the input samples, but
in a multi-level coder, it is possible to use a hybrid transform
incorporating the 32-band filtering of layers 1 and 2 as a basis.
If this is done, the 32 subbands from the QMF (quadrature
mirror filter) are each further processed by a 12-band MDCT
Figure 3-9. MPEG-4 audio with v2 as a superset of v1. v2 adds Parametric Stereo.
(modified discreet cosine transform) to obtain 384 output
coefficients.
Two window sizes are used to avoid pre-echo on transients. known as code excited linear predictive (CELP) coding for
The window switching is performed by the psycho-acoustic speech coding in the mid range of bit rates. Various forms of
model. It has been found that pre-echo is associated with the scalability are supported, including bitstream scalability that
entropy in the audio rising above the average value. To obtain can be applied at points in the transmission system.
the highest compression factor, non-uniform quantizing of the The use of object coding permits choices to be made at the
coefficients is used along with Huffman coding. This technique decoding point. For example, a concerto may be transmitted
allocates the shortest wordlengths to the most common code as two objects, orchestra and solo. Normal decoding would
values. present the complete work, but an instrumentalist could
decode only the orchestra object and perform the solo part
3.7 MPEG-2 Audio “live.” Similar approaches could permit coding of programs
Although originally designated MPEG audio levels 1, 2 and so that listeners could select a “mix minus” mode to eliminate
3, the systems are now more accurately known as MPEG-1 commentary from, say, a sporting event.
Level 1, etc. MPEG-2 defined extensions to MPEG-1 audio,
In 2001, a High Efficiency variant of MPEG-4 AAC was
and a new advanced coding system.
introduced. This is intended to provide superior performance
MPEG-2 permits the use of sampling at lower rates than at lower bit rates, typically around 128 Kbit/s. HE-AAC, also
MPEG-1. This is not strictly backward compatible, but known as aacPlus, is really a superset of AAC that adds an
requires only additional tables in an MPEG-1 decoder for enhancement technique, known as Spectral Band Replication
interoperability. (SBR). SBR is a bandwidth extension technique that enables
MPEG-2 BC (backward compatible) audio provides for 5.1 audio CODECs to deliver the same listening experience at
channels (five full-bandwidth channels plus a low-bandwidth approximately half the bit rate that the core CODEC would
low frequency effects channel). MPEG-2 BC has an MPEG-1 require, if operated on its own.
(2 channel) bit stream at its core and adds the multi-channel More recently, HE-AAC v2 was introduced, as illustrated
extensions in a form that will be ignored by an MPEG-1 decoder. by Figure 3-9. Again, this is a superset of v1 and adds a
MPEG-2 AAC (advanced audio coding) is a more Parametric Stereo (PS) enhancement. Parametric Stereo
sophisticated system with higher resolution filter banks and increases the coding efficiency by exploiting a parametric
additional coding tools. It offers significantly higher coding representation of the stereo image of a given input signal.
efficiency, but is not backward compatible. Given this superset architecture, HE-AAC v2 decoders are
also capable of decoding plain AAC bitstreams, as well as
3.8 MPEG-4 Audio bitstreams incorporating AAC and SBR data components.
MPEG-4 coding is based on objects. (See Section 4.4.2.) This provides a high degree of flexibility for broadcasters, as
MPEG-4 audio objects can represent natural or synthetic it contains all the technical components necessary for audio
sounds. For natural audio coding, the MPEG-4 toolkit includes compression over a wide range of bit rates.
MPEG-2 AAC as well as a variety of other tools, including
parametric encoding for very low bit rates and a technique

[Link]/video_audio 27
Primer

3.9 AC-3
The AC-3 audio coding technique, invented by Dolby
Laboratories, is used with the ATSC system instead of one
of the MPEG audio coding schemes. It is documented as
Figure 3-9-1. Overlapping blocks in AC-3 provide 100% redundancy.
ATSC Standard a/52. Subsequently, AC-3 was adopted as an
optional component of DVB, and by the Motorola Digicypher
II system. AC-3 is a transform-based system that obtains The masking model drives the requantizing process, which
coding gain by requantizing frequency coefficients. reduces the accuracy of each coefficient by rounding the
The PCM input to an AC-3 coder is divided into overlapping mantissa. A significant proportion of the transmitted data
windowed blocks as shown in Figure 3-9-1. These blocks consist of mantissa values.
contain 512 samples each, but because of the complete The exponents are also transmitted, but not directly as there is
overlap, there is 100 percent redundancy. After the transform, further redundancy within them that can be exploited. Within a
there are 512 coefficients in each block, but because of the block, only the first (lowest frequency) exponent is transmitted
redundancy, these coefficients can be decimated to 256 in absolute form. The remaining exponents are transmitted
coefficients using a technique called time domain aliasing differentially and the decoder adds the difference to the
cancellation (TDAC). previous value. Where the input audio has a smooth spectrum,
The input waveform is analyzed, and if there is a significant the exponents in several frequency bands may be the same.
transient in the second half of the block, the waveform will be Exponents can be grouped into sets of two or four with flags
split into two to prevent pre-echo. In this case, the number of that describe what has been done.
coefficients remains the same, but the frequency resolution will Sets of six blocks are assembled into an AC-3 sync frame.
be halved and the temporal resolution will be doubled. A flag The first block of the frame always has full exponent data, but
is set in the bit stream to indicate to the decoder that this has in cases of stationary signals, later blocks in the frame can use
been done. the same exponents.
The coefficients are output in floating-point notation as a
mantissa and an exponent. The representation is the binary
equivalent of scientific notation. Exponents are effectively scale
factors. The set of exponents in a block produce a spectral
analysis of the input to a finite accuracy on a logarithmic scale
called the spectral envelope. This spectral analysis is the input
to the masking model that determines the degree to which
noise can be raised at each frequency.

28 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

Section 4 – The Standards

Sections 2 and 3 introduced the technologies of video
and audio compression, and many of the tools used in
the standards. Over the years, there have been two main
standards bodies undertaking parallel development of video
and audio compression algorithms and standards, namely
MPEG and ITU. This section examines the history, structure
and evolution of the broadcasting related standards.

4.1 What is MPEG

MPEG is the Moving Pictures Experts Group, a committee that
comes under the joint control of the International Standards Figure 4-3. Evolution of MPEG.

Organization (ISO) and the International Electrotechnical

Commission (IEC). IEC handles international standardization
for electrical and electronic technologies; ISO handles virtually 4.3 Evolution of the Standards
everything else. At the start of the information technology Figure 4.3 illustrates the evolution of the video standards. It
age, ISO and IEC formed a joint technical committee (JTC1) can be seen that on occasions the ITU-T VCEG group and
to address IT issues. JTC1 has a number of working groups, MPEG WG 11 groups have collaborated to develop new video
including JPEG (Joint Photographic Experts Group) and coding standards. The resultant CODECs have then been
WG11, which is MPEG. issued by both bodies under separate names. The MPEG-2
The committee was formed in 1988 under the leadership video CODEC standard that arose from the partnership is
of MPEG convener Dr. Leonardo Chiariglione of Italy. It known in the ITU-T community as H.262.
established an enviable track record of generating standards The MPEG-4 part 2 video standard is a large standard and
that achieved widespread adoption: MPEG-1, MPEG-2, and still has a number of issues for commercial implementation.
the MP3 audio compression standard (MPEG-1 Audio, layer It is over-complex, containing many profiles, the vast majority
3). This reputation was somewhat tarnished by MPEG-4, not of which are unused by commercial applications. It includes
because of deficiencies in the standard, but as a result of a number of technical compromises, such that the syntax
the long delay in publishing licensing terms, and the strong is not as well structured and clear as it could be, making
adverse reaction to the first terms that were eventually implementation and interoperability more difficult. Some
published in early 2002. elements are not entirely clear and are open to interpretation.
There are some errors, such as the standard, the normative
4.2 Who are ITU reference software and normative bitstreams are sometimes
The International Telecommunication Union (ITU) is the United at odds with each other. In the extended period it took to
Nations specialized agency in the field of telecommunications. develop and standardize MPEG-4, technology moved onto
The ITU Telecommunication Standardization Sector (ITU-T) H.264/AVC, which undoubtedly gives better compression.
is a permanent organization of ITU. ITU-T is responsible for H.264 is a name related to the ITU-T line of H.26x video
studying technical, operating and tariff questions and issuing standards, while AVC relates to the ISO/IEC MPEG side of the
recommendations on them with a view to standardizing partnership project that completed the work on the standard,
telecommunications on a world-wide basis. In some areas after earlier development done in the ITU-T as a project
of information technology that fall within ITU-T's purview, the called H.26L. It is usual to call the standard H.264/AVC (or
necessary standards are prepared on a collaborative basis AVC/H.264 or H.264/ MPEG-4 AVC or MPEG-4 Part 10) to
with ISO and IEC. emphasize the common heritage. The name H.26L, harkening
A steering group known as, ITU-T SG 16 Q.6, is responsible back to its ITU-T history, is far less common, but still used.
for developing the actual standards. This group is better Occasionally, it has also been referred to as “the JVT CODEC”,
known as VCEG (Video Coding Experts Group). in reference to the JVT organization that developed it.

[Link]/video_audio 29
Primer

6
1st Generation
MPEG-2 Encoder In recent years Microsoft has entered the broadcast market
2nd Generation
MPEG-2 with an alternative video CODEC to AVC/H.264. Microsoft
MPEG-4
5
Encoder
H.264/AVC originally developed the Windows Media Video (WMV) format
H.263
3rd Generation
Encoder
as a proprietary CODEC for Internet streaming purposes. With
4

WMV version 9 it added a new Advanced Profile for broadcast

4th Generation
Mbit/s

3
Encoder
applications. In 2003, WMV-9 was submitted to SMPTE for
5th Generation
Encoder
standardization. The standard was officially approved in March
2
2006 as SMPTE 421M, which is also known as VC-1.
VC-1 provides similar performance to H.264/AVC, and
1
H.264 /
MPEG-4 part 10

0
potentially offers reduced processing and memory
1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005
requirements. With SMPTE standardization, it should find
Figure 4-3-1. Bandwidth requirements drop as codecs improve over time.
widespread use.
It should be noted that competing formats for the next
H.264/AVC concentrates solely on the video, and as a result generation DVD standard, HD DVD and Blu-ray, both mandate
has been easier and quicker to standardize than MPEG-4 part support for AVC/H.264 and VC-1.
2. In addition, it does not have to interoperate with many other
parts of the same standard. The H.264/AVC standards body 4.3.1 Video Performance
ensured that systems elements were taken care of elsewhere CODEC performance is simply the measure of how many
– for example, with a minor modification to the MPEG-2 bits it takes to achieve a particular visual quality in a video
Transport Stream it can carry H.264/AVC video as well as sequence. However, it is much more complex than it appears,
MPEG-2 video. As a result, H.264/AVC has become highly as the performance and “visual quality” is a subjective
attractive to the whole broadcast industry, which uses MPEG- combination of many things, e.g.:
2 Transport Stream (‘TS’). The approach of embedding H.264/
Sharpness of the video.
AVC video in an MPEG-2 TS means that it is backwards
compatible with the majority of the existing highly expensive Visual artifacts such as “blockiness” or “ringing”.
infrastructure – satellite transmission, TV/cable transmission, Color fidelity.
etc. To take advantage of the improved video quality and HD
possibilities that H.264/AVC offers, only the video elements at Video/audio synchronization.
the source and destination endpoints need updating. Smoothness of motion.
Although complex in itself (see ‘Complexity of H.264/AVC’), These visual features derive from the different types of
the standard has been well designed and well written. encoding used throughout the sequence, the differing frame
Relatively good reference software is also available. The result types and the allocation of bit usage variation during the
is that for engineers implementing H.264/AVC, although still a sequence. Whilst the subjective elements are hard to measure,
very large task, it is generally clearer to implement than MPEG- it is possible to quantify objective elements, such as the
4 part 2. Many companies recognize the maturity of the encoding and bit usage. There is some correlation between
standard, so there is now real commercial impetus behind it. the visual quality and the quantifiable parameters, for example,
such as PSNR (Peak Signal-to-Noise Ratio) measurement.

30 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

4.3.2 Licensing sampling rate of 44.1 kHz, has a data transfer rate of up to 1.5
Mbit/s. MPEG-1 succeeds in compressing video and audio so
The MPEG committee itself has no role in licensing. Under
that both may be transmitted within the same data rate!
the auspice of ISO and IEC, it requires that technology
included in its standards be licensable under “reasonable The CIF format is a compromise between European and
and non-discriminatory terms”, but there is no accepted American SIF (source input format) formats: spatial resolution
definition of “reasonable.” Licensing is the responsibility for 625 SIF (352x288) and Temporal Resolution 525 SIF
of the holders of the relevant patents, and typically this (29.97 Hz). This is the basis for video conferencing.
means many organizations throughout the world that have MPEG-1 can was designed for CIF images, and has no tools
contributed research and development, and wish to see some to handle interlaced images, so it had little obvious impact in
recompense. the world of broadcast television.
For MPEG-2, the patent holders grouped together and formed Before leaving MPEG-1, it is important to note what is
MPEG-LA (MPEG licensing authority). All the essential patents actually included in the standard, and how interoperability is
are certified by this group, and are licensed as a block to any achieved. The standard defines a tool set, the syntax of the bit
organization wishing to implement the standards. This worked stream, and the operation of the decoder. It does not define
well for MPEG-2, but as noted above, greater difficulties were the operation of the encoder – any device that produces
experienced with MPEG-4 part 2, and certainly impacted its a syntactically valid bit stream that can be decoded by a
commercial success. compliant decoder is a valid MPEG encoder. Also, it does
H.264/AVC (aka MPEG-4 part 10) also suffered from not define the quality of the picture, nor encoding quality.
protracted licensing issues, which is one reason why VC-1 This allows for the evolution of encoding technology without
was introduced. Eventually two separate groups of patent change to the standard, and without rendering existing
holders were formed and are represented by MPEG-LA and decoders obsolete. This model is used throughout the MPEG
Via Licensing. A license is required from both bodies for any standards. The success of this strategy is obvious; although
organization wishing to implement the H.264/AVC standard. MPEG-2 is used for video, MPEG-1, layer 2 audio is still in
use as the principal audio compression system in the DVB
4.4 MPEG-1 transmission systems today.
The MPEG-1 system, ISO/IEC 11172, is the first international
compression standard for motion imagery and was developed
4.5 MPEG-2
between 1988 and 1992. It uses DCT transforms, coefficient MPEG-1 was frozen (i.e., subsequent changes were allowed
quantization, and variable length coding in a similar manner to be editorial only) in 1991. In the same year the MPEG-
to JPEG, but also includes motion compensation for temporal 2 process was started, and MPEG-2 eventually became a
compression. standard in 1994. The initial goals were simple; there was
a need for a standard that would accommodate broadcast
It is in three parts:
quality video width. This required the coding of “full size”
System ISO/IEC 11172-1, the multiplex structure. standard definition images (704x480 at 29.97 Hz, and
ISO/IEC 11172-2, video coding. 704x576 at 25 Hz), and the ability to code interlaced video
efficiently.
ISO/IEC 11172-3, audio coding.
In many ways, MPEG-2 represents the “coming of age” of
MPEG-1 represented a remarkable technical achievement. It
MPEG. The greater flexibility of MPEG-2, combined with the
was designed to compress image streams with SIF picture
increased availability of large-scale integrated circuits, meant
size, 352x288 (25-Hz PAL) or 352x240 (30 Hz NTSC),
that MPEG-2 could be used in a vast number of applications.
and associated audio, to approximately 1.5 Mbits/s total
The success of MPEG-2 is best highlighted by the demise
compressed data rate. This rate is suitable for transport
of MPEG-3, intended for high-definition television. MPEG-3
over T1 data circuits and for replay from CD-ROM, and
was soon abandoned when it became clear that MPEG-2
corresponds approximately to the resolution of a consumer
could accommodate this application with ease. MPEG-2 is,
video recorder. A measure of this achievement may be seen
of course, the basis for both the ATSC and DVB broadcast
by comparing the numbers for an audio CD. A normal audio
standards, and the compression system used by DVD.
CD, carrying two-channel audio, at 16-bit resolution with a

[Link]/video_audio 31
Primer

Figure 4-5. MPEG-2 levels and profiles.

MPEG-2 was also permitted to be a moving target. By the Note: DVB and ATSC transport streams carry video and audio
use of profiles and levels, discussed below, it was possible to PES within “program” groupings, which are entirely different
complete the standard for one application, but then to move than “program streams” (these are used on DVD & CD).
on to accommodate more demanding applications in an
MPEG Transport Streams are normally constant bit rate but
evolutionary manner. Work on extending MPEG-2 continues
program streams are normally variable bit rate.
into 2002.
4.5.1 Profiles and Levels in MPEG-2
MPEG-2 is documented as ISO/IEC 13818, currently in 10
parts. The most important parts of this standard are: With certain minor exceptions, MPEG-1 was designed for one
task; the coding of fixed size pictures and associated audio
ISO/IEC 13818-1 Systems (transport and programs
to a known bit rate of 1.5 Mbits/sec. The MPEG-1 tools and
streams), PES, T-STD buffer model and the basic PSI
syntax can and have been used for other purposes, but such
tables: CAT, PAT, PMT and NIT.
use is outside the standard, and requires proprietary encoders
ISO/IEC 13818-2 video coding. and decoders. There is only one type of decoder compliant to
ISO/IEC 13818-3 audio coding. the MPEG-1 standard.

ISO/IEC 13818-4 MPEG test and conformance.

ISO/IEC 13818-6 data broadcast and DSMCC.
One of the major achievements of MPEG-2 defined in 13818-
1, the transport stream, is described in Section 8. The flexibility
and robustness of this design have permitted it to be used for
many applications, including transport of MPEG-4 and MPEG-
7 data.

32 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

At the outset, there was a similar goal for MPEG-2. It was The high-1440 level is a high definition scheme that doubles
intended for coding of broadcast pictures and sound, the definition compared to the main level. The high level not
nominally the 525/60 and 625/50 interlaced television only doubles the resolution but maintains that resolution with
systems. However, as the design work progressed, it was 16:9 format by increasing the number of horizontal samples
apparent that the tools being developed were capable of from 1440 to 1920.
handling many picture sizes and a wide range of bit rates. In
In compression systems using spatial transforms and
addition, more complex tools were developed for scalable
requantizing, it is possible to produce scalable signals. A
coding systems. This meant that in practice there could not be
scalable process is one in which the input results in a main
a single MPEG-2 decoder. If a compliant decoder had to be
signal and a “helper” signal. The main signal can be decoded
capable of handling high-speed bit streams encoded using all
alone to give a picture of a certain quality, but if the information
possible tools, it would no longer be an economical decoder
from the helper signal is added, some aspect of the quality
for mainstream applications. As a simple example, a device
can be improved.
capable of decoding high-definition signals at, say, 20 Mbits/
sec would be substantially more expensive than one limited to For example, a conventional MPEG coder, by heavily
standard-definition signals at around 5 Mbits/sec. It would be requantizing coefficients, encodes a picture with moderate
a poor standard that required the use of an expensive device signal-to-noise ratio results. If, however, that picture is locally
for the simple application. decoded and subtracted pixel-by-pixel from the original,
a quantizing noise picture results. This picture can be
MPEG devised a two-dimensional structure of profiles and
compressed and transmitted as the helper signal. A simple
levels for classifying bit streams and decoders. Profiles
decoder only decodes the main, noisy bit stream, but a more
define the tools that may be used. For example, bidirectional
complex decoder can decode both bit streams and combine
encoding (B-frames) may be used in the main profile, but
them to produce a low-noise picture. This is the principle of
not in simple profile. Levels relate just to scale. A high level
SNR (signal-to-noise ratio) scalability.
decoder must be capable of receiving a faster bit stream, and
must have more decoder buffer and larger frame stores than As an alternative, coding only the lower spatial frequencies in
a main level decoder. However, main profile at high level (MP@ a HDTV picture can produce a main bit stream that an SDTV
HL) and main profile at main level (MP@ML) use exactly the receiver can decode. If the lower definition picture is locally
same encoding/decoding tools and syntax elements. decoded and subtracted from the original picture, a definition-
enhancing picture would result. This picture can be coded into
Figure 4-5 shows the pairings of profile and level that are
a helper signal. A suitable decoder could combine the main
defined by MPEG-2 (Profiles on the horizontal axis, Levels
and helper signals to recreate the HDTV picture. This is the
on the vertical axis). Note that not all combinations are valid;
principle of spatial scalability.
only the completed pairings are defined in the standard. It is
a requirement of conformity to the standard that a decoder The high profile supports both SNR and spatial scalability as
at any profile/level must be able to decode lower profiles and well as allowing the option of 4:2:2 sampling.
levels. For example, an MP@ML decoder must be able to The 4:2:2 profile has been developed for improved
decode main profile at low level (MP@LL) and simple profile at compatibility with digital production equipment. This profile
main level (SP@ML) bit streams. allows 4:2:2 operation without requiring the additional
The simple profile does not support bidirectional coding, complexity of using the high profile. For example, an HP@
and so only I-and P-pictures will be output. This reduces the ML decoder must support SNR scalability, which is not a
coding and decoding delay and allows simpler hardware. The requirement for production. The 4:2:2 profile has the same
simple profile has only been defined at main level. freedom of GOP structure as other profiles, but in practice it is
commonly used with short GOPs making editing easier. 4:2:2
The Main Profile is designed for a large proportion of uses. The
operation requires a higher bit rate than 4:2:0, and the use of
low level uses a low-resolution input having only 352 pixels
short GOPs requires an even higher bit rate for a given quality.
per line. The majority of broadcast applications will require the
MP@ML subset of MPEG, which supports SDTV (standard The concept of profiles and levels is another development of
definition TV). MPEG-2 that has proved to be robust and extensible; MPEG-
4 uses a much more complex array of profiles and levels, to
be discussed later.

[Link]/video_audio 33
Primer

Figure 4-6. MPEG-4, Part 2 codec diagram.

4.6 MPEG-4 Performance at low bit rates remained a major objective

and some very creative ideas contributed to this end. Great
International standardization is a slow process, and
attention was also paid to error resilience, making MPEG-4
technological advances often occur that could be incorporated
very suitable for use in the error-prone environments, such as
into a developing standard. Often this is desirable, but
transmission to personal handheld devices. However, other
continual improvement can mean that the standard never
profiles and levels use bit rates up to 38.4 Mbits/s, and work is
becomes final and usable. To ensure that a standard is
still proceeding on studio- quality profiles and levels using data
eventually issued there are strict rules that prohibit substantive
rates up to 1.2 Gbits/s.
change after a certain point in the standardization process.
So, by the time a standard is officially adopted there is often a More importantly, MPEG-4 became vastly more than just
backlog of desired enhancements and extensions. Such was another compression system – it evolved into a totally new
the case with MPEG-2. As discussed above, MPEG-3 had concept of multimedia encoding with powerful tools for
been started and abandoned, so the next project became interactivity and a vast range of applications. Even the official
MPEG-4. “overview” of this standard spans 67 pages, so only a brief
introduction to the system is possible here.
At first the main focus of MPEG-4 was the encoding of video
and audio at very low rates. In fact, the standard was explicitly
optimized for three bit rate ranges:
Below 64 Kbits/s.
64 to 384 Kbits/s.
384 Kbit/s to 4 Mbits/s.

34 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

Figure 4-6-3. MPEG-4, Part 2 feature extract process.

4.6.1 MPEG-4 Standards Documents each be coded with the tools most appropriate to the job.
The objects may be generated independently, or a scene
The principal parts of the MPEG-4 standards are:
may be analyzed to separate, for example, foreground and
ISO/IEC 14496-1 Systems. background objects. In one interesting demonstration, video
ISO/IEC 14496-2 Visual. coverage of a soccer game was processed to separate the
ball from the rest of the scene. The background (the scene
ISO/IEC 14496-3 Audio.
without the ball) was transmitted as a “teaser” to attract a pay-
ISO/IEC 14496-4 Conformance Testing. per-view audience. Anyone could see the players and the field,
ISO/IEC 14496-6 Delivery Multimedia Integration but only those who paid could see the ball!
Framework (DMIF). The object-oriented approach leads to three key
ISO/IEC 14496-10 Advanced Video Coding. characteristics of MPEG-4 streams:

ISO/IEC 14496-12 ISO Base Media File Format. Multiple objects may be encoded using different techniques,
and composited at the decoder.
ISO/IEC 14496-14 MP4 File Format.
Objects may be of natural origin, such as scenes from a
ISO/IEC 14496-15 Advanced Video Coding (AVC) file camera, or synthetic, such as text.
format.
Instructions in the bit stream, and/or user choice, may
4.6.2 Object Coding enable several different presentations from the same bit
The most significant departure from conventional transmission stream.
systems is the concept of objects. Different parts of the The generalized system for object coding in MPEG-4 is shown
final scene can be coded and transmitted separately as in Figure 4-6. This diagram also emphasizes the opportunities
video objects and audio objects to be brought together, for user interaction within MPEG-4 systems – a powerful
or composited, by the decoder. Different object types may feature, particularly for video game designers.

[Link]/video_audio 35
Primer

Figure 4-6-4. MPEG-4, Part 2 encoder diagram.

These capabilities do not have to be used – MPEG-4 also be represented. The standard includes tools for shape
provides traditional coding of video and audio, and improves coding of rectangular and irregular objects, in either binary or
on MPEG-2 by offering improved efficiency and resilience gray-scale representations (similar to an alpha channel). The
to errors. However, the true power of MPEG-4 comes from concept is shown in Figure 4-6-3.
the architecture described above. The coding of objects
Similarly, MPEG-4 uses tools similar to those of MPEG-1
independently offers a number of advantages. Each object
and MPEG-2 for coding live audio, and AAC offers greater
may be coded on the most efficient manner, and different
efficiency. Multiple audio “objects” may be encoded separately
spatial or temporal scaling (see 4.6.3) may be used as
and composited at the decoder. As with video, audio objects
appropriate.
may be natural or synthetic.
4.6.3 Video and Audio Coding
4.6.4 Scalability
Many of the video coding tools in MPEG-4 are similar to those
In the context of media compression, scalability means the
of MPEG-2, but enhanced by better use of predictive coding
ability to distribute content at more than one quality level within
and more efficient entropy coding. However, the application of
the same bitstream. MPEG-2 and MPEG-4 part 2 both provide
the tools may differ significantly from earlier standards.
scalable profiles using a conventional model; the encoder
MPEG-4 codes video objects. In the simplest model a video generates a base-layer and one or more enhancement layers,
is coded in much the same way as in MPEG-2, but it is as shown in Figure 4-6-4. The enhancement layer(s) may be
described as a single video object with a rectangular shape. discarded for transmission or decoding if insufficient resources
The representation of the image is known as texture coding. are available. This approach works, but all decisions about
Where there is more than one video object, some may have quality levels have to be made at the time of encoding, and in
irregular shapes, and generally all will be smaller than a full- practice the number of enhancement layers is severely limited
screenbackground object. This means that only the active area (usually to one).
of the object need be coded, but the shape and position must

36 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

Figure 4-6-5. MPEG-4, Part 10 still uses 16x16 macroblock sizes, but adds many new block sizes and combinations for better efficiency.

4.6.5 Variable Macroblock Size Main Profile: This profile has largely been superseded
by High profile as the mainstream profile for consumer
One of the advantages of AVC/H.264 over MPEG-4 part 2
broadcast and storage devices.
is variable size macroblocks, as illustrated by Figure 4-6-5.
H.264/ AVC allows the 16x16 macroblocks to be sub-divided Extended Profile: A profile focussed on video streaming,
down to blocks as small as 4x4, to enhance the compression this has relatively high compression capabilities together
efficiency. In MPEG-2 and MPEG-4 part 2, there is just one with good resilience to data losses.
type of 'Intra' macroblock, containing a compressed image,
High Profile: The primary profile for broadcast and disk
which does not refer to any other macroblock.
storage devices, particularly for HDTV applications. It
To incorporate this efficiency, however, there are 26 types of includes extra coding efficiencies, for example, with the
AVC/ H.264 Intra macroblock. There are many other complex introduction of an 8x8 block transform. This is the profile
elements in H.264/AVC, such as the ‘CABAC’ Entropy used by HD DVD and Blu-ray disk formats.
coding, where bits are shared such that fractions of 1 bit may
High 10 Profile: This profile extends the High profile used
effectively represent a specific syntax element.
predominately for consumer devices; in particular, support
On the converse side, substantial attempts have been made for up to 10 bits per sample decoded picture precision.
to understand and reduce the computing power required. For
High 4:2:2 Profile: Primarily targeting professional devices
example, instead of a DCT, a simple well defined integer 4x4
that use interlaced video, this profile extends the High 10
transform could be utilized that only uses fast-to-execute shift,
Profile; in particular, support for the 4:2:2 chroma sampling
subtract and add operations.
format.
4.6.6 Profiles and Levels in MPEG-4 part 10 High 4:4:4 Profile: This profile extends the High 4:2:2
(AVC/H.264) profile. It supports up to 4:4:4 chroma sampling, up to
In a similar manner to MPEG-2, a series of profiles and levels 12 bits per sample, efficient loss less region coding and
have been specified for classifying bitstreams and decoders. RGB video coding transform that reduces color-space
The following seven profiles, or capabilities, are intended to transformation error. Note that this profile has been
target different types of application: removed from the standard in favor of developing a new
improved 4:4:4 profile.
Baseline Profile: Mainly targeted at applications with
limited computing resources, this profile is widely used in
videoconferencing and mobile devices.

[Link]/video_audio 37
Primer

H.264/AVC Profiles

Profile Chroma Format Max Sample Depth Max Bit Rate

Level 3 Level 5

Baseline 4:2:0 8 bits 10 Mb/s 135 Mb/s

Main 4:2:0 8 bits 10 Mb/s 135 Mb/s

Extended 4:2:0 8 bits 10 Mb/s 135 Mb/s

High 4:2:0 8 bits 12.5 Mb/s 168.75 Mb/s

High 10 4:2:0 10 bits 30 Mb/s 405 Mb/s

High 4:2:2 "4:2:0, 4:2:2" 10 bits 40 Mb/s 540 Mb/s

High 4:4:4 "4:2:0, 4:2:2, 4:4:4" 12 bits 40 Mb/s 540 Mb/s

Figure 4-6-6. H.264 profiles.

There are five main levels and a number of sub-levels 4.7 SMPTE VC-1
associated with each profile. Levels essentially relate to scale.
The basic mode of operation of VC-1 is the same as most
Each level specifies a maximum video bit rate and number
other video compression standards since H.261 and MPEG-1
of macroblocks per frame for example, as well as display
and involves block-based motion compensation and spatial
resolution. Display resolution may vary from 128x96 in level
transforms. A number of enhancements have been made to
1 to 4096x2304 in level 5.1. This covers the QCIF formats
give superior rate-distortion performance compared with older
used on mobile phones through to HDTV displays. Note that
CODECs, which allows it to compete with H.264/AVC.
1920x1080 and 4096x2048 are referred to as 2K and 4K
modes respectively in HDTV parlance. Together with innovations to improve quality, it has also been
designed to have a relatively low level of complexity, which
4.6.7 MP4 Container Format helps it to target a wide range of applications and bit rates.
MP4 is a multimedia container format standard. It is most- Some of the technologies used in VC-1 are described on the
commonly used to store MPEG-4 audio and video bitstreams, next page.
but also can be used to store other data such as subtitles
and still images. The MP4 format is based upon Apple's
QuickTime container format, but has been improved upon in
many ways. Like most modern container formats, MP4 allows
streaming over the Internet. The official file name extension
for MPEG-4 Part 14 files is .mp4, thus the container format is
often referred to simply as MP4. Devices that play .mp4 files
are referred to as MP4 players.
MPEG-4 part 15 specifies the format for carriage of H.264/
AVC elementary streams within an MP4 file.

38 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

VC-1 Profiles

Profile Level Chroma Format Max Sample Max Bit Rate Interlace
Depth (b/s) Support

Simple Low 4:2:0 8 bits 96 k

Medium 4:2:0 8 bits 384 k

Main Low 4:2:0 8 bits 2M

Medium 4:2:0 8 bits 10 M

High 4:2:0 8 bits 20 M

Advanced L0 4:2:0 8 bits 2M

L1 4:2:0 8 bits 10 M X

L2 4:2:0 8 bits 20 M X

L3 4:2:0 8 bits 45 M X

L4 4:2:0 8 bits 135 M X

Figure 4-7. SMPTE VC-1 profiles.

4.7.1 Variable Block Size Transform lower bit rates and complexity 16x16 blocks together with
1/2 pixel resolution can be used. For higher bit rates, motion
VC-1 shares the ability of AVC/H.264 to use smaller
vectors to 1/4 pixel resolution may be coded and the block
macroblocks than more traditional CODECs in order to
size reduced to 8x8.
increase coding efficiency and reduce the appearance of
artifacts such as ‘ringing’ at block boundaries. The standard Unlike H.264/AVC, motion vectors for 4x4 blocks can't be
8x8 transform can still be used, but this can be subdivided sent; this means that the complexity is lower, but at the cost of
into multiple 8x4, 4x8 or 4x4 transforms to produce the best reduced flexibility.
picture quality for each macroblock. The transform size can
4.7.3 Deblocking and Smoothing
be specified per block or per macroblock, but can also be
signaled at the frame level which can be useful in low bit rate Various methods are employed to reduce the ‘blocking’
situations to reduce the overhead. artifacts that are often visible in block-based CODECs,
especially at lower bit rates. VC-1 uses an in-loop deblocking
4.7.2 Motion Compensation filter to reduce these artifacts, which can affect multiple frames
The model for motion compensation in VC-1 is similar to if used as a predicted block in future frames.
that used in MPEG-1 and many other CODECs in which a A technique of overlap smoothing is also used to reduce
prediction for a block is made based upon a reference frame unintended block boundary discontinuities. This uses
and a displacement vector. transforms which slightly overlap the block boundaries to
There are four modes of motion compensation available in mitigate the edge artifacts.
VC-1 which are tailored to different decoder scenarios. For

[Link]/video_audio 39
Primer

4.7.4 Fading Compensation MPEG-7 defines a structure of descriptors and description

schemes that can characterize almost anything. In theory
Motion compensation tends to perform poorly when
at least, primitive elements such as color histograms and
attempting to compensate for fading effects in a scene. Fade
shapes can be combined to represent complex entities
compensation allows for a linear transform of pixel values to
such as individual faces. It may be possible to index material
be made to account for broad changes in luminance. This can
automatically such that the database can be searched for
in turn produce better results from the motion compensation
scenes that show, for example, President Clinton and U.S.
step.
Federal Reserve Chairman Greenspan together.
4.7.5 Profiles and Levels
However, MPEG-7 has not been widely adopted and
Figure 4-7 shows that VC-1 has three profiles; Simple, Main instead so-called ‘wrapper’ or ‘container’ formats such
and Advanced. The Simple and Main profiles need to be MXF are increasing in popularity, particularly in Production
encapsulated within an ASF wrapper, while Advanced profile environments. The key features of MXF are:
streams can exist as independent elementary streams.
Inter-operability.
The Advanced profile addresses many of the issues
Extensibility.
associated with the broadcast industry in that it supports
interlaced as well as progressive encoding. It is also easily Platform independence.
encapsulated in container and transport formats. MXF supports a number of different essence tracks, encoded
The three profiles have between two and five levels which with a variety of CODECs together with accompanying
broadly correspond to the scale of the video both in terms of metadata. It uses the KLV [Key-Length-Value] metadata
maximum bit rate and resolution. This gives a range suitable to system, which may refer to a local dictionary or external public
applications from mobile streaming to High Definition video. registries. An MXF metadata dictionary is standardized by
SMPTE.
4.8 MPEG-7 and MXF
4.9 MPEG-21
MPEG-7 is not about compression; it is about metadata,
also known as the “bits about the bits.” Metadata is digital The MPEG-21 standard is intended to define an open
information that describes the content of other digital data. framework for multimedia delivery from content creator to
In modern parlance, the program material or content, the content consumer. It is intended to enable transparent and
actual image, video, audio or data objects that convey the augmented use of multimedia resources across a wide range
information are known as data essence. The metadata tells of networks and devices. A major aspiration of the standard is
the world all it needs to know about what is in the essence. that it will put an end to illicit file sharing.

Anyone who has been involved with the storage of information, At its heart is a “Rights Expression Language” standard, as a
be it videotapes, books, music, etc., knows the importance means of sharing digital rights, permissions and restrictions.
and the difficulty of accurate cataloging and indexing. Stored It is an XML-based standard designed to communicate
information is useful only if its existence is known, and if it can machine-readable license information and do so in an
be retrieved in a timely manner when needed. "ubiquitous, unambiguous and secure" manner.

This problem has always been with us, and is addressed MPEG-21 is based on two essential concepts: the definition of
in the analog domain by a combination of labels, catalogs, a fundamental unit of distribution and transaction, termed the
card indexes, etc. More recently, the computer industry has Digital Item, and the concept of users interacting with them.
given us efficient, cost-effective, relational databases that MPEG-21 provides a framework in which one user interacts
permit powerful search engines to access stored information, with another user, and the object of that interaction is a Digital
provided the information is present in a searchable form. Item. MPEG-21 essentially defines the technology needed to
support users who wish to exchange, access, consume, trade
or manipulate Digital Items in an efficient and transparent way.
That is, it is a collection of technologies that designers of, say
DRM systems, could use.

40 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

Figure 5-1. Building a video sequence starting with a block of DCT coefficients.

Section 5 – Elementary Streams Blocks are assembled into macroblocks, which are the
fundamental units of a picture and which can be motion
An elementary stream is basically the raw output of an compensated. Each macroblock has a two-dimensional
encoder and contains no more than is necessary for a motion vector in the header. In B-pictures, the vectors can be
decoder to approximate the original picture or audio. The backward as well as forward. The motion compensation can
syntax of the compressed signal is rigidly defined in MPEG be field or frame based and this is indicated. The scale used
so that decoders can be guaranteed to operate on it. The for coefficient requantizing is also indicated. Using the vectors,
encoder is not defined except that it must somehow produce the decoder obtains information from earlier and later pictures
the right syntax. to produce a predicted picture. The blocks are inverse-
The advantage of this approach is that it suits the real world transformed to produce a correction picture that is added to
in which there are likely to be many more decoders than the predicted picture to produce the decoded output. In 4:2:0
encoders. By standardizing the decoder, they can be made coding, each macroblock will have four Y blocks and two
at low cost. In contrast, the encoder can be more complex color-difference blocks. To make it possible to identify which
and more expensive without a great cost penalty, but with the block describes which component, the blocks are sent in a
potential for better picture quality as complexity increases. specified order.
When the encoder and the decoder are different in complexity, Macroblocks are assembled into slices that must always
the coding system is said to be asymmetrical. represent horizontal strips of picture from left to right. In
The MPEG approach also allows for the possibility that MPEG, slices can start anywhere and be of arbitrary size, but
quality will improve as coding algorithms are refined while in ATSC they must start at the left-hand edge of the picture.
still producing bit streams that can be understood by earlier Several slices can exist across the screen width. The slice
decoders. The approach also allows the use of proprietary is the fundamental unit of synchronization for variable length
coding algorithms, which need not enter the public domain. and differential coding. The first vectors in a slice are sent
absolutely, whereas the remaining vectors are transmitted
5.1 Video Elementary Stream Syntax differentially. In I-pictures, the first DC coefficients in the slice
Figure 5-1 shows the construction of the elementary video are sent absolutely and the remaining DC coefficients are
stream. The fundamental unit of picture information is the DCT transmitted differentially. In difference pictures, correlation of
(discrete cosine transform) block, which represents an 8x8 these coefficients is not to be expected, and this technique is
array of pixels that can be Y, Cb or Cr. The DC coefficient is not appropriate.
sent first and is represented more accurately than the other
coefficients. Following the remaining coefficients, an end of
block (EOB) code is sent.

[Link]/video_audio 41
Primer

In the case of a bit error in the elementary stream, either the 5.2 Audio Elementary Streams
deserialization of the variable length symbols will break down,
Various types of audio can be embedded in an MPEG-2
or subsequent differentially coded coefficients or vectors will
multiplex. These types include audio coded according to
be incorrect. The slice structure allows recovery by providing a
MPEG layers 1, 2, 3, or AC-3. The type of audio encoding
resynchronizing point in the bit stream.
used must be included in a descriptor that a decoder will read
A number of slices are combined to make a picture that is the in order to invoke the appropriate type of decoding.
active part of a field or a frame. The picture header defines
The audio compression process is quite different from the
whether the picture was I, P or B coded and includes a
video process. There is no equivalent to the different I, P, and
temporal reference so that the picture can be presented at the
B frame types, and audio frames always contain the same
correct time. In the case of pans and tilts, the vectors in every
amount of audio data. There is no equivalent of bidirectional
macroblock will be the same. A global vector can be sent for
coding and audio frames are not transmitted out of sequence.
the whole picture, and the individual vectors then become
differences from this global value. In MPEG-2 audio, the descriptor in the sequence header
contains the layer that has been used to compress the audio
Pictures may be combined to produce a GOP that must
and the type of compression used (for example, joint stereo),
begin (in transmission order) with an I-picture. The GOP is the
along with the original sampling rate. The audio sequence is
fundamental unit of temporal coding. In the MPEG standard,
assembled from a number of access units (AUs) that will be
the use of a GOP is optional, but it is a practical necessity.
coded audio frames.
Between I-pictures, a variable number of P- and/or B-pictures
may be placed as was described in Section 2. A GOP may be If AC-3 coding is used, as in ATSC, this usage will be reflected
open or closed. In a closed GOP, the last B-pictures do not in the sequence header. The audio access unit (AU) is an AC-3
require the I-picture in the next GOP for decoding and the bit sync frame as described in Section 3.7. The AC-3 sync frame
stream could be cut at the end of the GOP. represents a time span equivalent of 1536 audio samples and
will be 32 ms for 48-kHz sampling and 48 ms for 32 kHz.
If GOPs are used, several GOPs may be combined to produce
a video sequence. The sequence begins with a sequence
start code, followed by a sequence header and ends with
a sequence end code. Additional sequence headers can
be placed throughout the sequence. This approach allows
decoding to begin part way through the sequence, as might
happen in playback of digital video disks and tape cassettes.
The sequence header specifies the vertical and horizontal
size of the picture, the aspect ratio, the chroma subsampling
format, the picture rate, the use of progressive scan or
interlace, the profile, level, and bit rate, and the quantizing
matrices used in intra and inter-coded pictures.
Without the sequence header data, a decoder cannot
understand the bit stream, and therefore sequence headers
become entry points at which decoders can begin correct
operation. The spacing of entry points influences the delay in
correct decoding that may occur when the viewer switches
from one television channel to another.

42 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

6.2 Time Stamps

After compression, pictures are sent out of sequence because
of bidirectional coding. They require a variable amount of
data and are subject to variable delay due to multiplexing and
transmission. In order to keep the audio and video locked
together, time stamps are periodically incorporated in each
picture.
A time stamp is a 33 bit number that is a sample of a counter
driven by a 90 kHz clock. This clock is obtained by dividing
the 27 MHz program clock by 300. Since presentation times
are evenly spaced, it is not essential to include a time stamp
in every presentation unit. Instead, time stamps can be
Figure 6-1. Header parameters within a Packetized Elementary Stream. interpolated by the decoder, but they must not be more than
700 ms apart in either program streams or transport streams.
Time stamps indicate where a particular access unit belongs
Section 6 – Packetized Elementary in time. Lip sync is obtained by incorporating time stamps
Streams (PES) into the headers in both video and audio PES packets. When
a decoder receives a selected PES packet, it decodes each
For practical purposes, the continuous elementary streams
access unit and buffers it into RAM. When the time-line count
carrying audio or video from compressors need to be broken
reaches the value of the time stamp, the RAM is read out. This
into packets. These packets are identified by headers that
operation has two desirable results. First, effective timebase
contain time stamps for synchronizing. PES packets can be
correction is obtained in each elementary stream. Second,
used to create Program Streams or Transport Streams.
the video and audio elementary streams can be synchronized
6.1 PES Packets together to make a program.

In the PES, an endless elementary stream is divided into

packets of a convenient size for the application. This size
might be a few hundred kilobytes, although this would vary
with the application.
Each packet is preceded by a PES packet header. Figure 6-1
shows the contents of a header. The packet begins with a
start-code prefix of 24 bits and a stream ID that identifies the
contents of the packet as video or audio and further specifies
the type of audio coding. These two parameters (start code
prefix and stream ID) comprise the packet start code that
identifies the beginning of a packet. It is important not to
confuse the packet in a PES with the much smaller packet
used in transport streams that, unfortunately, shares the same
name.
Because MPEG only defines the transport stream, not the
encoder, a designer might choose to build a multiplexer that
converts from elementary streams to a transport stream in
one step. In this case, the PES packets may never exist in an
identifiable form, but instead, they are logically present in the
Transport Stream payload.

[Link]/video_audio 43
Primer

Figure 6-2. PTS and DTS values (based upon PCR values) tell the decoder when to decode a picture and then when to present the picture.

6.3 PTS/DTS Figure 6-2 shows that when an access unit containing an
I-picture is received, it will have both DTS and PTS in the
When bidirectional coding is used, a picture may have to be
header and these time stamps will be separated by one
decoded some time before it is presented so that it can act
picture period. If bidirectional coding is being used, a P-picture
as the source of data for a B-picture. Although, for example,
must follow and this picture also has a DTS and a PTS time
pictures can be presented in the order IBBP, they will be
stamp, but the separation between the two stamp times is
transmitted in the order IPBB. Consequently, two types of time
three picture periods to allow for the intervening B-pictures.
stamp exist. The decode time stamp (DTS) indicates the time
Thus, if IPBB is received, I is delayed one picture period, P is
when a picture must be decoded, whereas a presentation
delayed three picture periods, the two Bs are not delayed at
time stamp (PTS) indicates when it must be presented to the
all, and the presentation sequence becomes IBBP. Clearly,
decoder output.
if the GOP structure is changed such that there are more
B-pictures are decoded and presented simultaneously so that B-pictures between I and P, the difference between DTS and
they only contain PTS. When an IPBB sequence is received, PTS in the P-pictures will be greater.
both I- and P- must be decoded before the first B-picture. A
The PTS/DTS flags in the packet header are set to indicate
decoder can only decode one picture at a time; therefore the
the presence of PTS alone or both PTS and DTS time stamp.
I-picture is decoded first and stored. While the P-picture is
Audio packets may contain several access units and the
being decoded, the decoded I-picture is output so that it can
packet header contains a PTS. Because audio packets are
be followed by the B-pictures.
never transmitted out of sequence, there is no DTS in an audio
packet.

44 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

Section 7 – Program Streams The problem of genlocking to the source does not occur in a
DVD player. The player determines the time base of the video
Program streams are one way of combining several PES with a local synchronizing pulse generator (internal or external)
packet streams and are advantageous for recording and simply obtains data from the disk in order to supply
applications such as DVD. pictures on that time base. In transmission, the decoder has to
recreate the time base at the encoder or it will suffer overflow
7.1 Recording vs. Transmission
or underflow. Thus, a transport stream uses program clock
For a given picture quality, the data rate of compressed video reference (PCR), whereas a program stream has no need for
will vary with picture content. A variable bit rate channel will the program clock.
give the best results. In transmission, most practical channels
are fixed and the overall bit rate is kept constant by the use of 7.2 Introduction to Program Streams
stuffing (meaningless data). A program stream is a PES packet multiplex that carries
In a DVD, the use of stuffing is a waste of storage capacity. several elementary streams that were encoded using the
However, a storage medium can be slowed down or speeded same master clock or system time clock (STC). This stream
up, either physically or, in the case of a disk drive, by changing might be a video stream and its associated audio streams,
the rate of data transfer requests. This approach allows a or a multichannel audio-only program. The elementary video
variable-rate channel to be obtained without capacity penalty. stream is divided into access units (AUs), each of which
When a medium is replayed, the speed can be adjusted to contains compressed data describing one picture. These
keep a data buffer approximately half full, irrespective of the pictures are identified as I, P, or B and each carries an AU
actual bit rate, which can change dynamically. If the decoder number that indicates the correct display sequence. One
reads from the buffer at an increased rate, it will tend to empty video AU becomes one program-stream packet. In video,
the buffer, and the drive system will simply increase the access these packets vary in size. For example, an I-picture packet
rate to restore balance. This technique only works if the audio will be much larger than a B-picture packet. Digital audio
and video were encoded from the same clock; otherwise, they access units are generally of the same size and several are
will slip over the length of the recording. assembled into one program-stream packet. These packets
should not be confused with transport-stream packets that
To satisfy these conflicting requirements, program streams
are smaller and of fixed size. Video and audio AU boundaries
and transport streams have been devised as alternatives. A
rarely coincide on the time axis, but this lack of coincidence
program stream works well on a single program with variable
is not a problem because each boundary has its own time-
bit rate in a recording environment; a transport stream works
stamp structure.
well on multiple programs in a fixed bit rate transmission
environment.

[Link]/video_audio 45
Primer

Section 8 – Transport Streams The operation of digital video production equipment is heavily
dependent on the distribution of a stable system clock for
A transport stream is more than a multiplex of many PES synchronization. In video production, genlocking is used, but
packets. In program streams, time stamps are sufficient to over long distances, the distribution of a separate clock is not
recreate the time axis because the audio and video are locked practical. In a Transport Stream, the different programs may
to a common clock. For transmission down a data network have originated in different places that are not necessarily
over distance, there is an additional requirement to recreate synchronized. As a result, the Transport Stream has to provide
the clock for each program at the decoder. This requires an a separate means of synchronizing for each program.
additional layer of syntax to provide PCR signals.
This additional synchronization method is called a PCR and
8.1 The Job of a Transport Stream it recreates a stable reference clock that can be divided
down to create a time line at the decoder, so that the time
The Transport Stream carries many different programs and
stamps for the elementary streams in each program become
each may use a different compression factor and a bit rate
useful. Consequently, one definition of a program is a set of
that can change dynamically even though the overall bit rate
elementary streams sharing the same timing reference.
stays constant. This behavior is called statistical multiplexing
and it allows a program that is handling difficult material to In a Single Program Transport Stream (SPTS), there will be one
borrow bandwidth from a program handling easy material. PCR channel that recreates one program clock for both audio
Each video PES can have a different number of audio and and video. The SPTS is often used as the communication
data PESs associated with it. Despite this flexibility, a decoder between an audio/video coder and a multiplexer. It is also
must be able to change from one program to the next and used in VOD applications. However, most broadcast transport
correctly select the appropriate audio and data channels. streams include multiple programs (MPTS). This larger
Some of the programs can be protected so that they can only aggregate contains control information, better known as
be viewed by those who have paid a subscription or fee. The signaling (SI) required to coordinate the entire transmission,
Transport Stream must contain CA information to administer such as an electronic program guide (EPG), as well as
this protection. The Transport Stream contains PSI to handle program specifics.
these tasks.
The transport layer converts the PES data into small packets
of constant size (adding stuffing bits if necessary) that are self-
contained. When these packets arrive at the decoder, there
may be jitter in the timing. The use of time division multiplexing
also causes delay, but this factor is not fixed because the
proportion of the bitstream allocated to each program need
not be fixed. Time stamps are part of the solution, but they
only work if a stable clock is available. The Transport Stream
must contain further data allowing the re-creation of a stable
clock.

46 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

The packet identification (PID). This thirteen-bit code is used

to distinguish between different types of packets. More will
be said about PID later.
The continuity counter. This four-bit value is incremented
by the multiplexer as each new packet having the same
PID is sent. It is used to determine if any packets are lost,
repeated, or out of sequence.
In some cases, more header information is needed, and if this
is the case, the adaptation field control bits are set to indicate
that the header is larger than normal. Figure 8-1 shows that
when this happens the extra header length is described by the
adaptation field length code. Where the header is extended,
the payload becomes smaller to maintain constant packet
length.

8.3 Program Clock Reference (PCR)

The encoder used for a particular program will have a 27-MHz
program clock. In the case of an SDI (serial digital interface)
input, the bit clock can be divided by 10 to produce the
encoder program clock. Where several programs originate in
Figure 6-1. ISO/IEC 13818-1 Transport Stream header parameters. the same production facility, it is possible that they will all have
the same clock. In case of an analog video input, the H-sync
period will need to be multiplied by a constant in a phase-
8.2 Packets locked loop to produce 27 MHz.
Figure 8-1 shows the structure of a transport stream packet. The adaptation field in the packet header is used periodically
The size is a constant 188 bytes and it is always divided into to include the PCR code that allows generation of a locked
a header and a payload. Figure 8-1a shows the minimum clock at the decoder. If the encoder or a remultiplexer has
header length is 4 bytes and contains no Adaptation field. In to switch sources, the PCR may have a discontinuity. The
this header, the most important information is: continuity count can also be disturbed. This event is handled
The sync byte. This byte is recognized by the decoder so by the discontinuity indicator, which tells the decoder to
that the header and the payload can be deserialized. expect a disturbance. Otherwise, a discontinuity is an error
condition.
The transport error indicator. This indicator is set if the error
correction layer above the transport layer is experiencing a
raw-bit error rate (BER) that is too high to be correctable. It
indicates that the packet may contain errors. See Section
10 – Introduction to DVB & ATSC for details of the error
correction layer.

[Link]/video_audio 47
Primer

Figure 8-2. Adding a PCR time stamp into a transport stream and then extraction and synchronizing with a decoder.

Figure 8-2 shows how the PCR is used by the decoder to 8.4 Packet Identification (PID)
recreate a remote version of the 27-MHz clock for each
A 13-bit field in the transport packet header contains the
program. The encoder clocks drive a constantly running
Packet Identification code (PID). The PID is used by the
binary counter, and the value of these counters are sampled
demultiplexer to distinguish between packets containing
periodically and placed in the header adaptation fields as the
different types of information. The transport-stream bit rate
PCR. The PCR is a 42-bit number that is represented by a
must be constant, even though the sum of the rates of all of
33-bit PCR base, plus a 9-bit PCR extension to provide higher
the different streams it contains can vary. This requirement
resolution. (The PCR base, like the PTS, is a 33-bit number
is handled by the use of null packets. If the real payload rate
that is a sample of a counter driven by a 90-kHz clock). The
falls, more null packets are inserted. Null packets always
packets generated by each encoder are given a different PID.
have the same PID, which is 8191 (thirteen ones in the binary
The decoder recognizes the packets with the correct PID for
representation).
the selected program and ignores others. At the decoder, a
voltage controlled oscillator (VCO) generates a nominal 27 In a given transport stream, all packets belonging to a given
MHz clock and this drives a local PCR counter. The local elementary stream will have the same PID. The demultiplexer
PCR is compared with the PCR from the packet header and can easily select all data for a given elementary stream simply
the difference is the PCR phase error. This error is filtered to by accepting only packets with the right PID. Data for an entire
control the VCO that eventually will bring the local PCR count program can be selected using the PIDs for video, audio and
into step with the header PCRs. Heavy VCO filtering ensures data streams such as subtitles or teletext. The demultiplexer
that jitter in PCR transmission does not modulate the clock. can correctly select packets only if it can correctly associate
The discontinuity indicator will reset the local PCR count and, them with the elementary stream to which they belong. The
optionally, may be used to reduce the filtering to help the demultiplexer can do this task only if it knows what the right
system quickly lock to the new timing. PIDs are. This is the function of the PSI.

MPEG requires that PCRs be sent at a rate of at least 10 8.5 Program Specific Information (PSI)
PCRs per second, whereas DVB specifies a minimum of 25
PCRs per second. PSI is carried in packets having unique PIDs, some of which
are standardized and some of which are specified by the
program association table (PAT), conditional access table
(CAT) and the transport stream description table (TSDT).

48 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

Figure 8-3. Table hierarchy in interleaved transport stream.

These packets must be included periodically in every transport A given network information table (NIT) contains details of
stream. The PAT always has a PID of 0, the CAT always has a more than just the transport stream carrying it. Also included
PID of 1, and the TSDT always has a PID of 2. These values are details of other transport streams that may be available
and the null-packet PID of 8191 are the only PIDs fixed by the to the same decoder, for example, by tuning to a different
MPEG standard. The demultiplexer must determine all of the RF channel or steering a dish to a different satellite. The NIT
remaining PIDs by accessing the appropriate tables. However, may list a number of other transport streams and each one
there are some constraints in the use of PIDs in ATSC and must have a descriptor that specifies the radio frequency,
DVB. In this respect (and in some others), MPEG and DVB/ orbital position, and so on. In DVB, additional metadata,
ATSC are not fully interchangeable. All DVB and ATSC known as DVB-SI, is included, and the NIT is considered to
transport stream must be MPEG-2 compliant (ISO/IEC 13818- be part of DVB-SI. This operation is discussed in Section 10 –
1), but not all MPEG-2 transport streams will be compliant with Introduction to DVB & ATSC. When discussing the subject in
the ATSC (A65) or DVB (EN 300 468) standards. general, the term PSI/SI is used.
The programs that exist in the transport stream are listed in Upon first receiving a transport stream, the demultiplexer must
the program association table (PAT) packets (PID = 0) that look for PIDs 0 and 1 in the packet headers. All PID 0 packets
carries the PID of each PMT packet. The first entry in the PAT, contain the PAT. All PID 1 packets contain CAT data.
program 0, is reserved for network data and contains the PID
By reading the PAT, the demultiplexer can find the PIDs of the
of network information table (NIT) packets. Usage of the NIT is
NIT and of each program map table (PMT). By finding the PMTs,
optional in MPEG-2, but is mandatory in DVB.
the demultiplexer can find the PIDs of each elementary stream.
The PIDs for entitlement control messages (ECM) and
Consequently, if the decoding of a particular program is
entitlement management messages (EMM) are listed in the
required, reference to the PAT and then the PMT is all that is
conditional access table (CAT) packets (PID = 1).
needed to find the PIDs of all of the elementary streams in the
As Figure 8-3 shows, the PIDs of the video, audio, and data program. If the program is encrypted, access to the CAT will
elementary streams that belong in the same program are listed also be necessary. As demultiplexing is impossible without
in the Program Map Table (PMT) packets. Each PMT packet a PAT, the lockup speed is a function of how often the PAT
normally has its own PID, but MPEG-2 does not mandate this. packets are sent. MPEG specifies a maximum interval of 0.5
The program number within each PMT will uniquely define seconds for the PAT packets and the PMT packets that are
each PMT. referred to in those PAT packets. In DVB and ATSC, the NIT
may reside in packets that have a specific PID.

[Link]/video_audio 49
Primer

Figure 9-1. Polar axis graph for analog video. Similar to vectorscope display. Figure 9-2. Quadrature Modulation.

Section 9 – Digital Modulation 9.2 Analog Modulation

MPEG systems encode and package video, audio, and other In the analog world, the amplitude or frequency of a carrier is
data. For storage, the resulting data stream may be recorded changed (modulated), according to the amplitude of an audio
on a hard drive or perhaps DVD. For other applications, or video signal, usually according to a linear law. An example
the MPEG data, usually in the form of a transport stream, where two quadrature-phase subcarriers are amplitude
has to be sent from one place to another, or to many modulated (as described later in Figure 9-3) can be found in
places. Television systems use cable, satellite and terrestrial the NTSC and PAL color modulation systems. When these
transmission in various ways for contribution, distribution and two linearly, amplitude modulated sub-carriers are combined,
broadcast. All of these transport mechanisms require that the the resulting signal has both simultaneous amplitude and
data be modulated on some carrier. This section provides a phase modulation.
brief introduction to the digital modulation schemes that are This familiar operation provides a useful example. We can think
used to achieve this. of the color information in two ways. It can be expressed in
This section discusses just the modulation techniques. Other polar form as a phase angle (related to hue) plus an amplitude
processes are necessary to make a data stream suitable for (related to saturation and color intensity), or in rectangular,
transmission, and will depend on both the modulation chosen “Cartesian” xy form as values of two color difference signals,
and the channel characteristics. These techniques will be B-Y and R-Y. If the color difference components are treated
discussed in Section 10. as “x” and “y” values on a graph, the two representations are
seen to be closely related, as illustrated in Figure 9-1. This
9.1 Principles of Modulation figure is similar to the well-known vectorscope display which
A continuous carrier conveys no information unless it is is essentially a B-Y and R-Y color component constellation
modified in some way by the information to be transmitted. diagram.
A carrier can be modified in three ways, by changing its
9.3 Quadrature Modulation
amplitude, frequency or phase. Frequency and phase are, of
course, closely related. Although generally treated as separate Figure 9.2 shows how we can modulate a carrier with two
modulation types, the distinction can become very blurred; different signals, using the technique known as “quadrature
some “frequency modulation” systems are implemented modulation.” A single carrier is split into two paths, and one
by “phase modulators” after performing integration on the path is delayed by a time equal to one-quarter of the cycle
modulating signal. Generally these types of modulators are time of the carrier or otherwise shifted in phase by 90 degrees
called “angle modulators” and are useful when constant to create two carriers in quadrature-phase. The two carriers
envelope power in the modulated signal is desired. are each amplitude modulated by an appropriate signal,
and the two modulated carriers are then added together.
This generates a single signal with amplitude and phase
determined by the amplitudes of the two modulating signals.

50 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

Demodulation is achieved by an almost identical process;

the received signal is sent to two demodulators. In each
demodulator the signal is multiplied by a local oscillator signal,
synchronized to the received signal. The two local oscillator
signals are 90 degrees apart, and each demodulator recovers
one axis of the quadrature modulation.

9.4 Simple Digital Modulation Systems

Most digital modulation systems use some form of quadrature
Figure 9-3. BPSK modulation plot (only two possible values).
modulation; the two axes are usually designated I (In phase)
and Q (Quadrature phase). Sometimes only one modulation
axis is used. In all cases, digital modulation is really just analog Digital modulation systems are frequently labeled by the
modulation where particular modulating signal levels are used modulation type, preceded by a number representing the
to uniquely define a particular combination of binary bits. number of states for each symbol. For example, 4QAM
These pre-defined analog signal levels are called symbols. describes quadrature amplitude modulation with four possible
All digital modulation schemes represent a compromise of states for each symbol. Four states can convey two bits of
bandwidth efficiency, robustness and complexity. Symbol information (00, 01, 10, 11), so the bit rate of a 4QAM system
rate is the principal factor in determining the bandwidth of is twice the symbol rate.
the transmitted signal. The symbol rate is the rate at which
The simplest digital modulation systems convey one bit of
the modulation is changed. With ideal filtering the positive-
information per symbol. Each symbol has two possible states,
frequency bandwidth can be as little as half the symbol rate
representing binary zero and binary one. The states may be
but this requires precise symbol time-sampling and filtering in
created by amplitude, frequency or phase modulation, but
the receiver. Therefore, typically the symbols are filtered with a
frequency and phase modulation are the most common.
little extra bandwidth of 5 to 20% to ease the receiver filtering
and sampling. So, with the typical double-sideband signals Binary frequency shift keying (BFSK or 2FSK) uses one carrier
of quadrature modulation, the bandwidth of the modulated frequency to represent a binary zero, and a different frequency
signal is about 5 to 20% larger than the symbol rate. Since to represent a binary one. Sometimes the frequency difference
the receiver needs to be band-limited to optimize signal-to- is very small, and is achieved by a phase modulator after
noise ratio, the symbol filtering is typically split between the integrating the symbols.
transmitter (limiting the occupied channel bandwidth) and the
Binary phase shift keying (BPSK or 2PSK) uses one phase of
receiver (limiting the noise bandwidth). The most common
the (constant amplitude) carrier to represent binary zero, and
way to split the symbol filtering is to take the square root of
the inverse (180 degrees phase shift) to represent binary one.
the desired symbol response and place identical square root
The different possible states of a symbol are usually shown
filters shapes in both the transmitter output and receiver input.
in a constellation diagram showing the various combinations
This way the combined response squares the square root to
resulting from permitted values of the I and Q modulating
recover the desired symbol response.
signals. The constellation diagram for BPSK, shown in Figure
Some simple digital modulation systems carry only one bit 9-3, is very simple; only one axis is used, and there are only
of information per symbol. In other words, each symbol may two permitted values.
represent one of two possible states, representing a binary
These systems can be very robust; the receiver needs only
zero or a binary one via the symbol polarity. In this case the bit
enough signal (or signal-to-noise ratio) to determine which of
rate of the system is the same as the symbol rate. However,
two possible states has been transmitted for each symbol.
other systems have many possible states for each symbol, so
However, they do not use spectrum efficiently; the bandwidth
they can convey more than one bit of information per symbol.
is nominally the same as the bit-rate so the bandwidth may
Generally the number of states is made to be a power of two,
be too large at the required bit rate. These systems are used
so the bit rate of the system is some integer multiple of the
on very difficult transmission paths, such as deep-space
symbol rate.
telemetry where signal-to-noise, rather than bandwidth, is the
limiting factor.

[Link]/video_audio 51
Primer

Figure 9-4. QPSK modulation plot (four possible values). Figure 9-5. 8PSK modulation plot (eight possible values).

9.5 Phase Shift Keying 9.5.2 DVB S2

BPSK or 2PSK was described in the previous section. DVB-S2 Digital Video Broadcast – Satellite – Second
Other forms of PSK modulation use both the I and Q axes. generation is an improved and updated specification designed
Quaternary phase shift keying (QPSK, also known as to supplement the DVB-S specification. It offers higher
quadrature phase shift keying) is the most common, and uses bandwidth than the original DVB-S specification at the cost of
two values on each axis. The constellation diagram is shown additional complexity and has been driven commercially by the
in Figure 9-4. QPSK has four possible states per symbol, so need for high definition satellite broadcast services. DVB-S2
each symbol carries two bits of information; one possible can closely approach the Shannon limit for the maximum data
mapping of states to binary values is shown in the Figure. If rate of transmission achievable in a given channel bandwidth
the I and Q symbols are properly timed, the QPSK signal will with a defined carrier to noise ratio (C/N) and offers up to
have a constant envelope power. Therefore, QPSK is used 30% greater capacity than DVB-S, but still provides backward
extensively in satellite communications where saturated power compatibility.
amplifiers with fixed output envelope power are used. DVB-S2 has two innovations over DVB-S, changing channel
8PSK is less common, but is used in professional satellite encoding parameters in real time using VCM (Variable Coding
transmission systems, particularly in Japan and the USA. The and Modulation) and ACM (Adaptive Coding and Modulation)
constellation diagram is shown in Figure 9-5. 8PSK carries Four modulation modes are available:
three bits of information in each symbol, so the bit rate is three
times the symbol rate and without special coding requires QPSK as in DVB-S.
more carrier power or a larger receive dish, since the signal-to- 8 PSK.
noise must be higher than that for QPSK.
16APSK.
9.5.1 Turbo 8PSK 32APSK.
Turbo 8PSK uses the advanced techniques of Turbo Coding DVB-S2 FEC uses a combination of binary BCH coding
forward error correction to increase the throughput of satellite combined with a very large, low-density parity check code
broadcast links by using 8PSK without increasing antenna (LDPC) first developed by Gallager in 1958. LDPC coding has
size and carrier power. Turbo coding FEC utilizes statistical renewed interest, eclipsing Turbo coding, due to the large
techniques for probabilistic data recovery and is discussed in computing power available today. LDPC decoders can provide
more detail in section 10 on advanced FEC techniques. receiver error thresholds within 0.7 dB of the Shannon limit. In
fact, in some cases, the signal-to-noise ratio can be negative.
Also, combined BCH and LDPC FEC are used in the China
DMB-T/H standard described later.

52 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

Figure 9-6. 4QAM modulation plot (four possible values). Figure 9-7. 16QAM modulation plot (16 possible values).

9.6 Quadrature Amplitude Modulation - QAM

Quadrature amplitude modulation (QAM) is the basis of many
transmission systems. Both the I and Q axes are used for
modulation and, depending on the particular variant, two or
more amplitude levels are permitted for each axis.
The simplest variant is 4QAM where just two values are used
for each axis, providing four possible states for each symbol.
The constellation diagram is shown in Figure 9-6, where it will
be seen that 4QAM is identical to QPSK, and carries two bits
per symbol.
16QAM uses four values on each axis, providing 16 possible
states. 16QAM systems carry four bits per symbol. If six
Figure 9-8. 32QAM modulation plot (32 possible values).
values are permitted for each modulation axis, there are a total
of 36 possible states. Five bits may be carried using only 32
states, so four of the possible combinations are not used in When good signal-to-noise can be guaranteed, even greater
32QAM. The constellation diagram for 16QAM is shown in constellation densities may be used. 64QAM uses eight values
Figure 9-7 and for 32QAM is shown in Figure 9-8. In 32QAM on each axis, and carries six bits per symbol. 64QAM is the
the four “corner” states are not used; these are the states most extensively used modulation scheme in cable systems
that would represent the highest amplitude and, therefore, the worldwide, as it provides a good trade-off between robustness
highest transmitter power. and compatibility with legacy cable infrastructures. 256QAM,
Figures 9-7 and 9-8 also help to show the trade off between used in some of the latest cable television systems, has 16
bit rate and robustness. In the presence of noise or jitter, the permissible values for each modulation axis, and carries eight
closer spacing of the states in 32QAM (at equal transmitted bits per symbol.
power) will make decoding errors more likely. Put another way,
the more possible states per symbol the better the signal to-
noise ratio required for a given error rate.

[Link]/video_audio 53
Primer

Figure 9-9. 4VSB modulation plot (four possible values). Figure 9-10. 8VSB modulation plot (eight possible values).

9.7 Vestigial Sideband Modulation – VSB the same constellation as BPSK. However, since only one
sideband is transmitted, the quadrature carrier cannot be
When a carrier is modulated, sidebands are generated above
used so QPSK is not possible. Recall that QPSK (or 4-QAM)
and below the carrier frequency. For example, a QAM system is 2-bits per symbol thereby needing only half the symbol-rate
with a symbol rate of 6 megasymbols per second will have for the same bit rate as 2VSB. Therefore, since QPSK needs
upper and lower sidebands each just over 3 MHz, requiring a bandwidth about equal to the symbol-rate, 2VSB with only
a nominal channel bandwidth of just over 6 MHz. However, one sideband would occupy the same bandwidth as QPSK for
using quadrature carriers, at least two bits per symbol are the same bit rate. Figure 9-9 shows the constellation diagrams
available if for the simplest constellation. for 4VSB and Figure 9-10 shows 8VSB, carrying respectively
To recover both amplitude and phase information (or both two and three bits per symbol.
axes of the quadrature modulation), both sidebands must be 8VSB modulation is used in the United States by the ATSC
recovered at the receiver. digital television standard. 4VSB was proposed originally,
providing two bits per symbol. However, it was found that
Vestigial sideband systems eliminate most of one sideband
8VSB, in conjunction with and extra parity bit using Trellis
prior to transmission, so only one modulation axis can be
coding (see Section 10.6), and convolutional inner error
recovered since the quadrature carrier component is not
correction provides the same date rate with improved signal-
cancelled due the missing lower sideband. (An alternative
to-noise performance.
interpretation is to say that the other modulation axis is
used so as to suppress the unwanted sideband.) 2VSB has

54 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

The number of carriers in an OFDM system can be very large.

DVB-T has options for 1705 or 6817 carriers (known as 2k
and 8k systems). Because the bandwidth allotted to each
carrier is small, the symbol rate is correspondingly reduced,
and the length of time to transmit each symbol is increased.
This is the key to OFDM’s tolerance to multipath interference.
In a single-carrier system, such as 8VSB, a high data rate
means that the symbol time is very short. In the case of the
ATSC digital television system, some 11 million symbols are
transmitted each second, giving a symbol duration of less
than 100 ns. This means that even a very short multipath
delay will create inter-symbol interference because the delayed
signal representing one symbol will arrive during reception of a
subsequent symbol.
Figure 9-11. Single carrier vs. multiple carrier.
In contrast, an OFDM system with thousands of carriers will
have a symbol time in the order of hundreds of microseconds
9.8 Coded Orthogonal Frequency Division (depending on the data rate, the number of carriers, and
the modulation used). Inter-symbol interference may be
Multiplex – COFDM
virtually eliminated by adding a “guard band” to each
In the above systems, a baseband signal is supplied to symbol,deliberately making the symbol longer than needed.
modulators that operate on a single carrier to produce This reduces the symbol rate, but only to a relatively small
the transmitted sideband(s). An alternative to a wideband degree. For example, if the nominal symbol duration is 200 µs,
system is one that produces many narrowband carriers a guard band of 50 µs will reduce the symbol rate by only 20%
at carefully regulated spacing. Figure 9-11a shows that a – and the elimination of inter-symbol interference may allow
digitally modulated carrier has a spectral null at each side. a higher order constellation to be used, perhaps more than
Another carrier of slightly higher or lower frequency can be compensating for this loss. This technique is not practical on a
placed here without interference as long as it has an integer wide-band single-carrier system. As an example, the same 50
number of cycles more or less per symbol. This creates an µs guard band with a 100 ns symbol time would reduce the
orthogonal carrier pair that can be independently modulated. data rate to a fraction of one percent!
Therefore, the two are mutually orthogonal as Figure 9-11b
However, OFDM is not fundamentally immune to multipath due
shows. This is the principle of OFDM (orthogonal frequency
to self-symbol interference causing carrier nulls. Therefore, any
division multiplexing). In practice, a variant known as coded
pilot carriers that are needed at the receiver must be scattered
orthogonal frequency division multiplexing (COFDM) improves
among the various data carriers. In fact, COFDM receivers
the performance dramatically in non-ideal channel conditions
typically use adaptive equalization to mitigate multipath as
by the use of FEC such as concatenated Viterbi convolutional
required with single carrier, high symbol-rate systems. This
and Reed Solomon coding, described in the next section.
tolerance to multipath interference also makes COFDM
COFDM is used in the digital video broadcasting (DVB)
systems well suited to single frequency networks where two
terrestrial system, DVB-T, the Japanese ISDB-T terrestrial
or more synchronized transmitters emit the same signal. A
system and in all mobile broadcast systems DVB-H, T-DMB,
receiver may, depending on its location and antenna system,
DAB-IP, ISDB-Tss and FLO.
receive signals from more than one transmitter at different
Each carrier in an OFDM system may be modulated by any of times. If the path lengths are radically different, the main signal
the techniques described in this section. In practice QAM is will probably be very much stronger than the secondary signal,
generally used, 16QAM and 64QAM being most common. It and interference will be minimal. If path lengths and signal
is not necessary to use all of the carriers. For example, if one strengths are similar, the guard band will prevent inter-symbol
part of the channel is known to be subjected to a high degree interference.
of interference, the affected carriers may be omitted.

[Link]/video_audio 55
Primer

9.9 China Digital Multimedia Broadcast- 9.10.1 ISDB-S Satellite System

Terrestrial/Handheld - DMB-T/H Launched in December 2000, ISDB-S enabled two
China has announced its own indigenous digital TV standard broadcasters to share a satellite transponder. It is also referred
designed for both fixed and mobile coverage. Called Digital to as BS-digital or CS-digital when the space segment is a
Multimedia Broadcast-Terrestrial/Handheld DMB-T/H (GB broadcast satellite or a communication satellite, respectively.
20600-2006), it is derived from two other standards: ADTB-T Up to eight transport streams can be used in all in a manner to
(similar to ATSC-T) and a variant of T-DMB. be agreed between the broadcasters sharing the transponder.
DMB-T/H is derived from work at Tsinghua University in Beijing The aggregate bit rate will depend on the transponder
and Jiaotong University in Shanghai and combines technology bandwidth and the modulation mode used. For example, for a
developed by both Universities. transponder of 34.5 MHz, the maximum rate including forward
error correction is 56.610 Mbits/s.
DMB-T/H provides SDTV and HDTV services, Digital Audio
Broadcast, Multimedia Broadcasting and data services to fixed Hierarchical modulation allows the modulation mode to be
receivers. For mobile receivers it provides SDTV, Digital Audio varied on a packet-by-packet basis within a 48-packet frame.
Broadcasting, Multimedia Broadcast and data services. Each packet is assigned a modulation slot. Four modulation
modes are supported BSPK(1/2), QPSK (to 7/8) and TC8PSK.
The standard uses the following technologies for transmission
The number of slots vary according to the mode used.
in a 8 MHz channel the UHF and VHF bands and has a
number of modes of operation defined for specific multimedia 9.10.2 ISDB-C Cable System
delivery types: The main feature of this system is that it transmits multiple
Support for Multi Frequency Networks (MFNs) and Single transport streams on a single 64QAM carrier. The system
Frequency Networks (SFNs). was developed in order to be able to retransmit efficiently the
information carried on ISDB-S signals. A maximum of 52.17
Bit randomization.
Mbits/s of information are transmitted typically on a BS-digital
FEC (binary BCH and LDPC coding). carrier. The information rate of a 64QAM/6 MHz signal is
QAM Modulation within OFDM – 64QAM, 32QAM, 16QAM, 29.162 Mbits/s. Hence, at least two cable television channels
4QAM, 4QAM-NR. must be used to retransmit information of a single BS carrier.
The full BS digital service consists of four broadcasters and
Very deep time domain interleaving.
occupies approximately 174 MHz including guard bands.
3 code rates, 3 frame sizes and 2 interleaving depths Using ISDB-C 8 cable channels would be required to carry
allowing a wide range of bit rates. this information, whereas as many as 29 channels would
be required using conventional cable transmission of one
3780 carrier mode (2 KHz spacing) with Frequency domain
transport stream per carrier.
interleaving.
There are 52 modulation slots plus 1 slot for the synchronizing
Dual pilot, single carrier mode.
header, TSMF (transport stream multiplexing frame).
Spread Spectrum techniques for SI transmission.
9.10.3 ISDB-T Terrestrial Modulation
9.10 Integrated Services Data Broadcasting The ISDB-T channel is divided into 13 segments (typically
(ISDB) 400-500 kHz wide), and a separate COFDM transmission
Integrated services data broadcasting (ISDB) is a development is used for each segment. All of the parameters affecting
that uses many modulation schemes and has been developed robustness (number of carriers, guard band length, modulation
for digital television services in Japan. It is designed to support type, convolution coding) may be chosen separately for each
hierarchical systems of many levels. It could be used, for layer of the hierarchy. For example, the most robust segment
example, to provide simultaneously low data rate reception might use a long guard band, QPSK modulation, and 1/2
under exceptionally difficult mobile conditions, intermediate convolution coding. The highest level could use a shorter
data rate (standard definition) for fringe-area static reception, guard band, 64QAM, and 7/8 convolution coding – providing
and high data rate (perhaps for HDTV) for good reception many times the data rate of the robust segment.
conditions.
The center segment may be used for partial reception,
There are four ISDB modulation systems currently in use in designed to allow a narrow band receiver to receive this
Japan: segment only.

56 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

9.11 Mobile Broadcast Systems

Mobile broadcast systems are increasingly important as
cellular phones and highly efficient digital video compression
techniques merge to enable TV and multimedia reception on
the move. COFDM channel coding techniques dominate this
area due to their robustness and ability to deal with multipath
interference and large Doppler shifts of carriers due to the
requirement for high speed reception in cars and trains.
The majority of mobile broadcast networks are designed
to be Single Frequency Networks (SFNs), i.e. the same
transmissions are made from all broadcast points in the
network at exactly the same frequency and time aligned,
using a GPS derived clock. This has the advantage of spectral
Figure 9-11-2. Mobile phone including mobile TV functionality. efficiency and relies on the ability of OFDM signals to reject
self interference from multipath signals using the inter-symbol
guard band.
In normal OFDM the complete channel bandwidth represents
Several standards for mobile TV have been authored by
a single layer. The carriers used are spaced out across the
industry bodies and commercial companies and all are
bandwidth at set multiples of a certain frequency. In ISDB-T
deploying, with some predominating in their parent geographic
the channel bandwidth of 5.6 MHz is divided up into 13
region. Several competing systems are deploying in parallel,
segments each having a bandwidth of 429 kHz. Hierarchical
leading to a requirement for multi-standard handsets able
transmission of ISDB-T is achieved by transmitting OFDM
to roam between networks using different transmission
segment groups having different transmission parameters.
standards. All mobile standards have focused on providing
These groups of layers constitute the layers. In non-
low power, and hence long battery life with highly robust RF
hierarchical modulation the same modulation scheme is used
transmissions. In a mobile environment handsets will have a
for all 13 segments.
small antenna at a height of 1.5-2 m from the ground. This has
9.10.4 ISDB-Tss Mobile Modulation lead to a series of innovations in advanced channel prediction
and FEC techniques. All the transmission standards discussed
Single Segment ISDB-T (ISDB-Tss) uses the central segment
are true broadcast, point to multipoint systems which only
of the 13 segment group to transmit services to mobile
require a forward link to the receiving terminal, most use
terminals and is covered in more detail in section 9.11.5.
a cellular back channel to provide interactive services, for
9.10.5 ISDB in Summary example links to web pages.

ISDB-S provides a means of applying different modulation Most mobile TV transmission systems use a number of
modes to multiple transport streams and transmitting them common methods to improve the transmission links immunity
in a 34.5 MHz channel on a single carrier. ISDB-C provides to the challenging RF environment. These include:
a means of transmitting multiple transport streams in a Robust sync acquisition mode to improve sync time and
single 6 MHz channel on a single carrier but with a common provide channel mode signaling.
modulation mode. ISDB-T is concerned with up to three
transmissions of the same transport stream in a single 6 MHz Deep time interleaving to reduce the impact of impulsive
channel. interference.

MPEG compression is already being used in broadcasting and Time slicing to reduce power consumption by enabling the
will become increasingly important in the future. This section handset to only receive on a low duty cycle.
discusses the additional requirements for digital television Variable Fast Fourier Transform (FFT) depth to provide trade
broadcasting, as implemented by the two principal DTV offs between SFN size.
Standards.
SFN operation for frequency efficiency.

[Link]/video_audio 57
Primer

TS bitrate

Time sliced:
Service 1, Service 2, Service 3

Bandwidth 1 2 3 1 2 3 1 2 3 1 2 Non-time sliced:

Service 4

Time
Figure 9-11-3. Services divided into time domain slots so mobile devices do not need to receive and parse the entire stream.

9.11.1 DVB-H 9.11.2 T-DMB

Digital Video Broadcast – Handheld DVB-H has been defined Digital Multimedia Broadcasting-Terrestrial (T-DMB)
by the DVB as its standard for transmission to mobile transmission is based on the Eureka 147 Digital Audio
terminals, and allows bandwidths of up to ~14 Mbits/s to Broadcasting (DAB) standard. Like DAB, T-DMB is designed
be received by mobile battery powered terminals. Robust for transmissions on radio frequency band III (VHF) and
reception at mobile speeds of up to several 100 KM/h is L Band. The stream is Reed Solomon FEC encoded (an
achievable. additional FEC layer to audio DAB) with a parity word of 16
bytes length; convolutional interleaving is then applied.
The standard has deployed a series of techniques to reduce
power and increase robustness of reception, including time In order to diminish the channel effects such as fading and
slicing of transmissions and an IP based transmission link with shadowing, the DMB modem uses OFDM-4DPSK modulation.
a Multiprotocol Encapsulation forward Error Correction (MPE- T-DMB is an ETSI standard (TS 102 427 and TS 102 428).
FEC). Its technical characteristics are:
T-DMB uses MPEG-4 Part 10 (H.264/AVC) for the video and
Time slicing. MPEG-4 Part 3 BSAC or HE-AAC V2 for the audio. The audio
and video is encapsulated in MPEG-2 Transport Streams.
MPE IP transmission stream.
MPE FEC (Optional). 9.11.3 DAB IP

Deep interleaving of data. DAB IP is a transmission system again derived from the
Eureka 147 DAB audio standard, but using the ETSI DAB
2K, 4K & 8K FFT Mode.
standard EN 300 401 v1.4.1 for transmission using Internet
Transmission Parameter Signaling (TPS) bits. Protocol (IP) as the transport mechanism.
The video coding standard used for DVB-H transmission has DAB IP has the advantage of having spectrum in Europe
been left for the network operator to select from DVB approved available and the ability to use the existing DAB audio
CODECs. In practice, SMPTE VC-1 and MPEG-4 Part 10 infrastructure for transmission hence saving network rollout
(H.264/AVC) have predominated in operational systems. cost. The downside is the limited RF channel bandwidth
common to all DAB based technologies of 1.7 MHz which
DVB-H has been specified to operate in both the existing TV
limits the bit rate and therefore number of TV channels that
VHF and UHF bands and the L Band with channel bandwidths
can be carried. In addition to TV services DAB radio is also
of 5,6,7 and 8 MHz. While the UHF band is broadly preferred
provided by most handsets.
spectrum, commercial issues have seen networks deploying in
the L Band successfully. IP transport has been chosen for its flexibility and the ability to
integrate with existing transmission networks and offer future
ETSI ratified standards for DVB-H include amongst others EN
migration to higher bandwidth radio transmission systems.
302 304 Transmission System for Handheld Terminals (DVB-H)
and TR 102 377 DVB-H Implementation Guidelines standards.

58 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

MPE-FEC RS FEC Convolutional FEC Modulation, Mobile Convolutional FEC RS FEC MPE-FEC
RS Encoder Encoder Encoder Channel & Demodulation Decoder Decoder RS Decoder

DVB-H only

DVB-T, DVB-H & DMB

DAB, DMB, DVB-T and DVB-H

Figure 9-11-4. Forward error correction and redundancy diagram for RF transmission.

Parameter T-DMB DVB-H

Channel Bandwidths (MHz) 1.712 5, 6, 7, 8
Error Protection Convolutional code + RS FEC Convolutional code + RS FEC+ MPE-FEC
MPE-FEC code rate No MPE-FEC Free selection (most likely 1⁄2 to 7⁄8)
Time interleaving 384 ms Practically up to 1000 ms depending on MPE-FEC selection,
typically 200-500 ms
Time Slicing Micro time slicing Time slicing
Protocol stack Raw MPEG-4 (i.e. no IP layer) IP Layer
Practical data rate 1.06 (@ 1.712 MHz channel, 1⁄4 GI 1⁄2 CR + RS-FEC) 3.32 –13.8 (@ 8 MHz channel, 1⁄4 GI QPSK 1⁄2 CR MPE-
FEC 2⁄3 – 1⁄8 GI 16QAM 3⁄4 CR MPE-FEC 5⁄6)
FFT Sizes 2 k, 1 k, 0.5 k, 0.25 k 8 k, 4 k, 2 k
Guard Intervals (us) 246, 123, 62, 31 224, 112, 56, 28, 14, 7
Inner Modulations Differential-QPSK QPSK, 16QAM, 64QAM
Convolutional code rates 1⁄4, 3⁄8, 1⁄2, 3⁄4 4⁄9, 4⁄7, 2⁄3, 4⁄5, 1⁄2, 2⁄3, 3⁄4, 5⁄6, 7⁄8
Figure 9-11-5. T-DMB and DVB-H comparison chart.

[Link]/video_audio 59
Primer

Mode Mode 1 Mode 2 Mode 3

Number of Segments 1 or 3
Segment Bandwidth 6000/14 = 428.57 kHz
Bandwidth 432.5 kHz (1 segment) 430.5 kHz (1 segment) 429.5 kHz (1 segment)
1.289 MHz (3 segments) 1.287 MHz (3 segments) 1.286 MHz (3 segments)
Carrier Spacing 250⁄63 = 3.968 kHz 125⁄63 = 1.984 kHz 125⁄126 = 0.992 kHz
Number of Carriers 109 217 433
Carrier Modulation QPSK, 16QAM, 64QAM, DQPSK
Effective Symbol Duration 252 µs 504 µs 1008 µs
Guard Interval 63 µs (1⁄4) 31.5 µs (1⁄8) 126 µs (1⁄4 )63 µs (1⁄8) 252 µs (1⁄4) 126 µs (1⁄8)
15.75 µs (1⁄16) 7.875 µs (1⁄32) 31.5 µs (1⁄16) 15.75 µs (1⁄32) 63 µs (1⁄16) 31.5 µs (1⁄32)
Inner Code Convolutional Code (1⁄2, 2⁄3, 3⁄4, 5⁄6, 7⁄8)
Outer Code Reed-Solomon (204, 188)
Interleaving Time and Frequency
Information Bit Rates (TS Rates) 280.85 kbps (DQPSK, Inner Code = 1⁄2, GI = 1⁄32) -1.787 Mbps (16QAM, Inner Code = 7⁄8, GI = 1⁄4) (1 segment)
0.842 Mbps (DQPSK, Inner Code = 1⁄2, GI = 1⁄32) -5.361 Mbps (16QAM, Inner Code = 7⁄8, GI = 1⁄4) (3 segments)
Figure 9-11-4. Three modes used by ISDB-T modulation (often, all three at the same time).

9.11.4 FLO 9.11.5 ISDB Single Segment

Forward Link Only – FLO is Qualcomm’s proprietary mobile TV ISDB Single Segment transmission is the Japanese standard
standard. The system is being submitted to standards bodies for mobile TV transmission and unlike the other mobile
for ratification and is expanding from its initial deployment in standards was designed into the Japanese digital terrestrial
the USA. FLO is again based on OFDM modulation using a transmission standard from inception, as part of its hierarchical
4 K FFT, but uses hierarchical modulation to provide service mode.
coverage to the edges of the transmission area with a reduced
ISDB single segment utilizes the fact that ISDB-T splits the
bit rate.
transmission channel into 13 segments each of which is an
FLO also uses time interleaving to reduce susceptibility to OFDM block. A transport capacity of 280.85 Kbps to 1.17873
impulsive interference and turbo coding for forward error Mbps is available per block depending on the chosen coding
correction (see section 10 on FEC). Its technical characteristics rate.
are:
The transmission parameters for ISDB are specified on a
OFDM 4 K FFT mode based. block by block basis, so that for mobile transmission a robust
channel can be coded. This feature is used with the central
Hierarchical modulation.
single segment (ISDB-Tss) to provide portable reception.
Service to the edge of the cell.
Time Interleaving.
Turbo coding.

60 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

Figure 10-1. Multiplexing, modulation, and transmission.

Section 10 – Introduction to DVB network operators, who will use an additional layer of error
correction as needed (error correction strategies are selected
& ATSC depending on the transmission channel). This layer should be
MPEG compression is already being used in broadcasting and transparent to the destination.
will become increasingly important in the future. This section A particular transmitter or cable operator may not want all of
discusses the additional requirements for digital television the programs in a transport stream. Several transport streams
broadcasting, as implemented by the two principal DTV may be received and a selection of channels may be made
Standards. and encoded into a single output transport stream using a
remultiplexer. The configuration may change dynamically.
10.1 An Overall View
Broadcasting in the digital domain consists of conveying the
ATSC (Advanced Television Systems Committee) is a U.S.
entire transport stream to the viewer. Whether the channel
organization that defines standards for terrestrial digital
is cable, satellite or terrestrial, the problems are much the
broadcasting. DVB refers to the Digital Video Broadcasting
same. Metadata describing the transmission must be encoded
Project and to the standards and practices established by the
into the transport stream in a standardized way. In DVB, this
DVB Project. This project was originally a European project,
metadata is called service information (DVB-SI) and includes
but produces standards and guides accepted in many areas
services such as teletext as well as details of programs carried
of the world. These standards and guides encompass all
both within itself and within other multiplexes.
transmission media, including satellite, cable and terrestrial
broadcasting. In broadcasting, there is much less control of the signal
quality and noise or interference is a possibility. This requires
Digital broadcasting has different distribution and transmission
some form of forward error correction (FEC) layer. Unlike the
requirements, as is shown in Figure 10-1. Broadcasters will
FEC used by the telecommunications network operators,
produce transport streams that contain several television
which can be proprietary, (or standardized as per European
programs. Transport streams have no protection against
Telecommunications Standard Institute (ETSI), which defines
errors, and in compressed data, the effect of errors is
DVB transmission over SDH and PDH networks), the forward
serious. Transport streams need to be delivered error-free
error connection (FEC) used in broadcasting must be
to transmitters, satellite uplinks and cable head ends. In this
standardized so that receivers will be able to handle it.
context, error free means a bit error rate (BER) of 1 in 10-11 or
better. This task is normally entrusted to telecommunications

[Link]/video_audio 61
Primer

The addition of error correction obviously increases the bit rate guide (EPG) information, such as the nature of a program, the
as far as the transmitter or cable is concerned. Unfortunately, timing and the channel on which it can be located, and the
reliable, economical radio and cable-transmission of data countries in which it is available. Programs can also be rated
requires more than serializing the data. Practical systems so that parental judgment can be exercised.
require channel coding.
DVB-SI must include the following tables over and above
10.2 Remultiplexing MPEG-PSI:

This is a complex task because a remultiplexer has to output Network Information Table (NIT). Information in one
a compliant bit stream that is assembled from parts of others. transport stream that describes many transport streams.
The required data from a given input transport stream can The NIT conveys information relating to the physical
be selected with reference to the program association table organization of the multiplex, transport streams carried via a
and the program map tables that will disclose the PIDs of the given network and the characteristics of the network itself.
programs required. It is possible that the same PIDs have Transport streams are identified by the combination of an
been used in two input transport streams; therefore, the PIDs original network ID and a Transport Stream ID in the NIT.
of one or more elementary streams may have to be changed. Service Description Table (SDT). Each service in a DVB
The packet headers must pass on the program clock transport stream can have a service descriptor and these
reference (PCR) that will allow the final decoder to recreate a descriptors are assembled into the service description
27 MHz clock. As the position of packets containing PCR may table. A service may be television, radio or teletext. The
be different in the new multiplex, the remultiplexer may need service descriptor includes the name of the service provider.
to edit the PCR values to reflect their new position on the time
Event Information Table (EIT). EIT is a table for DVB that
axis.
contains program names, start times, durations and so on.
The program map tables and program association tables
Time and Date Table (TDT). The TDT is a table that embeds
will need to be edited to reflect the new transport stream
a UTC time and date stamp in the transport stream.
structure, as will the conditional access tables (CAT).
DVB-SI also defines other optional tables including: bouquet
If the sum of the selected program stream bit rates is less
association table (BAT), running status table (RST), time offset
than the output bit rate, the remultiplexer will create stuffing
table (TOT) and the stuffing table (ST).
packets with suitable PIDs. However, if the transport streams
have come from statistical multiplexers, it is possible that The ATSC, like DVB, used the MPEG-2 private section table
the instantaneous bit rate of the new transport stream will to define several new tables. This set of new mandatory tables
exceed the channel capacity. This condition might occur defined by ATSC in A/65A is part of the program and system
if several selected programs in different transport streams information protocol (PSIP). ATSC PSIP must include the
simultaneously contain high entropy. In this case, the only following tables over and above the MPEG-PSI:
solution is to recompress and create new, shorter coefficients Terrestrial Virtual Channel Table (TVCT) defining, at a
in one or more bit streams to reduce the bit rate. minimum, MPEG-2 programs embedded in the transport
stream in which the TVCT is carried.
10.3 Service Information (SI)
Master Guide Table (MGT) defining the type, packet
In the future, digital delivery will mean that there will be a large
identifiers and versions for all the other PSIP tables in the
number of programs, teletext and services available to the
transport stream, except for the system time table (STT).
viewer and these may be spread across a number of different
transport streams. Both the viewer and the integrated receiver Rating Region Table (RRT) defining the TV parental guideline
decoder (IRD) will need help to display what is available and to system referenced by any content advisory descriptor
output the selected service. This capability requires metadata carried within the transport stream.
beyond the capabilities of MPEG-PSI (program specific System Time Table (STT) defining the current date and time
information) and is referred to as DVB-SI (service information). of day.
DVB-SI is considered to include the NIT, which is optional in
Event Information Table (EIT-n) defining the first four Event
MPEG transport streams.
Information Tables (EIT-0, EIT-1, EIT-2 and EIT-3) describing
DVB-SI is embedded in the transport stream as additional 12 hours of events (TV programs), each with a coverage of
transport packets with unique PIDs and carries technical 3 hours, and including all of the virtual channels listed in the
information for IRDs. DVB-SI also contains electronic program TVCT.

62 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

10.4 Error Correction

Error correction is necessary because conditions on long
transmission paths cannot be controlled. In some systems,
error detection is sufficient because it can be used to request
a retransmission. Clearly, this approach will not work with real-
time signals such as television. Instead, FEC is used in which
sufficient extra bits, known as redundancy, are added to the
data to allow the decoder to perform corrections in real time.
Briefly, R-S codes (a special case of the general BCH codes) Figure 10-2. Large RAM buffer for byte re-ordering.
add redundancy by adding extra parity data bytes (N bytes
for example) to a block of data bytes so that the R-S decoder
in the receiver can correct any or all of the bits in each (N/2)
byte(s) in the block of bytes. The R-S decoder also can
indicate erred bytes that it cannot correct as long as the bytes
in error in each block are not excessive. If none of the bits of
any of the bytes within the block (including the extra parity
bytes) are in error a zero byte-error syndrome is returned. This
zero condition is obtained irrespective of the data and makes
checking easy. In Transport Streams, the packets are always
188 bytes long prior to the addition of error-correction data.
The addition of 16 bytes of R-S redundancy (parity) produces Figure 10-3. Sequentially filling the array (left to right, top down).
a packet length of 204 bytes. In practice, transport streams
may use 204 byte packets even when FEC is not present. The
use of 16 stuffing bytes avoids reclocking the stream when
FEC is added or deleted. Note that the R-S parity bytes do not
always have to be 8 bits to match the packet data byte size.
For example, QAM-J83B uses a 7 bit R-S byte size on blocks
of 8 bit data bytes so there is not a 1:1 correspondence to the
packet data bytes.
In the event that the syndrome is non-zero, solving the
simultaneous equations will result in two values needed
for error correction: the erred byte and the bits within that Figure 10-4. Sending out data using vertical columns. With FEC, a burst of lost symbols
byte that are in error. However, if the number of erred bytes will not be impacted.
exceeds half the number of parity bytes, the bits in error
cannot be corrected. Unfortunately, in typical transmission
channels, the signal quality is statistical. This means that while data enters the RAM in rows and is then read out in columns.
single bits may be in error due to noise, on occasion a large The reordered data are now transmitted. Upon reception, the
number of bits, known as a burst, can be corrupted together. data are put back to their original order, or de-interleaved, by
This corruption might be due to lightning or interference from using a second RAM. The result of the interleaving process
electrical equipment and will overwhelm the block of R-S is that a burst of errors in the channel after de-interleaving
coded data and easily exceed the (N/2) number of correctable becomes a large number of single-byte errors in each R-S
bytes. data block spanning a large number of R-S data blocks, which
It is not economic to protect every code word against such are more readily correctable.
bursts, because they do not occur often enough. The solution When a burst error reaches the maximum correctable size,
is to use a technique known as interleaving. Figure 10-2 the system is vulnerable to random bit errors that make code
shows that when interleaving is used, the source data are words uncorrectable. The use of an inner code applied after
FEC coded, but prior to transmission, they are fed into a RAM interleave and corrected before de-interleave can prevent
buffer. Figure 10-3 shows one possible technique in which random errors from entering the de-interleave memory.

[Link]/video_audio 63
!"#$%&'()"*+",-%

Primer

Input Data
Soft In/ Soft In/ Output Data
Deinterleaver
Soft Out Soft Out

Interleaver

Figure 10-5. Interleaver diagram.

As Figure 10-3 shows, when this approach is used with a block At the decoder the front end receiver produces two streams
interleave structure, the result is a product code. Figure 10-4 of integers from the two n/2 parity streams for each bit of the
shows that interleave can also be convolutional, in which the data stream which corresponds to the likelihood that the bit
data array is sheared by applying a different delay to each row. is a 0 or 1. This is referred to as the soft bit. This integer data
Convolutional (Forney), or cross interleave, has the advantage is in the range -127 to +127 with -127 meaning certainly 0,
that less memory is needed to interleave and de-interleave. -100 means very likely 0, 0 means could be 1 or 0 and similar
values of + probability for 1 bits.
10.4.1 Turbo Coding
Key to turbo code is the use of the two likelihood streams
Turbo coding is a class of recently developed (1992) error
generated by the receiver to resolve differences between
correction codes designed to achieve maximal information
the two decoded parity bit streams. Each of the two parity
transfer over a limited bandwidth channel which is subject to
decoders generates a hypothesis for the payload sub block
noise.
pattern of m bits. The hypothesis bit patterns are compared
The key difference from established error coding techniques and if they differ the two decoders exchange the likelihoods
is that the receiver is designed to provide not only a received they have for each bit in the hypothesized m bit payload. Each
data bit stream, but also a likelihood measure for each bit decoder then incorporates the derived likelihood estimates for
being correct. the bits from the other decoder to generate a new hypothesis
At the transmit encoder, data is split into three sub blocks of for m. They then compare the new hypothesis until agreement
bits, as shown in Figure 10-5. The first sub block of m bits is is achieved.
the data payload. The second sub block is n/2 parity bits for This iterative process continues until the decoders reach a
the payload data computed using a Recursive Systematic converged solution for m on which they agree to be correct.
Convolution (RSC) code. The third block is n/2 parity bits A characteristic of some Turbo decoders is that the corrected
calculated for a known permutation of the payload data. BER can have a floor where the BER does not improve with
The transmitted data is therefore m+n bits and includes two increases in S/N but this BER is typically designed to be much
redundant but different sub blocks of parity data. better than the required BER.

64 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

Figure 10-6. QAM modulator.

Figure 10-7. PRBS used with input stream to disperse energy.

10.5 Channel Coding; Energy Dispersion little energy and be susceptible to interference. In practice,
randomizing is necessary to de-correlate the transmitted
Raw serial binary data is unsuitable for transmission for
spectrum from the data content. Figure 10-7 shows that when
several reasons. Runs of identical bits cause DC offsets and
randomizing or energy dispersal is used, a pseudo-random
lack a bit clock. There is no control of the spectrum and
sequence is added to the serial data before it is input to the
the bandwidth required is too great. In practical radio and
modulator. The result is that the transmitted spectrum is noise-
cable systems, a modulation scheme called a channel code
like with relatively stationary statistics. Clearly, an identical and
is necessary. Digital modulations schemes are discussed
synchronous sequence must be subtracted at the receiver
in Section 9. Figure 10-6 shows the application of these
as shown. Randomizing is typically (DVB and ATSC) not
principles to a 64QAM modulator.
applied to sync patterns like the TS sync bytes, to simply
In the schemes described above, the transmitted signal detection but this is not always true. For example the new
spectrum is signal dependent. Some parts of the spectrum China standard does randomize the TS sync bytes (payload
may contain high energy and cause interference to other agnostic) and determines frame timing from the symbol
services, whereas other parts of the spectrum may contain synchronization.

[Link]/video_audio 65
Primer

Figure 10-8. Convolutional coding.

10.6 Inner Coding In Trellis coding, the data are coded such that only certain
Trellis waveforms represent valid data. If only 64 of the Trellises
The inner code of a FEC system is designed to prevent
represent error-free data, then two data bits per symbol
random errors from reducing the power of the interleave
can be sent instead of three. The remaining bit is a form of
scheme. A suitable inner code can prevent such errors by
redundancy because Trellises other than the correct 64 must
giving an apparent increase to the SNR of the transmission.
be due to errors. If a Trellis is received in which the level of one
Typical inner codes are convolution codes where, for example,
of the symbols is ambiguous due to noise, the ambiguity can
an extra bit is added based on the previous bit sequence. The
be resolved because the correct level must be the one which
ratio of data bits to total bits is called the convolution code
gives a valid trellis. This technique is known as maximum-
rate. In Trellis coding (developed by G. Ungerboeck in 1982),
likelihood decoding made famous by A. Viterbi.
which can be used with multi-level signaling, the convolution
code is combined with the symbol constellation mapping The 64 valid Trellises should be made as different as possible
where several multi-level symbols are associated into a group. to make the system continue to work with a poorer signal-to-
The waveform that results from a particular group of symbols noise ratio. If the Trellis coder makes an error, the outer code
is called a Trellis. If each symbol can have eight levels, then will correct it.
in three symbols there can be 512 possible trellises. Prior to In DVB, standard Viterbi inner convolutional coding may be
Ungerboeck the inner convolution coding was independent used. Figure 10-8 shows that following interleave, the data are
of the symbol mapping or constellation, as in Figure 10-8. fed to a shift register. The contents of the shift register produce
By combining the processes, a substantial improvement in two outputs that represent different parity checks on the input
effective coding gain or S/N improvement was achieved. data so that bit errors can be corrected. Clearly, there will be
two output bits for every input bit; therefore the coder shown
is described as a 1/2 rate coder. Any rate between 1/1 and
1/2 would still allow the original data to be transmitted, but
the amount of redundancy would vary. Failing to transmit the
entire 1/2 output is called puncturing and it allows any required
balance to be obtained between bit rate and correcting power.

66 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

Figure 10-9. ATSC 8VSB RF from TS to modulation and upconversion.

Figure 10-10. DVB COFOM RF from TS to modulation and upconversion.

10.7 Transmitting Digits Figure 10-10 shows a DVB-T transmitter. Service information
is added as before, followed by the randomizing stage for
Figure 10-9 shows the elements of an ATSC digital transmitter.
energy dispersal (except for the packet 0x47 sync byte). Outer
Service Information describing the transmission is added to
R-S check symbols are added prior to interleaving. After the
the Transport Stream. This stream (except for the packet
interleaver, the inner coding process takes place, and the
0x47 sync byte) is then randomized prior to routing to an
coded data is fed to a COFDM modulator. The modulator
outer R-S error correction coder that adds redundant bytes
output is then upconverted to produce the RF output.
to the data of each TS packet. A convolutional interleave
process then reorders the data so that adjacent data in the At the receiver, the bit clock is extracted and used to control
Transport Stream are not adjacent in the transmission. An the timing of the whole system. The channel coding is
inner trellis coder and interleave is then used to produce a reversed to obtain the raw data plus the transmission errors.
multi-level signal for the vestigial sideband (VSB) modulator. The inner code corrects random errors and may identify
In ATSC the 0x47 sync byte is actually never coded and sent larger errors to help the outer coder after de-interleaving. The
by the transmitter. In place of that byte a special, 2-level, sync randomizing is removed and the result is the original transport
pulse is sent and the receiver always inserts a 0x47 into that stream. The receiver must identify the PAT, the Service
position so it does not have to decode the sync byte from the Information (SI) and PMT that the PAT points to so the viewer
randomization, R-S coding and Trellis coding. can be told what is available in the Transport Stream and the
selected program can be located in the multiplex.

[Link]/video_audio 67
Primer

Section 11 – Data Broadcast 11.1.1 Program Related Data

The previous sections have looked at the basics of an MPEG- The base MPEG-2 specification does not have any provision
2 transport stream and their main application of carrying for an EPG that can give information about the TV channels or
compressed video and audio streams, similar to conventional individual programs being carried on the transport stream. The
analog broadcasts. However one of the major advantages of regional variants ATSC, DVB and ARIB have used the MPEG-2
an MPEG-2 transport stream is that it can carry data as well private table syntax such as the EIT to provide additional data
as video and audio. Although analog television systems can about programs. These tables are required to be broadcast
and do carry data, their maximum data bandwidth is severely at regular intervals and give start times, synopsis and other
limited compared with the bandwidths possible on a transport information about services and programs.
stream. However even this level of information may not be enough and
This section will provide an overview of the different methods so there are a number of proprietary EPGs that extend the
provided by MPEG-2 and the regional variants (DVB, ATSC amount of information available and also provide facilities like
and ARIB (Association of Radio Industries and Businesses)) to enhanced searching, favorite channels and other augmented
encapsulate data within a transport stream. The next section services. These enhanced EPGs are usually permanently
will then deal with how the data is presented to the viewer by resident on the set top box and use a combination of the
the set top box (which does not strictly fall within the scope of standard table information and proprietary data formats.
MPEG-2). Usually this sort of program information is not time critical and
so cheap low bit rate techniques are used. In many cases
11.1 Applications the data is designed to be cached by the set top box so it
appears to be immediately available to the user, even if the
There are many different types of applications for data
actual total transmission time is relatively long.
broadcast over a transport stream, and each application
type may require different types of data with different timing 11.1.2 Opportunistic Data
requirements. For example, the type of data involved in
It is rarely possible to utilize the complete bandwidth of a
sending Internet traffic is very different from that needed
transport stream with video and audio streams, not least
to provide a firmware update for a set top box. A non-real
because of the need to handle the complex constraints of
time update of pricing information has very different timing
remultiplexing or table insertion. Opportunistic data systems
requirements from a quiz application where answers and
make use of this spare capacity by replacing some null
questions must be sent in close synchronization with video/
packets with useful data. However the bandwidth of this data
audio.
cannot be guaranteed and may be very small. Hence it can
MPEG-2 provides a large variety of different techniques to only be used for applications with no real-time constraints.
send data. The choice of technique is a trade-off between
Some applications using this sort of data could be file transfers
optimizing bandwidth (and hence the cost of providing
such as price lists or stock level data distribution via satellite to
the service) while meeting the timing requirements of the
all company locations. The only constraint is that the transfer
application.
must take place overnight and so the low bandwidth and
The types of applications can be loosely grouped by their real- hence cheap opportunistic data services can be used.
time requirements and level of interactivity as described in the
next sections.

68 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

11.1.3 Network Data 11.2 Content Encapsulation

A very common application is to simply use the broadcast The first stage in a data broadcast is to encapsulate the
transport stream as a carrier of network data. In the simplest data into a form suitable for transmission on a transport
case, a satellite link would provide a very high bandwidth stream. There are a wide variety of different data types and
interconnect between two geographically separated so the MPEG-2 standards provide a number of different
companies. This network link could be used to carry virtually encapsulation methods. The various country specific
any type of network protocol. standards such as the DVB and ATSC have further enhanced
However the most common current use is for high speed the basic MPEG-2 options to meet regional requirements, but
Internet downloads using just the IP protocol. In this scenario these all build upon the core MPEG-2 standards.
a low-speed Internet connection is used to send commands 11.2.1 MPEG Data Encapsulation
and page requests to an Internet server. However the Internet
server will then route any requested data via the satellite link at [Link] Data Piping
speeds much higher than that possible by an ISDN or average Data piping is used for simple asynchronous delivery of data
broadband connection. to a target set top box on the network. Its main application is
A variant on this application is used when only a slow back to send proprietary data in closed systems where the target
channel is available. In this case, for example, data related to set top box has been pre-programmed to look for specific
the current TV program can be delivered on demand over the data on certain PIDs. Data is carried directly in the payload of
transmitted signal without viewers being aware that they are MPEG-2 TS packets without any timing information. A typical
accessing an Internet site. application might be a nightly update of product stock levels
to all sales outlets throughout a region.
11.1.4 Enhanced TV
[Link] Data Streaming
In an enhanced TV application there is no back channel,
and so all data required must be sent over the transmitted Data streaming is used for the end-to-end delivery of data in
transport stream. Although this imposes obvious constraints asynchronous, synchronous or synchronized fashion. Data is
on the design and range of an application, it is still possible to carried as PES packets in a similar method to video and audio
produce a very wide range of games, quizzes and infomercials services. In fact, video and audio are really specific examples
where the viewer may be completely unaware that all of a synchronous data streaming service. As PES packets are
interaction is with the TV only. This is especially the case with signaled in the service information (SI) tables, and can carry
pre-recorded material, where it is relatively straightforward timing information, this approach is more flexible than data
to synchronize data pop-ups, such as quiz questions and piping but in practice is used in a similar fashion in proprietary
answers, with the audio/video. closed systems.

11.1.5 Interactive TV
The key difference between enhanced and interactive TV
is that interactive TV has a back channel to send or receive
highly personalized data. This greatly increases the range
of applications that can be supported, including the ability
to provide real time interaction with other viewers, such as
multi-player gaming or voting, as well as truly personalized
feedback.
A further extension of this is when high-speed Internet
connection is combined with a true back channel. This offers
seamless integration of personalized data with broadcast
television, while also permitting true VOD delivery of
programming or other material.

[Link]/video_audio 69
Primer

[Link] DSMCC – Digital Storage Medium Command [Link] Carousels

and Control
Carousels are intended for the periodic transmission of
The MPEG-2 DSM-CC specification (ISO/IEC 13818-6) information over a transport stream. Although the content
provides further ways of broadcasting data in the sections of a carousel can be changed in response to a request from
of a standard MPEG-2 private table. It was originally devised a target user it is more usual for the carousel to be regularly
as a way of supporting VOD delivery of program material repeated regardless of whether any target is listening or needs
across a network on a Transport Stream. The protocol has that data at that moment. A target that needs a specific data
been extended to be able to cope with both on-demand item is expected to simply wait until it is retransmitted.
delivery (using the MPE paradigm) as well as periodic delivery
There are two different types of carousels, object carousels
(using the carousel paradigm) of data across multiple network
and data carousels, and the main differences between them
providers
are that:
[Link] MPE – Multi-protocol Encapsulation Data carousels contain only modules of data of unspecified
Multi-protocol encapsulation (MPE) allows a datagram of any content; it is up to the target to know what to do with the
communication protocol to be transmitted in the section of a data it receives.
DSM-CC table via a Transport Stream. A datagram is a logical Object carousels contain identifiable data objects such
structure that contains all defining information about the data, as pictures, text files, or executable application files and
i.e., its size and contents, where it should be going and how it contains a directory listing of all objects in the carousel.
should get there.
Data carousels are often used for downloading new system
The most common application is Internet traffic where the software to a set top box whereas an object carousel is used
TCP/IP datagram carries information about the logical (IP) for shopping services, EPGs and to send applications and
addresses of the source and destination (target) as well as games.
the Media Access Control (MAC) address (a unique network
In both data and object carousels, items are repeated at
address) of the target. However MPE supports nearly any type
periodic intervals. However, object carousels make it easy to
of network protocol and is certainly not restricted to only TCP/
vary the repetition rate of individual objects. For example, the
IP data. Indeed, The DVB-H standard makes use of MPE to
EPG for the next hours viewing may repeat far more often than
stream audio and video elementary streams to mobile phones.
that for next month. The repetition rates for objects may be a
DVB-H typically uses the UDP/IP protocols instead of TCP/
commercial decision made by the service provider to maximize
IP, as the data is being broadcast. Many implementations also
bandwidth utilization.
use the real-time protocol (RTP) to facilitate audio and video
synchronisation (lip sync). An example DVB-H protocol stack Both object and data carousels are based upon the DSM-CC
is illustrated in Figure 11-1. extensions to the MPEG-2 specification ISO13818-6, with
specific extensions for the DVB, ARIB and ATSC systems.

70 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

Figure 11-1. MPEG Data Carousel structure.

[Link] Data Carousels If there are more related modules than can be grouped
together into a single DII message then a Supergroup can be
A data carousel does not contain any individual data items or
formed from multiple DII messages. These are referenced by a
directory structure but a single monolithic chunk of data. It is
DownloadServerInitiate (DSI) message.
up to the target user to know what the data is and what to do
with it. A one-layer data carousel contains a small number of modules
referenced in a single DII.
The structure is shown in Figure 11-1. A complete single
item of data is defined as a “module.” Transmission modules A two-layer data carousel contains DSI messages referencing
are split up into one or more blocks. Each block is sent as one or more DII messages. It may be smaller or larger than a
a section in the payload of a DownloadDataBlock (DDB) single carousel.
message, which follows the MPEG-defined private table A typical use for a 2-layer carousel would be for multi-
syntax. DDB messages can be sent in any order or at any
periodicity; hence a mechanism is needed to identify which language support. One group might convey the executable
DDBs belong to what modules. program in one module along with English text in a second
module. The second group could then just carry a single
A DownloadInfoIndication (DII) message is used to link the module with just French text, saving the overhead of repeating
DDBs for a module together. The information for more than the application module.
one module can be in a single DII message; this forms a
Group. Usually a group will contain logically related data
modules.

[Link]/video_audio 71
Primer

Figure 11-2. MPEG Object Carousel structure.

[Link] Object Carousels In essence, a BIOP is a method to exchange information

about an object being broadcast in the carousel. The BIOP
Object carousels are used to broadcast individually identifiable
may contain the object or may simply provide a pointer to the
items of identified data from a server to a receiver. These items
object. The BIOP may also indicate how to use the object,
are called objects and may be pictures, text files, programs,
including providing a link to where to download the application
a pointer to a video PID, a directory listing or service gateway
software needed to use the object.
of what is available in the carousel. Related objects grouped
and sent together as a single carousel form a service domain. Object carousels are similar to data carousels in that groups
Objects can be sent as often as required and different objects of objects are combined together to form modules. The basic
may have very different repetition rates. data carousel methodology is then used to transmit that data
using blocks, modules and DIIs. The key difference is that the
A key feature of object carousels is that all objects are sent
DSI is used to point directly to the Service Gateway directory
using the BIOP (broadcast inter-ORB protocol) paradigm.
object, which can then be used to find all other objects in the
Conventional software developers have been using ORB
carousel. This arrangement is shown in Figure 11-2.
(object request brokerage) for many years. BIOP extends the
basic system to support identifying and using objects in a
broadcast environment across different networks from different
service providers.

72 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

11.2.2 DVB Data Encapsulation Secondly, the A/90 uses only data carousels that can
effectively contain only a single object. The data carousel
DVB has adopted the standard MPEG-2 encapsulation
format has been extended to optionally include a DSM-
methods with only very minor variations, mainly to remove
CC adaptation header that may contain a PTS to enable
possible ambiguities that emerged from the original
synchronous data delivery. The MPEG concept of NPT within
specifications (EN 301 192). These include specifying a
a program is not supported. A/90 also defines additional
slightly different MPE format and imposing restrictions on
data types including its own version of DSM-CC MPE as well
the maximum data PES packet size. DVB has also taken
allowing IP data to be streamed using the PES format.
responsibility for ensuring that there can be no ambiguity
between data services that use IORs by providing unique Mainly due to its lack of support for object carousels, there
allocation of network IDs and server IDs. has been little adoption of A/90.
DVB has also defined two specific data streaming PES 11.2.4 ARIB Data Encapsulation
formats for DVB subtitling and for DVB teletext and have a
The Japanese ARIB B24 standard is arguably the simplest
standardized format for using data carousels to transmit set
of all systems. It does not support data piping or object
top box (STB) firmware updates. The DVB have also defined
carousels. Instead it uses the data carousels format to
a number of data descriptors and additional tables that are
send one or more entities by imposing a specific directory
discussed in the signaling and announcement sections below.
structure upon the data carousel content. ARIB does not allow
Most useful of these additions is the ability to use a descriptor
references to any entities that are not directly referenced in the
to provide a simple mask for the MAC addresses in an MPE
PMT and so has no need for either CORBA (common object
datagram. This enables simple support for narrow, multi or
request broker architecture) or NRT type linkages.
broadcast MPE services.

11.2.3 ATSC A/90 Data Encapsulation

The ATSC A/90 specification includes some significant
differences from both the DVB and the MPEG-2 standards.
First, A/90 does not use object carousels and the CORBA/IOR
system at all but instead uses a separate table, the Network
Resources Table (NRT), to provide the binding information
that links a referenced datum with its real location. Apart from
arguably simplifying the system operation it also enables direct
support for Internet URLs that is not supported by DVB or
MPEG.

[Link]/video_audio 73
Primer

Figure 11-3. Time division multiplexing DSMCC sections in TS packets.

11.3 Broadcasting Data Content An IOR can contain more than one profile body if the object is
available on more than one carousel and the set top box can
11.3.1 How Object Carousels Are Broadcast choose the easiest/quickest one to access.
A full explanation is beyond the scope of this document; the Taps are used to solve the problem that the actual PIDs
following description is a brief and much-simplified overview. used to broadcast DIIs, DDBs and video/audio streams are
(Also see Figure 11-3.) not known until immediately before transmission. Within the
Directory, file and stream objects are sent in the same method carousel therefore all references to PIDs are only made in
as data carousels i.e., in modules split into blocks are sent as terms of a tap; the association between a tap and a real PID
sections in the payload of a DownloadDataBlock (DDB). is made in the SI. This vastly simplifies re-multiplexing streams
across different networks.
A DownloadServerInitiate (DSI) message contains the location
of a special directory object called the service gateway. DSI 11.3.2 MPEG-2 Data Synchronization
messages are referenced in the SI and so form the starting
There is a need for data broadcasts to be synchronized in
point to work out what is in a specific object carousel. The DSI
some way with programs being broadcast. It is not really
references the DownloadInfoIndication (DII) that references the
practical to use the real-time delivery of a datum as the
DDB that contain the module in which the service gateway
synchronization method, except in very non-critical real time
object is sent.
applications such as updating sports scores where a few
Objects are referenced in a directory object using IORs (inter- seconds or more error is of no practical significance. However
operable object references). This contains all the information even a second or two could have a very big impact on, for
needed to access an object in the same service domain or on example, quiz shows where revealing the answer too early
another object carousel (including those broadcast on other could have serious consequences.
Transport Streams).
MPEG-2 provides different timing mechanisms for the different
The name given to the structure in the IOR that describes the types of data encapsulation. Data piping and MPE do not
location of an object is called a profile body that comes in two support any form of timing mechanism apart from near real-
flavors: time delivery of the data.
BIOP profile body – used only for objects within this service
domain.
Lite Options Profile Body – used for objects on other servers
or transport streams.

74 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

Data streaming PES packets can contain presentation The only new table is the application information table, which
time stamp (PTS) and possibly decoding time stamp (DTS) is defined for use with MHP services only. This contains a list
timestamps for synchronization with the system clock derived of all applications within a carousel, a pointer to their boot
from the PCR values in the stream. The mechanism is exactly class and optionally allows applications to be auto-started or
the same as for video or audio synchronization and decode. killed upon a channel change.
MPEG-2 data carousels have no timing information. However, [Link] ATSC Announcement
object carousels can contain a special object called a “stream
In contrast to the DVB the A/90 specification treats data
event” which contains timing information relative to the normal
events and services totally separately from video and audio
play time (NPT) of an individual television program. The NPT
services and has added a number of tables to support this
is not the same as the PCR as the NPT clock can be paused
paradigm. These are the data services table (DST), which list
during, for example, a commercial break. In other words the
the data services being sent and the data event table (DET).
NPT of a program can remain in full synchronization with the
The DET is exactly analogous to the EIT for conventional
program timeline, regardless of when it is transmitted or how it
program scheduling information and uses an identical syntax.
is split into commercial breaks.
A third table, the long term services table, is similar to the
11.3.3 Signaling Data Contents DET but provides long term schedule information beyond the
maximum 16 days provided by the DET.
Data services are first encapsulated according to the relevant
national or international standard and then transmitted via a 11.4 Content Presentation
transport stream to an STB.
The final stage of data broadcasting is running the application
However in order to make use of a data service the STB on the STB and presenting information to the viewer. It should
must first know that a data service exists and when it will be be noted that a Transport Stream is entirely agnostic to the
available (“announcement”). Secondly it must then be possible actual content it is carrying, provided of course that it is
to find and identify the various components of the data service encapsulated in the correct protocol for video, audio or data.
(“signaling”) when it is actually being broadcast. MPEG-2 only In other words the following section is almost completely
defines the PAT and PMT so in effect all announcement and divorced from MPEG-2, DVB or ATSC data broadcasting
signaling can only be made by inspection of the elementary standards as applications and data are just delivered by the
stream type and some very basic descriptors. Transport Stream to be used by the STB.
Although usable there are a large number of limitations to
11.4.1 Set Top Box Middleware
just using only the PAT and PMT, not least that scheduling
of data services or events is not possible. Both the DVB and In some cases, such as Internet data, the environment in
ATSC have therefore defined additional tables that significantly which the data is to be used is self-defined or relatively
increase the amount of information provided about any data unimportant. However in most cases, such as games and
services present. Note that in all cases the data-carrying PIDs applications it is critical that the data application is constructed
must be signaled as present with an entry in at least one PMT. to run properly on the target STB. In most cases the STB will
have a software application program interface (API) to run that
[Link] DVB Announcement connects the high level function calls from the application to
The DVB consider data services to be either associated with the software drivers that control the real hardware in the box
an individual event (for example a single quiz program) or and link to the data in the Transport Stream. This software
to be part of a service such as a sports channel. In effect it layer, which sits between the application and the drivers, is
conforms to the MPEG-2 paradigm where a data service is known as the “middleware.”
logically indistinguishable from a conventional audio or video There are however a very large variety of STB in use in the
stream. world. Their performance and feature sets run from very basic
It therefore has simply defined a number of additional boxes that can only just decode and display a basic standard
descriptors that are placed into either the EIT or the SDT table definition picture through to the latest units with powerful
which announce that a data service or event will be available. processing capabilities. Some modern units have high
The actual location of data services and components is then performance graphics chip sets and run full operating systems
signaled using the techniques described above. such as Linux or Windows. They are equipped with hard
disk drives for recording programs, have DVD ROMs for DVD
replay and can run conventional programs such as Microsoft
or Star Office packages.

[Link]/video_audio 75
Primer

A simple graphic designed for display on a new high 11.4.2 History of Open Middleware Standards
performance box might not even be displayable on an older
A couple of front-runners for a global Middleware standard
unit. Application developers would have to write programs
quickly emerged, the Multimedia Home Platform (MHP) from
completely differently using different programming languages
the DVB and the Digital TV Application Software Environment
for each type of set top box, middleware and operating
(DASE) from ATSC. Both standards build upon their respective
system, clearly inefficient and increasing the development
Data Encapsulation standards, EN 301 192 for the DVB
cost. There have been a number of attempts to standardize
and A/90 for ATSC, as illustrated in Fig. 11-4. The lack
boxes or at least their minimum functionality, where the
of interoperability and object carousel support has meant
functionality of the box is clearly defined and controlled by the
that DASE has not been widely adopted. CableLabs in the
broadcasters themselves in a very closed fashion.
USA decided to develop a standard that addressed these
Another approach has been taken by companies that provide deficiencies for the US cable TV market. They collaborated
a complete solution by offering a middleware along with the with DVB in its work on open middleware standards, leading
application development tools to develop compliant programs. to the MHP specification being used as the basis for the
These systems are more open in that the middleware can OpenCable™ Applications Platform (OCAP) in January 2002.
be licensed to run on a variety of different set top boxes with
With several organizations around the world striving for the
different performance or features as long as they meet the
same goal in creating open middleware systems, it obviously
basic required functionality.
made sense to seek some form of harmonization. For an
A proprietary data transmission scheme and a “walled garden” industry that is as global as the television industry, this is even
interactive service have been widely used as an interactive more important. The GEM (Globally Executable MHP) work
system, providing true interactivity via a back channel. Other item in DVB came about after a request from CableLabs
company offerings are also widely used throughout Europe to consider the unification of MHP with the original DASE
and applications can be delivered using standard object standard from ATSC.
carousels. There are many other proprietary systems.
The GEM specification was first published in February
A more open standard is an application environment defined 2003 and is a subset of MHP. It was designed to address
by the Multimedia Hypermedia Experts Group (MHEG). interoperability issues between the various open standard
A particular variant of this, MHEG-5, is optimized for low middleware specifications. These issues included:
performance low memory applications and set top boxes in
Interoperability problems that beset previous middleware
particular. MHEG-5 has had some success, notably in the UK
standards, such as OCAP and DASE.
where it is used for terrestrial broadcasts to STBs.
Elements of the transmission system; in particular,
The current trend is therefore towards standardizing upon an
modulation systems, delivery mechanisms and CA systems.
open middleware with a clearly defined programming API. This
has the benefit that the consumer can choose from a wide Specific network operator market requirements.
variety of set top boxes whilst enabling application developers ATSC then developed the ACAP (Advanced Common
to reuse code. Application Platform) standard. ACAP is primarily based on
GEM and DASE, with additional functionality from OCAP.
Indeed, equipment that is ACAP compliant, will also comply
with the GEM and OCAP standards. ACAP was intended to
provide access to the various advantages of these standards
to the ATSC markets, thus replacing the outdated DASE.

76 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

ARIB DVB Broadcast SCTE ATSC

Standard
Independent Content
Presentation
GEM
MHP
DASE
(TS 101 812)
OCAP

DVB Content
ARIB B24 ACAP A/90
(EN 301 192) Transmission
(A/101)

ARIB B20 DVB Content A/65

(EN 300 468) Signaling

MPEG-2 DSM-CC (ISO 13818-6)

MPEG-2 Transport Stream

Relationships between Open Middleware Standards

Figure 11-4. Each standard makes use of the 13818-6 standard.

11.4.3 The DVB Multimedia Home Platform (MHP) 11.4.4 ATSC Advanced Common Application Platform
(ACAP)
The multimedia home platform (MHP) defines a very
comprehensive API with several different profiles and levels The ACAP middleware standard is intended to provide content
to allow for different performance and cost STB and services. and application authors the assurance that their programs
Enhanced TV and Interactive TV (with back channels) and data will be received and run uniformly on all brands
are supported, and it also provides support for Internet and models of receivers. It is intended to apply to broadcast
connections, which is otherwise not directly supported by the systems and receivers for terrestrial broadcast and cable TV
other DVB standards. systems. In addition, the same architecture and facilities may
be applied to other transport systems, such as satellite.
MHP applications are delivered using standard DVB object
carousels and announced using the AIT table described At the heart of ACAP is the GEM framework. GEM is aimed at
earlier. This allows applications to be auto-started, paused or allowing varied organizations to create harmony in technical
stopped and for resources to be explicitly cached in order to specifications, such as the selection of a single execution
speed application boot times. The MHP standard is based on engine and (where possible) a common set of APIs. The goal
Java technology and encompasses a set of APIs derived from is such that applications and content will be interoperable
existing standards such as JavaTV, HAVI (home audio video across all GEM-based platforms.
interoperability) and DAVIC (Digital Audio Visual Council).
Even though other standards will not be completely
Applications can be either Java- or HTML-based and there are compatible with the full MHP specification, GEM ensures
a number of MHP software development kits (SDK) available that compatibility will be maintained where it is feasible to
from several different suppliers. Applications will have to do so. The set of technologies where functional equivalents
undergo a compliance test and be electronically signed to are allowed is negotiated as part of the technical dialogue
ensure both the interoperability and security of the system. between the DVB and each of the organizations wishing to
use GEM. Additionally, the GEM specification contains a list of
The MHP logo is also protected and STBs must undergo a
those other specifications with which it can be used.
compliance process before they can be sold with the logo.
Regular interoperability tests have taken place over the last In keeping with the goal of interoperability, signaling of ACAP
two years to ensure that all MHP applications can run on all services is restricted purely to MPEG; that is, from PMTs only.
MHP set top boxes from a variety of leading manufacturers. This is in contrast with MHP, which includes DVB specific
signaling.
MHP is the de facto standard in Europe, and services officially
started in Finland in August 2001. However, MHP is not just ACAP supports two usage profiles, one is Java only and the
a European standard as it has been adopted throughout the other is a combination of Java and XHTML (extended HTML).
world. For example it is mandated in South Korea for satellite
data broadcasting and in Australia for terrestrial broadcasting.

[Link]/video_audio 77
Primer

Section 12 – MPEG QoS Testing

The ability to analyze existing transport streams for compliance
is essential, but this ability must be complemented by an
ability to create transport streams that are known to be
compliant.

12.1 Testing Requirements - QoS

Although the technology of MPEG differs dramatically from
the technology that preceded it, the testing requirements are
basically the same. On an operational basis, the user wants
to have a simple, regular confidence check that ensures all is
well. In the event of a failure, the location of the fault needs to
be established rapidly. For the purpose of equipment design,
the nature of problems needs to be explored in some detail.
As with all signal testing, the approach is to combine the
generation of known valid signals for insertion into a system
Figure 12-1. Real-time/Deferred-time diagram.
with the ability to measure signals at various points.
One of the characteristics of MPEG that distances it most
from traditional broadcast video equipment is the existence of If more detailed analysis is required, this can be achieved by
multiple information layers, in which each layer is hoped to be simply recording that part of the Transport Stream for further
transparent to the one below. It is very important to be able analysis. This technique is known as deferred-time testing and
to establish in which layer any fault resides to avoid a fruitless can be used, for example, to examine the contents of a time
search. stamp.
For example, if the picture monitor on an MPEG decoder When used for deferred-time testing, an MPEG Transport
is showing visible defects, these defects could be due to a Stream analyzer is acting like a logic analyzer that provides
number of possibilities. Perhaps the encoder is faulty, and the data-interpretation tools specific to MPEG. As with all logic
transport stream is faithfully delivering the faulty information. analyzers, a real-time triggering mechanism is required to
On the other hand, the encoder might be fine, but the determine the time or conditions under which a capture will
transport layer is corrupting the data. In DVB, there are even take place. Figure 12-1 shows a typical block diagram of an
more layers such as energy dispersal, error correction, and analyzer illustrating the real-time section, storage section, and
interleaving. Such complexity requires a structured approach the deferred section. In real-time analysis, only the real-time
to fault finding, using the right tools. The discussion of protocol section operates, and a signal source needs to be connected.
analysis of the compressed data in this primer may help the For capture, the real-time section is used to determine when
user derive such an approach. Reading the discussion of to trigger the capture. Most analyzers include tools known as
another important aspect of testing for compressed television, filters that allow selective analysis to be applied before or after
picture-quality assessment, may also be helpful. This later capture.
discussion is found in the Tektronix publication, “A Guide to
Once the capture is completed, the deferred section can
Video Measurements for Compressed Television Systems.”
operate on the captured data and the input signal is no
12.2 Analyzing a Transport Stream longer necessary. There is also a good parallel in the storage
oscilloscope which can display the real-time input directly or
An MPEG Transport Stream has an extremely complex save it for later study.
structure, but an analyzer such as the Tektronix MTS400
Series can break down this structure into a logical presentation The MTS400 Series Analyzers combine both the real-time
of the contained data. This allows the user to determine and deferred analysis in a single tool. They also incorporate a
Transport Stream correctness. Much of this analysis can take sophisticated triggering mechanism, called CaptureVu, which
place in real time on a live Transport Stream. This includes the causes the bitstream to be stored in volatile memory (RAM)
programs hierarchy and what proportion of the stream bit rate for instant deferred analysis. This enables much faster debug
is allocated to each PID. iterations around the real-time analysis, capture and deferred
analysis loop.

78 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

Figure 12-2. Transport stream hierarchy from PAT to PMT, to program elements and PIDs.

12.3 Hierarchic View An analyzer creates the Hierarchic view by using the PAT and
PMT in the PSI data in the Transport Stream together with any
When analyzing an unfamiliar Transport Stream, the hierarchic
standards specific tables such as the DVB-SI EIT and SDT.
view is an excellent starting point because it provides a
The PIDs from these tables are displayed in the Summary
graphic view of every table, or SI, component in the bitstream.
pane associated with each table. PAT and PMT data are
Figure 12-2 shows an example of a hierarchic display such
fundamental to the operation of any demultiplexer or decoder;
as that provided by the MTS400. The tree display shows
indeed, if missing or incorrect, it is unlikely that equipment
the Transport Stream tables present together with contained
further downstream will be able to interpret the Transport
stream components. The user can very easily see how many
Stream at all. This situation is shown an MTS400 Series as
programs are present and the video and audio content of
PAT/PMT error.
each.

[Link]/video_audio 79
Primer

Figure 12-3. Electronic program guide. Figure 12-4. PSI/SI/PSIP table interval timing.

Figure 12-5. Isolated TR 101 290 tests, PIDs, and event logs.

Whilst the Hierarchic view is useful for displaying PSI and SI Figure 12-6. Bandwidth chart of each program.
contents, it is not particularly useful for navigating between
tables. The Summary pane in the MTS400 Series contains
a series of hyperlinks for quickly navigating the logical in the bitstream. For example, if a given PID is referenced in
connections between tables such as the DVB SDT and PMT a PMT, it should be possible to find PIDs of this value in the
as well as the PAT and PMT. In addition, the MTS400 includes bitstream. The consistency-check function makes such a
an EPG view that comprises the EIT and SDT contents. The comparison. Figure 12-5 shows a consistency-error from a
user may navigate directly to an individual EIT section of bitstream including two unreferenced packets.
interest from the EPG view (see Figure 12-3). The EPG view A Program Summary view graphically displays the proportions
also supports the ATSC and ARIB standards. of the transport stream allocated to each PID or program.
The ability of a demux or decoder to lock to a Transport Figure 12-6 shows an example display. The hierarchical view
Stream depends on the frequency with which the PSI data and the Program Summary show the number of elements
are sent. The PSI/SI rate option shown in Figure 12-4 displays in the transport stream and the proportion of bandwidth
the frequency of insertion of system information. PSI/SI allocated.
information should also be consistent with the actual content

80 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

12.4 Standards Compliance Checking

12.4.1 DVB Measurement Guidelines, TR 101 290
Founded in September 1993, the Digital Video Broadcasting
Project (DVB) is an industry- led consortium of over 260
broadcasters, manufacturers, network operators, software
developers, and regulatory bodies in over 35 countries
committed to designing global standards for the delivery of
digital television and data services. Although DVB Services
are focused on Europe, its standards are available on every
continent with more than 120 million DVB receivers deployed.
One of the most important DVB standards is TR 101 290. This
provides guidelines for measurement in DVB satellite, cable
and terrestrial digital television systems. It defines a number
of measurement techniques, such that the results obtained
are comparable when the measurement is carried out in
Figure 12-7. Summary of TR 101 290 tests and event log.
compliance with the appropriate definition. The general aim of
the tests is to provide a "health check" of the most important
elements of the TS; that is, the list of tests is not exhaustive. Figure 12-7 shows the DVB Tests view from the MTS400
The tests are split into three priority groups according to their Series. A colored LED shows the current state of each test,
importance for monitoring purposes: including amber for a test that was previously in error. The
view allows the user to drill down to individual sub-tests,
1. The first priority group lists a basic set of parameters
adjust test parameters and display the filtered log entries.
which are considered necessary to ensure that the TS
Where applicable, it also displays a list of PIDs that are subject
can be decoded. These are mainly Transport Stream
to each test.
synchronisation, dropped packets plus the PSI integrity
(PAT and PMT tables).
2. The second group lists additional parameters which are
recommended for continuous monitoring. These include the
PCR and PTS timestamps and table CRCs.
3. The third group lists additional parameters which could
be of interest for certain applications. These are mainly
concerned with the integrity of the other SI tables, such as
SDT and EIT.

[Link]/video_audio 81
Primer

Error Description TOA POA CM QOS TNC

PAT repetition interval error (100 ms < cycle time <= 200 ms) X
PAT repetition interval error (200 ms < cycle time <= 500 ms) X X
PAT not found for 500 ms (cycle time > 500 ms) X X X X X
Table 12-2. ATSC A/78 softens the one hard timer limit defined by TR 101 290. ATSC adds multiple timer limits (i.e., 1x, 2x, or 5x).

12.4.2 ATSC Transport Stream Verification, A/78 3. Component Missing (CM) Typically, one of the audio or
video elementary streams cannot be found.
The Advanced Television Systems Committee (ATSC) was
formed in 1982, as an international, non-profit organization 4. Quality Of Service (QOS) Parameters are out of specification
developing voluntary standards for digital television. The by such a margin that a significant fraction of the receivers
ATSC member organizations represent the broadcast and can be expected to produce flawed outputs.
associated industries. Specifically, ATSC develops television 5. Technically Non-Conformant (TNC) Violates the letter of the
standards among different communications media focusing standard, but in practice will have little effect on the viewing
on digital television, interactive systems, and broadband experience.
multimedia communications. The largest users of the ATSC
standards are the North American cable and terrestrial As might be expected, there is much commonality between
markets. the ATSC and DVB parameters monitored. The difference
is that ATSC specifies the error threshold for escalating
The A/78 is a Recommended Practice that provides a categorization. This can generally be described as:
common methodology for describing TS conformance
criteria. Specifically, the SI elements and parameters within an The official metric to twice the metric.
ATSC TS are verified for correct transmission. The RP uses Twice the official metric to 5 times the metric.
the premise that the severity of an error depends upon the
Over five times the metric.
magnitude of deviation. It specifies 5 groups of priority tests
associated with ranges of deviation from the standard for An example is the PAT repetition error, as shown in Table 12-2.
certain parameters. The 5 priorities, or severity levels, are:
1. Transport Stream Off Air (TOA) The station is technically off-
air, as the TS errors are so severe. Receivers will not be able
to tune and decode anything within this broadcast.
2. Program Off Air (POA) A major service is in error to the point
where decoders will not be able to decode the program.
This is likely to be caused by incorrect signaling.

82 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

Figure 12-9. Script files define structures of all present and new sections and tables.

Figure 12-8. Packet view supporting TS packets, SI packets, and hex view.

12.5 Packet View

As an alternative to checking for specific data in unspecified
places, it is possible to analyze unspecified data in specific
places, including individual Transport Stream packets, tables,
or PES packets. This analysis on the MTS400 Series is
known as the Packet view, because the analyzer automatically
parses and decodes the data and then displays its meaning. Figure 12-10. TR 101 290 flags CRC error based upon PSI and SI sections and
Figure 12-8 shows an example of an MPEG transport packet script files.

in hex view as well as Packet view. A selected field in the

Packet view automatically selects the corresponding bytes in 12.6 Syntax and CRC Analysis
the hex view for easier correlation.
To ship program material, the transport stream relies
Figure 12-8 shows an example of a PAT in the Packet view. completely on the accurate use of syntax by encoders.
The field structure in the Packet view is defined by script Without correct settings of fixed flag bits, sync patterns,
files. Script files are simply text files, such as that shown in packet-start codes, and packet counts, a decoder may
figure 12-9 that make use of pre-defined keywords. This misinterpret the bit stream. The syntax check function
allows for quick updates if the standards change and also the considers all bits that are not program material and displays
interpretation of proprietary SI, as no changes to the software any discrepancies. Spurious discrepancies could be due to
are required. transmission errors; consistent discrepancies point to a faulty
encoder or multiplexer. Figure 12-10 shows a syntax-error as
well as a missing cyclic redundancy check (CRC).
Many MPEG tables have checksums or CRCs attached for
error detection. The analyzer can recalculate the checksums
and compare them with the actual checksum. Again, spurious
CRC mismatches could be due to stream-bit errors, but
consistent CRC errors point to a hardware fault.

[Link]/video_audio 83
Primer

Figure 12-11. PCR plot over time.

12.7 Timing Analysis

The tests described above check for the presence of the Figure 12-12. Real-time program clock plots.

correct elements and syntax in the transport stream. However,

to display real-time audio and video correctly, the transport
Figure 12-11 shows a PCR display that indicates the positions
stream must also deliver accurate timing to the decoders. This
at which PCRs were received with respect to an average
task can be confirmed by analyzing the PCR and time-stamp
clock. The Data at Cursor pane displays the data associated
data.
with any selected PCR. PCR inaccuracy measurement is
The correct transfer of program-clock data is vital because this based upon the PLL clock recovery in a receiver. The analyzer
data controls the entire timing of the decoding process. PCR calculates the constant bitrate from the first 10 PCR values,
analysis can show that, in each program, PCR data is sent at which is then used as a basis for interpolating further values.
a sufficient rate and with sufficient accuracy to be compliant. These may then be compared with actual values. A high
The PCR data from a multiplexer may be precise, but pass filter is used to ensure that the plotted inaccuracies are
remultiplexing may put the packets of a given program at a consistent with PLL jitter.
different place on the time axis, requiring that the PCR data An alternate approach shown in Figure 12-12 provides a
be edited by the remultiplexer. Consequently, it is important to graphical display of PCR interval, inaccuracy, jitter, frequency
test for PCR inaccuracies after the data is remultiplexed. offset and drift, which is updated in real-time.

84 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

Figure 12-13. Packetized Elementary Stream Access Unit intervals and headers.
Figure 12-14. Transport, multiplex, and element buffer levels.

Figure 12-13 shows a time-stamp display for a selected defect will be seen on a repeat of the test because the same
elementary stream. The access unit, the presentation time, video signal will not result in the same transport stream. In this
and, where appropriate, the decode times are all shown. case, an absolutely repeatable transport stream is essential
so that the defect can be made to occur at will for study or
In MPEG, the reordering and use of different picture types
rectification.
causes delay and requires buffering at both encoder and
decoder. A given elementary stream must be encoded within Transport stream jitter should be within certain limits, but a
the constraints of the availability of buffering at the decoder. well-designed decoder should be able to recover programs
MPEG defines a model decoder called the T-STD (transport beyond this limit in order to guarantee reliable operation. There
stream system target decoder); an encoder or multiplexer is no way to test for this capability using existing transport
must not distort the data flow beyond the buffering ability of streams because, if they are compliant, the decoder is not
the T-STD. The transport stream contains parameters called being tested. If there is a failure, it will not be reproducible
VBV (video buffer verify) specifying the amount of buffering and it may not be clear whether the failure was due to jitter
needed by a given elementary stream. or some other noncompliance. The solution is to generate
a transport stream that is compliant in every respect and
The T-STD analysis displays the buffer occupancy graphically so
then add a controlled amount of inaccuracy to it so that
that overflows or underflows can be easily seen. Figure 12-14
the inaccuracy is then known to be the only source of
shows a buffering display. The output of a normal compressor/
noncompliance. The editor feature of the MTS400 is designed
multiplexer is of limited use because it is not deterministic. If a
to create such signals.
decoder defect is seen, there is no guarantee that the same

[Link]/video_audio 85
Primer

12.8 Elementary Stream Testing 12.9 Elementary Stream Analysis

Due to the flexible nature of an MPEG bitstream, the An elementary stream is a payload that the transport stream
number of possibilities and combinations it can contain is must deliver transparently. The transport stream will do so
almost incalculable. As the encoder is not defined, encoder whether or not the elementary stream is compliant. In other
manufacturers are not compelled to use every possibility; words, testing a transport stream for compliance simply
indeed, for economic reasons, this is unlikely. This fact makes means checking that it is delivering elementary streams
testing quite difficult because the fact that a decoder works unchanged. It does not mean that the elementary streams
with a particular encoder does not prove compliance. That were properly assembled in the first [Link] elementary
particular encoder may simply not be using the modes that stream structure or syntax is the responsibility of the encoder.
cause the decoder to fail. Thus, an elementary stream test is essentially testing the
encoder. It should be noted that an encoder can produce
A further complication occurs because encoders typically will
compliant syntax, and yet still have poor audio or video quality.
not produce the same bitstream if the video or audio input is
However, if the syntax is incorrect, a decoder may not be able
repeated. There is little chance that the same alignment will
to interpret the elementary stream. An elementary stream may
exist between the resultant I-, P- and B-pictures. If a decoder
be intermittently noncompliant if some less common mode of
fails a given test, it may not fail the next time the test is run,
operation is not properly implemented.
making fault-finding difficult. A failure with a given encoder
does not determine whether the fault lies with the encoder As transport streams often contain several programs that
or the decoder. The coding difficulty depends heavily on come from different encoders, elementary stream problems
the nature of the program material, and any given program tend to be restricted to one program, whereas transport
material will not necessarily exercise every parameter over the stream problems tend to affect all programs. If problems are
whole coding range. noted with the output of a particular decoder, then the Vclips
test streams should be run on that decoder. If these are
To make tests that have meaningful results, two tools are
satisfactory, the fault may lie in the input signal. If the Transport
required:
Stream syntax has been tested, or if other programs are
A known source of test signals that deliberately explore working without fault, then the particular elementary stream in
the whole coding range. This range includes bitstreams error should be analyzed.
compliant to the video standard concerned, but also
The main component of elementary stream analysis is
streams containing deliberate bitstream errors. These
verifying compliance against the relevant standards. The
signals must be deter ministic so that a decoder failure will
MTS4EA produces an output log of all violations and non-
give repeatable symptoms. The Tektronix Vclips product is a
conformances. The user may select any log entry and
collection of bitstreams designed to perform this task.
synchronise the video display and other views to that point in
An elementary stream analyzer that allows the entire syntax the bitstream for fast pinpointing of problems. A useful feature
from an encoder to be checked for compliance, such as the is overlays; there are many different types including average
Tektronix MTS4EA. quantisation and bits per macroblock.

86 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

Figure 12-16. Color-coding different types of blocks.

Figure 12-15. Advanced elementary stream analyzer showing bits per block.

Figure 12-15 shows a bits per macroblock display that

illustrates an encoder optimisation problem; many bits
are being used to encode the non-changing background
information. At a lower level of testing, Figure 12-16 shows a
decoded P frame overlaid with macroblock type information.
The details of a selected macroblock are also shown.
Figure 12-17. PSNR of reference file and test file.
Elementary stream analysis discussed so far has been single
ended; that is, a measurement of the output signal with no
regard to the quality of the original uncompressed video data.
If the original video is available for reference then the MTS4EA
can create a signal to noise ratio graph (Figure 12-17) and
visual difference display.

[Link]/video_audio 87
Primer

Figure 12-18. Reference audio and video files used to accurately detect A/V delay Figure 12-19. Generic multiprogram transport stream multiplexer.
when encoded into transport stream.

12.11 Creating a Transport Stream

Whenever the decoder is suspect, it is useful to be able to
generate a test signal of known quality. Figure 12-19 shows
that an MPEG transport stream must include Program Specific
Information (PSI), such as PAT, PMT, and NIT describing one
or more program streams. Each program stream must contain
its own PCR and elementary streams having periodic time
stamps.
A DVB transport stream will contain additional service
information, such as BAT, SDT and EIT tables. A PSI/SI editor
enables insertion of any desired compliant combination of PSI/
SI into a custom test bitstream.
Clearly, each item requires a share of the available transport
stream rate. The MTS400 Multiplexer provides a graph
showing the total bit rate used at each point in the bitstream,
Figure 12-20. Multiplexer application showing hierarchy, timeline, and available
as illustrated in Figure 12-20. The overall transport stream bit
bandwidth.
rate must remain constant, so the remainder of the bit rate is
used up by inserting stuffing packets with PIDs that contain all
12.10 Audio Visual Synchronization 1s, which a decoder will reject.
One of the major problems affecting broadcasting throughout
its entire history is synchronising audio and video information,
commonly referred to as lip sync. In a production facility, the
audio and video information typically takes different routes
before they are encoded and multiplexed together, thereby
leaving plenty of scope for delays to be introduced. The
MTS4EA includes an AV Delay feature that measures the delay
introduced by the encoder under test by using an input test
pattern, as shown in Figure 12-18. A graph is plotted of the AV
Delay, as well as errors generated for each point that exceeds
the threshold for the chosen standard.

88 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

12.12 PCR Inaccuracy Generation The advantage of this approach is that jitter of any required
magnitude can easily be added to any program simply by
The MPEG decoder has to recreate a continuous clock by
modifying the PCR data and leaving all other data intact. Other
using the clock samples in PCR data to drive a phase-locked
programs in the transport stream need not have jitter added.
loop. The loop needs filtering and damping so that jitter in the
In fact, it may be best to have a stable program stream to use
time of arrival of PCR data does not cause instability in the
as a reference.
clock.
For different test purposes, the time base may be modulated
To test the phase-locked loop performance, a signal
in a number of ways that determine the spectrum of the loop
with known inaccuracy is required; otherwise, the test is
phase error in order to test the loop filtering. Square-wave
meaningless. The MTX100 Generator can generate simulated
jitter alternates between values which are equally early or late.
inaccuracies for this purpose. Because it is a reference
Sinusoidal jitter values cause the phase error to be a sampled
generator, the MTX100 Generator has highly stable clock
sine wave. Random jitter causes the phase error to be similar
circuits and the actual output jitter is very small. To create
to noise.
the effect of jitter, the timing of the PCR data is not changed
at all. Instead, the PCR values are modified so that the PCR
count they contain is slightly different from the ideal. The
modified value results in phase errors at the decoder that are
indistinguishable from real jitter.

[Link]/video_audio 89
Primer

Figure 12-21. MPEG Carousel sections multiplexed into a transport stream.

interactive application. The main problems faced with

transmitting data broadcast applications may be summarized
as:
Timing.
Bit rates.
Signaling and component linkages (references).
The Tektronix Carousel Analyzer is designed to help diagnosis
of all of these problems, both at the data and object carousel
layers. Indeed, it allows analysis of bit rate and repetition
interval graphs for each node indicated in the protocol layers
shown in Figure 12-21.
Figure 12-22 shows a typical Carousel Analyzer view with
Services pane on the left and Message Log pane at the
bottom. The contents of an Object carousel are displayed,
Figure 12-22. Section timing of carousels.
together with the repetition interval and bit rate graphs for
the selected ‘Start’ object. If multiple nodes are selected for
either bit rate or repetition interval then each additional trend is
12.13 Data Broadcast Carousel Analysis added to the same graph for ease of comparison.
The description of carousels in Section 11 should lead most
One of the main timing issues is the boot time. That is,
readers to conclude that they are a complex animal. Part of
the amount of time it takes from the start of transmitting a
the complexity is driven by the trade-off between transmission
carousel to the point where the STB can execute it. This is
bandwidth and responsiveness of the user experience. This
especially important for interactive adverts, where the advert
results in a necessity to restrict bandwidth by repeating
to which the carousel is associated may only last for, say 20
group data at different rates, depending upon its likelihood
seconds.
to change. That is, football score updates will change much
more frequently than the weather forecast. Incorporating this The Carousel Analyzer incorporates many timing methods,
flexibility means that carousels typically consist of a large one of which is Cold Boot Time. This measures the time taken
number of components that need to be linked together to signal the carousel and download the service gateway, the
and signalled correctly to ensure the integrity of the overall boot class and all of the assets. This is repeated at regular
intervals within the bitstream.

90 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

Figure 12-23. Cold-boot times of the carousel across the file. Figure 12-24. Hierarchical view of entire service.

The boot class must be received before any of the assets. This
simulates the STB not knowing which carousel objects the
boot class depends upon until it has received and processed.
Directories must be received before any child objects, as this
timing method assumes no signaling caching. The end of each
cycle will be the end of the module containing the last asset
- this is an important distinction, as an asset may not be the
only one in that module.
The most common signaling problems are unresolved objects
and absent objects. Figure 12-24 shows that when an object
is found during analysis, but its key is not referred to in the
Service Gateway, it is displayed under an Unresolved Objects
node. However, when a directory object is found, the keys
of its child nodes can be resolved and its tree structure built
correctly, even though the directory object itself may still be Figure 12-25. Elements of service gateway.

unresolved.
Figure 12-25 shows that objects referenced, but not found are As might be expected, Tektronix also offers a Carousel
identified. The path in which the absent object is expected is Generator product for creating carousel contents within an
also marked. output Transport Stream. This is particularly useful in test
situations where the effects of varying parameters, such as
Occasionally, the signaling is sufficiently broken that a carousel
individual repetition intervals, may be quickly ascertained.
present in the Transport Stream is not found during analysis.
The Generator will create object carousels conforming to the
In this instance the carousel can be identified with Manual
MPEG-2, DVB, DTT (MHEG-5) or MHP standards.
Signaling from the related PID node. The Transport Stream
will then be re-analyzed with any DSM-CC sections on the
selected PID being processed.

[Link]/video_audio 91
Primer

Figure 13-1. Low blockiness from satellite distribution feed. Figure 13-2. Transcoded content with significantly higher blockiness (green plot) over
the same timeframe.

Section 13 – MPEG QoE Testing to clearly portray the high-motion video scene. In this case,
QoE testing examines the baseband impairments of the there are no protocol errors, but the picture frame or video
Elementary Stream and which are noticeable to viewers and sequence is made up of large 16x16 solid squares rather than
subscribers. a clean picture. Viewers often refer to this as a blocky video
problem. Figure 13-1 shows a low blockiness measurement
13.1 Testing Requirements - QoE on a TV program from satellite distribution at 4.1 Mbps. Figure
Quality of Experience (QoE) QoE is a metric that relies much 13-2 shows the same TV program over cable (256QAM), but
more upon our ability to notice a problem in the midst of only after it has been transcoded (or rate-shaped, clamped,
a TV program rather than a single bit error. For instance, if etc.) to a lower rate of 3.6 Mbps. The resulting changes cause
single bit error affected the far corner of a B-frame, then that significantly more blockiness artifacts over the same period
bit error would have a much lower impact to QoE than if the of time. In this case, the QoE has deteriorated enough that
error had landed in the middle of an I-frame. Subsequently, if viewers would recognize the blockiness issues in the video.
the single error in the I-frame ended up corrupting an entire Another video QoE issue occurs when the same video frame
slice of 16x16 pixel blocks, then the impact to QoE would occurs repeatedly for a long period of time (frozen or black
be huge. Therefore, to be able to measure the impact of an frames). Obviously, video frames repeat when there is no
error upon a TV program, it is critical to know exactly where activity, but at some point (e.g., 2 minutes), an alarm is needed
the error occurred. Another issue is that the transport may to alert the operator if the frozen frames are on purpose, or
be perfectly error free (no syntax or semantic errors), but the from a broken link or piece of equipment.
video is objectionable because there is not enough bandwidth

92 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

Figure 13-3. The Picture Quality Analyzer performs direct comparisons of video frames
before and after a live transcode.

Figure 13-4. Satellite ingest with audio loudness at 2 dB quieter than DialNorm.

When a reference signal is available for direct comparison as

in the case shown above from satellite and cable, an objective
picture quality rating can be made using the Tektronix
PQA600 Picture Quality Analyzer, or from the Picture Quality
Analysis software option in the MTS4000. In this case, a
stream is recorded from both satellite and cable, and then the
Picture Quality Analyzer will convert to YUV, perform spatial
and temporal alignments, and then measure the difference
between the two video clips. The basic Peak Signal to Noise
Ratio (PSNR) measurement is shown, but the Picture Quality
Analyzer also performs many more important tests such
as Picture Quality Rating, Attention model, etc. Figure 13-3
shows the Picture Quality Analyzer making a frame by frame
PSNR comparison between every frame of the satellite and
cable video clips.
The Picture Quality Analyzer has been pitted directly against
human view trials in a variety of studies. The IRT study11
Figure 13-5. Cable broadcast with audio at 2 dB louder than DialNorm. This is a 4 dB
(Institut fuer Rundfunktechnik in Munich, Germany) showed increase over the satellite feed.
extremely high correlation between the PQR results and
human viewer results using ITU-R_BT.500-1112 testing.
Audio is another element to QoE as it occasionally increases fines on broadcast operators who deviate beyond the agreed
to an unbearable level, or maybe drops out altogether over limits. With this agreement in place, TV viewers should now
a long period of time. To accurately track these problems, be able to switch from channel to channel without having to
loudness measurements have been defined for instantaneous adjust the audio levels. The same goes for the commercials
peaks (ITU-R BS.1770/177113), as well as short term and long or ad-inserts between programming. The loudness of
term filters for ATSC14, ITU, and EBU15. These measurement commercials must also stay within the agreed upon levels. As
algorithms allow test equipment to track audio over long an example of audio levels being altered, Figure 13-4 shows
periods of time and trigger on deviations outside allowable the satellite distribution signal at one audio level (-29 LKFS13)
levels. One method of tracking this is to use a DialNorm level while its cable-broadcast counterpart (Figure 13-5) is at a
as a target for normal dialog levels. Averaged or filtered levels different audio level (-25 LKFS). Both signals use a DialNorm
are allowed to rise or fall within a few dB of this set level. Many reference level of -27 LKFS.
governments have adopted this requirement and now impose

[Link]/video_audio 93
Primer

in sporting events is when the remote transmission link is

lost for a fraction of a second. The video breaks apart, but
the receiver is able to catch this QoS issue due to Transport
Error Indicator flags (TR 101 290 Priority 2.1 – Also bit #9 in
each transport packet). Although, if the decoded and blocky
or broken video is passed on to the network as a re-encoded
program , then the program will often have a perfect QoS, as
well as its compressed video syntax being perfect due to its
decode and then second encode process. The video may
still look like garbage (as seen in Figure 13-6), but according
to QoS and video syntax, the results are error free. To find an
Figure 13-6. Significant loss of RF or IP packets cause blockiness and slice errors. artifact, we must actually look at the decoded picture content
and determine if the frames contain objectionable artifacts
(blockiness, slice errors, frozen frames, etc.).
13.2 How are QoS and QoE interrelated? If we have a high BER or poor QoS, then it is inevitable that
Both QoS and QoE are interrelated, but their relationships are the QoE will deteriorate also (as in Figure 13-6). In this case,
not equal. Case in point: we do not care which frames the errors are landing in because
the rate is high enough to land in every frame. Therefore, the
1) A good QoE usually means a good QoS
QoE rating becomes highly objectionable. In this case, the
2) But a good QoS does not always mean you will have a QoS measurement is the key to isolating the problem.
good QoE. For example, Encoder bandwidth starvation or
When the QoS is quite good (no transmission errors), then
over-compression.
there is a very low correlation between QoS and QoE. In this
3) A bad QoS, or at least frequently occurring QoS issues case, it is more important to focus on the QoE.
usually lend toward a poor QoE. With enough bad bits
There is an important case where the QoS can be perfect, but
randomly distributed into all video and audio elements,
the QoE is highly objectionable. This is due to the common
poor quality of video and audio is bound to occur (see
case with networks running constant bit rate (CBR) video.
Figure 13-6).
Normal video with low motion may look decent, or even
4) A bad QoE is not always related to a bad QoS, but it is very good. But the scene occasionally changes from slow
always worth validating the QoS performance. Poor QoE moving objects (e.g., talking head) to something with a high
can come from misbehaving ad-inserted and any other rate of change (e.g., sporting event, quick successive scene
equipment (not related to TR 101 290). Poor QoE can come cuts, etc.). If the encoder does not have enough bandwidth
from over compression (this is perfectly legal). Poor QoE to accommodate the transition to the high-motion events,
can come from dropped IP packet, corrupt RF packets, or then the only recourse for the encoder is to start decimating
other transmission problems. the 8x8 blocks of video. At its worst, the 8x8 blocks may
only contain a single value representing the video luminance
Therefore, it is common to measure both QoE and QoS in
and color. This scenario is important for QoS and QoE
order to quickly identify the root cause of a problem and in
testing because the end-user notices a very objectionable
turn correct the issue.
TV program segment, although the difference between
One problem with QoS issues is that they can be easily hidden over compression and near total loss of signal are almost
or masked when the program is handed off from one network indistinguishable. Figures 13-1 and 13-2 show a program at
to another (e.g., dropped video packets). Once the program is both 4.1 and 3.6 Mbps with the lower rate having much more
decoded back to baseband for manipulation and re-encoding, blockiness due to high motion in a bandwidth starved stream.
all previous QoS errors are lost forever. After encoding a Figure 13-6 shows blockiness too, but due to a completely
second time, the transport and video protocol is now error different reason (missing packets). Therefore, QoE testing
free, but visually, there may be large slice errors seen in the is extremely important, but testing QoS helps to determine
middle of the video frames. At this point, then only thing the difference between malfunctioning equipment causing a
available to catch these artifacts is to decode the video and problem verses a poor transmission link.
audio content and look for anomalies. A common problem

94 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

Figure 13-7. Dual trigger levels for Frozen and Black video.

13.3 Using QoS and QoE to identify “real”

problems Figure 13-8. Dual trigger levels and gating for Blockiness.

Most facilities use an ingest, manipulate, and egress

architecture. Cable headends use a wide variety of video
equipment to capture, manipulate, and ultimately, broadcast.
Most all test points would be in the form of QPSK/8PSK,
8VSB, QAM, IP (GigE or 10G), or ASI.
Monitoring QoE aspects on live programming falls into several
different audio and video categories.
Video QoE
Video QoE includes monitoring video for frozen frames, black, Figure 13-9. VQS Target DialNorm settings.

and blockiness. In the case of frozen frames, a long series

of nearly identical frames should trigger a problem since live
TV programming is usually made up from moving video. We maintain that level for a significant amount of time. Otherwise,
know that there are many occasions where it is acceptable the monitor would trigger an alert every time the scene was
for a frame to be repeated many times, but at some point, the overly blocky, even for one single frame. Figure 13-8 shows
operator should be alerted to excessive repetitive frames. It the VQS1000 QoE software setup for Blockiness triggering
could be the sign of a broken encoder or multiplexer, or any when
of a number of pieces of hardware. Figure 13-7 shows the level of 90% is reached for over 2 seconds, or
VQS1000 QoE software setup for Frozen Frames that will
level of 50% is reached for over 5 seconds, or
be triggered if 90% of the frame is frozen for more than 15
seconds, or when 50% of the frame is frozen for more than level of 75% is reached at least 5 times within 30 seconds.
120 seconds. The requirements are similar for excessively Audio QoE
long periods of black. Blockiness is a little different in that it
Audio QoE includes monitoring audio for loudness levels. With
can occur in a small portion of the picture, or over the entire
such a high emphasis on audio levels today, it has become
picture. Blockiness will often occur during short scenes where
critical to measure overall loudness levels according to the
the video content is moving too quickly for the encoder to
new guidelines. Just as with video QoE monitoring, triggers for
faithfully compress all of the details. Therefore, the encoder
levels, deviations, and durations are important to minimize the
tends to throw away high frequency details rather than failing
many false positives than can occur. Figure 13-9 shows audio
completely. In this case, the monitor must be set to trigger
loudness limits must be within 3 dB of DialNorm.
once the blockiness level has crossed a threshold, and then

[Link]/video_audio 95
Primer

Figure 13-10. Terrestrial encoder generating negative DTS-PCR delta values (should be
impossible).

Troubleshooting Over-compression
Figure 13-11. Cable headend encoder generating negative DTS-PCR delta values
The VQS1000 QoE software allows for real-time monitoring of (should be impossible).
multiple RF and IP signals in both the transport layer as well
as the video and audio layer. When blockiness is found to be
excessive on in a TV program, the MTS4000 MPEG Analyzer had no problem at all on the SAP channel. It sounded like an
can measure the broadcast program as well as the ingest interoperability issue, so the MTS4000 MPEG Analyzer was
distribution feed. tuned to the local off-air channel to look at its TR 101 290
results. The TR 101 290 tests, and virtually every other test
When the QoE of a TV program is called into question, the
came up clean except for one. The MTS4000 MPEG Analyzer
MTS4000 MPEG Analyzer with its VQS1000 QoE software can
can graph the audio or video Decode Time Stamp (DTS)
quantify the magnitude of the audio and video issues. Once
value as it arrives, against its current PCR value. This delay is
the levels are proved to be unacceptable (as seen in Figure
always a positive number (representing buffer delay time) and
13-2), the MTS4000 MPEG Analyzer can quantify the ingest
by definition it is required to be between zero and one second.
program (as seen in Figure 13-1), and then a comparison can
The newly added SAP audio had its DTS-PCR delta ranging
be made between the two programs. In the example here,
above and below zero, which is by definition, never allowed to
the cable broadcast program has a stable QoE or blockiness
happen. Figure 13-10 shows the local broadcaster SAP audio
as long as the content does not contain too much action.
delay issue.
Once the scene changes quickly, the blockiness rating spikes
dramatically. In comparison, the ingest content is much less This same terrestrial broadcast feed was also sent to both
blocky and only varies slightly when the scenes change local cable companies for redistribution.
quickly. It can noted that the ingest content is coming into the Another similar audio example was found at the cable
facility at about 4.1 Mbps whereas the broadcast program is headend where the audio buffer delay was a negative
leaving at only 3.6 Mbps. Therefore, the blockiness issue can value, which is impossible for a set top box to support. The
be identified as a result of decoding and re-encoding without MTS4000 MPEG Analyzer once again showed the values
providing sufficient bandwidth to maintain the quality of the going below zero. Figure 13-11 shows the same audio
original picture. problem, but at a different facility.
Troubleshooting Interoperability In both cases, reconfiguring the encoder fixed the problem.
The MTS4000 can also be used to identify non-compliance The MTS4000 MPEG Analyzer with its unique DTS-PCR
issues when televisions and set top boxes begin to react measurement was able to quickly pinpoint the problem to the
abnormally to specific programming. In a local broadcast recent configuration change to the encoder, and thus take the
case, several viewers complained about inaudible audio on the blame off the small set of failing decoders throughout the city.
SAP channel (audio sounding like motorboats). Other viewers

96 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

While the MTS4000 provides real-time analysis, the

instrument's ability to capture events for deep analysis is
critical to identifying the root cause of problems. Especially
useful are the pre-triggers in CaptureVu that not only capture
an error, but also provide information leading up to the error.
This last troubleshooting example occurred with a DVR
from a major consumer electronics manufacturer that was
misbehaving whenever it was tuned to a Switched Digital
Video (SDV) channel. The MTS4000 discovered the device
was dropping the encryption key whenever a service was
added to or removed from the multiplex. Figure 13-12 shows
the MTS4000 TSCA setup screen for enabling a trigger on
a specific test or missing PID. By documenting the scenario
with screen shots and test reports, the DVR manufacturer was
contacted, and they provided a software update that solved
Figure 13-12. MTS4000 Transport Stream Compliance Analyzer: Trigger and capture the problem and eliminated many expensive truck rolls for the
enabled for missing encryption key packet.
Cable Operator.

Troubleshooting Transport Streams with CaptureVu

A key part of delivering a quality experience is finding the root
cause of problems in the transport stream. It's no longer good
enough just to spot a problem, reboot the set-top box and
hope that it goes away. The MTS4000 can uniquely go as
deep as is needed into the transport and elementary streams
to track down sources of picture anomalies, like discovering
that a closed caption stream contained too much information,
causing set-top box buffers to overflow and precipitate
automatic reboots.

[Link]/video_audio 97
Primer

Figure 13-13. MTS4000 Transport Stream Compliance Analyzer enabling the Figure 13-14. PID 800 DPI packets arrive about every 15 minutes.
CaptureVu and Recording for SCTE35 DPI packets.

In the case of much larger recordings, the MTS4000 can

Troubleshooting Ad Insertion
record for many hours. In the example shown in Figure 13-14
All broadcast companies fund their businesses based upon running at 4 Mbps, the MTS4000 could hold over 11 days of
selling advertising time between programs. The ads need to continuous programming. The figure also shows the PID 800
be inserted at very specific time, and often timed by SCTE3516 DPI packets arriving around 15 minutes apart, as you would
Digital Program Insertion (DPI) cueing. If these cue tones do expect for local ad-inserts.
not make it through, then the national ads go through and
With these SCTE35 DPI tools, troubleshooting and debugging
local revenue is lost. The MTS4000 MPEG Analyzer can trigger
missing DPI packets become relatively simple.
on the STCE35 packets as well as record programming to
disk before and after the event (pre-trigger). Figure 13-13
shows the MTS4000 Transport Stream Compliance Analyzer
enabling the CaptureVu and Recording feature based upon
SCTE35 packets. The trigger/record function allows the
recorded file to be as small as possible.

98 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

13.4 Summary Glossary

Troubleshooting video signals in a broadcast or cable facility AAC – Advanced Audio Coding.
requires the use of an MPEG Analyzer that provides multiple
AAU – Audio Access Unit. See Access unit.
input signals all running in parallel. The minimum set of test
points should be at ingress, ASI/IP switch, and egress. It is AC-3 – The audio compression scheme invented by Dolby
critical to measure QoE at both egress and ingest, but in the Laboratories and specified for the ATSC Digital Television
case of audio and video problems, QoS testing at multiple Standard. In the world of consumer equipment it is called
points in the facility may be needed to pinpoint the source Dolby Digital.
of the impairments. Critical capabilities in an MPEG Analyzer ACAP – Advanced Common Application Platform.
should include:
Access Unit – The coded data for a picture or block of sound
Transport stream generation, modification and analysis and any stuffing (null values) that follows it.
- Real and deferred time
A/D – Analog to digital converter.
- Including MPEG-2, H.264 & MPEG-4 AAC
- Multiplexer AES – Audio Engineering Society.

- Automatic error triggered recording and capture Anchor Frame – A video frame that is used for prediction. I
frames and P frames are generally used as anchor frames, but
A range of physical interfaces
B frames are never anchor frames.
- Multi-port ASI
ANSI – American National Standards Institute.
- RF (DVB-S2, 8-VSB & QAM-B)
- Video over IP analysis and generation (1Gbps and API – Application Program Interface.
10Gbps) ARIB – Association of Radio Industries and Businesses.
Comprehensive suite of software tools for analyzing all ASF – Advanced Systems Format (formerly "Advanced
layers of video Streaming Format"), a proprietary Microsoft streaming format.
- High accuracy RF layer analysis
Asynchronous Transfer Mode (ATM) – A digital signal
- Transport, Program & Elementary Streams protocol for efficient transport of both constant-rate and bursty
- Video & Audio Quality of Experience (QoE) information in broadband digital networks. The ATM digital
- Picture Quality stream consists of fixed-length packets called “cells,” each
containing 53 8-bit bytes – a 5-byte header and a 48-byte
A facility equipped with an analyzer having the feature set
information payload.
listed above should be able to resolve problems in minutes
rather than hours or days using alternative methods. ATM – See asynchronous transfer mode.
ATSC – Advanced Television Systems Committee.
ATVEF – Advanced Television Enhancement Forum. AU –
Access Unit.
BAT – Bouquet Association Table.
BCH – Bose, Ray-Chaudhuri, Hocquenghem Code. A BCH
code is a multilevel, cyclic, error-correcting, variable-length
digital code used to correct multiple random error patterns.
BER – Bit Error Rate.
BFSK – Binary Frequency Shift Keying.
BIOP – Broadcast Inter-ORB Protocol. Bit rate – The rate at
which the compressed bit stream is delivered from the channel
to the input of a decoder.
Block – A block is an array of pixel values or DCT coefficients,
usually 8-by-8 (8x8), representing luminance or chrominance
information.

[Link]/video_audio 99
Primer

Bouquet – A group of transport streams in which programs DRM – Digital Rights Management.
are identified by combination of network ID and PID (part of
DSI – DownloadServerInitiate.
DVB-SI).
DSMCC – Digital Storage Media Command and Control.
BPSK – Binary Phase Shift Keying.
DST – Data Services Table.
CA – Conditional Access. Information indicating whether a
program is scrambled. DTS – Decoding Time Stamp. Part of PES header indicating
when an access unit is to be decoded.
CAT – Conditional Access Table. Packets having PID (see
Section 8 – Transport Streams) codes of 1 and that contain DVB – Digital Video Broadcasting. Generally refers to the
information about the scrambling system. See ECM and EMM. European-initiated consortium of broadcasters, manufacturers,
regulatory bodies and others that created standards for the
CD – Compact disc.
delivery of digital television and data services. Includes DVB-C
CELP – Code Excited Linear Predictive. Channel Code – A (cable), DVB-S (satellite) and DVB-T (terrestrial) versions.
modulation technique that converts raw data into a signal that
DVB-H – Digital Video Broadcasting - Handheld.
can be recorded or transmitted by radio or cable.
DVB-SI – DVB Service Information. Information carried in a
CIF – Common Interchange Format. A 352x240 pixel format
DVB multiplex describing the contents of different multiplexes.
for 30 fps video conferencing.
Includes NIT, SDT, EIT, TDT, BAT, RST and ST (see Section 10
Closed GOP – A Group of Pictures in which the last pictures – Introduction to DVB & ATSC).
do not need data from the next GOP for bidirectional coding.
DVC – Digital Video Cassette.
Closed GOP is used to make a splice point in a bit stream.
DVD – Digital Versatile Disk or Digital Video Disk. Elementary
C/N – Carrier to Noise ratio.
Stream – The raw output of a compressor carrying a single
Coefficient – A number specifying the amplitude of a video or audio signal.
particular frequency or basis function in a transform.
ECM – Entitlement Control Message. Conditional access
CORBA – Common Object Request Broker Architecture. information specifying control words or other stream-specific
scrambling parameters.
COFDM – Coded Orthogonal Frequency Division Multiplex,
a modified form of OFDM. A digital modulation scheme using ECS – Enhanced Content Specification.
a very large number of carriers, each carrying a very low data
EIT – Event Information Table. Part of DVB-SI.
rate. Used by DVB-T.
EMM – Entitlement Management Message. Conditional
Compression – Reduction in the number of bits used to
access information specifying authorization level or services
represent an item of data.
of specific decoders. An individual decoder or a group of
CRC – Cyclic Redundancy Check. decoders may be addressed.
DAB – Digital Audio Broadcast. ENG – Electronic News Gathering. Term used to describe use
of video-recording instead of film in news coverage.
DAB-IP – Digital Audio Broadcast - Internet Protocol.
Entropy Coding – Variable length lossless coding of the digital
DAC – Digital-to-Analog Converter.
representation of a signal to reduce redundancy.
DASE – Digital TV Application Software Environment.
EOB – End of Block.
DAVIC – Digital Audio Visual Council.
EPG – Electronic Program Guide. A program guide delivered
DCT – Discrete Cosine Transform. by data transfer rather than printed paper.
DDB – DownloadDataBlock. ETSI – European Telecommunication Standard Institute.
DET – Data Event Table. FEC – Forward Error Correction. System in which redundancy
DFT – Discrete Fourier Transform. is added to the message so that errors can be corrected
dynamically at the receiver.
DII – DownloadInfoIndication.
FFT – Fast Fourier Transform.
DMB-T/H – Digital Multimedia Broadcast-Terrestrial/Handheld.
GEM – Globally Executable MHP.
Dolby Digital – See AC-3.

100 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

GOP – Group of Pictures. In transmission order a GOP starts MGT – Master Guide Table.
with an I-picture and ends with the last picture before the next
MHP – Multimedia Home Platform.
I-picture.
Motion Vector – A pair of numbers which represent the
HAVI – Home Audio Video Interoperability.
vertical and horizontal displacement of a region of a reference
HDTV – High Definition Television. picture for prediction.
HTML – HyperText Markup Language. MP@HL – Main Profile at High Level.
Huffman coding – A type of source coding that uses codes MP@LL – Main Profile at Low Level.
of different lengths to represent symbols which have unequal
MP@ML – Main Profile at Main Level.
likelihood of occurrence.
MPE – Multi-protocol Encapsulation.
IEC – International Electrotechnical Commission.
MPEG – Moving Picture Experts Group ISO/IEC JTC1/SC29/
Inter-coding – Compression that uses redundancy between
WG11, and the Standards developed by this Group.
successive pictures; also known as temporal coding.
MPEG-LA – MPEG License Agreements.
Interleaving – A technique used with error correction that
breaks up burst errors into many smaller errors. MPTS – Multi Program Transport Stream.

Intra-coding – Compression that works entirely within one MXF – Material eXchange Format.
picture; also known as spatial coding. NIT – Network Information Table. Information in one transport
IOR – Inter-operable Object Reference. stream that describes many transport streams.

IP – Internet Protocol. NPT – Normal Play Time.

I-pictures – Intra-coded Pictures. NRT – Network Resources Table.

IRD – Integrated Receiver Decoder. A combined RF receiver Null Packets – Packets of “stuffing” that carry no data but
and MPEG decoder that is used to adapt a TV set to digital are necessary to maintain a constant bit rate with a variable
transmissions. payload. Null packets always have a PID of 8191 (all ones).
(See Section 8 – Transport Streams.)
ISDB – Integrated Services Data Broadcasting, the digital
broadcasting system developed in Japan. OCAP – OpenCable™ Applications Platform.

ISDB-Tss – Single Segment ISDB-T. OFDM – Orthogonal Frequency Division Multiplexing.

ISO – International Organization for Standardization. ORB – Object Request Brokerage.

ITU – International Telecommunication Union. PAL – Phase Alternate Line.

JPEG – Joint Photographic Experts Group. PAT – Program Association Table. Data appearing in packets
having PID (see Section 8 – Transport Streams) code of zero
JTC1 – Joint Technical Committee of the IEC.
that the MPEG decoder uses to determine which programs
JVT – Joint Video Team. exist in a Transport Stream. PAT points to PMT, which, in turn,
points to the video, audio and data content of each program.
LDPC – Low-Density Parity Check Code.
PCM – Pulse Code Modulation. A technical term for an
Level – The size of the input picture in use with a given profile
analog source waveform, for example, audio or video signals,
(see Section 2 – Compression in Video).
expressed as periodic, numerical samples. PCM is an
MAC – Media Access Control. uncompressed digital signal.
Macroblock – The screen area represented by several PCR – Program Clock Reference. The sample of the encoder
luminance and color-difference DCT blocks that are all steered clock count that is sent in the program header to synchronize
by one motion vector. the decoder clock.
Masking – A psycho-acoustic phenomenon whereby certain PES – Packetized Elementary Stream.
sounds cannot be heard in the presence of others.
MDCT – Modified Discreet Cosine Transform.

[Link]/video_audio 101
Primer

PID – Program Identifier. A 13-bit code in the transport QSIF – One-quarter-resolution Source Input Format. See SIF.
packet header. PID 0 indicates that the packet contains a PAT
Quantization – a processing step that approximates signal
PID. (See Section 8 – Transport Streams.) PID 1 indicates a
values by allocating one of a number of pre-defined values.
packet that contains CAT. The PID 8191 (all ones) indicates
null (stuffing) packets. All packets belonging to the same RLC – Run Length Coding. A coding scheme that counts
elementary stream have the same PID. number of similar bits instead of sending them individually.

PLL – Phase Locked Loop. RRT – Rating Region Table.

PMT – Program Map Tables. The tables in PAT that point to RST – Running Status Table.
video, audio and data content of a transport stream. R-S – Reed-Solomon is a polynomial function used by DVB
Packets – A term used in two contexts: in program streams, for protecting up to 8 bytes within each transport packet.
a packet is a unit that contains one or more presentation RTP – Real-Time Protocol.
units; in transport streams, a packet is a small, fixed-size data
SBR – Spectral Band Replication.
quantum.
Scalability – A characteristic of MPEG-2 that provides for
Pixel – Picture element (sometimes pel). The smallest unit of
multiple quality levels by providing layers of video data.
an image, represented by one sample, or a set of samples
Multiple layers of data allow a complex decoder to produce
such as GBR or YCrCb.
a better picture by using more layers of data, while a more
Preprocessing – The video signal processing that occurs simple decoder can still produce a picture using only the first
before MPEG Encoding. Noise reduction, downsampling, cut- layer of data.
edit identification and 3:2 pulldown identification are examples
SDI – Serial Digital Interface. Serial coaxial cable interface
of preprocessing.
standard intended for production digital video signals.
Profile – Specifies the coding syntax used.
SDK – Software Development Kit.
Program Stream – A bit stream containing compressed
SDT – Service Description Table. A table listing the providers
video, audio and timing information.
of each service in a transport stream.
PS – Parametric Stereo.
SDTV – Standard Definition Television.
PS – Program Stream.
Shannon Limit – Stated by Claude Shannon in 1948, the
PSI – Program Specific Information. Information that keeps theorem describes the maximum possible efficiency of error-
track of the different programs in an MPEG transport stream correcting methods versus levels of noise interference and
and in the elementary streams in each program. PSI includes data corruption.
PAT, PMT, NIT, CAT, ECM and EMM.
SI – See DVB-SI.
PSI/SI – A general term for combined MPEG PSI and DVB-SI.
SIF – Source Input Format. A half-resolution input signal used
PSIP – Program and System Information Protocol. by MPEG-1.
PTS – Presentation Time Stamp. The time at which a Slice – A sequence of consecutive macroblocks.
presentation unit is to be available to the viewer.
SMPTE – Society of Motion Picture and Television Engineers.
PU – Presentation Unit. One compressed picture or block of
SNR – Signal-to-Noise Ratio.
audio.
SP@ML – Simple Profile at Main Level.
QAM – Quadrature Amplitude Modulation, a digital modulation
system. SPTS – Single Program Transport Stream.
QCIF – One-quarter-resolution (176x144 pixels) Common ST – Stuffing Table.
Interchange Format. See CIF. STB – Set Top Box.
QMF – Quadrature Mirror Filter. STC – System Time Clock. The common clock used to
QPSK – Quaternary Phase Shift Keying (also known as encode video and audio in the same program.
Quadrature Phase Shift Keying), a digital modulation system STT – System Time Table.
particularly suitable for satellite links.

102 [Link]/video_audio
 A Guide to MPEG Fundamentals and Protocol Analysis

Stuffing – Meaningless data added to maintain constant bit References:

rate.
1. ISO/IEC 13818 MPEG-2 System, Video, and Audio
Syndrome – Initial result of an error checking calculation.
Generally, if the syndrome is zero, there is assumed to be no 2. ISO/IEC 144961-10 MPEG-4 Part 10 Video, also known
error. as H.264

TCP/IP – Transmission Control Protocol/Internet Protocol. 3. EN 300 421 DVB-S QPSK Satellite

TDAC – Time Domain Aliasing Cancellation. A coding 4. EN 302 307 DVB-S2 Second Generation Satellite
technique used in AC-3 audio compression. 5. ITU-T J.83B QAM-B, also known as SCTE 07
T-DBM – Digital Multimedia Broadcasting -Terrestrial. 6. ATSC A/53 Digital Television Standard
TDT – Time and Date Table. Used in DVB-SI. 7. ETSI TR 101 290 DVB Measurement Guidelines
TOT – Time Offset Table. 8. SCTE 07 Digital Transmission for Cable Television
Transport Stream (TS) – A multiplex of several program 9. EN 50083-9 DVB-PI ASI and parallel
streams that are carried in packets. Demultiplexing is achieved
10. RFC 0791 Internet Protocol
by different packet IDs (PIDs). See PSI, PAT, PMT and PCR.
11. IRT / Tektronix Comparison of Subjective and Objective
Truncation – Shortening the wordlength of a sample or
Picture Quality Rating of MPEG-2 Video Report
coefficient by removing low-order bits.
12. ITU-R_BT.500-11 Methodology for the subjective
T-STD – Transport Stream System Target Decoder. A decoder
assessment of the quality of television pictures
having a certain amount of buffer memory assumed to be
present by an encoder. 13. ITU-R BS.1770/1771 Algorithms to measure audio
program loudness and true-peak audio level
TVCT – Terrestrial Virtual Channel Table.
14. ATSC A/85 Techniques for Establishing and Maintaining
UDP – User Datagram Protocol.
Audio Loudness for Digital Television
VAU – Video Access Unit. One compressed picture in
15. European Broadcast Union R 128 Loudness normalization
program stream.
and permitted maximum level of audio signals
VBV – Video Buffer Verify.
16. SCTE35 Digital Program Insertion Cueing Message for
VCO – Voltage Controlled Oscillator. Cable
VLC – Variable Length Coding. A compressed technique that
allocates short codes to frequency values and long codes to
infrequent values.
VOD – Video On Demand. A system in which television
programs or movies are transmitted to a single consumer only
when requested.
VSB – Vestigial Sideband Modulation. A digital modulation
system used by ATSC.
Wavelet – A transform using a basis function that is not of
fixed length but that grows longer as frequency reduces.
Weighting – A method of changing the distribution of the
noise that is due to truncation by pre-multiplying values.
XHTML – Extended HTML.
Y/C – Luminance and chrominance.

[Link]/video_audio 103
 Contact Tektronix:
ASEAN / Australasia (65) 6356 3900
Austria* 00800 2255 4835
Balkans, Israel, South Africa and other ISE Countries +41 52 675 3777
Belgium* 00800 2255 4835
Brazil +55 (11) 3759 7627
Canada 1 (800) 833-9200
Central East Europe and the Baltics +41 52 675 3777
Central Europe & Greece +41 52 675 3777
Denmark +45 80 88 1401
Finland +41 52 675 3777
France* 00800 2255 4835
Germany* 00800 2255 4835
Hong Kong 400-820-5835
India 000-800-650-1835
Italy* 00800 2255 4835
Japan 81 (3) 6714-3010
Luxembourg +41 52 675 3777
Mexico, Central/South America & Caribbean 52 (55) 56 04 50 90
Middle East, Asia and North Africa +41 52 675 3777
The Netherlands* 00800 2255 4835
Norway 800 16098
People’s Republic of China 400-820-5835
Poland +41 52 675 3777
Portugal 80 08 12370
Republic of Korea 001-800-8255-2835
Russia & CIS +7 (495) 7484900
South Africa +27 11 206 8360
Spain* 00800 2255 4835
Sweden* 00800 2255 4835
Switzerland* 00800 2255 4835
Taiwan 886 (2) 2722-9622
United Kingdom & Ireland* 00800 2255 4835
USA 1 (800) 833-9200

* If the European phone number above is not accessible,

please call +41 52 675 3777

Contact List Updated 10 February 2011

For Further Information

Tektronix maintains a comprehensive, constantly expanding collection of
application notes, technical briefs and other resources to help engineers
working on the cutting edge of technology. Please visit [Link]

Copyright © 2012, Tektronix. All rights reserved. Tektronix products are

covered by U.S. and foreign patents, issued and pending. Information in this
publication supersedes that in all previously published material. Specification
and price change privileges reserved. TEKTRONIX and TEK are registered
trademarks of Tektronix, Inc. All other trade names referenced are the service
marks, trademarks or registered trademarks of their respective companies.

05/12 EA/ 25W-11418-10