Multimedia

Multimedia Systems:
Compression Techniques

Ralf Steinmetz
Lars Wolf

Darmstadt University of Technology

Industrial Process and System Communications

Merckstraße 25 – 64283 Darmstadt – Germany

[Link]
1
 Multimedia Applications
User Interface
Documents

Programming Abstractions

File Sched- Synchron-

System uling ization Commun-
Database
System ication
Operating System
System
Quality-of-Service

Computer Technology

Compression
Storage Network
Audio Video

[Link]
2
 CONTENT
Multimedia
1. Motivation
2. Requirements
3. Fundamentals
4. Basic Encoding Steps
5. Basic Audio Coding Schemes
6. JPEG
7. H.261 (px64)
8. Further ITU Video Schemes (H.263, H.3xx)
9. MPEG-1
10. MPEG-2
11. MPEG-4
12. Wavelets
13. Fractal Image Compression
14. Conclusion

[Link]
3
 1. Motivation
Multimedia
Digital video in computing means:

for 1 hour 25 images/s

184 Mbit/s
82.9 Gbyte
(uncompressed) (uncompressed)

storage communication workstation

HW & SW

with 3 bytes/pixel
640*480 pixels/frame
25 images/s

[Link]
Hence, Compression is necessary
4
 2. Requirements
Multimedia
General
low delay

intrinsic scalability

high quality
compression

low complexity (e.g., ease of decoding)

efficient implementation (e.g., memory req.)

[Link]
5
 Requirements
Multimedia
Dialogue and retrieval mode requirements:
• Independence of frame size and video frame rate
• Synchronization of audio, video, and other media

Dialogue mode requirements:

• Compression and decompression in real-time
(e.g. 25 frames/s)
• End-to-end delay < 150ms

Retrieval mode requirements:

• Fast forward and backward data retrieval
• Random access within 1/2 s

Software and/or hardware-assisted implementation requirements

[Link]
6
 3. Fundamentals
Multimedia
Categories

entropy coding
hybrid
- ignoring semantics of the data coding
- lossless

source coding
entropy encoding - entropy
- based on semantic of the data and
- often lossy source
coding
channel coding
- adaptation to communication channel
- introduction of redundancy

[Link]
7
 Categories and Techniques
Multimedia
Run-Length Coding
Entropy
Huffman Coding
Coding Arithmetic Coding

Prediction DPCM
DM
FFT
Source Transformation
DCT
Coding Bit Position
Layered Coding Subsampling
Sub-Band Coding
Vector Quantization

JPEG
Hybrid MPEG
Coding H.261
proprietary: DVI RTV, DVI PLV,...

[Link]
8
 Entropy Coding: Run-Length
Multimedia
Assumption:
• Long sequences of identical symbols

Example:
... A B C E E E E E E D A C B...

compression

... A B C E M 6 D A C B...

symbol number of
occurrences
special flag

[Link]
9
 Entropy Coding: Huffman
Multimedia
Assumption:
• Some symbols occur more often than others
• E.g., character frequencies of the English language

Fundamental principle:
• Frequently occurring symbols are coded with shorter bit strings

[Link]
10
 Huffman Coding
Multimedia
Example:
• Characters to be encoded: A, B, C, D, E
• Given probabilities of occurrence:
• p(A)=0.3, p(B)=0.3, p(C)=0.1, p(D)=0.15, p(E)=0.15

coding tree probability symbol code

1 30% A 11
1
60
0 30% B 10

1
100 10% C 011
1 25
0
0 15% D 010
40
0 15% E 00

[Link]
11
 Entropy Coding: Huffman
Multimedia
Table and example of application to data stream
symbol code
A 11
B 10 A B D C A A E D C
C 011 11 10 010 011 11 11 00 010 011
D 010
E 00

[Link]
12
 Arithmetic Encoding (1)
Multimedia
Example:
• p(A)=0.2, p(B)=0.3, p(C)=0.5
Code generation
20% 30% 50%
0 0.2 0.5 1.0
A B C

20% 30% 50%

0 0.04 0.1 0.2
AA AB AC

20% 30% 50%

0.1 0.12 0.15 0.2
ACA ACB ACC

[Link]
13
 Arithmetic Encoding (2)
Multimedia
Table as result
lower bound upper bound output
A 0 0.2 -
AC 0.1 0.2 -
ACB 0.12 1.15 1
A 0 0.2 -
AA 0 0.02 0

Note
• some issues are subject to patents

[Link]
14
 Source Coding: DPCM
Multimedia
DPCM = Differential Pulse-Code Modulation

Assumptions:
• Consecutive samples or frames have similar values
• Prediction is possible due to existing correlation

Fundamental Steps:
• Incoming sample or frame (pixel or block) is predicted
by means of previously processed data
• Difference between incoming data and prediction is determined
• Difference is quantized

Challenge: optimal predictor

Further predictive coding technique:

• Delta modulation (DM): 1 bit as difference signal

[Link]
15
 Source Coding: Transformation
Multimedia
Assumptions:
• Data in the transformed domain is easier to compress
• Related processing is feasible

Example:
Fourier Transformation

time domain frequency

domain

Inverse
Fourier Transformation

FFT: Fast Fourier Transformation

DCT: Discrete Cosine Transformation

[Link]
16
 Source Coding: Sub-Band
Multimedia
Assumption:
• Some frequency ranges are more important than others

Example:
frequency spectrum of the signal

frequency
transformation / coding

Application:
• E.g., vocoder for speech communication

[Link]
17
 4. Basic Encoding Steps
Multimedia
video:
lossy
audio: lossy
lossless (sometimes lossless) lossless

data data quanti- entropy

source compressed
pre- pro-
data zation encoding data
paration cessing

e.g. e.g. e.g. e.g.

- resolution - DCT - linear - runlength
- frame rate - sub-band - DC, AC - Huffman
coding values

[Link]
18
 5. Basic Audio Coding Schemes
Multimedia
Background
• ITU driven activities

G.711: PCM
• with 64 kbps

G.722 differential PCM (DPCM)

• 48, 56, 64 kbps

G.723
• Multipulse-maximum Likelihood Quatizer (MP-MLQ): 6,3 kbps
• Algebraic Codebook Excitation Linear Prediction (ACELP) 5,3 kbps
• application: speech

[Link]
19
 Schemes for Video/Audio Conferencing
Multimedia
G.728: Low Delay Code Excited Linear Prediction (LD-CELP)
• 16 kbps
• one-way end to end delay less than 2 msec (due to CODEC algorithm)
• complex algorithm
• 16-18 MIPS in floating point required
• appr. 40 MIPS whole encoding and decoding

AV.253
• still “under consideration” at ITU
• 32 kbps

IS-54
• VSELP
• good for voice
• bad for music
• 13 kbps (appr. 8 kbps voice + 5.05 kbps forward error correction FEC)

[Link]
• driving force: Motorola (similar developments in Japan)
20
 Schemes for Mobile Telephone Networks
Multimedia
RPE-LTP (GSM)
• Regular Pulse Excitation - Long-Term Predictor
• used in European GSM: speech
• 13 kbps

GSM Half-Rate Coders

• 5.6 - 6.25 kbps
• quality and characteristics similar to RPE-LPT

[Link]
21
 Vocoder: e.g. Inmarsat IMBE Coder
Multimedia
Improved Multiband Excitation Coder IMBE
• application: maritime satellite communications
• 4,15 kbps for voice (plus 2,25 kbps for channel coding)

Principle: Vocoder
• (IMBE voiced and unvoiced individually for each frequency band)
200 - .DC modulator 200 -
300 Hz + lowpass 300 Hz

300 - DC 300 -

encoded Speech
Speech input

450 Hz + lowpass modulator 450 Hz

… … … …

2.800 - DC modulator 2.800 -

3.400 Hz + lowpass 3.400 Hz
…
replicated for
pitch analysis switch each frequency
band
Puls- Rausch-
generator generator

[Link]
22
 6. JPEG
Multimedia
“JPEG”: Joint Photographic Expert Group

International Standard:
• For digital compression and coding of continuous-tone still images:
• Gray-scale
• Color
• Since 1992

Joint effort of:

• ISO/IEC JTC1/SC2/WG10
• Commission Q.16 of CCITT SGVIII

Compression rate of 1:10 yields reasonable results (lt. Heinrichs,

Multimedia im Netz)

[Link]
23
 JPEG
Multimedia
Very general compression scheme

Independence of:
• Image resolution
• Image and pixel aspect ratio
• Color representation
• Image complexity and statistical characteristics

Well-defined interchange format of encoded data

Implementation in:
• Software only
• Software and hardware

“MOTION JPEG” for video compression

• Sequence of JPEG-encoded images

[Link]
24
 JPEG - Compression Steps
Multimedia

image image entropy

com-
pre- pro- encoding
source paration cessing pressed
quanti-
runlength
pixel
image zation image
predictor Huffman
block
MCU FDCT Arithm.

MCU: Minimum Coded Unit

FDCT: Forward Discrete Cosine Transformation

[Link]
25
 JPEG - Image Preparation
Multimedia
data units top

* * * *
* * * * line

left * * Yi right
CN
*
C2 Xi
C1
bottom
data units: samples in lossless mode, blocks with 8x8 pixels in other modes

Planes:
• 1 ≤ N ≤ 255 components Ci (e.g., one plane per color)
• Different resolution of individual components possible

Pixel resolution:
• 8 or 12 bit per pixel in lossy modes
• 2 to 16 bit per pixel in lossless mode

[Link]
26
 JPEG - Image Preparation
Multimedia
Example 4:2:2 YUV, 4:1:1 YUV, and YUV9 Coding
• Luminance (Y):
• brigthness
• sampling frequency 13.5 MHz
• Chrominance (U, V):
• color differences
• sampling frequency 6.75 MHz

[Link]
27
 JPEG - Image Preparation
Multimedia
Non-interleaved encoding:
top
* * * * * * *
left * * * * * * * right
* * * * * * *

bottom

Interleaved encoding:
C1 C2 C3
* * * * * * * * * * * * * * *
* * * * * * * * * * * * * * *
* * * * * * * * *
* * * * * * * * * * * * * * *

Minimum Coded Unit (MCU):

• Combination of interleaved data units of different components

[Link]
28
 JPEG - 4 Modes of Compression
Multimedia

lossy sequential DCT-based mode

(baseline mode)

expanded lossy DCT-based mode

lossless mode

hierarchical mode

[Link]
29
 JPEG - Baseline Mode
Multimedia
image image quanti- entropy com-
pre- pro- zation encoding
source paration cessing pressed

image image
8x8
blocks FDCT tables
tables tables

Baseline mode is mandatory for all JPEG implementations:

• Often restricted to certain resolution
• Often only three planes with predefined color set-up

Image preparation:
• Pixel resolution of p=8 bit
• 8 x 8 pixel blocks (data units)

[Link]
30
 JPEG - Baseline Mode: Image Processing
Multimedia
Forward Discrete Cosine Transformation (FDCT):
7 7
1 ( 2x + 1 ) uπ ( 2y + 1 ) vπ
S vu = --- C u C v
4 ∑ ∑ s yx cos ----------------------------- cos -----------------------------
16 16
x = 0y = 0

with:
1
cu, cv = ------- , for u, v= 0; else cu, cv = 1
2

Formula applied to each block for all 0 ≤ u, v ≤ 7:

• Blocks with 8x8 pixel result in 64 DCT coefficients:
• 1 DC-coefficient S00: basic color of the block
• 63 AC-coefficients: (likely) zero or near-by zero values

Different significance of the coefficients:

• DC: most important
• AC: less important

[Link]
31
 JPEG – Baseline Mode: Image Processing
Multimedia
FDCT transforms:
• blocks into blocks
• not pixels into pixels

Example:
• Calculation of S00

…
# # # # # # # # * * * * * * * *
# # # # # # # # * * * * * * * *
# # # # # # # # * * * * * * * *
# # # # # # # # … * * * * * * * *
# # # # # # # # * * * * * * * *
# # # # # # # # * * * * * * * *
# # # # # # # # * * * * * * * *
# # # # # # # # * * * * * * * *

[Link]
32
 JPEG - Baseline Mode: Quantization
Multimedia
Use of quantization tables for the DCT-coefficients:
• Map interval of real numbers to one integer number
• Allows to use different granularity for each coefficient

[Link]
33
 JPEG - Baseline Mode: Entropy Encoding
Multimedia
DC-coefficients:
• Compute the differences:
DCi-1 DCi

... block block ...

DIFF = DCi - DCi-1

• Use differences instead of the DCi values

[Link]
34
 JPEG - Baseline Mode: Entropy Coding
Multimedia
63 AC coefficients:
• Ordering in ‘zig-zag’ form
AC01 AC07
* * * * * * * *
DC * * * * * * * *
* * * * * * * *
* * * * * * * *
* * * * * * * *
* * * * * * * *
* * * * * * * *
AC70 * * * * * * * *
AC77

• reason: coefficients in lower right corner are likely zero

• Huffman coding of all coefficients:
Transformation into a code where amount of bits depends on frequency of
respective value
• Subsequent runlength coding of zeros

[Link]
35
 JPEG - Extended Lossy DCT-Based Mode
Multimedia
Pixel resolution 8 to 12 bit

Sequential image display:

• Top ➨ bottom
• Good for small images and fast processing

Progressive image display:

• Coarse ➨ fine
• Good for large and complicated images

[Link]
36
 JPEG - Extended Lossy DCT-Based Mode
Multimedia
Principle:
• Coefficients stored in buffer after quantization
• Order of pixel/block processing changed

By spectral selection:
• Selection according to importance of DC, AC value
• All DC values of whole image first
• All AC values in order of importance subsequently

By successive approximation:
• Selection according to position of bits
• First the most significant bit of all blocks
• Then the second significant bit of all blocks
• Until the least significant bit of all blocks

[Link]
37
 JPEG - Lossless Mode
Multimedia
Image preparation:
• On pixel basis (2-16 bit/pixel)

Image processing:
• Selection of a predictor for each pixel
code prediction
0 no prediction
1 x=A
c b 2 x=B
3 x=C
a x 4 x=A+B+C
5 x=A+((B-C)/2)
6 x=B+((A-C)/2)
7 x=(A+B)/2

Entropy coding:
• Same as lossy mode
• Code of chosen predictor and its difference to the actual value

[Link]
38
 JPEG - Hierarchical Mode
Multimedia
Coding of each image with several resolutions:
• Image scaling
• Differential encoding
• First, coded with lowest resolution image A
• Coded with increasing horizontal & vertical resolution image A’
• Difference between both images is computed B = A - A’
• Iteration for higher resolutions

Features:
• Requires more storage and higher data rate
• Fast decoding process
• Used for scalable video
• Similar to Photo-CD

[Link]
39
 7. H.261 (px64)
Multimedia
Video codec for audiovisual services at p x 64kbit/s:
• CCITT standard from 1990
• For ISDN
• With p=1,..., 30

Technical issues:
• Real-time encoding/decoding
• Max. signal delay of 150ms
• Constant data rate
• Implementation in hardware (main goal) and software

[Link]
40
 H.261 - Image Preparation
Multimedia
Fixed source image format

Image components:
• Luminance signal (Y)
• Two color difference signals (Cb,Cr)

Subsampling according to CCIR 601 (4:1:1)

Quarter Common Intermediate Format (QCIF) resolution:

• Mandatory
• Y: 176 x 144 pixel
• At 29.97 frames/s appr. 9.115 Mbit/s (uncompressed)

[Link]
41
 H.261 - Image Preparation
Multimedia
Common Intermediate format (CIF) resolution:
• Optional
• Y: 352 x 288 pixel
• At 29.97 frames/s appr. 36.46 Mbit/s (uncompressed)

Layered structure:
• Block of 8 x 8 pixels
• Macroblock of:
• 4 Y blocks
• 1 Cr block
• 1 Cb block
• Group of blocks (GOBs) of 3 x 11 macroblocks
• Picture:
• QCIF picture: 3 GOBs
• CIF picture: 12 GOBs

[Link]
42
 H.261 - Image Compression
Multimedia
Intraframe coding:
• DCT as in JPEG baseline mode

Interframe coding, motion estimation:

Frame 1 Frame 2

• Search of similar macroblock in previous image

• Position of this macroblock defines motion vector
• Search range is up to the implementation:
• i.e., motion vector may always be 0

[Link]
43
 H.261 - Image Compression
Multimedia
Interframe coding, further steps:
• Results:
• Difference between similar macroblocks
• Motion vector
• Difference of macroblocks:
• DCT if value higher than a specific threshold
• No further processing if value less than this threshold
• Motion vector:
• Components are coded yielding code words of variable length

Quantization:
• Linear
• Adaptation of step size constant data rate

[Link]
44
 8. Further ITU Video Schemes (H.263, H.3xx)
Multimedia
H.263
• extension to H.261
• bitrate: H.263 approx. 2.5 x H.261

Source Image Formats

Forma H.261 H.263

Pixels
t Encoder Decoder Encoder Decoder
SQCIF 128 x 96 optional required
QCIF 176 x 144 required required
CIF 352 x 144 optional optional
4CIF 704 x 576
not defined optional
16CIF 1408 x 1152
[Link]
45
 H.263
Multimedia
Differences of H.263 compared to H.261
• optinal PB-frames (2 combined pictures: 1 B- & 1 P-Frame)
• optional overlapped block motion compensation
• optional motion vector pointing outside image
• half pel motion compensation (instead of full pel)
• JPEG is the still picture mode
• no included error detection and correction
• ..

[Link]
46
 H.320, H.32x Family
Multimedia
H.320 specifies (as overview) videophone for ISDN

H.310
• adapt MPEG 2 for communication over B-ISDN (ATM)

H.321
• define videoconferencing terminal for B-ISDN (instead of N-ISDN)

H.322
• adapts H.320 for guaranteed QoS LANs (like ISO-Ethernet)

H.323
• videoconferencing over non-guaranteed LANs

H.324
• Terminal for low bit rate communication (over V.34 Modems)

[Link]
47
 9. MPEG-1
Multimedia
Motion Picture Expert Group (MPEG):

• ISO/IEC JTC1/SC29/WG11
• ISO IS 11172 since 3/93
• Based on experiences with JPEG and H.261
• Target: coding at about 1.5 Mbit/s for
• Audio and
• Video

Evolution:
• MPEG-1 was starting point
• additional MPEG standards follow
• MPEG-2:
• higher data rates for high-quality video
• MPEG-4:
• lower data rates for e.g. mobile communication
• additional functionality
[Link] based on analysis of image contents
48
 MPEG - Features
Multimedia
MPEG
audio video system

combined stream
coding data stream coding data stream
common buffer
management

Consideration of other standards:

• JPEG
• H.261

Symmetric and asymmetric compression

Constant data rate, should be < 1856 kbit/s

Target rate about 1.5 Mbit/s

[Link]
49
 MPEG - Video: Preparation Step
Multimedia
Fixed image format

Color subsampling:
• Y, Cr, Cb
• 4:2:0

Resolution:
• Should be at most 768 x 576 pixel
• 8 bit/pixel in each layer (i.e., for Y, Cr, Cb)
• 14 pixel aspect ratios
• 8 frame rates

No user defined MCU like JPEG

No progressive mode like JPEG

[Link]
50
 MPEG - Video: Processing Step
Multimedia
4 types of frames:

I-frames (intra-coded frames):

• Like JPEG
• Real-time decoding demands

P-frames (predictive coded frames):

• Reference to previous I- or P-frames
• Motion vector
• MPEG does not define how to determine the motion vector
• difference of similar macroblocks is DCT coded
• DC and AC coefficients are runlength coded

[Link]
51
 MPEG - Video: Processing Step
Multimedia
B-frames (bi-directional predictive coded frames):
• Reference to previous and subsequent (I or P) frames
• Interpolation between macro blocks

D-frames (DC-coded frames):

• Only DC-coefficients are DCT coded
• For fast forward and rewind

[Link]
52
 MPEG - Video Coding
Multimedia
Sequence of I-, P-, and B-frames:
I
B References
B
• I-Frames (Intracoded)
P
B • P-Frames (Predictive Coded)
B
• B-Frames (Bidirectionally Coded)
P
• (D-Frames (DC Coded))
I

Sequence:
• Defined by application
• E.g., I B B P B B P B B I B B P B B P B B…
• Order of transmission is different: I P B B ...

[Link]
53
 MPEG - Video: Implications
Multimedia
Random access
• at I-frames
• at P-frames: i.e. decode previous I-frame first
• at B-frame: i.e. decode I and P-frames first

Editing
• decoded data
• loss of quality (encode -> decode -> encode -> ...)
• application of all video editing functions
• encoded data (previous to entropy encoding)
• preservation of quality
• transition effects as function in the DCT domain
• morphing, non-block conform overlay very difficult
• encoded data
• preservation of quality
• today: too complex, if possible, i.e. need for entropy decoding

[Link]
54
 MPEG - Audio Coding: Fundamentals
Multimedia
80

fm = 0.25 1 4 kHz
60

Sound Pressure Level (dB)

40
av
masking
patterns
20

absolute threshold
0 of hearing

0.02 0.05 0.1 0.2 0.5 1 2 5 10 20

frequency (kHz)

Masking threshold in the frequence domain

• narrowband random noise
• depends on frequency

[Link]
55
 MPEG - Audio Coding: Fundamentals
Multimedia

80
100 dB

60

Sound Pressure Level (dB)

80

40

60

20 40

20
0

0.02 0.05 0.1 0.2 0.5 1 2 5 10 20

frequency (kHz)

Masking threshold in the frequence domain

• narrowband random noise
• depends on amplitude

[Link]
56
 MPEG - Audio Coding: Fundamentals
Multimedia
60 pre- simultaneous- post-masking-
40

masker

SLT
20

0
-50 50 100 150 ms 0 50 100 150 200
Dt tv

Masking in Time Domain

• after and before the event
• depends on (to some extent) amplitude

[Link]
57
 MPEG - Audio Coding
Multimedia
sub-band 32 quanti- entropy
coder &
coding zation frame
packing

psychoacoustical controls
model

Audio channel:
• Between 32 and 448 kbit/s
• In steps of 16 kbit/s

Definition of 3 layers of quality

• Layer 1: max. 448 Kbit/s
• Layer 2: max. 384 Kbit/s (most often used, also as MUSICAM in DAB)
• Layer 3: max. 320 Kbit/s

[Link]
58
 MPEG - Audio Coding
Multimedia
Compatible to encoding of CD-DA and DAT:
• Sampling rates:
• 32 kHz
• 44,1 kHz
• 48 kHz
• Sampling precision:
• 16 bit/sample

Audio channels:
• Mono (single, 1 channel)
• Stereo (2 channels)
• dual channel mode (independent, e.g., bilingual)
• optional: joint stereo (exploits redundancy and irrelevancy)

[Link]
59
 MPEG - Audio Coding: Application
Multimedia
DAB Digital Audio Broadcasting
• uses MPEG layer 2
• data compression also known as “MUSICAM”
(Masking pattern adapted Universal Subband Integrated Coding And
Multiplexing)

Delays
• max. of 30 ms encoding
• max. of 10 ms decoding
• based on VLSI

[Link]
60
 MPEG - Audio and Video Data Streams
Multimedia
Audio Data Stream Layers:
1. Frames
2. Audio access units
3. Slots

Video Data Stream Layers:

1. Video sequence layer
2. Group of pictures layer
3. Single picture layer
4. Slice layer
5. Macroblock layer
6. Block layer

[Link]
61
 Follow-Up MPEG Standards
Multimedia
MPEG-2
• 1993: Committee Draft
• 1994: Draft International Standard
• MPEG-2 Audio, MPEG-2 Video in March 1994, MPEG-2 System in June 1994
• Late 1994: International Standard
• Commercial MPEG-2 realizations available: digital TV

MPEG-3
• Initially HDTV
• MPEG-2 scaled up to subsume MPEG-3

MPEG-4
• Originally targeted at very low bit rates (less than 128 Kbit/s)
• Additional functionality based on analysis of image contents

[Link]
62
 10. MPEG-2
Multimedia
From MPEG-1 to MPEG-2
• Improvement in quality
• from VCR to TV to HDTV
• No CD-ROM based constraints
• higher data rates

Data rates
• MPEG-1: about 1.5 Mbit/s
• MPEG-2: 2-100 Mbit/s

Also later known as H.262

[Link]
63
 MPEG-2 Video
Multimedia
Inclusion of interlaced video format

Increase resolution, more than CCIR 601

Defined as:
• 5 profiles (simple, main,..)
• 4 levels (with increasing resolution,...)

Other additional features

• DCT coefficients may be coded with a non-linear quatization function

[Link]
64
 MPEG-2 Video: Scaling
Multimedia
Motivation
• analog: contiuous decrease in quality if errors occur
• digital: need for tolerance whenever error occur, i.e scaling

Option: Spatial scaling

• reduction of resolution
• approach
• image sampled with half resolution, then MPEG algorithms applied,
output processd with better FEC (base layer)
• Image decoded, substracted from original, to difference MPEG algorithms applied,
output processd with worseFEC (enhanced layer)

Option: Signal to Noise SNR scaling

• noise introduced by
• quantization errors and visible block structures
• approach
• Base layer: DCT output, more significant bits encoded with better FEC
[Link]
• Enhanced layer:DCT output, less significant bits encoded with worse FEC
65
 MPEG-2 Video Profiles und Levels
Multimedia
High Level ≤ 80 Mbit/s ≤ 100 Mbit/s
1920 pixels/line
1152 lines
High-1440 Level ≤ 60 Mbit/s ≤ 60 Mbit/s ≤ 80 Mbit/s
1440 pixels/line
1152 lines
Main Level ≤ 15 Mbit/s ≤ 15 Mbit/s ≤ 15 Mbit/s ≤ 20 Mbit/s
720 pixels/line
576 lines
Low Level ≤ 4 Mbit/s ≤ 4 Mbit/s
352 pixels/line
288 lines
Simple Main SNR Spatial High
Profile Profile Scalable Scalable Profile
Profile Profile
LEVELS
and No B-frames B-frames B-frames B-frames B-frames
PROFILES 4:2:0 4:2:0 4:2:0 4:2:0 4:2:0 or 4:2:2
Not Scalable Not Scalable SNR SNR Scalable SNR Scalable
Scalable or Spatial or Spatial
Scalable Scalable

[Link]
66
 MPEG-2 Audio
Multimedia
Up to
• 5 full bandwidth channels (surround system)
• left and right front
• center (in frot)
• left and right back with (x and y = 0.71)

Left for Stereo = Left_f + xCenter + yLeft_b

Right for Stereo = Right_f + xCenter + yRigtht_b

• and 7 multilingual/commentary channels

Improved quality at or below 64 kbit/s

Compatible to MPEG-1
• all MPEG-1 audio format can be processed by MPEG-2
• Only 3 MPEG-2 audio codec will not provide backward compatibility
(in the range between 256 - 448 Kbit/s)

[Link]
67
 MPEG-2 System
Multimedia
Steps
1. audio and video combined to “Packetized Elementary Stream (PES)“
2. PES(es) combined to “Program Stream” or “Transport Stream”

Program stream:
• Error-free environment
• Packets of variable length
• One single stream with one timing reference

Transport stream:
• Designed for “noisy“ (lossy) media channels
• Multiplex of various programs with one or more time bases
• Packets of 188 byte length

Conversion between Program and Transport Streams possible

[Link]
68
 11. MPEG-4
Multimedia
Goals

MPEG-4 (ISO 14496) Originally:

• Targeted at systems with very scarce resources
• To support applications like
• Mobile communication
• Videophone and E-mail
• Max. data rates and dimensions (roughly):
• Between 4800 and 64000 bits/s
• 176 columns x 144 lines x 10 frames/s

Further demand:
• To provide enhanced functionality
to allow for analysis and manipulation of image contents

[Link]
69
 MPEG-4: Components
Multimedia
Concept
• API for coded representation of objects
• not algorithm itself

Components
• definition of the object interface (as kind of API)
• mechanism to combine objects
• to construct the compression algorithms & profiles
• specification how to download new objects
• rules (syntax) how to parse all mentioned above

[Link]
70
 MPEG-4: Basic Idea
Multimedia
Code and manipulate individual objects:
• Video objects
• Audio objects

1
„Rhubarb 2
3
Rhubarb“
Audio Audio
object 1 video objects object 2

Include:
• Natural objects
• Synthectic objects

Coding steps:
• Analyse image contents => identify objects
• Code individual objects independently of each others

[Link]
71
 MPEG-4: Elements of Standard
Multimedia
Element 1: Set of coding tools
• To support
• efficient compression
• object-based operations
• scalability
• error robustness
• User can select appropriate tools from toolset
• Toolset is extendable

Element 2: Syntax for descriptions

• MSDL: MPEG-4 Syntactic Description Language
• To describe individual objects
• and operations on objects:
• decoding
• manipulation
• combination

[Link]
72
 MPEG-4: Standardization
Multimedia
Standardization is currently under way:
• 1993: Work on MPEG-4 started
• 1998 (?): International MPEG-4 standard finished

Parts of coding toolset will be drawn from existing standards:

• MPEG-1, MPEG-2, H.261, H.263

[Link]
73
 12. Wavelets
Multimedia
Motivation

DCT problems:
• at high compression ratio ⇒ block structure becomes visible
• scaling as add-on ⇒ additional effort
• DCT function is fixed ⇒ can not be adapted to source data

Goal for developing Wavelets:

• to get a better identification of which data is relevant to human perception
⇒ higher compression ratio

• iterative operation on whole image

⇒ overcomes visible block structures & introduces inherent scaling

[Link]
74
 Wavelets: Principle
Multimedia
Idea:
• To split image recursively by using high and low pass filters
read by
read by column
line lower
L c1 ...
frequencies
L
H transformed
d11 image with
L d1 2 reduced size
H
higher
L Low Pass H d1 3 frequencies
H High Pass

Next Steps:
• Quantization (according to importance) of transformed images
• Entropy encoding

[Link]
75
 Wavelet Compression / Decompression
Multimedia
Compressor

Forward Wavelet
Quantizer Encoder
Transformation

Inverse Wavelet
DeQuantizer Decoder
Transformation

Decompressor

• Wavelets are various functions, i.e. Function Class

• Application of Algorithm of Mallat for archieving efficiency

[Link]
76
 Wavelets: Further Issues
Multimedia
Edge detection reduces high frequencies:
• first extract detected edges
• then apply wavelets to such a filtered image

Application to video:

In-2 ...
In-1 In-1 - In-2
Image n In - In-1
Compute Wavelet
differences compressor

t Im t ...

[Link]
77
 13. Fractal Image Compression
Multimedia
Idea:
• To search for self-similarities in image by using affine transformations and
brightness-correction
• To build a function µ with image as fix point by coding similarity information

[Link]
78
 Fractal Image Decompression
Multimedia

[Link]
79
 14. Conclusion
Multimedia
JPEG:
• Very general format with high compression ratio
• SW and HW for baseline mode available

H.261:
• Established standard by telecom world
• Preferable hardware realization

MPEG family of standards:

• Video and audio compression for different data rates
• Asymmetric (focus) and symmetric

Proprietary systems: e.g. DVI Product

• Migration to the use of standards

Next steps: wavelets, fractals, models of objects

[Link]
80