SIP (2019), vol. 8, e30, page 1 of 16 © The Authors, 2019.

This is an Open Access article, distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives licence
([Link] which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is
unaltered and is properly cited. The written permission of Cambridge University Press must be obtained for commercial re-use or in order to create a derivative work.
doi:10.1017/ATSIP.2019.23

overview paper

A Comprehensive Video Codec Comparison

thorsten laude, yeremia gunawan adhisantoso, jan voges, marco munderloh and
jörn ostermann

In this paper, we compare the video codecs AV1 (version 1.0.0-2242 from August 2019), HEVC (HM and x265), AVC (x264),
the exploration software JEM which is based on HEVC, and the VVC (successor of HEVC) test model VTM (version 4.0 from
February 2019) under two fair and balanced configurations: All Intra for the assessment of intra coding and Maximum Coding
Efficiency with all codecs being tuned for their best coding efficiency settings. VTM achieves the highest coding efficiency in both
configurations, followed by JEM and AV1. The worst coding efficiency is achieved by x264 and x265, even in the placebo preset
for highest coding efficiency. AV1 gained a lot in terms of coding efficiency compared to previous versions and now outperforms
HM by 24 BD-Rate gains. VTM gains 5 over AV1 in terms of BD-Rates. By reporting separate numbers for JVET and AOM
test sequences, it is ensured that no bias in the test sequences exists. When comparing only intra coding tools, it is observed that
the complexity increases exponentially for linearly increasing coding efficiency.

Keywords: AV1, Codec comparison, HEVC, Video coding, VVC

Received 28 May 2019; Revised 22 October 2019

I. INTRODUCTION (IETF) as Internet Video Codec (NetVC). The finalization

of the standardization process was scheduled for 2017 but
For several decades, the market for standardized video initially delayed until the end of 2018 [16]. At the time of
codecs was dominated by the standardization groups ISO, writing this manuscript, the official status of NetVC is that
IEC, and ITU-T: MPEG-1 [1], MPEG-2/H.262 [2], H.263, the requirements for the standards were finalized in March
MPEG-4 Visual [3], and Advanced Video Coding (AVC, 2019 and that the submission of the codec specification for
also referred to as MPEG-4 Part 10 and H.264) [4,5] are approval is aspired for December 2019 [17].
some standards in this line. In 2013, the steady improve- Concurrently, ISO/IEC and ITU-T established the Joint
ment of video coding algorithms resulted in High Efficiency Video Exploration Team (JVET) in October 2015 to explore
Video Coding (HEVC) which was standardized as MPEG- technologies for a potential HEVC successor. For this pur-
H Part 2 by ISO/IEC and as H.265 by ITU-T [6]. A reference pose, a reference software called Joint Exploration Model
implementation of HEVC is available with the HM software (JEM) was developed which includes a variety of novel
[7]. Compared to its predecessor standard AVC, HEVC con- coding tools [18]. In the process of the JVET activities, it
siderably increases the coding efficiency. Depending on the was revealed that the new test model provides sufficient
selected configuration, HEVC achieves a 40–60 bit rate evidence to justify to formally start a new standardization
reduction while maintaining the same visual quality [8,9]. project [19]. The new standard is referred to as Versatile
After the finalization of HEVC, the research for further Video Coding (VVC) and is planned to be finalized in 2020.
improvements continued [10,11]. The Versatile Test Model (VTM) [20] was established to
More recently, new participants entered the market for assess the performance of VVC.
video codecs. Among the proposed codecs are VP8 [12], The purpose of the reference implementations and test
VP9 [13], Daala [14], and Thor [15]. The participants respon- models HM, JEM, and VTM is to enable the evaluation
sible for these codecs and many more participants (e.g. of new coding tools and to demonstrate one exemplary
Amazon, Facebook, Intel, Microsoft, Netflix) joined their and straight-forward implementation of the correspond-
efforts in the Alliance for Open Media (AOM) to develop ing standard. Not much optimization, e.g. for fast encod-
the video codec AV1. Furthermore, AV1 is a contender ing, was performed for these implementations. It is safe
for standardization by the Internet Engineering Task Force to assume that it is therefore unlikely that these reference
implementations will be deployed in real-world products.
Institut für Informationsverarbeitung, Leibniz University Hannover, Appelstr. 9A,
Hannover 30167, Germany
Instead, highly optimized encoders are used. Therefore,
we also evaluate the codecs x264 and x265 which imple-
Corresponding author:
Thorsten Laude
ment the AVC and HEVC standards, respectively. For these
Email: laude@[Link] two codecs, two presets are used. The medium preset is a

[Link] Published online by Cambridge University Press 1

 2 thorsten laude et al.

typical trade-off between coding efficiency and computa- We structure our answer to this question in the fol-
tional resource requirements while the placebo preset max- lowing four parts: choice of codec implementation, codec
imizes the coding efficiency at the cost of a considerable configuration, metrics, and test sequences.
amount of complexity [21].
Given these eight codec implementations – HM as
state-of-the-art, JEM, VTM, and AV1 as contenders, and
x264 (medium and placebo) as well as x265 (medium and
A) Codec implementations
placebo) as optimized encoders – it is of great interest to The difficulty of comparing video codecs starts with the
assess and compare their performance. This comparison difference between video coding standards and particular
can be performed in terms of coding efficiency but also in encoder implementations of these standards. The standards
terms of computational complexity. are only long text documents which cannot be evaluated
For some codecs, e.g. HM and JEM, straightforward in simulations. Only the implementations can be used for
comparability is given because both codecs share the same simulations. However, two encoder implementations pro-
foundation (with JEM being an extension of HM) and Com- ducing bitstreams compliant with the same standard can
mon Test Conditions are defined to configure both codecs be very different. One could distinguish between refer-
similarly [22]. To include AV1 or optimized encoders in ence implementations like HM and optimized encoders like
a fair comparison is more challenging because their soft- x265.
ware structures and working principles are fundamentally
different. This also explains why existing comparisons of
HEVC with JEM, VP8, VP9, or AV1 in the literature come
to different conclusions [13,23,24].
B) Encoder configurations
In this paper, we compare the codecs under well-defined Depending on the application and available computational
and balanced conditions. First, we analyze the difficulty of resources, encoders can be configured in many different
comparing video codecs in Section II. An overview of the ways. Among the choices to be made are restrictions dur-
technologies in the codecs is given in Section III. Based on ing the rate-distortion optimization [32] for partitioning
the analysis in the preceding sections, we introduce our two options to be tested, the decision which coding tools should
codec configurations which we use for the comparison in be enabled, and for parameters of the coding tools like
Section IV. In Section V and in Section VI, we compare the motion estimation search range. The x264 and x265 imple-
performance of the codecs in terms of coding efficiency and mentations allow the configuration of coding tools by pre-
complexity, respectively. Section VII concludes the paper. sets. Depending on the selected preset, a different trade-off
between computational complexity and coding efficiency is
made. When comparing the fastest preset (ultrafast) with
II. ON THE DIFFICULTY OF the most efficient preset (placebo), the bit rate can differ by
COMPARING VIDEO CODECS 179 for a 720p video encoded at the same quality [21].
Also, the tuning of the encoder can vary, e.g. it can
Our motivation for this manuscript emerged at the Pic- be tuned for PSNR or some subjective criterion. Only if
ture Coding Symposium (PCS) 2018 where we presented the codecs are tuned for the same criterion and if this
our codec comparison work [24] together with three other criterion corresponds to the metric used for the evaluation,
codec comparison works [25–27]. These four works com- the results are meaningful. This is, for example, the case
pared the same video coding standards. In doing so, the if the codecs are tuned for PSNR and BD-Rates are used for
findings of the works are quite different: for example, in one the evaluation.
work [24] HEVC is considerably better than AV1 while it is The group of pictures (GOP) structure is an important
the other way around in another work [27]. aspect of the encoder configuration as well to ensure a fair
The observation of inconclusive results is sustained when comparison. Depending on the available reference pictures,
other published works are studied. For example, Feldmann the efficiency of motion-compensated prediction can vary
finds that AV1 is up to 43 better than AV1 [28] while Grois considerably [33].
et al. find that HEVC is 30 better than AV1 [29]. Liu con- Intra coding is an essential part of all video coding appli-
cludes that on average AV1 is 45 better than AVC, the cations and algorithms: it is used to start transmissions,
predecessor of HEVC which is allegedly outperformed by for random access (RA) into ongoing transmissions, for
HEVC by 50, while being 5869 times as complex at the error concealment, in streaming applications for bit rate
same time [30]. An online codec comparison based on a adaptivity in case of channels with varying capacity, and
limited set of videos and configurations is available at [31]. for the coding of newly appearing content in the currently
Discussion among the authors of said conference session coded picture. However, pictures that are all-intra coded,
led to the conclusion that all of these very different numbers i.e. without motion-compensated prediction, can require
for the (apparently) same experiment are plausible. So the 10–100 times the bit rate of motion-compensated pictures
following question remains: to achieve the same quality [34]. Therefore, the number and
How can these numbers be so different while being cor- temporal distance of all-intra pictures greatly influence the
rect at the same time? coding efficiency.

[Link] Published online by Cambridge University Press

 a comprehensive video codec comparison 3

C) Metrics considered necessary because a single quaternary-tree par-

titioning was not flexible enough to meet the requirements
Different metrics can be employed to evaluate video codecs.
of prediction and transform coding at the same time. It
In most cases, BD-Rates are calculated following [35,36]. For
implied a certain amount of overhead to signal the indepen-
the BD-Rate, the average bit rate difference for the same
dent partitioning configuration of a CTU. This partition-
quality between four data points is calculated. PSNR and
ing scheme is replaced in JEM by a quarternary-tree plus
SSIM [37] are common metrics to measure the quality.
binary-tree (QTBT) block structure. CTUs (whose maxi-
A metric called Video Multimethod Assessment Fusion
mal size is increased from 64 × 64 to 128 × 128 to reflect
(VMAF) for the approximation of perceptual quality, which
increasing spatial video resolutions) are partitioned using
combines multiple methods by machine learning, gained
a quarternary-tree (the term quad-tree can be used as an
attention recently [38–40]. Also, subjective tests can be con-
alternative) followed by a binary tree. Thereby, CUs can be
ducted to assess the perceptual quality. For some kinds
square or rectangular. This more flexible partitioning allows
of video coding algorithms, e.g. when artificial content is
CUs, PUs, and TUs to have the same size which circum-
synthesized during the decoding process [41,42], subjective
vents the signaling overhead of having three independent
tests are inevitable because PSNR measurements are not
partitioning instances.
meaningful for the artificial content [43].
Intra prediction: In HEVC, the intra prediction consists
of the reference sample value continuation in 33 angular
D) Test sequences modes, one DC mode, and one planar mode. In JEM, the
The content which is encoded by video codecs is very number of angular modes is extended to 65. Addition-
diverse: videos can be camera-captured or computer- ally, the precision of the fractional pel filters for directional
generated, they can be produced with professional equip- modes is increased by using 4-tap instead of 2-tap filters.
ment or by consumer equipment, they can contain lots of Boundary filters are applied for more directional modes to
motion or no motion at all, the spatial and temporal resolu- reduce the occurrence of abrupt boundaries. The position-
tion can be low or very high (up to 4k or 8k), the bit depth dependent intra prediction combination (PDPC), which
can vary (typically either 8 or 10 bit per sample value), etc. combines the usage of filtered reference samples and unfil-
It is known, e.g. from [44], that different codecs perform tered reference samples, is used to improve the planar mode.
differently well depending on the type of content. There- Typically, there remains some redundancy between the
fore, the selection of test sequences covering said diversity luma component and the chroma components. To exploit
is important. this redundancy, a cross-component linear model (CCLM)
similar to, e.g. [52] is adopted in JEM. With this algorithm,
chroma blocks are predicted based on the corresponding
III. CODEC OVERVIEWS luma blocks.
Inter prediction: Multiple novel coding tools for inter
It is assumed that the reader is familiar with the technologies prediction are included in JEM. Sub-CU motion vec-
in AVC and HEVC. Various works [5,45–50] in the litera- tor prediction (using Alternative Temporal Motion Vector
ture give introductions to these codecs. The other codecs Prediction, ATMVP, and Spatial-temporal Motion Vector
are introduced in the following. Prediction, STMVP) allows the splitting of larger CUs into
smaller sub-CUs and the prediction of a more accurate
motion vector field for these sub-CUs via additional merge
A) JEM candidates. Then on a CU-level activatable Overlapped
JEM extends the underlying HEVC framework by modifi- Block Motion Compensation (OBMC) uses the motion
cations of existing tools and by adding new coding tools. information of neighboring sub-CUs in addition to the
In what follows, we briefly address the most important motion information of the currently coded sub-CU to pre-
modifications. A comprehensive review can be found in dict multiple signals for the current sub-CU which are
[51]. combined by a weighted average. Conceptually (without the
Block partitioning: In HEVC, three partitioning trees adaptivity) this can also be initially found in H.263. To cope
were used to further split Coding Tree Units (CTU) which with illumination changes between the current CU and the
had a maximum size of 64 × 64. The CTU was further reference block, a Local Illumination Compensation (LIC)
split into Coding Units (CU) using a quaternary-tree. For is defined. With LIC, the illumination is adjusted using a lin-
the leaf nodes of this quaternary-tree, the prediction mode, ear model whose parameters are derived by a least-squares
i.e. intra or inter, was determined. Subsequently, the CUs approach. To improve the prediction of content with non-
were partitioned into one, two, or four rectangular Predic- translative motion, JEM supports affine motion compen-
tion Units (PU) for which the parameters of the prediction sation. Multiple techniques are employed to improve the
mode were set independently and for which the prediction motion vector accuracy: The available motion information
was performed. A second quaternary-tree started on CU after block-wise motion compensation can be improved
level to partition the CU into Transform Units (TU) for using Bi-directional Optical Flow (BIO), a Decoder-side
which the transform coding was performed. This complex Motion Vector Refinement (DMVR) is applied in case of bi-
partitioning scheme with multiple quaternary-trees was prediction, and Pattern Matched Motion Vector Derivation

[Link] Published online by Cambridge University Press

 4 thorsten laude et al.

(PMMVD) is used to derive motion information for merged Furthermore, the context model states of already coded pic-
blocks at the decoder. The CU-level Locally Adaptive tures can be used as initialization of the state of the currently
Motion Vector Resolution (LAMVR) enables the signaling coded picture.
of motion vector differences with full-pel, quarter-pel, and
four-pel precision. Additionally, the precision of the internal
motion vector storage is increased to 1/16 pel (and 1/32 pel
B) VTM
for chroma). For the first version of VTM, which was developed in April
Transform coding: The transform coding techniques of 2018, a conservative approach was chosen for the inclusion
HEVC are very similar for different block sizes and different of new coding tools. The two main differences to HEVC
modes. For almost every case, a discrete cosine transform were a completely new partitioning scheme and the removal
(DCT-II) is used. Intra-coded 4 × 4 TUs constitute the only of coding tools and syntax elements which were not consid-
deviation as they are coded with a discrete sine transform ered as beneficial any more [55]. In subsequent versions of
(DST-VII). In contrast to that, JEM can rely on a greater VTM up to the current version 4.0, new coding tools were
variety of selectable core transforms from the DCT and DST steadily integrated into VTM. The new coding tools are dis-
families (DCT-II, DCT-V, DCT-VIII, DST-I, and DST-VII). cussed in the following. Some of them are known from JEM
Depending on the selected mode (intra or inter), and in while others were firstly introduced for VTM.
case of intra depending on the selected direction, a subset Partitioning: Similarly to JEM, the necessity for indepen-
of the available core transforms is formed and one trans- dent trees for mode selection, prediction, and transform
form from this subset is selected via rate-distortion (RD) coding was overcome in most cases by introducing a more
optimization. This technique is referred to as Adaptive Mul- flexible partitioning scheme in VTM. With this scheme,
tiple Transform (AMT). For big blocks (width or height is one tree is sufficient for the partitioning of CTUs which
equal to or larger than 64), the high-frequency coefficients can have a maximal size of up to 128 × 128. Then, the pre-
are automatically zeroed out as no meaningful information diction mode decision, the prediction, and the transform
is expected from them for signals which are encoded at this coding is applied to the same block. Namely, a nested struc-
block size. In addition to the higher variety of core trans- ture of quaternary, binary, and ternary splits is used for the
forms, JEM provides multiple other novel transform tech- partitioning in VTM. At first, the CTU is partitioned by
niques over HEVC: A Mode-Dependent Non-Separable a quaternary tree. Then, the leaf nodes of the quaternary
Secondary Transform (MDNSST) is applied between the tree are further split using a multi-type tree which allows
core transform and the quantization. Its purpose is to reduce binary and ternary splits. It is further noteworthy that for
remaining dependencies after the separable core transforms slices that are intra-only coded, the luma channel and the
which only address horizontal and vertical dependencies. chroma channels may have two independent partitioning
It is known that the Karhunen-Loève transform (KLT) is trees.
the only orthogonal transform which can achieve uncorre- Intra prediction: Compared to HEVC, the number of
lated transform coefficients with the extra benefit of efficient intra modes is increased from 33 to 67, including the pla-
energy compaction. At first glance, the drawback of the KLT nar mode, the DC mode, and 65 directional modes. Some
is that it is signal-dependent. It would be necessary to sig- adjustments were made to cope with non-square blocks
nal the transform matrix for a given block as part of the which can occur due to the new partitioning scheme.
bitstream. As this is unfeasible due to the considerable sig- Namely, some existing directional modes were replaced by
naling overhead, the KLT cannot be employed directly. To other wide-angle directional modes and for the DC mode
circumvent this drawback, the KLT is realized in JEM (here the mean value is calculated only for the reference samples
referred to as Signal-Dependent Transform or SDT) in such on the longer block side to avoid division operations. No sig-
a way that the transform matrix is calculated based on the naling changes were introduced by these two modifications.
most similar region within the already reconstructed signal. Cross-component Linear Models (CCLM) [56,57] were dis-
In-loop filtering: Adaptive Loop Filters (ALF) [53,54] cussed previously and are part of VTM. In HEVC, one
were studied intermediately during the standardization pro- row or column of references samples is available. In VTM,
cess of HEVC but were dismissed before the finalization of Multiple Reference Line (MRL) intra prediction allows the
the standard. With JEM, they return to the codec design. selection of one row or column of reference samples from
Wiener filters are derived to optimize the reconstructed sig- four candidate rows or columns. The selection is signaled
nal toward the original signal during the in-loop filtering as part of the bitstream. It is possible to further partition
stage. Another new in-loop filter in the JEM architecture intra-coded blocks into two or four parts via Intra Sub-
is a bilateral filter which smooths the reconstructed signal partitions (ISP). With ISP, the first sub-partition is predicted
with a weighted average calculation on neighboring sample using the available intra coding tools. The prediction error
values. ALF and the bilateral filter are applied in addition to is transform coded and the reconstructed signal for the sub-
Sample Adaptive Offset and the deblocking filter. The order partition is generated after the inverse transform. Then,
of filtering is: Bilateral – SAO – deblocking – ALF. the reconstructed signal is used as reference for the next
Entropy coding: The CABAC technique is enhanced by sub-partition. In contrast to deeper partitioning using the
a multiple-hypothesis probability estimation model and by normal partitioning algorithm, all sub-partition share the
an altered context modeling for the transform coefficients. same intra mode and thus no additional mode signaling

[Link] Published online by Cambridge University Press

 a comprehensive video codec comparison 5

is required. Further modifications compared to HEVC are Transform Coding: Similar to JEM, there is a Multiple
introduced by Mode Dependent Intra Smoothing (MDIS) Transform Selection (MTS) for the core transform. How-
which relies on simplified Gaussian interpolation filters for ever, the number of different transforms is reduced to three:
directional modes and by Position Dependent Intra Pre- DCT-II, DCT-VIII, and DST-VII. Also, the idea of zero-
diction Combination (PDPC) which combines unfiltered ing out the high-frequency coefficients for large blocks
reference samples and filtered reference samples. is adopted from JEM. With Dependent Quantization two
Inter prediction: For inter coding, the variety of merge quantizers with different representative values are intro-
candidates is extended. In addition to the previously duced. For each coefficient, one of the quantizers is selected
existing spatial and temporal candidates, history-based based on previously coded coefficients and a state-machine
and pairwise-averaged candidates are introduced. For the with four states.
history-based candidates, the motion information of pre- In-loop filtering: In addition to other minor changes, the
viously coded blocks is gathered using a first-in-first-out adaptive loop filters are adopted from JEM.
(FIFO) buffer. The pairwise-averaged candidates are cal- Entropy coding: Two states are used to model the prob-
culated by averaging a pair of other merge candidates. abilities for the update of the CABAC engine. In contrast
The Merge Mode with Motion Vector Difference (MMVD) to previous CABAC engines which relied on a look-up
enables the refinement of merge candidates by signaling an table for the update step, in VTM the update is calculated
offset. Affine Motion Compensated Prediction (with four based on said states following an equation. Other modifica-
or six parameters) including a merge mode and a predic- tions comprise the grouping of transform coefficients before
tion for the affine motion parameters improves the motion entropy coding and the related context modeling.
compensation for complex motion. The Subblock-based
Temporal Motion Vector Prediction (SbTMVP) is similar to
the Temporal Motion Vector Prediction (TMVP) of HEVC
C) AV1
but applied on the subblock level. Additionally, the refer- AV1 originates from the combination of multiple codecs
ence for the motion vector prediction is found by using an (VP9, Daala, and Thor) which were developed by members
offset based on the motion information of a spatially neigh- of the Alliance for Open Media. In this section, we review
boring block. With the Adaptive Motion Vector Resolution the distinguishing features of AV1. Additional information
(AMVR), the resolution can be adjusted on CU level based can be found in [58,59].
on the coded content. For translational motion vectors it Block partitioning: Similar to JEM, AV1 relies on an
can be set to quarter-pel, full-pel, or four-pel resolution. enhanced quarternary-tree partitioning structure. Pictures
For affine motion parameters, it can be set to quarter-pel, are partitioned into super-blocks (equivalent to CTUs) with
full-pel, or 1/16-pel resolution. To avoid increasing the com- a maximum size of 128 × 128. Super-blocks can be recur-
plexity of the rate-distortion check by a factor of three, sively partitioned into either square or rectangular shaped
the different resolutions are only tested if certain condi- blocks down to a minimum size of 4 × 4. The tree-based
tions are fulfilled. For the translational motion vector, the partitioning is extended by a wedge mode in which a rect-
four-pel resolution is only tested if the full-pel resolution angular block can be partitioned by a wedge into non-
is better than the quarter-pel resolution. For the affine rectangular parts for which different predictors are used.
motion parameters, the full-pel resolution and the 1/16-pel Thereby, the partitioning can be better adapted to object
resolution are only tested if the affine motion compensa- boundaries. The wedges can be selected from a wedge code-
tion with the quarter-pel resolution is the best mode. The book.
motion information for bi-prediction can be refined by Intra prediction: For intra prediction, AV1 provides the
using Bi-directional Optical Flow (BDOF, formerly BIO) following modes: a generic directional predictor, a Paeth
and Decoder-side Motion Vector Refinement (DMVR). In predictor, and a smooth predictor. The generic directional
both methods, the goal is the minimization of the difference predictor resembles the angular intra prediction as it is real-
between the two predictions from the two references. For ized in JEM and HEVC. It consists of an angular prediction
BDOF, this goal is achieved by using the optical flow, and in one of 56 different directions using a 2-tap linear inter-
for DMVR with a local search around the signaled motion polation with a spatial resolution of 1/256 pel. The Paeth
parameters. For CUs which are coded in merge mode or predictor and the smooth predictor of AV1 are conceptu-
skip mode, the CU can be split into two triangles along one ally similar to the planar mode in JEM and HEVC. The
of the two block diagonals. Each block can have a differ- Paeth predictor performs a prediction based on three pix-
ent merge candidate originating from a modified derivation els in neighboring blocks to the left, top, and top-left side.
process and blending is applied for the sample values on the The smooth predictor is based on the weighted averaging of
diagonal boundary. neighboring pixels from the left and top neighboring blocks
For a mode called Combined Inter and Intra Prediction and of interpolated pixels at the bottom and right of the
(CIIP), two predictions are generated: one with the regular current pixel. A chroma-only mode prediction consists of
inter prediction and one with a restricted version of the reg- using an already predicted, i.e. by other modes, luma sig-
ular intra prediction (only the DC, planar, horizontal, and nal to predict the chroma signal by a linear model with two
vertical modes). Then, the two predictions are combined parameters. The parameters are derived at the encoder and
using weighted averaging to form the final prediction. signaled as part of the bitstream. This mode is similar to the

[Link] Published online by Cambridge University Press

 6 thorsten laude et al.

cross-component linear model known from JEM. It is espe- lower res, but upscaled and restored using CDEF and guided
cially beneficial for screen content signals. A mode called restoration). Guided restoration supports Wiener filter and
Intra Block Copy [60], which is very similar to the Intra dual self-guided filter.
Block Copy mode known from the HEVC screen content Entropy coding: The entropy coding in AV1 is based
extension [61], is used to predict the currently coded block on the combination of a Multi-symbol Arithmetic Range
by copying a region of the same size from the already recon- Coder with Symbol Adaptive Coding. Thereby, a multi-
structed part of the current picture. This method is mainly symbol alphabet is encoded with up to 15-bit probabilities
beneficial for screen content signals. The block search adds a and an alphabet size of up to 16 symbols. With this entropy
considerable amount of complexity for intra coding. During coder, multiple binary symbols are combined into non-
the study of Intra Block Copy for the HEVC screen content binary symbols. This reduces the number of symbols which
extension, it was revealed and implemented in the reference need to be parsed by the entropy decoder. It is stated that
encoder HM-SCM that a hash-based search can be used to the efficiency is increased compared to a binary entropy
greatly increase the encoder speed with only a small loss encoder especially for lower bit rates due to reduced signal-
in coding efficiency. This approach was also adopted for ing overhead.
AV1 [60]. The hash-based search works well because screen
content signals tend to be noise-free. For high spatial res-
olutions, a super-resolution technique is applied. With this IV. ENCODER CONFIGURATIONS
technique, the video signal is downscaled and encoded at a
lower resolution. At the decoder, the signal is upscaled to its In this section, we elaborate on our experimental setup.
original spatial resolution. The exact versions of the different codecs are listed for
Inter prediction: The inter prediction in AV1 has access easy reproducibility of our experiments. Furthermore, all
to up to seven reference pictures of which one or two can parameters for the encoders are listed in Table 1 to enable
be chosen per block. For the compound mode, a weighted the configuration of the codecs in the same way. Some
combination of two references is performed. The weights parameters are redundant because they are implicitly set
can be varied smoothly or sharply within the block through when other parameters are set to certain values. For easier
the wedge-mode partitioning. Motion vectors can be pre- readability without going into details of encoder parameter
dicted at 8 × 8 block level by Dynamic Reference Motion selections, they are nevertheless noted to enable an under-
Vector Prediction. Similar to JEM, AV1 specifies an OBMC standing of the complete encoder configurations. For AV1
mode to refine the prediction at block boundaries by uti- we allowed 2-pass encoding as this results in an adaptive bit
lizing neighboring predictors. AV1 supports multiple global rate allocation comparable to the hierarchical GOP struc-
motion compensation models [62]: a rotation-zoom model tures used for HM, JEM, and VTM [25]. AV1 2-pass mode
with four parameters, an affine model with six parame- is not two passes of real full encoding, the first pass only per-
ters, and a perspective model with eight parameters. It is forms very fast statistics collection, hence not real coding or
asserted that these models are especially beneficial for the rate-distortion optimization. AV1 pure 1-pass mode is cur-
encoding of videos with video gaming content. Warping can rently under construction and is announced for the second
be applied by horizontal and vertical shearing using 8-tap half of 2019.
filters. The following versions of the codecs were used for this
Transform coding: AV1 supports multiple transforms: evaluation: version 1.0.0-2242-g52af439c8 for AV1, version
DCT, Asymmetric DST (ADST), flipped ADST, and Iden- 16.19 for HM, version 7.2 for JEM, version 4.0 for VTM,
tity. The identity transform is similar in spirit to the trans- version 155 for x264, version 2.8 for x265.
form skip mode of VTM, JEM and HM and beneficial, for For HM, JEM, and VTM, the configuration files from the
example, for screen content coding. The vertical and the common test conditions (CTC) with changes as required
horizontal transform can be selected independently from were used for the considered configurations.
the set of four available transforms. In total, 16 transform The following two configurations were used for our
combinations are possible this way. AV1 includes both, uni- experiments:
form and non-uniform quantization matrices for the quan- All Intra (AI): In the AI configuration, all pictures are
tization. Delta QP values can be signaled at superblock encoded self-contained, i.e. without any reference to previ-
level. ously coded pictures via motion compensated prediction.
In-loop filtering: For the in-loop filtering, AV1 combines The purpose of this configuration is to test the intra pre-
the constrained low-pass filter from the Thor codec with the diction tools and the transform coding for the prediction
directional deringing filter from the Daala codec into the errors produced by intra prediction. With this configuration
Combined Constrained Directional Enhancement (CDEF). it is ensured that all codecs operate based on the same con-
It is stated that this filter merging increases the quality of figuration as no encoder-specific optimizations like sophis-
the filtered picture while at the same time reducing the com- ticated hierarchical GOP structures can be used for intra
plexity compared to two separate filtering processes. Guided coding. For HM, JEM, and VTM, the all-intra configura-
restoration is a tool used after in-loop filtering and CDEF. It tion files from the CTC were used unaltered. The other
is both available for common single-resolution coding and encoders were configured by the parameters listed in Table 1
the super-resolution case (some frames initially coded at to encode as desired.

[Link] Published online by Cambridge University Press

 a comprehensive video codec comparison 7

Table 1. Parameters for the configuration of the codecs. Configuration for All Intra (AI): Disabling all inter prediction features. Configurations for
Maximum Coding Efficiency (MAX): Only one intra frame was coded. Unlike for AI, all tools were used unrestrictedly.

General HM, JEM, VTM -c <exp_config> Load parameters from experiment configuration file
-wdt <seq_width>, -hgt <seq_height> Width and height of sequence in pixel
-q <qp_val> QP value
-fr <seq_framerate> Frame per second of sequence
-fs <seq_frameskip> Frame skip of sequence
--InputChromaFormat<seq_chroma> Chroma format of sequence
--InputBitDepth<seq_bitdepth> Bit depth of sequence
--OutputBitDepth<seq_bitdepth> Bit depth of encoded sequence
--Level <seq_l evel > Level for sequence (decoder requirements)
AV1 --psnr, --tune=psnr Optimize encoding for PSNR metric, show PSNR value
--cpu-used=0 Slowest encoding for highest coding efficiency
-t 1 Single-threaded encoding
-passes 2 2-pass encoding: 1. pass for statistics, 2. pass for encoding
--c q -l evel <q p_va l > Constrained quality level, set to QP value
--end-usage=q Set to fixed QP mode
--i <seq_chroma> Chroma format of sequence
--input-bit-depth<seq_bitdepth> Bit depth of sequence
--bit-depth<seq_bitdepth> Bit depth of encoded sequence
--fps<seq_fps> Frame per second of sequence
-w <seq_width>, -h <seq_height> Width and height of sequence in pixel
x264, x265 --preset <profile> Set to either “medium” or “placebo”
--psnr, --tune=psnr Optimize encoding for PSNR metric, show PSNR value
--csv-log-level 1 (x265) Store information in CSV file
--csv <csv_filename> (x265)
--threads 1, --lookahead-threads 1 (x264) Set to single-threaded performance
--no-wpp, --frame-threads 1 (x265)
--input-depth <seq_bitdepth> Bit depth of sequence
--output-depth <seq_bitdepth> Bit depth of encoded sequence
--fps <seq_framerate> Frame per second of sequence
--input-res <seq_width> x <seq_height> Width and height of sequence in pixel
--input-csp <seq_chroma> Chroma format of sequence
--frames <seq_numframes> Number of frames in sequence
AI HM, JEM, VTM encoder_intra_<>.cfg Default configuration for All Intra
TemporalSubsampleRatio 1 Encode all pictures of the sequence
AV1 --lag-in-frames=0 Allow no future frames as reference
--kf-min-dist=1, --kf-max-dist=1 Set distance between keyframes to 1
x264, x265 --min-keyint 1, --keyint 1 Set distance between I-frames to 1
--no-scenecut Disable adaptive I-frame placement
--no-open-gop (x265) Set to closed gop
--no-cutree (x265) Disable lowresolution motion vector lookahead
--b-adapt 0
--bframes 0
--b-pyramid none (x264) Disable B-frame related operations
--no-b-pyramid (x265)
--no-weightb
--weightp 0 (x264) Disable weighted prediction for P-frames
--no-weightp (x265)
--qp <qp_val> QP value
--rc-lookahead 0 Disable slice-type decision lookahead
MAX HM, JEM, VTM encoder_randomaccess_<>.cfg Use default random access configuration as reference
IntraPeriod -1 Allow only 1 I-frame
AV1 --kf-m in-dist<seq_n umfram es> Allow only 1 I-frame
--kf-max-dist<seq_n umfram es>
--lag-in-frames=25 Enable forward referencing similar to RA configuration
--auto-alt-ref=1 Enable hierarchical GOP structure
x264, x265 --min-keyint -1, --keyint -1 (x265) Allow only 1 I-frame
--keyint infinite (x264)
--no-open-gop (x265) Set to closed gop
--crf <crf val> QP-comparable CRF value
--no-scenecut Disable adaptive I-frame placement

[Link] Published online by Cambridge University Press

 8 thorsten laude et al.

Maximum Coding Efficiency (MAX): The purpose of the calculation of meaningful BD-Rates it is considered as best
MAX configuration is to test all codecs at their respective practice to encode at the same PSNR value for all codecs
configurations for the highest coding efficiency. Naturally, to maximize the overlap of rate-distortion curves. Our pro-
considering that the codecs differ considerably in terms cedure is based on encoding the sequences with HM at the
of coding tools and encoder optimizations, the codecs are four QPs defined in the CTC (22, 27, 32, 37) at first. Then, the
not configured the same way for this configuration. Only QPs (or CRFs) of the other codecs were tuned to match the
one intra-only picture is encoded at the beginning of the PSNR of the HM-encoded representations of the sequences.
sequence. For HM, JEM, and VTM, the MAX configura- The test sequences were not chosen by ourselves but
tion is based on the Random Access configuration file with agreed upon by experts from the standardization bodies
minor changes such as disabling random access I pictures. MPEG and VCEG. It is generally believed that they are rep-
Almost no tools are disabled for the codecs in this configu- resentative enough for a comparison as they cover a wide
ration. The only exception is that the detection of scene cuts range of contents and spatial resolutions. In total, all 28 test
is disabled for x264 and x265 to avoid the dynamic place- sequences defined in [22] were coded. They are referred to
ment of I pictures. Furthermore, no tools for improving the as JVET test sequences. Based on their resolution and char-
subjective quality at the cost of PSNR quality are used as this acteristics, they are categorized into seven classes: Class A1
would imply a disadvantage for the corresponding codecs (4K), Class A2 (4K), Class B (1080p), Class C (WVGA),
in the PSNR-based evaluation. We used CRF for x264 and Class D (WQVGA), Class E (720p), and Class F (screen
x265 as it maximizes the coding efficiency. This allows the content with different resolutions). The characteristics of
encoder to adopt the QP on the local properties of the video some sequences in class F vary considerably from other
signal. The benefit is similar to the adaptive bit rate allo- sequences: In parts, they do not contain any motion, in
cation strategies of the other codecs. For our experiments, other parts all moving objects have the same motion direc-
2-pass encoding for x264 and x265 is not suitable for these tion and in other cases only very few different colors are
two codecs because it aims at rate-control encoding. Details present. These characteristics influence the efficiency of
on the parameters for the MAX configuration can be found video codecs, especially if the codecs incorporate distin-
in Table 1. guished coding tools for these characteristics [63,64]. The
x264 and x265 can be configured to either use closed or first picture of each sequence is visualized in Fig. 1 to give
GOP structures. For x264, the default is a closed GOP struc- an impression of the sequence characteristics.
ture, while it is the other way around for x265. Open GOPs The JVET sequences were also (completely/partly) used
are, for example, used in typical encoder configurations for in the development of VVC and HEVC. Theoretically, the
Blu-rays. The reasons why open GOPs are used for Blu- respective reference software should not be optimized for
rays are: (1) They are necessary to facilitate the small GOP the test set but work equally good for all sequences. How-
sizes used for Blu-rays. Otherwise, with closed GOPs, the ever, we believe that a potential bias toward HEVC and
coding would be very inefficient. (2) On Blu-rays, the qual- VVC due to the sequences should not be ruled out too eas-
ity of the encoded video does not change much compared ily. Therefore, we also encoded some sequences which are
to video streaming where quality and resolution can vary used by the AOM community and report separate results
considerably between different chunks, e.g. if the available for both test sets. We refer to the second test set as AOM test
bandwidth changes and different representations are deliv- sequences. Namely, we chose the first four 1080p sequences
ered. Hence, for Blu-rays, it is no problem to use references in alphabetical order since we believe that the other lower
outside of a GOP. However, today streaming is more impor- resolutions are today not that important anymore.
tant than Blu-rays. Therefore, we used closed GOPs for our
experiments.
All encoders support PSNR tuning while AV1, x264, and V. CODING EFFICIENCY
x265 also support the tuning for subjective quality. As the
latter is not supported by all encoders, a comparison with In this section, we discuss the coding efficiency results for
that tuning would be unfair. And even if one would only the JVET test sequences of our comparison with reference
consider the three encoders with subjective quality tuning, to Table 2 and Fig. 2 at first. To asses the coding efficiency
the results would be hard to compare. There are plenty of we measured BD-Rates. BR-Rates reveal the average bit rate
metrics which all allegedly approximate the subjective qual- savings at the same objective quality for multiple operat-
ity very well but yet come to different assessments of codecs. ing points which differ in bit rate and quality. Typically,
Therefore, we tuned all encoders for PSNR. Only because e.g. for standardization activities and for this manuscript,
the encoders were tuned for PSNR, the BD-Rates calcu- four operating points are used per BD-Rate value. Other
lated with PSNR as a quality metric for the experiments are implementations of the BD-Rate which allow the usage of
meaningful. an arbitrary number of operating points exist [65]. One
For each combination of codec, configuration, and test BD-Rate is calculated per codec pair and configuration and
sequence, four data points were encoded covering a wide sequence. For the data points in the table and the figure,
range of bit rates and qualities. For each data point, the the BD-Rates of all 28 sequences were averaged per codec
quantization parameters (QP) of the codecs need to be set. pair and configuration. So, each data point represents 224
The resulting PSNR and bit rate depend on the QP. For the simulations.

[Link] Published online by Cambridge University Press

 a comprehensive video codec comparison 9

Fig. 1. Overview of the JVET test sequences used for the comparison. The sequences are defined by the common test conditions [22].

In the table and the figure, each codec is compared to all Additionally, our main insights from the data are elabo-
other codecs. One example of how to read Table 2 is as fol- rated in the following.
lows: For the configuration MAX and the anchor codec HM, Compared to HM, the coding efficiency of x265 is unex-
x264 medium achieves a BD-Rate loss of 98, x264 placebo pectedly (given that both implement encoders for the same
a loss of 76, x265 medium a loss of 53, x265 placebo a loss standard) bad. This states true especially in the case of the
of 19, JEM a gain of 29, VTM a gain of 30, and AV1 a MAX configuration when all codecs are “let off the leash”.
gain of 24. Even for the placebo preset which maximizes the coding
For both configurations, the codecs rank as follows (from efficiency of x265, the BD-Rate loss of x265 is 19. It is worth
most efficient to less efficient): VTM – JEM – AV1 – HM – keeping this insight in mind when interpreting codec com-
x265 (placebo) – x265 (medium) – x264 (placebo) – x264 parisons for which x265 is used as HEVC implementation,
(medium). especially if a less efficient preset than placebo is configured.

[Link] Published online by Cambridge University Press

 10 thorsten laude et al.

Table 2. BD-Rates for the two configurations AI (all-intra prediction) and MAX (most efficient motion compensation configuration for each codec)
for the JVET test sequences. Negative numbers mean increased coding efficiency.

Test
AI x264 (Medium) x264 (Placebo) x265 (Medium) x265 (Placebo) HM JEM VTM AV1
Anchor
x264 (Medium) −3 −16 −22 −23 −39 −39 −32
x264 (Placebo) 3 −13 −19 −21 −37 −37 −30
x265 (Medium) 20 16 −7 −9 −26 −27 −17
x265 (Placebo) 30 25 7 −2 −21 −22 −11
HM 33 29 10 2 −20 −20 −10
JEM 67 61 37 27 25 −1 13
VTM 68 63 37 28 26 1 14
AV1 49 44 21 13 11 −11 −12
Test
MAX x264 (Medium) x264 (Placebo) x265 (Medium) x265 (Placebo) HM JEM VTM AV1
Anchor
x264 (Medium) −14 −19 −37 −47 −61 −62 −56
x264 (Placebo) 13 −8 −28 −40 −56 −57 −50
x265 (Medium) 29 15 −22 −34 −53 −53 −50
x265 (Placebo) 62 43 29 −15 −39 −40 −36
HM 98 76 53 19 −29 −30 −24
JEM 181 149 117 68 42 −1 5
VTM 183 150 120 70 44 1 7
AV1 147 114 108 62 33 −3 −5

Fig. 2. BD-Rates for the two configurations AI (all-intra prediction) and MAX (most efficient motion compensation configuration for each codec) for the JVET
test sequences. Each point represents the comparison of one codec against another codec. The “anchor” codec is indicated on the horizontal axis. The “test” codec
is indicated by the color of the point. Each point corresponds to one number in Table 2. Negative numbers mean increased coding efficiency.

AV1 gained a lot in terms of coding efficiency compared coding standards, which cannot be considered in this
to previous versions like in [24] and is now superior to manuscript.
the finalized codecs of this comparison (HM, x264, x265) Interestingly, the BD-Rates of AV1 and VVC – which
for all configurations. Furthermore, AV1 only falls shortly average in a 7 loss of AV1 – are not consistent over the
behind the upcoming VVC standard. Still, we point the different classes. For 4K Sequences, AV1 is farther behind
reader to the fact that there are commercial encoders avail- VVC with 20 loss, while for screen content and some low
able on the market, especially for the established video resolutions AV1 can outperform VVC.

[Link] Published online by Cambridge University Press

 a comprehensive video codec comparison 11

Table 3. BD-Rates for the two configurations AI (all-intra prediction) and MAX (most efficient motion compensation configuration for each codec) for
the AOM test sequences. Negative numbers mean increased coding efficiency.

Test
AI x264 (Medium) x264 (Placebo) x265 (Medium) x265 (Placebo) HM VTM AV1
Anchor
x264 (Medium) −4 −19 −24 −26 −40 −33
x264 (Placebo) 4 −16 −21 −23 −38 −30
x265 (Medium) 24 19 −5 −7 −24 −15
x265 (Placebo) 31 26 6 −1 −20 −10
HM 36 31 7 1 −20 −9
VTM 68 62 32 25 25 13
AV1 49 43 17 11 10 −12
Test
MAX x264 (Medium) x264 (Placebo) x265 (Medium) x265 (Placebo) HM VTM AV1
Anchor
x264 (Medium) −8 −19 −34 −40 −56 −48
x264 (Placebo) 9 −12 −28 −35 −52 −44
x265 (Medium) 26 16 −20 −25 −46 −41
x265 (Placebo) 53 41 25 −6 −32 −26
HM 73 59 34 7 −29 −23
VTM 142 122 88 49 41 6
AV1 105 89 75 38 32 −6

Table 4. Encoding time ratios for the two configurations AI (all-intra prediction) and MAX (most efficient motion compensation configuration for
each codec) relative to the encoding time of HM. Values over 1 indicate slower encoders compared to HM, ratios below 1 faster encoders.

Test Sequence Class

AI A1 A2 B C D E F
x264 (Medium) 0,020 0,025 0,008 0,010 0,011 0,007 0,008
x264 (Placebo) 0,09 0,11 0,04 0,05 0,06 0,03 0,04
x265 (Medium) 0,30 0,30 0,30 0,29 0,29 0,28 0,26
x265 (Placebo) 0,54 0,52 0,57 0,60 0,64 0,49 0,49
HM 1 1 1 1 1 1 1
JEM 32,80 28,88 41,07 51,24 57,82 29,67 42,55
VTM 22,11 18,60 28,43 36,27 39,39 21,18 21,49
AV1 9,17 9,60 9,66 10,57 11,83 6,25 6,82

Test Sequence Class

MAX A1 A2 B C D E F
x264 (Medium) 0,008 0,008 0,006 0,006 0,007 0,005 0,004
x264 (Placebo) 0,78 0,61 1,02 0,89 1,09 0,69 0,53
x265 (Medium) 0,02 0,02 0,02 0,01 0,02 0,01 0,01
x265 (Placebo) 0,51 0,52 0,65 0,41 0,32 0,40 0,34
HM 1 1 1 1 1 1 1
JEM 10,92 9,50 9,44 12,17 13,02 4,79 8,41
VTM 10,80 7,11 9,93 12,11 12,69 5,68 6,34
AV1 9,64 8,50 11,04 7,89 10,16 6,39 3,18

Considering that HM gains 47 and 40 over the decade- Secondly, we observe that the x264 and x265 encoders partly
long optimized AVC encoder x264 confirms the statements catch up on the reference implementations. Their leeway is
of [8,9] that HEVC outperforms AVC by 40–60 based on considerably reduced.
the configuration and application. As an additional experiment, the VMAF metric was cal-
The coding efficiency results for the AOM test sequences culated for the two contenders with the highest coding
are summarized in Table 3. We make two main observa- efficiency, namely VTM and AV1. For this experiment, the
tions for the data: Firstly, the numbers for the comparisons bitstreams of the MAX configuration were chosen. BD-
of HM, VTM, and AV1 relative to each other are within Rates were calculated based on the bit rate and the VMAF
a range of ±2 compared to the numbers for the JVET score as the quality metric. The content-dependency of the
sequences. From this observation, we conclude that there coding efficiency results manifests stronger than for the
is no noticeable bias in either of the two test sequence sets. conventional BD-Rate calculations based on bit rate and

[Link] Published online by Cambridge University Press

 12 thorsten laude et al.

Fig. 3. Encoding time ratios for the two configurations AI (all-intra prediction) and MAX (most efficient motion compensation configuration for each codec)
relative to the encoding time of HM. Values over 1 indicate slower encoders compared to HM, ratios below 1 faster encoders.

PSNR. While VTM gains up to 39 over AV1 for individual Table 5. Absolute per picture encoding times for the sequence 4k
Toddler Fountain. Times are given in the format hh:mm:ss. It is observed
4K sequences, AV1 expands the lead for the lower resolu-
that the encoding times vary between few seconds per picture and more
tions and screen content and mixed content sequences. On than one hour per picture.
average, VTM falls behind by 9.8 for the JVET sequences.
For the AOM sequences, VTM and AV1 perform equally Toddler Fountain (4K) Configuration
good in terms of VMAF-based BD-Rates with an average (hh:mm:ss) AI MAX
value smaller than 1. x264 (Medium) 00:00:01 00:00:03
x264 (Placebo) 00:00:03 00:03:39
x265 (Medium) 00:00:20 00:00:06
x265 (Placebo) 00:00:46 00:05:45
HM 00:01:09 00:05:34
VI. COMPLEXITY JEM 01:04:32 01:16:36
VTM 00:51:00 01:36:30
In this section, we discuss our findings for the complex- AV1 00:16:21 00:38:21
ity of the used codecs. For this purpose, we measured the
run times of the encoders and decoders on a homogeneous
cluster composed of Intel Xeon Gold 5120 CPUs. For easier
interpretability, all run times were normalized to the run
times of HM. Therefore, we refer to the numbers as time Fountain in Table 5. It is observed that encoding one pic-
ratios. Numbers greater than 1 indicate higher run times ture with x264 in the medium preset just takes a few sec-
compared to HM, values lower than 1 faster run times. onds. At the other end of the scale, modern codecs such
The results for the encoders are listed in Table 4 and visu- as JEM, VTM, or AV1 require more than half an hour or
alized in Fig. 3. Due to the large spread of encoding time even more of computation per picture. Hence, it can be
ratios (four orders of magnitude), the vertical axis has a concluded that even in highly multi-threaded set-ups real-
logarithmic scale. Depending on the configuration and test time encoding with these codecs configured for maximum
sequence, either the JEM or the AV1 encoders are the slow- coding efficiency is unfeasible.
est. It is without surprise that the x264 (medium) encoder is For AV1, the trade-off between coding efficiency and
the fastest. encoding complexity can be tuned using the cpu-used
Although it is common practice in academic and stan- parameter. This parameter was set to 0 for all of the pre-
dardization contributions to compare the complexity with sented experiments. With this value, the encoder is tuned
relative numbers, we believe that this complicates the per- for the highest coding efficiency but also for the highest
ception of how complex modern video codecs are. To facil- encoding complexity. To further study the impact of the
itate the assessment of the encoding times, we exemplarily cpu-used parameter, we conducted a comparison of AV1
list the absolute encoding times for the 4k sequence Toddler with cpu-used=0 versus AV1 withcpu-used=1. We

[Link] Published online by Cambridge University Press

 a comprehensive video codec comparison 13

Table 6. Decoding time ratios for the two configurations AI (all-intra which was not performed by the HM developers. x264 and
prediction) and MAX (most efficient motion compensation configuration
x265 do not include decoder implementations. Hence, they
for each codec) relative to the decoding time of HM. Values over 1
indicate slower decoders compared to HM, ratios below 1 faster decoders. are omitted in the table.
In the end, video coding is a trade-off between coding
Test Sequence Class efficiency and complexity. To assess how the codecs under
AI A1 A2 B C D E F review perform for this trade-off, we plot the BD-Rates of
HM 1 1 1 1 1 1 1 the codecs (relative to HM) over the encoding time ratio
JEM 2,58 2,46 2,74 2,89 3,11 2,55 3,07 (relative to HM as well) in Fig. 4. A least-squares regression
VTM 2,22 2,15 2,05 2,26 2,08 1,88 2,07 for a linear function was performed on the data. The result-
AV1 0,82 0,99 0,73 0,78 0,73 0,65 0,82
ing function is plotted along with 95 confidence intervals.
Test Sequence Class For the all-intra configuration, a linear trend is observed.
MAX A1 A2 B C D E F Considering the logarithmic horizontal axis it can be con-
cluded that increasing the coding efficiency linearly results
HM 1 1 1 1 1 1 1
JEM 6,74 7,86 7,67 8,62 11,51 7,42 6,01 in exponentially increasing complexity of the coding tools.
VTM 2,24 2,09 2,26 2,34 2,54 1,95 1,95 Although a similar trend is visible in the MAX data as well,
AV1 1,04 1,10 0,97 1,19 1,47 0,63 0,56 the confidence intervals are too large to draw solid conclu-
sions. The model fit by the regression is typically judged by
the coefficient of determination (R2 ). The range for R2 is
between 0 and 1, where 1 indicates that the model fits the
observed that by using cpu-used=1, the coding effi- data perfectly and 0 that the model does not fit the data at all.
ciency drops by 2.4 (BD-Rate) averaged over our test set The values for the two configurations are: R2AI = 0.97 and
while the encoding speed is roughly 2.5 times faster. R2MAX = 0.75.
The results for the decoders are listed in Table 6. Some In real-world applications, often commercial encoders
interesting observations can be made for the decoder side: are used. The reason is that the complexity of reference
JEM shifts a certain amount of complexity to the decoder, implementations is too high to allow a deployment in prod-
e.g. with the decoder-side motion refinement. This is the ucts. For these encoders, the trade-off between coding effi-
reason why the decoder run time ratio of JEM is very high, ciency and complexity can be configured depending on the
8× for MAX compared to HM. The decoding complexity requirements of the particular applications and systems.
of AV1 is similar to the HM decoding complexity for high- To perform such trade-offs with the reference implemen-
resolution sequences and slightly lower for low-resolution tations which we use for our comparison is not possible.
sequences. It should be considered that some extend of However, it is known from the literature that by using
software optimization was performed by the AV1 developers commercial encoder products the HEVC encoding process

Fig. 4. Trade-off of coding efficiency and encoder complexity (both relative to HM). A linear regression function is plotted with 95 confidence intervals. The
coefficients of determination for the regression are R2AI = 0.97 and R2MAX = 0.75.

[Link] Published online by Cambridge University Press

 14 thorsten laude et al.

can be sped-up by a factor of 30 with around 1 BD- [11] Laude T.; Ostermann J.: Deep learning-based intra prediction mode
Rate loss and by a factor of 300 for a BD-Rate loss of 12 decision for HEVC, in Proc. of 32nd Picture Coding Symp. (PCS),
Nuremberg, Germany, 2016, IEEE.
compared to HM [66].
[12] Bankoski J.; Wilkins P.; Xu Y.: Technical overview of VP8, an open
source video codec for the web, in 2011 IEEE Int. Conf. on Multimedia
and Expo, Hangzhou, CN, July 2011, 1–6, IEEE.
VII. CONCLUSION [13] Mukherjee D.; Bankoski J.; Grange A.; Han J.; Koleszar J.; Wilkins
P.; Xu Y.; Bultje R.: The latest open-source video codec VP9 – An
In this paper, we compared the video codecs AV1 (version overview and preliminary results, in 2013 Picture Coding Symp. (PCS),
1.0.0-2242 from August 2019), HM, JEM, VTM (version San Jose, US, December 2013, 390–393, IEEE.
4.0 from February 2019), x264, and x265 under two dif- [14] Valin J.-M.; Terriberry T.B.; Egge N.E.; Deade T.; Cho Y.; Montgomery
ferent configurations: All Intra for the assessment of intra C.; Bebenita M.: Daala: Building A Next-Generation Video Codec
coding (which is also applicable to still image coding) and From Unconventional Technology, arXiv:1608.01947, August 2016.
Maximum Coding Efficiency with all codec being tuned [15] Bjontegaard G.; Davies T.; Fuldseth A.; Midtskogen S.: The thor video
for their best coding efficiency settings. VTM achieves the codec, in 2016 Data Compression Conf. (DCC), Snowbird, US, March
highest coding efficiency in both configurations, followed 2016, 476–485, IEEE.
by JEM and AV1. The worst coding efficiency is achieved [16] IETF 100: NetVC WG: Internet Video Codec. Available at https://
by x264 and x265, even in the placebo preset for highest [Link]/meeting/100/materials/slides-100-netvc-chair-
slides/, 2017.
coding efficiency. AV1 gained a lot in terms of coding effi-
ciency compared to previous versions and now outperforms [17] Miller M.; Zanaty M.: Internet Video Codec (NETVC) Standard-
HM by 24 BD-Rate gains. VTM gains 5 on average over ization Website. Available at [Link]
about/, accessed 2019-03-26, 2019.
AV1 in terms of BD-Rates. For 4K Sequences, AV1 is far-
ther behind VVC with 20 loss. For the screen content [18] Joint Video Exploration Team (JVET) of ITU-T SG16 WP3 and
ISO/IEC JTC1/SC29/WG11: JEM reference software. Available at
sequences in the test set and some low resolutions, AV1 is
[Link] 2018.
even able to outperform VVC.
[19] Chiariglione L.: Press Release: Versatile Video Coding (VVC) project
starts strongly in the Joint Video Experts Team. Available at https://
[Link]/meetings/122, 2018.
REFERENCES [20] Joint Video Exploration Team (JVET) of ITU-T SG16 WP3 and
ISO/IEC JTC1/SC29/WG11: VTM reference software. Available at
[1] ISO/IEC 11172: Information technology–coding of moving pictures [Link] 2019.
and associated audio for digital storage media at up to about 1.5
Mbit/s, 1993. [21] Motokado: Comparison of ffmpeg’s x264 presets. Available at http://
[Link]/yoshi/2011/06/25/comparison-of-x264-presets/,
[2] ISO/IEC 13818–2: Generic Coding of Moving Pictures and Associated accessed 2019-03-18, 2011.
Audio Information – Part 2: Video/ITU-T Recommendation H.262,
1994. [22] Joint Video Exploration Team (JVET) of ITU-T SG16 WP3 and
ISO/IEC JTC1/SC29/WG11: JVET-G1010: JVET common test condi-
[3] ISO/IEC 14496–2: Information technology–coding of audiovisual tions and software reference configurations, in 7th Meeting, Torino,
objects–part 2: visual, 2000. IT, 13–21 July 2017.
[4] ISO/IEC 14496–10: Coding of Audiovisual Objects-Part 10: Advanced [23] Grois D.; Nguyen T.; Marpe D.: Coding efficiency comparison of
Video Coding/ITU-T Recommendation H.264 Advanced video cod- AV1/VP9, H.265/MPEG-HEVC, and H.264/MPEG-AVC encoders,
ing for generic audiovisual services, 2003. in 2016 Picture Coding Symp. (PCS), Nuremberg, DE, 2016, 1–5, IEEE.
[5] Ostermann J.; Bormans J.; List P.; Marpe D.; Narroschke M.; Pereira [24] Laude T.; Adhisantoso Y.G.; Voges J.; Munderloh M.; Ostermann J.:
F.; Stockhammer T.; Wedi T.: Video coding with H.264/AVC: tools, A comparison of JEM and AV1 with HEVC: coding tools, coding effi-
performance, and complexity. IEEE Circ. Syst. Mag., 4 (1) (2004), ciency and complexity, in Picture Coding Symp. (PCS), San Francisco,
7–28. CA, US, 2018, IEEE.
[6] ITU-T Recommendation H.265/ ISO/IEC 23008-2:2013 MPEG-H [25] Nguyen T.; Marpe D.: Future video coding technologies: a perfor-
Part 2: High Efficiency Video Coding (HEVC), 2013. mance evaluation of AV1, JEM, VP9, and HM, in 2018 Picture Coding
[7] JCT-VC: HEVC reference software HM. Available at [Link] Symp. (PCS), San Francisco, US, 2018, 31–35.
[Link]/. [26] Guo L.; De Cock J.; Aaron A.: Compression performance comparison
[8] De Cock J.; Mavlankar A.; Moorthy A.; Aaron A.: A large-scale of x264, x265, libvpx and aomenc for on-demand adaptive streaming
video codec comparison of x264, x265 and libvpx for practical VOD applications, in 2018 Picture Coding Symp. (PCS), San Francisco, US,
applications, in International Society for Optics and Photonics, San 2018, 26–30.
Francisco, CA, US, September 2016, 997116. [27] Chen Y. et al.: An overview of core coding tools in the AV1 video
[9] Hanhart P.; Rerabek M.; De Simone F.; Ebrahimi T.: Subjective qual- codec, in 2018 Picture Coding Symp. (PCS), San Francisco, US, June
ity evaluation of the upcoming HEVC video compression standard, 2018, 41–45, IEEE.
in SPIE Optical Engineering + Applications, San Diego, CA, US, [28] Feldmann C.: Multi-Codec DASH Dataset: An Evaluation of AV1,
October 2012, 84990V. AVC, HEVC and VP9 – Bitmovin. Available at [Link]
[10] Laude T.; Tumbrägel J.; Munderloh M.; Ostermann J.: Non-linear av1-multi-codec-dash-dataset/, 2018.
contour-based multidirectional intra coding. APSIPA Trans. Signal [29] Grois D.; Nguyen T.; Marpe D.: Performance comparison of AV1,
Inf. Process., 7 (11) (2018), 1–13. JEM, VP9, and HEVC encoders, in Applications of Digital Image

[Link] Published online by Cambridge University Press

 a comprehensive video codec comparison 15

Processing XL, Andrew G. Tescher, Ed., San Diego, US, February 2018, [48] Sullivan G.J.; Boyce J.M.; Chen Y.; Ohm J.-R.; Segall C.A.; Vetro A.:
vol. 10396, 120, SPIE. Standardized extensions of high efficiency video coding (HEVC).
IEEE J. Sel. Top. Signal Process., 7 (6) (2013), 1001–1016.
[30] Liu Y.: AV1 beats x264 and libvpx-vp9 in practical use case. Avail-
able at [Link] [49] Ohm J.-R.; Sullivan G.J.; Schwarz H.; Tan T.K.; Wiegand T.: Compar-
libvpx-vp9-in-practical-use-case/, 2018. ison of the Coding Efficiency of Video Coding Standards–Including
High Efficiency Video Coding (HEVC). IEEE Trans. Circ. Syst. Vid.
[31] Vatolin D.; Grishin S.; Kalinkina D.; Soldatov S.: MSU Video
Technol., 22 (12) (2012), 1669–1684.
Codecs Comparison. Available at [Link]
codec_comparison/codec_com_en.html [visited 2019-08-06], [50] Sze V.; Budagavi M.; Sullivan G.J.: High Efficiency Video Coding
2019. (HEVC) – Algorithms and Architectures, Integrated Circuits and
Systems, Springer International Publishing, Cham, 2014.
[32] Sullivan G.J.; Wiegand T.: Rate-distortion optimization for video
compression. IEEE Signal Process. Mag., 15 (6) (1998), 74–90. [51] Joint Video Exploration Team JVET of ITU-T SG16 WP3 and
ISO/IEC: JVET-G1001: Algorithm Description of Joint Exploration
[33] Haub F.; Laude T.; Ostermann J.: HEVC Inter Coding Using Deep
Test Model 7 (JEM 7), 2017.
Recurrent Neural Networks and Artificial Reference Pictures, arXiv
Preprint 1812.02137, December 2018. [52] Khairat A.; Nguyen T.; Siekmann M.; Marpe D.; Wiegand T.: Adaptive
cross-component prediction for 4 : 4 : 4 high efficiency video coding,
[34] Laude T.; Ostermann J.: Contour-based multidirectional intra coding
in ICIP, 2014, 3734–3738.
for HEVC, in Proc. of 32nd Picture Coding Symp. (PCS), Nuremberg,
Germany, 2016, IEEE. [53] Chen C.-Y.; Tsai C.-Y.; Huang Y.-W.; Yamakage T.; Chong I.S.; Fu C.;
Itoh T.; Watanabe T.; Chujoh T.; Karczewicz M; Lei S.: The adaptive
[35] Bjontegaard G.: VCEG-M33: calculation of average PSNR differences
loop filtering techniques in the HEVC standard, in International Soci-
between RD-curves, in ITU-T SG 16 Q 6. 13th Meeting, Austin, Texas,
ety for Optics and Photonics, vol. 8499, San Diego, US, October 2012,
USA, 2001.
849913.
[36] Bjøntegaard G.: VCEG-AI11: improvements of the BD-PSNR model,
[54] Tsai C.-Y.; Chen C.-Y.; Yamakage T.; Chong I.S.; Huang Y.; Fu C.; Itoh
in ITU-T SG 16 Q 6. 35th Meeting, Berlin, Germany, 2008.
T.; Watanabe T.; Chujoh T.; Karczewicz M.; Lei S.: Adaptive Loop Fil-
[37] Wang Z.; Bovik A.C.; Sheikh H.R.; Simoncelli E.P.: Image quality tering for Video Coding. IEEE J. Sel. Top. Signal. Process., 7 (6) (2013),
assessment: from error visibility to structural similarity. IEEE Trans 934–945.
Image Process., 13 (4) (2004), 600–612.
[55] Chen J.; Alshina E.: JVET-J1002: algorithm description for versatile
[38] Aaron A.; Li Z.; Manohara M.; Lin J.Y.; Wu E.C.-H.; Kuo C.-C.J.: Chal- video coding and test model, in 10th Meeting of the Joint Video Experts
lenges in cloud based ingest and encoding for high quality streaming Team (JVET), San Diego, US, 2018.
media, in 2015 IEEE Int. Conf. on Image Processing (ICIP), Quebec
[56] Boyce J.: JVET-J0096: BoG report on benchmark set tool selection, in
City, QC, Canada, September 2015, 1732–1736, IEEE.
10th Meeting of the Joint Video Experts Team (JVET), San Diego, US,
[39] Lin J.Y.; Liu T.-J.; Wu E.C.-H.; Kuo C.-C.J.: A fusion-based video 2018.
quality assessment (fvqa) index, in Signal and Information Processing
[57] Sullivan G., J.-R. Ohm. JVET-J1000: Meeting Report of the 10th
Association Annual Summit and Conf. (APSIPA), 2014 Asia-Pacific,
Meeting of the Joint Video Experts Team (JVET), San Diego, US,
Siem Reap, Cambodia, December 2014, 1–5, IEEE.
2018.
[40] Rassool R.: VMAF reproducibility: validating a perceptual practical
[58] Massimino P.: Paris Video Tech #4: AOM – AV1. How does it
video quality metric, in 2017 IEEE Int. Symp. on Broadband Multi-
work? Available at [Link]
media Systems and Broadcasting (BMSB), Cagliari, IT, June 2017, 1–2,
2017/07/[Link], 2017.
IEEE.
[59] Mukherjee D.; Su H.; Bankoski J.; Converse A.; Han J.; Liu Z.; Xu
[41] Norkin A.; Birkbeck N.: Film grain synthesis for AV1 video codec, in
Y.: “An overview of new video coding tools under consideration for
2018 Data Compression Conf., Snowbird, UT, US, March 2018, 3–12,
VP10: the successor to VP9,” SPIE Optical Engineering + Applica-
IEEE.
tions, vol. 9599, no. September 2015, pp. 95991E, 2015.
[42] Wandt B.; Laude T.; Rosenhahn B.; Ostermann J.: Extending hevc
[60] Li J. et al.: Intra block copy for screen content in the emerging AV1
with a texture synthesis framework using detail-aware image decom-
video codec, in 2018 Data Compression Conf., Snowbird, US, March
position, in Proc. of the Picture Coding Symp. (PCS), San Francisco,
2018, 355–364, IEEE.
US, June 2018.
[61] Xu X. et al.: Intra block copy in HEVC screen content coding exten-
[43] Wandt B.; Laude T.; Liu Y.; Rosenhahn B.; Ostermann J.: “Extending
sions. IEEE J. Emerg. Sel. Top. Circ. Syst., 6 (4) (2016), 409–419.
HEVC Using Texture Synthesis,” in IEEE Visual Communications and
Image Processing (VCIP), St Petersburg, US, 2017. [62] Parker S.; Chen Y.; Barker D.; de Rivaz P.; Mukherjee D.: “Global
and locally adaptive warped motion compensation in video compres-
[44] Akyazi P.; Ebrahimi T.: Comparison of compression efficiency
sion,” in 2017 IEEE Int. Conf. on Image Processing (ICIP), Beijing, CN,
between HEVC/H.265 and VP9 based on subjective assessments, in
September 2017, 275–279, IEEE.
International Conf. on Quality of Multimedia Experience (QoMEX),
Sardinia, Italy, 2018. [63] Laude T.; Ostermann J.: “Copy mode for static screen content coding
with HEVC,” in IEEE Int. Conf. on Image Processing (ICIP), Québec
[45] Wien M.: High Efficiency Video Coding – Coding Tools and Specifi-
City, Canada, 2015, IEEE.
cation, 1st ed., Springer, Berlin Heidelberg, 2015.
[64] Xu J.; Joshi R.; Cohen R.A.: Overview of the Emerging HEVC screen
[46] Bossen F.; Bross B.; Flynn D.: HEVC Complexity and Implementa-
content coding extension. IEEE Trans. Circ. Syst. Vid. Technol., 26 (1)
tion Analysis. IEEE Trans. Circ. Syst. Vid. Technol., 22 (12) (2013),
(2016), 50–62.
1685–1696.
[65] Valenzise G.: Bjontegaard metric – file exchange – MATLAB
[47] Sullivan G.J.; Ohm J.-R.; Han W.-j.; Wiegand T.: Overview of the high
central. Available at [Link]
efficiency video coding (HEVC) standard. IEEE Trans. Circ. Syst. Vid.
fileexchange/27798-bjontegaard-metric, accessed 2019-03-20, 2010.
Technol.2012), 1649–1668.

[Link] Published online by Cambridge University Press

 16 thorsten laude et al.

[66] Grois D.; Nguyen T.; Marpe D.: “Coding efficiency comparison of ISO/IEC JTC 1/SC 29/WG 11 (MPEG), where he contributes
AV1/VP9, H.265/MPEG-HEVC, and H.264/MPEG-AVC encoders,” to the MPEG-G standard (ISO/IEC 23092) and where he
in 2016 Picture Coding Symp. (PCS), Nuremberg, DE, 2016, 1–5, IEEE.
serves as co-editor of parts 2 and 5 of MPEG-G. His current
Thorsten Laude studied electrical engineering at the research ranges from information theory to bioinformatics.
Leibniz University Hannover with a specialization in
Marco Munderloh achieved his Dipl.-Ing. degree in com-
communication engineering and received his Dipl.-Ing.
puter engineering with an emphasis on multimedia infor-
degree in 2013. In his diploma thesis, he developed a motion
mation and communication systems from the Technical
blur compensation algorithm for the scalable extension of
University of Ilmenau, Germany, in 2004. His diploma the-
HEVC. He joined InterDigital for an internship in 2013.
sis at the Fraunhofer Institute for Digital Media Technology
At InterDigital, he intensified his research for HEVC. After
dealt with holographic sound reproduction, the so-called
graduating, he joined the Institut für Informationsverar-
wave field synthesis (WFS) where he helds a patent. Dur-
beitung at the Leibniz University Hannover where he is cur-
ing his work at the Fraunhofer Institute, he was involved in
rently pursuing the Ph.D. degree. He contributed to several
the development of the first WFS-enabled movie theater. At
standardization meetings for HEVC and its extensions. His
the Institut für Informationsverarbeitung of the Leibniz Uni-
current research interests are intra coding for HEVC, still
versity Hannover, Marco Munderloh wrote his thesis with a
image coding, and machine learning for inter prediction,
focus on motion detection in scenes with non-static cam-
intra prediction, and encoder control.
eras for aerial surveillance applications and received his
Dr.-Ing. degree in 2015.
Yeremia Gunawan Adhisantoso studied electrical engi-
neering at the Leibniz University Hannover with special-
ization on computer engineering. He received his Master’s Jörn Ostermann studied electrical engineering and com-
degree in 2019 with his thesis on “Auto-encoder for Domain munications engineering at the University of Hannover and
Adopted Head Pose Estimation”. He has been working on Imperial College London. He received the Dipl.-Ing. and Dr.-
various research topics in the field of electrical engineer- Ing. degrees from the University of Hannover in 1988 and
ing with interest in reachability analysis, deep learning, and 1994, respectively. In 1994, he joined AT&T Bell Labs. From
genome coding. 1996 to 2003, he was with AT&T Labs – Research. Since 2003,
he is a Full Professor and Head of the Institut für Informa-
Jan Voges studied electrical engineering at the Leibniz Uni- tionsverarbeitung at Leibniz University Hannover, Germany.
versity Hannover with specialization on communications Since 2008, Jörn Ostermann is the Chair of the Require-
engineering. He received his Dipl.-Ing. degree from Leibniz ments Group of MPEG (ISO/IEC JTC1 SC29 WG11). Jörn
University Hannover in 2015. After graduating, he joined the Ostermann received several international awards and is a
Institut für Informationsverarbeitung at the Leibniz Univer- Fellow of the IEEE. He published more than 100 research
sity Hannover where he is currently working as a research papers and book chapters. He is a coauthor of a graduate-
assistant toward his Ph.D. degree. In the second half of level text book on video communications. He holds more
the year 2018, he worked as a visiting scholar at the Carl than 30 patents. His current research interests are video
R. Woese Institute for Genomic Biology at the University of coding and streaming, computer vision, 3D modeling, face
Illinois at Urbana-Champaign. He is an active contributor to animation, and computer–human interfaces.

[Link] Published online by Cambridge University Press