data2vec: A General Framework for Self-supervised Learning in Speech,

Vision and Language

Alexei Baevski 1 Wei-Ning Hsu 1 Qiantong Xu 2 Arun Babu 1 Jiatao Gu 1 Michael Auli 1

Abstract Research in self-supervised algorithms has focused on in-

While the general idea of self-supervised learn- dividual modalities which results in specific designs and
arXiv:2202.03555v1 [[Link]] 7 Feb 2022

ing is identical across modalities, the actual al- learning biases. For example, in speech processing, there
gorithms and objectives differ widely because is no vocabulary of speech units over which we can define
they were developed with a single modality in a self-supervised learning task such as words in NLP1 and
mind. To get us closer to general self-supervised therefore several prominent models are equipped with mech-
learning, we present data2vec, a framework that anisms to learn an inventory of speech units (Baevski et al.,
uses the same learning method for either speech, 2020b; Hsu et al., 2021). A similar problem exists for com-
NLP or computer vision. The core idea is to pre- puter vision, where researchers either learn discrete visual
dict latent representations of the full input data tokens (Radford et al., 2021a; Bao et al., 2021), regress
based on a masked view of the input in a self- the input (He et al., 2021) or learn representations invariant
distillation setup using a standard Transformer ar- to data augmentation (Chen et al., 2020; Grill et al., 2020;
chitecture. Instead of predicting modality-specific Caron et al., 2021).
targets such as words, visual tokens or units of While learning biases are certainly helpful, it is often un-
human speech which are local in nature, data2vec clear whether they will generalize to other modalities. More-
predicts contextualized latent representations that over, leading theories on the biology of learning (Friston &
contain information from the entire input. Ex- Kiebel, 2009; Friston, 2010) imply that humans likely use
periments on the major benchmarks of speech similar learning processes to understand the visual world as
recognition, image classification, and natural lan- they do for language. Relatedly, general neural network ar-
guage understanding demonstrate a new state of chitectures have been shown to perform very well compared
the art or competitive performance to predom- to modality-specific counterparts (Jaegle et al., 2021).
inant approaches. Models and code are avail-
able at [Link]/pytorch/fairseq/ In an effort to get closer to machines that learn in gen-
tree/master/examples/data2vec. eral ways about the environment, we designed data2vec, a
framework for general self-supervised learning that works
for images, speech and text where the learning objective
1. Introduction is identical in each modality. The present work unifies the
learning algorithm but still learns representations individu-
Self-supervised learning builds representations of data with- ally for each modality. We hope that a single algorithm will
out human annotated labels which led to significant ad- make future multi-modal learning simpler, more effective
vances in natural language processing (NLP; Peters et al. and lead to models that understand the world better through
2018; Radford et al. 2018; Devlin et al. 2019; Brown et al. multiple modalities.
2020), speech processing (van den Oord et al., 2018; Schnei-
der et al., 2019; Baevski et al., 2020b) as well as computer Our method combines masked prediction (Devlin et al.,
vision (Chen et al., 2020; 2021b; Caron et al., 2021; Bao 2019; Baevski et al., 2020b; Bao et al., 2021) with the learn-
et al., 2021; He et al., 2021). Self-supervised representa- ing of latent target representations (Grill et al., 2020; Caron
tions have even enabled completely unsupervised learning et al., 2021) but generalizes the latter by using multiple net-
in tasks such as machine translation (Lample et al., 2018) work layers as targets and shows that this approach works
and speech recognition (Baevski et al., 2021). across several modalities. Specifically, we train an off-the-
shelf Transformer network (Vaswani et al., 2017) which we
1
Meta AI 2 SambaNova, work done while at Meta AI. Corre- use either in teacher or student mode (Illustration in Fig-
spondence to: Alexei Baevski <abaevski@[Link]>, Michael Auli
1
<michaelauli@[Link]>. This is true for many languages but not for certain Asian
languages.
 data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language

Images Speech Language

Model in teacher-mode

Original I like tea with milk

Teacher tracks
student
Predict model parameters
Model in student-mode
representation of
original input
Masked I like tea with milk

Figure 1. Illustration of how data2vec follows the same learning process for different modalities. The model first produces representations
of the original input example (teacher mode) which are then regressed by the same model based on a masked version of the input. The
teacher parameters are an exponentially moving average of the student weights. The student predicts the average of K network layers of
the teacher (shaded in blue).

ure 1): we first build representations of the full input data Chen et al., 2021b) as well as online clustering (Caron et al.,
whose purpose is to serve as targets in the learning task 2020). Similar to our work, both BYOL (Grill et al., 2020)
(teacher mode). Next, we encode a masked version of the and DINO (Caron et al., 2021) regress neural network rep-
input sample with which we predict the full data represen- resentations of a momentum encoder, but our work differs
tations (student mode). The weights of the teacher are an in that it uses a masked prediction task and we regress mul-
exponentially decaying average of the student (He et al., tiple neural network layer representations instead of just the
2019; Grill et al., 2020; Caron et al., 2021). Since differ- top layer which we find to be more effective. Moreover,
ent modalities have vastly different inputs, e.g., pixels vs. data2vec works for multiple modalities.
words, we use modality-specific feature encoders and mask-
The most recent work focuses on training vision Trans-
ing strategies from the literature.
formers (Dosovitskiy et al., 2020) with masked prediction
Since our method works with the latent network representa- objectives (Bao et al., 2021; He et al., 2021; Xie et al., 2021)
tions of the learner itself, it can be seen as a simplification whose performance surpasses supervised-only training on
of many modality-specific designs such as learning a fixed ImageNet-1K. Several of these methods predict visual to-
set of visual tokens (Radford et al., 2021a; van den Oord kens (Bao et al., 2021; He et al., 2021; Dong et al., 2022)
et al., 2017), or normalization of the input to create suitable learned in a separate step before pre-training (van den Oord
targets (He et al., 2021), or the learning of a vocabulary et al., 2017; Ramesh et al., 2021), during pretraining (Zhou
of discrete speech units (Baevski et al., 2020b; Hsu et al., et al., 2021), and others directly predict the input pixels (He
2021). Moreover, our target representations are continuous et al., 2021; Xie et al., 2021). Instead, data2vec predicts the
and contextualized, through the use of self-attention, which latent representations of the input data. Another difference
makes them richer than a fixed set of targets and/or targets to this body of work is that the latent target representations
based on local context such as used in most prior work. are contextualized, incorporating relevant features from the
entire image instead of targets which contain information
Experimental results show data2vec to be effective in all
isolated to the current patch, such as visual tokens or pixels.
three modalities, setting a new state of the art for ViT-B with
single models and ViT-L on ImageNet-1K, improving over
the best prior work in speech processing on speech recogni- Self-supervised learning in NLP. Pre-training has been
tion (Baevski et al., 2020b; Hsu et al., 2021) and outperform- very successful in advancing natural language understand-
ing a like for like RoBERTa baseline on the GLUE natural ing (McCann et al., 2017; Peters et al., 2018; Radford et al.,
language understanding benchmark (Liu et al., 2019). 2018; Baevski et al., 2019; Devlin et al., 2019; Yang et al.,
2019; Brown et al., 2020). The most prominent model is
BERT (Devlin et al., 2019) which solves a masked predic-
2. Related work
tion task where some of the input tokens are blanked out in
Self-supervised learning in computer vision. Unsuper- order to be predicted given the remaining input. For many
vised pre-training for computer vision has been a very active languages it is easy to determine word boundaries and most
area of research with methods contrasting representations methods therefore predict word or sub-word units for pre-
of augmentations of the same image, entirely different im- training. There is also work on knowledge distillation to
ages (Chen et al., 2020; Grill et al., 2020; Caron et al., 2021; obtain smaller BERT-style models, both for pre-training and
fine-tuning (Jiao et al., 2020).
 data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language

Compared to prior NLP algorithms, data2vec does not pre- ure 1). We first encode a masked version of the training
dict discrete linguistic tokens such as words, sub-words or sample (model in student mode) and then construct training
bytes but rather a continuous and contextualized representa- targets by encoding the unmasked version of the input with
tion. This has two advantages: first, the targets themselves the same model but when parameterized as an exponentially
are not predefined, nor is their number limited. This enables moving average of the model weights (model in teacher
the model to adapt to a particular input example. Second, mode; Grill et al. 2020; Caron et al. 2021). The target rep-
targets are contextualized, taking context information into resentations encode all of the information in the training
account. This is unlike BERT-style models which learn sample and the learning task is for the student to predict
a single embedding for each target which needs to fit all these representations given a partial view of the input.
instances of a particular target in the data.
3.1. Model architecture
Self-supervised learning in speech. Work in self- We use the standard Transformer architecture (Vaswani et al.,
supervised learning for speech includes autoregressive mod- 2017) with a modality-specific encoding of the input data
els (van den Oord et al., 2018; Schneider et al., 2019; borrowed from prior work:3 for computer vision, we use the
Baevski et al., 2020a; Chung et al., 2019) as well as bi- ViT-strategy of encoding an image as a sequence of patches,
directional models (Baevski et al., 2020b; Hsu et al., 2021; each spanning 16x16 pixels, input to a linear transforma-
Ao et al., 2021; Chen et al., 2021a). Two prominent mod- tion (Dosovitskiy et al., 2020; Bao et al., 2021). Speech
els, wav2vec 2.0 and HuBERT are based on predicting data is encoded using a multi-layer 1-D convolutional neural
discrete units of speech, either learned jointly during pre- network that maps 16 kHz waveform to 50 Hz representa-
training (Baevski et al., 2020b), or in an iterative pipeline tions (Baevski et al., 2020b). Text is pre-processed to obtain
approach (Hsu et al., 2021) where pre-training and cluster- sub-word units (Sennrich et al., 2016; Devlin et al., 2019),
ing alternate.2 Another line of work directly reconstructs which are then embedded in distributional space via learned
the input features (Eloff et al., 2019; Liu et al., 2021). embedding vectors. We detail these methods below (§4).
In comparison to wav2vec 2.0, data2vec directly predicts
contextualized latent representations without quantization. 3.2. Masking
HuBERT discretizes representations from different layers
After the input sample has been embedded as a sequence
across iterations and predicts these discretized units whereas
of tokens, we mask part of these units by replacing them
data2vec predicts the average over multiple layers. Similar
with a learned MASK embedding token and feed the se-
to other modalities, there is work on distilling larger self-
quence to the Transformer network. For computer vision,
supervised models into smaller models but primarily for the
we follow the block-wise masking strategy of Bao et al.
purpose of efficiency (Chang et al., 2021).
(2021), for speech we mask spans of latent speech represen-
tations (Baevski et al., 2020b) and for language we mask
Multimodal pre-training. There has been a considerable tokens (Devlin et al., 2019); §4 details each strategy.
body of research on learning representations of multiple
modalities simultaneously often using paired data (Aytar 3.3. Training targets
et al., 2017; Radford et al., 2021b; Wang et al., 2021; Singh
et al., 2021) with the aim to produce cross-modal representa- The model is trained to predict the model representations of
tions which can perform well on multi-modal tasks and with the original unmasked training sample based on an encoding
modalities benefiting from each other through joint train- of the masked sample. We predict model representations
ing (Alayrac et al., 2020; Akbari et al., 2021) with recent only for time-steps which are masked. The representations
methods exploring few-shot learning (Tsimpoukelli et al., we predict are contextualized representations, encoding the
2021). Our work does not perform multimodal training particular time-step but also other information from the
but aims to unifiy the learning objective for self-supervised sample due to the use of self-attention in the Transformer
learning in different modalities. We hope that this will en- network.4 This is an important difference to BERT (De-
able better multimodal representations in the future. vlin et al., 2019), wav2vec 2.0 (Baevski et al., 2020b) or
BEiT, MAE, SimMIM, and MaskFeat (Bao et al., 2021;
He et al., 2021; Xie et al., 2021; Wei et al., 2021) which
3. Method predict targets lacking contextual information. Below, we
data2vec is trained by predicting the model representations 3
While we used Transformer networks, alternative architectures
of the full input data given a partial view of the input (Fig- may be equally applicable.
4
2 In preliminary experiments, we found that additional context
Quantization is optional for wav2vec 2.0 (Baevski et al.,
information for the targets was helpful since masking some of the
2020b; Zhang et al., 2020) but helpful for noisy speech (Chung
time-steps when in teacher mode resulted in lower accuracy.
et al., 2021).
 data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language

detail how we parameterize the teacher which predicts the 3.4. Objective
network representations that will serve as targets as well as
Given contextualized training targets yt , we use a Smooth
how we construct the final target vectors to be predicted by
L1 loss to regress these targets:
the model in student-mode.
(
1
(yt − ft (x))2 /β |yt − ft (x)| ≤ β
L(yt , ft (x)) = 2
Teacher parameterization. The encoding of the un- 1
(|yt − ft (x)| − 2 β) otherwise
masked training sample is parameterized by an exponen-
tially moving average (EMA) of the model parameters where β controls the transition from a squared loss to an L1
(θ; Tarvainen & Valpola 2018; Grill et al. 2020; Caron et al. loss, depending on the size of the gap between the target yt
2021) where the weights of the model in target-mode ∆ are: and the model prediction ft (x) at time-step t. The advantage
of this loss is that it is less sensitive to outliers, however, we
∆ ← τ ∆ + (1 − τ ) θ need to tune the setting of β.

4. Experimental setup
We use a schedule for τ that linearly increases this parameter
from τ0 to the target value τe over the first τn updates after We experiment with two model sizes: data2vec Base and
which the value is kept constant for the remainder of train- data2vec Large, containing either L = 12 or L = 24 Trans-
ing. This strategy results in the teacher being updated more former blocks with H = 768 or H = 1024 hidden dimen-
frequently at the beginning of training, when the model is sion (with 4 × H feed-forward inner-dimension). EMA up-
random, and less frequently later in training, when good dates are performed in fp32 for numerical stability (Manohar
parameters have already been learned. We found it more et al., 2021).
efficient and slightly more accurate to share the parameters
of the feature encoder and the positional encoder between 4.1. Computer vision
the teacher and student networks.
We embed images of 224x224 pixels as patches of 16x16
pixels (Dosovitskiy et al., 2020). Each patch is linearly
Targets. Training targets are constructed based on the out- transformed and a sequence of 196 representations is input
put of the top K blocks of the teacher network for time-steps to a standard Transformer. We follow BEiT (Bao et al.,
which are masked in student-mode.5 The output of block 2021) by masking blocks of multiple adjacent patches where
l at time-step t is denoted as alt . We apply a normaliza- each block contains at least 16 patches with a random aspect
tion to each block to obtain âlt before averaging the top K ratio. Different to their work, we found it more accurate to
1
PL l
blocks yt = K l=L−K+1 ât for a network with L blocks
mask 60% of the patches instead of 40%. We use randomly
in total to obtain the training target yt for time-step t. This applied resized image crops, horizontal flipping, and color
creates training targets that are to be regressed by the model jittering (Bao et al., 2021). We use the same modified image
when in student mode. In preliminary experiments we found both in teacher mode and student mode.
that averaging performed as well as predicting each block
ViT-B models are pre-trained for 800 epochs. As batch
separately with a dedicated projection while enjoying the
size we use 2,048 for ViT-B and 8,192 for ViT-L. We
advantage of being more efficient.
use Adam (Kingma & Ba, 2015) and a cosine sched-
Normalizing the targets helps prevent the model from col- ule (Loshchilov & Hutter, 2016) with a single cycle where
lapsing into a constant representation for all time-steps and we warm up the learning rate for 40 epochs to 0.002 for
it also prevents layers with high norm to dominate the tar- ViT-B and for 80 epochs to 0.001 for ViT-L after which the
get features. For speech representations, we use instance learning rate is annealed following the cosine schedule. For
normalization (Ulyanov et al., 2016) without any learned ViT-B and ViT-L, we use β = 2, K = 6 and τ = 0.9998
parameters over the current input sample since neighbor- as a constant value with no schedule which worked well.
ing representations are highly correlated due to the small We use stochastic depth with rate 0.2 (Huang et al., 2016).
stride over the input data, while for NLP and vision we For ViT-L, we train for 1,600 epochs in total, the first 800
found parameter-less layer normalization (Ba et al., 2016) epochs use τ = 0.9998, we then reset the learning rate
to be sufficient. Variance-Invariance-Covariance regulariza- schedule and the teacher weights to the student and continue
tion (Bardes et al., 2021) also addresses this problem but for another 800 epochs with τ = 0.9999.
we found the above strategy to perform well and it does not
For image classification we mean-pool the output of the
introduce additional hyper-parameters.
last Transformer block and input it to a softmax-normalized
5
We generally use the output of the FFN prior to the last resid- classifier. We fine-tune ViT-B for 100 epochs and ViT-L
ual connection in each block as target. See the ablation in §5.4. for 50 epochs using Adam and a cosine schedule where
 data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language

we warmup up the learning rate for 20 epochs to 0.004 for

Table 1. Computer vision: top-1 validation accuracy on ImageNet-
ViT-B and for 5 epochs to 0.004 for ViT-L after which the
1K with ViT-B and ViT-L models. data2vec ViT-B was trained
learning rate follows the cosine schedule. We build on the for 800 epochs and ViT-L for 1,600 epochs. We distinguish be-
open source implementation of BEiT (Bao et al., 2021). tween individual models and setups composed of multiple models
(BEiT/PeCo train separate visual tokenizers and PeCo also distills
4.2. Speech processing two MoCo-v3 models).

Models are implemented in fairseq (Ott et al., 2019) and take

ViT-B ViT-L
as input 16 kHz waveform which is processed by a feature
encoder (Baevski et al., 2020b) containing seven temporal Multiple models
convolutions with 512 channels, strides (5,2,2,2,2,2,2) and BEiT (Bao et al., 2021) 83.2 85.2
kernel widths (10,3,3,3,3,2,2). This results in an encoder PeCo (Dong et al., 2022) 84.5 86.5
output frequency of 50 Hz with a stride of about 20ms Single models
between each sample, and a receptive field of 400 input MoCo v3 (Chen et al., 2021b) 83.2 84.1
samples or 25ms of audio. The raw waveform input to the DINO (Caron et al., 2021) 82.8 -
encoder is normalized to zero mean and unit variance. MAE (He et al., 2021) 83.6 85.9
The masking strategy for the Base model is also identical SimMIM (Xie et al., 2021) 83.8 -
to Baevski et al. (2020b): we sample p = 0.065 of all iBOT (Zhou et al., 2021) 83.8 -
time-steps to be starting indices and mask the subsequent MaskFeat (Wei et al., 2021) 84.0 85.7
ten time-steps. This results in approximately 49% of all data2vec 84.2 86.6
time-steps to be masked for a typical training sequence.
During pre-training we linearly anneal τ using τ0 = 0.999,
τe = 0.9999 and τn = 30, 000, average the top K = 8 total batch size of 256 sequences and each sequence is up
blocks as targets and found a simple L2 loss to work well. to 512 tokens. For downstream tasks, we fine-tune the pre-
We optimize with Adam (Kingma & Ba, 2015), with a trained model with four different learning rates (1 × 10−5 ,
peak learning rate of 5 × 10−4 for data2vec Base. The Base 2 × 10−5 , 3 × 10−5 , 4 × 10−5 ) and choose the one which
model uses a tri-stage scheduler which linearly warms up the performs best across all considered NLP downstream tasks.
learning rate over the first 3% of updates, holds it for 90%
of updates and then linearly decays it over the remaining 5. Results
7%. We train data2vec Base for 400K updates with a batch
size of 63 minutes of audio (61M frames). We follow the 5.1. Computer vision
fine-tuning regime of wav2vec 2.0 (Baevski et al., 2020b) To evaluate our approach for computer vision, we pre-
whose hyper-parameters depend on the labeled data setup. train data2vec on the images of the ImageNet-1K training
set (Deng et al., 2009) and fine-tune the resulting model
4.3. Natural language processing for image classification using the labeled data of the same
benchmark (§4.1). Following standard practice, models
We build on the BERT re-implementation RoBERTa (Liu
are evaluated in terms of top-1 accuracy on the validation
et al., 2019) available in fairseq (Ott et al., 2019). The input
set. We distinguish between results based on a single self-
data is tokenized using a byte-pair encoding (Sennrich et al.,
supervised model, and results which train a separate visual
2016) of 50K types and the model learns an embedding for
tokenizer on additional data (Bao et al., 2021) or distill other
each type (Devlin et al., 2019; Liu et al., 2019) Once the
self-supervised models (Dong et al., 2022).
data is embedded, we apply the BERT masking strategy to
15% of uniformly selected tokens: 80% are replaced by a Table 1 shows that data2vec outperforms prior work with
learned mask token, 10% are left unchanged and 10% are ViT-B and ViT-L in the single model setting and all prior
replaced by randomly selected vocabulary token; we do not work for ViT-L. Predicting contextualized latent represen-
use the next-sentence prediction task. We also consider the tations in a masked prediction setup can perform very well
wav2vec 2.0 strategy of masking spans of four tokens. compared to approaches which predict local targets such
as the original input pixels (He et al., 2021; Xie et al.,
For pre-training we use τ0 = 0.999, τe = 0.9999 and
2021), engineered image features (Wei et al., 2021) or visual
τn = 100, 000, K = 10 and set β = 4. The model is
tokens (Bao et al., 2021). It also outperforms prior self-
optimized with Adam over 1M updates using a tri-stage
distillation methods (Caron et al., 2021) which regressed
learning rate schedule (5%, 80% and 15% of updates for
the final layer of the student network while inputting two
warm-up, holding and linearly decaying, respectively). The
different augmented versions of an image to the student and
peak learning rate is 2 × 10−4 . We train on 16 GPUs with a
teacher networks.
 data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language

Table 2. Speech processing: word error rate on the Librispeech test-other test set when fine-tuning pre-trained models on the Libri-light
low-resource labeled data setups (Kahn et al., 2020) of 10 min, 1 hour, 10 hours, the clean 100h subset of Librispeech and the full 960h of
Librispeech. Models use the 960 hours of audio from Librispeech (LS-960) as unlabeled data. We indicate the language model used
during decoding (LM). Results for all dev/test sets and other LMs can be found in the supplementary material (Table 5).

Unlabeled Amount of labeled data

LM
data 10m 1h 10h 100h 960h
wav2vec 2.0 (Baevski et al., 2020b) LS-960 4-gram 15.6 11.3 9.5 8.0 6.1
HuBERT (Hsu et al., 2021) LS-960 4-gram 15.3 11.3 9.4 8.1 -
WavLM (Chen et al., 2021a) LS-960 4-gram - 10.8 9.2 7.7 -
data2vec LS-960 4-gram 12.3 9.1 8.1 6.8 5.5

5.2. Speech processing data provided by each task and report the average accuracy
on the development sets over five fine-tuning runs. We
For speech processing, we pre-train data2vec on the 960
compare to the published BERT results as well as to the
hours of speech audio data from Librispeech (LS-960). This
results we obtain by retraining RoBERTa in the current
dataset contains relatively clean speech audio from read
setup (Baseline; Liu et al. 2019) which provides a more
audiobooks in English and is a standard benchmark in the
suitable baseline to data2vec since we build on their open
speech community. To get a sense of performance in dif-
source code.
ferent resource settings, we fine-tune models for automatic
speech recognition using different amounts of labeled data, The results (Table 3) show that data2vec outperforms the
ranging from just 10 minutes to 960 hours. We also com- RoBERTa baseline. When we mask spans of four BPE
pare to other work from the literature, including wav2vec tokens with masking probability 0.35 (Baevski et al., 2020b),
2.0 (Baevski et al., 2020b) and HuBERT (Hsu et al., 2021), then results improve further.7 This strategy does not leave
two popular algorithms for speech representation learning tokens unmasked or uses random targets as for BERT (§4.3).
relying on discrete units of speech.
To our knowledge this is the first successful pre-trained
Table 2 shows improvements for all labeled data setups with NLP model which does not use discrete units (words, sub-
the largest gains for 10 minutes of labeled data (20% rela- words, characters or bytes) as the training target. Instead,
tive WER improvement). Our results suggest that learning the model predicts a contextualized latent representation
discrete units is not required when rich contextualized tar- emerging from self-attention over the entire unmasked text
gets are used and that learning contextualized targets during sequence. This enables a learning task where the model
pre-training improves performance. needs to predict targets with specific properties of the current
text sequence rather than representations which are generic
5.3. Natural language processing to every text sequence in which the particular discrete unit
occurs. Moreover, the set of training targets is not fixed, i.e.,
To get a sense of how data2vec performs for language, we not a closed vocabulary, and the model can choose to define
adopt the same training setup as BERT (Devlin et al., 2019) new target types as it sees fit, akin to an open vocabulary
by pre-training on the Books Corpus (Zhu et al., 2015) and setting.
English Wikipedia data over 1M updates and a batch size
of 256 sequences. We evaluate on the General Language 5.4. Ablations
Understanding Evaluation (GLUE) benchmark (Wang et al.,
2018) which includes tasks for natural language inference Layer-averaged targets. One of the main differences of
(MNLI, QNLI, RTE), sentence similarity (MRPC, QQP and our method compared to BYOL is the use of targets which
STS-B), grammaticality (CoLA), and sentiment analysis are based on averaging multiple layers from the teacher net-
(SST-2).6 We fine-tune data2vec separately on the labeled work (§3.3). This idea was partly inspired by the fact that the
6
top layers of wav2vec 2.0 do not perform as well for down-
MNLI (Multi Genre Natural Language Inference; Williams stream tasks as layers in the middle of the network (Baevski
et al. 2018), Stanford Question Answering Dataset (QNLI; Ra-
jpurkar et al. 2016), Recognizing Textual Entailment (RTE;Dagan mark (STS-B; Cer et al. 2018), Corpus of Linguistic Acceptability
et al. 2006; Haim et al. 2006; Giampiccolo et al. 2007; Bentivogli (CoLA; Warstadt et al. 2018), Stanford Sentiment Treebank (SST-
et al. 2009), and we exclude Winograd NLI task from our results 2; Socher et al. 2013)
similar to Devlin et al. (2019), Microsoft Research Paragraph 7
Note, that we used a cosine learning rate schedule for this
Corpus (MRPC; Dolan & Brockett 2005), Quora Question Pairs result.
benchmark (QQP), and the Semantic Textual Similarity Bench-
 data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language

Table 3. Natural language processing: GLUE results on the development set for single-task fine-tuning of individual models. For MNLI
we report accuracy on both the matched and unmatched dev sets, for MRPC and QQP, we report the unweighted average of accuracy and
F1, for STS-B the unweighted average of Pearson and Spearman correlation, for CoLA we report Matthews correlation and for all other
tasks we report accuracy. BERT Base results are from Wu et al. (2020) and our baseline is RoBERTa re-trained in a similar setup as BERT.
We also report results with wav2vec 2.0 style masking of spans of four BPE tokens with no unmasked tokens or random targets.

MNLI QNLI RTE MRPC QQP STS-B CoLA SST Avg.

BERT (Devlin et al., 2019) 84.0/84.4 89.0 61.0 86.3 89.1 89.5 57.3 93.0 80.7
Baseline (Liu et al., 2019) 84.1/83.9 90.4 69.3 89.0 89.3 88.9 56.8 92.3 82.5
data2vec 83.2/83.0 90.9 67.0 90.2 89.1 87.2 62.2 91.8 82.7
+ wav2vec 2.0 masking 82.8/83.4 91.1 69.9 90.0 89.0 87.7 60.3 92.4 82.9

Top-1 valid accuracy

40
Word error rate

80 84
GLUE score

30
70
82
20
60
80
1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12

K K K
(a) Speech (b) NLP (c) Vision

Figure 2. Predicting targets which are the average of multiple layers is more robust than predicting only the top most layer (K = 1)
for most modalities. We show the performance of predicting the average of K teacher layer representations (§3.3). The effect is very
pronounced for speech and NLP while for vision there is still a slight advantage of predicting more than a single layer.

et al., 2021; Pasad et al., 2021).

Table 4. Effect of using different features from the teacher model as
In the next experiment, we measure performance for all targets: we compare using the output of the self-attention module,
three modalities when averaging K = 1, . . . , 12 layers the feed-forward module (FFN) as well as after the final resid-
where K = 1 corresponds to predicting only the top layer ual connection (FFN + residual) and layer normalization (End of
similar to BYOL. For faster experimental turn-around, we block). Results are not directly comparable to the main results
since we use a reduced setup (§5.4).
train Base models with L = 12 layers in total. For speech,
we pre-train for 200K updates on Librispeech, fine-tune on
the 10 hour labeled split of Libri-light (Kahn et al., 2019) Layer WER
and report word error rate without a language model on dev- self-attention 100.0
other. For NLP, we report the average GLUE score on the FFN 13.1
validation set (§5.3) and for computer vision we pre-train FFN + residual 14.8
models for 300 epochs and report the top-1 accuracy on End of block 14.5
ImageNet (§5.1).
Figure 2 shows that targets based on multiple layers im-
proves over using only the top layer (K = 1) for all modal- different layers impact performance, we pre-train speech
ities. Using all layers is generally a good choice and only models on Librispeech using the features from different
slightly worse than a carefully tuned value of K. Neural layers as target features. Table 4 shows that the output of
networks build features over multiple layers and different the feedforward network (FFN) block works best while the
types of features are extracted at different layers. Using output of the self-attention block does not yield a usable
features from multiple layers enriches the self-supervised model. We believe this is because the self-attention output
task and improves accuracy. is before the residual connection and features are heavily
biased towards other time-steps. This issue is alleviated by
Target feature type. Transformers blocks contain several the use of the FFN features since these include the features
layers which can each serve as targets. To get a sense of how before the self-attention as well.
 data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language

6. Discussion more, the representation of each instance of a particular unit

(word/sub-word/character/byte) can differ for the masked
Modality-specific feature extractors and masking. Our prediction task. This enables to associate a different mean-
primary is to design a single learning mechanism for differ- ing to a particular depending on the context it occurs in. It
ent modalities. Despite the unified learning regime, we still also relieves the model from the need to learn a single set of
use modality-specific features extractors and masking strate- features for a target unit that fits all instances of this unit.
gies. This makes sense given the vastly different nature of
the input data: for example, in speech we learn from a very
high resolution input (16 kHz waveform) which contains Representation collapse. A common issue with algo-
hundreds of thousands of samples for typical utterances. rithms which learn their own targets is representation col-
To process this, we apply a multilayer convolutional neu- lapse. This occurs when the model produces very similar
ral network to obtain a 50 Hz feature sequence. For NLP, representations for all masked segments which results in
inputs are of vastly lower resolution in the form of much a trivial task (Jing et al., 2021). To deal with this issue,
shorter word sequences which can be directly embedded contrastive models such as wav2vec 2.0 (Baevski et al.,
in distributional space via a lookup table. The type of data 2020b) use the same target representation both as a positive
also impacts how we should mask the input to create a chal- and a negative example. BYOL (Grill et al., 2020) do not
lenging learning task: removing individual words provides optimize the teacher parameters to minimize the loss and Vi-
a sufficiently challenging task but for speech it is neces- cReg (Bardes et al., 2021) adds an explicit loss encouraging
sary to mask spans since adjacent audio samples are highly variance among different representations.
correlated with each other. We found that collapse is most likely to happen in the fol-
Relatedly, there has been recent work on a Transformer lowing scenarios: First, the learning rate is too large or the
architecture that can directly operate on the raw data of learning rate warmup is too short which can often be solved
different modalities without modality-specific feature en- by tuning the respective hyperparameters. Second, τ is too
coders (Jaegle et al., 2021). Their work is focused on su- low which leads to student model collapse and is then prop-
pervised learning for classification tasks and we believe that agated to the teacher. This can be addressed by tuning τ0 ,
our work is complementary. τe and τn . Third, we found collapse to be more likely for
modalities where adjacent targets are very correlated and
where longer spans need to be masked, e.g., speech. We
Structured and contextualized targets. One of the main
address this by promoting variance through normalizing tar-
differences of data2vec to most other masked prediction
get representations over the sequence or batch (Grill et al.,
work (Devlin et al., 2019; Baevski et al., 2020b; Ling et al.,
2020). For models where targets are less correlated, such as
2020; Bao et al., 2021; He et al., 2021; Wei et al., 2021) is
vision and NLP, momentum tracking is sufficient.
that the features of the training targets are contextualized
since the features are built with self-attention over the entire
unmasked input in teacher mode. And while BYOL (Grill 7. Conclusion
et al., 2020) and DINO (Caron et al., 2021) also use latent
Recent work showed that uniform model architectures can
target representations based on the entire input, their focus is
be effective for multiple modalities (Jaegle et al., 2021). In a
on learning transformation-invariant representations instead
similar spirit, we show that a single self-supervised learning
of structural information within a sample.
regime can be effective for vision, speech and language. The
One exception is HuBERT (Hsu et al., 2021) which builds a key idea is to regress contextualized latent representations
fixed set of discrete target units by clustering Transformer based on a partial view of the input. data2vec outperforms
layer representations. In comparison, data2vec has no limi- prior self-supervised algorithms on ImageNet-1K for ViT-B
tation on the number of target units. Instead of representing single models and ViT-L models, it improves over prior
each instance of particular discrete target unit with the same work on speech recognition for the low-resource setups of
set of features, data2vec can build target features that are Libri-light, and it outperforms RoBERTa for natural lan-
specific to the current sequence. guage understanding on GLUE in the original BERT setup.
For NLP, we believe data2vec is the first work that does not A single learning method for multiple modalities will make
rely on pre-defined target units. Most other work uses either it easier to learn across modalities and future work may
words, sub-words (Radford et al., 2018; Devlin et al., 2019), investigate tasks such as audio-visual speech recognition or
characters (Tay et al., 2021) or even bytes (Xue et al., 2021). cross-modal retrieval. Our approach still uses modality-
Aside, defining word boundaries is not straightforward for specific input encoders and we adopt modality-specific
some Asian languages. Contextualized targets enable inte- masking strategies. Future work may investigate a single
grating features from the entire sequence into the training masking strategy that is modality-agnostic as well as jointly
target which provides a richer self-supervised task. Further- training multiple modalities.
 data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language

Acknowledgements Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu,
J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M.,
We thank Brenden Lake, Dhruv Batra and Marco Baroni for Gray, S., Chess, B., Clark, J., Berner, C., McCandlish,
helpful discussions. We also thank Laurens van der Maaten S., Radford, A., Sutskever, I., and Amodei, D. Language
for feedback on an earlier version of the paper. models are few-shot learners. In Proc. of NeurIPS, 2020.

References Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P.,
and Joulin, A. Unsupervised learning of visual features by
Akbari, H., Yuan, L., Qian, R., Chuang, W.-H., Chang, S.-F., contrasting cluster assignments. arXiv, abs/2006.09882,
Cui, Y., and Gong, B. Vatt: Transformers for multimodal 2020.
self-supervised learning from raw video, audio and text,
2021. Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bo-
janowski, P., and Joulin, A. Emerging properties in self-
Alayrac, J.-B., Recasens, A., Schneider, R., Arandjelović, supervised vision transformers. arXiv, abs/2104.14294,
R., Ramapuram, J., Fauw, J. D., Smaira, L., Dieleman, S., 2021.
and Zisserman, A. Self-supervised multimodal versatile
networks, 2020. Cer, D. M., Diab, M. T., Agirre, E., Lopez-Gazpio, I., and
Specia, L. Semeval-2017 task 1: Semantic textual simi-
Ao, J., Wang, R., Zhou, L., Liu, S., Ren, S., Wu, Y., Ko, T.,
larity - multilingual and cross-lingual focused evaluation.
Li, Q., Zhang, Y., Wei, Z., Qian, Y., Li, J., and Wei, F.
In Proc. of SemEval, 2018.
Speecht5: Unified-modal encoder-decoder pre-training
for spoken language processing. arXiv, abs/2110.07205, Chang, H.-J., wen Yang, S., and yi Lee, H. Distilhubert:
2021. Speech representation learning by layer-wise distillation
of hidden-unit bert. arXiv, abs/2110.01900, 2021.
Aytar, Y., Vondrick, C., and Torralba, A. See, hear, and read:
Deep aligned representations, 2017. Chen, S., Wang, C., Chen, Z., Wu, Y., Liu, S., Chen, Z., Li,
Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normalization. J., Kanda, N., Yoshioka, T., Xiao, X., Wu, J., Zhou, L.,
arXiv, abs/1607.06450, 2016. Ren, S., Qian, Y., Qian, Y., Wu, J., Zeng, M., and Wei, F.
Wavlm: Large-scale self-supervised pre-training for full
Baevski, A., Edunov, S., Liu, Y., Zettlemoyer, L., and Auli, stack speech processing. arXiv, abs/2110.13900, 2021a.
M. Cloze-driven pretraining of self-attention networks.
In Proc. of EMNLP, 2019. Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A
simple framework for contrastive learning of visual rep-
Baevski, A., Schneider, S., and Auli, M. vq-wav2vec: Self- resentations. arXiv, abs/2002.05709, 2020.
supervised learning of discrete speech representations. In
Proc. of ICLR, 2020a. Chen, X., Xie, S., and He, K. An empirical study
of training self-supervised vision transformers. arXiv,
Baevski, A., Zhou, Y., Mohamed, A., and Auli, M. wav2vec abs/2104.02057, 2021b.
2.0: A framework for self-supervised learning of speech
representations. In Proc. of NeurIPS, 2020b. Chung, Y., Hsu, W., Tang, H., and Glass, J. R. An unsu-
pervised autoregressive model for speech representation
Baevski, A., Hsu, W.-N., Conneau, A., and Auli, M. Unsu- learning. Proc. of Interspeech, 2019.
pervised speech recognition. In Proc. of NeurIPS, 2021.
Chung, Y.-A., Zhang, Y., Han, W., Chiu, C.-C., Qin, J., Pang,
Bao, H., Dong, L., and Wei, F. Beit: BERT pre-training of
R., and Wu, Y. W2v-bert: Combining contrastive learn-
image transformers. arXiv, abs/2106.08254, 2021.
ing and masked language modeling for self-supervised
Bardes, A., Ponce, J., and LeCun, Y. Vicreg: Variance- speech pre-training. arXiv, abs/2108.06209, 2021.
invariance-covariance regularization for self-supervised
Dagan, I., Glickman, O., and Magnini, B. The pascal rec-
learning. arXiv, abs/2105.04906, 2021.
ognizing textual entailment challenge. Machine learning
Bentivogli, L., Clark, P., Dagan, I., and Giampiccolo, D. challenges, evaluating predictive uncertainty, visual ob-
The fifth pascal recognizing textual entailment challenge. ject classification, and recognizing textual entailment, pp.
In Proc. of TAC, 2009. 177–190, 2006.

Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei,
J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., L. Imagenet: A large-scale hierarchical image database.
Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., In Proc. of CVPR, 2009.
 data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Huang, G., Sun, Y., Liu, Z., Sedra, D., and Weinberger,
Pre-training of deep bidirectional transformers for lan- K. Deep networks with stochastic depth. arXiv,
guage understanding. Proc. of NAACL, 2019. abs/1603.09382, 2016.

Dolan, W. B. and Brockett, C. Automatically constructing a Jaegle, A., Gimeno, F., Brock, A., Zisserman, A., Vinyals,
corpus of sentential paraphrases. In Proc. of IWP, 2005. O., and Carreira, J. Perceiver: General perception with
iterative attention. arXiv, abs/2103.03206, 2021.
Dong, X., Bao, J., Zhang, T., Chen, D., Zhang, W., Yuan, L.,
Chen, D., Wen, F., and Yu, N. Peco: Perceptual codebook Jiao, X., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L.,
for bert pre-training of vision transformers, 2022. Wang, F., and Liu, Q. Tinybert: Distilling bert for natural
language understanding. arXiv, abs/1909.10351, 2020.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn,
D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, Jing, L., Vincent, P., LeCun, Y., and Tian, Y. Understand-
M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. ing dimensional collapse in contrastive self-supervised
An image is worth 16x16 words: Transformers for image learning, 2021.
recognition at scale. arXiv, abs/2010.11929, 2020.
Kahn, J., Rivière, M., Zheng, W., Kharitonov, E., Xu, Q.,
Eloff, R., Nortje, A., van Niekerk, B., Govender, A., Nortje, Mazaré, P., Karadayi, J., Liptchinsky, V., Collobert, R.,
L., Pretorius, A., Van Biljon, E., van der Westhuizen, E., Fuegen, C., Likhomanenko, T., Synnaeve, G., Joulin, A.,
van Staden, L., and Kamper, H. Unsupervised acoustic Mohamed, A., and Dupoux, E. Libri-light: A bench-
unit discovery for speech synthesis using discrete latent- mark for asr with limited or no supervision. arXiv,
variable neural networks. arXiv, abs/1904.07556, 2019. abs/1912.07875, 2019.

Friston, K. The free-energy principle: a unified brain theory? Kahn, J. et al. Libri-light: A benchmark for asr with limited
Nature reviews neuroscience, 2010. or no supervision. In Proc. of ICASSP, 2020.

Friston, K. and Kiebel, S. Predictive coding under the free- Kingma, D. P. and Ba, J. Adam: A Method for Stochastic
energy principle. Philosophical transactions of the Royal Optimization. In Proc. of ICLR, 2015.
Society: Biological sciences, 2009. Lample, G., Denoyer, L., and Ranzato, M. Unsupervised
Giampiccolo, D., Magnini, B., Dagan, I., and Dolan, B. The machine translation using monolingual corpora only. In
pascal recognizing textual entailment challenge. Proc. Proc. of ICLR, 2018.
of the ACL-PASCAL workshop on textual entailment and Likhomanenko, T., Xu, Q., Kahn, J., Synnaeve, G., and
paraphrasing, 2007. Collobert, R. slimipl: Language-model-free iterative
pseudo-labeling. arXiv, abs/2010.11524, 2021.
Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond,
P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Ling, S., Liu, Y., Salazar, J., and Kirchhoff, K. Deep con-
Guo, Z. D., Azar, M. G., Piot, B., Kavukcuoglu, K., textualized acoustic representations for semi-supervised
Munos, R., and Valko, M. Bootstrap your own la- speech recognition. In Proc. of ICASSP, 2020.
tent: A new approach to self-supervised learning. arXiv,
abs/2006.07733, 2020. Liu, A. T., Li, S.-W., and Lee, H.-y. Tera: Self-supervised
learning of transformer encoder representation for speech.
Haim, R. B., Dagan, I., Dolan, B., Ferro, L., Giampiccolo, IEEE/ACM Trans. on Audio, Speech, and Language Pro-
D., Magnini, B., and Szpektor, I. The pascal recognising cessing, 2021.
textual entailment challenge. Lecture Notes in Computer
Science, 2006. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D.,
Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V.
He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Mo- Roberta: A robustly optimized bert pretraining approach.
mentum contrast for unsupervised visual representation arXiv preprint arXiv:1907.11692, 2019.
learning. arXiv, abs/1911.05722, 2019.
Loshchilov, I. and Hutter, F. SGDR: stochastic gradient
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. descent with restarts. arXiv, abs/1608.03983, 2016.
Masked autoencoders are scalable vision learners. arXiv,
abs/2111.06377, 2021. Manohar, V., Likhomanenko, T., Xu, Q., Hsu, W.-N.,
Collobert, R., Saraf, Y., Zweig, G., and Mohamed, A.
Hsu, W.-N., Tsai, Y.-H. H., Bolte, B., Salakhutdinov, R., Kaizen: Continuously improving teacher using exponen-
and Mohamed, A. Hubert: How much can a bad teacher tial moving average for semi-supervised speech recogni-
benefit ASR pre-training? In Proc. of ICASSP, 2021. tion, 2021.
 data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language

McCann, B., Bradbury, J., Xiong, C., and Socher, R. Singh, A., Hu, R., Goswami, V., Couairon, G., Galuba,
Learned in translation: Contextualized word vectors. W., Rohrbach, M., and Kiela, D. FLAVA: A foun-
arXiv, abs/1708.00107, 2017. dational language and vision alignment model. arXiv,
abs/2112.04482, 2021.
Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng,
N., Grangier, D., and Auli, M. fairseq: A fast, extensible Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning,
toolkit for sequence modeling. In Proc. of NAACL System C. D., Ng, A., and Potts, C. Recursive deep models for
Demonstrations, 2019. semantic compositionality over a sentiment treebank. In
Proc. of EMNLP, 2013.
Park, D. S., Zhang, Y., Jia, Y., Han, W., Chiu, C.-C., Li, B.,
Wu, Y., and Le, Q. V. Improved noisy student training Tarvainen, A. and Valpola, H. Mean teachers are better role
for automatic speech recognition. Proc. of Interspeech, models: Weight-averaged consistency targets improve
2020. semi-supervised deep learning results, 2018.

Pasad, A., Chou, J.-C., and Livescu, K. Layer-wise analysis Tay, Y., Tran, V. Q., Ruder, S., Gupta, J., Chung, H. W.,
of a self-supervised speech representation model. arXiv, Bahri, D., Qin, Z., Baumgartner, S., Yu, C., and Metzler,
abs/2107.04734, 2021. D. Charformer: Fast character transformers via gradient-
based subword tokenization. arXiv, abs/2106.12672,
Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, 2021.
C., Lee, K., and Zettlemoyer, L. Deep contextualized
Tsimpoukelli, M., Menick, J., Cabi, S., Eslami, S. M. A.,
word representations. In Proc. of ACL, 2018.
Vinyals, O., and Hill, F. Multimodal few-shot learning
Radford, A., Narasimhan, K., Salimans, T., and Sutskever, with frozen language models, 2021.
I. Improving language understanding by generative pre- Ulyanov, D., Vedaldi, A., and Lempitsky, V. S. Instance
training. [Link] normalization: The missing ingredient for fast stylization.
com/openai-assets/research-covers/ arXiv, abs/1607.08022, 2016.
language-unsupervised/language_
understanding_paper.pdf, 2018. van den Oord, A., Vinyals, O., et al. Neural discrete repre-
sentation learning. In Proc. of NeurIPS, 2017.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G.,
Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, van den Oord, A., Li, Y., and Vinyals, O. Representation
J., Krueger, G., and Sutskever, I. Learning transferable learning with contrastive predictive coding. Proc. of NIPS,
visual models from natural language supervision. arXiv, 2018.
abs/2103.00020, 2021a.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones,
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention
Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, is all you need. In Proc. of NIPS, 2017.
J., Krueger, G., and Sutskever, I. Learning transferable
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and
visual models from natural language supervision. arXiv,
Bowman, S. R. GLUE: A multi-task benchmark and anal-
abs/2103.00020, 2021b.
ysis platform for natural language understanding. arXiv,
Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. Squad: abs/1804.07461, 2018.
100, 000+ questions for machine comprehension of text. Wang, W., Bao, H., Dong, L., and Wei, F. Vlmo: Unified
arXiv, abs/1606.05250, 2016. vision-language pre-training with mixture-of-modality-
experts. arXiv, abs/2111.02358, 2021.
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Rad-
ford, A., Chen, M., and Sutskever, I. Zero-shot text-to- Warstadt, A., Singh, A., and Bowman, S. Corpus of linguis-
image generation. arXiv, abs/2102.12092, 2021. tic acceptability. [Link] 2018.
Schneider, S., Baevski, A., Collobert, R., and Auli, M. Wei, C., Fan, H., Xie, S., Wu, C.-Y., Yuille, A., and Feicht-
wav2vec: Unsupervised pre-training for speech recog- enhofer, C. Masked feature prediction for self-supervised
nition. In Proc. of Interspeech, 2019. visual pre-training. arXiv, abs/2112.09133, 2021.

Sennrich, R., Haddow, B., and Birch, A. Neural machine Williams, A., Nangia, N., and Bowman, S. R. A broad-
translation of rare words with subword units. In Proc. of coverage challenge corpus for sentence understanding
ACL, 2016. through inference. In Proc. of NAACL, 2018.
 data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language

Wu, Z., Wang, S., Gu, J., Khabsa, M., Sun, F., and Ma, H.
CLEAR: contrastive learning for sentence representation.
arXiv, abs/2012.15466, 2020.
Xie, Z., Zhang, Z., Cao, Y., Lin, Y., Bao, J., Yao, Z., Dai, Q.,
and Hu, H. Simmim: A simple framework for masked
image modeling. arXiv, abs/2111.09886, 2021.

Xu, Q., Likhomanenko, T., Kahn, J., Hannun, A., Synnaeve,

G., and Collobert, R. Iterative pseudo-labeling for speech
recognition. Proc. of Interspeech, 2020.
Xue, L., Barua, A., Constant, N., Al-Rfou, R., Narang, S.,
Kale, M., Roberts, A., and Raffel, C. Byt5: Towards a
token-free future with pre-trained byte-to-byte models.
arXiv, abs/2105.13626, 2021.
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R.,
and Le, Q. V. Xlnet: Generalized autoregressive pretrain-
ing for language understanding. arXiv, abs/1906.08237,
2019.

Zhang, Y., Qin, J., Park, D. S., Han, W., Chiu, C.-C., Pang,
R., Le, Q. V., and Wu, Y. Pushing the limits of semi-
supervised learning for automatic speech recognition.
Proc. of NeurIPS SAS Workshop, 2020.

Zhou, J., Wei, C., Wang, H., Shen, W., Xie, C., Yuille, A.,
and Kong, T. ibot: Image bert pre-training with online
tokenizer, 2021.
Zhu, Y., Kiros, R., Zemel, R. S., Salakhutdinov, R., Urta-
sun, R., Torralba, A., and Fidler, S. Aligning books and
movies: Towards story-like visual explanations by watch-
ing movies and reading books. arXiv, abs/1506.06724,
2015.
 data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language

A. Appendix

Table 5. Speech processing: word error rate on the Librispeech dev/test sets when training on the Libri-light low-resource labeled data
setups of 10 min, 1 hour, 10 hours and the clean 100h subset of Librispeech. Models use the audio of Librispeech (LS-960) as unlabeled
data.
Unlabeled dev test
Model LM
data clean other clean other
10 min labeled
wav2vec 2.0 Base (Baevski et al., 2020b) LS-960 4-gram 8.9 15.7 9.1 15.6
Hubert Base (Hsu et al., 2021) LS-960 4-gram 9.1 15.0 9.7 15.3
data2vec Base LS-960 4-gram 7.3 11.6 7.9 12.3
1h labeled
wav2vec 2.0 Base (Baevski et al., 2020b) LS-960 4-gram 5.0 10.8 5.5 11.3
Hubert Base (Hsu et al., 2021) LS-960 4-gram 5.6 10.9 6.1 11.3
data2vec Base LS-960 4-gram 4.0 8.5 4.6 9.1
10h labeled
wav2vec 2.0 Base (Baevski et al., 2020b) LS-960 4-gram 3.8 9.1 4.3 9.5
Hubert Base (Hsu et al., 2021) LS-960 4-gram 3.9 9.0 4.3 9.4
data2vec Base LS-960 4-gram 3.3 7.5 3.9 8.1
100h labeled
Noisy student (Park et al., 2020) LS-860 LSTM 3.9 8.8 4.2 8.6
IPL (Xu et al., 2020) LL-60K 4-gram+Transf. 3.2 6.1 3.7 7.1
SlimIPL (Likhomanenko et al., 2021) LS-860 4-gram+Transf. 2.2 4.6 2.7 5.2
wav2vec 2.0 Base (Baevski et al., 2020b) LS-960 4-gram 2.7 7.9 3.4 8.0
Hubert Base (Hsu et al., 2021) LS-960 4-gram 2.7 7.8 3.4 8.1
data2vec Base LS-960 4-gram 2.2 6.4 2.8 6.8