Bioinformatics, 2023, 39(10), btad617

[Link]
Advance Access Publication Date: 10 October 2023
Original Paper

Sequence analysis
Investigation of the BERT model on nucleotide sequences
with non-standard pre-training and evaluation of different
k-mer embeddings

Downloaded from [Link] by guest on 29 April 2024

1, 1 1,
Yao-zhong Zhang *, Zeheng Bai , Seiya Imoto *
1
Division of Health Medical Intelligence, Human Genome Center, The Institute of Medical Science, The University of Tokyo, Minato-ku,
Tokyo 108-8639, Japan
*Corresponding authors. Division of Health Medical Intelligence, Human Genome Center, The Institute of Medical Science, The University of Tokyo,
Shirokanedai 4-6-1, Minato-ku, Tokyo 108-8639, Japan. E-mails: yaozhong@[Link] (Y.Z.) and imoto@[Link] (S.I.)
Associate Editor: Valentina Boeva

Abstract
Motivation: In recent years, pre-training with the transformer architecture has gained significant attention. While this approach has led to notable
performance improvements across a variety of downstream tasks, the underlying mechanisms by which pre-training models influence these
tasks, particularly in the context of biological data, are not yet fully elucidated.
Results: In this study, focusing on the pre-training on nucleotide sequences, we decompose a pre-training model of Bidirectional Encoder
Representations from Transformers (BERT) into its embedding and encoding modules to analyze what a pre-trained model learns from nucleo-
tide sequences. Through a comparative study of non-standard pre-training at both the data and model levels, we find that a typical BERT model
learns to capture overlapping-consistent k-mer embeddings for its token representation within its embedding module. Interestingly, using the
k-mer embeddings pre-trained on random data can yield similar performance in downstream tasks, when compared with those using the k-mer
embeddings pre-trained on real biological sequences. We further compare the learned k-mer embeddings with other established k-mer represen-
tations in downstream tasks of sequence-based functional prediction. Our experimental results demonstrate that the dense representation
of k-mers learned from pre-training can be used as a viable alternative to one-hot encoding for representing nucleotide sequences. Furthermore,
integrating the pre-trained k-mer embeddings with simpler models can achieve competitive performance in two typical downstream tasks.
Availability and implementation: The source code and associated data can be accessed at [Link]

1 Introduction In this study, we focus on nucleotide sequences and aim to un-

derstand what a pre-trained Bidirectional Encoder Representations
Pre-training with the transformer architecture (Vaswani et al.
2017) has achieved remarkable results across a variety of re- from Transformers (BERT) model (Devlin et al. 2018, Ji
search fields, including natural language processing (Devlin et al. 2021) learns during pre-training and how the acquired
et al. 2018) and computer vision (Dosovitskiy et al. 2020). In knowledge influences performance in downstream tasks.
the realm of computational biology, this methodology has Previous work (Ji et al. 2021) used attention weights to inter-
been used to learn representations for amino acid sequences pret regions that contribute to the predictive outcomes for
(Jumper et al. 2021, Rao et al. 2021, Elnaggar et al., 2022) specific input sequences. However, non-sequence-specific to-
and nucleotide sequences (Ji et al. 2021). Given the scarcity of ken embeddings learned by the pre-trained model have not
labeled data, the pre-training and fine-tuning paradigm lever- been extensively investigated. Here, we concentrate on the
ages abundant unlabeled data to learn general representa- non-sequence-specific lower-level token embeddings to pro-
tions. These representations are subsequently refined via fine- vide a global interpretation (Novakovsky et al. 2023) of the
tuning, utilizing a limited dataset of labeled data specific to pre-trained model. We decompose the BERT model into its
the target downstream task. While this training paradigm has embedding and encoding modules. We employ a non-stan-
resulted in significant performance improvements in a range dard pre-training method by incorporating randomness at
of downstream applications, the underlying mechanisms that both the data and model levels for comparative analysis. In
contribute to the efficacy of pre-trained models remain incom- addition to evaluating pre-trained BERT models, we investi-
pletely understood. Furthermore, current research tends to fo- gate the use of different k-mer embeddings in the sequence-
cus on model architectures that are reminiscent of those used based functional prediction tasks, including TATA promoter
in natural language modeling tasks. The unique characteris- prediction and transcription factor binding site (TFBS) pre-
tics inherent to biological sequences have been less extensively diction. We demonstrate potential applications of a BERT
explored. model pre-trained on randomly generated sequences. These

Received: 21 November 2022; Revised: 8 September 2023; Editorial Decision: 17 September 2023. Accepted: 9 October 2023
C The Author(s) 2023. Published by Oxford University Press.
V
This is an Open Access article distributed under the terms of the Creative Commons Attribution License ([Link] which
permits unrestricted reuse, distribution, and reproduction in any medium, provided the original work is properly cited.
2 Zhang et al.

applications encompass its utility as a dense k-mer token prediction for the contiguous masked k-mers. For instance,
representation and as a model weight initializer to accelerate for two consecutive masked tokens, GATCT and ATCTG,
the pre-training process. predictions maintaining consistent overlap (e.g. GATCT and
ATCTG) are preferable to inconsistent ones (e.g. GATCT
and GTTTG). (ii) Prediction of the non-trivially inferable nu-
2 Materials and methods cleotide within k-mers. For different pre-training datasets,
To understand what a BERT model captures during the pre- such as nucleotide sequences originating from different spe-
training phase, we decomposed a standard BERT model into cies, the first subtask remains data-independent. In contrast,
embedding and encoding modules. The embedding module the second subtask is data-dependent. Previous work (Liang
comprises embeddings that represent tokens, token types, and et al. 2022) on predicting nucleotides from their surrounding
positions. The encoding module employs the encoder compo- nucleotides has demonstrated variable performance across
nent of the transformer model (Vaswani et al. 2017), incorpo- different regions of human chromosomes. Specifically, repeat
rating multi-head attention and feed-forward layers. To regions exhibit higher accuracy, whereas coding regions show

Downloaded from [Link] by guest on 29 April 2024

analyze the pre-training of the BERT model on nucleotide lower accuracy, yet the overall accuracy exceeds 50%. Note
sequences, we utilized DNABERT (Ji et al. 2021). (In the sub- that the encoding module constitutes a significant portion of
sequent sections, the terms “BERT” and “DNABERT” are the model parameters. Taking the 5-mer DNABERT model
used interchangeably, referring to the BERT model applied to for instance, the encoding module accounts for 98.6% of
nucleotide sequences.) the total model parameters (amounting to 85 million). This
During pre-training, the BERT model is trained to predict substantial number of model parameters enables the model
masked tokens. In the context of nucleotide sequences, to make overlapping-consistent k-mer predictions across a
DNABERT uses a strategy of masking k contiguous k-mers, vast label search space during pre-training on nucleotide
to preclude trivial inference of the masked tokens based on sequences.
their adjacent k-mers. This masking approach is exemplified To illustrate the learning outcomes of the BERT model ap-
in a 5-mer illustration shown in Fig. 1, where five contiguous plied to nucleotide sequences, we adopted a non-standard
5-mers are masked during the pre-training phase. However, pre-training approach, introducing randomness at both the
upon decomposing a k-mer to the nucleotide level, it becomes data and model levels for a comprehensive analysis of the
apparent that specific nucleotides within the masked k-mers DNABERT model. At the data level, we generated entirely
may still be deducible from adjacent k-mers. As a conse- random nucleotide sequences, comprising 3 billion bases,
quence, for k contiguous masked k-mers, only a single nucleo- for comparative analysis with the DNABERT pre-trained on
tide cannot be deduced from neighboring k-mers. The the human reference genome. With the exception of the pre-
DNABERT model has a label search space of 4k þ 5 (with the training data difference, we maintained all other pre-training
five additional labels corresponding to special tokens used in settings consistent with DNABERT and trained a DNABERT
BERT, e.g. [CLS], [PAD]). However, the search space for model on the randomly generated nucleotide sequences. The
non-trivially inferable labels is reduced to 4 þ 5. Based on this contrastive analysis allows us to investigate the first subtask,
observation, the pre-training task of DNABERT can be di- given that the random sequences are devoid of biological in-
vided into two subtasks: (i) Overlapping-consistent k-mer formation that can be used to predict context-dependent

INPUT: [CLS] GTCTCGATCTGACCTTGTGA[SEP]

K-mer tokenization Label prediction space
and masking (contiguous k-length k-mers) for 5-mer in one mask position
0 ......... [CLS]
1 .................. GTCTC
Pre-training model
2 ..................... TCTCG prediction space
3 ....................... CTCGA
4 .......................... TCGAT
5 ............................ CGATC
6 [MASK] GATCT Red-colored nucleotides in masked
7 [MASK] ATCTG k-mers are the nucleotides that can
8 [MASK] TCTGA be directly deduced from the
9 [MASK] CTGAC surrounding unmasked k-mers.
10 [MASK] TGACC
11 ............................................. GACCT 4+5

12 ................................................ ACCTT Equivalent non-trivial

13 ................................................... CCTTG predition space
14 ...................................................... CTTGT
15 ......................................................... TTGTG
16 ............................................................ TGTGA
17 ........................................................................... [SEP]

Figure 1. K-mer token masking strategy used in DNABERT. A nucleotide sequence is first converted into an overlapping k-mer sequence, with special
tokens inserted at both ends of the sequence. For pre-training, partial tokens are masked to train the model for their prediction. To preclude the trivial
inference of a masked token from its immediately adjacent k-mers, DNABERT masks k contiguous k-mers. Compared with the prediction space of the
non-trivially inferable nucleotides (those that cannot be directly deduced from adjacent k-mers), the original prediction space is appreciably larger.
Investigation of the BERT model on nucleotide sequences 3

Pre-training Fine-tuning
DNABERT_human DNABERT_encRand DNABERT_dataRand

Human Human Fine-tuning

Data Random
Reference Reference task
sequences
Genome Genome

encoding
encoding (Re-random encoding encoding
Pre-trained Initialization)
BERT model
embedding embedding embedding
embedding

Downloaded from [Link] by guest on 29 April 2024

k-mer Position TokenType k-mer Position TokenType k-mer Position TokenType

Standard fine-tuning

Other model
architectures
(e.g. CNN)

Standard pre-training Non-standard pre-training Only using k-mer embeddings

Figure 2. Methodology for decomposition and analysis of the DNABERT model. The model is decomposed into embedding and encoding modules. To
investigate the DNABERT model, we conducted non-standard pre-training that incorporates randomness at both the data and model levels. At the data
level, we generated nucleotide sequences randomly, comprising 3 billion bases, for comparison with the DNABERT pre-trained on the human reference
genome. At the model level, we incorporated randomness into the encoding module through re-initializing the corresponding model weights. In the fine-
tuning stage, in addition to the standard fine-tuning, we also assessed simpler neural network structures that solely utilize the learned k-mer embeddings.

nucleotides. At the model level, we introduced randomness In the pre-training task on nucleotide sequences, all token
into the encoding module of DNABERT, as illustrated in type embeddings were set to be the same. For the pre-trained
Fig. 2. We re-initialized the learned model weights in the model on the human reference genome, we directly utilized
encoding module while retaining the embeddings from the the models provided by DNABERT (accessible at https://
pre-trained model. We further examined the “ablated” pre- [Link]/jerryji1993/DNABERT, abbreviated as
trained models fine-tuned for downstream tasks, comparing DNABERT_human). These models were trained on the hu-
them with other pre-trained models using the human refer- man reference genome (GRCh38). To perform the t-SNE
ence genome and random sequences. These comparative analysis, we extracted the weights of k-mer token embeddings
assessments assist to analyze the learning outcomes of the pre- from these provided models. With respect to the model pre-
trained DNABERT model on nucleotide sequences. trained on randomly generated sequences (referred to as
Furthermore, we extended our evaluation to include simpler DNABERT_dataRand), we employed DNABERT with its de-
model structures previously recognized as state-of-the-art. fault model hyperparameters for pre-training. We extracted
Specifically, we assessed DeePromoter (Oubounyt et al. 2019) the k-mer embeddings learned with randomly generated
and Convolutional Neural Network (CNN) (Zeng et al. sequences from the trained model. To compare the learned k-
mer embeddings with other widely used k-mer representa-
2016), integrating them with the pre-trained k-mer embed-
tions, we also visualized k-mer embeddings learned by dna2-
dings. In addition to the k-mer embeddings learned by
vec as well as one-hot encoding. Notably, neither of these two
DNABERT, we compared these models with two other com-
embeddings includes special tokens.
monly used k-mer representations of dna2vec (Ng 2017) and
Figure 3a and b illustrate the 5-mer embeddings learned by
one-hot encoding.
the DNABERT model with different pre-training datasets.
From the figures, we have the following two observations.
3 Results First, the special tokens appear distinctly separated from the
3.1 Overlapping-consistent k-mer embeddings are other k-mer tokens within the visualized space. Second,
among the k-mer tokens, there is a noticeable trend that
learned by DNABERT for its token representation
k-mers sharing the same prefix or suffix nucleotide strings of-
To examine k-mer embeddings learned by DNABERT, we ten cluster in close proximity. This is evident with k-mers like
performed t-distributed Stochastic Neighbor Embedding CTCCN, NGCTT, and NGAAT (“N” represents any nucleo-
(t-SNE) (Hinton and Roweis 2002) analysis using Scikit-learn tide from A, T, C, G) as depicted in Fig. 3a and b. Such
(version 1.1.1) (Pedregosa et al. 2011). Given that a k-mer local clustering based on shared prefix and suffix strings sub-
can be decomposed into nucleotides, we investigated k-mer sequently contributes to the formation of larger aggregated
embeddings in the context of their nucleotide composition. clusters centered around a common nucleotide sequence, such
Our analysis focused on k-mer token embeddings and ignored as -GAT-. For visualization purposes, we color-coded each
other embeddings, such as token type and position embed- k-mer based on the nucleotide occupying its central position
dings. Token types, also known as segment IDs, are utilized (--[A/T/G/C]--). This color-coding scheme facilitates the ease
to differentiate between two types of sequences in the input. of visualization and analysis of the k-mer embeddings. In
4 Zhang et al.

(a) (b)

Downloaded from [Link] by guest on 29 April 2024

(c) (d)

Figure 3. T-SNE plot of 5-mer embeddings generated by different methods (DNABERT_human, DNABERT_dataRand, dna2vec, and one-hot encoding).
DNABERT_human refers to the DNABERT model pre-trained on the human reference genome. DNABERT_dataRand is the DNABERT model pre-trained
on randomly generated sequences. Dna2vec is a k-mer embedding model based on word2vec trained on the human reference genome. One-hot
encoding is a commonly used method for representing categorical data, such as nucleotides. (a) 5-mer embeddings learned by DNABERT pre-trained on
human reference genome. (b) 5-mer embeddings learned by DNABERT pre-trained on randomly generated sequences. (c) 5-mer embeddings of dna2vec.
(d) 5-mer embeddings of one-hot encoding.

both Fig. 3a and b, four distinct clusters are readily observ- consistent k-mer representation”. This pattern is evident in
able, each corresponding to a nucleotide occupying the central both the k-mer embeddings from models pre-trained on both
position in a k-mer. Although randomly generated sequences real genomic sequences and random nucleotide sequences. The
lack inherent biological information, the contextual con- overlapping-consistent k-mer representation may be regarded
straints of neighboring overlapping k-mers are retained in the as an inherent feature that DNABERT captures for the token
random data. Compared with the DNABERT trained on the representation during pre-training. Its species-independent
human genome, the k-mer embeddings from DNABERT pre- character can be associated with the model’s generalization
trained on randomly generated sequences are lined up more ability. For comparative analysis, we also visualized the two
obviously. Here, we refer to the proximity of k-mer embed- commonly used k-mer embeddings of dna2vec and one-hot
dings with similar prefixes or suffixes as “overlapping- encoding, as shown in Fig. 3c and d. In the dna2vec
Investigation of the BERT model on nucleotide sequences 5

embedding, while not as pronounced as in Fig. 3a and b, the turning to performance metrics like AUROC and AUPRC,
affinity of k-mers with identical prefixes or suffixes is still ob- DNABERT_human, DNABERT_dataRand, and DNABERT_
servable, exemplified by k-mers like NGAGC and GATGN in encRand exhibit comparable levels of performance. Among the
Fig. 3c. No larger clusters centered around the middle posi- four models assessed, DNABERT_allRand shows the lowest
tion are discernible in the t-SNE plot generated using dna2vec. performance on both datasets. As illustrated in the figure,
As for the one-hot encoding, shown in Fig. 3d, the k-mers ap- DNABERT_dataRand and DNABERT_encRand exhibit
pear to be uniformly distributed. similar levels of performance. DNABERT_encRand retains
We then evaluated DNABERT pre-trained models and the pre-trained weights of its embedding module from
assessed the efficacy of different k-mer embeddings in two DNABERT_human but diverges in the weights associated with
downstream tasks of sequence-based functional prediction: the encoding module. The observed performance disparity be-
TATA promoter prediction and TFBS prediction. We first tween DNABERT_encRand and DNABERT_human indicates
evaluated the pre-trained DNABERT models in the context of the impact of encoding module, which takes the role of integrat-
the TATA promoter prediction task. Following the previous ing information from lower-level encoding and embedding

Downloaded from [Link] by guest on 29 April 2024

work (Oubounyt et al. 2019, Ji et al. 2021), we obtained the layers. DNABERT_dataRand was pre-trained on randomly
human and mouse positive promoter sequences from the generated data that lack inherent biological information. As a
EPDnew database (Dreos et al., 2017) . These positive sequen- result, its encoding module cannot learn meaningful sequence
ces are from regions ranging from 249 to 50 bp surrounding patterns from such random data. On the other hand, the encod-
the transcription start sites. The dataset comprises 29 598 ing module of DNABERT_encRand was randomly re-
positive human samples (TATA: 3065; non-TATA: 26 533) initialized. Regarding the functionality of identifying meaning-
and 25 110 positive mouse samples (TATA: 3305; non- ful sequence patterns, the encoding modules of
TATA: 21 805) with a length of 300 bp. Negative sequences DNABERT_dataRand and DNABERT_encRand perform simi-
were generated based on the characteristics of the positive larly. Besides that, the difference in the embedding module be-
samples. For samples containing the TATA motif, negative tween DNABERT_dataRand and DNABERT_encRand does
samples were selected from other genomic regions containing not lead to a significant performance difference in this down-
the TATA motif to preclude overly simplistic classifications stream task.
based solely on the presence of the TATA motif. For the non-
TATA sequences, the positive samples were generated by ran- 3.2 Investigation of 1-mer DNABERT models
dom substitution of subsequences. The complete dataset was In addition to evaluating 5-mer DNABERT models, we
divided into training, development, and test sets at an assessed 1-mer DNABERT models pre-trained on both the
80:10:10 ratio. Different pre-trained DNABERT models were human and randomly generated datasets. Distinct from the
fine-tuned utilizing the training and development sets, and 5-mer models, the 1-mer models are not influenced by over-
subsequently evaluated on the test set. lapping k-mer tokens. This feature allows the 1-mer model
The average performance of different DNABERT models, as to focus on the second subtask of predicting masked nucleo-
determined through ten independent runs, is shown in Fig. 4. tides using contextual information from adjacent nucleoti-
Besides DNABERT_human and DNABERT_dataRand, we des. The 1-mer model failed to be pre-trained on the random
compared with models where DNABERT_human had its data, but it converged successfully when pre-trained on the
encoding module or all modules randomly re-initialized. These human data. There are significant differences in model
models are named DNABERT_encRand and DNABERT_ weights and predicted nucleotide probabilities between mod-
allRand, respectively. Detailed numerical results are available els pre-trained on human data versus random data. We then
in Supplementary Table S1. When evaluated using single- evaluated the accuracy of masked tokens predicted by
threshold metrics (0.5) such as accuracy, F1-score (F1), and DNABERT_human using 100 000 samples randomly gener-
Matthews Correlation Coefficient (MCC), DNABERT_human ated from human genome (15% masking rate) and acquired
outperforms both DNABERT_dataRand and DNABERT_ an accuracy of 0.5484. This result is consistent with a prior
encRand on the human and mouse datasets. However, when study (Liang et al. 2022), which indicates that the base

(a) (b)

Figure 4. Performance comparison of different pre-trained DNABERT models fine-tuned on the TATA human (a) and mouse dataset (b).
DNABERT_human is the provided model pre-trained on the human reference genome. DNABERT_dataRand is pre-trained on randomly generated
sequences. DNABERT_encRand is based on DNABERT_human while the encoding module is randomly re-initialized. DNABERT_allRand is fine-tuning
from scratch with all parameters randomly initialized. We carried out ten independent runs using different random seeds and reported average scores
accompanied by their respective standard deviations.
6 Zhang et al.

prediction accuracy, given the surrounding context of 14 dna2vec model supports variant k-mer lengths, with all the k-
nucleotides, averages above 50%. These results from the 1- mer embeddings sharing a consistent dimension of 100.
mer models provide additional support to the observation Although the dna2vec k-mer dimension is smaller than that of
related to the DNABERT model pre-trained with overlap- DNABERT (768), its performance in terms of AUROC,
ping k-mer tokens and the model learns the overlapping- AUPRC, and MCC slightly lags behind models using
consistent k-mer embeddings for its token representation. DNABERT k-mer embeddings. For the DeePromoter model
More detailed information on 1-mer models can be found in using k-mer embeddings derived from DNABERT, despite
Section 5 of the Supplementary Material. having significantly fewer model parameters than the full
DNABERT model, the performances (both with human 5-
3.3 Evaluation of k-mer embeddings with a simpler mers and random 5-mers) are comparable to those achieved by
model in the TATA promoter prediction task DNABERT_dataRand and DNABERT_encRand on the hu-
Given the large number of model parameters within DNABERT, man dataset (marginally lower on the mouse dataset).
with encoding module accounting for 98.6% of the overall Interestingly, when employing k-mer embeddings of

Downloaded from [Link] by guest on 29 April 2024

model parameters, we then explored the utilization of models DNABERT pre-trained on random sequences, the
with simpler encoding layers for downstream tasks. K-mer DeePromoter model has an improved MCC result compared to
embeddings were investigated with a simplified model. We the model using k-mer embeddings of DNABERT pre-trained
assessed four distinct k-mer embeddings and adopted the on human data. This improvement is particularly notable in
DeePromoter framework (Oubounyt et al. 2019) as the foun- the mouse dataset. In conjunction with the observations from
dational architecture for the TATA promoter prediction task. the t-SNE plot shown in Fig. 3, these results suggest that the
Notably, compared with DNABERT, DeePromoter has a sig- overlapping-consistent k-mer embeddings learned by
nificantly reduced number of model parameters, as illustrated DNABERT stand out as a learning outcome for the token rep-
in Fig. 5c (or detailed in Supplementary Table S2). Despite its resentation within the embedding module. Additionally, the k-
fewer parameters, DeePromoter has been reported to achieve mer embeddings learned by DNABERT provide a potent alter-
a commendable performance. The k-mer embeddings under native to dna2vec and one-hot embeddings for encoding nucle-
evaluation included one-hot encoding (ranging from 1-mer to otide sequences in deep learning frameworks.
5-mer), dna2vec embeddings (for 4-mer and 5-mer), and 5-mer
embeddings extracted from DNABERT, utilizing different pre- 3.4 Evaluation on TFBS prediction tasks
training datasets (human genome and random data). Unlike We extended the evaluation of related models on TFBS predic-
the default fine-tuning procedure of DNABERT, the weights of tion tasks, including both motif discovery and motif occupancy
the embedding layer are no longer updated within the identification (Zeng et al. 2016). Different from motif discovery
DeePromoter framework. The position and token type embed- that utilizes negative samples crafted from shuffled positive
dings were not used in the simplified model. Our approach in- sequences preserving identical dinucleotide frequency, motif oc-
volved a comprehensive hyperparameter search, conducted on cupancy identification uses negative samples that are also from
the human development set. regions centering at a motif instance with matched size, GC,
As illustrated in Fig. 5a and b, DeePromoter using k-mer and motif strength. We evaluated a total of 690 motif discovery
embeddings from DNABERT has a better performance than datasets and 422 motif occupancy datasets. These datasets con-
those using the other two embedding methods of one-hot and tain input nucleotide sequences of 101 bp, which is shorter than
dna2vec. With the one-hot encoding, the number of k-mers the 300 bp input sequences used in the TATA promoter predic-
increases exponentially as k grows, resulting in the challenge tion task. Each dataset was randomly partitioned into 80% for
of sparse high-dimensionality. The AUROC of the one-hot training and 20% for testing. Within each training dataset, a
encoding model peaks at k ¼ 3 and subsequently declines as k subset of 12.5% was randomly selected to use as a development
increases. The best performance of the one-hot k-mer model set. We utilized the CNN model (Zeng et al. 2016) as a simpli-
remains inferior to that of the models using dna2vec and fied model for evaluating different k-mer embeddings. For com-
DNABERT k-mer embeddings. Dna2vec employs the word2- parative analysis, we also evaluated fine-tuned DNABERT
vec word embedding model, which is trained on a two-layer models. We conducted model hyperparameter tuning using the
neural network, to learn k-mer embeddings. The provided default hyperparameters previously established for the CNN

(a) (b) (c)

Figure 5. Performance of the DeePromoter model using different k-mer embeddings on the TATA human dataset (a) and mouse dataset (b). (c) The
number of model parameters in DeePromoter using different k-mer embeddings.
Investigation of the BERT model on nucleotide sequences 7

Downloaded from [Link] by guest on 29 April 2024

Figure 6. Boxplot of AUROC performance on 690 TFBS motif discovery datasets.

(Zeng et al. 2016) and DNABERT model as initial settings. DNABERT’s pre-trained k-mer embeddings are also very close,
Hyperparameter tuning for DNABERT was further optimized which are 0.8991 and 0.8965 for k-mer embeddings derived from
utilizing the development set of 20 randomly selected motif oc- the model pre-trained on human data and random sequence data,
cupancy datasets. For each training set, models were trained for respectively. The performance of fine-tuned DNABERT models is
ten epochs and the model with the highest MCC score on the shown in the last four columns of Fig. 6. DNABERT_human_ft
development set was saved. These stored models were used for achieves the highest median AUROC score of 0.9. Both
predictions on the corresponding test sets. DNABERT_encRand_ft and DNABERT_dataRand_ft exhibit
Figure 6 shows AUROC scores for CNN models using dif- comparable performance levels, with scores of 0.8652 and
ferent k-mer embeddings and fine-tuned DNABERT models 0.8624, respectively. These scores are lower than those
evaluated across 690 motif discovery datasets. We mainly fo- achieved by the CNN models utilizing DNABERT’s k-mer
cused on the median AUROC score of the 690 datasets as an embeddings. DNABERT_allRand_ft without pre-training has
overall assessment metric for different models. Additional
the lowest median AUROC scores among all the compared
metrics, including MCC, AUPRC, and F1, can be found in
models. These performance disparities among different pre-
Section 4.1 of the Supplementary Material. For one-hot
trained DNABERT models also demonstrate the impact of the
encoding, we evaluated k-mers of lengths 1, 4, and 5. The in-
encoding module in the downstream fine-tuning. Given the
corporation of one-hot encoding for 4-mer and 5-mer serves
considerable variation in sample sizes across the 690 datasets,
as a reference for comparative analysis against other learned
k-mer embeddings, aligning their vocabulary sizes within a ranging from 139 to 143 425 (with a median of 21 929.5 for
comparable scale range. The one-hot encoding of 1-mer yields training data), the intricate encoding module with its large
a median AUROC score of 0.8551, in line with the previously number of parameters poses challenges when relying solely on
reported result (Zeng et al. 2016). As the value of k increases the fine-tuning dataset for training. This is particularly pro-
to 4 and 5, the dimension of the input vector expands propor- nounced in the context of small-scale datasets. As a conse-
tionally to 256 and 1024, resulting in improved median quence, the necessity for adequate pre-training becomes
AUROC scores of 0.8929 and 0.8919 for 4-mer and 5-mer, re- apparent when employing a complex encoding module. In
spectively. Comparatively, the model using dna2vec embeddings such a scenario, the pre-training process plays a crucial role in
performs similarly, achieving a median AUROC score of 0.8918, enabling the successful training of intricate models. On the
despite the dimensionality of the k-mer vector being restricted to other hand, our observations indicate that the performance
100. The median AUROC scores of the CNN model using disparity between CNN models utilizing k-mer embeddings
8 Zhang et al.

Downloaded from [Link] by guest on 29 April 2024

Figure 7. Boxplot of AUROC performances on 422 TFBS motif occupancy identification datasets.

and the DNABERT fine-tuning models is not notably signifi- using a model pre-trained on human data. This consideration
cant for this particular task. becomes pertinent when accounting for the role of encoding
Similarly, we conducted evaluations on 422 TFBS motif oc- module. When there is a need to pre-train a new model,
cupancy identification datasets, as illustrated in Fig. 7. Given leveraging initial model weights from a model pre-trained on
that the motif occupancy identification is more challenging random data can accelerate the pre-training process. This
than the motif discovery task (Zeng et al. 2016), the median becomes particularly beneficial once overlapping-consistent
AUROC scores are generally lower, ranging from 0.78 to k-mer embeddings have been learned.
0.8465, compared to the scores observed in the motif discov- Figure 8 illustrates the exponential loss curve on the devel-
ery task. Models employing the same type of k-mer embed- opment set during pre-training with and without model ini-
dings exhibit a performance pattern that closely resembles tialization on the human reference genome. With the model
what was observed in the motif discovery task. One different initialization, the pre-training process approaches conver-
and interesting result is that the CNN model using 5-mer one- gence after 1500 steps. In contrast, pre-training without the
hot embedding achieves the highest median AUROC score of model initialization begins to show signs of convergence after
0.8465, which is even slightly higher than the DNABERT_ about 5500 steps. Given that pre-training on random se-
human_ft of 0.8398. Although it is difficult to perform strict quence data is non-species-specific, its utilization introduces
comparisons between the two, this result indicates that a sim- no additional bias into the newly pre-trained model.
pler neural network structure could be more suitable for tasks
involving relatively small-scale datasets.
4 Discussion
3.5 Acceleration of pre-training through model In this work, we conducted a comprehensive analysis of the
initializing using weights pre-trained on random data BERT model pre-trained on nucleotide sequences. Different
Pre-training a BERT model is widely recognized as a process from conventional analyses, we proposed to use a non-stan-
that is computationally intensive and time-consuming. dard pre-training approach that incorporated randomness at
Although a pre-trained model can be applied for cross-species both the data and model levels. We subsequently contrasted
applications, its efficacy may be confined to species with close these variant pre-trained models (or modules) with their stan-
phylogenetic relationships. For example, if a downstream ap- dard pre-trained counterparts, drawing a parallel to the
plication involves bacterial genomes, leveraging a model pre- “knockout” versus “wild type” comparisons commonly made
trained on bacterial data may offer advantages over directly in wet-lab experiments. Our experiments and analyses
Investigation of the BERT model on nucleotide sequences 9

Downloaded from [Link] by guest on 29 April 2024

Figure 8. Exponential loss curve on the development set of the pre-training with and without model initialization. The development set is randomly
sampled 20% of the total training dataset of the human reference genome. Applying model initialization using the weights of the model pre-trained on
random data demonstrates efficacy in reducing the number of steps needed for convergence.

revealed that when pre-training on the task of predicting con- parameters are difficult to be trained only using a limited fine-
tiguous masked overlapping k-mers, one key feature learned tuning dataset. In this study, we introduced two additional
for the token representation is the overlapping-consistent approaches to illustrate the impact of pre-training. Besides intro-
k-mer embeddings. These embeddings effectively capture the ducing randomness into the encoding module, we also conducted
similarity structure of overlapping k-mers and can be indepen- pre-training on totally random sequences. Compared with intro-
dently leveraged for various downstream tasks. To further re- ducing randomness into the encoding module, using the pre-
fine the pre-training task for nucleotide sequences, exploring trained model on random sequences achieves relatively close
tokenization methods informed by biological insights and de- results in the TATA promoter prediction and TFBS prediction
signing pre-training tasks of heightened biological significance tasks. The pre-trained model on randomized data can provide a
would be beneficial. pivot to investigate biologically insignificant perturbations of in-
For protein sequences, previous work (Yang et al. 2018) has put nucleotide sequences.
demonstrated that randomizing the UniPort sequences improves
downstream performance for some tasks, such as cytochrome
P450 thermostability. This improvement was hypothesized to 5 Conclusion
be related to a regularizing effect on the embedding model. In this study, we focused on nucleotide sequences and analyzed
Specifically, the randomization deters the embedding model the learning outcomes of a typical BERT model learned
from overfitting to a particular subset of protein sequences in through pre-training. We applied a non-standard pre-training
the pre-training dataset. In this study, we used randomization methodology to scrutinize different modules by incorporating
from a different motivation for analyzing the DNABERT model randomness at both the data and model levels. We demon-
for nucleotide sequences. Another notable difference between strated that the overlapping-consistent embedding of k-mers is
the two works is the unit of biological sequence representation. the outcome that the DNABERT model learns for its token rep-
DNABERT generally uses overlapping k-mers for representing resentation. We assessed the efficacy of utilizing different pre-
nucleotide sequences, whereas protein sequences utilize non- trained k-mer embeddings and compared them with other com-
overlapping amino acids as their foundational unit. Our assess- monly used k-mer representations in downstream tasks related
ment across two downstream tasks showed that k-mer embed- to sequence-based functional prediction. Additionally, we dem-
dings learned from randomized nucleotide sequences can also onstrated that utilizing a model pre-trained on random sequen-
be helpful in downstream tasks in the pre-training and fine- ces can accelerate the pre-training process.
tuning framework. These embeddings learned from random
data can provide close performance comparable to the overlap-
ping k-mer embeddings learned on real genomic data. Acknowledgements
Furthermore, k-mer embeddings trained on the random data The computing resources were provided by Human Genome
can provide an unbiased option of overlapping k-mer represen- Center, the Institute of Medical Science, the University of
tation for downstream tasks. Tokyo. The authors thank the anonymous reviewers for their
One commonly used approach to highlight the impact of pre- valuable comments.
training is the practice of training models from scratch. This
method skips the pre-training phase, moving directly to the fine-
tuning process. Typically, such scratch-training results in a lower Supplementary data
performance (Ji et al. 2021), as models with a large number of Supplementary data are available at Bioinformatics online.
10 Zhang et al.

Conflict of interest Hinton GE, Roweis S. Stochastic neighbor embedding. In: Advances in
Neural Information Processing Systems. p. 15. MA, United States:
None declared. MIT Press. 2002.
Ji Y, Zhou Z, Liu H et al. DNABERT: pre-trained bidirectional encoder
representations from transformers model for DNA-language in ge-
Funding nome. Bioinformatics 2021;37:2112–20.
Jumper J, Evans R, Pritzel A et al. Highly accurate protein structure pre-
This work was supported by Grant-in-Aid for Scientific
diction with alphafold. Nature 2021;596:583–9.
Research [JSPS KAKENHI grant number JP21K12104 to Liang Y, Grønbæk C, Fariselli P et al. Context dependency of nucleotide
Y.Z., grant number 21H03538 to S.I.] from Japan Society for probabilities and variants in human DNA. BMC Genomics 2022;23:
the Promotion of Science and the Uehara Memorial 1–15.
Foundation (to S.I.). Ng P. dna2vec: consistent vector representations of variable-length
k-mers. arXiv, arXiv:1701.06279, 2017, preprint: not peer reviewed.
Novakovsky G, Dexter N, Libbrecht MW et al. Obtaining genetics

Downloaded from [Link] by guest on 29 April 2024

References insights from deep learning via explainable artificial intelligence. Nat
Rev Genet 2023;24:125–37.
Devlin J, Chang M-W, Lee K et al. BERT: pre-training of deep bidirec-
Oubounyt M, Louadi Z, Tayara H et al. DeePromoter: robust promoter
tional transformers for language understanding. In: Proceedings of predictor using deep learning. Front Genet 2019;10:286.
2019 Conference of the North American Chapter of the Association Pedregosa F, Varoquaux G, Gramfort A et al. Scikit-learn: machine
for Computational Linguistics: Human Language Technologies, vol learning in python. J Mach Learn Res 2011;12:2825–30.
1, pp. 4171–4186, 2019. Rao RM, Liu J, Verkuil R et al. MSA transformer. In: Proceedings of the
Dosovitskiy A, Beyer L, Kolesnikov A et al. An image is worth 16x16 38th International Conference on Machine Learning, Cambridge
words: transformers for image recognition at scale. In: 9th MA, United States: PMLR pp. 139:8844-8856, 2021.
International Conference on Learning Representations, ICLR 2021. Vaswani A, Shazeer N, Parmar N et al. Attention is all you need. In: Advances
Dreos R, Ambrosini G, Groux R et al. The eukaryotic promoter data- in Neural Information Processing Systems. p. 30. NY, United States:
base in its 30th year: focus on non-vertebrate organisms. Nucleic Curran Associates Inc., Red Hook. 2017.
Acids Res 2017;45:D51–5. Yang KK, Wu Z, Bedbrook CN et al. Learned protein embeddings for
Elnaggar A, Heinzinger M, Dallago C et al. ProtTrans: Toward machine learning. Bioinformatics 2018;34:2642–8.
understanding the language of life through self-supervised Zeng H, Edwards MD, Liu G et al. Convolutional neural network archi-
learning. IEEE Trans Pattern Anal Mach Intell 2022;44: tectures for predicting DNA–protein binding. Bioinformatics 2016;
7112–27. 32:i121–7.