Bioinformatics, 36(4), 2020, 1037–1043

doi: 10.1093/bioinformatics/btz694
Advance Access Publication Date: 6 September 2019
Original Paper

Sequence analysis
Identifying enhancer–promoter interactions with
neural network based on pre-trained DNA
vectors and attention mechanism

Downloaded from [Link] by guest on 05 March 2025

Zengyan Hong1, Xiangxiang Zeng2,*, Leyi Wei 3,
* and Xiangrong Liu 1,
*
1
Department of Computer Science, Xiamen University, Xiamen 361005, China, 2School of Information Science and Engineering, Hunan
University, Changsha 410076, China and 3School of Software, Shandong University, Jinan 250000, China
*To whom correspondence should be addressed.
Associate Editor: John Hancock
Received on July 1, 2019; revised on August 13, 2019; editorial decision on September 1, 2019; accepted on September 4, 2019

Abstract
Motivation: Identification of enhancer–promoter interactions (EPIs) is of great significance to human develop-
ment. However, experimental methods to identify EPIs cost too much in terms of time, manpower and
money. Therefore, more and more research efforts are focused on developing computational methods to solve
this problem. Unfortunately, most existing computational methods require a variety of genomic data, which are
not always available, especially for a new cell line. Therefore, it limits the large-scale practical application of
methods. As an alternative, computational methods using sequences only have great genome-scale application
prospects.
Results: In this article, we propose a new deep learning method, namely EPIVAN, that enables predicting long-range
EPIs using only genomic sequences. To explore the key sequential characteristics, we first use pre-trained DNA vec-
tors to encode enhancers and promoters; afterwards, we use one-dimensional convolution and gated recurrent unit
to extract local and global features; lastly, attention mechanism is used to boost the contribution of key features, fur-
ther improving the performance of EPIVAN. Benchmarking comparisons on six cell lines show that EPIVAN performs
better than state-of-the-art predictors. Moreover, we build a general model, which has transfer ability and can be
used to predict EPIs in various cell lines.
Availability and implementation: The source code and data are available at: [Link]
Contact: xzeng@[Link] or weileyi@[Link] or xrliu@[Link]

1 Introduction chromosome conformation capture (Hi-C) (Rao et al., 2014) and

chromatin interaction analysis using paired end tag sequencing
A promoter is a DNA sequence that determines the position of a (ChIA-PET) (Nastaran et al., 2014), a large amount of data has
transcriptional start point and frequency of transcription (Boris been provided, which makes possible the study of EPIs in a compu-
et al., 2012). An enhancer is a50–1500-base pair (bp) DNA se- tational way.
quence that increases the transcriptional activity of specific genes, In recent years, many machine learning-based computational
such as promoters (Pennacchio et al., 2013). Enhancer–promoter methods have been proposed and proven to be effective for fast and
interactions (EPIs) are critical to the regulation of gene expression accurately identifying EPIs on a genomic scale. These methods can
and are closely related to the occurrence of human diseases be roughly divided into two categories: (i) genomic data based, and
(Williamson et al., 2011). Therefore, studying the EPI contributes (ii) sequence based. The first category of methods identifies EPIs by
greatly to human development. However, the mode of EPIs is com- training classifiers based on the features derived from genomic data.
plex: one enhancer can act on one or more target promoters, where- For example, Whalen et al. (2016) proposed TargetFinder, an EPIs
as one or more enhancers can also co-regulate one target promoter. prediction model trained with multiple genomic peak data, like
Enhancers and promoters have no fixed relationship in position DNase-seq, DNA methylation, transcription factor ChIP-seq, his-
(Arensbergen et al., 2014). Therefore, it requires considerable man- tone marks, CAGE and gene expression data, as model features.
power, time and money to study EPIs by biological experimental However, this kind of method is limited in that it requires relevant
methods. With the development of high-throughput sequencing knowledge about how to select genomic features. The other category
technology and biotechnologies, such as high-throughput of methods uses only information from genomic sequences to predict

C The Author(s) 2019. Published by Oxford University Press. All rights reserved. For permissions, please e-mail: [Link]@[Link]
V 1037
1038 [Link] et al.

EPIs. For instance, Yang et al. (2017) developed a predictive algo- Table 1. Summary of the original dataset in six cell lines
rithm using word embedding to generate features directly from gen-
omic sequences and trained a boosted tree ensemble model. They Cell lines Positive samples Negative samples
demonstrated that their model can capture the sequential characteris-
GM12878 2113 42 200
tics between true EPIs and non-EPIs. Interestingly, they also showed
that their sequential features are complementary to some genomic fea- HUVEC 1524 30 400
tures, contributing to the improvement in performance. Moreover, HeLa-S3 1740 34 800
Mao et al. (2017) proposed an attention-based neural network model, IMR90 1254 25 000
namely EPIANN, to identify EPIs. To further improve performance, K562 1977 39 500
EPIANN incorporates an attention mechanism and location-based NHEK 1291 25 600
feature decoding. Recently, Singh et al. (2019) proposed a deep learn-
ing-based predictive model, SPEID, combining the convolutional
neural network (CNN) with long short-term memory (LSTM). By
simplifying the SPEID model, Zhuang et al. (2019) constructed a pre-
dictive model that uses CNN coupled with transfer learning to train

Downloaded from [Link] by guest on 05 March 2025

their predictive model, which is denoted as SIMCNN for convenience
of discussion. Overall, these studies demonstrate that genomic
sequences themselves contain key information about whether
enhancers interact with promoters on a long-range scale in genome.
Using deep learning or natural language processing (NLP) techniques,
it is quite possible to explore the hidden information between
enhancers and their long-range target promoters. Although consider-
able progress has been done, some aspects still remain challenging for
existing studies. First, EPIs have cell line specificity and the interac-
tions are usually different in different cell lines (Butler and Kadonaga,
2001). Most existing methods can predict cell-line-specific EPIs, but
fail to accurately identify general EPIs simultaneously in various cell
lines. Second, using one-hot or word embedding for sequence encod-
ing has some limitations. For example, the word vectors trained by
small corpus contain insufficient information. Pre-training methods
have been very successful in image processing and natural language
processing, for example, pre-trained word vectors contain richer and
more accurate information, which helps models save training time
and computational resources and improve performance, especially for
large-scale prediction. Therefore, how to combine biological sequence
analysis with pre-training techniques is a challenge.
To address the above limitations, we propose EPIVAN
(Promoter–Enhancer Interaction predictor with pre-trained Vector
and Attention-based neural Networks), a new deep learning model
Fig. 1. The prediction pipeline of EPIVAN. It involves four steps: sequence embed-
for predicting EPIs based on genomic sequences only. The three con- ding, feature extraction, attention mechanism and prediction. In sequence embed-
tributions of this work are as follows: (i) we use DNA vectors pre- ding, the query sequences are encoded by pre-trained DNA vectors; Afterwards, the
trained with human whole genome sequences to encode the resulting DNA vectors of enhancers and promoters are fed into one-dimensional
enhancers and promoters. (ii) We use the attention mechanism to convolution and gated recurrent unit to extract local and global features. Then, in
boost the contribution of key features, thus improving the perform- attention mechanism, the contribution of key features to our model are enhanced.
ance of the model. (iii) We build a general model, which has transfer Finally, the result feature vector is fed into a sigmoid unit to predict EPIs
ability and can be used to predict EPIs in various cell lines.
Benchmarking comparison results on six cell lines show that line is 1:20. The details of datasets for each cell line is shown in
EPIVAN performs better than the existing models. We expect that Table 1.
our method is useful for biological researchers and complementary
to existing methods for computationally identifying EPIs.
2.2 Prediction framework of the proposed EPIVAN
In this work, we combined sequence analysis with natural language
processing to propose a new deep learning model, called EPIVAN,
which is capable of predicting EPIs based on enhancers and pro-
2 Materials and methods moter sequences only. The prediction procedure of EPIVAN is
shown in Figure 1. As we can see, it has three main steps: sequence
2.1 Data embedding, feature extraction and attention mechanism.
In this study, we used the same EPI dataset from TargetFinder as the Afterwards, the generated feature vector was fed to the final layer, a
original dataset for evaluating and comparing our model with existing sigmoid unit, to predict EPIs. The detail regarding how the EPIVAN
methods. The dataset includes EPIs in six human cell lines: GM12878 works step by step is described in the following subsections.
(lymphoblastoid cells), HUVEC (umbilical vein endothelial cells),
HeLa-S3 (ectoderm-lineage cells from a patient with cervical cancer),
IMR90 (fetal lung fibroblasts), K562 (mesoderm-lineage cells from a 2.2.1 Sequences embedding based on pre-trained DNA vectors
patient with leukemia) and NHEK (epidermal keratinocytes). It is Enhancer and promoter sequences are genomic sequences consisting
worth noting that TargetFinder used ENCODE and Roadmap of four bases: ‘A’ (adenine), ‘G’ (guanine), ‘C’ (cytosine), and ‘T’
Epigenomics annotations to identify active enhancers and promoters (thymine). Undetermined bases are annotated as ‘N’. SPEID,
in each cell. Further, based on high-resolution genome-wide Hi-C EPIANN and SIMCNN (both removed ‘N’ from the sequence)
data, they annotated each enhancer–promoter pair as interaction encoded each base into a four-dimensional one-hot vector, which is
(used as positive sample) or non-interaction (used as negative sample). easy to process by the computer. However, the limitations of using
They sampled 20 negative samples for each positive sample (positive one-hot encoding are that data storage consumes a lot of space; on
and negative samples have similar enhancer–promoter distance distri- the other hand, the models cannot capture the hidden association in-
bution), so that the ratio of positive to negative samples in each cell formation in the sequences, as the information between each vector
Predicting EPIs with EPIVAN 1039

is independent. PEP-WORD uses word embedding to generate cell- sequences, it requires much time to learn. Although the CNN model
specific representations for the DNA sequences of the enhancers and obtains interesting features only from short segments, its parallel
promoters in each cell line to serve as input to the Gradient Tree computing ability is strong. Therefore, we combine the CNN model
Boosting classifier. However, the corpus used to train word vectors with the RNN model for feature extraction. We first used one-di-
in PEP-WORD has only one cell line data (train different word vec- mensional convolution to extract local correlation features in
tors for different cell lines). Because the learning corpus is small, the sequences and then input them into Bi-GRU to extract global correl-
trained word vectors contain limited information. ation features.
To address the above limitations, we use the pre-trained DNA 1D convolution layer: Convolution requires a fixed input. We
vectors provided in dna2vec (Ng, 2017). dan2vec is a new method, did the same procedure with previous studies, like SPEID, EPIANN
based on the word2vec (Mikolov et al., 2013) word embedding and SIMCNN, all of which set the lengths of the promoters and
model, to compute the distributed representation of k-mers (a k-mer enhancers to 3000 and 2000, respectively. Singh et al. (2019) indi-
is a sequence fragment of length k) in DNA sequences. Unlike the cated that the length setting can contain all the relevant regions
word embedding used in PEP-WORD, dna2vec used the human gen- around the enhancer and promoter. Therefore, we also specify the
ome sequence as a learning corpus to embed k-mers into a 100-di- length of the two input sequences as 3000 (for enhancers) and 2000
mensional continuous vector space. The dna2vec used a larger (for promoters). We define 64 filters with a kernel size of 40. ReLU

Downloaded from [Link] by guest on 05 March 2025

learning corpus than PEP-WORD, so the learnt DNA vectors con- function was set as the activation function. The model learns 64 dif-
tain more sequence information. In dna2vec, the fact that summing ferent features for each sequence at this layer.
of DNA vectors is akin to nucleotides concatenation was verified in Max-pool layer: To reduce the complexity of the output and pre-
experiments [e.g. vec (‘ACATT’) – vec (‘ATT’) þ vec (‘GAG’)  vec vent overfitting, a pooling layer is often used after a convolution
(‘ACGAG’)]. This shows that using word embedding on DNA layer. In our work, we chose the max-pool layer with a pool window
sequences does capture useful information. size of 20 and a step size of 20. This means that the output matrix of
Kim (2014) demonstrated that, in NLP, it is advantageous to this layer is only one-twentieth the size of the input matrix.
perform the appropriate fine tune on pre-trained word vectors, Bi-GRU layer: Gated recurrent unit (GRU) is an improvement of
which makes the vectors more suitable for the specific task. the normal RNN, which has ability to solve long-term dependency
Therefore, we use fine tune strategy (i.e. the weight of the embed- problems. We assume that a segment of a sequence is related to both
ding layer was set to be trainable) to improve the performance of the previous sequence segments and the subsequent sequence seg-
our model. PEP-WORD verified in the experiment that when k ¼ 6, ments. Therefore, we chose to use the Bi-GRU (bi-directional GRU)
the computational efficiency of the model and the information com- to capture features. The output of the GRU was set as 50 dimen-
plexity of the vectors are most suitable. Here, we also chose k ¼ 6. sions. Because it is a Bi-GRU, two outputs in opposite directions are
The process of sequence embedding (see a specific example in concatenated for a total output dimension of 100.
Fig. 2) is described as follows:

1. Convert the enhancer and promoter sequences into sequences 2.2.3 Attention mechanism
consisting of words (6-mers), and if a word contains a ‘N’, the In the model, we added an attention layer behind the Bi-GRU layer
to adaptively learn the weight of the features. The attention mechan-
word is marked as ‘NULL’.
ism is originally proposed by Yang et al. (2016). They proposed a
2. Construct a dictionary containing 46 þ 1 words. hierarchical attention network for document classification. We as-
3. Convert each gene sequence into a sequence of word indexes sume that not all features in sequences are equally decisive for EPIs;
according to the dictionary (each word has its own unique therefore, we added this attention mechanism to enhance the contri-
index). bution of key features to our model. The formulas for the attention
4. Find the vectors in the pre-trained DNA vectors matrix accord- mechanism are described as follows:
ing to the index of each word, and form these vectors into a vec- hi ¼ tanhðWx fi þ bx Þ (1)
tor matrix as a representation of a sequence.
After the third step, we convert each DNA sequence into a  
exp hTi hx
2995  1 size index sequence (for enhancer) or a 1995  1 size index ai ¼ P  T  (2)
sequence (for promoter) and store these index sequences on the hard i exp hi hx
disk. These index sequences are the actual input for the model. X
v¼ ai fi (3)
i
2.2.2 Feature extraction
In NLP, researchers typically use the recurrent neural network where, hi , is a hidden representation of the ith feature, fi , from the
(RNN) model to analyze sequences. Compared with the CNN Bi-GRU layer. The importance of the feature is measured by the
model, the RNN model has the ability to solve long-term depend- similarity of hi and context vector, hx . The normalized weight, a, of
ency problem, which is important for sequences. However, the each feature was obtained by the softmax function. All feature vec-
RNN model cannot be calculated in parallel, and, for long tors were multiplied by their corresponding weights and then
summed as a final output vector, v. We randomly initialized Wx , bx
and hx , which were jointly learned during the training process.

2.3 Model training

We implemented our EPIVAN model in Keras (available at https://
[Link]/hzy95/EPIVAN). We used the batches of 64 samples to
train the model. The cross-entropy loss function was used to calcu-
late the error. We chose the Adam optimization algorithm (Kingma
and Ba, 2014) to update the neural network weights. To prevent
overfitting, we used batch normalization and dropout (with a ran-
dom drop probability P ¼ 0.5) (Srivastava et al., 2014) behind the
merge layer.
As described in Section 2.1, the dataset provided by
Fig. 2. An example of sequence embedding. This figure is only an example. The TargetFinder is an imbalanced dataset. In supervised deep learning,
length of the enhancer sequences and promoter sequences in the experiment were if the imbalanced dataset is used for training, the traditional classifi-
3000 and 2000, respectively cation algorithm with the total accuracy as the learning goal, will
1040 [Link] et al.

pay too much attention to the majority class; therefore, the accuracy Table 2. Prediction results of EPIVAN-specific in terms of AUROC
of the minority class is degraded. Mao et al. (2017) amplified sam- cross different cell lines
ples by shifting the fixed size window to the left or right on DNA
sequences to obtain new enhancer sequences and new promoter Training/test GM12878 HUVEC HeLa-S3 IMR90 K562 NHEK
sequences. The lengths of obtained enhancers and promoters are cell line
fixed at 3000 bp and 2000 bp, respectively. We used the same data
augmentation method to amplify the positive samples in the training GM12878 0.935 0.657 0.598 0.653 0.632 0.616
set to 20 times to achieve class balance. To compare the perform- HUVEC 0.582 0.930 0.651 0.605 0.586 0.628
ance of our model with other models, we fixed the training set and HeLa-S3 0.579 0.619 0.943 0.624 0.616 0.630
test set for each cell line. The training procedure is described as IMR90 0.573 0.601 0.605 0.938 0.558 0.612
follows: K562 0.601 0.607 0.629 0.611 0.949 0.621
NHEK 0.513 0.623 0.591 0.609 0.567 0.971
1. Randomly divide the original imbalanced dataset, Dc [c2ALL,
ALL¼{‘GM12878’, ‘HUVEC’, ‘HeLa-S3’, ‘IMR90’, ‘K562’, Note: The best performance in each cell line is given in boldface.
‘NHEK’}], into a training set, Dcimbltrain (90%), and a test set,

Downloaded from [Link] by guest on 05 March 2025

Dctest (10%), in each cell line. The ratio of positive and negative Table 3. Prediction results of EPIVAN-specific in terms of AUPR
samples in Dcimbltrain and in Dctest is 1:20. cross different cell lines
2. Amplify the positive sample in Dcimbltrain 20 times to produce a
balanced training set, Dcbltrain , in each cell line. Training/test GM12878 HUVEC HeLa-S3 IMR90 K562 NHEK
3. Train the model on Dcbltrain for 15 epochs. cell line
4. Evaluate the model on Dctest . GM12878 0.781 0.140 0.104 0.128 0.199 0.132
HUVEC 0.112 0.689 0.153 0.105 0.134 0.184
HeLa-S3 0.082 0.152 0.803 0.098 0.149 0.165
2.4 Evaluation metrics
IMR90 0.104 0.093 0.103 0.759 0.076 0.096
In this study, the datasets we used for performance evaluation are
K562 0.132 0.136 0.124 0.121 0.802 0.135
extremely imbalanced. Therefore, the evaluation metrics we used
are area under the receiver operating characteristic curve (AUROC) NHEK 0.085 0.159 0.184 0.111 0.071 0.868
(Hanley and Mcneil, 1982) and area under precision-recall curve
Note: The best performance in each cell line is given in boldface.
(AUPR) (Davis and Goadrich, 2006).
The receiver operating characteristic curve is plotted with sensi-
tivity as the vertical axis and false positive rate (1-specificity) as the sequential characteristics of one specific cell line is not able to accur-
horizontal axis, reflecting the relationship between sensitivity and ately predict the EPIs on other cell lines from the perspective of se-
specificity at different thresholds. The area under this curve is called quence itself, since the model trained on one specific cell line can
AUROC. The closer the value of AUROC is to 1 (the closer the learn only the interaction patterns of the enhancer and promoters in
curve is to the upper left corner), the better the performance of the that cell line. On the other hand, it also further confirms that the
model is. The ROC curve does not change due to the distribution of interaction of the enhancer and promoter has cell line specificity.
positive and negative samples; therefore, the AUROC is suitable as
an evaluation metric for the model used for imbalanced binary
classification. 3.2 A general model trained on all cell lines
The precision-recall curve is plotted with precision as the vertical Due to cell line specificity, we train the corresponding model for
axis and recall as the horizontal axis, reflecting the trade-off be- each cell line (i.e. to train six EPIVAN-specific models), which is
tween the precision of the model’s recognition of positive examples time consuming. Therefore, we built a general model, EPIVAN-
and the ability of the model to cover positive cases. The area under general, which can be used for any cell line. We hypothesized that in
the precision-recall curve is called AUPR. The closer the value of addition to cell line-specific features, EPIs also have some cell line
AUPR is to 1 (the closer the curve is to the upper right corner), the common features. Only when the training set contains enough dif-
better the performance of the model. ferent cell lines can the model capture the cell line common features.
Based on this hypothesis, we aggregated and disrupted the training
sets of the six cell lines to produce a new training set, DALL
bltrain (this
3 Results training set is balanced). We trained EPIVAN-general on DALL bltrain for
20 epochs and evaluated the model on the test set Dctest of each cell
3.1 Prediction ability of cell line-specific model on six line separately. The results of EPIVAN-general are given in Table 4.
cell lines Comparing EPIVAN-general with EPIVAN-specific, we find
The interaction of the enhancer and promoter has cell line specifi- that, on each cell line, the AUPR of EPIVAN-general is about 5%
city, and the interaction rules are different in different cell lines. lower than that of EPIVAN-specific, but the AUROC value of
Therefore, a model trained on a specific cell line may not be suitable EPIVAN-general is about 2% higher than that of EPIVAN-specific.
for other cell lines. As shown in the experiments discussed in Section 3.1, on one specif-
We trained our model by using the training procedure mentioned ic cell line, specificity features play a more important role in EPIs.
in Section 2.3. For convenience of discussion, our cell line-specific Compared with EPIVAN-specific, EPIVAN-general has the ability
model is denoted as EPIVAN-specific. to capture common features, but the ability to capture specific
Tables 2 and 3 show the AUROC and AUPR of EPIVAN- features is lower than that of EPIVAN-specific, which results in
specific for across cell line evaluation. As seen in Tables 2 and 3, the overall performance of EPIVAN-general not being as good
EPIVAN-specific has a good ability to predict EPIs when the training as that of EPIVAN-specific on a specific cell line. However,
and test sets are from the same cell line. However, the model EPIVAN-general is still suitable as a general model for predicting
achieved significantly decreased performance for cross cell line test, EPIs on various cell lines.
as compared with the model trained and test on the same cell line.
Meanwhile, we also compared the cross cell line prediction ability
of existing predictors. We found that the other models (SPEID, PEP- 3.3 Contributions of pre-trained DNA vectors and
WORD, EPIANN and SIMCNN) also performed almost the same attention mechanism
as EPIVAN-specific. Detailed results can be referred to their corre- In this section, we discuss how pre-trained DNA vectors and atten-
sponding references. The results demonstrate that using the tion mechanism can impact the predictive performance of our cell
Predicting EPIs with EPIVAN 1041

Table 4. AUROC and AUPR values of EPIVAN-general on each cell

line

Cell line GM12878 HUVEC HeLa-S3 IMR90 K562 NHEK

AUROC 0.955 0.934 0.952 0.952 0.951 0.974

AUPR 0.733 0.696 0.713 0.703 0.739 0.779

Downloaded from [Link] by guest on 05 March 2025

Fig. 4. Performance of eight models in terms of AUPR on six cell lines. (a) Shows the
performance of EPIVAN-specific after removing pre-trained vector or attention
mechanism. (b) Shows the performance of EPIVAN-general after removing pre-
trained vector or attention mechanism

pre-trained DNA vectors may not be suitable for identifying EPIs.

In order to fit the pre-trained DNA vector to current task (identifica-
tion of EPIs), the pre-trained DNA vector must be adjusted.
Although we used a fine-tuning strategy, the limited data scale for
Fig. 3. Performance of eight models in terms of AUROC on six cell lines. (a) Shows single one cell line made the pre-trained DNA vectors still not fit
the performance of EPIVAN-specific after removing pre-trained vector or attention well to current task. When the EPI dataset is small, the ordinary
mechanism. (b) Shows the performance of EPIVAN-general after removing pre- word embedding is more suitable for the current task than the pre-
trained vector or attention mechanism trained DNA vectors. Therefore, EPIVAN-spe-nopre performed bet-
ter than EPIVAN-specific. EPIVAN-general was trained on six cell
lines. In this case the EPI dataset is large enough, the pre-trained
line-specific model (EPIVAN-specific) and our general model
DNA vectors can adapt to the current task. The abundant informa-
(EPIVAN-general). We trained six models, which are defined as fol-
tion contained in the pre-trained DNA vectors helps EPIVAN-general
lows: (i) EPIVAN-spe-nopre: an EPIVAN-specific model (trained on
better capture cell line-common features. Thus, EPIVAN-general per-
Dcbltrain ) removing pre-trained DNA vectors and using ordinary word
formed better than EPIVAN-gen-nopre. In Figures 3 and 4, we can
embedding, (ii) EPIVAN-spe-onehot: an EPIVAN-specific model
see that the performance of EPIVAN-spe-onehot is worse than that of
(trained on Dcbltrain ) removing pre-trained DNA vectors and using
EPIVAN-specific and EPIVAN-spe-nopre, and the performance of
one-hot; (iii) EPIVAN-spe-noatt: an EPIVAN-specific model (trained
EPIVAN-gen-onehot is also worse than that of EPIVAN-general
on Dcbltrain ) removing attention mechanism, (iv) EPIVAN-gen-nopre:
and EPIVAN-gen-nopre. Because the information between each one-
an EPIVAN-general model (trained on DALL bltrain ) removing pre-trained hot vector is independent, encoding a DNA sequence with one-hot
DNA vectors and using ordinary word embedding, (v) EPIVAN-
embedding will lose a lot of useful information.
gen-onehot: an EPIVAN-general model (trained on DALL bltrain ) On each cell line, the AUROC and AUPR values of EPIVAN-
removing pre-trained DNA vectors and using one-hot and (vi)
specific were higher than those of EPIVAN-spe-noatt. The AUROC
EPIVAN-gen-noatt: an EPIVAN-general model (trained on DALL bltrain ) and AUPR values of EPIVAN-general were also higher than those of
removing attention mechanism. It is worth noting that the 100-di-
EPIVAN-gen-noatt. This indicates that the attention mechanism is
mensional representations of k-mers which were learnt during
more concerned with cell line-specific features; thus, the perform-
model training are called the ordinary word embedding.
ance of EPIVAN-specific without the attention layer was worse.
Then, we compared EPIVAN-specific with EPIVAN-spe-nopre,
Similarly, after removing the attention layer, the ability of EPIVAN-
EPIVAN-spe-onehot and EPIVAN-spe-noatt, and also compared
general to extract cell line-specific features was also reduced, leading
EPIVAN-general with EPIVAN-gen-nopre, EPIVAN-gen-onehot
to the decreased performance.
and EPIVAN-gen-noatt on six cell lines. Figures 3 and 4 show the
In summary, the pre-trained DNA vectors help the model to bet-
results of eight models on each cell line.
ter capture the cell line-common features. The attention mechanism
On each cell line, the AUROC and AUPR values of EPIVAN-
helps the model to better extract cell line-specific features. The two
spe-nopre were higher than those of EPIVAN-specific, but the
works together to greatly enhance the performance of the general
AUROC and AUPR values of EPIVAN-gen-nopre were lower than
model.
those of EPIVAN-general. Pre-trained DNA vectors were learned
through unsupervised learning on human genome. Ordinary word
embedding was learned through supervised learning for current
task. The human genome can be viewed as a big dataset of DNA 3.4 Improving the ability of EPIVAN-general on specific
sequences. The enhancer and promoter sequences of the six cell lines cell line
in our dataset are a small part of the human genome. Therefore, the In Sections 3.2 and 3.3, we demonstrated that the EPIVAN-general
pre-trained DNA vectors can better capture the relationship between model captured both cell line-specific and cell line-common features.
DNA k-mers than ordinary word embedding. However, the Although EPIVAN-general is versatile on six cell lines, it performed
1042 [Link] et al.

slightly worse than EPIVAN-specific on six cell lines. Therefore, we 2.3% higher than the runner up predictors on GM12878, HUVEC,
used a new training strategy to make EPIVAN-general more suitable HeLa-S3, IMR90, K562 and NHEK, respectively. Likewise, EPIVAN-
to the specific cell line by enhancing the learning of cell line-specific best outperforms existing predictors on four out of the six cell lines
features. The training process is described as follows: in terms of AUPR, with only two exceptions, in which our model is
slightly worse than the best PEP-WORD on IMR90 and K562 cell
1. Use EPIVAN-general mentioned in Section 3.3 as a pre-trained lines. It is worth noting that, although the AUPR value of PEP-
model. WORD is similar to that of EPIVAN-best, the AUROC value of
2. Train the pre-trained model on the training set, Dcbltrain , of the EPIVAN-best is about 10% higher than that of PEP-WORD. In
specified cell line, c, for 15 epochs. summary, our model performs better than state-of-the-art predic-
tors on specific cell lines.
3. Evaluate the model on the test set, Dctest , of the specified cell line, c.
A model trained with this training strategy is called EPIVAN-
3.5 Transfer ability of EPIVAN-general
best.
We have demonstrated that EPIVAN-general can be used well on
the cell lines contained in the training set. Can EPIVAN-general be

Downloaded from [Link] by guest on 05 March 2025

3.4.1 Comparison of our models on six cell lines used as a starting point for the prediction of new cell lines? We
Since EPIVAN-best is an enhancement of EPIVAN-general on each designed an experiment to verify the transfer ability of EPIVAN-
cell line, we compared these two models. At the same time, we also general. The transfer learning process is described below:
added EPIVAN-specific for comparison.
1. Take one of the six cell lines as a new cell line, new. The training
The experimental results of three different models on each cell line
in terms of AUROC and AUPR are shown in Tables 5 and 6, respect- sets of the other cell lines are aggregated as the training set,
ively. It can be observed that on each cell line, the AUROC value of DALLnew
bltrain (this training set is balanced).
EPIVAN-best is about 1% higher than that of EPIVAN-general, and 2. Pre-train the model on DALLnew
bltrain for 20 epochs to obtain the pre-
the AUPR value of EPIVAN-best is about 10% higher than that of trained model EPIVAN-general.
EPIVAN-general. After enhancing the learning of cell line-specific fea- 3. Train EPIVAN-general on training set, Dnew bltrain , of cell line new.
tures, EPIVAN-best performs much better than EPIVAN-general.
4. Evaluate the model on test set, Dnew test , of cell line, new.
EPIVAN-best even performs better than EPIVAN-specific. On
each cell line, the AUROC value of EPIVAN-best is about 2% A model transferred to a new cell line is called EPIVAN-transfer.
higher than that of EPIVAN-specific, and the AUPR value of Table 9 shows the results of EPIVAN-transfer on each new cell line. The
EPIVAN-best is about 3% higher than that of EPIVAN-specific. experimental results show that EPIVAN-general trained with only the
Although EPIVAN-best and EPIVAN-specific both built different data of five cell lines has good transfer ability and can be transferred to
models for different cell lines, EPIVAN-best, which uses EPIVAN- any new cell line. Therefore, EPIVAN-general trained with the data of
general as a pre-trained model, captures more cell line-common fea- six cell lines in Section 3.3 is more suitable as a pre-trained model for
tures than EPIVAN-specific. transfer learning (the weight of the general model is available at https://
[Link]/hzy95/EPIVAN/tree/master/model/generalModel).
3.4.2 Comparison of our model and state-of-the-art models on six
cell lines Table 7. Performance of different models in terms of AUROC on six
In this section, we further compare the performance of EPIVAN- cell lines
best with four state-of-the-art predictors, including SPEID, PEP-
Model/cell line GM12878 HUVEC HeLa-S3 IMR90 K562 NHEK
WORD, EPIANN and SIMCNN. On each cell line, each model used
the same training set and the same test set. Each compared predic- EPIVAN-best 0.965 0.950 0.960 0.962 0.959 0.985
tors was trained by the training process as mentioned in their own SPEID 0.916 0.904 0.923 0.915 0.922 0.950
corresponding literatures.
PEP-WORD 0.842 0.845 0.843 0.898 0.883 0.917
The results of our model (EPIVAN-best) and the four predictors on
EPIANN 0.919 0.918 0.924 0.945 0.943 0.959
six cell lines in terms of AUROC and AUPR are presented in Tables 7
SIMCNN 0.941 0.933 0.949 0.951 0.943 0.962
and 8, respectively. Since the training and test sets used for evaluation
are exactly the same as those of Mao et al. (2017), the results of
Note: The best performance in each cell line is given in boldface.
EPIANN and PEP-WORD are copied directly from Mao et al. (2017).
As we can see, the AUROC of EPIVAN-best is the highest of all mod-
els. Specifically, our model is 2.4%, 1.7%, 1.1%, 1.1%, 1.6% and Table 8. Performance of different models in terms of AUPR on six
cell lines
Table 5. AUROC value of three different models on six cell lines
Model/cell line GM12878 HUVEC HeLa-S3 IMR90 K562 NHEK
Model/cell line GM12878 HUVEC HeLa-S3 IMR90 K562 NHEK
EPIVAN-best 0.819 0.773 0.820 0.801 0.814 0.899
EPIVAN-best 0.965 0.950 0.960 0.962 0.959 0.985 SPEID 0.773 0.523 0.797 0.732 0.771 0.852
EPIVAN-general 0.955 0.934 0.952 0.952 0.951 0.974 PEP-WORD 0.807 0.76 0.803 0.868 0.836 0.88
EPIVAN-specific 0.935 0.930 0.943 0.938 0.949 0.971 EPIANN 0.723 0.616 0.702 0.77 0.673 0.861
SIMCNN 0.706 0.640 0.737 0.737 0.679 0.882
Note: The best performance in each cell line is given in boldface.
Note: The best performance in each cell line is given in boldface.

Table 6. AUPR value of three different models on six cell lines

Table 9. AUROC and AUPR values of EPIVAN-general transferred
Model/cell line GM12878 HUVEC HeLa-S3 IMR90 K562 NHEK
to the new cell line
EPIVAN-best 0.819 0.773 0.820 0.801 0.814 0.899
Cell line new GM12878 HUVEC HeLa-S3 IMR90 K562 NHEK
EPIVAN-general 0.733 0.696 0.713 0.703 0.739 0.779
EPIVAN-specific 0.781 0.689 0.803 0.759 0.802 0.868 AUROC 0.963 0.943 0.957 0.958 0.951 0.983
AUPR 0.809 0.725 0.811 0.796 0.798 0.897
Note: The best performance in each cell line is given in boldface.
Predicting EPIs with EPIVAN 1043

4 Conclusion References
In this article, we proposed a new model, EPIVAN, which quickly pre- Arensbergen,V. et al. (2014) In search of the determinants of enhancer–pro-
dicts EPIs by using only enhancer and promoter sequences. Compared moter interaction specificity. Trends Cell Biol., 24, 695–702.
with the existing models, EPIVAN with added pre-trained DNA vec- Boris,L. et al. (2012) Metazoan promoters: emerging characteristics and
tors and an attention mechanism, better captures the hidden informa- insights into transcriptional regulation. Nat. Rev. Genet., 13, 233–245.
tion features from the perspective of sequences. Experimental results Butler,J.E. and Kadonaga,J.T. (2001) Enhancer-promoter specificity mediated
on six cell lines indicate that EPIVAN performs better than the existing by DPE or TATA core promoter motifs. Genes Dev., 15, 2515–2519.
models. Moreover, our EPIVAN has the advantages of fast training Davis,J. and Goadrich,M. (2006) The relationship between precision-recall
speed and less data storage space. In particular, our method trains a and ROC curves. In: Proceedings of the 23rd International Conference on
general model that can be used for various kinds of cell lines. Machine Learning, pp. 233–240. ACM, Pittsburgh, PA.
Hanley,J.A. and Mcneil,B.J. (1982) The meaning and use of the area under a
Compared with other methods that build only a specific model for
receiver operating characteristic (ROC) curve. Radiology, 143, 29–36.
each cell line, our method significantly reduces the cost of training
Kim,Y. (2014) Convolutional neural networks for sentence classification.
models. The results of our experiment demonstrate that pre-trained
arXiv: 1408.5882.
DNA vectors and attention mechanism contribute greatly to the
Kingma,D.P. and Ba,J. (2014) Adam: a method for stochastic optimization.

Downloaded from [Link] by guest on 05 March 2025

performance of the model. It further confirms that our model is cap- arXiv: 1412.6980.
able of capturing both cell line-specific features and cell line-common Mao,W. et al. (2017) Modeling enhancer-promoter interactions with
features. Additionally, we also proved that our model has good transfer attention-based neural networks. bioRxiv, 219667.
ability and can be used as a pre-trained model for transfer learning. Mikolov,T. et al. (2013) Efficient estimation of word representations in vector
We have already seen the potential of pre-trained word vectors space. arXiv: 1301.3781.
in biological applications. In future work, we will study whether Nastaran,H. et al. (2014) Genome-wide map of regulatory interactions in the
there are better word embedding pre-training methods that can be human genome. Genome Res., 24, 1905–1917.
used on biological sequences to obtain pre-trained DNA vectors Ng,P. (2017) dna2vec: consistent vector representations of variable-length
containing more information. EPIVAN uses convolution and pool- k-mers. arXiv: 1701.06279.
ing to extract abstract local features, which leads the model to be Pennacchio,L.A. et al. (2013) Enhancers: five essential questions. Nat. Rev.
unable to retain the position information of k-mers. Therefore, we Genet., 14, 288.
can’t explain the information obtained by the model, i.e. EPIVAN is Rao,S.S.P. et al. (2014) A 3D map of the human genome at kilobase resolution
an uninterpretable model. Building an interpretable model is also reveals principles of chromatin looping. Cell, 159, 1665–1680.
worthwhile to study in the future. Singh,S. et al. (2019) Predicting enhancer-promoter interaction from genomic
sequence with deep neural networks. Quant. Biol., 7, 122–137.
Srivastava,N. et al. (2014) Dropout: a simple way to prevent neural networks
from overfitting. JMLR, 15, 1929–1958.
Funding Whalen,S. et al. (2016) Enhancer-promoter interactions are encoded by
complex genomic signatures on looping chromatin. Nat. Genet., 48,
The work was supported by the National Natural Science Foundation of
488–496.
China [61701340, 61472333, 61772441, 61472335, 61272152, 61872309
Williamson,I. et al. (2011) Enhancers: from developmental genetics to the gen-
and 41476118]; Project of marine economic innovation and development in etics of common human disease. Dev. Cell, 21, 17–19.
Xiamen [16PFW034SF02]; Natural Science Foundation of the Higher Yang,Y. et al. (2017) Exploiting sequence-based features for predicting
Education Institutions of Fujian Province [JZ160400]; Natural Science enhancer-promoter interactions. Bioinformatics, 33, i252–i260.
Foundation of Fujian Province [2017J01099]; Natural Science Foundation of Yang,Z. et al. (2016) Hierarchical attention networks for document classifica-
Tianjin City [18JCQNJC00500]; and National Key R&D Program of China tion. arXiv: 1707.00896.
[2017YFE0130600] Zhuang,Z. et al. (2019) A simple convolutional neural network for prediction
of enhancer–promoter interactions with DNA sequence data.
Conflict of Interest: none declared. Bioinformatics, 35, 2899–2906.