Received: 8 October 2022 Revised: 2 April 2023 Accepted: 27 June 2023

DOI: 10.1002/asi.24817

RESEARCH ARTICLE

Neural machine translation for in-text citation

classification

Iqra Safder 1 | Momin Ali 2 | Naif Radi Aljohani 3 | Raheel Nawaz 4 |

5
Saeed-Ul Hassan

1
Department of Computer Science,
National University of Computer & Abstract
Emerging Sciences, Lahore, Pakistan The quality of scientific publications can be measured by quantitative indices
2
Department of Computer Science, such as the h-index, Source Normalized Impact per Paper, or g-index. How-
Information Technology University,
ever, these measures lack to explain the function or reasons for citations and
Lahore, Pakistan
3
Faculty of Computing and Information
the context of citations from citing publication to cited publication. We argue
Technology, King Abdulaziz University, that citation context may be considered while calculating the impact of
Jeddah, Saudi Arabia research work. However, mining citation context from unstructured full-text
4
Staffordshire University, Stoke-on-
publications is a challenging task. In this paper, we compiled a data set com-
Trent, UK
5 prising 9,518 citations context. We developed a deep learning-based architec-
Department of Computing and
Mathematics, Manchester Metropolitan ture for citation context classification. Unlike feature-based state-of-the-art
University, Manchester, UK models, our proposed focal-loss and class-weight-aware BiLSTM model with
Correspondence
pretrained GloVe embedding vectors use citation context as input to outper-
Iqra Safder, Department of Computer form them in multiclass citation context classification tasks. Our model
Science, National University of improves on the baseline state-of-the-art by achieving an F1 score of 0.80 with
Computer & Emerging Sciences, Lahore,
Pakistan. an accuracy of 0.81 for citation context classification. Moreover, we delve into
Email: [Link]@[Link] the effects of using different word embeddings on the performance of the clas-
sification model and draw a comparison between fastText, GloVe, and spaCy
Funding information
Faculty Research Support Program pretrained word embeddings.
(FRSG) Fall 2022, National University of
Computer and Emerging Sciences

1 | INTRODUCTION publications, journals, and scholars (Iqbal et al., 2021;

Safder et al., 2022).
The scientific literature accounts for more than 100 million There is a debate over the usage of citation counts
publications in recent years, increasing to 1.5 million and other such quantitative measures for the evaluation
publications every year (Aljohani et al., 2021a, 2021b; of publications and journals' impact. These quantitative
Iqbal et al., 2022). To navigate through such extensive measures are justified for scientific evaluation (Kunnath
data and efficiently access scientific resources, we need a et al., 2020), but there are diverse views regarding the
solution that can help extract information from these type of studies that can be assessed using such metrics.
records and identify essential pieces of published works. However, these indexes give equal importance to all cita-
For this purpose, citations can be used to identify quality tions, whereas it has been established that all citations
work. Citations provide pointers that could be useful to are not equal; some are important while others are
summarize a research work. Besides this, citation counts merely incidental (Hassan et al., 2020).
are utilized to measure various indexes, such as the According to Britain (1970), “citation” is when an
h-index and g-index, which signify the importance of author cites a publication in their article, and the cited

J Assoc Inf Sci Technol. 2023;74:1229–1240. [Link]/journal/asi © 2023 Association for Information Science and Technology. 1229
 23301643, 2023, 10, Downloaded from [Link] by University Of Southampton, Wiley Online Library on [04/03/2024]. See the Terms and Conditions ([Link] on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
1230 SAFDER ET AL.

article is called a reference. There could be a variety of (Iqbal et al., 2021). Using the Extra tree classifier, they
reasons for citing a publication, and the relation between select the best 29 features out of 64 features. These
the citing and the cited publication should depict the selected features were used for training models. Using
importance of cited material. Boyack et al. (2018) and random forest, the supervised model achieved the best
Bornmann et al. (2020) explored the reasons for citing a results. Moreover, they presented a deep learning
publication, and according to them, a work may be cited model, the LSTM network. Using all 64 features, this
to provide background context of the problem at hand, model achieved an accuracy score of 92.57%.
make a comparison of different approaches to solving the Unlike most studies related to citation classification,
problem, and build upon an already published work by Hassan et al. (2018) took an unsupervised learning
utilizing a proposed algorithm or putting to work a sug- approach to tackle the classification task. The researchers
gested technique. To understand the purpose of citation clustered citations into two groups using a self-organizing
and the impact of cited work, it is vital to understand map (SOM). This approach allowed them to visualize the
the intention of the citing author. One way to under- data intuitively and provided a qualitative understanding
stand a citation's intention and function is by extracting of the features. They identified that the incidental class
and analyzing the citation context (Hassan et al., 2018; organizes into an independent cluster with adjacent neu-
Safder & Hassan, 2019). Citation analysis is a great tool rons. This shows that it is easier to separate incidental
to understand how a field of research has progressed citations from important ones; however, the classification
over time, map the flow of knowledge, and measure the task carries an inherent issue of class imbalance.
impact of research work (Arshad et al., 2019; Numerous studies have indicated the power of
Drongstrup et al., 2020; Safder et al., 2021). machine learning approaches to retrieve in-text influen-
Valenzuela et al. (2015) presented 12 features, ranging tial citations from scholarly publications. However, they
from citation counts to positional features that provide have significant shortcomings, such as the time-
information about the actual position of citation in the consuming feature engineering task and the lack of any
referred publication, for developing a binary classification data set annotated by human experts with which to train
model for citations. Using a data set of 450 annotated their machine learning models. In this paper, we present
citations, they developed a classification model with a the following contributions:
precision of 65% for recall of 90%. Their analysis showed
that self-citations and the total number of citations per • First, we used a comprehensive tagged data set of
section are the two best features for classifying the cita- 9,518 citation contexts taken from anthology research
tions. Combining the features presented by Zhu et al. papers published by the Association for Computa-
(2015) and Valenzuela et al. (2015), a study conducted by tional Linguistics (ACL). For citation categorization,
Pride and Knoth (2018) showed that by combining just the data set includes a six-class annotation scheme:
three features (abstract similarity; author overlap; and Use, Extend, CompareOrContrast, Motivation, Back-
the total number of direct citations) give better results ground, and Other. These six categories are combined
compared to the previously suggested features. This into two major categories: important and nonimpor-
research reinforces the findings of Zhu et al. (2015) and tant citations.
Valenzuela et al. (2015) regarding the high correlation • Second, we describe a tailored focal-loss and class-
between the number of in-text references and citation weight-aware bidirectional LSTM network to address
influence. This study also reveals that abstract similarity the underlying problem of class imbalance in citation
between the citing article and the reference is also one of text classification tasks. Furthermore, we contend that
the most predictive features for citation classification. using the tailored focal-loss method improves the over-
Extending the work of Valenzuela et al. (2015) and all cross-entropy function by a factor of ð1
pt Þγ and
Hassan et al. (2017a, 2017b) presented a set of 13 fea- that the suggested model surpasses the current state-
tures categorized into three groups; textual features, of-the-art methodologies.
context-based features, and cue word-based features. • Finally, we investigated word embedding models such
They experimented with different classifiers to develop as fastText, GloVe, and spaCy to improve the classifica-
a classification model, including support vector tion results. Our proposed model outperformed the
machines, random forest, K-nearest neighbors, Decision existing methods with a 0.80 F1 score using BiLSTM
Tree, and Naive Bayes. Random forest achieved the best and GloVe embeddings.
results with an area under the curve of 91%. Building on
this research work, Hassan et al. (2018) developed The rest of the paper has been organized as follows:
another classification model, which explored the effects Section 4 presents related work. Section 5 presents data
of using an extra tree classifier and other classifiers processing steps and proposed embedding techniques.
 23301643, 2023, 10, Downloaded from [Link] by University Of Southampton, Wiley Online Library on [04/03/2024]. See the Terms and Conditions ([Link] on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
SAFDER ET AL. 1231

Section 4 presents results. Concluding remarks and overall classification accuracy of 84% with a random for-
future research directions are presented in section 5. est classifier.
Furthermore, Kunnath et al. (2020) organized a 3C
Citation Context Classification task as part of the work-
2 | R E LA T E D WOR K shop on mining scientific publications. de Andrade and
Gonçalves (2020) designed a novel approach to citation
Over the years, numerous studies have been conducted to classification. They combine potentially complementary
propose different methods for citation classification pro- representations of text, including TF-IDF, LDA, and
posed. Finney (1979) proposed a classification scheme GloVe word embedding for representing statistical, topi-
based on citation location in the document and cue verbs cal, and contextual information, respectively, to classify
around it. An enhancement proposed by Garzone and the context of the citation. Using an ACL-ARC data set of
Mercer (2000) suggested 35 categories for classification of 3,000 annotated samples, Jurgens et al. (2018) achieved a
citations, including contrasting type, research type, devel- Macro F1 score of 0.20 on the citation purpose classifica-
opmental type, negational type, and affirmational type. tion task and Macro F1 of 0.59 on the citation influence
Teufel et al. (2006) presented a model that classified classification task.
citation purpose into nine categories. This taxonomy was Oesterling et al. (2021) presents multitask learning
broadly classified as positive, negative, and neutral cita- model that combines different modules designed to han-
tions. The data used for this research contains 2,829 dle the citation classification task from different perspec-
citations obtained from 116 research articles. These cita- tives. Their proposed model utilized a variety of features
tions are manually annotated. Using machine learning extracted from the data set (Jurgens et al., 2018). These
techniques, all the research articles are processed. The features include hand-generated linguistic features, TF-
results obtained after the processing are compared to the IDF features, and an LSTM with an attention model.
tags provided by the annotators. Hand-generated linguistic features are further divided
Another classification scheme based on four catego- into frequency-based features, positional features, num-
ries, fundamental idea, background, comparison, and ber of stop words, count of nonstop words found in both
technical basis, was suggested by Dong and Schäfer the citing and cited paper, and a feature that indicates
(2011). They use domain agnostic features from textual, the presence of words from a set vocabulary in records
physical and syntactical aspects to classify the citation categorized as COMPAREANDCONTRAST samples. As
purpose. To verify their classification results, they com- for TF-IDF features, the vectors were generated for given
pared them with the annotations provided by two human citation contexts. Lastly, the LSTM with attention model
annotators. Training for different ratios of train-test data is inspired by Cohan et al. (2019) structural scaffold
split, they found Naive Bayes to be the most effective model, where, for each token in a sentence, a word vector
model for the problem as it was least affected by the is generated. This word vector is obtained by combining
train/test data ratio and achieves good comparable results the embeddings generated by GloVe and ELMo. Then
for even 10% training data. They also presented an these vectors pass through a BiLSTM network. After-
ensemble style classification model with self-training wards, the hidden states of the network are fed into an
capability to up-sample the training data effectively. attention mechanism which gives a vector encoding of
Another research work focusing on identifying the input sentence.
important citations is proposed by Valenzuela et al. Aljohani et al. (2021a, 2021b) presented a model of
(2015). Using a supervised learning approach, they per- citation classification for a scholarly search system. The
form binary classification of citations (important or non- system utilized the Random Forest prediction model for
important) as well as multiclass classification, that is, feature engineering. This search system was tested on a
nonimportant (related work), nonimportant (compari- data set of 4,138 full-text articles indexed by PLOS ONE,
son), important (extending the work), and important with 31,839 unique references. Lu et al. (2020) developed
(using the work). Using a data set of 450 annotated cita- a model they call VGCN-BERT that utilizes Graph Con-
tions, they achieved an AUCPR of 0.80. Hassan volutional Networks and BERT to create a representation
et al. (2017a, 2017b) addresses the citation classification of textual data for text classification. Unlike other works
regarding the development presented in a publication. in citation classification, Aljohani et al. (2021a, 2021b)
They use eight previously published features and six used citation context only and applied CNN with fastText
novel features (including cue words, textual based and based pretrained embedding vectors on the data to per-
context-based features) for building their classification form citation classification. Using this approach, they
models. By employing five classification techniques on achieved a 90.6 F1 score on citation influence classifica-
an annotated data set of 465 citations, they achieve an tion and 72.3 F1 score on the multiclass classification of
 23301643, 2023, 10, Downloaded from [Link] by University Of Southampton, Wiley Online Library on [04/03/2024]. See the Terms and Conditions ([Link] on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
1232 SAFDER ET AL.

citation purpose. They propose using focal loss and class vocabulary used for training the word embeddings. The
weight functions to cater for class imbalance. vector assigned to the word is learned by training a model
in a manner similar to neural network training. The core
of word embeddings is dense distributed vector for each
3 | DATA AND METHODOLOGY word in subject vocabulary used for generating word
embeddings. This feature vector associated with the
This section presents the details of the proposed frame- word captures information regarding different aspects of
work. We use the data set compiled by Aljohani the word. Moreover, this representation of the word is
et al. (2021a, 2021b) for the experimentations. To gener- learned on the basis of usage of the word. As a result,
ate vector embedding, we use fastText, GloVe and spaCy words with similar meaning have similar vector represen-
pretrained vectors trained on 400 k words vocabulary tations. In comparison to word embedding, bag of words
Wikipedia data set to convert textual data into numeric approach provides a rigid representation of words that is
form. Afterwards, we apply the CNN and BiLSTM model not affected by semantic similarity of words.
to perform citation context classification with focal loss
for handling class imbalance.
3.4 | Word embedding using GloVe

3.1 | Data set GloVe is an unsupervised learning algorithm developed

by researchers at Stanford in 2014. It is a log bilinear
The Association for Computational Linguistics (ACL) regression model which combines features of two differ-
anthology of research articles provided us with a tagged ent model families, namely local context window
data set of 9,518 citation sentences. We combine the cita- methods and global matrix factorization (Pennington
tion methods given by Jurgens et al. (2018) and Teufel et al., 2014). It is used for obtaining vector representa-
et al. (2006) to develop a six-class annotation technique tions for words. Training is performed on aggregated
for citation classification. We proposed a six-class annota- global word pairs co-occurrence statistics from a corpus,
tion approach for citation classification: Use, Extend, which results in representations with interesting linear
Compare&Contrast, Motivation, Background, Future, substructures of the word vector space. GloVe is devel-
and None. These classes are then separated into two oped on basic idea that co-occurrence of words is primary
groups: important citations and nonimportant citations. source of information utilized by unsupervised algo-
rithms for learning word representations.
Basis idea of the GloVe is based on finding the co-
3.2 | Deep learning model architecture occurrence of words in a window. Consider a matrix
W which contains frequency of co-occurrence of word
In order to perform citation context classification, we pairs. An entry of the matrix wi,j represents the count of
propose a deep learning architecture. After experiment- ith and jth word co-occurrence in a window. We can con-
ing with different architectures and testing variety of sider GloVe as weighted least square regression with the
hyperparameters for number of layers and depth of archi- cost function as follows:
tecture, we finalized a deep learning architecture with
BiLSTM layer. We transform our textual data into word X
V   2
embedding using pretrained GloVe, fastText, and spaCy J¼ f W i,j wT wj þ bi þ bj
log W i,j , ð1Þ
i,j¼1
word embedding.

where V represents size of vocabulary, w Rd and w e  Rd

3.3 | Word embedding are vectors for the ith and jth words, and b's are biases.
The sum of W is the final output of the algorithm. In
In this work we explored different word vector embed- practice f(x) are usually chosen as follows:
ding, namely GloVe (Global Vectors for Word Represen-

tation) (Pennington et al., 2014), fastText1 and spaCy2 ðx=x max Þα x > x max
f ðx Þ ¼ : ð2Þ
(en core web lg). Word embedding is a representation of 1 otherwise
words for a subject vocabulary that allows similar words
to have similar representations. In word embedding tech-
nique each word is assigned a vector in a predefined vec- The model architecture of GloVe is shown in
tor space. The vector space is defined on the basis of Figure 1. Input to the model is one hot representation of
 23301643, 2023, 10, Downloaded from [Link] by University Of Southampton, Wiley Online Library on [04/03/2024]. See the Terms and Conditions ([Link] on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
SAFDER ET AL. 1233

word. Word embedding matrices act as weight matrices representations and can handle rare words or out of
in model and therefore the model generates a vector of vocabulary words as well. fastText also provides pre-
inner product of word vectors. The embedding matrices trained embedding vectors and for our work we have uti-
are updated by the gradient of the loss function described lized 1-million-word vectors file.
above in Equations (1) and (2) (Figure A1). We also experimented with word embedding gener-
Using GloVe word embedding we initialized our ated using fastText technique. We utilized pretrained fast
embedding vectors with the pretrained 300 dimension text word embeddings trained on UMBC web base cor-
embedding vectors. These embedding vectors are trained pus, Wikipedia 2017 and [Link] data set
using Wikipedia 2014 and Gigaword5 data. Initializing (Mikolov et al., 2017). In contrast to GloVe, this tech-
word embedding with pretrained vectors allows to per- nique is useful in cases where we have lot of out of
form transfer learning and also saves us training time. vocabulary words. This is because fastText is based on
character level embedding instead of word level embed-
ding which allows to accommodate out of vocabulary
3.5 | Word embedding using fastText words by combining character level embedding
information.
The fastText is an opensource library developed by
researchers at Facebook. Unlike GloVe that works on
word pair co-occurrence and is unable to handle out of 3.6 | Word embedding using spaCy
vocabulary words fastText works on the basis of character
level information as shown in Figure 2. By achieving the spaCy is pretrained word embedding which can be uti-
granularity of character level embedding fastText pro- lized for a variety of purposes. The word embedding con-
vides a word representation that is sum of character level sists of 685 k unique vectors. The data sources used to
train and obtain these word embedding include WordNet
3.0 (Fellbaum, 2005), GloVe common crawl (Pennington
et al., 2014), OntoNotes 5,3 and ClearNLP Constituent-
to-Dependency Conversion (Choi & Palmer, 2012).
spaCy models are developed following a four step
approach; embed, encode, attend, predict. First two steps
are utilized to generate word embedding. In the first step
an embedding table is used that maps sparse vectors
representing each token or word onto a dense, continu-
ous vector as shown in Figure 3. For second step, we take
as input a sequence of word vectors and computes a new
representation with each vector being associated with a
token. These vectors are representative of meaning of the
FIGURE 1 Architecture diagram for GloVe embedding subject token in context of the rest of sentence. GRU,
technique LSTM, and RNN architectures are used for getting these

F I G U R E 2 Architecture diagram
for fastText embedding technique
 23301643, 2023, 10, Downloaded from [Link] by University Of Southampton, Wiley Online Library on [04/03/2024]. See the Terms and Conditions ([Link] on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
1234 SAFDER ET AL.

F I G U R E 3 spaCy—Encode word
vectors as context aware vectors

FIGURE 4 Citation context classification model architecture

vectors. These vectors are made up of two parts. One part context of citations. Afterwards we apply global max-
is made by forward pass of the sentence through the net- pooling with the intuition that it allows our model to
work and the second part is made from backward pass. learn the most significant features of data and incorpo-
Then both these parts are stick together to get the final rates that information to the learned representation.
matrix. Using RNN, LSTM, or GRU architectures we Next, we apply regularization in order to avoid over-
manage to learn an intermediate representation of token fitting and for this purpose we utilize drop out layer. The
as a context aware vector as shown in Figure 3. output obtained from drop-out layer is passed on to
the dense layer with relu activation function. Again, we
repeat the last two steps and then final output passes
3.7 | Citation context representation through sigmoid activation function.
using BiLSTM One of the major issues faced in citation context clas-
sification is the lack of labeled data. Existing studies
Figure 4 presents the high level architecture of the shows that identifying incidental citations is easier com-
designed approach. The embedding is then fed into BiLSTM pared to important citations (Hassan et al., 2018). This
layer while helps the model to learn representation for results in class imbalance that affects the process of
 23301643, 2023, 10, Downloaded from [Link] by University Of Southampton, Wiley Online Library on [04/03/2024]. See the Terms and Conditions ([Link] on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
SAFDER ET AL. 1235

FIGURE 5 F1 and precision curves with different embeddings vectors using BiLSTM

automating citation classification. In order to cater for 4 | E X P E R I M E N T A T I O N AN D

class imbalance, we have employed classifier level balan- RESULTS
cing technique of focal loss.
Focal loss is a technique that gives hard-classified This section encapsulates the details of the experiments
samples greater weight than well-classified examples. In conducted to develop and finalize our model for citation
most data sets, there are some data samples that are easy classification task. Moreover, we provide a comparison
to classify. Even though these data samples were segre- for the results obtained as a result of these experiments
gated with 99% efficacy in training, unlike the more diffi- as well.
cult and convoluted data cases, the results were
unsatisfactory. The fundamental issue is that the easily
classified training samples contributed just as much to the 4.1 | Hyper parameter of CNN
total cross-entropy loss value as the more challenging data
points, which improved our overall accuracy when cor- This section will provide details about the configurations
rectly identified and thus should be given more weight. of our experimental setup for training our model.
Our multiclass data is divided into six categories, with Along with values chosen for our hyper-parameters,
Background and None being the most common. This we describe the pretrained word embeddings used for
results in well-classified data instances. We employed a our classification task. To train our MLP network we
focal-loss function to deal with the data imbalance, experimented with different sets of hyper-parameters'
which reduces the weight of well-classified samples while values. After extensive experimentation we set the
raising the weight of minority-class samples, which are values as: training epochs = 100, batch size = 200,
more difficult to classify. Equation (1) shows the generic optimizer = Adam, loss function = focal loss, and
cross-entropy function: learning rate = 0.001. We employ regularization in the
last couple of layers by using dropout mechanism
Cross entropy ¼
logðpt Þ, ð3Þ where dropout rate is set to 0.1.
In order to avoid model over-fitting, we employed
Focal loss ¼
ð1
pt Þγ logðpt Þ, ð4Þ early-stop mechanisms. For both our original data set as
well as up-sampled data set, we perform a train/test split
of 9:1. The weights in our network are updated via mini
where batch gradient descent over shuffled mini-batches with
200 samples per batch.

p y¼1
pt ¼ :
1
p otherwise
4.2 | Experimentation with different
embedding vectors using Citation Context
Equation (4) depicts the focal-loss function, which
boosts the general cross-entropy function by a factor This section provides comparison of results obtained with
of ð1
pt Þγ . different embeddings using BiLSTM. We have reported
 23301643, 2023, 10, Downloaded from [Link] by University Of Southampton, Wiley Online Library on [04/03/2024]. See the Terms and Conditions ([Link] on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
1236 SAFDER ET AL.

TABLE 1 F1 measure and the accuracy results of state-of-the-art studies for citation classification

Reported evaluation
Model No. of citations Type of feature measure
Teufel et al. (2006) IBk algorithm, k = 3 2,829 Cue words and pattern-based F1: 0.57
Acc: 0.77
Dong and Schäfer Naïve Bayes 1,768 14 textual, physical and F1: 0.66
(2011) syntactic
Abu-Jbara et al. SVM 14,000 10 lexical and structural F1: 0.575
(2013)
Valenzuela et al. Random Forest 465 12 contextual and textual F1: 0.75
(2015) AUCPR: 0.80
Hassan et al. (2017a, SVM and Random Forest 465 14 context, cue words and AUCPR: 0.84
2017b, 2018) textual
Small (2018) Logistic Regression 646,347 TF-IDF-based Acc: 0.92
Jurgens et al. (2018) SVM linear Kernel 1,969 30 structural, lexical, F1: 0.53
morphological and
grammatical and field-
based
Wang et al. (2019) CRF with POS and n-gram 3,500 10 pattern and polarity-based F1: 0.882
Tuarob et al. (2019) SVM 8,796 52 context-based and TF-IDF F1: 0.749
content-based
Aljohani CNN with fastText 9,518 Only citation context F1: 0.723
et al. (2021a, Acc: 0.721
2021b)
Baseline SVM with TF-IDF 9,518 Citation context with F1: 0.48
TF-IDF vector Acc: 0.503
Naïve Bayes with TF-IDF 9,518 F1: 0.28
Acc: 0.30
CNN + spaCy CNN with spaCyx 9,518 Only citation context F1: 0.76
Acc: 0.82
CNN + GloVe CNN with GloVe 9,518 Only citation context F1: 0.78
Acc: 0.77
LSTM + fastText LSTM with fastText 9,518 Only citation context F1: 0.74
Acc: 0.79
LSTM + spaCy LSTM with spaCy 9,518 Only citation context F1: 0.77
Acc: 0.78
LSTM + GloVe LSTM with GloVe 9,518 Only citation context F1: 0.73
Acc: 0.79
BiLSTM + fastText BiLSTM with fastText 9,518 Only citation context F1: 0.77
Acc: 0.76
BiLSTM + spaCy BiLSTM with spaCy 9,518 Only citation context F1: 0.77
Acc: 0.78
Proposed model BiLSTM with GloVe 9,518 Only citation context F1:0.80
Acc: 0.81

F1 score and precision metrics on validation data for each score in the early epochs. Although this trend is prevalent
embedding in comparison to others as depicted in across the embeddings, however, for GloVe embedding
Figure 5. F1 scores comparison across the spectrum of the spike is of much large amplitude as compared to
word embeddings depicts that GloVe embedding gives other options. Second, for fastText and spaCy embed-
better result as compared to other embeddings. A closer dings F1 score remains below that for GloVe as we pro-
analysis of the curve reveals the sharp increase in F1 gress with higher number of epochs.
 23301643, 2023, 10, Downloaded from [Link] by University Of Southampton, Wiley Online Library on [04/03/2024]. See the Terms and Conditions ([Link] on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
SAFDER ET AL. 1237

features related to the training data set have achieve great

results. However, in our case, we only utilize text for
citation context for the purpose of training our classifica-
tion model. This puts our approach apart from the other
approaches. Our proposed model does not require
detailed features of training data set. The data set used
for evaluation of baseline models is same as the one we
used for training our proposed model. Our proposed
model improves on Aljohani's work. By exploring effects
of using different architectures and embeddings we
obtained an increase of 0.8 units in F1 score.
Figure 6 draws a comparison of the improvement in
F1 score achieved by using our proposed model. The dot-
F I G U R E 6 Precision curve analysis of proposed model versus ted lines represent the F1 and accuracy curves for Aljo-
Aljohani et al. (2021a, 2021b) hani et al. (2021a, 2021b) approach, whereas solid lines
are for the scores achieved by using the proposed
approach. The graph shows that our proposed approach
The precision curves for the permutations of embed- achieved much better results with only few epochs of
dings with our best model makes it evident that GloVe training. And the scores remain stable for latter epochs.
embeddings supersedes other embeddings for the applica-
tion of citation classification. The graph depicts a clear
margin between the precision scores obtained with GloVe 5 | CONCLUDING REMARKS
embedding and with other embeddings. The remaining
two embeddings, namely fastText and spaCy en core web Metrics based on citations are the indicators used by
lg, have comparable score for precision with fastText hav- researchers to determine the significance of research
ing an upper hand for early epochs. We present a com- work. The most well-known metrics are h-index, impact
parison of results achieved through our proposed factor and source normalized impact per paper (SNIP).
approach and other methodologies opted by different These metrics are quantitative and take no account of the
research groups. Moreover, a PR curve shows the com- qualitative value of a citation in a scientific publication.
parison of scores between our proposed model and the These metrics use absolute citation counts and, therefore,
work by Aljohani et al. (2021a, 2021b). do not consider any qualitative measure to estimate the
importance of citation. In this work, we have proposed a
deep learning technique for citation classification using
4.3 | Evaluation of proposed models with citation context while addressing the class imbalance
baseline issue.
We explored various techniques and tested a diverse
Table 1 shows the comparison of the state-of-the-art range of models, including TF-IDF based models, the
models along with information pertaining to the features Naive Bayes approach, CNN based models and LSTM
and function used for citation classification in the respec- based deep learning models. Our results show that
tive study. Key details regarding each system are also using BiLSTM based deep learning models performs
given alongside in the table. This comparison of different better for citation classification. Moreover, we also
methodologies shows that our proposed model outper- employed different pretrained embedding models such
forms state-of-the-art model (Aljohani et al., 2021a, as GloVe, fastText, and spaCy to improve the classifica-
2021b) for multiclass citation classification task achieving tion results.
0.80 F1 score. All of our reported results against different Our research has explored some deep learning archi-
deep learning architectures including CNN, LSTM, and tectures and some class imbalance handling techniques
BiLSTM with embeddings GloVe, fastText, and spaCy to perform citation classification effectively. To further
give better results compared to state-of-the-art work of improve results, ensemble models can be utilized. More-
Aljohani et al. (2021a, 2021b). For majority, of the cases over, to cater to class imbalance, one interesting
permutations involving GloVe embedding achieve better approach would be to transform this problem into a mul-
F1 score. titask problem where we can try to predict other proper-
Some of the existing citation classification techniques ties such as the paper title. Since such data is present in
are also reported in Table 1 that utilize detailed set of abundance, our model would be able to learn a
 23301643, 2023, 10, Downloaded from [Link] by University Of Southampton, Wiley Online Library on [04/03/2024]. See the Terms and Conditions ([Link] on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
1238 SAFDER ET AL.

representation with information of paper titles incorpo- Fellbaum, C. (2005). WordNet and wordnets. In Encyclopedia of
rated, which could lead to improved results. language and linguistics. Elsevier.
Finney, B. (1979). The reference characteristics of scientific texts
(Master's thesis). The City University of London.
E N D N O T ES
Garzone, M., & Mercer, R. E. (2000). Towards an automated cita-
1
[Link] tion classifier. In Conference of the Canadian society for compu-
2
[Link] tational studies of intelligence (pp. 337–346). Springer.
3
[Link] Hassan, S. U., Akram, A., & Haddawy, P. (2017b). Identifying
important citations using contextual information from full text.
In 2017 ACM/IEEE joint conference on digital libraries (JCDL)
R EF E RE N C E S (pp. 1–8). IEEE Press.
Abu-Jbara, A., Ezra, J., & Radev, D. (2013). Purpose and polarity of Hassan, S. U., Aljohani, N. R., Tarar, U. I., Safder, I., Sarwar, R.,
citation: Towards nlp-based bibliometrics. In Proceedings of the Alelyani, S., & Nawaz, R. (2020). Exploiting tweet sentiments
2013 conference of the north American chapter of the association in Altmetrics large-scale data. arXiv preprint arXiv:
for computational linguistics: Human language technologies 2008.13023.
(pp. 596–606). ACL. Hassan, S. U., Imran, M., Iftikhar, T., Safder, I., & Shabbir, M.
Aljohani, N. R., Fayoumi, A., & Hassan, S. U. (2021a). An in-text (2017a). Deep stylometry and lexical & syntactic features based
citation classification predictive model for a scholarly search author attribution on PLoS digital repository. In International
system. Scientometrics, 126(7), 5509–5529. conference on Asian digital libraries (pp. 119–127). Springer.
Aljohani, N. R., Fayoumi, A., & Hassan, S. U. (2021b). A novel Hassan, S. U., Imran, M., Iqbal, S., Aljohani, N. R., & Nawaz, R.
focal-loss and class-weight-aware convolutional neural network (2018). Deep context of citations using machine-learning
for the classification of in-text citations. Journal of Information models in scholarly full-text articles. Scientometrics, 117(3),
Science, 49, 79–92. 1645–1662.
Arshad, N., Bakar, A., Soroya, S. H., Safder, I., Haider, S., Iqbal, S., Hassan, S. U., Aljohani, N. R., Alelyani, S., Nawaz, R., &
Hassan, S. U., Aljohani, N. R., Alelyani, S., & Nawaz, R. (2019). Bornmann, L. (2021). A decade of in-text citation analysis based
Extracting scientific trends by mining topics from Call for Papers. on natural language processing and machine learning tech-
Library Hi Tech. niques: An overview of empirical studies. Scientometrics,
Bornmann, L., Wray, K. B., & Haunschild, R. (2020). Citation 126(8), 6551–6599.
concept analysis (CCA): A new form of citation analysis Iqbal, S., Safder, I., Aljohani, N. R., Alfakeeh, A. S., Visvizi, A.,
revealing the usefulness of concepts for other researchers Nawaz, R., & Hassan, S. U. (2022). Who cites whom and how it
illustrated by exemplary case studies including classic books impacts the knowledge production process across disciplines?:
by Thomas S. Kuhn and Karl R. Popper. Scientometrics, A methodological insight. Science & Technology Libraries, 42,
122(2), 1051–1074. 201–214.
Boyack, K. W., van Eck, N. J., Colavizza, G., & Waltman, L. (2018). Jurgens, D., Kumar, S., Hoover, R., McFarland, D., & Jurafsky, D.
Characterizing in-text citations in scientific articles: A large- (2018). Measuring the evolution of a scientific field through
scale analysis. Journal of Informetrics, 12(1), 59–73. citation frames. Transactions of the Association for Computa-
Britain, J. M. (1970). Information and its users: A review with special tional Linguistics, 6, 391–406.
reference to the social sciences. Bath University Press. Kunnath, S. N., Pride, D., Gyawali, B., & Knoth, P. (2020). Overview
Choi, J. D., & Palmer, M. (2012). Guidelines for the clear style of the 2020 WOSP 3C citation context classification task. In Pro-
constituent to dependency conversion (Technical Report) ceedings of the 8th international workshop on mining scientific
(Vol. 1, p. 12). Center for Computational Language and Edu- publications (pp. 75–83). Association for Computational
cation Research, University of Colorado Boulder, Institute of Linguistics.
Cognitive Science. Lu, Z., Du, P., & Nie, J. Y. (2020). VGCN-BERT: Augmenting BERT
Cohan, A., Ammar, W., Van Zuylen, M., & Cady, F. (2019). Struc- with graph embedding for text classification. In European con-
tural scaffolds for citation intent classification in scientific pub- ference on information retrieval (pp. 369–382). Springer.
lications. arXiv preprint: arXiv:1904.01608. Mikolov, T., Grave, E., Bojanowski, P., Puhrsch, C., & Joulin, A.
de Andrade, C. M. V., & Gonçalves, M. A. (2020). Combining repre- (2017). Advances in pre-training distributed word representa-
sentations for effective citation classification. In Proceedings of tions. arXiv preprint arXiv:1712.09405.
the 8th international workshop on mining scientific publications Oesterling, A., Ghosal, A., Yu, H., Xin, R., Baig, Y., Semenova, L., &
(pp. 54–58). Association for Computational Linguistics. Rudin, C. (2021). Multitask learning for citation purpose classi-
Dong, C., & Schäfer, U. (2011). Ensemble-style self-training on cita- fication. arXiv preprint arXiv:2106.13275.
tion classification. In Proceedings of the 5th international joint Pennington, J., Socher, R., & Manning, C. D. (2014). GloVe: Global vec-
conference on natural language processing (pp. 623–631). Asian tors for word representation. In Proceedings of the 2014 conference
Federation of Natural Language Processing. on empirical methods in natural language processing (EMNLP)
Drongstrup, D., Malik, S., Aljohani, N. R., Alelyani, S., (pp. 1532–1543). Association for Computational Linguistics.
Safder, I., & Hassan, S. U. (2020). Can social media usage of Pride, D., & Knoth, P. (2018). Peer review and citation data in pre-
scientific literature predict journal indices of AJG, SNIP and dicting university rankings, a large-scale analysis. In Interna-
JCR? An altmetric study of economics. Scientometrics, tional conference on theory and practice of digital libraries
125(2), 1541–1558. (pp. 195–207). Springer.
 23301643, 2023, 10, Downloaded from [Link] by University Of Southampton, Wiley Online Library on [04/03/2024]. See the Terms and Conditions ([Link] on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
SAFDER ET AL. 1239

Safder, I., Batool, H., Sarwar, R., Zaman, F., Aljohani, N. R., Transactions on Knowledge and Data Engineering, 32(10),
Nawaz, R., Gabere, M., & Hassan, S. U. (2022). Parsing AUC 1881–1896.
result-figures in machine learning specific scholarly documents Valenzuela, M., Ha, V., & Etzioni, O. (2015). Identifying meaningful
for semantically-enriched summarization. Applied Artificial citations. In Workshops at the twenty-ninth AAAI conference on
Intelligence, 36(1), 2004347. artificial intelligence. Association for the Advancement of Artifi-
Safder, I., & Hassan, S. U. (2019). Bibliometric-enhanced informa- cial Intelligence.
tion retrieval: A novel deep feature engineering approach for Wang, M., Leng, D., Ren, J., Zeng, Y., & Chen, G. (2019). Sentiment
algorithm searching from full-text publications. Scientometrics, classification based on linguistic patterns in citation context.
119(1), 257–277. Current Science, 117(4), 606.
Safder, I., Mahmood, Z., Sarwar, R., Hassan, S. U., Zaman, F., Zhu, X., Turney, P., Lemire, D., & Vellino, A. (2015). Measuring
Nawab, R. M. A., Bukhari, F., Rabeeh, A. A., Alelyani, S., academic influence: Not all citations are equal. Journal of the
Aljohani, N. R., & Nawaz, R. (2021). Sentiment analysis Association for Information Science and Technology, 66(2),
for Urdu online reviews using deep learning models. Expert 408–427.
Systems, 38, e12751.
Small, H. (2018). Characterizing highly cited method and non-
method papers using citation contexts: The role of uncertainty.
How to cite this article: Safder, I., Ali, M.,
Journal of Informetrics, 12(2), 461–480.
Aljohani, N. R., Nawaz, R., & Hassan, S.-U. (2023).
Teufel, S., Siddharthan, A., & Tidhar, D. (2006). Automatic classifi-
cation of citation function. In Proceedings of the 2006 conference Neural machine translation for in-text citation
on empirical methods in natural language processing (pp. 103– classification. Journal of the Association for
110). Association for Computational Linguistics. Information Science and Technology, 74(10),
Tuarob, S., Kang, S. W., Wettayakorn, P., Pornprasit, C., Sachati, T., 1229–1240. [Link]
Hassan, S. U., & Haddawy, P. (2019). Automatic classification
of algorithm citation functions in scientific literature. IEEE
23301643, 2023, 10, Downloaded from [Link] by University Of Southampton, Wiley Online Library on [04/03/2024]. See the Terms and Conditions ([Link] on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
SAFDER ET AL.

Netwrok architecture of the designed LSTM

A P P EN D I X A

FIGURE A1
model
1240