JMIR INFODEMIOLOGY Bayani et al

Original Paper

Transformer-Based Tool for Automated Fact-Checking of Online

Health Information: Development Study

Azadeh Bayani1,2, MSc; Alexandre Ayotte1,2*, MSc; Jean Noel Nikiema1,2,3*, PhD
1
Laboratoire Transformation Numérique en Santé, LabTNS, Montreal, QC, Canada
2
Centre de recherche en santé publique, Université de Montréal et CIUSSS du Centre-Sud-de-l’Île-de-Montréal, Montreal, QC, Canada
3
Department of Management, Evaluation and Health Policy, School of Public Health, Université de Montréal, Montreal, QC, Canada
*
these authors contributed equally

Corresponding Author:
Azadeh Bayani, MSc
Laboratoire Transformation Numérique en Santé, LabTNS
7101 Av. du Parc, Montréal,
Montreal, QC, H3N 1X9
Canada
Phone: 1 4389980241
Email: [Link]@[Link]

Abstract
Background: Many people seek health-related information online. The significance of reliable information became particularly
evident due to the potential dangers of misinformation. Therefore, discerning true and reliable information from false information
has become increasingly challenging.
Objective: This study aimed to present a pilot study in which we introduced a novel approach to automate the fact-checking
process, leveraging PubMed resources as a source of truth using natural language processing transformer models to enhance the
process.
Methods: A total of 538 health-related web pages, covering 7 different disease subjects, were manually selected by Factually
Health Company. The process included the following steps: (1) using transformer models of bidirectional encoder representations
from transformers (BERT), BioBERT, and SciBERT, and traditional models of random forests and support vector machines, to
classify the contents of web pages into 3 thematic categories (semiology, epidemiology, and management), (2) for each category
in the web pages, a PubMed query was automatically produced using a combination of the “WellcomeBertMesh” and “KeyBERT”
models, (3) top 20 related literatures were automatically extracted from PubMed, and finally, (4) the similarity checking techniques
of cosine similarity and Jaccard distance were applied to compare the content of extracted literature and web pages.
Results: The BERT model for the categorization of web page contents had good performance, with F1-scores and recall of 93%
and 94% for semiology and epidemiology, respectively, and 96% for both the recall and F1-score for management. For each of
the 3 categories in a web page, 1 PubMed query was generated and with each query, the 20 most related, open access articles
within the category of systematic reviews and meta-analyses were extracted. Less than 10% of the extracted literature was
irrelevant; those were deleted. For each web page, an average of 23% of the sentences were found to be very similar to the
literature. Moreover, during the evaluation, it was found that cosine similarity outperformed the Jaccard distance measure when
comparing the similarity between sentences from web pages and academic papers vectorized by BERT. However, there was a
significant issue with false positives in the retrieved sentences when compared with accurate similarities, as some sentences had
a similarity score exceeding 80%, but they could not be considered similar sentences.
Conclusions: In this pilot study, we have proposed an approach to automate the fact-checking of health-related online information.
Incorporating content from PubMed or other scientific article databases as trustworthy resources can automate the discovery of
similarly credible information in the health domain.

(JMIR Infodemiology 2025;5:e56831) doi: 10.2196/56831

KEYWORDS
fact-checking automation; transformers; infodemic; credible health information; machine learning; automated; online health
information; misinformation; natural language processing; epidemiology; health domain
[Link] JMIR Infodemiology 2025 | vol. 5 | e56831 | p. 1
(page number not for citation purposes)
XSL• FO
RenderX
 JMIR INFODEMIOLOGY Bayani et al

journalist-crafted, gold-standard explanations designed to

Introduction substantiate the fact-check labels assigned to each claim [6,18].
With rapid progressions in the digital age, and the vast While synthetic datasets provide valuable contributions to
dissemination of textual information available online, the advancing automatic fact-checking efforts, they cannot fully
likelihood of coming across misinformation has surged [1,2]. address real-world challenges, particularly the need for real-time,
Misinformation refers to information that is untrue, incorrect, dynamic information [23]. Therefore, there is a need that claims
or deceptive in nature [3]. It is prevalent across various domains, and their associated evidence to be automatically extracted [24].
with social media being a particularly prominent source [4]. A study [25] developed a Large Language Model called
Indeed, many people seek health-related topics on modern TrumorGPT, which addresses limitations in fact-checking by
platforms and websites available online [5]. Inaccurate incorporating retrieval-augmented generation and using
health-related information, however, poses an even greater risk, continually updated knowledge graphs. This approach uses
as it can directly impact lives [6,7]. Health misinformation is few-shot learning, knowledge graph construction, and semantic
considered “a health-related claim or information which is not reasoning, which enhances the model’s ability to handle
correct due to a lack of scientific evidence or knowledge” [4,8]. fact-checking tasks effectively. Another recent survey [12]
The importance of trustworthy online health information became explored automated techniques for predicting the veracity of
particularly clear during the COVID-19 pandemic, which claims, relying on natural language processing, knowledge
triggered a new crisis known as the COVID-19 infodemic. An representation, and databases. This study identified common
infodemic refers to the excessive spread of false or misleading challenges in fact-checking research and emphasized the
information across both digital and physical spaces [9] causing importance of information retrieval and knowledge
confusion and detrimental outcomes, as it underscores the representation, particularly due to the rapid emergence of new
potential risks posed by inaccurate or deceptive information to claims.
individuals [3,10]. The infodemic often manifests across 4 key Therefore, a key element of fact-checking involves identifying
areas: scientific research, policy and health care practice, news credible sources, and for health information, leveraging
outlets, and social media platforms [11]. As a result, up-to-date scientific literature is essential as it is widely regarded
distinguishing between true and reliable information and as 1 of the most trustworthy references [26]. Indeed, numerous
falsehoods has become increasingly challenging. The platforms and databases provide access to health-related and
labor-intensive process of manually verifying information scientific literature, including Google Scholar, PubMed,
specifically in health-related fields demands expert oversight ScienceDirect, and Web of Science, among others. These
and consumes significant time [4,9,12]. Therefore, it is crucial databases can be used as a reliable source for the automation
to establish an automated fact-checking process to help users of all the processes.
identify the accuracy of health-related information available
online. Numerous organizations have established guidelines to aid users
in identifying trustworthy claims [27,28] where time-consuming
The fact-checking process involves evaluating the truthfulness manual recognition plays an important role in the process. In
of information and consists of 3 key tasks: claim detection, this pilot study, we proposed a novel automated evidence-based
evidence retrieval, and claim verification [12]. The first 2 tasks fact-checking approach that aims to identify and confirm
can be considered as factual verification, while the third focuses accurate, truthful information using scientific literature and
on assessing the accuracy of claims, which involves research databases as sources of truth. This exploratory
distinguishing reliable information from falsehoods to establish evaluation highlights how using this approach may help users
their factual validity [13]. measure the extent of confidence in a web page and make
Several studies have explored automating the fact-checking informed decisions about accepting the health-related
process, primarily focusing on misinformation in the form of information of a website. Thus, the objective was to assess the
fake news on websites [4,14,15] or social media [2,7,16-18]. truthfulness of health-related information through an
These studies have generated synthetic datasets as the gold evidence-based approach, without creating a synthetic database
standard to facilitate the automation of evidence-based of claims-evidence but leveraging PubMed as a reliable source
fact-checking. Thus, they compiled datasets comprising of fine-grained and up-to-date health-related information.
information or claims along with their corresponding evidence
from trusted sources. Models were then trained using these Methods
datasets to automate the fact-checking process [7,10,15,17-20].
Approximately 1000 web pages were provided by Factually
To create a database of verified claims, they used methods such
Health company on January 31, 2023. This company specializes
as modifying phrases from Wikipedia [20], manual selection
in identifying reliable health-content websites [29]. The web
of quotation sentences and handpicking of claims from health
pages were selected through random sampling within various
news sites [14,15,21], and automatic selection of verified claims
disease categories to ensure a balanced dataset while minimizing
that were manually done by experts of journalists from
the risk of overrepresentation of any single category. This
fact-checking websites [10]. For example, the FEVER dataset,
approach accounted for variations in the number of available
generated by modifying sentences taken from Wikipedia,
websites across disease categories. The web pages then
consisted of 185,400 claims [22]. PUBHEALTH is another
underwent manual cleaning. Redundant pages were removed,
dataset containing false, true, unproven, and a mixture of
and those unsuitable for research were excluded based on the
health-related claims. The dataset also had a column containing

[Link] JMIR Infodemiology 2025 | vol. 5 | e56831 | p. 2

(page number not for citation purposes)
XSL• FO
RenderX
 JMIR INFODEMIOLOGY Bayani et al

following criteria: pages primarily featuring video content, The Content Categories
pages related to clinical studies, pages resembling anecdotes To compare web page content with materials from the scientific
rather than factual health information, or pages that restricted literature database, it was essential to categorize the content,
data extraction by Python (Python Software Foundation) ensuring that comparisons were made within the relevant
libraries. subject. Three distinct thematic categories have been identified
After this process, a dataset comprising 538 web pages was for analysis: epidemiology, semiology, and management. In the
finalized. These web pages represented a diverse range of epidemiology category, we included all sentences related to the
diseases, including arthritis (81 pages), chronic obstructive statistics of a disease, the population, the frequencies, the causes,
pulmonary disease (79 pages), COVID-19 (66 pages), the risk assessment of the disease, and all public health-related
hypertension (66 pages), lung cancer (70 pages), prostate cancer information about the disease (eg, as of 2014, the global
(66 pages), and diabetes (110 pages). prevalence rate of rheumatoid arthritis was about 0.24%). In
the semiology category, we considered all sentences related to
The selection of diverse disease categories was intended to signs (eg, high blood pressure is another sign of the disease)
minimize potential bias in the analysis. However, our previous and symptoms (eg, this disease has symptoms such as pain,
study demonstrated that the selected diseases did not discomfort, weakness, fatigue). Finally, for the management
significantly impact classification results [29]. Using the URLs category, we considered all the sentences linked to therapeutic
of each web page, the content was extracted as text files using approach (eg, drug treatment and surgical intervention,
the “justext” library in Python, to remove additional links and prevention, and the element of paraclinical diagnosis of diseases
extraneous content from websites, such as navigation links, (eg, a complete medical examination carried out by a doctor
headers, and footers. can better determine if a person has chronic obstructive
The process included the following three steps: (1) Classification pulmonary disease and the degree of severity of the disease).
of web page content into 3 thematic categories, semiology,
Manual Annotation and Model Development
epidemiology, and management by evaluating various
transformer models, including bidirectional encoder Two authors (AB and AA) independently annotated 200 web
representations from transformers (BERT), SciBERT, and pages on a sentence-by-sentence basis considering the 3
BioBERT, as well as traditional models such as random forest categories of epidemiology, semiology, management, and neutral
(RF) and support vector machine (SVM), (2) automating the until reaching a roughly balanced amount of data across all
creation of PubMed queries combining “WellcomeBertMesh” classes [32]. We used the Cohen κ score to assess the agreement
and “KeyBERT” models, (3) automatic extraction of top 20 between the 2 reviewers AB and AA). Any discrepancies were
related literatures from PubMed, and (4) applying similarity resolved by the third author (JNN).
checking techniques of cosine similarity and Jaccard distance Neutral sentences were those that did not correspond to any of
to compare the content of extracted literature and web pages the defined thematic categories. Table 1 shows the distribution
vectorized using BERT tokenizer. As a reliable source of truth, of sentences for each category. The portable serverless text
PubMed was a suitable choice to find evidence for health-related annotation tool of MedTator-1.3-11 [33] was used for the
claims. PubMed, an open-source platform dedicated to annotation process. A total of 3 transformer models of BERT,
facilitating searches and retrieval of health-related literature, SciBERT, and BioBERT were used to classify the sentences
encompasses over 36 million papers [30]. into the 4 mentioned categories. The BERT model has
Classification of Web Page Contents demonstrated superior performance in several text classification
tasks [29,34,35]. SciBERT is an extension of BERT and is
One of the necessary stages before determining the veracity of trained on a vast corpus of scientific literature spanning multiple
a claim or information is to detect the sentences that need to be domains [36] and BioBERT is pretrained using an extensive
verified [31]. These claims are crucial to the content’s main corpus comprising PubMed abstracts (PubMed) and full-text
point but require verification through an annotation schema and articles from PubMed Central [37]. We have also conducted a
developing a benchmark for automated claim detection [14,31]. performance comparison between the transformer models and
To detect sentences that need to be verified, two major steps 2 traditional machine learning models: RF and SVM.
were taken: (1) the identification of 3 thematic categories of
content and (2) the classification of web page content according
to these categories.

Table 1. The distribution of classes.

Category Number of sentences
Neutral 3162
Semiology 851
Epidemiology 1171
Management 1066

[Link] JMIR Infodemiology 2025 | vol. 5 | e56831 | p. 3

(page number not for citation purposes)
XSL• FO
RenderX
 JMIR INFODEMIOLOGY Bayani et al

The “BertTokenizer” library has been used to tokenize the we applied the Bayesian optimization approach using the
incoming sentences, with the following parameters: We applied ‘BayesianOptimization’ library in Python. The hyperparameter
a maximum sequence length of 128 to standardize the size of tuning spaces are detailed in Table 2.
each input sentence. To optimize the model's hyperparameters,

Table 2. Hyper-parameter tuning search space.

Hyper-parameters Range Best trial
Learning rate 10–7, 10–2 3×10–5
Weight decay 10–5, 10–1 10-3
Number of epochs (1:5) 3
Batch size (8,16,32,64) 32

the foundation of the BioBert pretrained model [38]. Given that

Automating PubMed Query Generation our evidence for the websites primarily comprised health-related
Overview articles from PubMed, we selected this model. Its architecture
is rooted in the latest advancements in the biomedical field,
Literature extraction involved identifying scientific articles
prominently featuring Microsoft’s cutting-edge “PubMedBert”
within PubMed to support the process. To achieve this, the
as its core framework [38].
approach requires the formulation of a query by combining
keywords and Medical Subject Headings (MeSH) terms, which To enhance the accuracy of the subquery, the identified MeSH
can be extracted from web page content. This process included terms were initially organized according to their MeSH
three steps: (1) Automating PubMed subquery creation from categories to construct subsubqueries. The MeSH has a tree
MeSH terms and creating a subquery using the structure that is organized hierarchically, visually presenting
“WellcomeBertMesh” model, (2) Automating PubMed subquery descriptors in broader and narrower relationships. The top tier
creation from keywords using KeyBert model and creating a of the MeSH tree structure encompasses 19 comprehensive
subquery, and (3) Construction of the final query by combining categories. While these terms are not included in MeSH data
the different subqueries. maintenance and distribution, they can be used to search
PubMed by using the search term “category” [39]. Therefore,
Automating PubMed Subquery Creation Using MeSH
we have considered the MeSH terms under each head category
Terms Extracted by Transformers together using the “OR” operator in this subsubquery. Then,
All the MeSH terms were extracted from the text using a we constructed the subquery using the “AND” operator between
pretrained model of “WellcomeBertMesh,” which takes its extracted MeSH terms in different categories. The pseudo-code
inspiration from “BertMesh,” which undergoes the pretraining for this step is presented in Figure 1.
using the entire text of biomedical publications and is built upon

[Link] JMIR Infodemiology 2025 | vol. 5 | e56831 | p. 4

(page number not for citation purposes)
XSL• FO
RenderX
 JMIR INFODEMIOLOGY Bayani et al

Figure 1. MeSH (medical subject heading) subquery builder.

in extracting the key phrases [40], especially for long texts [41],
Automating PubMed Subquery Creation Using Key which aligns with our need of extracting the key phrases of the
Phrases Extracted by Transformers scientific papers. The extracted keywords were combined with
The key phrases from web page contents have been extracted the “AND” operator to create a subquery.
using the transformer model “KeyBERT” library, which is
Figure 2 shows the proposed pseudo-code to extract the
described in previous literature as having the best performance
keywords for the creation of the subquery.

[Link] JMIR Infodemiology 2025 | vol. 5 | e56831 | p. 5

(page number not for citation purposes)
XSL• FO
RenderX
 JMIR INFODEMIOLOGY Bayani et al

Figure 2. Key phrase extractor and subquery builder.

Figure 3 presents a comprehensive overview of the process used

Construction of the Final Query to construct the final PubMed query, summarizing the structure
The subqueries extracted from the preceding processes were and strategy behind its creation.
combined using the “OR” operator to construct the final query.

[Link] JMIR Infodemiology 2025 | vol. 5 | e56831 | p. 6

(page number not for citation purposes)
XSL• FO
RenderX
 JMIR INFODEMIOLOGY Bayani et al

Figure 3. Detailed process diagram for the development of the comprehensive final PubMed query.

irrelevant papers were removed and excluded from the final

Automating Related Literature Extraction process.
The final query was used to retrieve a compilation of articles,
from which the top open access 20 resulting papers were Similarity Detection and Fact-Checking
extracted. The “PMC_ids” of papers were extracted using the For the process of computing the similarity measure between
“Entrez” library of Python that provides integrated access to different sentences, for each disease, we randomly selected 5
PubMed Medline [42]. To evaluate the quality of our query web pages in our dataset. For each of the 3 predefined thematic
results, we conducted a comprehensive review of the obtained categories in a web page, 1 PubMed query was generated and
full-text papers. In our assessment of the extracted papers in with each query, the 20 most related, open access articles within
PubMed, those subjected to filtering within the systematic the type of systematic reviews and meta-analysis were extracted.
reviews and meta-analysis category exhibited more related The following steps were then carried out: (1) Categorizing the
papers to the subject of the research, compared with papers that extracted related literature content based on the 3 thematic
were not subject to such filtering. Consequently, we selected categories. This was necessary to analyze sentences (from
them to encompass a wider range of relevant articles. websites and scientific articles) that are relevant to the same
topics. (2) Comparing by thematic category, the content from
Finally, the automatically extracted papers were manually
scientific articles and web pages to identify similar sentences.
checked to be pertinent considering the title of the papers, the

[Link] JMIR Infodemiology 2025 | vol. 5 | e56831 | p. 7

(page number not for citation purposes)
XSL• FO
RenderX
 JMIR INFODEMIOLOGY Bayani et al

Finally, after conducting a manual evaluation of the identified Comparing the Content From Literature and Web Pages
similar sentences, we calculated the average number of to Identify Similar Sentences
categorized sentences for each randomly selected web page, as
For the sentence comparison, we used the BERT vectorizer to
well as the average number of credible sentences detected.
transform the texts into vectors. This allowed us to encode the
Credible sentences refer to those in the related literature that
semantic significance of sentences as numerical values,
demonstrated similarity with the sentences from the web pages.
facilitating the application of different similarity detection
Categorizing the Extracted Literature algorithms [43].
The more performant fine-tuned model on the web page contents Both scientific articles and web page sentences were transformed
was used to categorize literature contents into 3 thematic into vector representations, taking into account their respective
categories. This approach enabled us to facilitate a direct thematic categories. Subsequently, each web page sentence was
comparison between sentences sharing the same thematic compared with scientific article sentences of the same category
context. using the cosine similarity and Jaccard technique. A similarity
threshold of 87% was chosen to determine sentence selection,
ensuring that sentences with over 87% similarity were chosen.
Figure 4 shows the proposed pseudo-code for the
similarity-checking part.
Figure 4. Paper similarity detection.

For each disease, we randomly selected 5 web pages and similar sentences between a web page and its related reference
extracted both their related papers and similar sentences. It was papers was calculated.
due to the inherent variability and specificity of medical
information related to each disease. Diseases often exhibit
Ethical Considerations
unique characteristics, nuances, and clinical considerations. By This research relied solely on publicly accessible data and did
prioritizing diseases, we aimed to provide a more granular and not involve any human or animal participants, making it exempt
clinically relevant assessment of the similarity between the from the need for ethical approval. The study strictly adheres
sentences. The outcomes, comprising sentences from the web to established data privacy norms to prevent any compromise
pages and their corresponding similar sentences, underwent a of confidentiality or privacy. In addition, the project does not
manual verification by the authors to ensure semantic similarity include any direct involvement or interactions with individuals,
between them. Subsequently, the proportion of semantically thereby minimizing potential ethical issues. The University of

[Link] JMIR Infodemiology 2025 | vol. 5 | e56831 | p. 8

(page number not for citation purposes)
XSL• FO
RenderX
 JMIR INFODEMIOLOGY Bayani et al

Montreal’s Research Committee has carefully examined our The performance of transformer-based models (BERT,
methodology and affirmed that this study falls outside the scope BioBERT, and SciBERT) was compared to traditional machine
of Medical Research Involving. learning models (RF and SVM) for categorizing web page
content into four categories. BERT emerged as the most
Results effective model, consistently achieving superior precision, recall,
and F1-scores across all categories. Traditional models, in
This section elaborates on the results of each part of the contrast, demonstrated lower performance, particularly in terms
proposed pseudo-codes. of F1-scores, indicating limitations in balancing precision and
Classification of Web Page Contents recall effectively.
The annotation process for web page contents achieved a Cohen Table 3 illustrates the performance of the classification models
κ score of 87% among the 2 annotators (AA and AB), indicating used to classify the content of web pages. The performance
high agreement between the annotators and ensuring the matrix includes metrics such as precision, recall, and F1-score.
reliability of the data used for model evaluation.

Table 3. Performance evaluation of the BERT (Bidirectional Encoder Representations from Transformers) and machine learning models for web page
content classification across considered categories.

BERTa BioBERT SciBERT RFb SVMc

Classes Preci- Re- F1-score Preci- Re- F1-score Preci- Re- F1-score Preci- Re- F1-score Preci- Re- F1-score
sion call sion call sion call sion call sion call
Neutral 0.96 0.93 0.95 0.88 0.83 0.85 0.85 0.81 0.83 0.51 0.92 0.66 0.72 0.81 0.77
Semiology 0.91 0.94 0.93 0.81 0.81 0.81 0.77 0.79 0.78 0.96 0.05 0.09 0.71 0.59 0.64
Epidemiology 0.92 0.94 0.93 0.80 0.76 0.76 0.75 0.74 0.75 0.8 0.1 0.1 0.69 0.62 0.65
Management 0.95 0.96 0.96 0.83 0.89 0.89 0.83 0.87 0.85 0.59 0.58 0.59 0.74 0.73 0.74

a
BERT: Bidirectional Encoder Representations from Transformers.
b
RF: random forests.
c
SVM: support vector machines.

According to Table 3, among the transformer models, the BERT BERT model, indicating that they may not achieve a good
model had a promising performance with more than 93% recall balance between precision and recall. Therefore, the BERT
for neutral sentences, 94% for semiology and epidemiology, model was selected for the classification of the web page
and 96% for the management category. The model had an contents.
F1-score of 95% for neutral sentences, 93% for semiology and
The confusion matrix for the BERT model is shown in Figure
epidemiology, and 96% for management. The model had 96% 5, providing a detailed visualization of its classification
precision for neutral sentences, 91% for semiology, 92% for performance across the different categories.
epidemiology, and 95% for management. Also, traditional
models did not have high performance, the precision values for Figure 5 shows the confusion matrix for the BERT classifier,
both RF and SVM were relatively low in some classes, which correctly classified 0.93 of the neutral sentences, 0.94
indicating a high rate of false positives. Also, the F1-scores for for both the semiology and epidemiology sentences, and 0.96
both RF and SVM were generally lower compared with the for management sentences as true positives.

[Link] JMIR Infodemiology 2025 | vol. 5 | e56831 | p. 9

(page number not for citation purposes)
XSL• FO
RenderX
 JMIR INFODEMIOLOGY Bayani et al

Figure 5. Bidirectional encoder representations from transformers model performance: confusion matrix for the classification of web page sentences
into 3 thematic categories.

Similarity Detection and Fact-Checking

Automating PubMed Query Generation
Figure 6 illustrates the average percentage of credible
To extract relevant literature for the web pages categorized
information found in the 5 randomly selected web pages
thematically, a PubMed query was generated for each of the 7
categorized by related diseases. Credible information is defined
diseases. Each query retrieved the 20 most related papers. The
as sentences in the web pages that were successfully matched
titles of the retrieved papers were manually evaluated, and less
with corresponding sentences in PubMed articles.
than 10% were deemed irrelevant, demonstrating the
effectiveness of the generated queries. These irrelevant articles On average, 23% of the sentences on each web page were
were excluded from further analysis. identified as similar to statements in the scientific literature.
While this demonstrates the potential of the system to detect
This result highlights the utility of using MeSH terms and key
credible content, a significant challenge arose with false
phrases in constructing PubMed queries, which efficiently
positives. Some sentences achieved a similarity score exceeding
yielded pertinent literature. The generated weblinks for
80% but were semantically dissimilar upon closer inspection.
accessing the papers followed the format:
“[Link] with PMIDs obtained
directly from the PubMed queries.
Figure 6. The average number of credible sentences on web pages (red line) versus the average number of all sentences on each web page (blue line).
COPD: chronic obstructive pulmonary disease.

[Link] JMIR Infodemiology 2025 | vol. 5 | e56831 | p. 10

(page number not for citation purposes)
XSL• FO
RenderX
 JMIR INFODEMIOLOGY Bayani et al

For instance, the following sentences from an extracted paper the contents into credible or non-credible information, however,
and a web page had a similarity score of 88% yet conveyed the decision is more complex since there may be several
different meanings: ambiguities in the sentences. In addition, several parts of the
process depend on human judgment, which needs further
1. “Previous studies have documented residual symptoms that
research in the area. Building on this groundwork, our study
continue 12 weeks after the onset of acute COVID-19, known
applied a BERT-based classification approach to detect health
as post-acute or long COVID-19.”
information requiring verification and automatically proposing
2. “The acute phase of COVID itself can last for up to 14 days.” a sentence for this process. Previous studies relied on reviewer
selections to develop claim and evidence datasets, lacking
This highlights the need for more sophisticated approaches to
attempts to automate claim identification with real-world
accurately distinguish between syntactic similarity and genuine
resources [17,18,45].
semantic alignment.
In addition, rather than constructing a manual reference dataset
As an illustrative example, for the rheumatoid arthritis category,
as the evidence for verifiable sentences, we leveraged the
we randomly selected 5 web pages, each containing an average
PubMed database as our source of truth. We automated the
of 27 sentences distributed across 3 thematic categories:
detection of evidence for claims made on web pages in an
epidemiology, semiology, and management (represented by the
unsupervised approach, streamlining the verification process.
blue line). Among these, an average of 7 sentences per web
This aligns with previous studies [21,26] that used PubMed
page were deemed credible and successfully matched to
publications as evidence, using transformer models to generate
corresponding statements in the scientific literature (depicted
queries and retrieve documents from PubMed. We demonstrated
by the red line).
the effectiveness of using transformer models to extract MeSH
terms and key phrases from web page content, enabling the
Discussion efficient generation of PubMed queries. This approach facilitated
Principal Findings the retrieval of related articles from scientific references without
requiring supervision. According to a previous study [14], to
In the present pilot study, our objective was to automate aspects verify the veracity of the claims, it is crucial to translate them
of the fact-checking process for online health information. While into queries against the reference databases. However, other
previous research [21,26] has explored automation in various studies [6,20,22] created a knowledge database as the references
stages of fact-checking, such as evidence retrieval or claim to compare with the claims. Notably, Sarrouti et al [6]
identification, this pilot serves as an initial step toward achieving introduced a dataset comprising evidence-claim pairs, manually
full automation in the fact-checking process. Our approach annotated as SUPPORT, REFUTE, and NEUTRAL. They used
includes the automation of identifying verifiable sentences BERT-based models to create a realistic testing ground for
through a classification process. Notably, our study used a evidence-based fact-checking systems.
fine-tuned BERT model, which exhibited notable efficacy in
categorizing health-related sentences. Although BioBERT and To assess the alignment between claim sentences and extracted
SciBERT models have been reported to outperform BERT in references, we measured their similarity, a practice supported
various downstream tasks [36,37], in our investigation, the by [46]. This study underscores the necessity for a model in
BERT model demonstrated superior performance. This claim verification to measure the semantic similarity between
discrepancy could be attributed to BERT training on claims and verified factual knowledge or references. To compare
general-purpose texts, such as Wikipedia or Book Corpus [35], the semantic similarity, we used a transformer-based
which align more closely with the content of websites targeted representation that converted the textual content into vectorial
at general populations. In contrast, BioBERT and SciBERT are representation, allowing us to capture the contextual nuances
trained on more specialized texts, such as scientific publications of each sentence consistent with previous approaches [19,43,47].
[36,37]. This approach is more efficient and produces semantically richer
sentence representations than simply averaging the vectors of
Previous research [14,31,44] has shown that the identification words that appear in each sentence, and facilitates the similarity
of claim-worthy sentences or the recognition of key information detection for the algorithms [48]. We successfully identified
needing verification from reliable sources is a fundamental first factual evidence for 23% of the health-related information
step in automating the fact-checking process akin to our extracted from web pages, indicating the complexity inherent
approach. This process is commonly structured as a text in health information. Further research is required to enhance
classification task. The previous studies used human annotators contextual comparison between claims and verified references.
[44] or crowdsourcing [31] to tag claim-worthy sentences and Also, the cosine similarity outperformed the Jaccard distance
trained machine learning models to classify them. A previous measure for comparing the claims and evidence in this study,
study [14] focused on detecting claims within news and public which is different from the previous study [4], as they reported
information, assigning each sentence a likelihood score for that the Jaccard distance was better at the similarity selection
containing significant factual claims. Also, automating the measure. The reason may be due to differences in the nature of
fact-checking process is far from straightforward, as it the datasets in the 2 studies.
necessitates the utilization of artificial intelligence tools to
struggle with the complexity of text and context [10]. Studies
often considered the problem as a binary classification to split

[Link] JMIR Infodemiology 2025 | vol. 5 | e56831 | p. 11

(page number not for citation purposes)
XSL• FO
RenderX
 JMIR INFODEMIOLOGY Bayani et al

Limitations quality (eg, journal quality, impact factor for the domain) and
This study had several limitations. First, we faced a challenge the contemporaneity of the information (eg, date of publication,
in identifying sentences within the papers that closely matched retracted).
the content of the web pages. Numerous methods have been The retrieved articles may vary in quality, ranging from
devised to tackle this issue [19,43,46]; however, a high-impact studies to potentially outdated or retracted articles
comprehensive consideration of the complete meaning of that could influence the reliability of the fact-checking process
sentences requires further investigation. In addition, 77% of the and the conclusions drawn from matched content. Addressing
sentences did not have matching counterparts in the academic these characteristics within an automated process remains a key
literature that we retrieved. Regarding this proportion, 2 possible challenge. In our previous research, the credibility of the sources
assumptions can be made: either the sentences themselves were was automatically assessed [29]. In this study, while we evaluate
not valid or the algorithm was unable to locate their related comparability with scientific articles, developing a credibility
counterparts. Another potential reason could be that the scoring strategy for these articles is also necessary. Combining
sentences, though addressing a common subject such as the an algorithm that evaluates website credibility and assigns a
same medical condition, exhibited variations in meaning or credibility score to scientific articles with 1 that determines
contextual interpretation. Consequently, it would be premature truthfulness could significantly enhance the effectiveness of
to assert that these unmatched sentences are inherently not fact-checking. These models can change the structure of
credible, given the vast volume of published papers that renders sentences and may improve the possibility of finding more
comprehensive verification computationally infeasible. similar sentences. Finally, while the process could not be
Expanding the number of selected papers for comparison could automated entirely since each step needed human supervision
therefore increase the likelihood of identifying additional for the results, the suggested techniques have the potential to
relevant sentences in the literature. Nonetheless, quantifying substantially alleviate the human effort required to locate valid
the proportion of credible sentences offers valuable insights to information.
aid users in their trust assessment.
Conclusions
It is worth acknowledging that authors in the realm of
Our approach aimed to empower users in the decision-making
health-related data often simplify and rephrase content to cater
process regarding the truthfulness of information by providing
to their target audience, making it more challenging to identify
relevant evidence and enabling informed judgments. As a pilot,
credible references for their statements. Therefore, the
this research serves as an initial step toward exploring the
researchers propose exploring other models such as text
feasibility of automating fact-checking processes in health
generation models as potential solutions to address this particular
information. Specifically, the methods presented here could be
challenge including WordNet or sequence-to-sequence
applied to create tailored fact-checking workflows for specific
(Seq2Seq) models.
disease areas, such as diabetes, arthritis, or cancer, which were
A second limitation was the sample size of the academic papers among the categories included in this study. For instance,
used in the comparison. Due to the extensive volume of thematic categorization (eg, management and epidemiology)
health-related publications, the assessment was limited to a could improve the precision and relevance of fact-checking
selection of 20 papers. Expanding this scope to include more tools in health care contexts. Using state-of-the-art models such
papers per content type could enhance the discovery of factual as transformers may improve the performance of the model
evidence in PubMed publications. Thus, further investigation since the BERT embedding captures the meaning of the
into paper retrieval approaches is recommended. sentences [49]. The investigation also revealed that incorporating
PubMed publications as a trustworthy resource can enhance the
A third limitation was that, although the thematic categorization
discovery of similar credible information as evidence. Finally,
of web page content, such as epidemiology, semiology, and
while the process could not be entirely automated and required
management, ensured that the generated PubMed queries were
human supervision, the suggested techniques demonstrate
more precise and contextually relevant, the need for quality
significant potential for integration into fact-checking tools.
assessment of the extracted PubMed articles remains evident.
This integration could reduce the effort required to validate
While our method provides users with essential information to
health information, ultimately increasing accessibility and
assess the accuracy of health information, the ultimate
reliability for end-users. Future work should focus on expanding
determination of its truthfulness may depend on individual
the dataset and testing the approach in real-world scenarios to
judgment, expert evaluation, source credibility, scientific article
further refine its applicability across various health domains.

Acknowledgments
The authors express their sincere gratitude to Factually Health Company for generously providing a factual dataset covering 7
diseases and to IVADO Labs for their support.

Data Availability
The datasets generated and analyzed during this study are available from the corresponding author on reasonable request.

[Link] JMIR Infodemiology 2025 | vol. 5 | e56831 | p. 12

(page number not for citation purposes)
XSL• FO
RenderX
 JMIR INFODEMIOLOGY Bayani et al

Conflicts of Interest
This research was funded by the Mitacs acceleration program in partnership with Factually Health Company, and the IVADO
Funding for Collaborative Research in Data Science to Serve Sustainable Development.

References
1. Cuan-Baltazar JY, Muñoz-Perez MJ, Robledo-Vega C, Pérez-Zepeda MF, Soto-Vega E. Misinformation of COVID-19 on
the internet: infodemiology study. JMIR Public Health Surveill. Apr 09, 2020;6(2):e18444. [FREE Full text] [doi:
10.2196/18444] [Medline: 32250960]
2. Himelein-Wachowiak M, Giorgi S, Devoto A, Rahman M, Ungar L, Schwartz HA, et al. Bots and misinformation spread
on social media: implications for COVID-19. J Med Internet Res. May 20, 2021;23(5):e26933. [FREE Full text] [doi:
10.2196/26933] [Medline: 33882014]
3. Lee DKL, Ramazan O. Fact-checking of health information: the effect of media literacy, metacognition and health information
exposure. J Health Commun. Jul 03, 2021;26(7):491-500. [doi: 10.1080/10810730.2021.1955312] [Medline: 34292862]
4. Barve Y, Saini JR. Healthcare misinformation detection and fact-checking: a novel approach. IJACSA. 2021;12(10). [doi:
10.14569/IJACSA.2021.0121032]
5. Ecker UKH, Lewandowsky S, Cook J, Schmid P, Fazio LK, Brashier N, et al. The psychological drivers of misinformation
belief and its resistance to correction. Nat Rev Psychol. Jan 12, 2022;1(1):13-29. [doi: 10.1038/s44159-021-00006-y]
6. Sarrouti M, Abacha AB, Mrabet Y, Demner-Fushman D. Evidence-based fact-checking of health-related claims. Association
for Computational Linguistics; 2021. Presented at: Findings of the Association for Computational Linguistics: EMNLP
2021; 2025 January 22:3499-3512; Punta Cana, Dominican Republic. [doi: 10.18653/v1/[Link]-emnlp.297]
7. Sumayyia MD, Al-Madaney MM, Almousawi FH. Health information on social media. Perceptions, attitudes, and practices
of patients and their companions. Saudi Med J. Dec 2019;40(12):1294-1298. [FREE Full text] [doi:
10.15537/smj.2019.12.24682] [Medline: 31828284]
8. Suarez-Lledo V, Alvarez-Galvez J. Prevalence of health misinformation on social media: systematic review. J Med Internet
Res. Jan 20, 2021;23(1):e17187. [FREE Full text] [doi: 10.2196/17187] [Medline: 33470931]
9. Hang CN, Yu PD, Chen S, Tan CW, Chen G. MEGA: Machine Learning-Enhanced Graph Analytics for infodemic risk
management. IEEE J Biomed Health Inform. Dec 2023;27(12):6100-6111. [doi: 10.1109/JBHI.2023.3314632] [Medline:
37713230]
10. Altuncu E, Nurse J, Bagriacik M, Kaleba S, Yuan H, Bonheme L, et al. aedFaCT: scientific fact-checking made easier via
semi-automatic discovery of relevant expert opinions. arXiv:2305.07796. 2023. [doi: 10.36190/2023.27]
11. Eysenbach G. How to fight an infodemic: the four pillars of infodemic management. J Med Internet Res. Jun 29,
2020;22(6):e21820. [FREE Full text] [doi: 10.2196/21820] [Medline: 32589589]
12. Guo Z, Schlichtkrull M, Vlachos A. A survey on automated fact-checking. Transactions of the Association for Computational
Linguistics. 2022;10:178-206. [doi: 10.1162/tacl_a_00454]
13. Walter N, Cohen J, Holbert RL, Morag Y. Fact-checking: a meta-analysis of what works and for whom. Political
Communication. Oct 24, 2019;37(3):350-375. [doi: 10.1080/10584609.2019.1668894]
14. Hassan N, Arslan F, Li C, Tremayne M. Toward automated fact-checking: detecting check-worthy factual claims by
ClaimBuster. ACM; 2017. Presented at: Proceedings of the 23rd ACM SIGKDD international conference on knowledge
discovery and data mining; 2017 August 13-17:1803-1812; Halifax NS Canada. [doi: 10.1145/3097983.3098131]
15. Samadi M, Mousavian M, Momtazi S. Deep contextualized text representation and learning for fake news detection.
Information Processing & Management. Nov 2021;58(6):102723. [doi: 10.1016/[Link].2021.102723]
16. Zhang J, Featherstone JD, Calabrese C, Wojcieszak M. Effects of fact-checking social media vaccine misinformation on
attitudes toward vaccines. Prev Med. Apr 2021;145:106408. [doi: 10.1016/[Link].2020.106408] [Medline: 33388335]
17. Zhong B. Going beyond fact-checking to fight health misinformation: A multi-level analysis of the Twitter response to
health news stories. International Journal of Information Management. Jun 2023;70:102626. [doi:
10.1016/[Link].2023.102626]
18. Akhtar M, Cocarascu O, Simperl E. PubHealthTab: a public health table-based dataset for evidence-based fact checking.
Association for Computational Linguistics; 2022. Presented at: Findings of the Association for Computational Linguistics:
NAACL 2022; 2025 January 22:1-16; Seattle, United States. [doi: 10.18653/v1/[Link]-naacl.1]
19. Martín A, Huertas-Tato J, Huertas-García Á, Villar-Rodríguez G, Camacho D. FacTeR-Check: semi-automated fact-checking
through semantic similarity and natural language inference. Knowledge-Based Systems. 2022. [doi:
10.1016/[Link].2022.109265]
20. Sathe A, Ather S, Le TM, Perry N, Park J. Automated fact-checking of claims from Wikipedia. European Language
Resources Association; 2020. Presented at: Proceedings of the Twelfth Language Resources and Evaluation Conference;
2025 January 22:6874-6882; Marseille, France.
21. Deka P, Jurek-Loughrey A, P D. Improved methods to aid unsupervised evidence-based fact checking for online heath
news. JDI. 2022;3(4):474-504. [doi: 10.26421/JDI3.4-5]

[Link] JMIR Infodemiology 2025 | vol. 5 | e56831 | p. 13

(page number not for citation purposes)
XSL• FO
RenderX
 JMIR INFODEMIOLOGY Bayani et al

22. Thorne J, Vlachos A, Christodoulopoulos C, Mittal A. FEVER: a large-scale dataset for fact extraction and VERification.
ACL Anthology. 2018:1-9. [doi: 10.18653/v1/w18-5501]
23. Das A, Liu H, Kovatchev V, Lease M. The state of human-centered NLP technology for fact-checking. Information
Processing & Management. Mar 2023;60(2):103219. [doi: 10.1016/[Link].2022.103219]
24. Nakov P, Corney D, Hasanain M, Alam F, Elsayed T, Barrón-Cedeño A, et al. Automated fact-checking for assisting human
fact-checkers.. arXiv. Preprint posted online on March 13, 2021. [doi: 10.48550/arXiv.2103.07769]
25. Hang CN, Yu PD, Tan CW. TrumorGPT: query optimization and semantic reasoning over networks for automated
fact-checking. IEEE; 2024. Presented at: 2024 58th Annual Conference on Information Sciences and Systems (CISS); 2024
March 13-15:1-6; Princeton, NJ, USA. [doi: 10.1109/ciss59072.2024.10480162]
26. Deka P, Jurek-Loughrey A, Deepak. Unsupervised keyword combination query generation from online health related
content for evidence-based fact checking. ACM; 2021. Presented at: iiWAS2021: The 23rd International Conference on
Information Integration and Web Intelligence; 2021 November 29:267-277; Linz Austria. [doi: 10.1145/3487664.3487701]
27. Boyer C, Baujard V, Nater T, Scherrer J, Appel R. HONcode: a standard to improve the quality of medical/health information
on the internet and HON's 5th survey on the use of internet for medical and health purposes. In: J Med Internet Res. 1999.
Presented at: 6th Internet World Congress for Biomedical Sciences (INABIS 2000); 2000 December 7-16; Canada. [doi:
10.2196/jmir.1]
28. Padhy S, Rathore SS. An attention‐based deep learning model for credibility assessment of online health information.
Computational Intelligence. Jul 24, 2023;39(5):832-859. [doi: 10.1111/coin.12596]
29. Bayani A, Ayotte A, Nikiema JN. Automated credibility assessment of web-based health information considering health
on the net foundation code of conduct (HONcode): model development and validation study. JMIR Form Res. 2023;7:e52995.
[FREE Full text] [doi: 10.2196/52995] [Medline: 38133919]
30. Giustini D, Barsky E. A look at Google Scholar, PubMed, and Scirus: comparisons and recommendations. J Can Health
Libr Assoc. Sep 01, 2005;26(3):85-89. [doi: 10.5596/c05-030]
31. Konstantinovskiy L, Price O, Babakar M, Zubiaga A. Toward automated factchecking. Digital Threats. Apr 15,
2021;2(2):1-16. [doi: 10.1145/3412869]
32. Fernando KRM, Tsokos CP. Dynamically weighted balanced loss: class imbalanced learning and confidence calibration
of deep neural networks. IEEE Trans Neural Netw Learn Syst. Jul 2022;33(7):2940-2951. [doi:
10.1109/TNNLS.2020.3047335] [Medline: 33444149]
33. He H, Fu S, Wang L, Liu S, Wen A, Liu H. MedTator: a serverless annotation tool for corpus development. Bioinformatics.
Mar 04, 2022;38(6):1776-1778. [FREE Full text] [doi: 10.1093/bioinformatics/btab880] [Medline: 34983060]
34. González-Carvajal S, Garrido-Merchán EC. Comparing BERT against traditional machine learning text classification..
arXiv. Preprint published online May 26, 2020. [doi: 10.48550/arXiv.2005.13012]
35. Koroteev MV. BERT: a review of applications in natural language processing and understanding.. arXiv. Preprint published
online March 22, 2021. [doi: 10.48550/arXiv.2103.11943]
36. Beltagy I, Lo K, Cohan A. SciBERT: a pretrained language model for scientific text. ACL Anthology. 2019:3615-3620.
[doi: 10.18653/v1/d19-1371]
37. Lee J, Yoon W, Kim S, Kim D, Kim S, So C, et al. BioBERT: a pre-trained biomedical language representation model for
biomedical text mining. Bioinformatics. 2020;36(4):1234-1240. [FREE Full text] [doi: 10.1093/bioinformatics/btz682]
[Medline: 31501885]
38. You R, Liu Y, Mamitsuka H, Zhu S. BERTMeSH: deep contextual representation learning for large-scale high-performance
MeSH indexing with full text. Bioinformatics. 2021;37(5):684-692. [FREE Full text] [doi: 10.1093/bioinformatics/btaa837]
[Medline: 32976559]
39. Dhammi IK, Kumar S. Medical subject headings (MeSH) terms. Indian J Orthop. Sep 2014;48(5):443-444. [FREE Full
text] [doi: 10.4103/0019-5413.139827] [Medline: 25298548]
40. Priyanshu A, Vijay S. arXiv. 2022. [doi: 10.48550/arXiv.2211.07499]
41. Giarelis N, Kanakaris N. A comparative assessment of state-of-the-art methods for multilingual unsupervised keyphrase
extraction. In: Artificial Intelligence Applications and Innovations. Cham. Springer International Publishing; 2021:635-645.
42. Schuler GD, Epstein JA, Ohkawa H, Kans JA. Entrez: molecular biology database and retrieval system. Methods Enzymol.
1996;266:141-162. [doi: 10.1016/s0076-6879(96)66012-1] [Medline: 8743683]
43. Qadrud-Din J, Rabiou A, Walker R, Soni R, Gajek M, Pack G, et al. Transformer based language models for similar text
retrieval and ranking.. arXiv. Preprint published online on May 10, 2020. [doi: 10.48550/arXiv.2005.04588]
44. Pathak A, Shaikh MA, Srihari R. Self-supervised claim identification for automated fact checking.. arXiv.. :213-227.
Preprint published online on February 3, 2021. [doi: 10.48550/arXiv.2102.02335]
45. Kotonya N, Toni F. Explainable automated fact-checking for public health claims. ACL Anthology. 2020:7740-7754. [doi:
10.18653/v1/[Link]-main.623]
46. Barve Y, Saini JR. Detecting and classifying online health misinformation with 'Content Similarity Measure (CSM)'
algorithm: an automated fact-checking-based approach. J Supercomput. 2023;79(8):9127-9156. [FREE Full text] [doi:
10.1007/s11227-022-05032-y] [Medline: 36644509]

[Link] JMIR Infodemiology 2025 | vol. 5 | e56831 | p. 14

(page number not for citation purposes)
XSL• FO
RenderX
 JMIR INFODEMIOLOGY Bayani et al

47. Vijjali R, Potluri P, Kumar S, Teki S. Two stage transformer model for COVID-19 fake news detection and fact checking..
arXiv. Preprint published online on November 26, 2020. [doi: 10.48550/arXiv.2011.13253]
48. Nabożny A, Balcerzak B, Wierzbicki A, Morzy M, Chlabicz M. Active annotation in evaluating the credibility of web-based
medical information: guidelines for creating training data sets for machine learning. JMIR Med Inform. 2021;9(11):e26065.
[FREE Full text] [doi: 10.2196/26065] [Medline: 34842547]
49. Jawahar G, Sagot B, Seddah D. What does BERT learn about the structure of language? HAL open scince; 2019. Presented
at: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics; 2019 July 28 to August
2:3651-3657; Florence (Italy). [doi: 10.18653/v1/p19-1356]

Abbreviations
BERT: bidirectional encoder representations from transformers
MeSH: medical subject heading
RF: random forest
SVM: support vector machines

Edited by T Mackey; submitted 27.01.24; peer-reviewed by P Deka, CN Hang, O Ismaila; comments to author 14.03.24; revised
version received 08.05.24; accepted 24.12.24; published 21.02.25
Please cite as:
Bayani A, Ayotte A, Nikiema JN
Transformer-Based Tool for Automated Fact-Checking of Online Health Information: Development Study
JMIR Infodemiology 2025;5:e56831
URL: [Link]
doi: 10.2196/56831
PMID: 39812653

©Azadeh Bayani, Alexandre Ayotte, Jean Noel Nikiema. Originally published in JMIR Infodemiology
([Link] 21.02.2025. This is an open-access article distributed under the terms of the Creative Commons
Attribution License ([Link] which permits unrestricted use, distribution, and reproduction
in any medium, provided the original work, first published in JMIR Infodemiology, is properly cited. The complete bibliographic
information, a link to the original publication on [Link] as well as this copyright and license information
must be included.

[Link] JMIR Infodemiology 2025 | vol. 5 | e56831 | p. 15

(page number not for citation purposes)
XSL• FO
RenderX