Explainable Hallucination Detection in Large

Language Model Responses using Consistency

Analysis and Antonym Probing

Abhirrami M, Keshav Kumar K R, Jamuna S, and Vinitha A

Department of Computer Science and Engineering

College of Engineering, Guindy
Anna University, Chennai, India

Abstract. Large Language Models (LLMs) have shown strong capabil-

ity in generating fluent, coherent, and context-aware natural language
responses across diverse applications. However, they often produce hal-
lucinated outputs responses that are grammatically correct but factu-
ally incorrect or logically inconsistent. This issue becomes critical as
LLMs are increasingly used in domains such as education, research, and
decision-support systems, where reliability and trust are essential. Hal-
lucinations typically arise due to ambiguous inputs, lack of grounding in
factual knowledge, and the probabilistic nature of language generation.
To address this problem, this paper proposes an explainable halluci-
nation detection framework that integrates semantic consistency anal-
ysis, antonym-based negation probing, and transformer-based contex-
tual representations. The system evaluates response reliability by ana-
lyzing variations when inputs are rephrased or logically negated. A su-
pervised classification model is used to distinguish hallucinated and non-
hallucinated responses, while Explainable Artificial Intelligence (XAI)
techniques highlight key features influencing predictions. The proposed
approach improves the reliability, interpretability, and practical usability
of LLM-generated responses.

Keywords: Hallucination Detection, Large Language Models, Explain-

able Artificial Intelligence, Semantic Consistency Analysis, Negation Prob-
ing, Transformer Models, Natural Language Processing.

1 Introduction
The rapid growth of Large Language Models (LLMs) has significantly
influenced the development of intelligent text-generation systems. Their
ability to understand prompts and produce meaningful, well-formed re-
sponses has led to widespread adoption in areas such as virtual assis-
tance, academic support, information retrieval, and automated writing.
Even with these advancements, one major limitation remains unresolved:
LLMs can generate statements that appear credible and fluent while lack-
ing factual correctness or logical validity. These misleading outputs, com-
monly referred to as hallucinations, create serious concerns when such
2 Abhi M. et al.

models are used in environments that demand dependable and verifiable

information.
This issue becomes increasingly important as language models are in-
tegrated into practical applications that affect learning, analysis, and
decision-making. Since LLMs rely on statistical language patterns rather
than explicit fact-checking, they may produce unreliable answers when
prompts are unclear, context is incomplete, or the generated response
is not sufficiently grounded in valid knowledge. Recent developments in
transformer-based text analysis offer useful ways to examine the trust-
worthiness of generated content. For instance, semantic consistency meth-
ods can compare whether a model preserves the same meaning when a
question is asked in different forms, while negation-based testing can ex-
pose contradictions hidden within the response behavior. At the same
time, explainable AI techniques make the detection process more trans-
parent by identifying the words or textual patterns that influence the
final decision. Bringing these methods together supports the design of
a more reliable hallucination detection framework that strengthens the
transparency, usability, and overall dependability of Large Language
Models.

2 Problem Statement
The increasing use of Large Language Models in diverse application ar-
eas has raised serious concerns about the factual correctness and logical
consistency of their generated responses. Although LLMs are capable
of producing fluent and contextually relevant text, hallucinated outputs
can negatively influence decision- making in sensitive domains such as
education, research, healthcare support, and automated analysis. Tradi-
tional methods used to verify response reliability often depend on exter-
nal knowledge retrieval, repeated prompting strategies, or multi- stage
validation mechanisms. These approaches may improve detection accu-
racy, but they also introduce additional computational cost and are not
always suitable for real-time or lightweight deployment settings.
Furthermore, many existing hallucination detection systems operate in
a black-box manner, offering limited explanation for why a particu-
lar response is identified as unreliable. This lack of interpretability re-
duces user confidence and restricts their usefulness in practical environ-
ments. In some cases, hallucinated responses may even appear consis-
tent across multiple generations, making them more difficult to detect
through simple verification techniques. Therefore, there is a strong need
for a lightweight, software-based hallucination detection frame- work that
can efficiently recognize unreliable outputs while also providing clear
and interpretable explanations. Addressing this challenge is essential for
support- ing the safe, transparent, and trustworthy use of Large Lan-
guage Models in real-world applications.

3 Literature Review
Early studies on hallucination detection mainly depended on rule-based
checking methods and manual inspection, which were often limited in
 Title Suppressed Due to Excessive Length 3

handling large-scale language model outputs. These conventional ap-

proaches lacked flexibility and frequently struggled to identify subtle
factual mistakes, semantic drift, or hidden logical inconsistencies in gen-
erated text.
Recent progress in transformer-based natural language processing has
enabled more effective analysis of model-generated responses through
contextual embed- dings, semantic similarity scoring, and consistency-
based evaluation. Deep learn- ing approaches have improved hallucina-
tion detection by capturing fine-grained relationships between prompts
and generated answers. However, many of these methods involve high
computational cost, making them difficult to apply in real-time environ-
ments or lightweight deployment settings.
At the same time, several research efforts have explored contradiction
detec- tion, self-consistency checking, and natural language inference to
identify unreli- able outputs produced by language models. Although
these techniques can be effective, they are often designed as separate
processes and provide limited insight into the reasons behind a detection
result. This reduced interpretability affects practical trust and usability
in real-world applications.
The reviewed literature indicates the need for an integrated, efficient,
and explainable framework that can detect hallucinated responses with
lower compu- tational overhead. The proposed hallucination detection
system addresses these gaps by combining semantic consistency analysis,
negation-based probing, and interpretable classification, thereby improv-
ing the reliability and transparency of Large Language Model outputs.

Table 1. Comparison of Hallucination Detection Approaches

Approach Technique Limitation

Rule-based Verification Keyword rules Low robustness
Knowledge-based Checking External knowledge sources High computation cost
Consistency-based Methods Rephrasing analysis Misses repeated errors
NLI-based Verification Entailment detection Limited explainability

4 Methodology
4.1 Overall Architecture Description
The proposed Explainable Hallucination Detection Framework is de-
signed as a modular, software-oriented architecture that identifies hallu-
cinated outputs pro- duced by Large Language Models through semantic
analysis and interpretable decision-making. At the core of the frame-
work, a user query and its correspond- ing model-generated response are
provided as inputs, which are then processed through multiple analytical
stages to determine whether the response is reliable or hallucinated. Since
the system is entirely software-based, it avoids dependence on external
hardware and remains suitable for lightweight deployment settings.
4 Abhi M. et al.

Fig. 1. Overall architecture of the proposed Explainable Hallucination Detection

Framework

In the first stage, the text processing pipeline prepares both the input
prompt and generated answer through normalization, tokenization, and
linguistic clean- ing to ensure consistent representation. The processed
text is then examined using semantic consistency analysis, where alter-
nate forms of the same question are compared with the original response
to evaluate response stability. In paral- lel, a negation probing module
generates logically inverted or antonym-based variations of the query in
order to identify contradictions and inconsistencies in the generated out-
put. These semantic and logical signals provide valuable indi- cators of
hallucination beyond simple surface-level matching.
The extracted text representations are further encoded using transformer-
based contextual models, which capture deeper semantic relationships
between the prompt and the generated response. These embeddings, to-
gether with con- sistency and negation-based features, are merged in
a unified feature fusion layer that supports robust classification. A su-
pervised learning model then analyzes the combined feature space to
determine whether the output should be labeled as hallucinated or non-
hallucinated. To improve transparency, an Explainable Arti- ficial In-
telligence module highlights influential words or response segments that
contribute most strongly to the final prediction.
The outputs of all modules are integrated into a final visualization and
deci- sion layer that presents both the classification result and its sup-
porting explana- tion to the user. This layer enables easier interpreta-
tion of detection outcomes and improves confidence in system behav-
ior. The complete architecture is im- plemented using lightweight NLP
and transformer-based software components, ensuring low computational
complexity and practical usability. By combining semantic consistency
checking, negation-based analysis, supervised classification, and explain-
ability, the proposed framework provides a reliable and interpretable so-
lution for hallucination detection in Large Language Model responses.

4.2 Design Rationale

The proposed framework is developed to improve the dependability and
trans- parency of responses generated by Large Language Models. Since
 Title Suppressed Due to Excessive Length 5

hallucinated content may appear fluent and convincing even when it is

incorrect, the system is designed to examine reliability through multi-
ple complementary signals rather than relying on a single verification
strategy. To achieve this, the framework adopts a lightweight and fully
software-based approach that performs semantic and logical analysis
without requiring external hardware or complex infrastruc- ture.
The semantic analysis module uses rephrase-based consistency checking
to measure whether a model preserves the same meaning when a query
is expressed in alternative forms. Alongside this, the negation probing
module evaluates how the response changes when the input is logically
inverted or modified using op- posite concepts, helping expose hidden
contradictions. Transformer-based con- textual embeddings are then used
to capture deeper semantic relationships be- tween the prompt and the
generated answer, enabling richer feature modeling for hallucination de-
tection.
By combining semantic consistency signals, negation-aware indicators,
and contextual text representations, the framework builds a more com-
plete basis for classification while keeping computational overhead man-
ageable. In addition, Explainable Artificial Intelligence techniques are
incorporated to highlight the words or response segments that most
strongly influence detection outcomes. This design supports accurate, in-
terpretable, and efficient hallucination detec- tion, while preserving mod-
ularity and allowing future extension without adding significant system
complexity.

4.3 Module Design and Explanation

Data Handling Module The data handling module is responsible for
main- taining the quality and consistency of the dataset used for training
and evalua- tion. It performs operations such as data cleaning, normal-
ization, label verifica- tion, and duplicate removal to reduce noise and
incorrect annotations that may affect model learning. In addition, appro-
priate dataset partitioning is carried out to create balanced training, val-
idation, and testing subsets. This module ensures that the downstream
detection framework is trained on reliable inputs and evaluated in an
unbiased manner, thereby improving robustness and overall prediction
performance.
Text Preprocessing Module The text preprocessing module converts
raw question–answer pairs into a structured representation that can be
processed effectively by transformer-based models. It applies operations
such as tokeniza- tion, lowercasing where required, padding, truncation,
and attention mask gen- eration to maintain uniform input formatting
across samples. These steps help the model preserve contextual depen-
dencies while handling variable-length text. By transforming unstruc-
tured textual data into model-ready input tensors, this module supports
efficient semantic analysis and stable contextual representation learning
during both training and inference.
Rephrase Consistency Analyzer The rephrase consistency analyzer
exam- ines whether a generated response preserves its meaning when the
6 Abhi M. et al.

same query is expressed in alternative forms. Instead of checking only the

original answer, this module compares outputs across reworded versions
of the input to measure se- mantic stability. If a model produces incon-
sistent answers for equivalent queries, the response may indicate halluci-
nation caused by semantic drift or unreliable reasoning. This module is
therefore important for detecting errors that cannot always be captured
through direct answer matching or surface-level evaluation methods.
Negation Probe Module The negation probe module is designed
to detect logical inconsistency in generated responses by constructing
negated or seman- tically opposite versions of the input query. The cor-
responding outputs are then analyzed to identify contradictions, reversal
errors, or unstable reasoning behav- ior. This module helps reveal hal-
lucinations that arise when a model fails to maintain logical coherence
under altered semantic conditions. By introducing a targeted contra-
diction analysis mechanism, the framework gains an additional layer of
reliability assessment beyond semantic similarity alone.
Transformer Feature Extraction Module The transformer feature
ex- traction module uses contextual embedding models to capture rich
semantic and syntactic information from the relationship between the
query and its generated response. These embeddings represent deeper
textual meaning in a compact and informative form, allowing the system
to move beyond keyword-level compari- son. Since transformer repre-
sentations are sensitive to context and token interde- pendencies, they
provide a strong foundation for distinguishing between reliable and hal-
lucinated outputs in complex natural language settings.
Feature Fusion and Classification Module This module combines
multiple signals including semantic consistency scores, contradiction-
based indicators, and transformer-derived embeddings into a single in-
tegrated feature representation. The fused feature space is then passed
to a supervised classification model that predicts whether the generated
response should be labeled as hallucinated or non-hallucinated. By ag-
gregating complementary evidence from several analyti- cal stages, this
module improves detection strength while preserving computa- tional
efficiency and keeping the overall framework lightweight.
Confidence Calibration Module The confidence calibration module
is in- troduced to improve the reliability of predicted probability scores
generated by the classifier. Instead of relying solely on raw output con-
fidence, this module adjusts the prediction scores so that they better
reflect actual uncertainty in the decision process. Calibrated confidence
values help users interpret the degree of trust that should be assigned to
each prediction and support more dependable decision-making in prac-
tical deployment settings.
Explainability Module The explainability module enhances trans-
parency by incorporating Explainable Artificial Intelligence techniques
such as LIME to show which words or textual segments most strongly
influence the hallucination detection outcome. Rather than returning
only a final label, this component provides interpretable evidence that
 Title Suppressed Due to Excessive Length 7

helps users understand the reasoning behind the model’s decision. This
improves trust, supports error analysis, and increases the usability of the
framework without introducing major architectural complex- ity.

User Interface Module The user interface module presents detection

re- sults, confidence values, and explanation outputs in a simple and un-
derstandable format for end users. It is designed to display whether a
response is hallucinated, how strongly the system supports that predic-
tion, and which textual elements contributed to the outcome. By mak-
ing the framework outputs accessible and easy to interpret, this module
supports practical interaction and improves the overall usability of the
proposed hallucination detection system.

5 Implementation Details

The proposed hallucination detection system is implemented in Python

using lightweight natural language processing components and transformer-
based mod- els for contextual text understanding. Question–answer pairs
are first processed through preprocessing routines that prepare them in a
standardized format suit- able for semantic analysis and model inference.
The framework then encodes these inputs into dense contextual represen-
tations that preserve both semantic meaning and linguistic dependencies
between the prompt and the generated re sponse.
To examine response reliability, the system performs rephrase-based consis-
tency checking by comparing answers produced for alternate versions of
the same question. Logical stability is further analyzed through nega-
tion probing, where negated or semantically reversed queries are used to
identify contradictions and reasoning errors. The semantic consistency
scores, contradiction indicators, and transformer-derived embeddings are
merged into a unified feature representa- tion that captures multiple as-
pects of hallucination behavior.
The final prediction stage uses a lightweight supervised classifier to deter-
mine whether the generated response should be marked as hallucinated
or non- hallucinated. Confidence calibration is then applied so that pre-
diction scores better reflect the actual certainty of the model. To improve
transparency, ex- plainability is incorporated through word-level impor-
tance analysis, allowing the system to highlight influential terms that
contribute to each decision. All mod- ules are integrated into a simple
user interface that displays predictions, confi- dence values, and sup-
porting explanations. The overall system runs efficiently in a standard
CPU-based software environment, making it practical for lightweight de-
ployment and real-time usage.

6 Dataset Description

The proposed hallucination detection framework is evaluated using the

FEVER v1.0 (Fact Extraction and VERification) dataset, which is a
8 Abhi M. et al.

widely used bench- mark for factual verification in natural language pro-
cessing. This dataset con- tains textual claims paired with evidence an-
notations and class labels that indi- cate whether the claim is supported,
refuted, or lacks sufficient information. Its structured design makes it
suitable for studying factual correctness and semantic reliability in gen-
erated text.
The claims in the dataset are grouped into three categories: supported,
refuted, and not enough information. These categories correspond to
factually correct, factually incorrect, and unverifiable statements respec-
tively. For the purpose of hallucination detection, supported claims are
treated as non-hallucinated sam- ples, while refuted and not enough in-
formation instances are considered hallu- cinated outputs. This mapping
allows the dataset to be adapted effectively for binary hallucination clas-
sification.
Before training and evaluation, the dataset undergoes preprocessing steps
such as text cleaning, normalization, and formatting to ensure consis-
tency across sam- ples. The textual claims are then transformed into
question–answer style inputs so that they align with the behavior of
Large Language Model responses. This prepared dataset provides a re-
liable foundation for assessing semantic consis- tency, contradiction pat-
terns, and overall hallucination detection performance within the pro-
posed framework.

7 Performance Metrics

The proposed framework is evaluated using classification-based as well

as semantic and explainability-oriented performance measures.

7.1 Classification Metrics

Accuracy denotes the overall correctness of hallucination detection.

TP + TN
Accuracy =
TP + TN + FP + FN

Precision measures the proportion of correctly predicted hallucinated

responses.
TP
P recision =
TP + FP
Recall indicates the ability of the model to identify all hallucinated
responses.
TP
Recall =
TP + FN
F1-Score provides a balanced assessment of precision and recall.

2 × P recision × Recall
F1 =
P recision + Recall
 Title Suppressed Due to Excessive Length 9

7.2 Semantic and Explainability Metrics

Semantic Similarity Score evaluates consistency across rephrased re-
sponses.
A·B
Similarity =
∥A∥∥B∥
Contradiction Detection Rate measures the effectiveness of negation
probing.
ContradictionsDetected
CDR =
T otalN egationP robes
Explanation Fidelity reflects the reliability of explanation outputs.
P redictionaf tertokenremoval
F idelity =
Originalprediction

8 Limitations and Future Work

Although the proposed hallucination detection framework provides an ef-
fective combination of semantic consistency analysis, negation probing,
and explainable classification, several limitations still remain. The system
depends strongly on the behavior of the underlying language model being
examined, and in some cases a model may repeatedly generate similar
incorrect answers, reducing the influence of consistency-based verifica-
tion. The present negation strategy is largely rule- driven and may not
fully capture complex sentence structures, implicit meaning, or domain-
specific linguistic variations. In addition, the use of pre-trained trans-
former models improves contextual understanding but may still leave
gaps when handling highly specialized topics or uncommon real-world
scenarios.
Another limitation is that the framework currently relies only on tex-
tual analysis without direct integration of external knowledge bases for
factual ver- ification. While this design keeps the system lightweight, it
may restrict perfor- mance when dealing with claims that require pre-
cise world knowledge or domain- dependent evidence. The effectiveness
of the approach can also vary depending on the quality and diversity of
the training data, which may limit generalization across broader appli-
cation settings. Furthermore, although the system is efficient enough to
run in a standard CPU-based environment, larger-scale deployment or
high-throughput usage may introduce latency and computational chal-
lenges.
Future work can address these limitations by developing more advanced
nega- tion handling techniques that better capture contextual and lin-
guistic complexity. Additional improvements may include richer feature
fusion strategies, stronger domain adaptation, and the use of domain-
specific datasets to increase robustness in specialized applications. Inte-
grating external factual resources or retrieval- based support mechanisms
could further enhance reliability without sacrificing interpretability. In
the longer term, evaluating the framework across multiple Large Lan-
guage Models and real-world tasks would help strengthen its gener- aliz-
ability, transparency, and practical usefulness in trustworthy AI systems.
10 Abhi M. et al.

9 Conclusion
This work presents an explainable hallucination detection framework de-
signed to improve the reliability and trustworthiness of responses pro-
duced by large language models. By organizing the system into mod-
ular components including text preprocessing, rephrase-based consis-
tency checking, negation probing, contextual feature extraction, classifi-
cation, confidence calibration, and explainability, the framework achieves
both clarity and practical flexibility. Semantic consistency analysis and
contradiction-focused probing help identify unreliable outputs, while con-
textual transformer representations capture deeper meaning required for
accurate hallucination detection.
Implemented entirely in Python using lightweight natural language pro-
cessing techniques, the framework remains efficient and suitable for de-
ployment in standard CPU-based environments without requiring spe-
cialized hardware. Although limitations remain in handling repeated in-
correct answers, complex negation patterns, and highly domain-specific
content, the overall design provides a strong foundation for future refine-
ment through improved linguistic modeling, richer datasets, and external
factual support. Overall, the proposed framework offers an effective, in-
terpretable, and practical solution that can be integrated as a dependable
safety layer for real-world language model applications.

References
1. Lin, J., Tan, A. H., Ji, S.: TruthfulQA: Measuring how models mimic
human falsehoods. In: Proceedings of the 60th Annual Meeting of the
Association for Computational Linguistics (ACL), 2022.
2. Min, J., Tang, S., Wallace, M., Hajishirzi, H.: FactScore: Fine-grained
atomic evaluation of factual precision in long-form text generation. In:
Proceedings of EMNLP, 2023.
3. Yin, X., Chen, T., Zhang, Z.: SelfCheckGPT: Zero-resource black-
box hallucination detection for generative large language models. In:
Proceedings of EMNLP, 2023.
4. Nie, Y., Chen, H., Bansal, M.: Combining fact extraction and veri-
fication with neural semantic matching networks. In: Proceedings of
AAAI, 2019.
5. Thorne, M., et al.: FEVER: A large-scale dataset for fact extraction
and verification. In: Proceedings of NAACL, 2018.