Detection of Invalid-Premise Hallucinations in Large
Language Models
Using Generative AI Verification Techniques
A Comprehensive Analysis and Implementation
Prepared By: Aman Chandra H
USN: 1RVU23CSE039
Course: Fundamentals of Generative AI
Department: Computer Science and Engineering
Abstract
Large Language Models (LLMs) have demonstrated remarkable capabilities in natural language
generation. However, they frequently suffer from hallucinations, where responses are generated
confidently despite being factually incorrect or based on invalid assumptions. This project
presents a practical system to detect invalid-premise hallucinations, a specific but critical
hallucination type where the model accepts and responds to non-existent entities, versions, or
references. We implement and compare three hallucination detection strategies: LLM-as-a-Judge,
Natural Language Inference (NLI)-based verification, and Retrieval-Augmented Generation
(RAG)-based verification. Experiments are conducted using open-source and synthetically
generated datasets. Results demonstrate that retrieval-based verification provides the highest
reliability, while LLM-based self-verification often fails under subtle invalid premises. The system
is demonstrated through a live interface showcasing hallucination detection in real time.
Keywords: Generative AI, Hallucination Detection, Large Language Models, Invalid Premise,
Verification Systems, RAG
1 Introduction
Generative Artificial Intelligence models, particularly Large Language Models (LLMs), are widely used
in applications such as chatbots, decision support systems, education, and cybersecurity. Despite their
success, LLMs exhibit a major limitation known as hallucination, where models generate plausible but
incorrect information.
Among different hallucination types, invalid-premise hallucinations are particularly dangerous.
In such cases, the model does not challenge a false assumption embedded in the query and instead
generates a confident response. This behavior can mislead users, especially in technical and safety-
critical domains.
Examples of Invalid-Premise Hallucinations:
• Non-existent entities: “Explain how the Flux Capacitor from Back to the Future works in quantum
computing.”
• Fake references: “Summarize RFC 9999 on time travel protocols.”
• Impossible events: “Describe the results of the 2050 World Cup final.”
This project aims to build a practical hallucination detection system that identifies such failures
using multiple GenAI-based verification techniques.
2 Problem Statement
Modern LLMs often fail to detect when a user query is based on non-existent entities, fake references
or citations, future or impossible events, or missing/contradictory context.
Detection of Invalid-Premise Hallucinations in LLMs 1
Instead of refusing to answer or acknowledging the invalid premise, the model frequently produces
fabricated explanations, creating a false sense of authority and reliability.
Core Challenge: There is a lack of lightweight, practical systems that can automatically detect
invalid-premise hallucinations during inference time without significantly increasing response
latency or computational overhead.
3 Research Objectives
1. Design and implement a system capable of detecting invalid-premise hallucinations in LLM outputs
2. Implement multiple hallucination detection techniques using Generative AI concepts:
• LLM-as-a-Judge approach
• Natural Language Inference (NLI)-based verification
• Retrieval-Augmented Generation (RAG)-based verification
3. Compare effectiveness using quantitative metrics (precision, recall, F1-score, latency)
4. Demonstrate the system through a live interactive application
5. Provide insights into strengths and limitations of each approach
4 Types of Hallucinations Considered
This project focuses specifically on Invalid-Premise Hallucinations, defined as responses
that accept and elaborate on a false or non-existent assumption in the input query, without
questioning or rejecting it.
Categories:
• Non-existent software versions: “How do I use Python 4.0’s new features?”
• Fake RFCs or papers: “Explain the findings from paper ’A New Approach to Cold Fusion’ by Smith
et al. 2023.”
• References to missing tables/figures
• Fictional/impossible technologies
• Future events presented as fact
5 Related Work
Previous research includes:
• TruthfulQA benchmark
• FEVER (fact verification)
• HaluEval dataset
• LLM-as-a-Judge methods (MT-Bench, Chatbot Arena)
• Retrieval-Augmented Generation (RAG)
• Self-consistency and contradiction detection
Gap: Most works target general hallucinations. This project focuses narrowly on invalid-premise
cases — easier to evaluate and more practical for production systems.
6 System Architecture
6.1 Overall Workflow
1. User query → LLM generates response
2. Response → hallucination detection module
3. Parallel verification: LLM-Judge, NLI, RAG
4. Aggregate verdicts → final classification + confidence
Detection of Invalid-Premise Hallucinations in LLMs 2
5. Display results with explanations
6.2 Key Components
• Query Parser
• Response Generator (base LLM)
• Verification Engine (multi-method)
• Aggregation Module
• Streamlit-based User Interface
7 Generative AI Techniques Used
7.1 LLM-as-a-Judge
Secondary LLM evaluates whether answer accepts invalid premises.
Prompt: "Given the query: ’query’ and answer: ’answer’, determine if the answer
accepts any invalid premises..."
Advantages: simple, leverages reasoning Limitations: bias inheritance, variance
7.2 NLI-Based Verification
Decompose answer → atomic claims → NLI check against known facts/query.
Advantages: explainable, fine-grained Limitations: needs good decomposition
7.3 RAG-Based Verification
Retrieve evidence → verify answer against retrieved content.
Advantages: strong grounding Limitations: depends on retrieval quality
8 Dataset Description
8.1 Open-Source
• TruthfulQA
• FEVER
• HaluEval
8.2 Synthetic Dataset (1000 queries)
Categories: non-existent software (450), fake references (300), future events (250)
Manually labeled: correct refusal / hallucinated / partially hallucinated
9 Implementation Details
Technology stack:
• Python 3.9+
• HuggingFace Transformers, LangChain
• FAISS / ChromaDB
• OpenAI / Anthropic / open LLMs
• Streamlit
10 Results
Detection of Invalid-Premise Hallucinations in LLMs 3
10.1 Performance Comparison
Method Precision Recall F1 Latency (ms)
LLM-as-Judge 0.72 0.65 0.68 1250
NLI-Based 0.85 0.78 0.81 320
RAG-Based 0.92 0.88 0.90 450
Ensemble 0.89 0.86 0.87 670
Main finding: RAG-based verification is most reliable for invalid-premise hallucination detec-
tion.
11 Demo Description
Streamlit app featuring:
• Query input
• LLM selection
• Response + highlighting
• Multi-method verification results
• RAG evidence viewer
• Confidence scores
12 Limitations
• Retrieval coverage dependency
• Subtle premise tricks
• Detection (not prevention)
• Added latency
• Scope limited to invalid-premise type
13 Future Work
• Multi-modal & chain-of-thought hallucination detection
• Real-time correction
• Domain-specific fine-tuning
• Better latency optimization
• Larger / more diverse benchmarks
14 Conclusion
This project developed a practical invalid-premise hallucination detection system comparing
LLM-Judge, NLI and RAG approaches. RAG-based verification showed the best performance
(F1 0.90). The work provides a foundation for building more trustworthy Generative AI systems
in high-stakes domains.
15 References
1. Lin, S., et al. (2022). TruthfulQA: Measuring How Models Mimic Human Falsehoods.
2. Thorne, J., et al. (2018). FEVER: a Large-scale Dataset for Fact Extraction and VERification.
3. Li, J., et al. (2023). HaluEval: A Large-Scale Hallucination Evaluation Benchmark.
4. Zheng, L., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.
5. Lewis, P., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.
Detection of Invalid-Premise Hallucinations in LLMs 4
Fundamentals of Generative AI Project Report • Aman Chandra H • January 22, 2026
"Building Trustworthy AI through Robust Hallucination Detection"
Detection of Invalid-Premise Hallucinations in LLMs 5