UNIT III: Evaluation and Visualization of Information Retrieval System
Performance evaluation: Precision and recall, MRR, F-Score, NDCG, user-oriented measures. Visualization in Information System: Starting points, Query
Specification.
1. Introduction
Evaluation and visualization are crucial components of an Information Retrieval (IR) system. Evaluation helps us measure how well an IR system performs, while
visualization helps users interact with, understand, and refine their information needs. Performance evaluation in information retrieval (IR) measures how effec-
tively a system retrieves relevant documents for a query, focusing on precision, recall, and ranking quality. Key metrics include Precision@k, Mean Average Preci-
sion (MAP), and Normalized Discounted Cumulative Gain (NDCG), often evaluated using the Cranfield paradigm, which uses standard test collections (corpus,
queries, relevance judgments). This unit focuses on standard performance evaluation metrics and visualization concepts used in modern IR systems like search en-
gines, digital libraries, and recommender systems.
2. Performance Evaluation of IR Systems
Performance evaluation determines the effectiveness of an IR system in retrieving relevant documents for a given user query.
2.1 Basic Terminology
Consider a query issued by a user:
Relevant Documents (R): Documents relevant to the query
Retrieved Documents (A): Documents retrieved by the system
Relevant & Retrieved (Ra): Relevant documents that are retrieved
Confusion Matrix for IR
Relevant Not Relevant
Retrieved TP FP
Not Retrieved FN TN
Where:
TP (True Positive) = Relevant & Retrieved
FP (False Positive) = Retrieved but Not Relevant
FN (False Negative) = Relevant but Not Retrieved
3. Precision and Recall
3.1 Precision
Definition:
Precision measures the exactness of the system.
[ Precision = \frac{TP}{TP + FP} ]
Interpretation:
Out of all retrieved documents, how many are relevant?
Example:
Retrieved documents = 10
Relevant among them = 6
[ Precision = 6/10 = 0.6 ]
3.2 Recall
Definition:
Recall measures the completeness of the system.
[ Recall = \frac{TP}{TP + FN} ]
Interpretation:
Out of all relevant documents, how many were retrieved?
Example:
Total relevant documents in collection = 8
Retrieved relevant documents = 6
[ Recall = 6/8 = 0.75 ]
3.3 Precision–Recall Tradeoff
High Precision → fewer but more accurate results
High Recall → more results but may include noise
Diagram (Conceptual):
Precision
|\
|\
| \
| \
|____\______ Recall
4. F-Score (F-Measure)
Precision and Recall are often combined into a single metric.
4.1 F1-Score
[ F1 = 2 \times \frac{Precision \times Recall}{Precision + Recall} ]
Example:
Precision = 0.6, Recall = 0.75
[ F1 = 2 \times \frac{0.6 \times 0.75}{0.6 + 0.75} = 0.666 ]
4.2 Generalized Fβ-Score
[ F_\beta = (1 + \beta^2) \frac{PR}{\beta^2 P + R} ]
β > 1 → Recall is more important
β < 1 → Precision is more important
5. Mean Reciprocal Rank (MRR)
5.1 Definition
MRR evaluates systems where only the rank of the first relevant document matters (e.g., QA systems).
[ MRR = \frac{1}{|Q|} \sum_{i=1}^{|Q|} \frac{1}{rank_i} ]
Where:
|Q| = Number of queries
rankᵢ = Rank position of the first relevant document
5.2 Example
Query Rank of First Relevant Doc Reciprocal Rank
Q1 1 1.0
Q2 2 0.5
Q3 4 0.25
[ MRR = (1 + 0.5 + 0.25)/3 = 0.583 ]
6. Discounted Cumulative Gain (DCG) and NDCG
6.1 Motivation
Some documents are more relevant than others. Ranking quality matters.
6.2 DCG
[ DCG = rel_1 + \sum_{i=2}^{n} \frac{rel_i}{\log_2(i)} ]
Where relᵢ is relevance score at rank i.
Example:
Rank Relevance
1 3
2 2
3 1
[ DCG = 3 + 2/\log_2(2) + 1/\log_2(3) ]
6.3 Normalized DCG (NDCG)
[ NDCG = \frac{DCG}{IDCG} ]
Where IDCG is DCG of ideal (perfect) ranking.
Range: 0 to 1
7. User-Oriented Evaluation Measures
Traditional metrics ignore user satisfaction. User-oriented measures include:
7.1 User Satisfaction
Feedback surveys
Click-through rate (CTR)
7.2 Time-Based Measures
Time to find information
Dwell time on documents
7.3 Success Rate
Whether user completed intended task
7.4 Implicit Feedback
Mouse movements
Scrolling behavior
8. Visualization in Information Retrieval Systems
Visualization helps users explore, refine, and understand search results.
9. Starting Points in IR Visualization
Starting points help users begin interaction with the system.
9.1 Types of Starting Points
Query-based: Keyword input (Google-style)
Browsing-based: Categories, facets
Recommendation-based: Popular or trending items
9.2 Example Diagram
User → Search Box → Ranked Results → Interaction
10. Query Specification
Query specification refers to how users express information needs.
10.1 Types of Query Specification
1. Keyword Queries
o Single or multiple terms
o Example: "machine learning"
2. Boolean Queries
o AND, OR, NOT
o Example: "AI AND Healthcare"
3. Natural Language Queries
o Example: "What is precision in IR?"
4. Structured Queries
o Fields and filters
o Example: Author = "Smith" AND Year > 2020
10.2 Visual Support for Query Specification
Auto-suggestions
Query expansion hints
Faceted filters
Diagram:
User Query → Visual Aids → Refined Query → Better Results
11. Numerical Practice Problems
Problem 1
An IR system retrieves 15 documents, out of which 9 are relevant. Total relevant documents in collection are 12.
Precision = ?
Recall = ?
Solution:
Precision = 9/15 = 0.6
Recall = 9/12 = 0.75
Problem 2
For 5 queries, the rank of first relevant documents are: 1, 2, 3, 1, 5.
MRR = ?
Solution:
[ MRR = (1 + 1/2 + 1/3 + 1 + 1/5)/5 = 0.606 ]