Evaluation of information retrieval systems is important to determine their effectiveness and

efficiency in retrieving relevant information for users. There are different ways to evaluate
information retrieval systems, including the following:
Precision and Recall: Precision is the proportion of retrieved documents that are relevant to the user's
query, while recall is the proportion of relevant documents that are retrieved by the system. Precision
and recall are often used together to evaluate the effectiveness of a retrieval system.
F-measure: F-measure is the harmonic mean of precision and recall. It is used to evaluate the overall
performance of a retrieval system.
Mean Average Precision (MAP): MAP is the average precision of all relevant documents retrieved
for a given set of queries. It is often used to evaluate the effectiveness of a retrieval system over
multiple queries.
Normalized Discounted Cumulative Gain (NDCG): NDCG is a measure that takes into account the
relevance of retrieved documents and their ranking order. It is often used in evaluating web search
engines.
User Satisfaction: User satisfaction can be measured through surveys or user feedback. It evaluates
the overall user experience of the retrieval system.
System Efficiency: System efficiency can be measured through the time taken to retrieve results or
the system's processing power. It evaluates the speed and resource usage of the retrieval system.

Precision and Recall are important metrics used to evaluate the effectiveness of an information
retrieval system. They measure the quality of the results returned by a system by examining the
number of relevant documents retrieved versus the total number of documents retrieved.
Precision refers to the proportion of retrieved documents that are relevant to the user's query. It is
calculated as the number of relevant documents retrieved by the system divided by the total number of
documents retrieved by the system. A system with high precision will return mostly relevant
documents, while a system with low precision will return many irrelevant documents along with
relevant ones.
On the other hand, recall refers to the proportion of relevant documents that are retrieved by the
system. It is calculated as the number of relevant documents retrieved by the system divided by the
total number of relevant documents in the collection. A system with high recall will retrieve most of
the relevant documents, while a system with low recall will miss many relevant documents.
The balance between precision and recall depends on the user's needs and the nature of the search
task. For example, in a medical search task, high recall is important to ensure that all relevant
information is captured, even if it means retrieving some irrelevant information as well. In contrast, in
a legal search task, high precision is more important to avoid irrelevant information and ensure that
only relevant legal cases are returned.
Precision and recall are often used together to evaluate the effectiveness of an information retrieval
system. For example, the F1-score, which is the harmonic mean of precision and recall, is a common
measure used to evaluate the overall effectiveness of a retrieval system. By examining both precision
and recall, information retrieval systems can be optimized to meet the specific needs of users and
improve their overall performance.
Example 1: Suppose you're searching for information about "How to bake a chocolate cake" using a
search engine. If the search engine returns 10 results and only 2 of them are relevant to the query, the
precision would be 20%. If there were 30 relevant results in total and the search engine only returned
10 of them, the recall would be 33%.
Example 2: Suppose you're a doctor looking for information about a rare disease. You perform a
search and the search engine returns 5 results. If all 5 results are relevant to the disease you're
researching, then the precision would be 100%. However, if there were 20 relevant results in total and
the search engine only returned 5 of them, then the recall would be 25%.
Example 3: Suppose you're a student researching a term paper on "The History of the Civil Rights
Movement in the United States". You perform a search and the search engine returns 50 results. If 30
of those results are relevant to your research, then the precision would be 60%. If there were 100
relevant results in total and the search engine only returned 30 of them, then the recall would be 30%.
Exercises
1. Suppose you are a data analyst tasked with evaluating the performance of a search engine that
returns 1000 results for a given query. You are given a list of 50 relevant documents and asked to
calculate the precision and recall of the search engine. What are the precision and recall values for this
search engine?
2. Suppose you are a search engine developer tasked with improving the effectiveness of your search
engine. Your team has decided to focus on improving the precision of the system. What strategies
would you consider implementing to improve precision?

The F-measure is a measure of effectiveness that combines both precision and recall into a single
metric. It is often used to evaluate the overall performance of an information retrieval system. The F-
measure is the harmonic mean of precision and recall and is calculated as follows:
F-measure = 2 * (precision * recall) / (precision + recall)
The F-measure provides a single number that summarizes the overall effectiveness of the retrieval
system. It is especially useful when the distribution of relevant and non-relevant documents in the
collection is uneven.
The F-measure is important because it provides a single metric that captures both precision and recall.
This makes it a more comprehensive measure of retrieval system effectiveness than either precision or
recall alone. It is often used in the field of information retrieval to compare the performance of
different retrieval systems, or to evaluate the effectiveness of a single system over time.
When evaluating a retrieval system using the F-measure, the goal is to achieve a high score. A high F-
measure indicates that the system is returning a good balance of relevant and non-relevant documents.
However, achieving a high F-measure can be challenging because improving one aspect of the
system's performance (such as precision) often comes at the cost of the other aspect (such as recall).
Therefore, it is important to strike a balance between precision and recall when optimizing a retrieval
system for the highest F-measure.
Here are some strategies for overcoming these challenges:
Choosing the right value for the beta parameter: The F-measure combines precision and recall,
with the beta parameter determining the weight given to each. In most cases, a value of 1 is used,
which gives equal weight to precision and recall. However, if precision is more important than recall,
a value of less than 1 can be used, and if recall is more important than precision, a value greater than 1
can be used. Therefore, it's important to choose the right beta value based on the specific needs of
your information retrieval system.
Dealing with imbalanced datasets: In some cases, the dataset may be imbalanced, meaning that one
class is much more common than the other. In such cases, the F-measure can be biased towards the
more common class. To overcome this challenge, you can use other evaluation metrics such as area
under the ROC curve (AUC-ROC) or precision-recall curve (PRC). Alternatively, you can use
techniques such as undersampling or oversampling to balance the dataset before evaluating the
system.
Addressing multi-class classification: F-measure is originally designed for binary classification.
When it comes to multi-class classification, micro-averaging and macro-averaging are the most
commonly used approaches. Micro-averaging calculates the F-measure by treating all the instances as
a single large class, while macro-averaging calculates the F-measure for each class separately and
then takes the average. It's important to choose the right approach based on the specific needs of your
information retrieval system.
Considering the context and domain: Different information retrieval tasks require different
evaluation metrics. It's important to consider the context and domain of the problem when selecting
the evaluation metric. For instance, for text classification tasks, accuracy, F-measure, and AUC-ROC
are the commonly used metrics. However, for tasks like information retrieval or recommender
systems, the ranking-based evaluation metrics like mean average precision (MAP) and normalized
discounted cumulative gain (NDCG) are more appropriate.
Here's an example of how to calculate the F-measure for a retrieval system:
Suppose a search engine retrieves 100 documents in response to a user query, and 50 of those
documents are relevant. The search engine ranks the documents in decreasing order of relevance, so
the top k documents are assumed to be the most relevant. Let's say we want to evaluate the system's
performance based on the top 20 results. We also know that the precision at 20 is 0.75 and the recall at
20 is 0.6.
To calculate the F-measure at 20, we use the formula:
F-measure = 2 * (precision * recall) / (precision + recall)
Substituting in the values we have:
F-measure = 2 * (0.75 * 0.6) / (0.75 + 0.6) = 0.664
So the F-measure for the retrieval system at 20 is 0.664.

Exercise 1
Two Information retrieval systems, A and B, are being compared. Both are given the same query,
applied to a collection of 1000 documents. System A returns 420 documents, of which 50 are relevant
to the query. System B returns 90 documents, of which 25 are relevant to the query. Within the whole
collection there are in fact 80 documents relevant to the query.
Create a contingency table of the results for each system, and compute the following:
i. Recall
ii. Precision
iii. Accuracy
iv. F-score

Mean Average Precision (MAP) is a measure of how well an information retrieval system performs
over multiple queries. MAP is the average of the Precision values of each query for all relevant
documents retrieved for that query.
MAP is calculated as follows:
For each query, the relevant documents are identified and ranked by the retrieval system.
The precision is calculated for each relevant document that is retrieved for that query.
The average precision is calculated for that query by summing up the precision values and dividing by
the number of relevant documents.
The MAP is the average of all the average precision values for all queries.
MAP is an important evaluation metric for information retrieval systems because it measures the
average effectiveness of the system over a set of queries. In contrast, precision and recall only
evaluate the performance of the system for a single query.
A high MAP score indicates that the system is retrieving a large number of relevant documents across
all queries. A low MAP score may indicate that the system is not performing well across all queries,
and that the system needs to be improved.
MAP is commonly used in various information retrieval tasks such as web search, document retrieval,
and question answering systems. It helps search engine developers to improve the quality of the
search results and to better understand the strengths and weaknesses of the system for a given set of
queries.
Here are some examples that illustrate how Mean Average Precision (MAP) can be used:
Web search engine: Suppose a web search engine wants to evaluate its performance on a set of 100
queries. For each query, the search engine retrieves 10 documents and assigns a relevance score to
each document (e.g., 0 for non-relevant, 1 for somewhat relevant, 2 for highly relevant). The MAP is
then calculated by averaging the average precision values for all 100 queries. A higher MAP value
indicates better overall performance of the search engine.
Document retrieval system: Consider a document retrieval system that retrieves documents based on
a user's query. For a set of 50 queries, the system retrieves 20 documents for each query and assigns a
relevance score to each document. The MAP is calculated by averaging the average precision values
for all 50 queries. A higher MAP value indicates better overall performance of the document retrieval
system.
Question answering system: A question answering system retrieves answers to user queries from a
large database of documents. For a set of 30 questions, the system retrieves 5 answers for each
question and assigns a relevance score to each answer. The MAP is calculated by averaging the
average precision values for all 30 questions. A higher MAP value indicates better overall
performance of the question answering system.

Exercises
1. How can MAP be used to evaluate the performance of a web search engine?
2. What are some limitations of using MAP as an evaluation metric for information retrieval systems?
3. A web search engine wants to evaluate its performance on a set of 50 queries. For each query, the
search engine retrieves 20 documents and assigns a relevance score to each document. The relevant
documents for each query are known in advance. Calculate the MAP for the search engine and
interpret your result.
4. A document retrieval system retrieves documents based on a user's query. For a set of 30 queries,
the system retrieves 10 documents for each query and assigns a relevance score to each document.
The relevant documents for each query are known in advance. The system's MAP is 0.75. How can
the system improve its performance?
5. A question answering system retrieves answers to user queries from a large database of documents.
For a set of 20 questions, the system retrieves 5 answers for each question and assigns a relevance
score to each answer. The relevant answers for each question are known in advance. The system's
MAP is 0.65. How can the system be evaluated further?

Normalized Discounted Cumulative Gain (NDCG) is a measure used to evaluate the quality of a
ranked list of documents, such as the results of a web search engine. It takes into account both the
relevance of the retrieved documents and their ranking order, which makes it a more sophisticated
evaluation metric than simple precision or recall.
The NDCG score ranges from 0 to 1, with 1 being the best possible score. A score of 0 indicates that
none of the documents in the ranked list are relevant, while a score of 1 indicates that all of the
documents are perfectly relevant and are ranked in the ideal order.
To calculate NDCG, we first assign a relevance score to each document in the ranked list. This
relevance score can be binary (e.g., 1 if the document is relevant and 0 if it is not), ordinal (e.g., 1-3 if
the document is not relevant, somewhat relevant, or highly relevant), or graded (e.g., a score from 0 to
10 indicating the degree of relevance).
Next, we calculate the Discounted Cumulative Gain (DCG) of the ranked list. DCG is a measure of
the relevance of the documents in the list, where documents that are more highly ranked receive a
higher weight. The formula for DCG is:
DCG = rel1 + (rel2 / log2(2)) + (rel3 / log2(3)) + ... + (reln / log2(n))
Where rel1, rel2, rel3, ..., reln are the relevance scores of the documents in the ranked list, and n is the
number of documents in the list.
Finally, we calculate the Ideal Discounted Cumulative Gain (IDCG), which is the maximum possible
DCG for the given set of documents. This is done by assuming that the most highly relevant
documents are ranked at the top of the list and calculating the DCG for that ideal list.
The NDCG score is then calculated by dividing the DCG by the IDCG:
NDCG = DCG / IDCG
NDCG is often used in evaluating web search engines, where the goal is to retrieve the most relevant
documents for a given query and rank them in the most effective order. By taking into account both
relevance and ranking order, NDCG provides a more accurate measure of the quality of the search
results than simple measures like precision and recall.
Example
Let's say you are using a search engine to find information on the topic of "best laptops for gaming".
The search engine provides you with a list of 10 results, ranked from 1 to 10.
You read through the list and assign a relevance score to each document, based on how useful it is to
your search. You give the top-ranked document a relevance score of 5, the second-ranked document a
score of 4, the third-ranked document a score of 3, and so on.
Using this relevance score, you calculate the DCG of the list by summing up the relevance scores of
each document, weighted by their position in the list. For example, the DCG for the top three results
might be:
DCG@3 = 5 + 4/log2(3+1) + 3/log2(4+1) = 5 + 3.17 + 1.58 = 9.75
To calculate the IDCG, you assume that the most relevant documents are ranked at the top of the list
and calculate the DCG for that ideal list. For example, if the three most relevant documents are ranked
at the top of the list, the IDCG would be:
IDCG@3 = 5 + 4/log2(2+1) + 3/log2(3+1) = 5 + 4 + 1.58 = 10.58
Finally, you calculate the NDCG score by dividing the DCG by the IDCG:
NDCG@3 = DCG@3 / IDCG@3 = 9.75 / 10.58 = 0.92
This NDCG score tells you that the top three results provided by the search engine are highly relevant
to your search and are in a good order. If the search engine provides a higher NDCG score, it means
that the search results are even better.
Exercises
1. Suppose a web search engine retrieves 10 documents in response to a user query, and their
relevance scores (on a scale of 0 to 3) and positions in the ranking are as follows:
Document 1: relevance score=2, position=1
Document 2: relevance score=1, position=2
Document 3: relevance score=0, position=3
Document 4: relevance score=3, position=4
Document 5: relevance score=2, position=5
Document 6: relevance score=1, position=6
Document 7: relevance score=0, position=7
Document 8: relevance score=2, position=8
Document 9: relevance score=3, position=9
 Document 10: relevance score=1, position=10
Calculate the NDCG@5 for this list of retrieved documents.
2. Suppose a company is developing a new recommendation system for movies, and they want to
evaluate its performance using NDCG. They have a dataset of 1000 users and their ratings of 500
movies on a scale of 1 to 5. How can they use NDCG to evaluate the system?
User satisfaction is a measure that evaluates the overall user experience of an information retrieval
system. It focuses on the users' perception of the system, rather than the objective measures of system
performance like precision, recall, F-measure, or NDCG. User satisfaction is an important metric
because ultimately, the success of an information retrieval system depends on how well it satisfies the
users' information needs and expectations.
User satisfaction can be measured through surveys or user feedback. Surveys can be designed to ask
users about their satisfaction with various aspects of the system, such as the relevance of retrieved
documents, the ease of use of the interface, the speed of the system, and the overall usefulness of the
system. User feedback can be obtained through various channels, such as email, social media, or
customer support tickets. User feedback can provide insights into specific issues that users encounter
while using the system, and can help improve the system's usability and relevance.
User satisfaction is often used in conjunction with objective performance measures like precision,
recall, F-measure, or NDCG to evaluate the effectiveness of an information retrieval system. For
example, a system that has high precision and recall but low user satisfaction may not be successful
because users are not satisfied with the user experience. On the other hand, a system that has moderate
precision and recall but high user satisfaction may be more successful because users are satisfied with
the overall experience of using the system.

Here are some examples of how user satisfaction can be measured:

Surveys: A survey can be sent to users of an information retrieval system asking them to rate their
satisfaction with the system on a scale from 1 to 5, with 1 being very dissatisfied and 5 being very
satisfied. The survey can also ask specific questions about different aspects of the system, such as ease
of use, relevance of results, and speed of the system.
User feedback: Users can be encouraged to provide feedback on the system through various
channels, such as email, social media, or customer support tickets. This feedback can be analyzed to
identify common issues or complaints that users have about the system, and improvements can be
made to address these issues.
User testing: Users can be recruited to participate in user testing sessions, where they are observed
while using the system and asked to provide feedback on their experience. This can provide valuable
insights into usability issues and areas for improvement.
Net Promoter Score (NPS): NPS is a measure of customer loyalty and satisfaction. It asks users to
rate how likely they are to recommend the system to others on a scale from 0 to 10. Users who rate
the system 9 or 10 are considered "promoters," while those who rate it 6 or below are considered
"detractors." The NPS score is calculated by subtracting the percentage of detractors from the
percentage of promoters.
System efficiency is an important aspect to consider when evaluating a retrieval system. It refers to
the speed and resource usage of the system while retrieving and presenting results to the user. There
are two main metrics used to measure system efficiency: time taken to retrieve results and the
system's processing power.
The time taken to retrieve results is the time elapsed from when the user enters the query to the time
when the results are displayed on the screen. This metric is important because users expect to receive
results quickly and efficiently. If the system takes too long to retrieve results, users may become
frustrated and abandon their search.
Processing power is a measure of the system's ability to handle a large number of queries and provide
accurate results. It refers to the amount of resources required by the system to process a single query.
Systems with higher processing power can handle a larger number of queries in a shorter amount of
time and provide more accurate results.
It is important to note that system efficiency should be considered in conjunction with other
evaluation metrics such as precision, recall, and user satisfaction. A system that retrieves results
quickly but provides irrelevant or inaccurate results is not effective. Therefore, it is essential to strike
a balance between system efficiency and the accuracy of the results provided.

Here are some examples of how we might evaluate system efficiency in different contexts:
Search engine: When you use a search engine like Google or Bing, you expect the search results to
appear quickly and accurately. The time it takes for the search engine to retrieve and display the
results is one measure of system efficiency. Search engines also need to be able to handle a large
number of queries from users all over the world, so they need to have robust infrastructure to handle
the load.
E-commerce website: An e-commerce website like Amazon or Walmart needs to be able to handle a
large number of users searching for products and making purchases at the same time. System
efficiency in this context refers to the speed at which the website can load and display product pages,
process transactions, and provide customer support.
Healthcare system: In a healthcare system, system efficiency might refer to how quickly a doctor or
nurse can access patient records or lab results. The system needs to be able to handle large amounts of
data and provide quick and accurate results to ensure that patients receive the best possible care.
Financial system: In a financial system like a bank or stock trading platform, system efficiency is
critical for making timely and accurate trades. The system needs to be able to process transactions
quickly and securely, and handle large amounts of data without crashing or slowing down.

Exercise
1. You are tasked with evaluating the system efficiency of a new search engine. What specific
measures would you use to evaluate its efficiency, and how would you collect data for each measure?