CHAPTER 8: EVALUATION

1. In a medical test for a rare disease, out of 1000 people tested, 50 actually have the disease while 950 do
not. The test correctly identifies 40 out of the 50 people with the disease as positive, but it also wrongly
identifies 30 of the healthy individuals as positive. What is the accuracy of the test?
A) 97% B) 90% C) 85% D) 70%

2. A student solved 90 out of 100 questions correctly in a multiple-choice exam. What is the error rate of the
student's answers?
A) 10% B) 9% C) 8% D) 11%

3. In a spam email detection system, out of 1000 emails received, 300 are spam. The system correctly
identifies 240 spam emails as spam, but it also marks 60 legitimate emails as spam. What is the precision
of the system?
A) 80% B) 70% C) 75% D) 90%

4. In a binary classification problem, a model predicts 70 instances as positive out of which 50 are actually
positive. What is the recall of the model?
A) 50% B) 70% C) 80% D) 100%

5. In a sentiment analysis task, a model correctly predicts 120 positive sentiments out of 200 positive
instances. However, it also incorrectly predicts 40 negative sentiments as positive. What is the F1 score of
the model?
A) 0.8 B) 0.75 C) 0.72 D) 0.82

6. A medical diagnostic test is designed to detect a certain disease. Out of 1000 people tested, 100 have
the disease, and the test identifies 90 of them correctly. However, it also wrongly identifies 50 healthy
people as having the disease. What is the precision of the test?
A) 90% B) 80% C) 70% D) 60%

7. A teacher's marks prediction system predicts the marks of a student as 75, but the actual marks obtained
by the student are 80. What is the absolute error in the prediction?
A) 5 B) 10 C) 15 D) 20

8. The goal when evaluating an AI model is to:

A) Maximize error and minimize accuracy B) Minimize error and maximize accuracy
C) Focus solely on the number of data points used D) Prioritize the complexity of the model

9. A high F1 score generally suggests:

A) A significant imbalance between precision and recall B) A good balance between precision and recall
C) A model that only performs well on specific data points D) The need for more training data

10. How is the relationship between model performance and accuracy described?
A) Inversely proportional B) Not related C) Directly proportional D) Randomly fluctuating

11. A _____ is a table that lists the predicted values of an AI model and the actual/correct outcome values.
a) Classification Matrix b) Regression Matrix c) Confusion Matrix d) Deep learning Matrix

12. When both predicted value of the AI model and actual value are positive, it is called _____________
a) True Positive b) True Negative c) False Positive d) False Negative

13. Stmt1: The output given by the AI model is known as reality.

Stmt 2:The real scenario is known as Prediction.
(a) Both Stmt 1 and Stmt 2 are correct (b) Both Statement1 and Statement2 are incorrect
(c) Stmt 1 is correct but Stmt 2 is incorrect (d) Statement2 is correct but Statement1 is incorrect
CL-X/AI/Evaluation 1 Rosary School
14. Anjali has made a model which predicts the performance of students in the various examinations in India.
She collected the data of students’ performance with respect to state, age, school and curriculum. Her model
works with good accuracy and precision value. Which of the statements given below is incorrect?
(a) Data gathered with respect to state, age, school and curriculum is known as Testing Data.
(b) Data given to an AI model to check accuracy and precision is Testing Data.
(c) Training data and testing data are acquired in the Data Acquisition stage.
(d) Training data is always larger as compared to testing data.

15. Amaira made a Forest Fire detector system for which she had collected the dataset and used all the
dataset to train the model. Then, she used the same data to evaluate the model which resulted in the correct
answer all the time but was not able to perform with unknown dataset. Name the concept.
a) Best fit b) Overfitting c) underfitting d) Regression

16. Which condition of the evaluation does the when Prediction is YES and Reality is NO?
a) True Positive b) True Negative c) False Positive d) False Negative

17. Which evaluation parameter takes into consideration all the correct predictions?
a) Precision b) Recall c) Accuracy d) F1 score

18. Stmt 1: Overfitting is not recommended for evaluation of a model.

Stmt 2: This is because the model will simply remember the whole training set, and will therefore always
predict the correct label for any point in the training set.
(a) Both Stmt 1 and Stmt 2 are correct. (b) Both Stmt 1 and Stmt 2 are incorrect.
(c) Stmt 1 is correct but Stmt 2 is incorrect. (d) Stmt 2 is correct but Stmt 1 is incorrect.

19. It is one of the parameters for evaluating a model's performance which is defined as the percentage of
true positive cases versus all the cases where the prediction is true. Which of the following evaluation
parameters is this?
(a) Precision (b) Recall (c) F1 score (d) accuracy

20. With respect to evaluation, for which of the following does the prediction and reality match?
(a) True positive and False positive (b) True positive and True negative
(c) False positive and False negative (d) True positive and False negative

21. Which of these reflect the correct decisions by an AI model?

a) True Positive b) True Negative c) False Positive d) False Negative

22. ____ is the percentage of times the predictions out of all the observations are correct.
a) Precision Rate b) Recall c) Accuracy Rate d) F1 score

23. ____ is the rate at which desirable predictions turn out to be correct.
a) Precision Rate b) Recall c) Accuracy Rate d) F1 score

24. When the predicted value of the AI model is positive but actual value is negative, it is called ________
a) True Positive b) True Negative c) False Positive d) False Negative

25. Which of the following ethical concerns is related to taking the responsibility for the choice of evaluation
metrics.
a) Bias b) Accountability c) Transparency d) Translucency

CL-X/AI/Evaluation 2 Rosary School

 (Do the Q. 7,10,11,12,13,14 in your AI register)

1. Define Evaluation.
Evaluation is a process of understanding the reliability of any AI model, based on outputs by feeding the test
dataset into the model and comparing it with actual answers.
OR
Evaluation is a process that critically examines a program. It involves collecting and analyzing information
about a program’s activities, characteristics, and outcomes. Its purpose is to make judgments about a
program, to improve its effectiveness, and/or to inform programming decisions.
2. Which two parameters are considered for Evaluation of a model?
Prediction and Reality are the two parameters considered for Evaluation of a model. The “Prediction” is the
output which is given by the machine and the “Reality”is the real scenario, when the prediction has been
made?
3. What is True Positive?
 The predicted value matches the actual value
 The actual value was positive and the model predicted a positive value
4. What is True Negative?
 The predicted value matches the actual value
 The actual value was negative and the model predicted a negative value
5. What is False Positive?
 The predicted value was falsely predicted
 The actual value was negative but the model predicted a positive value
 Also known as the Type 1 error
6. What is False Negative?
 The predicted value was falsely predicted
 The actual value was positive but the model predicted a negative value
 Also known as the Type 2 error

7. What is meant by Overfitting, Under fitting and Perfect fitting of Data?

OVERFITTING
Overfitting is the condition where an AI model memorizes the training data instead of learning general
patterns, leading to excellent training performance but poor performance on new, unseen data.
 It performs very well on the training data.
 It performs poorly on new, unseen data, because it has essentially "memorized" the training
examples rather than learned how to generalize.
UNDERFITTING
Underfitting is a situation where an AI model is too simple to learn the patterns in the training data, leading
to poor performance on both training and unseen data.
PERFECT FITTING
Perfect fitting (also called a good fit or optimal fit) in AI and machine learning refers to a model that learns
the true patterns in the training data. As a result, it performs well on both the training data and new, unseen
data.

8. Why is evaluation important? Explain.

Evaluation is a process that critically examines a program. It involves collecting and analyzing information
about a program's activities, characteristics, and outcomes. Its purpose is to make judgments about a
program, to improve its effectiveness, and/or to inform programming decisions.
 Evaluation is important to ensure that the model is operating correctly and optimally.
 Evaluation is an initiative to understand how well it achieves its goals.
 Evaluations help to determine what works well and what could be improved in a Program.

CL-X/AI/Evaluation 3 Rosary School

9. What are the possible reasons for an AI model not being efficient? Explain reasons of an AI model
not being efficient:
a. Lack of Training Data: If the data is not sufficient for developing an AI Model, or if the data is missed while
training the model, it will not be efficient.
b. Unauthenticated Data / Wrong Data: If the data is not authenticated and correct, then the model will not
give good results.
c. Inefficient coding / Wrong Algorithms: If the written algorithms are not correct and relevant, Model will
not give desired output. Not Tested: If the model is not tested properly, then it will not be efficient.
d. Not Easy: If it is not easy to be implemented in production or scalable.
e. Less Accuracy: A model is not efficient if it gives less accuracy scores in production or test data or if it is
not able to generalize well on unseen data.
Q10. Identify which metric (Precision or Recall) is to be used in the following cases and why?
a) Email Spam Detection b) Cancer Diagnosis c) Legal Cases(Innocent until proven guilty)
d) Fraud Detection e) Safe Content Filtering (like Kids YouTube)

Case Metric Reason

a) Email Spam Precision We should avoid marking genuine (important) emails as spam. High
Detection precision means emails classified as spam are actually spam.
b) Cancer Diagnosis Recall Missing a cancer patient (False Negative) can be life-threatening.
High recall ensures most cancer cases are detected.
c) Legal Cases Precision It is more important to ensure that people identified as guilty are
(Innocent until proven actually guilty. This minimizes false accusations of innocent people
guilty) (False Positives).
d) Fraud Detection Recall Missing a fraudulent transaction (False Negative) can cause
significant financial loss. High recall helps detect as many fraud
cases as possible.
e) Safe Content Recall The priority is to block as much harmful or inappropriate content as
Filtering (like Kids possible, even if some safe content is mistakenly filtered. High recall
YouTube) minimizes harmful content being missed.

11. What is the purpose of Accuracy in AI and give the example of

equation?
Answer – The percentage of accurate predictions among all the
observations is what is meant by the term accuracy. A prediction is deemed
accurate if it agrees with reality. There are two circumstances in this case
where the Prediction and Reality match: True Positive and True Negative.

12. What is the purpose of Precision in AI and give the example of

equation?
Precision is defined as the percentage of true positive cases versus all the
cases where the prediction is true. That is, it takes into account the True
Positives and false positives.

13. What is the purpose of Recall in AI and give the example of

equation?
It can be described as the percentage of positively detected cases that are
positive. It takes into account both False Negatives and True Positives.

CL-X/AI/Evaluation 4 Rosary School

Q5. Examine the following case studies. Draw the confusion matrix and calculate metrics such as accuracy, precision,
recall, and F1-score for each one of them.
a. Case Study 1:
A spam email detection system is used to classify emails as either spam (1) or not spam (0). Out
of 1000 emails:
- 150 emails were correctly classified as spam.
- 50 emails were incorrectly classified as spam.
- 750 emails were correctly classified as not spam.
- 50 emails were incorrectly classified as not spam.
Confusion Matrix
The Confusion Matrix Actual (Yes) Spam Actual (No) Not Spam
Prediction- spam-Yes 150 50
Prediction- Not Spam 50 750

Total emails = 150 + 50 + 50 + 750 = 1000

Accuracy = (TP + TN) / Total = (150 + 750) / 1000 = 90%
Precision = TP / (TP + FP) = 150 / 200 = 0.75
Recall = TP / (TP + FN) = 150 / 200 = 0.75
F1 Score = 2* (precision * recall)/precision + recall
= 2*(0.75*0.75)/(0.75+0.75) = 0.75

b. Case Study 2:
A credit scoring model is used to predict whether an applicant is likely to default on a loan
(1) or not (0). Out of 1000 loan applicants:
90 applicants were correctly predicted to default on the loan.
40 applicants were incorrectly predicted to default on the loan.
820 applicants were correctly predicted not to default on the loan.
50 applicants were incorrectly predicted not to default on the loan.
Confusion Matrix
The Confusion Matrix Actual (Yes) Actual (No)
Prediction- Default-Yes 90 40
Prediction- Default-No 50 820
Accuracy = (TP + TN) / Total = (90 + 820) / 1000 = 91%
Precision = TP / (TP + FP) = 90/ 90+40 = 0.692
Recall = TP / (TP + FN) = 90 / 90+50 = 0.642
F1 Score = 2* (precision * recall)/precision + recall
= 2*(0.692*0.642)/(0.692+0.642) = 0.666

c. Case Study 3:
A fraud detection system is used to identify fraudulent transactions(1) from legitimate ones
(0). Out of 1000 transactions:
40 transactions were incorrectly identified as legitimate.
80 transactions were correctly identified as fraudulent.
850 transactions were correctly identified as legitimate.
30 transactions were incorrectly identified as fraudulent.
Confusion Matrix
The Confusion Matrix Actual (Yes) Fraudulent Actual (No) - Legitimate
Prediction- Fraudulent-Yes 80 30
Prediction- Legitimate-No 40 850

CL-X/AI/Evaluation 5 Rosary School

Accuracy = (TP + TN) / Total = (80+850) / 1000 = 93%
Precision = TP / (TP + FP) = 80/ 80+30 = 0.727
Recall = TP / (TP + FN) = 80 / 80+40 = 0.666
F1 Score = 2* (precision * recall)/precision + recall
= 2*(0.727*0.666)/(0.727+0.666) = 0.695

d. Case Study 4:
A medical diagnosis system is used to classify patients as having a certain disease (1) or not
having it (0). Out of 1000 patients:
800 patients were correctly diagnosed as not having the disease.
60 patients were incorrectly diagnosed as not having the disease
120 patients were correctly diagnosed with the disease.
20 patients were incorrectly diagnosed with the disease.
Confusion Matrix
The Confusion Matrix Actual (Yes) Disease Actual (No) Disease
Prediction- Disease-Yes 120 20
Prediction- Disease-No 60 800

Accuracy = (TP + TN) / Total = (120+800) / 1000 = 92%

Precision = TP / (TP + FP) = 120/ 120+20 = 0.857
Recall = TP / (TP + FN) = 120 / 120+60 = 0.666
F1 Score = 2* (precision * recall)/precision + recall
= 2*(0.857*0.666)/ (0.857+0.666) = 0.75

e. Case Study 5:
An inventory management system is used to predict whether a product will be out of stock
(1) or not (0) in the next month. Out of 1000 products:
50 products were incorrectly predicted to be out of stock.
800 products were correctly predicted not to be out of stock.
100 products were correctly predicted to be out of stock.
50 products were incorrectly predicted not to be out of stock.
Confusion Matrix
The Confusion Matrix Actual (Yes) out of stock Actual (No) out of stock
Prediction out of stock-Yes 100 50
Prediction- out of stock-No 50 800

Accuracy = (TP + TN) / Total = (100+800) / 1000 = 90%

Precision = TP / (TP + FP) = 100/ 100+50 = 0.6667
Recall = TP / (TP + FN) = 100/ 100+50 = 0.6667
F1 Score = 2* (precision * recall)/precision + recall
= 2*(0.6667*0.6667)/ (0.6667+0.6667)
= 0.6667 = 0.67

CL-X/AI/Evaluation 6 Rosary School