[Link]. Lectures Neural Networks Dr. Fatimah S.

Abdulsattar

Lecture 10: Model Interpretability vs Explainability

What is interpretability?
Interpretability refers to the visibility and understanding of the inner logic and mechanics
of the AI model. An AI model with high interpretability allows us to understand how the
components of the AI model (nodes and weights in deep neural network models) produce
a mapping between a system input and output.
 Interpretable AI describes how it makes the prediction.
What is explainability?
Explainability is the ability to describe the behavior of a system in understandable
language to humans. Explainability helps us understand what caused an AI system to
reach a prediction.
 Explainable AI describes why the AI model made a prediction.

White-box models vs. black-box models

White-box AI models have inputs and logic that are easy to see and understand. For
example, basic decision trees, which show a clear flow between each step, are not
difficult for the average person to decipher. White-box models tend to use more linear
decision-making systems that are easy to interpret, but can result in less accuracy or
fewer compelling insights or applications.
Black-box AI models are more complicated and offer less transparency into their inner
workings. The user generally doesn’t know how the model reaches its results. These
more complex models tend to be more accurate and precise. But because they are
difficult or impossible to understand, they come with concerns about their reliability,
fairness, biases and other ethical issues. Making black-box models more interpretable is
one way to build trust in their use.

Interpretability versus Explainability

Although interpretability and explainability are both important for understanding
Artificial Intelligence models, there are some key differences between the two concepts:
 Level of detail: Interpretability focuses on understanding the inner workings of the
models, while explainability focuses on explaining the decisions made. Consequently,
interpretability requires a greater level of detail than explainability.
 Model complexity: More complex AI models, such as deep neural networks, can be
difficult to interpret because of their intricate structure and the interactions between
different parts of the model. In these cases, explainability may be more viable, as it
focuses on explaining decisions rather than understanding the model itself.
 Communication: Interpretability concerns the understanding of the model by AI
experts and researchers, while explainability is more focused on communicating model

1|Page Computer Eng. Dept. M u s ta n s iri y a h U n iv.

[Link]. Lectures Neural Networks Dr. Fatimah S. Abdulsattar

decisions to end users. As a result, explainability requires a simpler and more intuitive
presentation of information.

The importance of Interpretability and Explainability

In general, interpretability and explainability are important because they provide insight
into how decisions are made by machine learning algorithms. This is especially important
in certain fields, such as medicine, where the choices made can have direct consequences
on people’s lives. Understanding how machine learning algorithms work can therefore
help to ensure that the decisions made by these algorithms are right and that errors are
minimized.
Interpretability and explainability are both essential to ensure that AI models are reliable,
secure, and adhere to ethical principles appropriate to the context. Here are some of the
reasons why these concepts are important:
 Responsibility: An AI model that is interpretable and explainable enables users to
understand the decision-making process and take into account the consequences of its
decisions. This is crucial to ensure accountability and transparency in the use of AI.
 Trust: Understanding AI models through interpretability and explainability can
increase user confidence in decisions made by AI-based systems. When users understand
how a model works and why it makes certain decisions, they are more likely to trust its
recommendations.
 Adaptation: An interpretable and explainable model allows developers to better
understand model performance and identify any problems or areas requiring
improvement. This facilitates adaptation and optimization of AI models over time.
 Regulatory compliance: Compliance with data protection and AI ethics regulations
often requires greater transparency in the decision-making process of AI models.
Interpretability and explainability are essential to ensure that models comply with these
requirements.
 Bias reduction: Understanding how AI models work through interpretability and
explainability enables the identification and reduction of bias in data and decision
making. This can help ensure that AI models are more equitable and do not discriminate
on the basis of sensitive characteristics, such as ethnicity, gender or disability

Levels of Explainability
 Global Explainability The aim of XAI at a global level is to explain the behavior of
the model across the entire dataset. It gives insights into the main factors influencing the
model and the overall trends and patterns observed. This is useful to explain to business
stakeholders how your model works.

2|Page Computer Eng. Dept. M u s ta n s iri y a h U n iv.

[Link]. Lectures Neural Networks Dr. Fatimah S. Abdulsattar

 Local Explainability The aim of XAI at the local level is to provide insights into why
a particular decision was made for a specific input.

Approaches to Explainability:
(1) Model-Agnostic Methods:
These methods can be applied to any machine learning model, regardless of its
architecture or complexity. They often treat the model as a black box and analyze its
input-output behavior to understand its decision-making process. Examples include
SHAP (SHapley Additive exPlanations) and LIME (Local Interpretable Model-Agnostic
Explanations).

(2) Model-Specific Methods:

These methods are tailored to specific types of models and leverage their internal
structures to provide explanations. For example, for linear regression models, the
coefficients and parameters can be used to understand the model's behavior. In deep
learning, techniques like activation maximization, saliency maps / Grad-CAM, and
layer-wise relevance propagation (LRP) can be used to understand how the model
processes information.

Intrinsic Interpretability vs. Post-hoc Explainability

Intrinsic Interpretability:
Some models are inherently interpretable due to their simple structure, like linear
regression or decision trees. Their explanations are directly derived from their structure
and parameters.

Post-hoc Explainability:
These methods are applied after a model has been trained, regardless of whether it's
intrinsically interpretable. They aim to provide insights into the model's decision-making
process after the fact. Post-hoc methods can be either model-agnostic or model-specific.

Use cases for explainable AI (XAI)

 Healthcare: Accelerate diagnostics, image analysis, resource optimization and medical
diagnosis. Improve transparency and traceability in decision-making for patient care.
Streamline the pharmaceutical approval process with explainable AI.

3|Page Computer Eng. Dept. M u s ta n s iri y a h U n iv.

[Link]. Lectures Neural Networks Dr. Fatimah S. Abdulsattar

 Financial services: Improve customer experiences with a transparent loan and credit
approval process. Speed credit risk, wealth management and financial crime risk
assessments. Accelerate resolution of potential complaints and issues. Increase
confidence in pricing, product recommendations and investment services.
 Criminal justice: Optimize processes for prediction and risk assessment. Accelerate
resolutions using explainable AI on DNA analysis, prison population analysis and crime
forecasting. Detect potential biases in training data and algorithms.

Gradient-weighted Class Activation Mapping (Grad-CAM)

Grad-CAM is an explainable AI (XAI) technique used to visualize the areas of an image
that are most important to a convolutional neural network's (CNN) classification decision.
It generates heatmaps highlighting these influential regions, providing insights into the
CNN's decision-making process.
The step wise process to generate CAMs are as follows:
i. Input:
A CNN model processes an image, and Grad-CAM focuses on a specific layer (usually
the last convolutional layer) and a particular class.
Let’s consider a CNN model 𝒇(𝒙) that takes an input image 𝒙 and predicts class scores.
 Let 𝒄 be the class of interest. The class-𝑐 logit is:
𝒚𝒄 = 𝒇(𝒙; 𝜽)𝒄
where the last convolutional feature maps are 𝑨 ∈ ℝ𝑲×𝑯×𝑾 , where 𝑲 is a number of
feature maps (channels), and 𝑯, 𝑾 are spatial dimensions.
ii. Calculate Gradients:
The gradients of the class logit, 𝒚𝒄 , are calculated with respect to each feature map, 𝑨𝒌𝒊𝒋 ,
of the chosen convolutional layer.
𝛛𝒚𝒄
, ∀ 𝒌 = 𝟏, … , 𝑲, 𝒊 = 𝟏, … , 𝑯, 𝒋 = 𝟏, … , 𝑾
𝛛𝑨𝒌𝒊𝒋

iii. Feature Map Weighting:

These gradients are used to weight the corresponding feature maps. Specifically, the
average gradient, 𝜶𝒄𝒌 , across each feature map channel, 𝑨𝒌𝒊𝒋 , is computed, and this average
serves as the weight for that channel. This is done by computing the average gradient by
the corresponding feature map (i.e., 𝜶𝒄𝒌 × 𝑨𝒌).
𝑯 𝑾
𝟏 𝛛𝒚𝒄
𝜶𝒄𝒌 =
𝑯𝑾 𝛛𝑨𝒌𝒊𝒋
𝒊 𝟏 𝒋 𝟏

𝒘𝒆𝒊𝒈𝒉𝒕𝒆𝒅_𝑨𝒌 = 𝜶𝒄𝒌 × 𝑨𝒌

4|Page Computer Eng. Dept. M u s ta n s iri y a h U n iv.

[Link]. Lectures Neural Networks Dr. Fatimah S. Abdulsattar

iv. Weighted Summation:

The weighted feature maps are summed together, resulting in a single matrix,
𝑳𝒄Grad-CAM , with the same spatial size of the convolutional layer. Then, ReLU (Rectified
Linear Unit) activation is applied to the summed matrix. This ensures that only positive
contributions (those that positively influence the target class) are retained, creating the
heatmap.
𝑲

𝑳𝒄Grad-CAM = ReLU 𝜶𝒄𝒌 𝑨𝒌

𝒌 𝟏

v. Upscaling:
The heatmap is upscaled to the original image size, allowing it to be overlaid on the input
image.
vi. Visualization:
The resulting heatmap, often displayed as a color-coded overlay, reveals the regions that
the CNN considered most relevant for classifying the input image as belonging to the
specified class.

Key Advantages of Grad-CAM:

 Localization: It accurately localizes the relevant parts of the image that influenced the
model’s decision.
 Model-Agnostic: Grad-CAM can be applied to a wide variety of CNN architectures
without the need for retraining.
 Interpretability: It helps in interpreting and explaining CNN predictions, which is
crucial in applications where understanding the basis of a decision is important (e.g.,
medical diagnostics, autonomous driving).

Use Cases:

5|Page Computer Eng. Dept. M u s ta n s iri y a h U n iv.

[Link]. Lectures Neural Networks Dr. Fatimah S. Abdulsattar

 Medical Imaging: Identifying which regions of an X-ray or MRI contributed most to

a diagnosis.
 Autonomous Driving: Understanding where the model is looking on the road to make
driving decisions.
 Natural Language Processing: Visualizing which words or phrases in an input text
are most influential in a classification decision.

SHAP (Shapley Additive Explanations)

It is the most widely used method in Explainable AI (XAI), due to the flexibility it
provides. SHAP values are model-agnostic, meaning they can be used to interpret any
machine learning model and come with the advantage of providing both local and global
level explanations, making our work easier.
SHAP values are a common way of getting a consistent and objective explanation of
how each feature impacts the model's prediction. They are based on game theory and
assign an importance value to each feature in a model. Features with positive SHAP
values positively impact the prediction, while those with negative values have a negative
impact.

Strategies for Applying SHAP to the Diabetes Dataset

SHAP offers an array of visualization tools for enhancing model interpretability, such as
(1) variable importance with the summary plot, (2) summary plot of a specific target,
and (3) dependence plot.

Example: Let us understand how Shapley’s values work with a hands-on example on
the diabetes dataset. This dataset is available to the public in Kaggle. First, load and read
the dataset.

You can see that we have features (X) like glucose level, blood pressure, etc., and the
target is ‘Outcome’. If it is 1, then we predict the patient to have diabetes and be healthy
if it is 0.
Second, a simple XGBoost model is trained on the training data by splitting the dataset
into 75% for training and 25% for testing. Third, after training, the model is used to
make predictions on the test data. Fourth, Let’s understand how to interpret SHAP
values locally – for a single prediction (Local interpretation).

6|Page Computer Eng. Dept. M u s ta n s iri y a h U n iv.

[Link]. Lectures Neural Networks Dr. Fatimah S. Abdulsattar

Here, the base value is the average prediction of the model. The contribution from each
feature is shown in the deviation of the final output value from the base value. Blue
represents positive influence, and pink represents negative influence (high chances of
diabetes).

(1) Variable Importance with Summary Plot

In this plot, features are ranked by their average SHAP values showing the most
important features at the top and the least important ones at the bottom. This helps to
understand the impact of each feature on the model’s predictions.

Below is the interpretation that can be made from the above graphic:
 We can observe that the red and blue colors occupy half of the horizontal rectangles
for each class. This means that each feature has an equal impact on the classification of
both diabetes (label=1) and non-diabetes (label=0) cases.
 However, Glucose, Age, and BMI are the first three features with the most predictive
power.
 On the other hand, Pregnancies, SkinThicknes, Insulin, and BloodPressure do not
contribute as much as the first three features.

Summary Plot on a Specific Label (Global interpretation)

Using this approach can provide a more granular overview of the impact of each feature
on a specific outcome (label).
In the example below, shap_values[1] is used to represent the SHAP values for instances
classified as label 1 (having diabetes).

7|Page Computer Eng. Dept. M u s ta n s iri y a h U n iv.

[Link]. Lectures Neural Networks Dr. Fatimah S. Abdulsattar

From the above graphic:

 Y-axis represents the features ranked by their average absolute SHAP values, similar
to the graphic in image 2.
 X-axis represents SHAP values. Positive values for a given feature push the model’s
prediction closer to the label being examined (label=1). In contrast, negative values push
towards the opposite class (label=0).
 An individual with a high glucose (red dots) level is likely to be diagnosed with
diabetes (positive outcome), while a low glucose level leads to not being diagnosed with
diabetes.
 Similarly, aging patients are more likely to be diagnosed with diabetes. However, the
model seems uncertain about the diagnosis for younger patients.

One way of dealing with this ambiguity for the Age attribute is using the dependence
plot to gain more insights.

Dependence Plot
Unlike summary plots, dependence plots show the relationship between a specific
feature and the predicted outcome for each instance within the data. This analysis is
performed for multiple reasons and is not limited to gaining more granular information
and validating the importance of the feature being analyzed by confirming or challenging
the findings from the summary plots or other global feature importance measures.
As can be seen from the plot below, the dependence plot reveals that patients under 30
have a lower risk of being diagnosed with diabetes. In contrast, individuals over 30 face
a higher likelihood of receiving a diabetes diagnosis.

8|Page Computer Eng. Dept. M u s ta n s iri y a h U n iv.

[Link]. Lectures Neural Networks Dr. Fatimah S. Abdulsattar

SHAP values Pros

 Transparency and Trust: By quantifying the contribution of each feature, SHAP
values increase the transparency and trustworthiness of machine learning models.
 Feature Importance: They provide insights into the global importance of features and
their impact on model predictions.
 Model Comparison: SHAP values facilitate the comparison of different models by
analyzing the contribution patterns of features.

SHAP values Cons

 Computational Expense: Calculating SHAP values can be computationally intensive,
especially for complex models and large datasets.
 Complex Interpretation: While SHAP provides detailed explanations, interpreting
these explanations can be challenging, requiring a deep understanding of the model and
the problem domain.
 Approximation Errors: In some cases, the SHAP values are approximations, which
may introduce errors in the explanations.

LIME (Local Interpretable Model-Agnostic Explanations)

It is widely used to explain black box models at a local level. When we have complex
models like CNNs, LIME uses a simple, explainable model to understand its prediction.
To make it even easier to understand, let’s see how LIME works in a step-wise manner:
1. Define your local point: Choose a specific prediction you want to explain (e.g., why
an image was classified as a cat by a CNN).

9|Page Computer Eng. Dept. M u s ta n s iri y a h U n iv.

[Link]. Lectures Neural Networks Dr. Fatimah S. Abdulsattar

2. Generate variations: Create slight variations of the input data (e.g., slightly modified
pixels in the image).
3. Predict with the original model: Pass the input to the CNN and get the predicted
output class for each variation.
4. Build an explainer model: Train a simple linear model to explain the relationship
between the variations and the model’s predictions.
5. Interpret the explainer: Now, you can interpret the explainer model with any method
like feature importance, etc. to understand which features played a crucial role in the
original prediction.

Benefits of LIME:
 Model-Agnostic: LIME can be applied to any deep CNN model, regardless of its
internal architecture.
 Localized Explanations: It provides insights into why a specific image was classified
in a particular way, which is crucial for complex black-box models.
 Visualizations: LIME generates helpful visualizations, such as heatmaps or highlighted
regions, to help humans understand which parts of the input image contributed most to
the model's decision.

Limitations of LIME
 Local Interpretability: Focuses on individual predictions, lacking insights into
overall model behavior.
 Sample Dependence: Explanation consistency varies with sampled instances, leading
to interpretation differences.
 Simplicity of Explanation Models: Simplified models (e.g., linear models) may not
fully capture complex model behaviors.
 Feature Space Approximation: Sampling may not accurately reflect real-world data
distributions, especially in high-dimensional spaces.
Permutation Feature Importance (PIMP)
10 | P a g e Computer Eng. Dept. M u s ta n s iri y a h U n iv.
[Link]. Lectures Neural Networks Dr. Fatimah S. Abdulsattar

Permutation Importance is an explainable AI technique that identifies the most

influential features in a machine learning model by measuring the drop in model
performance when a feature's values are randomly shuffled. A larger decrease in
performance after shuffling indicates the feature's greater importance to the model's
predictions. This method provides global interpretability by focusing on the overall
impact of each feature on the model's outcome and is applicable to any machine learning
model.

How Permutation Importance Works

1. Get a trained model and a dataset: Start with a pre-trained machine learning model
and a dataset, typically a test or validation set.
2. Measure initial performance: Evaluate the model's performance on the original,
unshuffled dataset.
3. Shuffle a single feature: Select one feature and randomly shuffle its values within
the dataset. This breaks the relationship between that feature and the target variable.
4. Measure performance on the shuffled data: Make predictions with the model using
the dataset containing the shuffled feature and measure the model's performance again.
5. Calculate the performance drop: The difference between the initial performance
and the performance with the shuffled feature quantifies how much the model relied on
that feature. A significant drop means the feature was important.
6. Repeat for all features: This process is repeated for every feature in the dataset, and
the average of performance drops is taken to get a reliable estimate.

PIMP Pros

11 | P a g e Computer Eng. Dept. M u s ta n s iri y a h U n iv.

[Link]. Lectures Neural Networks Dr. Fatimah S. Abdulsattar

 Feature importance provides a highly compressed, “global” insight into the model’s
behavior.
 PIMP does not require you to retrain the model.

PIMP Cons:
 PIMP is dependent on model error. PIMP cannot be relied in cases where we do not
have the true outcome data.
 Features need to be uncorrelated which is very difficult in real world problems.
 Not doing enough permutations in the computation of the feature importance can lead
to false/inaccurate results. On the other hand, running a lot of permutations involves long
processing times.

12 | P a g e Computer Eng. Dept. M u s ta n s iri y a h U n iv.