Module 5

XAI Methods
XAI Techniques - Local Interpretable Model-Agnostic Explanations (LIME) -
Understanding Mathematical representation of LIME - Shapley Additive
exPlanations (SHAP) - Diverse Counterfactual Explanations (DiCE) - Layer wise
Relevance Propagation (LRP).
Dr Vinothini A
Local Interpretable Model-Agnostic
Explanations (LIME)
• Local Interpretable Model-Agnostic Explanations (LIME), proposed by
Ribeiro, Singh, and Guestrin (2016), is an approach for fitting
surrogate models.
• LIME- an algorithm that can explain the predictions of any classifier or
regressor in a faithful way, by approximating it locally with an
interpretable model.
• Surrogate models are trained to approximate the predictions of the
underlying black box model.
• Local surrogate models are interpretable models that are used to
explain individual predictions of black box machine learning models.
LIME
Figure 2: Explaining individual predictions of competing classifiers trying to determine if a
document is about Christianity or Atheism. The bar chart represents the importance given to
the most relevant words, also highlighted in the text. Color indicates which class the word
contributes to (green for Christianity, magenta for Atheism).
LIME on Image
• The explanation produced by LIME is obtained by the following:

f → Black-box model.
g → Interpretable surrogate model
Belongs to a simple model family 𝐺(linear regression, decision tree, rule-based
model).
𝜋𝑥 𝑧 →Proximity function (locality), kernel function
Measures how close a perturbed sample 𝑧 is to the instance of interest 𝑥.
L(f,g, 𝜋𝑥 ​) → Local fidelity loss
Captures how well the surrogate 𝑔approximates the black-box 𝑓locally around 𝑥.
Ω(g) → Complexity penalty.
Keeps the explanation interpretable (sparse, simple).
Examples:
L1 regularization (forces many coefficients to zero), Restrict depth of decision tree.
Local Weighting (Proximity Kernel)
• Each perturbed point z is weighted by:
Algorithm
• Pick an instance 𝑥 to explain.
• Create many similar but slightly different samples (𝑧𝑖′ ​).
• See how the black-box model 𝑓 behaves on them.
• Weight them by closeness to the original 𝑥.
• Train a simplified, sparse linear model (K-Lasso).
• The coefficients 𝑤 tell us which features mattered most.
✓ The black-box model’s complex decision
function f (unknown to LIME) is
represented by the blue/pink
background, which cannot be
approximated well by a linear model.
✓ The bold red cross is the instance being
explained. LIME samples instances, gets
predictions using f, and weighs them by
the proximity to the instance being
explained (represented here by size).
✓ The dashed line is the learned
explanation that is locally (but not
globally) faithful.
For each sample 𝑖 from 1to 𝑁:
Shapley Additive exPlanations (SHAP)
• SHAP was introduced in 2017 by Lundberg et al.(Lundberg, S. M., &
Lee, S. I. A unified approach to interpreting model
predictions. NeurIPS. 2017)
• SHAP is a method that enables a fast computation of Shapley values
and can be used to explain the prediction of an instance x by
computing the contribution (Shapley value) of each feature to the
prediction.
• SHAP is an XAI approach that uses the Shapley value from cooperative
game theory to provide interpretable and explainable insights into the
factors that are most relevant and influential in the model's
predictions.
Shapley values come from cooperative game theory (Lloyd
Shapley, 1953).
•Imagine a game where multiple players cooperate to earn a
reward.
•The question: How do we fairly distribute the total reward
among players based on their contributions?
In XAI,
•Players = features in the model.
•Reward = model prediction (or prediction minus baseline).
•Goal = attribute each feature’s contribution fairly.
Compuation of Shapely Values
• Let’s choose feature A (a player in game theory) and compute the
Shapley values, , as the weighted sum of the marginal
contributions on different sets s:
The calculation of SHAP values can be computationally intensive, especially for
models with many features. There are two major approaches to calculate SHAP
values:
• Exact SHAP Values: This approach computes the Shapley values through
exhaustive enumeration of all possible combinations of feature subsets, which is
feasible for small feature sets.
• Approximate SHAP Values: For larger datasets, approximate methods are used.
Techniques such as Kernel SHAP and Tree SHAP are popular methods to estimate
SHAP values efficiently.
Example
Consider the given dataset.

ID x₁ x₂
1 0 0
2 0 1
3 1 0
4 1 1

Assume the model :

𝑓 𝑥 = 3 + 2𝑥1 + 4𝑥2
Explain the prediction for the instance ID 4 (x₁=1, x₂=1).
Solution :
Prediction: 𝑓 1 1 = 3 + 2 ⋅ 1 + 4 ⋅ 1 = 9.

Calculate Value function 𝑣 𝑆 = 𝔼 𝑓 𝑥 ∣ 𝑥𝑆

• The baseline is the model’s expected output when no feature
information is available.
Mathematically:
𝑣 ∅ =𝔼𝑓 𝑋
where:
𝑓 𝑋 =model’s prediction function
𝔼 𝑓 𝑋 =expected prediction, usually the average prediction over the
training dataset
• So, the baseline is the starting point before adding contributions from
any feature.
Calculate Shapely values for X1
Calculate Shapely values for X2
Check Efficiency
• Find sum of Shapley values :φ₁ + φ₂ = 1.0 + 2.0 = 3.0
Find the difference between prediction and baseline:
Difference= f(1,1) - v(∅) = 9 - 6 = 3
Thus the sum of Shapley values Matches correctly with the difference.
Interpretation
• Baseline prediction = 6
Final prediction for the instance = 9
Shapley attribution:
- x₁ contributes +1
- x₂ contributes +2
Thus, x₂ has a larger influence in raising the prediction above the baseline.
Layer-Wise Relevance Propagation(LRP)
• Layer-wise Relevance Propagation (LRP) is an Explainable AI technique
applicable to neural network models, where inputs can be images,
videos, or text.
• LRP calculates something called relevance in an iterative fashion from
output class neurons to the first input neurons.
• LRP achieves this by propagating the output prediction score
backward through the network layer by layer, redistributing the
output “relevance” back to the input features.
• LRP follows a relevance conservation rule similar to energy
conservation:
𝑙+1 𝑙
෍ 𝑅𝑗 = ෍ 𝑅𝑖
𝑗 𝑖
• That is, the total relevance at one layer is redistributed to the
previous layer without gain or loss.
• This ensures that the explanation is consistent and that the sum of all
input relevances equals the model’s output score.
Once the relevance scores for the input layer are computed, they can
be visualized as heatmaps:
• Red regions: Positive relevance — features that support the decision.
• Blue regions: Negative relevance — features that contradict the
decision.
• In image classification, this means highlighting the parts of the image
most responsible for the model’s output.
1. LRP-0 (Basic Rule / ε-rule baseline)

Formula:
𝑎𝑖 𝑤𝑖𝑗
𝑅𝑖 = ෎ 𝑅𝑗
σ𝑖 𝑎𝑖 𝑤𝑖𝑗
𝑗
• The simplest version distri butes relevance based purely on weighted
activations.
• Works for fully connected and convolutional layers.
• Problem: Can be unstable if the denominator is close to zero
(especially with negative activations).
2. LRP-ε (Epsilon Rule)

𝑎𝑗 𝑤𝑗𝑘
𝑅𝑗 = ා 𝑅𝑘
෍ 𝑎𝑗 ′ 𝑤𝑗 ′ 𝑘 + 𝜖
𝑗′
𝑘
where:
• 𝑅𝑘 :relevance of neuron k in the upper (next) layer
• 𝑅𝑗 :relevance of neuron j in the current layer
• 𝑎𝑗 :activation of neuron j
• 𝑤𝑗𝑘 :weight connecting neuron j → k
• 𝜖: small positive stabilizing constant
Gamma Rule (LRP-γ)

+
𝑎𝑗 𝑤𝑗𝑘 + 𝛾𝑤𝑗𝑘
𝑅𝑗 = ා 𝑅𝑘
෍ 𝑎𝑗 ′ 𝑤𝑗 ′𝑘 + 𝛾𝑤𝑗+′𝑘
𝑗′
𝑘
where:
• 𝑅𝑘 :relevance of neuron k in the next layer
• 𝑅𝑗 :relevance of neuron j in the current layer
• 𝑎𝑗 :activation of neuron j
• 𝑤𝑗𝑘 :weight connecting neuron j → k
• 𝑤𝑗+′𝑘 ​=max(wjk​,0): positive part of the weight
• 𝛾 ≥ 0: scaling factor (hyperparameter) controlling the emphasis on positive
contributions
• The parameter γ controls by how much positive contributions are
favored.
• As γ increases, negative contributions start to disappear.
• The prevalence of positive contributions has a limiting effect on how
large positive and negative relevance can grow in the propagation
phase. This helps to deliver more stable explanations.
LRP-αβ (Alpha-Beta Rule)

𝑎𝑖 𝑤𝑖𝑗+ 𝑎𝑖 𝑤𝑖𝑗−
𝑅𝑖 = ෎ 𝛼 + −𝛽 − 𝑅𝑗
σ𝑖 𝑎𝑖 𝑤𝑖𝑗 σ𝑖 𝑎𝑖 𝑤𝑖𝑗
𝑗
with the constraint that 𝛼 − 𝛽 = 1.
• Separates positive (supportive) and negative (inhibitory) contributions.
• By adjusting α and β, you can control focus:
• α = 1, β = 0 → only positive evidence
• α = 2, β = 1 → includes some negative evidence
• Used when we want to understand both why a model made a decision and
why not others.
LRP-z⁺ Rule (Positive Contributions Only)

+
𝑎𝑖 𝑤𝑖𝑗
𝑅𝑖 = ෎ + 𝑅𝑗
σ𝑖 𝑎𝑖 𝑤𝑖𝑗
𝑗
• Considers only positive weights.
• Ideal for networks with ReLU activations, where negative activations
carry little meaning.
• Ensures non-negative relevance maps , easier for visualization.
LRP-COMPOSITE (Layer-Adapted Rules)

• Combines different LRP rules for different layer types:

• LRP-ε for dense and convolutional layers
• LRP-γ for ReLU layers
• LRP-αβ for normalization or pooling layers
• This approach gives the most accurate and stable explanations in
deep CNNs and transformers.
Diverse Counterfactual Explanations (DiCE)
• Exploring “what-if” scenarios is an important way to inspect a
machine learning (ML) model.
• The DiCE library helps you to understand an ML model by generating
“what-if” data points that lead to the desired model output.
• “what-if” data points are known as counterfactuals, described by the
following question:
Given that the model’s output for input 𝑥 is Y, what would be the
output if input is changed to 𝑥 ′ ?
• When inspecting a classifier, for instance, we are often interested in
knowing the changes to that will lead to a desired predicted class.
• For a regressor, we may be interested in the changes to that lead to a
desired output range.
• Ideally, these changes should be proximal(minimal changes) to bring
out the local decision logic of the classifier, sparse to highlight a
limited set of features, and diverse to show the different ways in
which the same outcome can be achieved.
• The DiCE library provides an easy interface to generate such
counterfactual examples for any ML model.
• Another important metric for counterfactual examples is
their feasibility.
• If the changes in a counterfactual example are not feasible (e.g.,
outside the possible range of a particular feature), then the example
is less useful.
• Specifying the possible ranges for continuous features (and possible
values for categorical features) is one of the simplest forms of
feasibility.
• Counterfactual examples generated by DiCE are closely related
to necessity and sufficiency conditions for causing a given model
output.
• Necessity and sufficiency provide an intuitive way to explain a model’s
output. Given an input x and the model output y, a feature
value xi is necessary for causing the model output y if
changing xi changes the model output, while keeping every other
feature constant.
Finally, the counterfactuals for an input data point can be used to derive a local importance score for each
feature.
The local feature importance score ranks features by their frequency of being changed in the generated
counterfactuals. Among all the features, necessary features are likely to be changed more often to generate
proximal counterfactuals and therefore will receive a higher score.
DiCE methods
• To generate counterfactuals, DiCE implements two kinds of methods: model-
agnostic and gradient-based.
• Model-Agnostic: These methods apply to any black-box classifier or regressor.
They are based on sampling nearby points to an input point, while optimizing a
loss function based on proximity (and optionally, sparsity, diversity and
feasibility).
Currently supported methods are:
• Randomized Search
• Genetic Search
• KD Tree Search (for counterfactuals from a given training dataset)
• Gradient-Based: These methods apply to differentiable models, such as those
returned by deep learning libraries like tensorflow and pytorch. They are based
on an explicit loss minimization based on proximity, diversity and feasibility.
• import dice_ml
• DiCE supports sklearn, tensorflow and pytorch models.
• Given the train dataset, construct a data object for DiCE. Since
continuous and discrete features have different ways of perturbation,
specify the names of the continuous features. DiCE also requires the
name of the output variable that the ML model will predict.
d = dice_ml.Data(dataframe=train_dataset, continuous_features=['age',
'hours_per_week'], outcome_name='income')
# Using sklearn backend
m = dice_ml.Model(model=model, backend="sklearn")
# Using method=random for generating CFs
• exp = dice_ml.Dice(d, m, method="random")
method parameter specifies the explanation method.
DiCE supports three methods for sklearn models: random sampling,
genetic algorithm search, and kd-tree based generation.
• e1 = exp.generate_counterfactuals(x_test[0:1], total_CFs=2,
desired_class="opposite")
• e1.visualize_as_dataframe(show_only_changes=True)

• generate_counterfactuals method is the query instances on which

counterfactuals are [Link] can be a dataframe with one or more
rows.
import dice_ml
from dice_ml.utils import helpers

# Load data and model

data = dice_ml.Data(dataframe=df, continuous_features=['income', 'age'],
outcome_name='loan_status')
model = dice_ml.Model(model=model_obj, backend="sklearn")

# Generate counterfactuals
exp = dice_ml.Dice(data, model)
query_instance = [Link][0:1]
dice_exp = exp.generate_counterfactuals(query_instance, total_CFs=3, desired_class="opposite")
dice_exp.visualize_as_dataframe()
• DiCE frames counterfactual generation as a multi-objective
optimization problem, balancing proximity, diversity, and feasibility
while ensuring a desired prediction outcome.
 Feasibility Constraints
• To ensure that generated 𝑥 ′ are valid:
• Categorical features must take discrete values.
• Immutable features (e.g., gender, race) must remain unchanged.
• So, the optimization is subject to:
• 𝑥𝑗′ = 𝑥𝑗 if feature 𝑗 is immutable
• and
• 𝑥𝑗′ ∈ 𝒟𝑗 if 𝑗 is categorical
• 3. Combined Optimization
• The overall optimization problem becomes:
• where 𝜆1 , 𝜆2 , 𝜆3 are hyperparameters controlling the trade-off
between validity, proximity, and diversity.