BCSE418L

Explainable Artificial Intelligence

Dr Suguna M
Associate Professor
SCOPE, VIT Chennai
 Module 4
XAI Models
Ante-hoc Explainability (AHE) models - Post-hoc Explainability (PHE)
models - Interactive Machine Learning (IML) - Black Box Explanation
through Transparent Approximation (BETA) models - Hybrid Models.
XAI Models
Ante-hoc Explainability (AHE) models
• Ante-hoc Explainability (AHE) models are machine learning and deep
learning models that are inherently interpretable by design.
• Unlike post-hoc methods (where we try to explain an already-trained
“black-box” model), ante-hoc models are constructed in such a way
that their structure and functioning can be directly understood and
interpreted without requiring extra explanation tools.
Ante-hoc Explainability (AHE) models
Examples of Ante-hoc Explainability Models
• Linear/Logistic Regression
• Coefficients show direct influence of features on output.
• Decision Trees
• Path from root to leaf gives a clear explanation of predictions.
• Rule-based Models (e.g., RIPPER, Bayesian Rule Lists, Anchors)
• Use “if–then” rules that are easy for humans to follow.
• Generalized Additive Models (GAMs) and Explainable Boosting Machines
(EBMs)
• Show how each feature contributes to the outcome, separately and
additively.
• Case-based Reasoning & Prototype Networks(ProtoPNet)
• Classify based on similarity to interpretable “prototypes” or “examples.”
• Sparse Linear Models & Monotonic Models
• Enforce sparsity or monotonicity constraints for interpretability.
Ante-hoc Explainability (AHE) models
Characteristics of Ante-hoc Explainability Models
• Transparency by Design – The model’s parameters, architecture, or
rules are inherently interpretable.
• No Need for External Explainers – Explanations come directly from
the model itself, not from add-on methods like SHAP or LIME.
• Trustworthy in Sensitive Domains – Especially important in
healthcare, law, and finance, where interpretability is as critical as
accuracy.
• Trade-off – Usually less complex than deep black-box models, which
may limit predictive power for certain tasks.
Post-hoc Explainability (PHE) models
• Post-hoc Explainability (PHE) models are black-box machine learning
or deep learning models (like Random Forests, Gradient Boosting, or
Deep Neural Networks) that are explained after training using
additional tools or techniques.
• Since these models are highly accurate but complex, their decision-
making process is not inherently interpretable.
• Post-hoc explainability methods aim to "open the black box" and
provide human-understandable explanations without altering the
original model.
Post-hoc interpretability means
that we use an interpretability
method after the model is
trained.
Types of Post-hoc Explainability Techniques

1. Feature Attribution Methods

• Identify which features contributed most to a prediction.
• Examples:
✓ LIME (Local Interpretable Model-agnostic Explanations)
✓ SHAP (SHapley Additive exPlanations)
✓ Integrated Gradients (for neural networks)
✓ Feature Importance in tree ensembles
Types of Post-hoc Explainability Techniques
Visualization-based Methods
• Make the inner working of models visible.
• Examples:
✓ Saliency Maps (highlight influential pixels in images)
✓ Grad-CAM (class activation maps for CNNs)
✓ t-SNE / UMAP embeddings for feature space visualization
Example-based Explanations
• Explain predictions by showing similar instances.
• Examples:
✓ Counterfactual Explanations (e.g., “If your income were higher by $5000, you’d get
the loan”)
✓ Prototype & Criticism (show typical and atypical examples from training data)
Surrogate Models
• Train a simpler, interpretable model to mimic the complex one.
• Examples:
• Decision Tree surrogate for a neural network or ensemble
• Rule-based approximation
Interpretability methods to explain Black-Box
Model
• The most effective modeling approaches today are “black boxes”
models with mathematical behavior too complicated to directly
interpret the effect that individual input features have on the model’s
output. Fortunately, several interpretability methods exist which can
be layered on top of any arbitrary black box model to approximate
how:
✓ Individual predictions are locally influenced by feature values (local
interpretability)
✓ Individual features impact predictions globally across the model
(global interpretability)
Local Model-Agnostic Methods

• Ceteris Paribus Plots

• Individual Conditional Expectation (ICE)
• LIME
• Counterfactual Explanations
• Scoped Rules (Anchors)
• Shapley Values
• SHAP
Global Model-Agnostic Methods

Global Model-Agnostic Methods

• Partial Dependence Plot (PDP)
• Accumulated Local Effects (ALE)
• Feature Interaction
• Functional Decomposition
• Permutation Feature Importance
• Leave One Feature Out (LOFO) Importance
• Surrogate Models
• Prototypes and Criticisms
Ceteris Paribus Plots

• Ceteris paribus plots, in the context of machine learning, are

visualizations that show how a model's prediction changes when one
input feature is varied while all other features are held constant
(ceteris paribus, meaning "all other things being equal").
• These plots help in understanding the influence of individual features
on the model's output and can be used for model explanation and
comparison
How changing individual features changes the predicted number of bike rentals.
• X-axis: Number of bikes rented 2 days ago
• Y-axis: Predicted bike rentals for today.
• The line rises sharply: if there were more bikes rented 2 days ago, the model predicts more rentals today.
• Interpretation: This feature has the strongest influence — the model is capturing momentum.
Ceteris paribus curves for the bike prediction task for different models: linear model, random forest, SVM,
and decision tree.
• pyCeterisParibus is a Python library based on an R package CeterisParibus.
It implements Ceteris Paribus Plots.
• They allow understanding how the model response would change if a
selected variable is changed.
• It’s a perfect tool for What-If scenarios. Ceteris Paribus is a Latin phrase
meaning all else unchanged.
• These plots present the change in model response as the values of one
feature change with all others being fixed.
• Ceteris Paribus method is model-agnostic .It works for any Machine
Learning model. The idea is an extension of PDP (Partial Dependency Plots)
and ICE (Individual Conditional Expectations) plots.
• It allows explaining single observations for multiple variables at the same
time.
Individual Conditional Expectation (ICE)

An Individual Conditional Expectation (ICE) plot is a visualization

technique used to understand and interpret complex machine learning
models.
It displays the relationship between a specific feature and the model's
predictions for individual data points.
By examining these plots, practitioners can gain insights into how a
model relies on specific features, identify issues with model
predictions, and guide feature selection for model training.
ICE plots for the bike rental prediction.
The underlying prediction model is a random forest.
All curves seem to follow the same course, so there are no obvious interactions.
CE curves for the random forest predicting bike rentals. Lines are colored by the season.
ICE plots are boxplots showing the distributions of humidity per season.
Local interpretable model-agnostic
explanations (LIME)
• Local surrogate models are interpretable models that are used to
explain individual predictions of black box machine learning models.
Local interpretable model-agnostic explanations (LIME), proposed by
Ribeiro, Singh, and Guestrin (2016), is an approach for fitting
surrogate models. Surrogate models are trained to approximate the
predictions of the underlying black box model.
import lime
import lime.lime_tabular
Model to classify a ball either as a football or
a basketball.
Interpreting model decision using LIME
• It turns out that our classifier was correctly predicting a ball as a
football because of the human body parts, not because of the ball
itself.
• So our classifier wasn’t trying to classify between a football and a
basketball, but to classify between a human body parts and a
basketball.
• Obviously, it’s not what we want and thus, we shouldn’t trust our
model only based on its accuracy.
Counterfactual explanation
• A counterfactual explanation describes a causal situation in the form:
“If X had not occurred, Y would not have occurred.”
• For example: “If I hadn’t taken a sip of this hot coffee, I wouldn’t have
burned my tongue.” Event Y is that I burned my tongue; cause X is
that I had a hot coffee.
• Thinking in counterfactuals requires imagining a hypothetical reality
that contradicts the observed facts (for example, a world in which I’ve
not drunk the hot coffee), hence the name “counterfactual.”
• The ability to think in counterfactuals makes us humans so smart
compared to other animals.
• import dice_ml
• DiCE: Diverse Counterfactual Explanations for Machine Learning
Classifiers
• e1 = exp.generate_counterfactuals(x_test[0:1], total_CFs=2,
desired_class="opposite")
e1.visualize_as_dataframe(show_only_changes=False)
Scoped Rules (Anchors)
• The anchors method explains individual predictions of any black box
classification model by finding a decision rule that “anchors” the
prediction sufficiently.
• A rule anchors a prediction if changes in other feature values do not
affect the prediction.
• Anchors utilizes reinforcement learning techniques in combination
with a graph search algorithm to reduce the number of model calls
(and hence the required runtime) to a minimum while still being able
to recover from local optima.
Checking Wine Quality
• Anchors are high-precision rules that "anchor" a prediction by
showing which features, when fixed, keep the prediction stable.
• Example for a loan approval model:
• “If income > ₹50,000 and credit score > 700, then the model always
predicts ‘Approved’, regardless of other features.”
pip install anchor-exp
from anchor import utils
from anchor import anchor_tabular
Shapley values
• A prediction can be explained by assuming that each feature value of
the instance is a “player” in a game where the prediction is the
payout. Shapley values – a method from coalitional game theory – tell
us how to fairly distribute the “payout” among the features.
 Import shap for shapley values
import shap # `pip install shap` if neeed
We use the shap_values method from the SHAP library to get Shapley values.
We use the explainer method from the SHAP library to get Shapley values along with other data.
SHAP- SHapley Additive exPlanations
• SHAP is a Python library (and framework) that applies Shapley values
to explain machine learning models.
• It quantifies how each feature contributes to individual
[Link] tree-based models (like XGBoost, LightGBM),
linear models, and even deep learning models.
• Visualizes feature contributions with plots like:Force plots, Summary
plots, Waterfall Plots etc
SHAP-Summary Plot
Beeswarm plot
Waterfall plot
Partial Dependence Plot (PDP)
• A Partial Dependence Plot (PDP) shows the relationship between a
feature (or features) and the predicted outcome of a machine
learning model, while averaging out the effects of all other features.
• It helps answer questions like:
• “How does the model’s prediction change if this feature changes,
keeping everything else constant on average?”
 Partial Dependence Plot (PDP
display = PartialDependenceDisplay.from_estimator(
model,
X_test,
features,
kind='individual', # 'average' for PDP, 'individual'
for ICE, 'both' for both
subsample=50, # limit number of ICE lines
grid_resolution=20,
random_state=42
)
#[Link]("PDP for 'MedInc'")
[Link]("ICE Plot for 'MedInc'")
[Link]()
Accumulated Local Effects(ALE)
• Accumulated local effects describe how features influence
the prediction of a machine learning model on average.
• ALE plots are a faster and unbiased alternative to partial
dependence plots (PDPs).
• If features of a machine learning model are correlated, the
partial dependence plot cannot be trusted.
Accumulated Local Effects(ALE)
• pip install pyALE
• from pyale import ale
Feature Grid Binning
• The range of the feature Xj is divided into intervals (bins), typically
based on quantiles.
• This ensures each bin contains a roughly equal number of samples.
Local Effects Within Bins
• For each bin ALE measures how the prediction changes locally when
Xj changes from zk to zk+1, keeping all other features fixed.
Average Local Effects per Bin
• Average the differences over all data points that fall into the bin. This
gives the local effect for that bin.
Accumulate Effects
• Starting from a reference point (usually zero or the first bin),
accumulate the average effects over bins to get the accumulated local
effect.
Centering
• The final ALE function is centered to have a mean of zero:
Leave One Feature Out (LOFO) Importance

• The intuition behind LOFO Importance: If dropping a feature makes

the predictive performance worse, then it was an important feature.
If dropping the feature keeps performance unchanged, it was not
important.
• When you use LOFO Importance as information for feature selection,
beware of the interpretation: LOFO only indicates how the model
performance reacts to individually removing features.
• LOFO doesn’t give us the information on how the performance
changes when removing 2 or more features at once.
Surrogate model
• A global surrogate model is an interpretable model that is trained to
approximate the predictions of a black box model.
• Their purpose is to explain the behavior of the original model by
providing a transparent proxy that mimics its output.
Surrogate model
Perform the following steps to obtain a Surrogate model
• Select a dataset .This can be the same dataset that was used for training the
black box model or a new dataset from the same distribution. You could even
select a subset of the data or a grid of points, depending on your application.
• For the selected dataset get the predictions of the black box model.
• Select an interpretable model type (linear model, decision tree, …).
• Train the interpretable model on the dataset and its predictions.
• You now have a surrogate model.
• Measure how well the surrogate model replicates the predictions of the black
box model.
• Interpret the surrogate model.
Prototypes
• Prototypes are representative examples from the dataset that best summarize
the main patterns learned by a model.
• They act like “typical” examples of a class or cluster.
• Used mainly in instance-based explanations.
• Provide intuitive understanding of what the model considers “normal” for a class.
• Useful in domains like image recognition, text classification, and anomaly
detection.
Example:
• Suppose a model classifies animals.
• Class “Cat” prototype: an image of a typical domestic cat.
• Class “Dog” prototype: an image of a common dog breed.
Criticisms
• Criticisms are examples that are not well-represented by the prototypes,
highlighting atypical or unusual cases.
• They help identify gaps, biases, or rare patterns in the model or dataset.
• Provide insight into what the model struggles with.
• Useful for model debugging, fairness checks, and dataset improvement.
• Often, they are far from prototypes or in regions of low density in feature space.
• Optimization techniques like Maximum Mean Discrepancy (MMD) can be used to
select criticisms.
Example:
• In animal classifier:
• Criticism for “Cat”: a hairless cat or wild cat, which is very different from the prototype.
• Criticism for “Dog”: a very small or unusual breed that is not like the prototype.
Interactive Machine Learning (IML)
• Interactive Machine Learning (IML) is the design and implementation
of algorithms and intelligent user interface frameworks that facilitate
machine learning with the help of human interaction.
• interactive machine learning (iML) has acquired increased attention
of researchers on account of its human-in-the-loop modality and
relatively efficient resource utilization.
Types of Interactive Machine Learning

IML defines two distinct types of interactions:

✓ Human machine interaction
✓ Human feedback interaction.
• Human-Machine Interaction: In this paradigm, the machine and human
collaborate to generate the desired outputs. In this model, human interaction
can be used in a variety of ways, including selecting relevant features for analysis,
influencing the learning schedule, and identifying the significance of individual
data points.
• Human Feedback Interaction: In this model, the machine generates outputs
based on the provided data, and humans provide feedback to the machine that
helps augment the model for the next iteration. The algorithm can refine itself by
engaging in multiple human observations and correcting its accuracy until it
becomes more reliable.
iML
• Interactive machine learning (iML) is an active machine learning
technique in which models are designed and implemented with
human-in-the-loop manner.
• End-users participate in model building process by iteratively feeding
training parameters, inspecting model outputs and providing
feedback on intermediate results
Where does iML suit ?
• Interactive ML may suit
• Robust ML
• Trustworthy ML
• Low Resource ML
Merit Oriented Architecture
Adversial Attacks
• Deep learning algorithms are known for their vulnerability to adversarial attacks .
Adversaries can craftily manipulate legitimate inputs, which may be
imperceptible to human eye, but can force a trained model to produce incorrect
outputs.
• Adversarial threat model is comprised of the following three dimensions, namely:
• the attack surface : the overall data processing pipeline of machine learning from input to
output and then action.
• adversarial capabilities
• adversarial goals.
• .Evasion attack , poisoning attack and exploratory attacks are the main scenarios
considered under the attack surface.
• The aforementioned attacks can further be dissected into training phase and
testing phase attacks from theadversarial capabilities point of view.
Adversial Attacks
• Adversarial attacks have real impacts on the robustness of a deep
learning and other standard machine learning methods.
• Therefore, exploring potential adversarial attacks and building a
robust machine learning has been the focus of machine learning
researchers.
• The fundamental assumption is that assuring the explainability and
interpretability of black-box models by having the human-in-the-loop
reduce the vulnerability of machine learning models to adversarial
attack.
Adversial Attacks
• Data injection, data modification and logic corruption are among the
strategies of training phase adversarial capability attacks.
• White-box and black-box attacks are among the testing phase capability
attacks.
• Adversarial goal attack, on the other hand, infer adversary from the
incorrectness of the target model.
• It is also further classified as confidence reduction, misclassification,
targeted misclassification and source/target misclassification.
• Adversarial attacks have real impacts on the robustness of a deep learning
and other standard machine learning methods. Therefore, exploring
potential adversarial attacks and building a robust machine learning has
been the focus of machine learning researchers
iML to prevent Attacks
• iML has been used as a way to explain and explore model vulnerabilities to
adversarial attacks .
• Exploration and explanation of model vulnerabilities to test-phase or poisoning
adversarial attacks from the perspective of models, data instances, features, and
local structures.
• A white-box exploratory attack based approach, interactively visualizes neurons
and their connections inside a DNN that are strongly activated or suppressed by
an adversarial attack.
• In addition to explaining potentially perturbated inputs and models, iML based
methods propose frameworks that masks the discriminatory biases of black-box
classifiers.
• This plays a vital role to compensate the effects of perturbated inputs on a given
model.
iML for Model Building
• In addition to letting the user detect potential adversarial
attacks and managing its impacts, iML has also been used to
directly engage users in the model building process.
• This helps to avoid both training phase and test phase
adversarial attacks as the user will be there validating inputs
and intermediate results.
• A graph based framework, where user feedback is represented
as edges and nodes are the models is a good example of such
applications.
• In iML for Model building each iteration, the algorithm
proposes a model, and the user either accepts it or reveals a
specific mistake in the proposal.
• The feedback is correct only with probability p > 1/2 (and
adversarially incorrect with probability 1 −p), i.e.,
• thealgorithm must be able to learn in the presence of arbitrary
noise
Trust worthy iML
• To design and develop AI-based systems that users and the larger
public can justifiably trust, one needs to understand how machine
learning technologies impact trust Toreini et al. [2020].
• The trustworthiness of AI-based systems is directly related to how the
user is confident about the decisions made by the machine learning
components.
• This may include its perception about both the intelligent model and
knowledge
Low Resource Machine Learning

• Low resource machine learning is a process of building an analytical model

employing optimal resource utilization techniques.
• However, most machine learning algorithms have tradeoff between accuracy and
resource utilization. As it
• is stated in Preuveneers et al. [2020], the most accurate model might be
prohibitively expensive to computationally
• evaluate on a resource constrained environment. Consequently, the problem of
building accurate and high performance machine learning models has been
achieved at the expense of resource (data and computing) utilization.
• Sufficient amount of data and computing resources are not always at the stake.
• Besides, some problems may also require to be run on low resource setting. For
instance, the use of pervasive devices and robots to build model is one valid
scenario
Categories for Low Resouce ML
• small data machine learning and pervasive machine learning.
• Machine learning algorithms usually requires a large volume of data in
order to yield accurate result
• However, big data is not always at stake to be used in some problem
domains like under-developed languages,clinical trials, biomedical science
and etc.
• Achieving small data machine learning requires optimal utilization of data.
• Engaging domain experts in the model building process in an interactive
way would result in optimal utilization of
• important data items in a way contributing to the development of accurate
models with small data.
Application of iML
• has been applied to various game and entertainment research problems. A sketch-rnn, Ha and
Eck ,
• present an interactive recurrent neural network (RNN) that help to construct stroke-based
drawings of common objects.
• Besides their model encode existing sketches into a latent vector, and generate similar looking
sketches conditioned on the latent space.
• emphasize is on the design of a deep reinforcement learning agent that can play from feedback
alone. This algorithm takes advantage of the structural characteristics of text-based games.
• Moreover, the application of iML in motion-driven music systems
• Gillian. Due to the complications to design a robust player-recognition or
• motion recognition system using standalone iML system, an enhanced model has been presented
.
• They proposed an iML solution for Unity3D game engine in the form of a visual node system
supporting classification
• (with k-nearest neighbour), regression (with m
Pervasive Machine Learning

• Interactive machine learning can also yield a better performance in

pervasive computing environment where there are low computing
resources.
• For instance, Frazier and Riedpresent techniques to enable low
resource computation
• for deep reinforcement learning agents complex behaviors in 3D
virtual environments.
• Specifically, they considered an environment with high degree of
aliasing,
Black Box Explanation through Transparent
Approximation (BETA) models
• BETA (Black Box Explanations through Transparent Approximations),
a novel model agnostic framework, closely connected to Interpretable
Decision Sets (if-then rules).
• BETA learns a compact two-level decision set in which each rule
explains part of the model behaviour unambiguously.
• It uses an objective function so that the learning process is optimised
for high fidelity (high agreement between explanation and the
model), low unambiguity (little overlaps between decision rules in the
explanation), and high interpretability (the explanation decision set is
lightweight and small).
• BETA, a novel model agnostic framework for explaining the behavior
of any black-box classifier by simultaneously optimizing for fidelity to
the original model and interpretability of the explanation.
• Problem of constructing global explanations of black box classifier.
Explanations generated Beta on depression
dataset when approximating a deep neural
network
• BETA has two level decision sets as the representation of approximations. Œ
• The basic building block of this structure is a decision set which is a set of
if-then rules that are unordered. Œ
• The two level decision set can be regarded as a set of multiple decision
sets, each of which is embedded within an outer if-then structure, such
that the inner if-then rules represent the decision logic employed by the
black box model while labeling instances within the subspace characterized
by the conditions in the outer if-then clauses.
• Consequently, we refer to the conditions in the outer if-then rules as
neighborhood descriptors and the inner if-then rules as decision logic rules.
BETA RULE SET
✓ Outer Rule = Neighborhood descriptor
✓ Inner Subrules = Decision logic
✓ Defaults = fallback explanations
This is a two-level decision set:
• Outer condition (𝑞𝑖 → (defines a subgroup of people (like Age < 50
and Male = Yes)).
• Inner condition (𝑠𝑖 + Class label 𝑐𝑖( specific rules that apply within
that subgroup (like Past-Depression = Yes and Insomnia = No … →
Healthy)).
• Default rules → cover people who don’t fit any subgroup.
• default function - computes the majority class label (assigned by the
black box model)
• All the instances in the training data which do not satisfy anyrule in R
a are assigned with the majority label.
• For each instance which is assigned to more than one rule in R, we
break ties
• Tie breakers are rules which has a higher agreement rate with the
blackbox model.
• BETA constructs a small number of compact decision sets (sets of if-then
rules) each of which captures the behavior of the given black box model in
certain parts of the feature space.
• To ensure that the resulting explanations are faithful to the original model,
Beta chooses approximations based on how well they mimic the original
model in terms of assigning class labels to instances.
• Beta framework also unambiguously specif€ies the rationale used for
assigning labels to instances in any part of the feature space by ensuring
that each decision set and the corresponding decision rules explain non-
overlapping parts of the feature space.
• To ensure that the resulting explanations are interpretable, BETA not only
employ an intuitive rule based representation but also focus on minimizing
its complexity in terms of the number of rules, predicates etc.
• While the expressive power of two level decision sets is the same as
that of other rule based models (e.g., decision sets\lists\trees), the
nesting of if-then clauses in a two level decision set representation
enables the optimization algorithm to select neighborhod descriptors
and decision logic rules such that higher fidelity can be obtained with
minimal complexity thus resulting in more compact approximations
compared to conventional decision sets.
Rule 1: If Age < 50 and Male = Yes(Outer Rule = Neighborhood descriptor )
InnerRule1.1
If Past-Depression = Yes AND Insomnia = No AND Melancholy = No → Healthy
• ➝ Even though he had past depression, the absence of insomnia and
melancholy suggests recovery.
• InnerRule 1.2
If Past-Depression = Yes AND Insomnia = Yes AND Melancholy = Yes AND
Tiredness = Yes → Depression
• ➝ Multiple symptoms (insomnia, melancholy, tiredness) reinforce a depression
diagnosis.
Rule 2: If Age ≥ 50 and Male = No
Rule 2.1
If Family-Depression = Yes AND Insomnia = No AND Melancholy = Yes
AND Tiredness = Yes → Depression
• ➝ Family history + emotional + physical symptoms → high depression
risk.
Rule 2.2
If Family-Depression = No AND Insomnia = No AND Melancholy = No
AND Tiredness = No → Healthy
• ➝ No family history + no symptoms → likely healthy.
Default Rules
• Applied when a person doesn’t fall into Case 1 or Case 2.
Default 1:
If Past-Depression = Yes AND Tiredness = No AND Exercise = No AND
Insomnia = Yes → Depression
Default 2:
If Past-Depression = No AND Weight-Gain = Yes AND Tiredness = Yes AND
Melancholy = Yes → Depression
Default 3:
If Family-Depression = Yes AND Insomnia = Yes AND Melancholy = Yes AND
Tiredness = Yes → Depression
Evaluating BETA rules
• The goal of creating approximations which can meaningfully explain
the behavior of any black box model is guided by the following
properties:
✓ Fidelity
✓ Unambiguity
✓ Interpretability
✓ Interactivity