Lecture Topic: Explainable Artificial Intelligence (XAI)
1. Introduction
Modern AI systems — especially deep learning models — often act as “black boxes.”
They make highly accurate predictions, but we don’t always know why they made a particular
decision.
Example:
A neural network classifies an X-ray as “malignant,” but a doctor cannot see which region or
feature contributed to that prediction.
➡️This lack of transparency leads to trust, accountability, and ethical concerns.
Explainable AI (XAI) aims to make AI systems’ decisions understandable to humans —
without sacrificing performance.
2. Definition
Explainable Artificial Intelligence (XAI) refers to techniques and methods that
make the behavior, decisions, and predictions of AI systems transparent,
interpretable, and understandable to humans.
Goal:
Enable users to trust, manage, and effectively use AI by understanding how it works and why
it makes certain outputs.
3. Why Explainability Matters
Reason Description
Trust Users need to trust model outputs, especially in high-stakes domains
(healthcare, finance).
Accountability Organizations must justify AI decisions to regulators and stakeholders.
Debugging Developers can identify model bias or data errors.
Ethical AI Ensures fairness, transparency, and non-discrimination.
Regulations Laws like GDPR require “right to explanation” for automated decisions.
4. The “Black Box” Problem
Deep neural networks have millions of parameters and complex nonlinear operations — making
it hard to trace how an input leads to an output.
Example:
A self-driving car model decides to brake suddenly — was it because of a shadow, a stop sign,
or a pedestrian?
Without interpretability, we cannot verify or correct the cause.
5. Levels of Explainability
Type Description Example
Global Understanding the model as a Which features generally affect
Explainability whole. predictions?
Local Understanding one specific Why did the model reject this loan
Explainability decision. applicant?
6. Categories of Explainable Models
A. Intrinsically Interpretable Models
These models are simple and transparent by design.
Examples:
● Linear Regression
● Decision Trees
● Rule-Based Models
● K-Nearest Neighbors (KNN)
They provide direct explanations:
E.g., “Income and credit score positively influence loan approval.”
B. Post-Hoc Explanation Methods
Used to explain black-box models (e.g., deep learning, ensembles).
They do not modify the model but analyze its outputs.
Examples:
● Feature importance
● Visualization of activations
● Surrogate models
● Counterfactual explanations
7. Key Techniques in Explainable AI
1. Feature Importance
Measures how much each input feature contributes to predictions.
Helps identify dominant or biased features.
Example: In a house price model, square footage might contribute 60%, location 30%, age 10%.
2. LIME (Local Interpretable Model-Agnostic Explanations)
● Creates local linear models around a prediction to explain it.
● Works for any model (model-agnostic).
Idea:
Perturb input data slightly → observe output changes → fit a simple model to approximate local
behavior.
Example:
For a text classifier, LIME highlights words that influenced the sentiment prediction.
3. SHAP (SHapley Additive exPlanations)
● Based on game theory (Shapley values).
● Distributes the prediction among all input features fairly.
Interpretation:
Each feature’s SHAP value shows its positive or negative contribution to the final output.
Example:
In credit scoring, SHAP can show that high income (+0.4) increased approval probability, while
high debt (-0.3)decreased it.
4. Saliency and Heatmaps (for CNNs)
Visualize which parts of an image activated the network most.
● Grad-CAM (Gradient-weighted Class Activation Mapping): highlights the regions
influencing a classification.
● Used in medical imaging, object detection, etc.
Example:
In a chest X-ray classifier, Grad-CAM can show which lung region led to the “pneumonia”
decision.
5. Counterfactual Explanations
Explain predictions by showing what needs to change in the input to alter the output.
Example:
“If your income increased by $2,000, your loan would have been approved.”
Helps users understand how to improve outcomes.
6. Surrogate Models
Approximate a complex model with a simpler, interpretable one (like a decision tree) to explain it
globally.
8. Example Use Case: Medical Diagnosis
● Black-box model: CNN trained to classify skin lesions.
● Explanation methods: Grad-CAM + SHAP.
● Output: Highlights lesion regions influencing classification and lists key texture/color
features.
● Benefit: Builds doctor’s confidence and allows error checking.
9. Trade-off: Accuracy vs. Interpretability
Model Type Example Accuracy Interpretability
Simple (white-box) Decision Tree, Linear Regression Moderate High
Complex (black-box) Deep Neural Networks, Ensemble High Low
Researchers aim to achieve both through explainable deep learning or hybrid models.
10. Applications of XAI
Domain Application
Healthcare Explaining disease predictions from medical images.
Finance Transparent loan approvals and fraud detection.
Autonomous Understanding decision-making in safety-critical moments.
Vehicles
Legal Systems Ensuring fairness in sentencing algorithms.
Human Resources Preventing bias in hiring decisions.
11. Challenges in Explainable AI
● Balancing accuracy vs. transparency.
● Defining what counts as a “good explanation.”
● Risk of misinterpretation of model behavior.
● Computational cost for large models.
● Maintaining privacy while explaining outputs.
12. Future Trends
● Causality-based explanations: linking features to actual cause-effect relationships.
● Natural language explanations: models that “talk back” (e.g., “I classified this as spam
because...”).
● Human-in-the-loop AI: collaboration between humans and interpretable models.
● Regulatory integration: mandatory explainability in high-risk AI systems (e.g., EU AI
Act).