CS282BR: Topics in Machine Learning
Interpretability and Explainability
Hima Lakkaraju
Assistant Professor
Harvard Business School + Computer Science
Background
Strong understanding: Linear algebra, probability,
algorithms, machine learning (cs181 or equivalent),
programming in python, numpy, sklearn;
Familiarity with statistics, optimization
2
Motivation
Machine Learning is EVERYWHERE!!
[ Weller 2017 ]
Motivation: Why Model Understanding?
Input This model is
relying on incorrect
features to make
this prediction!! Let
me fix the model
Model understanding facilitates debugging.
Model Understanding
Predictive Prediction = Siberian Husky
Model
Motivation: Why Model Understanding?
This prediction is
Defendant Details biased. Race and
gender are being
used to make the
prediction!!
Model Understanding
Model understanding facilitates bias detection.
Race
Crimes
Gender
Predictive
Prediction = Risky to Release
Model
[ Larson et. al. 2016 ]
Motivation: Why Model Understanding?
Model Understanding
Increase
Loan Applicant Details
salary by I have some means
50K + pay for recourse. Let me
credit card go and work on my
promotion and pay
bills on time
my bills on time.
Model understanding helps
for next 3 provide recourse to individuals
months
who are adversely to
affected by model predictions.
get a loan
Predictive
Prediction = Denied Loan
Model
Loan Applicant
Motivation: Why Model Understanding?
Model Understanding
Patient Data This model is using
If gender = female, irrelevant features when
if ID_num > 200, then sick predicting on female
subpopulation. I should
If gender = male, not trust its predictions
25, Female, Cold if cold = true and cough = true, then for that group.
32, Male, No
31, Male, Cough
Model understanding helps assess
sick if and when to trust
model predictions when making decisions.
Predictions
Healthy
Sick
Predictive Sick
Model .
.
Healthy
Healthy 7
Sick
Motivation: Why Model Understanding?
Patient Data Model Understanding
If gender = female, This model is using
if ID_num > 200, then sick irrelevant features when
predicting on female
25, Female, Cold If gender = male, subpopulation. This
32, Male, No if cold = true and cough = true, cannot be approved!
31, Male, Cough then sick
.
Predictions
Healthy
Sick
Predictive Sick
Model .
.
Healthy
Healthy 8
Sick
Motivation: Why Model Understanding?
Utility Stakeholders
Debugging End users (e.g., loan applicants)
Bias Detection Decision makers (e.g., doctors, judges)
Recourse Regulatory agencies (e.g., FDA, European
commission)
If and when to trust model predictions
Researchers and engineers
Vet models to assess suitability for
deployment
Achieving Model Understanding
Take 1: Build inherently interpretable predictive models
[ Letham and Rudin 2015; Lakkaraju et. al. 2016 ]
Achieving Model Understanding
Take 2: Explain pre-built models in a post-hoc manner
Explainer
[ Ribeiro et. al. 2016, 2018; Lakkaraju et. al. 2019]
Inherently Interpretable Models vs.
Post hoc Explanations
Example
In certain settings, accuracy-interpretability trade offs may exist.
[ Cireşan et. al. 2012, Caruana et. al. 2006, Frosst et. al. 2017, Stewart 2020]
Inherently Interpretable Models vs.
Post hoc Explanations
can build interpretable + complex models might
accurate models achieve higher accuracy
Inherently Interpretable Models vs.
Post hoc Explanations
Sometimes, you don’t have enough data to build your model
from scratch.
And, all you have is a (proprietary) black box!
[ Ribeiro et. al. 2016 ]
Inherently Interpretable Models vs.
Post hoc Explanations
If you can build an interpretable model which is also adequately
accurate for your setting, DO IT!
Otherwise, post hoc explanations come to the rescue!
Let’s get into some details!
Next Up!
Define and evaluate interpretability
somewhat!
Taxonomy of interpretability evaluation
Taxonomy of interpretability based on
applications/tasks
Taxonomy of interpretability based on methods
[Doshi-Velez & Kim, 2017]
17
Motivation for Interpretability
ML systems are being deployed in complex high-
stakes settings
Accuracy alone is no longer enough
Auxiliary criteria are important:
Safety
Nondiscrimination
Right to explanation
18
Motivation for Interpretability
Auxiliary criteria are often hard to quantify
(completely)
E.g.: Impossible to enumerate all scenarios violating safety
of an autonomous car
Fallback option: interpretability
If the system can explain its reasoning, we can verify if
that reasoning is sound w.r.t. auxiliary criteria
19
Prior Work: Defining and Measuring
Interpretability
Little consensus on what interpretability is and how
to evaluate it
Interpretability evaluation typically falls into:
Evaluate in the context of an application
Evaluate via a quantifiable proxy
20
Prior Work: Defining and Measuring
Interpretability
Evaluate in the context of an application
If a system is useful in a practical application or a
simplified version, it must be interpretable
Evaluate via a quantifiable proxy
Claim some model class is interpretable and present
algorithms to optimize within that class
E.g. rule lists
You will know it when you see it!
21
Lack of Rigor?
Yes and No
Previous notions are reasonable
Important to formalize these notions!!!
However,
Are all models in all “interpretable” model classes equally
interpretable?
Model sparsity allows for comparison
How to compare a linear model with a decision tree?
Do all applications have same interpretability needs?
22
What is Interpretability?
Defn: Ability to explain or to present in
understandable terms to a human
No clear answers in psychology to:
What constitutes an explanation?
What makes some explanations better than the others?
When are explanations sought?
23
When and Why Interpretability?
Not all ML systems require interpretability
E.g., ad servers, postal code sorting
No human intervention
No explanation needed because:
No consequences for unacceptable results
Problem is well studied and validated well in real-world
applications trust system’s decision
When do we need explanation then?
24
When and Why Interpretability?
Incompleteness in problem formalization
Hinders optimization and evaluation
Incompleteness ≠ Uncertainty
Uncertainty can be quantified
E.g., trying to learn from a small dataset (uncertainty)
25
Incompleteness: Illustrative Examples
Scientific Knowledge
E.g., understanding the characteristics of a large dataset
Goal is abstract
Safety
End to end system is never completely testable
Not possible to check all possible inputs
Ethics
Guard against certain kinds of discrimination which are too
abstract to be encoded
No idea about the nature of discrimination beforehand
26
Taxonomy of Interpretability Evaluation
Claim of the research should match the type of the evaluation!
27
Application-grounded evaluation
Real humans (domain experts), real tasks
Domain experts experiment with exact application
task
Domain experts experiment with a simpler or partial
task
Shorten experiment time
Increases number of potential subjects
Typical in HCI and visualization communities
28
Human-grounded evaluation
Real humans, simplified tasks
Can be completed with lay humans
Larger pool, less expensive
Potential experiments
Pairwise comparisons
Simulate the model output
What changes should be made to input to change the
output?
29
Functionally-grounded evaluation
No humans, just proxies
Appropriate for a class of models already validated
E.g., decision trees
A method is not yet mature
Human subject experiments are unethical
What proxies to use?
Potential experiments
Complexity (of a decision tree) compared to other other
models of the same (similar) class
How many levels? How many rules?
30
Open Problems: Design Issues
What proxies are best for what real world
applications?
What factors to consider when designing simpler
tasks in place of real world tasks?
31
Taxonomy based on applications/tasks
Global vs. Local
High level patterns vs. specific decisions
Degree of Incompleteness
What part of the problem is incomplete? How incomplete
is it?
Incomplete inputs or constraints or costs?
Time Constraints
How much time can the user spend to understand
explanation?
32
Taxonomy based on applications/tasks
Nature of User Expertise
How experienced is end user?
Experience affects how users process information
E.g., domain experts can handle detailed, complex
explanations compared to opaque, smaller ones
Note: These taxonomies are constructed based on intuition
and are not data or evidence driven. They must be treated as
hypotheses.
33
Taxonomy based on methods
Basic units of explanation:
Raw features? E.g., pixel values
Semantically meaningful? E.g., objects in an image
Prototypes?
Number of basic units of explanation:
How many does the explanation contain?
How do various types of basic units interact?
E.g., prototype vs. feature
34
Taxonomy based on methods
Level of compositionality:
Are the basic units organized in a structured way?
How do the basic units compose to form higher order
units?
Interactions between basic units:
Combined in linear or non-linear ways?
Are some combinations easier to understand?
Uncertainty:
What kind of uncertainty is captured by the methods?
How easy is it for humans to process uncertainty?
35
Questions??
Relevant Conferences to Explore
ICML
NeurIPS
ICLR
UAI
AISTATS
KDD
AAAI
FAccT
AIES
CHI
CSCW
HCOMP 37
Breakout Groups
Say hi to your neighbors! Introduce yourselves!
What topics are you most excited about learning as part of
this course?
Are you convinced that model interpretability/explainability
is important?
Do you think we can really interpret/explain models
(correctly)?
What is your take on inherently interpretable models vs. post
hoc explanations? Would you favor one over the other? Why? 38