Occlusion method to obtain saliency

maps for CNN

Author: Thomas Eriksson

Supervisor: Mattias Ohlsson

A Bachelor report presented for the degree of

Bachelor of Science

Department of Astronomy and Theoretical Physics

Lund University
Sweden
Spring 2020
Abstract
This Bachelor project will study convolutional neural networks created for
image classification. Furthermore, it will specifically use an explanatory
model for how the network decided a certain classification output. This
is to increase the interpretability of the network. However, the completeness
of the explanatory model needs to be high for it to be useful. A saliency
map of how valuable each image pixel is for the classification will be created,
by occluding parts of the image. The MNIST dataset was used, which con-
tains handwritten digits. The main points of research were to study ways
to occlude or filter parts of the image. Among the researched topics were
the size of the filter, the number of filters and how the filtered pixels should
be alternated. The occlusion method to obtain saliency maps was compared
with rivalling methods, such as deepLIFT. The conclusion was that the large
amount of computational power needed limits the use of occlusion based
methods, but the high completeness makes it useful for niche purposes.

1
Contents
1 Introduction 3
1.1 General introduction . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Interpretability and completeness . . . . . . . . . . . . . . . . 3
1.3 Importance of an interpretable neural networks . . . . . . . . 4
1.4 Project introduction . . . . . . . . . . . . . . . . . . . . . . . 4

2 Related work 5
2.1 Occlusion based saliency maps . . . . . . . . . . . . . . . . . . 5
2.2 LIME/Anchor . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.3 DeepLIFT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

3 Method 7
3.1 Method introduction . . . . . . . . . . . . . . . . . . . . . . . 7
3.2 Filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3.2.1 Size of filter . . . . . . . . . . . . . . . . . . . . . . . . 9
3.2.2 Number of filters and type of filter method . . . . . . . 10
3.2.3 Optimization and comparison . . . . . . . . . . . . . . 11

4 Results and discussion 12

4.1 Comparing filters . . . . . . . . . . . . . . . . . . . . . . . . . 12
4.1.1 How many pixels should be flipped? . . . . . . . . . . . 13
4.1.2 How large should the filters be? . . . . . . . . . . . . . 13
4.1.3 How many filters are required? . . . . . . . . . . . . . 14
4.1.4 How can the pixels be alternated? . . . . . . . . . . . . 15
4.2 Defining features of figures . . . . . . . . . . . . . . . . . . . . 16
4.3 Comparing with deepLIFT . . . . . . . . . . . . . . . . . . . . 19
4.4 Occlusion saliency maps uses . . . . . . . . . . . . . . . . . . . 21

5 Conclusions 22

6 Appendix 23

2
1 Introduction
1.1 General introduction
Machine learning is a fast growing and wide field. There are many divisions
of the field that are paramount, artificial neural networks (ANN) being one of
them. The model of the ANN is loosely built on how the human brain works,
and one of the applications that ANNs are most known for is computer vision.
In computer vision it is common to use convolutional neural networks (CNN),
which uses sparsely connected, shared weights [1]. As the CNNs become more
complex, it is also harder to have an intuition of what they base their decision
on. This project will be working with explanatory artificial intelligence (XAI)
of CNNs with the use of saliency maps.

1.2 Interpretability and completeness

When discussing XAI, two of the central concepts is interpretability and
completeness of an explanatory model. The interpretability is how well the
explanatory model can be understood by humans. Therefore, there is a
subjectivity in the interpretability depending on the human interpreting the
explanatory model. The completeness is how well the explanatory model rep-
resents the actual model. An explanatory model with perfect completeness
would be a mathematical derivation of all the weights and nodes. However,
then the interpretability would be low, even for domain experts. One could
imagine an explanatory model which is very easily understood, but does not
reflect the actual model in any significant way. Then the interpretability is
high, but the completeness low. Clearly, the aim is to maximize the inter-
pretability as well as the completeness of the explanatory model, and herein
lies the challenge [2].
To gain knowledge about an explanatory model’s interpretability, F. Doshi-
Velez and B. Kim define three possible ways to evaluate them [3]. The first is
application-grounded evaluations, where experts evaluate the interpretabil-
ity of the explanatory model. If the AI is built for diagnostics a doctor
would evaluate the explanatory model. Secondly, a non-expert can perform
human-grounded metrics. Since the interpretability of a system has a subjec-
tive value depending on the viewer, it is important to distinguish the former
from the latter. The human-grounded and application-grounded metrics are
separate from each other, and an explanatory model can have a high in-
terpretability when considering application-grounded metrics, but low when
human-grounded. Finally, one can try and create some formal definition of
the interpretability, which then can be quantified and give scores to explana-

3
tory models without human interaction. This is then called functionally-
grounded evaluation. All of this is visualized in [3] using the figure seen in
figure 1.

Figure 1: Taxonomy of evaluation explanatory models. Image form F. Doshi-

Velez and B. Kim. [3].

1.3 Importance of an interpretable neural networks

There are many reasons why greater interpretability of ANNs is needed. A
vacuum in the interpretability of the ANNs is created, since the program-
mer does not know what the ANN has picked up during training. Having
experience creating ANNs will not necessarily lead to networks being more
interpretable. Therefore, there is a great need for explanatory models.
Having interpretable ANNs is crucial to knowing whether network works
correctly or not. If the user does not know which features the ANN has
based its decision on, conclusions could be drawn on false basis, for instance,
making classifications of cats based on the background. In a few cases it
is of utmost importance that the decision is correct, say if the network is
used for cancer diagnostics or self-driving cars. Furthermore, if the tool is
not interpretable the usability will decrease substantially. It is also common
sense that an increase of knowledge in a subject leads to faster and better
development of that subject. It is implied in the EU’s white paper regarding
AI that a fundamental understanding will soon be legally required for safe
usage in many fields [4].

1.4 Project introduction

As discussed in a study by Leilani H. Gilpin et al. [2], there are three main cat-
egories of XAI. First, one can look at individual parts of the model to increase

4
the interpretability. For ANNs that means looking at layers independently, or
even nodes. The second approach is to have explanation-producing systems,
where the networks themselves are interpretable to a higher degree. Exam-
ples are attention-based networks or disentanglement representation [2]. The
third and final focuses on the processing of the model, and tries to simplify
it. One possible way to do so is to use linear proxy models such as LIME
[5]. A proxy model is useful for some domains where its characteristics can
be explained in a simpler fashion. Decision trees often aim to simplify ex-
planation of how models work, although unfortunately they tend to become
quite complex in themselves. However, this project will focus on saliency
maps. Saliency maps give weighted values to the inputs on how important
they are for the output. Saliency maps belong to the third category of XAI
that explains the processes of the model [2].
This project aims to visualize what a CNN, built for classification, bases
its predictions on. The CNN will be built and trained to acquire a high test
accuracy. The test images will then be slightly perturbed before being used
as inputs. The difference in output, relative to the output of the unperturbed
image, is then stored. This will be repeated several times until a saliency
map of the pixels can be obtained. The method will be used on the MNIST
database [6]. The desired effect of the perturbations is to occlude parts of
the image to see how important they are for the classification. Therefore,
the method will also be called occlusion based. A detailed description of
the project is given in section 3. The evaluation will mainly be application-
grounded.
Comparison will primarily be done with backpropagating saliency maps.
Simply put, the backpropagating saliency maps are created by starting with
an output and then step by step going backwards through the network until
the input-layer is reached. However, the values in the input-layer will not
recreate the image but rather give values of importance for the classification.
These are then used to create the saliency maps. Note that there is a funda-
mental difference between the occlusion based and backpropagating methods,
as the first is a black box model and the latter is not. This is because the
weights are needed to propagate backwards through the network.

2 Related work
2.1 Occlusion based saliency maps
The most common use for occlusion saliency maps is to have them as a
reference map to check if the backpropagating saliency maps give an accurate

5
result. This is due to the larger computational power required to create
occlusion saliency maps compared to backpropagating models. Examples of
this use of occlusion maps can be seen in the paper by M. Ancona et al.
[7], where the occlusion maps are used as a reference but are not the main
topic of research. One example of occlusion based saliency map can be seen
in the work by Matthew D. Zeiler et al. [8], where it is used to investigate
the individual layers to gain knowledge about how the CNN functions. This
is a mixture of the main categories of XAI discussed in section 1.4, as the
processes of specific parts of the network is studied. However, in most of
the studies the research is limited to the occlusion methods with a square
of background colour travelling across the image. Since there is no/little
variation regarding the filter, size of filter, number of filters and placement
of filters a void of information appears to prevail.
Some human-grounded studies on saliency maps give insight on how much
interpretability is gained from using these maps. Even though they use an
algorithm called Layerwise Relevance Propagation and not occlusion, the
knowledge gained should apply to all saliency maps. They arrive at the
conclusion that while saliency maps are helpful, they need to be a part of
a larger explanatory model. The need for a non-instance based explanatory
model is evident [9].

2.2 LIME/Anchor
Contrary to this project, LIME uses a proxy model for interpretability. The
proxy model in use can vary, but it should be a local interpretable model-
agnostic explanation. This is what the acronym LIME stands for. The
proxy model needs to be interpretable. That is the main point of it, that
humans can understand what happens in that local area. These interpretable
models can be constructed to allow the complexity to decrease at a local level.
Often a linear approximation is used. Another key factor to LIME is that
it is model-agnostic. The main idea of LIME is to construct an explanatory
model which does not depend on the machine learning model [5]. When
LIME is used for image classifying CNNs, super-pixels will be constructed.
Super-pixels are pixels that have approximately the same contributions to
the classification, which then will be grouped together to create a super-
pixel. The pixel values in a super-pixel is set to the same value, that is the
average of the pixels in that super-pixel. One super-pixel should correspond
to one feature. The explanation model that will be obtained in LIME is
usually which super-pixels are the most useful for a specific classification.
The super-pixels that are essential to the classification together constructs
what is called the anchor. Everything except the anchor could be changed

6
Figure 2: Image from Ribeiro (2018) discussion on anchors. The anchor
constitutes what the CNN ”defines” as a beagle in this case. [10]

and the classification should still hold. The anchor of a classification of a

beagle can be seen in figure 2 [10].

2.3 DeepLIFT
DeepLIFT is a backpropagating explanation model. First, reference values
for inputs and outputs will be defined. A reference input value can be some
default value or some other natural base value for the model. Inputs reference
values for MNIST is an image with all black pixels. The reference output
values is then the values from having the network predict with the reference
inputs. The choice of the reference input value is crucial for the model. For a
prediction, the difference between an output value and its reference value can
be calculated. This difference can then be attributed, based on contribution,
to the various input-reference differences. The sum of all contributions is the
prediction output. When all is done, there is one contribution value for each
input. These contribution values can be plotted to create a saliency map
[11].

3 Method
3.1 Method introduction
There are two general ideas behind getting a saliency map over the image.
The first is a perturbation driven approach. This is done by introducing
various perturbations in the input, and tracking how that affects the output.

7
The second is by backpropagating through the gradients of the network. In
this project a brute force perturbation method will be used. It will sometimes
be referred to as occlusion based also, since the idea is to block out features
that the CNN uses to classify.
In this project the MNIST citegawd dataset was used. It contains 70000
handwritten digits in images with the size of 28x28 pixels. The CNN will be
trained on a subset of the total images, and then tested on another subset.
A network of the CNN model with two convolutional layers, and maxpooling
in-between was used. An MLP connected the convolutional layers to the
output nodes. A test accuracy of 99% was obtained. Since this project aims
to work with XAI this short description of the network will be sufficient.
The evaluation of the interpretability will be application-based. This is
because the dataset in use is MNIST, and anyone can make an assessment
of what constitutes a specific digit. Attempts of functionally-grounded eval-
uations were made, when comparing the result with deepLIFT. The main
evaluations will still be a qualitative assessment, focusing on whether or not
the features that the saliency map presents are plausible.

Figure 3: Showing the procedure to create the saliency map, with the dazed
lined procedures being done once all iterations are finished.

The trained CNN was used to predict digits on perturbed MNIST images.
Much of this project’s work was done by qualitatively investigating methods

8
to perturb the images. The perturbation methods will be discussed below
in section 3.2. The pixels that were perturbed was noted in an array. Once
the image is perturbed it was used as an input for the trained CNN. The
difference between the output value obtained from the perturbed image and
the original image was then stored in an array with the saliency map values.
However, since pixels are chosen randomly in many of the filters there is a
need to keep track of how many times a pixel has been used for normalization
purposes. Figure 3 visualizes the procedure. Each iteration will build up the
value array and the normalization array, but only once all the iterations were
done was the saliency map created. This procedure is iterated many times,
where the results seem to converge at 105 − 106 iterations for the majority
of the filters. An iteration contains all the steps seen in figure 3, except
the dazed-line steps. These two final steps is where the normalization array
and value array together create the saliency map. This is done after all the
iterations have been done. There is one iteration per filtered image.

3.2 Filters
The main idea behind selecting filters is to visualize what the defining features
are for each digit. Features are attributes that the digits posses, which may
or may not be important for the classification. Features can also be a lack of
certain attributes. A feature for the digit one might be that it has a straight
line, but another might be that it has nothing else around it. To be able to
find the important features the filter needs to block them. If too much is
filtered, noise can take the upper hand. It is therefore important to select
the optimal filter.

3.2.1 Size of filter

The first perturbation method was that every pixel had a probability of
being flipped to the opposite value, that is if it is zero(black) then it is set
to 255(white) and if it is non-zero it is set to zero. An example filter can be
seen in figure 4.a. A few probabilities for flipping were tried, and it became
clear that at high probabilities the noise became dominant.
The next step was to make larger local alterations, such as 2x2, 3x3 or
5x5 filters and see if larger features could be extracted. The idea is that the
probability of removing complete features with flipping random pixels is far
too low, and therefore there is a need for a more localized filter. All of the
sizes were tested using a simple flip. It is probable that the most efficient
size of filter depends on the size of the features. Therefore, if this method
is to be used on larger images than 28x28 it is probable that another filter

9
size is preferred. This is because the number of pixels used to block out a
feature completely is probably larger. Therefore, if a certain filter size is
the most proficient in feature extraction, it is then only applicable on this
specific dataset. Since features can be smaller or larger in the image, even
in the same dataset, it could be of interest to make multiple filter sizes when
working on a new dataset. When using only one filter there is no need for a
large number of iterations, instead the filter simply travels across the entire
image once. Since there is no random factor there is nothing to gain in doing
more iterations.

(a) Many 1 pixel flips. (b) Three 3x3 flip filters. (c) One 3x3 random.

Figure 4: MNIST images with various filters applied.

3.2.2 Number of filters and type of filter method

Once the size of the filters has been established, the next step was to look
at the number of filters. Having smaller filters, but more than one, can give
both the capacity of covering entire features while still having the benefits of
being a small filter. The placements of the squares were randomized. Three
3x3 filters can be seen in figure 4.b.
Two other filter methods were tested. The first was to give the value of
the background rather than flipping the value. For this dataset it corresponds
to the filter setting all its pixels to zero(black). A problem with this approach
is that it has no way to account for the background. The CNN might give a
specific classification because of the lack of certain pixel-values just as much
as the presence of some. Therefore, it does not seem like a filter method with
a high completeness. The third tested filter method was to give a randomized
value in the grayscale range to each pixel in the filter. An example of this
can be seen in figure 4.c.

10
3.2.3 Optimization and comparison
The final addition to the filters was optimization. The idea was that the
interesting data will mainly be around the digit, and not at the edge of the
image. By reducing the number of inputs that is far from the digit, the
desired results can be obtained quicker. The first attempt of an optimization
was done by adding all the grayscale values together in an array. The array is
the sum of all the MNIST images together. This leads to each array element
being the sum of the corresponding grayscale values in the MNIST figures
together. This can then be used as a probability distribution for drawing
values at where to place the filter. This is visualized in figure 5. Here the first
two MNIST images are added together, and then printed. Since the figure
5.c will have pixel values over the maximal of 255 it gives a false picture, but
helps to visualize the idea. All the MNIST images were added together, and
not just these two. However, this method was abandoned, since values at
the edge of where digits appear in the images would often only be used once.
All pixels need to be drawn many times to have a statistical foundation. If
a saliency map of one of the digits that are at the edge of the probability
distribution is desired, the map would look completely inaccurate.
A simpler optimization was done with a check if the filter was within
the digit or not. If it was not within the digit it had a probability of being
rejected. One thing to keep in mind is the domain of the CNN. If there are
no digits in the training set at the edge of the image, but in the filtration
that area is suddenly flipped, it might give strange/incorrect results.

(a) (b) (c)

Figure 5: Conceptual images for probability distribution optimization at-

tempt.

From the various tests a filter method was established, and then com-
pared to deepLIFT and LIME/anchor. DeepLIFT was compared with this

11
project’s method by the use of structural similarity index(SSIM), but be-
cause of the fundamental differences between the methods, the SSIM scores
did not contain much valuable information.

4 Results and discussion

This section will start with looking at the the various filters and approaches
used. These will be presented and discussed in a qualitative investigation on
how well the various methods detect crucial parts of the classification. One
method will then be selected and compared with deepLIFT and LIME. A
discussion about where this selected method can be used instead of leading
methods, and then finally a general discussion will be presented. Note that
even though only a few images are used to present the result, many more
were investigated. The testing was not done solely on the ones shown.
The black and white images show how the MNIST figures have been
filtered. The red, white and sometimes blue images show the saliency map.
Higher contrast of red means that those pixels have a high positive impact on
the classification of the figure. White means that is does not contribute to the
classification. If blue pixels are seen, that means that those pixels contribute
negatively to the classification. In other words, changing the pixel would lead
to better classification. It should be said that with the method used there
can be parts of the image which gives negative contributions to the output
but still is shown as red or white when using multiple filters at once. These
unwanted effects need to be weighted against the potential of detecting more
complex features.
Note that the network has been trained multiple times between the sec-
tions. Therefore, the features that the network uses for classification can vary
between sections. This means that the same image can yield two different
saliency maps depending on what the network detected during that training.
However, the same trained network is used when one property is analysed.
For example, section 4.1.1 uses the same trained network to analyse the num-
ber of pixels, but then a newly trained network is used in 4.1.2 to analyse
the size of the individual filters.

4.1 Comparing filters

Here the results from using various filters will be presented. These results
will largely correspond to the methods discussed in section 3.2.

12
4.1.1 How many pixels should be flipped?
Two examples when random pixels were flipped are presented in figure 6.
Figure 6.a and 6.c show examples of how an occlusion can look. As can
clearly be seen, the percentage of filtered pixels cannot be too large, otherwise
the noise will be too dominant. It is important to note that the figures are
normalized individually, and does not share intensity-value relations. While
comparing figure 6.b and 6.d, it might seem intuitive that figure 6.c gives
a better map since more of it is of higher intensity. This is however false,
the general higher intensity is noise from coupling high impact pixels with
all the rest. While there still are features that can be extracted from the
saliency map, they are less clear against the background. Indeed, this is
what intuition also tells us, flipping too many pixels should yield more noise.
There should be a larger possibility of the 10% flip to catch more of the larger
feature, but none can be seen here that were not captured by the 1% flip.
From now on a few percentages of the image will be filtered.

(a) 1% flipped (b) 1% flipped (c) 10% flipped (d) 10% flipped

Figure 6: The percentage of filtered pixels tested, here using 1x1 flipping
filters.

4.1.2 How large should the filters be?

The next step now is to look at various sizes of occlusion. This was done
with a single flipping filter, and results may vary when using other filter
methods. As discussed in the section 3.2, these sizes will be dependent on
the database. Size dependency will also exist within datasets, as feature sizes
will vary even there. The results can be seen in figure 7, with four different
sizes of filters. All of these only walk across the image once, since more would
be redundant. In figure 7.a one filtering of size 2x2 is shown, along with its
saliency map in figure 7.e. Filter using 3x3 is shown in figure 7.b and 7.f.,
filter using 4x4 shown in figure 7.c and 7.g and finally filter using 5x5 shown
in figure 7.d and 7.h. In this example all the features can be seen in all the

13
sizes. The 2x2 filter gives a more detailed view of individual pixels, while the
bigger filters seem to emphasize the feature in the middle of the image more.
Which filter size that yields the highest completeness depends on whether
the middle feature actually is the most important or not.
Moving forward, the smaller filters, 2x2 and 3x3, will be regarded as the
superior size for MNIST as they still detected all the features.

(a) 2x2 (b) 3x3 (c) 4x4 (d) 5x5

(e) 2x2 (f) 3x3 (g) 4x4 (h) 5x5

Figure 7: Testing of filter size, with filtered inputs and saliency maps.

4.1.3 How many filters are required?

The next step is to investigate the number of filters in the image. The knowl-
edge gained from section 4.1.1 and 4.1.2 is now useful to select which combi-
nations of filters should be used. A few percentages of the image should be
filtered using 2x2 or 3x3 filters. From this, four combinations were selected:
three 2x2 filters, four 2x2 filters, two 3x3 filters, and four 3x3 filters. All of
these filters are still under the 10% of the image as seen in section 4.1.1, to
minimize the noise and make the features more interpretable. Examples of
all of these filters and results can be seen in figure 8.
Going from three 2x2 filters to four 2x2 filters does not change much for
the feature detection when it comes to the digit one. However, with digit
seven it detects more of the digit as an important feature as seen in the
appendix figure 15.a and 15.b. It can also be seen that the features selected
does not change much when it comes to the fours, as they seem to have a

14
rigorous feature in between the peaks. This can be seen in figure 16 in the
appendix.
The same tendencies can be seen with the 3x3 filters. All the features
were detected although the features cover a larger area. When using two 3x3
filters there seem to be less noise in the general background. In both figure
15.g and 16.g the same effect can be seen, where the features are slightly
larger but the map otherwise have less noise. Not much was gained from
using three 3x3 filters, but more noise was added. For this dataset three 2x2
or two 3x3 filters seem to give the most complete result.

(a) 3st 2x2 (b) 4st 2x2 (c) 2st 3x3 (d) 3st 3x3

(e) 3st 2x2 (f) 4st 2x2 (g) 2st 3x3 (h) 3st 3x3

Figure 8: Testing number of filters, with filtered inputs and saliency maps.

4.1.4 How can the pixels be alternated?

Then next step is to further investigate how pixel values can be modified
within the filters. Up until now the filtering method has been a simple flip,
i.e. pixels have been switched from black to white and vice versa. Now two
other methods will be tested. The first is by far the most common in current
research, which uses the background of the images as new values for the pixels
in the filter. For MNIST the background is always zero/black. The other
is random values, which for MNIST correspond to random grayscale values.
In figure 9 the results are presented. As can be seen, using the background
as the filter method means that there is no alternation for the background.
Therefore, if any features are in the background of the image they cannot be

15
detected. However, it does give more details over the digit itself. For filters
with random pixel values (figure 9.c) the results looks like a combination
between the saliency maps seen in figure 7.e and 7.g. It gives a bit more
detail than the simple flip, and out of the tested alternations the random
value seems to yield the most satisfactory results.

(a) Flip (b) Background (c) Random values

(d) Flip (e) Background (f) Random values

Figure 9: Testing various alternations to pixel, including flip, background

values and randomized values.

It is however important to state that there is not one selection of a filter

that does everything satisfactorily, there is always a trade-off. If an overview
of all features in a figure is required, then perhaps many smaller filters are
preferred. If main features of what distinguishes one figure from the other is
desired, then larger filters are more useful. A combination of them side-by-
side might give higher completeness than they can achieve individually. It is
largely the same for filtering methods. The random value and flip are rela-
tively close to each other, with the same trade-off as smaller or larger filters.
However, filtering the image as the background gives completely different
results, with its own strengths and weaknesses.

4.2 Defining features of figures

Based on the discussion in section 4.1, we from now on use two 3x3 filters
with random value to investigate specific features of digits in MNIST, to test

16
if the network is more interpretable.
In some cases it is easily understood why that features is dominant. One
of the clearest examples is images with a number four. In figure 10 the
saliency map of fours can be seen. What they all have in common is that
the area between the two peaks in a four is the high intensity area. This is
because if a filter fills in this area it becomes a nine instead. Indeed, the four
is the only figure with two independent peaks with an area between them.
This gives an interpretable explanation of what information the CNN uses
to classify a figure as a four.

(a) (b)

(c) (d)

Figure 10: The features of the number four.

In some other cases it is not as clear. For example, sevens can be seen
having as sharply defined feature, but the reason behind it is not as clear.
Figures 11.a and 11.b give the information that there seems to be two crucial
points in the figure. This is not what would be expected, since these parts
of the seven are identical to parts of a slightly tilted number one. Perhaps it
could be explained by looking at the saliency maps of the ones, as in figure
11.c and 11.d. Here some clarity is gained since the ones seems to have their
main features in the background around them. The CNN seems to define
ones as a line without anything around it, and if there is anything around,
it stops being a one. Now the sevens saliency map makes more sense. With
that said, it could very well be a simple case of overtraining when it comes
to the seven.

17
 (a) Saliency map of seven (b) Saliency map of seven

(c) Negative space illustrated by ones (d) Negative space illustrated by ones

Figure 11: The features of the number seven and the number one.

Various digits with less dominant features is shown in figure 12. Here the
saliency maps does not provide clearly interpretable features. Figure 12.a
shows a digit two, with a few features having high intensity. Meanwhile,
figure 12.b shows many features of equal importance. A possible explanation
for the bottom feature of figure 12.a is the resemblance of the digit 3, if the
area of this feature is filled with white pixels.
Two chaotic examples are shown in figure 12.c and 12.d. Here the ran-
domized filters are almost as likely to make the image look more like a four
as it is to make it look less like a four. As mentioned before, the blue areas
are where changing the image results in better classification.

18
 (a) two with concentrated feature (b) Two with spread features

(c) Chaotic five (d) Chaotic four

Figure 12: Detected feature in the number two and chaotic numbers.

4.3 Comparing with deepLIFT

Because of the fundamental differences between deepLIFT and this project’s
method there will mainly be a qualitative comparison between them. The
main focus points will be if the information gained in section 4.1 and 4.2 can
be obtained with deepLIFT. Thereafter, there is a discussion if deepLIFT
can detect features unavailable for occlusion based saliency maps. If a more
direct comparison with a functionally-grounded evaluation is wanted, then
occlusion maps filters need to be selected to match deepLIFT. First, one
would need to use the background filter instead of flip or random value. Sec-
ondly, the background needs to be accounted for since deepLIFT accounts for
the background. This is far from this project’s main method, and therefore a
direct functionally-grounded evaluation between them is not desirable. If the
called for result is a saliency map which can be used as the correct answer
for gradient based saliency map, then this would be the required approach.
When comparing deepLIFT with this project’s occlusion method a few
observations can be made. The first is that the feature detected in the
images with the digit four is harder to detect with deepLIFT. In figure 13.b
the deepLIFT equivalent of figure 10.a is shown. Here the most important
feature seems to be the two bars in the four, and not the space between
them. However, deepLIFT does not have any possible way to mark the
area between the peaks as important and that is why there is fundamental

19
difference between deepLIFT and this project’s method. The same drawback
can be seen with the number one in figure 13.a. The space around the
number cannot be accounted for, and therefore deepLIFT seems to loose
completeness.

(a) (b) (c) (d)

Figure 13: DeepLIFT saliency maps of various numbers.

In other cases where the features are within the digit itself deepLIFT
should be as complete or even more complete than this project’s method.
Examples of this can be seen in figure 13.c and 13.d. From the study of
the features of sevens, it is known that they are within the digit itself. In
deepLIFT a detailed map over each pixel can be seen. From figure 13.c it
seems like it was the middle feature from figure 7.a-7.d that indeed was the
most important. For the number nine in figure 13.d the top part can be
seen as being important, which is what is expected since it would create a
four otherwise. It can also be seen that the left part of the circle in the nine
is important, perhaps because it starts looking like a three if that part is
removed. Here it seems more beneficial to use deepLIFT then any occlusion
method.
A functionally-based evaluation was done using structural similarity index
(SSIM). One example of SSIM evaluation is shown in figure 14. Figure 14.a
shows the background for the number 5. Figure 14.b shows the saliency
map obtained using a 1 pixel background filter. In figure 14.c the occlusion
saliency map can be seen when the background is taken into account, and
next to it in figure 14.d is the deepLIFT saliency map. These two images
were then evaluated using SSIM. Since this is a sidetrack from the main
method the math of SSIM will not be discussed. However, if a value of 0 is
obtained then there is no similarity between the images, and if a value of 1
is obtained it is the same image. SSIM between figure 14.c and 14.d gave
a value of 0.4. While this value seems low, some similarities in the general
structure can still be seen in when qualitatively comparing figure 14.c and
14.d.

20
 (a) Background (b) Occlusion (c) Combination (d) DeepLIFT

Figure 14: The background, occlusion method, a combination of the two

former and deepLIFT saliency maps of the digit 5 are shown.

LIME and anchor are closer to this project’s method in its results since it
can detect the features seen in the background around the figure. However, it
will not yield a saliency map that can be compared using SSIM or any other
functionally-based evaluation tool. One possible way to construct LIME
super pixels is to divide the image into the parts which contribute positively
to the classification, and those that contribute negatively. This binary system
results in high interpretability, but lower completeness.

4.4 Occlusion saliency maps uses

As discussed in L. G. Leispin et al. [2], the assessment for how well a saliency
map performs is determined by its completeness. A complete saliency map
would give the exact importance of each pixel. It is widely accepted that oc-
clusion based saliency maps achieve higher completeness then gradient based
ones. However, to achieve that, it needs much more computational power.
Occlusion methods are more computationally heavy since they need many it-
erations to obtain results, while backpropagating methods only goes through
the weights once. Therefore, use of occlusion methods is limited. The most
obvious is using occlusion maps as a form of correction when creating the gra-
dient based maps. It is then of utmost importance that the occlusion maps
detect as many features as possible, since otherwise these features will per-
haps go unnoticed through-out the next generation of gradient based saliency
maps. Therefore, it is relevant research to try and maximize the features de-
tected in the occlusion saliency maps. Other possible uses, over gradient, is
when a higher order of certainty of the completeness is needed. Examples of
this might be if decisions with a big impact on human lives, in one way or
another. A self-driving car using trained networks should perhaps be tested
with occlusion based saliency maps before being used. The saliency maps are
not enough though since they are instance based. To get a fuller picture of

21
how a classifying CNN works multiple tools should be used, where saliency
maps is one of them.

5 Conclusions
When using CNN it is crucial to be able to interpret its output. A useful tool,
when it comes to visual classifying CNN’s, is creating saliency maps of what
parts are of importance. These should be seen as a part of a larger explana-
tory model, partly because no model is perfect, and gaps of completeness
should be expected, and partly because its nature as instance based explana-
tory model is limited. It needs to be combined with a global explanatory
model. Nonetheless, saliency maps give a rare combination of high com-
pleteness and interpretability.
Occlusion based saliency maps are limited in its practical use since they
require much computational power. Therefore, the future is probably in gra-
dient based methods, such as deepLIFT. For now, occlusion based methods
do have higher general completeness and therefore there are still some niche
uses for them. The first being the use of the occlusion methods results as a
way to correct the gradient based methods. The second being when a saliency
map needs to have very high completeness in itself, such as for self-driving
cars.
This project evolved a method through testing various filters. The result
ended up being that a few smaller sized filters with random grayscale values
gave the most complete result. Some parts of the selection is obvious, while
other parts is harder. If two filters contradict one and other, and it is not
obvious which one gives the more complete picture, the decision becomes
hard.
It has throughout this project been shown that even in occlusion maps
there are many variables that can be altered. Generally it is a trade-off
between having higher feature detection and more noise. Larger and more
filters are able to construct more complex filters, which then in turn can
detect more complex features. However, as the size and number of filters
increase so does the noise. The noise will heavily alter the values of all the
non-essential areas, and therefore decrease the completeness in these areas. If
occlusion maps are desired, then the best approach would be to test various
filters to find the sweet spot for that dataset.

22
6 Appendix

(a) 3st 2x2 (b) 4st 2x2 (c) 2st 3x3 (d) 3st 3x3

(e) 3st 2x2 (f) 4st 2x2 (g) 2st 3x3 (h) 3st 3x3

Figure 15: Testing number of filters, with filtered inputs and saliency maps.

(a) 3st 2x2 (b) 4st 2x2 (c) 2st 3x3 (d) 3st 3x3

(e) 3st 2x2 (f) 4st 2x2 (g) 2st 3x3 (h) 3st 3x3

Figure 16: Testing number of filters, with filtered inputs and saliency maps.

23
References
[1] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning.
MIT Press, 2016. [Link]

[2] Leilani H. Gilpin, David Bau, Ben Z. Yuan, Ayesha Bajwa, Michael
Specter, and Lalana Kagal. Explaining explanations: An overview of
interpretability of machine learning. 2018 IEEE 5th International Con-
ference on Data Science and Advanced Analytics (DSAA), pages 80–89,
2018.

[3] Finale Doshi-Velez and Been Kim. Towards a rigorous science of inter-
pretable machine learning. arXiv, 2017.

[4] European Commisson. White paper on artificial intelligence - a european

approach to excellence and trust. Office for Official Publications of the
European Communities, 2020.

[5] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. “why should
i trust you?”: Explaining the predictions of any classifier. In Proceed-
ings of the 22nd ACM SIGKDD International Conference on Knowledge
Discovery and Data Mining, KDD ’16, page 1135–1144, New York, NY,
USA, 2016. Association for Computing Machinery.

[6] Yann LeCun and Corinna Cortes. MNIST handwritten digit database.
2010.

[7] Marco Ancona, Enea Ceolini, A. Cengiz Öztireli, and Markus H. Gross.
A unified view of gradient-based attribution methods for deep neural
networks. CoRR, abs/1711.06104, 2017.

[8] Matthew D. Zeiler and Rob Fergus. Visualizing and understanding con-
volutional networks. In David Fleet, Tomas Pajdla, Bernt Schiele, and
Tinne Tuytelaars, editors, Computer Vision – ECCV 2014, pages 818–
833, Cham, 2014. Springer International Publishing.

[9] Ahmed Alqaraawi, Martin Schuessler, Philipp Weiß, Enrico Costanza,

and Nadia Berthouze. Evaluating saliency map explanations for convolu-
tional neural networks. Proceedings of the 25th International Conference
on Intelligent User Interfaces, Mar 2020.

[10] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Anchors:
High-precision model-agnostic explanations. In Thirty-Second AAAI
Conference on Artificial Intelligence, 2018.

24
[11] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning
important features through propagating activation differences. CoRR,
abs/1704.02685, 2017.

[12] Scott M Lundberg and Su-In Lee. A unified approach to interpreting

model predictions. In Advances in neural information processing sys-
tems, pages 4765–4774, 2017.

25