The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B3-2021

XXIV ISPRS Congress (2021 edition)

FUSING MULTI-MODAL DATA FOR SUPERVISED CHANGE DETECTION

Patrick Ebel1, , Sudipan Saha1 , Xiao Xiang Zhu1,2 ∗

1
Data Science in Earth Observation (SiPEO), Technical University of Munich (TUM), Munich, Germany - ([Link],
[Link])@[Link]
2
Remote Sensing Technology Institute (IMF), German Aerospace Center (DLR), Wessling, Germany - [Link]@[Link]

Commission III, WG 6

KEY WORDS: change detection, multi-modal, fusion, synthetic aperture radar (SAR), optical, deep learning.

ABSTRACT:

With the rapid development of remote sensing technology in the last decade, different modalities of remote sensing data recorded via
a variety of sensors are now easily accessible. Different sensors often provide complementary information and thus a more detailed
and accurate Earth observation is possible by integrating their joint information. While change detection methods have been tradi-
tionally proposed for homogeneous data, combining multi-sensor multi-temporal data with different characteristics and resolution
may provide a more robust interpretation of spatio-temporal evolution. However, integration of multi-temporal information from
disparate sensory sources is challenging. Moreover, research in this direction is often hindered by a lack of available multi-modal
data sets. To resolve these current shortcomings we curate a novel data set for multi-modal change detection. We further propose
a novel Siamese architecture for fusion of SAR and optical observations for multi-modal change detection, which underlines the
value of our newly gathered data. An experimental validation on the aforementioned data set demonstrates the potentials of the
proposed model, which outperforms common mono-modal methods compared against.

1. INTRODUCTION as in e.g. the presence of clouds. Exemplary full-scene obser-

vations for one ROI are portrayed in Fig. 1.
In a time of rapidly evolving urban landscapes and nature mod-
ified due to climate change our planet faces a rapid transform-
ation of its surface area. This transformation is being continu-
ously monitored by modern satellite systems like Copernicus’
Sentinel mission. The automated recognition of changes ob-
served by these repeated observations is the task of change de-
tection (CD). CD is a prominent and long-standing challenge
in remote sensing (Malila, 1980); on one hand because of the
Earth’s dynamic nature and the need to quantify change, on the
other due to the variety of land cover and the persistent chal-
lenge of the task. Recent progress in deep learning greatly be-
nefited previous application to change detection in satellite data
(Ball et al., 2017) (Zhu et al., 2017), which is the approach fol-
lowed as well in our work. However, most preceding publica-
tions do not consider the fusion of multiple sensors and thereby
misses on the opportunity to utilize the variety of Earth ob-
servation data available. Our work specifically addresses the
challenge of multi-modal bi-temporal change detection, where
(multi-spectral) optical as well as ground range detected syn-
thetic aperture radar (SAR) measurements are available at both
considered time points. Fusing these two modalities poses a
difficult problem, as both domains are very different from one
another: First, in terms of viewpoint geometry—while our op-
tical data is orthorectified, the SAR measurements are sideway-
looking. Second, multi-spectral optical data provides a view
on the surface characteristics of the target, whereas SAR obser-
vations provide information on its physical properties. Finally,
SAR data is challenging to work with and contains speckle ef-
Figure 1. Multi-modal observations and change map for
fects that a change detector must learn to interpret as noise and
exemplary ROI ’Paris’. Rows: Sentinel-2 data (RGB channels).
not raise any false alarms about. On the other hand, SAR as
Sentinel-1 data (VV-polarized). Change maps. Columns: Time
an active sensor does not suffer from drawbacks of optical im-
point 1. Time point 2. The images highlight the differences
agery, such as sensitivity to light conditions or bad weather due
between both modalities and the potential complementary
∗ Corresponding author information they may provide to benefit change detection.

This contribution has been peer-reviewed.

[Link] | © Author(s) 2021. CC BY 4.0 License. 243
 The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B3-2021
XXIV ISPRS Congress (2021 edition)

The central research question addressed in this work is whether very high resolution images of the same modality, and thus with
and to which extent information from multiple sources bene- relatively smaller domain differences.
fits the detection of changes in remote sensing data. For this
sake we build on preceding work in change detection (Daudt Furthermore, the related work of (Liu et al., 2016), (Zhang et
et al., 2018b)(Saha et al., 2019). We design a novel convolu- al., 2016), (Zhan et al., 2018) (Saha et al., 2019), (Ferraris
tional encoder-decoder architecture that fuses the multi-modal et al., 2020) and (Saha et al., 2021) consider the challenging
information and processes them in a supervised Siamese fash- case where the pre-change observation may be captured by a
ion. Furthermore, we collect a data set for multi-modal change sensor different from the one recording the post-change im-
detection and propose an experimental design to investigate the age. Whereas the first five consider an unsupervised training
research question. paradigm, (Saha et al., 2021) extends the prior work and pro-
poses a method for self-supervised change detection between
In sum, the contributions of our work are given by: First, we pairs of Sentinel-1 and Sentinel-2 imagery. Moreover, these
design a novel architecture that ingests both optical and SAR methods have in common that representations of change are
data and processes the multi-modal information through a Sia- learned in scenarios where no sufficient amount of labeled train-
mese network. Second, we collected SAR observations to com- ing data is available, with potential effects on the quality of the
plement an established data set of optical images, providing a learned features. While we also consider very heterogeneous
multi-modal change detection data set. Third, we experiment- pairings of SAR and optical data, our work differs in the sense
ally evaluate the proposed architecture, train it on the curated that data is curated for our study to allow for a supervised train-
data set and test it to highlight the benefits of multi-modal data ing procedure. In addition, we focus on the data fusion case
for change detection in remote sensing. where both modalities are available as pre- and post-change in-
puts to the model.
The remainder of the paper is organized as follows: The context
of related work is provided in section 1.1. The methodology These earlier contributions constitute fusion of multi-modal re-
and the novel network architecture are introduced in section 2. mote sensing data as a well-established research area in change
The experimental design and results are reported in section 3. detection and provides a vital starting point for our own con-
Finally, sections 4 and 5 close our work with a discussion and tributions. To sum up, our work extends on the existing re-
conclusion, respectively. search by acknowledging the existence of very heterogeneous
and more complex scenes, in which change may not be con-
1.1 Related work strained to an individual class of land cover or objects in par-
ticular. Specifically, we build on the efforts of (Daudt et al.,
Change detection is a longstanding challenge in remote sens- 2018b) and their hand-annotated data set of optical satellite ob-
ing and original methods date back accordingly into the past, servations to combine it with progress in data fusion for change
constituting a long and rich body of literature on the problem. detection. For this purpose, we curate co-registered and tem-
A classical approach, post-classification comparison, first se- porally aligned SAR observations for each of the bi-temporal
mantically segments pre- and post-change images individually change images, and demonstrate their purpose by introducing a
and then computes the change map as the difference of the novel Siamese network architecture for data fusion. The pres-
labels. In comparison, direct-multidate classification follows ence of a sufficiently large and hand-annotated data set allows
a single-step approach via decision tree change detection on for supervised training for bi-modal change detection—which
stacked features integrating across both spectral & temporal is in contrast to most of the preceding work that, due to lack
properties (Singh, 1989). A third classical technique is com- of training data, focuses on unsupervised methods. Taken to-
pound classification, which maximizes the posterior distribu- gether, these are the key characteristics that differentiate our
tion of change and non-change assignments (Bruzzone et al., contribution from prior work.
1999)(Bruzzone et al., 2004) in the predicted change map.

With the advent of deep learning and its adoption by the remote 2. METHOD
sensing community (Ball et al., 2017) (Zhu et al., 2017), change
detection in Earth observation took a paradigm shift from the We build on recent work in change detection for remote sensing
classical methods described above and the application of deep and propose a deep neural network that is capable of integrating
neural networks became the leading approach. The seminal data from multiple sources. Specifically, we consider a Siamese
work of (Daudt et al., 2018b) collects data of Sentinel-2 obser- network (Chicco, 2021) with a U-net architecture (Ronneber-
vations and considers well-established neural network architec- ger et al., 2015). In Section 2.1 we introduce the Siamese net-
tures for CD trained supervisedly on hand-annotated data. Our work. Section 2.2 briefly outlines the usage of Siamese network
work builds upon this study by extending the the collected data in homogeneous (single-sensor) change detection. Finally the
with novel SAR observations and advances the methodology network for multi-sensor change detection is detailed in Section
by proposing a novel data fusion neural network architecture 2.3. Triplet loss is also used (Dong and Shen, 2018).
for multi-modal change detection.
2.1 Siamese network
In terms of prior work on multi-modal change detection the fol-
lowing publications are of relevance. (Orsomando et al., 2007) Siamese networks were first proposed in context of image match-
detects change on a stack of SAR images and one multispectral ing (Bromley et al., 1993). A Siamese network consists of twin
observation. (Jiang et al., 2020) performs a transformation sep- networks (or parts thereof) that generally share weight yet ac-
arating semantics and style to project SAR and optical images cept different inputs of the same dimensionality. Weight sharing
into a shared feature space. (Zhang et al., 2018) uses a Siamese ensures that two similar inputs are mapped to alike represent-
architecture to detect building and tree changes between point ations in the feature space since they are processed through a
cloud data and aerial observations (Chen et al., 2019) proposes a shared set of non-linear functions. The outputs of the twin net-
recurrent Siamese network for detecting change on multi-sensor works are processed through an energy function that computes

This contribution has been peer-reviewed.

[Link] | © Author(s) 2021. CC BY 4.0 License. 244
 The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B3-2021
XXIV ISPRS Congress (2021 edition)

S1 S2
pre post pre post

2 → 16 → 16 2 → 16 → 16 13 → 16 → 16 13 → 16 → 16

16 → 32 → 32 16 → 32 → 32 16 → 32 → 32 16 → 32 → 32

32 → 64 → 64 → 64 32 → 64 → 64 → 64 32 → 64 → 64 → 64 32 → 64 → 64 → 64

64 → 128 → 128 → 128 64 → 128 → 128 → 128 64 → 128 → 128 → 128 64 → 128 → 128 → 128

128 → 128 → UP 2

512 → 128 → 128 → 64

64 → 64 → UP 2

256 → 64 → 64 → 32
32 → 32 → UP 2

128 → 32 → 16
16 → 16 → UP 2

64 → 16 → 2

Legend: out
CONV max pool S1 shared
transp. CONV concat S2 shared

Figure 2. Proposed multi-modal Siamese architecture for CD. The network consists of two encoder branches for each sensore and a
decoder part integrating the features from earlier layers. Each encoder branch processes its corresponding modality’s bi-temporal
samples, SAR and multispectral optical, in two passes. The extracted features get forwarded via skip connections in a U-Net like
fashion and then concatenated. Figure style adopted from (Daudt et al., 2018a).

a similarity metric between the highest level feature representa- as backbone architecture for Siamese CD. U-net is appealing
tions for each of the inputs propagated through the model. Con- for both tasks as the architecture is composed of processing in-
trastive loss is generally used owing to its ability to increase formation across two distinct pathways, one that preserve res-
the distance between dissimilar pairs and decrease the distance olution while being relatively shallow (processing information
between similar pairs (Koch et al., 2015). In sum, Siamese net- about the where of content) and the other being deep and wide
works are an appealing architecture for tasks that benefit from but with less spatial resolution (focusing on the what of con-
similarity of discrepancy-sensitive feature learning when com- tent). The Fully Convolutional Siamese - Concatenation (Sia-
paring two or more input types. mese (S2 only)) architecture of (Daudt et al., 2018b) uses U-net
as backbone with a 10 layer encoder to process the pre-change
2.2 Siamese network for homogeneous CD and post-change observations via two passes of one Siamese
branch. Skip connections in the network’s decoder compon-
The Siamese models used in the context of homogeneous CD
ent concatenate multi-scale information coming from the two
benefit of the principles described above. Two weight-sharing
encoding streams of the pre- and post change images. Our pro-
networks (or components of a single network) are used for high-
posed method builds on and extends the Siamese (S2 only) net-
level feature extraction from the pre-change and post-change
work to a multi-modal model, as detailed next.
images. After, the inferred high-level features are processed
through a decision network. The decision network segregates 2.3 Multi-sensor Siamese architecture
the changed pixels from the unchanged ones. Rather than just
processing the highest-level feature from the last weight-sharing The Siamese network architecture proposed in this work (abbre-
layer, multi-level features from multiple layers are often con- viated as: ours) consists of two encoder branches, one per con-
catenated to obtain a multi-scale representation of the change sidered sensor type, and a decoder part integrating the features
information (Rahman et al., 2018). Considering the related- from the preceding layers on a multi-scale basis. Each encoder
ness of the pixelwise change detection problem to semantic seg- branch processes its modality’s bi-temporal samples, SAR and
mentation, U-net (Ronneberger et al., 2015) is generally used multispectral optical, in two passes. The extracted features get

This contribution has been peer-reviewed.

[Link] | © Author(s) 2021. CC BY 4.0 License. 245
 The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B3-2021
XXIV ISPRS Congress (2021 edition)

forwarded via skip connections in a U-Net like fashion and then 3.2 Experiments & Results
concatenated at the individual levels of depth. Fig. 2 depicts the
proposed Siamese neural network architecture. To address the research question stated in section 1 we train
the neural network proposed in section 2 on the data set intro-
Similar to the Siamese (S2 only) baseline, the encoder part con- duced in section 3.1 and compare it against baseline models
sists of 10 convolutional layers (of 3×3px2 kernel size, a stride utilizing just a single sensor as well as networks utilizing both
of 1 px and a padding of 1 px), each followed by operators of sensors but exercising less guidance on the fusion process. The
batch normalisation, rectified linear units (ReLU) and dropout baselines compared against are given as follows:
(p = 0.2). In the encoder, every convolution blocks indicated
in Fig. 2 is followed by a layer of max pooling (of 2 × 2 px2 1. Siamese (S2 only) that corresponds to the FC-Siam-conc
kernel size and a stride of 2 px). interleaved max pooling oper- setup from (Daudt et al., 2018a) as detailed in Section 2.2
ations. The network’s decoder component consists of 14 trans- and relies on S2 inputs only.
posed convolution layers (of 3 × 3 px2 kernel size, a stride of
2 px, a padding of 1 px and an padding of 1 px), followed by 2. Siamese (S1+S2) that follows the Siamese (S2 only) archi-
operators of batch normalisation, ReLU and dropout (p = 0.2). tecture but with stacked S1 and S2 observations combined
At the end of each of each differentiable upsampling block fol- into a singly input tensor and processed jointly. That is,
lows a layer of replication padding as well as a concatenation other than our model, no explicitly separate processing of
layer stacking earlier decoder features in a U-Net manner. modalities is taking place.

The described encoder-decoder architecture follows a conven- 3. U-Net (S2 only) that stacks the channels from pre-change
tional hourglass style with the bottleneck being the widest part and post-change images into a single image and then pro-
and fewer kernels at the start as well as the end of the network. cesses it through a U-Net treating change detection as a
The last decoding layer reduces the features into a change map semantic segmentation task.
of only two bands representing changed and unchanged pixels
accordingly, with a log softmax nonlinearity appplied. The pre- 4. U-Net (S1+S2 only) that works similarly as U-Net (S2
dicted class, i.e. whether change or non-change, is then given only) but in addition to stacking pre- and post-change S2
by taking the maximum value across both bands in the output. images it also combines S1 and S2 cross-modality into a
singly input tensor and handles everything jointly without
imposing further constraints on the structure of informa-
3. EXPERIMENTS AND ANALYSIS tion processing.

3.1 Data All networks considered are trained in a supervised manner via
the ADAM optimizer (Kingma and Ba, 2014) with a weight
To conduct experiments we collect and process a multi-modal decay of 1e−4 and an exponential decay learning rate scheduler
data set and acquire SAR observations as follows: The geo- on a cross-entropy loss on the collected data set. The cross-
spatial locations of each ROI and the acquisition dates of their entropy cost function is weighted according to
original observations Sentinel-2 multi-spectral observations are
read from the meta information of the ONERA CD data set npositive (ntotal − npositive )
2 × λF P × ,2 ×
(Daudt et al., 2018b). The (ascending orbit) Sentinel-1 SAR ntotal ntotal
observations are downloaded via Google Earth Engine (Gorel-
ick et al., 2017) and coordinate-transformed via GDAL (Warm-
for the respective classes, where we set λF P = 10 as a para-
erdam, 2008) to match the coordinate system of the original
meter and ntotal and npositive denote the number of total and
optical data. Exemplary full-scene observations for one ROI
positively labeled pixels in the training split, respectively. The
are illustrated in Fig. 1. Finally, all full-scene images are sliced
networks are trained on batches of 32 samples. Random rota-
online into patches of sizes 96 pixels × 96 pixels with a stride
tions (in steps of 90 degrees) or mirroring (on the vertical axis)
of 1 pixel between spatially adjacent patches. The train and
are applied as data augmentation steps at equally distributed
test splits of the data set are as defined in (Daudt et al., 2018b).
chances to synthetically increase the training set size.
The SAR and multi-spectral optical patches are value-clipped
in the intervals [-25, 0] and [0, 1000], respectively. Finally,
The goodness of predictions are evaluated in terms of the well-
patches are normalized to the unit range for input to the net-
established metrics of precision, recall and F1 score, given by
works. For the baselines this is done via z-standardizing each
patch individually (such that the extreme values are guaranteed
to be taken per patch) as suggested in (Daudt et al., 2018a). TP
precision = ,
For our proposed Siamese fusion network we experimentally TP + FP
observed that standardizing in absolute terms outperforms z-
scoring, so patches are rescaled according to their modality’s
theoretically obtainable range rather than z-scoring, preserving TP
recall = ,
band-wise information in absolute terms across observations. TP + FN

To facilitate future research in remote sensing on bi-temporal

2 × precision × recall
change detection of multi-modal satellite observations we wish F 1 score = ,
to share our data with the scientific community. Our SAR ob- precision + recall
servations specifically collected and preprocessed for this study
can be found online on [Link] where T P , F P and F N denote true positives, false positives
multimodalCD_ISPRS21. and false negatives, respectively. The F1 score is the harmonic

This contribution has been peer-reviewed.

[Link] | © Author(s) 2021. CC BY 4.0 License. 246
 The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B3-2021
XXIV ISPRS Congress (2021 edition)

model precision recall F1 score

ours 0.602 0.561 0.581
Siamese (S2 only) 0.680 0.494 0.573
Siamese (S1+S2) 0.699 0.412 0.519
U-Net (S2 only) 0.762 0.394 0.519
U-Net (S1+S2) 0.562 0.255 0.351
Table 1. Performance of the evaluated change detection models
on the ONERA test split. The proposed multi-modal network
outperforms the considered baseline and is strongest in terms of
both recall and F1 score.

mean of both precision as well as recall and provides a summary

statistics of a considered method’s overall accuracy.

Results are reported in Table 1 and show that the proposed

model outperforms the considered baselines. It is evident from
the reported numbers that our fusion-based method outperforms
the other S2-only models, accomplishing a considerably im-
proved recall score and an overall increase in terms of F1 metric
as well. Remarkably, solely feeding S1 and S2 combined inputs
to the standard Siamese and U-Net architectures does not guar-
antee any increase in performance but may even be detrimental.
This may indicate that fusing diverse modalities such as S1 and
S2 together necessitates more guidance (as provided by our pro-
posed architecture) than merely stacking them together. The
predictions of the proposed model and the second best method,
the Siamese (S2 only) baseline, on data of three exemplary ROI
are displayed in Fig. 3. The results show that both models
share many of the correctly predicted changes, indicating that
these pixels may exhibit change that is clearer to detect than
more ambiguous change in other parts of the scenes. Interest-
ingly, our proposed model has a tendency to correctly detect
more change, but it may also be more prone to false alarms—a
circumstance that is discussed further in section 4.

4. DISCUSSION

Change detection in remote sensing poses a challenging task as

the typical scenes considered by practitioners are very complex.
The images utilized for training and testing in this study are
constituted by spatial arrangements of many objects which are
themselves often not constrained or clearly defined in terms of Figure 3. Exemplary change predictions. Rows: Proposed
their land cover or object class. While complementary views on method and optical-only Siamese baseline model. Column:
the scene (as given by multi-modal observations) can ease this Three different ROI. Chonqqing, Duba and Las Vegas. Labels:
uncertainty and be of benefit, bridging the difference between White (true positive), green (false positive), violet (false
two very heterogeneous domains and integrating sensor inform- negative). The results show that the proposed model is more
ation is all by itself a nontrivial task. One contribution of our sensitive to change, yet may also be more prone to false alarms,
work is to provide the scientific community with the needed as compared to the baseline.
data to collectively address this challenge, as well as propos-
ing a novel deep neural network architecture demonstrating the to the optical-only baseline) as exemplified in Fig. 3. While this
benefits of multi-modal change detection. point highlights the complementing nature of both modalities, it
may put SAR data at a disadvantage when evaluating on labels
Our work builds on the original data set of (Daudt et al., 2018a) driven by optical information, raising the more principal ques-
and its high-quality annotations hand-labeled by introspecting tion of what modification of pixel intensities in which modality
Sentinel-2 data. While the presence of labels allows for su- should actually count as a change in the ground truth.
pervised training and competitive performances, it may as well
pose a limitation to our study as the provided supervision may Furthermore, the results presented in section 3.2 demonstrated
not always capture change perfectly. An example is given in the benefits of multi-modal data in combination with our de-
Fig. 4, where the upper right image quartile displays clear signed architecture, but the relative improvement over the strong
change between the pre- and post-change Sentinel-1 observa- second best method is not very large. This may reflect the or-
tions but the subtle differences are barely visible in the RGB der of improvement provided by the proposed neural network.
plots of the Sentinel-2 data and consequently not annotated in Alternatively, a saturation effect of the F1 score just around
the labels. This may lead to predicted change where there is 0.6 may be natural on the considered data set, the given train-
none annotated, eventually raising the false positives (compared ing split size and the challenging test split scenes. Similarly

This contribution has been peer-reviewed.

[Link] | © Author(s) 2021. CC BY 4.0 License. 247
 The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B3-2021
XXIV ISPRS Congress (2021 edition)

but further opens up the research towards this direction by mak-

ing available a novel data set and encouraging further research
in the direction. In addition to improving the proposed archi-
tecture, our future work will focus on extending the proposed
method for integrating more than two modalities. We will also
extend the proposed method for more challenging problem of
multi-class or semantic change detection.

ACKNOWLEDGEMENTS

The work is supported by the German Federal Ministry of Edu-

cation and Research (BMBF) in the framework of the interna-
tional future AI lab “AI4EO – Artificial Intelligence for Earth
Observation: Reasoning, Uncertainties, Ethics and Beyond”,
Grant number: 01DD20001.

REFERENCES

Ball, J. E., Anderson, D. T., Chan Sr, C. S., 2017. Comprehens-

ive survey of deep learning in remote sensing: theories, tools,
and challenges for the community. Journal of Applied Remote
Sensing, 11(4), 042609.

Bromley, J., Guyon, I., LeCun, Y., Säckinger, E., Shah, R.,
1993. Signature verification using a” siamese” time delay
neural network. Advances in neural information processing sys-
tems, 6, 737–744.

Bruzzone, L., Cossu, R., Vernazza, G., 2004. Detection of land-

cover transitions by combining multidate classifiers. Pattern
Recognition Letters, 25(13), 1491–1500.

Bruzzone, L., Prieto, D. F., Serpico, S. B., 1999. A neural-

Figure 4. Multi-modal observations and change map for statistical approach to multitemporal and multisource remote-
exemplary ROI ’Abu Dhabi’. Rows: Sentinel-2 data (RGB sensing image classification. IEEE Transactions on Geoscience
channels). Sentinel-1 data (VV-polarized). Change maps. and remote Sensing, 37(3), 1350–1359.
Columns: Time point 1. Time point 2. The images indicate
discrepancies between SAR versus multispectral data and the Chen, H., Wu, C., Du, B., Zhang, L., Wang, L., 2019. Change
change labels, raising a question about the semantics of change. detection in multisource VHR images via deep Siamese convo-
lutional multiple-layers recurrent neural network. IEEE Trans-
(Daudt et al., 2018a) reported considerable gains in the range of actions on Geoscience and Remote Sensing, 58(4), 2848–2864.
poorer performances, but the degree of improvements as well
Chicco, D., 2021. Siamese neural networks: An overview. Arti-
decreased around the marks observed in our study. In sum,
ficial Neural Networks, 73–94.
we are positive that future research building on our data will
provide valuable insights with respect to this point and advance Daudt, R. C., Le Saux, B., Boulch, A., 2018a. Fully convolu-
the state of the art of multi-modal change detection. tional siamese networks for change detection. 2018 25th IEEE
International Conference on Image Processing (ICIP), IEEE,
5. CONCLUSION 4063–4067.

Daudt, R. C., Le Saux, B., Boulch, A., Gousseau, Y., 2018b.

This work addressed the challenge of multi-modal bi-temporal
Urban change detection for multispectral earth observation us-
change detection. We investigated the central research question
ing convolutional neural networks. IGARSS 2018-2018 IEEE
of whether multi-modal fusion approaches benefit bi-temporal
International Geoscience and Remote Sensing Symposium,
change detection. To evaluate this question on a substantially
IEEE, 2115–2118.
large amount of train and test images, we extended an existing
and well-established single-sensor S2 data set by complement- Dong, X., Shen, J., 2018. Triplet loss in siamese network for ob-
ing it with corresponding S1 images curated for this study. Fur- ject tracking. Proceedings of the European conference on com-
thermore, we proposed a novel architecture for bi-modal fusion puter vision (ECCV), 459–474.
based change detection that integrates information from both
SAR as well as optical sensors. The results show that bi-modal Ferraris, V., Dobigeon, N., Cavalcanti, Y., Oberlin, T., Chabert,
fusion improves result over single-sensor approach. Though the M., 2020. Unsupervised change detection for multimodal re-
improvement is not large, further improvement in multi-modal mote sensing images via coupled dictionary learning and sparse
fusion architecture can potentially improve the result. We em- coding. ICASSP 2020-2020 IEEE International Conference
phasize that the contribution of this work is not only limited on Acoustics, Speech and Signal Processing (ICASSP), IEEE,
to devising a novel approach for multi-sensor change detection, 4627–4631.

This contribution has been peer-reviewed.

[Link] | © Author(s) 2021. CC BY 4.0 License. 248
 The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B3-2021
XXIV ISPRS Congress (2021 edition)

Gorelick, N., Hancher, M., Dixon, M., Ilyushchenko, S., Thau, Zhang, Z., Vosselman, G., Gerke, M., Tuia, D., Yang, M. Y.,
D., Moore, R., 2017. Google Earth Engine: Planetary-scale 2018. Change detection between multimodal remote sensing
geospatial analysis for everyone. Remote sensing of Environ- data using siamese CNN. arXiv preprint arXiv:1807.09562.
ment, 202, 18–27.
Zhu, X. X., Tuia, D., Mou, L., Xia, G.-S., Zhang, L., Xu, F.,
Jiang, X., Li, G., Liu, Y., Zhang, X.-P., He, Y., 2020. Change Fraundorfer, F., 2017. Deep learning in remote sensing: A com-
detection in heterogeneous optical and SAR remote sensing im- prehensive review and list of resources. IEEE Geoscience and
ages via deep homogeneous feature fusion. IEEE Journal of Se- Remote Sensing Magazine, 5(4), 8–36.
lected Topics in Applied Earth Observations and Remote Sens-
ing, 13, 1551–1566.

Kingma, D. P., Ba, J., 2014. Adam: A method for stochastic

optimization. arXiv preprint arXiv:1412.6980.

Koch, G., Zemel, R., Salakhutdinov, R., 2015. Siamese neural

networks for one-shot image recognition. ICML deep learning
workshop, 2, Lille.

Liu, J., Gong, M., Qin, K., Zhang, P., 2016. A deep convolu-
tional coupling network for change detection based on hetero-
geneous optical and radar images. IEEE transactions on neural
networks and learning systems, 29(3), 545–559.

Malila, W. A., 1980. Change vector analysis: an approach for

detecting forest changes with landsat. LARS symposia, 385.

Orsomando, F., Lombardo, P., Zavagli, M., Costantini, M.,

2007. SAR and optical data fusion for change detection. 2007
Urban Remote Sensing Joint Event, IEEE, 1–9.

Rahman, F., Vasu, B., Van Cor, J., Kerekes, J., Savakis, A.,
2018. Siamese network with multi-level features for patch-
based change detection in satellite imagery. 2018 IEEE Global
Conference on Signal and Information Processing (GlobalSIP),
IEEE, 958–962.

Ronneberger, O., Fischer, P., Brox, T., 2015. U-net: Convo-

lutional networks for biomedical image segmentation. Interna-
tional Conference on Medical image computing and computer-
assisted intervention, Springer, 234–241.

Saha, S., Bovolo, F., Bruzzone, L., 2019. Unsupervised

multiple-change detection in vhr multisensor images via deep-
learning based adaptation. IGARSS 2019-2019 IEEE Inter-
national Geoscience and Remote Sensing Symposium, IEEE,
5033–5036.

Saha, S., Ebel, P., Zhu, X. X., 2021. Self-supervised Multis-

ensor Change Detection. arXiv preprint arXiv:2103.05102.

Singh, A., 1989. Digital change detection techniques using

remotely-sensed data. International Journal of Remote Sensing,
10(6), 989–1003.

Warmerdam, F., 2008. The geospatial data abstraction library.

Open source approaches in spatial data handling, Springer, 87–
104.

Zhan, T., Gong, M., Jiang, X., Li, S., 2018. Log-based trans-
formation feature learning for change detection in heterogen-
eous images. IEEE Geoscience and Remote Sensing Letters,
15(9), 1352–1356.

Zhang, P., Gong, M., Su, L., Liu, J., Li, Z., 2016. Change de-
tection based on deep feature representation and mapping trans-
formation for multi-spatial-resolution remote sensing images.
ISPRS Journal of Photogrammetry and Remote Sensing, 116,
24–41.

This contribution has been peer-reviewed.

[Link] | © Author(s) 2021. CC BY 4.0 License. 249