SINGLE IMAGE CLOUD DETECTION VIA MULTI-IMAGE FUSION

Scott Workman1 M. Usman Rafique2 Hunter Blanton2 Connor Greenwell2 Nathan Jacobs2
1 2
DZYNE Technologies University of Kentucky

ABSTRACT globe. Consider the scenario of transitioning to a new sen-

sor. Instead of collecting large amounts of new annotations, a
Artifacts in imagery captured by remote sensing, such as
arXiv:2007.15144v1 [[Link]] 29 Jul 2020

method is needed that can function with minimal supervision.

clouds, snow, and shadows, present challenges for various
In this work we explore how recent advances in multi-image
tasks, including semantic segmentation and object detection.
fusion can be extended to support cloud detection.
A primary challenge in developing algorithms for identifying
First, we design an architecture for weakly-supervised
such artifacts is the cost of collecting annotated training data.
multi-image fusion that learns to estimate image quality.
In this work, we explore how recent advances in multi-image
Then, we describe two approaches which take advantage of
fusion can be leveraged to bootstrap single image cloud de-
the resulting quality network to produce a cloud detector.
tection. We demonstrate that a network optimized to estimate
To support the training and evaluation of our methods, we
image quality also implicitly learns to detect clouds. To sup-
collect a large dataset of overhead images captured at varying
port the training and evaluation of our approach, we collect
timesteps and varying levels of cloud cover. Our contribu-
a large dataset of Sentinel-2 images along with a per-pixel
tions include: 1) an analysis of multi-image fusion on real
semantic labelling for land cover. Through various exper-
data, 2) two approaches for identifying clouds that require
iments, we demonstrate that our method reduces the need
limited supervision, and 3) an extensive evaluation, achieving
for annotated training data and improves cloud detection
state-of-the-art results on a benchmark dataset.
performance.
Index Terms— weakly-supervised learning, multi-image
2. APPROACH
fusion, segmentation, clouds
Our approach for identifying clouds uses multi-image fusion
1. INTRODUCTION as a form of bootstrapping, reducing the need for annotated
training data. We start by describing the architecture for
As overhead imagery captured via remote sensing becomes multi-image fusion and then describe how we extend this
more abundant, it is increasingly relied upon as an important architecture for detecting clouds.
source of information for understanding locations and how
they change over time. For example, methods have been pro-
2.1. Multi-Image Fusion
posed for extracting roads [1], detecting buildings [2], esti-
mating land cover [3], and interpreting the effects of natural We apply multi-image fusion to take a stack of images over
disasters [4]. Unfortunately, various artifacts contained in the the same region, I = {I1 , . . . , IK }, where Ij ∈ Rh×w×3 , and
captured imagery, such as clouds, snow, and shadows, nega- produce a fused image, F = φ(I), such that F is free of arti-
tively impact the performance of these methods. facts. Our approach is inspired by the recent work of Rafique
Clouds and their properties have long been researched due et al. [9]. There are two main steps: 1) estimating a per-pixel
to their impact on weather and climate processes [5]. In an quality mask for each image then using the qualities to com-
empirical study Wylie et al. [6] analyze cloud cover over a 22 pute a fused image and 2) passing the fused image through a
year period using atmospheric sounding, finding that approxi- segmentation network to produce a per-pixel semantic label-
mately 75 percent off all observations indicated clouds. Given ing. When trained end-to-end, this architecture learns to esti-
their high frequency, clouds present persistent challenges for mate per-pixel image qualities that can be used to produce a
interpreting overhead imagery and many methods have been fused image with reduced artifacts, without requiring explicit
proposed for identifying them [7, 8]. labels.
The primary challenge is that the appearance of clouds
can vary dramatically and collecting manually annotated data 2.1.1. Dataset
is time consuming and expensive. This issue is further com-
pounded by the various sensor types and resolutions of satel- To support the training of our methods, we collected Sentinel-
lite imagery, as well as differences in locations around the 2 imagery from the state of Delaware with varying levels of
cloud cover. Starting from a bounding box around the state,
we generated a set of non-overlapping tiles using the stan-
dard XYZ style spherical Mercator tile. For each tile, we col-
lected a semantic labeling from the Chesapeake Land Cover
dataset [3], removing tiles without valid labels. For each re-
maining tile, we randomly downloaded six Sentinel-2 images
(RGB bands) from the year 2019 that satisfied the constraint
of having between 10% and 50% cloud cover in the parent
Sentinel-2 image strip. This process resulted in 1033 unique
locations and 6198 images (of size 512×512). Figure 1 shows
some example images from our dataset.

2.1.2. Method
Each image Ij is first passed through a quality network which
outputs a per-pixel quality mask Qj ∈ Rh×w for each pixel
p, such that Qj (Ij (p)) ∈ [0, 1]. Given quality masks for each
image, a relative quality score at each pixel is computed by
applying a softmax across images:
eQj (p)
Q∗j (p) = PK . (1)
Qk (p)
k=1 e

The final fused image Fj is obtained by averaging the images

weighted by the relative quality score:
K
X
Fj (p) = Ij (p)Q∗j (p). (2)
j=1

The fused image Fj is passed through a segmentation network Fig. 1. Examples from our dataset for multi-image fusion.
to produce a per-pixel labeling. The entire architecture, both (top) Land cover labeling from the Chesapeake Land Cover
quality network and segmentation network, are optimized us- dataset [3]. (bottom) Images of the same location with vary-
ing a cross-entropy loss function. ing cloud cover.

2.1.3. Architecture Details

2.2.1. Quality Calibration
For the quality network, we use a slightly modified U-Net [10]
with the same number of layers but a quarter of the feature We apply Platt scaling (which we refer to as quality calibra-
maps compared to the original work. The final activation tion) to transform the outputs of the quality network into a
is a sigmoid. For the segmentation network, we build on distribution over classes (cloud/not cloud). In practice, this
LinkNet [11], a modern, lightweight segmentation architec- means we fit a logistic regression model:
ture that follows an encoder/decoder approach. Specifically,
1
we use LinkNet-34, which is LinkNet with a ResNet-34 [12] P (y = 1|Qj (p)) = , (3)
encoder. We initialize the encoder with weights from a net- 1+ eβ0 Qj (p)+β1
work pretrained on ImageNet.
where β0 and β1 are two learned parameters.

2.2. Detecting Clouds

2.2.2. Fine-Tuning the Quality Network
The quality network learns to identify artifacts in the train-
ing data that negatively impact the final segmentation, for ex- Alternatively, we employ transfer learning, freezing all lay-
ample clouds and regions of no data. We describe two ap- ers of the quality network except the final three convolutional
proaches which use the quality network, trained for multi- layers (the last upsampling block and the final 1 × 1 convolu-
image fusion, as a starting point for learning a cloud detector tion). Then, we fine-tune the network for cloud detection. We
(per-pixel binary classification). For these methods, we use optimize the network using the following loss function:
the dataset recently introduced by Liu et al. [13] with 100
training images and 20 testing images. L = Lbce + (1 − Ldice ) (4)
 Image (1 of 6) Quality Image (2 of 6) Quality Fused Target Prediction

Fig. 2. Qualitative examples of multi-image fusion. (left) Example images and estimated quality masks. (right) The fused
image produced using the relative quality scores, the target segmentation mask, and our prediction.

where Lbce is binary cross entropy, a standard loss function

Table 1. Quantitative evaluation for cloud detection.
used in binary classification tasks, and Ldice is the dice coef-
ficient, which measures spatial overlap. Method TPR TNR mIoU Accuracy
Liu et al. [13] 0.963 0.945 89.47% 95.87%
2.3. Implementation Details Ours (threshold) 0.982 0.878 81.78% 91.73%
Ours (calibrate) 0.933 0.967 88.50% 95.42%
Our methods are implemented using the PyTorch [14] frame- Ours (fine-tune) 0.962 0.967 91.24% 96.51%
work and optimized using RAdam [15] with Lookahead [16]
(k = 5, α = .5). The learning rate is λ = 10−4 (10−2 when
fine-tuning). We train all networks with a batch size of 10 against a baseline, Ours (threshold), that naı̈vely thresholds
for 100 epochs and train on random crops of size 416 × 416. the quality masks at .5 (treating anything below the threshold
For multi-image fusion, we randomly sample 4 images per as a cloud). The baseline, which requires no direct supervi-
location during training. sion, is able to correctly classify over 91% of pixels. Applying
quality calibration, Ours (calibrate), to the output of the qual-
3. EVALUATION ity network improves upon this result. Ultimately fine-tuning,
Ours (fine-tune), outperforms all baselines, achieving supe-
We evaluate our methods both qualitatively and quantitatively rior results than Liu et al. [13]. Next, we evaluate the ability
through a variety of experiments. of our approach to identify clouds with varying number of
training images (Figure 3). For this experiment, we trained
3.1. Visual Analysis of Multi-Image Fusion on Real Data each model on a randomly selected subset of the training data
and fine-tuning was limited to 30 epochs. As observed, our
Previous work on multi-image fusion used training data aug- proposed approaches require very few annotated images to
mented with synthetic clouds [9]. In our work, we train and produce reasonable cloud detection results. Finally, Figure 4
evaluate our approach using real images with varying levels shows some example predictions using our best method.
of cloud cover. Figure 2 shows example output from our net-
work (described in Section 2.1), including: example images
alongside the estimated quality mask, the fused image using 4. CONCLUSION
the relative quality scores, the target label from the Chesa-
peake Land Cover dataset [3], and our prediction. The esti- We presented methods for detecting clouds that require min-
mated quality masks clearly identify artifacts in the imagery, imal supervision. Our key insight was to take advantage of
such as clouds. multi-image fusion, which learns to capture the notion of im-
age quality, as a form of pretraining. To support our methods,
we introduced a large dataset of images with varying lev-
3.2. Quantitative Analysis of Cloud Detection
els of cloud cover and a corresponding per-pixel land cover
Using the dataset recently introduced by Liu et al. [13], we labelling. Using this dataset, we showed results for multi-
quantitatively evaluate our methods ability to detect clouds. image fusion on real-world imagery. Finally, we presented a
Table 1 shows the results of this experiment. We compare quantitative evaluation of cloud detection, ultimately achiev-
 Image Target Prediction Error

Fig. 3. Evaluating the impact of the number of training im- Fig. 4. Example cloud detection results using Ours (fine-
ages on cloud detection accuracy. tune). The error image (right) shows false positives (nega-
tives) color coded as purple (yellow).
ing state-of-the-art results on an existing cloud detection
benchmark dataset. [9] Muhammad Usman Rafique, Hunter Blanton, and Nathan Ja-
cobs, “Weakly supervised fusion of multiple overhead im-
5. REFERENCES ages,” in IEEE/ISPRS Workshop: Large Scale Computer Vi-
sion for Remote Sensing Imagery (EARTHVISION), 2019. 1,
[1] Anil Batra, Suriya Singh, Guan Pang, Saikat Basu, CV Jawa- 3
har, and Manohar Paluri, “Improved road connectivity by joint [10] Olaf Ronneberger, Philipp Fischer, and Thomas Brox, “U-Net:
learning of orientation and segmentation,” in CVPR, 2019. 1 Convolutional networks for biomedical image segmentation,”
[2] Benjamin Bischke, Patrick Helber, Joachim Folz, Damian in International Conference on Medical Image Computing and
Borth, and Andreas Dengel, “Multi-task learning for segmen- Computer-Assisted Intervention, 2015. 2
tation of building footprints with deep neural networks,” in [11] Abhishek Chaurasia and Eugenio Culurciello, “LinkNet: Ex-
ICIP, 2019. 1 ploiting encoder representations for efficient semantic segmen-
[3] Caleb Robinson, Le Hou, Kolya Malkin, Rachel Soobitsky, tation,” in IEEE Visual Communications and Image Process-
Jacob Czawlytko, Bistra Dilkina, and Nebojsa Jojic, “Large ing, 2017. 2
scale high-resolution land cover mapping with multi-resolution [12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,
data,” in CVPR, 2019. 1, 2, 3 “Deep residual learning for image recognition,” in CVPR,
[4] Jigar Doshi, Saikat Basu, and Guan Pang, “From satellite im- 2016. 2
agery to disaster insights,” in NeurIPS Workshop on AI for [13] Cheng-Chien Liu, Yu-Cheng Zhang, Pei-Yin Chen, Chien-
Social Good, 2018. 1 Chih Lai, Yi-Hsin Chen, Ji-Hong Cheng, and Ming-Hsun Ko,
[5] Kuo-Nan Liou, “Influence of cirrus clouds on weather and “Clouds classification from sentinel-2 imagery with deep resid-
climate processes: A global perspective,” Monthly Weather ual learning and semantic image segmentation,” Remote Sens-
Review, vol. 114, no. 6, pp. 1167–1199, 1986. 1 ing, vol. 11, no. 2, pp. 119, 2019. 2, 3
[6] Donald Wylie, Darren L Jackson, W Paul Menzel, and John J [14] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan,
Bates, “Trends in global cloud cover in two decades of hirs Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmai-
observations,” Journal of climate, vol. 18, no. 15, pp. 3021– son, Luca Antiga, and Adam Lerer, “Automatic differentiation
3031, 2005. 1 in PyTorch,” in NIPS Autodiff Workshop, 2017. 3
[7] Pengfei Li, Limin Dong, Huachao Xiao, and Mingliang Xu, “A [15] Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xi-
cloud image detection method based on SVM vector machine,” aodong Liu, Jianfeng Gao, and Jiawei Han, “On the variance
Neurocomputing, vol. 169, pp. 34–42, 2015. 1 of the adaptive learning rate and beyond,” in ICLR, 2020. 3
[8] Fengying Xie, Mengyun Shi, Zhenwei Shi, Jihao Yin, and [16] Michael Zhang, James Lucas, Jimmy Ba, and Geoffrey E Hin-
Danpei Zhao, “Multilevel cloud detection in remote sensing ton, “Lookahead optimizer: k steps forward, 1 step back,” in
images based on deep learning,” IEEE Journal of Selected Top- NIPS, 2019. 3
ics in Applied Earth Observations and Remote Sensing, vol. 10,
no. 8, pp. 3631–3640, 2017. 1