SEN12MS – A CURATED DATASET OF GEOREFERENCED MULTI-SPECTRAL

SENTINEL-1/2 IMAGERY FOR DEEP LEARNING AND DATA FUSION

M. Schmitt1 , L. H. Hughes1 , C. Qiu1 , X. X. Zhu1,2

1
Signal Processing in Earth Observation, Technical University of Munich, Munich, Germany
2
Remote Sensing Technology Institute, German Aerospace Center (DLR), Oberpfaffenhofen, Wessling

KEY WORDS: Data Fusion, Dataset, Machine Learning, Remote Sensing, Multi-Spectral Imagery, Synthetic Aperture Radar (SAR),
Optical Remote Sensing, Sentinel-1, Sentinel-2, Deep Learning

ABSTRACT:

This is a pre-print of a paper accepted for publication in the ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Infor-
mation Sciences. Please refer to the original (open access) publication from September 2019.
arXiv:1906.07789v1 [[Link]] 18 Jun 2019

The availability of curated large-scale training data is a crucial factor for the development of well-generalizing deep learning methods
for the extraction of geoinformation from multi-sensor remote sensing imagery. While quite some datasets have already been published
by the community, most of them suffer from rather strong limitations, e.g. regarding spatial coverage, diversity or simply number
of available samples. Exploiting the freely available data acquired by the Sentinel satellites of the Copernicus program implemented
by the European Space Agency, as well as the cloud computing facilities of Google Earth Engine, we provide a dataset consisting of
180,662 triplets of dual-pol synthetic aperture radar (SAR) image patches, multi-spectral Sentinel-2 image patches, and MODIS land
cover maps. With all patches being fully georeferenced at a 10 m ground sampling distance and covering all inhabited continents during
all meteorological seasons, we expect the dataset to support the community in developing sophisticated deep learning-based approaches
for common tasks such as scene classification or semantic segmentation for land cover mapping.

1. INTRODUCTION of the dataset, which is designed to suit the needs of the remote
sensing community. SEN12MS contains full multi-spectral infor-
The availability of curated annotated datasets is of crucial impor- mation in geocoded imagery and is described in details through-
tance for the development of machine learning models for infor- out the remainder of this paper.
mation retrieval from remote sensing data. While classic shallow
learning approaches could easily be trained on comparably small 2. THE DATA BASIS
datasets such as, e.g., the famous Indian Pines scene (Baum-
gardner et al., 2015), modern deep learning requires large-scale We exploit freely available satellite(-derived) data to form the ba-
data to reach the desired generalization performance (Zhu et al., sis of the dataset. On the one hand, we make use of SAR and
2017). However, computer vision usually deals with conventional multi-spectral imagery provided by Sentinel-1 and Sentinel-2, re-
photographs of everyday objects, whereas remote sensing data is spectively. On the other hand, we add land cover information de-
more versatile and much more difficult to interpret. Therefore, rived from observations acquired by the MODIS system. Details
massive databases of labeled imagery such as ImageNet (Deng of the three basic data sources are provided in the following.
et al., 2009) do not yet exist in the remote sensing domain, al-
though there have been first steps into that direction; a certainly 2.1 Sentinel-1
non-exhaustive overview of existing scientific datasets of anno-
tated remote sensing imagery can be found in Tab. 1. Additional The Sentinel-1 mission (Torres et al., 2012) consists of currently
datasets, which were mostly provided in the frame of machine two polar-orbiting satellites, equipped with C-band SAR sensors,
learning competitions and are not described and discussed in sci- which enables them to acquire imagery regardless of the weather.
entific papers, can be found in a the private link list of (Rieke, Sentinel-1 works in a pre-programmed operation mode to avoid
2019). conflicts and to produce a consistent long-term data archive built
for applications based on long time series. Depending on which
In order to support deep learning-related research in the field of SAR imaging mode is used, resolutions down to 5 m with a
remote sensing, we have published the SEN1-2 dataset in 2018, wide coverage of up to 400 km can be achieved. Furthermore,
which is comprised of about 280,000 pairs of corresponding Sentinel-1 provides dual polarization capabilities and very short
Sentinel-1 SAR and Sentinel-2 optical images (Schmitt et al., revisit times of about 1 week at the equator. Since highly precise
2018). Since SEN1-2 was mainly intended for bridging the gap spacecraft positions and attitudes are combined with the high ac-
between classical computer vision problems and remote sensing, curacy of the range-based SAR imaging principle, Sentinel-1 im-
e.g. image-to-image translation tasks, the provided data were ages come with high out-of-the-box geolocation accuracy (Schu-
strongly simplified: For Sentinel-1, we just provided vertically bert et al., 2015).
polarized (VV) imagery in dB scale, and for Sentinel-2 we re-
duced the original multi-spectral data tensors to RGB images For the Sentinel-1 images in the SEN12MS dataset, again ground-
with adjusted histograms, whereas none of the images came with range-detected (GRD) products acquired in the most frequently
any form of geolocation information. Based on feedback from the available interferometric wide swath (IW) mode were used.
community, with this paper, we now publish a follow-on version These images contain the σ 0 backscatter coefficient in dB scale
 Dataset Nr. of Image Size Data Source Description Reference
Images
UC Merced Land Use 2,100 256 × 256 Color aerial 21 balanced land use classes (Yang and Newsam, 2010)
Dataset images
SAT-4 500,000 28 × 28 Color aerial 4 agricultural classes over (Basu et al., 2015)
imagery (R, G, continental USA
B, NIR)
SAT-6 405,000 28 × 28 Color aerial 6 land cover classes over (Basu et al., 2015)
imagery (R, G, continental USA
B, NIR)
Brazilian Coffee 51,004 64 × 64 SPOT 2 imbalanced classes (Penatti et al., 2015)
Scenes Dataset describing non-coffee and
coffee
USGS SIRI-WHU 1 10,000 × 9,000 Color aerial 4 classes over Montgomery (Zhong et al., 2015)
image County, Ohio, USA
Brazilian 1,311 64 × 64 RapidEye (G, R, 4 imbalanced classes (Nogueira et al., 2016)
Cerrado-Savanna NIR) describing Cerrado-Savanna
Dataset vegetation
SIRI-WHU 200 200 × 200 Google Earth 12 classes describing urban (Zhao et al., 2016)
areas in China
Inria Aerial Image 360 1,500 × 1,500 Color aerial 2 classes (building / not (Maggiori et al., 2017)
Labeling Dataset imagery building) for 10 cities in
Austria and USA
2017 IEEE GRSS 57 from Sentinel-2, 17 local climate zone classes (Yokoya et al., 2018)
Data Fusion Contest 447 × 377 to Landsat,
1,461 × 1,222 OpenStreetMap
DeepGlobe – Road 8,570 1,024 × 1,024 Worldview-2/-3, 1 target class: roads over India (Demir et al., 2018)
Extraction GeoEye-1 (R, G, and Thailand
B)
DeepGlobe – 24,586 650 × 650 Worldview-3 1 target class: buildings in Las (Demir et al., 2018)
Building Detection Vegas, Shanghai, Paris,
Karthoum
DeepGlobe 1,146 2,448 × 2,448 Worldview-2/-3, 7 land cover classes (Demir et al., 2018)
– Land Cover GeoEye-1 (R, G,
Classification B)
DOTA 2,806 800 × 800 Color aerial 188,282 instances of 15 object (Xia et al., 2018)
to images classes, each labeled by a
4,000 × 4,000 quadriliteral
EuroSAT 27,000 64 × 64 Sentinel-2 10 classes (Helber et al., 2018)
SEN1-2 564,768 256 × 256 Sentinel-1 and corresponding pairs of SAR (Schmitt et al., 2018)
Sentinel-2 (single-pol intensity) and
optical (RGB) image pairs
without annotations
38-Cloud 17,601 384 × 384 Landsat 8 (R, G, single target class: clouds (Mohajerani and Saeedi, 2019)
B, NIR)
BigEarthNet 590,326 up to Sentinel-2 43 Corine Land Cover classes (Sumbul et al., 2019)
120 × 120 over Europe
SEN12MS 541,986 256 × 256 Sentinel-1, globally distributed; MODIS this paper
Sentinel-2, Land Cover maps can either be
MODIS Land used as labels or auxiliary data
Cover

Table 1. Non-exhaustive list of curated datasets for deep learning in remote sensing.

for every pixel at a pixel spacing of 5 m in azimuth and 20 m in data any further.
range. In order to exploit the full potential of Sentinel-1 data,
SEN12MS contains both VV and VH polarized images. 2.2 Sentinel-2

For precise ortho-rectification, restituted orbit information was The Sentinel-2 mission (Drusch et al., 2012) currently comprises
combined with the 30 m-SRTM-DEM or the ASTER DEM for two identical polar-orbiting satellites in the same orbit, phased at
high latitude regions where SRTM is not available. As for the 180◦ to each other. The mission is meant to provide continuity for
SEN1-2 dataset, we intend to leave any further pre-processing, multi-spectral imagery of the SPOT and LANDSAT kind, which
e.g. speckle filtering, to the end user and do not manipulate the have provided information about the land surfaces of our Earth
for many decades. With its wide swath width of up to 290 km
and its high revisit time of 5 days at the equator (based on two ROI Season
satellites) under cloud-free conditions, the Sentinel-2 mission is
specifically well-suited to vegetation monitoring within the grow-
ing season. Query Module

For the SEN12MS dataset, we provide the full multi-spectral im-

Fetch Images Clip Images to ROI
age cubes as extracted from the original, precisely georeferenced [Link]() [Link]()
Sentinel-2 granules. The only manipulation we carried out was
to implement a sophisticated mosaicking workflow to avoid the
download of cloud-affected images (cf. Section 3.1). Quality Score Module

2.3 MODIS Land Cover

Cloud Score Submodule Shadow Score Submodule
MODIS (the Moderate Resolution Imaging Spectroradiometer)
is the main instrument on board of the Terra and Aqua satellites.
Terra’s orbit around the Earth is timed so that it passes from north
Image Merging Module
to south across the equator in the morning, while Aqua passes
south to north over the equator in the afternoon. Terra MODIS
Threshold Quality Score to Sort Images by Share of
and Aqua MODIS acquisitions cover the whole Earth with an ap- get Cloud Mask Poor Pixels
proximately daily revisit frequency – at a band-dependent resolu-
tion of 250 m to 1000 m. Based on calibrated MODIS reflectance
data, hierarchical classification following the land cover classifi-
cation system (LCCS) scheme, and sophisticated post-processing Cloud-free
Image
for class-specific refinement incorporating prior knowledge, aux-
iliary information and temporal regularization based on a Markov
random field, annually updated global land cover maps for the Figure 1. Workflow of the GEE-based procedure for cloud-free
years 2001–2016 are provided as MCD12Q1 V6 dataset at a Sentinel-2 image generation presented in (Schmitt et al., 2019).
ground sampling distance of 500 m (Sulla-Menashe et al., 2019).
To add land cover information to the Sentinel-1/Sentinel-2 patch- three main modules, which are carried out for every ROI. These
pairs constituting the core of the SEN12MS dataset, we add four- ROIs result from two uniform random samplings over the land-
band MODIS land cover patches created from 2016 data at an masses of the Earth and the urban areas across the globe, respec-
upsampled pixel spacing of 10 m. The first of the provided tively.
bands contains land cover following the International Geosphere-
Biosphere Programme (IGBP) classification scheme (Loveland While the procedure is described in detail in (Schmitt et al.,
and Belward, 1997), while the remaining bands contain the LCCS 2019), a short summary of the three modules is as follows:
land cover layer, the LCCS land use layer, and the LCCS sur-
face hydrology layer (Di Gregorio, 2005). The schemes’ classes (1) The Query Module for loading images from the catalogue.
are listed in Tab. 2. According to (Sulla-Menashe et al., 2019), In this module, for the specified ROI all Sentinel-2 images
the overall accuracies of the layers are about 67% (IGBP), 74% available for a specified time period are selected.
(LCCS land cover), 81% (LCCS land use), and 87% (LCCS sur-
face hydrology), respectively. This should be kept in mind when (2) The Quality Score Module for the calculation of a quality
using the land cover data as labels for training scene classification score for each image. In this module, every pixel of each
or semantic segmentation models, as these accuracies will con- Sentinel-2 image is assigned a score that considers the like-
stitute the upper bound of actually achievable predictive power lihood it is affected either by clouds or by shadow.
– even if validation accuracies of 100% are reached. If the land
cover information is not utilized as annotation, but as auxiliary (3) The Image Merging Module for mosaicking of the selected
data source, similar caution should be had. images based on the meta-information generated in the pre-
ceding modules. First, the quality scores are thresholded to
determine cloud and shadow masks for each image. After-
3. GOOGLE EARTH ENGINE FOR DATA
wards, the images are sorted by their amount of poor pixels.
PREPARATION
The best images are finally merged into a cloud-free mosaic.
As for the SEN1-2 dataset, we have again utilized Google Earth
Engine (Gorelick et al., 2017) to generate a large-scale dataset of Since the Sentinel-1 images and the MODIS land cover data are
corresponding multi-sensor remote sensing image patches. While not affected by clouds, in these cases no complicated mosaick-
we basically used the same pipeline as described in (Schmitt et ing processes are required and the data are simply exported in a
al., 2018), including the random sampling of ROIs for the mete- straight-forward manner later on.
orological seasons of the northern hemisphere, we added a more
sophisticated mosaicking workflow for the generation of cloud- 3.2 Data Export
free short-term Sentinel-2 mosaics.
3.1 Mosaicking of Cloud-Free Sentinel-2 Images For SEN12MS we have utilized the same random ROIs as for
SEN1-2. The same holds for the meteorological seasons as
The general mosaicking workflow to produce cloud-free defined for the northern hemisphere. After preparation of the
Sentinel-2 images for a given region of interest (ROI) and a spec- Sentinel-1 images and the cloud-free Sentinel-2 mosaics for ev-
ified time period is depicted in Fig. 1. In essence, it consists of ery ROI and season, we export them together with the land cover
 Class IGBP value LCCS LC value LCCS LU value LCCS SH value
Evergreen Needleleaf Forests 1 11 - -
Evergreen Broadleaf Forests 2 12 - -
Deciduous Needleleaf Forests 3 13 - -
Deciduous Broadleaf Forests 4 14 - -
Mixed Broadleaf/Needleleaf Forests - 15 - -
Mixed Broadleaf Evergreen/Deciduous Forests - 16 - -
Mixed Forests 5 - - -
Dense Forests - - 10 10
Open Forests - 21 20 20
Sparse Forests - 22 - -
Natural Herbaceous - - 30 -
Dense Herbaceous - 31 - -
Sparse Herbaceous - 32 - -
Shrublands - - 40 40
Closed (Dense) Shrublands 6 41 - -
Open (Sparse) Shrublands 7 43 - -
Shrubland/Grassland Mosaics - 42 - -
Woody Savannas 8 - - -
Savannas 9 - - -
Grasslands 10 - - 30
Permanent Wetlands 11 - - -
Woody Wetlands - - - 27
Herbaceous Wetlands - - - 50
Herbaceous Croplands - - 36 -
Croplands 12 - - -
Urban and Built-Up Lands 13 - 9 -
Cropland/Natural Vegetation Mosaics 14 - - -
Forest/Cropland Mosaics - - 25 -
Natural Herbaceous/Croplands Mosaics - - 35 -
Tundra - - - 51
Permanent Snow and Ice 15 2 2 2
Barren 16 1 1 1
Water Bodies 17 3 3 3

Table 2. MODIS land cover classes as represented by four different schemes: IGBP, LCCS land cover, LCCS land use, LCCS surface hydrology. The
NoData value is set to be 255 (Sulla-Menashe and Friedl, 2018).

data at a scale of 10 m in the form of GeoTiffs. In this context, e.g. no data areas, clouds, or jet streams. Some examples for
it has to be noted that the 10 m scale is defined at the equator patch types that were discarded in this step are displayed in Fig. 2.
by Google Earth Engine, which corresponds to an angular reso- After this final inspection step, a total of 180,662 patch triplets
lution of 0.0001◦ . This angular resolution leads to significantly remained, which comprise the final SEN12MS dataset. The lo-
smaller pixel widths for regions with latitudes deviating from 0◦ . cations of the final ROI scene locations are displayed in Fig. 3.
We therefore used GDAL (Warmerdam, 2008) to transform all
exported data from the WGS84 lat/lon georeference to their lo-
cal UTM coordinate representation, while resampling to actually
square pixels of 10 m×10 m.

3.3 Data Curation

After the export of the data from the GEE servers to local storage,
we followed an inspection protocol similar to the one proposed
in our previous work (Schmitt et al., 2018): First, each triplet of
full scene images was converted to a visually perceivable format
(gray-scale images for Sentinel-1 and MODIS Land Cover, RGB
images for Sentinel-2) and displayed to a remote sensing expert.
If either of the three images contained very large no-data areas,
large non-detected clouds, or strong artifacts resulting from the
cloud-adaptive mosaicking, the triplet was discarded. After this
first inspection, only 252 out of the originally downloaded 600 Figure 2. Examples for manually removed SAR (top row) and optical
scenes were kept in the dataset. These remaining scenes were patches (bottom) row).
then tiled into patches of 256 × 256 pixels in size. Again, we
have implemented a stride of 128 pixels, resulting in an overlap
between adjacent patches of 50%. We think, a 50% overlap is the 4. THE SEN12MS DATASET
ideal trade-off between patch independence and maximization of
the number of samples. After the tiling, 216,596 patch triplets The final SEN12MS dataset contains 180,662 patch triplets
were available for a second inspection. In this second inspection, (Sentinel-1 dual-pol SAR, Sentinel-2 multi-spectral, MODIS
all patches were again visually inspected by remote sensing ex- land cover) in the form of multi-channel GeoTiff images and re-
perts in order to avoid patches containing artefacts or distortions, quires 421.3 GiB of storage. Some patch triplet examples are
 Figure 3. Final distribution of the ROIs over all inhabited land masses of the Earth.

shown in Fig. 4 to give an impression of the rich and versatile

information contained in the dataset.

4.1 Structure of the Final Dataset

As mentioned in Section 3.1, the dataset is based on randomly

sampled regions of interest, resulting from four different seed
values: 1158, 1868, 1970, and 2017. These four different
seed values are related to the four meteorological seasons de-
fined for the northern hemisphere: winter (1 December 2016
to 28 February 2017), spring (1 March 2017 to 30 May 2017),
summer (1 June 2017 to 31 August 2017), and fall (1 Septem-
ber 2017 to 30 November 2017). This leads to a tree-like
structure of the dataset into four branches: ROIs1158 spring,
ROIs1868 summer, ROIs1970 fall, and ROIs2017 winter.
Each of those branches again is divided into several sub-branches
corresponding to the individual ROIs (or scenes, respectively)
derived from the corresponding random number seed. The full
dataset structure is depicted in Fig. 5.

The Sentinel-1 data can be recognized by the abbreviation s1,

the Sentinel-2 data by s2, and the MODIS land cover data
by lc; the individual patches can be identified by the token
pXXX where XXX denotes a unique identifier number per patch.
Thus, the file naming convention follows the following scheme:
ROIsSSSS SEASON DD [Link], where SSSS denotes the seed
value, SEASON denotes the meteorological season as defined for
the northern hemisphere, DD denotes the data identifier, and XXX
denotes the patch identifier.

Of course, we are aware that the seasonal structuring of the

dataset is only of little semantic worth since we have taken the
seasons of the northern hemisphere as a reference. To allow end-
users a sub-structuring of the dataset taking semantically mean-
ingful seasons into account, we provide the file [Link]
Figure 4. Images extracted from 3 example patch triplets. Each column
with the metadata of the dataset. It declares, which scenes ac-
shows (from top to bottom): False color Sentinel-1 SAR (R: VV, G: VH,
tually were acquired in spring, summer, winter, and fall from a B: VV/VH), Sentinel-2 RGB, Sentinel-2 SWIR, IGBP Land cover,
climatic point of view. LCCS Land cover. Note that while the GSD of all patches is upsampled
to 10 m, the actual resolution varies from 10 m (Sentinel-2 RGB) to
While we forgo to define a fixed train/test split, we think this 500 m (land cover).
can easily be achieved by end-users considering their individual
needs: Deterministic splits into disjunct training and test sets can
 Figure 5. Tree structure of the final dataset.

be achieved via the meteorological seasons, or via the individual a stride of 10 (leading to an output pixel spacing of 100 m). The
ROIs. overall accuracy (OA), average accuracy (AA) and Kappa coef-
ficient calculated based on a grid of manually annotated control
4.2 Dataset Availability points can be seen in Tab. 4. It has to be noted that for sake of this
evaluation, we combined the classes Open Forests (LCCS 20) and
The SEN12MS dataset is shared under the open access license Forest/Cropland Mosaic (LCCS 25) to a joint Open Forests class,
CC-BY and available for download at a persistent link provided and Natural Herbaceous (LCCS 30) with Herbacoeus Croplands
by the library of the Technical University of Munich (TUM): (LCCS 36) and Natural Herbaceous/Croplands Mosaic (LCCS
[Link] This paper must be 35) to a simple Herbaceous class, as it is very difficult for hu-
cited when the dataset is used for research purposes. man annotators to distinguish those classes by their only subtle
differences.
5. APPLICATION TO LAND COVER MAPPING The second network we used was the fully convolutional
DenseNet for semantic segmentation (Jégou et al., 2017), aim-
In order to provide an example for the usefulness of the dataset ing at assigning a class label to every pixel of the input im-
with regard to the development of land cover classification age. Here, we used full-sized Sentinel-2 10-band patches (i.e.
solutions, we have trained two state-of-the-art deep convolu- 256 × 256 pixels) as input and processed the city of Rome for test
tional neural network architectures for predicting LCCS land use purposes. The result is also depicted in Fig. 6, and the accuracy
classes (cf. Tab. 3). The first network, ResNet-110 (He et al., metrics, calculated analogue to the Munich case, are described in
ResNet-110 DenseNet
Tab. 5.

Upsampling type n/a Deconvolution OA Kappa AA

Input size 64 × 64 × 10 256 × 256 × 10 MODIS 53.0% 0.38 37.5%
Batch size 16 4 Predicted 65.1% 0.51 43.6%
Loss Categorical cross-entropy
Optimizer Nesterov Adam
Initial LR 0.0005 0.0001 Table 4. Accuracy metrics of the LCCS maps for the Munich scene. The
LR schedule ReduceOnPlateau predicted result was achieved by patch classification based on a
ResNet-110 CNN. The evaluation is based on manually labeled
reference points.
Table 3. Baseline network training configurations.
OA Kappa AA
2016), was designed for image classification, i.e. for assigning a
MODIS 56.0% 0.37 33.7%
single class label to the input image. For the presented baseline
Predicted 63.3% 0.44 39.4%
experiment, we cut images of 64 × 64 pixels from the Sentinel-
2 samples from the summer subset, and used the following ten
Table 5. Accuracy metrics of the LCCS maps for the Rome scene. The
bands as channel information: B2 (Blue), B3 (Green), B4 (Red), predicted result was achieved by semantic segmentation based on a
B8 (Near-infrared), B5 (Red Edge 1), B6 (Red Edge 2), B7 (Red DenseNet CNN. The evaluation is based on manually labeled reference
Edge 3), B8a (Red Edge 4), B11 (Short-wavelength infrared 1), points.
and B12 (Short-wavelength infrared 2). We then used the major-
ity LCCS land use class from each of the 64×64 patches as scene By comparing the resulting maps to the original MODIS-derived
label for that patch. Due to the unconventional multi-channel LCCS land use map, as well as an image extracted from Google
configuration of the data, we trained the network from scratch Earth, it can be seen that in both cases the resolution of the
rather than relying on any pre-trained weights. In order to test map was successfully enhanced so that more details can be re-
the predictive power of this network, we applied it to the area of trieved, while an overall agreement between the low-resolution
the city of Munich, with the test image being pre-processed with MODIS map and the corresponding predicted high-resolution re-
the same GEE-based procedure as described in Section 3.1. The sult is preserved. It has to be highlighed that both test areas are
resulting land use map is shown in Fig. 6 (top row), and was cre- not contained in the SEN12MS dataset, so that the results can
ated by sliding the ResNet-110 on an input Sentinel-2 image with serve as a first indicator of the strong generalization capability
 (a) (b) (c) (d)

(e) (f) (g) (h)

Barren (LCCS 1) Permanent Snow and Ice (LCCS 2) Water Bodies (LCCS 3) Urban and Built-Up Lands (LCCS 9)
Dense Forests (LCCS 10) Open Forests (LCCS 20) Forest/Cropland Mosaics (LCCS 25) Natural Herbaceous (LCCS 30)
Natural Herbaceous/Croplands Mosaics (LCCS 35) Herbaceous Croplands (LCCS 36) Shrublands (LCCS 40)

Figure 6. Examples from Munich (top) and Rome (bottom) for the predictive power of the dataset: (a) and (e) Optical images extracted from Google
Earth, (b) and (f) MODIS-derived LCCS land use map with 500m resolution, (c) predicted LCCS land use map with 100m resolution using
classification network, (g) predicted LCCS land use map with 10m resolution using semantic segmentation network, (d) and (h) zoom-in of the red
rectangles in the respective predicted LCCS land use maps. While the area in the red rectangle is completely defined as urban in the original
MODIS-derived product, more details are visible in the predicted results.

provided by the versatility of the dataset. This holds even more USGS SIRI-WHU), individual countries (e.g. SAT-4/5 or Deep-
so since only a small subset of the dataset (namely the patches Globe - Road Extraction), or a single continent (e.g. BigEarth-
of the summer season) has been used for training the networks Net).
used in the experiments. This impression is also confirmed by
the independently evaluated accuracy metrics. We expect that by
exploiting the whole dataset, i.e. both sensor modalities, all ROIs Wile this versatility is the major strength of SEN12MS, it also
and all seasons, powerful classifiers for large-scale mapping ap- poses the greatest challenge: In comparison to, e.g., ImageNet,
plications can be developed. Besides that, it has to be mentioned which also provides a lot of versatility and thus tries to model the
that training on low-resolution labels of course creates a form of whole world, the number of samples in SEN12MS is still fairly
label noise. Using specifically adapted strategies aiming at so- small. As can be seen from the example results provided in Sec-
called label super-resolution (Malkin et al., 2019) might be able tion 5, the dataset nevertheless seems to hold the potential to train
to provide even better results. powerful, well-generalizing models even under data-scarce train-
ing situations. Besides, we believe that further benefit will arise
from combining the existing datasets, e.g. BigEarthNet, with
6. DISCUSSION SEN12MS in the frame of transfer learning to enlarge the amount
of data a model can learn patterns from.
As can be seen from Tab. 1, SEN12MS is among the five largest
datasets when only the sheer number of image patches is consid-
ered. However, in the end, SEN12MS contains a lot more data
and is consequently much bigger than its competitors: First and 7. SUMMARY AND CONCLUSION
foremost, its patches are of size 256 × 256 pixels instead of just
28×28 (SAT-4/6) or 120×120 (BigEarthNet) pixels. Besides, the
spectral information content is also much higher, as SEN12MS With this paper, we publish the SEN12MS dataset, which con-
contains full multi-spectral Sentinel-2 imagery and, in addition, tains 180,662 triplets of Sentinel-1 dual-polarimetric SAR data,
dual-polarimetric Sentinel-1 SAR data, while most other datasets Sentinel-2 multi-spectral images, and MODIS-derived land cover
– besides EuroSAT, SEN1-2, and BigEarthNet – only contain maps. With its large patch size, its global scene distribution, and
color imagery with 3 or 4 bands. Last, but not least, it has to its wealth of versatile remote sensing information, it can be con-
be mentioned that SEN12MS is the most versatile dataset regard- sidered to be the largest remote sensing dataset available to date.
ing scene distribution, as it covers all regions of the Earth over We hope it will foster the development of well-generalizing ma-
all meteorological seasons, while most of the other datasets are chine learning models for a more sophisticated automatic analysis
restricted to fairly small areas (e.g. Brazilian Coffee Scenes or of Sentinel satellite data.
 ACKNOWLEDGEMENTS Penatti, O. A. B., Nogueira, K. and dos Santos, J. A., 2015. Do
deep features generalize from everyday objects to remote sensing
This work is jointly supported by the Helmholtz Association and aerial scenes domains? In: Proc. CVPR Workshops, pp. 44–
51.
under the framework of the Young Investigators Group SiPEO
(VH-NG-1018), the German Research Foundation (DFG, grant Rieke, C., 2019. Awesome satellite imagery
SCHM 3322/1-1), and the European Research Council (ERC) un- datasets. [Link]
awesome-satellite-imagery-datasets. [Online; ac-
der the European Unions Horizon 2020 research and innovation cessed 15-March-2019].
programme (grant agreement ERC-2016-StG-714087, Acronym:
So2Sat). Schmitt, M., Hughes, L. H. and Zhu, X. X., 2018. The SEN1-
2 dataset for deep learning in SAR-optical data fusion. In: IS-
PRS Ann. Photogramm. Remote Sens. Spatial Inf. Sci., Vol. IV-1,
REFERENCES pp. 141–146.

Basu, S., Ganguly, S., Mukhopadhyay, S., DiBiano, R., Karki, Schmitt, M., Hughes, L. H., Qiu, C. and Zhu, X. X., 2019. Aggre-
M. and Nemani, R., 2015. DeepSat: a learning framework for gating cloud-free Sentinel-2 images with Google Earth Engine.
satellite imagery. In: Proc. ACM SIGSPATIAL. Art. No. 37. In: ISPRS Annals of the Photogrammetry, Remote Sensing and
Spatial Information Sciences.
Baumgardner, M. F., Biehl, L. L. and Landgrebe,
D. A., 2015. 220 band AVIRIS hyperspectral im- Schubert, A., Small, D., Miranda, N., Geudtner, D. and Meier, E.,
age data set: June 12, 1992 Indian Pine Test Site 3. 2015. Sentinel-1A product geolocation accuracy: Commission-
[Link] ing phase results. Remote Sensing 7(7), pp. 9431–9449.

Demir, I., Koperski, K., Lindenbaum, D., Pang, G., Huang, J., Sulla-Menashe, D. and Friedl, M. A., 2018. User guide to Collec-
Basu, S., Hughes, F., Tuia, D. and Raska, R., 2018. DeepGlobe tion 6 MODIS land cover (MCD12Q1 and MCD12C1) product.
2018: A challenge to parse the earth through satellite images. In: USGS: Reston, VA, USA.
Proc. CVPR Workshops, pp. 172–181.
Sulla-Menashe, D., Gray, J. M., Abercrombie, S. P. and Friedl,
Deng, J., Dong, W., Socher, R., Li, L., Li, K. and Fei-Fei, L., M. A., 2019. Hierarchical mapping of annual global land cover
2009. ImageNet: A large-scale hierarchical image database. In: 2001 to present: The MODIS Collection 6 land cover product.
Proc. CVPR, pp. 248–255. Remote Sens. Environ. 222, pp. 183 – 194.

Di Gregorio, A., 2005. Land cover classification system: classi- Sumbul, G., Charfuelan, M., Demir, B. and Markl, V., 2019.
fication concepts and user manual: LCCS. Food & Agriculture BigEarthNet: A large-scale benchmark archive for remote sens-
Org. ing image understanding. arXiv:1902.06148.

Drusch, M., Del Bello, U., Carlier, S., Colin, O., Fernandez, V., Torres, R., Snoeij, P., Geudtner, D., Bibby, D., Davidson, M.,
Gascon, F., Hoersch, B., Isola, C., Laberinti, P., Martimort, P. Attema, E., Potin, P., Rommen, B., Floury, N., Brown, M. et al.,
et al., 2012. Sentinel-2: ESA’s optical high-resolution mission for 2012. GMES Sentinel-1 mission. Remote Sens. Environ. 120,
GMES operational services. Remote Sens. Environ. 120, pp. 25– pp. 9–24.
36.
Warmerdam, F., 2008. The Geospatial Data Abstraction Library.
Gorelick, N., Hancher, M., Dixon, M., Ilyushchenko, S., Thau, Springer, Berlin, pp. 87–104.
D. and Moore, R., 2017. Google earth engine: Planetary-scale
geospatial analysis for everyone. Remote Sens. Environ. 202, Xia, G., Bai, X., Ding, J., Zhu, Z., Belongie, S., Luo, J., Datcu,
pp. 18–27. M., Pelillo, M. and Zhang, L., 2018. Dota: A large-scale dataset
for object detection in aerial images. In: Proc. CVPR, pp. 3974–
He, K., Zhang, X., Ren, S. and Sun, J., 2016. Deep residual 3983.
learning for image recognition. In: Proc. CVPR, pp. 770–778.
Yang, Y. and Newsam, S., 2010. Bag-of-visual-words and spatial
Helber, P., Bischke, B., Dengel, A. and Borth, D., 2018. Intro- extensions for land-use classification. In: Proc. ACM SIGSPA-
ducing EuroSAT: A novel dataset and deep learning benchmark TIAL, pp. 270–279.
for land use and land cover classification. In: Proc. IGARSS,
pp. 204–207. Yokoya, N., Ghamisi, P., Xia, J., Sukhanov, S., Heremans, R.,
Tankoyeu, I., Bechtel, B., Saux, B. L., Moser, G. and Tuia, D.,
Jégou, S., Drozdzal, M., Vazquez, D., Romero, A. and Bengio, 2018. Open data for global multimodal land use classification:
Y., 2017. The one hundred layers tiramisu: Fully convolutional Outcome of the 2017 IEEE GRSS data fusion contest. IEEE J.
densenets for semantic segmentation. In: Proc. CVPR Work- Sel. Topics Appl. Earth Observ. 11(5), pp. 1363–1377.
shops, pp. 11–19.
Zhao, B., Zhong, Y., Xia, G. and Zhang, L., 2016. Dirichlet-
Loveland, T. and Belward, A., 1997. The international geosphere derived multiple topic scene classification model for high spatial
biosphere programme data and information system global land resolution remote sensing imagery. IEEE Trans. Geosci. Remote
cover data set (DISCover). Acta Astronautica 41(4-10), pp. 681– Sens. 54(4), pp. 2108–2123.
689.
Zhong, Y., Zhu, Q. and Zhang, L., 2015. Scene classification
Maggiori, E., Tarabalka, Y., Charpiat, G. and Alliez, P., 2017. based on the multifeature fusion probabilistic topic model for
Can semantic labeling methods generalize to any city? The Inria high spatial resolution remote sensing imagery. IEEE Trans.
aerial image labeling benchmark. In: Proc. IGARSS, pp. 3226– Geosci. Remote Sens. 53(11), pp. 6207–6222.
3229.
Zhu, X. X., Tuia, D., Mou, L., Xia, G., Zhang, L., Xu, F. and
Malkin, K., Robinson, C., Hou, L., Soobitsky, R., Samaras, D., Fraundorfer, F., 2017. Deep learning in remote sensing: A com-
Saltz, J., Joppa, L. and Jojic, N., 2019. Label super-resolution prehensive review and list of resources. IEEE Geosci. Remote
networks. In: Proc. ICLR, pp. 1–9. Sens. Mag. 5(4), pp. 8–36.
Mohajerani, S. and Saeedi, P., 2019. Cloud-Net: An
end-to-end cloud detection algorithm for Landsat 8 imagery.
arXiv:1901.10077.
Nogueira, K., Dos Santos, J. A., Fornazari, T., Silva, T. S. F.,
Morellato, L. P. and Torres, R., 2016. Towards vegetation species
discrimination by using data-driven descriptors. In: Proc. PRRS.