Deep Learning for Computer Vision

Segmentation

Muhammad Faisal
Research Associate @ VisPro Lab & Intelligent Machines Lab
Information Technology University, Lahore
Outline

■ What is object segmentation?

■ Applications

■ Semantic segmentation
■ From image classification to semantic segmentation
■ Fully convolutional networks
■ Learnable Upsampling
■ Skip connections
■ FCN8s, Dilated Convolutions, U-Net, ...

■ Instance segmentation
■ Simultaneous Detection and Segmentation
■ Mask R-CNN

■ Domain Adaptation of Semantic Segmentation

2
Object segmentation: applications

Dr. Verónica Vilaplana: UPC

3
Image Segmentation: Computer Vision Tasks

CS231- Fei-Fei Li 4
Semantic Segmentation

■ Label every pixel:

recognize the class of
every pixel

■ Do not differentiate
instances, only care
about pixels

■ A key part of Scene

Understanding

CS231- Fei-Fei Li,

Dr. Verónica Vilaplana: UPC 5
From classification to semantic segmentation

Dr. Verónica Vilaplana: UPC

6
 From classification to semantic segmentation

Farabet et al, “Learning Hierarchical Features for Scene Labeling,” TPAMI 2013
Pinheiro and Collobert, “Recurrent Convolutional Neural Networks for Scene Labeling”, ICML 2014 7
From classification to semantic segmentation

Problem: Very inefficient! Not reusing shared

features between overlapping patches

8
 From classification to semantic segmentation

▪ Dense prediction: fully convolutional, end to end, pixel-to-pixel network

CS231- Fei-Fei Li 9
From classification to semantic segmentation

Problem:
convolutions at original image resolution will be
very expensive

10
 From classification to semantic segmentation

Long, Shelhamer, and Darrell, “Fully Convolutional Networks for Semantic Segmentation”, CVPR 2015
Noh et al, “Learning Deconvolution Network for Semantic Segmentation”, ICCV 2015 11
Learnable upsampling: recovering spatial shape

12
 Learnable upsampling: recovering spatial shape

Noh et al, “Learning Deconvolution Network for Semantic Segmentation”, ICCV 2015
13
 Learnable upsampling: recovering spatial shape

Noh et al, “Learning Deconvolution Network for Semantic Segmentation”, ICCV 2015
14
 Resolution: spectrum of deep features

▪ Problem: coarse output

▪ Combine where (local, shallow) with what (global, deep)

cf. Hariharan et al. CVPR15 “hypercolumn”

15
 Fine details: skip connections

Long, Shelhamer, and Darrell, “Fully Convolutional Networks for Semantic Segmentation”, CVPR 2015
16
Semantic segmentation: Typical Architectures

▪ Downsampling path: extracts coarse features

▪ Upsampling path: recovers input image resolution
▪ Skip connections: recovers detailed information
▪ Post-processing (optional): refines predictions (CRF)

▪ Other architectures:
▪ U-Net (Ronnemberger, 2015)
▪ SegNet (2015)
▪ DeepLab: ‘atrous’ convolutions + spatial pyramid + CRF (Chen, ICLR 2015)
▪ CRF-RNN: FCN + CRF as Recurrent NN (Zheng, ICCV 2015)
▪ Fully Convolutional DenseNets (Jégou, 2016)
▪ Dilated convolutions (Yu, 2016) 17
Semantic segmentation architectures: SegNet

▪ Encoder-Decoder architecture
▪ Using upsampling instead of deconvolutions
▪ Uses Max value indices for upsampling instead of bilinear or other upsampling
methods

SegNet: 2015 18
 Semantic segmentation architectures: UNet

▪ A contracting path and an expansive path

▪ Adds convolutions in the upsampling path (“symmetric” net)
▪ Skip connections: concatenation of feature maps

Dense UNet: Where

each D-block is a
DenseNet block

Ronneberger et al, “U-Net: Convolutional Networks for Biomedical Image Segmentation”, arXiv 2015
19
Huang et al, “Densely connected convolutional networks”, arxiv Aug 2016
 Semantic segmentation architectures: Dilated convolutions

▪ Systematically aggregate multiscale contextual information without losing

resolution

▪ Large Memory requirements

Yu, Koltun, Multi-scale context aggregation by dilated convolutions, 2016

20
 Semantic segmentation architectures: DeepLab

▪ Atrous spatial Pyramid pooling

▪ No decoder there..?
▪ Fully connected CRF for
post-processing

DeepLab: Semantic Image Segm. with Deep Conv. Nets, Atrous Convolution, and Fully Connected CRFs, Chen, et al, 2015
21
 FCN-8s vs DeepLab vs Dilated Convolutions

FC8: Fully Convolutional Networks for Semantic Segmentation, Long, Darrell, Shelhamer, 2014-2016
DeepLab: Semantic Image Segm. with Deep Conv. Nets, Atrous Convolution, and Fully Connected CRFs, Chen, et al, 2015
Dilated convolutions: Multi-scale context aggregation by dilated convolutions, Yu, Koltun, 2016 22
 Instance Semantic segmentation

CS231: Fei-Fei- Li
23
 Instance Semantic segmentation

▪ Detect instances, categorize and label every pixel

▪ Labels are class-aware and instance-aware

Arnab,Torr “Pixelwise instance segmentation with a dynamically instantiated network”, CVPR 2017 24
 Instance segmentation: Multi-task cascades

Dai et al, “Instance-aware Semantic Segmentation via Multi-task Network Cascades”, CVPR 2016 25
 Instance segmentation: Multi-task cascades

Dai et al, “Instance-aware Semantic Segmentation via Multi-task Network Cascades”, CVPR 2016 26
 Instance segmentation: Mask R-CNN

▪ Extension of Faster R-CNN to instance segmentation

▪ A Fully Convolutional Network (FCN) is added on top of the CNN features
of Faster R-CNN to generate a mask (segmentation output).
▪ This is in parallel to the classification and bounding box regression network of
Faster R-CNN
▪ RoIAlign instead of RoIPool to properly aligning extracted features with input

He et al, Mask R-CNN, 2017 27

 Instance segmentation: Mask R-CNN

▪ Masks are combined with classifications and bounding boxes from Faster R-CNN

He et al, Mask R-CNN, 2017 28

 Instance segmentation: Mask R-CNN

▪ Results on COCO dataset

▪ MNC and FCIS were winners of COCO 2015 and 2016

He et al, Mask R-CNN, 2017

MNC: Dai et al, “Instance-aware Semantic Segmentation via Multi-task Network Cascades”, CVPR 2016
FCIS: Li et al., “Fully convolutional instance-aware semantic segmentation”, CVPR 2017
29
Domain Adaptation: Problem Statement

■ How can we learn, using labeled data from a source distribution, a low-error
classifier for another related target distribution?

■ Exp: Can we train classifiers with photos already collected and annotated, and
make the classifiers work well on mobile camera images?

30
Domain Shift

■ Synthetic to real distribution

■ Geological position, illumination changes, camera, weather conditions
■ Day to night shifting
■ Appearance variability, resolution variations

[1] No More Discrimination: Cross City Adaptation of Road Scene Segmenters. CVPR 2017.
[2] Boosting Domain Adaptation by Discovering Latent Domains. CVPR 2018.
[3] Unsupervised Domain Adaptation for semantic segmentation via class balanced self training. ECCV 2018.
31
Domain Adaptation

■ When domain adaptation is possible?

■ How to tune Model parameters and hyperparameters with no labeled
information from target?

Underlying idea: Try to move closer the two distributions

32
Domain Adaptation: Main Approaches

■ Feature-based methods/Find new representation spaces

■ Find a common space where source and target are close (projection, new features, etc)

■ Adjustment/Iterative methods
■ Modify the model by incorporating pseudo-labeled information

■ Reweighting/Instance-based methods
■ Correct a sample bias by reweighting source labeled data: Make source data to look like
target data
■ Output based methods
■ Make the output of both the domains to look similar

33
 Domain Adaptation: Output Space

Yi-Hsuan et al, Learning to Adapt Structured Output Space for Semantic Segmentation , CVPR 2018 34
 Domain Adaptation: Output Space

Yi-Hsuan et al, Learning to Adapt Structured Output Space for Semantic Segmentation , CVPR 2018 35
Domain Adaptation: For Satellite Imagery Built-up Region Segmentation

36
Available Datasets

■ Pascal VOC
■ MS-COCO
■ Berkeley Deep Drive (10K images with 19 classes)
■ GTA5(24966 images with 19 classes)
■ SYNTHIA (9400 images with 19 classes)
■ Cityscapes(5000 images with 19 classes)
■ Cross-City dataset (13000 images with 13 classes)

37
Video Object Segmentation

■ Labelling independently moving object(s) in videos.

■ Challenges:
■ Moving Camera
■ Occlusion
■ Deforming Nature of Objects

Input Output:
RGB Video Segmentation
38
Video Object Segmentation

VOS from Static Images (CVPR-2017)

Given the mask of initial frame, the
FusionSeg (CVPR-2017) approach tries to predict the output next
Takes RGB Frame and Optical Flow as input. frame.

SegFlow (ICCV-2017)

Train a network to
simultaneous predict the
segmentation and optical
flow.

39
Video Object Segmentation

■ Exploiting Geometric Constraints on Dense Trajectories for Motion Saliency

40
Exploiting Geometric Constraints on Dense Trajectories for Motion
Saliency

X-Displacement Y-Displacement Epipolar Distance Motion Image

RGB GroundTruth MpNet (CVPR-2017) EpO (our motion)

Comparison of our EpO Network with Mp-Net

41
Exploiting Geometric Constraints on Dense Trajectories for Motion
Saliency

X-Displacement Y-Displacement Epipolar Distance Motion Image

RGB GroundTruth MpNet (CVPR-2017) EpO (our motion)

Comparison of our EpO Network with Mp-Net 42

Exploiting Geometric Constraints on Dense Trajectories for Motion
Saliency

GroundTruth LVO (ICCV-2017) ARP (CVPR 2018)

STP (ECCV–2018) PDB (ECCV-2018) EpO+ (our fusion)

43
Summary

■ UNet is a successful and widely used Segmentation Network

■ Semantic Segmentation in static images have reached to an acceptable level.
■ Instance Segmentation still needs improvements.
■ There is a lot of room for improvement for Segmentation in Videos.
■ Exploiting temporal properties of videos in videos

■ Domain Adaptation is a challenging problem.

44