Robotics Technologies
Lec# 14 Computer Vision for Mobile Robots
3D Semantic Segmentation with LiDAR and
Camera Images
Computer Vision Overview
❑ Image Formation using Camera
❑ Edge & Corner detection
❑ Feature Extraction
❑ 3D Objects and Stereo Vision
❑ Object Detection and Recognition
❑ Segmentation
❑ Motion & Tracking
Computer Vision.
sky • street
1. Camera-Imaging • people
• building
2. Object Detection • mountain
building • tourism
3. 3D Imaging • cloudy
• brick
4. Scene Categorization tree •…
5. Scene Tagging building
6. Image Parsing & banner
Segmentation
street lamp
find pedestrians?
• Scene
Understanding? market
people • outdoor/indoor
• city/forest/factory/etc. 57
Svetlana Lazebnik
What is Computer Vision
• Computer vision is a field of computer science
• works on enabling computers to see,
• identify and process images in the same way that
human vision does, and
• then provide appropriate output.
• It is like imparting human intelligence and instincts
about vision to a computer.
Computer Vision vs Human Vision
The goal of Computer Vision
Make computers understand images.
(e.g. photos, videos, medical images)
The goal of computer vision
• Compute the 3D shape of the world
ZED 2i
Camera
Stereo Vision
extracts depth information from a scene by comparing two
or more images captured from slightly different
perspectives.
● Adds depth to the image.
● Stereo Occupancy Grids
3D Reconstruction (RGB/RGBD)
The goal of computer vision
• Recognize objects and people
What Kind of information?
Semantic Geometric
Information Information
Semantic Information: Scene and Context
outdoor city traffic
Semantic Information: Object Recognition
tree
window
person
bike
Object Detection
Object Recognition
Recognition and Detection
Image/Video Recognition
Motion and Optical Flow
estimates the motion of pixels in a video by generating a
dense field of 2D displacement vectors that represent
movement between two consecutive frames.
[Link]
Tracking
[Link]
Segmentation
the process of dividing a digital image into multiple
regions based on pixels called segments
Semantic Segmentation
Assigns a class label to every pixel in an image
allowing a machine to understand the image at a pixel-level detail by grouping pixels into categories
like "road," "sky," "person," or "car"
person
bike
• Task: Label every pixel
• Don’t differentiate instances
Instance segmentation
• instance segmentation distinguishes individual
objects within those categories, even if they are the
same class (e.g., each car is labeled separately).
• Instance segmentation is more detailed as it
provides unique masks for each object instance.
Object Segmentation vs Detection
Pixel-level labels Bounding box labels
Category only Category + instance
3D Semantic
Segmentation with
LiDAR and Camera
Images
● Enhances autonomous driving and
computer vision through multi-modal
data fusion.
● Objective: Improve scene
understanding via combined
● LiDAR and
● camera data
● IMU
Motivation
● Scene Understanding: Critical for safe
autonomous vehicle navigation.
● Complementary Sensors:
○ LiDAR: Precise 3D geometry and spatial
structure.
○ Cameras: Rich visual appearance and
texture details.
○ IMU
● Limitation of Single-Modality Systems:
Often lack sufficient context in
complex scenes.
● Objective: Fuse LiDAR, IMU and
camera data to improve 3D semantic
segmentation.
Problem Statement
● Input Modalities:
○ 3D LiDAR point clouds.
○ 2D multi-view camera images
○ IMU
● Output: Semantic label for each
LiDAR point.
● Dataset: Down-scaled NuScenes
dataset.
● Class Scope: 16 semantic
classes.
Methodology
● Multi-Modal Pipeline:
○ 3D Backbone: Point
Transformer V3 (PTv3) for 3D
feature extraction.
○ 2D Backbone: DINOv2 for 2D
image feature extraction.
○ Fusion: Combines features
into a unified representation.
○ Prediction: Per-point
semantic classification.
3D Feature Extraction with Point
Transformer V3
● PTv3 Architecture:
Transformer-based backbone
optimized for point clouds.
● Serialization: Efficient
mechanisms for processing
point clouds.
● Point-Level Feature
Generation:
○ Outputs a 64-dimensional
feature vector per 3D point.
2D Feature Extraction with DINOv2
● Backbone: ViT-S/14 architecture.
● Self-Supervised Learning:
Trained via self-supervised
learning.
● Patch-Level Features: Image
features extracted at the patch
level.
● Layer Selection: Layers 9 and 12
evaluated for optimal
performance.
2D–3D Fusion Strategies
● Direct Fusion:
○ Multiplies aligned point and
image features.
○ Provided limited gains in
experiments.
● Projection-Based Fusion:
○ Projects 3D points into 2D image
space.
○ Retrieves corresponding patch
tokens.
Projection Fusion Mechanism
● 3D-to-2D Projection: Projects 3D
points onto 2D image planes.
● Patch Token Retrieval: Identifies
corresponding image patches.
● Multi-Camera Averaging:
Averages features across six
cameras to enhancerobustness.
Dataset and Training Setup
● Dataset Splits:
○ Training: 323 samples.
○ Validation: 40 samples.
○ Test: 41 samples.
● Loss Functions: Combination of
Cross-Entropy and Lovasz-Softmax
losses.
● Optimizer: AdamW.
● Training Duration: 10 epochs.
Quantitative Results
● Baseline (PTv3): mIoU of
0.5935.
● Projection Fusion: mIoU
improved to 0.6975.
● Convergence: Multi-
modal models converge
faster than single-modal
models.
Qualitative Results
● Improvements:
○ Segmentation of roads, vegetation,
and vehicles.
○ Better boundary definition in
complex scenes.
● Challenges:
○ Fine details and small or rare classes
remain difficult to segment.
Conclusion
● Key Finding: Multi-modal fusion
significantly improves 3D
semantic segmentation.
● Effective Strategy: Projection-
based fusion is the most effective
strategy tested.
● Benefit: DINOv2 features enhance
spatial predictions and overall
accuracy.
Future Work
● Full nuScenes Dataset: Training
on the complete dataset to
enhance generalization.
● Multi-Layer DINOv2 Fusion:
Exploring fusion of multiple
DINOv2 feature layers.
● Attention-Based Fusion: Using
attention mechanisms to weight
feature contributions.
● Joint 2D–3D Pretraining: Jointly
pretraining the 2D and 3D
backbones.
Computer Vision Applications
Intelligent Vehicles
Smart cars
Tesla?
Warehouse Robot
Mobile Robots
Simultaneously Localization and Mapping (SLAM)
a technology that allows robots to build a map of an unknown
environment while simultaneously tracking their own position
within that map.
Credit: Han Wang
Visual Odometry
a technique used by robots, autonomous vehicles to estimate their
own motion (ego-motion) by analyzing consecutive images from a
cameras.
Credit: Kuan Xu
3D Body Tracking (Kinect)
3D Scanning
Questions