Introduction to
image segmentation
DEEP LEARNING FOR IMAGES WITH PYTORCH
Michal Oleszak
Machine Learning Engineer
Image segmentation
Image segmentation partitions the image into multiple segments on the pixel level
Each pixel in an image is assigned to a particular segment
Three types of segmentation:
Semantic segmentation
Instance segmentation
Panoptic segmentation
DEEP LEARNING FOR IMAGES WITH PYTORCH
Semantic segmentation
Each pixel classified into a class
All pixels belonging to the same class are treated equally
DEEP LEARNING FOR IMAGES WITH PYTORCH
Instance segmentation
Distinguishes between different instances of the same class
Background often not segmented
DEEP LEARNING FOR IMAGES WITH PYTORCH
Panoptic segmentation
Combines semantic and instance segmentations
Assigns a unique label to each instance of an object
Classifies background at pixel level
DEEP LEARNING FOR IMAGES WITH PYTORCH
Data annotations
image = [Link]("images/British_Shorthair_36.jpg")
mask = [Link]("annots/British_Shorthair_36.png")
transform = [Link]([
[Link]()
])
image_tensor = transform(image)
mask_tensor = transform(mask)
print(f"""Image shape: {image_tensor.shape}
Mask shape: {mask_tensor.shape}""")
Image shape: [Link]([3, 333, 500])
Mask shape: [Link]([1, 333, 500])
DEEP LEARNING FOR IMAGES WITH PYTORCH
Understanding the mask
Dataset documentation:
Pixel Annotations: 1: Foreground 2: Background 3: Not classified
Unique mask values:
mask_tensor.unique()
tensor([0.0039, 0.0078, 0.0118])
Pixel values are divided by 255 :
1 / 255 = 0.0039 - object
2 / 255 = 0.0.0078 - background
3 / 255 = 0.0118 - unclassified
DEEP LEARNING FOR IMAGES WITH PYTORCH
Creating a binary mask
binary_mask = [Link]( [Link]() :
mask_tensor == 1/255, Condition
[Link](1.0),
[Link](0.0), Value to use if condition met
)
Value to use otherwise
to_pil_image = [Link]() Transform mask to PIL image
mask = to_pil_image(binary_mask)
[Link](mask) Display mask image
DEEP LEARNING FOR IMAGES WITH PYTORCH
Segmenting the object
object_tensor = image_tensor * binary_mask Multiply image with the binary mask
Transform object to PIL image
to_pil_image = [Link]()
object_image = to_pil_image(object_tensor) Display object image
[Link](object_image)
DEEP LEARNING FOR IMAGES WITH PYTORCH
Let's practice!
DEEP LEARNING FOR IMAGES WITH PYTORCH
Instance
segmentation with
Mask R-CNN
DEEP LEARNING FOR IMAGES WITH PYTORCH
Michal Oleszak
Machine Learning Engineer
Faster R-CNN
DEEP LEARNING FOR IMAGES WITH PYTORCH
Mask R-CNN
DEEP LEARNING FOR IMAGES WITH PYTORCH
Pre-trained Masked R-CNN in PyTorch
from [Link] import \ Import the Mask R-CNN model
maskrcnn_resnet50_fpn
Load pre-trained model
model = maskrcnn_resnet50_fpn(pretrained=True) Load test image and transform to tensor
[Link]()
image = [Link]("cat_and_laptop.jpg")
transform = [Link]([
[Link]()
])
image_tensor = transform(image).unsqueeze(0)
with torch.no_grad():
prediction = model(image_tensor)
Pass image tensor to the model
DEEP LEARNING FOR IMAGES WITH PYTORCH
Model outputs
Labels Class probabilities
prediction[0]["labels"] prediction[0]["scores"]
tensor([ tensor([
17, 73, 76, 73, 67, 42, 63, 84,73, 65, 0.9981, 0.9672, 0.9061, 0.6893, 0.3729,
17, 73, 73, 73, 84, 72, 76, 76,17, 15 ...,
]) 0.0745, 0.0705, 0.0623, 0.0610, 0.0508
])
Class names
Masks
print(class_names[17], class_names[73])
prediction[0]["masks"]
cat laptop
tensor([[[[0., 0., 0., ..., 0., 0., 0.],
...]]]])
DEEP LEARNING FOR IMAGES WITH PYTORCH
Soft masks
Unique mask values
prediction[0]["masks"].unique()
tensor([0.0000e+00, 5.9713e-08, ...,
9.9989e-01, 9.9990e-01])
Mask R-CNN masks:
Values between 0 and 1
Represent the model's confidence that each pixel belongs to the object
Provide more nuanced information than binary masks
Can be binarized by thresholding if needed
DEEP LEARNING FOR IMAGES WITH PYTORCH
Displaying soft masks
masks = prediction[0]["masks"] Extract masks and labels from prediction
labels = prediction[0]["labels"]
Iterate over top two objects, plotting the
for i in range(2): original image
[Link](image)
For each object, plot the semi-transparent
[Link](
masks[i, 0], mask
cmap="jet",
Add title and display
alpha=0.5,
)
[Link](
f"Object: {class_names[labels[i]]}"
)
[Link]()
DEEP LEARNING FOR IMAGES WITH PYTORCH
Displaying soft masks
DEEP LEARNING FOR IMAGES WITH PYTORCH
Let's practice!
DEEP LEARNING FOR IMAGES WITH PYTORCH
Semantic
segmentation with
U-Net
DEEP LEARNING FOR IMAGES WITH PYTORCH
Michal Oleszak
Machine Learning Engineer
Semantic segmentation
No distinction between different instances of the same class
Useful for medical imaging or satellite image analysis
Popular architecture: U-Net
DEEP LEARNING FOR IMAGES WITH PYTORCH
U-Net architecture
Encoder:
Convolutional and pooling layers
Downsampling: reduces spatial dimensions while increasing depth
DEEP LEARNING FOR IMAGES WITH PYTORCH
U-Net architecture
Decoder:
Symmetric to the encoder
Upsamples feature maps with transposed convolutions
DEEP LEARNING FOR IMAGES WITH PYTORCH
U-Net architecture
Skip connections:
Links from the encoder to the decoder
Preserve details lost in downsampling
DEEP LEARNING FOR IMAGES WITH PYTORCH
Transposed convolution
Upsamples feature maps in the decoder: increases height and width while reducing depth
Transposed convolution process:
1. Insert zeros between or around the input feature map
2. Perform a regular convolution on the zero-padded input
DEEP LEARNING FOR IMAGES WITH PYTORCH
Transposed convolution in PyTorch
import [Link] as nn
upsample = nn.ConvTranspose2d(
in_channels=in_channels,
out_channels=out_channels,
kernel_size=2,
stride=2,
)
DEEP LEARNING FOR IMAGES WITH PYTORCH
U-Net: layer definitions
class UNet([Link]): Encoder:
def __init__(self, in_channels, out_channels):
super(UNet, self).__init__() Convolutional blocks
def conv_block(self, in_channels, out_channels):
self.enc1 = self.conv_block(in_channels, 64)
return [Link](
self.enc2 = self.conv_block(64, 128)
nn.Conv2d(in_channels, out_channels),
self.enc3 = self.conv_block(128, 256)
[Link](inplace=True),
self.enc4 = self.conv_block(256, 512)
nn.Conv2d(out_channels, out_channels),
[Link] = nn.MaxPool2d(kernel_size=2, stride=2)
[Link](inplace=True)
)
self.upconv3 = nn.ConvTranspose2d(512, 256,
kernel_size=2, stride=2)
self.upconv2 = nn.ConvTranspose2d(256, 128, Pooling layer
kernel_size=2, stride=2)
self.upconv1 = nn.ConvTranspose2d(128, 64,
Decoder:
kernel_size=2, stride=2)
Transposed convolutions
self.dec1 = self.conv_block(512, 256)
self.dec2 = self.conv_block(256, 128)
Convolutional blocks
self.dec3 = self.conv_block(128, 64)
[Link] = nn.Conv2d(64, out_channels, kernel_size=1)
DEEP LEARNING FOR IMAGES WITH PYTORCH
U-Net: forward method
def forward(self, x): Pass input through encoder's convolutional
x1 = self.enc1(x)
x2 = self.enc2([Link](x1))
blocks and pooling layers
x3 = self.enc3([Link](x2))
x4 = self.enc4([Link](x3)) Decoder and skip connections:
Pass encoded input through transpose
x = self.upconv3(x4)
x = [Link]([x, x3], dim=1) convolution
x = self.dec1(x)
Concatenate with corresponding
x = self.upconv2(x)
x = [Link]([x, x2], dim=1)
encoder output
x = self.dec2(x)
Pass through convolution block
x = self.upconv1(x)
x = [Link]([x, x1], dim=1) Repeat for all decoder steps
x = self.dec3(x)
Return output of the last decoder step
return [Link](x)
DEEP LEARNING FOR IMAGES WITH PYTORCH
Running inference
model = UNet()
[Link]()
image = [Link]("[Link]")
transform = [Link]([[Link]()])
image_tensor = transform(image).unsqueeze(0)
with torch.no_grad():
prediction = model(image_tensor).squeeze(0)
[Link](prediction[1, :, :])
[Link]()
DEEP LEARNING FOR IMAGES WITH PYTORCH
Let's practice!
DEEP LEARNING FOR IMAGES WITH PYTORCH
Panoptic
segmentation
DEEP LEARNING FOR IMAGES WITH PYTORCH
Michal Oleszak
Machine Learning Engineer
Panoptic segmentation challenge
Original image Semantic segmentation
Instance segmentation Panoptic segmentation
DEEP LEARNING FOR IMAGES WITH PYTORCH
Panoptic segmentation workflow
Combining semantic and instance segmentation can be complex:
Overlaps
Ensuring unique instance IDs
Our workflow:
1. Generate semantic masks
2. Combine them into a single mask
3. Initialize the panoptic mask as the semantic mask
4. Generate instance masks
5. Iterate over instance masks and overlay detected objects onto the semantic mask
DEEP LEARNING FOR IMAGES WITH PYTORCH
Semantic masks
model = UNet() Instantiate the model
Produce semantic masks for the input
with torch.no_grad():
image
semantic_masks = model(image_tensor)
print(semantic_masks.shape) Choose highest-probability class for each
pixel
[Link]([1, 3, 427, 640])
semantic_mask = [Link](
semantic_masks, dim=1
)
DEEP LEARNING FOR IMAGES WITH PYTORCH
Instance masks
model = MaskRCNN() Load instance segmentation model
with torch.no_grad():
Produce instance masks
instance_masks = model(image_tensor)[0]["masks"]
print(instance_masks.shape)
[Link]([80, 1, 427, 640])
DEEP LEARNING FOR IMAGES WITH PYTORCH
Panoptic masks
panoptic_mask = [Link](semantic_mask) Initialize panoptic mask as semantic_mask
Iterate over instance masks
instance_id = 3
for mask in instance_masks: Set panoptic mask to instance ID where
panoptic_mask[mask > 0.5] = instance_id mask > 0.5
instance_id += 1
Increase instance ID counter
DEEP LEARNING FOR IMAGES WITH PYTORCH
Let's practice!
DEEP LEARNING FOR IMAGES WITH PYTORCH