1. What is Image Segmentation?

2. Why do we Need Image

Segmentation?
3. The Different Types of Image
Segmentation
4. Region-Based Segmentation
5. Edge Detection Segmentation
6. Segmentation based on Clustering
7. Facebook’s Mask R-CNN
Framework
8. Summary of Image Segmentation
Techniques

What is Image Segmentation?

Let’s understand image segmentation using
a simple example. Consider the below
image:
There’s only one object here – a dog. We
can build a straightforward cat-dog
classifier model and predict that there’s a
dog in the given image. But what if we have
both a cat and a dog in a single image?

We can train a multi-label classifier, in that

instance. Now, there’s another caveat – we
won’t know the location of either
animal/object in the image.
That’s where image localization comes into
the picture (no pun intended!). It helps us to
identify the location of a single object in the
given image. In case we have multiple
objects present, we then rely on the
concept of object detection (OD). We can
predict the location along with the class for
each object using OD.

Before detecting the objects and even

before classifying the image, we need to
understand what the image consists of.
Enter – Image Segmentation.

So how does image segmentation work?

We can divide or partition the image into
various parts called segments. It’s not a
great idea to process the entire image at
the same time as there will be regions in
the image which do not contain any
information. By dividing the image into
segments, we can make use of the
important segments for processing the
image. That, in a nutshell, is how image
segmentation works.
An image is a collection or set of different
pixels. We group together the pixels that
have similar attributes using image
segmentation. Take a moment to go
through the below visual (it’ll give you a
practical idea of image segmentation):
 Source : [Link]
Object detection builds a bounding box
corresponding to each class in the image.
But it tells us nothing about the shape of the
object. We only get the set of bounding box
coordinates. We want to get more
information – this is too vague for our
purposes.
Image segmentation creates a pixel-wise
mask for each object in the image. This
technique gives us a far more granular
understanding of the object(s) in the image.
Why do we need to go this deep? Can’t all
image processing tasks be solved using
simple bounding box coordinates? Let’s
take a real-world example to answer this
pertinent question.

Why do we need Image Segmentation?

Cancer has long been a deadly illness.
Even in today’s age of technological
advancements, cancer can be fatal if we
don’t identify it at an early stage. Detecting
cancerous cell(s) as quickly as possible can
potentially save millions of lives.
The shape of the cancerous cells plays a
vital role in determining the severity of the
cancer. You might have put the pieces
together – object detection will not be very
useful here. We will only generate bounding
boxes which will not help us in identifying
the shape of the cells.
 Image Segmentation techniques make a
MASSIVE impact here. They help us
approach this problem in a more granular
manner and get more meaningful results. A
win-win for everyone in the healthcare
industry.

Source: Wikipedia
Here, we can clearly see the shapes of all
the cancerous cells. There are many other
applications where Image segmentation is
transforming industries:
 Traffic Control Systems
 Self Driving Cars
 Locating objects in satellite images
There are even more applications where
Image Segmentation is very useful. Feel
free to share them with me in the comments
section below this article – let’s see if we
can build something together.

The Different Types of Image Segmentation

We can broadly divide image segmentation
techniques into two types. Consider the
below images:

Can you identify the difference between

these two? Both the images are using
image segmentation to identify and locate
the people present.
 In image 1, every pixel belongs to a
particular class (either background or
person). Also, all the pixels belonging to a
particular class are represented by the
same color (background as black and
person as pink). This is an example of
semantic segmentation
 Image 2 has also assigned a particular
class to each pixel of the image. However,
different objects of the same class have
different colors (Person 1 as red, Person 2
as green, background as black, etc.). This
is an example of instance segmentation

Let me quickly summarize what we’ve

learned. If there are 5 people in an image,
semantic segmentation will focus on
classifying all the people as a single
instance. Instance segmentation, on the
other hand. will identify each of these
people individually.
So far, we have delved into the theoretical
concepts of image processing and
segmentation. Let’s mix things up a bit –
we’ll combine learning concepts with
implementing them in Python. I strongly
believe that’s the best way to learn and
remember any topic.

Region-based Segmentation
One simple way to segment different
objects could be to use their pixel values.
An important point to note – the pixel values
will be different for the objects and the
image’s background if there’s a sharp
contrast between them.
In this case, we can set a threshold value.
The pixel values falling below or above that
threshold can be classified accordingly (as
an object or the background). This
technique is known as Threshold
Segmentation.
If we want to divide the image into two
regions (object and background), we define
a single threshold value. This is known as
the global threshold.

If we have multiple objects along with the

background, we must define multiple
thresholds. These thresholds are
collectively known as the local threshold.
Let’s implement what we’ve learned in this
section. Download this image and run the
below code. It will give you a better
understanding of how thresholding works
(you can use any image of your choice if
you feel like experimenting!).
First, we’ll import the required libraries.
from [Link] import
rgb2gray
import numpy as np
import cv2
import [Link] as
plt
%matplotlib inline
from scipy import ndimage
view rawimport_library.py hosted
with by GitHub
Let’s read the downloaded image and plot
it:
image
=[Link]('[Link]')
[Link]
[Link](image)
view rawread_image_1.py hosted
with by GitHub
It is a three-channel image (RGB). We need
to convert it into grayscale so that we only
have a single channel. Doing this will also
help us get a better understanding of how
the algorithm works.
gray = rgb2gray(image)
[Link](gray,
cmap='gray')
view rawgray_scale.py hosted
with by GitHub
Now, we want to apply a certain threshold
to this image. This threshold should
separate the image into two parts – the
foreground and the background. Before we
do that, let’s quickly check the shape of this
image:
[Link]
(192, 263)
The height and width of the image is 192
and 263 respectively. We will take the
mean of the pixel values and use that as
a threshold. If the pixel value is more than
our threshold, we can say that it belongs to
an object. If the pixel value is less than the
threshold, it will be treated as the
background. Let’s code this:
gray_r =
[Link]([Link][0]*gray
.shape[1])
for i in
range(gray_r.shape[0]):
if gray_r[i] >
gray_r.mean():
gray_r[i] = 1
else:
gray_r[i] = 0
gray =
gray_r.reshape([Link][0],gr
[Link][1])
[Link](gray, cmap='gray')
view
rawglobal_threshold.py hosted
with by GitHub
Nice! The darker region (black) represents
the background and the brighter (white)
region is the foreground. We can define
multiple thresholds as well to detect
multiple objects:

gray = rgb2gray(image)
gray_r =
[Link]([Link][0]*gra
[Link][1])
for i in
range(gray_r.shape[0]):
if gray_r[i] >
 gray_r.mean():
gray_r[i] = 3
elif gray_r[i] > 0.5:
gray_r[i] = 2
elif gray_r[i] > 0.25:
gray_r[i] = 1
else:
gray_r[i] = 0
gray =
gray_r.reshape([Link][0],g
[Link][1])
[Link](gray, cmap='gray')
view rawlocal_threshold.py hosted
with by GitHub
 There are four different segments in the
above image. You can set different
threshold values and check how the
segments are made. Some of the
advantages of this method are:
 Calculations are simpler
 Fast operation speed
 When the object and background have high
contrast, this method performs really well
But there are some limitations to this
approach. When we don’t have significant
grayscale difference, or there is an overlap
of the grayscale pixel values, it becomes
very difficult to get accurate segments.

Edge Detection Segmentation

What divides two objects in an image?
There is always an edge between two
adjacent regions with different grayscale
values (pixel values). The edges can be
 considered as the discontinuous local
features of an image.
We can make use of this discontinuity to
detect edges and hence define a boundary
of the object. This helps us in detecting the
shapes of multiple objects present in a
given image. Now the question is how can
we detect these edges? This is where we
can make use of filters and convolutions.
Refer to this article if you need to learn
about these concepts.
The below visual will help you understand
how a filter colvolves over an image :

Here’s the step-by-step process of how this

works:
 Take the weight matrix
 Put it on top of the image
 Perform element-wise multiplication and get
the output
 Move the weight matrix as per the stride
chosen
 Convolve until all the pixels of the input are
used
The values of the weight matrix define the
output of the convolution. My advice – it
helps to extract features from the input.
Researchers have found that choosing
some specific values for these weight
matrices helps us to detect horizontal or
vertical edges (or even the combination of
horizontal and vertical edges).
One such weight matrix is the sobel
operator. It is typically used to detect
edges. The sobel operator has two weight
matrices – one for detecting horizontal
edges and the other for detecting vertical
edges. Let me show how these operators
look and we will then implement them in
Python.
Sobel filter (horizontal) =
121
000
-1-2-1

Sobel filter (vertical) =

-
01
1
-
02
2
-
01
1

Edge detection works by convolving these

filters over the given image. Let’s visualize
them on this article.
image =
 [Link]('[Link]')
[Link](image)
view rawread_image_2.py hosted
with by GitHub

It should be fairly simple for us to

understand how the edges are detected in
this image. Let’s convert it into grayscale
and define the sobel filter (both horizontal
and vertical) that will be convolved over this
image:
# converting to grayscale
gray = rgb2gray(image)
 # defining the sobel filters
sobel_horizontal =
[Link]([[Link]([1, 2, 1]),
[Link]([0, 0, 0]),
[Link]([-1, -2, -1])])
print(sobel_horizontal, 'is a
kernel for detecting
horizontal edges')

sobel_vertical =
[Link]([[Link]([-1, 0,
1]), [Link]([-2, 0, 2]),
[Link]([-1, 0, 1])])
print(sobel_vertical, 'is a
kernel for detecting vertical
edges')
view rawsobel_filters.py hosted
with by GitHub
Now, convolve this filter over the image
using the convolve function of
the ndimage package from scipy.
out_h = [Link](gray,
sobel_horizontal,
mode='reflect')
out_v = [Link](gray,
sobel_vertical,
mode='reflect')
# here mode determines how the
input array is extended when
the filter overlaps a border.
view
rawconvolving_sobel_filters.py hos
ted with by GitHub
Let’s plot these results:
[Link](out_h,
cmap='gray')
view rawplot_1.py hosted with
by GitHub
 [Link](out_v,
cmap='gray')
view rawplot_2.py hosted with
by GitHub

Here, we are able to identify the horizontal

as well as the vertical edges. There is one
more type of filter that can detect both
horizontal and vertical edges at the same
time. This is called the laplace operator:
11 1
-
1 1
8
11 1

Let’s define this filter in Python and

convolve it on the same image:
kernel_laplace =
[Link]([[Link]([1, 1, 1]),
[Link]([1, -8, 1]),
[Link]([1, 1, 1])])
print(kernel_laplace, 'is a
laplacian kernel')
view rawlaplacian_filter.py hosted
with by GitHub

Next, convolve the filter and print the

output:
 out_l = [Link](gray,
kernel_laplace,
mode='reflect')
[Link](out_l, cmap='gray')
view
rawconvolving_laplacian_filter.py
hosted with by GitHub

Here, we can see that our method has

detected both horizontal as well as vertical
edges. I encourage you to try it on different
images and share your results with me.
Remember, the best way to learn is by
practicing!
Image Segmentation based on Clustering
This idea might have come to you while
reading about image segmentation. Can’t
we use clustering techniques to divide
images into segments? We certainly can!
In this section, we’ll get an an intuition of
what clustering is (it’s always good to revise
certain concepts!) and how we can use of it
to segment images.
Clustering is the task of dividing the
population (data points) into a number of
groups, such that data points in the same
groups are more similar to other data points
in that same group than those in other
groups. These groups are known as
clusters.
One of the most commonly used clustering
algorithms is k-means. Here, the k
represents the number of clusters (not to be
confused with k-nearest neighbor). Let’s
understand how k-means works:
 1. First, randomly select k initial
clusters
2. Randomly assign each data point to
any one of the k clusters
3. Calculate the centers of these
clusters
4. Calculate the distance of all the
points from the center of each cluster
5. Depending on this distance, the
points are reassigned to the nearest
cluster
6. Calculate the center of the newly
formed clusters
7. Finally, repeat steps (4), (5) and (6)
until either the center of the clusters
does not change or we reach the set
number of iterations
The key advantage of using k-means
algorithm is that it is simple and easy to
understand. We are assigning the points to
the clusters which are closest to them.
Let’s put our learning to the test and check
how well k-means segments the objects in
an image. We will be using this image, so
download it, read it and and check its
dimensions:
pic = [Link]('[Link]')/255
# dividing by 255 to bring the
pixel values between 0 and 1
print([Link])
[Link](pic)
view rawread_image_3.py hosted
with by GitHub
It’s a 3-dimensional image of shape (192,
263, 3). For clustering the image using k-
means, we first need to convert it into a 2-
dimensional array whose shape will be
(length*width, channels). In our example,
this will be (192*263, 3).
pic_n =
[Link]([Link][0]*[Link]
ape[1], [Link][2])
pic_n.shape
view
rawreshaping_image.py hosted
with by GitHub
(50496, 3)
We can see that the image has been
converted to a 2-dimensional array. Next, fit
the k-means algorithm on this reshaped
array and obtain the clusters.
The cluster_centers_ function of k-means
will return the cluster centers and labels_
function will give us the label for each pixel
(it will tell us which pixel of the image
belongs to which cluster).
from [Link] import
KMeans
kmeans = KMeans(n_clusters=5,
random_state=0).fit(pic_n)
pic2show =
kmeans.cluster_centers_[kmeans.
labels_]
view [Link] hosted with
by GitHub
I have chosen 5 clusters for this article but
you can play around with this number and
check the results. Now, let’s bring back the
clusters to their original shape, i.e. 3-
dimensional image, and plot the results.
cluster_pic =
[Link]([Link][0],
[Link][1], [Link][2])
[Link](cluster_pic)
view [Link] hosted with
by GitHub

Amazing, isn’t it? We are able to segment

the image pretty well using just 5 clusters.
I’m sure you’ll be able to improve the
segmentation by increasing the number of
clusters.
k-means works really well when we have a
small dataset. It can segment the objects in
the image and give impressive results. But
the algorithm hits a roadblock when applied
on a large dataset (more number of
images).
It looks at all the samples at every iteration,
so the time taken is too high. Hence, it’s
also too expensive to implement. And since
k-means is a distance-based algorithm, it is
only applicable to convex datasets and is
not suitable for clustering non-convex
clusters.
Finally, let’s look at a simple, flexible and
general approach for image segmentation.

Mask R-CNN
Data scientists and researchers at
Facebook AI Research (FAIR) pioneered a
deep learning architecture, called Mask R-
CNN, that can create a pixel-wise mask for
each object in an image. This is a really
cool concept so follow along closely!
Mask R-CNN is an extension of the
popular Faster R-CNN object detection
architecture. Mask R-CNN adds a branch to
 the already existing Faster R-CNN
outputs. The Faster R-CNN method
generates two things for each object in the
image:
 Its class
 The bounding box coordinates
Mask R-CNN adds a third branch to this
which outputs the object mask as well.
Take a look at the below image to get an
intuition of how Mask R-CNN works on the
inside:
 Source: [Link]
1. We take an image as input and pass
it to the ConvNet, which returns the
feature map for that image
2. Region proposal network (RPN) is
applied on these feature maps. This
 returns the object proposals along with
their objectness score
3. A RoI pooling layer is applied on
these proposals to bring down all the
proposals to the same size
4. Finally, the proposals are passed to
a fully connected layer to classify and
output the bounding boxes for objects. It
also returns the mask for each proposal
Mask R-CNN is the current state-of-the-art
for image segmentation and runs at 5 fps.

Summary of Image Segmentation

Techniques
I have summarized the different image
segmentation algorithms in the below
table.. I suggest keeping this handy next
time you’re working on an image
segmentation challenge or problem!
Algorithm Description Advantage Limitation
 s s
When
there is no
a. Simple significant
calculations grayscale
b. Fast difference
Separates operation
or an
the objects speed
overlap of
Region- into different
Based regions c. When the the
Segmentati based on object and grayscale
on some background pixel
threshold have high values, it
contrast, becomes
value(s).
this method very
performs difficult to
really well get
accurate
segments.
Edge Makes use It is good Not
Detection of for images suitable
Segmentati discontinuou having when there
on s local better are too
features of contrast many
 an image to edges in
detect the image
edges and and if there
between
hence is less
objects.
define a contrast
boundary of between
the object. objects.
Segmentati Divides the Works a.
on based pixels of the really well Computati
on image into on small on time is
Clustering homogeneo datasets too large
us clusters. and and
generates expensive.
excellent b. k-means
clusters. is a
distance-
based
algorithm.
It is not
suitable for
clustering
non-
convex
 clusters.
Gives three a. Simple,
outputs for flexible and
each object general
in the approach
image: its b. It is also High
Mask R-
class, the current training
CNN
bounding state-of- time
box the-art for
coordinates, image
and object segmentati
mask on

Image Segmentation using K-

means

i) Importing libraries and Images

Import matplotlib, numpy, OpenCV
along with the image to be
segmented.
import matplotlib as plt
import numpy as np
import cv2
path = '[Link]'
img = [Link](path)

ii) Preprocessing the Image

Preprocess the image by converting it
to the RGB color space. Reshape it
along the first axis to convert it into a
2D vector i.e. if the image is of the
shape (100,100,3) (width, height,
channels) then it will be converted to
(10000,3). Next, convert it into the
float datatype.
img =
[Link](img,cv2.COLOR_BGR2R
GB)
twoDimage = [Link]((-1,3))
twoDimage =
np.float32(twoDimage)

iii) Defining Parameters

Define the criteria by which the K-
means algorithm is supposed to
cluster pixels.
The ‘K’ variable defines the no of
clusters/groups that a pixel can
belong to (You can increase this value
to increase the degree of
segmentation).
criteria =
(cv2.TERM_CRITERIA_EPS +
cv2.TERM_CRITERIA_MAX_ITER, 10,
1.0)
K = 2
attempts=10

iv) Apply K-Means

The K variable randomly initiates K
different clusters and the ‘center’
variable defines the center of these
clusters. The distance of each point
from these centers is computed and
then they are assigned to one of the
clusters. Then they are divided into
different segments according to the
value of their ‘label variable’.
ret,label,center=[Link](twoD
image,K,None,criteria,attempts,c
v2.KMEANS_PP_CENTERS)
center = np.uint8(center)
res = center[[Link]()]
result_image =
[Link](([Link]))
Output:
K-Means Output
2. Image Segmentation using
Contour Detection

i) Importing libraries and Images

Import OpenCV, matplotlib, numpy
and load the image to memory.
import cv2
import [Link] as plt
import numpy as np
path = '[Link]'
img = [Link](path)
img = [Link](img,(256,256))

ii) Preprocessing the Image

1. Convert the image to grayscale.
2. Compute the threshold of the
grayscale image(the pixels above
 the threshold are converted to
white otherwise zero).
3. Apply canny edge detection to
the thresholded image before
finally using the ‘[Link]’
function to dilate edges detected.
 Also Read – Learn Image
Thresholding with OpenCV
 Also Read – OpenCV Tutorial –
Erosion and Dilation of Image
gray =
[Link](img,cv2.COLOR_RGB2G
RAY)
_,thresh = [Link](gray,
[Link](gray), 255,
cv2.THRESH_BINARY_INV)
edges =
[Link]([Link](thresh,0,25
5),None)
Output:
iii) Detecting and Drawing
Contours
1. Use the OpenCV find contour
function to find all the
open/closed regions in the image
and store (cnt). Use the -1
subscript since the function
returns a two-element tuple.
2. Pass them through the sorted
function to access the largest
contours first.
3. Create a zero pixel mask that
has equal shape and size to the
original image.
4. Draw the detected contours on
the created mask.
cnt =
sorted([Link](edges,
cv2.RETR_LIST,
cv2.CHAIN_APPROX_SIMPLE)[-2],
key=[Link])[-1]
mask = [Link]((256,256),
np.uint8)
masked = [Link](mask,
[cnt],-1, 255, -1)

iv) Segmenting the Regions

In order to show only the segmented
parts of the image, we perform a
bitwise AND operation on the original
image (img) and the mask (containing
the outlines of all our detected
contours).
Finally, Convert the image back to
RGB to see it segmented(while being
comparable to the original image).
dst = cv2.bitwise_and(img, img,
mask=mask)
segmented = [Link](dst,
cv2.COLOR_BGR2RGB)
Output:
Contour Detection
output

3. Image Segmentation using

Thresholding

i) Importing libraries and Images

Import numpy, scikit-image,
matplotlib, and OpenCV.
import numpy as np
import [Link] as plt
from [Link] import
threshold_otsu
import cv2
path ='[Link]'
img = [Link](path)

ii) Preprocessing the Image

Convert the image to the RBG color
space from BGR in order to finally
convert it to grayscale.
img_rgb=[Link](img,[Link]
OR_BGR2RGB)
img_gray=[Link](img_rgb,cv
2.COLOR_RGB2GRAY)

iii) Segmentation Process

Create a “filter_image” function that
multiplies the mask (created in the
previous section) with the RGB
channels of our image. Further, they
are concatenated to form a normal
image.
Next, we calculate the threshold
(thresh) for the gray image and use it
as a deciding factor i.e. values lying
below this threshold are selected and
others are discarded. This creates a
mask-like (img_otsu) image that can
later be used to segment our original
image.
Finally, apply the “filter_image”
function on the original image(img)
and the mask formed using
thresholding (img_otsu)
def filter_image(image, mask):
r = image[:,:,0] * mask
g = image[:,:,1] * mask
b = image[:,:,2] * mask
return [Link]([r,g,b])

thresh =
threshold_otsu(img_gray)
img_otsu = img_gray < thresh
filtered = filter_image(img,
img_otsu)
Output:

4. Segmentation using Color

Masking

i) Import libraries and Images

Import OpenCV and load the image to
memory.
import cv2
path ='[Link]'
img = [Link](path)

ii) Preprocessing the Image

OpenCV default colorspace is BGR so
we convert it to RGB. Next, we
convert it to the HSV colorspace.
rgb_img = [Link](img,
cv2.COLOR_BGR2RGB)
hsv_img = [Link](rgb_img,
cv2.COLOR_RGB2HSV)

iii) Define the Color Range to be

Detected
Define the RGB range for the color we
want to detect. Use the OpenCV in
range function to create a mask of all
the pixels that fall within the range
that we defined. It will later help to
mask these pixels.
light_blue = (90, 70, 50)
dark_blue = (128, 255, 255)
# You can use the following
values for green
# light_green = (40, 40, 40)
# dark_greek = (70, 255, 255)
mask = [Link](hsv_img,
light_blue, dark_blue)

iv) Apply the Mask

Use the bitwise AND operation to
apply our mask to the query image.
result = cv2.bitwise_and(img,
img, mask=mask)
Output:
Color masking (blue)