Autumn 2025 Department of AI, IIT Kharagpur

AI61201: Visual Computing With AI/ML

Module 6: Feature Design for Image Analysis

Dr. Somdyuti Paul

1
Low, Mid and High Level Image Features
• An image feature is a distinctive attribute extracted from an image that can be used to represent, analyze, or
understand its content.

• Image features could be broadly categorized into three types:

• Low-level features: simple attributes directly extracted from raw pixel values that represent the lowest
level of abstraction, such as color histograms, edges, corners.
• Mid-level features: provide a higher level of abstraction of the image such as shape, texture and
keypoints.
• High level features: highly abstract and complex representations of an image that capture semantic and
contextual information, such as class labels, bounding boxes, segmentation masks.

• The low, mid and high level features share a hierarchical relationship, i.e. features at higher levels of
abstraction can be constructed using those extracted at lower levels of abstraction.

• Feature selection and/or extraction is often a preprocessing step in ML pipelines for image understanding
and analysis.

2
 Canny Edge Detector
• The Canny operator (Canny, 1986) is still one of the most widely used edge
detection operators in image processing and computer vision tasks.

• Edge detection using the Canny operator involves the following steps: -1 0 1 -1 -2 -1
-2 0 2 0 0 0
1. Noise Reduction : given an image, Gaussian smoothing is performed to reduce the
impact of noise on edge detection. -1 0 1 1 2 1
1 − x2 +2y2 hx(x, y) hy(x, y)
f′(x, y) = f(x, y) * g(x, y), where g(x, y) = e 2σ
2πσ 2
Sobel operators along
2. Gradient Computation: gradients are then computed to identify areas of signi cant horizontal and vertical
changes in intensity (i.e. edges) using gradient operators such as the Sobel kernels. directions
Gx(x, y) = f′(x, y) * hx(x, y), Gy(x, y) = f′(x, y) * hy(x, y)

Gy(x, y)
2 2
M(x, y) = Gx(x, y) + Gy(x, y) , Θ(x, y) = tan−1
Gx(x, y)
3



fi
 Canny Edge Detector
3. Non-maximum suppression: if a pixel is not the local maximum in the direction of the gradient, θ, then it is
set to 0. This step ensures that the edges are thin and only the strongest edges are preserved:

{0 otherwise
M(x, y) if M(x, y) = m where m = max Wθ(x, y)
M′(x, y) = , where Wθ(x, y) is the neighborhood in the

gradient direction θ.

4. Double Thresholding: all pixels are classi ed into strong, weak and non-edges based on two thresholds:
1 if M′(x, y) ≥ τH (Strong edges)
E(x, y) = 2 if τL ≤ M(x, y) < τH (Weak edges)
0 if M′(x, y) < τL (Non-edges)

5. Edge tracking by Hysteresis: pixels classi ed as strong edges are retained in the nal edge map, and the
weak edges are retained if and only if they are connected to strong edges, by considering the 8-connected

neighborhood around the pixel classi ed as an weak edge.

4

fi
fi
fi
fi
Canny Edge Detector

Original Image Smoothed Image Gradient Magnitude

Non-maximum Suppression Result 5 Thresholding and Edge Tracking Result

 Harris Corner Detector
• Corners are points in an image where the intensity changes signi cantly in multiple directions, making them
useful for various computer vision tasks like feature matching for object recognition, tracking, registration etc.

• Harris corner detector (Harris and Stephens, 1988) is a popular algorithm for detecting corner features based
on rst order derivatives of intensity.

• Corner detection with Harris detector involves the following steps:

1. Gradient Calculation: The intensity gradients Gx and Gy are computed in the horizontal and vertical
directions (for example, using the Sobel operator).

2. Structure Tensor: the second moments of the gradients are then computed as the structure tensor matrix:
2
∑u,v w(u, v)Gx (u + x, v + y) ∑u,v w(u, v)Gx(u + x, v + y)Gy(u + x, v + y)
A(x, y) = , where w(x, y)
∑u,v w(u, v)Gx(u + x, v + y)Gy(u + x, v + y) ∑u,v w(u, v)Gy2(u + x, v + y)
is an window function (such as Gaussian), that weighs the pixels in the neighborhood.

6
fi
fi
Harris Corner Detector
3. Corner Response function: The “cornerness” of each pixel is then quanti ed as follows:
2 2
R(x, y) = det(A(x, y)) − α ⋅ (Trace(A(x, y))) = λ1λ2 − α(λ1 + λ2)
where λ1 and λ2 are the eigenvalues of the structure tensor matrix A(x, y), and α is an empirically
determined constant that is typically set to a value in the range [0.04,0.06]

4. Thresholding and Non-maximum Suppression: the corner response is thresholded to identify potential
corner points. NMS over a local window is often applied to keep the local maxima.

The values of corner response R can be interpreted as follows:

• Flat regions: λ1 and λ2 are small, R is positive and small.
• Edges: λ1 > > λ2 or vice versa, R < 0
• Corners: λ1 and λ2 are large, and λ1 ∼ λ2, R is large.

fi
Harris Corner Detector

8
 Keypoint Detection
• Keypoints are distinct and repeatable points of interest in an
image that can be used to identify and match features between
di erent images or between di erent views of the same scene.

• Keypoints are located at positions in the image where the

surrounding features are unique or distinctive.

• Keypoints are typically chosen to be robust to changes in

scaling, rotation and illumination.

• Keypoints play a key role in providing features for visual search

and object matching.

Example of Keypoints: the 68 point facial landmarks

9
ff
ff
 Scale Invariant Feature Transform (SIFT)
• Scale Invariant Feature Transform (SIFT) (Lowe, 2004) is a powerful local feature (keypoint) detection operator
that is based on scale-space representation of an image using di erence of Gaussians (DoGs).

• The steps involved in detecting key points using SIFT are as follows:

1. Scale Space Extrema Detection

a. The image is Gaussian blurred at di erent scales (σ values) and resolutions.
2 2
1 x +y
Lσ(x, y) = Gσ(x, y) * I(x, y), where Gσ(x, y) = exp(− )
2πσ 2 2σ 2
b. Di erence of Gaussians (DoG): the di erence of Gaussians at successive values of σ is computed as
follows:
Dσ(x, y) = Lσ(x, y) − Lkσ(x, y), where k = 2 (typically).

10
ff
ff
ff
ff
 Scale Invariant Feature Transform (SIFT)
Gσ
− Gσ
−
Gkσ Gkσ
− −
Gk2σ Gk2σ
−
IM×N I M2 × N2 − ⋯
⋮ ⋮
− −
GkN−1σ
− GkN−1σ
−
GkN σ GkN σ
First octave
Next octave
11
Scale Invariant Feature Transform (SIFT)
2. Extrema Detection and Re nement

a. Extrema Detection: the local maxima and minima of the DoG

images in both spatial and scale dimensions are identi ed by
comparing each pixel of the DoG responses with its 26 neighbors
in scale space.
Adjacent
b. Accurate Keypoint Localization: the keypoint localization is Scales
improved by tting a 3D quadratic function around the detected
extremum.

c. Extrema Re nement: the detected DoG extremes are ltered to DoG Response
eliminate unreliable candidates for keypoints by checking the
following: Extrema detection in scale-space.
• Whether the extrema has high contrast
• Whether the extrema is not located on or near an edge.
The extrema that do not meet these conditions are rejected.
12
fi
fi
fi
fi
fi
 Scale Invariant Feature Transform (SIFT)
3. Orientation Histogram
a. The gradient of the 16 × 16 local patch around the keypoint is computed at the location and scale of each
keypoint:
2 2

( ∂x ) ( ∂y ) ( ∂y / ∂x )
∂IG(x, y) ∂IG(x, y) ∂IG(x, y) ∂IG(x, y)
m(x, y) = + , θ(x, y) = tan−1

b. To achieve rotation invariance, the orientations of the local patch are computed relative to the keypoint’s
orientation as θ′(x, y) = (θ(x, y) − θ(x0, y0)) mod 2π

c. Using the orientations θ′(x, y), a 8-bin gradient

orientation histogram is constructed for each
4 × 4 quadrant.
d. The contribution of each gradient orientation to
its corresponding bin is the Gaussian weighted
magnitude of the gradient.
Local patch 4 × 4 quadrants
13


Scale Invariant Feature Transform (SIFT)
4. SIFT Descriptor

a. The histograms from all quadrants are concatenated to form a 128 dimensional feature descriptor for each
keypoint.
b. The SIFT feature descriptor is normalized to a unit-length vector to achieve contrast invariance.

14
Scale Invariant Feature Transform (SIFT)
• SIFT features can be used to establish correspondence between images that have di erent scales, rotations
and other transforms.
• The concept of matching keypoints is employed in image stitching, image registration, object recognition and
object tracking, 3D reconstruction etc.

15

ff
Histogram of Oriented Gradients
• The Histogram of Oriented Gradients (HoG) is a widely used feature descriptor for object detection.
• It captures the distribution of gradient orientations in localized portions of an image, providing a robust representation of
the object's shape and appearance.
• The steps to compute the HoG descriptors are outlined as follows:
Block
1. Gradient Computation: the horizontal and vertical gradients are Image
computed and magnitude and orientation of the gradient vectors are
initially computed.
Histogram
2. Cell histogram Computation: the pixels are divided into non-
overlapping cells, and the gradient orientation histograms are
computed for each cell. The vote of each pixel towards the gradient
histogram is weighted by its gradient magnitude.
3. Histogram Normalization: the cells are grouped into overlapping
Cell
blocks, and the histograms from all cells of a block are concatenated
and normalized to achieve contrast invariance. The nal HoG
descriptor are the normalized histograms from all blocks in an image.

16
fi
 Histogram of Oriented Gradients

• The nal HoG descriptor provides a compact representation of the gradient information in the image.
• Cell histograms capture local gradient details, which are important for distinguishing ne textures and
shapes.
• Normalization over blocks ensure that the feature descriptors are invariant to changes in contrast.
• The HoG descriptor e ectively balances capturing details and computational e ciency.

17
fi
ff
ffi
fi
Pedestrian Detection Using HoG
• HoG descriptors were shown to be particularly e ective for pedestrian
detection (Dalal and Triggs, 2005).
• The HoG descriptors from overlapping blocks were used to train a
linear support vector machine classi er for pedestrian detection.

18
fi
ff
 Hough Transform
• The Hough transform (Hough, 1962) is an image analysis tool used to detect curves of speci c shapes, such
as lines, circles, ellipses etc. from an image.

• The Hough transform operates on an edge map and relies on letting edges vote for plausible locations of
speci c shapes.

• The Hough transform is robust to the presence of noise, and can connect “broken” edges in the edge map to
detect the desired shape.

• The Hough transform can be generalized to any geometric shape.

• The Hough transform is computation and memory intensive.

19
fi
fi
Hough Transform
Hough Transform for Line Detection

1. Line parameterization: A line is parameterized as follows:

xcos(θ) + ysin(θ) − r = 0, where r is the distance of the origin from the line, and θ
θ is the angle of the line normal with the x-axis. r
(x, y)
Thus, each (r, θ) pair represents an unique line in the image space.

2. Voting or Accumulation:
Line representation
• A discrete 2D array called the accumulator is constructed by quantizing the
parameter space into discrete bins.
• For each edge pixel (x, y) in the image space, r is computed for the range of
values of θ.
• For each pair (r, θ) thus computed, the corresponding entry in the accumulator
array is incremented.
20
Hough Transform
Hough Transform for Line Detection

3. Line Detection: the accumulator array is appropriately thresholded to nd peaks which correspond to the
detected lines, i.e. if a peak occurs at (r, θ), there is a line at a distance of r from the origin, whose normal
makes an angle θ with the x-axis.

r
θ (degrees)
Input image Edge map Accumulator Visualization Detected Lines
21

fi
Hough Transform
Hough Transform for Circle Detection

1. Circle parameterization: A circle is represented by 3 parameters, the centre coordinates (x0, y0), and its
radius r.
2 2 2
(x − x0) + (y − y0) = r

2. Voting or Accumulation:
• In the case of circle, the accumulator array is 3-dimensional, owing to the three parameters.
• For any edge point (x, y) and each possible radius r, and centre coordinates (x0,y0) are computed as
x0 = x − rcos(θ) and y0 = y − r sin(θ), by varying the angle θ between the edge point and the circle
centre between 0 to 2π.

3. The peaks in the accumulator array correspond to the detected circles.

22
Hough Transform Results

Line Detection

Circle detection
23
 Local Binary Pattern
• Local Binary Patterns (LBP) is a feature extraction technique that is designed to
capture local texture information by comparing each pixel to its neighbors.
• Each pixel is assigned a binary code by comparing its intensity to its neighboring 1 4 16 0 0 1
pixels.
53 9 7 1 0
• The computation of LBP involves the following steps:
5 105 37 0 1 1
1. De ning the neighborhood: the neighborhood around each pixel in the image is
de ned. For example the 8 neighbors in a 3 × 3 window. 8 bit LBP code: 00101101
Decimal code: 45
2. Pixel comparisons: each pixel is compared with its neighborhood pixels. If its
intensity is greater than the neighborhood pixel, the corresponding comparison LBP computation on an example
outcome is assigned a value of 1, else 0. neighborhood
3. LBP code: the binary LBP code for each pixel is obtained by grouping the above
comparison outcomes in the local neighborhood in a particular order, usually
clockwise.
4. For a pixel having an LBP code (b7, b6, ⋯b0), where bi ∈ {0,1}, the LBP value
7
bi ⋅ 2i
∑
is
i=0 24
fi
fi
Local Binary Pattern
Texture LBP Image LBP Histogram LBP of
Test patch
test patch
• The histogram
computed from the
LBP could be used for
the purpose of texture
matching.
• A new texture could be
matched by comparing
its LBP histogram with
the histogram of the
textures present in the
database (for example,
using K-Nearest
Neighbors). LBP histogram of
test patch

25
 Gray Level Co-occurence Matrix
• The Gray Level Co-occurence Matrix (GLCM) o ers a powerful technique for characterizing the spatial
relationships between pixel intensities.

• GLCM represents how often two pixel intensities occur together in a pre-de ned spatial relationship.
• The spatial arrangement of pixel pairs is de ned by the following:
• Pixel o set: distance between two pixels
• Direction: the direction in which the pixel pairs are considered

• For each such spatial arrangement of the pixel pairs of an image having N gray levels, the GLCM is a N × N
matrix.

• The element P(i, j) of the GLCM matrix P represents the number of times gray level i occurs in the speci ed
spatial relationship with gray level j.

• The GLCM can be normalized to sum to 1 to get the likelihood distribution of pixel cooccurrences.

26
ff
fi
ff
fi
fi
 Gray Level Co-occurence Matrix
• The following features are commonly extracted from each
GLCM to capture texture information:
(i − j)2 ⋅ P(i, j)
∑
Contrast:
•
i,j
2
∑
Energy: P(i, j)
•
i,j
P(i, j)
∑ 1 + |i − j|
Homogeneity:
•
i,j
∑i,j (i − μx)( j − μy) ⋅ P(i, j)
Correlation: , where
• σxσy
∑ ∑ ∑ ∑
μx = i⋅ P(i, j), μy = j⋅ P(i, j) and
i j j i
σx2 = (i − μx)2 ⋅ σy2 = ( j − μy)2 ⋅
∑ ∑ ∑ ∑
P(i, j), P(i, j)
i j j i

∑
Entropy: − P(i, j)log(P(i, j))
•
i,j
Gray Level Co-occurence Matrix
Texture GLCM feature

Test patch

GLCM feature

GLCM features computed using 8 spatial relationships (2 pixel o sets and 4 directions)
ff