IMAGE PRE-PROCESSING
Updated from Dr Nidhal Abdulaziz
slides
References:
“Introduction to Robotics: Analysis, Control, Applications” by
Saeed B. Niku, John Wiley and Sons, 2011.
“Digital Image Processing”, 3rd ed., Gonzalez and Woods,
Pearson, 2010.
Forsyth & Ponce, Computer Vision: A Modern Approach.
Richard Szeliski, Computer Vision: Algorithms and Applications
(available online).
“Image Processing, Analysis and Machine Vision”, Sonka, Hlavac,
and Boyle, Cengage Learning, 2015.
Content
Definition of Image pre-processing
Categories of Image pre-processing:
Pixel Brightness Transformations
Position Dependent Brightness Corrections
Gray-scale Transformations
Thresholding
Neighbourhood Operations
Pixel Coordinate Transformation
3
Introduction
The pre-process steps aims to prepare the image data for
the specific image processing or analysis task.
It aims to suppressing undesired distortions on the input
image, enhances some of its features for further
processing, or make transformation
Geometric transformations of images (e.g., rotation,
scaling, translation) are considered as an example
Image pre-processing: Input and output are intensity images
4
Introduction
In the following, we’ll study three categories of the image pre-
processing:
Pixel Brightness Transformation: each pixel’s value is
modified independently of others.
Neighbourhood Operations: the new value of a pixel
depends on its neighboring pixels.
Pixel Coordinate Transformation:the spatial positions of
pixels are changed rather than their intensity values, e.g.
rotation
5
I. Pixel Brightness Transformations
They modify pixel brightness; the transformation depends on
the properties of the pixel itself
Two classes of pixel brightness transformations:
Position Dependent Brightness Corrections:
Modifies the brightness of the pixel, taking into account its original
brightness and its position in image
Gray-scale Transformations:
Change brightness without regard to position in image
6
I.1 Position Dependent Brightness Corrections
Ideally, the sensitivity of image acquisition and digitization
devices should NOT depend on position in the image, but this
assumption is not valid in many practical cases:
• The sensitivity of the lenses is not uniform across all the scene
capture
• Uneven object illumination is also a source of degradation.
• The degradation can be suppressed using a multiplicative
error coefficient e(i,j) obtained from a reference image g(i,j)
with known brightness, e.g. constant, after being captured.
I.2 Gray-scale Transformation
Does not depend on position of pixel in image
Gray-scale transformations used mainly when image is viewed
by human observer and contrast enhancement may be beneficial
Transformation T of original brightness p from scale [p0, pk] into
brightness q in a new scale [q0, qk] is given by q=T(p)
• The piecewise linear function a enhances the
image contrast between brightness values p1
and p2.
• The Function b is called brightness
thresholding and results in a black-and-white
image;
• The straight line c denotes the negative
transformation.
42
Other Functions
• Image with intensity levels in the
range [0, L-1]
• The general form of the log
transformation is s=c*log(1+r).
• It maps a narrow range of low
intensity values in the input into a
wider range of output values.
• The opposite is true to higher values
of input levels
• See also power-law (Gamma)
Some basic intensity transformation
transformation functions. All
curves were scaled to fit in the
range shown
Histogram Equalization
To enhance the contrast of an image, Histogram
Equalisation algorithm can be used.
It aims to create an image with equally distributed
brightness levels over the whole brightness scale
accomplishes this by effectively spreading out the most
frequent intensity values.
Example
X-ray CT image of a lung
11
Algorithm
Cumulative
histogram
Approximated
by a sum in
digital images
Examples (1)
Expands the histogram to include larger portion of the
available range of gray levels.
Examples (2)
Original image
brightened
when an image is brightened, its
histogram distribution simply
shifts, in this case by 30 points.
When the contrast of the image
is increased the distribution of
more contrast
pixel gray levels is expanded
Figure 11.16 Increasing the contrast in an image expands the histogram to
include new
Examples(3)
Thresholding
Assigning pixels to foreground and background or other
regions based on whether below a threshold or above it
Original image Thresholded at 100 Thresholded at 150
Thresholding
Gray-level Thresholding
Computationally inexpensive and fast
Oldest segmentation method
Transformation of input image f to output binary image g
g(i,j)=1 for f(i,j)>T
g(i,j)=0 for f(i,j)≤T
Where T is threshold, g(i,j)=1 for image elements of objects, and
g(i,j)=0 for image elements of background (or vice versa)
…Continued
If objects do not touch each other and if foreground gray-
levels are clearly distinct from background gray-levels,
thresholding is a suitable segmentation method
Correct threshold selection is crucial for successful
segmentation
Selection can be determined interactively
Can be result of some threshold detection method
Examples
19
Adaptive Thresholding (1)
Global threshold is determined from the whole image f
T=T(f)
Adaptive thresholding:
use variable thresholds
Threshold values vary over the image as a function of local
image characteristics
Local thresholds are position dependent
T=T(f, fc)
Where fc is image part in which threshold is determined
20
Adaptive Thresholding (2)
Basic thresholding has many modifications
One possibility is to segment an image into regions of pixels
with gray-levels from a set D (band) and into background
otherwise
g(i,j)=1 for f(i,j) in D
=0 otherwise
• Approach can work as a border
detector:
Some gray-levels between the dark
objects and Lighter background will
only be found in the object borders.
21
Optimal Thresholding
Optimal thresholding
Alternative approach that seeks to model histogram of image
using weighted sum of two or more probability densities
with normal distribution
Threshold is set as gray-level closest to minimum
probability between maxima of normal distributions
Results in minimum error segmentation
Difficulty with these methods is in estimating normal
distribution parameters together with uncertainty that
distribution may be considered normal,see Otsu algorithm22
2. Neighbourhood Operation (Local Pre-processing)
They use a small neighborhood of a pixel in an input image
to get a new brightness value in the output image.
They are also called filtration
First Classification: methods are divided into two groups
according to the goal of the processing
Smoothing
Aims to suppress noise or other small fluctuations in image
Equivalent to the suppression of high frequencies in the
Fourier transform domain
• Unfortunately, smoothing also blurs all sharp edges that
bear important information about the image
Sharpening:
Based on local derivatives of image function
Other classifications may depend on the linearity of operation or the
domain (intensity of frequency) 23
Linear Transformation
Linear transformation
Linear operations calculate the resulting value in the output
image pixel f(i, j) as a linear combination of brightnesses in a
local neighborhood O of the pixel g(i,j) in the input image
Contribution of pixels in neighborhood O weighted by
coefficients h
f (i, j) h(i m, j n)g(m, n)
(m,n)O
This is equivalent to discrete convolution with kernel h
which is called a convolution mask
24
Defining convolution
• Let g be the image and h be the kernel. The output of
convolving g with h is denoted h * g.
( h g)[i, j] h[i m, j n] g[i,j]
kernel is “flipped” m,n
Convention:
• MATLAB functions: conv2, filter2, imfilter Source: F. Durand
…Continued
• Make use of a kernel or convolution mask {m1, m2,,,}
• Operates on neighbourhood which have similar properties to the
pixel being processed.
• Has many applications such as filters, edge finders, and
morphology.
• The principle operation is (S is normalizing value):
R ( A m1 B m2 C m3 E m4 F m5 G m6 I m7 J m8 K m9 ) S
Image Smoothing
Uses redundancy in image neighbourhood to suppress noise
The average is computed only from points in the
neighbourhood
It acts as a low pass filter and introduce new gray values
Can effectively eliminate impulse noise or degradation
appearing as thin stripes
Does not work if degradations are large blobs or thick stripes
Poses the problems of blurring sharp edges
Need to consider methods which are edge preserving
27
Moving Average
• Let’s replace each pixel with a weighted average of its
neighborhood
• The weights are called the filter kernel
• What are the weights for the average of a 3x3
neighborhood?
1 1 1
1 1 1
1 1 1
“box filter”
Source: D. Lowe
Averaging is a special case of discrete convolution
For a 3 x 3 neighborhood, the convolution mask h is
1 1 1
1
[1 1 1]
9
1 1 1
The significance of the pixel in the center of the
convolution mask h or its 4-neighbors is sometimes
increased
Two commonly used smoothing filters: Gaussian and
Butterworth
1 1 1 1 2 1
1
1
h [1 2 1] h [2 4 2]
10 16
1 1 1 1 2 1
66
Other Averaging Convolution Masks
Practice with Linear Filters
0 0 0
0
0
1
0
0
0
?
Original
Source: D. Lowe
Practice with Linear Filters
0 0 0
0 1 0
0 0 0
Original Filtered
(no change)
Source: D. Lowe
Practice with Linear Filters
0 0 0
0
0
0
0
1
0
?
Original
Source: D. Lowe
Practice with Linear Filters
0 0 0
0 0 1
0 0 0
Original Shifted left
By 1 pixel
Source: D. Lowe
Practice with Linear Filters
1 1 1
1 1 1 ?
1 1 1
Original
Source: D. Lowe
Practice with Linear Filters
1 1 1
1 1 1
1 1 1
Original Blur (with a
box filter)
Source: D. Lowe
Practice with Linear Filters
0 0 0 1 1 1
0
0
2
0
0
0
- 1
1
1
1
1
1
?
(Note that filter sums to 1)
Original
Source: D. Lowe
Practice with Linear Filters
0 0 0 1 1 1
0
0
2
0
0
0
- 1
1
1
1
1
1
Original Sharpening filter
- Accentuates differences with local
average
Source: D. Lowe
Neighborhood Averaging with Convolution Masks
1. Original image
2. Corrupted with noise
3. 3x3 neighborhood averaging
4. 5x5 neighborhood averaging
Prague Castle Averaging
75
Examples of Averaging
Sharpening
Source: D. Lowe
Sharpening
What does blurring take away?
– =
original smoothed (5x5) detail
Let’s add it back:
+ =
original detail sharpened
Edge Detectors
Edges are pixels where function changes abruptly.
Edge detectors are a collection of very important local
image pre-processing methods used to locate changes in
intensity function
They are important for image perception and robust to
changes of illumination and viewpoint
They often used in image analysis for finding region
boundaries 88
Edge Detection output
Usually results in a line drawing of the image.
The lines represent changes in planes, textures, lines, and
colors, etc..
Intuitively, most semantic and shape
information from the image can be
encoded in the edges
More compact, it can reduce unnecessary
information
…Continued
Edge is a vector variable with two components:
Magnitude: Magnitude of the gradient
Direction: Gradient direction gives direction of maximum growth of
function
Edges are caused by a variety of factors:
surface normal discontinuity
depth discontinuity
surface colordiscontinuity
illuminationdiscontinuity
Source: Steve Seitz
Edge Identification
• An edge is a place of rapid change in the image intensity
function
intensity function
image first derivative
(along horizontal scanline)
edges correspond to
extrema of derivative
Derivatives with convolution
For 2D function f(x,y), the partial derivative is:
f (x, y) f (x , y) f (x, y)
lim
x 0
Fordiscrete data, we can approximate using finite differences:
f (x, y) f (x 1, y) f (x, y)
x 1
To implementabove as convolution, what would be the associated
filter?
Source: K. Grauman
Derivatives with convolution
For 2D function f(x,y), the partial derivative is:
f (x, y) f (x , y) f (x, y)
lim
x 0
Fordiscrete data, we can approximate using finite differences:
f (x, y) f (x 1, y) f (x, y)
x 1
Source: K. Grauman
Partial Derivatives of an Image
f (x, y) f (x, y)
x y
-1 or
1
-1 1 1 -1
Which shows changes with respect to x?
Finite Difference Filters
Other approximations of derivative filters exist:
Operators which are able to detect edge direction are represented by a collection of
masks
Source: K. Grauman
Roberts Operator
One of the oldest
Very easy to compute as it uses only a 2 x 2 neighborhood of
current pixels
Its masks are
The magnitude of the edge is computed as
5
2
Prewitt Operator
Approximates the first derivative
The gradient is estimated in eight possible directions
Convolution result of greatest magnitude indicates the gradient
direction
In practice we use the x and y kernels
53
5
4
Sobel operator
Often used as a simple detector of horizontality and
verticality, in which case only the first two of three masks
are used
If the first mask response is y and the second mask response
is x, the edge strength is |x|+|y| and direction is arctan(y/x)
5
5
Edge Detection: Continued.
Except in binary images, edges are generally not ideal.
First and second derivatives may be used
I I
I ,
x y
f(x)
255
0 x
Ideal edge
f(x)
x
Real edge
I I
2 2
255
(I ) magnitude f '(x)
x y
x
0
Edge(min peak)
f ''(x)
2
I 2
I
I 2 , 2
x
2
x y Edge(zero crossing)
The gradient magnitude |grad g(x,y)| and gradient
direction are continuous image functions calculated as
g 2 g 2
| gradg(x, y) | ( ) ( )
x y
arg( g , g )
x y
Where arg(x,y) is the angle from the x axis to (x,y)
57
•Sometimes we are interested only in edge magnitudes without regard
to their orientations.
•The Laplacian may be used.
•The Laplacian has the same properties in all directions and is
therefore invariant to rotation in the image.
2 g(x, y) 2 g(x, y)
g(x, y)
2
x 2
y2
58
Sharpness
Image sharpening has the objective of making edges
steeper – the sharpened image is intended to be observed
by a human
The sharpened output image f is obtained from the input
image g as
f(i,j)=g(i,j) + C *S(i,j)
Where C is a positive coefficient which gives the strength of
sharpening and S(i,j) is a measure of image function
sheerness, calculated using a gradient operator
59
Gradient operators as a measure of edge sheerness can be
divided into three categories:
Operators approximating derivatives of the image
function using differences.
Operators based on zero-crossings of the image function
second derivative
Operators which attempt to match an image function to
parametric model of edges
Individual gradient operators that examine small local
neighborhoods are convolutions and can be expressed by
convolution masks as shown before and next.. 60
Laplace Operator
Approximating the second derivative which gives the edge
magnitude only
Approximated in digital images by a convolution sum
Laplacian operator responds doubly to some edges in image:
Since the Laplace operator detects rapid changes in
intensity, it can sometimes highlight the transition on
both sides of an edge, leading to a double response.
3 x 3 mask h is often used
6
1
Edge Detection : Cont.
Laplacian2 is Laplacian of Gaussian (LoG).
It first applies a Gaussian smoothing filter to
a) Original image suppress noise before Laplacian to detect edges.
b) Laplacian 1 Sobel might detect more noise as edges in
c) Laplacian 2 images with significant noise.
d) Sobel operator
Laplacian provides better localization of edges
e) Robert’s edge since it is based on double derivatives
Other Effects with High-Pass Filters
Non-Linear Transform: Median Filtering
The median of a finite list of real numbers is simply found
by ordering the list and selecting the middle member.
Lists are often constructed to be odd in length to secure
uniqueness.
Median filtering is a non-linear smoothing method that
reduces the blurring of edges
The median in the neighborhood is not affected by
individual noise spikes and so median smoothing
eliminates impulse noise quite well
64
Examples
(a) Original image, (b)Corrupted by noise,
(c) 3x3 median filter, (d) 7x7 median filter
Median Filter Implementation
This algorithm exploits the fact that
as the window moves across a row
by one column, the only change to its
contents is to lose the leftmost
column and replace it with a new
right column—for a median window
of m rows and n columns, mn−2m
pixels are unchanged and do not
need re-sorting
66
…Continued
Main disadvantage of median filtering in a rectangular
neighborhood is its damaging of thin lines and sharp corners
This can be avoided if another shape of neighborhood is used.
For instance, if horizontal/vertical lines need preserving, a
neighborhood such as that in the below can be used
67
I.3. Pixel Coordinate Transformation
Transform is linear with respect to the coefficients
If pairs of corresponding points in both images are known,
coefficients may be determined using a set of linear equations
Bilinear transform: the mapping of the pixel coordinate can be
approximated by a bilinear transform for which four pairs of
corresponding points are sufficient to find the transformation
coefficients
Affine transformation: simpler version of Bilinear Transform
where three pairs of corresponding points are sufficient to find
coefficients
Includes typical geometric transformations such as rotation,
translation, scaling, and skewing
68