Lecture 1 Images and image filtering

Reading
• Szeliski, Chapter 3.1-3.3
What is an image?
What is each part of an image?
• Pixel -> picture element

‘138’

y
I(x,y)

x
Image as a 2D sampling of signal
• Signal: function depending on some variable with
physical meaning.

• Image: sampling of that function.

– 2 variables: xy coordinates
– 3 variables: xy + time (video)
– ‘Brightness’ is the value of the function for visible light

• Can be other physical values too: temperature,

pressure, depth …
Danny Alexander
Example 2D Images

Danny Alexander
Sampling in 1D

• Sampling in 1D takes a function, and returns a

vector whose elements are values of that
function at the sample points.

Danny Alexander
Sampling in 2D

• Sampling in 2D
takes a function
and returns a
matrix.
Danny Alexander
Grayscale Digital Image
• Pixel -> picture element

‘127’

y
I(x,y)

x
Grayscale Digital Image

Brightness
or intensity

x y

Danny Alexander
Image Histogram
Image Histogram
Quantization
Quantization Effects – Radiometric
Resolution

8 bit – 256 levels 4 bit – 16 levels 2 bit – 4 levels 1 bit – 2 levels

What is an image?

Digital Camera

We’ll focus on these in this class

(More on this process later) The Eye

Source: A. Efros
What is an image?
• A grid (matrix) of intensity values
255 255 255 255 255 255 255 255 255 255 255 255

255 255 255 255 255 255 255 255 255 255 255 255

255 255 255 20 0 255 255 255 255 255 255 255

255 255 255 75 75 75 255 255 255 255 255 255

=
255 255 75 95 95 75 255 255 255 255 255 255

255 255 96 127 145 175 255 255 255 255 255 255

255 255 127 145 175 175 175 255 255 255 255 255

255 255 127 145 200 200 175 175 95 255 255 255

255 255 127 145 200 200 175 175 95 47 255 255

255 255 127 145 145 175 127 127 95 47 255 255

255 255 74 127 127 127 95 95 95 47 255 255

255 255 255 74 74 74 74 74 74 255 255 255

255 255 255 255 255 255 255 255 255 255 255 255

255 255 255 255 255 255 255 255 255 255 255 255

(common to use one byte per value: 0 = black, 255 = white)

What is an image?
• Can think of a (grayscale) image as a function f from R2 to R:
– f (x,y) gives the intensity at position (x,y)

f (x, y)

– A digital image is a discrete (sampled, quantized) version of this function

Image transformations
• As with any function, we can apply operators to
an image

g (x,y) = f (x,y) + 20 g (x,y) = f (-x,y)

• Today we’ll talk about a special kind of operator,

convolution (linear filtering)
Filters
• Filtering
– Form a new image whose pixel values are a
combination of the original pixel values
• Why?
– To get useful information from images
• E.g., extract edges or contours (to understand shape)
– To enhance the image
• E.g., to remove noise
• E.g., to sharpen and “enhance image”
– A key operator in Convolutional Neural Networks
Canonical Image Processing problems
• Image Restoration
– denoising
– deblurring
• Image Compression
– JPEG, HEIF, MPEG, …
• Locating Structural Features
– corners
– edges
 Question: Noise reduction
• Given a camera and a still scene, how can you
reduce noise?

Take lots of images and average them!

What’s the next best thing?
Source: S. Seitz
Image filtering
• Modify the pixels in an image based on some
function of a local neighborhood of each pixel

10 5 3 Some function
4 5 1 7
1 1 7

Local image data Modified image data

Source: L. Zhang
Linear filtering
• One simple version of filtering: linear filtering (cross-
correlation, convolution)
– Replace each pixel by a linear combination (a weighted
sum) of its neighbors
• The prescription for the linear combination is called the
“kernel” (or “mask”, “filter”)
10 5 3 0 0 0
4 6 1 0 0. 0 8
1 1 8 0 5
1 0.5

Local image kernel Modified image data

[ [
data [ ]
Cross-correlation
Let be the image, be the kernel (of
size 2k+1 x 2k+1), and be the
output image

This is called a cross-correlation

operation:
[
[ ]
[
• Can think of as a “dot product” between]local
neighborhood and kernel for each pixel
]
2d correlation

h
Convolution
• Same as cross-correlation, except that the
kernel is “flipped” (horizontally and vertically)

This is called a convolution operation:

• Convolution is commutative and associative

2d convolution

f
Convolution

Adapted from F. Durand

Mean filtering

0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 10 20 30 30 30 20 10
0 0 0 90 90 90 90 90 0 0 0 20 40 60 60 60 40 20

=
0 0 0 90 90 90 90 90 0 0 0 30 60 90 90 90 60 30

*
1 1 1
1 1 1 0 0 0 90 90 90 90 90 0 0 0 30 50 80 80 90 60 30
0 0 0 90 0 90 90 90 0 0 0 30 50 80 80 90 60 30
1 1 1
0 0 0 90 90 90 90 90 0 0 0 20 30 50 50 60 40 20
0 0 0 0 0 0 0 0 0 0 10 20 30 30 30 30 20 10

0 0 90 0 0 0 0 0 0 0 10 10 10 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0
Mean filtering/Moving average
Mean filtering/Moving average
Mean filtering/Moving average
Mean filtering/Moving average
Mean filtering/Moving average
Mean filtering/Moving average
Linear filters: examples

0 0 0

* 0
0
1
0
0
0

Original

Source: D. Lowe
What image operation does filtering
with this kernel perform?
([0 0 0; 0 1 0; 0 0 0])

ⓘ Start presenting to display the poll results on this slide.

Linear filters: examples

0 0 0

=
* 0
0
1
0
0
0

Original Identical image

Source: D. Lowe
Linear filters: examples

0 0 0

* 1
0
0
0
0
0

Original

Source: D. Lowe
What image operation does filtering
with this kernel perform? ([0 0 0; 1 0
0; 0 0 0])

ⓘ Start presenting to display the poll results on this slide.

Linear filters: examples

0 0 0

=
* 1
0
0
0
0
0

Original Shifted left by 1 pixel

Source: D. Lowe
Linear filters: examples

1 1 1

=
* 1
1
1
1
1
1

Original Blur (with a mean

filter)

Source: D. Lowe
Linear filters: examples

-
0 0 0 1 1 1

* 0
0
2
0
0
0
1 1 1
1 1 1
=
Sharpening
Original
filter
(accentuates
edges)

Source: D. Lowe
Sharpening

Source: D. Lowe
Linear filters: examples

1 0 -1
2 0 -2
1 0 -1
Sobel

Vertical Edge
(absolute value)
David Lowe
Linear filters: examples

1 0 -1
2 0 -2
1 0 -1
Sobel

Vertical Edge
(absolute value)
David Lowe
Linear filters: examples

1 2 1
0 0 0
-1 -2 -1
Sobel

Horizontal Edge
(absolute value)
David Lowe
Smoothing with box filter revisited

Source: D. Forsyth
Gaussian kernel

Source: C. Rasmussen
Gaussian filters

= 1 pixel = 5 pixels = 10 pixels = 30 pixels

Mean vs. Gaussian
filtering
Gaussian filter
• Removes “high-frequency” components from
the image (low-pass filter)
• Convolution with self is another Gaussian

* =
– Convolving twice with Gaussian kernel of width
= convolving once with kernel of
width
Source: K. Grauman
Sharpening revisited
• What does blurring take away?

– =
original smoothed (5x5) detail
(This “detail extraction” operation
Let’s add it back: is also called a high-pass
filter)

+α =
original detail sharpened
Photo credit: [Link]
Sharpen filter
+
blurred
image unit impulse
image
(identity kernel
with single 1 in
center, zeros
elsewhere)

scaled impulse Gaussian Sharpen filter

Sharpen filter

unfiltered

filtered
“Optical” convolution
Camera
shake

= *
Source: Fergus, et al. “Removing Camera Shake from a Single Photograph”, SIGGRAPH 2006

Bokeh: Blur in out-of-focus regions of an image.

Source: [Link]
Filters: Thresholding
Linear filters
• Can thresholding be implemented with a linear
filter?
Can thresholding be
implemented with a linear
filter?

ⓘ Start presenting to display the poll results on this slide.

Questions?