Preliminaries Digital Image Processing DIP vs.

CV Computer Vision State-of-the-art in CV Challenges in CV

01 Introduction
CS-867 Computer Vision
Enrollment Code: 129564830

Dr. Mohsin Kamal

Associate Professor
[Link]@[Link]

School of Electrical Engineering and Computer Science (SEECS)

National University of Sciences and Technology (NUST), Pakistan

1
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

1 Preliminaries

2 Digital Image Processing

3 DIP vs. CV

4 Computer Vision

5 State-of-the-art in CV

6 Challenges in CV

2
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

1 Preliminaries

2 Digital Image Processing

3 DIP vs. CV

4 Computer Vision

5 State-of-the-art in CV

6 Challenges in CV

3
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

"One picture is worth more than ten thousand words."

Arthur Brisbane (1911)

Figure : TESLA’s self driving car.

4
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

Prerequisites

This course is appropriate as a first course for UG/PG students

with a EE/CE/CS background, which should have prepared them
with these essential prerequisites:
Good working knowledge of MATLAB or OpenCV with
Python or C/C++ (or willingness and time to learn quickly),
data structures
Linear algebra, calculus and probability
While a background in image processing would certainly be
beneficial.

5
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

Textbook

[Link]
6
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

Reference book

Multiple View Geometry in Computer Vision by Richard Hartley

and Andrew Zisserman (latest edition) 7
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

Grading policy

Quiz 10%
Assignments 10%
Research Project 10%
Mid Term Exam 25%
Final Exam 45%
These weightages may vary as the semester progresses

8
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

1 Preliminaries

2 Digital Image Processing

3 DIP vs. CV

4 Computer Vision

5 State-of-the-art in CV

6 Challenges in CV

9
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

Image Formation: Reflected Light

The colors that we

perceive are determined
by the nature of the light
reflected from an object.
For example, if white light
is shone onto a green
object most wavelengths
are absorbed, while green
light is reflected from the
object

10
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

What is a digital image?

An image is a 2D projection of the 3D scene.

Mathematically, an image is a 2D function f (x, y ), where x
and y are spatial (plane) coordinates, and f (x, y ) is the
amplitude of the function at (x, y ) called intensity or gray
level.
A digital image is a f (x, y ), where x, y and f (x, y ) are all
finite and discrete quantities. An image point at (x, y ) is
called a picture element, image element, pel or pixel.

11
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

What is a digital image?

A digital image is a representation of a two-dimensional image
as a finite set of digital values, called picture elements or pixels.

12
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

What is a digital image?

Common image formats include:

1 sample per point (B&W or Grayscale)
3 samples per point (Red, Green, and Blue)
4 samples per point (Red, Green, Blue, and "Alpha", a.k.a.
Opacity or Transparency)

13
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

What is a digital image?

- Pixel values typically represent gray levels, colors, heights,

opacities etc.
- Remember digitization implies that a digital image is an
approximation of a real scene

14
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

What is Image Processing?

Image processing is a method to perform some operations on an

image, in order to get an enhanced image or to extract some
useful information from it.

15
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

What is Digital Image Processing?

Digital image processing is the transformation of a digital

image by means of a digital computer.
Image processing starts with an image and (usually)
produces a modified version of that image - e.g.,
enhancement, restoration, compression, i.e., the output is
also an image.
Machine vision / computer vision aims to understand the
content of images at a higher level so as to take decisions
based on visual input.

16
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

Image Acquisition

Figure : An example of the digital image acquisition process. (a) Energy

("illumination") source. (b) An element of a scene. (c) Imaging system. (d) Projection of
the scene onto the image plane. (e) Digitized image. 17
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

Sampling and Quantization

Figure : Generation of a digital image. (a) Continuous image. (b) A scan line from A to B in the continuous
image, used to illustrate the concepts of sampling and quantization. (c) Sampling and quantization. (d) Digital scan
line. 18
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

Result of Sampling and Quantization

A digital image is an approximation of a real world scene.

19
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

Digital Image Representation

Number of bits required to store an image are:

b =M ×N ×k
k
Image having 2 intensity levels
k = bit depth (bits per pixel) of an image.
For a grey scale image, there are 256 intensity levels
(0-255), hence, k = 8.
0 represents black and 255 the white.

20
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

Coordinate Convention

Standard convention in programming is to place the image origin

at top, left corner
21
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

1 Preliminaries

2 Digital Image Processing

3 DIP vs. CV

4 Computer Vision

5 State-of-the-art in CV

6 Challenges in CV

22
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

From Image Processing to Computer Vision

Low, mid and high-level processes

23
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

From Image Processing to Computer Vision

Figure : Example of Low Level Processing

24
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

From Image Processing to Computer Vision

Low, mid and high-level processes

25
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

From Image Processing to Computer Vision

Figure : Example of Mid Level Processing - Segmentation2 of an

image into regions
2
Image segmentation is a commonly used technique in digital image processing and analysis to partition an
image into multiple parts or regions, often based on the characteristics of the pixels in the image. 26
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

From Image Processing to Computer Vision

Low, mid and high-level processes

27
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

From Image Processing to Computer Vision

Figure : Example of High Level Processing

28
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

From Image Processing to Computer Vision

29
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

1 Preliminaries

2 Digital Image Processing

3 DIP vs. CV

4 Computer Vision

5 State-of-the-art in CV

6 Challenges in CV

30
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

What is vision?

31
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

What is (computer) vision?

32
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

What is (computer) vision?

33
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

What is computer vision?

The ability of computers to see.

Extract descriptions of the world from pictures or sequences
of pictures (videos) [Forsyth & Ponce, 2003].
"Vision is the process of discovering from images what is
present in the world, and where it is" (David Marr).
Computer vision is the study of the analysis of pictures and
videos in order to achieve results similar to those done by
human.

34
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

Computer Vision

Make computers
understand images and
videos.
What kind of scene?
Where are the cars?
How far is the building?

35
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

Vision for perception, interpretation

36
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

The goal of computer vision

To bridge the gap between pixels and "meaning"

What we see What a computer sees

37
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

Focus of computer vision

Computer Vision focuses on the following problems:

What information should be extracted from the visual
sensors?
How is this information extracted?
How should it be represented?
How the information be used to allow a robotics system to
perform its tasks?

38
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

Overview of classical computer vision

39
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

Why vision?
Images are everywhere

40
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

Why computer vision matters?

41
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

A story about computer vision

In 1966, Marvin Minsky3 at MIT asked his undergraduate

student Gerald Jay Sussman to "spend the summer linking a
camera to a computer and getting the computer to
describe4 what it saw". We now know that the problem is slightly
more difficult than that. (Szeliski 2009, Computer Vision)

3
Founder, MIT AI project
4
Image understanding 42
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

A story about computer vision

In 1966, Marvin Minsky at MIT asked his undergraduate student

Gerald Jay Sussman to "spend the summer linking a camera to
a computer and getting the computer to describe what it saw".
We now know that the problem is slightly more difficult than that.
"Needless to say, Sussman didn’t make the deadline,". "Vision
turned out to be one of the most difficult and frustrating
challenges in AI over the next four decades. As machine vision
expert Berthold Horn once pointed out, Sussman opted never to
work in vision again."

43
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

Ridiculously brief history of computer vision

1966: Minsky assigns computer vision as an

undergrad summer project
1960’s: interpretation of synthetic worlds
1970’s: some progress on interpreting selected
images
1980’s: ANNs come and go; shift toward
geometry and increased mathematical rigor
1990’s: face recognition; statistical analysis in
vogue
2000’s: broader recognition; large annotated
datasets available; video processing starts; vision
& graphis; vision for HCI; internet vision, etc.

44
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

1 Preliminaries

2 Digital Image Processing

3 DIP vs. CV

4 Computer Vision

5 State-of-the-art in CV

6 Challenges in CV

45
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

Optical character recognition (OCR)

-Technology to convert scanned docs to text

-If you have a scanner, it probably came with OCR software

License plate readers

Digit recognition, AT&T labs [Link]
http: Automatic_number_plate_
//[Link]/~yann/ recognition

46
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

Face detection
Many new digital cameras now detect faces e.g., Canon, Sony,
Fuji, ...

47
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

Smile detection

The smile shutter flow

- A camera is smart enough to catch every smile.
- In Smile Shutter Mode, camera can automatically trip the
shutter at just the right instant to catch the perfect expression.

48
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

Object recognition (in supermarkets)

LaneHawk by Evolution Robotics
A smart camera is flush-mounted in the checkout lane, continuously watching for items.
When an item is detected and recognized, the cashier verifies the quantity of items that
were found under the basket, and continues to close the transaction. The item can
remain under the basket, and with LaneHawk,you are assured to get paid for it.

49
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

Vision-based biometrics
The remarkable story of Sharbat
Gula, first photographed in 1984
aged 12 in a refugee camp in
Pakistan by National Geographic
photographer Steve McCurry, and
traced 18 years later to a remote
part of Afghanistan where she was
again photographed by McCurry.
"How the Afghan Girl was Identified by Her Iris Patterns" Read the National Geographic accepted and
story at published this conclusion in a sec-
[Link]
ond cover issue featuring Sharbat
Gula, 18 years after the first, and
the Society launched their "Afghan
Girl’s Fund" to assist the education
of Muslim girls in cultures that dis-
courage or prohibit female educa-
tion.

50
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

Login without a password

Face recognition systems now beginning to appear more widely

[Link]
51
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

Object recognition (in mobile phones)

52
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

Special effects: motion capture

53
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

Sports

54
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

Smart cars

MobileEye ([Link]
Vision systems currently in many car models 55
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

Industrial robots

Vision-guided robots position nut runners on wheels

56
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

Medical imaging

Image guided surgery

[Link]
3D imaging MRI, CT research/
medical-vision-group
57
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

Vision as a source of semantic information

58
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

Vision as a source of semantic information

Object categorization

59
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

Vision as a source of semantic information

Scene and context categorization

60
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

Vision as a source of semantic information

Qualitative spatial information

61
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

1 Preliminaries

2 Digital Image Processing

3 DIP vs. CV

4 Computer Vision

5 State-of-the-art in CV

6 Challenges in CV

62
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

Scale variation
Viewpoint variation

Illumination conditions

63
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

Occlusion

Deformation

Background clutter

Intra-class variation

64
CS-867 Computer Vision by Dr. Mohsin Kamal
Preliminaries Digital Image Processing DIP vs. CV Computer Vision State-of-the-art in CV Challenges in CV

Publication venues in Computer Vision

Journals:
1 IEEE Trans. on Pattern Analysis and Machine Intelligence
(TPAMI)
2 Internal Journal of Computer Vision (IJCV)

3 IEEE Trans. on Image Processing

Conferences
1 Conf. of Computer Vision and Pattern Recognition (CVPR),
once a year
2 International Conference on Computer Vision (ICCV), once
every two years
3 Europe Conference on Computer Vision (ECCV), once
every two years
65
CS-867 Computer Vision by Dr. Mohsin Kamal