Computer Vision (CSL536)

Dr. Praveen Kumar

CSE Department, VNIT Nagpur
 4
General Information
Lecture slot: B Labs: Mostly assignment based
E-mail: praveenkumar@[Link],
MS Teams/[Link]: Platform for all course related
interactions, materials, announcement etc.
Study material: Lecture slides and Video recordings (as
far as possible) will be made available
References/Books: “Computer Vision: Algorithms and Applications” by
Richard Szeliski, Springer, 2010
“Digital Image Processing”, R. C. Gonzalez & R. E. Woods, Pearson Education,
3rd edition
“Image Processing, Analysis, and Machine Vision”, 2nd ed., M. Sonka, V.
Hlavac, & R. Boyle. Cengage Learning, 2009.
5
Why study this course?
Images (and videos) have become ubiquitous as are
systems that extract and use visual information

Surveillance
Autonomous Vehicles Credit: Yeong Nam, Flickr CC License Factory Automation
Credit: smoothgrover22, Flickr CC License Credit: KUKA Roboter GmbH, Bachmann

Human-Computer
Medical Imaging Interaction Visual Effects
Credit: National Cancer Institute Credit: AntMan3001, Flickr CC License
Credit: Vancouver Film School
6
Why study this course?
Applications of computer vision are exploding
especially in recent past
Retail and Retail Security (Amazon Structural Health Monitoring (Oilwell
Go, Virtual Try-on, StopLift) Inspection, Drone-based Bridge
Inspection and 3D Reconstruction)
Healthcare (Blood Loss
Detector, DermLens) Document Understanding (Optical
Character Recognition, Robotic
Agriculture (SlantRange, Process Automation)
Cainthus - Livestock facial
recognition)
Augmented Reality (TechSee
Banking and Finance (Mobile
Visual Support, Warehouse and
Deposit, Insurance Risk Profiling)
Enterprise Management)
Remote Sensing (Land Use
Understanding)
7
Why study this course?
Very active research area with long history and rapidly evolving

Most importantly, it is interesting stuff

with immediate real life practical impact
8
Course outcomes
By the end of the semester, you should be
able to:
• Look at a real life problem and identify if
Computer vision can appropriately address
or solve it (or atleast a part)
• If so, identify what techniques and
algorithms that you studied might be
applicable
• Apply them and develop a practical solution
9
What is Computer Vision?
A field that seeks to automate and endow a
computing framework with the ability to interpret
and understand images the way humans do.

What & where are the objects in the scene? how are they are moving ?

Can a machine answer these questions or perform these human vision task?
10
Image Processing Vs. Computer Vision
• Image Processing
– Image in, image out
– Usually low level techniques (eg,
noise removal, edge detection)
– Quantitative measurements
• Computer Vision
– Extracting symbolic descriptions
– Higher level techniques (eg,
object recognition)
– Semantic output
• IVP techniques are often used
in computer vision
11
Vision is multidisciplinary

Computer
Graphics

HCI

From wiki
13
Scope of this course
The continuum from image processing to
computer vision can be divided up into low-,
mid- and high-level processes
Low Level Process Mid Level Process High Level Process
Input: Image Input: Image Input: Attributes
Output: Image Output: Attributes Output: Understanding
Examples: Noise Examples: Feature Examples:
removal, image extraction, region Recognition, Scene
sharpening segmentation etc. understanding,
autonomous navigation
14
Low-Level: Image Adjustments
15
Low-Level: Image Grayscale
16
Low-Level: Exposure
17
Low-Level: Image Saturation
18
Low-Level: Image Hue
19
Low-Level: Image Edges
20
Low-Level: Image Enhancement
21
Low Level : Color info & processing
Mid-Level Vision
- Morphological processing
- Model Fitting
- Segmentation
- Motion object detection
- tracking
- Panoramas
Mid-Level: Morphological Processing
Mid-Level: Model Fitting
Mid-Level: Feature extraction and
matching
Mid-Level: Panorama Stitching
Mid-Level: Segmentation
Background Subtraction

=
Mid-level: Moving object tracking
High-Level Vision
Semantics! Hardest!
- Image classification, tagging
- Object recognition, detection
- Instance segmentation
- Activity Recognition
- Pose Estimation etc.
High-Level: Classification
- What is in the image?
High-Level: Tagging
- What are ALL the things in the image?
High-Level: Detection
- What are ALL the things in the image?
- Where are they?
High-Level: Instance Segmentation
High-Level: Activity classification

Aerobics
High-Level: Pose Estimation
37

WHY IS VISION HARD?

 Although seems easy for people, vision is very
difficult for computers
Vision is an amazing feat of natural intelligence
Is that a
Visual cortex occupies about 50% of Macaque queen or a
brain bishop?
More human brain devoted to vision than
anything else
 A little story about Computer Vision
In 1966, Seymour Papert (some anecdotes say it was Marvin
Minsky) at MIT asked a group of his undergraduate students led
by Gerald Jay Sussman to “spend the summer linking a camera to
a computer and getting the computer to describe what it saw”. We
now know that the problem is slightly more difficult than that.
(Szeliski 2009, Computer Vision)

Aimed to develop a platform

to automatically segment
background/foreground and
extract non-overlapping
objects from real-world
images
 Why is CVhard?
When machines “view” images,
all they see are numbers that
represent individual pixels.

animal or not?
 1. High-dimensional data ⇒ heavy
computational requirements
• Dealing with lot of data that needs to be processed and
made sense of.
• Things start to get really difficult for computer vision the
higher the resolution of images. Ex. HD video at 30 fps = 3
*1920 * 1080 * 30 ~= 180 million pixels/sec
• Real time processing of all that no.’s!
 2. Loss of Information
• Digitising process, noise, compression etc.
• 3D (or 4D) to 2D (or 3D)
• Many practical use cases are inverse model applications
• No knowledge of how an image was taken or camera parameters
• But need to model the real world in which picture/video was taken
(shape, lighting, color, objects, interactions)
 3. Interpretation is needed
• Very difficult thing for machines to deal with.
• We use accumulated learning and memory (called a
priori knowledge
• Therefore with more machine learning capability, there has
been great advancement in CV also.
• …but really we don’t understand the recognition process
Visual Recognition Challenges: illumination
Visual Recognition Challenges: Deformation
Visual Recognition Challenges: Occlusion
Visual Recognition Challenges: Intraclass variation
Challenges: Background clutter
 Visual Recognition Challenges
Realistic scenes are crowded, cluttered, have overlapping objects.
 4. No complete models of the
human visual system exist
• Existing models largely related to
subsystems, not holistic
• What is perceived, and what is
cognized?
• When is an object important for
a task, and when is the context
important?
• Verifiability of mathematical/
physical models non-trivial
Structure of The Human Eye
Illusory perceptions
Simple scene right?
Dark square
Light square

Edward Adelson
Edward Adelson
Really!

Edward Adelson
Brightness Adaptation & Discrimination
Color Perception
 Optical Illusions
• Human visual system is
good at picking out
structure from noisy,
incomplete, and missing
data

• We make and use

assumptions about the
real world to do this

• Optical illusions occur

when these assumptions
are incorrect
 Optical Illusions (more)

Rabbit or duck? Faces or Flower Vase ?

Eskimo or Tribal chief ?

Is this the face of a lady or

a word?

How many legs does this elephant have?

 Optical Illusions (and more…)

Is this picture still or moving?

Stare at the cross in the middle of
the image and think circles
Optical Illusions (Even more…)
Are the dots in between the Focus on the dot in the
squares white, black or grey? middle and then move
your head backwards
and forwards.

Is the ladder going

up or down?
61

BRIEF HISTORY OF
SUCCESSFUL APPLICATIONS
 History of applications
1970s: Optical Character Recognition (OCR)
1980s: Machine vision, Smart cameras
1990s: Machine vision in manufacturing environments,
Biometrics, Medical imaging, Recording devices, Video
surveillance
2000s: More biometrics, Better medical imaging,
Object/Face detection, object tracking, HCI applications
2010s: Autonomous navigation, Google Goggles, Vision on
social media …. Everywhere around us!
 Optical character recognition(OCR)
Technology to convert scanned docs to text
If you have a scanner, it probably came with OCR

software

License plate readers Handwritten Digit recognition

[Link]
Machine vision for Factory automation
Automated inspection, Aiding Manual Assembly,
Vision-guided Robots ….upto recent Industry 4.0
Security andsurveillance
Face detection andmore…

• Most digital cameras can detect faces…

Face detection andmore…
SONY
“Smile Shutter”

• Some can detect blinking or smiling…

 Biometrics

Fingerprint scanners are Face recognition systems are

now on many new laptops beginning to appear more widely
and other devices
Object recognition
Vision-based interactive interfaces
 Smart cars

Mobileye : Vision
systems on high
end BMW, GM,
Volvo models

Pedestrian collision warning

Forward collision warning Lane
departure warning
Headway monitoring and warning
Autonomous cars arehere!
Current state of theart
• Can computers match (or beat) human vision?
• Yes and no (but mostly no!)
• Humans are much better at “hard” things
• Computers can be better at “easy” things
• Though getting good at harder things using machine
& deep learning techniques.
74

RECENT PROJECTS @ VNIT

Vision based proactive guard for bank ATMs

Patent pending:

System and method for performing surveillance of an ATM. Indian patent application no.
201611009410A. Filed on 17/3/2016 and published on 22/9/2017

Computer Science and Engineering Department

 Problem and how to prevent it

September 1, 2017: Source: (Express)

[Link]
Jan 3 2018 : Source (TOI) needs-greater-vigilance-4823142/
[Link]
lost-8-times-the-money-it-did-in-16-17/articleshow/[Link]
 Competitive advantages
1. Pro-active prevention vs. Passive CCTV recording : real time
abnormality detection
2. Minimize manpower cost and/or improve efficiency in 24x7
monitoring of multiple locations
3. Video summarization and efficient search for forensic
investigation
4. Cost effective RGB-D sensor for high accuracy
- RGB, Depth and Skeleton data
Demo Video:Pro-active ATM Security Guard
 Camera based online classroom
attendance management System
• Works with low resolution panoramic
camera covering large classroom
• Rapidly and accurately marks
attendance online/real time
• Doesn’t require active participation of
students
• No need to store in advance any
personal identity information like
biometric
• Automatically generate useful
cumulative statistics/reports
• Send appropriate timely intimations
Vision based classroom
analytics
Student engagement/ emotion analysis
Automated extraction of Tabular data
Intelligent Traffic Signal Time
Management
 Other traffic surveillance applications

Wrong way driving

detection

Stopped vehicle
detection
Alphanumeric Digits Extraction from
an Indian Number Plate for SIH'20
 Opportunities in CV
• This is just a taste of the state of the art.
• Some of these are less than 5 years old, most
less than 10
• This is a very active research area, and rapidly
changing
• Many new apps in the next 5 years

Next class we will start to see how it all works …