Computer Vision

(B22EF0701)
Dr. Nimrita Koul
AY2025-26
Syllabus
• Unit 1: Introduction to computer vision, Image formation: Geometric primitives and
transformation, Photometric image formation, The Digital Camera, Sources, Shadows and
Shading: Local shading models-point, line and area sources; photometric stereo, Camera
calibration.

• Unit2: Image processing: Point operators, Linear filtering, Fourier transforms, Feature
detection and matching: Points and patches, Edge detection: Estimating Derivatives with
Finite Differences, Noise, and Edges and Gradient-based Edge Detectors.

• Unit 3: Segmentation using clustering methods: Human Vision and applications,

Segmentation by graph theoretic clustering, Fitting: Hough transform, fitting lines, Fitting
curves, Structure from motion: Two frame structure from motion, Dense motion estimation:
Parametric motion, Spline based Motion

• Unit 4: 3D reconstruction: Shape from X, Active range finding, Surface representation, Point-
based representations, Volumetric representations, Model-based reconstruction,
Recognition: Object detection, Face recognition, Instance recognition, category recognition
Text and Reference Books
TEXTBOOKS:
1. Computer vision: algorithms and applications, Richard Szeliski, Springer
Science & Business Media;2010.
2. Forsyth, David ,and Jean Ponce. Computer vision: A modern
[Link],2011.

REFERENCEBOOKS:
1. Shapiro LG, Stockman GC. Computer Vision: Theory and Applications.
2001.
2. 2. Trucco, Emanuele, and Alessandro Verri. Introductory techniques for
3-D computer vision. Vol. 201. Englewood Cliffs: Prentice Hall, 1998.
Course Outcomes
CO# Course Outcomes POs PSOs

CO1 Understand the basics of image formation, Camera 1to5 1,2

CO2 Apply morphological process and region growing methods for 1to5 1,2
image processing.

CO3 Make use of threshold techniques for Feature detection, feature 1to5 1,2
matching, and edge detection in images.

CO4 Use clustering-based segmentation methods for image synthesis. 1to5 1,2

CO5 Analyze reconstruction algorithm for 3D objects 1to5 1,2

CO6 Evaluate appropriate techniques for object recognition and 1to5 1,2
detection in computer vision-based applications.
Unit 1 - Contents
• Introduction to computer vision
• Image formation: Geometric primitives and transformation
• Photometric image formation
• The Digital Camera
• Sources
• Shadows and Shading
• Local shading models-point, line and area sources;
• Photometric Stereo
• Camera calibration
Introduction to Computer
Vision
“An Image is worth a Thousand Words”
Computer Vision
• A field of artificial intelligence that enables computers to acquire,
process, and interpret visual information from images or videos, with
the goal of understanding and making decisions about the physical
world.

• Goal: Extract meaningful information → recognition, detection,

reconstruction, tracking, etc.

• CV Pipeline (Image acquisition -> Image Processing -> Feature

Extraction-> Inference). It combines image processing, pattern
recognition, and machine learning to perform tasks such as object
detection, recognition, segmentation, tracking, and 3D reconstruction.
Human Vision
• Visual cortex occupies about 50% of a Macaque brain.
• Largest area of human brain is devoted to processing visual
information than any other kind of information.

From the image on left you

can get lots of information,
such as:

What kind of scene?

Where are the cars?
How far is the building?
Brief History of Computer Vision
• 1966: Minsky assigns computer vision as an undergrad summer
project
• 1960’s: interpretation of synthetic worlds
• 1970’s: some progress on interpreting selected images
• 1980’s: ANNs come and go; shift toward geometry and increased
mathematical rigor
• 1990’s: face recognition; statistical analysis in vogue
• 2000’s: broader recognition; large annotated datasets available;
video processing starts
Fields Related to Computer Vision
An Image
What is an Image?
• An Image is a two dimensional function “f” from R2 to RM.
• f(x, y) represents the intensity at spatial coordinates (x,y).

Source Credits: Stanford University, Course: Introduction to Computer Vision

A Digital Image
• A sampled and quantized version of an image, represented as a
finite 2D array (matrix) of pixels, each pixel storing discrete
intensity or color values.

• It is defined over a rectangle, with a finite range:

f:[a, b] x [c, d] -> [0,255]
What kind of information can we extract from an Image?
• Metric 3D information
• Semantic information
Image Formation:
Geometric Primitives and Transformations
Image Formation
• Image formation is about how images are captured and how the
real 3D world projects into 2D images.

• We will first establish a vocabulary for describing the geometry of

a scene.

• Then we will understand the image formation process that

produced a particular image given a set of lighting conditions,
scene geometry, surface properties, and camera optics.
Geometric Primitives and Transformations
• The light from the real-world objects is captured by a camera and
projected on to an image place to form an image.
• The objects are represented in images using the primitives – points,
lines and regions (surfaces). Thus, the basic building blocks of images
are points, lines, and regions (surfaces).
• A point corresponds to a single pixel location.
• A line/edge represents the boundary or direction of intensity change.
• A region is a collection of connected pixels with similar properties (e.g.,
intensity, color, texture).

These primitives form the foundation for representing shapes and

objects in an image.
Geometric Primitives and Transformations
• The geometric primitives that form the basic building blocks used
to describe three-dimensional shapes (images) are - points, lines
and planes.
• A point corresponds to a single pixel location.
• A line/edge represents the boundary or direction of intensity
change.
• A region is a collection of connected pixels with similar properties
(e.g., intensity, color, texture).

These primitives form the foundation for representing shapes and

objects in an image.
Geometric Primitives – 2D Points
Geometric Primitives - 2D Lines
2D Lines
Geometric Primitives - 2D Conics, 3D Points, 3D Planes
3D Lines and 3D Line Equation
3D Quadrics
2D Transformations
Geometric Transformations
• Transformations define how points or regions in an image are
mapped to new positions.

• They are expressed as mathematical operations on pixel

coordinates.
Basic Set of 2D Planar Transformations
Translation
Scaling
Rotation
• Rotates an image around the origin by angle θ:
Reflection
• Reflection is the mirror image of original object. In other words, we
can say that it is a rotation operation with 180°.

• In reflection transformation, the size of the object does not

change.
Rotation + Translation
Scaled Rotation
Shear
X-Shear
Y-Shear
Composite Transformation
• If a transformation of the plane T1 is followed by a second plane transformation T2, then the result itself
may be represented by a single transformation T which is the composition of T1 and T2 taken in that
order. This is written as T = T1∙T2. Composite transformation can be achieved by concatenation of
transformation matrices to obtain a combined transformation matrix.
• A combined matrix −[T][X] = [X] [T1] [T2] [T3] [T4] …. [Tn] , where [Ti] is any combination of
• Translation
• Scaling
• Shearing
• Rotation
• Reflection
• The change in the order of transformation would lead to different results, as in general matrix
multiplication is not cumulative, that is [A] . [B] ≠ [B] . [A] and the order of multiplication.
• The basic purpose of composing transformations is to gain efficiency by applying a single composed
transformation to a point, rather than applying a series of transformation, one after another.
• For example, to rotate an object about an arbitrary point (Xp, Yp), we must carry out three steps −
• Translate point (Xp, Yp) to the origin.
• Rotate it about the origin.
• Finally, translate the center of rotation back where it belonged
Affine Transformation
Projective Transformation (Homography)
Hierarchy of 2D Transformations
• Previously defined 2D transformations can be thought of as a set
of 3 x3 matrices operating on 2D homogeneous coordinate
vectors.

• They form a nested set of groups – they are closed under

composition and have an inverse that is a member of the same
group. Each simpler group is a subgroup of the more complex
group below it.
Hierarchy of 2D Transformations
Co-Vectors
3D Transformations
3D Transformations
3D Transformations
Rotation + Translation
3D Rotations
Axis/Angle (exponential twist)
Unit Quaternions
• Quaternions are a very popular representation for pose and for pose interpolation in
computer graphics.

• A unit quaternion is a unit length 4-vector whose components can be written as q =

(qx; qy; qz; qw) or q = (x; y; z;w) for short.

• Unit quaternions live on the unit sphere ||q|| = 1 and antipodal (opposite sign)
quaternions, q and -q, represent the same rotation (Figure 2.6). Other than this
ambiguity (dual covering), the unit quaternion representation of a rotation is unique.

• This representation is continuous, i.e., as rotation matrices vary continuously, you

can find a continuous quaternion representation, though the path on the quaternion
sphere may wrap all the way around before returning to the “origin” qo = (0; 0; 0; 1).
3D to 2D Projections
Perspective
• The most used projection in computer graphics and computer vision is true 3D perspective.
Here, points are projected onto the image plane by dividing them by their z component.
Using inhomogeneous coordinates, this can be written as
• Computer graphics systems often use a
form of two-step projection that first
projects 3D coordinates into normalized
device coordinates (x; y; z) ∈ [-1,1]x[-
1,1]x[0,1], and then rescales these
coordinates to integer pixel coordinates
using a viewport transformation. The
(initial) perspective projection is then
represented using a 4 x 4 matrix:
Camera Models
Camera
• The camera is the mechanism by which we can record the world
around us and use its output - photographs - for various
applications.
Simple Pinhole Camera
• This camera system can be designed by placing a barrier with a small aperture between the
3D object and a photographic film or sensor.
• As Figure 1 shows, each point on the 3D object emits multiple rays of light outwards. Without
a barrier in place, every point on the film will be influenced by light rays emitted from every
point on the 3D object. Due to the barrier, only one (or a few) of these rays of light passes
through the aperture and hits the film.
• Therefore, we can establish a one-to-one mapping between points on the 3D object and the
film. The result is that the film gets exposed by an “image” of the 3D object by means of this
mapping. This simple model is known as the pinhole camera model
Formal Construction of the Pinhole Camera Model
• In this construction, the film is commonly
called the image or retinal plane.

• The aperture is referred to as the pinhole O or

center of the camera.

• The distance between the image plane and the

pinhole O is the focal length f.

• Sometimes, the retinal plane is placed between

O and the 3D object at a distance f from O. In
this case, it is called the virtual image or virtual
retinal plane.

• The projection of the object in the image plane

and the image of the object in the virtual image
plane are identical up to a scale (similarity)
transformation.
How do we use Pinhole Cameras?
• Let P = [ x y z]T be a point on some 3D object visible to the pinhole camera. P will be
mapped or projected onto the image plane Π’ , resulting in point1 P’ = [x’ y’] T .
Similarly, the pinhole itself can be projected onto the image plane, giving a new point
C’ .
• Here, we can define a coordinate system [i j k] centered at the pinhole O such that
the axis k is perpendicular to the image plane and points toward it. This coordinate
system is often known as the camera reference system or camera coordinate
system. The line defined by C’ and O is called the optical axis of the camera system.
• Recall that point P’ is derived from the projection of 3D point P on the image plane
Π’. Therefore, if we derive the relationship between 3D point P and image plane
point P’, we can understand how the 3D world imprints itself upon the image taken
by a pinhole camera.
• The triangle P’C’O and the triangle formed by P, O and (0, 0, z) are similar triangles.
• Therefore, using the law of similar triangles we find that
In pinhole camera, we assume that the aperture is a single point. In most real-world scenarios, however, we
cannot assume the aperture can be infinitely small.
• As the aperture size increases, the
number of light rays that passes through
the barrier
• consequently increases. With more light
rays passing through, then each point on
the film may be affected by light rays from
multiple points in 3D space, blurring the
image.

• We cannot reduce the aperture size

indefinitely because a smaller aperture
size causes fewer light rays to pass
through, resulting in crisper but darker
images.

• Therefore, we arrive at the fundamental

problem presented by the pinhole
formulation: can we develop cameras
that take crisp and bright images?
We need more light, but big pinholes cause blur
Cameras and Lenses
• In modern cameras, the above conflict between crispness and brightness is
reduced by using lenses, devices that can focus or disperse light.
• If we replace the pinhole with a lens that is both properly placed and sized,
then it satisfies the following property:
• All rays of light that are emitted by some point P are refracted by the lens such that they
converge to a single point P’ in the image plane. Therefore, the problem of most of the
light rays blocked due to a small aperture is removed.

• Lenses have a specific distance for which objects are “in focus”. This property is also
related to a photography and computer graphics concept known as depth of field, which
is the effective range at which cameras can take clear photos.

• Camera lenses they focus all light rays traveling parallel to the optical axis to one point
known as the focal point (Figure 5). The distance between the focal point and the
center of the lens is commonly referred to as the focal length f.

• Light rays passing through the center of the lens are not deviated
Thin Lens
Photometric Image Formation
Photometric Image Formation
• Photometric image formation explains how light interacts with
objects and how that interaction gets recorded as pixel values.

• I.e., it explains how light intensity values (brightness, color) in

an image are determined by:
• the scene properties (surface reflectance, material, color),
• the illumination (light sources, direction, intensity, spectrum),
• the camera response (sensor sensitivity, exposure, lens).
• Important components of photometric image formation are – light
sources, surface reflectance models, camera response.
Photometric Image Formation
Illumination – Types/Models of light sources
1. Point source : A single small source located at some position (like a bare bulb or flashlight). Light rays
spread out radially from that point.
• Characteristics:
• Sharp highlights: Intense bright spot where the reflection angle matches the viewing direction.
• Hard shadows: Well-defined shadow boundaries (umbra).Example: Torchlight on a wall → one bright spot and sharp-
edged shadow behind an object.
2. Directional Source (E.g., Sunlight): This source is effectively at infinite distance, so rays are parallel when
they reach the scene.
Characteristics:
• Lighting appears uniform across the scene.
• Shadows are sharp (if sun is small in angular size).
• Shadow orientation is the same every where. E.g., midday sunlight issues parallel rays that create crisp shadows
pointing in the same direction.
3. Area source: Light comes from a finite surface area (like a window, sky, or large fluorescent panel). Rays
arrive from multiple directions instead of just one. E.g., Overcast sky or a big photography softbox creates
shadows that are barely visible, surfaces are lit smoothly.
Characteristics:
• Soft shadows: Edges of the shadow are blurred (penumbra).
• Diffuse lighting: Even illumination, less contrast.
These models help explain shading, shadows, and brightness in images. They are important for photometric
stereo and shape-from-shading tasks, where the type of light gives the cues for 3D reconstruction.
• Just like the construction of the pinhole model that relates a point P in 3D
space with its corresponding point P’ in the image plane, we can have
equation that relates P and P’ in lens-based camera:

• We can have an equation for camera with lens as well, however, in the
pinhole model z’ = f, while in this lens-based model, z’ = f +z0.

• This derivation takes advantage of the paraxial or “thin lens” assumption. So,
it is called the paraxial refraction model.

• Thin lens assumption – “For the angle θ that incoming light rays make with
the optical axis of the lens, the paraxial assumption substitutes θ wherever
sin(θ) is used. This approximation of θ for sin θ is true as θ approaches 0.”
Pincushion and Barrel Distortion in Images
Light at surfaces
Many effects occur when light strikes a
surface. It could be
• Reflected – Mirror
• Transmitted – Skin, glass
• Scattered – Milk
• Travel along the surface and leave at
some other point
• Absorbed
Bidirectional Reflectance Distribution Function (BRFD)
Lighting, Reflectance and Shading
Specular Reflection
Ideal specular reflection is mirror reflection
– Perfectly smooth surface
– Incoming light ray is bounced in single direction
– Angle of incidence equals angle of reflection
Diffuse Surface
• Ideal diffuse material
reflects light equally in all
directions
• View-independent
• Matte, not shiny materials:
– Paper
– Unfinished wood
– Unpolished stone
Diffuse Reflection
• Beam of parallel rays shining on a surface
– Area covered by beam varies with the angle between the beam and the normal
– The larger the area, the less incident light per area
– Incident light per unit area is proportional to the cosine of the angle between the normal
and the light rays
• Object darkens as normal turns away from light
• Lambert’s cosine law (Johann Heinrich Lambert, 1760)
• Diffuse surfaces are also called Lambertian surfaces
Light, Sources of Light.
Shadows, Shades
Light
• Special light sources
• Point sources
• Distant point sources
• Area sources
• Point light source is like light bulbs. It is an infinitesimally small
point that radiates light equally in all directions. Light vector varies
across receiving surface, intensity drops off proportionally to the
inverse square of the distance from the light. Reason for inverse
square falloff: Surface area of sphere A = 4πr2.
• Standard nearby point source model and distant source.
Shadows
• Give additional cues on scene lighting, contact points and depth
cues.
Photometric Stereo
Photometric Stereo
• It is a method to recover surface normals (shape) using multiple
images taken under different known lighting conditions.
• Assumptions:
• Same viewpoint (camera fixed).
• Only lighting changes.
• Equation: 𝐼=𝜌(𝑛⋅𝑠)
where I = intensity,
ρ = albedo (surface reflectivity),
n = surface normal,
s = light source direction.
With ≥3 light sources, we can solve for surface normals.
Photometric Stereo
Photometric Stereo is a technique to estimate the surface normals (A
surface normal is a unit vector that is perpendicular to a surface at a
given point.) of an object using multiple images taken from a fixed
camera viewpoint but under different lighting directions.
Key Idea:
For a Lambertian surface, pixel intensity is proportional to the dot
product between the light direction and the surface normal. By capturing
several images with varying illumination, one can solve for the surface
normal at each pixel.
Output: Dense surface normal map → can be integrated to recover
object shape/geometry.
Applications: 3D shape recovery, texture mapping, material inspection,
cultural heritage digitization.
Goal of Photometric Stereo
• The goal of photometric stereo is to recover per-pixel surface
normals (A surface normal is a unit vector that is perpendicular
to a surface at a given point.) and intrinsic reflectance of surface
(albedo) — and thus the 3D shape — from images of an object
captured under varying, known lighting directions with a fixed
camera.
• Setup:
• A single fixed camera.
• Multiple images of the same object taken under different known lighting
directions.
• Assumes a Lambertian reflectance model (diffuse reflection, intensity
depends only on angle between light and normal).
Number of Images Needed for Photometric Stereo:
• Minimum: 3 images with non-coplanar light directions (to solve for 3D normal).
• In practice: more images → more robust estimation (least squares solution).
Output:
• A normal map of the surface.
• Normals can be integrated to reconstruct a height map or full 3D surface
Limitations:
• Assumes Lambertian surfaces (fails on shiny/specular or transparent objects).
• Sensitive to cast shadows and interreflections.
• Requires calibrated lighting directions.
Applications:
• Shape recovery in archaeology and cultural heritage.
• Industrial inspection (detecting fine surface defects).
• Face and biometric modeling.
• Robotics and 3D perception.
Camera Intrinsics &Extrinsic Parameters
Modelling the projection from 3D space to the
digital images
• All the results derived will use the pinhole model, but they also hold for the
paraxial refraction model.
• As discussed earlier, a point P in 3D space can be mapped (or projected) into
a 2D point P’ in the image plane Π’ . This R3 → R2 mapping is referred to as a
projective transformation.
• This projection of 3D points into the image plane does not directly correspond to what
we see in actual digital images for several reasons. First, points in the digital images are,
in general, in a different reference system than those in the image plane.
• Second, digital images are divided into discrete pixels, whereas points in the image
plane are continuous.
• Finally, the physical sensors can introduce non-linearity such as distortion to the
mapping.
To account for these differences, we will introduce a number of additional
transformations that allow us to map any point from the 3D world to pixel
coordinates
Camera Matrix Model and Homogeneous Coordinates

• The camera matrix model describes a set of important parameters that affect how a world
point P is mapped to image coordinates P’. These parameters are represented in matrix form.
• The first parameters, cx and cy, describe how image plane and digital image coordinates
can differ by a translation.
• Image plane coordinates have their origin C’ at the image center where the k axis intersects
the image plane. Digital image coordinates typically have their origin at the lower-left corner
of the image. Thus, 2D points in the image plane and 2D points in the image are offset by a
translation vector [cx, cy]T . To accommodate this change of coordinate systems, the
mapping now becomes:
Mapping pixels to physical measurements
• Points in digital images are expressed in pixels, while points in image plane are represented
in physical measurements (e.g., centimeters). To accommodate this change of units, we
must introduce two new parameters k and l in camera matrix.
• These parameters, whose units would be something like pixels per cm , correspond to the
change of units in the two axes of the image plane. k and l may be different because the
aspect ratio of a pixel is not guaranteed to be one. If k = l, we often say that the camera has
square pixels. We adjust our previous mapping to be:

However, from above equation, we see that this projection P → P’ is not linear, as the operation divides
one of the input parameters (z). We can use homogeneous coordinates to represent this non-linear
projection as a matrix-vector product despite its nonlinearity.
Homogeneous Coordinates
Complete Camera Matric Model
Camera Intrinsic Parameters
Intrinsic parameters describe how the camera
maps 3D points in camera coordinates to 2D
points in the image plane. They are independent of
the scene and only depend on the camera’s
internal geometry and optics.
Important Parameters:
• Focal length (𝑓𝑥,𝑓𝑦): Scaling factors in
horizontal and vertical directions, often
measured in pixels. Focal length depends on the
units used to measure pixels
• Principal point (𝑐𝑥,𝑐𝑦): The intersection of the
optical axis with the image plane (ideally the
image center).
• Skew (𝑠s): Non-orthogonality between image
axes (usually zero in modern cameras).
• Pixel aspect ratio: Ratio of horizontal to vertical
pixel dimensions.
Intrinsic Matrix (Calibration Matrix K)

• This matrix uses independent focal lengths fx and fy for the sensor x and y dimensions.
• The entry s encodes any possible skew between the sensor axes due to the sensor not being mounted
perpendicular to the optical axis
• (cx; cy) denotes the image center expressed in pixel coordinates.
• The image center is also often called the principal point, although in optics, the principal points are 3D
points usually inside the lens where the principal planes intersect the principal (optical) axis.
•. The intrinsic parameters are required for camera calibrations. They are essential in 3D reconstruction,
Structure-from-motion and augmented reality.
• Errors in intrinsics affect geometric accuracy. E.g., wrong Focal length gives wrong depth perception
Extrinsic Parameters
Complete mapping from a 3D point to the image plane
• This completes the mapping from a 3D point P in an arbitrary world reference
system to the image plane.
• Thus, the full projection matrix M consists of the two types of parameters
introduced above: intrinsic and extrinsic parameters.
• All parameters contained in the camera matrix K are the intrinsic parameters,
which change as the type of camera changes.
• The extrinsic parameters include the rotation and translation, which do not
depend on the camera’s build.
• Overall, we find that the 3 × 4 projection matrix M has 11 degrees of freedom:
5 from the intrinsic camera matrix, 3 from extrinsic rotation, and 3 from
extrinsic translation.
Recap - The projective camera: intrinsic and extrinsic
parameters
• Extrinsic Parameters: Since camera may not be at the origin, there
is a rigid transformation between the world coordinates and the
camera coordinates
• Intrinsic parameters: Since scene units (e.g., cm) differ image
units (e.g., pixels) and coordinate system may not be centered in
image, we capture that with a 3x3 transformation comprised of
focal length, principal point, pixel aspect ratio, and skew
 Digital Camera
Lens → Aperture → Shutter → Sensor → ADC → Image Processor → Storage.
Internals of a Digital Camera
1. Lens System: Focuses light from the scene onto the image sensor. Determines field of view,
depth of field, and distortions (radial, tangential).
2. Aperture: Adjustable opening that controls the amount of light entering. Measured in f-stops
(e.g., f/2.8). Also affects depth of field (smaller aperture → deeper focus).
3. Shutter: Controls exposure time (how long the sensor collects light). Types: mechanical or
Electronic (rolling/global shutter).
4. Image Sensor: Converts light into electrical signals. Types: CCD (Charge-Coupled Device) or
CMOS (Complementary Metal-Oxide Semiconductor). Covered with a color filter array
(e.g., Bayer pattern) for RGB capture.
5. Analog-to-Digital Converter (ADC): Converts analog voltages from the sensor into digital
pixel values (discrete intensities).
6. Image Processor: Performs in-camera processing: demosaicing, white balance, noise reduction,
compression (JPEG, RAW).
7. Storage & Output: Stores images in formats like RAW, JPEG, PNG. Provides data to memory
card or computer.
Internals of a digital camera
Internals of a Digital Camera
Camera Calibration
Color Cameras
Color Cameras
• Human eye has 3 types of cone cells.
• Cameras have:
• 1. Filter wheel
• 2. Prism (with 3 sensors)
• 3. Filter mosaic … and X3
Camera Calibration
Camera Calibration
• We need to know the intrinsic camera parameters to know the transformation
from the real, 3D world into digital images.
• For an arbitrary camera, we may not know these parameters. However, we
have access to the images taken using this camera takes.
• We can estimate the extrinsic and intrinsic camera parameters from the
images taken by this camera. This process is known as camera
calibration.
• We do this by solving for the intrinsic camera matrix K and the extrinsic
parameters R, T from Equation 10.
Camera Calibration
Camera Calibration
Camera Calibration