Multiple View Geometry 1

Last updated: 20-04-2025

Szeliski R., Computer Vision - Algorithms and

Applications, Springer, 201, Ch. 12.
Solem, J. E., 2012. Programming Computer Vision
with Python: Tools and algorithms for analyzing
images. "O'Reilly Media, Inc.“, Ch. 5. Dr. Zulfiqar Habib, Professor
[Link]
Topics 2

o Stereo vision
o Depth estimation
o Structure from motion (3D reconstruction)
Recovering 3D from Images
3
How can we automatically compute 3D geometry from
images?
What cues in the image provide 3D information?
 Introduction… 4

❑ Humans have the natural ability

to perceive 3D world.
❑ When conventional cameras
capture the pictures, the depth
information is lost due to the
projection of a scene onto the 2D
image plane.
 3D Applications
5

❑ Recovering the 3D structure of a scene from a

single image has become an important
research problem in the present years due to
its wide range of applications.
3D Printing
3D Applications
6
Self Driving Cars
Robot Navigation
3D Applications
7
3D Games / Movies
Introduction
Monocular Depth Cues…
8
▪ Interposition
o When one object partly blocks your view of another, you
perceive the partially blocked object as farther away
▪ Linear perspective
o Parallel lines that are known to be the same distance apart
appear to grow closer together, or converge, as they recede
into the distance
▪ Texture Gradients
o The texture of objects tend to become smoother as the object
gets farther away, suggesting that more detailed textured
objects are closer.
Introduction
Monocular Depth Cues
9
▪ Relative Size
o Larger objects are perceived as being closer to the viewer,
and smaller objects as being farther away
▪ Light and Shadow
o When objects are darker or shaded they are perceived as
farther away.
▪ Motion Parallax
o When you ride in a moving vehicle and look at the side window, the
objects you see appear outside move in opposite direction
o Objects seem to be moving in different speeds-the ones that are closer to
you seem to move faster, whereas objects far behind seem to move slower
Introduction
Types of Existing Approaches
10

Manual: Currently used in high-quality cinematic

2D-to-3D conversion workflows.
Interactive (Semi-automatic): A combination of
user-input and computer vision algorithms.
Automatic: Entirely based on computer vision
algorithms or convolutional neural networks.
Stereo Reconstruction 11
The Stereo Problem
– Shape from two (or more) images
– Biological motivation

Why do we have two eyes?

 Stereograms
12

Magic Eye
 Stereograms
13

Magic Eye
Stereograms Anaglyphs 14
Epipolar Geometry 15
Epipolar geometry is the geometry of stereo vision. When two
cameras view a 3D scene from two distinct positions, there are a
number of geometric relations between the 3D points and their
projections onto the 2D images that lead to constraints between the
image points. These relations are derived based on the assumption
that the cameras can be approximated by the pinhole camera model.
o Two views of the same object
o Suppose I know the camera positions and camera
matrices
o Given a point on left image, how can I find the
corresponding point on right image?
 Epipolar Geometry 16
o Two cameras take a picture of the same scene from
different points of view.
o The epipolar geometry then describes the relation
between the two resulting views.
Depth Estimation 17
To find the depth of a point in a stereo image pair, we can use
the triangulation based on the disparity of the corresponding
points in the two images.
The depth Z of a point in 3D space can be calculated
Using the following general triangulation formula:
𝑍=(𝐵⋅𝑓)/𝑑
where
B is the baseline distance between the two cameras c & c’.
f is the focal length in pixels of the cameras.
d = (x-x’) is the disparity of the corresponding points
in pixels.
Depth Estimation: Example 18
Given a stereo camera setup with a baseline distance B = 0.2 meters
and a focal length f = 1000 pixels for both cameras.
If positions in the left and right images are 116 &
100 pixels, resp., calculate its depth.

Sol. Disparity d = 116-100 = 16 pixels

Using the formula 𝑍 = (𝐵⋅𝑓)/𝑑
𝑍 = (0.2×1000)/16
=12.5 meters
Depth Estimation: Real Example 19
Given a pair of stereo pictures of iPhone 15 pro max camera with a
baseline distance B = 6 cm and a focal length f = 7 mm for both
pictures. If positions of a point in the right and left images are 1800 &
1350 pixels, resp., calculate its depth.
Sol.
Using the formula 𝑍 = (𝐵⋅𝑓)/𝑑
f in pixels = fmm x image width (pixels) / sensor width (mm)
= 7 * 3213 / 9.8 = 2295 pixels
d = 1800 – 1350 = 450
AI tools can help to find
𝑍 = 6*2295/450 =30.6 cm sensor width of camera
Example: Depth Estimation From Stereo Video 20
Structure from Motion 21
Structure from motion (SfM) is the process of estimating
the 3D structure of a scene from a set of 2D images.
Applications:
o 3D scanning
o 3D printing
o Geosciences
o Cultural heritage
o Biological vision
o Augmented reality, etc.
Structure from Motion 22
If the images are taken with a single calibrated camera, then the
3-D structure and camera motion can only be recovered up to
scale. Up to scale means that you can rescale the structure and
the magnitude of the camera motion and still maintain
observations. For example, if you put a camera close to an
object, you can see the same image as when you enlarge the
object and move the camera far away. If you want to compute
the actual scale of the structure and motion in world units, you
need additional information, such as:
o The size of an object in the scene
o Information from another sensor, for example, an odometer.
 Structure from Motion from Two Views
23
For the simple case of structure from two stationary
cameras or one moving camera, one view must be
considered camera 1 and the other one camera 2.
In this scenario, the algorithm assumes that camera
1 is at the origin and its optical axis lies along
the z-axis.
1. SfM requires point correspondences between
images. Find corresponding points either by
matching features or tracking points from image
1 to image 2.
Structure from Motion from Two Views
24
2. To find the pose of the second camera relative to the first
camera, you must compute the fundamental matrix. Use the
corresponding points found in the previous step for the
computation. The fundamental matrix describes the epipolar
geometry of the two cameras. It relates a point in one camera
to an epipolar line in the other camera.
Structure from Motion from Two Views
25
3. Find the orientation and the location of the second camera in
the coordinate system of the first camera. The location can
only be computed up to scale, so the distance between two
cameras is normalized to 1 unit.
4. Determine the 3-D locations of the matched points. Because
the pose is up to scale, when you compute the structure, it has
the right shape but not the actual size.
5. Finally display the reconstruction, and visualize the camera
poses
Example: 3D Reconstruction 26
Example: 3D Reconstruction 27
Example: 3D Reconstruction 28
Watch @ YouTube 29
Watch @ YouTube 30
Extra Stuff 31
Depth Estimation 32

The triangulation formula used to calculate the depth of a point in a stereo

image pair is derived from geometric principles and the similar triangles
formed by the cameras, the point in 3D space, and their projections onto
the image planes. Consider the following setup:
•Two cameras are observing the same point in 3D space from different
viewpoints.
•The distance between the camera centers (the baseline) is denoted by 𝐵.
•The focal length of each camera is denoted by 𝑓.
•The disparity between the projections of the point onto the two image
planes is denoted by 𝑑.
 Depth Estimation 33

In the stereo setup, the cameras, the observed point in C1 C2

3D space, and their projections onto the image planes *--------------------*
form similar triangles. This is illustrated in the /| /|
diagram. / | / |
/ | / |
*---P----------------*---P'
• C1 and 𝐶2 are the centers of Camera 1 and Camera 2, respectively.
• 𝑃 and 𝑃′′ are the 3D point and its projections onto the image planes of
Camera 1 and Camera 2, respectively.
Depth Estimation 34

Using the similar triangles formed by the cameras, the C1 C2

3D point, and their projections, we can write: *--------------------*
𝑍/𝑓 =𝑋/𝑥 /| /|
𝑍/𝑓=(𝑋−𝐵)/(𝑥−𝑑) / | / |
/ | / |
*---P----------------*---P'
• 𝑍 is the depth of the point in 3D space.
• 𝑋 is the x-coordinate of the point in 3D space.
• 𝑥 is the x-coordinate of the projection of the point onto the image plane.
• 𝑑 is the disparity between the projections of the point onto the two image
planes.
Depth Estimation 35

By equating the expressions for 𝑍/𝑓, we can solve for 𝑍: C1 C2

𝑋/𝑥 = (𝑋−𝐵)/(𝑥−𝑑) *--------------------*
𝑋(𝑥−𝑑) = (𝑋−𝐵)𝑥 /| /|
𝑋𝑥−𝑋𝑑 = 𝑋𝑥−𝐵𝑥 / | / |
𝑋𝑑 = 𝐵𝑥−𝑋𝑥 / | / |
𝑍 = 𝐵𝑓/𝑑 *---P----------------*---P'

This is the triangulation formula derived from the similar triangles formed
by the stereo setup. It represents the relationship between the baseline
distance 𝐵, the focal length of the cameras 𝑓, the disparity 𝑑, and the depth 𝑍
of a point in 3D space.
Depth Estimation 36

From Convergence

Human performance: up to 6-8 feet

Epipolar Geometry 37
The figure depicts two pinhole cameras
looking at point X. In real cameras, the
image plane is actually behind the focal
center, and produces an image that is
symmetric about the focal center of the
lens. Here, however, the problem is
simplified by placing a virtual image OL and OR: the centers of symmetry
plane in front of the focal center of the two cameras lenses.
i.e. optical center of each camera lens to X: the point of interest in both
cameras.
produce an image not transformed by the xL and xR: are the projections of
symmetry. point X onto the image planes.
Epipolar Geometry 38
Each camera captures a 2D image of the
3D world.
This conversion from 3D to 2D is
referred to as a perspective
projection and is described by the
pinhole camera model.
It is common to model this projection OL and OR: the centers of symmetry
operation by rays that derive from the of the two cameras lenses.
X: the point of interest in both
camera, passing through its focal center. cameras.
Each emanating ray corresponds to a xL and xR: are the projections of
single point in the image. point X onto the image planes.
Epipolar Geometry 39
Epipole or Epipolar Point
Since the optical centers of the cameras
lenses are distinct, each center projects
onto a distinct point into the other
camera's image plane. These two image
points, denoted by eL and eR, are
called epipoles or epipolar points. Both OL and OR: the centers of symmetry
epipoles eL and eR in their respective of the two cameras lenses.
X: the point of interest in both
image planes and both optical cameras.
centers OL and OR lie on a single 3D xL and xR: are the projections of
line. point X onto the image planes.
Epipolar Geometry 40
Epipolar Line
The line OL–X is seen by the left camera
as a point because it is directly in line
with that camera's lens optical center.
However, the right camera sees this line
as a line in its image plane. That line
(eR–xR) in the right camera is called OL and OR: the centers of symmetry
an epipolar line. Symmetrically, the of the two cameras lenses.
X: the point of interest in both
line OR–X seen by the right camera as a cameras.
point is seen as epipolar line eL–xLby the xL and xR: are the projections of
left camera. point X onto the image planes.
Epipolar Geometry 41
An epipolar line is a function of the position of
point X in the 3D space, i.e. as X varies, a set
of epipolar lines is generated in both images.
Since the 3D line OL–X passes through the
optical center of the lens OL, the corresponding
epipolar line in the right image must pass
through the epipole eR (and correspondingly
for epipolar lines in the left image). All OL and OR: the centers of symmetry
epipolar lines in one image contain the of the two cameras lenses.
epipolar point of that image. In fact, any line X: the point of interest in both
which contains the epipolar point is an cameras.
epipolar line since it can be derived from some xL and xR: are the projections of
point X onto the image planes.
3D point X.
Epipolar Geometry 42
Epipolar Plane
As an alternative visualization, consider
the points X, OL & OR that form a plane
called the epipolar plane.
The epipolar plane intersects each
camera's image plane where it forms
lines—the epipolar lines. OL and OR: the centers of symmetry
of the two cameras lenses.
All epipolar planes and epipolar lines X: the point of interest in both
intersect the epipole regardless of cameras.
where X is located. xL and xR: are the projections of
point X onto the image planes.
 Epipolar Geometry 43
Epipolar Constraint & Triangulation
If the relative position of the two cameras is
known, this leads to two important
observations:
1. Assume the projection point xL is known,
and the epipolar line eR–xR is known and the
point X projects into the right image, on a
point xR which must lie on this particular OL and OR: the centers of symmetry
epipolar line. This means that for each point of the two cameras lenses.
X: the point of interest in both
observed in one image the same point must
cameras.
be observed in the other image on a known xL and xR: are the projections of
epipolar line. point X onto the image planes.
Epipolar Geometry 44
This provides an epipolar constraint:
The projection of X on the right camera
plane xR must be contained in
the eR–xR epipolar line. All points X
e.g. X1, X2, X3 on the OL–XL line will verify
that constraint.
It means that it is possible to test if two points
correspond to the same 3D point. Epipolar OL and OR: the centers of symmetry
constraints can also be described by of the two cameras lenses.
X: the point of interest in both
the essential matrix or the fundamental
cameras.
matrix between the two cameras. xL and xR: are the projections of
point X onto the image planes.
 Epipolar Geometry 45
2. If the points xL and xR are known,
their projection lines are also known.
If the two image points correspond to
the same 3D point X the projection
lines must intersect precisely at X.
This means that X can be calculated
from the coordinates of the two image OL and OR: the centers of symmetry
points, a process called triangulation. of the two cameras lenses.
X: the point of interest in both
cameras.
xL and xR: are the projections of
point X onto the image planes.
Examples: Epipolar Lines 46
Example: Epipolar Lines 47
Structure From Motion (SFM)
48
The number of camera poses and 3D points that can be
estimated depends on the number of images and the number of
unique feature points:
If you have two images captured by a stationary camera
observing a scene and each image contains six feature points.
Four of these feature points are observed in both images. Then
you need a minimum of two feature points that are observed in
both images.
If there are four images and each image contains eight unique
feature points, there will be four camera poses and eight 3D
points that can be estimated.