CSE578: Computer Vision
Spring’20 2-View Geometry
Anoop M. Namboodiri
Center for Visual Information Technology
IIIT Hyderabad, INDIA
Case 1: Planar World
Plane: Z=0 X
x1 x2
What is the relation between x1 and x2?
Case 1: Planar World
• Projection equation of points on a plane:
⎡ X ⎤
⎢ ⎥ ⎡ X ⎤
Y ⎢ ⎥
x = K⎡⎣ R t ⎤⎦ X = K⎡⎣ r1 r2 r3 t ⎤⎦⎢ ⎥ = K⎡ r1
⎣ r2 ⎤
t ⎦⎢ Y ⎥ = HX,
⎢ 0 ⎥
⎢ ⎥ ⎢ 1 ⎥
1 ⎣ ⎦
⎣ ⎦
where H is a 3 × 3 non-singular matrix
• Now if we consider two different views of the
same world point, we get:
−1
x1 = H1X; x 2 = H 2 X = H 2 H x = H 21x1. 1 1
x1 = H12 x 2 ; x 2 = H 21x1
Case 2: Same Camera Center
X Arbitrary
world
x1 x2
What is the
relation between
x1 and x2?
C
Case 2: Same Camera Center
• Projection equation for two cameras with same C:
x1 = K1R1⎡⎣ I -C ⎤⎦ X
x 2 = K 2 R 2 ⎡⎣ I -C ⎤⎦ X
= K 2 R 2 ( K1R1 ) K1R1 ⎡⎣ I -C ⎤⎦ X
−1
−1
= K 2 R 2 ( K1R1 ) x1
= H 21x1
where H is a 3 × 3 non-singular matrix
x1 = H12 x 2 ; x 2 = H 21x1
Case 3: Generic World and Cameras
x1 x2
l1 l2
o1 e1 e2 o2
Epipolar Geometry
X • All world points that map
to x1 in I1 (pre-image of x1)
I1 I2
map to a line l2 in I2,
called an epipolar line.
x1 (vice-versa)
x2
l1 l2 • The image of o1 in I2 (e2) is
an epipole. (vice-versa)
o1 e1 e2 o2
• The plane containing these is called the epipolar plane.
• These result in a set of constraints, which are referred to
as the epipolar constraints and the resulting geometry is
called the epipolar geometry.
Epipolar Constraint: Essential Matrix
X λ1x1 = X
I1 I2 λ2 x 2 = RX + T
= R ( λ1x1 ) + T
x1 x2
l2
T̂λ2 x 2 = T̂Rλ1x1 + 0
l1
e1 e2
λ2 x T2 T̂x 2 = λ1x T2 T̂Rx1
o1 o2
x T2 T̂Rx1 = 0
T T
x Ex1 = 0 or x Ex 2 = 0
Cross product of any 2 1
two vectors: AxB can ⎡ 0 −Az Ay ⎤
be written as the ⎢ ⎥
matrix product ÂB, Â = ⎢ Az 0 −Ax ⎥
where: ⎢ ⎥
⎢⎣ Ay Ax 0 ⎥
⎦
Epipolar Constraint: Fundamental Matrix
X
I1 I2 x1 = K1X
x2 = K2X
x1 x2 T −T −1
l1 l2 x K T̂RK x = 0
2 2 1 1
o1 e1 e2 o2 T
x Fx 2 = 0
1
Both Essential and Fundamental matrices are 3x3
and are independent of the world point.
Summary
1. Two views of a planar world are related by a
homography. (H: non-singular 3x3 matrix)
2. Two views or an arbitrary world from the same
camera position are related by a homography.
3. Two views of an arbitrary world are constrained by
epipolar geometry
1. Strong Calib.: Essential Matrix: E; 𝑥&' 𝐸𝑥) = 0
2. Weak Calib.: Fundamental Matrix: F; 𝑥& ' 𝐹𝑥) = 0
Stereo
1. We see a slightly
different image of
the world through the
two eyes.
2. The shift in image is
proportional to the
the distance to the
object.
3. The roadside trees are
left behind, while the
farther mountains
follow you.
[Link]
Stereo
Z f
xr
Object br
baseline
Right camera b
bl
xl
Left camera
𝑏. 𝑏1 𝑥. 𝑥1 𝑓. 𝑏
+ = + Z=
𝑍 𝑍 𝑓 𝑓 (𝑥. + 𝑥1 )
Stereo Geometry
• Farther the point,
smaller the disparity
and vice versa
• A large baseline can
give more reliable
estimates of depth.
But, matching may
become harder
• Basic step: Identify common points in camera views
o How do we find the images of a single world point in two
views?
o Do they have the same intensity/color?
o What else can be same/similar?
Identifying Common Points
• Find a world point in 2 or more views
• Appearance is the only clue to identify them
• Individual pixel colors are similar very often.
However, the match is too noisy.
• Match a (small) neighborhood of colors from
one image to a similar neighborhood in others
• Will work if local surface is fronto-parallel and
images have similar magnification
• Foreshortening can happen when viewing
an oblique surface
• Many ambiguities. We need a lot of help!
Some Examples
Matching Patches
• Compare 𝑚×𝑚 patches from two views.
• Form vectors 𝑣 and 𝑣′ of length 𝑚) from them
• Matching scores between patches:
• Sum of Absolute Difference (SAD): 𝑣 − 𝑣′ &
• Sum of squared difference (SSD): 𝑣 − 𝑣′ )
>
<= <
• Correlation:
>
<>< <= <=
>
<? = <?
• Normalized correlation: . Range: [−1, 1].
>
<? > <? <? = <? =
where 𝑣̅ and 𝑣’̅ are vectors with respective patch-mean
colour subtracted.
• Invariant to affine changes in intensity/colour.
Constraints on Matching
• Epipolar: Match lies on the epipolar line of the pixel
• Colour Constancy: The appearance/colour does not change from
one view to another
• Uniqueness: A point on left image can match with only one point
on the right and vice versa
• Ordering or Monotonicity: If point A is to the left of B in view 1, it will
to the left of B in view 2 also. (Violated if great difference in depth
exists)
• Continuity: Disparity values vary smoothly (violated at occlusion
boundares)
• Sparse correspondence: only for good feature points
• Dense correspondence: a match for every pixel
Epipolar: Reduced Search and Rectification
• The search is limited to a line if fundamental matrix is known
(weakly calibrated)
• Simplest if left and right cameras have same image plane and
pure X-translation between them.
o Fundamental matrix has a simple form and Epipolar constraint
reduces to 𝑦′ = 𝑦.
o Matches constrained to lie in the same scan line
• Rectification: A rotation of the camera (to make image planes
parallel) and a change in K matrix (focal length, image
center).
o Can be represented using a homography 𝐻 to align one image
plane to the other
o Or, homographies 𝐻& , 𝐻) to align them to a third plane.
Stereo Rectification
• Reproject images
onto a common
plane parallel to the
line between the
optical centers
• Pixel motion is
horizontal after this
transformation
C. Loop and Z. Zhang
“Computing Rectifying
Homographies for Stereo
Vision”, IEEE Computer Vision
and Pattern Recognition 1999
Rectification: Example