Using Many Cameras As One
Using Many Cameras As One
Robert Pless
Department of Computer Science and Engineering
Washington University in St. Louis,
Box 1045, One Brookings Ave, St. Louis, MO, 63130
pless@[Link]
Abstract
1
A raxel defines how a pixel samples the scene. This sam-
pling is assumed to be centered around a ray starting at a
point X,Y,Z, with a direction parameterized by (φ, θ). This
pixel captures light from a cone around that ray, whose as-
pect ratio and orientation is given by (fa , fb , Υ). The light
intensity captured may also be attenuated, these radiometric
quantities may differ for every pixel.
For the geometric analysis of multiple images, we sim-
plify this calibration so that it only includes the definition of
the ray that the pixel samples. This gives a simpler calibra-
tion problem which requires determining, for each pixel, the
Plücker vectors of the sampled line. Since Plücker vectors
are required for the mathematical analysis presented later,
the following section gives a brief introduction.
Figure 2. (Top) The generalized imaging model [13] ex-
presses how each pixel samples the light-field. This sam-
pling is assumed to be centered around a ray starting at a 2.2 Plücker Vectors
point X,Y,Z, with a direction parameterized by (φ, θ), rel-
ative to a coordinate system attached to the camera. The In order to describe the line in space that each pixel sam-
simplified model captures only the direction of the ray, pa- ples in this more general camera setting, we need a mech-
rameterized by its Plücker vectors q, q 0 . anism to describe arbitrary lines in space. There are many
parameterizations of lines, but Plücker vectors [18] give a
convenient mechanism for the types of transformations that
are required. The Plücker vectors of a line are a pair of 3-
lines [19].
vectors: q, q 0 , named the direction vector and moment vec-
The “generalized camera model” [13] is briefly intro-
tor. q is a vector of any length in the direction of the line.
duced in Section 2.1, and encompasses the imaging geom-
Then, q 0 = q × P , for any point P on the line. There are
etry of all the camera designs discussed above. The main
two constraints that this pair of vectors must satisfy. First,
contribution of this paper is to express a multi-camera sys-
q · q 0 = 0, and second, the remaining five parameters are
tem in this framework and then to derive the structure from
homogeneous, their overall scale does not affect which line
motion constraint equations for this model. The differen-
they describe. It is often convenient to force the direction
tial structure from motion constraints give an error function
vector to be a unit vector, which defines a scale for the ho-
which defines a minimization problem for motion estima-
mogeneous parameters.
tion (summarized in Section 3). Considering the Fisher In-
The set of all points that lie on a line with these Plücker
formation Matrix of this error function makes it possible to
vectors is given by:
give quantitative comparisons of different camera designs
in terms of their ability to estimate ego-motion (Section 4). (q × q 0 ) + αq, ∀α ∈ R. (1)
2
RKhx, y, 1i> , and a moment vector q × T . The imaging
geometry of the entire multi-camera system is represented q2 · (Rq10 + R(T × q1 )) + q20 Rq1 = 0.
simply by the collective list of the rays sampled by the indi- This completely defines how two views of a point constrain
vidual cameras. the discrete motion of a generalized camera. Using the con-
For a single camera, a differential motion on the im- vention that [T ]x is the skew symmetric matrix such that
age plane defines a differential change in the direction [T ]x v = T × v for any vector v, we can write:
of the sampled ray. For a camera not centered at the
origin, the differential change in the direction vector is: Generalized Epi-polar Constraint
∂q(x,y)
= RKhdx, 0, 0i> , and ∂q(x,y) = RKh0, dy, 0i> . T
dx dy
∂q 0 (x,y)
q2 T Rq10 + q2T R[T ]x q1 + q20 Rq1 = 0. (4)
The differential change in the moment vector is: dx =
0
RKhdx, 0, 0i> × T , and ∂q dy (x,y)
= RKh0, dy, 0i × T . >
In Section 3.2, this will be used to define the relation- For standard perspective projection cameras, q10 = q20 =
ship between optic flow measurements in each camera and 0, and what remains: q2T R[T ]x q1 = 0, is the classical epi-
the motion of the entire system. We proceed first, however, polar constraint defined by the Essential matrix. Strictly
with the constraints relating discrete motions of the camera speaking, this is not exactly analogous to the epi-polar con-
system to the image captured by each camera. straint for standard cameras. For a given point in one image,
the above equation may have none, one, several, or an infi-
nite number of solutions depending on what the exact cam-
3 Motion Estimation era geometry is. The name generalized epi-polar constraint,
however, is fitting because it describes how two correspond-
This section presents the structure from motion con- ing points constrain the relative camera motions.
straint equations for generalized cameras. We present first Given the camera transformation R, T and correspond-
the discrete motion equations — how a pair of correspond- ing points, it is possible to determine the 3D coordinates of
ing points constrain the rotation and translation between the world point in view. Using Equation 1, and transform-
camera viewpoints. Then we present the differential mo- ing the first line into the second coordinate system, solving
tion constraints relating continuous camera motions to optic for the position of the point in space amounts to finding the
flow. intersection of the corresponding rays. This requires solv-
ing for the parameters α1 , α2 , which is the corollary of the
3.1 Discrete Motion depth in typical cameras:
R((q1 × q10 ) + α1 q1 ) + T = (q2 × q20 ) + α2 q2
Suppose, in two generalized images, we have a corre-
spondence between pixel (x1 , y1 ) in the first image and Collecting terms leads to the following vector equation
pixel (x2 , y2 ) in a second image. This correspondence im- whose solution allows the reconstruction of the 3D scene
plies that the rays sampled by these pixels (hq1 , q1 0 i, and point P in the fiducial coordinate system:
hq2 , q2 0 i) must intersect in space. When the camera system
Generalized Point Reconstruction
undergoes a discrete motion, the fiducial coordinate systems
are related by an arbitrary rigid transformation. There is a α1 Rq1 − α2 q2 = (q2 × q20 ) − R(q1 × q10 ) − T,
rotation R and a translation T which takes points in the first
solve above equation for α1 , and use below
coordinate system and transforms them into the new coor-
dinate system. P = q1 × q10 + α1 q1
After this rigid transformation, the Plücker vectors of the
first line in the second coordinate system become: 3.2 Continuous Motion
hRq1 , Rq10 + R(T × q1 )i . (2) In the differential case, we will consider the image of a
point P in space which is moving (relative to origin of the
A pair of lines with Plücker vectors hqa , qa 0 i, and hqb , qb 0 i camera coordinate system) with a translation velocity ~t, and
intersect if and only if: an angular velocity ω~ . The instantaneous velocity of the 3D
point is:
qb · qa0 + qb0 · qa = 0. (3) ~ × P + ~t.
Ṗ = ω
For a point in space to lie on a line with Plücker vectors
This allows us to write down the constraint given by the hq, q 0 i, the following must hold true:
correspondence of a point between two images, combining
Equations 2 and 3 P × q − q 0 = ~0;
3
As the point moves, the Plücker vectors of the lines in- Generalized Optic Flow Equation
cident upon that point change. Together, the motion of the
point and the change in the Plücker vectors of the line inci- −(~t × q) × q + q̇ 0 × q − (~
ω · q)q 0
~ ×q+
q̇ = ω (9)
dent on that point must obey: α
∂
(P × q − q 0 ) = ~0, or,
dt For standard perspective projection cameras, q 0 = 0, and
0
Ṗ × q + P × q̇ − q̇ 0 = 0 q̇ = 0, and this simplifies to the standard optic flow equa-
tion (for spherical cameras):
In terms of the parameters of motion, this gives:
ω × P + ~t) × q + P × q̇ − q̇ 0 = ~0.
(~ (~t × q) × q
q̇ = −(~
ω × q) −
α
ω × P + ~t) × q = q̇ 0 − P × q̇.
(~ (5)
One approach to finding the camera motion starts by
which are constraints relating the camera motion and the finding an expression relating the optic flow to the motion
line coordinates incident on a point in 3D for a particu- parameters that is independent of the depth of the point.
lar rigid motion. On the image plane, the image of this This differential form of the epi-polar constraint is:
point is undergoing a motion characterized by its optic flow
(u, v). In Section 2.3, we related how moving across an
image changes the rays that are sampled. This allows us to q̇ × ((~t × q) × q) = −(~
ω × q) × ((~t × q) × q).
write how the coordinates of the Plücker vectors incident on
a point in 3D must be changing: This same process can be applied to generalized cam-
eras, giving:
∂q ∂q
q̇ = u+ v,
∂x ∂y Generalized Differential Epi-polar Constraint
∂q 0 ∂q 0
q̇ 0 = u+ v, (6) ~ × q) × ((~t × q) × q − q̇ 0 × q + (~
(q̇ + ω ω · q)q 0 ) = 0, (10)
∂x ∂y
so that we can consider q̇ and q̇ 0 to be image measurements. which, like the formulation for planar and spherical cam-
Then we can substitute Equation 1 into Equation 5 to get: eras, is bilinear in the translation and the rotation and in-
dependent of the depth of the point. The next section in-
ω × ((q × q 0 ) + αq) + ~t) × q = q̇ 0 − ((q × q 0 ) + αq) × q̇.
(~ troduces a tool to calculate the sensitivity to noise of this
or, constraint for any specific camera model.
ω ×(q×q 0 ))×q+α(~
(~ ω ×q)×q+~t×q = q̇ 0 −(q×q 0 )×q̇−αq×q̇. 3.3 Fisher Information Matrix
(7)
The following identities hold when |q| = 1, Let p~ be a vector of unknowns, (in our problem,
0
ω × (q × q )) × q = (~
(~ 0
ω · q)(q × q ), and, h~t, ω
~ , α1 , . . . αn i), and Z be the set of all measurements (in
our case the u, v that are used to calculate q̇, q̇ 0 using equa-
ω × q) × q) = (~
(~ ω · q)q − ω~,
tion 6). The Fisher Information Matrix, F , is defined to
0 0
(q × q ) × q̇ = −(q · q̇)q, be [22]1 :
Simplifying Equation 7 and collecting terms gives: >
∂ ln p(Z|p) ∂ ln p(Z|p)
0 0 0 F = E[ ].
ω ·q)q −~
α((~ ω +q × q̇)+~t×q = q̇ +(q · q̇)q −(~ω ·q)(q ×q ) ∂p ∂p
(8)
This vector equation can be simplified by taking the cross For any unbiased estimator of the parameter p, the
product of each side with the vector q, which gives: Cramer-Rao inequality guarantees that the covariance ma-
trix of the estimator, (E[(p − p̂)(p − p̂)> ]), is at least as
ω × q) + (~t × q) × q = q̇ 0 × q − (~
−α(~ ω · q)q 0 − αq̇. great as the inverse of the Fisher information matrix:
Collecting the terms that relate to the distance of the point E[(p − p̂)(p − p̂)> ] ≥ F −1 .
along the sampled ray, then dividing by α gives:
If we model the probability density function of errors
−(~t × q) × q + q̇ 0 × q − (~
ω · q)q 0
ω × q + q̇) =
(−~ in the motion field measurements as Gaussian distributed,
α 1 This presentation follows [8] and [21], which use the Fisher Informa-
which can be written more cleanly to define the optic flow tion Matrix to study ambiguities and uncertainties for motion estimation
for a generalized camera under differential motion: with standard cameras.
4
zero mean, and covariance σ 2 I, then the Fisher Information 4 Analysis
matrix can be written:
Different visual system designs are appropriate for dif-
1 X ∂hi > ∂hi ferent environments. Within the context of estimating ego-
F(~t,~ω,α1 ,...αn ) = 2 ,
σ i=1...n ∂p ∂p motion, the relevant parameters are the system motion and
the distances to points in the scene. The environment is de-
where hi is the function that defines the exact optic flow at fined as the distribution of these parameters that the system
pixel i for a particular set of motion parameters and scene will experience. This is a distribution over the set of param-
depths (these subscripts are to remind the reader that F is eters (~t, ω
~ , α1 , . . . αn ). In our simulation, we choose this
dependent on the these particular parameters). distribution D as follows:
To determine the form of hi , we start with the general- • ~t chosen uniformly such that |~t| < 1.
ized optic flow equation:
• ω
~ chosen uniformly such that |~
ω | < 0.01.
−(~t × q) × q + q̇ 0 × q − (~
ω · q)q 0 1 1
~ ×q+
q̇ = ω • αi chosen uniformly such that 0 ≤ αi ≤ 1.
α
Each point in this distribution has an associated Fisher
and multiply through by the α and re-arrange terms to Information Matrix, the (randomized, numerical) integra-
get: tion of this matrix over many samples from this distribution
is a measure of the fitness of this camera system in this en-
q̇ 0 × q − αq̇ = −α~ ω · q)q 0
ω × q + (~t × q) × q + (~ vironment:
X
FD = F(~t,~ω,α1 ,...αn )
Equation 6 defines how moving on the image plane
(~
t,~
ω ,α1 ,...αn )∈D
changes the Plücker coordinates of the ray being sampled.
Define A and B to be the pair of 2 × 3 Jacobians relating Figure 3 displays FD for a number of different camera
motion on the image to changes in the direction of sampled designs. Each of these situations merits a brief discussion:
ray such that:
1. A standard video camera with a 50◦ field of view.
q̇ = ~uA, The covariances reflect the well known ambiguities be-
tween some translations and rotations.
q̇ 0 = ~uB,
2. The flow fields from a stereo pair have the same ambi-
where ~u =< u, v > is the optic flow measured on the im- guities as a single camera. The covariance matrix has
age. Then we can rewrite the generalized optic flow equa- a larger magnitude because two cameras capture twice
tion as: the amount of data. This matrix (and these ambigui-
ties) do not apply to algorithms which use correspon-
~uB[q] − α~uA = −α~ ω · q)q 0 , or
ω × q + (~t × q) × q + (~ dence between the stereo imagery as part of the motion
~u(B[q] − αA) = −α~ω × q + (~t × q) × q + (~ω · q)q 0 (11) estimation process.
3. A “stereo” camera pair in which one camera looks
Defining C = (B[q] − αA)> ((B[q] − αA)(B[q] − backwards does not have the standard rotation-
αA)> )−1 , we can then isolate the optic flow and get the translation ambiguity, but has a new confusion be-
terms we need to define the Fisher Information Matrix2 : tween rotation around the axis connecting the cameras
and rotation around the common viewing axis of the
hi = ~u = (α~ ω · q)q 0 )C.
ω × q + (~t × q) × q − (~ cameras.
It is thus possible to create an analytic form giving 4. A camera system with cameras facing in opposite di-
the Fisher Information Matrix for a particular environment rections but aligned along their optic axis is perhaps
(camera motion and scene structure). In the following sec- the best design for two standard cameras. The “line”
tion we use this to measure of how well a particular vision ambiguity (discussed under camera design 1) is still
system can measure its ego-motion in a variety of environ- present
ments. 5. Three cameras aligned along three coordinate axes
2 Recall that the matrices A, B, C, and the Plucker vectors q, q 0 are show a small covariance between translation and rota-
different at each pixel, we have dropped the subscripts to facilitate the tion. This may be an artifact of the strong covariances
derivations between each camera pair.
5
6. Six cameras placed to view in opposite directions
along coordinate axes show no ambiguities.
7. Six cameras placed to view in every direction, but
not as matched pairs along each viewing axis. This
situation models the “Argus Eye” system [3], which
reported very accurate estimates of system rotation,
again arguing that the covariance shown here may be (1)
an artifact of the strong covariance (in rotation) be-
tween each camera pair.
These results fit published empirical data which finds
that system rotation and translation is much better con-
strained for a multi-camera system (the Argus Eye), than (2)
for single cameras [3, 2] (see Figure 4).
In this work we have carried out an analytic comparision
of different types of cameras (as opposed to different algo-
rithms) for the ego-motion estimation problem. As tech-
nology and computational power increase, the effectiveness (3)
of visual algorithms will be limited only by inherent sta-
tistical uncertainties in the problems they are solving. The
Fisher Information Matrix is a powerful analysis technique
that can apply to any problem which involves searching for
a parameter set that minimizes an error function. Design- (4)
ing camera systems optimized for particular environments
is necessary to make computer vision algorithms successful
in real world applications.
References (5)
6
[17] Tomas Pajdla. Stereo with oblique cameras. International
Journal of Computer Vision, 47(1):161–170, 2002.
[18] J. Plücker. On a new geometry of space. Philisophical Trans-
actions of the Royal Society of London, 155:725–791, 1865.
[19] Steven M Seitz. The space of all stereo images. In Proc. In-
ternational Conference on Computer Vision, volume I, pages
26–33, 2001.
Figure 4. (Middle) The Argus eye system [3] was con- [20] H Y Shum, A Kalai, and S M Seitz. Omnivergent stereo. In
structed as a system of cameras, but can also be modelled Proc. International Conference on Computer Vision, pages
as a single non-central projection imaging system. Empiri- 22–29, 1999.
cal studies demonstrate that ego-motion estimation is better
constrained for the Argus system than a single camera [2]. [21] S. Soatto and R. Brockett. Optimal structure from motion:
(Left) A sphere represents the set of all possible transla- Local ambiguities and global estimates. In CVPR, 1998.
tion directions. Each point is colored by the error in fitting [22] H W Sorenson. Paramerer Estimation, Principles and Prob-
image measurements to that translation direction (darker is lems. Marcel Dekker, New York and Basel, 1980.
higher error). For a single camera, this error surface has an [23] Peter Sturm. Mixing catadioptric and perspective cameras.
extended minimum region, indicative of the difficulty in de- In Proc. of the IEEE Workshop on Omnidirectional Vision,
termining the exact translation. (Right) Using all cameras 2002.
simultaneously, the best translation direction is very well
constrained. [24] T Svoboda, T Pajdla, and V Hlavac. Epipolar geometry for
panoramic cameras. In Proc. European Conference on Com-
puter Vision, 1998.
[8] K. Daniilidis and M. E. Spetsakis. Understanding noise sen- [25] Rahul Swaminathan, Michael Grossberg, and Shree Nayar.
sitivity in structure from motion. In Y. Aloimonos, editor, Caustics of catadioptric cameras. In Proc. International Con-
Visual Navigation: From Biological Systems to Unmanned ference on Computer Vision, 2001.
Ground Vehicles, chapter 4. Lawrence Erlbaum Associates,
Mahwah, NJ, 1997.
[9] Christopher Geyer and Kostas Daniilidis. Structure and mo-
tion from uncalibrated catadioptric views. In Proc. IEEE
Conference on Computer Vision and Pattern Recognition,
2001.
[10] Christopher Geyer and Kostas Daniilidis. Properties of the
catadioptric fundamental matrix. In Proc. European Confer-
ence on Computer Vision, 2002.
[11] Joshua Gluckman and Shree Nayar. Planar catadioptic
stereo: Geometry and calibration. In Proc. IEEE Confer-
ence on Computer Vision and Pattern Recognition, volume 1,
1999.
[12] Joshua Gluckman and Shree K. Nayar. Ego-motion and om-
nidirectional cameras. In ICCV, pages 999–1005, 1998.
[13] Micheal D Grossberg and Shree Nayar. A general imaging
model and a method for finding its parameters. In Proc.
International Conference on Computer Vision, volume II,
pages 108–115, 2001.
[14] Rajiv Gupta and Richard Hartley. Linear pushbroom cam-
eras. IEEE Transactions on Pattern Analysis and Machine
Intelligence, 19(9):963–975, 1997.
[15] R Andrew Hicks and Ruzena Bajcsy. Catadioptric sen-
sors that approximate wide-angle perspective projections.
In Proc. IEEE Conference on Computer Vision and Pattern
Recognition, pages 545–551, 2000.
[16] Randall Nelson and John Aloimonos. Finding motion pa-
rameters from spherical flow fields (or the advantage of hav-
ing eyes in the back of your head). Biological Cybernetics,
58:261–273, 1988.