IMAGE PROCESSING

Construction of holographic images using cross

sample modified laplacian method

Abstract—This paper presents a important issues in computer vision.

shape-from-focus method,which is Techniques executing this task are often
improved with regard to the referred to as shape-from-x, where x is a
mathematical operator used for visual cue such as shading, texture,
contrast measurement, the selection of contour, focus, stereo, and motion [1].
the neighborhood size, surface Blurring phenomena due to defocusing
refinement through interpolation, and are among the important cues for depth
surface postprocessing. Three- recovery. Shape-from focus is a method
dimensional models of living human where a sequence of images is taken
faces are presented with such a high while changing the focus setting in small
resolution that single hairs are visible. steps. One image of such a sequence,
Key words—Autofocusing, depth further on referred to as focus series,
estimation, depth-from focus, facial contains a focused region of the imaged
measurement, focus analysis, focusing, object as well as an unfocused
focus measure, holography, shape background. By determining the focus
estimation, shape-from-focus, 3-D setting, that locally optimizes image
reconstruction. focus, 3-D information about the imaged
object is retrieved [2]. If the object is
moving and cannot be expected to
I. INTRODUCTION remain in a certain position during the
capturing time, an alternative method for
THE acquisition of 3-D information out 3-D measurement is holography, where
of 2-D images is one of the most part of the 3-D information about the
recorded object is stored by This is done optically, if thehologram
superimposing the back-scattered light was stored on an analogue recording
coherently with a reference wave. material. It is realized by illuminating
Through this, the amplitude and the the hologram with the complex
phase information of the scattered light conjugate reference beam, which
field are stored as an interference produces the so-called real image, a 3-D
pattern, for example in an analogue light field with one-to-one
photosensitive material. This kind of correspondence to therecorded object. It
acquisition of 3-D data has the has to be digitized into a set of 2-D
tremendous advantage of storing the projections, because of the 2-D nature of
high-resolved information in an digital recording [Link] result of
extremely short amount of time, namely the holographic reconstruction is a set of
in the duration of the recording laser images containing focused and
pulse (35 ns for the setup described unfocused regions similar to the focus
here). However, the only way to get series gained through subsequent
access to the stored information is recording. This is why the same methods
reconstructing the hologram. designed for shape-from-focus in
microscopy are applicable for
holographic reconstructions [3]–[7].

This paper discusses improvements of

shape-from-focus methods, applicable to
all kind of focus series. The application,
Fig. 1. Optical reconstruction of the that provides the exemplary material for
hologram and digitalization of a set of the discussion, is the analogue
2-D projections at different axial holographic recording of living human
positions. faces. For that kind of application, the
full advantage of the extremely short
recording time arises since motion
artifacts caused by changes of the face
due to mimic, breathing, or heartbeat can plane with the coordinate . The
be avoided. The completely focused distinction between focused and
image, extracted from the focus series in unfocused image parts is realized with a
combination with the 3-D information, so-called focus criterion, a mathematical
serves as precisely fitting texture. Such operator which measures the local image
highly resolved textured 3-D models of contrast of each pixel in each image with
living human faces are of. great interest the axial position with the aid of a local
in many kinds of applications, among pixel neighborhood. These values are
others in medicine and forensicscience further on referred to as focus values. A
[8]–[14]. maximization of the focus values along
the direction localizes the focused
regions and produces information about
the 3-D geometry of the imaged object
(see Fig. 2). This process is further on
called surface extraction. In addition to
the topometrical data, the procedure can
be used to generate an image with
unlimited depth of field by compounding
the detected sharp patches from each
image into one image described by . This
so-called texture image can be used for
texturing the resulting 3-D computer

Fig. 1(a) shows exemplarily one image models. The pixel-to-pixel

from the focus series and (b) screenshot correspondence between texture and

of the resulting, textured 3-D computer geometry, which is inherent to the

model of the face. described process, makes any further

. registration between the model and the

SHAPE-FROM-FOCUS texture expendable [see Fig. 1(b)].

The considered focus series may consist Pioneering work in microscopic shape-

of gray-scale images, each characterized from-focus was done by Nayar [2]. It is

by their intensity distribution in the used for a wide range of applications

like the measurement of firearm bullets the digital image and assigning a
and cartridge cases and for machine quantitative value to a whole image or a
tools for inspection purposes [15], for specified part. In shape-from-focus, a
material structure analysis in sharpness value is assigned to every
mineralogical research [16] or for micro pixel. It is calculated using a local pixel
structures such as a micro-cogwheel[17]. neighborhood. To find a suitable focus
The optimization of the depth-form- criterion for holographic facial
focus procedure depends strongly on measurement, a quantitative comparison
thenatural frequency content of the of the extracted facial surface was
texture of the object to be recorded. For performed with the help of a simulated
objects with low intrinsic contrast, active focus series with different amounts of
illumination can be used [18]. In the case added noise. The evaluated criteria
of human faces, the intrinsic contrast of were among others Tenengrad [31],
the skin is sufficient for shape-from [19], Squared Gradient, Absolute
focus. Gradient, Sum Modified Laplacian
OPTIMIZED FOCUS CRITERIA (SML) [2], Variance, and Cross Sum
The quintessence of the depth-from- Modified Laplacian (XSML). The latter
focus procedure is thedetection of is a newly developed modification of
focused image parts, the same task that Nayars SML operator, presented for the
is performed by auto focus algorithms first time in this paper. The Laplacian in
used in many different applications. The the SML takes only direct neighbors in
performance of the different auto focus `the x- and y-directions into account.
algorithms depends strongly on the Additionally, also the diagonal
properties of the object to be focused on, neighbours could be included, weighted
as well as those of the imaging with a factor of to compensate for their
[Link] of research was performed larger distance to the central point where
for finding the suitable algorithm for the is a square pixel neighborhood of the
different applications [19]–[26]. The parameter can be adjusted to the
core of an autofocusing algorithm is a characteristic size of texture elements in
mathematical operator, a focus criterion, the recorded scene.
working on the pixel representation of
A detailed discussion on the criteria and
their performance would go beyond the
scope of the paper. An in-depth study
can be found in [7]. It demonstrates that
the newly invented XSML criteria Fig. 2. For each fixed lateral point (x,y),
produces the lowest deviation between the focus values F(x,y,z) are
the extracted and a reference surface for maximized,
any amount of added noise. The XSML which leads to 3-D geometry
operator shows a slightly higher information about the object.
robustness against noise than the SML- On the other hand, small neighborhoods
criterion due to the increased number of are very sensitive to noise. The lower the
points used for the calculation. signal-to-noise ratio, the larger the
ADAPTIVE SELECTION OF neighborhood should be. Even in the
NEIGHBORHOOD SIZE absence of noise, the minimal
The choice of the neighborhood size is a neighborhood size is limited to a certain
critical step in the procedure of surface extent by the feature size of the texture
extraction. The lateral resolution of the of the recorded object. With a
ex- tracted surface is limited by the size neighborhood smaller than any feature it
of the neighborhood, since structures is impossible to distinguish between
smaller than this size cannot be resolved. focused or defocused image regions.
Thus, the neighborhood size should be When choosing a neighborhood size,one
chosen as small as possible. always has to compromise between
lateral resolution and robustness against
noise. . In regions of high intrinsic
contrast, like the mustache of the
proband, a high-quality surface can be
extracted using a neighborhood size of 3
 neighborhood size d along the direction
Z and finding the

Fig. 4. Histograms of confidence values

for a typical facial data set calculated
with a neighborhood size of 3 (a), 5 (b),

Fig. 3. Exemplary height maps extracted and 11 pixels (c). An increasing

from a holographic data set with a neighborhood size leads to the

neighborhood size of 3 (a) and 17 (b) appearance of a second peak belonging

to reliable surface points.

pixelspixels with a lateral resolution so maximum. The higher the maximum in

high that even single hairs can be comparison to the mean sharpness value,

distinguished. In regions of lower the more reliable is the generated surface

intrinsic contrast, however, the surface [Link] defined as follows:

extracted with a neighborhood size of 3 (1)

is affected by noise. Using a with

neighborhood size of 17 pixels on the (2)

other hand allows for a noise resistant The normalization leads to confidence

surface extraction in regions of lower values between zero and one, a higher

intrinsic contrast, while the lateral value indicates a more reliable surface

resolution is reduced. To overcome this point. The confidence values of one

tradeoff, an adaptive algorithm was surface point differ for different

developed choosing locally the smallest neighborhood sizes. The concept of the

possible neighborhood size so that the adaptive algorithm is to start with the

surface is not corrupted by noise. It is smallest neighborhood size and accept

based on the definition of a confidence extracted surface points only if they are

value for each found surface point and reliable. If not, a larger neighborhood is

several neighborhood sizes . The surface considered. Reliability can be defined

extraction is done by comparing with the help of the confidence values by

sharpness values calculated with a

selecting a threshold . The thresholds are [Link] the maximal
determined by regarding the histogram neighborhood size to a certain value,
of the confidence values created with there are still points where no reliable
different neighborhood sizes for a surface value could be found. In order to
typical data set of a recorded face. fill these gaps, the surface extracted with
the maximal neighborhood can be
Fig. 4 shows such histograms for a smoothed by weighted averaging.
neighborhood size of 3 pixels (a), 5 The confidence values act thereby as
pixels (b), and 11 pixels (c). For a small weights. These values are used to fill the
neighborhood size, very few reliable gaps if desired. The result can be seen in
surface points are present. The histogram
shows mainly a normal distribution of Fig. 6. It shows a surface not corrupted
confidence values of unreliable points by noise with the maximal achievable
with a slight asymmetry caused by the lateral resolution. 4An automated
reliable points. Increasing the size of the selection is also possible, but is not
neighborhood, the number of reliable necessary, as is explained in the
points increases,so that a second peak following.
appears with its center shifting to higher . Fig. 5. Threshold � against
confidence values. Unreliable points can neighborhood size with exponential fit.
be excluded if the threshold is chosen The thresholds were averaged over 25
such that the first peak is eliminated. In values, the error bars represent the
this manner, the neighborhood size standard deviation.
depending thresholds were determined
for 25 different data sets manually.4

Fig. 5 shows the mean thresholds, as

well as an exponential fit. The resulting
curve is further on used for automated
threshold selection and the distinction
between reliable and unreliable surface
Fig. 6. Height map created with the
points without manually analyzing the
presented algorithm corresponding to the
height maps created with a fixed represents the threshold for points to be
neighborhood size shown in Fig. 3. considered.
tions. Thus, focus information is also
available only at discrete positions and
with it surface information. Hence, the
choice of the interslice distance limits
the maximal achievable axial resolution.
An interpolation of the discrete focus
Fig. 7. Section of the texture extracted
values would provide continuous
from a holographically recorded face
information and consequently a
digitized with a resolution of 40 _m (a).
potentially higher resolution. Nayar [2]
A screenshot of the 3-D computer model
proposed an interpolation method
of the marked section is shown in (b).
based on a Gaussian distribution (3)
Another example can be seen in Fig. 7. It
By choosing the maximal focus value
shows a section of the texture image
and its two neighbors, the unknown
extracted form a holographic recording,
parameters and can be calculated just by
digitized with a resolution of 40 m. The
solving a system of algebraic equations.
adaptive approach presented in this
With this, continuous values are
paper permitted the creation of a closed
produced that are not restricted anymore
surface with a local resolution so high
to [Link]
that single hairs are visible in the 3-D
alternative approach is to fit the
model.
Gaussian model mentioned in (3) to a
SURFACE INTERPOLATION
particular chosen number of points. For
The set of 2-D images, from which the
the selection of points used for the fit it
surface information is extracted,
was found best to first determine the
represents the holographical image at
height of the maximum above the
discrete position.
average focus values and consider only
points in the upper half of that interval
(dashedline in Fig. 8). Additionally, the
Fig. 8. Exemplary focus profile with
points have to lie in an interval of 20
fitted Gaussian curve. The dashed line
tomograms before and behind the
maximal value, to eliminate noisy data appear if using large neighborhood sizes.
points coincidentally lying above the An example can be seen in the height
threshold but far away from the profile in Fig. 9. The steps in the height
maximum. The fitting algorithm was map are much larger than the interslice
implemented using a downhill simplex [Link] steps are lessened
method in multidimensions[27, p. 408] considerably through the Gaussian fit, as
proposed by Nelder and Mead [28]. Fig. can be seen in Fig. 9. The height profile
8 shows the whole profile with the fitted gained with the fit is shifted for better
curve. The center of the Gaussian serves comparability.
as new [Link] interpolation SURFACE POSTPROCESSING
method is based upon three focus values The previous sections presented new
exactly met by the curve, while the fit methods invented for the improvement
method takes up to twenty points into of the classical depth-from-focus
account and finds thecurve by [Link], despite sophisticated
minimizing the overalldeviation. That surface extraction methods,little noisy
makes the fit more accurate and more surface fluctuation cannot be avoided
stable in the presence of noise. However, [Link] denoising is done with
its computational effort is much higher low-pass filtering, which reduces
since an iterative optimization step is noise, but also blurs sharp features and
[Link] holographic facial details. We found that a smoothing
measurement, where computational time method proposed by Tasdizen [29] based
is not as important as in real time on anisotropic diffusion of normals
applications, the Gaussian fit proved worked perfectly for the postprocessing
to be a useful tool for surface of facial [Link] diffusion of
refinement. A quantitative comparison normals is the natural expansion of the
of the interpolation methods mentioned anisotropic diffusion method for 2-D
above and an additional one can be images introduced by Perona and Malik
found in [7].Additionally to the [30] to curved surfaces in the 3-D space.
increased robustness against noise, the Smoothing in regions of high total
fit also eliminates step like structures in curvature is suppressed through the
the extracted surface,which especially introduction of a nonconstant diffusion
coefficient depending on the total This paper demonstrated that
curvature. the classical shape-from-focus
method is perfectly suited for
the extraction of 3-D geometry
and texture information from
holographic reconstructions,
especially
Fig. 10 shows a facial model gained with for the creation of highly
the presented adaptive algorithm and resolved facial [Link]
Gaussian fitting without any evaluated several mathematical
postprocessing (a), with isotropic operators for contrast
smoothing (b) and with anisotropic measurement and found that
diffusion of normals.5 While small the best criterion for this
features are almost obliterated in (b) they application is the newly
are nearly as pronounced in (c) as in the
developed XSML criterion, a
raw [Link] generation of continuous
modification of SML operator
surface data through the Gaussian fit or
proposed by Nayar [2].
Gaussian interpolation is obligatory for
Additionally, we presented an
the appliance of the anisotropic diffusion
algorithm for the automated
of normals. For discrete surfaces, as they
adaption of the neighborhood
originate from classic shape-from-focus,
size to the conditions of the
the anisotropic smoothing does not
data.
behave [Link] all improvements
presented in this paper applied, the
computation time on a standard PC for
facial models as shown in this paper
increases from 1 min for the standard
procedure to approximately 30 min,
whereas the Gaussian fitting requires the
most of the additional time

Conclusions
The neighborhood size is chosen
as small as possible while
preventing
corruption through noise. With
this, an optimal lateral
resolution can be achieved. It
was demonstrated that in facial
measurement single hairs can
be visualized through this
procedure. It was also shown that a Fig. 11. Schematic top view, side view,
Gaussian fit to the focus profile and photograph of the holographic
improves the quality of the surface camera for portrait holography.
tremendously. Artifacts are
eliminated and the influence of noise is REFERENCES
reduced, while at the same time a
continuous surface not restricted to the [Link] journals and magazines&website
digitization positions is created. This is [Link] search engine
compulsory for the appliance of
anisotropic smoothing methods, which
were also presented in this paper.
The holographic method for facial
measurement is superior to conventional
measurement techniques since it
captures 3-D information of the face
with one single laser pulse of 35-ns
[Link] that extremely short amount
of time, no movement artifacts can occur
and highly resolved facial models can be
created for various applications.