See discussions, stats, and author profiles for this publication at: [Link]

net/publication/328685112

Projection-based Augmented Reality in Engineering Applications

Conference Paper · March 2002

CITATIONS READS

4 3,152

3 authors, including:

André Stork Raffaele de Amicis

Fraunhofer Institute for Computer Graphics Research IGD Oregon State University
161 PUBLICATIONS   1,320 CITATIONS    190 PUBLICATIONS   1,400 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

status View project

Cloudifacturing View project

All content following this page was uploaded by Raffaele de Amicis on 02 November 2018.

The user has requested enhancement of the downloaded file.

 Stork et. al. PBAR in Engineering Applications

Projection-based Augmented Reality in Engineering Applications

André Stork, Fraunhofer-Institut für Graphische Datenverarbeitung, Darmstadt

Oliver Bimber, Fraunhofer Center for Research in Computer Graphics, Providence, RI
Raffaele de Amicis, GRIS, Department of Computer Science, Technical University, Darmstadt

Abstract

Augmented Reality (AR) superimposes computer-generated graphics onto the user's view of the real
world. Today, AR applications can occasionally be found in the late phases of the product development
process (PDP), such as in training and in maintenance. Head-mounted displays (HMDs) are typically used
for AR applications. HMDs still show deficiencies in terms of ergonomics and image quality. Looking at
the development of VR technology, it can be stated that VR began its triumphal procession only after
replacing HMDs by immersive projection technology (IPT) such as PowerWalls, CAVEs, Virtual Tables.
Projection-based Augmented Reality enhances such environments towards augmented reality applica-
tions. We believe that this concept opens new application possibilities for AR – especially in the early
phases of the product development process.
In this paper we will describe the concept of Projection-based Augmented Reality (PBAR), introduce
current hardware prototypes and discuss applications of PBAR configurations within the following areas:
augmented design review, hybrid assembling simulation, hybrid modeling/sketching and visual inspection
of augmented physical parts.

Introduction

Head-attached displays (such as head-mounted displays) are the traditional output devices for Aug-
mented Reality. They have first been developed in the mid-sixties and still today own the display mono-
pole in AR field. However, in contrast to the well established projection technology that is mainly applied
for Virtual Reality (VR) applications, head-mounted displays have barely improved over the previous
years. HMDs still show deficiencies in terms of ergonomics and image quality, i.e. resolution, viewing
angle, etc. (see Fig. 1). Current models are still far away from being “ultimate displays“. High-quality,
lightweight, and inexpensive, 3D see-through HMDs are not available today. The compromises which
have to be made to the hardware-technology hinder AR from having its break-through. Alternative ap-
proaches such as video-mixing are evaluated and further developed by some researchers. Video-mixing
superimposes the computer-generated information, e.g. 3D graphics, onto a video stream in the back-
ground. This approach supports mutual occlusion between real and virtual objects and allows for a pixel-
precise registration of both environments. However, due to the technological limitations of the miniature
displays, the virtual and the real environments are be perceived in the low resolution that is supported by
the displays.

Fig. 1: Two representative Head Mounted Displays (left: Kaiser Electro-Optics, right: VPI)

1/8
 Projection-based Augmented Reality (PBAR) is a new concept that extends the capabilities of projec-
tion-based VR (immersive projection technology) towards AR by using semi-transparent mirrors as opti-
cal combiners. Commonly known stereo glasses (active or passive ones) are sufficient to see a 3-
dimensional scene, consisting of a real/physical part and a virtual part. PBAR detaches the display device
from the user’s head and consequently solves some of the technological, ergonomic and economic prob-
lems that can be attributed to traditional head-attached devices.

Optional ceiling
projection

semi-
transparent
mirror

stereo rear
projection

Real car- virtual

front engine

Fig. 2: Sketch of a PBAR set-up built from a stereo rear projection display, a semi-transparent mirror
and a physical model

For the product development process this approach is of special relevance, since VR hardware technol-
ogy in form of PowerWalls - as already established in many companies – can be used by just extending
them with inexpensive semi-transparent mirrors and the according software technology. Thus, PBAR
shows not only advantages with respect to technological criteria but also economical benefits.

In contrast to HMDs that evolved to an all-purpose AR display medium over the previous 35 years,
PBAR offers application specific solutions that adapt the display and interaction technology to problem
specific tasks. By doing this, some of the technological limitations that are related to HMDs –and which
still today remain unsolved– can be improved. , Tracking within a well controlled environment, for in-
stance, becomes a less crucial problem since highly precise tracking technology can be used in a station-
ary set-up. Since PBAR does not apply head-attached miniature displays, but rather detaches the display
technology from the user a high and scalable resolution can be provided. Finally, the focal-length problem
caused by HMDs is improved, since the graphics is displayed near their real world position. It should not
be left unmentioned that not all AR scenarios can be supported with PBAR. Especially mobile applica-
tions, or applications that require a direct or close manipulation can mostly not be supported. However,
many applications in the product development process can benefit from PBAR, such as augmented design
review, hybrid assembling simulation and modelling/sketching. In the remainder of this paper we will
discuss some applications of PBAR, but first we present related work topics. Then, we describe the con-
cept of Projection-based Augmented Reality and introduce current hardware prototypes. Finally we con-
clude and give an outlook to further work topics.
Related Work

In contrast to traditional front or rear-projection systems that apply opaque canvases or ground glass
screens, transparent projection screens don't block the observer's view to the real environment behind the
display surface. Therefore, they can be used as optical combiners that overlay the projected graphics over
the simultaneously visible real environment. Pronova’s HoloPro system (Pronova, 2001) is such a trans-
parent projection screen. It consists of a multi-layered glass plate that has been laminated with a light-
directing holographic film. The holographic elements on this film route the impinging light rays into spe-
cific directions, rather than to diffuse them into all directions (as it is the case for traditional projection
screens). This results in a viewing volume of 60° horizontal and 20° vertical range in front of the screen,
in which the projected images are visible. Regular projectors can be used to rear-project onto a HoloPro
screen. However, they have to beam the images from a specific vertical angle (36.4°) to let them appear
within the viewing volume. Originally, the HoloPro technique has been developed to support bright pro-
jections at daylight and some researchers already begin to adapt this technology for Augmented Reality
purposes (Ogi, Yamada, Yamamoto & Hirose, 2001).
However, several drawbacks (mainly due to the applied holographic film) can be related to this tech-
nology:

• Limited and restricted viewing area.

• Static and constrained alignment of projector and projection plane (and therefore a no flexibility
or mobility).
• Low resolution of the holographic film (the pattern of the holographic elements are well visible
on the projection plane).
• Reduced see-through quality due to limited transparency of non-illuminated areas.

Computer augmented whiteboards have been a topic of research over the past years. As traditional
whiteboards, a computer augmented or electronic whiteboard presents an environment shared by multiple
users, which can be used to discuss, draw or present ideas. We’ll analyze the previous work with the fo-
cus on the interaction aspects relevant to our system.
The videoWhiteboard (Tang and Minneman, 1991) is one of the early systems that provides tools for
shared drawing. A camera captures the drawings and the gestures of each participant on the whiteboard
and transmits it to the remote user where it is projected giving the illusion that they are on opposite sides
of the same whiteboard. The Clearboard (Ishii and Kobayashi, 1992) is based on the same concept adding
gaze awareness by overlapping the image of the user and the drawing on the same display surface. Both,
the videoWhiteboard and the Clearboard are suitable for remote conferencing. Another pioneering exam-
ple of a whiteboard application is the Tivoli (Pedersen, McCall, Mora and Halasz, 1993) system, that has
been designed support collaborative meetings. It uses a pen as input device and offers the combination of
a traditional graphical user interface that is displayed on Xerox Liveboard and a gesture interface.
A different interaction approach is applied for the BrightBoard (Stafford-Fraser and Robinson, 1996).
Here, the user controls the computer by drawing simple sketches or characters on the board. Through
image processing the computer distinguishes the sketches from the rest of the drawing. Written words, for
instance, can be interpreted as computer commands. Thus, a seamless transition between handwriting and
computer control is achieved.
FlatLand (Mynatt, Igarashi, Edwards and LaMarca, 1999) is a system designed to support typical
whiteboard applications. A touch-sensitive whiteboard (the SmartBoardTM) accepts normal whiteboard
marker input as well as stylus input. Captured strokes are projected onto the board. Gestures serve for
managing the displayed visual layout.
HoloWall (Matsushita and Rekimoto, 1997) is a wall-size computer display that allows users to inter-
act with their fingers, hands, bodies, or even with physical objects. Inputs are recognized through infrared
lights (IR) and a IR video camera. Since the HoloWall can detect two or more hands (or fingers), multi-
hand interaction is possible. Also, user body posture/position or even physical objects that reflect IR light,
such as a document showing a 2D-barcode which identifies it, provide an enriched interaction environ-
ment.
Experimental Hardware-Setups

Since PBAR is an extension of immersive projection technology it can be combined with different IPT
set-ups. Up to now, we built-up two PBAR prototypes for the support of engineering scenarios:

• A Barco Baron Virtual Table (Barco, 2001) that applies rear-projection and active shuttering using
CrystalEyes Shutter glasses (see Fig. 3). In this case, two stereo images are projected sequentially
onto the projection surface and shuttering is synchronized via an infrared signal.
• A portable two-screen front-projection system that applies passive shuttering using polarized light. In
this case, four beamers project pre-filtered stereo images simultaneously onto the two projection
planes. The images are separated via polarized glasses. This is illustrated in figure 4.

In both cases the semi-transparent mirror consists of a mobile Aluminium rack that holds a tiltable
wooden frame. The frames comes with a conventional white board which is replaced by a half-silvered
mirror (a 40” x 60” large, and 10mm thick glass pane which has been laminated with a half-silvered mir-
ror foil (3M, 2001)) that simultaneously transmits and reflects light. Thus, we refer to it as “transflective
board” (Bimber, Stork and Branco, 2001). Technically, the transflective board is used as an optical com-
biner that reflects stereoscopic 3D graphics off an arbitrary display surface. Using the optical combiner,
this graphics is spatially merged with the surrounding real environment.

semi-transparent semi- transparent

mirror mirror

‚shine through’
model
physical
model Virtual Table

Fig. 3: Extended Virtual Table: on the right a table-like back-projection system (Virtual Table – here a
Barco Baron) with the semi-transparent mirror; on the left the physical model of a car

Fig. 4: The projection device used for this scenario is a mobile two-screen system. The dashed arrow
illustrate the flow of light. The dashed lines outline the reflected projection planes.

The set-ups are combined with different tracking technology. In one case an electromagnetic tracking
device (Ascension’s Flock of Birds (Ascension, 2001)) is used to support head-tracking and tracking of
spatial input devices. In the other case an optical tracking system (AR-Tracking, 2001) is used which
allows highly precise 3D tracking of the hand and head positions. In contrast to the magnetic tracker, the
user is not faced with cables which gives the system a new acceptance level.

In addition, we apply a tool that is usually used to track real markers and cleaning pads on a white-
board – the Mimio (Dunkane, 2001). The Mimio is a hybrid (ultrasonic and infrared) 2D tracking system
for planar surfaces which is more precise and less susceptible to distortion than an electromagnetic track-
ing device. The Mimio together with the transflective board can provide the same interaction behaviour as
a white-board.

Mixed Reality Techniques and Tools

In terms of providing a correct superimposition of the rendered graphics over the real environment, we
apply the following techniques (these techniques have been describe in full detail in (Bimber, Encarnação
& Branco, 2001):
• Rendering: Since the rendered graphics is perceived as reflection by looking at the mirror, several
transformations have to be applied to see an unreflected virtual augmentation and to provide correct
stereo separation. This is realized by introducing two additional transforms to the transformation
pipelines of the rendering framework. A view transformation reflects the viewpoints (the left and
right camera positions) that are located on the one mirror side (the side of the stereo display) over the
mirror plane. A model transformation reflects the virtual scene that is located on the other side of the
mirror (the side of the real environment) vice versa. Since these transformations are affine, they can
be simply integrated into a hard- or software-implemented transformation pipeline (such as the one
realized by OpenGL (Neider, et al., 1993)). Consequently, only minimal modifications to an existing
rendering framework have to be made to support planar mirrors as optical combiners in combination
with stereo displays. Furthermore, no additional computational cost is required for these transforma-
tions during rendering.
• Optical Distortion: Optical distortion is caused by the mirror and the projection planes because of
refraction and the curvature of the display or the mirror. We have developed several pre-distortion
methods that are applied within the 3D spatial space, as well as within the 2D image space.
• Tracking Distortion: When electromagnetic tracking devices are used, we apply smoothening filters
to filter high-frequent sub-bands, and positional pre-distortion with the pre-sampled magnetic field of
the working volume to minimize non-linear distortion, since electromagnetic tracking devices intro-
duce non-linear tracking-distortion and noise over an extensive working volume.
• System Delay: In addition to distortion, end-to-end system delay or lag causes a “swimming effect”
(virtual objects appear to float around real objects). To reduce the swimming effect, we apply predic-
tion filters (Kalman filters (Azuma, 1995)) for orientation information and linear prediction for posi-
tion information.

Interaction

Direct interaction in front of the transflective board as well as indirect and remote interaction tech-
niques with virtual objects ‘behind’ the transflective board are supported. A variety of techniques have
been implemented. For example, a tracked pen is used for direct manipulation in front of the board, as
well as for remote interaction behind the board. In addition, a transparent hand-held tablet (which is also
tracked) is applied to feature two-handed interaction (direct or remote). Virtual objects can be exchanged
between both sides of the board: they can be picked with the pen – either directly or remotely – and can
then be pushed, pulled or beamed through the mirror. A virtual laser beam casts from the pen through the
mirror to move and place virtual objects behind the board’s tangent plane. Additionally, the user’s view-
point can be combined with the pen direction to compute an appropriate selector. The ARToolkit optical
tracking system (Kato, Billinghurst, Blanding & May, 1999) is employed to detect different paper mark-
ers within the real environment. The markers are used to track real world objects or as placeholder for
multi-media information (e.g., images, video or textual information). Since different multi-media contents
are attached to specific markers, they can easily be exchanged by replacing the markers, or simply be
moved within the real environment. The tracked pen and also the pad feature sketch-based interaction
(Bimber, Encarnação, & Stork, 2000): two or three-dimensional freehand sketches are used to build ob-
jects by reconstructing their shapes from the sketches, or to interact with the objects (e.g. by sketching
assembling steps, etc.).

Engineering Applications

Today, maintenance and training are the major application fields for Augmented Reality from the in-
dustry’s point of view (ARVIKA, 2001). Augmented Reality will conquer the earlier phases of the prod-
uct development process where VR is more heavily used. AR can compensate for the shortcomings of
VR, such as fine and precise tactile and haptic feedback. Here, mobility is not a crucial issue. Thus, sta-
tionary set-ups that ensure high quality images and a more robust tracking than mobile ones will be pre-
ferred for these purposes.

We envision four application scenarios for projection-based AR in the engineering process:

• Augmented design review: VR systems are mainly used for design review. But the lack of reality
especially concerning haptic and tactile feedback still limits the acceptance of VR as a decision
base. This lack can be compensated by full functional rapid prototyping (RP) parts that give the
user a real tactile and also aural feedback. To produce many RP parts with, for instance, different
colors and materials is cost intensive. The combination of RP parts with virtual overlays that let
these parts appear in different colors/materials is a promising ‘best-of-two-worlds’-approach.

• Hybrid assembling: The feasibility evaluation of the assembling/disassembling processes is a

common application of VR whereby the product exists in a digital form, only. But products are
frequently altered during their lifecycle. In these situations, the products already exist physically
and the need to assemble virtual models into physical mock-ups (PMU) arises.

Fig. 5: Hybrid assembly sequence: upper left: virtual model ‘above’ the Virtual Table; upper right: user
moves virtual part ‘through’ the mirror; lower left: distant interaction with virtual part; lower right: virtual
part assembled into real printer (images taken from Bimber, Encarnação & Branco, 2001)
 • Hybrid modeling/sketching: Although CAS and VR systems have changed the styling and de-
sign process considerably, clay models still have their place in the design process. During a re-
view of a physical model, requirements to change some of its parts/features appear. Today, these
modification requirements are expressed verbally. An Augmented Reality setup would allow to
virtually sketch the change onto/over the PMU.

Fig. 6: Free-form surface sketched ‘over’ physical model

• Visual inspection of molded parts: The development of a product comprises the design of tools
to manufacture the product parts. Molded parts shrink when they cool down. Thus, the tools are
not simply the inverse product geometry. When starting a new production line, the produced
parts have to be compared with the intended geometry – the virtual parts. Again, Augmented Re-
ality can give us an efficient to use visual inspection environment.

Conclusion and Future Work

Traditional Augmented Reality displays, such as head-mounted devices entail a number of technologi-
cal and ergonomic drawbacks that prevent their usage in a many of application areas. With the objectives
to overcome some of these drawbacks and to address an efficient and problem specific application of
Augmented Reality technology within the engineering domain, we have presented the early stages of a
new projection-based Augmented Reality device – the transflective board. Compared to head-attached
AR displays, the application of spatial projection displays for Augmented Reality tasks feature an im-
proved ergonomics, a theoretically unlimited field-of-view, a high and scalable resolution, and an easier
eye accommodation (Raskar, 1998). Since the technology, as well as the consequential advantages are
derived from the well established projection-based VR concept, a seamless combination of VR and AR is
also imaginable. However, several shortcomings can be related to our approach. Self-reflection of the user
in some situations, for example, is one of the drawbacks of the transflective board. Our current approach
to address this problem is to integrate the transflective board into engineering-related application scenar-
ios in such a way that these reflections are minimized or avoided. Future research will tackle further tech-
nological and usability issues, such as mobility, robust tracking, efficient interaction and the evaluation of
implemented scenarios with the industry. Another topic for future work is the extension towards multi-
user scenarios as sketched in figure 7.

Fig. 7: The Virtual Showcase: Multi-user AR set-up using a

transflective cone (Bimber, Fröhlich, Schmalstieg, and Encar-
nação 2001)
 Acknowledgement

The work described in this paper has been supported in part by the European Commission Grant #IST-
2000-28169 (SmartSketches project).

References

3M, Corp. (2001) P-18. Retrieved from the World Wide Web:
[Link]
Advanced Realtime Tracking, [Link], 2001
ARVIKA – Augmented Reality in Industrial Applications (2001), Retrieved from the World Wide Web:
[Link]
Ascension Technologies. Corp. (2001) Flock of Birds. Retrieved from the World Wide Web: [Link]
[Link]/products/flockofbirds/[Link].
Azuma, R.(1995) Predictive Tracking for Augmented Reality. Ph.D. dissertation, University of North Carolina at Chapel
Hill, TR95-007.
Barco, Inc. (2001). Baron. Retrieved from the World Wide Web:
[Link]
Bimber, O., Encarnação, L.M. and Stork, A. (2000). A Multi-Layered Architecture for Sketch-based Interaction within
Three-dimensional Virtual Environments. Computers and Graphics – The international Journal of Systems and Applica-
tions in Computer Graphics, vol. 24, no. 6, pp. 851-867.
Bimber, O., Encarnação, L.M. and Stork (2001), A. Seamless integration of virtual reality in habitual workplaces. Jour-
nal for Industrial Science, Munich University of Technology, vol.55, no.2., pp. 103-112.
Bimber, O., Stork, A., and Branco, P. (2001) Projection-based Augmented Engineering. Human-Computer Interaction
International (HCII’01), vol. 1, pp. 787-791.
Bimber, O., Fröhlich, B., Schmalstieg, D., and Encarnação, L.M. (2001) The Virtual Showcase. IEEE Computer
Graphics & Applications, vol. 21, no.6, pp. 48-55.
Bimber, O., Encarnação, L.M. and Branco, P. (2001) The Extended Virtual Table: An Optical Extension for Table-Like
Projection Systems. To appear in Presence: Tele-operators and Virtual Environments, vol.10, no. 6.
Dunkane, Corp. (2001) Mimio. Retrieved from the World Wide Web:
[Link]
Ishii, H. and Kobayashi, M. (1992). ClearBoard: A Seamless Medium for Shared Drawing and Conver-sation with Eye
Contact. Proceedings of the Conference on Computer Human Interaction CHI’92, pp. 525-532.
Matsushita, N. and Rekimoto, J. (1997). HoloWall: Designing a Finger, Hand, Body, and Object Sensitive Wall. In
proceedings of UIST’97, 209-210.
Mynatt, E. D., Igarashi, T., Edwards, W. K. and LaMarca, A. (1999). Flatland: New Dimensions in Office
Whiteboards. In proceedings of the Conference on Computer Human Interaction CHI’99, pp. 346-353.
Neider, J., Davis, T., and Woo, M. (1993) OpenGL programming Guide. Addison-Wesley Publ., ISBN 0-201-63274-8.
Ogi, T. Yamada, T. Yamamoto, K. and Hirose, M. (2001). Invisible Interface for Immersive Virtual World. To appear at
the Immersive Projection Technology Workshop (IPT’01), Stuttgart, Germany.
Pedersen, E. R., McCall, K., Mora, T. P. and Halasz, F. G. (1993). Tivoli: An Electronic Whiteboard for Informal
Workgroup Meetings. In proceedings of INTERCHI’93, pp. 391-398.
G+B Pronova GmbH. (2001) HoloPro. Retrieved from the World Wide Web: [Link]
Raskar, R., Welch, G., and Fuchs, H. (1998). Spatially Augmented Reality. In proceedings of First IEEE Workshop on
Augmented Reality (IWAR'98), San Francisco, CA.
Stafford-Fraser, Q. and Robinson, P. (1996). BrightBoard: A Video-Augmented Environment. Proceedings of the
Conference on Computer Human Interaction CHI’96, pp. 134-141.
Tang, J. C. and Minneman, S. L. (1991). VideoWhiteboard: Video Shadows to Support Remote Collaboration.
Proceedings of the Conference on Computer Human Interaction CHI’91, pp. 157-168.

View publication stats