A Comparative Study of Rendering Devices for

Safety-Critical Applications in Operative Control

Rooms
Syed Muhammad Umair Arif Pramit Mazumdar Federica Battisti
Department of Engineering Department of Engineering Department of Engineering
Roma Tre University Roma Tre University Roma Tre University
Rome, Italy Rome, Italy Rome, Italy
[Link]@[Link] [Link]@[Link] [Link]@[Link]

Abstract—Recent advancements in augmented reality have space. Bernardo et al. [5] performed VR simulation to train
helped bridging the gap between real world and virtual environ- surgeons with different cases by designing game scenarios to
ments. In this context, augmented reality can be used to assist reduce the learning time and to improve the study of complex
operators in control rooms by providing extra information in
the decision making process and to improve the user attention. human anatomy. The implementation of AR for industrial
In this paper we focus on application of augmented reality in safety applications is technically complex due to the presence
industrial safety applications in a control room environment. Our of many influencing factors (e.g., human, ergonomics, interac-
work is divided into three modules. First, we develop a safety- tion, hardware). One such example is provided in [6], where
critical application for emergency fire scenarios. Then, we make the authors used Microsoft HoloLens to improve the assem-
it available on two rendering devices: a flat 2D laptop screen and
on Microsoft HoloLens. Finally, to check the usability and the bly process and human safety. Authors used AR in human
effectiveness of the application, subjective tests are performed. surgery applications to avoid human errors in during tasks.
Index Terms—Augmented Reality, HoloLens, Control Room, All required useful information was displayed on HMD. In
User Interfaces, Human Interaction, Firefighter the automotive industry, AR has been used for guiding the car
assembly process. Rentzos et al. [7] built an AR application to
I. I NTRODUCTION guide mechanics in manufacturing and development. For user
Augmented reality (AR) is a combination of computer- interface and interaction author used a tablet with touch screen
generated graphics (in the form of graphics, videos, symbols, capabilities. Another example in the automotive industry is
texts, etc.) and elements belonging to the real world. Hand- in [8] where the authors used AR to check visual features
held devices, head-mounted, and transparent see-through dis- of car fenders, and the quality of car parts. The author used
play interfaces are the main sources to augment the infor- monitors and projectors to visualize augmented data. Mourtzis
mation on the different environment according to a particular et al. [9] used AR to support operators in the papermaking
application. As an example, AR can support operators in in- industry, where to locate and maintain inventories is a big
dustrial environments thanks to the use of optical see-through problem. Palmarini et al. [10] provide an innovative process
displays that allow continuous connection of the operator with for industries to improve human performances by providing
real world while accessing useful information for carrying extra information in the form of texts and videos for operators
out specific tasks. AR, therefore, opens an important field of training and step by step real-time work instructions. The
research with plenty of real world applications [1]. The general author also carried out a study on whether or not AR is
goal is to improve the user experience and effectiveness in required for solving industrial problems. Hincapie et al. [11]
many real life applications such as medical sciences, human- used AR for aeronautical maintenance, to handle small and
safety, or industrial training. big complex aircraft parts. The author proposed an AR-based
AR has already found application in healthcare-related ac- solution to reduce the duration of training time and location
tivities to help surgeons and doctors to improve human health related problems. LED monitors were adopted to visualize the
and safety. Mahmood et al. [2] used Microsoft HoloLens to augmented information. Mourtzis et al. [12] used AR as a
visualize medical ultrasound images in real-time. Vaughan remote maintenance service tool to reduce industry mainte-
et al. [3] used Virtual Reality (VR) for orthopedic surgery nance costs that often exceeds 30 percent of the operating
to improve the training level of surgeons and doctors in costs. The author developed a cloud-based service-oriented
critical human bone surgery through the implementation of AR system for maintenance and information exchange. Utzig
pre-operative planning. et al. [13] used AR glasses to facilitate the collaboration
Rambani et al. [4] used augmented interface for junior among multiple operators through a remote meeting service
orthopedic trainees to operate surgical instruments into 3D for aircraft maintenance to improve and keep track of all
performed activities. Segovia et al. [14] used AR as a real- have access for safety reasons. Naturally, the extension of
time data analysis tool to monitor key performance indicators such zone depends on several factors such as the material that
of each workstation to control product and process quality. started the fire, the behavior of wind, or the presence of nearby
The author used smartphone interfaces for data visualization. critical spots such as gas tanks, gasoline pumps, forests, etc.
Crescenzio et al. used AR in aircraft maintenance training and Another important key factor is the usability of the developed
operations support. The author used the head-mounted display application with an augmented display interface. It is important
for daily interior and exterior inspection of C.172P aircraft to that the operators can use it in stress condition during the task
ensure the operators checked all defined safety procedures. and with least interaction delay.
Stoltz et al. [15] explored the use of AR in warehouses and
their limitations according to specific tasks. Makris et al. B. Application development
[16] focused on human safety in everyday tasks in a robotic To develop our application, first we collect from the fire-
environment and demonstrated that AR can support operators fighters [18] the list of 2511 materials that are the most
to ensure human safety with the help of markers and symbols. common cause of fires. For each material, their international
Quandt et al. [17] explored industrial applications for remote code, the Kemler-ONU [19] is reported. Moreover, depending
maintenance in the wind energy environment. on the specific material, the following information is given:
In the use of AR, the mode of interaction is a very important the radius of the dangerous zones, the primary hazard, the
factor that significantly influences user experience. As an secondary hazard, and the optional hazard.
example Microsoft HoloLens can be used with air gestures To optimize the query in the database, we defined as primary
(air taps) and head movements, or with a wireless keyboard keys for the search the Kemler-ONU and the name of the
and mouse. Incidentally, it has been observed that an easier material. Kemler-ONUs consist of 7 numeric digits: the first
interaction with the augmented content can also be achieved 2 or 3 digits define the substance radioactivity, and the last 4
with 2D screens [1]. However, such 2D displays are not digits identify the material, based on its chemical name and
technologically capable to induce immersive quality for the its classification.
viewers. Based on the available information, We designed a user-
In this work, we develop an application for increasing the friendly cross-platform web application that can be used
safety of firefighters in case of fire. To test the usability of also by non-technical users. The application aims to support
the developed application we perform subjective tests. The firefighters for high-grade safety operation planning. This
application is projected on two different interfaces; augmented also incorporates real-world 2D maps for particular location
on Microsoft HoloLens and displayed on flat 2D LED screen understanding (in satellite, terrain and map views).
of a laptop. Moreover, we prepared a questionnaire to assess The developed web-page is divided into two sections: first,
how the subjects liked the safety-critical application and also the operator can enter the information on latitude and longitude
which interface was preferred for such an application. of the fire (to localize where the firefighters should act), and
The rest of the paper is organized as follows: Section II secondly, a search field for inserting the name of the material
describes the developed application, Section III describes the or its Kemler-ONU code. When both the fields are inserted,
performed tests, Section IV reports the obtained results and the application will return information such as the nature of the
finally in Section V the conclusions are drawn. hazard, and the radius of the dangerous zone. The right panel
of the web-page accordingly shows the exact location of the
II. P ROPOSED METHOD accident and two concentric circles: the inner one, indicating
The proposed method aims at the definition of a user- the no-entry zone, and the outer one, indicating the area that
friendly interface and interaction that can support the activities is accessible only by the firefighters.
of firefighters without interruption in the event of a fire. In regard to the marked circles, We used two different types
of circles: high contrast (Figure 1) and low contrast (Figure
A. Application requirements 2). It is observed that for the high contrast circles, the colors
The first step of our work has been to define the user are more saturated but the map information are less visible.
(firefighter) requirements. In this regard, we interviewed a pool On the contrary, low contrast circles provide better visibility
of firefighters to learn what are the operational procedures in of underlying maps.
case of a fire and to collect their requests on how their work In the design of our application we considered the fact that
could be supported. users may suffer from information overloading [20]. For this
The operational control room requires two basic information reason we provide only the basic information to the operators
related to fire notification. First, the specific location which in two modes: visual and textual.
caught fire, and secondly, the material that started the fire.
The primary information is extremely important because C. Application and Network architecture
it allows the operators to know which is the danger zone To optimize and manage the application architecture we
around the place where the fire started. Moreover, two areas distributed the processes on the remote cloud and on a local
can be identified within the danger area: one in which only database server as shown in Fig. 3. Chemical information
the firefighters are allowed and one in which no one should of all materials (i.e., formula names, Kemler-ONU numbers,
 Fig. 1: User interface with high contrast colour.

Fig. 3: Network architecture.

at the University of Roma Tre. They were asked to wear any

vision correcting devices (glasses or contact lenses) that they
normally wear.
A. Test Environment
The experiments were carried out in an office environment
Fig. 2: User interface with low contrast colour. to simulate the real conditions of an operative control room in
which such an application should be used. Microsoft HoloLens
has a low processing ability to render map application in
primary, secondary, optional hazards, and circles radius infor- satellite and map view and cannot fastly handle the search
mation) were saved in the local database. After the inputs query within the 2511 chemical materials in the database. To
are inserted in the interface of the application, a query is address this problem we use a server-client architecture to
performed in the remote database thorough local network render satellite and map view with the search query. Moreover,
connection. Location services, coordinates and maps layers to minimize the processing delay, the client sends a query to
were handled remotely by the HERE maps cloud server [21]. the server for searching a specific material in the database, the
Maps rendering and other related geographical information server processes all information and sends back to the client
were processed on the remote server and the pre-rendered the graphics overlays.
graphics were then sent to the local server for visualization. B. Test Execution
D. Application interaction At the start of each test verbal instructions are provided
As previously mentioned, the comfort in the interaction is to each participant to clearly describe the purpose of the
one of the common influencing factors of the user experience application, how to use it and how to interact and operate
in performing particular tasks. For this reason, we experi- the Microsoft HoloLens. Each participant involved in the
mented the usability of the implemented application on two subjective test took part in two sessions. Once by viewing
different rendering devices: 2D 15.6” UHD (3840 x 2160) the application in Microsoft HoloLens (Figures 4) and subse-
60Hz IPS screen and Microsoft HoloLens. With the 15.6” quently, viewing it on a flat 2D laptop screen (Figure 5).
UHD (3840 x 2160) 60Hz IPS display, the operators need After all instructions are provided, the users start the sub-
to open a web-page and interact with it by using mouse jective test by viewing the developed application through the
and keyboard. Whereas, the Microsoft HoloLens provides a Microsoft HoloLens by using a Bluetooth mouse to interact
hands-free interaction control by voice commands and air taps. with the web application. In the application, subjects enter a
However, in an emergency, the voice controls might not be particular location corresponding to the place where the fire
favorable due to the presence of background noise. During a started, and the name of the material that caused the fire.
stressful situation such as an emergency, human errors have a The corresponding target location is displayed in the right
big chance to occur. In fact most tasks are bounded with time part of the application. Figure 5 shows the interface with a
constraints and delays in interaction are not acceptable. critical location on the right part of the panel and both the
high and low contrast circles. It can be noted thay the same
III. S UBJECTIVE T ESTING information is provided for both rendering devices (HoloLens
To assess the usability of the designed augmented applica- and 2D laptop screen). Subjects are also asked to switch the
tion we used HoloLens and a 2D laptop screen to simulate a map view between map and satellite views.
control room environment. We used a total of 21 naive subjects Moreover, we evaluate the usability of the developed ap-
drawn from a pool of students in the Engineering Department plication and compare user preferences between the flat 2D
 Fig. 4: Subjective test execution on Microsoft HoloLens. Fig. 5: Subjective test execution on 2D screen.

display and an augmented display. In this regard, we prepare users considered the content to be more clear on the 2D
a set of questionnaire for the participants of the subjective test. screen rather than on the HoloLens (Q5 and Q6) because of
The questionnaire included a total of 10 questions on a 5-step the viewing distance from the eye of the projected image.
scale (from ’worst’ to ’best’): Finally, in Q7, users are asked to evaluate the quality of
the executed application, in terms of this application provides
• Q1: How was the interaction with the HoloLens by using
complete information that might be useful during tasks without
the wireless mouse?
considering the rendering device. In this case, the largest part
• Q2: How was the interaction with the application on the
of users was satisfied with the information provided by the
2D screen?
application and the easiness in the interaction with it. The
• Q3: How was your experience with the 2D screen?
mean and standard deviation of the ratings given for Questions
• Q4: How was your experience with HoloLens?
1 to 7 are provided in Table I. Surprisingly, although the 2D
• Q5: How clear is the content information on the 2D
rendering was preferred in the 7 questions, when the users
screen?
were asked to express their preference on the clarity of the
• Q6: How clear is the content information on the
content and on the preferred rendering system, half of the
HoloLens?
users selected the HoloLens and half of them the 2D screen.
• Q7: Rate how user-friendly the application is on a scale
Anyhow, the HoloLens were preferred because of wide virtual
on 1 to 5 (where 5 is the best);
FOV that makes ease in visual searching of information about
• Q8: The content information is more clear on 2D screen
geographical area information on maps.
or HoloLens?
• Q9: Which is your preferred rendering for the augmented Question Mean Standard Deviation
content? Q1 4.00 1.09
• Q10: Which method is the best for obtaining surrounding Q2 4.76 0.43
Q3 4.71 0.56
area information on the map? Q4 3.80 1.03
Q5 4.66 0.57
IV. R ESULTS Q6 3.76 1.18
Q7 4.33 0.79
To verify the usability of the application on the two ren-
dering systems, we analyze the opinions collected with the
subjective test. Figure 6 reports the bar plots of the subjective TABLE I: Mean value and Standard Deviation of the collected
scores of questions from 1 to 7. It can be noticed that the scores.
questions related to the usability and the overall quality of the
experience with the HoloLens (Q1 and Q4) do not indicate a Question Microsoft HoloLens 2D Monitor
high acceptance of the systems. The MOS is spread and no real Q8 52% 48%
preference is shown. In this case, one important influencing Q9 52% 48%
factors is the reduced interaction speed with AR contents. At Q10 76% 24%
the same time, Q2 and Q3 indicate that the subjects agreed
on the usability of the 2D rendering system. Moreover, the TABLE II: Results in percentage for Q8-Q10.
 From the above analysis, we may conclude that users are [5] A. Bernardo, “Virtual reality and simulation in neurosurgical training,”
more comfortable with the 2D screen rendering method due World neurosurgery, vol. 106, pp. 1015–1029, 2017.
[6] G. Evans, J. Miller, M.I. Pena, A. MacAllister, and E. Winer, “Evaluating
to its quick and reliable interaction feedback. In contrast, the microsoft hololens through an augmented reality assembly applica-
HoloLens enriches the user experience because the wide tion,” in Degraded Environments: Sensing, Processing, and Display
virtual field of view provides the possibility of accessing a 2017. International Society for Optics and Photonics, 2017, vol. 10197,
pp. 1–15.
wide amount of useful information while performing the tasks. [7] L. Rentzos, S. Papanastasiou, N. Papakostas, and G. Chryssolouris,
Moreover, HMDs implement an optical see-through interface “Augmented reality for human-based assembly: using product and
for data visualization and the operator can have continuous process semantics,” IFAC Proceedings Volumes, vol. 46, no. 15, pp.
98–101, 2013.
visual contact with real-world during the task that makes the [8] S. Nolle and G. Klinker, “Augmented reality as a comparison tool in
user more comfortable and confident in using AR. A possible automotive industry,” in 2006 IEEE/ACM International Symposium on
explanation for the obtained results is that users are very Mixed and Augmented Reality. IEEE, 2006, pp. 249–250.
[9] D. Mourtzis, V. Samothrakis, V. Zogopoulos, and E. Vlachou, “Ware-
confident in interacting with the 2D screen by using mouse house design and operation using augmented reality technology: A
and keyboard while, for all of them, it was the first experience papermaking industry case study,” Procedia CIRP, vol. 79, pp. 574–
with the HoloLens. 579, 2019.
[10] R. Palmarini, J. A. Erkoyuncu, and R. Roy, “An innovative process
As shown by Q8, Q9 and Q10 percentages (Table II), the to select augmented reality (ar) technology for maintenance,” Procedia
users are open to new technologies and, even if up to now 2D CIRP, vol. 59, pp. 23–28, 2017.
15.6” UHD (3840 x 2160) 60Hz IPS screen is the preferred [11] M. Hincapié, A. Caponio, H. Rios, and E. G.ález Mendı́vil, “An
introduction to augmented reality with applications in aeronautical
rendering method, they expect that augmented reality will be maintenance,” in 2011 13th International Conference on Transparent
the media for delivering extended information. To this aim, it Optical Networks. IEEE, 2011, pp. 1–4.
is useful to underline that the user experience with HoloLens [12] D. Mourtzis, V. Zogopoulos, and E. Vlachou, “Augmented reality
application to support remote maintenance as a service in the robotics
may be affected by the hardware capabilities in terms of industry,” Procedia CIRP, vol. 63, pp. 46–51, 2017.
visual definition or equipment usability and comfort. However, [13] S. Utzig, R. Kaps, S. M. Azeem, and A. Gerndt, “Augmented reality
augmented reality is a young technology. We may expect that, for remote collaboration in aircraft maintenance tasks,” in 2019 IEEE
Aerospace Conference. IEEE, 2019, pp. 1–10.
if properly supported by ad-hoc software applications, the [14] D. Segovia, M. Mendoza, E. Mendoza, and E. González, “Augmented
hardware performances will improve thus reducing, or even reality as a tool for production and quality monitoring,” Procedia
inverting, the gap among 2D and AR experience. Computer Science, vol. 75, pp. 291–300, 2015.
[15] V. Stoltz, M.-[Link] Giannikas, J. McFarlane, [Link] Strachan, J. Um, and
R. Srinivasan, “Augmented reality in warehouse operations: opportuni-
V. CONCLUSIONS ties and barriers,” IFAC-PapersOnLine, vol. 50, no. 1, pp. 12979–12984,
2017.
In this paper, we analyze the usability of AR in a defined [16] S. Makris, P. Karagiannis, S. Koukas, and A.-S. Matthaiakis, “Aug-
application scenario. In particular, we propose a framework mented reality system for operator support in human–robot collaborative
assembly,” CIRP Annals, vol. 65, no. 1, pp. 61–64, 2016.
for supporting the activities of firefighters in emergencies. [17] M. Quandt, B. Knoke, C. Gorldt, M. Freitag, and K.-D. Thoben,
The usability of the developed augmented application for “General requirements for industrial augmented reality applications,”
control room is tested by using two rendering methods: a Procedia CIRP, vol. 72, no. 1, pp. 1130–1135, 2018.
2D 15.6” UHD (3840 x 2160) 60Hz IPS screen and the [18] National Fire Academy, “National fire academy italy,”
[Link] orld.
HoloLens. To carry out our study, we performed subjective [19] National Fire Brigade Corps, “Kemler onu,”
tests and administered questionnaires to the subjects to detect [Link]
[20] C.H. Tsai and J.Y. Huang, “Augmented reality display based on user
their opinions. We found out that, although the 2D rendering behavior,” Computer Standards and Interfaces, vol. 55, pp. 171–181, 2018.
method is preferred, the HoloLens allows the users to access [21]
a HERE, “Here maps all additional functions and products information,”
larger amount of information while performing the task and, at [Link]
the same time, to keep the visual contact with the real world.
We believe that with appropriate training we could enhance
the user satisfaction concerning the use of the HoloLens and
for that reason, we are actively working to explore more
possibilities of AR in industries.

R EFERENCES
[1] J. Peddie, Augmented Reality: where we will all live, Springer, 2017.
[2] F. Mahmood, E. Mahmood, R.G. Dorfman, J. Mitchell, F.U. Mahmood,
S.B. Jones, and R. Matyal, “Augmented reality and ultrasound education:
initial experience,” Journal of cardiothoracic and vascular anesthesia,
vol. 32, no. 3, pp. 1363–1367, 2018.
[3] N. Vaughan, V.N. Dubey, T.W. Wainwright, and R.G. Middleton, “A
review of virtual reality based training simulators for orthopaedic
surgery,” Medical engineering and physics, vol. 38, no. 2, pp. 59–71,
2016.
[4] R. Rambani, W. Viant, J. Ward, and A. Mohsen, “Computer-assisted
orthopedic training system for fracture fixation,” Journal of surgical
education, vol. 70, no. 3, pp. 304–308, 2013.
 (a) Q1 (b) Q2

(c) Q3 (d) Q4

(e) Q5 (f) Q6

(g) Q7
Fig. 6: Scores collected from the questionnaires administered to the subjects.