IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS, VOL. 54, NO.

5, OCTOBER 2024 589

ARMedicalSketch: Exploring 3D Sketching for

Medical Image Using True 2D-3D Interlinked
Visualization and Interaction
Nan Zhang , Tianqi Huang , and Hongen Liao , Senior Member, IEEE

Abstract—In traditional clinical practice, doctors often have to I. INTRODUCTION

deal with 3D information based on 2D-displayed medical images.
N CLINICAL practice, how medical images are interacted
There is a considerable mismatch between the 2D and 3D di-
mensions in image interaction during clinical diagnosis, making
image manipulation challenging and time-consuming. In this study,
I with significantly affects the accuracy and efficiency of im-
age operations for treatment, which is crucial for both health-
we explored 3D sketching for medical images using true 2D-3D care providers and patients. The prevalent method in clinics
interlinked visualization and interaction, presenting a novel AR
environment named ARMedicalSketch. It supports image display
involves using 2D flat displays, operated through conventional
enhancement preprocessing and 3D interaction tasks for original devices such as a mouse and keyboard. Physicians often perform
3D medical images. Our interaction interface, based on 3D au- complex 3D tasks using these 2D-rendered medical images,
tostereoscopic display technology, provides both floating 3D display such as segmenting 3D lesions, planning 3D surgical paths,
and 2D tablet display while enabling glasses-free visualization. and conducting 3D minimally invasive procedures. However,
We presented a method of 2D-3D interlinked visualization and
interaction, employing synchronized projection visualization and dealing with medical images in a 3D format through 2D-based
a virtual synchronized interactive plane to establish an integrated visualization and interaction presents challenges. Complex 3D
relationship between 2D and 3D displays. Additionally, we utilized structures of human tissues are not adequately captured without
gesture sensors and a 2D touch tablet to capture the user’s hand 3D depth perception, hampering effective observation. Tasks
information for convenient interaction. We constructed the proto- such as surgical path planning based on 3D images are dif-
type and conducted a user study involving 23 students and 2 clinical
experts. The controlled study compared our proposed system with ficult in a 2D setting, and switching between display layers
a 2D display prototype, showing enhanced efficiency in interacting is inconvenient and less precise. To address these issues, we
with medical images while maintaining 2D interaction accuracy, propose using medical 3D sketching [1] for 3D interactions
particularly in tasks involving strong 3D spatial correlation. In the with medical images. This approach is critical but challenging. It
future, we aim to further enhance the interaction precision and helps overcome the limitations of 2D-based methods, enhancing
application scenarios of ARMedicalSketch.
both the precision of medical image operations and the quality
Index Terms—3D aerial display, augmented reality, autoste- of patient care.
reoscopic visualization, human-computer interaction. As computer technology advances, exploration and research
into 3D sketching within virtual reality (VR) [2], [3], [4] and aug-
mented reality (AR) [5] have gained momentum. Some research
based on 2D display [6] can also achieve 3D sketching, which
Manuscript received 24 October 2023; revised 7 May 2024 and 2 July 2024;
accepted 15 July 2024. Date of publication 2 August 2024; date of current is not the focus of this study. The demand for 3D sketching is
version 18 September 2024. This work was supported in part by the National extensive, particularly in various design fields involving manual
Natural Science Foundation of China under Grant U22A2051, Grant 82027807, work. Compared with 2D sketching, 3D spatial sketching offers
and Grant 62101305, in part by the National Key Research and Development
Program of China under Grant 2022YFC2405200, in part by the Intelligent a clear advantage, providing a “what you see is what you get”
Healthcare Tsinghua University under Grant 2022ZLB001, and in part by the human-computer interaction (HCI) experience. Nevertheless,
Tsinghua-Foshan Innovation Special Fund under Grant 2021THFS0104. This achieving accurate and efficient 3D sketching is a challenging
article was recommended by Associate Editor S. Cao. (Corresponding author:
Hongen Liao.) task. Two primary difficulties in achieving precision and effi-
Nan Zhang was with the Department of Biomedical Engineering, School of ciency in 3D sketching involve ensuring users possess a correct
Medicine, Tsinghua University, Beijing 100084, China. He is now with the perception of 3D space and the ability to interactively translate
Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
(e-mail: zhangnan@[Link]). their mental 3D sketches effectively [7].
Tianqi Huang is with the School of Biomedical Engineering, Shanghai Jiao In the realm of VR/AR-based 3D sketching [4], head-mounted
Tong University, Shanghai 200030, China (e-mail: huangtianqi@[Link]). displays (HMDs) [3], [8] are a significant approach for imparting
Hongen Liao is with the Department of Biomedical Engineering, School of
Medicine, Tsinghua University, Beijing 100084, China, and also with the School 3D spatial perception, offering advantages such as real-time
of Biomedical Engineering and the Institute of Medical Robotics, Shanghai Jiao rendering and high-quality graphics. However, several draw-
Tong University, Shanghai 200240, China (e-mail: liao@[Link]). backs, such as inducing discomfort like nausea, vomiting, and
Color versions of one or more figures in this article are available at
[Link] vertigo, have yet to be effectively addressed [9], [10]. Given the
Digital Object Identifier 10.1109/THMS.2024.3432735 demanding workload of clinical experts, there is a compelling

2168-2291 © 2024 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See [Link] for more information.

Authorized licensed use limited to: Noakhali Science and Technology University. Downloaded on October 05,2025 at 10:02:22 UTC from IEEE Xplore. Restrictions apply.
 590 IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS, VOL. 54, NO. 5, OCTOBER 2024

need for a 3D visualization method that does not rely on HMDs. surfaces or the use of 2D tablets as input devices have shown
Naked-eye-based methods emerge as a viable alternative, en- promise in addressing this challenge. For instance, Arora et al.
compassing autostereoscopic [11], [12], [13], holographic [14], [22] developed a hybrid interactive drawing system based on
[15], and volumetric [16], [17] displays. Among them, the 3D HMDs. Their hybrid mode combines the precision of 2D plane
autostereoscopic display stands out as a full-color 3D visual- interaction with the immersive qualities of 3D air interaction,
ization technique supporting real-time interaction. Zhang et al. offering users greater flexibility. Drey et al. [4] expanded on
[11] proposed a floating autostereoscopic display system with this concept with “VRSketchIn,” enhancing 2D and 3D hy-
in-situ interaction, leveraging a novel light-field symmetrical brid interactions. The system features six-degrees-of-freedom
multiview-point rendering algorithm and incremental gestures. interaction and real-time tracking for input devices (pens and
It is noteworthy, however, that limited attention has been devoted tablets), earning recognition from artists, product designers,
to 3D autostereoscopic display-based 3D sketching methods and and other creative professionals. Jiang et al. [23] proposed an
systems, warranting further exploration in this domain. immersive 3D drawing system called “HandPainter,” wherein
Based on the considerations mentioned earlier, this study users wear HMDs and interact using gloves to draw on their
aims to explore 3D sketching for medical images using naked- palms. User studies have demonstrated that this system enables
eye-based visualization. We propose ARMedicalSketch, a novel more accurate spatial interaction compared with drawing on a
medical 3D sketching environment featuring true 2D-3D inter- tablet or using a 3D controller.
linked visualization and interaction. The main contributions of Moreover, some studies focus on aiding users in reproducing
this work are as follows. target sketches by providing visual guidance or stroke optimiza-
1) The definition of a design space for 3D medical im- tion. Tsang et al. [24] introduced an image-guided, pen-based
age sketching, as well as the presentation of interaction suggestive interface for sketching 3D wireframe models. This
metaphors arising from the integration of naked-eyes- environment combines 3D sketching with gestures, concept
based 2D and 3D visualization and interaction with a images, and suggestions, resulting in a compelling and effective
floating 3D autostereoscopic display and tablet. design tool. Yu et al. [25] proposed a method for curve and
2) The concept and implementation of ARMedicalSketch surface drawing in an immersive environment based on HMD
involve a sketching application that combines a user- technology. Their 3D drawing optimization framework incorpo-
controlled tablet and floating 3D display, enabling seam- rates functions such as automatic curve network connection and
less content control through hand gestures. It integrates 3D patch prediction, enhancing the reasonableness and aesthetics of
mid-air sketching with 2D surface sketching for mutual the output compared with user-input three-dimensional spatial
responsiveness to changes. paths.
3) First insights regarding usage patterns of ARMedicalS- While HMD-based VR/AR sketching research has yielded
ketch, based on a one-hour-long usability walkthrough valuable interaction solutions, there remains an unmet need for a
study with twenty-three participants. Comparative user naked-eye-based 3D sketching system, free from the constraints
experiments demonstrate greater efficiency in 3D-based of glasses. This work aims to address that gap and contribute to
interaction compared with 2D. the field.

II. RELATED WORK

B. Medical Image Sketching
Our work is strongly influenced by previous work on HMD-
In the medical context, research on 3D sketching remains
based 3D sketching VR/AR environments and medical image
relatively scarce. Relevant studies can be categorized into two
sketching studies.
main areas: education, and clinical diagnosis and treatment.
In the field of medical education, 3D sketching aids users in
A. VR/AR 3D Sketching
rapidly grasping complex anatomical structures and physiolog-
3D sketching holds a significant role in HCI [4]. As early ical dynamics. For instance, Saalfeld et al. [26] introduced a
as 1995, a VR-based 3D sketch drawing system was introduced semi-immersive 3D user interface that allows users to sketch
[3], marking an early milestone in the development of VR-based intricate vascular structures and vessel pathologies by drawing
3D geometry creation and manipulation tools. This system centerlines in 3D. A qualitative evaluation affirmed the prac-
facilitated the creation and manipulation of 3D tube and wire ticality and user-friendliness of their approach. Subsequently,
geometries. While sketching systems based on predefined sur- they presented computer graphic techniques [27] to enhance and
face shapes serve practical purposes in specific scenarios, they facilitate the creation and annotation of complex sketches, re-
can limit user creativity. Freeform 3D space sketching, on the sulting in more lucid, expressive, and comprehensible resources
other hand, demands a more sophisticated HCI design. In recent for medical teaching, treatment planning, and patient education.
years, HMD-based sketching systems have emerged [18], [19], In contrast to the realm of medical education, 3D sketching
[20], [21] to address this challenge. Common approaches include in clinical diagnosis and treatment is a demanding endeavor
constraining input dimensions and providing visual guidance to characterized by the need for efficiency and precision. The effi-
improve sketch alignment with users’ intentions. ciency and precision of these tasks often have a direct bearing on
One of the challenges in rendering 3D space is managing subsequent diagnosis and treatment outcomes. Therefore, within
the high degree of interaction freedom. Virtual 2D interactive the domain of medical space mapping, there is a heightened

Authorized licensed use limited to: Noakhali Science and Technology University. Downloaded on October 05,2025 at 10:02:22 UTC from IEEE Xplore. Restrictions apply.
 ZHANG et al.: ARMEDICALSKETCH: EXPLORING 3D SKETCHING FOR MEDICAL IMAGE 591

display of 3D sketch schemes of medical images. Given the

precision challenges inherent in interactions within environ-
ments with high spatial freedom, we utilize a 2D tablet as a
reliable input mechanism. Additionally, to ensure users accu-
rately perceive and position themselves relative to both the 3D
and 2D displays, we have designed a virtual synchronized inter-
active plane. The 3D-human interaction offers intuitiveness and
2D-human interaction ensures precision. Our 2D-3D interlinked
mode helps bridge the visualization and interaction advantages
present in these two distinct dimensions.
The combination of naked-eye-based 2D and 3D displays with
freehand interaction facilitates an easy-to-perceive and engaging
3D medical sketching environment. It allows multiple users to
view without glasses and engage effortlessly. The user’s hand
Fig. 1. System schematic. in the prototype can be replaced with any required input device.
For instance, an optical tracking pen can be used for higher
input precision, and a force feedback device can be employed
emphasis on optimizing interaction efficiency and precision. for perceiving the tactile sense of medical anatomical structures,
Heckel et al. [28] proposed sketch-based editing tools for tumor and so forth.
segmentation in 3D medical images using a 2D interface. Ijiri
and Yokota [29] introduced an innovative system for the intuitive IV. PROOF OF CONCEPT IMPLEMENTATION
and efficient refinement of medical volume segmentation by A. Medical Image Pro-Processing
modifying multiple curved contours, supporting both topolog-
ical manipulations and contour shape modifications. Johnson In our system, a specialized interface for medical image
et al. [30] explored VR and hybrid 2D/3D sketch-based inter- data has been developed. The workflow involves reading and
faces and visualizations for medical image segmentation, high- reorganizing the original medical images, transforming them
lighting the growing demand for 3D sketching in this domain. into 3D texture data optimized for use in the AR environment.
Human medical images inherently comprise 3D data, and This 3D texture data is then utilized in the AR environment for
many manual processing tasks in medical image analysis can be both 2D slice display and 3D volume rendering. To enhance
effectively summarized as 3D sketching. Therefore, a compre- medical image visualization, we have implemented tailored
hensive platform capable of supporting medical images from raw enhancements based on the 3D texture data.
data to 3D sketch processing is needed. Such a platform could In 2D display, an approach based on histogram normalization
encompass features such as anatomical feature markers and using 3D volumetric data was adopted to address challenges
surgical pathway planning, presenting a valuable contribution posed by the limited grayscale range and low image contrast
to the field of medical image analysis. often observed in medical images. This method effectively dis-
tributes voxel grayscale values from the medical 3D texture data
uniformly within the entire grayscale range of 0–1, significantly
III. OVERVIEW OF THE ARMEDICALSKETCH enhancing image clarity. Each voxel value (with dx : dy : dz =
Despite numerous studies having been conducted in the do- 1 : 1 : 1, representing the equal spatial resolution in the x, y,
mains of VR/AR 3D sketching and medical image sketching, and z dimensions resulting from preprocessing) is considered
systems enabling 3D spatial sketching through naked-eye dis- as a pixel. When statistically analyzing the grayscale range, the
plays have been rarely investigated. In this study, we present entire 3D volume data is considered as a collective, as opposed
ARMedicalSketch (Fig. 1), a novel medical 3D sketching en- to individual 2D planes. Similarly, during grayscale transfor-
vironment that supports genuine 2D and 3D displays and free mation, the alteration and distribution of grayscale values are
interaction based on the user’s hand. The displayed content in 2D computed using the complete 3D volume data as a unified set.
and 3D is dynamically linked in real-time, with the content in one Fig. 3 illustrates the effects of histogram normalization based on
dimension synchronously adjusting whenever a user interacts 3D volumetric data, enhancing medical image interpretation.
with the other dimension. In this study, the 3D volume rendering is implemented using
By combining naked-eye floating 3D and 2D images, our AR the raycasting algorithm. During the rendering process, we
medical sketching environment serves as an effective and precise sample and synthesize from the front to the back of the image
entry point for handling 3D sketching tasks in medical imaging. using the substitution.
The environment consists of three parts (Fig. 2): medical image ˙ i = (1 − Colȯri−1 .a) ∗ Colori + Colȯri−1 .
Color
pro-processing (Section IV-A), naked-eye-based 2D and 3D
display (Section IV-B), and interlinked visualization and inter- Here, Colori represents the color of the sample point, and
action (Section IV-C–D). ˙ i is the cumulative displayed color at position i. To
Color
The system employs 3D aerial autostereoscopic visualization improve the rendering effect, we incorporate three rendering
technology to provide true 3D visualization, enabling real-time parameters, I, D, and K, into the sampling process, defined by

Authorized licensed use limited to: Noakhali Science and Technology University. Downloaded on October 05,2025 at 10:02:22 UTC from IEEE Xplore. Restrictions apply.
 592 IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS, VOL. 54, NO. 5, OCTOBER 2024

Fig. 2. Overview of our approach.

Fig. 3. Histogram normalization enhancement based on three-dimensional

volume data. (a) Original medical image. (b) Processed medical image.

the substitutions.
Colori .a = D × pow (Colori .a, K)
Colori .rgb = I × Colori .rgb × Colori .a,
where I adjusts the color intensity of the sample point, D
determines the sampling depth, and K fine-tunes the contrast Fig. 4. Three-dimensional volume rendering enhancement visualization.
between colors at the sample point. Voxel values are mapped (a) Original medical image volume rendering. (b)–(d) Rendering results after
to a pseudocolor palette, allowing for selective enhancement parameter adjustments and pseudocolor mapping.
of specific aspects of the volume rendering display. Moreover,
we integrate boundary enhancement into the rendering process
to accurately showcase the range of data. Fig. 4(a) displays the Subsequently, the EIA is showcased on a liquid crystal display
original volume rendering of the medical image, while Fig. 4(b)– (LCD), and the images are converged into a 3D light field
(d) demonstrates the rendering outcomes after adjusting render- through a lens array. To achieve spatial interaction, we project
ing parameters and applying pseudocolor mapping. The data the 3D parallaxes into the air using an aerial imaging plate (AIP).
border can be tailored for display, including adjustments to AIP can project incident light into the air at identical angles. For a
transparency, color, and width. comprehensive understanding of the AIP’s structure and internal
light path, we refer to [11]. A light-field symmetrical multiview
B. True 2D and 3D Visualization Without Glasses rendering algorithm is employed to rectify viewpoint inversion
during pixel resampling, ensuring the correct multiview parallax
The system employs glasses-free naked-eye 2D tablet and 3D of the 3D parallax in the air. The viewing angle of the floating
floating display technology. For authentic 2D visualization, a 3D display is calculated as follows:
touch-enabled 2D display tablet is utilized. In this section, our
primary focus is elucidating the methodology for realistic 3D ϕ = 2 × arctan (pitch/2gap)
visualization. We employ the floating autostereoscopic display
technology, as depicted in Fig. 5. Initially, the system calculates where pitch represents the diameter of a single lens, and
the real-time capture of 3D parallaxes within the virtual AR envi- gap denotes the separation between the lens array and LCD.
ronment using a virtual camera array, resulting in the generation The advantage of the AIP-based floating 3D display is to
of an elemental image array (EIA) through pixel resampling. present undistorted proportional real images. Consequently, by

Authorized licensed use limited to: Noakhali Science and Technology University. Downloaded on October 05,2025 at 10:02:22 UTC from IEEE Xplore. Restrictions apply.
 ZHANG et al.: ARMEDICALSKETCH: EXPLORING 3D SKETCHING FOR MEDICAL IMAGE 593

Fig. 5. Rendering and display procedure of the floating 3D image, encompassing multiviewpoint image capture and subsequent 3D image reconstruction through
a lens array.

calibrating the physical position in the real world, we establish

a vertical projection relationship between 3D and 2D displays.
Our proposed naked-eye visualization approach eliminates
the need for wearable devices, offering an accessible and user-
friendly experience. The integration of 2D and 3D displays
enhances the precision of signal input through the tablet, while
the 3D display facilitates easily accessible 3D depth perception.

C. 2D-3D Interlinked Visualization

Enhanced visualization of medical 3D texture data is utilized
in our proposed platform for both the 2D tablet and a floating
3D display. In our research, we provide a 2D-3D interlinked
visualization method to seamlessly integrate 2D projection and
3D spatial representation. It ensures a close correlation between
the content displayed in 2D and the floating 3D projection. The Fig. 6. 2D-3D synchronized projection. (a) Horizontal plane. (b) and (c) Free
2D display serves as a projection, offering an accurate and easy- angle plane. First row: 3D volume rendering; second row: corresponding 2D
to-use display for users, while the floating 3D display brings the displays.
content to life in a spatial context, providing comprehensive and
immersive understanding. virtual plane based on model-space coordinates. During the
To achieve the integration, we adopt a parameter-based con- rendering process, we assign the voxel values to the coordinates
trol system and a virtual synchronized interactive plane that gov- of the virtual plane and incorporate them into the final output
erns the content displayed in both 2D and 3D. This means that screen color values using a front-to-back raycasting rendering
the same set of parameters is employed to regulate visualization approach. This integration places the interactive plane and the
in both dimensions, ensuring synchronicity and cohesiveness 3D medical images under a unified rendering framework, allow-
between the 2D projections and the 3D spatial representations. ing the depiction of their accurate tangential relationship, which
At the same time, the interactive plane will be presented in the is pivotal in achieving seamless synchronization and spatial
floating 3D display image, representing the orientation of the coherence.
2D plane in the floating 3D display. The control system includes three core parameters Ndir ,
In the volume rendering process, 3D medical images are Nslice and Npos . Ndir is a 3D vector representing the normal
visualized as a whole in space. Consequently, establishing a cor- vector of the synchronized interactive plane, Nslice is the dis-
rect spatial overlay relationship between an independent virtual tance from the center point to the interactive plane in 3D medical
interaction plane and the 3D medical images becomes nontrivial. image volume data, and Npos is the real-time interaction point
A virtual plane can typically either be entirely displayed in location of the user on the medical image. These parameters
front of or behind the volume-rendered medical images, but control critical aspects such as orientation, position, and ap-
fails to accurately represent the spatial pose relationship between pearance, allowing for a seamless transition between 2D and
them. Therefore, we adopted a technique grounded in volume 3D, enhancing the overall user experience and comprehension
rendering graphics pipelines to cohesively integrate the virtual of the medical data. Fig. 6 illustrates the results of 2D-3D
interaction plane with the 3D medical images. We utilize the interlinked visualization. By adjusting the parameters Ndir and
fragment shader program to compute the coordinates of the Nslice , changes in the 3D volume rendering image and the
Authorized licensed use limited to: Noakhali Science and Technology University. Downloaded on October 05,2025 at 10:02:22 UTC from IEEE Xplore. Restrictions apply.
 594 IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS, VOL. 54, NO. 5, OCTOBER 2024

TABLE I
CONFIGURATION PARAMETERS OF THE PROTOTYPE

Fig. 7. Setup of the prototype.

interactive plane within it can be observed. Clearly, a greater

set of parameters can be configured, such as those used to store
user interaction results.
Our 2D-3D interlinked visualization method leverages the
synergy between 2D projection and 3D spatial representation,
unifying them under a common set of parameters. This inno-
vative approach not only enriches the visualization experience
but also maximizes the potential for effective interpretation and
interaction with complex medical data.

D. 2D-3D Interlinked Interaction With Hand and Tablet

To facilitate seamless HCI within the environment, we employ
a fully user-centric interaction mode, leveraging the user’s hands Fig. 8. (a) 2D-3D interlinked visualization. (b) Rendered EIA imaging of
extensively. This interaction is made possible through a sophis- floating 3D display.
ticated integration of sensory inputs from two critical sources:
the 2D tablet and gesture sensors.
The user’s hands serve as the focal point of interaction, configured with the specifications in Table I. The prototype con-
acting as conduits for input into our system. The interaction tains two LCD panels, an MLA, an AIP, a gesture recognition,
process is orchestrated through a combination of sensory data and a computer workstation. LCD 1 is utilized for the floating
obtained from the 2D tablet and gesture sensors. The 2D tablet, 3D display and LCD 2 for 2D display and touch.
equipped with touch-sensitive capabilities, forms a pivotal com- Fig. 8(a) shows the 2D-3D interlinked visualization with a
ponent of this dynamic interplay, enabling precise and intuitive floating 3D image and the corresponding tumor image in 2D
touch-based inputs from the user. Complementing the 2D tablet, tablet display. We used a skeleton model to demonstrate the 3D
gesture sensors play a crucial role in enriching the interaction floating horizontal parallax. Fig. 8(b) shows a rendered EIA of
dynamics. The sensors capture the nuances of hand gestures and the skeleton model for the floating 3D display, and Fig. 9 shows
the positioning of fingertips. By amalgamating the touch-based the 3D aerial autostereoscopic display with horizontal parallax.
inputs from the 2D tablet and the hand data from gesture sensors, In the prototype interaction experiment, users can utilize ges-
our system hopes to achieve a comprehensive and sophisticated tures to modify the display interaction plane, resulting in changes
understanding of the user’s intent and actions. in both 2D and 3D images [Fig. 10(a) and (b)]. Furthermore,
Our 2D-3D interlinked interaction approach leverages the when users draw the tumor boundary on the touchscreen, the
user’s hands for intuitive interaction. The integration of sensory floating 3D display synchronously presents the drawn contours
inputs from the 2D tablet and gesture sensors offers a cohesive, [Fig. 10(c) and (d)]. The rendering rate exceeded 110 frames per
enriched interaction experience, contributing to effective inter- second. More control gestures can be designed to meet a variety
pretation and interaction with intricate medical data. of medical sketching objectives.

V. IMPLEMENTATION AND EVALUATION B. Application Walkthroughs

A. System Architecture and Technical Details We demonstrate the 3D sketching capabilities of our proposed
The proposed prototype is realized through Unity 2020.3.8. system through tumor segmentation and tumor puncture path
Considering the hardware construction, the prototype (Fig. 7) is planning.
Authorized licensed use limited to: Noakhali Science and Technology University. Downloaded on October 05,2025 at 10:02:22 UTC from IEEE Xplore. Restrictions apply.
 ZHANG et al.: ARMEDICALSKETCH: EXPLORING 3D SKETCHING FOR MEDICAL IMAGE 595

Fig. 12. Tumor puncture pathing view in (a) frontal and (b) top-down perspec-
tives.

the 3D autostereoscopic aerial display (bold green outline).

Fig. 9. 3D aerial autostereoscopic display with parallax effect.
After completing the drawing of a tumor boundary, a search
is conducted within the medical 3D volume data based on the
drawn boundary. Using the tumor voxel values for growth, a
3D segmented tumor label is generated, and then a 3D surface
model of the label is generated based on the Marching Cubes
algorithm.
2) Scenario 2. Path Planning: Fig. 12 showcases the tumor
puncture path sketching effect. Users adjust the synchronous in-
teraction interface to the suitable tumor puncture position using
the left hand and then mark the puncture starting position on the
2D touchscreen using the right hand. Then, the left-hand inter-
acts to change the synchronous interaction plane Nslice value.
A real-time puncture path is generated in the 3D volume ren-
dering, synchronously changing with the Nslice , and when the
user marks the second puncture termination position on the 2D
Fig. 10. User prototype interaction experiment. (a) and (b) Gestures alter 2D- tablet, the final puncture path is determined and generated. The
3D display content. (c) and (d) Touch gestures draw the tumor boundary. coordinates of the completed sketching points and lines can be
adjusted according to the user’s preference. The provided sketch-
ing system assists users in achieving the desired drawing results.

C. User Study
1) Purpose: The control test evaluated the following two
parameters. First, training effectiveness: a) determining whether
the system improved the sketching efficiency of medical im-
ages; b) determining whether the system improved the accuracy
of sketching medical images. Second, system performance:
a) determining the system’s usability and participant satisfac-
tion, and b) determining the degree of psychological burden on
participants caused by system use.
2) Participants: Twenty-three biomedical engineering stu-
dents volunteered to participate in the user study and completed
Fig. 11. Synchronized effect of tumor boundary sketching. (a) 2D drawing the consent form. Despite 11 having a basic medical knowledge
interface. (b) Touch drawing of tumor boundary. (c) and (d) Bold 3D spatial foundation, participants had no actual expertise in organ diag-
boundary.
nosis of medical images. Participants ranged in age from 21 to
33 years (average = 25 years) and included 13 males and 10
1) Scenario 1. Tumor Segmentation: Fig. 11 illustrates the females. None of the participants in the study were involved
synchronous effect of tumor boundary sketching. Users adjust in the system’s creation. Three participants were assigned to
the synchronous interaction interface to the appropriate tumor test the prototype, and based on their feedback, we established
position using their left hand and then draw the tumor boundary the system’s learning curve process time in procedures. The
on the 2D tablet using their right hand. Simultaneously, the remaining twenty participants were randomly assigned to an
drawn boundary appears in the volume rendering image of experimental group or a control group, each with ten students.

Authorized licensed use limited to: Noakhali Science and Technology University. Downloaded on October 05,2025 at 10:02:22 UTC from IEEE Xplore. Restrictions apply.
 596 IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS, VOL. 54, NO. 5, OCTOBER 2024

Fig. 13. Labeling experiment of Willis’s circle. (a) Willis’s circle enhanced
rendering through the floating 3D display. (b) Users marked anatomical points
through the 2D tablet.

Willis’s circle, recognized as one of the complex anatomical

structures at the base of the human brain, is the intersection
of several important arteries. We collected five MR images of
Willis’s circle, split them into a comparable sample set, and
minimized differences in position. Similarly, we collected and
Fig. 14. Results of the user study on training effectiveness. (a) Task 1 involved
processed ten healthy liver CT images. identifying liver regions. (b) Task 2 focused on labeling key points of Willis’s
3) Procedures: We set up two prototypes. Our proposed pro- circle. (∗∗: p<0.01).
totype was Prototype 1, which featured a naked-eye floating 3D
TABLE II
display alongside a 2D tablet. Prototype 2 replaced the floating RESULTS FOR THE QUESTIONNAIRES: MEAN (INTERQUARTILE RANGE)
3D display with a standard 2D flat display for visualizing 3D
medical models rendered using a single virtual camera, and also
along with a 2D tablet. Different prototypes were utilized by
different groups.
The experiment comprised five stages: (1) introduction, (2)
initial testing, (3) learning, (4) follow-up testing, and (5) ques-
tionnaire period. Initially, around 20 min were allocated for frustration. For the SUS questionnaire, responses were recorded
each participant to be introduced to their prototype and learn on a five-point Likert scale and converted to a usability score
about the structure of Willis’s circle and liver imaging region. ranging from 0 to 100, where higher SUS scores indicate better
Subsequently, volunteers underwent two tasks for the first time. usability. In the workload assessment, participants rated each
Task 1 involved selecting any region within the liver images, dimension on a scale from 0 to 10, and provided pairwise
while Task 2, a more challenging test, focused on Willis’s comparisons to determine their relative importance as weights.
circle (Fig. 13), where volunteers were required to mark the Overall workload scores, ranging from 0 to 100, were calcu-
anterior communicating artery and the top of the basilar artery lated based on these ratings, with lower scores indicating lower
by interacting with their prototype. Participants then spent 20 workload levels.
min learning about their prototype, during which they were Table II shows two questionnaires’ results on system per-
encouraged to engage freely with the system during the learning formance, including the score mean, interquartile range, and
process. Following this learning phase, volunteers underwent significance levels (one-way ANOVAs). There was significance
the same two tests for a second time. We documented the time in the SUS questionnaire.
and marked locations. 4) Clinical Expert Evaluation: We invited an orthopedic sur-
Fig. 14 presents the results, including mean scores, standard geon with seven years of clinical experience and a neurosurgeon
deviations, and significance across prototypes for interaction du- with three years to provide expert feedback after using our
ration and accuracy across two tests. In the accuracy assessment, system. Both acknowledged that the glasses-free 3D spatial
the precision of the liver experiment was measured by users’ HCI technology provides an intuitive, immersive visualization
ability to identify points within the liver region, while the preci- experience, enhancing comprehension and evaluation of intri-
sion of Willis’s experiment was gauged by positional deviation cate anatomical structures for improving precision and safety
distances. A critical p-value of 0.05 is used for the hypothesis test in medical imaging tasks. The integration of a 2D touch-screen
to determine whether the interprototype difference is significant, tablet with 3D floating image projection enables seamless an-
and significant findings are depicted in Fig. 14 asterisks. ANOVA notation and manipulation on a 2D plane, with correspond-
tests were used to obtain p-values. Prototype 1 exhibited higher ing 3D effects, facilitating flexible surgical path planning and
completion efficiency in Task 2. procedural guidance. The platform’s versatility across various
Participants were also asked to complete two questionnaires: minimally invasive surgical tasks, including tumor resection
(1) assessing system usability using the System Usability Scale and endovascular interventions, offers a novel adjunctive tool
(SUS) [31]; (2) evaluating workload with the NASA-TLX as- to optimize surgical efficiency and success rates.
sessment tool [32], gauging cognitive load with mental demand, However, for more intricate surgical procedures, further im-
physical demand, temporal demand, performance, effort, and provements and optimizations may be necessary to align with
Authorized licensed use limited to: Noakhali Science and Technology University. Downloaded on October 05,2025 at 10:02:22 UTC from IEEE Xplore. Restrictions apply.
 ZHANG et al.: ARMEDICALSKETCH: EXPLORING 3D SKETCHING FOR MEDICAL IMAGE 597

practical requirements, particularly in adapting to interpatient spatial interaction through the projected visual association of 2D
variations and addressing unforeseen challenges across clinical and 3D visualizations [22].
scenarios and specialties. Limitations in the clarity and stability Leveraging the strengths in visualization and interaction,
of the floating 3D images could affect the observation and the integration of ARMedicalSketch into medical environments
interpretation of fine details and structures. Recommendations holds considerable promise. As an innovative HCI system de-
include improving the quality and clarity of the floating 3D veloped for medical use, it provides users with glasses-free
images to ensure precise visualization, and fortifying system 3D spatial perception and an intuitive, precise mode of inter-
stability and real-time responsiveness to facilitate relevant sup- action. The system serves diverse diagnostic, therapeutic, and
port. educational purposes by customizing interactive interfaces to
suit various clinical scenarios. For instance, in neurosurgery or
interventional cardiology, ARMedicalSketch enhances diagnos-
VI. DISCUSSION tic capabilities by visualizing intricate case images, facilitates
Overall, our prototype was well-received by the volunteers segmentation and planning of 3D radiation targets in radiology,
and clinical experts. Based on the results of the user experi- and aids in explaining lesion presentations to patients and their
ments, Prototype 1 demonstrated superior performance and user families, thereby enhancing comprehension of treatment strate-
acceptance. In Fig. 14, the completion time for Task 2 indicated gies and surgical procedures. Additionally, the naked-eye 3D
a significant improvement in efficiency for Prototype 1 when floating display and gesture interaction module can be combined
handling challenging 3D medical tasks. In Table II, the results independently to form a fully floating, noncontact 3D HCI
of the SUS revealed that users appreciated Prototype 1 for its system, extending ARMedicalSketch’s potential application de-
system usability and demonstrated a high level of acceptance. mands to the operating room, including path planning display
Results from both the initial and follow-up tests showed a and intraoperative navigation.
slight improvement in task completion efficiency and accuracy, The system is compatible with standard DICOM medical im-
demonstrating the usability of the proposed prototypes. age formats for seamless integration into existing systems within
User experiments confirm that the proposed true 2D-3D medical institutions, facilitating easy access and manipulation
interlinked visualization and interaction efficiently facilitates of patient medical images by healthcare professionals. While
interactive tasks based on 3D medical images. Fig. 14 shows introducing ARMedicalSketch into clinical settings requires
that the combined display mode of floating 3D visualization and careful considerations, it will strictly adhere to relevant regu-
projected 2D display significantly improves user interaction effi- latory standards and undergo validation to ensure continuous
ciency compared with traditional 2D displays when dealing with compliance. This includes robust data security, stringent user
complex 3D image tasks (such as Task 2). For medical image privacy protection, and system stability measures. Regarding
tasks with weak 3D spatial correlation (such as Task 1), the hardware deployment, it is implemented in sealed facilities with
efficiency of Prototype 1 and traditional 2D devices are similar. integrated tablets, facilitating strict adherence to disinfection re-
These findings align with previous research in related fields. quirements in line with healthcare institution hygiene standards.
Studies in surgical applications [33] and anatomical education Furthermore, ongoing plans for feedback iteration mechanisms
[34] have consistently shown that 3D visualization improves are in place to better align with the needs and preferences of
performance in 3D image recognition tasks. Additionally, a clinical users. Ultimately, as a medical device, ARMedicalS-
comparative study [35] of 3D and 2D camera-based laparoscopic ketch will need to comply with relevant regulatory standards
procedures revealed that interactions facilitated by 3D visualiza- and regulations in the future to ensure its safety, efficacy, and
tion significantly reduced error rates. Table II further indicates compliance.
that spatial gesture interaction based on 3D autostereoscopic Our research addresses the gap in naked-eye 2D-3D fusion
aerial display is acceptable and does not significantly increase visualization and interaction. However, this work still has several
user workload. Therefore, the proposed naked-eye 2D-3D visual limitations. There is room for improvement in the interaction
interaction interface is conducive to assisting volunteers in rapid accuracy of the prototype. In this study, we captured interaction
spatial localization. information through users’ hands, but the positioning informa-
One of the innovative aspects of our work lies in the fusion tion at the fingertip or obtained by finger touch is relatively
visualization of true 2D and 3D. In Fig. 14, the interaction coarse. Future plans include designing a spatial pen with 3D
accuracy of users with Prototypes 1 and 2 is similar and does optical tracking and 2D tablet touch functions, replacing hand
not significantly differ. When it comes to 3D spatial tasks, touch while retaining quick gesture control. Additionally, we
Prototype 1 shows a slightly more accurate representation, which recruited 23 volunteers for this study, and aim to expand the
is speculated to be due to the spatial perception provided by participant pool for a comprehensive evaluation, comparing our
the floating 3D autostereoscopic display, facilitating the user’s system with HMDs, traditional 2D flat displays, and freehand
visual judgment. The result of the SUS in Table II suggests that handheld tablets in 3D medical sketching in future work.
introducing floating 3D visualization in the naked-eye display
significantly enhances the usability level of the system. These
results indicate that our proposed true 2D-3D visualization in- VII. CONCLUSION
terface provides a high-precision 3D spatial interaction method, In this article, we explored the realm of 3D sketching for medi-
achieving an interaction accuracy level comparable to that of cal images by leveraging the potential of true 2D-3D interlinked
a 2D interface. This result is promising, achieving precise 3D visualization and interaction, and presenting a novel medical
Authorized licensed use limited to: Noakhali Science and Technology University. Downloaded on October 05,2025 at 10:02:22 UTC from IEEE Xplore. Restrictions apply.
 598 IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS, VOL. 54, NO. 5, OCTOBER 2024

3D sketching AR environment named ARMedicalSketch. In [12] H. Liao, T. Inomata, I. Sakuma, and T. Dohi, “3-D augmented reality
clinical practice, 3D interaction based on 2D medical image for MRI-guided surgery using integral videography autostereoscopic im-
age overlay,” IEEE Trans. Biomed. Eng., vol. 57, no. 6, pp. 1476–1486,
displays is challenging due to a mismatch between dimensions, Jun. 2010.
and current 3D drawing methods with HMDs limit prolonged use [13] N. Zhang, T. Huang, X. Zhang, C. Hu, and H. Liao, “Omnidirectional
for medical professionals. Our ARMedicalSketch presents a new 3D autostereoscopic aerial display with continuous parallax,” J. Opt. Soc.
Amer. A, vol. 39, no. 5, pp. 782–792, 2022.
approach by offering an interaction interface devoid of the need [14] A. Maimone, A. Georgiou, and J. S. Kollin, “Holographic near-eye dis-
for glasses, presenting a seamless fusion of both floating 3D dis- plays for virtual and augmented reality,” ACM Trans. Graph., vol. 36, no. 4,
play and 2D tablet display capabilities. This amalgamation em- pp. 1–16, 2017.
[15] T. Inoue and Y. Takaki, “Table screen 360-degree holographic display
powers users to interact with 3D medical imaging tasks without using circular viewing-zone scanning,” Opt. Express, vol. 23, no. 5,
the encumbrance of wearing glasses. We introduced a method pp. 6533–6542, 2015.
of 2D-3D interlinked visualization and interaction, employing [16] D. L. MacFarlane, “Volumetric three-dimensional display,” Appl. Opt.,
vol. 33, no. 31, pp. 7453–7457, 1994.
synchronized projection visualization and a virtual synchronized [17] R. Balakrishnan, G. W. Fitzmaurice, and G. Kurtenbach, “User interfaces
interactive plane to establish an integrated relationship between for volumetric displays,” Computer, vol. 34, no. 3, pp. 37–45, 2001.
2D and 3D displays. The outcomes gleaned from our user study [18] D. Keefe, R. Zeleznik, and D. Laidlaw, “Drawing on air: Input techniques
for controlled 3D line illustration,” IEEE Trans. Visual. Comput. Graph.,
unveil a promising trajectory. When compared with traditional vol. 13, no. 5, pp. 1067–1081, Sep./Oct. 2007.
2D interaction interfaces, our true 2D-3D system significantly [19] S.-H. Bae, R. Balakrishnan, and K. Singh, “ILoveSketch: As-natural-as-
amplifies the efficiency of 3D interactive tasks related to medical possible sketching system for creating 3D curve models,” in Proc. 21st
Annu. ACM Symp. User Interface Softw. Technol., 2008, pp. 151–160.
image visualization while preserving precision and accuracy. [20] R. C. Zeleznik, K. P. Herndon, and J. F. Hughes, “SKETCH: An interface
ARMedicalSketch provides a novel glasses-free 3D spatial in- for sketching 3D scenes,” in Proc. ACM SIGGRAPH 2006 Courses, 2006,
teraction method tailored specifically for interactive tasks in 3D Art. no. 9-es.
[21] D. F. Keefe, D. A. Feliz, T. Moscovich, D. H. Laidlaw, and J. J. LaViola
medical images. In the future, our aim is to refine the precision Jr, “CavePainting: A fully immersive 3D artistic medium and interactive
of interactions, expand the spectrum of application scenarios experience,” in Proc. Symp. Interactive 3D Graph., 2001, pp. 85–93.
for ARMedicalSketch, and even introduce machine learning [22] R. Arora, R. Habib Kazi, T. Grossman, G. Fitzmaurice, and K. Singh,
“SymbiosisSketch: Combining 2D & 3D sketching for designing detailed
methods, such as automated feature detection and segmentation, 3D objects in situ,” in Proc. CHI Conf. Hum. Factors Comput. Syst., 2018,
as well as connect to large-scale language models, to improve pp. 1–15.
the intelligence of the system. This will unlock its full potential [23] Y. Jiang et al., “Handpainter-3D sketching in VR with hand-based physical
proxy,” in Proc. CHI Conf. Hum. Factors Comput. Syst., 2021, pp. 1–13.
in medical imaging and beyond. [24] S. Tsang, R. Balakrishnan, K. Singh, and A. Ranjan, “A suggestive
interface for image guided 3D sketching,” in Proc. SIGCHI Conf. Hum.
Factors Comput. Syst., 2004, pp. 591–598.
ACKNOWLEDGMENT [25] E. Yu, R. Arora, T. Stanko, J. A. Bærentzen, K. Singh, and A. Bousseau,
The authors would like to thank Z. Liao and J. Liu for helping “Cassie: Curve and surface sketching in immersive environments,” in Proc.
CHI Conf. Hum. Factors Comput. Syst., 2021, pp. 1–14.
design and print the prototype frame. [26] P. Saalfeld, A. Stojnic, B. Preim, and S. Oeltze-Jafra, “Semi-immersive
3D sketching of vascular structures for medical education,” in Proc.
VCBM/MedViz, 2016, pp. 123–132.
REFERENCES [27] P. Saalfeld, S. Oeltze-Jafra, S. Saalfeld, U. Preim, O. Beuing, and B. Preim,
[1] D. Gasques, J. G. Johnson, T. Sharkey, and N. Weibel, “What you sketch is “Sketching and annotating vascular structures to support medical teaching,
what you get: Quick and easy augmented reality prototyping with PintAR,” treatment planning and patient education,” in Proc. Eurographics (Dirk
in Proc. CHI Conf. Hum. Factors Comput. Syst., 2019, pp. 1–6. Bartz Prize), 2017, pp. 5–8.
[2] N. Zhang, H. Wang, T. Huang, X. Zhang, and H. Liao, “A VR environment [28] F. Heckel, J. H. Moltz, C. Tietjen, and H. K. Hahn, “Sketch-based editing
for human anatomical variation education: Modeling, visualization and tools for tumour segmentation in 3D medical images,” Comput. Graph.
interaction,” IEEE Trans. Learn. Technol., vol. 17, pp. 391–403, 2024. Forum, vol. 32, no. 8, pp. 144–157, 2013.
[3] M. F. Deering, “HoloSketch: A virtual reality sketching/animation tool,” [29] T. Ijiri and H. Yokota, “Contour-based interface for refining volume
ACM Trans. Comput.-Human Interact., vol. 2, no. 3, pp. 220–238, 1995. segmentation,” Comput. Graph. Forum, vol. 29, no. 7, pp. 2153–2160,
[4] T. Drey, J. Gugenheimer, J. Karlbauer, M. Milo, and E. Rukzio, “VRS- 2010.
ketchin: Exploring the design space of pen and tablet interaction for 3D [30] S. Johnson, B. Jackson, B. Tourek, M. Molina, A. G. Erdman, and
sketching in virtual reality,” in Proc. CHI Conf. Hum. Factors Comput. D. F. Keefe, “Immersive analytics for medicine: Hybrid 2D/3D sketch-
Syst., 2020, pp. 1–14. based interfaces for annotating medical data and designing medical de-
[5] J. H. Shuhaiber, “Augmented reality in surgery,” Arch. Surg., vol. 139, vices,” in Proc. ACM Companion Interactive Surfaces Spaces, 2016,
no. 2, pp. 170–174, 2004. pp. 107–113.
[6] L. Besançon, M. Sereno, L. Yu, M. Ammi, and T. Isenberg, “Hybrid [31] J. Brooke, “SUS: A quick and dirty usability scale,” in Usability Evaluation
touch/tangible spatial 3D data selection,” Comput. Graph. Forum, vol. 38, Industry, P. W. Jordan, B. Thomas, I. L. McClelland, and B. Weerdmeester,
no. 3, pp. 553–567, 2019. Ed. London, U.K.: CRC Press, 1996, pp. 4–7.
[7] M. D. B. Machuca, W. Stuerzlinger, and P. Asente, “Smart3DGuides: [32] S. G. Hart and L. E. Staveland, “Development of NASA-TLX (Task
Making unconstrained immersive 3D drawing more accurate,” in Proc. Load Index): Results of empirical and theoretical research,” in Advances
25th ACM Symp. Virtual Reality Softw. Technol., 2019, pp. 1–13. in Psychology, vol. 52. Amsterdam, The Netherlands: Elsevier, 1988,
[8] C. R. Spitzer and C. Spitzer, Ed., Digital Avionics Handbook, 1st ed., CRC pp. 139–183.
Press, 2000. [Online]. Available: [Link] [33] C. Lin et al., “Three-dimensional visualization technology used in pan-
[9] Y. Gao, Y. Liu, D. Cheng, Y. Wang, and S. Optoelectronics, “A review on creatic surgery: A valuable tool for surgical trainees,” J. Gastrointestinal
development of head mounted display,” J. Comput.-Aided Des. Comput. Surg., vol. 24, no. 4, pp. 866–873, 2020.
Graph., vol. 28, no. 6, pp. 896–904, 2016. [34] T. Itamiya et al., “A novel anatomy education method using a spatial reality
[10] H. Hua, “Enabling focus cues in head-mounted displays,” Proc. IEEE, display capable of stereoscopic imaging with the naked eye,” Appl. Sci.,
vol. 105, no. 5, pp. 805–824, May 2017. vol. 11, no. 16, 2021, Art. no. 7323.
[11] N. Zhang, T. Huang, X. Zhang, and H. Liao, “35.1: A novel in-situ [35] S.-H. Kong et al., “Comparison of two-and three-dimensional camera sys-
interactive 3D floating autostereoscopic display system with aerial imaging tems in laparoscopic performance: A novel 3D system with one camera,”
plate,” in Proc. SID Symp. Dig. Tech. Papers, 2021, vol. 52, pp. 244–247. Surg. Endoscopy, vol. 24, pp. 1132–1143, 2010.

Authorized licensed use limited to: Noakhali Science and Technology University. Downloaded on October 05,2025 at 10:02:22 UTC from IEEE Xplore. Restrictions apply.