Module 5 - Creating Cross-

Platform Augmented Reality and

Virtual Reality
Understanding 3D graphics
 Basic 3D Graphics Concepts
 3D graphics simulate a three-dimensional environment
using two-dimensional screens, requiring certain
fundamental concepts.
• Coordinate Systems:
• World Space: The global coordinate system that defines the
position and orientation of objects in the 3D scene.
• Local Space: The coordinate system relative to an object, used
to define its size, rotation, and position within the scene.
• Camera Space (View Space): The space viewed from the
camera's perspective, showing what the player/user will see.
Cont.,
• Mesh: A 3D model composed of vertices, edges, and faces
that form the object’s surface. In AR/VR, objects in the
virtual world are represented by meshes.
• Vertices and Polygons: Vertices are points in 3D space,
and when connected, they form polygons, usually triangles
or quads, which define the surface of a 3D object.
• Textures: 2D images applied to the surface of a 3D model
to give it color, detail, or appearance (e.g., roughness,
smoothness).
• Shaders: Small programs that run on the GPU, responsible
for determining how an object looks by controlling its
surface properties, light interaction, and visual effects (like
reflections or transparency).
Cont.,
 Rendering in AR/VR
 Rendering is the process of generating a 2D image from 3D
objects using a virtual camera. In AR/VR, rendering must
be real-time to ensure smooth and responsive interactions.
• Real-Time Rendering: AR/VR applications require real-
time rendering at high frame rates (often 60 FPS for AR,
90–120 FPS for VR) to maintain immersion and avoid
motion sickness.
• Field of View (FOV): The FOV defines how wide the
camera's view is. In VR, a wider FOV (usually around 110°
or higher) is needed to create an immersive experience.
• Stereoscopic Rendering: VR requires rendering two
slightly different perspectives (one for each eye) to create a
stereoscopic 3D effect, allowing depth perception. This is
crucial for immersion in VR.
Cont.,
 Lighting and Shadows
 Lighting is a key element in making 3D graphics appear realistic
and immersive.
• Types of Lights:
• Directional Light: Simulates light from a faraway source, like the sun.
• Point Light: Emits light from a single point in all directions (e.g., a light
bulb).
• Spotlight: Emits light in a specific direction within a cone, like a
flashlight.
• Ambient Light: A general, indirect light that simulates global
illumination from the environment.
• Shadows: Adding shadows gives depth and realism. In AR,
shadows help virtual objects blend with the real world, grounding
them in the scene.
• Global Illumination (GI): Simulates the way light bounces
around a scene, producing more realistic lighting and reflections,
important for creating immersive environments.
Cont.,
 AR and VR Specific Concepts
 AR-Specific Concepts:
• Environment Tracking: AR relies on tracking the user’s
environment to correctly place virtual objects in the real world.
Techniques like SLAM (Simultaneous Localization and Mapping)
are used to map the physical world in real-time.
• Anchors: Points in the real world where virtual objects are placed.
These objects remain in the same location even if the camera moves,
ensuring they are "anchored" to the real environment.
• Occlusion: This refers to the effect where real-world objects block
the view of virtual objects. AR systems can use depth sensors or
cameras to create a depth map and render virtual objects accordingly,
providing a sense of realism.
• Lighting Estimation: AR systems often estimate the lighting
conditions of the real world and apply these lighting conditions to
virtual objects to make them appear more integrated with the real
world.
Cont.,
 VR-Specific Concepts:
• Immersion and Presence: The goal of VR is to create a
sense of presence, where users feel like they are truly
inside the virtual environment. This involves high-quality
3D rendering, realistic lighting, and minimal latency.
• 6DoF (Six Degrees of Freedom): In VR, users have six
degrees of freedom — three for movement (X, Y, Z) and
three for rotation (pitch, yaw, roll). The virtual environment
must track these motions to adjust the camera accordingly.
• Latency: In VR, latency is critical. Any lag between user
movements and updates to the virtual scene can cause
discomfort or motion sickness. Low-latency rendering (less
than 20 ms) is essential for smooth VR experiences.
 Spatial Audio
 Audio is an important part of AR/VR experiences, helping
enhance immersion. Spatial audio involves rendering sound
based on the position of objects in 3D space and the user's
head position. This allows users to perceive where sounds
are coming from, adding another layer of realism.
 Interaction in AR/VR
• Hand Tracking and Gesture Recognition: In VR, users
interact with virtual objects using controllers or hand
tracking. In AR, interactions may involve gestures or
tapping on the screen.
• Haptics: VR experiences often use haptic feedback
(vibrations or resistance) to simulate touch and enhance
immersion.
Cont.,
 Physics Simulation
• Physics simulations in AR/VR ensure that objects behave in a
realistic manner. For example, virtual objects should follow the
laws of gravity, momentum, and collision detection. Realistic
physics helps reinforce the user’s sense of presence in VR and
immersion in AR.
• Rigid Body Physics: Simulates solid, non-deformable
objects.
• Soft Body Physics: Simulates objects that can deform (like
rubber).
• Collision Detection: Ensures that virtual objects don’t pass
through one another and interact realistically.
Cont.,
 Optimization for AR/VR
 AR/VR applications require extensive optimization to
ensure smooth performance across different hardware
platforms.
• Level of Detail (LOD): Simplifying the geometry or
texture of distant objects to reduce computational load.
• Baked Lighting: Pre-calculating lighting effects
(especially global illumination and shadows) to reduce
runtime computational load.
• Occlusion Culling: Not rendering objects that are out of
view or blocked by other objects to save processing power.
Cont.,
 Frameworks and APIs
 Several frameworks and APIs help developers build AR/VR
applications with 3D graphics:
• Unity and Unreal Engine: Both game engines are highly
popular for AR/VR development and provide 3D graphics
capabilities.
• WebXR: A web standard for delivering VR and AR
experiences directly in web browsers.
• ARCore (Google) and ARKit (Apple): SDKs that provide
tools for building AR experiences on Android and iOS
devices, respectively.
• OpenXR: A cross-platform API standard for VR and AR that
allows developers to write a single codebase that works
across multiple VR/AR hardware platforms.
Virtual Camera
 Virtual Camera in VR (Virtual Reality)
 In VR, the virtual camera mimics the user’s head movement and
controls the view into the fully immersive virtual environment. This
gives the user the impression of looking around a virtual world in 3D
space.
 Key Concepts in VR Camera Systems:
• Stereoscopic Rendering:
• In VR, two virtual cameras are used—one for each eye—positioned slightly
apart (typically the distance between the user's eyes, called the
interpupillary distance or IPD). This simulates binocular vision, allowing
the user to perceive depth and get a 3D view of the world.
• 6 Degrees of Freedom (6DoF):
• The virtual camera in VR tracks both position and orientation with 6DoF
(degrees of freedom). This allows users to:
• Move along the X, Y, Z axes (translation: forward-backward, up-down, left-right)
• Rotate around the X, Y, Z axes (rotation: pitch, yaw, roll)
• This means if a user moves their head or body in the real world, the virtual
camera adjusts to reflect those movements, making it feel like they are
actually "inside" the virtual world.
Cont.,
• Field of View (FOV):
• The virtual camera’s FOV simulates the user’s natural peripheral vision. In
VR, a wider FOV (typically between 90° and 120°) is used to create a more
immersive experience. Narrow FOVs can reduce immersion and lead to
discomfort.
• Latency and Frame Rate:
• A low-latency virtual camera system is critical in VR. Any delay in
translating the user’s real-world movements to changes in the virtual
camera’s position and orientation can cause motion sickness. A high frame
rate (90–120 FPS) is also necessary for smooth, realistic rendering.
• Camera Frustum:
• The virtual camera in VR uses a frustum (a truncated pyramid shape) to
define what is visible to the user. Objects inside the frustum are rendered,
while those outside it are culled to save computational power.
• Head-Tracking Sensors:
• VR systems use head-tracking sensors (gyroscopes, accelerometers, and
sometimes external tracking cameras) to update the position and
orientation of the virtual camera in real-time, ensuring that the user’s view
responds naturally to head movements.
Cont.,
 Virtual Camera in AR (Augmented Reality)
 In AR, the virtual camera works differently than in VR, as it is blended
with a live video feed of the real world. The virtual camera places virtual
objects into the real-world environment, ensuring they align with real-
world physics and perspectives.
 Key Concepts in AR Camera Systems:
• Camera Overlay:
• In AR, the virtual camera captures and displays the real-world environment
through the device's physical camera (smartphone, tablet, AR glasses, etc.). The
virtual objects are then superimposed or “augmented” onto this real-world view.
• Real-World Anchoring:
• The virtual camera in AR must ensure that virtual objects are anchored to
specific locations in the real world. This is done using techniques like:
• SLAM (Simultaneous Localization and Mapping), which helps the system
understand the spatial layout of the environment.
• Marker-based tracking, where the camera recognizes specific patterns or images in
the real world to place virtual objects.
• Markerless tracking using planes, surfaces, and feature points detected in the real
world to position virtual objects.
Cont.,
• Camera Alignment and Calibration:
• To make virtual objects appear as though they exist in the real world, the AR
virtual camera must align its perspective with the real camera’s FOV, focal length,
and orientation. This ensures that virtual objects appear correctly scaled and in
the right position relative to the physical world.
• Depth Perception and Occlusion:
• Some AR systems, especially those on advanced devices like HoloLens or Magic
Leap, use depth sensors along with the virtual camera to create a depth map of
the real world. This enables virtual objects to interact more naturally with the
physical environment, including occlusion (where real objects block virtual
objects and vice versa).
• Lighting Estimation:
• To integrate virtual objects seamlessly into the real world, AR systems often use
the virtual camera to estimate the lighting conditions of the surrounding
environment. This data allows the system to adjust the lighting, shadows, and
reflections of virtual objects to match the real-world scene.
• Real-Time Rendering:
• AR requires real-time rendering of virtual objects with constant feedback from the
camera and environment sensors. Virtual objects need to move and react based
on camera motion (as the user moves their device or head), so the virtual camera
Cont.,
 Differences Between AR and VR Virtual Cameras
• Viewpoint:
• In VR, the virtual camera represents the user’s head and defines the entire
view of the virtual world.
• In AR, the virtual camera operates on top of a live feed from the physical
camera and merges virtual objects into that real-world view.
• Immersion:
• In VR, the virtual camera is fully responsible for creating the user's
immersive environment, controlling what the user sees in the entirely virtual
space.
• In AR, the virtual camera augments reality by placing digital elements
within the real-world context, rather than creating a fully immersive
environment.
• Rendering:
• In VR, the camera must render a fully 3D environment from scratch,
handling everything from textures and lighting to shadows and reflections.
• In AR, the camera must handle both the real-world feed and the rendering of
virtual objects that interact seamlessly with the real world (including
lighting, scaling, and positioning).
Cont.,
 Tracking Technologies for Virtual Cameras
 Both AR and VR rely on tracking technologies to ensure the
virtual camera provides an accurate, responsive view:
• Inside-out Tracking:
• Used in AR and some VR systems, inside-out tracking uses
cameras and sensors on the device itself to track movement
relative to the environment (e.g., Microsoft HoloLens, Oculus
Quest).
• Outside-in Tracking:
• In VR, this system uses external cameras or sensors placed
around the room to track the user's position (e.g., HTC Vive,
PlayStation VR).
• SLAM (Simultaneous Localization and Mapping):
• Common in AR, SLAM uses the camera feed along with depth
sensing and image recognition to create a map of the
Cont.,
 Optimizing Virtual Camera Performance
 To ensure a smooth experience in AR and VR, virtual
cameras need to be optimized for performance:
• Frame Rate: High frame rates (usually 60 FPS for AR, 90-
120 FPS for VR) are essential to avoid latency, motion
sickness, and disorientation.
• Latency: Especially in VR, the virtual camera must respond
to user movements with extremely low latency (less than 20
milliseconds) to maintain immersion and comfort.
• Depth Sensing: In AR, depth sensing improves the realism
of virtual objects by allowing accurate placement and
occlusion based on the real-world environment.
Degree of Freedom
 Degrees of Freedom (DoF) in the context of AR
(Augmented Reality) and VR (Virtual Reality) describe the
number of independent movements or motions that a body
or object can have within a 3D space. The concept of DoF is
essential for understanding how users can interact with
and move within virtual or augmented environments.
 Types of Degrees of Freedom
 In AR/VR, there are typically two levels of freedom:
3 Degrees of Freedom (3DoF)
6 Degrees of Freedom (6DoF)
 These degrees describe both the rotation and translation
movements.
Cont.,
 3 Degrees of Freedom (3DoF)
 3DoF allows users or objects to rotate in 3 dimensions without
changing their position in space. This is typically seen in more basic
VR systems and some AR applications where movement is restricted to
head or device orientation.
 The three rotational movements (axes) are:
• Yaw (rotation around the vertical axis, looking left or right)
• Pitch (rotation around the lateral axis, looking up or down)
• Roll (rotation around the longitudinal axis, tilting left or right)
 In VR, a 3DoF system would allow the user to look around the virtual
world (head orientation) but not move forward, backward, or sideways.
• Examples of 3DoF Devices:
• Google Cardboard
• Gear VR
• Basic AR on mobile phones (where movement is limited to the rotation of
the device).
Cont.,
 6 Degrees of Freedom (6DoF)
 6DoF allows both rotation and translation, meaning users
can move freely in space and rotate around different axes. It
enables much more immersive and interactive experiences,
particularly in VR, where users can walk, crouch, reach, and
interact with objects in the environment.
 The 6DoF system includes:
• Rotational Movements (3DoF):
• Yaw (left-right head rotation)
• Pitch (up-down head movement)
• Roll (tilting head side to side)
• Translational Movements (3DoF):
• Surge (forward-backward movement along the X-axis)
• Heave (up-down movement along the Y-axis)
• Sway (side-to-side movement along the Z-axis)
Cont.,
 In 6DoF, users can:
• Move forward, backward, up, and down in the 3D space.
• Look around by rotating their head in any direction.
• Interact with objects by reaching out, moving, or physically
walking in a room.
• Examples of 6DoF Devices:
• Oculus Quest 2
• HTC Vive
• PlayStation VR 2
• HoloLens (for AR with 6DoF head tracking)
Practical Differences between 3DoF
and 6DoF
• Immersion:
• 3DoF offers limited immersion since the user can only rotate their
viewpoint without moving their body in space. It’s good for seated
experiences where users don’t need to physically move.
• 6DoF provides a fully immersive experience by allowing users to move
around a virtual or augmented environment. The ability to walk, crouch,
and reach increases the sense of presence.
• Interaction:
• 3DoF limits interaction to head or device rotation, often requiring external
controllers or stationary experiences.
• 6DoF offers more natural, intuitive interaction where users can directly
interact with the environment by walking up to objects, picking them up, or
moving around them.
• Hardware:
• 3DoF systems typically rely on gyroscopes and accelerometers to track
head orientation or device rotation.
• 6DoF systems require inside-out tracking (sensors on the device track
the environment) or outside-in tracking (external sensors track the
device) to monitor both rotational and positional movements.
Virtual Reality Toolkit
 The Virtual Reality Toolkit (VRTK) is an open-source
development toolkit designed to streamline the process of
creating VR applications. It provides a set of reusable, pre-built
components that help developers quickly build immersive and
interactive VR experiences without needing to start from
scratch. VRTK is compatible with multiple VR hardware
platforms and game engines like Unity, making it a powerful
tool for VR development.
 Key Features of VRTK
 Cross-Platform Support: VRTK is designed to work across
various VR hardware platforms, including:
1. Oculus Rift / Quest
2. HTC Vive
3. Valve Index
4. Windows Mixed Reality
Cont.,
Unity Integration: VRTK is primarily designed to be used with
the Unity game engine, one of the most popular platforms for VR
development. It provides Unity-specific components that make it
easier to build, test, and deploy VR applications within Unity.

Pre-Built VR Interactions: VRTK offers pre-built functionality for

common VR interactions, which saves significant development
time. Some key interaction features include:
1. Teleportation: Provides standard and customizable teleportation
mechanisms to navigate virtual spaces.
2. Grab and Interact with Objects: Allows users to pick up, move, and
interact with virtual objects naturally using hand controllers.
3. User Interfaces (UI): Tools to create VR-friendly UI components like
buttons, sliders, and menus.
4. Locomotion: VRTK includes multiple movement systems such as
continuous movement, dash, and teleportation to suit different types of
VR experiences.
Common Use Cases for VRTK
 Common Use Cases for VRTK
Game Development: VRTK provides developers with a fast way
to create interactive VR games with mechanics like grabbing
objects, using weapons, solving puzzles, or navigating virtual
worlds.
Training Simulations: For industries like healthcare, military,
or manufacturing, VRTK simplifies the creation of training
simulations that require physical interactions, task-based
activities, or navigation in virtual environments.
Educational Experiences: VRTK’s interaction system can be
used to build immersive educational tools that allow students to
explore and manipulate scientific models, historical
reconstructions, or environmental simulations in VR.
Virtual Tours: VRTK can be used to create virtual walkthroughs
of spaces like real estate, museums, or tourist destinations, with
interaction and exploration enabled by locomotion systems.