What you can see here is that in this 360 video, I have agency, I have control.

Because it was
recorded in 360, it was captured all around the camera, and I can choose to look at Michael or
not. Next, we're going to look at stereoscopic displays, so the concept of rendering a different
view, a slightly different view for each eye. There is a little shift between the pupils, and that little
shift then creates a depth view for our eyes. So it's actually quite cool. So here, we have the
cardboard-vr-display source code. This has really a few demos that I think are cool. Here,
we see an example of a rotating cube, and that is just 3D, okay? So we're just looking at it in
3D, but now, I'm going to start the stereoscopic view here, the stereoscopic display, and now,
what it does is it creates a rendering for each eye, left eye, right eye. There is a tiny little shift
between these eyes and this has to do with the interpupillary distance, the IPD, and that is
something that can be adjusted in some headsets as well. So basically, we're not seeing two
cursors and two cubes. Actually, having you put this on your face, you actually then perceive
depth. You see one cursor. If it's done well, it gives you this illusion of depth, and that is the
whole key behind stereoscopic displays. This is really the main thing behind cardboard. So
remember, cardboard, phone goes inside and then we actually have a rendering for each eye
with these barrel distortions. Left eye, right eye.
Then we perceive it like this, like you see in the video, rendering for each eye. Cool. So that was
that. I think that is really a first important thing. We're creating this immersive view, if you will, a
rendering for each eye that gives you then a sense of depth, and we are starting to see things
and we are starting to call this a virtual reality. 360 photos and videos by and in themselves are
not virtual reality. They're often associated with virtual reality, but they only become a virtual
reality when we see them through a stereoscopic viewer like the cardboard with a rendering for
each eye. Because then our head movement can be tracked just the rotation, not normally the
position. So we cannot actually walk inside 360 photos and videos, and we're starting to get
control. We're gaining some agency over some of the content, at least in terms of rotation. My
next example is a little bit different. It's a different display technology. It also creates a virtual
reality display, but in this case, you're actually stepping inside what is called a Cave Automatic
Virtual Environment inside a CAVE. Here, I'm showing it based on the example that a colleague
of mine at the University of Michigan gave to me. Ted Hall, he is in our MIDEN, and he shows
one of the applications he was working on for some of the researchers that allows him to
explore a mammoth and look at it in 3D. What was really impressive here is how the view is
rendered just for him. He wears these funny antennas and there's some tracking system, an
OptiTrack System in the room that actually tracks these antennas and allows him to actually
walk around. If you look at the video one more time, you will see how the view seems to be like
somehow distorted and overlaid. That is the effect that is being created to give him a depth
sense. So these will shift further away the further the virtual content is away from him and they
will becoming more overlapping the closer the content is to him. When you go to the movies and
watch a 3D movie, you put on your glasses that they give you, but then sometimes, you
probably take them off because you're tired or not, or you just feel like, hey, let's look at this
from a VR perspective. These 3D glasses are actually counteracting the shift and they're putting
it together in front of you. There's usually some shadow display that can be active or passive,
but some shadow display that then shows you just the image for the left eye and then the image
for the right eye. Then this is such a high frequency that you don't even notice and it will create
a 3D image for you that you perceive as depth. Let's look at actually, head tracking and also
control tracking if you will. We have virtual reality motion controllers. They can be tracked in
three degrees of freedom or six degrees of freedom. Let's look at this in a little bit more detail.
Three degrees of freedom, like the cardboard, looks like this, and then we can track along the x,
so pitch. We can track along y, yaw, but just rotation. We can track along z, but just again
rotation, so roll. You cannot actually move along these axis but you can rotate your head. This is
me in cardboard and this is pitch, yaw, and roll if you will. This is just three degrees of freedom,
and if you can actually, also adjust your position. The second example here is the hollow kit, the
one that I had earlier. It's a device wrapper just like cardboard, but a little different for AR.
Phone goes inside, we can see through. If we expose the camera, we can actually do some
tracking, and we can do positional tracking. Now, in addition to head, roll, pitch, and yaw, we
can also actually, determine whether the device moves along the z-axis, along the x-axis, and
along the y-axis. We can do this through tracking techniques like SLAM, which is simultaneous
localization and mapping of the device in 3D space. These are some computer algorithms that
work based off the camera feed, and allow us to determine some motion. Optical flow, if you
will. Optical flow would be computing a vector as I move the device this way. We are producing
a vector from here to there. Or if you will, what happens with the camera feed, all the pixels
actually shift to the right. That way we can actually figure out that the device moved to the left.
Now, that is what I'm showing here in this illustration. This has six degrees of freedom. We have
also positional changes plus or minus x. Positional changes along the y-axis, and positional
changes along the z-axis. Now, we do have position, and orientation, and we refer to this as six
degrees of freedom tracking. Again, with this tracking, you can walk around in a 3D space. You
can not just look around like with 360 video, but you can then move around. This gives you a lot
of agency in the content. That's then when it starts to become hard for designers and
developers that have to design for that agency. Here I have an example of the Oculus Go. The
Oculus Go is technically a three degrees of freedom headset. It comes with the three degrees of
freedom controller. The controller will always be on the right down there. We can figure out its
rotation, but we cannot figure out where it actually moves in 3D space. It is really a little limited,
but you can already do quite a bit just by being able to look around and rotate the controller. If
we use this laser interaction here, this laser controller, laser pointer interaction, you can reach
objects at a distance and you actually do have quite a bit of control already over the virtual
reality content. Here we do have the Go. That's the controller I just showed, and that's the Go,
the device that goes on your head. It is three degrees of freedom. No travel, just rotation, and
the same for the controller. Just a little bit of a pointing device, but whether I have it here or
there doesn't make a difference. Really, the difference is rotation. This makes really for quite
some differences in terms of how you design and develop and then also how you can use virtual
reality applications. We're going to learn more about this in other segments of the MOOC. When
it comes to six degrees of freedom, a very good way to demonstrate this is through the Oculus
Quest. You usually draw a boundary and then you can walk inside those bounds. That's then
supposed to be like a safe zone. With six degrees of freedom, both for the controller and for the
head, I can now walk around and move the controller in 3D space, and I'll be tracked all the
time. That is really quite cool. This one has three degrees of freedom with a six degrees of
freedom tracking for both the head and the controller. Now, we're going to talk about outside-in
and inside-out tracking. In a nutshell, the difference between outside-in tracking is that you have
some sensors in the environment that are pointed towards you and they track you as you move
around in a certain area that is covered by these sensors. Inside-out tracking means you
actually have those centers inbuilt into the device, the cameras are actually inside the device.
I'm going to show you two examples here. My first example here for outside-in tracking is
actually the original Oculus. The original Oculus required sensors to be placed on a table
usually in front of you. These sensors here, were actually pointing towards you. Then they are
creating a sensor field, and they can see you in this range and you had to make sure that these
sensors are overlapping. There's some triangulation happening, and what they're actually
looking for is light patterns emitted from the controller and also from the headset. My second
example is the Oculus Rift S, which is basically the new version of the Rift. It actually has the
cameras inbuilt into the device. Now, this makes it possible to actually do both the head-tracking
and also tracking the controllers in six degrees of freedom without any external sensors. The
cameras are on the device and basically this is the field of view that these cameras have. It's an
approximation it's not absolutely correct here. You can see that the controllers are actually
captured this way. In fact, the Oculus Rift S and the Quest even they also have additional
cameras on the side. Even when you have the controllers next to the device or even below the
device, the device can still see the controllers. This is actually quite an advancement over
outside-in tracking. When you have these stands on the table, they could only see sensors, and
usually the table would include things, so as you were doing things below the table you couldn't
be tracked anymore. Now with inside-out tracking you actually have these sensors on the
device and so anything that is visible, in a certain area around them can be tracked. When you
put your hands behind your back, obviously this will not work, but that's fine. Here I have an
example of the HTC Vive with a form of outside-in tracking. It's technically not absolutely
correct, but this is what they usually looks like. You have a sensor in the environment and then
the controller here is coming towards the sensor and it's being tracked in six degrees of
freedom. Here's an example of me demonstrating their Rift S and I often do this. I'm pointing out
the different cameras on the device and I am explaining how the Rift S can actually see. Has
cameras on top, downwards to the left and right and front, and has a really good field of view for
these cameras to actually track. I'm making some comparisons to the Oculus Rift original sensor
stands that you see in the back. Let's talk about hand tracking. Hand tracking now has actually
become a standard on the latest generation of VR devices. I'm going to introduce you to a hand
tracking in two steps. First, I'll show you what this used to be like with a device called the Leap
Motion, which I have here. The Leap Motion is a USB tethered device that would be mounted on
top of the headset, and it creates a field of view, a window in which you can actually do hand
tracking. I'm going to show you this first, and then I'm going to show you how hand tracking
actually works on the Oculus Quest, which we have there. Now it's essentially inbuilt, so built
into these devices. This is Leap Motion. It's really not very big and creates this window here.
Then I can actually see my hands nicely articulated. In this example, I'm just moving my hands
up and down and I can actually draw, if you will, with left and right and all the fingers. This is
cool, first person view. This is from the app. Looking down. I'm fancying device on the side.
This make sense. In this example I'm going to show Oculus hand tracking. Here under settings
in the device, we do have hands and controllers, and you can see hand tracking and auto
enabling hand tracking is actually toggled to on. Switched on. Now that means that I can just put
down my controllers. Look at my hands, and there they are. I'm going to do pass through. You'll
see what I'm looking at here on my hands and go back into VR and it's nicely mapped. It's
relatively accurate. It feels pretty good. I am touching their correct fingers, and this would
actually be a gesture, so lights up becomes active. What we're going to do is, well we can scroll
and we can scroll sideways as well. I'm going to close this and can scroll sideways. It feels
relatively good you can see the rendering of my hands. It still seems relatively accurate.
Scroll up and down with this pinch to drag gesture. One thing that is a little bit, if an app doesn't
support hand tracking like here, this is the browser. I can actually browse with hand tracking.
Most normal websites would work, but then when you go and look at whether this now works in
A-frame, then at this stage I have to disappoint you. I have to grab the controllers. We're going
to do the passthrough, controllers over there. We're going to grab them, and we're going to go
into the VR immersive view. There we go. There it is. Hand tracking and A-frame for default
doesn't work yet, but this is probably going to happen very soon, just not at the time of this
recording. I'm going to exit out with the Oculus button and this concludes my little demo of hand
tracking and controllers. You can actually seamlessly go back and forth between hands and
controllers. Pick up the controllers and press a button and there they are again and give it a little
bit of rest and then look at your hands, give it a bit of time for the recognition to work. We do
have hand tracking now built into these devices. I talked about VR gloves earlier and the
reasons I don't have them, but the point of VR gloves is not necessary to tracking of the hands,
it was actually the creation of the haptic sensation so that you actually feel like you're touching
something. That can only be accomplished through some tricks by actually mapping the
physical to the virtual environment so that for example, you are perceiving in VR a surface here
and just happens to be also your desk that you are touching. If you can create these nice
overlays between the physical and the virtual, you actually do get a sensation of physical
feedback even without varying additional things like gloves and whatnot. Most of my examples
were actually visual, so what does it take to create a virtual reality? Virtual reality is
predominantly visual so in that sense it's not wrong. However, a very, very important part of
creating immersion is actually the audio around the user, so my last example will focus on
spatial audio. Here I could have chosen many examples. One that I like is actually Tilt Brush
and I want to show it to you in two ways: first, without sound and then with sound.
Now we have seen the main ingredients of Virtual Reality, both on the visual side and then on
the tracking side, so stereoscopic views or alternatively creating a cave like monitor wall-size
displays around the user, so you're inside a virtual reality headset, if you will, or for the headset
that we've talked about stereoscopic views. We've learned a little bit about interactions and also
hand tracking, and then spatial or positional audio as well. These together actually create a rich
set of sensations and this is really what you want to accomplish so that your users feel
immersed in Virtual Reality. The first thing's I'm going to revisit Beat Saber and we're going to
look at it from the perspective of all these kinds of technologies that we've just learned about. In
terms of the VR technologies that we have just learned about, Beat Saber really actually
chooses those. It creates a stereoscopic display through the headset, it supports six degrees of
freedom tracking which is really necessary. You need to be able to move in space, and also it
does work actually with both outside-in and inside-out tracking, but the example that I'll show
you here is using the Oculus Rift S with inside-out tracking. Down, down. I have to follow these
directions when slicing. Oops, we should have used the other Saber, but that way you see what
happens when things go wrong. Avoid those, and then just blue and now red.
This one you can cut it into any direction. The synchronous cuts are fun, but I would like it when
you get to slice like crazy like this. Now comes my favorite part of song.
Up, up, any, any, down, down, down, right, down, down, left, down, down, right, down, down,
any, any. We've made one mistake so far, but is okay. You're not pro.
I have to be careful that I don't hit the cable or the case, because I don't want it to disconnect
again.
Down, and this one, I always find find it funny when they turn around and then you have to do
the last minute changes. This one is cool. Slice, slice, slice, up..