Flat UI with 3D Viewport: A Different VR Experience (Tutorial)

Note: This tutorial assumes you have a basic working knowledge of Unity, including layer configuration and creating assets such as rendertextures.

In my off time I’m working on prototype experiments for Suburban Nightmare, a prequel of sorts to A Night in the Woods. ANITW has a retro presentation inspired by Ultima: Underworld, one of the first 3D texture-mapped PC games. In Underworld, the player sees the world through a tiny viewport window:

Ultima: Underworld screenshot

A Night in the Woods emulates this approach in a simplified form, doing away with the mouse cursor and sticking to modern first-person mouselook controls. I chose this approach both because I wanted the game to be simple, and because a lack of coding ability. Here’s a screenshot of ANITW for comparison:

A Night in the Woods screenshot

For Suburban Nightmare, I want to up the ante in several ways. I’m going to have a more complicated game system that switches between mouselook and a free mouse cursor, and I also wanted badly to experiment with virtual reality. Most VR games go for full immersion, surrounding the player with a 3D world. I wanted to try a different approach: Have a “virtual monitor” the player looks at in VR, with a flat user interface and a viewport that renders stereo 3D. This way you can continue using standard WASD locomotion and other traditional game design practices without inducing nausea or discomfort in VR.

Before I explain the VR implementation I came up with, I’m going to explain the setup I have for the non-VR (or as I like to call it, muggle 3D) version of the game. Understanding the default muggle setup will make it easier to leap into VR.

Here’s a shot of my current, very early prototype, which re-uses assets from ANITW:

ANITW Prequel screenshot

There are a couple things to note here. For now, the HUD is faked with a single transparent texture; it will eventually be expanded into a functional UI. The viewport has 256-color palletized graphics and is accurately downscaled to match a 320×200 resolution. There are also image effects happening full-screen in addition to any rendering effects within the 3D viewport. I achieve this with a two-tier rendering setup.

1. The Original Two-Tier Rendering Setup

Instead of rendering directly to the screen, the camera attached to the player object (currently a modified Unity 5 FPSController prefab) outputs to a rendertexture called Viewport. The rendertexture size is set to the pixel dimensions of the viewport window in the HUD, and also has its filter mode set to “point” so those big pixels aren’t smoothed out.

Figure 1. Player camera with rendertexture.

Figure 1. Player camera with rendertexture.

Figure 2. Player camera rendertexture.

Figure 2. Player camera rendertexture.

Now, we’re going to create a layer called “HUD”. This layer allows us to make sure the player camera never accidentally renders our HUD layout, and makes sure the primary camera that renders to the screen only renders the HUD. After you create the layer, remove “HUD” from the culling mask for ViewportCamera (notice how the culling mask above says “Mixed …”).

Next, create an empty game object called “ScreenRender”. This will hold all the stuff we use to actually draw things onto the screen. Here’s a glimpse of what our final setup will look like, with a stationary camera and a UI canvas with two RawImage objects:

Figure 3. The ScreenRender group.

Figure 3. The ScreenRender group.

Create a camera, called “HUDCamera”, that is a child of ScreenRender. Set its culling mask so that it only renders the HUD layer and nothing else. The quickest way to do this is to set the culling mask to “Nothing” and then just check “HUD”.

Figure 4. The HUDCamera is set to only render the HUD.

Figure 4. The HUDCamera is set to only render the HUD.

Because I like things to be nice and tidy, I place HUDCamera at position and rotation (0,0,0) so that it is always in the same location as its parent. In my scene I have the ScreenRender group located below the world geometry. Since the HUDCamera will never accidentally render the game world, and vice versa, in reality you can put it anywhere you like.

Next we create a UI canvas, that will contain the fake HUD image and Viewport rendertexture. The UI system in Unity will automatically create the necessary Canvas object if there isn’t one in the scene, so just create a RawImage object from the GameObject menu:

Figure 5. Creating a RawImage object.

Figure 5. Creating a RawImage object.

Drag the Canvas object onto ScreenRender group so it’s a child of ScreenRender, and rename it “HUDCanvas”. (You can actually name objects whatever you want, but these are the names I gave them in screenshots.)  Name the RawImage object “FakeHUD”.

Configure HUDCanvas as follows:

Figure 6. HUDCanvas configuration.

Figure 6. HUDCanvas configuration.

With this setup, the HUD will scale to fill the screen vertically at any resolution, so you don’t have to worry about scripting FOV changes or anything like that to keep everything sized correctly, as long as you have a 4:3 aspect ratio virtual screen like mine. If your HUD image is 16:9 or 16:10, the left and right edges may get cut off on some displays.

Now let’s set up our fake HUD. You can use any transparent texture for this part. Since my virtual screen is 320×200, and my placeholder HUD image fills the whole screen, I set the width and height to match:

Figure 7. FakeHUD configuration.

Figure 7. FakeHUD configuration.

Now we create another RawImage, called “Viewport” that will display the viewport rendertexture. Since my HUD image partially overlaps the viewport, it’s important that Viewport is above FakeHUD in the hierarchy. (In a Canvas object, child order determines render order, with the bottom object rendered on top.) My Viewport is configured to match the pixel size of the window, and has its position moved so it renders in the correct place (you may have to fiddle with the “Pivot” setting to get things pixel-aligned properly):

Figure 8. Viewport configuration.

Figure 8. Viewport configuration.

With this setup you should have a little window into the player object’s world, rendered within a HUD using a stationary camera. You can apply image effects either to the player object camera, or to the HUDCamera, depending on what you want to do. For example, my player camera has ambient occlusion, and my HUDCamera has a slight Fisheye effect to give the impression of CRT monitor screen curvature. My HUDCamera also has a Bloom effect to imitate color bleed and general blurriness of a CRT monitor.

Here’s a video of this setup in motion (note I also have a faked low framerate for the viewport that is independent of the HUD and mouse cursor):

2. Onward to VR

Now for the tricky part! We need to take this same basic setup and make it work in VR. I’m using the Oculus Rift DK2, but this doesn’t touch the bare metal of the hardware so it can apply to any VR platform. To make this work in VR we need to treat the HUD as a virtual screen that’s floating in front of us in 3D space. We also need the viewport to have depth, which means rendering a different image for each eye. So our FPSController player camera has to be two cameras, with some separation in the X axis to create depth. Since the viewport is going to be a small part of what we’re looking at when we wear the HMD, we probably don’t need to worry about IPD (inter-pupillary distance) calibration and can just set an arbitrary stereo separation. This worked fine in my experiments, and means we don’t have to go through the hassle of trying to set up a second OVRCameraRig that ignores head tracking.

Instead we just duplicate our player camera, set one’s X coordinate to -0.01, and the other to +0.01. We then set them to each render to a different rendertexture, which we’ll call “Viewport_VR_Left” and “Viewport_VR_Right”.

Figure 9. One of the VR player cameras.

Figure 9. One of the VR player cameras.

The cameras are grouped with “Viewport_VR” as their parent, for simplicity’s sake. We can make any mouselook code rotate Viewport_VR instead of messing with both cameras directly. Remember to set their culling masks to not render the HUD layer.

Next we need to place a camera for HMD rendering. With the Oculus Unity Integration plugin, that means we drop an OVRCameraRig prefab into the ScreenRender group, to replace our previous HUDCamera. I once again set my position and rotation to (0,0,0) for cleanliness. Make sure to set your VR cameras to only render the HUD layer, and have a clear flag of solid black. (If we wanted to we could create an environment to surround our virtual screen, even an actual 3D model of a CRT monitor, but for our purposes an endless black void is good enough.)

Since our HUD rendering is no longer attached to a static camera, we need to change our HUDCanvas from screen space to worldspace. I’ve also renamed the canvas to “VRCanvas”:

Figure 10. VRCanvas configuration.

Figure 10. VRCanvas configuration.

Be sure to set the Pos Z coordinate to a reasonable distance from the OVRCameraRig. Remember that number, because we’ll use it again later. The Event Camera is set to the left eye of the OVRCameraRig.

We have a VR camera and two different rendertextures, so we need to make sure that each eye only sees one rendertexture. This is how we get stereo 3D on a flat surface. This means creating two more layers, “Viewport_VR_Left” and “Viewport_VR_Right”. Each RawImage that shows a Viewport rendertexture will be assigned to one of these layers, and each eye in the VR cam will have its culling mask set to render only one of these layers.

Now for a little trick: Canvas objects don’t play nice when you mix child objects into different layers, so we need a separate canvas for each Viewport RawImage. Duplicate VRCanvas twice and rename the child objects until you get a setup like this:

Figure 11. Two canvases, one for each Viewport RawImage.

Figure 11. Two canvases, one for each Viewport RawImage.

Notice the layer assignment (make sure the children keep the same layer), and that the Pos Z is the same as our VRCanvas. Both of these canvases should perfectly overlap.

Each Viewport_VR RawImage (the child of each canvas) is configured similarly to how the monoscopic Viewport was, except we assign the Viewport_VR_Left rendertexture to the left one, and Viewport_VR_Right to the other.

Next, we need to tell our OVRCameraRig to properly exclude each of these canvases so the left eye only sees the left Viewport, and the right eye only sees the right Viewport. This is a simple matter of modifying the culling mask of LeftEyeAnchor and RightEyeAnchor so that they render the HUD layer and their respective Viewport_VR layer. Here’s the left eye:

Figure 12. LeftEyeAnchor culling mask configuration.

Figure 12. LeftEyeAnchor culling mask configuration.

If you view this setup in the Rift, you will have a stereoscopic viewport that’s like a little cutout window into a 3D world! There’s only one problem: Our 320×200 resolution is creating a lot of artifacts. That’s because 320×200 is a terrible resolution for stereo 3D; even slight view changes will make a big difference with those chunky pixels. That means in VR we have to increase the resolution of our viewport, and add anti-aliasing.

To do this, simply double or triple the rendertexture resolutions for the left and right player cameras. In my test, I went with a rendertexture resolution of 600×316 for each eye. This was a good tradeoff between comfortable viewing in VR, and system performance. If you’ve got a more powerful machine, you can probably go higher. I also changed the filtering mode for these rendertextures from “point” to “bilinear”, and added the AntiAliasing image effect to each camera. Smoothing out the images reduces the “shimmer” effect you get in VR from fine detail.

The other effects I had in place for my virtual CRT did not work well in VR either, so I had to drop them. That meant goodbye fake low framerate, because in VR, framerate is king. I also dropped the bloom effect to increase clarity on the Rift display. (The viewport cameras themselves can use bloom just fine, however.) Since the HUD is a flat plane, I kept it at 320×200 with point filtering, and it looked just fine. You’ll have more leeway with visual effects on your HUD than your viewport.

Here’s a video of the final rig:

So there you have it, a neat “fake 3D TV” kind of effect in VR, using a pretty simple setup with rendertextures. There are a few remaining kinks to work out, such as window violations—where the depth in the viewport “pokes out” beyond the plane of the HUD. This can really hurt your eyes because it is geometrically impossible. The mouse cursor especially is headache-inducing when it passes over objects that are visually behind it, but in front of it depth-wise. This can be fixed with a combination of altering the player bounding box (which affects gameplay, so tune early!) and adjusting the stereo separation between the left and right eye in the player object.

My blogging is supported by lovely folks through Patreon. Become a patron today and make sure I keep writing, making games, and doing all the stuff that I do.