Developing with Oculus: Mastering the Oculus SDK. Volga Aksoy Software Engineer, Oculus

SPEAKER Developing with Oculus: Mastering the Oculus SDK Volga Aksoy Software Engineer, Oculus Michael Antonov Chief Software Architect, Oculus

Table of Contents New SDK Features Direct-to-Rift Latency Testing & Dynamic Prediction in DK2 Pixel Luminance Overdrive Real-time Engine Integration Tips SDK rendering vs. App rendering Troubleshooting Judder FOV, texture resolution Asymmetric/symmetric frustum issues Left/right separate eyes vs one buffer Multi-threading and timing Looking Into The Future Time Warp Improvements SDK Roadmap

New SDK Features: SPEAKER Direct To Rift

Extended Mode Headset shows up as an OS Display Application must place a window on Rift monitor Icons and Windows in a wrong place Windows compositor handles Present There is usually at least one extra frame of latency More latency if no CPU & GPU sync is done

Direct To Rift Outputs to Rift without Display being a part of the desktop Headset not seen by the OS Avoid jumping windows and icons Decouples Rift v-sync from OS compositor Avoids extra GPU buffering for minimum latency Use ovrhmd_attachtowindow Window swap chain output is directed to the Rift Hope to see Direct Mode become the longer term solution

Direct To Rift - Mirroring Direct to Rift Supports duplicating the image to a Window Minimum overhead Can be disabled by ovrhmdcap_nomirrortowindow Possible to display a different image instead, such as messages in this screen

Direct To Rift - Implementation LibOVR shims Direct3D/GL Runtime Must call ovr_initializerenderingshim() before D3D use Window Swap Chain has an associated Rift Primary Buffer On DK2, back buffer is copied and rotated to Primary for display For best quality, use reported resolution User/Kernel Mode Video Driver Shim Hides Rift from OS, allowing us to find and render to it Allows flipping with Rift's v-sync (no composition queue)

Direct To Rift - Challenges so Far (1/2) Symptom: Rift doesn t turn on Optimus (NVidia + Intel HD GPU Laptops) Discrete GPU not connected to HDMI port SW has to create a bridge and copy data Use NVidia driver bridge in Win 7/8; Win 8.1 cross adaptor support Works on Razer Blade laptops On some machines copy is slow (10ms+ making it unusable) Multi-GPU (SLI/Crossfire) Rift plugged into the wrong adapter, need to copy

Direct To Rift - Challenges so Far (2/2) DisplayLink Conflicting driver shim; working with Display Link to resolve. Too slow for Rift output. Multi-Monitors May sync to wrong Display => V-sync tear or Judder Resolved with upcoming driver update Other Software Shims ASUS Splendid color, etc. Plan: Adding debugging logic, fix incrementally

Direct To Rift - Future Move Distortion/composition to a Separate Process Remove Shims from the Application Avoid state-related rendering issues Support Time warp layer composition Asynchronous Time Warp (TBD) Multi-GPU TW rendering Implementing Time warp on secondary GPU with Optimus Researching better GPU pipelining / preemption

New SDK Features: SPEAKER Latency Testing & Dynamic Prediction in DK2

VR Motion-To-Photon Latency Input New Image USB Game Engine Write Display Pixel Switching

VR Motion-To-Photon Latency Keeping latency low is crucial for a solid VR experience Goal is < 20 ms, and hopefully close to 5 ms We re almost there! (stay tuned for some numbers)

Latency Testing in DK1 Required custom hardware: Oculus Latency Tester App goes into latency testing mode Too cumbersome, ultimately not very practical Minimal developer utilization

Latency Testing in DK2 DK2 can: Sample top-right pixel color Time stamp of color change Messaged tohost PC via USB SDK handles rest of the latency orchestration on host PC

Latency Testing in DK2 SDK can internally time everything up to the point of calling Present()/ SwapBuffers() (i.e. pre-present). Post-present timing is a black-box to the VR app SDK relies on DK2 latency testing for post-present timing

Latency Testing in DK2 Variable frames primarily due to differences in: OS GPU Render APIs Driver settings

Latency Testing in DK2 At Oculus, we ve seen post-present latencies From 0 to 4+ frames That s a delta of: 50+ ms @ 75 Hz!

Latency Testing in DK2 High-level implementation PC renders top-right pixel with given color & records time HMD pings PC with detected color & time stamp of pixel SDK matches up detected color & calculates time difference Latency is reported after enough successive matches

Latency Testing in DK2 Final latency calculated by SDK as: Time difference from final HMD pose sampling to present/swap + Post-present delay duration from DK2 latency tester

Latency Testing in DK2 If (for any reason) timing fails SDK falls back to HMD-specific best-guess value Latency info accessible to VR app See OculusWorldDemo HUD info ( Space to view)

OculusWorldDemo showing HUD Info

Latency Testing in DK2 Timing info can be N/A because: Not full-screened on HMD Not running v-sync'ed Unaccounted for gamma curve in back-buffer (srgb, gamma 2.2 etc.)

Latency Testing in DK2 Ren: Latency from point left-eye HMD pose was queried from SDK

Latency Testing in DK2 Ren: Latency from point left-eye HMD pose was queried from SDK TWrp: Latency from EndFrame()

Latency Testing in DK2 Ren: Latency from point left-eye HMD pose was queried from SDK TWrp: Latency from EndFrame() PostPresent Latency from the point GPU executed Present() / SwapBuffer

Latency Testing in DK2 Captured from Mac Book Pro w/ GeForce GT 750M on Windows 8.1 ~0.0 ms Present, w/ TW at <7 ms Mid-display-scan lands @ ~10 ms We can now force display *IMMEDIATELY after Present()! (*when using direct-mode)

Latency Testing in DK2 Extended-mode, PostPresent will usually be multiple of 13.3 ms PostPresent will increase Ren & TWrp

Latency Testing in DK2 So we have motion-to-photon latency from SDK Then what? Remember with 0.2 SDK, you provided a prediction time offset for every HMD pose query? Enter dynamic prediction

Dynamic Prediction in DK2 0.4 SDK internally knows how much prediction the app needs to stabilize head tracking VR app no longer needs to worry about prediction.

New SDK Features: SPEAKER Pixel Luminance Overdrive

Pixel Luminance Overdrive Artifact called 2-frame-rise-delay Display fails to hit requested luminance levels fast enough Accounting for this in 0.4 SDK

Normal Driving Over Driving Pixel driving voltage Pixel transmission Transmission after one frame period Pixel driving voltage Pixel transmission Transmission after one frame period 1 frame Time 1 frame Time Pixel Luminance Overdrive

Pixel Luminance Overdrive Current formula in distortion shader code: overdrivecolor = newcolor + (newcolor - prevcolor) * overdrivescale; For-each-RGB, newcolor > prevcolor, overdrivescale = 0.1 For-each-RGB, newcolor < prevcolor, overdrivescale = 0.05 Currently in D3D10/11. Coming to OpenGL and D3D9.

Pixel Luminance Overdrive SDK needs to hold onto last presented frame Requires *v-sync *otherwise GPU won t know the contents of display Yet another reason to make sure v-sync is always on

Pixel Luminance Overdrive Current implementation fixes many 2-frame-rise artifacts e.g. in OculusWorldDemo, look through windows outside, or outside at the cypress trees against the sky Except the staircase and fireplace

Pixel Luminance Overdrive Black smear Anomalous behavior in 2-frame-rise-delay response Extremely-dark-green slightly-dark-green transitions Evaluating fine tuned LUTs for the anomalous zone Replace overdrive scale factors by small texture look-ups

Real-time Engine Integration Tips

Real-time Engine Integration Tips SDK rendering vs. App rendering Troubleshooting Judder FOV, texture resolution Asymmetric/symmetric frustum issues Split vs. Shared Eye Textures Multi-threading and timing

Real-time Engine Integration Tips: SPEAKER SDK Rendering vs. App Rendering

SDK rendering vs. App rendering 0.2 SDK didn t do any rendering Only provided parameters needed for proper rendering New rendering backend in SDK 0.4 Takes over critical rendering features App gives SDK L & R eye textures to ovrhmd_endframe()

SDK rendering vs. App rendering SDK 0.4 finishes rendering with the following features: Barrel distortion with chromatic aberration & time warp Internal latency testing & dynamic prediction Low-latency v-sync and flip (even better with direct-to-rift) Pixel Luminance Overdrive Health & Safety Warning

SDK rendering vs. App rendering Our Goal: Allow every engine to use SDK rendering Unity 3D, Unreal Engine 4 use SDK rendering All SDK rendering code is open source, and we highly encourage experimentation. If your engine cannot use SDK rendering, then *please* let us know why not Some valid reasons exist

SDK rendering vs. App rendering Don t have your engine s source code (e.g. XNA)? Need to support new render API? (e.g. AMD s Mantle)? Need a stereo 3D post-fx with distortion (a la CryEngine)? Need to do post-distortion rendering (e.g. debug info)? Show stopper bugs or missing features in our SDK?

SDK rendering vs. App rendering In some cases, what you think is unsupported by our SDK, might actually be supported, but we still want to know! Ping us. We will listen.

SDK rendering vs. App rendering Recommended SDK integration path: Follow OculusRoomTiny for bootstrapping the SDK Demonstrates both SDK rendering & App rendering Then move onto OculusWorldDemo for advanced features

Real-time Engine Integration Tips: SPEAKER Troubleshooting Judder

Judder aka Jitter-in-time Very few complained about judder on the DK1 DK2 has low-persistence + precise tracking & visuals Tracking hitches magnified when working accurately (Think uncanny valley of VR tracking)

Judder Reasons Low fps: Too much work per-frame (provide GFX options!) Highly variable prediction: Unpredictable frame loads Bad refresh rate: Not v-syncing with the HMD display Bad multi-threading: Incorrect usage of timing in SDK

Real-time Engine Integration Tips: SPEAKER FOV & Eye Texture Resolution

FOV & Eye Texture Resolution SDK provides recommended eye texture resolution Eye buffer texture resolution can be modified per-frame Pushed into ovrhmd_endframe() SDK will account for changes during distortion rendering e.g. Can drop eye resolution for better perf

FOV & Eye Texture Resolution Runtime modification can be tested in OculusWorldDemo Hit Tab & enable Render Target->Dynamic Res Scaling Will get better with layered distortion rendering for UI (e.g. high-res UI vs. lower-res 3D scene)

FOV & Eye Texture Resolution Symmetric Projection SDK provides recommended FOVs FOV port: Left, Right, Top, Bottom tangent angles Helpers to convert FOV Port to projection matrix Please: Do not cook up your own projection matrix that looks good enough using your own magic constants. We have seen games with divergent stereo-parallax. BAD! Asymmetric Projection

FOV & Eye Texture Resolution FOV can be set based on your game s needs Set via ovrhmd_configurerendering() SDK will generate new distortion mesh and account for changes during distortion rendering Changing FOV every frame not recommended

Wide FOV HMDs have locked physical FOV Unlike classic monitors, not an arbitrary setting Virtual FOV settings larger than HMD physical FOV are *generally wasteful *could modify FOV to account for time warp filling

Narrow FOV Modified virtual FOV respects physical FOV Otherwise, the view will look like it s viewed from *binoculars (*should be a separate zooming feature)

FOV & Eye Texture Resolution Outer FOV angle > Inner FOV angle? In real-life nose limits inner FOV Better HMD screen utilization by providing larger total horizontal FOV with less frustum overlap DK1 FOVs more asymmetric than DK2 Asymmetric/off-center projection to the rescue!

Real-time Engine Integration Tips: SPEAKER Asymmetric/Symmetric Frustum

Asymmetric/Symmetric Frustum Some engines always expect a symmetric camera frustum Many sub-systems can be fixed, but can be cumbersome Camera frameworks with only symmetric FOV options Deferred rendering interpolation artifacts Culling and LOD algorithms Many more

Asymmetric/Symmetric Frustum Make FOV port symmetric Inner FOV & Outer FOV = max(outer FOV, Inner FOV)

Asymmetric/Symmetric Frustum Make FOV port symmetric Inner FOV & Outer FOV = max(outer FOV, Inner FOV) Clipped-symmetric frustums similar to asymmetric-frustums

Asymmetric/Symmetric Frustum Caveat: Wasted pixel draw on distortion clipped area Perf issue? Scissor 3D scene render to visible distortion area.

Union Camera? Culling can be expensive Especially if done once for each HMD eye camera Some engines cull once from center eye & live with culling artifacts

Union Camera? A unionized HMD camera will allow culling once without artifacts Let us know if you d want a helper function for this Takes in: 2x camera transforms & 2x projections Returns: a camera transform & projection

Real-time Engine Integration Tips: SPEAKER Split vs. Shared Eye Textures

Split vs. Shared Eye Textures To share or not to share? SDK allows splitting two eye textures vs. sharing a single render target for the eye textures ovreyerenderdesc provides viewport for each scenario OculusWorldDemo option Shared RenderTarget

Split vs. Shared Eye Textures Combined Eye Texture - Pros: Per-draw alternate-eye rendering without switching render targets Pipelining on some GPU types Can save state & constant uploads Single-quad post-fx on both eyes

Split vs. Shared Eye Textures Split Eye Texture - Pros: If needed separable work load Smaller texture per-distortion Can modify eye texture resolutions independently w/o wasting space Accidental cross-eye sampling with time warp not a problem

Real-time Engine Integration Tips: SPEAKER Multi-Threading & Timing

Multi-threading & Timing Input devices are usually the first thing the game processes HMD also an input device! Some multi-threaded engines have 2+ frame input latency @ 75 Hz, that s 25+ ms latency given! Please use time warp no matter what!

Multi-threading & Timing e.g. producer/consumer two-thread engine i.e. main & render thread On main thread sample HMD with thread-safe functions: ovrhmd_getframetiming() + ovrhmd_gettrackingstate()

Multi-threading & Timing New API call coming soon: ovrhmd_geteyeposes() Returns HMD Pose + L & R Eye Poses No need for viewadjust by VR app Can be used on main thread & re-used on render thread

Multi-threading & Timing 0.4 SDK time warp only fixes up orientation latency To decrease positional latency: Resample HMD pose via ovrhmd_geteyepose(s) before 3D-scene draw on *render thread*

Multi-threading & Timing Time warp needs precise timing triggers on render thread ovrhmd_beginframe() or ovrhmd_beginframetiming() ovrhmd_endframe() or ovrhmd_endframetiming() No other API call directly affects SDK timing each frame!

Multi-threading & Timing Time warp parameters sent to SDK via ovrhmd_endframe() ovrposef renderpose[2] & ovrtexture eyetexture[2] Important: Time warp doesn t care about when and via what SDK call renderpose[2] was generated ovrhmd_gettrackingstate(), ovrhmd_geteyepose() etc.

Looking Into The Future: SPEAKER Time Warp Improvements

Time Warp Re-projects rendering for a later point in time Done simultaneously with distortion Reduces perceived latency Accounts for DK2 rolling shutter 0.4 SDK only handles orientation, positional possible

Synchronous Time Warp 0.4 SDK supports synchronous time warp Each frame does Distortion + TW after both eyes render Time warp uses later sensor reading Judder if we don t hit frame rate Dropping a frame can be very uncomfortable to the user Much worse than low-fps on regular monitors Can we use Time warp to fill in frames? Run game engine at half rendering speed

Time Warp OculusWorldDemo has features to show Time Warp in action. Hit C to pause rendering 3D scene Time Warp will continue to do its thing

Time Warp You can keep rotating the HMD to look around in the last 3D scene render Artifact: Missing scenery is black Artifact: UI warps with the rest of the 3D scene

Asynchronous Time Warp Time warp in a separate thread Best hope for reducing Judder Fill in the frames based on old rendering Implemented on Android for Gear VR Works well for smoothing head tracking on 30 fps game Requires: GPU Priorities GPU Preemption

Async Time Warp Sample Diagram CPU Eye Render Thread Left Right Left Right Left Right CPU Time warp Thread TW TW TW TW GPU Command Processor Left R- 1 TW R- 2 Left TW Right L- 1 TW L- 2 Right TW Frame N Frame N+1 Frame N+2 Frame N+3

Asynchronous Time Warp - PC Current OS/Windows limitations No fine-grained GPU Preemption GPU rendering priorities are not exposed Working with GPU Vendors AMD supports compute shader preemption NVidia supports GPU context priorities internally Exploring Adaptive and Cooperative Techniques Generating extra TW data just in case Checking TW time from game code

Time Warp Layers May need separate time warp transformations for: World-space FPS avatar/cockpit-space Static UI-space Allows feeding controller input into worldspace time warp while avatar-space time warp works as usual Solution: Layered time warp composition

Looking Into The Future: SPEAKER SDK Roadmap

SDK Roadmap Short Term Driver Improvements & Fixes Multi-Monitor fixes to v-sync related latency & judder Reduce post-present latency Handle Optimus and multi-gpu scenarios Linux Support Asynchronous/Adaptive Time warp Reduce Judder by generating extra frames when needed Time warp Layers Handle Cockpit and Overlays separately from the scene

SDK Roadmap Long Term Runtime DLL Better forward compatibility with HW & Runtime Updates Less frequent API versioning Advanced VR Composition (TBD) Support Layered VR composition with time warp Better state management & Preemption Overlays from secondary application (launcher etc.) Audio See Brian Hook s talk on VR Audio

Questions? SPEAKER Michael Antonov michael.antonov@oculusvr.com Volga Aksoy volga.aksoy@oculusvr.com @volgaksoy

OCULUS TM