GearVRf Developer Guide

GearVRf Developer Guide Welcome to GearVRf development! The Gear VR Framework (GearVRf) allows you, the Android developer, to write your own stereoscopic 3D virtual reality Java applications for the Samsung Galaxy Note 4 and Galaxy S6. GearVRf manages VR application tasks, which reduces your application code. With the GearVRf API, you can focus on your 3D models and animation - GearVRf does the heavy lifting. Table of Contents GearVRf Developer Guide (Online) GearVRf Overview Thread Management Scene Graph and Scene Objects Getting Started Hardware Requirements Software Requirements GearVRf Build Instructions Sample GearVRf Applications GearVRf Application Development Building and Running GearVRf Applications Writing a Basic GearVRf Application Step 1 Implement Your Activity Step 2 Implement Your VR World Step 3 Implement onstep Writing More Advanced GearVRf Applications Implementing GVRActivity Setting Up the Camera Rig and Cameras Creating the SceneGraph Rendering GearVRf Developer Guide 1

GearVRf Developer Guide > GearVRf Overview GearVRf provides tools to speed up development of advanced features in high quality VR applications. Available EGL extensions (including dual scan, front buffer, MSAA, and tile rendering) allow the best render quality. GearVRf is a native code 3D rendering engine with an Android library interface. You can build non-trivial content using only built-in objects. You can add new objects (such as scene objects with or without GL shaders) derived from classes or by overriding some methods - GearVRf takes care of all hardware handholding. You can do just about everything in Java - all source code is published, so you can easily add to or tweak native code. Thread Management One key constraint of embedded GPU programming is that there is only one GL context. That is, all GPU commands must come from the same thread - the GL thread. The GPU should always be busy; therefore, the GL thread cannot be the main GUI thread. When starting GearVRf, your Android app creates the GL thread, puts the phone into stereoscopic mode, and supplies a pair of callback methods that run the app's startup and per-frame code on the GL thread. GearVRf provides methods for any thread to schedule runnable callbacks to the GL thread. All these callbacks mean that GearVRf programming is event-oriented on the GL thread in just the same way that Android programming is event-oriented on the GUI thread. Running two independent event systems on two independent threads does mean that you have to think about IPC whenever your Android Activity code on the GUI thread interacts with the GearVRf code on the GL thread. However, dual-thread operation also creates another huge section of your application that can take advantage of event atomicity. That is, callback events are method calls from a main loop - neither the GUI thread nor the GL thread ever runs more than one callback at one time, and each callback has to run to completion before any other callback can start on that thread. Your GL callbacks do not have to write code to keep other GL callbacks from seeing data structures in a partially updated state. GearVRf Developer Guide 2

Scene Graph and Scene Objects Your startup code builds a scene graph made up of scene objects, and your per-frame code then manipulates the scene graph in real time. Each scene object has a 4x4 matrix that describes its position, orientation, and zoom relative to its parent. Each scene object can parent other scene objects, so complex objects can be composed of multiple small objects, each with its own shape and appearance, with all changing in synchrony. Each scene object provides methods to change its components using a lazy update pattern, which means that multiple updates per event cost very little more than a single update. You make a scene object visible by adding a surface geometry and a skin. The geometry is a mesh of 3D triangles. GearVRf provides methods to build simple rectangular quads, and to load more complex meshes from files built by 3D model editors. Each material class contains the shader type, the GL identifier of a shader, values for all shader parameters, texture, and other uniform mappings. Each shader has two parts: a vertex shader and a fragment shader. The vertex shader is called for each vertex of each visible triangle and can compute triangle-specific values that are passed to the fragment shader, which draws each pixel of each visible triangle. GearVRf contains standard shaders that provide methods, such as simply sampling a texture (a bitmap image in GPU memory), without applying any lighting effects. You can create custom shaders by supplying vertex and fragment shaders and by declaring names to bind Java values to. The GL shader language is very simple and C-like; you can learn a lot by reading a few of the shaders in the sample applications. GearVRf Developer Guide 3

GearVRf Developer Guide > Getting Started You can run sample GearVRf applications and create your own GearVRf applications and run them on supported devices. Hardware Requirements Samsung Galaxy Note 4 and Samsung Galaxy S6 Samsung Gear VR headset Software Requirements JDK 1.7 or above Oculus Mobile SDK 0.5.1 Android SDK (version 5.0.1, Lollipop, is preferred) and/or KitKat SDK (version 4.4.x ) GearVRf Build Instructions In Eclipse, you can build the Gear VR framework in order to run sample and your own GearVRf applications. Prerequisites 1. Latest GearVRf 1. In https://github.com/samsung/gearvrf, click Download ZIP 2. Unzip the file to path / to / convenient / local / directory / (the GearVRf directory) 2. JDK 1.7 or above Download it from http://www.oracle.com/technetwork/java/javaseproducts/downloads/index.html. 3. Eclipse IDE 1. If you already have Eclipse from a previous ADT bundle, go to step 4. GearVRf Developer Guide 4

2. If you do not have Eclipse: 1. Download Eclipse IDE for Java Developers from https://eclipse.org/downloads/ 2. Download Android Development Tools (ADT) for Eclipse and configuration instructions from http://developer.android.com/sdk/installing/installing-adt.html. 4. NDK plugin with Eclipse CDT (C/C++ Development Tooling) Download it and configuration instructions from http://tools.android.com/recent/usingthendkplugin NOTE: Do NOT install a CDT version higher than 8.0.2 or you will encounter parser errors at the JNI level, even though these will not affect building and running your GearVRf application. 5. Oculus Mobile SDK 0.5.1 Download it from https://developer.oculus.com/downloads/#sdk=mobile Put the Oculus SDK folder at the same directory level as your GearVRf directory. For example, if your GearVRf directory path is / path / to / convenient / local / directory / GVRf / put the Oculus SDK in / path / to / convenient / local / directory / ovr_mobile_sdk / NOTE: Remove version information from the pathname. 6. Android SDK (version 5.0.1, Lollipop, is preferred) and/or KitKat SDK (version 4.4.x ) After you set up Android Development Tools in Eclipse: 1. Click Window > Android SDK Manager 2. Download the newest version of Android SDK Tools and build tools. 3. Download Android 5.0.1 (API 21) and 4.4.x in Android SDK Manager. Configuration 1. Properly configure Eclipse: 1. For Android Lollipop, ensure the current Java compiler level is at least 1.7: 1. Click Window > Preference > Java > Compiler > JDK Compliance > Compiler compliance level 2. Verify the compiler level is 1.7 or above. 3. If it is NOT, set it to 1.7. 2. Ensure the JDK version 1.7 or above is in Eclipse: 1. Click Window > Preference > Java > Installed JREs 2. Verify the JDK version is 1.7 or above. 3. If it is NOT, click Add and follow the instructions to put it in. 3. Click Window > Preferences > Android > Build, and UNcheck Force error when external jars contain native libraries 4. If NOT using Eclipse with the ADT bundle, properly configure the Eclipse Memory Settings for Android development NOTE: For GearVRf development, it is recommended to set Xms = 512m and Xmx = 1024m in order to not encounter Java heap size errors. 5. Click Project > UNcheck Build Automatically 6. Click Project > Clean..., checkmark Start a build immediately, and checkmark Build only the selected projects GearVRf Developer Guide 5

2. If you are on a case-sensitive file system (such as for Linux), you may need to correct a typo in the Oculus source code in order to compile. If you get a compile error in jni/vrmenu/vrmenucomponent.cpp about #include "Android/Log.h" then change it to #include "android/log.h" Build 1. Import VRLib. In Eclipse Package Explorer: 1. Right-click Import... 2. In the Import window, click Android > Existing Android Code Into Workspace 1. Click Next > 2. Click Browse... 3. Navigate to the VRLib directory (an Oculus SDK sub-directory). 4. Click OK 5. Click Finish 3. If a Maven plugin (such as Eclipse Luna) is installed, disable the plugin. Right-click and UNcheck VRLib > Maven > Disable Maven Nature 2. Import the Gear VR framework. In Eclipse Package Explorer: 1. Right-click Import... 2. In the Import window, click Android > Existing Android Code Into Workspace 1. Click Next > 2. Click Browse... 3. Navigate to the Framework directory (a GVRf sub-directory). 4. Click OK 5. Click Finish 3. Clean and build the VRLib and Gear VR framework. In the Eclipse toolbar: 1. Click Project > Clean... 2. In the Clean window: 1. Checkmark Clean projects selected below 2. Checkmark VRLib 3. Click OK 3. After the Build Project window closes, click Project > Clean... 4. In the Clean window: 1. Checkmark Clean projects selected below 2. Checkmark Framework 3. Click *OK* NOTE: The VRLib and GearVRf builds are done. GearVRf Developer Guide 6

Optional Get details about the GearVRf API by generating GearVRf Javadoc files in a directory that you specify. In Eclipse toolbar: 1. Click Project > Generate Javadoc... 2. In the Generate Javadoc window: 1. Javadoc command: (Pathname to the Javadoc executable file) Typically, in the bin directory of the Java directory. 2. Click on the plus-icon to expand the Framework listing. 3. Checkmark src 4. Select Use standard doclet 5. Destination: (Convenient local directory to contain the Javadoc files) 6. Optional Specify VM options: NOTE: You may encounter errors if VM options is not specified. 1. Click Next > 2. Click Next > 3. VM options: -bootclasspath [path to your Android jar file] (Typically the pathname ends in.../android/sdk/android.jar) 7. Click Finish Sample GearVRf Applications Sample GearVRf applications, available in the GearVRf SDK, can provide you with valuable insight into writing your own VR applications. Solar System (gvrsolarsystem) Eye Picking (gvreyepickingsample) A solar system with rotating planets revolving around a rotating sun * Asynchronous loading of meshes and textures is used. * Loading order is by priority and no mesh or texture is loaded twice. * Animation is used to create the illusion of rotation and revolution. Colors of objects (bunnies and rectangles) change to red when an object is at the center of the view Dynamic Gallery (gvropacityanigalery) * OnStep is used to update scene objects colors. * Implements a color custom shader. Images and video displayed on a board * OnStep is used to decide and define which animation to apply, based on HMD orientation. * A video scene object is displayed. * Animation is used to switch images and video smoothly. * Post-effect converts the view to sepia colors. GearVRf Developer Guide 7

GearVRf Application Development GearVRf runs full-screen and in a single Android View of its own, so you do not need any themes, menus, or layout files. But GearVRf apps can use Android resources (such as strings, ints, drawables, assets, and res/raw files). For details about developing your own GearVRf applications: Writing a Basic GearVRf Application Writing More Advanced GearVRf Applications Building and Running GearVRf Applications You can import, build, and run GearVRf applications (your own or sample apps) in Eclipse. To build and run GearVRf applications in Eclipse: 1. For your own GearVRf apps only, copy the appropriate device XML file from the GearVRf SDK to your application's assets folder. GearVRf provides two xml files for you to use: gvr_note4.xml for Galaxy Note 4 devices, and gvr_s6.xml for Galaxy S6 devices. 2. For your own and sample GearVRf apps, copy your Oculus device key from the Oculus SDK to the application's assets folder. 3. Import the GearVRf application code: In Eclipse Package Explorer: 1. Right-click Import... 2. In the Import window, click Android > Existing Android Code Into Workspace 1. Click Next > 2. Click Browse... 3. Navigate to the application code. 4. Click OK 5. Click Finish 4. Clean and build the application: In the Eclipse toolbar: 1. Click Project > Clean... 2. In the Clean window: 1. Checkmark Clean projects selected below 2. Checkmark [Application] 3. Click OK 5. Run the application: NOTE: You may need to apply for a signature file for your device with Oculus. For application and use details, refer to the online signature generator (https://developer.oculus.com/tools/debug/ ) 1. Connect an Android mobile device to your local machine. 2. In the Eclipse Package Explorer, right-click on the application Run As > Android Application GearVRf Developer Guide 8

GearVRf Developer Guide > Writing a Basic GearVRf Application To become familiar with the basics of using the GearVRf API, write a fundamental application. The GearVRf sample application simplesample, used as an example in this section, presents a 3D view of a GearVRf banner in a VR world with a white background. GearVRf runs full-screen and in a single Android View of its own, so you do not need any themes, menus, or layout files. But GearVRf apps can use Android resources (such as strings, ints, drawables, assets, and res/raw files). Step 1 Implement your activity by extending GVRActivity To implement your activity, your application needs to extend GVRActivity to meet the application needs. Call GVRActivity setscript to initialize your application by providing a device XML file and a GVRScript object (SampleViewManager in the example below) to create your own scene graph. Modify the wizard-generated Activity so that it extends GVRActivity (instead of extending android.app.activity) and change its oncreate() method so that it calls setscript() instead of setcontentview(). To extend GVRActivity for a sample application and a supported device (Samsung Galaxy Note 4): package org.gearvrf.simplesample; import android.os.bundle; import org.gearvrf.gvractivity; public class GalleryActivity extends GVRActivity { @Override protected void oncreate(bundle icicle) { super.oncreate(icicle); setscript(new SampleViewManager(), "gvr_note4.xml"); } } GearVRf Developer Guide 9

Step 2 Implement your VR world by extending GVRScript The main task of your application is to extend the script class to meet the needs of your application. Create a new class Script that extends GVRScript and implements the oninit() and onstep() callbacks. Your application needs to implement the oninit function: Set the background color. Load the mesh and texture to be used for scene objects. NOTE: In simplesample, default loading of mesh data is used. Create each scene object and set its material and position. Add scene objects to the scene graph. To extend the GVRScript: package org.gearvrf.simplesample; import org.gearvrf.gvrcontext; import org.gearvrf.gvrscene; import org.gearvrf.gvrsceneobject; import org.gearvrf.gvrscript; import org.gearvrf.gvrtexture; import android.graphics.color; public class SampleViewManager extends GVRScript { private GVRContext mgvrcontext; @Override public void oninit(gvrcontext gvrcontext) { *// save context for possible use in onstep(), even though that is empty* *// in this sample* mgvrcontext = gvrcontext; *// set background color* GVRScene scene = gvrcontext.getmainscene(); scene.getmaincamerarig().getleftcamera().setbackgroundcolor(color.white); scene.getmaincamerarig().getrightcamera().setbackgroundcolor(color.white); *// load texture* GVRTexture texture = gvrcontext.loadtexture("gearvr.jpg"); *// create a a scene object (this constructor creates a rectangular scene* *// object that uses the standard 'unlit' shader)* GVRSceneObject sceneobject = new GVRSceneObject(gvrContext, 4.0f, 2.0f, texture); *// set the scene object position* sceneobject.gettransform().setposition(0.0f, 0.0f, -3.0f); } *// add the scene object to the scene graph* scene.addsceneobject(sceneobject); GearVRf Developer Guide 10

Step 3 Implement the onstep method for each frame This is where your application needs to adjust the SceneObject for each frame. If you need to update shader uniforms, you should do that in this function. If your application needs to adjust scene properties for each frame or a particular frame, the onstep method is right place to change them. For OpenGL ES applications, shader uniform update happens for each frame. If you have uniform updates, you need to perform the updates in onstep. NOTE: The onstep method could be empty (as in simplesample, which as a static scene, and the perspective only changes based on user-initiated HMD motion). To change the scene for the next frame: } @Override public void onstep() { } GearVRf Developer Guide 11

GearVRf Developer Guide > Writing More Advanced GearVRf Applications With a little more code, you can create 3D stereoscopic VR worlds! Implementing GVRActivity To implement your activity, your application needs to extend GVRActivity to meet the application needs, by providing a device XML file and a GVRScript object. For details, see step 1 in the basic GearVRf application section. Setting Up the Camera Rig and the Left and Right Cameras GearVRf will create the camera rig by default. Its parameters do not need to be adjusted; however, applications may move the camera rig. The HMD sensor automatically adjusts camera orientation; your app does not need code for this. Camera background color can be set for each eye; however, they are typically the same. Camera background color and post-effect data can be set by the application. Post-effects are applied to each camera. To set the background of the cameras and the position of the camera rig: // set camera background color mgvrcontext.getmainscene().getmaincamerarig().getleftcamera().setbackgroundcolor(0.0f, 0.0f, 0.0f, 1.0f); mgvrcontext.getmainscene().getmaincamerarig().getrightcamera().setbackgroundcolor(0.0f, 0.0f, 0.0f, 1.0f); // set up camerarig position (default) mgvrcontext.getmainscene().getmaincamerarig().getownerobject().gettransform().setposition(0.0f, 0.0f, 0.0f); GearVRf Developer Guide 12

Creating the Scene Graph The scene graph - the VR world - is a hierarchical tree of scene objects. Each scene object is a tree node with one parent and one or more child scene objects. Applications must build a scene graph. Your app needs to set up camerarig for the root scene object of the scene graph, but does not need to set up camerarig for each lower-level scene object. To create a scene graph at initialization time, get the GearVRf main scene (the root scene object) from GVRContext. To create the scene graph by getting its root scene object: GVRScene scene = mgvrcontext.getmainscene(); Scene Objects Populate your VR world's scene graph scene object tree by adding scene objects to the root scene object and to other lower-level scene objects. The most common way is to load mesh data using the GearVRf wrapped Assimp library. Assimp can load many 3D file formats. After a mesh object is created, it should be attached to a scene object OR added to the render data of a scene object. Typically, load each texture from a file and then add each texture to a material. In managing materials, assign a shader program and a texture to the material, and add the material to a scene object. A shader program is composed of a vertex shader and a fragment shader. There are 7 pre-defined shaders and 2 pre-defined post-effect shaders for applications to use. Applications can simply set material shadertype (to track application shader mapping) to one of the pre-defined shaders. A common pre-defined shader is the unlit shader. To create a scene object from mesh and texture loaded from files: // load mesh using assimp GVRMesh spheremesh = mgvrcontext.loadmesh("sphere.obj"); // create pre-defined scene object: texture mesh object GVRSceneObject bumpysphere = new GVRTextureMeshObject(mGVRContext, spheremesh, mgvrcontext.loadtexture("bumpytexture.png")); To create a scene object with shader-only material via render data: // load mesh object GVRMesh spheremesh = gvrcontext.loadmesh("sphere.obj"); // get material GVRMaterial spherematerial = new GVRMaterial(gvrContext, mscreenshader.getshaderid()); GearVRf Developer Guide 13

// create render data GVRRenderData sphererenderdata = new GVRRenderData(gvrContext); // set mesh and material for render data sphererenderdata.setmesh(spheremesh); sphererenderdata.setmaterial(spherematerial); // create scene object sphereobject = new GVRSceneObject(gvrContext); sphereobject.attachrenderdata(sphererenderdata); Managing Transforms in a Scene Graph After scene objects are added to the scene graph, each scene object can be controlled by transforms. To set the position of a scene object and rotate it about an axis with a pivot point: GVRSceneObject rotator = new GVRTextureBoardObject(mGVRContext, 2.0f, 1.0f, rotatortextures.get(i)); rotator.gettransform().setposition(0.0f, 0.0f, -5.0f); float degree = 360.0f * i / (rotatortextures.size() + 1); rotator.gettransform().rotatebyaxiswithpivot(degree, 0.0f, 1.0f, 0.0f, 0.0f, 0.0f, 0.0f); Rendering Opaque scene objects are drawn back-to-front, in render order. Transparent objects are drawn front-to-back. The renderer will automatically sort scene object data. However, the application may set the render order (lower render value first). Standard constants are shown below; however, any integer values are valid. Render Type = Render Value Render Order Background = 1000 First Geometry = 2000 Second Transparent = 3000 Third Overlay = 4000 Last After your startup code has built a scene graph, GearVRf enters its event loop. On each frame, GearVRf starts its render pipeline, which consists of four main steps. The first three steps run your Java callbacks on the GL thread. The final step is managed by GearVRf. 1. GearVRf executes any Runnable you added to the run-once queue. Queue operations are thread-safe. You can use the GVRContext.runOnGlThread() method from the GUI or background threads in the same way you use Activity.runOnUiThread() from non-gui threads. The analogy is not exact: runonglthread() always enqueues its Runnable, even when called from the GL thread. GearVRf Developer Guide 14

2. GearVRf executes each frame listener you added to the on-frame list. GearVRf includes animation and periodic engines that use frame listeners to run time-based code on the GL thread, but you may add frame listeners directly. A frame listener is like a Runnable that gets a parameter telling you how long it has been since the last frame. An animation runs every frame until it stops, and morphs a scene object from one state to another. A periodic callback runs a standard Runnable at a specified time (or times) as a runonglthread() callback. You can run a sequence of animations either by starting each new animation in the previous animation's optional on-finish callback or by starting each new animation at set times from a periodic callback. 3. GearVRf runs your onstep() callback, which is the place to process Android or cloud events and make changes to your scene graph. As a rule of thumb, an animation changes a single scene object's properties; onstep() changes the scene graph itself. Of course, you can use onstep() to start an animation that will make a smooth change. 4. GearVRf renders the scene graph twice, once for each eye. 1. Rendering determines what a camera can see and draws each visible triangle to a buffer in GPU memory. 2. Any post-effects you have registered for an eye are applied the buffer for step 4.a in registration order. (Typically, you will use per-eye registration to run the same effect with different per-eye parameters, but you can run different effects on each eye, perhaps adding different debugging info to each eye.) A post-effect is a shader that is a lot like the shaders that draw scene objects' skins; the big difference is that while a material shader's vertex shader may be quite complex (adding in lighting and reflections), a post-effect is a 2D effect, and all the action is in the fragment shader, which draws each pixel. GearVRf includes pre-defined post-effect shaders, and it is easy to add your own. 3. One last shader applies barrel distortion to the render buffer, and draws the barrel distortion to the screen. When the user views the screen though a fish-eye lens, the undistorted image will (nearly) fill the field of view. This step is not programmable, except in so far as you provide an XML file with screen size information at start-up time.. GearVRf Developer Guide 15