1 RemoteUI: A High-Performance Remote User Interface System for Mobile Consumer Electronic Devices Daniel Thommes, Student Member, IEEE, Ansgar Gerlicher, Member, IEEE Qi Wang, Member, IEEE, Christos Grecos, Senior Member, IEEE Abstract Modern consumer electronic devices can easily integrate into distributed computing environments. By applying network-based user interface systems to control these devices, innovative applications become feasible. However, several requirements like high responsiveness and low bandwidth consumption need be satisfied. Particularly, to be applicable in wireless and mobile local and wide area networks, they must circumvent low available data rates and high network latency. We propose a new solution called RemoteUI, which works with abstract user interface (UI) descriptions and their prioritized, incremental remote replication. The system combines different approaches from thin client computing and widget based UI systems and is further optimized in the context of wireless networks and mobile consumer devices. To demonstrate the advantages of our solution, we experimentally compare the proposed RemoteUI System with Virtual Network Computing (VNC) implementations and an HTML-based solution1. Index Terms Remote User Interfaces, Mobile Consumer Electronic Devices, VNC, Web4CE, HTML I. INTRODUCTION A growing number of consumer electronic devices can be integrated into IP networks allowing them to use and offer pervasive services. Particularly, the transfer of user interfaces (UIs) to remote control devices is an essential technique to facilitate such services in a user-friendly and locationindependent manner. Scheduling the next TV-recording on a TV set or setting the temperature at home with the help of a smart climate control device can be carried out from remote locations. Versatile mobile consumer electronic devices like smartphones or tablet computers are today's remote controls "to go". They are connected to the Internet and capable of presenting rich user interfaces for a comprehensive number of applications. In this way they become a desirable amendment 1 Daniel Thommes is with the Institute of Applied Science, Stuttgart Media University, Stuttgart, Germany ( Ansgar Gerlicher is with the Institute of Applied Science, Stuttgart Media University, Stuttgart, Germany ( Qi Wang is with the Audio Visual Communications and Networks Group, School of Computing, University of the West of Scotland, Paisley, PA1 2BE, United Kingdom ( Christos Grecos is with the Audio Visual Communications and Networks Group, School of Computing, University of the West of Scotland, Paisley, PA1 2BE, United Kingdom ( Fig. 1. Using mobile devices to remotely control other consumer electronic devices within LAN or WAN environments. or even replacement for traditional remote controls. They even allow remote control use cases that have not been possible before. Fig. 1 illustrates two of these use cases where a smartphone is used to control consumer electronic hardware from a local and a wide area network, respectively. Furthermore, the mentioned mobile devices are able to offer innovative network services themselves. For example, they can make personal information management functionality or rich media applications available to other network devices. There are situations where more convenient or better-suited input equipment for the control of these services are either preferable or even mandatory. Fig. 2 shows such a scenario in which the driver of a car controls the smartphone through the car s head unit, typically an onboard thin client computer with a larger screen. The latter mirrors the mobile phone s user interface, transmitted via a wireless personal or local area network (WPAN or WLAN), and allows controlling it with touch gestures. This solution provides a safer and more Fig. 2. Using specialized input devices to control the mobile device in a WLAN or WPAN environment.
2 convenient way for the driver to enjoy the applications running on the mobile phone. To realize the described use cases, a remote user interface system is needed that fulfills the following requirements: The complexity of both client and server software should be low, because in a consumer electronic environment the majority of devices have reduced processing power. The responsiveness of the user interface, although being transferred over a possibly slow network connection, should be high for good user experience. The remote control protocol s required bandwidth should be reduced to accomplish the desired fast reaction times in slow networks and lower the network traffic in scenarios where users are charged on a traffic amount basis. The transferred remote user interface must be adaptable to the client device, e.g. allowing high-quality client-side scaling and rotation-specific layouts. To achieve high user acceptance, the system should enable the usage of native UI-elements on client side to represent a system-specific design and behavior. The server component of the system must be portable to a variety of platforms including platforms that do not natively support graphics rendering (for example due to the absence of a display). There must be a possibility for server-initiated updates to allow asynchronous events occurring on the server to be propagated to the client. The aim of the RemoteUI Project is to develop technologies for realizing a high-performance, portable remote user interface system satisfying these requirements. In this paper, we propose the RemoteUI System and present the results of an in-depth comparison with existing well-known solutions. The remainder of this paper is structured as follows. In Section II we discuss existing work in the areas of thin client computing and remote control solutions. In Section III the architecture of the RemoteUI System and the functionality of its protocol are defined. For our experimental evaluation we implemented prototypes for the proposed system, and two existing representative solutions, as described in Section IV. With these prototypes we accomplished different experimental evaluations introduced in Section V. In Section VI we describe the setup and the results of the bandwidth and traffic volume comparison tests. In Section VII the test bed of the latency tests and experiment results are discussed. We summarize our observations and conclude in Section VIII. II. RELATED WORK Transferring graphical user interfaces from a server to a client via network connections can be realized through two major approaches. The first approach originated from the thin client concept, and typically employs pixel information or low-level graphic primitives for this purpose. For instance, systems like Virtual Network Computing (VNC) , Muse  and the system proposed by Simoens et al.  retrieve the required bitmap information from the framebuffer at the server. Systems like the Remote Desktop Protocol (RDP, based on ITU-T T.128 ), THINC  and BASS  work as a mirror graphic driver and intercept graphic commands. X- Server  introduces a graphic abstraction layer and Joveski et al.  intercept the latter for accessing graphic primitives. Information gathered from these sources is encoded with special run-length codecs like ZRLE or TRLE  or existing codecs like PNG, JPEG, H.264 or Theora. Client and server are usually connected persistently over TCP or RTP. These systems have various advantages. Typically the (thin) client software has relatively low complexity and displays a pixelaccurate reproduction of the server's UI. Employing optimized codecs, these systems perform adequately when transferring video streams. However disadvantages exist regarding client adaptation, for example client-side scaling is possible but often not optimal on mobile devices. Additionally the bandwidth requirements of these solutions are relatively high and the responsiveness in high-latency environments suffers from the necessity for full client-server round trips. There are various applications of thin client systems especially VNC-based solutions in consumer electronic environments. Haraikawa et al. propose the µvnc system , which transfers the UI of consumer electronic hardware through VNC. In this way a video recorder can be controlled with the help of a regular VNC client on a PC. To lower the complexity of the server software and supersede the necessity of a real framebuffer, µvnc uses pre-rendered images. Mainly for the realization of remote video servers, the RVU Protocol  uses an approach similar to RDP to transfer user interfaces. Videos can be embedded into these UIs using specialized video codecs. Consumer electronic devices implementing such technology, which shall support special mobile-based UI delivery options in the near future, have already been shipped. To mirror the user interface of a mobile consumer electronic device to a car's head-unit, Bose et al.  describe a solution that extends VNC with special carrelated features. The device can be connected via USB, WLAN or WPAN. The European project Mobithin  has focused on the thin client computing with mobile devices. Different proposed systems - use parts of the MPEG4 standard to accomplish high performance. The other major approach utilizes widget descriptions to transfer user interfaces, allowing a high-quality client-side rendering and scaling. These descriptions are often formulated in XML languages supporting specialized elements for widgets like ToggleButton or Progressbar. Examples are HAViML , Verhoeven and Dees protocol , LRDP  and QML . A similar approach explores HTML to realize and transfer rich user interfaces. Sonnenberg  proposes to utilize HTML for the integration of mobile devices into cars. Following the Web4CE standard , a consumer electronic device can offer a UPnP Remote UI service  in the form of an XHTML 1.0 page. The Web4CE standard also addresses a common problem of HTML- and XML-based solutions that use HTTP as the underlying protocol. A so-called NotifSocket object is defined that allows the server to send updates to the client through a persistent
4 (a) Fig. 4. (a) The left UI-Tree being rendered on the phone, (b) UI-Tree manipulation (b) shall be displayed through UI-Tree manipulations, which are triggered by server commands. The client receives and instantly executes the commands, which includes rendering the resultant tree. To allow the manipulation, each node in the UI-Tree carries a unique ID that is used for addressing. Fig. 4 (b) exemplifies this process. The left tree is transformed into the right by replacing text and image information in different nodes. For certain widgets a round trip with the server is necessary to determine their appearance. The server for instance defines the state of a toggle button after it is pressed. During the following round-trip the button is in a pending state. If the round-trip time is high, this new state should be presented to the user. We call this phenomenon the additional-state problem that can generally occur in remote UI systems. The RemoteUI Client in future versions will provide special modified widgets that are capable of displaying these additional states. B. Incremental Transfer and Rendering Although the amount of data required to express a user interface with the RemoteUI Protocol is relatively small, there are situations where the transfer of a complete UI-Tree takes a long time. This can be the case in scenarios with low available bandwidth or the UI containing many images. To provide fast user feedback, the RemoteUI-Server incrementally sends the UI-Tree as soon as the first nodes become ready for transfer. The RemoteUI-Client incrementally receives, builds up and renders the local copy of the tree. This results in a row-by-row display of the user interface. If the first row of the layout can be displayed within less than a second, we assume this as sufficient for the required timely user feedback. Additionally, a less conspicuous progress indicator can be part of the interface to indicate a pending receive process. C. Prioritization The RemoteUI Protocol offers a mechanism to prioritize different user interface and command objects. The server processes and transmits objects with high priority first, which intentionally causes a delay for lower prioritized objects especially in low bandwidth scenarios. Because the client is able to incrementally render these objects, this technology reduces the user interface latency. In a preliminary approach, widgets displaying pixel data (ImageViews) are separated from the UI-Tree and transferred as lower-prioritized SetData commands. Part of our research is the development of algorithms to determine the optimal priority from input parameters such as visibility, traffic volume, operational importance, available bandwidth and network latency. By continuously applying these algorithms at runtime, the system will adapt to changing environment conditions to offer the highest possible performance. D. Caching and Prefetching Theoretically a UI-Tree does not need to be transferred more than once. Although there are screen switches that are realized by replacement of the tree, it is likely that a tree will be reused once loaded. Avoiding the retransfer of resources like pixel-images used in a UI-Tree can reduce the required traffic volume significantly. To profit from this fact, the RemoteUI System caches resources on the client. Additionally, during idle-times, a low-priority process prefetches resources that are likely to be required later on. A tree that contains a resource that has already been fetched only needs to contain a reference to it, which reduces the user interface latency. IV. PROTOTYPIC IMPLEMENTATIONS As platform for all prototypic implementations we chose the Linux-based operating system described by McLoughlin  later on referred to as target platform. This system is available for different kinds of mobile consumer electronic devices like smartphones, tablets and multimedia hardware. Additionally a specialized version is available for TV systems predestining the platform for our prototypic implementations. A. RemoteUI System Prototype We realized the RemoteUI System with the components described in Section III for the above mentioned target platform. Fig. 5 shows the applications that have been developed by us. While the RUI Client was realized as a stand-alone-application, the server functionality was divided into different applications. It was an important requirement for us to avoid manipulations of the operating system or the necessity of gaining root privileges to run our applications. Access to the user interface on the target platform is allowed