Approach of In-Vehicle Infotainment development on open source software. May22-24, 2013 @San Francisco Takeshi Hoshina

Tizen Developer Conference Approach of In-Vehicle Infotainment development on open source software May22-24, 2013 @San Francisco Takeshi Hoshina 1

Self-introduction

Agenda What is IVI system? How We collaborate with Tizen IVI Introduction of Toyota source code UI Manager Vehicle Information Control (VIC) Home screen Summary Q&A 3

What is IVI system? Portable Device IVI Other ECU RSE Rear view camera Example In-Vehicle Infotainment (IVI) system i.e. Navigation, Multimedia, Telematics/Call Assistant Portable Device and other ECUs are connected to IVI

How We collaborate with Tizen Toyota developed reference platform based on Tizen IVI Released SW packages to tizen.org include Weston plugin Device input controller Pulse Audio plugin Vehicle information control (VIC) plugin for automotive message broker Car simulator Sample home screen Sample apps Reference Platform Ver0.5 (RPF0.5) Connecting Car, Driver and Internet ico

TIZEN IVI RPF0.5 on Tizen IVI RPF0,5 OSS RPF0,5 sample code Closed( Proprietary etc.) Portable Device IVI App Layer ECU Apps HomeScreen Native Apps HTML5 Apps Native App Tizen-IVI EFL Tizen-IVI EFL UI-component Webkit-EFL Remote UI Remote UI Remote UI Wayland/Weston Pulse Audio UI Layer weston -plugin pulse -plugin System Layer Murphy AMB Policy DB amb -plugin CorePF Layer Device Input Controller Cgroups Smack Car Simulator OEM CAN MW internet Tizen IVI on Nexcom NDiS 166

UI Manager Multi Window Manager Multi Input Manager Multi Sound Manager Vehicle Information Control Home Screen

UI-Manager Focus Requirements Easy-to-use user interface Simple and Smart architecture to reduce SW complexity Robustness under limited memory resource on embedded CPU Automotive reliability Mobile Apps portability Toyota UI-Manager Utilize OSS community efforts and common code base Adapt software layer architecture Use plugin and common IPC Support mechanism to avoid driver distraction

Role of UI-Manager IVI Remote-UI UI-Manager (View) Multi Sound Mgr. For Audio Tel/TTS Multi Input Mgr. For touch SW steering SW STT Multi Window Mgr. Center Display Window Mgr. MID/Cluster Display Tizen-IVI Apps (Control or Model) Portable Device Proxy Navigation Apps Home Screen Radio TV Air-conditioner Camera Cluster ECU (Model) Portable device Audio Air Conditioner Cluster

MWM Usage example with multi layers Service Apps Data Type Cluster ECU Camera Back Camera Front Camera MID/Cluster Informatio n Super imposed Menu Apps AV 3D 2D 3D RGB Wayland + UI-Manager Center display Remote U/I 2D 3D Navigation Route guidance 2D 3D remote UI (Drawing API) Map 3D 2D IVI Portable Device

MWM functional requirements Sophisticated user interface Multi layers for IVI High performance - Wayland/Weston plugin High scalability - Pluggable OEM specific drawing App coexistence with car OEM applications Monitor Window operation of applications Apps execution environment (Web, Native)

Why do we want to use Wayland/Weston? Simple and Smart Wayland/Weston = Display Server + Window Manager Direct rendering No use of X11 rendering protocol High scalability Easy to add OEM original features Drawing method 3D animation, etc Car OEM original UI devices (Haptic device, etc)

MWM feature(rpf0.5) Basic window functions Create and delete two or more windows Manage it's position, width and height, visible/invisible and effects Manage two or more windows within one display device Window Layer Management Free arrangement of the layers Batch control of windows on the layers Window Layout Management (2 types) Compositing window manager using Normal Apps, Pop-Up and so on. Tiling window manager using Home-Screen and so on

MWM software block diagram (RPF0.5) HomeScreen Native App HTML5 App UX-FW(API) UI Component Webkit2-EFL OpenGL/EGL API EFL Wayland API Weston Application Mgr App Manifest App Manifest App Manifest Configurations Wayland-Server IVI Shell Plugin Loader IVI Common RemoteGL Proxy Multi Window Manager Multi Input Manager Remote UI Broker Other ECU Weston 別 ECU OpenGL ESv2/EGL Function call WebSocket Protocol Wayland Protocol

MIM functional requirements Sophisticated user interface High efficiency - Weston plug-in High scalability - Can add OEM specific input device Manage input events Abstract Input Event Map Input Event and each application Arbitrate Input Event conflict Suppress Input Event by vehicle condition Near future, MIM manages the input data even for other ECU

MIM feature (RPF0.5) Basic Input Device Mouse (Touch-Screen), Keyboard, Joystick Mapping Apps and Input devices Mapping by rule-based DB Deliver the event to the apps Support Car OEM specific devices Device Input Controller

MIM software block diagram (RPF0.5) HomeScreen UX-FW(API) Native App HTML5 App UI Component OpenGL/EGL API Wayland API Weston Wayland-Server IVI Shell Plugin Loader IVI Common RemoteGL Proxy Multi Window Manager Multi Input Manager Remote UI Broker WebSkeleton Webkit2-EFL EFL Other ECU Weston 別 ECU Function call WebSocket Protocol Wayland Protocol Device Input Controllers Remote リモートタッチ touch Haptic ハプテック HW スイッチ switch 入力 Touch タッチ input 入力

MSM functional requirements Sophisticated user interface High efficiency - pulse audio plugin High scalability - Pluggable Car OEM specific sound functions App Coexistence with car OEM apps Apps execution environment (Web, Native) Monitoring and Arbitration sound operation(*) of each application (*) Mixing/attenuation/pause/Mute/Cut off etc. Zone-control Multi sound zone (front and rear etc.) Near future, Remote-UI MSM manages the sound (in/out) even for other ECU

MSM feature (RPF0.5) Basic Audio functions Dynamic Volume Control Dynamic Mixing Control User operation (Audio apps, Mute button etc.) Other situation (Phone call etc.) Manage apps using Sound Arbitrate sound operation from each application (Priority scheduling by rule-based DB) Cooperation with the Home screen Mute, Cancel, get attribute notify status change to application

MSM software block diagram (RPF0.5) HomeScreen Native App HTML5 App libpulse In/out WebSkeleton control In/out Webkit2-EFL EFL libpulse In/out Other ECU PulseAudio Hook Pulse Audio UIFW-PlugIn Hook Rule-based DB out in control ALSA out PulseAudio Protocol WebSocket Protocol Wayland Protocol

VIC functional requirement VIC caches vehicle data sent by the ECU. Vehicle data is offered by the demand from the Vehicle apps. VIC uploads vehicle data to the center. Vehicle apps controls each ECU. 3 Gate Way Vehicle apps Automotive BUS 4 2 Vehicle Information control 1 IVI

VIC feature (RPF0.5) High efficiency Work as Automotive Message Broker plug-in Manage Vehicle Data Abstract vehicle data Data format for in-vehicle LANs may change Two Data format type Real-time data in the cache History data in the vehicle-information-db Two API property type Common API OEM specific API Non Disclosure Car simulator for software debug Can simulate the running state of the vehicle Steering, Acceleration pedal, Brake Pedal etc. Increase efficiency for debugging during vehicle test API for car simulator and vehicle middleware are the same

VIC Software Block Diagram (RPF0.5) Native HTML5 HTML5 Apps Apps Apps Apps Apps Apps Apps Apps Apps Probe Manager D-Bus plugin AMB WebSocket sink Other ECU AMB User info User Manager Device info History Weston Vehicle info History (standard) Vehicle info History (OEM) History (standard) AMB core History (OEM) OEM VIC API VIC plugin Standard VIC API WebSocket Source VIC MW Car Simulation TIZEN RPF0.5 Car Drv CAN HW WebSocket socket CAN 27

VIC topics Policy of API Definitions Contents Provider needs open standard vehicle API Car OEM can not support all standard API Depend on each car OEM Personal information protection policy Safety, Security policy Business strategy (probe data property etc.) Depend on each vehicle model Equipment of vehicle package option (Navigation, ADAS, etc) Automotive LAN grade (e.g. not connected some ECU) We need to define open standard API! But, We should be provided with the access control every API (Property)

Home Screen feature (RPF0.5) User Menu Simple Tile Menu (using Window Manager) Status Bar Launcher (with sample Apps) On Screen Application control regarding window, input event, sound Interrupt Action Change View for Driver Distractio (interrupted by vehicle inf. via Automotive Message Broker)

Overview of Sample Home Screen Local Apps Navigation Audio System Controller Home Screen Native Apps Web Apps API G-book Entune Navigation Audio UI manager EFL HTML5 POSIX System call (Process/File/device) PF Navigation G-Book Entune Tizen IVI (with UI Manager +VIC) Linux kernel Navi core: maps, route search and guidance

Home Screen & System Controller Home Screen System Controller Layout Policy DB each Destination/Vehicle OEM Home Screen Main Menu Status Bar On Screen Apps Controller Resource control Life cycle (launch,) Access control Multi User Device Controller Vehicle Controller Native Apps Request resource HTML Apps AppsFW lib Apps Framework PF lib UX-Framework Tizen API EFL AMB PulseAudio Murphy etc.

HS topics Sample Screen Layers (RPF0.5) Event handling (Input Layer) On Screen Message (ONS layer) Apps Display (Apps Layer) Apps Menu (Home Screen Layer) status bar, live thumbnail, Apps launcher etc. Input ONS APPS HS Multi Layer Apps Menu (Tile Window) RPF0.5 sample code

HS topics Change View for Driver Distraction In IVI system, application must follow the rule and the law. Only trusted applications are allowed to display during driving. UI-Component is libraries for trusted application. Home Screen (or alternate system app) Make invisible during driving No control even if driving Un-trusted Apps Trusted Apps Reduce development cost and time by using UI-Component. UI Component

UI Manager Multi Window Manager Multi Input Manager Multi Sound Manager Vehicle Information Control Home Screen ico

Download http://download.tizen.org/snapshots/2.0alpha/ivi-wayland/latest/images/ ivi-wayland/ Packages ico-vic-amb-plugin: Automotive Message Broker (AMB) plugin for delivering vehicle information to applications. ico-uxf-weston-plugin: Weston compositor plugin for UI management. ico-uxf-device-input-controller: Touch screen and USB steering wheel input device calibration. ico-uxf-homescreen-sample-apps: a set of sample applications from Toyota including AR navigation, sound and vehicle information controller applications. These apps are examples of how applications behave when the vehicle state changes. i.e. running, stop ico-vic-carsimulator: CAN bus emulator ico-uxf-pulse-plugin: a policy enforcement plugin to Pulse Audio ico-uxf-homescreen: sample UI for application launcher and manager. 36

Summary Toyota delivered OSS packages UI-Manager Vehicle Information Control Sample home screen Keep up to date on Tizen IVI Let s collaborate in OSS community

Any questions? Thank you!! hoshina@takeshi.tec.toyota.co.jp

Backup 39

MWM TOPICS sequence chart App Home Screen OpenGL /EGL API Wayland API Wayland Server IVI Shell Weston Multi Window Manager Weston Compositor Multi Input Manager OpenGL/EGL Create Wayland Surface HomeScree WebSocket Protocol Wayland Protocol Create Shell Surface Shell Surface Create EGL Surface EGL Surface EGL Surface EGL Surface EGL Surface Positioresize - layer Shell Surface EGL Surface HomeScree Notify Window Resize surface Notify Window PositioSize Layer Window HomeScree Westo ( ) ( ivi-shell ) (HomeScree )

MWM TOPICS sequence chart App Home Screen OpenGL /EGL API Wayland API Wayland Server IVI Shell Weston Multi Window Manager Weston Compositor Multi Input Manager OpenGL/EGL Surface Surface WebSocket Protocol Wayland Protocol Surface Window Surface Surface EGL Surface HomeScree Surface Surface Surface Surface Surface Surface Window Surface HomeScree

MIM sequence chart (RPF0.5) App App App HomeScreen Wayland API Wayland Weston Multi Window Manager Multi Input Manager Device Input Controllers Remote Touch Touch Panel Steering Switch Input Devices Surface Surface ID WebSocket Protocol Wayland Protocol

MSM sequence chart (RPF0.5) App Home Screen Pulse Audio Pulse Audio UIFW-PlugIn ALSA Drv (A) (A ) ( ) A / / / / A A B / A A A ( ) B PID (A) ( ) ( ) ( / ) (B) ( / ) (A ) ( ) ( ) WebSocket Protocol Wayland Protocol

MWM TOPICS Application monitoring HomeScreen Native Apps HTML5 Apps App-FW Tizen-IVI EFL Tizen-IVI EFL UIcomponent Webkit-EFL Multi Window Manager Wayland/Weston

HS topics Change View for Driver Distraction Home Screen (or alternate system app) UI Component for Web Apps for Native Apps Murphy Automotive Message Broker VIC plugin. Vehicle Information If you want to use policy based decision making, "Murphy" is useful for it. CAN

HS topics UI Components functional overview Text labels Radio button Check box Tab panel Audio setting panel Element type Driving mode visibility Allowed with restriction below: - (Font size > limit) && (Text string length < limit) - If Text label supports text-scrolling, then scrolling is not allowed. (should stop scrolling) Animation (MNG, GIFanim ) Slider Accordion list Anchor and mode transition Text Input form Dropdown list box Date picker Allowed with restriction below: - Animation should stop. (only still image is allowed) Allowed with restriction below: - 2 value slider is not allowed, only 1 value. - Accordion should not be animated. Not allowed.