How Mobility is Affecting Embedded Devices and Development Webinar March 27, 2014
Your Presenter Chris Ault Product Manager, QNX Software Systems
3 Agenda Trends Choices of graphics technologies for embedded Android HTML5 Open GL Qt Rendering content from various graphics technologies User interface requirements: Multi-touch gestures, Browser/ Webkit Accessing media content, media detection, device access Going Mobile: Security & Power Management Wrap it all up / Summary
4 What is driving user interfaces? Consumer market impacting all traditional embedded markets Consumer, medical, auto, industrial Younger generation creates strong culture of mobile device integration Innovative and booming application market OEMs wanting to focus on differentiation and accelerate their innovation curve with fewer resources Don t want to do all of the non value-add integration And we GET it!!!
Mobile influence in Embedded UI Features 5 Growing demand and preference for standards-based solutions Consumer UI features emerging in embedded devices Smooth transitions between screens and animations Rich graphical assets (designer driven as opposed to developer driven) Multimedia, 2D/3D Speech interfaces (recognition, text-to-speech, speech-to-text) Handset integration Cloud connectivity apps, content, device synchronization and over-the-air updates Multi-modal interaction through gestures (swipe, pinch, tap), multi-touch, buttons, speech,
Mobile influence in Embedded Silicon trends 6 From CPUs with discrete graphics and peripherals to highly-integrated System-on-Chip (SoC) High performance CPU engines 32-bit CPU: single, dual, & quad core (e.g. ARM core or Intel Atom ) High performance 2D/3D graphics engines with multi-display capabilities Integration of high performance GPU engines on the SoC General and device/market specific I/O complement Audio, Video encode / decode, CAN, USB, Bluetooth, Ethernet, I2C, SPI, SATA, GPIO, UART, PCI, memory controller, Bluetooth, WIFI, GPS, DSP, Modem, Security (e.g. TrustZone, crypto engines) Hardware-assist for virtualization Power management (system-wide) SoC price points to meet specific market needs Silicon vendors leveraging investment in mobile
Improve time to market 7 QNX helped one automotive customer cut award to SOP time from 3 years to 13 months QNX is committed to further driving down development time while increasing functionality Complete software ecosystem, already pre-integrated Modularly architected around software reuse and user interface re-skinning
8
Example: Audi FPK Driver Information Display Confidential Information of 9 http://youtu.be/wlm6ld23d0g
Audi FPK Driver Information Display Confidential Information of 10 http://youtu.be/wlm6ld23d0g
11 Lowrance SonicHub Marine Chartplotter and GPS http://www.lowrance.com/en-us/products/weather-entertainment/sonichub-en-us.aspx
12 Example of Smartphone Integration Lowrance SonicHub SonicHub integrates with Lowrance HDS chartplotter system Detects the presence of iphone and content Presents the ability to access the iphone content via the chartplotter UI http://www.youtube.com/watch?v=dt57ty5du4s
Consumer vs. Embedded Systems lifecycle Confidential Information of 13 Applications Smartphones Embedded systems Year 0 Year 5 Year 10 Year 20
14 Graphics: One Size Doesn t fit all
Native UI framework? Confidential Information of 15 C/C++ graphical environments Examples: Qt, Elektrobit EB GUIDE, Crank Software Storyboard, OpenGL ES Advantages Generally better performance, smaller size, quicker booting compared to HTML5 or Android-based UI Ability to leverage existing in-house experience with tools, developers, workflows Who is building your UI: embedded developers or mobile app developers? Challenges Tools often proprietary and not (generally) used for cross-platform mobile development Qt addresses mobile development with the upcoming Qt 5.2 release Often necessary to integrate native environment with other application frameworks
16 Android: Advantages Much more than a UI operating system (Linux-based), VM (Dalvik), frameworks, apps, application store Broadly adopted in smartphone space 80% smartphone market share in Q2, 2013 (Strategy Analytics Aug 01, 2013) Large developer community Lots of tools, libraries, SDKs, and documentation Tons of apps Free (depending on use)
Android: Challenges Fragmentation: Rapid releases and customizations ~13 Android releases since 2008 Typically long cold boot times (e.g. > 60 seconds) Designed around suspend to RAM (assumes battery backup) Native and multimedia support No direct support for some Microsoft codecs (e.g. wma); no ipod connectivity Native code discouraged High Safety or Security level certifications are very difficult if not impossible Commercial licensing and deployment Patent licensing agreements may be required 70 percent of all Android smartphones sold in the U.S. are now receiving coverage under Microsoft s patent portfolio. (http://www.microsoft.com/enus/news/press/2012/jan12/01-12lgpr.aspx) Value-added services (i.e. Google maps, voice, earth) may require a license fee Ice Cream Sandwich Honeycomb 23.3% 34.1% 37.9% Confidential Information of Jelly Bean Froyo Gingerbread Source: http://developer.android.com/ Eclair 17
18 HTML5: Overall picture HTML5 activity very high Available on every major OS platform Available on all smartphone platforms: Android, BlackBerry, ios, Windows Phone Four main categories of usage For a built-in UI For an app container environment For mobile integration (e.g. mobile-> embedded device) For cloud-based applications
HTML5 decision factors Reskinning/personalization/customization Cascading Style Sheets (CSS) provides easy reskinning HTML5 is an and excellent fit here; UI reskinning as part of their DNA Code reuse across scalable product lines HTML5 runs on mobile devices and on embedded devices Mobile integration Mobile takes over the embedded device UI No embedded UI on the device -- webserver Mobile app use on embedded device: use MirrorLink, Miracast, ipod or equivalent for expansion Downloadable 3rd party applications Isolated execution container -- inline frames (iframe) + sandbox element Other HTML5 considerations Boot time, application performance, memory requirements, access to system services 19
OpenGL ES as an HMI Benefits Cross-platform industry standard (www.khronos.org/opengles) Direct API mapping to hardware acceleration in many GPUs high performance Default for certain class of apps e.g. avionics displays, instrument clusters, navigation applications especially where certifications are required Challenges Many necessary components of an HMI outside OpenGL ES scope Frame buffer management, graphical composition, windowing, event handling input management Expert OpenGL developers are hard to find Requires understanding of matrix math, 3D transforms, texture operations Shader language (GLSL) not easily debugged, shaders are a unique specialty Often developed and tuned to a specific configuration GPU vendor-specific OpenGL ES extensions and optimizations Confidential Information of 20
The Qt Framework 21 Cross platform software development framework Implemented in C++ API in C++ with binding to JavaScript, Python, Ruby, etc. Available on all desktop platforms, embedded and mobile Excellent documentation and learning resources User Interface Technologies Native Widgets (push buttons, combo boxes, etc.) QML (JavaScript based declarative UI) HTML5 (integrates WebKit) OpenGL (acceleration on GL or GL/ES2) Non-graphical, middle-ware components Network, SQL, XML, printing, PDF generation, etc. Easy to interface via SOAP, REST, XMLRPC Regular expressions, containers, powerful string handling, etc.
22 The Qt Framework continued Fully cross platform IDE (Qt Creator) Integration with Visual Studio, Xcode, Eclipse/Momentics Framework for building installers Framework for building custom, domain specific UI editors Built-in support for cross-compilation Easy switching between local execution (developer desktop) and deployment to target for short development cycles Built-in on-device debugging
Qt is Used Everywhere Confidential Information of 23 From Desktop Applications to Embedded & Mobile Devices 5000+ Companies from 70+ industries use Qt
24 Qt on QNX Latest versions of Qt 5 ported, integrated and optimized for the QNX realtime operating system All source code is upstreamed to http://qt-project.org and is a part of Qt 5.2 Digia regression-testing Qt 5.3 on QNX nightly Full integration High-performance QNX OpenGL ES library and driver integration and testing QNX Screen composition manager integration to support multiple UI technologies Well defined and optimized Qt <-> OS services layer through PPS (Persistent Publish Subscribe) Complete integration of Qt 5 with the QNX OS and the QNX CAR Platform for Infotainment In use today by many customers across many markets (industrial, medical, automotive, defense & aerospace ) Ongoing support services from QNX Software Systems (www.qnx.com) and key partners such as Digia (http://qt.digia.com) and KDAB (www.kdab.com)
25 Rendering Graphics Content
26 Rendering from multiple input sources 2015 Audi TT Dashboard: multiple UI technologies HTML5, OpenGL ES, Qt, video Mix and match graphical components to create compelling UIs Required deep graphics programming skill to arrange the framebuffers and handle the transformations (e.g., scaling, translation, rotation, alpha blending, etc.) Need to create a single unified graphical interface from multiple UI technologies
Multiple UI Rendering - example Confidential Information of 27 Example with 4 UI technologies Background wallpaper: HTML5 Video: From iphone or ipod Navigation: OpenGL ES Control application: Qt5 Transparently combine multiple off-screen buffers onto the display Windows can be moved around, zoomed in / out, rotated, or have transparency effects applied without requiring the application to redraw or even be aware Hardware and virtual layer support Full hardware acceleration Multi-modal input device support
28 Multi-touch gestures Multi-touch and gestures are being supported by an increasing set of mobile devices Oscilloscopes, vending machines, GPS plotters, ATM payment systems, etc. etc. Composite gestures may send multiple notifications to the application as you continue to touch or move your fingers on the screen Discrete gestures send only a single notification to the application Your customers are expecting this behaviour you need to implement it Composite Swipe Pinch Rotate Two-finger Pan Discrete Tap Double tap Triple tap Long press Press and tap Two-finger tap
Web Browser 29 Support for HTML5 standards Audio/ video (through QNX multimedia technology) Geo-location Canvas WebSockets Offline applications Session storage Enhanced browser High performance panning and scrolling Pixel-accurate zooming Physics-based scrolling and panning JavaScript JIT
30 Browser UI HTML5 engine and browser Browser components / framework Engine is highly optimized and tuned version of WebKit GPU accelerated CSS transitions Media WebKit Browser control API WebGL support (JavaScript access to OpenGL ES 2.0) Fully integrated for QNX SDP 6.6 platforms Fully featured browser provided WebCore JavaScript Core Browser UI implemented in HTML5 itself Audio Memory manager Full browser source allows complete customization Best HTML5 compatibility See html5test.com mobile browser results OS adaptation Highest HTML5 performance* MIME type content filters HTTP framework OS services
BlackBerry 10 obliterates ios and Android yet again in browser showdown Browser UI 31 Browser components / framework WebKit Browser control API BlackBerry 10 obliterates ios and Android yet again in browser showdown: Media WebCore JavaScript Core BlackBerry users have had to deal with what was undeniably the worst mobile web browser on the planet Audio Memory manager BlackBerry sought to change that with its new BlackBerry 10 operating system, and we re not sure it could have possibly done a better job OS adaptation the BlackBerry 10 browser is more than three times faster than its next-closest competitor MIME type content filters HTTP framework OS services http://bgr.com/2014/02/06/blackberry-10-browser-test-iphone-android/
32 Poll Question 1: Which graphics technology are you planning to use in your next project? A. HTML5 B. OpenGL ES C. Qt D. Electrobit / Crank E. Other POLL
33 Accessing media content and media detection
User Interfaces and Content Considerations: Your to-do list Confidential Information of Audio / video playback Play, pause, stop, seek, status events, volume, Media detection Apple iphone/ipod (iap, ipod video out) Android/Windows (MTP) USB mass media Metadata synchronization and management Title, genre, artist, album, album art, etc. Playlist support M3U, itunes, PLS, ASX, RMP, B4S, WPL, XSPF Integrated audio codecs MP3, WMA9, AAC, AAC+, PCM/WAV, MPEG2, MPEG2.5 Integrated video codecs H.264, MPEG4, XVID, VC-1, WMV9, DIVX Input Multi-touch, gestures 34
35 1. Detecting device attachments 2. Obtaining device information 3. Publishing device information 4. Detecting devices in media applications 1 4 3 2
36 2 4 1 1. Starting a synchronization 2. Extracting file and media metadata 3. Storing file and media metadata 3 6 4. Extracting playlist entries 5 5. Storing playlist entries 6. Retrieving synchronized metadata
37 2 1,2 1. Configuring the rendering service 2. Controlling playback 3. Outputting audio and video 2 2 3 3
38 Going Mobile: Security and Power Management
39 Mobility requires Security Devices are carried, roaming across multiple free insecure networks root permissions give the attacker full control of the system ALERT: Feb. 22, 2014: Apple Security flaws Mac computers even more exposed "It's as bad as you could imagine, that's all I can say," said Johns Hopkins University cryptography professor Matthew Green. authorities had 100 percent success rate in breaking into iphones Embedded devices contain corporate data, not just: Contacts, e-mails Calendars, Facebook http://www.reuters.com/article/2014/02/22/us-apple-flaw-idusbrea1l01y20140222
40 Mobility requires Security Securely packaging, installing, and launching apps: Apps get downloaded from untrusted providers and may contain malicious threats You need a packaging mechanism that validates and bundles your app's resources as well as your app s capabilities, properties, and launch icon When launching an app on the device, you need a mechanism to safely launch and contain the app in an isolated sandbox Manages app lifecycle, starting/switching/ending apps Define how apps interact with system (event control, background execution, etc.) There needs to be a mechanism to oversee and contain the system resources which the app can access during its runtime
41 Mobility requires Security The OS needs to provide more protection against hackers in mobile networks Permitting a thread to elevate to root permission to do an operation is too coarse Processes and threads need access to system-level resources YES! We know to which system resources a process or thread needs access YES! User-input needs access to keyboard driver and interrupts File I/O needs access to the filesystem Neither of these need access to mmap() or fork() (for example) We have a good idea of which system-level privileges each process and thread needs access So MAP it out!
42 Mobility requires Security The OS should provide much more fine-grained control of system privilege levels control settings that govern and protect which operations a process can perform, with granularity down to the system-call level no longer have to give processes root access to the entire system Breaks root into multiple separate capabilities that comprise root authority Individual capabilities can be assigned to processes that need access to each specific resource But no other resources Compromised processes only have a tiny subset of privileged operations available Even if they become root
43 Power saving opportunities Interrupt Clock Time Timer Timers, Interrupts, clock ticks, all causing the CPU to wakeup and consume power
44 Power saving opportunities Event/ Power Wakeup Time 10 separate events/wakeups
45 Lazy interrupts Time Interrupts that don't require real-time precision can be assigned a latency value Best used for non-realtime requirements Keyboard input, etc.
46 Tolerant timers Time Timers that don't require real-time precision can be assigned a tolerance value Classic example is a UI response timer set in seconds Allows you to specify how much tolerance the kernel is allowed when scheduling your thread after the timer fires
Tickless kernel Handle 3 coordinated events rather than 10 events/wakeups 47 Time The OS still has ticks, and everything runs as normal BUT: Only when the system goes completely idle does the kernel "turn off" clock ticks, In reality it slows down the clock so that the next tick interrupt occurs just after the next active timer is to fire
48 Poll Question 1: Do you have plans to implement embedded designs with smartphone-like functionality? A. Yes, in the near term (0-6 months) B. Yes, in the mid-term (6-12 months) C. Yes, but further out (1 year +) D. No POLL
QNX is advancing user interfaces today Confidential Information of 49 Mining Control In-Flight Entertainment
Dalian Eastern Display: Already Doing This 50
51 Call to Action
QNX SDK for Apps & Media Free software development kit QNX SDK for Apps & Media Confidential Information of Confidential QNX Software Information Systems of Limited 52 52 HTML5 Engine Highly optimized HTML5 Browser and Engine HTML5 Application Development Javascript application frameworks Authorization manager, launcher, installer Multimedia Management Media detection, synchronization Metadata handling (album art, etc) Playlist support Multimedia Playback Audio / video control (play, pause, stop, ) Integrated audio / video codecs Multimedia Device Connectivity ipod (iap, ipod video out), Android (MTP) support HTML5 Engine Multimedia Management HTML5 Multimedia Multimedia Playback QNX SDP 6.6 HTML5 Application Development Multimedia Device Connectivity
QNX Software Development Platform 6.6 Software Stack Confidential Information of 53 QNX Momentics Tool Suite Qt 5.2 3 rd Party UIs Storyboard EB Guide QNX Neutrino RTOS Screen Composition Manager GPU Acceleration Display Controller Device Input Font Rendering Fonts Core OS Neutrino ukernel Adaptive Partitioning High Availability Multicore Networking FileSystems Connectivity Manager Utilities Database Hardware support Cortex A15 Cortex Cortex A15 A9 Cortex Cortex A9 A8 Cortex Intel A8 Core tm Intel Core Intel tm Atom tm Intel Freescale Atom tm QorIQ tm
A continuum of Integrated technologies for building mobile-like devices Confidential Information of Confidential QNX Software Information Systems of Limited 54 54 The only platform that brings the user experience of mobile devices to secure and reliable embedded systems Automotive Infotainment Systems QNX CAR Platform for Infotainment 2.1 Android Speech recognition MirrorLink Bluetooth (integrated) Over-the-Air updates NAV systems Systems with Complex HMIs QNX SDK for Apps and Media 1.0 Multimedia Media detection, playback ipod and Android connectivity HTML5 SDK, browser, engine Traditional Embedded Systems QNX SDP 6.6 Security Power management Screen UI Multi-touch Video capture USB OTG
QNX Accelerator Kits www.qnx.com 55
QNX Accelerator Kit Ti AM335, Freescale I.MX6. Free. Reference Applications and Demos Thermostat Instrument Cluster Audio Meter High Availability demos Connectivity Wireless Diver Ethernet MMC-SD USB Serial UART SPI, I2C support IDE Integration Momentics project Confidential Information of 56 Advanced Graphics and Video Qt 5 demo applications Crank Storyboard Demo OpenGL ES graphics & demos
57 Summary Driven by the smartphone generation, embedded device manufacturers are being forced to build more compelling UIs Building smartphone-like UIs requires many subsystems Multimedia Audio, video codecs Graphics subsystem that makes it easy to control, render from multiple sources Mobile devices require heightened security regimes Improvements in power and thermal management Building these devices involves integration of many subsystems, but can be made easier for the device manufacturer
58 Questions? Chris Ault Product Manager, QNX Software Systems
2013. QNX, QNX CAR, NEUTRINO, MOMENTICS, AVIAGE and other product names are trademarks of BlackBerry Limited, which are registered and/or used in certain jurisdictions, and used under license by QNX Software Systems Limited. The information herein is for informational purposes only and represents the current view of QSSL as of the date of this presentation. Because QSS must respond to changing market conditions, it should not be interpreted to be a commitment on the part of QSSL, and QSSL cannot guarantee the accuracy of any information provided after the date of this presentation. QSSL MAKES NO WARRANTIES, REPRESENTATIONS OR CONDITIONS EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.