Trends in Video Processing for Mobile Systems

Transcription

1 Trends in Video Processing for Mobile Systems Intel Corporation - Client Computing Group Jean-Pierre Giacalone Principal Engineer

2 Introduction We are in a phase of acceleration in media technology. This is driven by how we consume media and use our devices. We see a race in resolution, color gamut, frame rate, dynamic range. We observe new domains becoming reality (360 Videos, Augmented Capture, Virtual Reality) In this talk we will: Understand and acknowledge these evolutions Look at key ingredients in Systems and SOCs List the associated big challenges and call for actions 2

3 Rapid Pace of Innovation in Video Technologies Driven by a Shift to IP-Based Content Delivery Twitter Sling Box 7 th Gen Consoles Facebook Tivo Blue-Ray Netflix DVR YouTube Netflix Streaming Hulu+ Smart TV Tablet GoPro Hero HD Netflix UHD Direct TV 4K YouTube 360 Video 8K Coming soon! VCR Cable DBS Satellite Yahoo Google VOD Apple TV HBO Go Amazon Prime HDR Amazon Fire TV AOL Broadband Smart Phone Roku Chromecast HDTV On the Horizon: Dolby Vision Netflix HDR 8K Broadcasts UHD Blue-Ray VR Media Amazon Prime Instant Video Video Technology race is escalating. New vectors of visual quality & immersion are emerging. 3

4 Streaming Media Exploding on all Form Factors Subscriber to OTT TV services 92.1M 332.2M ( 14-19) Juniper Research By 2019 nearly 80% of internet traffic will be driven by video Cisco 3x! Nearly 7 in 8 Americans reported watching online video, with more than half doing so daily. Comscore Video usages growing thanks to several key enablers 4

5 Million Units The 4K ramp Enabling more and better pixels 4K TV and desktop volumes growing rapidly OS & Microsoft * software ready for 4K since Windows * 8.1 Content creation: Adobe *, CyberLink *, DivX *, etc. ready Games/Office/Productivity suites: all ready for 4K Intel equipping 4K panels All-In-Ones Desktop+AIO 4K & 5K Panels Volume (Mu) % TAM 10% 8% 6% 4% 2% 0% 23.6 Available now IPS class 4K/60Hz 300nits 1000:1 contrast ratio Wibtek A24 1. DisplaySearch Expect to see 10b SDR first out, then 10b HDR. Next immediate train we are boarding is 8K. 5

6 Why HDR? Higher Brightness Dynamic Ecosystem 100 NITS range (per BT709) Better Colors NITS range Wider Contrast & Color Range Very Bright Spots Higher details (10b) HDR provides a significantly better viewing experience. Start to see aggregation of full eco-system and standards. Intel Restricted Secret 6

7 Key HDR ingredients and data flows Decode: HEVC 10b 420 & possibly 444 Requires Protected Video Path and HDCP2.2 encryption to remote display. Assemble/Convert SDR / HDR conversions Decoded Streams combinations HDR10/BD-A -> 1 stream Dolby Vision -> 2 streams Base layer (BT709) Enhancement layer (BT2020) Blend and HDR Display control SDR GFx HDR to SDR 1 stream, HDR Codec Convert Compose HDR SDR Display I/F SDR Monitor Scaling in HDR color space and dynamic range (FHD content) Blending in HDR color space and dynamic range Media pipe GPU or Display Engine SDR GFx HDR Display support (Metadata, target tone mapping, local dimming) 1 stream, HDR Codec HDR Convert GFx to HDR HDR Compose HDR Display support Display I/F HDR to HDR HDR Monitor SDR GFx Display Engine 2-stream, HDR Codec Assemble HDR Convert GFx to HDR HDR Compose HDR Display Support Display I/F HDR (2 images) HDR Monitor 7

8 Welcome to the 360-Video World! Objective Use Cases Letting users capture and interact with videos in a naturalistic and immersive way Immersive Video Conferencing, Virtual Reality Content Creation, Action Cameras, Video Blogging, Remote Home Monitoring Typical System Characteristics Ranges 3-8 USB 2.0 Cameras ⁰ HFOV of each camera ⁰ VFOV of each camera 30-60% Overlap in HFOV between two cameras 1080p Capture resolution Picture Credits: Giroptic 8

9 360-Video ingredients Intel Core i7-5557u Avg 50% CPU Load FPS Avg 60% GPU Load 9000x1000 Final Resolution (6 cameras) 9

10 The Augmented Reality Promise Professional Consumer Military Education Design/Eng. Industrial Healthcare Gaming Entertainment Sports/Fitness Shopping Aid Combat assistance Remote info, Recon., field HMD Reading extension Classroom support Enhanced education Interactive prototyping Data visualizatio n Simulation Ideation assistance and tools Remote diagnostics/ maintenanc e Workflow efficiency (e.g. machine assistance) Remote diagnostics Surgery assistance Game in real life Picture-in- picture Tracking HMD Cycling, motorcycling, skiing, etc. Interactive ads and signage Price/ availability alerts Visualization & try-on Language translation Memory assistance Directions Notification s Home repair assistance Many potential use cases for AR Leading adopters: Gaming, Entertainment, as purchase with several ancillary uses following 10

11 HARD SENSING SOFT SENSING Contextual Sensing the World Activity Location Environment Fusion Algorithms Device Usage Calendar Raw Data Feature Extraction Classification Inference Guidance X Z Y Accelerometer Std dev (x,y,z) Mean, min, max (gravity angle) Decision Tree Running Sitting Walking Light UV Shapes Orientation Acceleration Location Audio MFCC Std dev Dynamic range Bandwidth GMM Chatting Listening to music Commuting Enablers: Video Intensity gradients Contours Color Kernel Machine Bayesian Net Sparse Bundle Adjustment John is in front of me We are in a room Always-on, ultra low power sensing and I/O (camera) Effective workload partitioning based on available processing resources Context primitives & Fusion of information Picture Credits: Recon, Fitbit 11

12 Guidance Indoor example with AOS Google* Tango Support Architecture Application-level APIs Vision SW Sensor SW Camera APIs pppppppppp HAL OpenVX Processing Primitives IPU Sensors Hub Camera 1 Multi-cameras I/F Camera 2 Camera 3 CPU GPU Main Camera Indoor Guidance is a very promising domain: User- or Drone-level navigation support inside a building No GPS available (although WiFi localization is potentially present) Approaches like Google* Tango couple sensors and vision together Optimal partitioning with low power operations is key Pictures Credits: Google 12

13 Virtual Reality: Excitement & Potential are strong Professional Consumer Military Education Design/Eng. Industrial Healthcare Gaming Entertainment Sports/Fitness Shopping FPS/action Training and simulations Combat planning Virtual classrooms Virtual Field trips (Geography, History) Conferencin g & meetings Modeling/ prototype visualizatio n (e.g., architect walks inside his design) Training Best Known Methods Training Remote appointments Vehicle simulation Fantasy/RPG Indie games (Horror, puzzle, exploration) Concerts, events, news Movies, Video Virtual travel Training & simulation Virtual shopping malls Virtual Travel, Realty 13

14 Premium VR Components HMD Premium VR requirements for best immersion imply significant high performance system-level ingredients to be present. Head-mounted display system optics and refresh rate must be comfortable for the eyes (90 fps min per eye and highest pixels density possible 4K, typically). HDMI USB High quality image link needed with full color resolution (HDMI). Components addition via standard, high performance link (SS-USB). Game Simulation CPU Game Rendering Gfx 14

15 Research Challenges Efficient codec without patent war (!) New Open Media Alliance is a possible good funnel to get there. Quick add-up of new codec at 4K/8K capabilities and long battery lifetime. DRM end-to-end solutions 4K and 8K deployments are exposing Studios master material. New concept combining protected video paths and fraud tracing (forensics) on the way. Augmented Reality wide deployment Deep learning, data fusion and inference algorithms efficient implementation. Partitioning over SOC ingredients. Low power scheduling. 15

16 Call to Action Rally around key technologies to deploy these new usages Collaborate on standards (codec) to efficiently deploy 4K/8K Build partnerships to optimize HDR (playback and capture) Innovate in new technologies for faster adoption of 360 Video and VR Collaborate on new AR-based usages 16

17