Using Mobile Processors for Cost Effective Live Video Streaming to the Internet Hans-Joachim Gelke Tobias Kammacher Institute of Embedded Systems Source: Apple Inc.
Agenda 1. Typical Application 2. Available Processors. Processor Architectures. Interfaces 5. Software Implementation 6. Practical Experiences 7. Further Applications 2
Typical Video Signal Processing Pipeline 2nd Video Source Video Input Color Space Conversion Scaling Picture in Picture h.26/h.265 Encoder Audio Audio/Video Mux Encryption Transport Protocol Packer Forward Error Correction Ethernet Output Recorder
Typical Video Signal Processing Pipeline 2nd Video Source Video Input Color Space Conversion Scaling Picture in Picture h.26/h.265 Encoder Audio Audio/Video Mux Encryption Transport Protocol Packer Forward Error Correction Ethernet Output Recorder
Typical Video Signal Processing Pipeline 2nd Video Source Video Input Color Space Conversion Scaling Picture in Picture h.26/h.265 Encoder Audio Audio/Video Mux Encryption Transport Protocol Packer Forward Error Correction Ethernet Output Recorder
Typical Video Signal Processing Pipeline 2nd Video Source Video Input Color Space Conversion Scaling Picture in Picture h.26/h.265 Encoder Audio Audio/Video Mux Encryption Transport Protocol Packer Forward Error Correction Ethernet Output Recorder
Typical Video Signal Processing Pipeline 2nd Video Source Video Input Color Space Conversion Scaling Picture in Picture h.26/h.265 Encoder Audio Audio/Video Mux Encryption Transport Protocol Packer Forward Error Correction Ethernet Output Recorder
Typical Video Signal Processing Pipeline 2nd Video Source Video Input Color Space Conversion Scaling Picture in Picture h.26/h.265 Encoder Audio Audio/Video Mux Encryption Transport Protocol Packer Forward Error Correction Ethernet Output Recorder
Typical Video Signal Processing Pipeline 2nd Video Source Video Input Color Space Conversion Scaling Picture in Picture h.26/h.265 Encoder Audio Audio/Video Mux Encryption Transport Protocol Packer Forward Error Correction Ethernet Output Recorder
Typical Video Signal Processing Pipeline 2nd Video Source Video Input Color Space Conversion Scaling Picture in Picture h.26/h.265 Encoder Audio Audio/Video Mux Encryption Transport Protocol Packer Forward Error Correction Ethernet Output Recorder
Typical Video Signal Processing Pipeline 2nd Video Source Video Input Color Space Conversion Scaling Picture in Picture h.26/h.265 Encoder Audio Audio/Video Mux Encryption Transport Protocol Packer Forward Error Correction Ethernet Output Recorder
Typical Video Signal Processing Pipeline 2nd Video Source Video Input Color Space Conversion Scaling Picture in Picture h.26/h.265 Encoder Audio Audio/Video Mux Encryption Transport Protocol Packer Forward Error Correction Ethernet Output Recorder
How is a Mobile Processor Defined? Multi Core s Audio and Video Processing Units Camera Graphics Processing Display DSP Connectivity Source block diagram: Qualcomm Positioning Systems Copyright Time Warner
How is a Mobile Processor Defined? Multi Core s Audio and Video Processing Units Camera Graphics Processing Display DSP Connectivity Source block diagram: Qualcomm Positioning Systems Copyright Time Warner
How is a Mobile Processor Defined? Multi Core s Audio and Video Processing Units Camera Graphics Processing Display DSP Connectivity Source block diagram: Qualcomm Positioning Systems Copyright Time Warner
How is a Mobile Processor Defined? Multi Core s Audio and Video Processing Units Camera Graphics Processing Display DSP Connectivity Source block diagram: Qualcomm Positioning Systems Copyright Time Warner
How is a Mobile Processor Defined? Multi Core s Audio and Video Processing Units Camera Graphics Processing Display DSP Connectivity Source block diagram: Qualcomm Positioning Systems Copyright Time Warner
How is a Mobile Processor Defined? Multi Core s Audio and Video Processing Units Camera Graphics Processing Display DSP Connectivity Source block diagram: Qualcomm Positioning Systems Copyright Time Warner
How is a Mobile Processor Defined? Multi Core s Audio and Video Processing Units Camera Graphics Processing Display DSP Connectivity Source block diagram: Qualcomm Positioning Systems Copyright Time Warner
How is a Mobile Processor Defined? Multi Core s Audio and Video Processing Units Camera Graphics Processing Display DSP Connectivity Source block diagram: Qualcomm Positioning Systems Copyright Time Warner
How is a Mobile Processor Defined? Multi Core s Audio and Video Processing Units Camera Graphics Processing Display DSP Connectivity Source block diagram: Qualcomm Positioning Systems Copyright Time Warner
How is a Mobile Processor Defined? Multi Core s Audio and Video Processing Units Camera Graphics Processing Display DSP Connectivity Source block diagram: Qualcomm Positioning Systems Copyright Time Warner
How is a Mobile Processor Defined? Multi Core s Audio and Video Processing Units Camera Graphics Processing Display DSP Connectivity Source block diagram: Qualcomm Positioning Systems Copyright Time Warner
How is a Mobile Processor Defined? Multi Core s Audio and Video Processing Units Camera Graphics Processing Display DSP Connectivity Source block diagram: Qualcomm Positioning Systems Copyright Time Warner
How is a Mobile Processor Defined? Multi Core s Audio and Video Processing Units Camera Graphics Processing Display DSP Connectivity Source block diagram: Qualcomm Positioning Systems Copyright Time Warner
How is a Mobile Processor Defined? Multi Core s Audio and Video Processing Units Camera Graphics Processing Display DSP Connectivity Source block diagram: Qualcomm Positioning Systems Copyright Time Warner
How is a Mobile Processor Defined? Multi Core s Audio and Video Processing Units Camera Graphics Processing Display DSP Connectivity Source block diagram: Qualcomm Positioning Systems Copyright Time Warner
How is a Mobile Processor Defined? Multi Core s Audio and Video Processing Units Camera Graphics Processing Display DSP Connectivity Source block diagram: Qualcomm Positioning Systems Copyright Time Warner
How is a Mobile Processor Defined? Multi Core s Audio and Video Processing Units Camera Graphics Processing Display DSP Connectivity Source block diagram: Qualcomm Positioning Systems Copyright Time Warner
The Players Snapdragon 810 TK1 and TX1 X7-Z8700 Sources: Qualcomm, Nvidia, Intel, Samsung, Apple, Allwinner 5
The Players Snapdragon 810 TK1 and TX1 X7-Z8700 Exynos Apple A8 Sources: Qualcomm, Nvidia, Intel, Samsung, Apple, Allwinner 5
The Players Snapdragon 810 TK1 and TX1 X7-Z8700 Exynos Apple A8 Allwinner A1 Sources: Qualcomm, Nvidia, Intel, Samsung, Apple, Allwinner RK288 5
Hardware Resources 6-bit ARM A57 2MB L2-Cache 1 2 6-bit ARM A5 512kB L2-Cache 1 2 GPU Video Encoder Video Decoder Video Interface Video Processor 6
Hardware Resources 6-bit ARM A57 2MB L2-Cache 1 2 Operating Mode 6-bit ARM A5 512kB L2-Cache 1 2 GPU Video Encoder Video Decoder Video Interface Video Processor 6
Hardware Resources 6-bit ARM A57 2MB L2-Cache 1 2 6-bit ARM A5 512kB L2-Cache 1 2 GPU Video Encoder Video Decoder Video Interface Video Processor 6
Hardware Resources 6-bit ARM A57 2MB L2-Cache 1 2 6-bit ARM A5 512kB L2-Cache 1 Sleep 2 Mode GPU Video Encoder Video Decoder Video Interface Video Processor 6
Hardware Resources 6-bit ARM A57 2MB L2-Cache 1 2 6-bit ARM A5 512kB L2-Cache 1 2 GPU Video Encoder Video Decoder Video Interface Video Processor 6
Hardware Resources 6-bit ARM A57 2MB L2-Cache 1 2 6-bit ARM A5 512kB L2-Cache 1 2 GPU Video Encoder 192/ 256 Core GPU OpenGL, CUDA, DX12, AEP, OpenCL Video Interface Video Decoder Video Processor 6
Hardware Resources 6-bit ARM A57 2MB L2-Cache 1 2 6-bit ARM A5 512kB L2-Cache 1 2 GPU Video Encoder Video Decoder Video Interface Video Processor 6
Hardware Resources 6-bit ARM A57 2MB L2-Cache 1 2 6-bit ARM A5 512kB L2-Cache 1 2 H.26/AVC & H.265/HEVC 096 2160 pix/0 Hz GPU Video Encoder Video Decoder Video Interface Video Processor 6
Hardware Resources 6-bit ARM A57 2MB L2-Cache 1 2 6-bit ARM A5 512kB L2-Cache 1 2 GPU Video Encoder Video Decoder Video Interface Video Processor 6
Hardware Resources 6-bit ARM A57 2MB L2-Cache 1 2 6-bit ARM A5 512kB L2-Cache 1 2 GPU H.26/AVC & H.265/HEVC 096 2160 pix/60 Hz/10 bit Video Encoder Video Decoder Video Interface Video Processor 6
Hardware Resources 6-bit ARM A57 2MB L2-Cache 1 2 6-bit ARM A5 512kB L2-Cache 1 2 GPU Video Encoder Video Decoder Video Interface Video Processor 6
Hardware Resources 6-bit ARM A57 2MB L2-Cache 1 2 HDMI 2.0 096 2160pix, Display Serial Interface (DSI) Video Encoder GPU 6-bit ARM A5 512kB L2-Cache 1 2 Video Decoder Video Interface Video Processor 6
Hardware Resources 6-bit ARM A57 2MB L2-Cache 1 2 6-bit ARM A5 512kB L2-Cache 1 2 GPU Video Encoder Video Decoder Video Interface Video Processor 6
Hardware Resources 6-bit ARM A57 2MB L2-Cache 1 2 6-bit ARM A5 512kB L2-Cache 1 2 GPU Scaling etc. Video Encoder Video Decoder Video Interface Video Processor 6
Hardware Resources 6-bit ARM A57 2MB L2-Cache 1 2 6-bit ARM A5 512kB L2-Cache 1 2 GPU Video Encoder Video Decoder Video Interface Video Processor 6
Mobile Processor Peripherals CSI DSI I2S I2S SDIO PCIe 7
Processors available on industrial modules Nvidia TK1 module Snapdragon 810 module Allwinner A1 module Sources: Avionic Design, Intrinisic, Theobroma Systems 8
GStreamer: A Multimedia Framework Generic Multimedia Processing Device Mobile Processor Images: Public Domain 9
GStreamer: A Multimedia Framework Generic Multimedia Processing Device 2nd Video Source Video Input Color Space Conversion Mobile Processor Picture Scaling in Picture H.26/H.265 Encoder Audio Audio/Video Mux Encryption Transport Protocol Packer Forward Error Correction Ethernet Output Recorder 9 Images: Public Domain
GStreamer: A Multimedia Framework Generic Multimedia Processing Device Mobile Processor Video Processing Images: Public Domain 9
GStreamer: A Multimedia Framework Generic Multimedia Processing Device Mobile Processor Video Processing Network Images: Public Domain 9
GStreamer: A Multimedia Framework Generic Multimedia Processing Device Mobile Processor Video Processing Network Network Images: Public Domain 9
GStreamer: A Multimedia Framework Generic Multimedia Processing Device Mobile Processor Linux Kernel GStreamer Video Processing Network Network Images: Public Domain 9
GStreamer Internals Gstreamer Core Framework 10
GStreamer Internals Sources: File Camera TCP/UDP Gstreamer Core Framework 10
GStreamer Internals Sources: File Camera TCP/UDP Gstreamer Core Framework Sinks: File Display TCP/UDP 10
GStreamer Internals Containers: Audio & Video MP / MKV Transport- Protocols: HTTP/RTSP Filters: Converters Mixers Sources: File Camera TCP/UDP Gstreamer Core Framework Sinks: File Display TCP/UDP 10
GStreamer Internals Containers: Audio & Video MP / MKV Transport- Protocols: HTTP/RTSP Filters: Converters Mixers Sources: File Camera TCP/UDP Gstreamer Core Framework Encoders: Audio (MP / AAC) Video (H.26/H.265) HW-accelerated Sinks: File Display TCP/UDP 10
GStreamer Internals Containers: Audio & Video MP / MKV Transport- Protocols: HTTP/RTSP Filters: Converters Mixers Sources: File Camera TCP/UDP Gstreamer Core Framework Sinks: File Display TCP/UDP Encoders: Audio (MP / AAC) Video (H.26/H.265) HW-accelerated Decoders: Audio (MP / AAC) Video (H.26/H.265) HW-accelerated 10
GStreamer Internals Containers: Audio & Video MP / MKV Transport- Protocols: HTTP/RTSP Filters: Converters Mixers Sources: File Camera TCP/UDP Gstreamer Core Framework Sinks: File Display TCP/UDP Encoders: Audio (MP / AAC) Video (H.26/H.265) HW-accelerated Decoders: Audio (MP / AAC) Video (H.26/H.265) HW-accelerated OpenMAX OS Kernel 10
GStreamer Internals Containers: Audio & Video MP / MKV Transport- Protocols: HTTP/RTSP Filters: Converters Mixers Sources: File Camera TCP/UDP Gstreamer Core Framework Sinks: File Display TCP/UDP Encoders: Audio (MP / AAC) Video (H.26/H.265) HW-accelerated Decoders: Audio (MP / AAC) Video (H.26/H.265) HW-accelerated Access to HWaccelerated CODECs OpenMAX OS Kernel 10
OpenMAX (Open Media Acceleration) Multimedia Application GStreamer Core Software User Space Gstreamer Plugins OpenMAX (OMX) Libraries Bellagio (OpenMAX IL) Software Kernel Space Linux Kernel Modules / Drivers Video DMA Mobile Processor HW accelerated Video Encoder HW accelerated Video Decoder 11
Macro Block vs. Quad Tree Procedure H.26 H.265 Macro Block Procedure Quad Tree Procedure Source: London Bus Museum 12
Comparison between H.26 and H.265 Video Quality PSNR vs. Bitrate for H.26 and H.265 1
Comparison between H.26 and H.265 Video Quality PSNR vs. Bitrate for H.26 and H.265 H.265 11Mbps H.26 19Mbps 1
Comparison between H.26 and H.265 Video Quality PSNR vs. Bitrate for H.26 and H.265 H.265 0Mbps H.26 67Mbps H.265 11Mbps H.26 19Mbps 1
Your Mobile Processor Application? Video Conferencing Android TV Source: Google Mobile Processor Medical Imaging Lecture recording streaming Source: PMK Driving Assistance Source: Google 1