Enriching and fulfilling experience to Eyes & Ears- Tizen Multimedia. Rajaram Naganur Ruchika Saxena

Transcription

1 Enriching and fulfilling experience to Eyes & Ears- Tizen Rajaram Naganur Ruchika Saxena

2 Progression Overview Framework Case Study: Camcorder Adding Experience Web Conclusion 2

3 Overview Features Open AL Pulse Audio ALSA 3

4 Overview Features Web Applications Native Applications Native APIs Web APIs Web Framework Native Framework Linux Kernel Device Drivers 4

5 Progression Overview Framework Case Study: Camcorder Adding Experience Web Conclusion 5

6 Adding Experience (APIs) API Common Usage Example State Transition Know Privileges & header file Common Usage Create & Destroy Prepare & Unprepare Start & Stop Get, Set & Unset Call Back Functions Handle These: Errors & Interruptions Information: Change in State, Data Packets 6

7 Adding Experience (APIs) Camera Recorder Player Utilities Preview & Capture Setting Applications Multi Shot Focus ISO My Secret Portion Face Detect Flash Cloud Photo Effects HDR Location Day Camera Ace Scene Theatre Effects Zoom Auto WB Exposure Watcher Instagram Dubsmash 7

8 Adding Experience (APIs) Camera Recorder Player Utilities Video Recorder Audio Recorder Applications MP4, 3GP Bit Rate Recording Motion (Slow, Fast) Resolution Orientation M4A & AMR Sample Rate Channels (Mono, Stereo) Recording Time/Size Audio Stream CB Spy Cam: AV Recorder Limit Recording Time Lapse Record 8

9 Adding Experience (APIs) Camera Recorder Player Utilities Player Music Applications Sound Type User Agent & Cookies Audio Latency Display Seek, Loop Subtitle Play Rate & Capture Sound Type Equalizer Metadata ex: Album, Genre, Album Art Sampling rate, Channels MP3, AAC, WMA, M4A, 3GA, WAV and AMR Player (Video & Music) M3U Play TED Talks MX Player Create Thumbnail Fast & Slow video Browsing 9

10 Adding Experience (APIs) Camera Recorder Player Utilities Image Video Applications Encode & Decode Rotation Crop Resize Color Space Conversion HW Acceleration Transcoding MPEG4, H264, H263 AAC, AMR NB 3GP, MP4 fps & Resolution Direct Video (HW & SW) and Audio Codec Editors AVI Photo Effects Collage Art MP3 Cutter Real-time Communication 10

11 Adding Experience (APIs) Audio Open AL Sound Manager FM Radio Audio IO Tone Player WAV Player Scan Tune Seek Up/Down Play volume control Record Raw PCM Play Raw PCM Buffer size Channels, sample size, sample rate Play DTMF Call supervisory Proprietary tones Start, Stop, Duration Play Waveform Audio file Format: WAV, OGG Play, Pause, Stop 11

12 Adding Experience (APIs) Audio Open AL Sound Manager Context Time-of-arrival difference Play 3D Audio Source2 Buffer2 Auditory Cues Source Play Set Source Doppler Effect Source1 Set Listener Sound directly goes to ear Sound passes through head on the way to ear HRTF Head Related Transfer Function Buffer1 Listener 12

13 Adding Experience (APIs) Audio Open AL Sound Manager Audio Scenarios Multiple sources, single sink Single source, multiple sinks 1. Speaker Play via speaker 2. Connect Ear-jack Play via Ear-jack? 3. Connect Bluetooth Play via BT Headphone? 4. Remove Bluetooth Headphone Play via Ear-jack? 13

14 Incoming Call End Call Adding Experience (APIs) Audio Open AL Sound Manager Audio role (ringtone) Focus change callback Request focus Acquire focus Release focus Audio Stream Focus* Collaborative approach to acquire, release, handle focus loss Audio role (media) Focus change callback Request focus Acquire focus Start playback Release focus Pause Acquire focus Resume play Volume Control Get/set output volumes Volume changed notification *Feature planned in Tizen

15 Progression Overview Framework Case Study: Camcorder Adding Experience Web Conclusion 15

16 Framework GStreamer Screen Mirror Audio Player FW Streaming FW Camera FW Recorder FW WiFi Miracast Audio FW Media Content FW Media Editing FW GStreamer Pulse Audio Plugins Plugins Muxer / Demuxer Video Audio Codecs FFMPEG ALSA Image Codec 16

17 Framework GStreamer Screen Mirror Audio Formats Supported Type Audio Codec (Decoder) List AAC, MP3, WMA7/8, AMR, LPCM Video Codec (Decoder) H.263, MPEG-4, H.264 Audio Codec (Encoder) Video Codec (Encoder) Container Format Streaming Supported AMR-NB H.263, MPEG-4 MP4, 3GP, AVI, WMV, ASF, MKV, TS MP3, AAC, AMR, AC3, WMA, OGG, WAV HTTP, HTTP Live Streaming, HTTP Progressive, RTSP 17

18 Framework GStreamer Screen Mirror Audio GStreamer GStreamer Source Plugin Generates data Plug-in Pool Pipeline Src Plug-in Filter Plug-in Sink Plug-in Filter Plugin Processing data Sink Plugin Rendering data Multiplatform Pipeline Intelligent Plugin Dynamic Load Capability Element Interfaces Pads Element s Interfaces More than 250 plugins 18

19 Framework GStreamer Screen Mirror Audio GStreamer OMX Plugin GStreamer Pipeline Src Plug-in Codec Plug-in Sink Plug-in Ready OMX GStreamer plugin Open MAX IL Core Codec OMX Component No Integration Effort to use Open MAX compliant HW codec 19

20 Framework GStreamer Screen Mirror Audio WiFi Miracast Display seamlessly multimedia content on big screens UIBC User Interface Back Channel 20

21 Framework GStreamer Screen Mirror Audio Applications Media Player System Sound Game Sound Camera Shutter Ringtone Alarm ETC Audio Core APIs Player Audio IO OpenAL Sound Manager Bluetooth FW Audiosrc / audio sink Audio FW BlueZ PulseAudio Audio HAL ALSA ALSA module Source/sink Audio HAL ALSA/Tiny ALSA PulseAudio BlueZ module Source/sink Audio plug-in BlueZ Audio IPC (Unix Domain Socket) 21

22 Framework GStreamer Screen Mirror Audio Media Player System Sound Game Sound Camera Shutter Ringtone Alarm Pulse Audio Sink Input #0 Sink Input #1 Sink Input #5 Sink Input #2 Sink Input #3 Sink Input #4 Sound Server Policy module Basic interaction between PA clients (Apps) and PA Server ALSA module Sink #0 <Default Sink> Pulse Audio Speaker Software mixing of multiple audio Plugin architecture Low & accurate latency 22

23 Framework GStreamer Screen Mirror Audio module-tizen Tizen-audio-policy Pulse Audio Extensions on Tizen* module-xxx Stream manager Device manager HAL manager Stream Manager Device Manager HAL Manager stream_map.json device_map.json Device dependent configurations *Feature planned in Tizen

24 Framework GStreamer Screen Mirror Audio Apps Sound server( PulseAudio) ALSA alsa-lib control PCM alsa-lib Unified audio interface Sound device drivers APIs Sound driver devices(/dev) Linux Kernel Sound drivers mixer playback capture PCM interface Volume control Hardware 24

25 Progression Overview Framework Case Study: Camcorder Add Experience Web Conclusion 25

26 Web Web Applications W3C/HTML 5 WebKit Media (Gstreamer Port) Device APIs Framework Features Audio & Video Element * Get User Media * Web Audio * & Web Speech * Device APIs to launch: Music Player, Video Player, Image Viewer FM Radio, Control Playback * Reference & Limitations: 3c_api/w3c_api_m.html Applications Hungama Box TV nexgtv MP3 Cutter 26

27 Progression Overview Framework Case Study: Camcorder Add Experience Web Conclusion 27

28 Case Study: Camcorder Camera Application Camera/Camcorder Framework GStreamer V4L2 Preview Pipeline Camera Src Queue Video Sink App Src Video Enc Queue Recorder Pipeline Mux File Sink ALSA PulseAudio ALSA Audio Src Audio Enc Queue 28

29 Progression Overview Framework Case Study: Camcorder Add Experience Web Conclusion 29

30 Conclusion Easy effective way of developing apps Device manufacturer friendly, Plugin based framework 30

31 Thank You