Multimedia Playback & Streaming

Transcription

1 Multimedia Playback & Streaming Shadab Rashid Jam 16 September 28 th, 2012

2 What are you interested in? Making multimedia apps for Consuming Audio/Video Dealing with content providers, looking for An application/client solution A compatible platform DRM options Protocols & formats supported 2

3 What we will cover Multimedia API capabilities Overview and usage C/C++, Qt/QML/Cascades Formats & Streaming protocols supported Digital Rights Management Sample applications & some code 3

4 Multimedia API Overview 4

5 Core API capabilities Audio/Video playback: Local or remote resource (streaming) For Video: Use windows within user interface UI Controls (buttons, progress slider bar.) Playlist control (next track, previous track ) Metadata query and setting (artist, album info ) Audio/video routing: Audio via headphones, speakers, TV etc Second screen via HDMI 5

6 Core API capabilities (continued) Audio volume control: Through device buttons or API Now Playing Area: Set artist, album, get next track/previous track commands Invoke core media previewer (or any other ones) In previewer mode or the app mode Register to be a media previewer yourself 6

7 Native SDK Context C++/Qt (QML) - Cascades API Wrapper classes Easy to use with Cascades UI elements C API mm-renderer BlackBerry Platform Services (BPS) 7

8 Multimedia API C++/Qt (QML) Cascades

9 Some core classes bb/multimedia/*.hpp MediaPlayer NowPlayingConnection AudioRecorder SystemSound And other classes for enum types. 9

10 Let s see some code Play - Pause functionality Using QML/Cascades And using C++/Qt To the Momentics IDE Attribution: 10

11 MediaPlayer setup (QML) 11

12 MediaPlayer control (QML) 12

13 MediaPlayer setup - C++/Qt 13

14 Event Handling C++/Qt 14

15 Concurrent streams You can play numerous streams at the same time Create multiple instances of MediaPlayer A sample app: RockMediaSample 15

16 Video Playback Just create a ForeignWindowControl Attach the window to the MediaPlayer Scaling, resizing, repositioning Great performance Smooth and responsive on interaction 16

17 VideoPlayerSample Another sample app! Beta2 version - already public on: github.com/blackberry/cascades-community-samples Full Link:

18 Creating the ForeignWindowControl 18

19 Attaching the windowid 19

20 Multimedia API C API Native SDK

21 mm-renderer overview API steps: Configure connect & create context Attach audio and video output Attach media URI Play! 21

22 Code Snippets 22

23 Event handling C API Done through BlackBerry Platform Services (BPS). Playback state, buffer state, position update Network status Errors 23

24 C API some relevant header files mm/renderer.h Initialize contexts & start playback bps/mmrenderer.h Monitor a specific context (query playback states, position, errors, etc) bps/mediacontroller.h & bps/mediaplayer.h 2 controllers on the device: Buttons & Now Playing service Connect to Now Playing service to determine what is playing (meta data, duration, position) Request & control playback for the active media player on the system Setting metadata 24

25 C++/Qt with C BPS API Qt Signals 25

26 Audio routing Use Audio Manager API Header files at: audio/*.h Direct sounds to other devices: Headphones Speakers HDMI TV and some more Attribution: 26

27 Apps heavy with audio effects? OpenAl Comes bundled with our SDK Possible choice for game developers Has more granular & low-level audio functionalities Can be used for cross platform solutions IDE even has a sample app! 27

28 Formats Streaming protocols DRM

29 Formats Video + Audio Video Audio Image 3GP, 3GP2 M4A M4V MOV MP4 MPEG-4 MKV MP2PS, MP2TS AVI ASF, WMV, WMA AVI F4V MP3 MKA AAC AWB AMR WAV OGG (new) BMP, WBMP JPG GIF PNG TIFF (new) SGI TGA 29

30 Codecs Video H.263, H.264 MPEG-2, MPEG-4 MJPEG Xvid VC-1 VP6 Sorenson Spark Audio AAC AC-3 (new: Dolby Digital plus) MP3 PCM AMR-WB AMR-NB QCELP FLAC (new) Windows Media Audio (9, 10, 10,Professional, Lossless) Vorbis FLAC PCM GSM6.10 G729 G722 30

31 Recommended Formats File format Component Codec Local: MP4 Video H.264 For remote streaming: Any supported containers/streams Audio AAC BlackBerry 10 will support all the formats that PlayBook supports and more 31

32 Supported Streaming Protocols HTTP HTTPS HTTP Progressive Download RTSP HLS v3 Http Live Streaming 32

33 HTTP Live Streaming Supports live data (unbounded multimedia stream) Of course, prerecorded content too Client intelligently switches streams (bit rates) Based on bandwidth availability Allows encryption of media over HTTPS Custom DRM schemes for content providers 33

34 Digital Rights Management Mainly related with content providers & studios 1) HLS with 128 AES 2) 3 rd party DRM providers 3) Microsoft PlayReady 4) Adobe Flash Access (FAXS) web only 34

35 1) HLS with 128 AES (DRM continued) On our non-rooted platform, it is impossible for one app to gain access to the protected content buffers of another Internally, it is safe; externally, clear text AES keys are being transmitted in M3U files. If your app can protect the AES keys, then you can use this approach More details on the platform s security with NDA (for studios, content providers etc) The security team can be part of the engagement as well 35

36 2) 3 rd party DRM providers RIM has active engagements with some 3 rd party DRM providers for supporting BlackBerry 10 for product launch RIM can link content providers to these companies They will also entertain introductions to other similar companies (that studios might know or prefer) A port to a new platform could take 6-9 months for these companies Sooner the process starts, the better 36

37 3) Microsoft PlayReady The client solution would be using RIM s keys on the device Hence, legal partnership would be required Limited to Progressive Download protocol 37

38 4) Adobe Flash Access Quick solution (for web only) Re-using your web infrastructure in the app 38

39 Other formats/protocols? Often use services from other companies Local Proxy server Decrypt and transform to HLS (or any other supported formats) Multimedia API Application 39

40 Integration Capabilities Experience

41 Media Previewer Just want to play a file? Invoke the Media Previewer! Register your app to be a previewer! Quickly, to the Dev Alpha B! 41

42 Unbound invocation (Media Previewer) Just use the URI 42

43 Bound invocation (Media Previewer) Set the target as well as the URI For setting the app mode specifically, use: sys.mediaplayer.previewer.app 43

44 Leverage other apps Share links, playlists, artist/album with others BBM, Twitter, Facebook Calendar events, Universal Search Notifications + Universal Inbox 44

45 Leverage other devices Second screen via HDMI Other libraries for wireless communication DLNA, UPnP 45

46 Don t forget about the UX! User Experience guidelines Navigation Pane Overflow Menu Context Menu Peek, Swipe, Gestures Many more. 46

47 Let s look at an actual app Nobex Radio (beta) I swear this is the last switch to HDMI! 47

48 Ending thoughts.. Dive into the IDE and sample apps! Documentation sites, header files for multimedia API Other relevant sessions: Native API Camera JAM 15, 9:30 AM Web API Multimedia JAM 34, 9:30 AM Invocation Framework JAM 20, 3:30 PM BBM JAM 47 & 49, 9:30 AM & 1:00 PM PIM API JAM 18, 10:45 AM Questions? 48 You can drop by the Ask the expert sessions: Roundtables: 6-7 PM One-on-One: Wed 2 3 PM Drop In: Thu 12 to 1 PM

49 Live free & code hard! Shadab Atttribution: 49