Multimedia Playback & Streaming Shadab Rashid Jam 16 September 28 th, 2012
What are you interested in? Making multimedia apps for Consuming Audio/Video Dealing with content providers, looking for An application/client solution A compatible platform DRM options Protocols & formats supported 2
What we will cover Multimedia API capabilities Overview and usage C/C++, Qt/QML/Cascades Formats & Streaming protocols supported Digital Rights Management Sample applications & some code 3
Multimedia API Overview 4
Core API capabilities Audio/Video playback: Local or remote resource (streaming) For Video: Use windows within user interface UI Controls (buttons, progress slider bar.) Playlist control (next track, previous track ) Metadata query and setting (artist, album info ) Audio/video routing: Audio via headphones, speakers, TV etc Second screen via HDMI 5
Core API capabilities (continued) Audio volume control: Through device buttons or API Now Playing Area: Set artist, album, get next track/previous track commands Invoke core media previewer (or any other ones) In previewer mode or the app mode Register to be a media previewer yourself 6
Native SDK Context C++/Qt (QML) - Cascades API Wrapper classes Easy to use with Cascades UI elements C API mm-renderer BlackBerry Platform Services (BPS) 7
Multimedia API C++/Qt (QML) Cascades
Some core classes bb/multimedia/*.hpp MediaPlayer NowPlayingConnection AudioRecorder SystemSound And other classes for enum types. 9
Let s see some code Play - Pause functionality Using QML/Cascades And using C++/Qt To the Momentics IDE Attribution:http://www.flickr.com/photos/zooboing/4361038612/sizes/o/in/photostream/ 10
MediaPlayer setup (QML) 11
MediaPlayer control (QML) 12
MediaPlayer setup - C++/Qt 13
Event Handling C++/Qt 14
Concurrent streams You can play numerous streams at the same time Create multiple instances of MediaPlayer A sample app: RockMediaSample 15
Video Playback Just create a ForeignWindowControl Attach the window to the MediaPlayer Scaling, resizing, repositioning Great performance Smooth and responsive on interaction 16
VideoPlayerSample Another sample app! Beta2 version - already public on: github.com/blackberry/cascades-community-samples Full Link: https://github.com/blackberry/cascades-community-samples/tree/master/videoplayersample
Creating the ForeignWindowControl 18
Attaching the windowid 19
Multimedia API C API Native SDK
mm-renderer overview API steps: Configure connect & create context Attach audio and video output Attach media URI Play! 21
Code Snippets 22
Event handling C API Done through BlackBerry Platform Services (BPS). Playback state, buffer state, position update Network status Errors 23
C API some relevant header files mm/renderer.h Initialize contexts & start playback bps/mmrenderer.h Monitor a specific context (query playback states, position, errors, etc) bps/mediacontroller.h & bps/mediaplayer.h 2 controllers on the device: Buttons & Now Playing service Connect to Now Playing service to determine what is playing (meta data, duration, position) Request & control playback for the active media player on the system Setting metadata 24
C++/Qt with C BPS API Qt Signals 25
Audio routing Use Audio Manager API Header files at: audio/*.h Direct sounds to other devices: Headphones Speakers HDMI TV and some more Attribution:http://www.iconspedia.com/icon/black-tv-18924.html 26
Apps heavy with audio effects? OpenAl Comes bundled with our SDK Possible choice for game developers Has more granular & low-level audio functionalities Can be used for cross platform solutions IDE even has a sample app! 27
Formats Streaming protocols DRM
Formats Video + Audio Video Audio Image 3GP, 3GP2 M4A M4V MOV MP4 MPEG-4 MKV MP2PS, MP2TS AVI ASF, WMV, WMA AVI F4V MP3 MKA AAC AWB AMR WAV OGG (new) BMP, WBMP JPG GIF PNG TIFF (new) SGI TGA 29
Codecs Video H.263, H.264 MPEG-2, MPEG-4 MJPEG Xvid VC-1 VP6 Sorenson Spark Audio AAC AC-3 (new: Dolby Digital plus) MP3 PCM AMR-WB AMR-NB QCELP FLAC (new) Windows Media Audio (9, 10, 10,Professional, Lossless) Vorbis FLAC PCM GSM6.10 G729 G722 30
Recommended Formats File format Component Codec Local: MP4 Video H.264 For remote streaming: Any supported containers/streams Audio AAC BlackBerry 10 will support all the formats that PlayBook supports and more 31
Supported Streaming Protocols HTTP HTTPS HTTP Progressive Download RTSP HLS v3 Http Live Streaming 32
HTTP Live Streaming Supports live data (unbounded multimedia stream) Of course, prerecorded content too Client intelligently switches streams (bit rates) Based on bandwidth availability Allows encryption of media over HTTPS Custom DRM schemes for content providers 33
Digital Rights Management Mainly related with content providers & studios 1) HLS with 128 AES 2) 3 rd party DRM providers 3) Microsoft PlayReady 4) Adobe Flash Access (FAXS) web only 34
1) HLS with 128 AES (DRM continued) On our non-rooted platform, it is impossible for one app to gain access to the protected content buffers of another Internally, it is safe; externally, clear text AES keys are being transmitted in M3U files. If your app can protect the AES keys, then you can use this approach More details on the platform s security with NDA (for studios, content providers etc) The security team can be part of the engagement as well 35
2) 3 rd party DRM providers RIM has active engagements with some 3 rd party DRM providers for supporting BlackBerry 10 for product launch RIM can link content providers to these companies They will also entertain introductions to other similar companies (that studios might know or prefer) A port to a new platform could take 6-9 months for these companies Sooner the process starts, the better 36
3) Microsoft PlayReady The client solution would be using RIM s keys on the device Hence, legal partnership would be required Limited to Progressive Download protocol 37
4) Adobe Flash Access Quick solution (for web only) Re-using your web infrastructure in the app 38
Other formats/protocols? Often use services from other companies Local Proxy server Decrypt and transform to HLS (or any other supported formats) Multimedia API Application 39
Integration Capabilities Experience
Media Previewer Just want to play a file? Invoke the Media Previewer! Register your app to be a previewer! Quickly, to the Dev Alpha B! 41
Unbound invocation (Media Previewer) Just use the URI 42
Bound invocation (Media Previewer) Set the target as well as the URI For setting the app mode specifically, use: sys.mediaplayer.previewer.app 43
Leverage other apps Share links, playlists, artist/album with others BBM, Twitter, Facebook Calendar events, Universal Search Notifications + Universal Inbox 44
Leverage other devices Second screen via HDMI Other libraries for wireless communication DLNA, UPnP 45
Don t forget about the UX! User Experience guidelines Navigation Pane Overflow Menu Context Menu Peek, Swipe, Gestures Many more. 46
Let s look at an actual app Nobex Radio (beta) I swear this is the last switch to HDMI! 47
Ending thoughts.. Dive into the IDE and sample apps! Documentation sites, header files for multimedia API Other relevant sessions: Native API Camera JAM 15, 9:30 AM Web API Multimedia JAM 34, 9:30 AM Invocation Framework JAM 20, 3:30 PM BBM JAM 47 & 49, 9:30 AM & 1:00 PM PIM API JAM 18, 10:45 AM Questions? 48 You can drop by the Ask the expert sessions: Roundtables: 6-7 PM One-on-One: Wed 2 3 PM Drop In: Thu 12 to 1 PM
Live free & code hard! Shadab Rashid @surashid shrashid@rim.com Atttribution:http://www.flickr.com/photos/lorenjavier/5686291142/ 49