Delivering live multicam content to smart devices through cloud platforms. Werner Ramaekers EVS - speaker Johann Schreurs EVS Maher Khatib - EVS

Transcription

1 Delivering live multicam content to smart devices through cloud platforms Werner Ramaekers EVS - speaker Johann Schreurs EVS Maher Khatib - EVS

2 Most people in the tech community sit in the basement learning Klingon and inventing the future of television and quite frankly we re not interested. Nigel Walley, managing director Decipher Connected TV debate at IBC 2012 in Amsterdam

3 Multi-screen media consumption Source : Google/Ipsos/Sterling, 2012

4 Modes of multi-screening Source :

5 Live sports broadcast production : A lot of content is left unused

6 Use available content for complementary multi-screening

7 Multi-camera plan for live event 33 cameras

8 Adaptive bitrate streaming for live TV HD-SDI Different delay wrt live event Low bitrate encoder In the truck or at the broadcaster Medium bitrate High bitrate HTTP IPTV

9 Adaptive bitrate streaming for live TV with multiple camera angles HD-SDI encoder encoder encoder encoder Low bitrate Medium bitrate High bitrate encoder encoder encoder encoder Doesn t scale for multiple camera angles All encoders must be in the production truck. Requires a lot of extra bandwidth at the truck.

10 Constraints & challenges No extra broadcast resources required

11 Constraints & challenges Live delivery of complimentary content On TV Connected screen

12 Constraints & challenges Content rights protection

13 Constraints & challenges Scalable distribution to end-users

14 Constraints & challenges No extra broadcast resources required Or as little as possible Live delivery of content Or as fast as possible Content protection Scalable distribution to end-users

15 delivering multi-camera replays to smart devices

16 Platform for delivering multi-camera replays to smart devices

17 Example production workflow Create event Extract clip for each camera angle Transfer each clip to Central Approve & enrich each clip Transcode each clip to ABR Publish <120 seconds

18 Workspace : approve & enrich clip

19 Transcoding : cloud services Multiple cameras Adaptive Bitrate : multiple bitrates

20 Transcoding : how to select bitrates for HLS HLS player behaviour on ios 3 bitrates available for a sample clip Bandwidth available > 800 Kbit/s 1 2 3

21 Transcoding : Content protection in HLS

22 Platform for delivering multi-camera replays to smart devices

23 CCast deployed the platform for a soccer tournament 38 Production cameras 22 cameras made available

24 Availability of content to end users Clean : 6x Real time (1min. for 10sec. Clip) 1st camera : 11x real time 1st slow-motion camera : 15x real time All 22 cams : 20x real time

25 CCast deployed the platform for a soccer competition

26 Average time / visit? minutes Audience sample : end-users

27 Opportunities provided by the platform Integration with social networks, statistics services, archives, Non-linear tv consumption Content repurposing Highlights, recaps, goals,..

28 Q & A

29 Platform for delivering multi-camera replays to smart devices

30 SMPTE Meeting Presentation Delivering multi-camera content to smart devices through cloud platforms Werner Ramaekers, Product Development Manager EVS, 16 Rue Bois St Jean, 4102 Seraing, Belgium, Johann Schreurs, General Manager New Media Broadcast EVS, 16 Rue Bois St Jean, 4102 Seraing, Belgium, Maher Khatib, EVP Products Architecture and Technology EVS,16 Rue Bois St Jean, 4102 Seraing, Belgium, Written for presentation at the 2012 SMPTE Annual Technical Conference Abstract. "Broadcasters must engage a new generation of multitasking viewers who no longer sit passively in front of their television sets but browse the internet and interact with social media while watching TV. Rather than risk losing viewers, broadcasters can provide original premium content including unseen camera angles and highlights to viewers via second screen devices. The large amount of unused content that sits on live TV production servers can be used to enrich the user experience and maximize the value of already available content. This paper will explore technology challenges in building open and scalable platforms to deliver high quality experiences on second screen devices, including: Best practices in building near-live multi-camera replay platforms on top of standard live production environments Overcoming challenges in cloud-based production and delivery to multiple screens Integration with social networks, archives, stats and other third-party content. Keywords. Second screen, multi-screen, mobile video, cloud, file-based workflow, HLS, transcoding. (The SMPTE disclaimer is on a footer on this page, and will show in Print Preview or Page Layout view.) The authors are solely responsible for the content of this technical presentation. The technical presentation does not necessarily reflect the official position of the Society of Motion Picture and Television Engineers (SMPTE), and its printing and distribution does not constitute an endorsement of views which may be expressed. This technical presentation is subject to a formal peer-review process by the SMPTE Board of Editors, upon completion of the conference. Citation of this work should state that it is a SMPTE meeting paper. EXAMPLE: Author's Last Name, Initials Title of Presentation, Meeting name and location.: SMPTE. For information about securing permission to reprint or reproduce a technical presentation, please contact SMPTE at jwelch@smpte.org or (3 Barker Ave., White Plains, NY 10601).

31 Introduction New Challenges for Traditional Broadcasters: Multitasking and Social Interaction Consumers clearly watch TV to keep in touch with current events, watch highlights or stay on top of entertainment news. But over the years habits have changed: viewers want to stay permanently connected to the world. In effect, they re consuming lots of video by choosing what they want to see and at which precise moment. For example, YouTube now provides 4 billion streams a day 1 and 92% of viewers 2 say they watch video over the internet. At the same time, they re seeking information about the subjects they follow ( Where did I see that actor? or What was the result of the previous match? or Who s performing the original soundtrack? ). And, like never before, they re interacting with social networks about what they re watching ( I want to buy that soundtrack, I like it, I vote for candidate 1, This is a photo of me watching the event ). Twitter saw more than 150 million tweets about the Olympics during London , with an astounding 80,000 tweets per minute published after Usain Bolt won the gold medal in the 200m final. Also, some Olympic athletes increased their Facebook fan numbers by about 4000% during the Games 4. As viewers use a remote control to flip channels, they re also increasingly searching for content through wireless devices such as tablets and iphones to stay informed. 20% of consumers are willing to pay more to be able to access services across multiple devices 5. This new trend poses a challenge for broadcasters who wish to attract more eyeballs towards the television set and away from connected devices which provide a variety of data. They have to face new challenges like How to keep viewers watching my channel during commercials or How to keep viewers attention while the match is at a standstill. It s not only about preventing the audience from switching to other networks, but also answering their needs by offering interesting content which they can share and comment on. Maximizing recorded content affordably to create new revenue streams One sure way to keep viewers tuned in is to offer quality content as well as enriching the viewing experience. Broadcasters need to avoid thinking in terms of reaching the lowest common denominator when it comes to subject matter. The goal should be to focus on the best possible programming which can engage the majority of the audience. Broadcasters have to anticipate needs and questions, and innovate by giving viewers original extra content while they re watching the main program on TV. This additional content should of course generate significant new revenues. These could come from the huge internet video market: 60 hours of video are uploaded every minute and 3 billion videos are monetized every week 6. According to an Accenture survey, 23% of viewers pay a subscription to access video over the internet, while 11% pay through a pay-per-view mechanism 7. Providing interesting content in relation to live events must generate revenues through a new way of delivering advertising on connected devices, additional fees or new ways to stay in touch with the community: brand loyalty, social communities, online shopping or paid for games. And, of course, to offer this type of service it needs to be affordable. Additional costs and resources should be kept to a minimum. The cost for traditional broadcasters to put events on air are already high so any new content should offer a high return on investment rather than create additional costs. Traditional TV usually offers one video at a time per screen which 2

32 means that content recorded by all the other cameras and other equipment at the event aren t providing value at that moment, making the return on these investments very low. Ideally, the proposed content should come from existing production infrastructure, at least in part, to maximize the investment. Although most recorded content is unseen on the main screen, it often contains interesting moments which can be monetized. This paper proposes a platform for allowing the creation and distribution of content that adds to the live TV viewer experience. The proposed platform will enable access to content that wasn t chosen to be part of the final program (PGM) transmitted to TV screens due to of scarcity of resources or time. The platform will allow the creation of third-party applications for connected screens that augment the experience of the end user. Production environments for live sports events Traditional television broadcast technology limits Capturing and broadcasting a live sports event from a venue involves a lot of technology challenges and components that work together in real-time to bring the event onto the television screen in the living room. Viewers are excited by the action during the game and want to get the best view of it. If we take the example of a live soccer game, the action will be captured by multiple cameras around the field. The director creating the game s program which is broadcast on air will switch from camera to camera, from live to replay, during the game to give viewers the best angle on what s happening on the field. The number of cameras capturing the live action varies between 4 and 22, or even more for high profile events like World Championships 8 (see examples in Figure 1). Fig. 1: camera plans for different sports 3

33 All the cameras are connected to video servers that continuously record the action and are timesynchronized. This allows the director to enrich the live program with replays of the most thrilling or remarkable actions that happen on the field. Obviously, while a replay is being played out from one of the video servers, the acquisition of the incoming video from the cameras by this video server must not be interrupted. Except for tournaments, a venue will usually host one game per week or fortnight. The production environment is therefore installed in an Outside Broadcast or mobile production truck so that it can be reused cost-efficiently for games at multiple venues. During the live broadcast, the video signal from the director s desk is sent to a playout server which will often use a satellite link to get the video stream into the broadcast infrastructure for delivery to the living room. Television broadcast technology limits the director to one channel. For an event where 12 cameras capture the live event - and video servers are recording all 12 cameras - only one camera actually makes it into the live broadcast at any given time unless a video effect (DVE) is applied like picture-in-picture, split screen, or other. This means that in the example of an event captured live by 12 cameras less than 10% of the content being recorded is being used to generate value. Because of the live nature of the event being broadcast and the short duration of transitions during the game, the director has to limit the number of replays he puts on air during the live broadcast. This means that the available content is under-utilized and a lot of value remains untapped. Adding value from unseen content One way of enriching the live broadcast user experience is to give viewers access to highlights or remarkable events from every available camera angle. The video for each camera angle is already recorded continuously by the servers in a time-synchronized way with all the other cameras capturing the live event. For the best user experience, the replay needs to be available as near to the live event happening at the venue as possible so that it s still fresh in the minds of viewers. Challenges at the live production side Broadcast infrastructure with a satellite link isn t a suitable option for making extra content available to enrich live coverage. However the technology provided by the internet is a good match for the on-demand interaction pattern of this enrichment. Since we re looking at a way to enrich the experience of watching a live game on television, it should be no surprise that the number of users using a connected screen application to access this enrichment will initially be a lot smaller than the number of viewers watching the television broadcast. This means that the cost of the resources used to produce and distribute unseen content must match the target audience. At any given moment during a game, not all camera angles may be appropriate for distribution to the public. So the ideal solution will allow broadcasters to decide which cameras to make available for any given event or highlight. The internet connection at the live production site will have limited bandwidth so it will be necessary to find the optimal way to get the most interesting content out of the video servers and onto the smart devices in the users hands. Since we re looking for a way to add value using video content that s already available, we must also limit the amount of extra hardware needed to convert the content stored at broadcast quality on the video servers to a level suitable for delivery over the internet to ipads and other 4

34 smart devices. Video stored on the servers can have a bitrate up to 550 Mbps per video channel which is clearly too high for delivery into the home. To address the issue of limited and unpredictable bandwidth and CPU usage on the smart device at the viewer s end we need to provide the enriched experience using adaptive bitrate streaming solutions. This will guarantee the best video quality possible and a smooth viewer experience, but it brings the complexity of having to create and manage multiple versions of the same piece of content. The distribution of unseen content is marked by a highly unpredictable load and number of concurrent users. To address these issues the solution needs to integrate with existing CDN providers and bring the content as close to the end user as possible. Distributing content over a CDN brings its own challenge in that the solution needs to provide a way of enforcing the rights of the content owner in a flexible way. The rights owner may put a geographical limit on the content made available over the internet, and they may also want to enforce a limit on the time during which the content is available. Content protection therefore needs to be an important part of the platform architecture. From a content management point of view destined to enrich the user experience, the platform needs to provide the following major services : Access to and extraction of resources at the venue Implementation of the production logic by associating extracted resources with relevant metadata Publication and distribution of the content in a flexible and scalable manner General architecture of the platform Fig. 2: General architecture The proposed platform is composed of two main parts: the agent is located at the venue where the live event is happening and the central server is located at a datacenter, usually outside the venue. 5

35 The agent and central server are connected over a reliable network connection that ideally has low latency with most of the transfer coming from the agent to the central server. The responsibility of the agent is to act as a proxy for the central server to the content being recorded by the video servers. The responsibility of the central server is to create, set up and manage the production logic with two major resources, content and metadata. The content would be delivered mainly by the agent, but it can also come from third-party providers through an ingest API. The proposed platform architecture takes into account the limited bandwidth and computing resources at the venue. Since not all camera angles are interesting at all times during the live broadcast, the architecture is based on the extraction of video clips. For example, a typical clip for a remarkable event during a live soccer game has a duration of 20 seconds. To explain the architecture of the platform, we ll follow a workflow that s used to make unseen content available from the production site at the venue to connected devices. Create Event Extract Clip From Camera Angle Transfer Clip Approve & Enrich Clip Transcode Clip Publish Agent in the production truck at the venue An event will be captured from multiple angles, each by a different camera, and since the cameras are time-synchronized in the video servers the proposed architecture allows an operator to define an event as a [time-in, time-out] couple and add a title as metadata to retrieve the clip further along in the workflow. The agent in the production truck at the venue is the software responsible for communication with the local video servers and computing resources. The agent is the only additional server we need to add to the existing production resources and it acts as a gateway for the central server (central) to the production truck. Once an event is created in the truck, the agent will notify the central that video files will be transmitted in the near future. The agent will then take care of extracting a clip for each camera from the relevant video server using the same [time-in, time-out] couple. The video for each angle will be created in a proxy format of which the bitrate settings are compatible with the highest quality available to connected smart devices. As soon as a clip for a camera angle has been extracted from the video servers the agent schedules its transfer to the central. Once the transfer is completed, the central will proceed with the next steps of the workflow to approve and publish the newly-generated content to the content delivery sub-system. As mentioned above, agent and central need to be connected over a reliable network connection, however the platform provides different interfaces to select the appropriate transfer policy. For example, if the agent and central are connected over a wired network connection with low latency the content transfer can be done using simple (S)FTP. While in the case of a high latency connection, the content transfer can, for example, make use of UDP-based solutions. 6

36 The central server in the datacenter and the workspace The central server is the controller entity for our workflow and has several responsibilities. Below is a description of the most important ones. Production context management Located in the datacenter, the central is responsible for creating and managing the context of the production for which the content is being created. Once a production is created and configured on the central server, a production identifier is generated and needs to be communicated to the operator to configure the agent. The production identifier is the first element that will be used to create a common working context between the agent and the central. This allows a given agent, located in the broadcast truck, to be used for different productions - a production being linked to a content rights owner. Once the production context is created, the agent can retrieve the production setup information: camera settings, clip settings, formats, encoding settings and other system configurations. This set up information will permit the agent to generate the appropriate content and data as expected by the central. Content approval The next step in the workflow is to approve one or more camera angles for each event as it becomes available on the central. The second major responsibility of the central is therefore to make available for approval the newly received clips. This procedure gives the operator access to manage the approval of clips with multiple camera angles created by one or several agents. The workspace (See figure 3) provides the operator with specific views of the generated content and data with the ability to: Approve one or more camera angles for publication in the timeline Add extra metadata for each individual event like a time code marker, keywords, the name of the player in the video, etc. Select the most appropriate thumbnail for the event Fig. 3: Production workspace and end-user screen Web Services API provider The third responsibility of the central is to provide an API to access the production data: events list, timing, metadata and the content of all the available camera angles for each event. 7

37 The central provides the data in the form of a timeline. Each event in the timeline contains: The metadata related to the event (e.g. title, timecode, keywords, thumbnails, etc.) The different clips for each of the approved camera angles The central platform and the use of cloud platforms In the context of our platform deployment, our objective is to limit the additional hardware and resources needed at the production facility. To this end, the central platform can be integrated with cloud-based platforms that provide transcoding and media distribution services. The central is itself hosted in a cloud infrastructure and designed to support a multi-tenant deployment model. Time constraints on the workflow The aim of the presented workflow is to enrich end-users experience during live sports broadcasts. To this end, we need to guarantee the delivery of the events and the associated content in a timely manner. The whole process needs to happen in a very short amount of time (e.g. less than 120 seconds), starting from the moment at which the action occurred at the venue. For example here are the main actions that need to be achieved in less than two minutes: Clip the action at the venue Extract the video for each camera angle from the different video servers Transfer content, thumbnails and metadata for each camera angle to the server Transcode each video into multiple bitrates Distribute the timeline, content and metadata to connected devices The above-stated constraints, minimizing extra resources at the venue and the timely delivery of the content, lead us to move the transcoding service into the central located in the cloud. The transcoding challenge Adaptive bitrate protocol To create the best user experience, the system should be able to deliver adaptive bitrate streaming. This consists of creating multiple renditions of each video, available for the player to dynamically select the streaming bitrate as a function of the quality of the network and the available bandwidth. Android and ios based devices both support HTTP Live Streaming 9 created by Apple and submitted as an Internet-Draft to the IETF. The HTTP Live Streaming protocol specifies that a compliant player must first obtain a playlist file and then obtain each of the segments in the playlist. The playlist and segments are retrieved over HTTP and can be served by any web server. There are no special requirements at the server side. Multi-bitrate transcoding In our architecture, the central is designed to use external interfaces to communicate with the transcoding service. The transcoding layer is responsible for scheduling videos to be 8

38 transcoded by the external service, and then create the corresponding playlist descriptor according to the HTTP Live Streaming protocol. The external transcoding service needs to provide multiple bitrates for each video so that the player at the client end, can dynamically adjust the chosen video stream as a function of the available bandwidth. Bitrates vary as a function of the video capabilities of the smart devices. As transcoding is generally a very CPU-intensive operation, the transcoding service must be built on top of a highly efficient and scalable platform. The number of concurrent transcoding jobs per event is simply: the number of camera angles multiplied by the number of bitrate profiles. As we need to deliver our content in the fastest time, the simplest approach is to run the transcoding jobs in parallel for each event. Consequently, the external transcoding service must be able to scale transcoding processors to perform the jobs queued by the central in parallel. Once a transcoding job is successfully completed, the central will be notified in order to complete the next steps defined in the workflow. Availability of the event in the timeline The HTTP Live Streaming protocol is based on the notion of a variant playlist that points to a playlist for each available bitrate. Once all the bitrates have been transcoded, the variant playlist for a video can be finalized and at this point, the corresponding camera angle can be marked as available in the timeline. The availability of the first camera for an event will trigger its availability in the timeline. Bitrate selection and optimization in real deployment The player provided by the smart device s operating system will dynamically select the stream to provide the best experience for the observed bandwidth on the device. This decision can t be influenced by the user or the application developer. It would be inefficient to provide the video in bitrates that would never be selected by the player. Because of the high computing cost of transcoding we need to optimize the choice of bitrates that we want to make available. The selection of bitrates needs to be based on the algorithm that s used by the player to switch to a lower or higher bitrate. Content distribution : the role of the CDN The last major interface to the outside services used by the central is the publishing layer. This layer is responsible for communicating with one or multiple standard CDN services that will ensure the delivery of our content to end-users. This distribution policy was chosen to isolate distribution scalability issues from the central s scalability itself. On other hand, the use of HTTP Live Streaming (HLS) to distribute the content is perfectly compatible with major CDN services and policies. Protecting the rights of the content owner Rights holders usually want to limit their offering to their subscribers. Aside from the commercial advertising model, this is one of the major business models used to monetize their content. One of the services a CDN offers is to geo-block access to the content from users who aren t located in the region where content rights have been cleared. This is a first step but it isn t a sufficient measure to prevent access to the content by unauthorized users. The use of the HLS model to distribute the content enables the use of caching services provided by delivery networks. However this can represent a content security breach to our limited 9

39 access requirement. Therefore, the platform needs to take control of the cached playlists and video segments, limiting their access to only authorized end-users. To this end, we can rely on the capabilities of HLS to encrypt the video content and to provide the corresponding keys through a separate service only accessible by the transcoding service and the player. (See Figure 4). Fig. 4: HTTP Live Streaming Architecture This approach allows multiple use cases: Access to the key(s) from the end-user application can be protected using HTTPS and a username/password combination. This could be a username/password combination entered by the user or even a combination known only by the application. The keys can be rotated as required. We can choose to use one key for the whole production, different keys on a per video segment basis, or randomly change the key at any point in the timeline. Preventing access to the key store can be used to lock access to the content whenever needed. In the case of ios applications it s also possible to let the playlist use a URI that isn t a URL, but uses a scheme only known by the app on the smart device. This allows the application to handle the retrieval of the decryption keys in an optimized way. This is not supported on Android. Storing the encryption keys requires its own dedicated and separately scalable keystore. The keystore can be implemented in the cloud as a dynamically scalable resource as we know that this service will be highly solicited. Scalability and high-availability of the platform The platform provides high availability and scalable operations at multiple levels. Two of the most important are: 10

40 I. Production subsystems: The scalability of the production workflow is guaranteed by the different subsystems. In some deployments multiple agents can feed content into the same central server. On the other hand, central servers can be distributed across different regions with one master and multiple slaves. The master central is usually the closest access point to the venue and receives content from the connected agents. The other central servers operating in slave mode receive their content through replication services. Failover operation is achieved through a regular high availability (HA) model. When the master is down, one of the slaves is elected to operate in master mode, with all the connected agents redirecting their content to it. II. Distribution subsystems: The proposed CDN interface offers the possibility of connecting one or multiple distribution networks at the same time, and automatically handles their redundancy. The platform takes advantage of the CDN s standard scalability and high availability. Real-life deployment for Pay TV soccer Based on our platform, a premium pay TV channel developed a tablet application to deliver complementary and innovative services to improve their users experience with the second screen model. Four types of content are provided through the developed application: Multiple camera angle content: the ability to replay each of the best moments of the match from any of the camera positions around the field. Bonus content: the ability to replay the best moments from the Super-Motion cameras on the field. Statistics: editorial analysis is inserted into the game timeline using an API for external content. Related content: additional content such as interviews and match listings are also provided to enhance the quality and exclusivity of the user experience. During the deployment, we had to overcome some technical challenges related to the satellite broadband connection latency having a high round trip time (RTT). To optimize the user experience, we implemented a priority mechanism to manage the publication of the events to the end-user. Opportunities provided by the platform Integration with social networks and other third-party services Studies have shown that while watching television viewers frequently use laptops, smart phones or tablets to interact with social networks like Facebook and Twitter. The platform presented in this paper can be used by rights holders to create second-screen applications that combine unseen content from multiple camera angles with social network interaction. Allowing users to share interesting events through social networks influences other users to switch channel to the live game (or live event) on the TV. 11

41 On the other hand, the platform allows for the integration of third-party services like player statistics and video archives. The modular architecture of the platform provides the necessary means to bring content coming from different sources together and combine them as additional metadata or content to be distributed to the end user. Non-linear TV consumption After the broadcast of the live game is over, all the production facilities at the venue usually need to be moved to the next event at a different venue. All the content stored on the production facilities will be erased to create the necessary space for the next production. The platform offers archiving services to be activated at the end of the workflow. The content is available in a reusable form on the central and in a transcoded format on the CDN. The rights holder can keep accessible timelines and all available content in the cloud infrastructure at all times after the event. Since all the events created for the production are available online at the central, broadcasters can repurpose the content, creating new highlights and event summaries, and making them available directly through the CDN. Conclusion Live sports events are still a big reason for people to watch TV, but the use of smart devices during broadcast is rising and the attention of viewers is shared between the television screen and the smart device. It s possible to generate additional value by extracting unseen interesting content from live productions and make it available on a smart device and, by doing so, to capture viewers attention for ad-based models. This paper proposes an open platform that enables complex workflows (from content ingest to distribution) and overcomes important technological challenges such as: making the content available to the end-user near-live, Content value decreases over time, imposing an important time constraint on the production workflow. minimizing the extra resources needed at the venue. This is important to reduce operational costs and also because they re based on mobile infrastructures. allowing distribution of the content as widely as possible without requiring special server software, but also complying with the rights of the content to prevent access by unauthorized users Cloud services and infrastructure are a great fit with the proposed platform to enrich the experience of a live event, and also enables the non-linear consumption of TV which aligns with how consumers watch video on the internet. 1 Advanced Television, YouTube, January 23, percent of respondents, Video-Over-Internet Consumer Survey, Accenture: Olympic (and Twitter) records, Twitter Blog, August 12, Olympic athletes score big on Facebook, Wildfire, August 13, Video-Over-Internet Consumer Survey, Accenture: Advanced Television, YouTube, January 23, Video-Over-Internet Consumer Survey, Accenture: cameras, UEFA EURO 2012 TV production fact sheet, UEFA, June, HTTP Live Streaming, R. Pantos Ed., March 23,