HTML5 Video. Current situation. Version 1.1. February 8, 2011. Herman van Dompseler / Bas Zoetekouw. SURFnet/Kennisnet Innovation Programme

HTML5 Video Current situation Version 1.1 February 8, 2011 Herman van Dompseler / Bas Zoetekouw SURFnet/Kennisnet Innovation Programme

The SURFnet/Kennisnet Innovation Programme is made possible by funding from the Ministry of Education, Culture and Science. This document has been published under the Dutch version of Creative Commons Licence Attribution 3.0 Unported. For more information about this licence, see http://creativecommons.org/licenses/by/3.0/nl/

Contents SUMMARY... 4 1 INTRODUCTION... 5 1.1 Demonstration website... 6 1.2 More information... 6 2 BACKGROUND... 7 2.1 Current situation regarding video in web browsers... 7 2.2 Codecs and formats for embedded video... 7 2.3 New kid on the block (Why HTML5/WebM/VP8?)... 10 2.4 Features of HTML5 and the <video> tag... 11 2.5 Sources... 12 3 SUPPORTING HTML5 VIDEO... 13 3.1 The <video> tag in web browsers... 13 3.2 Web browser and video matrix... 15 3.3 Mobile access to video... 16 3.4 Mobile devices and video matrix... 17 3.5 What if HTML5 is not supported?... 17 3.6 Sources... 19 4 ADVANCED FEATURES OF HTML5 VIDEO... 20 4.1 Subtitle and annotations... 20 4.2 WebSockets... 21 4.3 Digital Rights Management... 22 4.4 HTML5 Canvas... 23 4.5 Full screen... 24 4.6 Recordings and peripherals... 24 4.7 Support matrix... 25 4.8 Sources... 25 5 SUPPORT ON THE SERVER SIDE... 26 5.1 Distributing video on the Web... 26 5.2 Distributing on-demand video... 26 5.3 Back-end support for HTML5 compatible formats... 29 5.4 Miscellaneous... 34 5.5 Sources... 35 6 CONCLUSION... 36 6.1 Outcomes... 37 6.2 Recommendations... 38 3

Summary The most recent standard for creating web pages is HTML5. This latest version of HTML makes it possible, for the first time, to play video right in a browser without using plug ins such as Flash Player. In this technology scouting, the possibilities of HTML5 in the area of video are considered, and recommendations are made to the SURFnet/Kennisnet communities regarding the support of HTML5 for web video. There are a number of advantages to being able to incorporate videos directly into a web page. The most important of these is that it becomes much easier to create and play videos on the Web. In order to make use of this new technology, the video must be available in a format that is supported by the user s browser. Unfortunately, at the moment there is no single format that is supported by every browser. To support HTML5 video on all platforms, then, every video must be made available in both the H.264 and WebM formats. Note though, that this applies to the next generation of browsers (in particular Internet Explorer 9 and Firefox 4), which are expected in 2011. Although the new browsers already allow videos to be played on an HTML5 page without difficulty, HTML5 video has a number of advanced features that are not supported by browsers, or by only a few. These include playing videos full screen and technical security for copyright protected videos. The technology is advancing rapidly in this area, however, and these features are expected to be eventually implemented in most browsers. Very few changes are required on the server side of things to support HTML video; in general, video files can be provided in the same way as in other technologies. The only area where modifications are required is streaming, live video. HTML5 video has gained a lot of attention in the development community and it looks as if it will become the new standard for web-based video, although there are a number of hurdles to be cleared first, specifically with respect to browser support. The main recommendation of this report is to take this new technology into account when developing video applications. The most important consideration is that videos must be made available in both the H.264 and WebM formats. In addition, it is already possible to support HTML5 video by having browsers that do not support HTML5 fall back on Flash Player, which can also play the H.264 format. 4

1 Introduction Video and video related applications are playing a growing role in education. In the past few years, we have seen a steady stream of new infrastructure facilities that support video, like SURFnet s and Kennisnet s video portals (Teleblik, SURFmedia, etc.).. Many independent initiatives have also been launched, and educational institutions are also making use of existing cloud services such as YouTube. What all these platforms have in common is that they use the World Wide Web to distribute videos. Distributing video on the Web has always been problematical, however. There are many different, mutually incompatible video formats, and not every format can be played on every platform and in every browser. In addition, separate browser plug ins are required to play videos (for example Adobe Flash Player), and these have all kinds of limitations and security problems. However, new advances have recently been made in this area. The W3C, which monitors World Wide Web standards, is working on a new version (version 5) of HTML that defines a new standard for incorporating videos into web pages and playing them directly in the browser, without requiring a separate plug-in. Google launched a new video format, WebM, in early 2010 that has an entirely open specification (in the sense that not only the specification is publically available like for H.264, but also the use is totally free) and makes it possible to distribute high-quality video on the Web without having to purchase licences for the technology. With this new format, Google is trying to stimulate the use of video on the web even more. These developments may have significant implications for the video platforms currently used in Dutch education and at Dutch universities and research centres. Under the auspices of the SURFnet/Kennisnet Innovation Programme, SURFnet has therefore carried out a technology scouting on HTML5 and WebM and what they mean for the use of video on the Web. The present document is the final report on that study. The purpose of the technology scouting was to determine: whether the use of HTML5-based video can be useful for the video services provided by SURFnet, Kennisnet and Dutch educational institutions; whether SURFnet, Kennisnet, and their direct target groups should support the open WebM video format; whether WebM and HTML5 video could eventually replace proprietary video codecs and solutions; whether it already makes sense to invest in HTML5 compatible video services, or whether it is still too early to do so. This report is intended mainly for readers who are somewhat familiar with web pages and video. Detailed accounts of the use of web-based video, or the technology behind web pages, digital video and web-streaming are therefore not given. Section 2 does look briefly at the background of the relevant technology, and provides a review of the current situation of video on the Web. In Section 3 looks more closely at how HTML5 video is supported and the various video formats that can be used by the most important web browsers. In Section 4 various advanced features of video on the Web are discussed, and how they are supported in HTML5. Section 5 is about to the server. The changes that need to be made to web servers to support HTML5 video are considered, and an overview is provided of which software packages can provide suitable video formats for HTML5. In Section 6, contains the conclusions of the technology scouting and makes a number of recommendations concerning how to deal with video on the Web within the context of HTML5. 5

1.1 Demonstration website A basic demonstration website was developed in parallel with this report. A short film has been made available on the website is various formats, so that readers can try out HTML5 video for themselves and see which video formats their browsers do and do not support. The website URL is http://demo.mediamosa.surfnet.nl/html5/. 1.2 More information F. Alons, Educatieve videotoepassingen in het voortgezet onderwijs, 2008, http://web.kennisnet2.nl/portal/onderzoek/onderzoeken/overig/educatievevideotoepassingen N. Verbeij, De magie van video en leren, 2009, http://www.vka.nl/artikelen/publicaties/de_magie_van_video_en_leren W3C HTML Working Group, http://www.w3.org/html/wg/ The WebM project, http://www.webmproject.org/ 6

2 Background This section discusses the use of videos in web browsers. It describes how videos can be played in web browsers, and reviews the various video formats that can be used in browsers. Before describing how the <video> tag works in HTML5, we first look at how videos are currently played in web browsers. 2.1 Current situation regarding video in web browsers In order to play video on an HTML page, the video must be embedded in the page. The standard way of embedding video in an HTML page is to include an <object> tag on the page. When watching a video on YouTube (for example http://www.youtube.com/watch?v=qid93uxvhrw, about the SURFnet lightpaths), a button labelled embed is displayed underneath the video. This button opens a screen that contains a code. You can use this code to embed the video in question in an HTML page. The code belonging to this film is: <object width="640" height="385"> <param name="movie" value="http://www.youtube.com/v/qid93uxvhrw?fs=1&hl=nl_nl"></param> <param name="allowfullscreen" value="true"></param> <param name="allowscriptaccess" value="always"></param> <embed src="http://www.youtube.com/v/qid93uxvhrw?fs=1&hl=nl_nl" type="application/x shockwave Flash" allowscriptaccess="always" allowfullscreen="true" width="640" height="385"></embed> </object> The code contains all kinds of information that the web browser uses to embed the object in the HTML page. For example, it gives the size of the object: 640x385 pixels. YouTube defines a simple basic object, with the URL http://www.youtube.com/v/qid93uxvhrw?fs=1&hl=nl_nl being opened by means of the HTTP protocol. In addition to the video, this URL provides an Adobe Flash video player that plays the video. The <embed> code, which is used by browsers that do not recognise the <object> code, repeats the information; it explicitly reports that the application is a Flash application, however. 2.2 Codecs and formats for embedded video Flash is only one of the formats that can be embedded in an <object>. There are many other video formats that can be played on an HTML page. The format denotes a video technology. A video technology consists of codecs, containers, servers, players, protocols and players. Microsoft, for example, used to refer to this collection of elements as Windows Media. The format often has the same name as the container used in a specific video technology. A container is a kind of envelope that groups together separate video and audio files, subtitles, menus and other elements of a complete video file. In other words, a container is a file that refers to yet other files. Examples include Apple s QuickTime format and the Ogg format. 7

The best-known formats at the moment are: Flash; Microsoft Windows Media/Silverlight; QuickTime; H.264 (MPEG 4); Ogg. SURFmedia, SURFnet s video service, supports several of these formats. Below is an example of a Microsoft object as used in SURFmedia: <object id='iivbtldr8ndsie0ggt5lqzvm' classid='clsid:22d6f312 B0F6 11D0 94AB 0080C74C7E95' standby='loading Media Player components...' type='application/x oleobject' width='1024' height='640'> <param name='filename' value='mms://wmvstream.kennisnet.nl/vpx/5/vjxbwryoffstsla9lozd01j1' /> <param name='autostart' value='true' /> <embed type='application/x mplayer2' src='mms://wmvstream.kennisnet.nl/vpx/5/vjxbwryoffstsla9lozd01j1' autostart='1' name='iivbtldr8ndsie0ggt5lqzvm' width='1024' height='640'/> </embed> </object> This object provides the same kind of information as shown in the example on the previous page, for example the video size, 1024x640 pixels. The filename parameter shows the origin of the video. In this case, the source was a Windows media server ( wmvstream.kennisnet.nl ) using the mms protocol. The term protocol describes how data transfer proceeds between computers. The two main protocols used to play video are: HTTP: Hyper Text Transfer Protocol is the standard protocol in the application layer for data transfer on the Internet. HTTP makes it possible to download a video and play the part that has already loaded before the entire file has been received. This is known as progressive download. RTSP: Real Time Streaming Protocol was developed by RealNetworks, Netscape and Columbia University and is better suited to streaming than HTTP. RTSP differs from HTTP in that it permits more communication between a player and a server. RTSP makes it possible to issue commands to a streaming server that control video files, for example to dynamically adjust the bandwidth if it is inadequate. Microsoft s MMS protocol is one of the various versions of the RTSP protocol. Adobe also uses a similar but incompatible format for streaming in Flash called RTMP. More information about servers and protocols can be found in Section 5. It is important to remember that every format has its own way of embedding videos in HTML pages by means of the <object> tag. Every format is associated with a different video player, 8

and the protocol the way in which the format is sent from the server to the player also differs from one format to the next. That means that the <object> tag does not work automatically in every browser and on every platform. Some browsers and platforms do not support certain protocols or formats. For example, a Windows Media stream will not run out of the box on Apple platforms. Besides the format itself, there is one another important difference between the various formats: the codec or compression technology used. Data compression requires a codec. The word codec is a portmanteau of coder-decoder. A codec is a program that compresses large video files (such as AVI) into smaller files so that they can be streamed on the Internet, and decompresses them again for viewing on a player. Compression makes the file smaller. Unless a special lossless codec is used, compression will result in poorer image quality than the original stream. Lossless codecs cannot compress data as much as lossy codecs, which throw out data and therefore sacrifice quality. The URL in the <object> tag that refers to the video file is in fact referring to the video file s container. The container in turn refers to the separate video and audio elements. The codec compresses these separate elements. The next table shows how the most important formats of the moment differ from one another. Format Flash Windows Media / Silverlight QuickTime H.264 Ogg Container Flv/f4v asf Mov MPEG4 Ogg Video player Flash/swf application Windows Media / Silverlight QuickTime player Flash app / Silverlight / QuickTime Cortado or via WM or QT Protocol Rtmp http / rtsp / mms http / rtsp http / rtmp / rtsp http Audio codec MP3 / AAC Wma / AAC Aifc / AAC AAC Vorbis / flac Video codec VP6 / H.264 Wmv / VC 1 / H.264 Sorensen / H.264 H.264 Theora Proprietary / open License need for encoding Licence needed for playing prop prop prop open open X X X X To summarise the table, note that browsers today have three key <object> tags that open (1) an Adobe Flash Player (2) a Microsoft Windows Media player or (3) an Apple QuickTime 9

player. Each of these three formats has its own proprietary codec and its own method of playing video. There is also one open video format, H.264, which all these players can handle. Although its specification is publicly available, H.264 is the only one for which a licence fee must be paid. There is also the Ogg suite, which has its own full set of tools available and is universally accessible. H.264 is an important format, in particular because it is used by a number of large companies and in effect almost anyone is able to play it. It allows for high compression rates without too much loss of image quality. A video file can be reduced to a relatively small size while retaining a relatively high-quality image. 2.3 New kid on the block (Why HTML5/WebM/VP8?) In May 2010, Google launched a new video format called WebM. Google was careful to position WebM as an open and free video format for the Internet. It delivers high-quality audio and video and can be accessed and used by everyone. In terms of underlying philosophy, WebM most resembles the Ogg format. The following table compares the formats discussed above and WebM. Format Flash Win Media Silverlight QuickTime H.264 Ogg WebM Container Flv/f4v asf Mov MPEG4 Ogg WebM (Matroska) Video player Flash/swf application Windows Media / Silverlight QuickTime player Flash app / Silverlight / QuickTime Cortado (java) HTML5 <video> tag Protocol rtmp http / rtsp / mms http / rtsp http / rtmp / rtsp http http Audio codec MP3 / AAC Wma / AAC Aifc / AAC AAC Vorbis / flac Vorbis Video codec VP6 / H.264 Wmv / VC 1 / H.264 Sorensen / H.264 H.264 Theora VP8 Prop / open prop prop prop open open open Licence for encoding License for playing X X X X The most important features of WebM are that it is open, freely available, and can be used free of charge. The other formats are either proprietary or require a licence agreement. In addition, WebM was developed to play videos directly in the browser, using the new HTML5 technology and related <video> tag. That eliminates the need to have a plug in (Flash, Silverlight or QuickTime) to play video. To play WebM video in HTML5, the browser must support HTML5 and the WebM format. In the following section we look more closely at HTML5 browser support. First, however, we examine the main features of HTML5 in the following section. 10

2.4 Features of HTML5 and the <video> tag HTML5 is a collection of W3C standards with new features for dynamic web pages. One of these is the <video> tag. Another is the graphics <canvas> tag. HTML5 also offers smarter forms, for example search boxes or an e-mail box. A further additional feature is an offline storage database for storing status data in the web browser. Geolocation technology is also part of HTML5. For more information about HTML5, view the presentation at http://slides.html5rocks.com/. For more information about HTML5 in a broader context than in this report, see http://diveintohtml5.org/. This study focuses on the video features in HTML5. The <video> tag is used to embed video in a web page. Unlike the examples given earlier in this section, the <object> tag is no longer necessary. The standard does not impose restrictions on the container or audio or video codec to be used. In theory, then, all formats and codecs are permitted within the <video> tag. The browser determines which video format is supported, and that means that browser developers will ultimately decide which formats are supported. At the moment, the HTML5 specification does not prescribe any one format. In the following section we look in greater detail at possible combinations of browsers and video formats. The video tag has the following attributes for adjusting the presentation. Attributes Value Description Audio muted Default status Autoplay autoplay The video plays automatically. Controls controls Shows video buttons such as a play button. Height [pixels] Video height Loop loop The video will start playing again after it has finished. Preload preload The videos loads along with the page. Src [URL] Video URL Width [pixels] Video width Besides the specific attributes listed above, the <video> tag supports all standard HTML5 attributes. That makes it easy to create and manipulate dynamically controlled and designed videos using JavaScript and CSS. HTML5 Code An example of a simple <video> tag to play the H.264 film MY_MOVIE.mp4 in 320 by 240 pixels is: <video src="my_movie.mp4" width="320" height="240" id="movie" preload controls /> The id tag can be used to manipulate the video object in JavaScript and CSS, for example to adjust its size, pause and play the video, etc. The preload and controls options indicate that the video may be loaded while the page is opening and that the screen should show control buttons, for example to play or pause the video. 11

Note that this <video> tag only works in browsers that support the H.264 format. In the course of this document, we will add code to this tag so that the video becomes suitable for all browsers. 2.5 Sources HTML5: http://diveintohtml5.org/video.html HTML5: http://www.html5video.org/ HTML5: http://en.wikipedia.org/wiki/html5_video HTML5; http://www.w3schools.com/html5/tag_video.asp HTML5 test: http://html5test.com/ QuickTime: http://images.apple.com/euro/quicktime/pdf/quicktime7_user_guide.pdf MS expression: http://www.microsoft.com/netherlands/expression/ MS Silverlight: http://www.microsoft.com/silverlight/ MS codecs: http://msdn.microsoft.com/en us/library/ff819508(vs.85).aspx Silverlight codecs: http://msdn.microsoft.com/en us/library/cc189080(vs.95).aspx All about Ogg: http://www.xiph.org/ Apple Flash: http://en.wikipedia.org/wiki/flash_video H.264 overall: http://www.wowzamedia.com/h.264.html WebM: http://www.webmproject.org/ Related SURFnet reports: http://www.surfnetkennisnetproject.nl/attachments/2235334/ts_open_live_streamin gdefinitief.pdf http://video.surfnet.nl/streamingvideoformaten/rapport_streaming_video_formaten.p df 12

3 Supporting HTML5 video The previous section described how the new <video> tag in HTML5 makes it possible to play video right in a web browser without using a plug in. This appears to be the ideal solution for web-based video, because it eliminates the need for a proprietary video player or software to play video on any platform. But that is still some way off. To begin with, only recent browser versions support HTML5 at all. The various browsers also implement the <video> tag in widely varying ways, meaning that certain videos may not be playable in every browser. All new browsers do support the <video> tag, but the video formats they support differ from one to the next. In section 3.3 we look at the support in mobile devices. 3.1 The <video> tag in web browsers Below, the five most important web browsers today are compared: 1. Internet Explorer; 2. Firefox; 3. Chrome; 4. Safari; 5. Opera. In Section 3.3, the support of mobile devices is considered. For each webbrowser, it is considered whether the most recent version supports HTML5, and whether the beta version will do so, once it becomes available. We restrict ourselves to only the most important formats, which are available on differing platforms. Besides the formats we describe, many software producers also support their own formats, but these are not universally useable. Internet Explorer Internet Explorer is the web browser provided by Microsoft. It is only available on Windows platforms. Current version: 8 The current version of Internet Explorer does not support the <video> tag. Beta version: 9 The latest Microsoft features related to HTML5 video can be found at http://msdn.microsoft.com/en us/ie/ff468705.aspx#_html5_video_audio. With respect to HTML5 and the <video> tag, there will be support for: Container: MP4, WebM 1 Video: H.264, VP8 1 Audio: AAC, mp3, Vorbis 1 Firefox Firefox is a popular browser developed by the Mozilla community. It is available for Windows, OSX, and Linux. 1 WebM does not come standard with Internet Explorer, but by installing the WebM codec in Windows, users can play WebM videos in IE9. 13

Current version: 3.6 This version supports the <video> tag for the following formats. Container: Ogg Video: Theora Audio: Vorbis Beta version: 4.0 Important version 4 HTML5 video features can be found at https://developer.mozilla.org/en/media_formats_supported_by_the_audio_and_video_elements. There is quite a lot of information there about HTML5 and video. Firefox will support WebM: Container: WebM, Ogg Video: VP8, Theora Audio: Vorbis Chrome Chrome is Google s web browser and is available for Windows, OSX, and Linux. Version 5 and up support HTML5 video. Version 6 and up also support WebM. Current version: 7.0 This version supports the <video> tag for the following formats. Safari Container: Ogg, WebM, MP4 Video: Theora, vp8, H.264 Audio: Vorbis, AAC Safari is Apple s web browser. Apple has always supported H.264 and the MP4/MOV container. Safari does that as well in version 3 and up. Current version: 5.0 This version supports the <video> tag for the following formats. Opera Container: MP4 Video: H.264 Audio: AAC Opera is a web browser that was developed in Norway. The Opera company is now part of Nokia. The browser is available for Windows, OSX, and Linux. Current version: 10.6 This version supports the <video> tag for the following formats. Container: Ogg, WebM Video: Theora, VP8 Audio: Vorbis Beta version: 11 The new version of Opera has optimisations for HTML5, but will support the same formats as version 10.6. 14

3.2 Web browser and video matrix All the various versions and supported video formats are shown in a single matrix below. Container Internet Explorer Firefox Chrome Safari Opera 8 9 3.6 4.0 6+ 3+ 10.6+ MP4 Ogg WebM 2 Video codec H.264 Theora VP8 2 Audio codec MP3 AAC Vorbis 2 Combined Ogg/Theora/Vorbis MP4/H.264/AAC/ WebM 2 The table shows that there is no video combination at present that works on every browser with HTML5 <video> tag. At the moment, using the <video> tag to embed videos involves offering a combination of all three video formats. Considering only the new version of Firefox, a combination of at least two video formats is necessary. Possible combinations are: MP4 and Ogg; MP4 and WebM. The most important transitions will be when Internet Explorer launches version 9 with H.264 support and when Firefox launches version 4.0 with WebM support. Since VP8 (the video codec used in WebM) offers better quality and compression than Theora, WebM would appear to be the obvious choice. However, the quality of the video will be very dependent on the encoded used: a good Theora encoder might give better results than a bad VP8 encoder, even though the VP8 codec is more advanced. The main conclusion is that the best way to make HTML5 video available is with the following combination of formats (it should be noted that Chrome supports both formats): MP4 for IE 9 and Safari 3+; WebM for Firefox 4, Chrome 6+ and Opera 10.6+. 2 After manual installation of the codec. 15

Supporting these two formats is the best way of preparing video for the future. These conclusion reflects what YouTube currently has to say about HTML5 support at http://www.youtube.com/html5. HTML5 code As we described above, videos must be made available in multiple formats. In order to make the HTML5 video referred to in Section 2Error! Reference source not found. playable in all browsers, the <video> tag offer has to offer multiple source files. The new example is: <video id="movie" width="320" height="240" preload controls> <source src="my_movie.mp4" /> <source src="my_movie.webm" type='video/webm; codecs="vp8, vorbis"' /> </video> If we want to offer Ogg format in addition to MP4 and WebM, then we can add the following source as a third option within the <video> tag: <source src="my_movie.ogv" type='video/ogg; codecs="theora, vorbis"' /> 3.3 Mobile access to video In addition to web browsers for PCs and laptops, people today are doing more and more browsing with their mobile devices. For the following four mobile devices/software we consider their support for HTML5 video: 1. Apple s ios for iphone and ipad; 2. Google s Android; 3. Windows Phone 7, Microsoft s mobile operating system; 4. Opera Mini, an Opera browser for various mobile platforms. ios for iphone and ipad IPhones and ipads are Apple devices and use the ios operating system. They use Safari and support the MP4 container and H.264 video, just like other Apple devices and software. The current version of ios is version 4. 16

Android The Android web browser is based on the same technology as the Chrome browser and uses a version of the browser engine. Like ios, Android supports H.264 in an MP4 container. The current version of Android, version 2.3, additionally offers support for WebM. Windows Phone 7 This operating system for mobile devices uses a version of Internet Explorer similar to Internet Explorer 8. Windows Phone 7 does not yet support the HTML5 <video> tag. Opera Mini Opera Mini does not yet support the HTML5 <video> tag. 3.4 Mobile devices and video matrix All the versions have once again been combined in a single matrix. ios 4+ Android 2+ Windows Phone 7 Opera Mini Combined MP4/H.264/AAC WebM/VP8/Vorbis (vanaf 2.3) At the moment, the use of the HTML5 <video> tag does not depend on mobile browser support. The two most important mobile devices do use H.264 video, but regrettably, the mobile Windows platform does not yet support it. In terms of the supported codecs, the mobile browser makes no difference, because MP4/H.264 has to be offered anyway for desktop browsers. The code presented earlier therefore also works on mobile devices, if they support HTML5 at all. It should be noted, however, that the H.264 video codec and the AAC audio codec should not be encrypted with the most advanced settings. The most complex encryptions are not supported on mobile platforms and are not playable there. It is therefore advisable to encode H.264 with the baseline profile and AAC with the low complexity profile whenever support for mobile devices is desired. One can for example use the iphone-profile in Handbrake (http://www.handbrake.fr/ ) 3.5 What if HTML5 is not supported? The HTML <video> tag is used to play video in a browser without a plug-in. At the moment, only the newest browsers can do this. Visitors using older browsers who wish to watch the video will require a mechanism allowing them to use a plug in to play the video. To allow visitors to play the video on a plug in if the <video> tag cannot be decoded, an <object> tag can be placed within the <video> tag. This can be any random object, for example Silverlight or Flash. Flash is currently the most popular object for playing a video on a web page. 17

The advantage of Flash is that it can also play an H.264 video. That eliminates the need for separate encoding; the H.264 can be reused to play the Flash object. HTML5 code In the example given below, a JavaScript code has been added to the code in order to generate an <object> tag. This code means that the video can be played in any browser. The JavaScript code was borrowed from SURFmedia and MediaMosa. <video id="movie" width="320" height="240" preload controls> <source src="my_movie.mp4" /> <source src="my_movie.webm" type='video/webm; codecs="vp8, vorbis"' /> <script type='text/javascript' src='http://player.vpcore.snkn.nl/swfobject licensed.js'></script> <div id='mediaspace'>this div will be replaced</div> <script type='text/javascript'> var s1 = new SWFObject('http://player.vpcore.snkn.nl/player licensed.swf','ply','352','308','9'); s1.addparam('allowfullscreen','true'); s1.addparam('allowscriptaccess','always'); s1.addparam('wmode','opaque'); 4'); s1.addparam('flashvars','streamer=rtmp://wowstreaming.snkn.nl/vod&file=vpx/5/my_movie.mp s1.addvariable('type', 'mp4'); s1.write('mediaspace'); </script> </video> If the browser does not support the offered videos, the Javascript code is called, which generates the following <embed>-tag: <embed type="application/x-shockwave-flash" src="http://player.vpcore.snkn.nl/player-licensed.swf" id="ply" name="ply" quality="high" allowfullscreen="true" allowscriptaccess="always" wmode="opaque" flashvars="streamer=rtmp://wowstreaming.snkn.nl/vod& true" file=vpx/5/my_movie.mp4&skin=http://player.vpcore.snkn.nl/classic.zip&autostart= width="352" height="308"> 18

3.6 Sources IE 8: http://www.microsoft.com/netherlands/windows/internet explorer/ IE 9: http://windows.microsoft.com/nl nl/internet explorer/products/ie 9/features IE 9 HTML5: http://msdn.microsoft.com/en us/ie/ff468705.aspx#_html5_video_audio IE 9 WebM: http://windowsteamblog.com/windows/b/bloggingwindows/archive/2010/05/19/another follow up on html5 video in ie9.aspx FF 3.6: http://www.mozilla europe.org/nl/firefox/ FF 4.0: http://www.mozilla.com/en US/firefox/beta/technology/ FF HTML5: https://developer.mozilla.org/en/media_formats_supported_by_the_audio_and_video_el ements Chrome: http://www.google.com/chrome/intl/en/more/index.html Chromium: http://www.chromium.org/chromium projects Safari 5: http://www.apple.com/nl/safari/whats new.html#html5 Safari develop: http://developer.apple.com/library/safari/#documentation/audiovideo/conceptual/using_ HTML5_Audio_Video/Introduction/Introduction.html Opera features: http://www.opera.com/docs/specs/productspecs/ HTML5 according to YouTube: http://www.youtube.com/html5 HTML5: http://diveintohtml5.org/video.html 19

4 Advanced features of HTML5 video This section looks at a number of HTML5 video technologies that go beyond simply playing videos. A number of these technologies have already been integrated into a few browsers, but others are only in the planning stage, or form part of the HTML5 specification that is not yet available in browsers. Bear in mind that although the HTML5 specification has been drawn up by the W3C, it is ultimately the developers of the individual browsers that dictate which features their browser will include and what parts of the standard it will support. And although browser developers have representatives in the W3C, browsers are known to deviate regularly from the standard (for whatever reason). That is true not only of Microsoft, which was famous (in the past) for releasing browsers that were incompatible with current Web standards, but also of other browser developers that had a habit of implementing features not specified in the standard while ignoring certain ones that were. A number of the features discussed below are nothing more than gimmicks, but others are more fundamental in nature and constitute part of the development process that must take place before HTML5 can replace Flash or Silverlight entirely. 4.1 Subtitle and annotations It is often useful to synchronise text or other data with a video or audio clip. One good example is subtitling for a video; another is a set of PowerPoint slides synchronised with a video of a lecture. Although there are various proposals for how this can be achieved, such as SMIL, these have a broader scope than video subtitles and are difficult to use with HTML5 video. There is another proposal that suggests an easier way of solving this problem for synchronised text such as subtitles. The idea is to add tracks containing various forms of synchronised text to a <video> element. The remaining rendering of the text is left to the browser. A video tag with subtitles would take the following form: <video id="movie" width="320" height="240" preload controls> <source src="my_movie.mp4" /> <source src="my_movie.webm" type='video/webm; codecs="vp8, vorbis"' /> <track src="subs_en.vtt" kind= subtitles srclang="en" label= English for the hearing impaired /> <track src="subs_se.vtt" kind= subtitles srclang= se label= Swedish /> <track src="subs_jp.vtt" kind= subtitles srclang="jp" label= Japanese /> </video> In this example, the code defines three text tracks for English, Swedish and Japanese subtitles in addition to the MP4 and WebM video tracks. The subtitle format is in the WebVTT (Web Video Text Tracks) format, a simple format that indicates the starting and finishing time of the subtitle and its content. An WebVTT file looks like this: 20

WEBVTT FILE 1 00:00:20,000 > 00:00:24,400 Altocumulus clouds occur between six thousand 2 00:00:24,600 > 00:00:27,800 and twenty thousand feet above ground level. 3 00:00:32,080 > 00:00:34,320 At these great heights, the air is very cold. In addition to plain text, it is also possible to add more advanced layout markers. All this is still in the proposal stage, and is not yet available in any browser. Subtitles can already be shown, however, in HTML5 videos using JavaScript. An example of what is currently possible (and functional in Firefox and Chrome) can be found at http://www.xiph.org/video/vid1.shtml. See the image below, which was copied from this web page. What is in fact happening is that the subtitles are being loaded in JavaScript. JavaScript uses the HTML5 video API to determine which point the video has reached, so that the right subtitle can be displayed at the right time. It is possible to synchronise other content with an HTML5 video in the same manner. By identifying in JavaScript which point the video has reached, it becomes possible, for example, to substitute a slide in an <img> element at the right time. 4.2 WebSockets It is often difficult to incorporate real time data into web pages. Although all kinds of solutions have been offered for using JavaScript to retrieve data within a web page (for example by means of XMLHttpRequests over HTTP connections), it is not possible to create a full duplex 21

connection to which the server can send data as soon as it becomes available. On top of this, all current connection options come with the overhead of setting up the HTTP connection and exchanging the HTTP headers. HTML5, however, specifies a mechanism WebSockets that makes full duplex connections possible. This makes it possible to use a web application to set up a real-time full-duplex connection with the server. The biggest advantage of this is that it reduces the overhead and latency required for data exchanges. An example at http://www.websockets.org/quantum.html shows a stock market website. Every second, clients request updates from the site. WebSockets would make it possible to reduce the amount of bandwidth wasted on overhead from 665Mbps to 1.6Mbps. Technically speaking, WebSockets works in the following manner. The client (usually the browser) is responsible for initiating the connection by opening a WebSocket to a server. To establish the connection, the client and the server must complete a handshake: GET /text HTTP/1.1 Upgrade: WebSocket Connection: Upgrade Host: www.example.org... HTTP/1.1 101 WebSocket Protocol Handshake Upgrade: WebSocket Connection: Upgrade... Once the handshake is completed, the client and the server can exchange both text and binary frames with only a few bytes of overhead being required. WebSockets is available in Chrome 6, and Safari 5. However, recently some security issues have surfaced, causing a number of browser developers to disable Websockets for now. Because of this reason, websockets are not currently enabled in Firefox 4 or Opera 11. 4.3 Digital Rights Management The distribution of media content on the Internet is a difficult point for many traditional media companies because of a general requirement that users should not be able to store or copy the distributed content locally. This is true both for content that has to be paid for (such as ondemand film rental and streaming from NetFlix) and for content that is freely available (for example at BBC iplayer or its Dutch counterpart, www.uitzendinggemist.nl). Browser plug ins currently used to play videos, such as Adobe Flash and Microsoft Silverlight, offer support at different levels to restrict access to digital content. This is known as Digital Rights Management (DRM). To begin with, it may be difficult to save the video stream in the browser by choosing the Save As option in the context menu, even if the video stream is not encrypted for network distribution. At a lower level, it is still possible to hijack the video stream and copy it to the hard disk when it is loaded to the browser. To prevent that, some servers encode the entire video stream. However, the plug in that plays their video must then know the encryption key that the server uses. Flash, Silverlight and other players have many different methods of exchanging such keys and decoding videos without giving users the option of saving the video. These methods do not yet exist for HTML5, however. The HTML5 specification makes no mention of DRM and exchanging keys, and although there have been calls to include this in the specification, opinions on the matter remain divided. 22

Although nothing has been specified, then, browser developers can build in their own DRM functionalities. Only one developer has done that so far: Apple, in Safari and its ios browser for the iphone and ipad. Moreover, the DRM is only available in Apple s version of HTTP Dynamic Streaming (see Section 5), which works as follows. In Apple s version of HTTP Dynamic Streaming, the video is sliced into segments of approximately 10 seconds each. As described in Section 5, a playlist in an M3U8 file indicates which segment the browser is supposed to display at what time. The playlist can also contain a reference to an encryption key, however. It will then look like this. #EXTM3U #EXT X MEDIA SEQUENCE:7794 #EXT X TARGETDURATION:15 #EXT X KEY:METHOD=AES 128,URI=https://priv.example.com/key.txt #EXTINF:15, http://media.example.com/movie 1.ts #EXTINF:15, http://media.example.com/movie 2.ts #EXTINF:15, http://media.example.com/movie 3.ts To play the video, the browser must first retrieve the encryption key, in this example from https://priv.example.com/key.php?r=52. That location is usually password-protected, and the key can only be downloaded by authorised users. Once the browser has retrieved the key, it can use it to decode the specified media clips, after which they can be displayed to the user. It is then no longer possible to hijack the video stream and save it on the network infrastructure. Obviously, this scheme is not totally secure: the browser has stored the key somewhere, so with some effort, it should be possible to retrieve the key from the browser cache or memory. In practice, however, produces of video turn out to find this security method acceptable, at least its current implementation on Apple platforms. A few content providers in the United States (including Hulu and NetFlix) are already using this secure method to send streaming video to their customers to play on their ipads and iphones. 4.4 HTML5 Canvas HTML5 is introducing a <canvas> element that can be used to draw graphics, pictures or other graphic elements. This is a very useful feature for web applications in general, but it will also be useful for HTML5 video: it makes it possible to load videos into a canvas and manipulate the images, resulting in all sorts of special effects. For examples, see the next screenshots. [http://www.craftymind.com/2010/04/20/blowing up html5 video and mapping it into 3d space/] 23

This feature is only a gimmick at the moment, but it offers all sorts of possibilities in future, for example in gaming. HTML5 canvas for video is available in the newest versons of all major browsers. 4.5 Full screen The HTML5 specification stipulates that the browser must not make it possible to play HTML5 videos in full screen. This is to prevent websites from taking over someone s entire screen and inducing users to click on fraudulent links or to reveal their password or credit card details to fraudulent websites. Many users demand full screen video, however, and browser developers appear to be ignoring the W3C s objections. Videos can be played in full screen in both Firefox and Safari. Chrome and Opera users can use an extension to watch full-screen video. 4.6 Recordings and peripherals In the case of video-conferencing, for example, web applications must have access to the user s webcam and microphone. Flash and Silverlight allow applications access to webcams and microphones, but that is not yet the case for HTML5 applications. Developers are, however, working on regulating access to webcams and microphones (and peripherals in general) in the HTML5 standard. The W3C is proposing a <device> tag that a web page can use to indicate that it wants access to a certain peripheral. The idea is that the browser should allow the user to select the device. Once the user has issued approval, the web page would then be able to control the device directly and, for example, send streaming video to a server (for example using a WebSocket). This feature is not yet available in any browser, although Google has announced that it will be implementing it in Chrome. 24

4.7 Support matrix The table below shows whether and how the various browsers support the features described above. HTML5 Flash IE9 Firefox 4 Chrome 8 Safari 5 Opera 10 Subtitles 3 3 3 3 3 WebSockets 4 DRM HTML5 canvas Full-screen video Webcam/microphone 5 It is clear that many of the advanced features have yet to be implemented in the browsers. 4.8 Sources Subtitles: http://blog.gingertech.net/2008/12/12/attaching subtitles to html5 video/ Subtitles: http://www.w3.org/wai/pf/html/wiki/media_textassociations Subtitles: http://blog.gingertech.net/wp content/uploads/2010/10/webdirsouth2010/index.html slide1 Subtitles: http://www.annodex.net/~silvia/itext/elephant_no_skin.html Subtitles: http://www.whatwg.org/specs/web-apps/current-work/multipage/video.html Subtitles: http://www.whatwg.org/specs/web-apps/current-work/webvtt.html WebSockets: http://www.websockets.org/ WebSockets: http://tools.ietf.org/html/draft ietf hybi thewebsocketprotocol 03 WebSockets: http://www.w3.org/tr/2008/wd html5 20080122/ broadcast DRM: http://www.w3.org/bugs/public/show_bug.cgi?id=10902]. DRM for Dynamic streaming: http://developer.apple.com/library/ios/ documentation/networkinginternet/conceptual/streamingmediaguide/usinghttplivestre aming/usinghttplivestreaming.html Peripherals <device> tag: http://www.whatwg.org/specs/web apps/current work/complete/commands.html devices Peripherals in Chrome: http://www.youtube.com/watch?v=tc3 NpIGx1U&feature=related] WebSockets: http://cometdaily.com/2010/11/04/state of websocket support/ 3 In JavaScript. 4 Available, but standard setting is off. 5 Has been announced for later versions. 25

5 Support on the server side As was discussed in Section 3, the lack of browser support means there are quite a few impediments to adopting HTML Video. We now turn our attention to the server side. This includes all systems used to place videos on the Web, such as web servers, streaming video servers, video encoders en transcoders. This is a somewhat broader definition than simply the web servers needed to send the video to the users. This section also looks at the way in which video is delivered to end users and related issues. This covers such things as the various protocols used to deliver a video, and the methods used to ensure that video streams play smoothly. This section is structured as follows. Firstly, the way in which video is delivered to browsers is considered. A distinction is made between on demand content and live streaming content, and the impact of the various distribution methods on the use of HTML5 video is explored. Then a number of commonly used media encoders and streamers are discussed and various additional technologies for delivering video are looked at. 5.1 Distributing video on the Web There are various ways to place videos on the World Wide Web. The first point to be considered is the difference between live video and on demand content. Live video often involves television-like applications, for example broadcasts of live events or the streaming of real time videos. On-demand video, on the other hand, means videos that the user watches whenever it suits him. There are various popular ways to transport the video stream for both live and on demand video. The technologies overlap partially, but they also differ from one another in a number of key ways. For live video, it is more common to use technologies that transmit the video smoothly to a large number of users at once, for example by adjusting the video quality automatically to the connection quality or the available bandwidth. In the case of on demand content, on the other hand, the problem is how to manage the huge numbers of video files; not all users start watching the video at the same time, so each user needs to be server individually, and the problems tend to be related to fast-forwarding. 5.2 Distributing on-demand video Standard HTTP The easiest way to distribute static on demand content is to use video files that can be downloaded over standard HTTP. Because transmission is to only one user at a time, the video file can simply be placed on the web server like any other file and offered over HTTP to the client, who can then download it like a normal file. It makes no difference to the server whether the client plays the video in Flash Player, another browser plug in, or an HTML5- compatible browser; it is up to the browser or browser plug in to ensure that the video is played properly in the user s browser. In this respect, it is unnecessary to make any adjustments in the web server to accommodate HTML5 video. This system has one major disadvantage for users, however. They can only play the video from the start, and can only fast forward to a point later in the video after the video has downloaded to that point. In some cases and in some video formats, the end of the video file contains essential meta-data, and the video will not start to play until the entire file has been downloaded. 26

Streaming video servers In the past, this problem was solved by using streaming video servers (for example Flash Media Server). These servers interpret the video formats themselves and maintain an index of which point in the video file should be linked to which video time. By also building continuous two way communication into the protocol (for example by using an alternative protocol to HTTP) for the client-server connection (for example RTMP in the case of Flash Media Server), the client can tell the server which segment of the video it wishes to receive. If the user indicates that he wants to start playing the video at a particular point, the server will send the relevant segment to client. This is not entirely straightforward in the case of a simple HTTP download. The disadvantage of this method is that a separate server is required especially for the videos, and that the alternative protocols used may be blocked by firewalls. HTTP pseudo streaming An alternative that has been around for some years now is HTTP pseudo streaming. In HTTP pseudo streaming, a small program (for example a PHP script or Apache module) is added on top of the HTTP web server that is able to interpret the video s index. The client can then send an HTTP GET request for a particular point in the video, after which the pseudo streaming module will provide the correct video segment. The disadvantage of pseudo streaming is not only that an extra bit of software has to be added to the server, but also that the client has to provide special support. One example of an application that supports this is the JW Player (see http:// www.longtailvideo.com/players/jw-flv-player/). Range requests Support for playing HTML5 video is built into the browser, with a version of HTTP pseudostreaming being used. The support makes use of an existing HTTP feature, the Range Request, that allows the client to request a specific portion of a file on the server. The browser has to find out which byte range is linked to which time code in the video, however, so that it can determine what segment of the video file it should request in order to begin playing the video at 6m30s, for example. The browser retrieves this information from the meta-data at the beginning or end of the video file. To do this, it first retrieves the start of the video file. If there is no meta-data, the browser retrieves the end of the file and uses the meta-data it finds there. It can then determine where it should begin reading and either start to play the video at the beginning or at a point that the user has indicated, by requesting only the relevant segment of the video file from the server. Support for this method depends largely on the browser and video format. For example, Chrome 8 supports this method only for WebM and MP4 formats, Firefox 3 only for Ogg, Safari only for MP4, and IE9 does support it at all. Distributing live video The situation with respect to live streaming video differs from that of on demand content. In the former case, video is constructed in real time and transmitted (simultaneously) to different clients, which in their turn display the video in real time. The seeking problem that plays such a crucial role in on demand content is almost nonexistent here. After all, the video is loading in real time and it is not possible to fast forward. A more important concern in live streaming is to be able to provide a large number of users with a stable video stream. As a result, streaming video is often transmitted over UDP, whereas traditional on demand video content is transmitted over HTTP (and therefore TCP). The main advantage of using TCP is that it virtually guarantees that any packet sent will also arrive. The implementation method the client has to re-request the packet if it turns out that one is missing makes this method of data transmission less suitable for live streaming, however; not only is there a 27

larger server overhead involved in keeping track of which packets have and have not been received and asking for packets to be resent, but if the connection is slow, there are delays in the transmission. It is often better for the performance of streaming media to simply skip a missing packet (and as a result, to leave out a frame in the output) than to wait for the dropped packet and stop the video in the meantime. Note though, that in practice sometime the disadvantages of using TCP for live video are disregarded because the advantages (like few or no firewall traversal problems) are deemed more important. Protocols The main transport protocols for streaming video are currently RTP/RTSP (used by Silverlight and QuickTime, for example), RTMP (used by Flash), and despite the objections described above standard HTTP. RTP/RTSP (Real time Transport Protocol/Real Time Streaming Protocol) is an open standard [RFC2326,RFC3350] for the transport of streaming video and audio. Server and client can send control messages to each other over RTSP; the client can request a video, it can request a stream at a higher or lower quality, and it can send information about the quality of what it has received back to the server. The actual content is delivered over RTP (over UTP 6 ). This streaming protocol is supported by QuickTime, Real Media, and Microsoft Silverlight. Current HTML5 browsers do not support RTP/RTSP streams, but some browser developers are working to remedy this. The beta versions of Chrome appear to offer rudimentary support for RTP streams for HTML5 video elements. Firefox is also expected to support this. It is unclear at the moment to what extent Microsoft and Apple will follow suit, because they are banking on a different technology for live streaming in HTML5 (see below). RTMP (Real Time Messaging Protocol) is a closed protocol originally developed by Macromedia (now Adobe) to use with the Flash plug in. Adobe released an RTMP specification in 2009. RTMP has been through several development stages, and now includes a version (RTMPT) for packaging RTMP in HTTP (because standard RTMP is often blocked by firewalls), and a version that transmits encrypted videos (RTMPE). RTMP is used mainly for Flash Player at the moment, and is supported on the server side by Flash Media Server and Wowza. HTML5 browsers do not currently support RTMP, and there are no signs that developers are working on enabling them to do so. For the time being, then, RTMP streams cannot be used for HTML5 video. The third way to transport live video is HTTP Dynamic Streaming. This method, which is suitable for both live video streams and on demand video, divides the video into small segments of 3 10 seconds, which are then sent piecemeal to the client. The client is then responsible for connecting up the separate segments smoothly and for displaying the entire video to the user. An added advantage is that it is relatively easy to achieve dynamic bitrate control. See also Section 5.4 below. Unfortunately, three different versions of this technology are in use, developed by Microsoft, Adobe, and Apple: Microsoft IIS Smooth Streaming In this case, the segments of video are stored on the server in an MP4 container (H.264 video and AAC audio). If the client requests a certain video segment, an extra server module looks up the correct segment in the MP4 file; that segment is then sent 6 TCP is basically also possible, but UDP is in fact usually used. 28

to the client. The necessary server support is only available in Microsoft IIS, and browser support is limited to Microsoft Silverlight. Adobe HTTP Dynamic Streaming Adobe saves individual segments of video in separate F4F files, whose format is based on MP4. Separate metadata (F4M) and index data (F4X) files are also required. The client first downloads the metadata in the F4M file and decides on that basis which F4F files it will request from the server. Once again, the server needs an extra module to interpret the index data in the F4X file. On the browser side, this only works for Flash plug ins at the moment. Apple HTTP Live Streaming In Apple s version of HTTP Live Streaming, every segment of video is stored in a separate MPEG TS container on the server. A playlist is also required in the form of an M3U8 file that specifies, in text, which segment contains which portion of the video. After requesting the M3U8 file, the client can use it to determine which video segments should be downloaded. The server does not require separate support. On the client side, the protocol is supported by Apple ios3 (recent iphones and ipads) and the Safari browser. Other browser developers have expressed an interest in supporting this format. Apple has submitted its version of HTTP Dynamic Streaming to the IETF as the Internet Standard. Of the three technologies, Apple s version is the easiest to set up on the server side, and the company has a massive userbase with all of its iphone and ipad users. Furthermore, the protocol also has some support for DRM (see Section 4.3). There is therefore a good chance that Apple s version of HTTP Dynamic Streaming will become the new de facto standard for streaming video on the Web. The implementation of Apple HTTP Live Streaming is extremely suitable for HTML5 video. Indeed, the websites that use this technology (including Apple itself, but also NetFlix and Akamai) already deliver these videos to their customers on an HTML5 web page with a <video> tag. 5.3 Back-end support for HTML5 compatible formats Now that we have reviewed how to deliver the video payload to users, it is time to look at how to generate video streams, and what software is available for streaming HTML5-compatible video. Generally speaking, the video will already be available in a digital format. That format might be the DV output of a video camera, a stream from a TV satellite, or an MPEG 2 file uploaded by a user. Before such content can be used in an HTML5 <video> tag, it has to be converted into a browser-supported format and codec. As we explained in Section 3, the three most important video formats that can be used with HTML5 video are: a WebM container with VP8 video and Vorbis audio; an MP4 container with H.264 video and AAC audio; an Ogg container with Theora video and Vorbis audio. Because not all web browsers support all formats and codecs, it is important for the server or video back end that offers the video to support different formats, so that it can deliver to all the various browsers. In this section, we look at various widely used video servers and back ends and which of the above-mentioned formats that they support. Note that this list contains both software that only encodes or transcodes video but does not stream, software that only streams ready-made 29

files on the network but does not modify the format, and software that can both stream and encode/transcode. The software that is reviewed here is only a subset of what is available. In particular in the area of encoders and transcoders, many programs are available that can generated the desired formats. FFmpeg (encoding only) FFmpeg is a popular open source tool for converting and transcoding video. It offers multifaceted libraries for decoding and encoding various audio and video formats and can read and write in many different container formats. It is available for various platforms, including Windows, OSX, and Linux, and is used as a back end in a number of media servers, such as MediaMosa. FFmpeg has long supported H.264 video, AAC audio and the MP4 container, Theora video, Vorbis audio and the Ogg container. Version 0.6 and up support WebM containers and VP8 Video (both decoding and encoding). FFmpeg does not support streaming video; it is only intended for converting video into different formats. Handbrake Handbrake is an open-source video transcoder, that is widely used for encoding H.264-type videos. It also supports Theora video, but not in Ogg containers. Also WebM iand VP8 s not supported. Handbrake is particularly useful for encoding videos for use on mobile devices: it has many predefined profiles for different devices, like the iphone and ipod. FFmpeg2theora FFmpeg2theora is a simple program to encode an Ogg format video. It is available for Windows, OSX and Linux. GStreamer (encoding and streaming) GStreamer is a multimedia framework suitable for opening, transcoding and filtering audio and video files. It is therefore frequently used as a media back end for content delivery networks. GStreamer has long supported AAC and Vorbis audio, H.264 and Theora video, and the MP4 and Ogg containers. It has supported WebM and VP8 since June 2010. GStreamer also has output models for various stream types, including HTTP (over Icecast), RTSP/RTP, and RTMP for all available video formats. Flumotion (encoding and streaming) Flumotion is an open source platform for online video streaming developed by Fluendo. There are open source and a commercial versions available; the biggest difference between the two lies in the codecs provided. Flumotion uses GStreamer as the back end, and like GStreamer it supports all three commonly used formats. Due to royalty issues, however, the H.264 and AAC codecs are not provided with the open source version. WebM and VP8 have been supported since version 0.8 (both open source and commercial). The commercial version only supports HTTP streaming. The open source version can also stream over RTSP/RTP. 30

Microsoft Expression Encoder & Windows Media Services (encoding and streaming) Windows Media Services is a server for streaming audio and video. It was developed by Microsoft and is available free of charge with Windows Server 2008. Expression Encoder is Microsoft s video encoding system. Expression Encoder supports H.264 video and AAC audio, in addition to Windows Media formats. It does not support Vorbis audio, Theora video, or VP8 video. Expression Encoder output can be streamed over Windows Media Services (to Silverlight clients) using Microsoft s version of HTTP Dynamic Streaming. QuickTime Pro & QuickTime Streaming Server/Darwin Streaming Server (encoding and streaming) QuickTime Pro is a video editor and encoder for OSX (and also available for Windows). QuickTime Streaming Server is a streaming server for OSX, and comes free with the Server edition van OSX. Darwin Streaming Server is the open source version of the QuickTime Streaming Server. QuickTime Streaming Server is only available for OSX, but Darwin Streaming Server also works on Windows and Linux. QuickTime Pro supports encoding in various video formats, but of the three most commonly used formats for HTML5, only H.264 video with AAC audio is available. The resulting MP4 stream can be distributed over the QuickTime Streaming Server and the Darwin Streaming Server. Streaming is possible over RTSP/RTP. Flash Media Live Encoder & Flash Media Server (encoding and streaming) Flash Media Live Encoder is Adobe s encoding system for (live) video, and is available on Windows and OSX. The resulting video can be distributed over the Flash Media Server. The combination is mainly intended for clients with a Flash plug in. At the moment it only supports H.264 and AAC, and not Theora, VP8, Vorbis, or the Ogg or WebM containers. Streaming is supported over RTMP and the Adobe version of HTTP Live Streaming. Red5 (streaming only) Red5 is an open source Flash server that can also stream Flash video. At the moment it does not support Theora or VP8 video or Vorbis audio, but only H.264 video and AAC audio. Streaming is only possible over RTMP. Real/Helix Universal Server (streaming only) Helix Universal Server is a streaming server developed for Real. It is available for Windows and Linux. Helix Universal Server only supports H.264 video and AAC audio in MP4 containers. It supports live streaming over RTSP/RTP and RTMP. It also supports Apple HTTP Live Streaming. Icecast (streaming only) Icecast is a streaming media server that originally focused on audio. Today it also supports Theora video in Ogg containers. It does not support H.264 or Theora video, AAC audio, or the MP4 and WebM containers. Streaming is only over plain HTTP. 31

Wowza (streaming only) Wowza Media Server is a streaming server developed mainly for H.264 content. It is available for Windows, OSX, and Linux. Wowza only supports H.264 video and AAC audio in MP4 containers. It can stream MP4 over RTSP/RTP, RTMP, and HTTP, and also supports the Apple and Microsoft versions of HTTP Live Streaming. It does not support the WebM and Ogg containers, Theora and VP8 video or Vorbis audio. 32

Overview The table below reviews whether and to what extent popular video formats support HTML5 and their support for streaming over various protocols. Note that transcoders and streamers are combined in this table; an empty white square indicates that the product is not suitable for either streaming or transcoding. If the category is marked, it means that the application basically does support streaming or transcoding, but not for this format. Encode as: Transport over: Ogg/Vorbis/ Theora Mp4/H.264/ AAC WebM/VP8/ Vorbis HTTP RTSP/RTP RTMP FFmpeg (Vanaf versie 0.6) Handbrake ffmpeg2theora Gstreamer (Sinds juni 2010) (Ogg via icecast) Flumotion (alleen in commerciële versie) (niet in commerciele versie) Microsoft Expression Encoder & Windows Media Services (alleen MP4) QuickTime pro & QuickTime/Darwin Streaming Server (alleen MP4) (alleen MP4) Flash Media Live Encoder & Flash Media Server (alleen MP4) Red5 (alleen MP4) Helix Universal Server (alleen MP4) (alleen MP4) (alleen MP4) Icecast (alleen ogm) Wowza (alleen MP4) (alleen MP4) (alleen MP4) 33

5.4 Miscellaneous Dynamic quality control Dynamic quality control means that when a video is being played, the client and the server together determine the highest bandwidth that can be delivered and automatically adjust the quality of the video stream to suit that bandwidth. In actual practice, the server often has a video file available in several bitrates and sizes, allowing the client to select the best quality that it can still manage. Automatic adjustments like these traditionally require a dialogue between the client and the server, and until now this has been implemented in a browser plug in and a streaming server component. There is really only one method of implementing dynamic quality control in the current implementation of HTML5: by means of HTTP Dynamic Streaming on Apple platforms. As described earlier, Apple s version of HTTP Dynamic Streaming slices the video into segments of 5-10 seconds. The browser requests the segments from the server one by one. It is possible, however, to offer each segment in a number of different qualities (and therefore sizes), allowing the browser to select the best quality for each segment that still fits within the available bandwidth. This is a good method of dynamic quality control: it does not require extra communication between browser and server, and the server does not need to provide special support other than making the video available in varying qualities. As mentioned above, however, this method is only available in Apple devices (ios3) and Safari 5. IPv6 IPv6 is the most recent version of the Internet Protocol, designed to succeed IPv4, the underlying Internet transport protocol for the past 30 years. The main driving force for the switch from IPv4 to IPv6 is the foreseeable IPv4 address exhaustion: IPv4 only has address space for four billion computers, whereas in IPv6 that will be 10 34. The introduction of IPv6 is not really an issue for HTML5 video. All modern browsers support IPv6, and there is no reason to assume that IPv6 support for the specific <video> element will become a problem. Multicast Multicast is a transport technology over an IP infrastructure in which a single data stream can be sent automatically to a large number of end receivers on the Internet without the sender needing to send the data to each receiver separately. The transmitter thus only sends the stream once, while it still can be received by many users. This is in contrast to the usual unicast, in which the server has a separate connection with each individual client. Multicast has clear advantages for live video, specifically when it comes to bandwidth usage: a server can basically deliver video to millions of clients using multicast while using only the bandwidth required for a single video stream. One key requirement is that the video stream must be distributed over UDP (for example over RTSP/RTP). The problem, however, is that there are only a few Internet providers that support multicast on IPv4. The lack of support makes it virtually impossible to use multicast to distribute video at the moment. Multicast has been accorded a more prominent place within the IPv6 protocol, and the hope is that after the wide-scale roll-out of IPv6 in the next few years, it will be possible to make more use of multicast for media streams. 34

Multicast streams are not very supported in HTML5 Video tags. Most HTML5 video files will be distributed over HTTP, and because that is based on TCP, multicast will not be possible. Browsers that can play RTSP/RTP streams in HTML5 video should basically be able to use multicast to receive the media stream. Currently, however, there are no browsers that have that capability. 5.5 Sources Adaptive HTTP Streaming: http://blog.gingertech.net/2010/10/09/adaptive-httpstreaming-for-open-codecs/ RTSP for Chrome: http://code.google.com/p/chromium/issues/detail?id=25573 Microsoft IIS Smooth Streaming: http://learn.iis.net/page.aspx/626/smooth streaming technical overview/ Microsoft IIS Smooth Streaming: http://www.microsoft.com/downloads/en/details.aspx?displaylang=en&familyid=03d225 83 3ed6 44da 8464 b1b4b5ca7520] Adobe HTTP Dynamic Streaming [http://www.adobe.com/products/httpdynamicstreaming/ Apple HTTP Live Streaming: http://www.appleinsider.com/articles/09/07/08/apple_launches_http_live_streaming_stan dard_in_iphone_3_0.html Apple HTTP Live Streaming: http://tools.ietf.org/html/draft pantos http live streaming 01 Comparison of media streamers: http://en.wikipedia.org/wiki/comparison_of_streaming_media_systems FFmpeg: http://www.ffmpeg.org/ VLC: http://www.videolan.org/vlc/ Gstreamer: http://www.gstreamer.net/ Flumotion: http://www.flumotion.net/ Flendo: http://www.fluendo.com/ Windows Media Services: http://www.microsoft.com/windows/windowsmedia/forpros/server/server.aspx QuickTime: http://docs.info.apple.com/article.html?artnum=120145 Flash Media Server: http://www.adobe.com/products/flashmediaserver/ Red5: http://osflash.org/red5 Helix Universal Server: http://www.realnetworks.com/products-services/helix-serverproxy.aspx Icecast: http://www.icecast.org/ Wowza: http://www.wowzamedia.com/ 35

6 Conclusion This technology scouting has discussed the use of videos in web browsers. The <video> tag in HTML5 is a new way of embedding video in a web page. Using this technology will simplify the use of videos in web browsers. It has become possible to play video without using a plug in in an <object> tag. That means that, theoretically, there are no longer any restrictions with respect to the video format, container, or codec to be used. It used to be the plug in that was responsible for playing a video correctly; now, with this new technology, it is the web browser. Because there are many different web browsers on many different playing devices (PCs, laptops, mobile devices), there are also many different ways of implementing the functionality of the <video> tag. Not every browser supports every video format. The three audio and video formats used at the moment are: 1. an MP4 container with H.264 video and AAC audio; 2. an Ogg container with Theora video and Vorbis audio; 3. WebM, a WebM container with VP8 video and Vorbis audio. Research has shown that it takes multiple versions of these formats to play video on all devices using HTML5. The combination of MP4 and Ogg or of MP4 and WebM is sufficient. The <video> tag provides for a mechanism to link multiple video formats to a single tag. As a result, only a single segment of code is required to run video on all playing devices. The video-codec that is used in the WebM-format, VP8, is more modern that Theora, and thus potentially allows for better compression and higher quality. Therefore, WebM seems to be the better choice. Note, however, that the current version of Firefox, 3.6, does not yet support WebM; this support will arrive in version 4, which will be released in 2011. The HTML5 <video> tag is a new technology that will ultimately have to compete with existing video players, such as QuickTime, Silverlight (Windows Media) and Flash. Flash is the most widely used at the moment and offers a number of advanced features that HTML5 does not yet support. This study considered Digital Rights Management (DRM) and the ability to play a video full screen. It is precisely in these two areas that HTML5, unlike Flash, lacks an all-round solution. This is one reason to continue using Flash for the time being. Sooner or later, however, we expect that HTML5 will also support these advanced video functionalities. On the other hand, there are also advanced functionalities that are much easier to add when using HTML5. This is because the <video> tag is part of the web page itself, making it possible to control it with JavaScript and CSS. It has become much easier, for example, to add subtitling to a video and control it from the web page. It is also possible to manipulate the video image in a <canvas> tag. Because HTML5 is embedded in the web page, streaming technology has also been simplified. The server does not require much technology to display a video in the browser. Because streaming is over HTTP, a video can be managed in precisely the same way as an HTML page or an image, and the video can be delivered from a web server. For advanced streaming technologies, existing plug-ins use other protocols, such as RTSP, MMS and RTMP. These protocols provide for functionalities such as bandwidth negotiation between server and plug in. They also make it easy to fast forward and backward in a video. HTML5 also offers the fast-forward/backward function, but it does not provide other advanced server functionalities (as yet). This study examined a number of video servers and considered the level of support they provide for streaming HTML5 video. The study revealed that only the Flumotion open source platform is suitable for streaming all three major formats for use with HTML5. 36

6.1 Outcomes In the introduction, a number of goals for this technology scouting have been established. From the information in the previous chapters, the questions asked in the introduction can be answered as follows: Can the use of HTML5-based video be useful for the video services provided by SURFnet, Kennisnet and Dutch educational institutions? As this report shows, HTML5-based video offers a number of new possibilities, in particular when it comes to integration of the video with the rest of the web page. Moreover, for playing HTML5-based video no separate browsers plugins are required. Therefore, HTML5-based video can indeed be very useful in the video services offered by SURFnet, Kennisnet, and the Dutch educational institutions. The main disadvantage of HTML5-based video is that for now, there is no single format that can be playing in all browsers. Therefore, two versions of the same video are required to be able to serve all clients. It is expected to on the long term HTML5 Video will be the default way of embedding video in web pages. However, this will take several years. Should SURFnet, Kennisnet, and their direct target groups support the open WebM video format? WebM is a very attractive format for videos on the web, because it is a fully open format, and no royalties need to be paid for its use. This is in particular the case for the research and education sector that are served by SURFnet and Kennisnet. To use HTML5-based video for distribution of video, it is necessary to offer a WebM-format video (in addition to a video in MP4-format) to be able to serve all clients. SURFnet, Kennisnet, and their direct target groups should therefore indeed consider offering their videos in this format. The main alternative is the Ogg-format, which is also open and freely available. However, Ogg (Theora) is a bit less advanced than WebM (VP8). Therefore, if a choice has to be made between the two, WebM should be preferred. Could WebM and HTML5 Video eventually replace proprietary video codecs and solutions? In the short term, too few browsers support HTML5 video to replace the use of separate video-plugins with HTML5-based video. However, once the next version of Microsoft Internet Explorer (version 9) and Mozilla Firefox (version 4) are released and most users have switched to these new versions, the use of HTML5-based videos becomes feasible. In the long term, the use of proprietary browser plugins can be phased out, and all video can be served using HTML5. The situation for WebM support is somewhat more complicated. The main platforms that will not support WebM in short term are Internet Explorer on Windows, and Apple browsers (both Safari and the integrated browsers on ipad and iphone). In practice, Internet Explorer will be able to play WebM video if the correct codec in installed in Windows Media. Apple platforms, however, will lack support for WebM. Therefore, it is expected that WebM alone will not suffice to serve all users, and in addition an MP4-format will have to be offered. Does it make sense to invest in HTML5 compatible video services at this time, or is it still too early to do so? 37

For a full switch to HTML5-based video it is still too early. However, it would be wise to prepare to make a possible future switch to HTML5-based video easier. Recommendations for this scenario are given in the next section. 6.2 Recommendations There is no doubt that HTML5 <video> is going to happen. These recommendations provide a number of guidelines for using HTML5 video. We have looked at the video formats currently used in various browsers as well as the formats that are growing increasingly popular. MP4/H.264/AAC has the best support at the moment. It is currently the best choice for video encoding. MP4 will also be used in the future because it is the most important format in new mobile environments, in particular on Apple devices. The licensing fees associated with H.264 will make an alternative inevitable, however. The best alternative probably is WebM (WebM/VP8/Vorbis). This open format, launched in 2010, is rapidly gaining ground, together with HTML5. WebM is attractive because of the combination of its high quality and its free and open specification. In the near future, we expect most major browsers to start supporting WebM; only Apple will probably not support WebM very soon. For now, though, WebM has too little support in the browsers to completely replace MP4 and H.264 at the present time. Ogg/Theora/Vorbis is another open and free format, which has been available for quite some time. Although this format does not offer quite the same video compression quality as VP8 and H.264, it is basically a good alternative, and the technology has already been around for a while. It has not been widely accepted, however, mainly because of lack of support in the main browsers and video plugins. If a decision is taken to begin working with HTML5, we recommend using MP4, with Flash Player as the fall-back option. The same MP4 format can be played in a Flash plug in if the browser does not support the <video> tag. That makes the video universally playable. Preferably, a second encoding in WebM should be added right away to lay the necessary groundwork for the future. In the future, the WebM format should be used as the basis to serve most browsers. For Apple platforms, an MP4 format can be offered as a fall-back. We also recommend converting any existing video in a proprietary format (for example Windows Media or Flash) to WebM and MP4. The new video can be played in the same environment with the same players but can also be used as HTML5 video. If Digital Rights Management (DRM) is an issue within the organisation, then the only formats that will support it are proprietary ones, i.e. Windows Media or Flash. The best option in that case is to delay distributing the video in HTML5. The video can, however, already be encoded in MP4. 38