Jason Livingston CPC

Transcription

1 Jason Livingston CPC

2 Background New FCC regula=ons require closed cap=ons from TV broadcasts to be available when these videos are delivered by Internet Protocol (IP), such as on the web and mobile devices. SMPTE created a new specifica=on called SMPTE Timed Text (2052) to address the challenges of bringing closed cap=ons from broadcast TV to IP video. Why is Closed Cap=oning required? It s the law More than 48 million Americans have hearing loss For noisy environments (gyms, restaurants, airports, etc.)

3 Brief History of Closed Cap=oning Early Developments Design decisions and limita=ons of 1970s and 80s technology con=nue to affect the development of the latest standards Requirements Limita.ons at the.me Low bandwidth analog transmission Nearly stateless (can jump into the stream at any point and start decoding) Decoder can be implemented with only a few KB of memory Simple anima=on (smooth roll- up) that enhances comprehension Support for mul=ple language tracks simultaneously Support for (limited) text formaang and posi=oning around the screen Character set that can accommodate most Roman alphabet- based languages Very limited processing power and memory available for decoders Must not interfere with exis=ng NTSC receivers NTSC fps only Character set, font, size, background color and opacity, etc. are fixed in the spec and can't be changed by the cap=on author

4 Demonstra=on of CEA- 608 Closed Cap=oning Features

5 Move to CEA- 708 Digital Cap=oning With the switch to digital ATSC broadcasts, a new closed cap=oning spec called CEA- 708 was created to greatly enhance the capabili=es of closed cap=ons, while also preserving backwards compa=bility with analog equipment Support for more text languages (including the possibility of adding Unicode characters for languages such as Chinese, Japanese, Korean, Cyrillic, etc.) [P16 code] More simultaneous languages (63 vs. 4) Proper support for frame rates other than 29.97fps (e.g , fps) 16x9 cap=on posi=oning Cap=oner- specified font styles and more formaang choices Cap=oner- specified background opacity, font edge decora=ons, etc. Carries CEA- 608 backwards compa=bility bytes for use with older equipment (e.g. analog output from an ATSC tuner)

6 Move to CEA- 708 Digital Cap=oning (cont d.) Despite the many advantages and new features of CEA- 708, we are s=ll stuck with the limita=ons of CEA Most cap=on authoring tools s=ll target the CEA- 608 spec only Common cap=on interchange files (SCC, CAP, etc.) are CEA- 608 only Primary language cap=ons must be backwards compa=ble with CEA- 608 (implies that many new features of CEA- 708 cannot be used) Many QC hardware and sokware do not properly display CEA- 708 cap=ons that make use of the new features As a result, most broadcast chains con=nue to use CEA- 608 cap=ons CEA- 608 data is translated to CEA- 708 near the end of the broadcast chain

7 Move to IP/web Delivery Streaming video should replicate broadcast cap=ons as closely as possible, so that people who rely upon them do not get a second class experience. Prior efforts for web cap=oning formats: SAMI, SMIL, SRT - do not support, or too cumbersome to support, all CEA- 608 features TTML (DFXP) many players only support a limited subset (e.g. no posi=oning controls) Na=ve CEA- 608/708 embedded data - requires decoder support in every player (not trivial), not supported by all video container formats

8 Move to IP/web Delivery (cont d) Desired features for a new cap=on format: "Mezzanine" format - one master file for both broadcast and IP delivery Raw CEA- 608/708 data for broadcast TV A more easily processed format for web- based players Must work together with exis=ng cap=on authoring tools and standard prac=ces Support both live (real- =me) and post (VOD) transmission Support both text- based and rendered (bitmap) image- based cap=ons. Format agnos=c: should work with any video codec/wrapper and streaming system, both as an embedded stream and as a sidecar file format Must address all of the FCC and other legal requirements for closed cap=ons

9 CEA- 608 vs. TTML Methodology Na=ve CEA- 608/708 closed cap=on data is transmiqed as a stream of commands and text, at a fixed number of bytes per frame of video. The data is not human readable it has to be parsed to be understood. CEA- 608 Representa=on: MUSIC Frame Data a cdd5 d c 942f Meaning Pop- on mode Row 15 TAB Indent MU SI C Clear Screen Display Cap=on Contrast with =med text format e.g. TTML, which is a document containing markup and text, and is mostly human readable: <p begin='00:00:00:09' end='00:00:02:01> MUSIC </p>

10 Challenges of emula=ng CEA- 608 in TTML W3C TTML is a very full featured format for authoring =med text, but there are a number of aspects of the spec that work very differently from the CEA- 608/708 spec that cap=oners and broadcasters are used to. Cap=on authoring tools may someday author as na=ve TTML or CEA- 708; however currently most cap=oning tools target CEA- 608 which uses a totally different model TTML had no standard provision for including na=ve CEA- 608/708 data The XML data (=ming, posi=oning, styles, content) needed to parse each cap=on is split into mul=ple areas of the file; it does not exist as a single accessible unit of informa=on that can readily be fed into the decoder Typically you need to parse the en=re XML structure to understand its contents. This requires having access to all the data in memory before you start decoding, which is not possible for live broadcasts. Fully conveying all of the proper=es of CEA- 608 cap=ons required very verbose syntax and repe==on of informa=on TTML anima=on (using <set>) enables discrete changes at fixed points in =me, but not smooth anima=on like in CEA- 608 roll- up cap=ons

11 Examples of some 608 features that are hard to represent in TTML Posi=oning cap=on regions: using <set> anima=on in header requires repe==on of the =me code in both the head and the body, e.g.: <head> <region xml:id='pop1'> <set begin='00:00:03:04' end='00:00:06:08' tts:origin='10% 79%'/> <set begin='00:00:07:08' end='00:00:09:05' tts:origin= 40% 10%'/> </region> </head> <body> <p region='pop1' begin='00:00:03:04' end='00:00:06:08'> I'M AT THE LEFT<br/>OF THE SCREEN</p> <p region='pop1' begin='00:00:07:08' end='00:00:09:05'> NOW MY NAME APPEARS<br/>AT THE BOTTOM OF THE SCREEN</p> </body> Downsides: Requires repea=ng each =me code in both the header and the body - - not bandwidth or memory efficient Decoder must hold both structures in memory and apply inheritance rules to each cap=on Difficult to combine with chunking (live streaming)

12 Examples of some 608 features that are hard to represent in TTML (cont d) Proposed solu=on: Allow qs:origin and qs:extent aqributes on <p>, crea=ng a much more compact representa=on which works with chunking. Example: <p region='pop1' begin='00:00:03:04' end='00:00:06:08' tts:origin='10% 79%'>I'M AT THE LEFT<br/>OF THE SCREEN</p> Advantages: Saves bandwidth and memory =me codes are only specified once per cap=on All necessary data is located in one element no difficult inheritance rules Compa=ble with chunking (live streaming) Change proposed for TTML 1.1, see: hqp://

13 Examples of some 608 features that are hard to represent in TTML (cont d) Split (non- con=guous region) cap=ons and different text alignments including indented lines, e.g.: I'M AT THE LEFT OF THE SCREEN. [ Note the 2nd line is indented, not flush or centered ] Map each CEA- 608 row (line) to one region: <p region='pop1' begin='00:00:08:02' end='00:00:09:20' tts:origin='10% 79.33%>I'M AT THE LEFT</p> <p region='pop2' begin='00:00:08:02' end='00:00:09:20' tts:origin='15% 84.67%'>OF THE SCREEN.</p> Or, use one region for all lines, breaking with <br/>, inden=ng with transparent space: <p region='pop1' begin='00:00:08:02' end='00:00:09:20' tts:origin='10% 79.33%'>I'M AT THE LEFT<br/> <span xml:space='preserve' tts:backgroundcolor='transparent'> </ span>of THE SCREEN.</p> Since CEA- 608 cap=ons frequently make use of indenta=on other than lek/center/right jus=fied, the former approach (always puang each line into its own region) makes the overall document look and work more consistently, as each CEA- 608 row always maps to one region and vice versa. It closely emulates the way CEA- 608 cap=on authoring and decoding works.

14 Upcoming Improvements to TTML / SMPTE TT Specifica=on Chunking of the tunneled CEA- 608/708 data sec=on - - work in progress Currently the decoder is responsible for implemen=ng smooth scrolling for roll- up cap=ons; this may be improved in a future version of TTML

15 Other challenges related to broadcast to IP/web cap=on conversion Live broadcast Typically cap=oned by a real- =me court reporter using a stenograph (special keyboard), or by a voice cap=oner using speech recogni=on sokware. Real- =me cap=oner feeds CEA- 608 data into the broadcast hardware Broadcast hardware then translates the CEA- 608 cap=on data into CEA- 708 data as the last step before ATSC encoding/transmission. The video encoder for the IP stream needs to extract the CEA- 608/708 data from the broadcast feed, and generate a chunked SMPTE 2052 stream in real- =me. These chunked segments would either be embedded into the video data, or delivered as separate files to the player. An alterna=ve is to use a player which can decode na=ve CEA- 608/708 data, which eliminates the conversion to SMPTE 2052 step. Video encoder simply needs to copy the CEA- 608/708 data bytes from the broadcast stream into the web video stream Requires the player to implement a broadcast cap=ons decoder, e.g. Apple's QuickTime Player sokware for Mac/Windows and ios

16 Other challenges related to broadcast to IP/web cap=on conversion Re- broadcast of Edited Content Broadcast videos are oken served with different commercials or edited content for the web versions Re- cap=oning these videos from scratch is a costly and =me consuming process Edi=ng systems do not necessarily preserve the exis=ng closed cap=ons at all, and even if they do, the cap=ons are not automa=cally conformed to match the edited versions Tools now exist which can automa=cally conform the original cap=ons to match an edited version by using a Edit Decision List (EDL) generated from the edi=ng sokware The conformed cap=ons can then be saved as na=ve CEA- 608/708, or converted to SMPTE 2052 or other formats

17 Other challenges related to broadcast to IP/web cap=on conversion Mobile Device Support Na=ve video APIs embedded into such devices don't necessarily have closed cap=oning support, or may support only one par=cular format To support a wide variety of mobile devices, providers may need to supply the cap=ons in mul=ple formats Ideally all devices will eventually support SMPTE 2052 and/or CEA- 608/708, but this is not the case yet A workaround is to create a custom player app which is tailored to one par=cular provider Gives the content provider full control over what cap=on formats can be supported by the device Involves more work since each provider must create, test, and maintain a custom app

18 Other challenges related to broadcast to IP/web cap=on conversion Legal rights to the cap.ons Some closed cap=ons are copyrighted by a different en=ty than the owner of the video rights Cap=on rights may only cover the original TV broadcast and may not allow for conversion of the cap=on assets into different formats; or may not allow the cap=ons to be edited It is important for all video content creators and distributors to consider the legal ramifica=ons of having a 3rd party author the cap=ons, especially if that 3rd party maintains rights or restric=ons on the cap=ons they provide The Na=onal Associa=on for the Deaf (NAD) has tes=fied before the US Copyright Office that closed cap=ons and other accessibility technologies should be exempted from copyright and an=- circumven=on measures under copyright law

19 Best Prac=ces for Cap=oners Although CEA- 708 and TTML have many more features and op=ons than CEA- 608, cap=on authors will probably con=nue to author for the CEA- 608 spec for backwards compa=bility. It is recommended that new post produced content be authored in pop- on mode. Pop- on cap=ons are easier to read for people who rely on cap=ons, and deliver a more consistent and reliable experience across a range of plaxorms. Roll- up cap=ons are typically lower quality and harder to read, and should only be used for live cap=oning. Ensure that the legal rights for the cap=ons allow the content providers and distributors to make format conversions, to re- broadcast the cap=ons when the content has been edited and via a wide range of delivery mechanisms, etc.

20 Best Prac=ces for Content Creators Authoring the cap=ons in- house can eliminate legal issues and cut down on cap=oning =me and expense Request that new cap=on projects be done in pop- on mode, except for (near) live produc=ons Cap=ons should ideally be stored in a separate mezzanine format in addi=on to being embedded into the video asset (file and/or tape). Re- use and re- purposing of standalone cap=on files is faster and easier than later extrac=ng the cap=on data from the video asset. Ensure that you have in- house capability to handle cap=on- related tasks such as edi=ng and conversion, ideally in an automated fashion, even if you outsource the labor intensive parts of the cap=oning process

21 Best Prac=ces for Content Providers / Distributors Depending on the sokware you have available, you may need to specify that content providers deliver only a limited subset of closed cap=oning files types; or, with more advanced sokware you can accept a wider range of file types Edi=ng and re- purposing of exis=ng cap=on assets (e.g. commercial removal and segmenta=on) can usually be done with at least some degree of automa=on. Accept and deliver SMPTE 2052 and/or na=ve CEA- 608/708 cap=on data to meet the FCC safe harbor regula=ons

22 Best Prac=ces for Web/Mobile Player Developers Support SMPTE 2052 and/or implement a na=ve CEA- 608/708 broadcast closed cap=ons decoder to meet the FCC safe harbor regula=ons Watch out for "supported" formats that are not fully supported (e.g. limited subset of TTML supported by many current web players), or don't support required features (e.g. SubRip SRT) Ensure all FCC mandated decoder features (cap=on posi=oning and formaang, user controls, etc.) are fully implemented

23 Conclusion The original design constraints of CEA- 608 con=nue to affect the development of new closed cap=oning standards such as SMPTE While hardware and sokware are s=ll catching up with specifica=ons such as CEA- 708 that have been around for years, new video formats and distribu=on technologies are always on the horizon, thus closed cap=oning will con=nue to present new challenges in all aspects of the video produc=on and distribu=on pipeline. Boqom Line: The SMPTE 2052 specifica=on will provide a long term solu=on for CEA- 608/708 closed cap=oning for both broadcast and IP delivery, and other cap=oning and sub=tling use cases

24 About the Presenter Jason Livingston is a developer and product manager with CPC Closed Cap=oning. He is well known for providing closed cap=oning sokware solu=ons to the industry. His recent projects include development of cap=oning sokware with speech recogni=on capabili=es, and implementa=on of the latest SMPTE and CEA closed cap=oning standards. jason@cpcweb.com Tel:

25 Closed Captioning Challenges for IP Video Delivery Jason Livingston - CPC Introduction New FCC regulations require closed captions from TV broadcasts to be available when these videos are delivered by Internet Protocol (IP), such as on the web and mobile devices. This presents a number of challenges in content authoring, asset management, and delivery. To address these challenges, SMPTE created a new specification called SMPTE 2052 (SMPTE Timed Text). This paper will discuss the new regulations and best practices for the different workflows involved, such as: file- based authoring of closed captions for broadcast and IP compatibility, translating existing CEA- 608 and CEA- 708 broadcast closed captions data into SMPTE 2052, common pitfalls and workarounds, and current SMPTE activities to help address these challenges. More than 48 million Americans have hearing loss and can rely on closed captions (CC) to comprehend broadcast TV shows (Lin, Niparko, & Ferrucci, 2011). Captions are also used in noisy environments such as airports, restaurants, and construction sites. However, these viewers cannot always find closed captions when watching Internet video because most closed caption files are created and authored only for TV broadcast. Recent changes to FCC closed captioning laws requires broadcasters to add captioning to Internet distributed video file formats over the next couple of years. To reach a greater audience and to comply with these new laws, content creators should begin repurposing TV closed captioning data files for Internet video. Caption data files used for TV broadcast often are challenging to work with because they contain special coding, formatting, and timing that are difficult to repurpose. Also, TV broadcast video is almost always edited before being distributed over the Internet, causing the original TV caption data file not to match the Internet video version of the show. Brief history of broadcast closed captioning technology In 1973, when broadcasters first started experimenting with line 21 to deliver closed captions via NTSC, there were a number of technical challenges to overcome: All analog transmission path Very limited processing power and memory available for decoders Must not interfere with existing NTSC receivers Despite these challenges, the engineers found some clever solutions to create a functional closed captioning specification. The Television Decoder Circuitry Act of 1990 gave the FCC authority over the broadcast of closed captions and mandated decoders in most televisions sold since The spec gradually evolved into the EIA- 608 (later CEA- 608) spec which is still used today. Some of the key advantages of the spec were: System to transmit low bandwidth digital data in an analog NTSC signal

26 Resilient to analog interference / dropouts Nearly stateless (can jump into the stream at any point and start decoding) Decoder can be implemented with only a few KB of memory Simple animation (smooth roll- up) that enhances comprehension Support for multiple languages simultaneously (e.g. English and Spanish) Support for (limited) text formatting and positioning around the screen (important to avoid covering up critical information on screen such as lower third titles) Character set that can accommodate most Roman alphabet- based languages (English, Spanish, French, German, Dutch, Portuguese) Some of the downsides are: CEA- 608 and line 21 were designed to work at 29.97fps only, although later revisions define how to pack the same data into video at other frame rates, most libraries of legacy content are authored for 29.97fps even when the associated video exists at a different frame rate (e.g fps) Mixed command & data stream means you have to parse all the data to make sense of it - - it's not human readable, e.g.: a cdd5 d c 942f [RCL][ROW15][TAB] MU SI C [EDM][EOC] Character set, font, size, background color and opacity, etc. are fixed in the spec and can't be changed by the caption author; understandable since in the 1980's the idea of real- time rendering of multiple character sets and font styles in would be extremely difficult to achieve in a consumer device Although many of the original technical limitations disappeared with the switch to digital broadcast and the exponential increases in available processing power and memory, many of the design limitations of the original closed captioning spec remain with us today and have a large influence on the development of related technologies such as SMPTE Furthermore, there are very large libraries of existing content authored with CEA- 608 captions which need to remain forwards compatible with new systems. Move to CEA- 708 Digital Captioning With the switch to digital ATSC broadcasts, a new closed captioning spec called EIA- 708 (later CEA- 708) was created to greatly enhance the capabilities of closed captions, while also preserving backwards compatibility with analog equipment by: A) Carrying CEA- 608 data (for use in down- converted analog outputs with line 21) B) Emulating 608- style formatting and control codes so that CEA- 708 data can closely represent the intent of the CEA- 608 captions. CEA- 708 also offers many advantages over CEA- 608 such as:

27 Support for more text languages (including the possibility of adding Unicode characters for languages such as Chinese, Japanese, Korean, Cyrillic, etc.) [P16 code] More simultaneous languages (63 vs. 4) Proper support for frame rates other than 29.97fps (e.g , fps) 16x9 caption positioning Captioner- specified font styles and more formatting choices Captioner- specified background opacity, font edge decorations, etc. Carries CEA- 608 backwards compatibility bytes for use with older equipment (e.g. analog output from an ATSC tuner) Although native CEA- 708 offers many advantages, as of Q there are still very few tools for authoring native CEA- 708 captions, and most of the caption file formats in standard use (Scenarist.SCC, Cheetah.CAP,.TDS, etc.) only store CEA- 608 caption data. For these reasons, captions are almost always authored as CEA- 608 and only translated to CEA- 708 just prior to ATSC broadcast. These translated captions only make use of the limited subset of 608- like commands and styles, and can't take advantage of the new features exclusive to CEA Most caption authoring tools continue to target CEA- 608 to ensure that the captions do not make use of features that can't be represented in the CEA- 608 backwards compatibility bytes. Another complication is that much of the equipment used in the broadcast chain, such as caption encoders/decoders and monitors used to QC the captions, only support the limited 608- like subset of the CEA- 708 spec. Supporting the full CEA- 708 spec is apparently not a high priority because there is virtually no content authored to make use of it. This chicken- and- egg problem has slowed down the development of native CEA- 708 authoring tools and more advanced QC tools. As a result, the vast majority of new content continues to target the features available in CEA- 608 only, meaning CEA- 608 compatibility will remain a critical point for some time to come. Moving to IP/web delivery As content increasingly becomes available via IP and other non- broadcast delivery methods, providing closed captioning has become an important challenge to overcome, not only due to the legal requirements but to reach a wider audience and provide an equivalent TV experience without discriminating against people with disabilities. Ideally the IP video experience should replicate broadcast captions as closely as possible so that people who rely upon them do not get a second class experience. Prior efforts Many streaming video formats have supported some kind of closed captioning or subtitling capability for some time, such as Synchronized Accessible Media Interchange (SAMI), Synchronized Multimedia Integration Language (SMIL), and W3C Timed Text Markup Language (TTML), also known as DFXP. However, while better than not having captions at all, many of these technologies did not fully reproduce (or were too cumbersome to make use of) all of the features possible in TV broadcast CEA- 608 closed captions. Additionally, compatibility and feature completeness is often an issue, for example

28 many web players will accept a limited subset of DFXP (TTML) but will not play the file properly at all if it makes use of TTML features outside of that supported subset. Thus a standard which requires a specified set of features to be implemented by the player is desired. Some web video formats support embedded CEA- 608/708 closed captions, and some devices feature CEA- 608/708 decoders just like broadcast TV receivers have. This means these devices display exactly the same closed captions as a broadcast receiver does. The downside of this approach is that implementing a CEA- 608/708 decoder into every web video player technology is not a trivial task, especially for web developers who are not familiar with the closed captioning concepts used in broadcast, and each implementation is proprietary. While this is great for those players that support embedded CEA- 608/708 captions, it is not a universal solution for the broadest possible application of closed captioning. Desirable features for a new captioning format The challenge that SMPTE is addressing with the 2052 spec is to provide both broadcast- compatible CEA- 608/708 closed captions as well as a timed text based format which may be easier to implement in a broad spectrum of player software and devices. Ideally any new format that the industry standardizes on, especially if it is enshrined as a safe harbor format by the FCC, should achieve the following goals: Function as a "Mezzanine" format - a master file compatible with both broadcast TV and IP delivery o Raw CEA- 608/708 data for broadcast TV compatibility o A more easily processed format for video players that don't have CEA- 608/708 decoders, but should still replicate the look and feel of CEA- 608/708 captions as closely as possible Must work together with existing caption authoring tools and standard practices Be based on existing standards for backwards compatibility and ease of implementation Support both live (real- time) and post (VOD) transmission Support both text- based and rendered (bitmap) image- based captions. Some player devices don't have the capability to render fonts over video in real- time, so for these devices a pre- rendered bitmap image such as PNG can be alpha blended over the video. Format agnostic - Since IP/web uses a lot of different video formats and streaming technologies, a new format should work with any video codec/wrapper and streaming system, both as an embedded (data within the stream) format and as a sidecar file format Must address all of the FCC and other legal requirements for closed captions Challenges of emulating CEA- 608 in TTML The W3C TTML spec that SMPTE- TT is based on is a very full featured format for authoring timed text, but there are a number of aspects of the spec that work very differently from the CEA- 608/708 spec that

29 captioners and broadcasters are used to. These challenges require some careful consideration and workarounds. Caption authoring tools may someday author as native TTML with support for TTML features that aren't in CEA- 608/708; however currently most captioning tools target CEA TTML had no standard provision for including CEA- 608/708 raw data; this was added in SMPTE 2052 so that SMPTE TT files can be used for both broadcast and IP/web delivery The data (timing, positioning, styles, content) needed to parse each caption is split into multiple areas of the file, and does not exist as a single accessible unit of information that can readily be fed into the decoder Typically you need to parse the entire XML structure to understand its contents. This requires having access to all the data in memory before you start decoding, which is not possible for live broadcasts. (e.g. there is no way to know what caption styles and positions will be used later on in the live broadcast) Fully conveying all of the properties of CEA- 608 captions required very verbose syntax and repetition of information - - larger memory and bandwidth requirements Although TTML has a concept of animation (using <set>), this involves discrete changes at fixed points in time, not smooth animation over time like in CEA- 608 roll- up captions Examples of some 608 that was hard to represent in TTML Positioning caption regions: using <set> animation in header requires repetition of the time code in both the head and the body, e.g.: <head> <region xml:id='pop1'> <set begin='00:00:03:04' end='00:00:06:08' tts:origin='10% 79%'/> </region> </head> <body> <p region='pop1' begin='00:00:03:04' end='00:00:06:08'> I'M AT THE LEFT<br/>OF THE SCREEN</p> </body> Downside: Requires repeating each time code in both the header and the body (not bandwidth or memory efficient); not compatible with chunking (live streaming) Proposed solution: Allow tts:origin and tts:extent attributes on <p>, creating a much more compact representation which works with chunking. Proposed for TTML 1.1, see: Example: <p region='pop1' begin='00:00:03:04' end='00:00:06:08' tts:origin='10% 79%'>I'M AT THE LEFT<br/>OF THE SCREEN</p>

30 Split (non- contiguous region) captions and different text alignments including indented lines I'M AT THE LEFT OF THE SCREEN. [ Note the 2nd line is indented, not flush left.] Map each CEA- 608 row (line) to one region: <p region='pop1' begin='00:00:08:02' end='00:00:09:20' tts:origin='10% 79.33%>I'M AT THE LEFT</p> <p region='pop2' begin='00:00:08:02' end='00:00:09:20' tts:origin='15% 84.67%'>OF THE SCREEN.</p> or, use one region for all lines, breaking with <br/>: <p region='pop1' begin='00:00:08:02' end='00:00:09:20' tts:origin='10% 79.33%'>I'M AT THE LEFT<br/> <span xml:space="preserve"> </span>of THE SCREEN.</p> xml:space="preserve" is necessary to preserve the whitespace used for indenting. Since CEA- 608 captions frequently make use of indentation and justification other than center justified, the former approach (always putting each line into its own region) makes the total document look and work more consistently, as each <p> maps to one CEA- 608 row and vice versa. It also more closely emulates the way CEA- 608 caption authoring and decoding works. Upcoming improvements to CEA- 608 to TTML conversion Chunking of the tunneled CEA- 608/708 data section - - work in progress Currently the decoder is responsible for implementing smooth scrolling for roll- up captions - - this may be improved in a future version of TTML Other challenges related to broadcast to IP/web caption conversion Live broadcast Live TV broadcasts are captioned by a real- time court reporter using a stenograph (special keyboard), or by a voice captioner using speech recognition software, where the captioner acts like an interpreter and repeats what is being said by others. These real- time captioning systems feed CEA- 608 data into the broadcast caption encoder installed at the TV station. Then these encoders translate the CEA- 608 caption data into CEA- 708 data as the last step before ATSC encoding/transmission. For live IP broadcast or webcast, the encoder which compresses the video for the IP stream needs to extract this CEA- 608/708 data from the broadcast stream (typically baseband HD- SDI video or an encoded MPEG- 2 transport stream via ASI or IP) and generate a chunked SMPTE 2052 stream in real- time. These chunked segments would either be embedded into the video data, or delivered as separate

31 files. The player/decoder must locate the appropriate data source in order to decode them. This requires the appropriate software infrastructure to tie the separate video and caption assets together. An alternative is to use a player which can decode CEA- 608/708, which eliminates the conversion to SMPTE 2052 step. In that case, the web encoder simply needs to copy the CEA- 608/708 data bytes from the broadcast feed into the IP/web feed. This requires the player to implement a CEA- 608/708 decoder. An example of this is the HTTP Live Streaming (HLS) format and Apple's QuickTime Player software for Mac/Windows and ios. Re- broadcast of Edited Content Broadcast videos are often served with different commercials or edited content for the web versions. The FCC requires these edited videos to also maintain the closed captions, although the implementation deadlines are a little later for edited content. Many of the editing systems used today do not preserve the existing closed captions at all, or if they do, the captions are not automatically conformed to match the edited versions. Re- captioning these videos from scratch is a costly and time consuming process. Fortunately there are now caption tools which can automatically conform the original captions to match an edited version by using a Edit Decision List (EDL) generated from the editing software. This can also work with content that is automatically re- edited, for example by automatic commercial removal or replacement systems, as long as they generate an EDL. The edited captions can then be saved as native CEA- 608/708, or converted to SMPTE 2052 or other formats. Mobile Device Support Many mobile devices such as smart phones and tablets can play video streams. However the native players embedded into such devices don't necessarily have closed captioning support, or may support only one particular format. This means to broadcast to a widest possible variety of mobile devices, broadcasters may need to supply the captions in multiple formats. Ideally all devices will eventually support SMPTE 2052 and/or CEA- 608/708, but this is not the case yet. A workaround is to create a custom player app which is tailored to the infrastructure used by that particular provider. This custom app could support virtually any caption format that the developer wants to implement. This gives the content provider full control over what caption formats can be supported by the device, but involves more work since each content provider must create, test, and maintain a custom app on multiple platforms. Legal rights to the captions Some closed captions are copyrighted by a different entity than the owner of the video rights. The caption rights may only cover the original TV broadcast and may not allow for conversion of the caption assets into different formats, or may not allow re- broadcast of the captions over IP/web or when the video source has been edited. It is important for all video content creators and distributors to consider the legal ramifications of having a 3rd party author the captions, especially if that 3rd party maintains any rights or restrictions on the captions they provide.

32 The National Association for the Deaf (NAD) has testified before the US Copyright Office that closed captions and other accessibility technologies should be exempted from copyright and anti- circumvention measures under copyright law, so it is possible that this may change in the future. Best practices for captioners Although CEA- 708 and TTML have many more features and options than CEA- 608, caption authors will probably continue to author for the CEA- 608 spec for backwards compatibility. Hopefully this will improve in time with better access to authoring tools which support native CEA- 708 and/or TTML authoring, and better QC equipment at the TV stations to properly QC these more feature- rich captions. It is recommended that new post produced content be authored in pop- on mode. Pop- on captions are easier to read for people who rely on captions, and deliver a more consistent and reliable experience across a range of platforms. Roll- up captions are typically lower quality and harder to read, and should only be used for live captioning. Ensure that the legal rights for the captions allow the content providers and distributors to make format conversions, to re- broadcast the captions when the content has been edited and via a wide range of delivery mechanisms, etc. Best practices for content creators Authoring the captions in- house can eliminate legal issues and cut down on captioning time and expense Request that new caption projects be done in pop- on mode, except for (near) live productions Captions should ideally be stored in a separate mezzanine format in addition to being embedded into the video asset (file and/or tape). Re- use and re- purposing of standalone caption files is faster and easier than later extracting the caption data from the video asset. Ensure that you have in- house capability to handle caption- related tasks such as editing and conversion, ideally in an automated fashion, even if you outsource the labor intensive parts of the captioning process Best practices for content distributors Depending on the software you have available, you may need to specify that content providers deliver only a limited subset of closed captioning files types; or with more advanced software you can accept a wider range of file types Editing and re- purposing of existing caption assets (e.g. commercial removal and segmentation) can usually be done with at least some degree of automation. Accept and deliver SMPTE 2052 and/or native CEA- 608/708 caption data to meet the FCC safe harbor regulations

33 Best practices for web/mobile device developers Support SMPTE 2052 and/or implement a native CEA- 608/708 broadcast closed captions decoder to meet the FCC safe harbor regulations Watch out for "supported" formats that are not fully supported (e.g. limited subset of TTML supported by many current web players) or don't support required features (e.g. SubRip SRT) Ensure all FCC mandated decoder features (caption positioning and formatting, user controls, etc.) are fully implemented Conclusion The original design constraints of CEA- 608 continue to affect the development of new closed captioning standards such as SMPTE Although closed captioning is often thought of as a single technology, it actually encompasses and affects a number of related technologies and specifications, creating challenges for developing implementations that meet the FCC regulations. While hardware and software are still catching up with specifications such as CEA- 708 that have been around for years, new video formats and distribution technologies are always on the horizon, thus closed captioning will continue to present new challenges in all aspects of the video production and distribution pipeline. The SMPTE 2052 specification will provide a long term solution for CEA- 608/708 closed captioning and other captioning and subtitling use cases as well. About the Author Jason Livingston is a developer and product manager with CPC Closed Captioning. He is well known for providing closed captioning software solutions to the industry. His recent projects include development of captioning software with speech recognition capabilities, and implementation of the latest SMPTE and CEA closed captioning standards. jason@cpcweb.com Tel: