NHK STRL is studying technologies that take advantage of broadcasting and telecommunications networks to offer new services and enhance programming. We participated in at IPTV Forum s standardization activities that led to Hybridcast technical specifications being issued in March 2013. TV sets based on these specifications came to market not long after, and NHK launched Hybridcast services in September. We also worked on advanced Hybridcast and Hybridcast for 8K Super Hi-Vision broadcasting. In addition, we contributed to the standardization of advanced Hybridcast at IPTV Forum Japan. To provide more attractive hybrid services, we are conducting R&D on technologies that use program related information and social networking services (SNS). We studied methods to direct Internet users to various programs and related services by linking the broadcasters programrelated information with open data. We also studied ways to promote TV viewing by analyzing the viewing behaviors of SNS users. We developed a system to allow users to share CG content while communicating with each other through an SNS. We studied a technology to provide program-related information matching viewers preferences by monitoring their viewing behavior and inferring their interests. We developed a method to analyze the viewers behavior based on gazes, prototyped a viewing behavior monitoring system, and experimentally studied the relation between gazes and viewers interests. Along with our efforts to improve the convenience of hybrid services, we are developing technologies to provide secure and reliable services. We developed a new authorization scheme to control different license management systems for broadcast and telecommunications services in an integrated framework. We developed a verification system for the next-generation Conditional Access System (CAS) and contributed to standardization activities at the Information and Communications Council. As a cryptography technology to provide more secure and reliable services, we developed an attribute-based encryption scheme with which personal information is stored securely in the cloud and only service providers who meet certain conditions can decrypt the information. To protect content, we developed a cryptography method capable of tracing the key owner. For program distribution over a communications network, we developed a distributed broadcasting system that enables easy timeshift viewing of past programs over a network and a push content delivery method to distribute information via communication channels in synchronization with broadcast content. We also developed a hybrid live streaming system linking peer-to-peer (P2P) and content delivery network (CDN) distribution. 3.1 Technologies for the convergence of broadcasting and telecommunications 3.1.1 Hybridcast We are continuing our research on Hybridcast, which is a service platform to provide flexible and extensible new services that take advantage of the characteristics of broadcasting and telecommunications networks. In FY 2013, we released the first version of this service to the public and continued with R&D to make the service more efficient and standardization. We showed the appeal of Hybridcast by demonstrating it to visitors at the NHK STRL Open House and other exhibitions at home and abroad. Practical use of Hybridcast The IPTV Forum published the first edition of Hybridcast technical specifications in March 2013. (1)(2)(3) Operational specifications for receivers were also created and standardized, and manufacturers started selling TV sets supporting Hybridcast in June. NHK launched Hybridcast services (Figure 1) in September 2013 and followed up by launching more in December. Prior to the launch, NHK STRL had participated in Hybridcast promotion projects and contributed to the studies on services to be offered by NHK, facilities and equipment, and operational Figure 1. Example of Hybridcast services 16 NHK STRL ANNUAL REPORT 2013
TV screen Tablet display Preview management window Application preview window Figure 4. Application development support tool Figure 2. Example of Hybridcast services at the NHK STRL Open House (Interactive picture drawing application) Tablet devices TV screen Figure 3. Example of Hybridcast services at the NHK STRL Open House (Musical instrument sessions application) aspects. We demonstrated a wide variety of Hybridcast services together with commercial broadcasters and pay TV service providers at the NHK STRL Open House 2013, using Hybridcast compliant receivers developed in cooperation with manufacturers and telecommunications carriers. We exhibited a home screen service, active program guide, scroll news, and TV60 Minogashi-Natsukashi, a service to provide past programs on demand.. We also exhibited an interactive picture drawing application (Figure 2), musical instrument sessions application (Figure 3), multilingual closed-captions and the TV note service, which TV links tablet devices and synchronizes service provision with the progress of programs. We improved the public s understanding of Hybridcast and showed the feasibility of various services. Examples of Hybridcast services to be offered by commercial broadcasters were also exhibited (4). Dissemination of Hybridcast and the growth of its services will require an environment to enable participation by a wide range of application developers. Whereas a versatile HTML authoring tool can be used to develop the Hybridcast applications themselves, which are written in HTML 5, an operational verification is also needed to preview how the developed service functions on various devices. Thus, we developed an application development support tool capable of previewing the linkage of programs and portable devices on a PC and exhibited it at the NHK STRL Open House (Figure 4) (5). Technologies for advanced Hybridcast We are studying the technical specifications for the next version of Hybridcast (called advanced Hybridcast) and Hybridcast for the 8K Super Hi-Vision era. The new functions include: a system model of non-broadcast managed applications allowing various service providers other than broadcasters to participate; a recording and reproduction functionality; precise synchronization using time stamps; and a sophisticated device linkage interface. We developed an application distribution management system (Figure 5) to study non-broadcast managed applications. It can register and distribute applications and issue signature keys to authenticate them. We exhibited a system model for other service providers than broadcasters to provide applications at the NHK STRL Open House. With the launch of 8K Super Hi-Vision (SHV) broadcasting in 2020 in mind, we studied possible 8K Hybridcast services that could take advantage of the large, extremely high-definition screen of this new form of broadcasting (Figure 6). We exhibited a live broadcasting service for covering marathons at the NHK STRL Open House. A high-definition map to show the current location is displayed together with multiple windows showing video coverage of the race and tweets from social networking services (SNS) provided by broadcasting and tele- Figure 5. Application distribution management system NHK STRL ANNUAL REPORT 2013 17
Broadcast video Live distribution Map Ranking data displayed SNS casting and drafted a report on the different forms of hybrid broadcasting in the world, including Hybridcast and Europe s HbbTV. In the Future of Broadcasting Television (FOBTV) group, we participated in a study on a system model based on the use cases submitted to date. At the Technical Plenary/Advisory Committee Meeting (TPAC) of the World Wide Web Consortium (W3C) working on HTML5 standardization, NHK reported on the establishment of Hybridcast specifications at the domestic IPTV Forum Japan in March 2013 and on the launch of Hybridcast services in September. These reports conveyed the advantages of Hybridcast to the world. On the domestic front, we worked on standardization of advanced Hybridcast and HTML5 browser specifications at the IPTV Forum Japan. We will continue with our research on more sophisticated services and standardization for Hybridcast. We will also investigate hybrid services for the SHV age and standardize SHV data broadcasting. Figure 6. Example of 8K SHV Hybridcast services (Marathon live broadcasting) communications. Together with other divisions involved in the development of SHV, we studied the SHV data broadcasting and designed a reference model of the protocol stack based on MPEG Media Transport (MMT) we also investigated data coding, transmission methods and closed captions. Hybridcast standardization At ITU-R, we contributed to a draft new Recommendation for the general and technical requirements of hybrid broad- (1) IPTVFJ STD-0010, Integrated Broadcast-Broadband System Specification Version 1.0, IPTV Forum Japan (2) IPTVFJ STD-0011, HTML5 Browser Specification Version 1.0, IPTV Forum Japan (3) M. Takechi: Technical Specifications and Trend of Hybridcast, NHK STRL R&D, No. 142, pp. 12-19 (2013) (in Japanese) (4) A. Baba, H. Oomata, K. Matsumura, M. Takechi, S. Sunasaki: Prototype Hybridcast Services at NHK STRL, Open House 2013, ITE Technical Report, Vol. 37, No. 41, pp. 17-20 (2013) (in Japanese) (5) K. Majima, K. Matsumura, Y. Hironaka: Development of Hybridcast Application Development and Distribution System, NHK STRL R&D, No. 142,pp. 20-27 (2013) (in Japanese) 3.1.2 SNS and Program viewing analysis We did R&D on technologies to make use of program related information and social networking services (SNS) to provide more attractive services that exploit the advantages of broadcasting and telecommunications. Information network for broadcasting programs We studied ways to direct Internet users from one program to various services or other programs by utilizing broadcasters program guide information and related information and to form a community using SNSs to bring excitement to the viewing experience. We devised a method to convert the program guide information in the Electronic Program Guide (EPG) to the Linked Open Data (LOD) format, link it with open data such as National Diet Library Authority Data and Twitter, and associate programs with program related information and social information (1). We prototyped an application to guide users from one program to related programs and information by web-based visualization (Figure 1). For behavior analysis of users, we analyzed the results of experiments on the teleda social TV platform developed in FY 2011 and found that users fall into different types, such as those who prioritize their own preferences and those who are influenced by others behavior. Based on the analysis, we compiled service design guidelines. We also devised a method to estimate users viewing trends such as viewing times and programs viewed from their tweets on Twitter and use those estimates to categorize users. Identifying influential users on SNSs based on the categorization and providing them with effective program recommendation services should promote TV viewing among SNS users. Virtualization of video content production We studied virtualization of video content production based on the server side video creation so that Internet users can easily produce their own video content. We built an SNS-style CG content creation system (Figure 2) (2), in which SNS users can communicate through videos production using the servers on the network and exhibited it at the NHK STRL Open House. This system allows users to re-create and share content derived from already posted content. Users Figure 1. Program-related information navigation application with open data 18 NHK STRL ANNUAL REPORT 2013
Figure 3. Example of gaze behavior estimation system in operation Figure 2. SNS-style CG content creation system can easily produce derived content by using a graphical user interface (GUI). We conducted experiments with about 100 users after we had posted sample content and demonstrated the system and method. Regarding linkage with Hybridcast, we developed a system to display sign-language CG animations on tablet devices concurrently with a broadcast and exhibited the system at the NHK STRL Open House. We also developed a simple TVML (TV program Making Language) player by using Web GL. It enables real-time runnning of TVML scripts sent with broadcasting without installing special software on the terminal. We also worked on enhancing the functionality of the CG creation engine. We developed a new TVML player using the latest graphics library. The player can display more characters through the use of high-speed processing and improve highquality translucent display. Acquisition and analysis of viewing behaviors We are researching technologies for the automatic acquisition and analysis of viewing status and the estimation of viewers interests, in a domestic TV viewing environment with a view to developing new services that provide program-related information matching viewers preferences. We developed a method for measuring a viewer s body movements and head pose and a machine learning system that estimates whether the viewer is gazing at the TV screen or not. Experiments showed that gaze behavior can be judged accurately (3). We also developed a method that combines Bayesian inference with an estimation based on the intensity of facial expressions, which has improved the accuracy of facial expression recognition. The research on gaze behavior estimation and on facial expression recognition were conducted in cooperation with Tokyo City University and the University of Tokyo, respectively. Part of the research on gaze behavior estimation was conducted under a government-commissioned project from the Ministry of Internal Affairs and Communications, titled Strategic Information and Communications R&D Promotion Programme (SCOPE). We prototyped a viewing behavior monitoring system incorporating the method of estimating gaze behavior (Figure 3). We used the system to study the relation between gaze behavior and inferred viewer interests through TV viewing experiments. The results demonstrated the possibility of inferring viewers interests by correlating gaze behavior with closed caption data that appears on screen during gaze periods. We also developed a method to selectively extract keywords from closed caption data that belong to particular classes derived from Wikipedia category information. This method enabled selective extraction of keywords identifying the names of people and places, which are often for the focus of viewer interest. (1) K. Ariyasu, T. Nakagawa: A Proposal of TV Program Information Network Formation Method by Linked Open Data, Proc. of FIT2013, no.2, F-035, pp. 363-364 (2013) (in Japanese) (2) M. Douke, H. Kaneko, S. Inoue: SNS Style CG Content Creation System Enables Users to the Easy Chain of Re-areation, Proc. of ITE Annual Conference, 2-3, 2013 (in Japanese) (3) Takahashi, Clippingdale, Okuda, Yamanouchi, Naemura, Shibata: Automatic rating of video contents based on viewer s behaviors in living space, SITIS2013, pp. 6-13 (2013) 3.1.3 Security technologies To support the convergence of broadcasting and telecommunications, we are researching integrated authentication federation technology for broadcast and telecommunications, a next-generation conditional access system (CAS), cryptography, and device-linked technology. Integrated authentication federation technology for broadcast and telecommunications We made progress with our R&D on an integrated authentication technology to enable secure and easy use of personalized services by linking license management schemes used in broadcast and telecommunications services. Depending on the types of channels and terminals used, different methods are used by broadcasters and telecommunications providers for access control and content protection. To view the same content in different environments, individual contract management is required for each service. To simplify this situation for subscribers to these services, we developed a license distribution management system embodying a new authorization scheme that merges different license management methods - a conditional access system for broadcast services and a digital rights management system for telecommunications services - in an integrated framework (1). A prototype NHK STRL ANNUAL REPORT 2013 19
Broadcast content viewing Receiver with built-in downloadable CAS EMM server Authentication / authorization server system based on this method was exhibited at the NHK STRL Open House (Figure 1). This system enables content viewing rights obtained from a single user authentication to be carried over to different services of broadcasting and telecommunications. To put it to practical use, we also made proposals to incorporate the next-generation conditional access system in commercial standards. We developed the TV note system to provide personalized services securely while efficiently collecting information on viewers interests through a linkage with authentication services. The system was exhibited as a service model of Hybridcast at the NHK STRL Open House. We also studied the specifications of the authorization scheme and a device linkage method in order to apply the authentication framework to actual services. Next-generation CAS technology Telecommunications content viewing EMM: Entitlement Management Message (Broadcast license) DRM: Digital Rights Management (Telecommunications license) VOD server / DRM server Tablet terminal Figure 1. Integrated access control system for broadcast and telecommunications We are researching next-generation CAS technology for the SHV test broadcasting in 2016. In FY 2013, we developed a verification system and contributed to the standardization of the scrambling subsystem and associated information subsystem for ultra-high definition television at the Information and Communications Council. The scrambling subsystem uses a cryptography algorithm on the transmission side, and the associated information subsystem uses a conventional conditional access system to manage the contract information in a way that is consistent with the access control functions of current services. For ease of maintenance and improvement of security, we also devised a downloadable conditional access system that sends the software updates to the receivers involved in processing associated information through broadcasting and telecommunications networks. These methods were reflected in a report issued by the Broadcasting System Committee of the Information and Communications Council. Cryptography algorithm for hybrid services We researched cryptography technologies to provide secure and reliable hybrid services taking advantage of broadcasting and telecommunications. Service providers need information about viewers in order to personalize services. Meanwhile, viewers should not receive unexpected services from unknown providers. An efficient way to meet these requirements is to store viewers information on a cloud. We studied an attribute-based encryption scheme (2) that encrypts the viewer s information to protect his/her privacy and allows only the providers meeting a policy set by the viewer based on attributes such as reliability of service provider to decrypt the information. By modifying the way of describing attributes, we found a way to use a smaller ciphertext and secret key. We evaluated the encryption on a computer. This research was conducted in cooperation with the Advanced Industrial Science and Technology. In our research on cryptography to protect content, we developed a method that can trace the owner of a decryption key that has been illegally copied. To deal with possible cryptanalysis, we developed a way to update the scrambling method that that will work on existing as well as new receivers without increasing the transmission volume. Interactive content technology for device linkage services We studied technologies to enable broadcast content to be shown on devices other than receiver equipped TVs Called Augmented TV, this technology captures images of the TV screen with a camera of a mobile terminal and processes the captured image on the mobile terminal. It provides a new viewing experience by displaying computer graphics (CG) of program-related characters on the mobile terminal in accurate synchronization with the TV screen. In FY 2013, we built a system to produce and present program and associated CGs in a virtual space displayed on a mobile terminal. The system works in an easy and integrated way by using TV program Making Language (TVML) (3). We also achieved frame-level precision synchronization (under 0.03 s) between the TV image and the mobile terminal by using a mobile terminal s camera to capture synchronization markers, graphic time data, on the broadcast content shown on the TV display. (1) C. Yamamura, Y. Nishimoto, A. Fujii: Development of an authorization system for license delivery control - application of its system to CAS/DRM -, ITE Annual Conference, 18-11 (2013) (in Japanese) (2) N. Attrapadung, G. Hanaoka, K. Ogawa, G. Ohtake, H. Watanabe, S. Yamada: Ciphertext-Policy Attribute-Based Encryption over Range Attributes, SCIS, 2E4-1 (2014) (3) H. Kawakita, H. Kaneko, S. Inoue, A. Fujii, T. Nakagawa: Production Platform of 3DCG Content Connected with TV Program - System Architecture for Augmented TV -, ITE Annual Conference, 2-5 (2013) (in Japanese) 3.1.4 IP content delivery technology With the aim of distributing programs over a network, we are researching a distributed server-based system for providing broadcast programs over a network and IP content delivery technologies for future hybrid services. Distributed server-based broadcasting system We are researching a distributed server-based broadcasting system for time-shift viewing of past programs over IP networks. The accumulation, reprocessing and distribution of broadcast content is handled by servers so that viewers can view any of the previously programs as they like. In FY 2013, we upgraded the time-associated zapping system prototyped in FY 2012 to enable program zapping by tags such as cast name, in addition to zapping by on-air time. We developed a database to manage tags associated with broadcast on-air time and incorporated it in a distributed file system capable of storing and retrieving video, sound and related information of programs using broadcast time as ID. The resulting acquisition of associated tags in synchronization with video and audio lets viewers zap quickly without stress. We also developed a program viewing player that shows the tags 20 NHK STRL ANNUAL REPORT 2013
For zapping by time For zapping by tag Broadcast Telecommunications Push Synthesized display on broadcast content Player name Hybridcast receiver Push delivery by high-precision synchronization Figure 1. Program viewing player for time-shift zapping system Program related information (Player name, etc.) associated with a program in addition to information for time zapping (Figure 1) and a system for collecting viewing logs and visualizing them by video frame. We will use this system to acquire viewing data categorized by broadcast on-air time and tag. In addition, we advanced with our development of a program viewing system that converts the resolution of programs stored in the distributed file system on the server side according to the display capabilities of the viewing terminal. Hybrid content distribution technologies We are researching a push content delivery method to enable synchronized distribution over communication channels of the information to be incorporated and displayed in each video frame of broadcast content (Figure 2). We began study in FY 2012 on a distribution method to control the timing of sending information on the distribution server so that the information arrives to many viewers at the same time. In FY 2013, we devised a distribution control method to minimize the difference in arrival times (jitter) of periodically distributed information in a quasi-network environment. The results of simulations showed that this method suppresses jitter in a large-scale distribution service (1). Figure 2. Example of Push content delivery method Hybrid P2P-CDN live streaming technology In light of the results of live streaming experiments at the London Olympics in 2012, we developed a hybrid live streaming system linking peer-to-peer (P2P) distribution and content delivery network (CDN) distribution (Figure 3). A distribution support server enables terminals not supporting P2P to receive live streaming video from a terminal supporting P2P. We also developed a method that uses this server to suppress differences in the video reproduction timing between terminals receiving P2P distributed content and the terminals receiving CDN distributed content. We evaluated the system implemented on a commercial cloud and confirmed that it worked (2). (1) S. Tanaka, S. Nishimura, Y. Endo: A Study of Determining Standard RTT in Push Content Delivery Method for Synchronization between Broadcast Programs and Push Contents, IEICE General Conference, B-6-38 (2014) (in Japanese) (2) S. Nishimura, S. Tanaka, Y. Endo: Development of a Hybrid P2P- CDN Live Streaming System, ITE Winter Annual Convention, 12-9 (2013) (in Japanese) CDN Distribution edge Live encoder Cache server Distribution support server P2P distribution server Reception request Request transfer Distribution status monitor Direct reception P2P stream P2P network Segment reception Non-P2P-compliant terminals P2P-compliant terminals Figure 3. Hybrid P2P-CDN live streaming system NHK STRL ANNUAL REPORT 2013 21