BT Media & Broadcast. Research Paper

Transcription

1 BT Media & Broadcast Research Paper

2 BT Media and Broadcast Research Paper Ultra High Definition Video Formats and Standardisation Version 1.0 April 2015 Mike Nilsson Page 1

3 Contents Page 1 Introduction 6 2 Enhanced Resolution 7 3 High Dynamic Range The Benefit of High Dynamic Range The dynamic range of the human visual system The non-linearity of the human visual system The mapping of linear light to code levels The mapping of pixel code levels back to linear light Black level: how dark should displays be? The mapping of pixel code levels to linear light in the presence of ambient light The current state of high dynamic range capture and display technology Interest and Experience in Hollywood Dolby Cinema 22 4 Wider Colour Gamut The CIE RGB Colour Space The CIE XYZ Colour Space Perceptually Uniform Colour Spaces Colour Television The need for a Wider Colour Gamut Wider Colour Gamut Standards Conversion between Colour Gamuts How large a Colour Gamut is needed? Display Technology for Wider Colour Gamut 31 5 Higher Frame Rate The artistic impact of frame rate Historical choices of frame rate The motion blur jerkiness trade-off Subjective evaluation of moving picture quality 35 Page 2

4 5.5 The impact of lighting frequency on frame rate 38 6 Standardisation ITU-R Parameters for Digital Television High Dynamic Range Colorimetry conversion Higher Frame Rates IN STRICTEST 41 CONFIDENCE 6.2 SMPTE UHD Parameter Values High Dynamic Range Colour Volume Metadata Digital Cinema Colour Equations On-going Activities MPEG High Efficiency Video Coding The Future of Video Coding Standardisation High Dynamic Range and Wide Colour Gamut DVB UHD-1 Phase UHD-1 Phase UHD-2 Eco-design requirements for electronic displays IN STRICTEST 49 CONFIDENCE EBU DTG Blu-ray Disk Association HDMI Forum The Digital Cinema Initiatives (DCI) The Forum for Advanced Media in Europe (FAME) UHD Alliance and UHD Forum 54 7 Abbreviations 55 Page 3

5 BT Media and Broadcast contacts Name Business Area / Role Phone John Ellerton Head of Media Futures john.ellerton@bt.com Jonathan Wing Head of Sales jonathan.wing@bt.com BT Research and Innovation contacts Name Business Area / Role Phone Mike Nilsson Research and Innovation mike.nilsson@bt.com Steve Appleby Research and Innovation steve.appleby@bt.com Page 4

6 Executive summary BT Media and Broadcast provides services to broadcast and media organisations worldwide, including the international carriage of Ultra High Definition Television signals. We commissioned our colleagues in BT Research and Innovation to examine the current international status of technology and standardisation of Ultra High Definition Television to aid in our own understanding and future product development. This paper is the result, which we were so impressed with, we decided to release as a resource to the professional broadcast community. We hope it is useful do get in touch if you IN have STRICTEST feedback or CONFIDENC comments. UHD televisions are now retailing in significant numbers, and services, such as those offered by Netflix and Amazon, are starting to appear in the market. But while these services offer higher resolution than HD services, further improvement could be made in due course to provide an even better viewing experience. Future television systems should be capable of producing an experience that is IN either STRICTEST closer to real CONFIDENCE life or is capable of more accurately recreating the artistic intent of the storyteller. To this end, increased resolution, wider colour palette, higher frame rate and an improvement in the dynamic range of the images when used together, have the potential to provide viewers with a better visual experience compared to current television applications and provide a viewer IN STRICTEST with a stronger CONFIDENCE sense of being there. Many standardisation organisations around the world have been, and are continuing to be, very active in the area of UHD TV standardisation. DVB has created standards for what it terms UHD-1 phase 1, effectively the parameters of HDTV but with enhanced resolution, similar to the deployments of Netflix and Amazon. DVB is now working on the commercial requirements for two subsequent phases, the first of which is expected to add support for higher dynamic range, wider colour gamut, and higher frame rates, and the second of which is planned to add support for even higher, 8K, resolution. The HEVC compression standard, almost essential to make delivery of UHDTV commercially feasible, has been approved, with the recently approved second version including support for higher bit-depths and enhanced chroma formats. MPEG is currently studying whether HEVC is optimal, as currently standardised, to support higher dynamic range and wider colour gamut, and if not, is expected to launch a standardisation activity during 2015 to address the deficiencies. There is widespread interest in backwards compatibility of future UHDTV services, which may have higher dynamic range, wider colour gamut, and higher frame rates, with first generation UHDTV services with only enhanced resolution. Technology to achieve this is still under development, and the ultimate success or failure of backwards compatibility will depend on how efficiently it could be implemented compared to simple but inefficient simulcasting. There are, at the writing, many problems and few agreed solutions, but there are many lively discussions taking place in the standards community could be the year in which significant advances are made to the technical standardisation of the second phase of UHDTV. This report begins with sections describing the science and technology of UHD television: enhanced resolution, higher dynamic range, wider colour gamut and higher frame rate. This is followed by a section describing the current state of standardisation in various bodies that are essential to the standardisation of UHD television services, ITU-R, SMPTE, MPEG and DVB, while also providing a brief status update on EBU, DTG, Blu-ray Disk Association, HDMI Forum, Digital Cinema Initiatives (DCI), the Forum for Advanced Media in Europe (FAME), the UHD Alliance and the UHD Forum. We consider that while the enhanced resolution of the first phase of deployments of UHDTV services could provide undoubtedly better picture quality than current HDTV services when viewed from an optimal viewing distance, the combination of human visual acuity, screen size, and home viewing distances could make the improvement over HDTV in the home environment less pronounced. We have observed the growing feeling within the standardisation community that to make UHDTV reach the mass market, additional features would be needed beyond the enhanced resolution achieved with the first phase of deployment. In particular, enhancements that that do not depend so critically on viewing distance would be beneficial. Fortunately these features, higher dynamic range, wider colour gamut, and higher frame rates, are becoming technologically IN STRICTEST feasible CONFIDENCE and are generating much interest within the industry. This has perhaps been most noticeable in the film industry. Dolby Cinema, which was announced in December 2014, and which is claimed to deliver high dynamic range with enhanced colour, is planned for theatrical exhibition in The Blu-ray Disc Association is developing a next generation format that will support UHD, a wide colour gamut, and high dynamic range, with the first players and titles being expected before the end of The broadcast TV industry is close behind, with DVB aiming to develop specifications to enable deployment of second phase UHD services in the timeframe. Page 5

7 1 Introduction Ultra High Definition TV has attracted a good deal of attention lately as a result of a push from TV vendors and some content providers, notably Netflix. However, to date, the focus has been solely on increased resolution. Ultra High Definition though, is about more than just increasing the number of pixels, being also about increasing the dynamic range, widening the colour gamut and increasing the frame rate. When viewing TV on a popular size of screen in a typical home environment, it is these additional improvements which may ultimately provide the most benefit to the viewer. The quality of television is therefore set to improve in multiple dimensions over the next few years. There is still heated debate about the standards that will be used, but it seems clear that managing legacy and dealing with multiple dimensions of TV improvements will be a challenge for the broadcast industry. The purpose of this document is to provide a thorough review of the current status IN STRICTEST of UHD standards CONFIDENCE with a view to obtaining a clearer understanding of the challenges and opportunities ahead. Page 6

8 2 Enhanced Resolution Display and capture technology is advancing. Today there are both video cameras and displays capable of resolutions higher than full High Definition TV. These resolutions are frequently referred to as either 4K or Ultra High Definition TV (UHDTV) and have resolution 3840x2160, that is, four times the resolution of full High Definition TV. Figure 1 shows the resolution of UHD relative to the earlier formats of Standard Definition (SD), HD-ready (720p HD), and High Definition (HD). Figure 1. The resolution of UHD relative to earlier, lower, resolutions. UHD televisions are now retailing in significant numbers 1, enabling potentially much better picture quality than current HD televisions. BT worked with the BBC during 2014 to deliver coverage of the 2014 football World Cup Final in UHD resolution live to the BT Tower, using HEVC video compression and MPEG DASH technology, delivering content from BT Wholesale Content Connect over BT Infinity superfast fibre optic broadband through BT Home Hub 5 routers to three set top boxes connected to UHDTVs and directly to two internet-enabled UHDTVs. The picture quality achieved was very impressive. The quality of the UHD signal was undoubtedly much better than HD delivered by Freeview, when viewed near to the screen. The ITU-R describe optimal viewing distance in image heights (H) for various digital image systems 2, recommending 6H for Standard Definition, 4.8H for 720p HD, 3.2H for HD, and 1.6H for UHD. Such optimal viewing distances are often calculated by considering normal human visual acuity to be 20/20, meaning that a letter, such as the letter E, can be identified when top to bottom it subtends an angle IN of STRICTEST 5 arc minutes. CONFIDENCE This corresponds to 1 arc minute per horizontal feature (three lines and two spaces). It also corresponds to one dark to light cycle every 2 arc minutes. One dark to light cycle every 2 arc minutes equals 30 cycles per degree, which can be represented with 60 pixels per degree. This has been taken as standard acuity for TV applications, although clearly some viewers have worse eye sight and some have better eye sight. The screen diagonal size, S, measured in inches, required to achieve a resolvable number of horizontal pixels, R, when viewed from a distance D, measured in centimetres, is given by the following equation, assuming visual acuity of 60 pixels per degree and 16:9 picture aspect ratio. 1 ( ) 2 "Guidelines on metrics to be used when tailoring television programmes to broadcasting applications at various image quality levels, display sizes and aspect ratios", Recommendation ITU-R BT (03/2010), IN STRICTEST CONFIDENCE I Page 7

9 However, there is reason to believe that viewers will not sit as close as 1.6H to IN their STRICTEST television, and CONFIDENCE hence would not achieve the ITU-R optimal viewing experience. A survey 3 of 102 BBC employees in 2004 reported that the median viewing distance for viewing the main TV in the home was 2.7m. The BBC carried out a more extensive survey of the UK population during the summer of , collecting information from 2185 people who have a television in their home, and finding the median viewing distance was 2.63m. While the 2004 survey found a median screen height of 32.5cm, the median height from the 2014 survey was 49cm, which, assuming a 16:9 picture aspect ratio, corresponds to a screen diagonal size of 39.3 inches. The 2014 survey also found that the median relative viewing distance is 5.5 times the screen height (5.5H). The 2014 survey, by asking respondents how they would expect the size of their next television to compare to that of their current one, found that about half the respondents expected to buy a larger screen next time. When asked to estimate the ideal television size for their current home, assuming that money were no object, they responded with a median ideal diagonal size of 48 inches. By assuming that people would need to sit three picture heights (3H) or closer to a UHD screen to get a resolution improvement over HD, the BBC deduced from the 2014 survey that 10.2% of viewers would be able to observe a benefit from UHD with their current television, and 22.9% would if they acquired their ideal television. These two surveys show no significant change in television viewing distances over a period of 10 years. As this is likely to be significantly influenced by the size of rooms in UK homes, it would seem reasonable to expect little change in the coming years. Currently 55 inch and 65 inch diagonals are popular for UHD televisions. There would appear little reason to believe that this would change in the near future, as it is in part determined by the size and layout of rooms in which people watch television. A viewing distance of 1.6H, which the ITU-R consider to be optimal for viewing UHD content, when calculated for a television with 65 inch diagonal, corresponds to a viewing distance of about 1.3m. It seems unlikely given the results of the two BBC surveys that many people would sit this close to their UHDTV. Table 1 shows the resolution of various popular TV formats and the screen size that would be needed for optimal viewing at a distance of 2.63m, the median viewing distance found in the 2014 BBC survey. It is unlikely that viewers would want to, and be able to, acquire screens with diagonal size of 148 inches, so that they could both maintain STRICTEST a viewing CONFIDENCE distance of 2.63m and achieve optimal, as defined by the ITU-R, viewing of UHD content. While some viewers may compromise on viewing distance and screen size to get such a UHD experience, it seems likely that the majority would not, and hence they may not get the full benefit of UHDTV resolution. Format Horizontal Resolution Screen Size (Inches) SD p HD HD 1920 IN STRICTEST 68 CONFIDENCE UHD Table 1. The screen size needed for optimal viewing of popular TV formats from 2.63m. The screen would not need to be so large if the requirement were relaxed from achieving optimal viewing of UHD content to achieving a better experience than watching HD content. Again referring to Table 1, any screen with diagonal size larger than 68 inches would enable more resolution than HD to be observed from 2.63m. However, there is some disagreement with this whole methodology. Spencer 5 has reported results of subjective tests in which viewers assessed content from a distance of 30cm, controlled using a head mount. By using the methodology described above, the 3 Results of a survey on television viewing distance, N. E. Tanton, June A Survey of UK Television Viewing Conditions, Katy C. Noland and Louise H. Truong, January 2015 Page 8

10 viewers should have been limited to seeing benefits as the pixel density increased IN STRICTEST to about 300 pixels CONFIDENCE per inch, and then seeing no further improvement as the pixel density was increased. However, the presented results show improvements at 500 pixels per inch, and yet further improvements at 1000 pixels per inch. Spencer explains these results by stating that the human eye is able to detect differences represented by even finer structures, referencing tests of vernier acuity, which involve distinguishing the relative alignment of parallel lines, that show that humans can distinguish details five to ten times smaller than standard resolution measurements predict. While this is an isolated study, it does suggest that further investigation of the benefits of the increased resolution of UHDTV, when viewed on televisions of desirable size at normal home viewing distances, would be worthwhile. 5 How much higher can mobile display resolution go?, Lee Spencer, Page 9

11 3 High Dynamic Range The human visual system has a significantly larger dynamic range than that supported by current television systems. Display devices that are able to support a much larger dynamic range than conventional televisions systems are starting to appear in the market. They achieve the larger dynamic range by extending the range both towards darker as well as lighter values as a result of fitting a conventional LCD panel with a spatially varying backlight, which could be either a projector or a panel of LEDs which are individually addressable. A key technical challenge for video delivery is how to signal the higher dynamic range while also supporting legacy standard dynamic range displays. Some proprietary technology is already entering the market, and various standardisation bodies, including SMPTE, the EBU and MPEG are considering the issues. This section provides an overview of the benefits of using a higher dynamic range IN STRICTEST for images and CONFIDENCE video, the capability of the human visual system, and the capability of emerging display technology. Additionally, the issues and proposed solutions for integrating higher dynamic range into existing end to end television systems are discussed. 3.1 The Benefit of High Dynamic Range Figure 2 is an example of an image where the use of a higher dynamic range would IN STRICTEST improve the CONFIDENCE viewing experience. With a standard dynamic range, it is not possible to have visible details in both the shaded parts of the image and the parts that are in full sunshine. Figure 2. An image that would benefit from high IN STRICTEST dynamic range. CONFIDEN The use of a high dynamic range is not simply making images brighter, which in itself can be effective, but enabling detail to be seen in both dark and light areas. Figure 3 shows an example of a histogram of the pixel luminance values of an image, represented with both standard and high dynamic range. Many of the pixels have about the same luminance in both cases, and only some pixels, the lowlights and the highlights in the image, have lower or greater luminance in the high dynamic range variant. Luminance is measured in the derived SI units of candela per square metre (cd/m 2 ), also known as the nit (1 nit = 1 cd/m 2 ). Typically displays are limited to a dynamic range of 100:1, and a typical maximum luminance of 100 cd/m 2 due to legacy limitations of CRT display technology and the derived current specification for display interfaces. The human visual system, as described later, is capable of perceiving a much higher dynamic range and higher levels of luminance. Figure 4 illustrates the wide range of luminance values that occur in real world scenes, labelling specific features with approximate values of light level. Page 10

12 Relative Frequency Luminance (cd/m2) Standard Dynamic Range Image High Dynamic IN STRICTEST Range Image CONFIDENCE Figure 3. An example pixel luminance histogram for a standard and high dynamic range image. 4,000 nits 10,000 nits 50,000 nits 100 nits 40 nits 2 nits Figure 4. Examples of approximate light levels in real world scenes. 3.2 The dynamic range of the human visual system The human visual system has a huge dynamic range, being able to cope with starlight at 10-4 cd/m 2 and bright sunshine at 10 5 cd/m 2. Hood and Finkelstein 6 report values ranging from 10-6 to 10 cd/m 2 for scotopic IN STRICTEST light levels CONFIDEN where light transduction is mediated by the rod cells in the eye, and from 0.01 to 10 8 cd/m 2 for the photopic range where the cone cells are active. And in the overlap, called the mesopic range, both rods and cones are involved. At any one time, the human visual system is able to operate over only a fraction of this enormous range. This subset is called the simultaneous or steady-state dynamic range. It shifts to an appropriate light sensitivity due to various mechanical, photochemical and neuronal adaptive processes 7, so that under any lighting conditions the effectiveness of human vision is maximised. 6 From D. C. Hood and M. A. Finkelstein, Visual sensitivity, in Handbook of Perception and Human Performance, K. Boff, L. Kaufman, and J. Thomas, Eds., vol. 1. Wiley, New York, J. A. Ferwerda, Elements of early vision for computer graphics, IEEE Computer IN Graphics STRICTEST and Applications CONFIDENCE 21, 5, 22-33, luthuli.cs.uiuc.edu/~daf/courses/rendering/papers3/ pdf. Page 11

13 The simultaneous dynamic range over which the human visual system is able to IN function STRICTEST can be CONFIDENCE defined as the ratio between the highest and lowest luminance values at which objects can be detected, while being in a state of full adaptation. In a given room that contains a display device, the human visual system will typically be in a steady state of full adaptation. Figure 5, taken from Hood and Finkelstein, shows the overall range of the human visual system compared to the ranges of its approximate steady state as well as the range for typical Low Dynamic Range (LDR) and High Dynamic Range (HDR) displays. While the overall dynamic range of the human visual system and these display devices is known, less is known about the range of the human visual system in the steady-state. Kunkel and Reinhard 8 have reported a sequence of psychophysical experiments, carried out with the aid of a high dynamic range display device, where they determined the simultaneous dynamic range of the human visual system, finding the human visual system to be capable of distinguishing contrasts over a range of 3.7 log units, equivalent to a range of about 1:5000, under specific viewing conditions. They also found that the dynamic range is affected by stimulus duration, the contrast of the stimulus and the background illumination, which they claimed accounts for the different dynamic ranges being reported in the literature. Figure 5. The dynamic range of the human visual system. Jenny Read, neuroscientist at the University of Newcastle, talked at DVB-EBU HDR Workshop in June on how the Human Visual System reacts to high dynamic range screens, stating that encoding schemes must be defined by taking into account human contrast sensitivity. It is also necessary to take into account the human visual system adaptation states, where light to dark adaptation is slow, typically taking between 8 and 30 minutes, and where dark to light adaptation is much quicker, typically taking just a few minutes. Daly et al 10 report studies to find the dynamic range that is preferred by human observers. They used a Dual Modulation Research Display, as shown in Figure 6, and referenced by Hammer 11, which is capable of supporting a dynamic range from cd/m 2 to 20,000 cd/m 2, that is, a contrast ratio of 5,000,000:1; and which supported the DCI P3 colour gamut. The experiment used several realistic and synthetic stimuli in a dark viewing environment. They found for diffuse reflective images a dynamic range between 0.1 and 650 cd/m 2 matched the average preferences, but to satisfy 90% of the population, a dynamic range from to about 3000 cd/m 2 is needed. They claim that since a display should be able to produce values brighter than the diffuse white maximum, as in specular highlights and emissive sources, they conclude that the average preferred maximum luminance for highlight reproduction satisfying 50% of viewers is about 2500 cd/m 2, increasing to marginally over cd/m 2 to satisfy 90%. They conclude that there would be a benefit from more capable displays, as the preferred luminances found in their study exceed even the best of consumer displays today. 8 T. Kunkel and E. Reinhard, A reassessment of the simultaneous dynamic range of the human visual system, Proceedings of the 7th Symposium on Applied Perception in Graphics and Visualization, ISBN , pp , July DVB-EBU HDR Workshop, IRT, Munich, Germany, 17 June 2014, 10 S. Daly, T. Kunkel, S. Farrell & Xing Sun: Viewer Preferences for Shadow, Diffuse, Specular, and Emissive Luminance Limits of High Dynamic Range Displays. SID Display Week Hammer, High-Dynamic-Range Displays, Page 12

14 Figure 6. The experimental set up used by Daly et al. to assess preferred dynamic range. Hanhart, Korshunov, and Ebrahimi 12 report a set of subjective experiments to investigate the added value of higher dynamic range. Seven test video sequences at four different peak luminance levels were assessed using the full paired comparison methodology. Pairs of the same sequence with different peak luminance levels were displayed side-by-side on a Dolby Research HDR RGB backlight dual modulation display (aka Pulsar ), which is capable of reliably displaying STRICTEST video content CONFIDENCE at 4000 cd/m 2 peak luminance. Their results, as shown in Figure 7, show that the preference of an average viewer increases logarithmically with the increase in the maximum luminance level at which the content is displayed, with 4000 cd/m 2 being the most attractive option of those tested. Figure 7. Results of subjective evaluation of HDR Video using pair comparison by Hanhart, Korshunov, and Ebrahimi. 3.3 The non-linearity of the human visual system Ernst Heinrich Weber studied of the human response to a physical stimulus in a quantitative fashion, finding, in what is now known as Weber's law, that the just-noticeable difference between two stimuli is proportional to the magnitude of the stimuli, that is, an increment is judged relative to the previous amount. Gustav Theodor Fechner used Weber's findings to construct a psychophysical scale in which he described the relationship between the physical magnitude of a stimulus and its (subjectively) perceived intensity. 12 Subjective Evaluation of Higher Dynamic Range Video, Philippe Hanhart, Pavel Korshunov, and Touradj Ebrahimi, SPIE Optical Engineering + Applications, Applications of Digital Image Processing XXXVII. Page 13

15 Fechner's law states that subjective sensation is proportional to the logarithm of IN the STRICTEST stimulus intensity. CONFIDENCE Fechner scaling has been found to apply to the human perception of brightness, at moderate and high brightness, with perceived brightness being proportional to the logarithm of the actual intensity 13. At lower levels of brightness, a more accurate description is given by the de Vries-Rose law which states that the perception of brightness is proportional to the square root of the actual intensity. 3.4 The mapping of linear light to code levels Current television systems typically support content with a range of brightness from about 0.1 cd/m 2 to about 100 cd/m 2. Consequently, it is the de Vries-Rose characteristic, described above, that has been designed into television systems to date, where a function known as gamma or gamma correction, is used to map brightness (linear light) to a scale in which each increment corresponds to a constant perceptual change. This function is known as the Opto-Electrical IN STRICTEST Transfer CONFIDENCE Function (OETF). In analogue television systems this characteristic made the whole range of intensities equally sensitive to noise, whereas in digital television systems it enabled the number of bits needed to represent each sample to be minimised 14. This characteristic is shown in Figure 8, where mappings, as specified in ITU-R Recommendation BT.709, to both 8 and 10 bit code levels are shown. The use of 10 bit codes, as increasingly used in video production, gives little benefit over 8 bits if BT.709 is still used, as the extra bits are the least significant bits, which are often discarded for display on 8 bit devices, and which at best could reduce the minimum black level, that is, make the blacks even blacker. While using BT.709, the extra two bits cannot increase the brightness, even though this would often be more desirable than increasing the maximum darkness. ITU-R Recommendation BT.2020, a more recent specification than BT.709, defines higher frame rates and a wider colour space, but the same OETF as BT.709. These and other ITU-R Recommendations are discussed in further later in this research paper. The dynamic range of a video signal can be increased by using an alternative Opto-Electrical Transfer Function (OETF) to that specified in BT.709. Digital Imaging and Communications in Medicine (DICOM) is a standard for handling, storing, printing, and transmitting information in medical imaging. This includes in Part an Opto-Electrical Transfer Function STRICTEST from linear CONFIDENCE light to 10 bit code levels, also shown in Figure 8, and valid between 0.05 and 4000 cd/m2 whereby each 1-bit luminance increment is equally visible according to Barten 16. Jenny Read stated at the DVB-EBU HDR Workshop in September 2014 that the film industry has used log-curves for many years that are quite near the human visual system and are much more efficient than the BT.709 Transfer Function used for TV today. A logarithmic OETF is used to map 14 stops of linear light, that is, a dynamic range of about 16,000:1, to a 10 bit signal, and is satisfactory to capture the Cineon format. Miller et al 17 of Dolby Laboratories have proposed a new OETF to provide higher dynamic range for video and movie production and distribution, which has recently been standardised by the Society of Motion Picture Engineers as SMPTE ST 2084:2014, High Dynamic Range Electro-Optical Transfer Function of Mastering Reference Displays, which is described later in this research paper. This is also based on the work of Barten, and is also shown in Figure 8. But there is no industry consensus on the relevance of the work of Barten. The BBC in its white paper argues that Miller s proposal links the camera OETF to the absolute brightness of the display and that this has potentially far reaching consequences. 13 Weber Fechner law, 14 C. A. Poynton, Digital Video and HDTV: Algorithms and Interfaces, Electronics & Electrical, Morgan Kaufmann series in computer graphics and geometric modelling, ISBN , Barten, P.G.J., Physical model for the Contrast Sensitivity of the human eye. Proc. SPIE 1666, (1992); Barten, P.G.J., Spatiotemporal model for the Contrast Sensitivity of the human eye and its temporal aspects. Proc. SPIE (1993); and P. G. J. Barten, Formula for the contrast sensitivity of the human eye, Proc. SPIE-IS&T Vol. 5294: , Jan Scott Miller, Mahdi Nezamabadi and Scott Daly. Perceptual Signal Coding for More Efficient Usage of Bit Codes. SMPTE Motion Imaging Journal : doi: /j BBC White Paper WHP 283, Non-linear Opto-Electrical Transfer Functions for IN High STRICTEST Dynamic Range CONFIDENCE Television, T. Borer, July Page 14

16 Linear Luminance (cd/m2) The BBC argues that previously the photographic, movie and television industries IN STRICTEST have all always CONFIDENCE worked with relative, rather than absolute, luminance levels. The BBC argues that changing to absolute luminance levels will require significant changes to the way television is produced and viewed; and that it is not clear that the large dynamic range of 10 7 is needed for image display, when the simultaneous dynamic range of the eye is about They have developed their own proposal for an OETF, as shown in Figure 8, and which is imagined to be consistent with their submissions to ITU-R WP6C. There is on-going discussion of high dynamic range in standardisation bodies. The formal standardisation process for high dynamic range signal and exchange format, including the consideration of suitable Opto-Electrical Transfer Functions, is on-going in ITU-R WP6C (RG-24) and in SMPTE 10-E as described later in this report. MPEG is currently investigating the effectiveness of existing compression technologies with high dynamic range video, and the means to support standard and high dynamic range video within a single coded representation BT.709 with 8 bit code levels BT.709 with 10 bit code levels DICOM Part 14 Perceptual Quantiser, SMPTE ST 2084:2014 BBC HDR Proposal - 10 bit code levels Code Levels 3.5 The mapping of pixel code levels back to linear light Figure 8. The mapping of linear light to code levels. While Cathode Ray Tubes (CRT) were the only or most popular display device for IN television, STRICTEST there CONFIDENCE was no need to specify an inverse to the OETF, known as the Electro-Optical Transfer Function (EOTF), as the inverse function was provided implicitly from the physics of cathode ray tubes. Fortunately, this function was well matched to the human visual system. It was only in 2011, by which time the consumer market for CRTs had almost completely disappeared, that a standard for an EOTF, ITU-R Recommendation BT , was finally agreed. This effectively documents the characteristics of CRT displays. The EOTF is not necessarily best defined as the inverse of the OETF: the combination of the OETF and the EOTF yields the total transfer function, sometimes known as the end-to-end gamma or system gamma. Some believe that this total transfer function should be a power law function with the exponent value dependent on the lighting condition, for example, with values of 1.0, 1.25, and 1.5 being appropriate for bright, dim, and dark surrounding environments 19 IN. The fundamental basis for interpreting any visual signal is the transfer function, the description of how to convert the signal, that is, the digital code values, to optical energy. It is therefore this EOTF and not the OETF IN STRICTEST that truly defines CONFIDENC the intent of visual signal code values. The vast majority of content is colour graded (either live in the camera, or during post production) according to artistic preference while viewing on a reference standard display. 19 Report ITU-R BT , (03/2014), The present state of ultra-high definition IN television. STRICTEST CONFIDENCE Page 15

17 The EOTF, when applied to code values with a given bit depth, 8, 10 or 12 bits etc., IN STRICTEST should ideally CONFIDENCE avoid the viewing of discontinuities in tone reproduction, and hence the EOTF and bit depth should be matched to the contrast sensitivity function of the human visual system. A well respected model for the contrast sensitivity function of the human visual system was developed by Peter Barten 16, and has been referenced by many electronic imaging studies and standards. This complex model, based on physics, optics, and some experimentally determined parameters, has been shown to align well with many visual experiments spanning several decades of research. Dolby Laboratories have used the Barten model directly to compute an optimized perceptual EOTF 17. This function is defined in Figure 9. This is defined in terms of absolute luminance levels viewed on the display screen, not in terms of absolute luminance at capture. This Perceptual Quantiser curve has nearly a square root behaviour (slope = -1/2) at the darkest light levels, consistent with the Rose-DeVries law, and then rolls off to a constant zero slope for the highest light levels, consistent with the log behaviour of Weber s law. Between those extreme luminance regions, it exhibits varying slopes, IN STRICTEST and throughout CONFIDENCE the mid luminance levels it exhibits a slope similar to the gamma non-linearities of BT ( ) Figure 9. The EOTF derived from the perceptual quantiser function, as proposed Dolby. Dolby Laboratories reported 17 the results of subjective tests with real images with peak luminance up to 600cd/m 2, comparing BT.1886 scaled for a peak luminance of 1000cd/m 2 with the perceptual quantiser based EOTF scaled to 1000cd/m 2, and scaled to 10,000cd/m 2. They reported finding that with both variants of the perceptual quantiser, 10 bits of bit depth were sufficient to avoid visible quantisation steps on all eight test images, whereas the use of BT.1886 required less bits for light (white) images, typically one bit less of bit depth, but needed more bits for the dark (black) images, including needing more than 12 bits of bit depth for one image. These results are consistent with the known limitations of BT.1886 that it has greater precision for lighter regions than darker regions, and effectively wastes code values at the light end, and does not have enough at the dark end. It is generally thought that it would be difficult to operate legacy infrastructures with bit depth greater than 12 bits, and most live production and broadcast environments still operate at the 10 bit level. Hence it is commonly agreed that applying BT.1886 to a higher dynamic range is not feasible as the bit depth required would be too high. However, there is no consensus that the perceptual quantiser proposed by Dolby Laboratories is the best solution: there are concerns about its being defined in terms of absolute light levels, and other functions have been proposed, as are discussed elsewhere in this report. 3.6 Black level: how dark should displays be? How dark or black a region of a display can appear depends on two factors: the minimum emission from the display and the amount of ambient light that is reflected. The effective display black level, L black, can be calculated, as in the equation below, as the sum of the display minimum light emission, L min, known as dark current in the days of CRT, IN STRICTEST and meaning CONFIDEN the lowest level of luminance that comes out of the display; and a product of the display screen reflectivity, R display, and the ambient light level, E ambient. The impact therefore of higher levels of ambient light is to raise the minimum black level, and consequently to reduce the dynamic range of the image, as the maximum intensity is mostly unchanged with ambient light. ( ) Mantiuk et al. 20 have reported an experiment to determine the highest luminance level that cannot be discriminated from absolute black as the surrounding luminance is varied. They showed viewers a patch in the centre of a screen with absolute black on one side of the patch and non-zero luminance on the other, and asked viewers to choose the side that was brighter, or choose randomly 20 Mantiuk, R. and Daly, S. and Kerofsky, L. (2010), The luminance of pure black: exploring the effect of surround in the context of electronic displays. In: Proc. of Human Vision and Electronic Imaging XXI, IS&T/SPIE's IN STRICTEST Symposium CONFIDENCE on Electronic Imaging. Page 16

18 Black Level (Logarithm to the base 10 of values measured in cd/m2) if they looked the same. Different values of non-zero luminance and of surrounding IN STRICTEST luminance were CONFIDENCE tested. Two viewing distances were used, 1.4m and 4.7m, so that the size of the square patch corresponded to 6.1 and 1.8 visual degrees. They converted these results of just detectable differences from absolute black so they could be plotted as a function of ambient light rather than the luminance of surrounding pixels. The results are shown in Figure 10, with the experimental results labelled as HVS Small Patch and HVS Larger Patch. It can be seen that as the ambient illumination is increased, the lowest level of black that can be distinguished from absolute black increases. Also shown in Figure 10 is the black level of a black diffuse surface with reflectance of 3%, such as black velvet, and the performance of three displays: a CRT with minimum light emission of 1cd/m 2 and 3% reflectance; a conventional CCFL (cold cathode fluorescent lamp) backlight LCD with minimum light emission of 0.8cd/m 2 and 1% reflectance; and a modern LED-backlight LCD with spatially uniform back-light dimming with minimum light emission of cd/m 2 and 1% reflectance.1.3/2.4 The CRT appears grey compared to the diffuse black (velvet) for ambient light below 300lux (about 2.5 on the horizontal axis of Figure 10), a level of brightness found in an office or a very well lit room in a home 21. For the CCFL-LCD, this threshold is 100lux (2.0 in Figure 10), typical of a room in a home. This is because the display effective black level is higher than the luminance of a diffuse black surface, due to a combination of reflectance and the minimum light emissions of the displays. The experimental results indicate that the eye can appreciate even deeper black than a diffuse black surface, and that of the considered display technologies, only the LED-LCD display can satisfy the demands of the human visual system, and only at levels below about 1.6lux (0.2 in Figure 10, where the LED-LCD curve crosses the HVS IN Larger STRICTEST Patch curve), CONFIDENCE an indoor illumination level that could be considered near pitch black. The problem is not the minimum light emissions which are very low, but the reflectance of ambient light from the screen. It appears unlikely that very low reflectance coatings will be possible, but this is unlikely to matter, as a reflectance of about 1% is likely to acceptable to almost all viewers as there are not many objects in the real-world that would have lower reflectivity and thus appear darker than a display CRT CCFL-LCD LED-LCD Diffuse Black HVS Small Patch HVS Larger Patch Deep Twilight Dark Room Home Office Ambient Luminance (Logarithm to the base 10 of values measured in Lux) Figure 10. Comparison of the black levels of displays and human detection capability. 3.7 The mapping of pixel code levels to linear light in the presence of ambient light Although standards including BT.1886 and IEC (srgb 22 ) specify the mapping of pixel code values to light levels, the actual perceived light level on a screen also depends on the ambient light and the reflectivity of the screen Page 17

19 Pixel Luminance (Logarithm to the base 10 of values measured in cd/m2) Mantiuk et al. 23 have developed a display model that combines the standardised IN gamma STRICTEST mapping CONFIDENCE with a factor to include the reflections of ambient light. This display model, shown in the equation below, includes the maximum brightness and dynamic range of the display device, and the viewing conditions, the amount of the ambient light that is reflected from the screen. Such reflected light increases the luminance of the darkest pixels shown on the display, thus reducing the available dynamic range. ( ) ( ) ( ) ( ) They claim that most CRT and LCD displays can be modelled with this equation, where the displayed luminance, L d, is calculated as a function of the luma (pixel value), L, and the display gamma, γ, the peak display luminance, L max, the display black level, which is the luminance of a black pixel displayed in a perfectly dark room, L black, the display screen reflectivity, R display, and the ambient light level, E ambient. Figure 11 shows the mapping of pixel code levels to linear light in the presence of different levels of ambient light using this equation from Mantiuk et al. for a display with peak display luminance 80cd/m 2, display black level of 1cd/m 2, reflectivity of 1% and gamma of 2.2. It can be seen that the effective dynamic range of a display gets compressed due to screen reflections, making lower pixel values almost indistinguishable. The dynamic range of nearly 2600:1 of srgb is reduced IN to STRICTEST only 75:1 in a CONFIDENCE dimly lit room, and to only 6:1 in sun light srgb : Dynamic Range 2584:1 Dim Room (20 lux) : Dynamic Range 75:1-1.0 Light Room (150 lux) : Dynamic Range 54:1 Sun light (5000 lux) : Dynamic Range 6: Luma pixel value (0.005 to 1.000) Figure 11. The mapping of pixel code levels to linear light in the presence of ambient light. Mantiuk et al. addressed this issue with adaptive tone mapping. As ambient light increases, the image gets brighter to avoid dark tones, which are the most affected by the display reflections. For the outdoors illumination, many of the bright pixels are clipped to the maximum value. An example of the actual result of their tone mapping, and how it differs for different ambient light conditions, is shown in Figure R. Mantiuk, S. Daly and L. Kerofsky, Display Adaptive Tone Mapping, ACM Transactions IN STRICTEST on Graphics, CONFIDENCE Vol. 27, No. 3, Article 68, Publication date: August Page 18

20 Figure 12. An image tone-mapped for three different ambient IN STRICTEST illumination conditions. CONFIDENCE Kunkel and Daly 24 stated that current displays are very thin and highly reflective perpendicular to the display: they are similar to mirrors! The earlier LCD displays had a matte, diffuse effect, with relatively low reflectivity, but the matte caused some image blur. Glossy LCDs eliminate the blurring effect, but at the expense of more reflections, making it important to position the screen and the viewer to minimise light sources affecting the experience. While this is controllable to some extent in the home environment, it is not so for outdoor use and signage. They state that as screens get brighter and brighter, the reflection of the viewer, illuminated by the light from the display, becomes more of an issue, but screen manufacturers may be reluctant to address this issue in the short term because such a TV looks good when turned off: the current design popular for marketing. Sharp have developed technology to minimize screen reflection, which they refer STRICTEST to as moth CONFIDENCE eye 25. This technology, showcased in October 2012 at CEATEC, Japan's largest consumer electronics show, involves applying an anti-reflecting coating to LCD panels based on technology similar to the nanostructure of a moth s eyes: the surface of a moth's eyes is covered with bumps and valleys that absorb oncoming light, enhancing night vision. Unlike conventional anti-reflection technology, Sharp s claimed its new LCD offers more vivid colour images and higher contrast. It demonstrated 60, 70 and 80 inch moth eye panels at CEATEC based on its Aquos large-screen TVs. Sharp said its panel technology is ready for deployment in commercial products for indoor use. But Kunkel and Daly comment that although moth-eye displays are diffuse, with even lower reflectivity than the earlier matte screens, they currently suffer from not being scratch resistant. Thicker displays were demonstrated at CES Kunkel and Daly claim these allow better audio because larger speakers can be included, and that they also allow for better backlight modulation. They also comment on curved screens, stating that curvature may help with screen surface side effects, and may enhance immersion, and that curved displays may be good for mobile use as they may allow the user to more easily avoid reflections from external light sources. 3.8 The current state of high dynamic range capture and display technology Modern digital motion imaging sensors can originate linear video signals having dynamic ranges up to about 14 stops, that is, ranges up to about 16,000:1. This dynamic range is similar to the simultaneous dynamic range of the human visual system 18. Such cameras include the Arri Alexa and Amira, various models by Canon including the EOS C300 and EOS C500, the Red Epic, and various models by Sony including the A7S and the F65. Cinema5D have reported results of tests carried out on a selection of cameras 26. They used the DSC labs XYLA-21, a high quality LEDbacklit transmissive chart that displays 21 stops of dynamic range: each vertical IN bar STRICTEST in the chart represents CONFIDENCE one stop of light. The chart is filmed with each camera in turn in a completely dark room using the same very sharp Zeiss 50mm CP2 T/2.1 makro lens with interchangeable mount adjusted for the camera bayonet. Each camera was set to its native ISO and the F-stop of the lens was 24 Timo Kunkel and Scott Daly, Dolby Labs, Inc., SMPTE Monthly Webcast: Lessons in Light: From Reality via Display to the Eye Page 19