High Definition Display System Based on Digital Micromirror Device

Transcription

1 High Definition Display System Based on Digital Micromirror Device Robert J. Gove, Vishal Markandey, Stephen W. Marshall, Donald B. Doherty, Gary Sextro, Mary DuVal Digital Imaging, Texas Instruments Inc. Dallas, Texas This paper describes a high definition display system based on the Digital Micromirror Device (DMD), a spatial light modulator developed at Texas Instruments. The system was designed to provide a rapid prototyping environment for the development and evaluation of system technology for DMD based high definition displays. The system design was based on the Scan-Line Video Processor (SVP), a programmable video processor developed at Texas Instruments. A multiple SVP parallel processing architecture was designed to handle the computational requirements of high definition video processing algorithms. Algorithms that were developed and implemented on this system include progressive scan conversion, scaling, degamma, picture controls, and image bit plane modulation for digital display. 1. Introduction The DMD is a new kind of semiconductor technology that combines electronic, mechanical, and optical functionality to create an all digital display. This technology has demonstrated the potential to provide a compelling alternative to competing technologies such as CRTs and LCDs, in applications including standard television displays, computer monitors, motion pictures, and high definition displays. Several display systems based on this technology have been developed and successfully demonstrated, including a high definition display system. Publications from Texas Instruments provide details of DMD semiconductor technology [3][7], as well as display system development [2][5][8]. The high definition display system was designed as a flexible prototype that provides an effective platform to develop and evaluate video processing algorithms, as well as to judge the impact of DMD technology in the display of high definition video. Figure 1 shows a functional block diagram of the system. Three DMDs (R/G/B) of resolution 2048x1152 pixels per DMD are used. Figure 1 Figure 1 shows a functional block diagram of the high- definition display system prototype utilizing Texas Instruments' innovative DMD technology

2 The number of pixels actually used for display varies depending on the source format. Table 1 provides a listing of display resolutions used for various video formats. The system is capable of displaying data derived from the following formats: SMPTE 240 & 260, NTSC, PAL, SECAM, Wide Video formats, and VGA. Table 1 provides a listing of display resolutions used for various video formats In Table 1, the column entitled "Source Line Resolution" shows the number of active lines of video obtained from the source for each format. This data is modified in our system to create the sizes listed in the column titled "DMD Display Resolution". Typically, this modification involves A/D conversion at appropriate rates to achieve the number of pixels per line needed, and vertical interpolation in the signal processing part of the system to obtain the needed number of data lines. While the SMPTE signals are input as Y/Pr/Pb data, the other signals can be Composite, S-Video, or RGB. These signals are decoded if necessary in the front end video interface. The analog signals are digitized and subjected to signal processing operations such as motion adaptive progressive scan conversion, scaling, color space conversion, and degamma (degamma is required because unlike CRTs, DMDs are linear displays with no inherent gamma characteristics). The formatter translates the R/G/B data into bit plane level data which is then used to drive the DMD display using a Pulse Width Modulation (PWM) technique. A control computer to used to download algorithm code into the SVPs, specific color space conversion matrices and look up tables (LUTs), and user specified picture control parameters. 2. System Architecture The digital signal processing required for the functions mentioned above is performed using a parallel processing architecture based on the Scan-Line Video Processor (SVP), a programmable video processor developed at Texas Instruments [9]. The core of the SVP is a one dimensional array of one-bit processing elements (PEs) which form a SIMD architecture. Each PE corresponds to one pixel on a line of video data. Current SVPs support 960 pixels per processor. An SVP has a 40 bit wide data input register (DIR), and 24 bit wide data output register (DOR). Data input by the DIR, computation by the PEs, and data output by the DORs are concurrent operations, each having independent clocks. The data rates of the DIR and DOR are both 33MHz. Current SVPs were originally designed for standard television, and can run into certain limitations when required to handle high definition data. Examples of such limitations are data I/O rate requirements,

3 data processing rate requirements, and the maximum number of pixels per video line. SMPTE 240 data with square pixel format is input at 71.1 MHz. This data rate is significantly higher than the 33MHz that the SVP DIR and DOR can handle. Also, Table 1 shows that there are more pixels per line for all the source formats, than one SVP can handle. To overcome these limitations, a multiple SVP based parallel processing system architecture was designed. In this architecture, a video line is partitioned into four blocks as shown in Figure 2. Figure 2 represents the multiple SVP based parallel processing architecture, which overcomes the limitations of single SVP architecture For the case of SMPTE 240 input, each partition contains 480 pixels with a block to block overlap of 20 pixels. The overlap is provided to accommodate horizontal filtering operations. These blocks of pixels are sent to individual SVPs configured in parallel. By doing this, each SVP has to process only 480 pixels per line which is well within its capability. Also, the data rate input into each SVP is now 71.1/4=17.775MHz. This data rate is within the capability of the SVP DIR and DOR. Each of the four branches of the parallel SVP configuration contains 9 SVPs, leading to a total of 36 SVPs in the system. The 9 SVPs per branch execute the various signal processing algorithms mentioned above. Figure 3 shows a block diagram of the parallel processing architecture as it is used for SMPTE 240 square pixel data. The analog signals are digitized and sent to the line slicer, where each video line is divided into four separate channels. Parallel Processing Architecture as it is Used for SMPTE 240 Square Pixel Data Figure 4 shows a diagram of the line slicer. It consists of an input multiplexer (not shown), slow down logic, and multiple first in-first out (FIFO) buffer memories. The multiplexer is used to select between data provided by the A/Ds or direct digital input as from SMPTE 260 format sources. This data is passed to slow down logic which compensates for differences in speed between the A/Ds and FIFOs. Four sets of FIFOs

4 are used, each corresponding to one branch of the system's parallel processing architecture. Figure 4 shows a diagram of the line slicer. It consists of an input multiplexer (not shown), slow down logic, and multiple first in-first-out (FIFO) buffer memories. Each set contains two FIFOs, for even and odd pixels. The slow down logic makes either the odd or even pixel input data (fs) available to the corresponding FIFOs at fs/2.. The outputs of the four FIFO sets then drives the corresponding signal processing channel. 3. Video Signal Processing The signal processing portion of the system performs interlace to progressive scan conversion, scaling, picture control operations, color space conversion, and degamma LUT operations. SVPs are used to perform most of these operations except for color space conversion and degamma. The system architecture was designed to be a flexible testbed that would allow the development and evaluation of various signal processing algorithms. The programmable nature of SVPs, color space coefficients, and degamma LUTs enables the system functionality to be changed by simply downloading new software. This allows for the evaluation of various architecture options and their associated tradeoffs. As can be seen from Figures 1 and 3, Color Space Conversion and Degamma LUT operations can be performed on the data either before or after it is subjected to the SVP signal processing operations. This enables the system to accept input data in either YC or RGB form and to have various signal processing modes such as those outlined in Figure 5. Further flexibility is provided for the cases where Progressive Scan Conversion and Scaling are performed on RGB data. In these cases, degamma operation can be performed either before or after the processing for Progressive Scan Conversion and Scaling. Color space conversion matrix coefficients corresponding to various data types such as NTSC, PAL, SECAM, SMPTE 240 or SMPTE 260 can be downloaded to the "Color Space Conversion" block from the control computer. Similarly, degamma LUTs corresponding to conventional power curve or SMPTE gamma curve can be downloaded to the "Degamma" block. 3.1 Progressive Scan Conversion DMD displays are progressive, i.e. complete frames are shown, rather than individual fields as in interlace displays. Many of the video data formats are interlace and this data

5 has to be converted to progressive form for display on the DMD. This is achieved by interlace to progressive scan conversion. Progressive scan conversion is the process of creating new scan lines between existing lines, which provides double the vertical sampling rate of interlaced scanning. Typical progressive scan algorithms [1][4][6] compute the missing field lines by means of a motion adaptive weighted combination of inter and intra-field data. Motion detection is a critical step because failure to detect motion can result in the use of inter-field data in moving parts of the video, where it can cause the appearance of "tearing" artifacts. On the other hand, oversensitive motion detection can cause the motion detector to be triggered by noise, resulting in intra-field data in still parts of the picture. This can lead to noticeable loss of resolution in the video. Thus, there is a need to balance the algorithm's motion sensitivity with the ability to provide good resolution. We have developed motion detection techniques that explicitly addresses this issue and provides good picture quality in both moving and still picture areas. Several progressive scan conversion algorithms were designed and tested on this system. The following discussion addresses one of these algorithms: Figure 6 shows a block diagram representing the progressive scan algorithm A block diagram representing the progressive scan algorithm is shown in Figure 6. Motion detection is performed by taking frame differences as shown in Figure 7, where f0,...,f3 represent individual fields of data. For each field, the solid lines represent the video data that is available from the source.

6 Motion detection is performed by taking frame differences as shown in Figure 7 The dashed line represents the location (X) where the progressive scan algorithm creates new as follows: The absolute difference A-B is computed and subjected to non-linear operation NL1 (Figure 8) which performs thresholding to eliminate small difference signal values due to noise. Non-Linear Operation NL1 Figure 8 shows a diagram representing non-linear operation NL1 which performs thresholding to eliminate small difference signal va lues due to noise It also reduces the resolution of the difference signal to 4 bits. This signal is then subjected to temporal filtering. Referring to Figure 6, the temporal filter computes the maximum of motion signals from (A-B,C-D,E-F) and assigns this as the current motion signal value. This signal is then subjected to vertical and horizontal low pass filtering. The filtering operations are useful for spreading out the motion signal spatially and temporally. This spreading is particularly important along moving edge boundaries, where conservative motion detection can lead to "tearing" artifacts in the video. However, when noise is present in the motion signal, errors or false detections can be spread to surrounding pixels via the filtering process. This spatial propagation of errors is particularly offensive for highly detailed still scenes, where false detections can lead to a perceived lack of resolution due to the use of intra-field components in the interpolation process. The use of thresholding limits the spatial propagation of errors in the motion

7 signal, thus providing more reliable motion detection for both still and moving scenes. After filtering and non-linear operations, the motion signal is normalized to values ranging between 0 and 1. This normalized signal is then used as a weighting factor for the inter and intra-field components in the interpolation process to produce the output Y' as shown in Figure 6. While this example of progressive scan conversion was shown for luminance data (Y), it is also relevant to RGB data, where the same processing is performed on each of the R/G/B components. In the case of YC data, the chroma component can be converted to progressive scan rate by line double, line average, or the motion adaptive processing described above. 3.2 Scaling Scaling is the process of changing displayed image resolution (number of active lines per frame and number of active pixels per line). Video scaling functionality has been provided in our system for two reasons: 1. Maximization of the number of active DMD pixels to obtain bright, high resolution display, 2. Recent offerings of 16:9 TVs provide several display modes corresponding to various scale factors, such as "Panorama Mode", "Movie Mode", "Wide Mode" etc. Various display modes are provided as user options to fill the complete 16:9 screen with video material, such as letterbox movies on laser disk. The lack of standard specifications for letterbox leads to a wide variation in the size of active areas in letterbox movies. Attempts are generally made to cover this variety of letterbox sizes by providing several scale factors representative of typical letterbox sizes. Scaling Algorithm Based on Bilinear Interpolation for 2:3 Scaling Figure 9 shows the scaling algorithm based on Bilinear interpolation for 2:3 scaling Algorithms have been designed and implemented for the following scale factors: 3:4, 5:6, 2:3, 9:10, and 1:2. Two sets of scaling algorithms were designed, based on Bilinear and Cubic interpolation. Examples of these algorithms for 2:3 scaling are provided in Figure 9 and Figure 10 respectively.

8 Scaling Algorithm Based on Cubic Interpolation for 2:3 Scaling Figure 10 shows the scaling algorithm based on Cubic interpolation for 2:3 scaling The equations for computing the output lines 1,2,3 for cubic interpolation are: Line 1 = A B C D Line 2 = B C D E Line 3 = C D E (Beginning of next set) 3.3 Picture Controls and Degamma The following picture controls are provided: Hue, Saturation, Sharpness, Brightness, and Contrast. Each control function has 256 adjustment levels. The control settings can be downloaded through the computer interface while the projector is in operation. Degamma LUTs are also programmable through the computer interface. A choice in degamma LUTs is provided for various input formats. The degamma LUTs are based on the following characteristics:

9 After the video signal has been subjected to progressive scan conversion, scaling, picture controls, and degamma operations, it is sent to the formatter. 4. Data Formatting and Display The formatter transforms the data into a format specific to DMD display. A line segment mapper is used to combine the data from the four parallel signal processing channels. These signals contain some overlap due to the requirements of horizontal filtering operations. The line segment mapper removes this overlap. The data is also modified to accommodate the DMD size. The number of active pixels per horizontal line can vary depending on the signal format, as shown in Table 1. This data is padded with zeros to extend the horizontal line size to the DMD line size (2048 pixels). This data is then split into individual bit planes, inverted or mirrored if required (special display features), and sent to the DMD. The DMD is a digital device with ON and OFF states. Various intensities required to display natural video are created by controlling the amount of ON versus OFF time of individual DMD elements using a Pulse Width Modulation (PWM) technique. Figure 11 provides an explanation of the PWM technique using a 4 bit example for simplicity.

10 PWM Technique using a 4-bit Example for Simplicity Figure 11 provides an explanation of the PWM technique using a 4-bit example for simplicity The total frame time (1/60 sec) is divided into individual bit times, where the time allocated to each bit is proportional to the binary weight of that bit. Thus, the MSB is allocated half of the frame time (1/120 sec), MSB-1 is allocated half the time of MSB, MSB-2 is allocated half the time of MSB-1 and so on. A given gray scale is displayed at a specific pixel by moving the DMD element corresponding to that pixel to the ON position for the bits that are ' 1 ' for that gray scale's binary representation, and turning the DMD element to OFF position for the bits that are '0' for that gray scale's binary representation. For example, a 4 bit representation of number 10 is Its gray scale is created at a specific pixel location by turning the DMD element ON for the time periods corresponding to the MSB and MSB-2 and turning them OFF for the time periods corresponding to the MSB-1 and MSB-3 (LSB) time period. The observer's visual system integrates the binary data to perceive gray scale intensity corresponding to number 10. PWM displays can cause annoying visual artifacts if the PWM is implemented without consideration for human perception. Visual perception of intensities is based on integration of incident energy for a finite time period. Imbalances in energy during this integration period can result in scene content (such as object motion or intensity changes), or viewer eye motion. For example, consider again the 4 bit example in Figure 11. Object motion, intensity changes, or image noise can alter a specific grey scale level from a value of 8 to a value of 7. For the coding pattern shown in Figure 10, 8 has binary representation of 1000 and 7 has binary representation of The transition energy of an 8 to 7 transition is much higher than the steady state energy associated with either 8 or 7. This causes a "flash" to be perceived at the transition. Artifacts also occur if the eye's integration is interrupted due to eye motion (e.g. blinking, darting of eyes to look at different parts of the scene, normal head movements). These artifacts are corrected by coding the bits to minimize perceptual energy imbalances [2].

11 Summary The DMD based high definition display system described above has been built and is operational. The programmable nature of the system architecture has allowed the development and evaluation of video processing algorithms and system technology tradeoffs for high definition as well as standard definition displays. Current work is focusing on MPEG-based digital video. References 1. T. Doyle, and M. Looymans, "Progressive Scan Conversion Using Edge Informtaion," Signal Processing of HDTV II, Proceedings of the Third International Workshop on HDTV, August R.J. Gove, "DMD Display Systems: The Impact of an All-Digital Display," Society for Information Display International Symposium, june L.J. Hornbeck, INVITED PAPER "Current Status of the Digital Micromirror Device (DMD) for Projection Telelvision Applications," International Electron Devices Technical Digest, p , V. Markandey et al., "Motion Adaptive Deinterlacer for DMD (Digital Micromirror Device) Based Digital Television," IEEE Transactions on Consumer Electronics, Vol. 40, No. 3, August J.B. Sampsell, "The Digital Micromirror Device and Its Applications to Projection Displays," Society for Information Display International Symposium, May K. Sato et al., "Adaptive Intra-Field Progressive Scan Converter," ITEC' C. Tew et al. "Electronic Control of a Digital Micromirror Device for Projection Displays," 1994 IEEE Solid-Sate Circuits Digest of Technical Papers, Vol. 37, p J. M. Younse, "Mirrors On A Chip," IEEE Spectrum, November M. Yugami et al., "EDTV With Scan-Line Video Processor," IEEE Transactions on Consumer Electronics, Vol. 38, No. 3, August 1992.