Video and Image Processing Up Conversion Example Design

Transcription

1 Video and Image Processing Up Conversion Example Design October 2007, ver 4.0 Application Note 427 Introduction The Altera video and image processing up conversion example design demonstrates up conversion from a standard definition video stream in national television system committee (NTSC) format to a high definition output resolution ( ). The example design provides a framework for rapid development of video and image processing designs using the library of parameterizable MegaCore functions available in the Altera Video and Image Processing Suite. These functions can be used to perform the following common video functions: Color Space Conversion Gamma Correction Chroma Resampling Deinterlacing Alpha Blended Mixing 2D FIR Filtering 2D Median Filtering Scaling Line Buffer Compilation 1 For more information about these MegaCore functions, refer to the Video and Image Processing Suite User Guide. In the example design, video source is input through an analog composite port on an Altera Video Input Daughtercard. The daughtercard generates a digital output in BT656 format, and connects to an Altera Cyclone II DSP development board via a daughtercard connector. The processed video stream is output via the VGA connector which is available on the Altera Cyclone II DSP development board. In the FPGA, a number of common video functions are performed on the input stream including chroma resampling, deinterlacing, color space conversion and scaling. The design shows by example, how to use DSP Builder for modeling and simulating the data path for an imaging application. The combined MATLAB and Simulink environment provide an easy interface to import and export data to and from the DSP Builder design to read and write video files, while verifying the functionality of the Cyclone II design. Altera Corporation 1 AN

2 Video and Image Processing Up Conversion Example Design In addition, an easy path to system integration of the video processing data path with NTSC video input, VGA output and an external DDR2 memory controller is demonstrated with SOPC Builder. The video up conversion design is implemented using a combination of Altera MegaCore functions from the Video and Image Processing Suite. The video functions use common Avalon Streaming (Avalon-ST) data interfaces and Avalon Memory-Mapped (Avalon-MM) control interfaces to facilitate connection of a chain of video functions and video system modeling. f f Refer to the Interfaces chapter in the Video and Image Processing Suite User Guide for a full description of how these interfaces are implemented. For more information about the Avalon-MM and Avalon-ST interfaces, refer to the Avalon Memory-Mapped Interface Specification and the Avalon Streaming Interface Specification. DSP Builder is a digital signal processing (DSP) development tool that interfaces The MathWorks industry-leading system-level DSP tool Simulink with the Altera Quartus II development software. DSP Builder provides a seamless design flow in which you can perform algorithmic design and system integration using MATLAB and Simulink and then port the design to hardware description language (HDL) files for use in the Quartus II software. The automatically generated HDL files are at the register transfer level (RTL). They are optimized for use in the Quartus II software for rapid prototyping. f For more information on DSP Builder, refer to the DSP Builder User Guide. SOPC Builder is a system development tool, allowing the user to create hardware and software system modules with a customized set of system peripherals. SOPC Builder automatically creates the bus arbitration logic connecting the individual components together to create an overall system. f For more information on SOPC Builder, refer to the SOPC Builder User Guide. 2 Altera Corporation

3 Installing the Example Design Installing the Example Design The example design files are included on the Video Development Kit, Cyclone II Edition CD-ROM or can be downloaded as a zip file from the Altera website. Figure 1 shows the directory structure for the example design files when they have been extracted from the zip file. Figure 1. Example Design Directory Structure Video_IP_Example_Design_<version> Contains top level block design file (Video_IP_Example_Design.bdf), Quartus II settings file (Video_IP_Example_Design.qsf), Quartus II project file (Video_IP_Example_Design.qpf), DSP Builder model file (example_design_data_path.mdl), SOPC Builder project file (video_system_sopc.sopc), and PLL files for the DDR2 Controller (ddr_pll_cycloneii*.*). altera_avalon_i2c Contains SOPC Builder components (<module name >_hw.tcl and VHDL files) for the I 2 C controller that communicates with the digital composite input card. example_design_controller Contains SOPC Builder components (<module name >_hw.tcl and VHDL files) for the Avalon master used by the I 2 C controller and NTSC composite input module. frame_buffer_beta Contains a beta version of the Frame Buffer MegaCore function. The frame buffer block can perform a double or triple buffering function and is inserted between the other Video and Image Processing Suite MegaCore functions and the VGA output. ntsc_composite_input Contains SOPC Builder components (<module name >_hw.tcl and VHDL files) which decode the video signal from the digital composite input card. vga_output Contains SOPC Builder components (<module name >_hw.tcl and VHDL files) for the VGA output driver. docs Contains this document (AN-427.pdf). Notes to Figure 1: (1) The Video_Ip_Example_Design_<version> directory also includes other files that are included for convenience but can be regenerated using the tools described in this application note. (2) If you want to use the Frame Buffer MegaCore function in other designs, you can copy the frame_buffer_beta directory to your Altera IP v7.2 installation directory (C:\altera\72\ip by default). The Frame Buffer MegaCore function will then be accessible in the Quartus II software using the MegaWizard Plug-In Manager or from the System Contents tab in SOPC Builder. The top level directory Video_IP_Example_Design_<version> contains a Quartus II project file (Video_IP_Example_Design.qpf), DSP Builder model file (example_design_data_path.mdl), and SOPC Builder project file (video_system_sopc.sopc). These files can be opened to explore the design as described in Review and Simulate the Example Design on page 16. Altera Corporation 3

4 Video and Image Processing Up Conversion Example Design Possible Video System Configurations Many new and exciting innovations, such as high definition television (HDTV) and digital cinema, revolve around video and image processing and this technology's rapid evolution. Leaps forward in image capture and display resolutions, advanced compression techniques, and video intelligence are the driving forces behind the technological innovation. The move from standard definition (SD) to high definition (HD) represents a 6 increase in data that needs to be processed. Video surveillance is also moving from common intermediate format (CIF) ( ) to D1 format ( ) as a standard requirement, with some industrial cameras even moving to HD at Military surveillance, medical imaging, and machine vision applications are also moving to very high resolution images. With expanding resolutions there is a need for high performance while keeping architectures flexible to allow for quick upgradeability. All of the processing functions within the Altera Video and Image Processing Suite are configurable to allow the user to satisfy common performance and cost requirements over a wide range of SD and HD resolutions. In addition, the processing functions have common data and control interfaces, providing a flexible and extendible framework for developing complex video pre- and post-processing data paths. This section describes a number of video systems that could be constructed using the Video and Image Processing Suite MegaCore functions. Single Video Channel Input The video up conversion example design is a good starting point for developing single video stream input applications. The example design demonstrates how to construct a video data path that performs deinterlacing, chroma resampling, color space conversion and scaling. The Altera Video and Image Processing Suite MegaCore functions all have common data and control interfaces. These common interfaces make it easy to extend the data path in the example design to perform additional functions such as filtering, gamma correction or picture in picture mixing with a static image from memory. 4 Altera Corporation

5 Possible Video System Configurations Figure 2 shows a block diagram of a single video channel input system. Figure 2. Single Video Channel Input System Composite Video Input (NTSC) NTSC Interface Avalon-ST data interface Avalon-MM control interface Deinterlacing Chroma Resampling Color Space Conversion (YCbCr->RGB) Gamma Correction Scaling 2D 5x5 FIR Filter (Sharpening) Nios II Processor Static Image Picture-in- Picture Mixing Triple Buffer VGA Controller Video Output (VGA) External Memory The system uses a Nios II processor for control processes such as writing gamma values to the gamma corrector look up table and writing the relative location of a static image in on-chip memory to the picture-inpicture mixing function. The standardization on Avalon-MM control interface ports makes Nios II a convenient choice for run-time control. Multiple Video Channel Input Video systems frequently contain multiple video streams, which are processing in parallel and synchronized. The video up conversion example design is a good starting point for developing multiple video stream input applications. You could extend the example design to process two video stream inputs via the dual composite Video Input Daughtercard. Like the Single Video Channel Input, the Avalon-MM control interface on the picture-inpicture mixer makes it easy to control the run time parameterization using a Nios II processor. Control parameters include foreground image location relative to the background image. Altera Corporation 5

6 Video and Image Processing Up Conversion Example Design Figure 3 shows a block diagram of a two video channel input system. Figure 3. Multiple Video Channel Input System Composite Video Input (Interlaced) Avalon-ST data interface Avalon-MM control interface NTSC Interface Deinterlacing Chroma Resampling Color Space Conversion (YCbCr->RGB) Gamma Correction Scaling 2D 5x5 FIR Filter (Sharpening) Nios II Processor Picture-in- Picture Mixing VGA Controller Video Output (VGA) Triple Buffer Composite Video Input (Interlaced) External Memory Triple Buffer NTSC Interface Deinterlacing Chroma Resampling Color Space Conversion (YCbCr->RGB) Gamma Correction Scaling 2D 5x5 FIR Filter (Sharpening) Note that the two triple buffer blocks perform frame synchronization for the two video streams. Other Video Interface Standards The up conversion example design provides a framework to develop video systems with NTSC video input and VGA video output. This requires appropriate hardware ports on the DSP development board, analog to digital conversion and FPGA hardware interface components. However, a variety of different hardware interface standards and protocols are seen in applications throughout the broadcast, consumer, automotive, surveillance, medical and document imaging markets. 6 Altera Corporation

7 Possible Video System Configurations These interfaces include phase alternation line (PAL), digital video interface (DVI), high definition multimedia interface (HDMI), component or YCbCr (with separate luminance and chrominance signals), S-Video or Y/C (with separate luminance and color signals), serial digital interface (SDI) and high definition SDI (HD-SDI). The programmable nature of the FPGA lends itself well to migrating systems that support different interfaces. The example design could be used as a framework for development of such FPGA hardware interface components. This can be achieved by implementing SOPC Builder components with Avalon-ST interfaces as used in the example design. The DSP Builder tool also allows development and test of video processing data paths independent of FPGA hardware interface development. For example, you could develop a video system with SDI video input and output ports using the Stratix II GX board, and the Altera SDI/HD-SDI MegaCore function. Figure 4 shows a block diagram of a system with different video interface standards. Figure 4. System with Different Video Interface Standards SDI Video Source SDI Avalon-ST data interface Avalon-MM control interface Deinterlacing Chroma Resampling Color Space Conversion (YCbCr->RGB) Scaling Triple Buffer SDI SDI Video External Memory Altera Corporation 7

8 Video and Image Processing Up Conversion Example Design Functional Description Figure 5 shows a simple block diagram for the video and image processing up conversion example design. Figure 5. Example Design Block Diagram DSP Builder NTSC Video Input - Composite Input on Video Daughtercard Video Up Conversion Frame Buffer HD Video Output - VGA Controller DDR2 SDRAM SOPC Builder The video and image processing up conversion example design is implemented using hardware interface components and parameterizable hardware processing blocks from the Video and Image Processing Suite. A video stream is input via one of the composite input ports on the Video Input Daughtercard. The main block of the example design performs an up conversion of the standard definition video input (NTSC format ) and outputs a high definition video stream ( ), displaying the results through the VGA connector. DDR2 SDRAM memory is used to buffer the video frames from both a frame buffer block and the deinterlacing hardware component of the video up conversion data path. The up conversion block demonstrates how to connect together video processing functions from the Video and Image Processing Suite. 8 Altera Corporation

9 Functional Description NTSC Video Input Figure 6 shows a simple block diagram of the three components of the NTSC video input subsystem. Figure 6. NTSC Video Input Block Diagram Example Design Controller I 2 C Controller Daughtercard Clock (~25 MHz) SOPC System Clock (130 MHz) Y/C Syncs Video Data Capture Dual-Clock FIFO 8 Ready Valid Data NTSC Composite Input The NTSC input into the system requires three SOPC Builder components to work together: the I 2 C Controller, the Example Design Controller and the NTSC Composite Input block. These are shown in Figure 6, with a dotted line delineating each SOPC Builder component. The NTSC Composite Input block operates similarly to its counterpart the VGA Output block. It is written in VHDL which can be found in the <install_dir>\ntsc_composite_input folder. Video data in YCbCr 4:2:2 format and associated synchronization signals are input into the Video Data Capture block from the Video Input Daughtercard. This block identifies active video parts of the picture and inserts just this data into the Dual-Clock FIFO. A standard flow controlled interface on the output of the FIFO allows this data to be read by the next part of the system (the video processing data path in the case of the example design) at the system clock rate of 130 MHz. Altera Corporation 9

10 Video and Image Processing Up Conversion Example Design If the FIFO becomes empty, then the interface will not assert valid and the rest of the system will wait. The FIFO should never become full - if this were to happen then data from the video data capture block would be lost and unpleasant visual artefacts would result. The TVP5146 video decoder chip on the Video Input Daughtercard performs analog-to-digital conversion of the video input signals and needs to be configured before it can be used. An I 2 C interface is provided for this purpose. The I 2 C Controller is an Avalon-MM slave which, when controlled by an Avalon-MM master, performs I2C reads and writes on an external bus as requested by that master. The Example Design Controller is an Avalon-MM master which directs the OpenCores I 2 C Controller to perform the necessary start-up sequence for the TI5146 chip, and then sends a signal to the NTSC Composite Input block telling it to start inputting data. The NTSC Composite Input block provides an Avalon-MM slave port for this purpose. The slave port has one register, one bit of which is significant: The least significant bit is a GO bit - when this is logic '1' the NTSC Composite Input block tries to input data, when it is logic '0' the NTSC Composite Input block does nothing. The I 2 C master is written in VHDL which can be found in the directory <install_dir>\altera_avalon_i2c. The example design controller VHDL can be found in <install_dir>\example_design_controller. Video Up Conversion Figure 7 shows a block diagram of the video up conversion data path subsystem. Figure 7. Video Up Conversion Data Path Block Diagram 640 x 480 Interlaced 60 Hz YCbCr 4:2:2 640 x 480 Progressive 30 Hz YCbCr 4:2:2 640 x 480 Progressive 30 Hz YCbCr 4:4:4 640 x 480 Progressive 30 Hz RGB 1024 x 768 Progressive 30 Hz RGB 8 Deinterlacer MegaCore Chroma Resampler MegaCore Color Space Converter MegaCore Scaler MegaCore 8 Ready Valid Data Two Avalon-MM master interfaces 10 Altera Corporation

11 Functional Description The video up conversion data path performs all of the video processing required to convert from an NTSC format input to a VGA output. It is composed entirely of Altera Video and Image Processing MegaCore functions and is assembled in DSP Builder, where it can be simulated independently of the rest of the system (see Simulate the Data Path Component in DSP Builder on page 30). The entire data path is exported from DSP Builder as a single SOPC Builder component, with standard Avalon-ST input and output interfaces and two Avalon-MM master ports. The data path makes use of four MegaCore functions connected in sequence to perform conversion. First a Deinterlacer converts from 60 Hz interlaced to 30 Hz progressive. Next a Chroma Resampler interpolates to convert from 4:2:2 subsampled color data to full 4:4:4 color data. This is followed by a Color Space Converter which transforms between the YCbCr and RGB color spaces. Finally a Scaler scales the input image up to using bicubic interpolation. Timing and Data Format Adapters The next set of components in the data path is a set of adapters. The video up conversion subsystem processes an 8-bit wide data stream, one color plane every sample, but the frame buffer expects a 24-bit wide data stream, one pixel every sample. An Avalon-ST Data Format Adapter SOPC Builder block is used to transform the incoming 1 8-bit stream, into a 3 8-bit stream. The Avalon- ST Data Format Adapter block is a ready-latency 0 block whereas the rest of the data path is built with ready-latency 1 components. The Avalon-ST Data Format Adapter block is therefore connected using two Avalon-ST Timing Adapter blocks that convert between the ready-latency 1 stream and ready-latency 0 streams as shown in Figure 8. Figure 8. Data Stream Adapter Block Diagram Ready Ready Ready Ready Avalon-ST Avalon-ST Avalon-ST Valid Valid Valid Valid Timing Data Format Timing Data Data Data Data Adapter Adapter Adapter ready-latency 1 rules ready-latency 0 rules ready-latency 0 rules ready-latency 1 rules Altera Corporation 11

12 Video and Image Processing Up Conversion Example Design Frame Buffer Component Figure 9 shows a simple block diagram of the Frame Buffer component and its connections to external RAM via SOPC Builder. Figure 9. Frame Buffer Component Block Diagram Ready Valid Data 24 Memory Writer Memory Reader 24 Ready Valid Data SOPC Builder Arbitration Logic DDR2 The Frame Buffer block is a beta version of a new Video and Image Processing Suite MegaCore function which can be found in the directory <install_dir>\frame_buffer_beta. The purpose of this block is to provide a triple buffering function which allows the input and output sides to run asynchronously and at different frame rates. In the example design, this is necessary because the output of the video data path is progressive 30 fps, but the VGA output cannot run slower than 60 fps. The Frame Buffer block inputs and outputs flow-controlled streams of video data over standard interfaces of the type described in Data and Flow Control Signals on page 18. The input and output data interfaces have 24-bit wide data ports and output all three color planes in parallel, with blue on the least significant bits and red on the most significant bits. This makes the output interface compatible with the input interface on the VGA Output block. The block uses three equal-sized frame buffers stored in external memory. The base address of this memory in the Avalon-MM address space is set at 0x There must be 8MByte of free memory at this address. 12 Altera Corporation

13 Functional Description The external memory is a double data rate (DDR) RAM in the example design, and is accessed via SOPC Builder arbitration logic as shown in Figure 9. SOPC Builder is responsible for sharing access time to the DDR between the frame buffer block and other blocks in the design. The block includes a writer, which writes the input stream into one of the buffers, and a reader, which reads the output stream from another (never the same) buffer. There is always one buffer which is neither being written to nor read from. This "spare" buffer is required to allow the input and output to run at differing frame rates. The frame buffer operates by swapping frames according to the following algorithms. Each time the input finishes writing a frame of data into a buffer: Wait until the spare buffer has data that has already been displayed, then start writing the next input frame into the spare buffer. The buffer just written into then becomes the new spare buffer. No frame is dropped in this configuration. Each time the output finishes reading a frame of data from a buffer: If the spare buffer has data which has not yet been displayed then start reading the next output frame from the spare buffer. The frame buffer just read from then becomes the new spare buffer. Otherwise, start reading the next output frame from the same buffer (a frame is repeated). VGA Output Figure 10 shows a simple block diagram of the VGA Output component. Figure 10. VGA Output Component SOPC System Clock (130 MHz) VGA Clock (65 MHz) Ready Valid Data 24 Dual-Clock FIFO VGA Syncs Generator R G B Syncs The VGA Output is an SOPC Builder component written in Verilog HDL. The source code can be found in the directory <install_dir>\vga_output. Altera Corporation 13

14 Video and Image Processing Up Conversion Example Design A stream of video data is input into the block over a standard flow controlled interface. This interface includes a 24-bit wide data port because red, green, and blue data is input into the block in parallel, so that all of the data for a pixel can be input in a single clock cycle. The least significant eight bits form the blue channel, the green is in the middle and the most significant eight bits carry the red channel. The interface also includes ready and valid lines for flow control. f Refer to Data and Flow Control Signals on page 18 for details of the flow controlled interface. The video data is input via a flow controlled interface, so data is not transferred on every clock cycle, only those clock cycles where valid = '1', and the clock therefore need not be the same rate as the VGA output clock. The clock actually used at this end is therefore the SOPC System clock, which runs at 130MHz in the example design. Data is input into a dual-clock FIFO which provides clock domain crossing to the 65MHz VGA clock and also provides a queue where pixels can wait when the VGA output is in blanking and does not need pixel data. If this FIFO ever becomes full, then the flow controlled interface will indicate that the VGA output is not ready for data and earlier parts of the pipe will stop. The FIFO should never become empty while the system is running. If it does, then the situation could arise that there is no pixel data available when the VGA Syncs Generator needs it. In this case, the VGA output module will generate red pixels on its outputs (R:255, G:0, B:0) as an indication of data starvation. Synchronization and blanking signals for VGA running the standard video electronics standards association (VESA) resolution are calculated and output by the VGA Syncs Generator block. This block also pulls data out of the dual-clock FIFO and presents it on R, G, and B outputs when the syncs indicate the VGA should output active picture data. Signals from the VGA Syncs Generator form the output of the VGA Output block, and are wired directly to pins connected to the 2C35 board's VGA digital-to-analog converter (DAC) in the example design. System Requirements This section describes the hardware and software requirements to run the video up conversion example design. 14 Altera Corporation

15 System Requirements Hardware Requirements The video and image processing up conversion example design requires the following hardware components: Cyclone II EP2C70 DSP Development Board. NTSC video source with composite output. The example design has been verified using: An Apple Video ipod. (An Apple TV out cable is required to connect an Apple Video ipod to the composite input.) An IBM Thinkpad T41 laptop with ATI v8.133 drivers generating the NTSC video source. (A S-Video to phono cable, S-Video to RCA, or a S-Video lead -> SVHS-Phono Adapter -> phono lead connection is required to connect a Thinkpad T41 to the Altera Video Input Daughtercard composite input.) A Sony Handycam DCR HC-46 Other video sources outputting NTSC can be used including video cameras. An Altera Video Input Daughtercard (with two separate input channels of composite input). A monitor or display with a VGA connector supporting resolution. A VGA cable to connect the Cyclone II VGA output to the monitor 1 The Cyclone II EP2C70 DSP Development Board, Video Input Daughtercard and reference design are included in the Altera Video Development Kit, Cyclone II Edition. Software Requirements The up conversion example design is supported on Windows XP only. Ensure that the software provided with the development kit is installed onto your PC. f For more information on the software installation, refer to the DSP Development Kit, Cyclone II Edition, Getting Started User Guide. You must have the following software installed on your PC: Quartus II software, version 7.2 MegaCore IP Library, version 7.2 The MathWorks release R14 SP3, R2006a, R2006b, or R2007a DSP Builder version This application note assumes that you have installed the software into the default locations. Altera Corporation 15

16 Video and Image Processing Up Conversion Example Design Review and Simulate the Example Design This section reviews the Data Path component in DSP Builder and describes how to simulate the Data Path component in DSP Builder. Review the Data Path Component in DSP Builder To review the video and image processing up conversion data path design perform the following steps: 1. Run the MATLAB software. 2. In the Current Directory browser, browse to the top level install directory. 3. Choose Open (File menu) and select the file example_design_data_path.mdl 4. Review the top level DSP Builder model, as shown in Figure 11. Figure 11. Top Level Model of the Example Design The example design shows how to connect together a chain of video functions for simulation within the DSP Builder environment. The next section describes each of the components and the connectivity of the data and flow control signals. 16 Altera Corporation

17 Review and Simulate the Example Design Altera Video and Image Processing MegaCore Functions The example design contains four Altera Video and Image Processing MegaCore functions: Deinterlacer, Chroma Resampler, Color Space Converter and Scaler. After installing any of the Video and Image Processing Suite MegaCore functions, you should run the alt_dspbuilder_setup_megacore command in the MATLAB command window. The MegaCore functions are available from the Altera DSP Builder Blockset in the Video and Image Processing library as displayed in Figure 12. Figure 12. Video and Image Processing Suite in DSP Builder Altera Corporation 17

18 Video and Image Processing Up Conversion Example Design Video functions can be added to a DSP Builder model design by simply dragging and dropping the MegaCore function blocks to the model window. Data and Flow Control Signals All of the MegaCore functions support processing of images with from one to three color planes using Avalon-ST and Avalon-MM interfaces. f Refer to the Interfaces chapter in the Video and Image Processing Suite User Guide for a full description of these data and flow control signals. Two MegaCore functions can therefore be connected together by connecting the common signal types valid to valid, ready to ready and data to data. The model based view in DSP Builder of the connection between the Color Space Converter and the Scaler block is show in Figure 13. Figure 13. Connections Between the Color Space Converter and Scaler Video Input Block - Video Source The example design uses a DSP Builder Simulation-only block to drive a video stream through the video processing functions for simulation. Figure 14 shows the Video Source block. 18 Altera Corporation

19 Review and Simulate the Example Design Figure 14. Video Source Block The Video Source block extracts video frames from a multimedia file (vip_car.avi) and transmits them using the image streaming protocol described in the Video and Image Processing Suite User Guide. f For more information on the Video Source block, refer to the DSP Builder Reference Manual. The Video Source block has one input signal and two output signals: Input: ready Data output: data Valid output: valid The block produces valid output data one clock cycle after the ready input signal is driven high. The valid output signal is driven high when the clock is valid. In this example, the Video Source block is configured to output 8-bit wide data in sequence, one color plane every clock cycle, over the data output port as shown in Figure 15. Figure 15. YCbCr Format Pixel Data at a 4:2:2 Sampling Rate Y Cb Y Cr Y Cb Y Cr 1 The file name and location of the input file is a parameter of the Video Source block. Before running the simulation, you should check that the input file vip_car.avi is in the current MATLAB directory. Alternatively, you can change the Input file name parameter of the Video Source block to the absolute path of the multimedia file of your choice. The Video Source block can extract video frames from a wide range of multimedia file types, including still bitmap pictures, provided that the proper codecs are installed on your system. Altera Corporation 19

20 Video and Image Processing Up Conversion Example Design Video Output Block - Video Sink The example design uses a DSP Builder Simulation-only block to write video streams of data to an audio-video interleaved (AVI) file during simulation. Figure 16 shows the Video Sink block. Figure 16. Video Sink Block The Video Sink block builds video frames from the incoming stream of data and uses an user-specified encoder to store them in an AVI file. The Video Sink block is configured to output the file vip_car_out.avi in the current MATLAB directory by default. 1 For more information on the Video Sink block, refer to the DSP Builder Reference Manual. Video Frame Counter The example design contains Simulink based blocks that provide a runtime count of the number of frames processed at any point in the data path during a simulation. Figure 17 shows the output Frame Counter blocks at the end of the data path. Figure 17. Frame Counter Blocks When running a simulation, the Input Frames and Output Frames blocks display the number of frames processed in decimal mode. 5. Double-click on the Frame Counter - Output block to display the Function Block Parameters dialog box as shown in Figure 18 on page Altera Corporation

21 Review and Simulate the Example Design Figure 18. Frame Counter - Output Block Parameters Dialog Box To count frames correctly, the frame width and frame height are set to the resolution of the video at the point in the data path where the frame counter is connected. The number of channels in sequence is set to the number of color planes transported in sequence at that point on the data path. For the frame counter placed after the Scaler block, the frame width, frame height, and number of channels in sequence are set to the format output by the Scaler block. 1 The number of channels in sequence for the frame counter placed at the beginning of the data path is not 3 because the frame counter is counting 4:2:2 subsampled frames and in this case the number of color planes in sequence is Click Cancel to close the Frame Counter - Output Block Parameters dialog box. Altera Deinterlacer MegaCore Function The example design deinterlaces the interlaced video input using the Deinterlacer MegaCore function provided with the Altera Video and Image Processing Suite. Figure 19 on page 22 shows the data and control ports on the Deinterlacer block. Altera Corporation 21

22 Video and Image Processing Up Conversion Example Design Figure 19. Deinterlacer Block 7. Double-click on the Deinterlacer block to display the MegaWizard Plug-In Manager Parameter Settings page (Figure 20). Figure 20. Parameters for the Deinterlacer MegaCore Function 22 Altera Corporation

23 Review and Simulate the Example Design The deinterlacer is parameterized to process an interleaved image of resolution pixels, with 2 color planes in sequence due to the 4:2:2 sampling rate of the interleaved input image. Weave deinterlacing is applied in this example and requires a frame buffer stored in off-chip memory so that lines from different fields can be woven together. For this reason, the weave deinterlacer has a built in double-buffering function. When in weave mode, the deinterlacer has two 64-bit Avalon-MM master ports. These must be connected to an external memory with enough space to store four full fields of video data. The base address of frame buffers represents the address in the Avalon-MM address space where the base of the frame buffer memory is to be located and there must be at least 1.2MByte of free RAM at this location. (7.4MByte is used by the frame buffer block at a base address of 0x ) 8. Click Cancel to close the Deinterlacer MegaWizard page. External Memory Simulation Block The example design uses a DSP Builder Simulation-only block to allow simulation of an external RAM memory. The memory model is designed for use with the Altera Video and Image Processing MegaCore functions, but can be used for other purposes. Data can be written to (or read from) the RAM model via Avalon-MM read and write master ports. In the example design, the RAM model is connected to the Deinterlacer function. 1 Note that the external RAM model can be used for simulation only and will not generate HDL if the design is compiled with Signal Compiler. Figure 21 shows the External RAM memory block in DSP Builder. Figure 21. External RAM Memory Block Altera Corporation 23

24 Video and Image Processing Up Conversion Example Design 9. Double-click on the External RAM block to view the parameter options as shown in Figure 22. Figure 22. Function Block Parameters for the External RAM 10. Click Cancel to close the External RAM Function Block Parameters dialog box. Chroma Resampler The example design resamples the subsampled video input using the Chroma Resampler MegaCore function provided with the Altera Video and Image Processing Suite. Figure 23 on page 25 shows the data and control ports on the Chroma Resampler block. 24 Altera Corporation

25 Review and Simulate the Example Design Figure 23. Chroma Resampler Block 11. Double-click on the Chroma Resampler block to display the MegaWizard Plug-In Manager Parameter Settings page (Figure 24). Figure 24. Parameters for the Chroma Resampler MegaCore Function In this example, the Chroma Resampler converts the YCbCr 4:2:2 subsampled pixel stream to a 4:4:4 sampled stream. 12. Click Cancel to close the Chroma Resampler MegaWizard page. Altera Corporation 25

26 Video and Image Processing Up Conversion Example Design Color Space Converter The example design converts the YCbCr color space to the RGB color space using the Color Space Converter MegaCore function provided with the Altera Video and Image Processing Suite. Figure 25 shows the data and control ports on the Color Space Converter block. Figure 25. Color Space Converter Block 13. Double-click on the Color Space Converter block to display the MegaWizard Plug-In Manager Parameter Settings page (Figure 26). Figure 26. Parameters for the Color Space Converter MegaCore Function 26 Altera Corporation

27 Review and Simulate the Example Design 14. Select the Operands tab to display the coefficients used to convert between the YCbCr and RGB color spaces. See Figure 27. Figure 27. Operands for the Color Space Converter MegaCore Function 15. Click Cancel to close the Color Space Converter MegaWizard page. Scaler The example design scales the input video from standard definition (SD) NTSC resolution of pixels to the high definition (HD) resolution pixels using the Scaler MegaCore function provided with the Altera Video and Image Processing Suite. Altera Corporation 27

28 Video and Image Processing Up Conversion Example Design Figure 28 shows the data and control ports on the scaler block. Figure 28. Scaler Block 16. Double-click on the Scaler block to display the MegaWizard Plug-In Manager Parameter Settings page (Figure 29). Figure 29. Resolution Parameters for the Scaler MegaCore Function The scaler convert from standard definition ( ) to high definition ( ) image resolution. The clipping feature is not used. 28 Altera Corporation

29 Review and Simulate the Example Design 17. Select the Algorithms and Precision tab (Figure 30). Figure 30. Algorithm and Precision Page for the Scaler MegaCore Function The scaler uses the Bicubic scaling algorithm with 16 vertical and horizontal phases. 1 The Coefficients tab is only used when you select the Polyphase scaling algorithm and is not used in the example design. 18. Click Cancel to close the Scaler MegaWizard page. Altera Corporation 29

30 Video and Image Processing Up Conversion Example Design Simulate the Data Path Component in DSP Builder The simulation of video systems is a computationally expensive task, and can take several hours to perform a gate level simulation of a frame of high resolution video. However, DSP Builder can accelerate simulation of the Video and Image Processing Suite MegaCore functions (by a factor of approximately 30) using transaction level simulation. This increase opens the door to rapid prototyping and experimentation leading to higher image quality. This section describes how to simulate the design using both fast functional simulation and traditional cycle accurate simulation. For convenience, the example design contains pre-generated sample input and output video files in.avi format. The vip_car_out.avi file contains several frames of simulated video. To view the image input and output files, open the.avi files in Windows Media Player. You can change a parameter for any of the MegaCore functions described in the previous section by double-clicking the appropriate block in DSP Builder to display the MegaWizard interface. The associated files are regenerated with the new parameters when you click Finish in the MegaWizard interface. Fast Functional Simulation The example model contains a Simulation Accelerator block. When this block displays Bit-accurate simulation (faster), all Video and Image Processing Suite blocks are simulated at the transaction level, resulting in significant simulation acceleration compared to traditional cycle-accurate simulation (Figure 31). Figure 31. Simulation Accelerator Block set for Bit-Accurate Simulation f For more information about fast functional simulation, refer to the Using the Simulation Accelerator chapter in the DSP Builder User Guide. algebraic_loop_cut_dil Block This block is used to prevent Simulink from detecting an algebraic loop between the Deinterlacer and the two Avalon-MM master interfaces. 30 Altera Corporation

31 Review and Simulate the Example Design f This issue is discussed in the DSP Builder Release Notes and Errata as an errata: Deinterlacer Ports Cannot Be Simulated with Avalon-MM Master Block. The design can be synthesized but it cannot be simulated correctly if the External RAM Wait States Per Write parameter is set to anything other than To simulate the design, select Start from the Simulation menu in the example_design_data_path model window. 1 Any existing vip_car_out.avi file is overwritten when you run a new simulation. You may want to archive the previous video files before you run a simulation. Simulation may fail to start if the previous video output file is open in your video player. 2. Allow the simulation to run for at least one output frame. The frame counter blocks provides run time feedback on simulation progress. 1 The output file is locked while the simulation is running or paused, and it is usually not possible to view the output video until the simulation has finished or is stopped manually. However, it is possible to make a copy of the output file while the simulation is running. Depending on the encoder chosen in the Video Sink block, this incomplete file might be readable by your media player. c If you stop the simulation, you will not be able to resume from the same point and the simulation must be restarted from the beginning. Cycle Accurate Simulation Switching between fast functional and cycle-accurate simulation is very simple. To perform a traditional cycle accurate simulation of the design, double-click the Simulation Accelerator block to set Cycle-accurate simulation mode (Figure 32). Figure 32. Simulation Accelerator Block set for Cycle-Accurate Simulation You can then simulate the design in exactly the same way as for fast functional simulation. Altera Corporation 31

32 Video and Image Processing Up Conversion Example Design Review the System Integration Using SOPC Builder This section covers the following steps: Review the Data, Control and SOPC Builder Interfaces Review the Final System using SOPC Builder Review the Data, Control and SOPC Builder Interfaces To integrate the example design data path block into the overall system as a SOPC Builder component, the system must include Avalon-ST and Avalon-MM interface blocks. The Avalon Memory-Mapped Interface Specification defines the transfer interaction between a peripheral and the interconnect logic that connects the component to the rest of the system. The interconnect logic, also known as the system interconnect fabric, is generated automatically by SOPC Builder. DSP Builder is closely integrated with SOPC Builder. SOPC Builder can automatically detect a DSP Builder model XML file (.mdlxml), and include it in the component list for integration into a larger system. The design contains two data interfaces: Avalon-ST Sink and Avalon-ST Source. The former handles the streaming data into the data path, and the latter handles the output data from the data path. The sink and source blocks bound the synthesizable region of the design, and tell DSP Builder which interfaces to expose in SOPC Builder. The Avalon-ST Sink block is shown in Figure 33. Figure 33. Avalon-ST Sink Block 32 Altera Corporation

33 Review the System Integration Using SOPC Builder The Avalon-ST Source block is shown in Figure 34. Figure 34. Avalon-ST Source Block In addition, the design contains two Avalon-MM Master interface blocks as shown in Figure 35. Figure 35. Avalon-MM Master Interface Blocks The Read and Write Avalon-MM masters bound the synthesizable region in a similar way to the Avalon-ST data interface bus limiters. Altera Corporation 33

34 Video and Image Processing Up Conversion Example Design The External RAM block is for simulation only, and is not synthesizable. The connection of the Deinterlacer to the external DDR2 memory is performed in SOPC Builder. The data and control interface's port blocks define the synthesizable boundary of the data path design that will interface to other system components in SOPC Builder. A description of your model is written out as a.mdlxml file when you run Signal Compiler. f Refer to these DSP Builder User Guide for information about compiling your design. A full compilation is not strictly necessary for integration of the design within SOPC Builder. You can alternatively use the DSPBuilder command alt_dspbuilder_mdl2xml to write the.mdlxml file by performing the following steps: 1. Save your example_design_data_path.mdl model file 2. At the MATLAB prompt, type the command: alt_dspbuilder_mdl2xml(gcs) Review the Final System using SOPC Builder In this section, you will review the Video and Image Processing Up Conversion example design using SOPC Builder, analyzing the individual modules that make up the system in the Quartus II software. The final system includes the following modules: Example Design Controller OpenCores I 2 C Master NTSC Composite Input Example Design Data Path (DSP Builder design) Timing and Data Format Adapters Frame Buffer VGA Output Pipeline Bridges DDR2 SDRAM Controller 34 Altera Corporation

35 Review the System Integration Using SOPC Builder To review the Quartus II project that describes the up conversion system, perform the following steps: 1. Run the Quartus II design software by choosing Quartus II 7.2 from the Programs section of the Windows Start menu. 2. Choose Open Project from the File menu. Browse to the directory where you installed the Video_IP_Example_Design and select the Video_IP_Example_Design.qpf file. (This file contains project definitions for the video up conversion example design.) 3. Choose Open from the File menu in the Quartus II software and select the file Video_IP_Example_Design.bdf. Click on Open to display this top-level file as shown in Figure 36. Figure 36. Top Level Video_IP_Example_Design.bdf file in Quartus II 4. In the Quartus II software, confirm that the Quartus II Fitter Physical Synthesis options are enabled (Assignments -> Settings -> Fitter Settings -> Physical Synthesis Optimizations) and the Physical Synthesis Effort is set to Extra. This is required to meet the system timing requirements. 5. Choose SOPC Builder from the Tools menu. Altera Corporation 35

36 Video and Image Processing Up Conversion Example Design The SOPC Builder window appears as shown in Figure 37. Figure 37. Final System with example_design_data_path from DSP Builder Integrated into a Video System SOPC Builder automatically detects the DSP Builder generated data path module because the.mdlxml file (example_design_data_path.mdlxml) is located in the same directory as the SOPC Builder component file (video_system_sopc.sopc). 1 You can specify the location of a DSP Builder system in another directory by adding this directory to the global libraries in the Quartus II project (Assignments -> Settings -> Libraries -> Global libraries). 36 Altera Corporation

37 Review the System Integration Using SOPC Builder The video up conversion data path module (which is located in DSPBuilder Systems > example_design_data_path_interface) refers to the synthesizable section of the DSP Builder system described in the previous section. (see Figure 38). It provides the following interfaces: Two Avalon-ST data interfaces: The Avalon_ST_Sink interface inputs the image pixels into the data path The Avalon_ST_Source interface outputs the processed image stream Two Avalon-MM control interfaces: The read master interface (Avalon_MM_Read_Master) to read data from the external DDR2 memory The write master interface (Avalon_MM_Write_Master) to write data to the external DDR2 memory Figure 38. Custom Hardware in the Video Up Conversion System Altera Corporation 37

38 Video and Image Processing Up Conversion Example Design 1 The DSP Builder up conversion subsystem could be built directly in SOPC Builder by wiring up MegaCore function blocks from the list of Video and Image Processing modules. However, DSP Builder is required to simulate the data path and perform early design verification. The custom hardware components: example_design_controller, ntsc_composite_input, opencores_i2c_master, and vga_output appear in the Video IP Example Design module list. The Frame Buffer BETA MegaCore function block is available in the list of Video and Image Processing modules. These components are described in the Functional Description on page 8. The Frame Buffer BETA block is added to the design and configured in SOPC Builder. 6. Click on the my_alt_vip_vfb instance in the SOPC Builder Module Name column to open the parameterization interface for the Frame Buffer BETA block (Figure 39). Figure 39. Parameters for the Frame Buffer MegaCore Function 38 Altera Corporation

39 Review the System Integration Using SOPC Builder Most of the parameters are self explanatory: The input resolution is with the RGB color samples streamed as 3 8-bits in parallel. The triple buffering function is configured to repeat frames. This is necessary because the VGA output cannot run slower than 60 fps and the output of the upscaling video data path is progressive 30 fps but frame dropping is not necessary and is consequently not allowed. To perform its function, the frame buffer uses 7.4Mbyte of memory at base address 0x The FIFO depth and burst target parameters configure the behavior of the Memory Writer and Memory Reader components shown in Figure 9 on page 12. To transmit and receive data to and from the memory in bursts, the Reader and the Writer components each have a small FIFO to store read and write data. The two FIFO depth parameters control the sizes of these FIFOs. Write requests from the Memory Writer are issued in sequence to the SOPC Builder arbitration logic layer when the write master FIFO has buffered at least as many words as specified by the burst target parameter. The Memory Writer then keeps issuing write requests until its FIFO is empty. The Memory Writer also starts a write burst when it reaches the end of a line of pixels regardless of the number of data words currently stored in the FIFO. Read requests from the Memory Reader are issued when the read master FIFO contains at least as many available spaces as specified by the burst target parameter. Read requests that have been issued but have not yet been fulfilled have a reserved place in the FIFO and do not count as available space. The Memory Reader stops issuing read requests when the FIFO is full, including pending read requests. 7. Click Cancel to close the Frame Buffer BETA block dialog box. Two Avalon-MM Pipeline Bridge components are used between the memory master interfaces of the Deinterlacer and Frame Buffer Megacore functions and the DDR2 SDRAM Controller Megacore function. 8. Double-click on pipeline_bridge to open the parameterization interface for the first Pipeline Bridge component (Figure 40 on page 40). 9. Click Cancel to close the Avalon-MM Pipeline Bridge dialog box. 10. Double-click on pipeline_bridge_1 to open the parameterization interface for the second Pipeline Bridge component. Altera Corporation 39

40 Video and Image Processing Up Conversion Example Design Figure 40. Parameters for pipeline_bridge Notice that the sizes of the data ports for the Pipeline Bridge components are different. The data width is 128 for the Pipeline Bridge connected to the Frame Buffer and 64 for the Pipeline Bridge connected to the up conversion subsystem built with DSP Builder. This is consistent with the parameterization of the Deinterlacer and Frame Buffer MegaCore functions. 11. Click Cancel to close the second Avalon-MM Pipeline Bridge dialog box. 12. Click Exit to close SOPC Builder after you review the components within the system but leave the Quartus II project open for the next section. Adding Files to the Quartus II Project Generating the SOPC Builder system, automatically adds all the HDL files associated with the custom hardware modules to the Quartus II software for compilation. 1 It may be necessary to regenerate the SOPC Builder system if the project is moved. See the Troubleshooting section DDR2 Fails in Analysis and Synthesis or In Timing Analysis on page Altera Corporation

41 Set Up the Hardware and Configure the FPGA Set Up the Hardware and Configure the FPGA 'This section covers the following steps: Set Up the Cyclone II Video Development Board Configure the Cyclone II Device Set Up the Cyclone II Video Development Board The hardware analysis requires the Cyclone II DSP Development Board, Video Input Daughtercard, power cable and USB Blaster cable included in the Video Development Kit, Cyclone II Edition. In addition, you require a composite input cable, VGA output cable, NTSC video source (for example, DVD player, ThinkPad T41, Video ipod, or video camera), and monitor with VGA input capable of displaying resolutions. To set up the Cyclone II DSP development board, perform the following steps: 1. Remove power from the Cyclone II DSP Development Board by disconnecting the power cable. 2. Connect one end of the USB cable to the USB port on your PC. 3. Connect the other end to the 10-pin header labeled (J21) on the Cyclone II DSP Development Board. 4. Connect the Video Input Daughtercard to the daughtercard connector on the Cyclone II DSP Development Board. 5. Connect the video source composite in cable to the input connector marked J41 on the video daughter card. 6. Connect one end of the VGA cable to a VGA monitor, capable of Hz. Connect the other end to the VGA connector labeled (J35) on the Cyclone II DSP Development Board. 7. Re-apply power to the Cyclone II DSP Development Board. f f For details of installing the USB Blaster software driver on the host PC (located at <quartus_install_dir>\drivers\usb-blaster), refer to the USB-Blaster Download Cable User Guide. For details of the Cyclone II DSP Development Board, refer to the DSP Development kit, Cyclone II Edition Getting Started User Guide. Altera Corporation 41

42 Video and Image Processing Up Conversion Example Design Configure the Cyclone II Device You can configure the Cyclone II device by downloading the SOF image to the DSP development board. To do so, perform the following steps: 1. In the Quartus II software, choose Programmer (Tools menu). 2. Choose Save As (File menu). 3. In the Save As dialog box, type Video_IP_Example_Design.cdf in the File Name box. 4. In the Save As type list, make sure you select Chain Description File. 5. Click Save. 6. In the Mode list of the Programming window, make sure JTAG is selected. 7. Click Hardware Setup to configure the programming hardware. Confirm that the Hardware Setup dialog box appears. 8. From the Hardware column, select USB Blaster. 9. Click Close to exit the Hardware Setup window. 10. In the Programming window, turn on the Program/Configure check box on the same line as Video_IP_Example_Design.sof. 11. Click Start. The programmer begins to download the configuration data to the FPGA. The Progress field displays the percentage of data that is downloaded. A message appears when the configuration is complete. 1 If you are not using a licensed version of the Video and Image Processing Suite, a message appears indicating that you are running a time-limited configuration file on your target hardware. 12. Press the USER RESET button on the board and confirm that a scaled video stream is displayed on the monitor. You have now successfully completed the Video and Image Processing Up Conversion design example walkthrough. 42 Altera Corporation

43 Conclusion Conclusion The Video and Image Processing Up Conversion example design provides a convenient framework for rapid development of video and image processing designs using the library of parameterizable MegaCore functions available in the Altera Video and Image Processing Suite. The FPGA supports high levels of parallel processing data flow structures that are important for efficient implementation of image processing algorithms. The suite of development tools that include DSP Builder and SOPC Builder allows you to build an image processing data path and integrate into a video system. Troubleshooting This section contains troubleshooting information for the following issues: DDR2 Fails in Analysis and Synthesis or In Timing Analysis Quartus II Compilation fails to meet F Max of 130MHz DDR2 Fails in Analysis and Synthesis or In Timing Analysis The example design makes use of the DDR2 MegaCore function. The Quartus II project includes an absolute path specifying the installed locations of this product and assumes that it has been installed in its default location (starting with c:\altera\72\ip). The DDR MegaCore function scripts include absolute file paths which are created when the MegaCore function variation is generated. For these reasons, it may be necessary to regenerate the MegaCore function if the project is moved in a different directory or a different computer. Use the following procedure to regenerate the MegaCore function variation if compilation fails because of DDR related errors: 1. Open Video_IP_Example_Design.bdf. 2. Double-click on Video_IP_Example_Design_SOPC_Component to open the SOPC System in SOPC Builder. 3. Make sure that the Quartus II user and global library paths are correct for the project (see Review the Final System using SOPC Builder on page 34). 4. Click Generate in SOPC Builder. 5. Close SOPC Builder and recompile the Quartus II project. Altera Corporation 43

44 Video and Image Processing Up Conversion Example Design Quartus II Compilation fails to meet F Max of 130MHz Ensure that the physical synthesis effort (Assignments -> Settings -> Fitter Settings -> Physical Synthesis Optimizations) is set to Extra in the Quartus II software as shown in Figure 41. Figure 41. Quartus II Physical Synthesis Assignments 44 Altera Corporation