What is Video? 2102642 Computer Vision and Video Electronics Chapter 7 Video Signals and Standards Suree Pumrin, Ph.D. 1 Video is a collation of images presented so fast they give the illusion of motion; A programme is divided into logical scenes, and a scene is composed of 1 or more camera shots; A shot is a single motion of a camera in time and can involve camera motion like panning, zooming, or tracking; Objects in a camera shot can also move; Shots can transition using fade in/out, fade to black, wipes, or the more usual hard cut 2 Type of Color Video Signals Component video -- each primary is sent as a separate video signal. The primaries can either be RGB or a luminance-chrominance chrominance transformation of them (e.g.,., YIQ, YUV). Best color reproduction Requires more bandwidth and good synchronization of the three components Composite video -- color (chrominance) and luminance signals are mixed into a single carrier wave. Some interference between the two signals is inevitable. S-Video (Separated video, e.g.,., in S-VHSS VHS) -- a compromise between component analog video and the composite video. It uses two lines, one for luminance and another for composite chrominance signal. Analog Video (I) Video information that is stored using television video signals, film, videotape or other non-computer media. Each frame is represented by a fluctuating voltage signal known as an analog wave form or composite video. Composite analog video has all the video components: brightness, colour and synchronization. Then combined into one signal for delivery. Usage: television Problems: color blending, low clarity, high generation lost, difficult to edit 3 4
Analog Video (II) Analog Television Standards Two main analog video formats NTSC (National Television Systems Committee) Consist of 525 scan lines drawn every 1/30 second Early color transmitter and line amplifier non-linearities caused so many errors that NTSC was said to stand for, Not Twice the Same Color.. PAL was developed as an improvement to avoid differential phase and gain transmission problems. PAL (Phase Alternate Line) Standard television format used in the European, UK, Australia and a South Africa Consist 625 scan lines drawn every 1/25 second Television usually has a 4:3 aspect ratio For every 1 pixel down there are 1.333 pixels across Aspect ration is the comparison of width to height for a viewing area Digital TV has an aspect ratio of 16:9 (wide screen) Interlaced Scanning 5 6 Color Difference Signals Video signals consist of Luminance, Chrominance, and Synchronization. Luminance and Chrominance are derived (matrixed) from the Red, Green, and Blue video signals. Luminance is the electrical signal that approximates the perception of brightness. It is also the signal that corresponds to the black and white (monochrome) video signal. Chrominance is derived from the color components of video reduced to two Color Difference signals. The two color difference signals have three important characteristics: They are not redundant. They disappear on monochrome source material. They can be bandlimited without noticeable effect. 7 8
9 10 Simple TV Horizontal Line Waveform Simple TV Vertical Waveform One raster line of the analog video signal consists of a continuous voltage that varies as the luminance of the scene, left to right, a negative going pulse denoting the initiation of retrace, and two short periods of blanking to ensure that edges of the (usually off screen) signals are truly black. 11 The vertical blanking period starts at the bottom of the raster. A series of vertical blanked lines (with serrations double synchronization pulses at blanking level), then an extended negative going serrated pulse to initiate vertical retrace, then a long blanking interval before active video begins top to bottom. 12
NTSC, PAL, AND SECAM The three international standards are more similar than different. All three systems reduce the bandwidth of the chrominance signals. All are compatible to B & W and consist of baseband luminance and two modulated chrominance signals. NTSC and PAL use analog quadrature modulation of I and Q signals. PAL and SECAM alternates color difference signals on alternate lines. PAL shifts the color signal for color subcarrier +/- 90 degrees on alternate lines. SECAM uses frequency modulation for color subcarrier. 13 Computers and Media Today, movies, TV, music, images are all created professionally in digital format; Domestic consumers also create and use media digitally: video (camcorder home movies, digital TV, DVDs) images (digital cameras, professional journalists) music (MP3 music, SoulSeek CDs) text of course So media and computers are pretty inseparable now and media and networks and computers are becoming inseparable; 14 Video Camera Input Video camera input In recent past, when you input video, needed a video capture card Video signals are analog - like TV, or non-digital video cameras Video capture boards transform analog video into digital and compress the file along the way Video files take up a lot of space Digital Video (I) Digital Video Files associated with video are MPEG, Quick Time, and AVI Digitized video is really a series of still images that are flashed to the screen Frame rate Need at least 24 frames per second to look like smooth action 15 16
Digital Video (II) Digital Video Input Digital Video Digital video cameras can connect to computer with USB or Firewire Don t t need a video capture board Videoconferencing Digital video technology enables people to conduct meetings via the Web People can see and communicate with each other Web cams Also Live cams that take pictures nearly continuously at a site and broadcast the pictures to the Internet 17 18 Video Conferencing (I) Video Conferencing High Quality Video Conferencing Usually video conferencing allows participants to use a whiteboard Part of the screen that allows shared input Can write or draw in this space For high quality video conferencing Need a high quality connection Dedicated link Yet, you can do low-resolution video conferencing via the Internet 19 20
Video Conferencing (II) Video Conferencing Web cam is needed to do low-resolution video conferencing Sits on top of the computer monitor All participating sites need a Web Cam and video conferencing software Microsoft Netmeeting works ok Images are small and can be jerky 21 Digital Video Encoding Video is 25 fps of synchronised images and audio; To display a single image of TV-quality video requires 720 Kbytes, so without compression this is 97.2 GBytes for a 90 minute movie; That s s huge, so clearly video must be compressed!!! There are formats such as.avi, QuickTime,.ram,.rm. (Real Networks),.wma. wma,,.wmp. wmp, but the ones that matter for video coding are the MPEG family; 22 Image and Video Coding Scope of Image and Video Coding Standards Only the Syntax and Decoder are standardized: Optimization beyond the obvious Complexity reduction for implementation Provides no guarantees of quality Input (image / video) Output (image / video) Pre- Processing Post-Processing & Error Recovery Encoding Decoding Scope of Standard Standard H. 261 H. 263 MPEG 1 MPEG 2 Low Main High 1440 High MPEG 4 MPEG 7 SIF 4:2:0 4:2:2 4:2:0 4:2:2 4:2:0 4:2:0 Various - Table of standards Digitisation format CIF/ QCIF S-QCIF/ QCIF SIF Compressed rate X 64 kbps <64kbps <1.5Mbps <4Mbps <15Mbps <20Mbps <60Mbps <80Mbps <80Mbps <100Mbps 5kbps tens Mbps - Example applications Video conferencing over LANs Video conferencing over low bits rate channels VHS quality video storage VHS quality video recording Digital video broadcasting High definition TV (4/3) High definition TV (16/9) Versatile multimedia coding standard Content description standard 23 24
Video Encoding principles There are 2 basic components, intra-,, and inter-frame comparisons; All video (and image) encoding standards use intra- frame compression based on sub-sampling, sampling, choosing 1 pixel or calculating an average Chroma Subsampling How to decimate for chrominance? 4:4:4 No chroma subsampling,, each pixel has Y, Cr and Cb values. 4:2:2 Horizontally subsample Cr, Cb signals by a factor of 2. 4:1:1 Horizontally subsampled by a factor of 4. 4:2:0 Subsampled in both the horizontal and vertical dimensions by a factor of 2.. Theoretically, the chroma pixel is positioned between the rows and columns as shown in the figure. 4:1:1 and 4:2:0 are mostly used in JPEG and MPEG 25 26 Sampling Formats Video Coding Layer (VCL) 27 28
Motion Compensation (I) Second component is inter-frame comparison; Motion compensation is identifying motion between adjacent frames and transmitting only the differences except shot bounds; Doing this on pixels is too fine-grained because cameras boom, tilt, pan, zoom, shake, and objects move, so pixel- pixel overlays would always have differences; Frames are divided into pixel aggregates called blocks and motion compensation is computed between equivalent blocks; This allows a graceful and effective encoding of deliberate camera and object motion Motion compensation (II) Camera panning right Notice equivalent blocks 29 30 Similar to chessboard moves Predictive Coding I B P B P Intra (I) frame, Predicted (P) frame, Bidirectional (B) frame 31 32
Evolution of MPEG standards MPEG-1 ( 93)... for 1.5 Mbit/s transmission, near VHS- quality (0.81 Gigabytes for 90 min). Encoding based on blocks; MPEG-2 2 ( 95)(... allows data rates up to 100 MBit/s. Used for digital TV, DVDs and professional broadcasting units; MPEG-4 4 ( 01)(... suitable for lower data rates between 10 KBit/s and 1 MBit/s.. Good quality object-based based encoding that will enable object-based based interactions for users, but just out of reach for now; MPEG 1 Video compression standard Adopts similar compression to H.261 Digitisation format SIF (525 line, 625 line) Types of frames supported I, P, & B frames 33 34 MPEG-1 1 encoding 352x288 pixels per frame @ 25 fps giving near-vhs quality at 1.5 Mbps; can be decoded on an (old) PC, encoding requires hardware; MPEG stream has I-, I, P-P and B-frames B in a given pattern; I-frame is a JPG image; each frame divided into 16x16 pixel blocks and in B-B and P-P frames, equivalent blocks are compared and a motion vector generated if possible; MPEG-1 1 encoding I-,, P-P and B-frames B form a pattern depending on the encoder used ours has an I-frameI every 12 frames (2 per sec); Encoders are not perfect and the 396 motion vectors in a frame (1 per macroblock) ) can sometimes be incorrect and have rogues giving pixelation; MPEG-2 2 is the same principle except 720x576 pixels and is used for digital TV; MPEG-4 4 is object based compression, based on identifying, tracking and encoding object layers which are rendered on top of each other, with huge potential for interaction; 35 36
MPEG 2 Main profile at the main level Digital broadcasting Digitisation format 4:2:0 Bite rate 5 Mbps 15Mbps Frame types I, P & B IBPBPBPBPBPBPBP (I frame every 0.5 sec) GOP 15 What about MPEG-4 4? MPEG-4 4 is object-based encoding Basically, analyse video, identify and track any moving objects across frames, compress the objects independently of the background Need to account for movement of the object across its background, and deformation of the object as it turns, rotates, changes 37 38 Object-based based compression Object-based based playback Original video Original video Video object planes Video object planes Composited video 39 40
Object-based based Recomposition Object based compression Original video Video object planes Composited video Object-based based video compression is really hard, computationally, requiring very clever video analysis, and we can t t do it automatically for general video Object-based based video processing appears on-screen in special effects in movies, TV adverts, but this is semi- automatic Automatic object-based based video processing appears in tennis and football matches (hoardings on pitch side, logo in centre-circle, circle, etc) which involves real-time processing 41 42 Object based compression You ll see MPEG-4 encoding Sony camcorders, but they re not object-based; based; When we do get object-based based video compression then that will open huge opportunities for interactive access with video content, but not yet; Meanwhile, even flat MPEG-4 4 offers reduced bandwidth and higher quality than MPEG-1 43 How much video is there? Up to 200 channels of content available per day, offering 20,000+ hours of content per week, if we assume an average of 15 hours programming per day per channel; There are 48 million hours per year of unique broadcasts from the 33,000 TV stations that operate simultaneously world-wide; wide; 2,000 TV stations in the US with 3.6M hours of content; 14 TB of info from TV series worldwide There are 4,500,000 hours per day of surveillance video from 14,000 airport terminals alone. Add in CCTV from cities, campus Then factor in home movies from camcorders 44
So what do we do with video? Video capture is a solved problem; Video editing is a professional s s task; Video storage is easy because of compression standards; Video distribution is easy; Video playback is easy; What s s left? Background Centre for Digital Video Processing (CDVP) concentrates on developing techniques for video analysis and supporting indexing, browsing, retrieval and summarisation, of digital video. Build a system which is used by real users with real information needs. finding the stuff! 45 46 Centre for Digital Video Processing (CDTV) Most CDVP research goes into a single demonstration system called This does recording, analysis, searching, browsing, and playback of digital video, currently broadcast TV (but with strictly limited access), and also educational videos, TV news, and nursing videos Users selected programmes from TV schedule, capture video, detect shots, scenes and keyframes Users browse programme keyframes and/or search for content, and then playback CDVP records TV News, the teletext,, and closed captions, aligned with video. CDTV uses the text as a basis for simple search. Problems solved? No, because that s s just using the closed caption text; Its not analyzing the video (or image) CONTENT; Some project examples: Using a trinocular camera for pedestrian modeling; Using a combination of visible spectrum, and infra-red red spectrum cameras for object detection; Sports summarization; 47 48