CCD. 500,000 to 10,000,000 Photosites

Transcription

1 Digital Video

2 Video Starts off as Analog Information Just as in Imaging, the information contained in video is analog by nature Intensity Color Speed / Motion (24/30 Frames per Second) Digital Technology allows us to convert it to bits, store it and manipulate it much easier than its analog counterpart

3

4 The CCD Imager CCD 500,000 to 10,000,000 Photosites

5 Converting the Video Frame to Bits Think of Video Frames as individual Images, stacked front to back 30 frames/sec!

6 Producing Digital Video Video capture Editing Playback

7 Advantages and Disadvantages of Digital Video Advantages Scalable to different playback systems Random access to frames Easy to Edit More playback options Potential for interactivity Disadvantages Special hardware/software needed for production High playback and storage requirements

8 Compression: Coping with Large Files Compression is an encoding process that filters the original file in several successive stages Without powerful compression we would NOT be able to produce CDs, DVDs, or Video Downloads over the Internet

9 Digital Compression Concepts Compression techniques are used to replace a file with another that is smaller Decompression techniques expands the compressed file to recover the original data -- either exactly or in facsimile A pair of compression/decompression techniques that work together is called a codec for short

10 Types of Codecs Codecs that upon decompression always reproduce the original file exactly are called lossless codecs Codecs that reproduce only an approximation of the original file upon decompression are called lossy codecs Codecs that take approximately the same amount of time to compress and decompress a file are referred to as symmetric codecs By contrast, codecs that feature simple fast decompression but significantly slower compression are called asymmetric codecs

11 Codec Methods Syntactic encoding (also called entropy encoding) ) methods attempt to reduce the redundancy of symbolic patterns in a file without any regard to the type of information represented Semantic methods consider special properties of the type of information represented to reduce nonessential information in a file Hybrid methods combine both syntactic and semantic methods

12 Video Compression The basic assumption in video compression is that the image is almost the same as the last one, so only record the differences. More complicated schemes look both forward and backward, that is, they also compare to key images that are coming up.

13 Other Brute Force Methods for Reducing Demands Frame rate adjustment adjusts for with slower CPUs helps keep video and audio synchronized Lower resolution on individual frames sometimes used in conjunction with smaller display window

14 Compressing Video Video compression employs both spatial and temporal compression spatial techniques compress individual frames temporal methods compress data in frames over time QuickTime and AVI (Audio Video Interleaved) are two popular (and incompatible with each other) formats used on PCs

15 Temporal Compression in Video Lossy strategies for eliminating redundancy of information between frames employ temporal compression -- referred to as interframe compression Sequence of frames are considered together key frames difference frames Used in QuickTime and DVI

16 Video compression standards Mpeg1 up to 1.5 mbits/sec Popular for web videos Mpeg2 Current Standard for DVDs Up to 15 mbits/sec Mpeg3 New Multimedia standard Gaining popularity for higher quality video on the web

17 MPEG 2- The Mother of all Video Compression!.so far Uses : Temporal and Spatial Redundancy Basically it predicts what subsequent frames of video are going to be based on previous and future frames It encodes that knowledge such that only one out of 12 frames has a complete set of digital binary information.the others have a combo of binary and vector information 40:1 Compression Ratio..makes DVDs possible

18 DVD Digital Versatile Disk Up to 133 minutes of high-resolution video, in letterbox or pan-and-scan format, with 720 dots of horizontal resolution (The video compression ratio is typically 40:1 using MPEG-2 compression.) Soundtrack presented in up to eight languages using 5.1 channel Dolby digital surround sound Subtitles in up to 32 languages

19 High Definition Video Much higher resolution and quality 1980 by 1080 pixels resolution HD video has been introduced to the market Limited HD material available but increasing on cable, satellite and over the air HD DVDs being designed These DVDs will hold up to 50 GB of information! Analog over the air channels being phased out in favor of digital broadcasts Uses MPEG2 or MPEG4 Compression

20 The Desktop Video System Basic Components Analog Source Video Capture Card CPU Secondary Storage Monitor Edit and Playback Control

21 Editing Digital Video Clip Logging Assembling Transitions dissolves wipes, etc. Rotoscoping Frame Editing (Digital Effects) Compositing keying titling

22 Compositing and Rotoscoping..First we have a Mountain

23 Then we design a plane

24 Mountain and Plane..Together!!

25 Digital Video The Entire Process Illustrated

26 Video Overview Video (like audio) requires analog signal to be digitized

27 Video Overview Analog video (videotape) is linear it has a start, middle, and end Digitized video allows random access DVD is a good example

28 Video Overview Video file size Frame rate (TV 30 frames/second) Image size Color depth Frames/second x image size x color depth / 8 = estimated file size of 1 second of video

29 Video Compression Two types of compression Lossless preserves the exact image (large file) Lossy eliminates data Standards established for compression MPEG (Motion Pictures Experts Group) 20:1 compression ratio (1MB to 50KB)

30 Video Compression Compression software programs Apple s QuickTime (.MOV) Lossy compression with 5:1 to 25:1 ratios Windows Media Player (.AVI) Uses Audio Video Interleave (AVI) Format Examples

31 Video Compression Incorporating transitions Superimposing titles and animations Applying special effects Synchronizing sound with the video Applying filters for blurring, morphing, etc.

32 Video on the Web Can be a drawback instead of an asset Animation or still images should be explored as an option Quality vs. cost

33 Video on the Web Video transfer types HTTP (HyperText Transfer Protocol) Small video clips RTSP (Real-Time Streaming Protocol) Continuous-playing technology Large video files and live broadcasts

34 Video on the Web Streaming video very popular Reduce start time significantly Players RealPlayer Windows Media Player QuickTime Player File formats (AVI and MPEG)

35 Video Data Size size of uncompressed video in gigabytes 1920x x x x x120 1 sec min hour hours 671, , , , , image size of video 1280x720 (1.77) 640x480 (1.33) 320x x120

36 Video Bit Rate Calculation width * height * depth * fps compression factor = bits/sec width pixels(160, 320, 640, 720, 1280, 1920, ) height pixels (120, 240, 480, 485, 720, 1080, ) depth bits (1, 4, 8, 15, 16, 24, ) fps frames per second (5, 15, 20, 24, 30, ) compression factor (1, 6, 24, )

37 Effects of Compression storage for 1 hour of compressed video in megabytes 1920x x x x x120 1:1 671, ,598 99,533 24,883 6,221 3:1 223,949 99,533 33,178 8,294 2,074 6:1 111,974 49,766 16,589 4,147 1,037 25:1 26,874 11,944 3, :1 6,718 2, bytes/pixel, 30 frames/sec

38 Be Careful... mpeg 200:1, jpeg 24:1 analog source digital representation compressed representation vs

39 Data Rate Another View Size/Hour 128 Kbs 60 MB 384 Kbs 170 MB 1.5 Mbs 680 MB 3.0 Mbs 1.4 GB 6.0 Mbs 2.7 GB 25 Mbs 11.0 GB

40 Perceptual Coding Encode source signal using lossy compression Lossless algorithms typically reduce signal by 3:1 Must use lossy algorithm to get adequate compression Hide errors where humans will not see or hear it Study hearing and vision system to understand how we see/hear Masking refers to one signal overwhelming/hiding another (e.g., loud siren or bright flash) Audio perception is khz but most sounds in low frequencies (e.g., 2 khz to 4 khz) Visual perception influenced by edges and low frequencies

41 What is JPEG - Joint Photographic Experts Group Still image compression, intraframe picture technology MJPEG is sequence of images coded with JPEG MPEG - Moving Picture Experts Group Many standards MPEG1, MPEG2, and MPEG4 Very sophisticated technology involving intra- and interframe picture coding and many other optimizations => high quality and cost in time/computation H.261/H.263/H Video Conferencing Low to medium bit rate, quality, and computational cost Used in H.320 and H.323 video conferencing standards

42 Coding Overview Digitize Subsample to reduce data Intraframe compression Remove redundancy within frame (spatial compression) Interframe compression Remove redundancy between frames (temporal compression) Symbol coding Efficient coding of sequence of symbols

43 Digitizing Modify color space 24 bit RGB => 15 or 16 bit RGB 24 bit RGB => YUV (8 bit Y, 4 bit U, 4 bit V) 24 bit RGB => 8 bit color map Encode only 1 field Reduce frame rate Film is 24 fps so why encode 30 frames of video Is 20 fps good enough? 18? 15? 12? 8? 4?... Variable frame rate?

44 Block Transform Encoding DCT Zig-zag Quantize Run-length Code Huffman Code

45 original image Block Encoding DCT AC components run-length code Huffman code DC component zigzag Quantize coded bitstream < 10 bits (0.55 bits/pixel)

46 Result of Coding/Decoding original block reconstructed block errors

47 Discrete Cosine Transform F[u,v] = where 4C(u)C(v) n 2 n-1 n-1 j=0 k=0 1 C(w) = 2 1 f(j,k) cos for w=0 (2j+1)uπ 2n for w=1,2,,n-1 cos (2k+1)vπ 2n Inverse is very similar DCT better at reducing redundancy than Discrete Fourier Transform but computationally expensive

48 DCT vs DFT original signal recovered from DCT recovered from DFT

49 Inter-frame Compression Pixel difference with previous frame If previous pixel very similar, skip it Send sequence of blocks rather than frames If previous block similar, skip it or send difference Motion compensation Search around block in previous frame for a better matching block and encode position and error difference Search in future frame Average block in previous and future frame

50 Background JPEG still image (1 bpp) Symmetric codec MPEG Interleaved audio and video Low cost decoder at expense of high cost encoder (asymmetric) H.26x Video conferencing standard (QCIF, CIF, 4CIF, and 16CIF) Variable bit rates and coding flexibility

51 MPEG Standards MPEG1 - vhs quality (1992) CIF images, 4:2:0 sampling, 1.5 Mbs Frame encoding MPEG2 - broadcast quality (1994) CCIR 601 images, 4:2:2 sampling, 15 Mbs Interlaced and progressive scanning Frame and field encoding

52 MPEG Technology Picture coding types Motion compensation Bit rate control Picture order

53 Do MPEG slides HERE Frame types I, P, B, and Bi frames Jargon GOP, macroblock, slice Motion vectors Predicted and interpolated vectors

54 Compression Formats Cinepack (2 bpp) Inter-frame, RGB15, software playback Motion JPEG (1-4 bpp) Intra-frame, DCT+Quantization, good for editing MPEG (0.5-2 bpp) Inter-frame, DCT+Quantization+Motion Compensation, excellent for playback H.261/H.263 Inter-frame, DCT+Quantization+Motion Compensation, block stream, excellent for conferencing

55 Quantitative Measures of Quality Mean Squared Error n-1 n-1 1 MSE = [ f(i,j) - f (i,j) ] 2 n 2 i=0 i=0 where f(i,j) is original image and f (i,j) is image after compression and decompression Perceptual Measures Perceptual Distortion Measure (Heeger) Picture Quality Scale (Miyahara, et.al.) Spatial/Temporal Measure (Webster, et.al.)

56 Discussion Where are the lossy steps? Quantization and subsampling before coding How do you choose quantization matrix? Standard proscribes matrix based on psychophysics Vary quantization by scaling Q matrix (i.e. MQUANT) Custom designed Q matrix Can improve compressioin by using motion compensation (MC) Comparison of standards JPEG - still image uses fixed Q matrix H video conferencing uses MC and variable quantization MPEG - video playback uses MC and variable quantization