Compressive Sensing. Examples in Image Compression. Lecture 4, July 30, Luiz Velho Eduardo A. B. da Silva Adriana Schulz

Transcription

1 Compressive Sensing Examples in Image Compression Lecture 4, July, 09 Luiz Velho Eduardo A. B. da Silva Adriana Schulz

2 Today s Lecture Discuss applications of CS in image compression Evaluate CS efficiency Review important definitions and theorems

3 Software Matlab functions for CS imaging Optimization algorithms Data Demos aschulz/cs/course.html

4 Experimental Setup 4 images of size (a) Phantom (b) Lena (c) Camera man (d) Text

5 Frequency Distributions (a) Phantom (b) Lena (c) Camera man (d) Text

6 Acquisition Images are stored as a matrix of pixels

7 Noiselets RIP orthogonal self-adjoint Φ =

8 Sparsity 1 0 2

9 Reconstruction Acquisition: y = Φ Ω x

10 Reconstruction Acquisition: y = Φ Ω x

11 Reconstruction Acquisition: y = Φ Ω x Reconstruction: y = Θ Ω s s = min s s l1 s. t. x = Ψ s { y = Θs y Θs l2 ɛ

12 Reconstruction Acquisition: y = Φ Ω x Reconstruction: y = Θ Ω s s = min s s l1 s. t. x = Ψ s { y = Θs y Θs l2 ɛ

13 Basic CS Images are sparse in the Ψ domain No noise is added Ordinary images are only approximately sparse : (a)

14 Basic CS Images are sparse in the Ψ domain No noise is added Ordinary images are only approximately sparse : (a) (b) Figure: Different visualizations of Lena s DCT.

15 Forcing Sparsity DCT Example: (a) Original Image (b) Original DCT (c) Original DCT (d) Result Image (e) Result DCT (f) Result DCT

16 First Experiment x Ψ s force sparsity Ψ s S x S Acquisition: y = Φ Ω x S Reconstruction of x via l 1 minimization Compare x and x S using (Peak Signal to Noise Ratio)

17 First Result Measurements( 3 ) Figure: Basic CS experiment: k-sparse Lena. Theorem Reconstruction is exact if M S µ 2 (Φ, Ψ) log N

18 Varying S Measurements( 3 ) 3.5k Sparse 6k Sparse k Sparse 14k Sparse

19 DCT Linear Compression

20 DCT Linear Compression

21 DCT Linear Compression We measure M N coefficients on the upper-left corner

22 DCT Linear Compression We measure M N coefficients on the upper-left corner

23 DCT Linear Compression Reconstruction: x = Ψ s

24 DCT Linear Compression

25 DCT Linear Compression Problem: we don t measure all the significant coefficients and measure some zero coefficients!

26 CS: Linear + Noiselets Measurements Reconstruction: s = min s s l1 s. t. Φ Ω Ψ s = y x = Ψ s

27 Results: CS Linear Compression [DCT l N] [DCT l L] [DCT l LN] Measurements( 3 ) Measurements( 3 ) (a) 3.5K-Sparse (b) 6K-Sparse Measurements( 3 ) Measurements( 3 ) (c) K-Sparse (d) 14K-Sparse

28 Coherence Before Boundary: best to use linear compression After Boundary: best to use CS Measurements( 3 ) DCT l 1 N DCT l 1 LN

29 Different Images What happens when we consider the other images? Measurements( 3 ) Measurements( 3 ) (e) Linear Compression (f) Compressive Sensing (a) Phantom (b) Lena (c) Camera man (d) Text

30 Different Images What happens when we consider the other images? 3 (f) Compressive Sensing (a) Phantom 2 0 Measurements( ) (e) Linear Compression 0 3 Measurements( ) 1 (b) Lena (c) Camera man 0 1 (d) Text 0 2

31 Different Images What happens when we consider the other images? phantom lena cameraman text Measurements( 3) (a) Linear Compression (a) Phantom 2 0 (b) Compressive Sensing 0 0 Measurements( 3) 1 (b) Lena (c) Camera man 0 1 (d) Text 0 2

32 Different Sparsity Domains What happens when we consider different sparsity domains? nz = 00 (a) DCT nz = 00 (b) Block DCT nz = 00 (c) Wavelets

33 Different Sparsity Domains What happens when we consider different sparsity domains? nz = 00 (a) DCT nz = 00 (b) Block DCT nz = 00 (c) Wavelets Measurements( 3 ) DCT l 1 N B_DCT l 1 N DWT l 1 N

34 But this is not for real... All this happens because we FORCE sparsity! Theorem Reconstruction is exact if M S µ 2 (Φ, Ψ) log N Important Parameters: µ - Noiselets are highly incoherent with all considered domains S - We are tempering with it! - Lets stop and see what happens...

35 Sparsity Errors When Φ are Noiselet measurements the RIP holds and we can use: Theorem Let s S be the best S-sparse approximation of s. If the RIP holds, the solution s to s = min s s l1 subject to Θ Ω s = y obeys s s l2 1 S s s S l1

36 Let s verify this theorem empirically

37 Let s verify this theorem empirically Measurements( 3 ) k Sparse 6k Sparse k Sparse 14k Sparse Measurements( 3 ) Test image Lena Measurements Sparsity is forced Sparsity is not forced M = k S = 3.5k = 28.8 = 26.6 M = 25k S = 6k =.7 = 27.8 M = 35k S = k = 33.0 =.2 M = k S = 14k = 34.9 = 31.5

38 Let s verify this theorem empirically Measurements( 3 ) k Sparse 6k Sparse k Sparse 14k Sparse Measurements( 3 ) Test image Lena Measurements Sparsity is forced Sparsity is not forced M = k S = 3.5k = 28.8 = 26.6 M = 25k S = 6k =.7 = 27.8 M = 35k S = k = 33.0 =.2 M = k S = 14k = 34.9 = 31.5

39 Different Sparsity Domains [DCT l L] 2 [B_DCT l L] 2 [DCT l N] 1 [B_DCT l N] 1 [DWT l N] 1 [SVD l N] 1 [TV l N] 1 Measurements( 3 ) Measurements( 3 ) (b) Phantom (c) Lena Measurements( 3 ) Measurements( 3 ) (d) Camera man (e) Text

40 Different Sparsity Domains [DCT l L] 2 [B_DCT l L] 2 [DCT l N] 1 [B_DCT l N] 1 [DWT l N] 1 [SVD l N] 1 [TV l N] 1 Measurements( 3 ) Measurements( 3 ) (b) Phantom (c) Lena Measurements( 3 ) Measurements( 3 ) (d) Camera man (e) Text

41 Different Sparsity Domains [DCT l L] 2 [B_DCT l L] 2 [DCT l N] 1 [B_DCT l N] 1 [DWT l N] 1 [SVD l N] 1 [TV l N] 1 Measurements( 3 ) Measurements( 3 ) (b) Phantom (c) Lena Measurements( 3 ) Measurements( 3 ) (d) Camera man (e) Text

42 Different Sparsity Domains Total Variation (TV) min x x TV subject to y = Φ Ω x l 1 -norm of the (appropriately discretized) gradient.

43 Different Sparsity Domains [DCT l L] 2 [B_DCT l L] 2 [DCT l N] 1 [B_DCT l N] 1 [DWT l N] 1 [SVD l N] 1 [TV l N] 1 Measurements( 3 ) Measurements( 3 ) (b) Phantom (c) Lena Measurements( 3 ) Measurements( 3 ) (d) Camera man (e) Text

44 Different Sparsity Domains Singular Value Decomposition (SVD) Calculated for each image! Upper bounds - insights into performance limitations.

45 Different Sparsity Domains [DCT l L] 2 [B_DCT l L] 2 [DCT l N] 1 [B_DCT l N] 1 [DWT l N] 1 [SVD l N] 1 [TV l N] 1 Measurements( 3 ) Measurements( 3 ) (b) Phantom (c) Lena Measurements( 3 ) Measurements( 3 ) (d) Camera man (e) Text

46 Measurement Errors Measurements: y = Φ Ω x + n n is a random variable with normal distribution Theorem If the RIP holds and n l2 ɛ, the solution s to s = min s s l1 s.t. Θ Ω s y l2 ɛ obeys s s l2 1 S s s S l1 + ɛ

47 Gaussian Errors Measurements( 3 ) var = 0.1 var = 0.5 var = 1 var = 2 var = 3 var = 4 var = 5 var =

48 Compression Number of Measurements Number of Bits To compress we have to quantize the acquired data!

49 Quantization Each measurement may assume K values.

50 Quantization Larger quatization steps: less bits Smaller quatization steps: less errors

51 Rate Rate = number of bits per pixel Original Image: N pixels 8 bits/pixel Acquired Measurements: M N coefficients H y bits/coefficients Rate = M N H y bits/pixel If Rate < 8 data is being compressed!

52 The Rate Distortion Criteria

53 The Rate Distortion Criteria = which is the best compression scheme?

54 Rate Distortion 90 Measurements x 3 step = 0.01 step = 0.02 step = 0.05 step = 0.1 step = 0.2 step = 0.5 step = 1 step = 2 step = 3 step = 4 step = 5 step = step = step = step =

55 Rate Distortion Rate step = 0.01 step = 0.02 step = 0.05 step = 0.1 step = 0.2 step = 0.5 step = 1 step = 2 step = 3 step = 4 step = 5 step = step = step = step =

56 Rate Distortion Rate step = 0.01 step = 0.02 step = 0.05 step = 0.1 step = 0.2 step = 0.5 step = 1 step = 2 step = 3 step = 4 step = 5 step = step = step = step =

57 Rate Distortion

58 Rate Distortion Rate step = 0.01 step = 0.02 step = 0.05 step = 0.1 step = 0.2 step = 0.5 step = 1 step = 2 step = 3 step = 4 step = 5 step = step = step = step =

59 Both Sparsity and Measurement Errors A real example: real image and quantized measurements. Recovery Strategies: Ψ = DCT Ψ = block DCT Ψ = DWT TV- minimization Ψ = SVD - of each image!

60 Both Sparsity and Measurement Errors Jpeg00 DCT l N B_DCT l N DWT l N TV N SVD l N Rate Rate (b) Phantom (c) Lena Rate Rate (d) Camera man (e) Text

61 Sparsity Quantization Errors x x l2 ɛ q + 1 x S x l1 S, }{{} ɛ s The reconstruction error in CS is of the order of the maximum of the quantization (ɛ q ) and sparsity errors (ɛ s ).

62 (a) DCT (b) DWT (c) TV (d) SVD Figure: Rate of image Lena

63 (a) DCT (b) DWT (c) TV (d) SVD Figure: Measurements of image Lena

64 Lets wrap it up! CS Applications Image acquisition and compression Interesting, but Limitations... Tomorow Applications in graphics and vision One Pixel Camera Dual Photography