Part II Redundant Dictionaries and Pursuit Algorithms

Size: px

Start display at page:

Download "Part II Redundant Dictionaries and Pursuit Algorithms"

Barbra Adams
8 years ago
Views:

1 Aisenstadt Chair Course CRM September 2009 Part II Redundant Dictionaries and Pursuit Algorithms Stéphane Mallat Centre de Mathématiques Appliquées Ecole Polytechnique

2 Sparsity in Redundant Dictionaries Bases are minimum set to decompose signals. Natural languages use redundant dictionaries. Use of larger dictionaries incorporating more patterns to represent complex signals f R N D = {φ p } p Γ with φ p = 1 and Γ = P > N. How to construct sparse representations in D? What is the impact of redundancy on the mathematics and applications?

3 Let D = {φ p } p Γ Dictionary Approximation be a redundant dictionary of P vectors. The best approximation of f from a sub-family orthogonal projection in V Λ = Vect{φ p } p Λ f Λ = a[p] φ p p Λ {φ p } p Λ is its Stability: {φ p } p Λ must be a Riez basis of V Λ : there exists 0 <A Λ B Λ such that a[p] R Λ, A Λ a[p] 2 a[p] φ p 2 B Λ a[p] 2. p Λ p Λ p Λ

4 Best M-Term Approximation The best M-term approximation support Λ minimizes f f Λ with Λ = M. A best M-term approximation minimizes a Lagrangian: L 0 = f f Λ 2 + T 2 Λ. If is an orthonormal basis then D Λ = {p : f,φ p T } In general, finding Λ is an NP-hard problem.

5 Compression Applications Compute a best M-term approximation f Λ = p Λ a[p] φ p with Λ = M. Compression with uniform quantization f Λ = p Λ Q(a[p]) φ p Total bit budget: R = log 2 ( P M ) + µm R M log 2 (P/M) Increasing P reduces D = f f Λ but increases R. If f f Λ 2 = O(M α ) then D(R) =R α log(p/r) α.

6 Wavelets for Cartoon Images Theorem: If f is uniformly approximation gives C α f f M 2 = O(M α ) then an M-term wavelet Theorem: If f is piecewise C α with finite length contours then f f M 2 = O(M 1 ) so D(R) =R 1 log(n/r) Result valid for all bounded variation functions. f j =-8 f, ψ H jn f h x j = -7 j = -6 y f, ψv jn f, ψjn D (b) Wavelet Transform (c) Zoom γ

7 Bandlets for Geometric Regularity Wavelet coefficients inherit the geometric regularity f j =-8 j = -7 j = -6 f, ψ H jn f h y f, ψv jn f, ψjn D (b) Wavelet Transform (c) Zoom γ 1. Segmentation of wavelet coefficients. b Regular square Edge square Small Edge square 2. Geometric flow in edge squares along the direction of regularity. w (a) (a ) Wavelet Transform Quadtree (a) 3. 1D wavelet S transform along 3 S the flow = 2 (b) (c) (d) bandlet transform Corner square S 1 S 2 (b ) (c ) (d ) (b ) (c ) ( S 1 (b) (c) ( S

8 Bandlet Approximations A bandlet dictionary is a union of orthonormal bases. The best bandlet approximation (best geometry) which minimizes L 0 = f f M 2 + T 2 M can be computed with O(N log N) operations. C α C α Theorem If f is piecewise with piecewise contours f f M 2 = O(M α ) so D(R) =R α log(p/r) α

9 Application of Photo ID Compression

10 just nice...

11 Xlets Beyond Wavelets Xlets take advantage of the image geometric regularity: bandlets, curvelets, contourlets, edglets, wedglets... Failure to improve wavelet approximations for natural images. Can wavelet approximations be asymptotically improved? I don t think so for static natural images.

12 Denoising in Redundant Dictionaries Measure a signal plus a Gaussian white noise: X[n] =f[n]+w [n] for 0 n < N. Orthogonal projection estimator in selected in a dictionary: V Λ = Vect{φ p } p Λ X Λ = p Λ a[p] φ p. Risk: If Λ is fixed then E{ W Λ 2 } = σ 2 Λ. Oracle choice: find Λ which minimizes E{ f X Λ 2 } = f f Λ 2 + E{ W Λ 2 }. E{ f X Λ 2 } = f f Λ 2 + σ 2 Λ

13 Penalized Estimation Choosing Λ is a model selection problem. Theorem: For satisfies T>σ 2 log e P ( ) Λ = arg min X X Λ 2 + T 2 Λ Λ Γ E{ f X Λ 2 } = 4 min ( f f Λ 2 + T 2 Λ + σ 2) Λ Γ Noisy image Wavelet thresholding Bandlet thresholding PSNR = 22db PSNR = 25.3db PSNR = 26.4db

14 Let D = {φ p } p Γ Greedy Matching Pursuits be a dictionary of P > N vectors. A best M-term approximation minimizes the Lagrangian: L 0 = f f Λ 2 + T 2 Λ Finding Λ is in general an NP complete problem. Greedy choice of the approximation vectors. The approximation of f over φ p0 D yields f = f,φ p0 + Rf f 2 = f,φ p0 2 + Rf 2 To minimize the residual error we choose φ p0 = arg max φ p D f,φ p

15 Matching Pursuit Iterations Initialize For each R 0 f = f m>0 φ pm = arg max φ p D Rm f,φ p R m f = R m f,φ pm + R m+1 f R m f 2 = R m f,φ pm 2 + R m+1 f 2 It results: f = f 2 = M 1 m=0 M 1 m=0 R m f,φ pm + R M f R m f,φ pm 2 + R M f 2

16 Time-Frequency Decompositions Dictionary of time-frequency atoms: D = { φ p (t) = 1 w(t u } 2 j 2 j ) e iξt (j,u,ξ) Γ It includes Dirac and Fourier bases, wavelets and window Fourier atoms. $ # " ^ ($)! % t % $ " ( t )! 0 u t

17 Time Frequency Matching Pursuits f(t) ! 2" 250 t t 1

18 f(t) Speech Signals 1000 t ! 2" t 0 1 Applications: denoising, compression (video), pattern recognition, but instabilities.

19 Orthogonal Matching Pursuit Initialize For each R 0 f = f m>0 φ pm orthogonalize the projections: = arg max φ p D Rm f,φ p u m = φ pm m 1 l=0 φ pm,u l u l 2 u l. R m+1 f = Rm f, u m u m 2 + R m+1 f After N iterations, it gives decomposition in an orthogonal basis: f = N 1 m=0 R m f, u m u m 2.

20 Matching versus Basis Pursuit Greediness is not optimal Signal f Time-Frequency Coefficients Matching Pursuit Basis Pursuit Orthogonal Matching Pursuit

21 Basis Pursuit Find a sparse representation f = a[p] φ p ( ) p Γ 1/q by minimizing an l q norm : a q = a[n] q l 0 n Γ norm : number of non-zero a[n] Not convex for q < 1. l 1 norm : a 1 = a[n]. n Γ. l q balls q =0 q =0 q q =05.. q q =1 q = q =2

22 Basis Pursuit Approximation A basis pursuit computes a sparse approximation f = p Γ solution of a convex optimization problem: ã = arg min a R P a 1 ã[p] φ p Lagrangian formulation: find subject to f p Γ ã which minimizes L 1 (a) = 1 2 f p Γ a[p] φ p 2 + T a 1 a[p]φ p ɛ.

23 Basis Pursuit Denoising Original Image Noisy Image Wavelet Thresh. Basis Pursuit with Wavelet Cosine Diction.

24 Computation of l1 Minimization Many iterative algorithms to minimize L 1 (a) = 1 2 f p Γ a[p] φ p 2 + T a 1 Simplest one: iterative thresholding. Initialisation b = Φf = { f,φ p } p Γ and a 0 =0. For k>0 Gradient step: ã k = a k + γ(b ΦΦ a k ) ( Soft thresholding: a k+1 =ã k max 1 γt ) ã k, 0.

25 Exact Recovery Suppose that the signal is sparse f = p Λ can we recover Λ with pursuit algorithms? With matching pursuits, need that Exact Recovery Criteria a[p] φ p = p Λ f,φ p φ p V Λ. C(R m f,λ c )= max q Λ c R m f,φ q max p Λ R m f,φ p ERC(Λ) = sup C(h, Λ c ) = max h V Λ q Λ c p Λ < 1. φ p, φ q.

26 ERC Recovery Theorem: The approximation support Λ of f V Λ is exactly recovered by an orthogonal matching pursuit or a basis pursuit if ERC(Λ) < 1. Theorem (stability): If f is not exactly sparse, an orthogonal matching pursuit f M with M = Λ iterations satisfies: f f M ( 1+ Similar result for a basis pursuit. Λ ) A Λ (1 ERC(Λ)) 2 f f Λ.

27 Dictionary Coherence Dictionary mutual coherence µ(d) = sup φ p, φ q (p,q) Γ 2 Dirac-Fourier dictionary: µ(d) =N 1/2. Theorem: ERC(Λ) < 1 if Λ < 1 2 ( 1+ 1 ) µ(d) Exact recovery is possible for sufficiently sparse signals in incoherent dictionaries.

28 2nd. Conclusion Redundant dictionaries can improve approximation, compression, denoising. Finding optimal approximation is NP complete but can be approximated with matching or basis pursuits. May be used for pattern recognition but problems of instabilities. The stability depends upon the dictionary coherence. Major applications to inverse problems, superresolution and compress sensing.

Disjoint sparsity for signal separation and applications to hybrid imaging inverse problems

Disjoint sparsity for signal separation and applications to hybrid imaging inverse problems Giovanni S Alberti (joint with H Ammari) DMA, École Normale Supérieure, Paris June 16, 2015 Giovanni S Alberti