Sparsity & Co.: An Overview of Analysis vs Synthesis in Low-Dimensional Signal Models. Rémi Gribonval, INRIA Rennes - Bretagne Atlantique

Transcription

1 Sparsity & Co.: An Overview of Analysis vs Synthesis in Low-Dimensional Signal Models Rémi Gribonval, INRIA Rennes - Bretagne Atlantique remi.gribonval@inria.fr

2 2 METISS Rennes Speech and audio modeling and processing Sparse representations models and algorithms Audio and spoken content processing and description Audio source localization and separation Music signal and content processing R. GRIBONVAL - Journées INRIA Apprentissage

3 Sparse Signal / Image Processing + Compression, Source Localization, Separation, Compressed Sensing... R. GRIBONVAL - Journées INRIA Apprentissage 3

4 Geometry of Inverse Problems Signal space ~ R d «Interesting» signals x Nonlinear Approximation = Sparse reconstruction Linear projection y = Mx Thanks to M. Davies, U. Edinburgh Observation space ~ R m m<<d R. GRIBONVAL - Journées INRIA Apprentissage 4

5 Sparse Atomic Decompositions x Dz Signal Image (Overcomplete) dictionary of atoms Representation Coefficients R. GRIBONVAL - Journées INRIA Apprentissage 5

6 Supporting Evidence: Sparsifying Transforms Audio : time-frequency representations (MP3) ANALYSIS SYNTHESIS Black = zero Images : wavelet transform (JPEG2000) ANALYSIS SYNTHESIS White = zero R. GRIBONVAL - Journées INRIA Apprentissage

7 Sparsity and inverse problems Uniqueness guarantees R. GRIBONVAL - Journées INRIA Apprentissage

8 Mathematical foundations Bottleneck ( ) : fewer equations than unknowns = ill-posed... Theoretical & algorithmic breakthrough ( ) : Uniqueness of the sparse solution = well-posed! x 0, x 1 Ax 0 = Ax 1 x 0 = x 1 if are sparse enough, then Bounded complexity algorithms to reconstruct Ax 0 = Ax 1 x 0 = x 1 x 0 Thresholding, Matching Pursuits, Convex Optimization,... [Mallat & Zhang 1993] [Gorodnistky & Rao 1997] [Chen Donoho & Saunders 1999] R. GRIBONVAL - Journées INRIA Apprentissage

9 CoSparse models and inverse problems R. GRIBONVAL - Journées INRIA Apprentissage 9

10 CoSparse models and inverse problems R. GRIBONVAL - Journées INRIA Apprentissage 10

11 Analysis vs Synthesis: Cosparsity with S. Nam, M. Davies, M. Elad small-project.eu R. GRIBONVAL - Journées INRIA Apprentissage

12 Sparsifying Transforms = Atomic Decompositions? ANALYSIS SYNTHESIS Black = zero Analysis = transform z = Ωz small number of nonzero coefficients z 0 := k z k 0 Synthesis = decomposition x = Dz = k z k d k R. GRIBONVAL - Journées INRIA Apprentissage

13 Analysis priors for regularization / data separation 1 arg min x 2 y Mx2 2 + λωx p Much empirical evidence of success Ω with associated to TV norm, curvelet coefficients, undecimated Haar... Starck & al 2003, Elad & al 2007, Portilla 2009, Selesnick & Figueiredo 2009, Afonso, Bioucas-Dias & Figueiredo 2010, Pustelnik & al 2011,... Few theoretical guarantees Donoho & Kutyniok 2010, Candès & al 2010, Nam & al 2011, Vaiter & al Which signals can be well recovered? Better understand analysis model (New) analysis algorithms with performance R. GRIBONVAL - Journées INRIA Apprentissage

14 Equivalence? With tight frames Yes... but some troubling facts: only one analysis representation infinitely many synthesis representations R. GRIBONVAL - Journées INRIA Apprentissage

15 Analysis without synthesis the 2D finite difference operator Cousin of TV norm Rudin, Osher, Fatemi 1992 x =(x ij ) i +1,j i, j i, j +1 finite differences Ω DIF x V = H TV like reconstruction min x Ωx 1 s.t. y Mx 2 There is NO dictionary such that x = DΩx, x R. GRIBONVAL - Journées INRIA Apprentissage

16 Introducing the cosparse model Sparse synthesis model D z s.t. x = Dz Synthesis dictionary Representation Support = location of nonzeroes Cosparse analysis model Analysis operator Ω Representation Ωx Cosupport = location of zeroes k := sparsity = dimension of subspace := cosparsity = codimension of subspace R. GRIBONVAL - Journées INRIA Apprentissage 16

17 CoSparse models and inverse problems R. GRIBONVAL - Journées INRIA Apprentissage 17

18 CoSparse models and inverse problems Coparse model with tight frames = «structured» sparse model Union of subspaces model [Lu & Do 2008] Uniqueness results for cosparse model [Nam & al 2011] R. GRIBONVAL - Journées INRIA Apprentissage 18

19 (Co)sparse recovery algorithms R. GRIBONVAL - Journées INRIA Apprentissage

20 Optimization principles / algorithms Idealized problem Portilla 2009, Selesnick & Figueiredo 2009, Afonso & al 2010 Nam & al 2011 : uniqueness guarantees Convex relaxation Starck & al 2003, Elad & al 2007, Kutyniok & Donoho 2010, Candès & al 2010 Nam & al 2011, Vaiter & al 2011 : recovery guarantees Greedy analysis pursuit (GAP) ~ analysis-omp ~ agressive IRLS Nam & al 2011 : definition + recovery guarantees Iterative cosparse projections ~ analysis-iht Gyries & al 2011: definition + recovery guarantees ˆx A L0 := arg ˆx A L1 := arg min x:y=mx Ωx 0 min x:y=mx Ωx 1 + Noise aware variants with y Mx 2 + Structured co-sparsity R. GRIBONVAL - Journées INRIA Apprentissage 20

21 Results: finite difference operator Uniqueness guarantee m 2554 Fourier subsampling d = = slices = radial lines m = 3032 = 4.7% R. GRIBONVAL - Journées INRIA Apprentissage 21

22 Results: finite difference operator (analysis) Linear dependencies = fewer small-dimensional subspaces (links with matroids?) R. GRIBONVAL - Journées INRIA Apprentissage 22

23 Robustness to approximate cosparsity Second order 2D finite difference operator 50 Fourier lines Original TV GAP R. GRIBONVAL - Journées INRIA Apprentissage

24 What s next? R. GRIBONVAL - Journées INRIA Apprentissage

25 Summary Traditional Synthesis Model Alternate Analysis Model Synthesis dictionary of atoms x = Dz = i zi di z 0 dimension Analysis operator ωi, x = 0 for many rows of Ωx 0 dimension Ω «Lego» model: building blocks «Carving out» model: constraints Low-dimension = few atoms Low-dimension = many constraints Ex: man-made codes in communications Ex: coupling with laws of physics 2 ( x R. GRIBONVAL - Journées INRIA Apprentissage 25 1 c2 t2 x) D =0

26 What s next? New cosparse recovery algorithms with guarantees GAP & Analysis-L1 [Nam & al 2011 a, Vaiter & al ] Analysis Iterative Hard Thresholding [Gyries & al 2011]... Group cosparsity [Nam & al 2011 b] Hybrid sparse / cosparse models [Figueiredo & al 2011] Learning / designing dictionary / analysis operator [Ophir & al 2011, Yaghoobi & al 2011, Peyré & Fadili 2011] [Nam & al 2011 b] R. GRIBONVAL - Journées INRIA Apprentissage

27 Data Deluge + Jungle Sparsity: historically for signals & images bottleneck = large-scale algorithms Signals Images Graph data Social networks Brain connectivity Hyperspectral Satellite imaging Spherical geometry Cosmology, HRTF (3D audio) New exotic or composite data bottleneck = dictionary/operator design/learning Vector valued Diffusion tensor R. GRIBONVAL - Journées INRIA Apprentissage

28 Scientific challenges... Fundamental objectives 1. Understand the coupling between sparsity and underlying physical phenomena 2. Exploit underlying structures & invariances (graphs, geometric manifolds,...) 3. Build generic formalisms to model composite data (multichannel, multimodal,... ) A key challenge : Data-adaptive models through learning from a corpus R. GRIBONVAL - Journées INRIA Apprentissage

29 Cosparsity and physical models? with S. Nam, N. Bertin, G. Chardon, L. Daudet echange.inria.fr R. GRIBONVAL - Journées INRIA Apprentissage

30 3D scene acquisition (ANR project ECHANGE) Acoustic pressure field y = p(r, t) Nyquist sampling High dimension 10 6 microphones / m samples / second cosparse model from the wave equation (Ωy) D =0 D = domain without acoustic source operator = from Laplacian Ωy = r y 1 2 y c 2 t 2 + initial & boundary conditions R. GRIBONVAL - Journées INRIA Apprentissage

31 2D field reconstruction (simulation) 2D+t vibrating plate 60x60x120 2 sources 60 microphones, random location known boundaries Reconstructed 2D+t field Structured-GAP (GRASP) Ground truth Cosparse reconstruction R. GRIBONVAL - Journées INRIA Apprentissage 31

32 Nearfield Acoustic Holography Random antenna & sparsity: from 1920 to 80 microphones [Peillot & al 2011] Cosparsity? R. GRIBONVAL - Journées INRIA Apprentissage

33 Dictionary learning with K. Schnass, F. Bach, R. Jenatton R. GRIBONVAL - Journées INRIA Apprentissage

34 Dictionary learning for sparse representations Sparse modeling = choose a dictionary patch extraction Training patches y n = Ax n, 1 n N Training image database Unknown dictionary Unknown sparse coefficients sparse learning  = edge-like atoms [Olshausen & Field 96, Aharon et al 06, Mairal et al 09,...] = shifts of edge-like motifs [Blumensath 05, Jost et al 05,...] R. GRIBONVAL - Journées INRIA Apprentissage

35 Theoretical Dictionary Learning Formalisation of the problem : N observed data y n = Ax n R d Y = AX Comon unknown dictionary A, Unknown coefficients X + sparsity hypothesis Goal: to identify A Appraoches Independent Component Anal. = asymptotic framework Proposed framework : L1 minimisation min A,X Y=AX X 1 R. GRIBONVAL - Journées INRIA Apprentissage

36 2D Numerical example Y = A 0 X 0 3 N = 1000 Bernoulli Gaussian training samples A θ0,θ 1 θ 0 θ 1 A 1 Y 1 θ0,θ Empirical observations a) Global minima match angles of the original basis b) There is no other local minimum. R. GRIBONVAL - Journées INRIA Apprentissage 36

37 Sparsity vs coherence weakly sparse sparse 1 p N = 1000 Bernoulli Gaussian training samples incoherent N = 1000 Bernoulli Gaussian training samples N = 1000 Bernoulli Gaussian training samples N = 1000 Bernoulli Gaussian training samples µ = cos(θ 1 θ 0 ) 1 coherent p p Empirical probability of erroneous minima Missed local minimum µ Wrong local minimum µ p p Wrong global mimimum µ Missed global minimum µ Rule of thumb: perfect recovery if: a) µ<1 p b) N is large enough (many training samples) R. GRIBONVAL - Journées INRIA Apprentissage 37

38 Recent Theoretical Guarantees [G. & Schnass 2010]: Identifiability conditions for invertible A Control on sufficient number of training samples N for identifiability with high probability (under Bernoulli- Gaussian model on X) in dimension d [Bach,Jenatton, Gribonval] (work in progress) Robustness to noise Robustness to outliers N Cdlog d min A,X 1 2 Y AX2 F + λx 1 R. GRIBONVAL - Journées INRIA Apprentissage

39 INRIA & Machine Learning? INRIA, Machine Learning & Signal Processing! R. GRIBONVAL - Journées INRIA Apprentissage

40 Signal Processing & Machine Learning Exploit Sparsity Analog level: object = data vector y x k a k = Ax few k... «Semantic» level: object = function f(y) α n K(y, y n ) few n Data model = atoms of a dictionary Function model = kernel Compression, representation,... Classification, regression,... R. GRIBONVAL - Journées INRIA Apprentissage 40

41 Signal Processing & Machine Learning Learn from a collection Analog level: object = data vector... «Semantic» level: object = function y 1,...,y N learning  y 1,...,y N learning ˆf( ) Dictionary Learning: infer model from examples Machine Learning: infer function from examples R. GRIBONVAL - Journées INRIA Apprentissage 41

42 Signal Processing vs Machine Learning? Signal Processing meets Machine Learning model / learn from data algorithms (scalability) high-dimensional statistics, large-scale optimization exploit sparsity +SP specific: hardware constraints/design random acoustic antenna compressed ECG A/D converter R. GRIBONVAL - Journées INRIA Apprentissage

43 Machine Learning vs Signal Processing? Machine Learning meets Signal Processing model / learn from data algorithms (scalability) high-dimensional statistics, large-scale optimization exploit sparsity +Machine Learning : fundamental complexity / performance tradeoffs Statistical Learning!= Machine Learning R. GRIBONVAL - Journées INRIA Apprentissage

44 !"###$%&# R. GRIBONVAL - Journées INRIA Apprentissage

45 INRIA? Strong existing task force but Spectrum of required skills ever increasing... data specific expertise (biomedical / audiovisual /...) + applied math + large-scale optim. + probability & stats + complexity theory... Need to secure / consolidate INRIA position Hiring: Junior Researchers & R&D Engineers Revisit organisation of Themes? R. GRIBONVAL - Journées INRIA Apprentissage

46 Reconstruction parcimonieuse: garanties théoriques Théorème : soit z = Ax 0 si si x 0 0 k 0 (A) x 0 0 k p (A) alors x 0 = x 0 alors x 0 = x p avec x p = arg min Ax=Ax 0 x p [Donoho & Huo 01] : paires de bases, notion de cohérence, p=1 [Gribonval & Nielsen 2003] [Donoho & Elad 2003] : dictionnaires, cohérence [Tropp 2004] : Orthonormal Matching Pursuit, cohérence cumulative [Candès, Romberg & Tao 2004] : dictionnaires aléatoires, isométrie restreintes [Gribonval & Nielsen 2007] : résultats d interpolation 0<p<1 R. GRIBONVAL - Journées INRIA Apprentissage

47 Géométrie des normes Lp Strictement convexe p>1 Convexe pour p=1 Non convexe p<1 Fait: la solution est parcimonieuse {x tel que b = Ax} R. GRIBONVAL - Journées INRIA Apprentissage

48 Stabilité y = Ax, x 0 =1 A =[a 1,...,a N ] Parcimonie idéale Parcimonie approchée R. GRIBONVAL - Journées INRIA Apprentissage

49 Robustness R. GRIBONVAL - Journées INRIA Apprentissage

50 Robustness to noise d=200 n=240 m=160 l=180 R. GRIBONVAL - Journées INRIA Apprentissage 50

51 Robustness to noise Original TVDantzig TVQC GAP TV R. GRIBONVAL - Journées INRIA Apprentissage 51

52 Geometry of Inverse Problems Signal space ~ R d Nonlinear Approximation = Sparse reconstruction x Linear projection y = Mx «Interesting» signals Uniqueness given the model Sparse model (spark of D) Cosparse model Provably good algorithms, with bounded complexity greedy methods convex optimization iterative hard thresholding... Sparse model Cosparse model???? Thanks to M. Davies, U. Edinburgh Observation space ~ R m m<<d R. GRIBONVAL - Journées INRIA Apprentissage 52

53 !"###$%&# Thanks Cosparse model: small-project.eu Sangnam Nam (INRIA, France) Mike Davies (University of Edinburgh, UK) Miki Elad (The Technion, Israel) Acoustic imaging echange.inria.fr/ Laurent Daudet, Gilles Chardon François Ollivier, Antoine Peillot Nancy Bertin... Graphical design: Jules Espiau (INRIA, France) R. GRIBONVAL - Journées INRIA Apprentissage

54 Sparsity & Projections In Signal Processing and Machine Learning Sparsity: approximate high-dimensional object with few parameters ex: signal, image Inverse problem: accurately reconstruct object from incomplete observation ex: tomography x Representation Sparse representation y few nonzero components y Ax High-dim. Object Random projections intentionally reduce dimension, with stable reconstruction theoretical guarantees algorithms of bounded complexity ex: compressed sensing, compressed machine learning b Inverse Problem Observations Random Projection b = My fewer equations than unknowns R. GRIBONVAL - Journées INRIA Apprentissage 54

55 What s next? New cosparse recovery algorithms with guarantees GAP & Analysis-L1 [Nam & al 2011 a, Vaiter & al ] Analysis Iterative Hard Thresholding [Gyries & al 2011]... Group cosparsity [Nam & al 2011 b] Hybrid sparse/cosparse models [Figueiredo & al 2011] Learning/designing analysis operators [Ophir & al 2011, Yaghoobi & al 2011, Peyré & Fadili 2011] Connections with PDEs... [Nam & al 2011 b] More applications... R. GRIBONVAL - Journées INRIA Apprentissage