Part II Redundant Dictionaries and Pursuit Algorithms

Save this PDF as:
 WORD  PNG  TXT  JPG

Size: px
Start display at page:

Download "Part II Redundant Dictionaries and Pursuit Algorithms"

Transcription

1 Aisenstadt Chair Course CRM September 2009 Part II Redundant Dictionaries and Pursuit Algorithms Stéphane Mallat Centre de Mathématiques Appliquées Ecole Polytechnique

2 Sparsity in Redundant Dictionaries Bases are minimum set to decompose signals. Natural languages use redundant dictionaries. Use of larger dictionaries incorporating more patterns to represent complex signals f R N D = {φ p } p Γ with φ p = 1 and Γ = P > N. How to construct sparse representations in D? What is the impact of redundancy on the mathematics and applications?

3 Let D = {φ p } p Γ Dictionary Approximation be a redundant dictionary of P vectors. The best approximation of f from a sub-family orthogonal projection in V Λ = Vect{φ p } p Λ f Λ = a[p] φ p p Λ {φ p } p Λ is its Stability: {φ p } p Λ must be a Riez basis of V Λ : there exists 0 <A Λ B Λ such that a[p] R Λ, A Λ a[p] 2 a[p] φ p 2 B Λ a[p] 2. p Λ p Λ p Λ

4 Best M-Term Approximation The best M-term approximation support Λ minimizes f f Λ with Λ = M. A best M-term approximation minimizes a Lagrangian: L 0 = f f Λ 2 + T 2 Λ. If is an orthonormal basis then D Λ = {p : f,φ p T } In general, finding Λ is an NP-hard problem.

5 Compression Applications Compute a best M-term approximation f Λ = p Λ a[p] φ p with Λ = M. Compression with uniform quantization f Λ = p Λ Q(a[p]) φ p Total bit budget: R = log 2 ( P M ) + µm R M log 2 (P/M) Increasing P reduces D = f f Λ but increases R. If f f Λ 2 = O(M α ) then D(R) =R α log(p/r) α.

6 Wavelets for Cartoon Images Theorem: If f is uniformly approximation gives C α f f M 2 = O(M α ) then an M-term wavelet Theorem: If f is piecewise C α with finite length contours then f f M 2 = O(M 1 ) so D(R) =R 1 log(n/r) Result valid for all bounded variation functions. f j =-8 f, ψ H jn f h x j = -7 j = -6 y f, ψv jn f, ψjn D (b) Wavelet Transform (c) Zoom γ

7 Bandlets for Geometric Regularity Wavelet coefficients inherit the geometric regularity f j =-8 j = -7 j = -6 f, ψ H jn f h y f, ψv jn f, ψjn D (b) Wavelet Transform (c) Zoom γ 1. Segmentation of wavelet coefficients. b Regular square Edge square Small Edge square 2. Geometric flow in edge squares along the direction of regularity. w (a) (a ) Wavelet Transform Quadtree (a) 3. 1D wavelet S transform along 3 S the flow = 2 (b) (c) (d) bandlet transform Corner square S 1 S 2 (b ) (c ) (d ) (b ) (c ) ( S 1 (b) (c) ( S

8 Bandlet Approximations A bandlet dictionary is a union of orthonormal bases. The best bandlet approximation (best geometry) which minimizes L 0 = f f M 2 + T 2 M can be computed with O(N log N) operations. C α C α Theorem If f is piecewise with piecewise contours f f M 2 = O(M α ) so D(R) =R α log(p/r) α

9 Application of Photo ID Compression

10 just nice...

11 Xlets Beyond Wavelets Xlets take advantage of the image geometric regularity: bandlets, curvelets, contourlets, edglets, wedglets... Failure to improve wavelet approximations for natural images. Can wavelet approximations be asymptotically improved? I don t think so for static natural images.

12 Denoising in Redundant Dictionaries Measure a signal plus a Gaussian white noise: X[n] =f[n]+w [n] for 0 n < N. Orthogonal projection estimator in selected in a dictionary: V Λ = Vect{φ p } p Λ X Λ = p Λ a[p] φ p. Risk: If Λ is fixed then E{ W Λ 2 } = σ 2 Λ. Oracle choice: find Λ which minimizes E{ f X Λ 2 } = f f Λ 2 + E{ W Λ 2 }. E{ f X Λ 2 } = f f Λ 2 + σ 2 Λ

13 Penalized Estimation Choosing Λ is a model selection problem. Theorem: For satisfies T>σ 2 log e P ( ) Λ = arg min X X Λ 2 + T 2 Λ Λ Γ E{ f X Λ 2 } = 4 min ( f f Λ 2 + T 2 Λ + σ 2) Λ Γ Noisy image Wavelet thresholding Bandlet thresholding PSNR = 22db PSNR = 25.3db PSNR = 26.4db

14 Let D = {φ p } p Γ Greedy Matching Pursuits be a dictionary of P > N vectors. A best M-term approximation minimizes the Lagrangian: L 0 = f f Λ 2 + T 2 Λ Finding Λ is in general an NP complete problem. Greedy choice of the approximation vectors. The approximation of f over φ p0 D yields f = f,φ p0 + Rf f 2 = f,φ p0 2 + Rf 2 To minimize the residual error we choose φ p0 = arg max φ p D f,φ p

15 Matching Pursuit Iterations Initialize For each R 0 f = f m>0 φ pm = arg max φ p D Rm f,φ p R m f = R m f,φ pm + R m+1 f R m f 2 = R m f,φ pm 2 + R m+1 f 2 It results: f = f 2 = M 1 m=0 M 1 m=0 R m f,φ pm + R M f R m f,φ pm 2 + R M f 2

16 Time-Frequency Decompositions Dictionary of time-frequency atoms: D = { φ p (t) = 1 w(t u } 2 j 2 j ) e iξt (j,u,ξ) Γ It includes Dirac and Fourier bases, wavelets and window Fourier atoms. $ # " ^ ($)! % t % $ " ( t )! 0 u t

17 Time Frequency Matching Pursuits f(t) ! 2" 250 t t 1

18 f(t) Speech Signals 1000 t ! 2" t 0 1 Applications: denoising, compression (video), pattern recognition, but instabilities.

19 Orthogonal Matching Pursuit Initialize For each R 0 f = f m>0 φ pm orthogonalize the projections: = arg max φ p D Rm f,φ p u m = φ pm m 1 l=0 φ pm,u l u l 2 u l. R m+1 f = Rm f, u m u m 2 + R m+1 f After N iterations, it gives decomposition in an orthogonal basis: f = N 1 m=0 R m f, u m u m 2.

20 Matching versus Basis Pursuit Greediness is not optimal Signal f Time-Frequency Coefficients Matching Pursuit Basis Pursuit Orthogonal Matching Pursuit

21 Basis Pursuit Find a sparse representation f = a[p] φ p ( ) p Γ 1/q by minimizing an l q norm : a q = a[n] q l 0 n Γ norm : number of non-zero a[n] Not convex for q < 1. l 1 norm : a 1 = a[n]. n Γ. l q balls q =0 q =0 q q =05.. q q =1 q = q =2

22 Basis Pursuit Approximation A basis pursuit computes a sparse approximation f = p Γ solution of a convex optimization problem: ã = arg min a R P a 1 ã[p] φ p Lagrangian formulation: find subject to f p Γ ã which minimizes L 1 (a) = 1 2 f p Γ a[p] φ p 2 + T a 1 a[p]φ p ɛ.

23 Basis Pursuit Denoising Original Image Noisy Image Wavelet Thresh. Basis Pursuit with Wavelet Cosine Diction.

24 Computation of l1 Minimization Many iterative algorithms to minimize L 1 (a) = 1 2 f p Γ a[p] φ p 2 + T a 1 Simplest one: iterative thresholding. Initialisation b = Φf = { f,φ p } p Γ and a 0 =0. For k>0 Gradient step: ã k = a k + γ(b ΦΦ a k ) ( Soft thresholding: a k+1 =ã k max 1 γt ) ã k, 0.

25 Exact Recovery Suppose that the signal is sparse f = p Λ can we recover Λ with pursuit algorithms? With matching pursuits, need that Exact Recovery Criteria a[p] φ p = p Λ f,φ p φ p V Λ. C(R m f,λ c )= max q Λ c R m f,φ q max p Λ R m f,φ p ERC(Λ) = sup C(h, Λ c ) = max h V Λ q Λ c p Λ < 1. φ p, φ q.

26 ERC Recovery Theorem: The approximation support Λ of f V Λ is exactly recovered by an orthogonal matching pursuit or a basis pursuit if ERC(Λ) < 1. Theorem (stability): If f is not exactly sparse, an orthogonal matching pursuit f M with M = Λ iterations satisfies: f f M ( 1+ Similar result for a basis pursuit. Λ ) A Λ (1 ERC(Λ)) 2 f f Λ.

27 Dictionary Coherence Dictionary mutual coherence µ(d) = sup φ p, φ q (p,q) Γ 2 Dirac-Fourier dictionary: µ(d) =N 1/2. Theorem: ERC(Λ) < 1 if Λ < 1 2 ( 1+ 1 ) µ(d) Exact recovery is possible for sufficiently sparse signals in incoherent dictionaries.

28 2nd. Conclusion Redundant dictionaries can improve approximation, compression, denoising. Finding optimal approximation is NP complete but can be approximated with matching or basis pursuits. May be used for pattern recognition but problems of instabilities. The stability depends upon the dictionary coherence. Major applications to inverse problems, superresolution and compress sensing.

Cheng Soon Ong & Christfried Webers. Canberra February June 2016

Cheng Soon Ong & Christfried Webers. Canberra February June 2016 c Cheng Soon Ong & Christfried Webers Research Group and College of Engineering and Computer Science Canberra February June (Many figures from C. M. Bishop, "Pattern Recognition and ") 1of 31 c Part I

More information

Disjoint sparsity for signal separation and applications to hybrid imaging inverse problems

Disjoint sparsity for signal separation and applications to hybrid imaging inverse problems Disjoint sparsity for signal separation and applications to hybrid imaging inverse problems Giovanni S Alberti (joint with H Ammari) DMA, École Normale Supérieure, Paris June 16, 2015 Giovanni S Alberti

More information

Compressive Sensing. Examples in Image Compression. Lecture 4, July 30, Luiz Velho Eduardo A. B. da Silva Adriana Schulz

Compressive Sensing. Examples in Image Compression. Lecture 4, July 30, Luiz Velho Eduardo A. B. da Silva Adriana Schulz Compressive Sensing Examples in Image Compression Lecture 4, July, 09 Luiz Velho Eduardo A. B. da Silva Adriana Schulz Today s Lecture Discuss applications of CS in image compression Evaluate CS efficiency

More information

Morphological Diversity and Sparsity for Multichannel Data Restoration

Morphological Diversity and Sparsity for Multichannel Data Restoration Morphological Diversity and Sparsity for Multichannel Data Restoration J.Bobin 1, Y.Moudden 1, J.Fadili and J-L.Starck 1 1 jerome.bobin@cea.fr, ymoudden@cea.fr, jstarck@cea.fr - CEA-DAPNIA/SEDI, Service

More information

Wavelet analysis. Wavelet requirements. Example signals. Stationary signal 2 Hz + 10 Hz + 20Hz. Zero mean, oscillatory (wave) Fast decay (let)

Wavelet analysis. Wavelet requirements. Example signals. Stationary signal 2 Hz + 10 Hz + 20Hz. Zero mean, oscillatory (wave) Fast decay (let) Wavelet analysis In the case of Fourier series, the orthonormal basis is generated by integral dilation of a single function e jx Every 2π-periodic square-integrable function is generated by a superposition

More information

Low-Dimensional Models for Complex Structured Data

Low-Dimensional Models for Complex Structured Data Low-Dimensional Models for Complex Structured Data Jackie Ma (Technische Universität Berlin) DEDALE Kick-off Meeting October 22-23, 2015 Jackie Ma (TU Berlin) WP2 CEA, Saclay/Paris 1 / 11 Low-Dimensional

More information

Stable Signal Recovery from Incomplete and Inaccurate Measurements

Stable Signal Recovery from Incomplete and Inaccurate Measurements Stable Signal Recovery from Incomplete and Inaccurate Measurements EMMANUEL J. CANDÈS California Institute of Technology JUSTIN K. ROMBERG California Institute of Technology AND TERENCE TAO University

More information

Several Views of Support Vector Machines

Several Views of Support Vector Machines Several Views of Support Vector Machines Ryan M. Rifkin Honda Research Institute USA, Inc. Human Intention Understanding Group 2007 Tikhonov Regularization We are considering algorithms of the form min

More information

Minimizing beam-on time during radiotherapy treatment by total-variation regularization

Minimizing beam-on time during radiotherapy treatment by total-variation regularization Minimizing beam-on time during radiotherapy treatment by total-variation regularization Xing Lab, Department of Radiation Oncology, Stanford. Zhu, L. & Xing, L. (2009) Search for IMRT inverse plans with

More information

Picture Perfect: The Mathematics of JPEG Compression

Picture Perfect: The Mathematics of JPEG Compression Picture Perfect: The Mathematics of JPEG Compression May 19, 2011 1 2 3 in 2D Sampling and the DCT in 2D 2D Compression Images Outline A typical color image might be 600 by 800 pixels. Images Outline A

More information

Stable Signal Recovery from Incomplete and Inaccurate Measurements

Stable Signal Recovery from Incomplete and Inaccurate Measurements Stable Signal Recovery from Incomplete and Inaccurate Measurements Emmanuel Candes, Justin Romberg, and Terence Tao Applied and Computational Mathematics, Caltech, Pasadena, CA 91125 Department of Mathematics,

More information

Bag of Pursuits and Neural Gas for Improved Sparse Coding

Bag of Pursuits and Neural Gas for Improved Sparse Coding Bag of Pursuits and Neural Gas for Improved Sparse Coding Kai Labusch, Erhardt Barth, and Thomas Martinetz University of Lübec Institute for Neuro- and Bioinformatics Ratzeburger Allee 6 23562 Lübec, Germany

More information

Clarify Some Issues on the Sparse Bayesian Learning for Sparse Signal Recovery

Clarify Some Issues on the Sparse Bayesian Learning for Sparse Signal Recovery Clarify Some Issues on the Sparse Bayesian Learning for Sparse Signal Recovery Zhilin Zhang and Bhaskar D. Rao Technical Report University of California at San Diego September, Abstract Sparse Bayesian

More information

Solving Inverse Problems with Piecewise Linear Estimators: From Gaussian Mixture Models to Structured Sparsity

Solving Inverse Problems with Piecewise Linear Estimators: From Gaussian Mixture Models to Structured Sparsity 1 Solving Inverse Problems with Piecewise Linear Estimators: From Gaussian Mixture Models to Structured Sparsity Guoshen YU* 1, Guillermo SAPIRO 1, and Stéphane MALLAT 2 1 ECE, University of Minnesota,

More information

Lecture Topic: Low-Rank Approximations

Lecture Topic: Low-Rank Approximations Lecture Topic: Low-Rank Approximations Low-Rank Approximations We have seen principal component analysis. The extraction of the first principle eigenvalue could be seen as an approximation of the original

More information

Admin stuff. 4 Image Pyramids. Spatial Domain. Projects. Fourier domain 2/26/2008. Fourier as a change of basis

Admin stuff. 4 Image Pyramids. Spatial Domain. Projects. Fourier domain 2/26/2008. Fourier as a change of basis Admin stuff 4 Image Pyramids Change of office hours on Wed 4 th April Mon 3 st March 9.3.3pm (right after class) Change of time/date t of last class Currently Mon 5 th May What about Thursday 8 th May?

More information

α α λ α = = λ λ α ψ = = α α α λ λ ψ α = + β = > θ θ β > β β θ θ θ β θ β γ θ β = γ θ > β > γ θ β γ = θ β = θ β = θ β = β θ = β β θ = = = β β θ = + α α α α α = = λ λ λ λ λ λ λ = λ λ α α α α λ ψ + α =

More information

Sparse recovery and compressed sensing in inverse problems

Sparse recovery and compressed sensing in inverse problems Gerd Teschke (7. Juni 2010) 1/68 Sparse recovery and compressed sensing in inverse problems Gerd Teschke (joint work with Evelyn Herrholz) Institute for Computational Mathematics in Science and Technology

More information

Lecture 5 Least-squares

Lecture 5 Least-squares EE263 Autumn 2007-08 Stephen Boyd Lecture 5 Least-squares least-squares (approximate) solution of overdetermined equations projection and orthogonality principle least-squares estimation BLUE property

More information

IK-SVD: Dictionary Learning for Spatial Big Data via Incremental Atom Update

IK-SVD: Dictionary Learning for Spatial Big Data via Incremental Atom Update IEEE COMPUTING IN SCIENCE AND ENGINEERING, VOL. XX, NO. XX, XXX 2014 1 IK-SVD: Dictionary Learning for Spatial Big Data via Incremental Atom Update Lizhe Wang 1,KeLu 2, Peng Liu 1, Rajiv Ranjan 3, Lajiao

More information

SPARSE REPRESENTATIONS APPLICATIONS ON COMPUTER VISION AND PATTERN RECOGNITION

SPARSE REPRESENTATIONS APPLICATIONS ON COMPUTER VISION AND PATTERN RECOGNITION SPARSE REPRESENTATIONS APPLICATIONS ON COMPUTER VISION AND PATTERN RECOGNITION Computer Vision Group Electronics Laboratory Physics Department University of Patras www.upcv.upatras.gr www.ellab.physics.upatras.gr

More information

CHAPTER 5 CORONARY ANGIOGRAM VIDEO COMPRESSION USING WAVELET BASED CONTOURLET TRANSFORM AND REGION OF INTEREST TECHNIQUE

CHAPTER 5 CORONARY ANGIOGRAM VIDEO COMPRESSION USING WAVELET BASED CONTOURLET TRANSFORM AND REGION OF INTEREST TECHNIQUE 123 CHAPTER 5 CORONARY ANGIOGRAM VIDEO COMPRESSION USING WAVELET BASED CONTOURLET TRANSFORM AND REGION OF INTEREST TECHNIQUE 5.1 INTRODUCTION Coronary angiograms play an important role in the diagnosis

More information

COMPUTATIONAL INTELLIGENCE (INTRODUCTION TO MACHINE LEARNING) SS16. Lecture 2: Linear Regression Gradient Descent Non-linear basis functions

COMPUTATIONAL INTELLIGENCE (INTRODUCTION TO MACHINE LEARNING) SS16. Lecture 2: Linear Regression Gradient Descent Non-linear basis functions COMPUTATIONAL INTELLIGENCE (INTRODUCTION TO MACHINE LEARNING) SS16 Lecture 2: Linear Regression Gradient Descent Non-linear basis functions LINEAR REGRESSION MOTIVATION Why Linear Regression? Regression

More information

Analysis of data separation and recovery problems using clustered sparsity

Analysis of data separation and recovery problems using clustered sparsity Analysis of data separation and recovery problems using clustered sparsity Emily J. King a, Gitta Kutyniok a, and Xiaosheng Zhuang a a Institute of Mathematics, Universität Osnabrück, Osnabrück, Germany

More information

c 2003 Society for Industrial and Applied Mathematics

c 2003 Society for Industrial and Applied Mathematics SIAM J. SCI. COMPUT. Vol. 4, No. 4, pp. 148 143 c 3 Society for Industrial and Applied Mathematics WAVELET ALGORITHMS FOR HIGH-RESOLUTION IMAGE RECONSTRUCTION RAYMOND H. CHAN, TONY F. CHAN, LIXIN SHEN,

More information

A Multichannel Spatial Compressed Sensing Approach for Direction of Arrival Estimation

A Multichannel Spatial Compressed Sensing Approach for Direction of Arrival Estimation A Multichannel Spatial Compressed Sensing Approach for Direction of Arrival Estimation Aris Gretsistas and Mark D. Plumbley Queen Mary University of London Centre for Digital Music Mile End Road, E1 4NS,

More information

Level Set Framework, Signed Distance Function, and Various Tools

Level Set Framework, Signed Distance Function, and Various Tools Level Set Framework Geometry and Calculus Tools Level Set Framework,, and Various Tools Spencer Department of Mathematics Brigham Young University Image Processing Seminar (Week 3), 2010 Level Set Framework

More information

Variational approach to restore point-like and curve-like singularities in imaging

Variational approach to restore point-like and curve-like singularities in imaging Variational approach to restore point-like and curve-like singularities in imaging Daniele Graziani joint work with Gilles Aubert and Laure Blanc-Féraud Roma 12/06/2012 Daniele Graziani (Roma) 12/06/2012

More information

A VARIANT OF THE EMD METHOD FOR MULTI-SCALE DATA

A VARIANT OF THE EMD METHOD FOR MULTI-SCALE DATA Advances in Adaptive Data Analysis Vol. 1, No. 4 (2009) 483 516 c World Scientific Publishing Company A VARIANT OF THE EMD METHOD FOR MULTI-SCALE DATA THOMAS Y. HOU and MIKE P. YAN Applied and Computational

More information

Geometric Tools for Image Compression

Geometric Tools for Image Compression Geometric Tools for Image Compression Michael Wakin, Justin Romberg, Hyeokho Choi, Richard Baraniuk Department of Electrical and Computer Engineering, Rice University 6100 Main St., Houston, TX 77005 Abstract

More information

Point Lattices in Computer Graphics and Visualization how signal processing may help computer graphics

Point Lattices in Computer Graphics and Visualization how signal processing may help computer graphics Point Lattices in Computer Graphics and Visualization how signal processing may help computer graphics Dimitri Van De Ville Ecole Polytechnique Fédérale de Lausanne Biomedical Imaging Group dimitri.vandeville@epfl.ch

More information

Sparsity-promoting recovery from simultaneous data: a compressive sensing approach

Sparsity-promoting recovery from simultaneous data: a compressive sensing approach SEG 2011 San Antonio Sparsity-promoting recovery from simultaneous data: a compressive sensing approach Haneet Wason*, Tim T. Y. Lin, and Felix J. Herrmann September 19, 2011 SLIM University of British

More information

Lecture 3: Linear methods for classification

Lecture 3: Linear methods for classification Lecture 3: Linear methods for classification Rafael A. Irizarry and Hector Corrada Bravo February, 2010 Today we describe four specific algorithms useful for classification problems: linear regression,

More information

Enhancement of scanned documents in Besov spaces using wavelet domain representations

Enhancement of scanned documents in Besov spaces using wavelet domain representations Enhancement of scanned documents in Besov spaces using wavelet domain representations Kathrin Berkner 1 Ricoh Innovations, Inc., 2882 Sand Hill Road, Suite 115, Menlo Park, CA 94025 ABSTRACT After scanning,

More information

From Sparse Approximation to Forecast of Intraday Load Curves

From Sparse Approximation to Forecast of Intraday Load Curves From Sparse Approximation to Forecast of Intraday Load Curves Mathilde Mougeot Joint work with D. Picard, K. Tribouley (P7)& V. Lefieux, L. Teyssier-Maillard (RTE) 1/43 Electrical Consumption Time series

More information

Introduction to Convex Optimization for Machine Learning

Introduction to Convex Optimization for Machine Learning Introduction to Convex Optimization for Machine Learning John Duchi University of California, Berkeley Practical Machine Learning, Fall 2009 Duchi (UC Berkeley) Convex Optimization for Machine Learning

More information

IN THIS paper, we address the classic image denoising

IN THIS paper, we address the classic image denoising 3736 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 15, NO. 12, DECEMBER 2006 Image Denoising Via Sparse and Redundant Representations Over Learned Dictionaries Michael Elad and Michal Aharon Abstract We

More information

Sparse Coding and Dictionary Learning for Image Analysis

Sparse Coding and Dictionary Learning for Image Analysis Sparse Coding and Dictionary Learning for Image Analysis Part I: Optimization for Sparse Coding Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro ICCV 09 tutorial, Kyoto, 28th September 2009

More information

About the Ingham type proof for the boundary observability of a square and its generalization in N-d

About the Ingham type proof for the boundary observability of a square and its generalization in N-d About the Ingham type proof for the boundary observability of a square and its generalization in N-d M. Mehrenberger University of Strasbourg (France), INRIA TONUS team Rome 015 Rome, September 30, 015

More information

Compressed Sensing with Non-Gaussian Noise and Partial Support Information

Compressed Sensing with Non-Gaussian Noise and Partial Support Information IEEE SIGNAL PROCESSING LETTERS, VOL. 22, NO. 10, OCTOBER 2015 1703 Compressed Sensing with Non-Gaussian Noise Partial Support Information Ahmad Abou Saleh, Fady Alajaji, Wai-Yip Chan Abstract We study

More information

Some stability results of parameter identification in a jump diffusion model

Some stability results of parameter identification in a jump diffusion model Some stability results of parameter identification in a jump diffusion model D. Düvelmeyer Technische Universität Chemnitz, Fakultät für Mathematik, 09107 Chemnitz, Germany Abstract In this paper we discuss

More information

PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 4: LINEAR MODELS FOR CLASSIFICATION

PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 4: LINEAR MODELS FOR CLASSIFICATION PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 4: LINEAR MODELS FOR CLASSIFICATION Introduction In the previous chapter, we explored a class of regression models having particularly simple analytical

More information

ELEC 4801 THESIS PROJECT

ELEC 4801 THESIS PROJECT School of Information Technology and Electrical Engineering ELEC 4801 THESIS PROJECT Thesis: Speech Compression Using Wavelets Name: Nikhil Rao Student No: 33710745 Supervisor: Dr John Homer Submitted

More information

t := maxγ ν subject to ν {0,1,2,...} and f(x c +γ ν d) f(x c )+cγ ν f (x c ;d).

t := maxγ ν subject to ν {0,1,2,...} and f(x c +γ ν d) f(x c )+cγ ν f (x c ;d). 1. Line Search Methods Let f : R n R be given and suppose that x c is our current best estimate of a solution to P min x R nf(x). A standard method for improving the estimate x c is to choose a direction

More information

FFT Algorithms. Chapter 6. Contents 6.1

FFT Algorithms. Chapter 6. Contents 6.1 Chapter 6 FFT Algorithms Contents Efficient computation of the DFT............................................ 6.2 Applications of FFT................................................... 6.6 Computing DFT

More information

INPAINTING COLOR IMAGES IN LEARNED DICTIONARY. Bijenička cesta 54, 10002, Zagreb, Croatia

INPAINTING COLOR IMAGES IN LEARNED DICTIONARY. Bijenička cesta 54, 10002, Zagreb, Croatia 2th European Signal Processing Conference (EUSIPCO 212) Bucharest, Romania, August 27-31, 212 INPAINTING COLOR IMAGES IN LEARNED DICTIONARY Marko Filipović 1, Ivica Kopriva 1 and Andrzej Cichocki 2 1 Ruđer

More information

Evolutionary denoising based on an estimation of Hölder exponents with oscillations.

Evolutionary denoising based on an estimation of Hölder exponents with oscillations. Evolutionary denoising based on an estimation of Hölder exponents with oscillations. Pierrick Legrand,, Evelyne Lutton and Gustavo Olague CICESE, Research Center, Applied Physics Division Centro de Investigación

More information

Image Compression Using Gabor Filter

Image Compression Using Gabor Filter Mr.B.H.Deokate, Dr. P. M. Patil and Mr.S.S. Majgaonkar 28 Image Compression Using Gabor Filter Mr.B.H.Deokate, Dr. P. M. Patil and Mr.S.S. Majgaonkar Abstract: The data is the basic component of information.

More information

Statistical Machine Learning

Statistical Machine Learning Statistical Machine Learning UoC Stats 37700, Winter quarter Lecture 4: classical linear and quadratic discriminants. 1 / 25 Linear separation For two classes in R d : simple idea: separate the classes

More information

Regression Using Support Vector Machines: Basic Foundations

Regression Using Support Vector Machines: Basic Foundations Regression Using Support Vector Machines: Basic Foundations Technical Report December 2004 Aly Farag and Refaat M Mohamed Computer Vision and Image Processing Laboratory Electrical and Computer Engineering

More information

Wavelet Analysis Based Estimation of Probability Density function of Wind Data

Wavelet Analysis Based Estimation of Probability Density function of Wind Data , pp.23-34 http://dx.doi.org/10.14257/ijeic.2014.5.3.03 Wavelet Analysis Based Estimation of Probability Density function of Wind Data Debanshee Datta Department of Mechanical Engineering Indian Institute

More information

Minimum-Volume Box Containing a Set of Points

Minimum-Volume Box Containing a Set of Points Minimum-Volume Box Containing a Set of Points David Eberly Geometric Tools, LLC http://www.geometrictools.com/ Copyright c 1998-2016. All Rights Reserved. Created: July 25, 2015 Contents 1 Introduction

More information

FOURIER SERIES BASED PHASE MODEL FOR STATISTICAL SIGNAL PROCESSING

FOURIER SERIES BASED PHASE MODEL FOR STATISTICAL SIGNAL PROCESSING EE 602 TERM PAPER FOURIER SERIES BASED PHASE MODEL FOR STATISTICAL SIGNAL PROCESSING - Chong yung chi Instructor: Dr. Rajesh Hegde Presented by: Mounika Bopuddi (Y8104037) Richa Tripathi (Y3288) ABSTRACT

More information

An integrated data compression scheme for power quality events using spline wavelet and neural network

An integrated data compression scheme for power quality events using spline wavelet and neural network An integrated data compression scheme for power quality events using spline wavelet and neural network S.K. Meher a, A.K. Pradhan b,, G. Panda c a Department of Electronics and Communication Engineering,

More information

Applications to Data Smoothing and Image Processing I

Applications to Data Smoothing and Image Processing I Applications to Data Smoothing and Image Processing I MA 348 Kurt Bryan Signals and Images Let t denote time and consider a signal a(t) on some time interval, say t. We ll assume that the signal a(t) is

More information

Coordinate Descent for L 1 Minimization

Coordinate Descent for L 1 Minimization Coordinate Descent for L 1 Minimization Yingying Li and Stanley Osher Department of Mathematics University of California, Los Angeles Jan 13, 2010 Li, Osher (UCLA) Coordinate Descent for L 1 Minimization

More information

COMPRESSIVE SENSING WITH HIGHLY COHERENT DICTIONARIES

COMPRESSIVE SENSING WITH HIGHLY COHERENT DICTIONARIES COMPRESSIVE SENSING WITH HIGHLY COHERENT DICTIONARIES SARA COHEN Abstract. Compressive sensing is an emerging field based on the discovery that sparse signals and images can be reconstructed from highly

More information

Robust principle component analysis (RPCA) for seismic data denoising

Robust principle component analysis (RPCA) for seismic data denoising Robust principle component analysis (RPCA) for seismic data denoising Jinkun Cheng, Ke Chen and Mauricio D. Sacchi Department of Physics, University of Alberta Summary Principal component analysis (PCA)

More information

Applied Algorithm Design Lecture 5

Applied Algorithm Design Lecture 5 Applied Algorithm Design Lecture 5 Pietro Michiardi Eurecom Pietro Michiardi (Eurecom) Applied Algorithm Design Lecture 5 1 / 86 Approximation Algorithms Pietro Michiardi (Eurecom) Applied Algorithm Design

More information

Neural Networks. CAP5610 Machine Learning Instructor: Guo-Jun Qi

Neural Networks. CAP5610 Machine Learning Instructor: Guo-Jun Qi Neural Networks CAP5610 Machine Learning Instructor: Guo-Jun Qi Recap: linear classifier Logistic regression Maximizing the posterior distribution of class Y conditional on the input vector X Support vector

More information

Sublinear Algorithms for Big Data. Part 4: Random Topics

Sublinear Algorithms for Big Data. Part 4: Random Topics Sublinear Algorithms for Big Data Part 4: Random Topics Qin Zhang 1-1 2-1 Topic 1: Compressive sensing Compressive sensing The model (Candes-Romberg-Tao 04; Donoho 04) Applicaitons Medical imaging reconstruction

More information

Image Super-Resolution via Sparse Representation

Image Super-Resolution via Sparse Representation 1 Image Super-Resolution via Sparse Representation Jianchao Yang, Student Member, IEEE, John Wright, Student Member, IEEE Thomas Huang, Life Fellow, IEEE and Yi Ma, Senior Member, IEEE Abstract This paper

More information

Markov chains and Markov Random Fields (MRFs)

Markov chains and Markov Random Fields (MRFs) Markov chains and Markov Random Fields (MRFs) 1 Why Markov Models We discuss Markov models now. This is the simplest statistical model in which we don t assume that all variables are independent; we assume

More information

Linear Algebra Review. Vectors

Linear Algebra Review. Vectors Linear Algebra Review By Tim K. Marks UCSD Borrows heavily from: Jana Kosecka kosecka@cs.gmu.edu http://cs.gmu.edu/~kosecka/cs682.html Virginia de Sa Cogsci 8F Linear Algebra review UCSD Vectors The length

More information

An Improved Reconstruction methods of Compressive Sensing Data Recovery in Wireless Sensor Networks

An Improved Reconstruction methods of Compressive Sensing Data Recovery in Wireless Sensor Networks Vol.8, No.1 (14), pp.1-8 http://dx.doi.org/1.1457/ijsia.14.8.1.1 An Improved Reconstruction methods of Compressive Sensing Data Recovery in Wireless Sensor Networks Sai Ji 1,, Liping Huang, Jin Wang, Jian

More information

Numerisches Rechnen. (für Informatiker) M. Grepl J. Berger & J.T. Frings. Institut für Geometrie und Praktische Mathematik RWTH Aachen

Numerisches Rechnen. (für Informatiker) M. Grepl J. Berger & J.T. Frings. Institut für Geometrie und Praktische Mathematik RWTH Aachen (für Informatiker) M. Grepl J. Berger & J.T. Frings Institut für Geometrie und Praktische Mathematik RWTH Aachen Wintersemester 2010/11 Problem Statement Unconstrained Optimality Conditions Constrained

More information

Fast Compressed Sensing Based High Resolution Image Reconstruction

Fast Compressed Sensing Based High Resolution Image Reconstruction International Journal of New Technology and Research (IJNTR) ISSN:24-4116, Volume-2, Issue-3, March 16 Pages 71-76 Fast Compressed Sensing Based High Resolution Image Reconstruction Dr G Vijay Kumar Abstract

More information

From Maxent to Machine Learning and Back

From Maxent to Machine Learning and Back From Maxent to Machine Learning and Back T. Sears ANU March 2007 T. Sears (ANU) From Maxent to Machine Learning and Back Maxent 2007 1 / 36 50 Years Ago... The principles and mathematical methods of statistical

More information

Cyber-Security Analysis of State Estimators in Power Systems

Cyber-Security Analysis of State Estimators in Power Systems Cyber-Security Analysis of State Estimators in Electric Power Systems André Teixeira 1, Saurabh Amin 2, Henrik Sandberg 1, Karl H. Johansson 1, and Shankar Sastry 2 ACCESS Linnaeus Centre, KTH-Royal Institute

More information

NMR Measurement of T1-T2 Spectra with Partial Measurements using Compressive Sensing

NMR Measurement of T1-T2 Spectra with Partial Measurements using Compressive Sensing NMR Measurement of T1-T2 Spectra with Partial Measurements using Compressive Sensing Alex Cloninger Norbert Wiener Center Department of Mathematics University of Maryland, College Park http://www.norbertwiener.umd.edu

More information

MPEG, the MP3 Standard, and Audio Compression

MPEG, the MP3 Standard, and Audio Compression MPEG, the MP3 Standard, and Audio Compression Mark ilgore and Jamie Wu Mathematics of the Information Age September 16, 23 Audio Compression Basic Audio Coding. Why beneficial to compress? Lossless versus

More information

Noisy and Missing Data Regression: Distribution-Oblivious Support Recovery

Noisy and Missing Data Regression: Distribution-Oblivious Support Recovery : Distribution-Oblivious Support Recovery Yudong Chen Department of Electrical and Computer Engineering The University of Texas at Austin Austin, TX 7872 Constantine Caramanis Department of Electrical

More information

Exact shape-reconstruction by one-step linearization in electrical impedance tomography

Exact shape-reconstruction by one-step linearization in electrical impedance tomography Exact shape-reconstruction by one-step linearization in electrical impedance tomography Bastian von Harrach harrach@math.uni-mainz.de Institut für Mathematik, Joh. Gutenberg-Universität Mainz, Germany

More information

JPEG compression of monochrome 2D-barcode images using DCT coefficient distributions

JPEG compression of monochrome 2D-barcode images using DCT coefficient distributions Edith Cowan University Research Online ECU Publications Pre. JPEG compression of monochrome D-barcode images using DCT coefficient distributions Keng Teong Tan Hong Kong Baptist University Douglas Chai

More information

Chapter 6. Orthogonality

Chapter 6. Orthogonality 6.3 Orthogonal Matrices 1 Chapter 6. Orthogonality 6.3 Orthogonal Matrices Definition 6.4. An n n matrix A is orthogonal if A T A = I. Note. We will see that the columns of an orthogonal matrix must be

More information

Curvelets A Surprisingly Effective Nonadaptive Representation For Objects with Edges

Curvelets A Surprisingly Effective Nonadaptive Representation For Objects with Edges Curvelets A Surprisingly Effective Nonadaptive Representation For Objects with Edges Emmanuel J. Candès and David L. Donoho Abstract. It is widely believed that to efficiently represent an otherwise smooth

More information

DSL Spectrum Management

DSL Spectrum Management DSL Spectrum Management Dr. Jianwei Huang Department of Electrical Engineering Princeton University Guest Lecture of ELE539A March 2007 Jianwei Huang (Princeton) DSL Spectrum Management March 2007 1 /

More information

Compressive Acquisition of Sparse Deflectometric Maps

Compressive Acquisition of Sparse Deflectometric Maps Compressive Acquisition of Sparse Deflectometric Maps Prasad Sudhakar*, Laurent Jacques*, Adriana Gonzalez Gonzalez* Xavier Dubois +, Philippe Antoine +, Luc Joannes + *: Louvain University (UCL), Louvain-la-Neuve,

More information

Discrete Optimization

Discrete Optimization Discrete Optimization [Chen, Batson, Dang: Applied integer Programming] Chapter 3 and 4.1-4.3 by Johan Högdahl and Victoria Svedberg Seminar 2, 2015-03-31 Todays presentation Chapter 3 Transforms using

More information

FINITE ELEMENT : MATRIX FORMULATION. Georges Cailletaud Ecole des Mines de Paris, Centre des Matériaux UMR CNRS 7633

FINITE ELEMENT : MATRIX FORMULATION. Georges Cailletaud Ecole des Mines de Paris, Centre des Matériaux UMR CNRS 7633 FINITE ELEMENT : MATRIX FORMULATION Georges Cailletaud Ecole des Mines de Paris, Centre des Matériaux UMR CNRS 76 FINITE ELEMENT : MATRIX FORMULATION Discrete vs continuous Element type Polynomial approximation

More information

Computer Networks 56 (2012) 2049 2067. Contents lists available at SciVerse ScienceDirect. Computer Networks

Computer Networks 56 (2012) 2049 2067. Contents lists available at SciVerse ScienceDirect. Computer Networks Computer Networks 56 (212) 249 267 Contents lists available at SciVerse ScienceDirect Computer Networks journal homepage: www.elsevier.com/locate/comnet Structural analysis of network traffic matrix via

More information

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 15, NO. 5, MAY

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 15, NO. 5, MAY IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 15, NO. 5, MAY 2006 1071 Wavelet-Domain Approximation and Compression of Piecewise Smooth Images Michael B. Wakin, Student Member, IEEE, Justin K. Romberg, Member,

More information

Big Data & Scripting storage networks and distributed file systems

Big Data & Scripting storage networks and distributed file systems Big Data & Scripting storage networks and distributed file systems 1, 2, in the remainder we use networks of computing nodes to enable computations on even larger datasets for a computation, each node

More information

VOL. 2, NO. 4, May 2012 ISSN ARPN Journal of Science and Technology All rights reserved.

VOL. 2, NO. 4, May 2012 ISSN ARPN Journal of Science and Technology All rights reserved. Analysis of Various Image Compression Techniques 1 G.M.Padmaja, 2 P.Nirupama 1 Senior Assistant Professor in CSE Dept, BVRIT 2 Associate Professor in CSE Dept, SIET 1 padmaja.gmp@gmail.com, 2 nirupama.cse1@gmail.com

More information

Computational Optical Imaging - Optique Numerique. -- Deconvolution --

Computational Optical Imaging - Optique Numerique. -- Deconvolution -- Computational Optical Imaging - Optique Numerique -- Deconvolution -- Winter 2014 Ivo Ihrke Deconvolution Ivo Ihrke Outline Deconvolution Theory example 1D deconvolution Fourier method Algebraic method

More information

Couple Dictionary Training for Image Super-resolution

Couple Dictionary Training for Image Super-resolution IEEE TRANSACTIONS ON IMAGE PROCESSING 1 Couple Dictionary Training for Image Super-resolution Jianchao Yang, Student Member, IEEE, Zhaowen Wang, Student Member, IEEE, Zhe Lin, Member, IEEE, Scott Cohen,

More information

α = u v. In other words, Orthogonal Projection

α = u v. In other words, Orthogonal Projection Orthogonal Projection Given any nonzero vector v, it is possible to decompose an arbitrary vector u into a component that points in the direction of v and one that points in a direction orthogonal to v

More information

Monte Carlo Simulation

Monte Carlo Simulation 1 Monte Carlo Simulation Stefan Weber Leibniz Universität Hannover email: sweber@stochastik.uni-hannover.de web: www.stochastik.uni-hannover.de/ sweber Monte Carlo Simulation 2 Quantifying and Hedging

More information

Pixel enlargement in high-speed camera image acquisition based on 3D sparse representations

Pixel enlargement in high-speed camera image acquisition based on 3D sparse representations Pixel enlargement in high-speed camera image acquisition based on 3D sparse representations A. Hirabayashi, N. Nogami, J. White Ritsumeikan University College of Information Science and Engineering Kusatsu,

More information

An Information-Theoretic Approach to Distributed Compressed Sensing

An Information-Theoretic Approach to Distributed Compressed Sensing An Information-Theoretic Approach to Distributed Compressed Sensing Dror Baron, Marco F. Duarte, Shriram Sarvotham, Michael B. Wakin and Richard G. Baraniuk Dept. of Electrical and Computer Engineering,

More information

International Doctoral School Algorithmic Decision Theory: MCDA and MOO

International Doctoral School Algorithmic Decision Theory: MCDA and MOO International Doctoral School Algorithmic Decision Theory: MCDA and MOO Lecture 2: Multiobjective Linear Programming Department of Engineering Science, The University of Auckland, New Zealand Laboratoire

More information

BIL Computer Vision Mar 5, 2014

BIL Computer Vision Mar 5, 2014 BIL 719 - Computer Vision Mar 5, 2014 Image pyramids Aykut Erdem Dept. of Computer Engineering Hacettepe University Image Scaling This image is too big to fit on the screen. How can we reduce it? How to

More information

Signal Processing for Speech Recognition

Signal Processing for Speech Recognition Signal Processing for Speech Recognition Once a signal has been sampled, we have huge amounts of data, often 20,000 16 bit numbers a second! We need to find ways to concisely capture the properties of

More information

Sparse Signal Recovery: Theory, Applications and Algorithms

Sparse Signal Recovery: Theory, Applications and Algorithms Sparse Signal Recovery: Theory, Applications and Algorithms Bhaskar Rao Department of Electrical and Computer Engineering University of California, San Diego Collaborators: I. Gorodonitsky, S. Cotter,

More information

Image Compression through DCT and Huffman Coding Technique

Image Compression through DCT and Huffman Coding Technique International Journal of Current Engineering and Technology E-ISSN 2277 4106, P-ISSN 2347 5161 2015 INPRESSCO, All Rights Reserved Available at http://inpressco.com/category/ijcet Research Article Rahul

More information

Sparsity & Co.: An Overview of Analysis vs Synthesis in Low-Dimensional Signal Models. Rémi Gribonval, INRIA Rennes - Bretagne Atlantique

Sparsity & Co.: An Overview of Analysis vs Synthesis in Low-Dimensional Signal Models. Rémi Gribonval, INRIA Rennes - Bretagne Atlantique Sparsity & Co.: An Overview of Analysis vs Synthesis in Low-Dimensional Signal Models Rémi Gribonval, INRIA Rennes - Bretagne Atlantique remi.gribonval@inria.fr http://www.irisa.fr/metiss/members/remi/talks

More information

A NEW LOOK AT CONVEX ANALYSIS AND OPTIMIZATION

A NEW LOOK AT CONVEX ANALYSIS AND OPTIMIZATION 1 A NEW LOOK AT CONVEX ANALYSIS AND OPTIMIZATION Dimitri Bertsekas M.I.T. FEBRUARY 2003 2 OUTLINE Convexity issues in optimization Historical remarks Our treatment of the subject Three unifying lines of

More information

7 Gaussian Elimination and LU Factorization

7 Gaussian Elimination and LU Factorization 7 Gaussian Elimination and LU Factorization In this final section on matrix factorization methods for solving Ax = b we want to take a closer look at Gaussian elimination (probably the best known method

More information

Modelling, Extraction and Description of Intrinsic Cues of High Resolution Satellite Images: Independent Component Analysis based approaches

Modelling, Extraction and Description of Intrinsic Cues of High Resolution Satellite Images: Independent Component Analysis based approaches Modelling, Extraction and Description of Intrinsic Cues of High Resolution Satellite Images: Independent Component Analysis based approaches PhD Thesis by Payam Birjandi Director: Prof. Mihai Datcu Problematic

More information

Probabilistic Models for Big Data. Alex Davies and Roger Frigola University of Cambridge 13th February 2014

Probabilistic Models for Big Data. Alex Davies and Roger Frigola University of Cambridge 13th February 2014 Probabilistic Models for Big Data Alex Davies and Roger Frigola University of Cambridge 13th February 2014 The State of Big Data Why probabilistic models for Big Data? 1. If you don t have to worry about

More information