# Introduction to Inverse Problems (2 lectures)

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 Introduction to Inverse Problems (2 lectures) Summary Direct and inverse problems Examples of direct (forward) problems Deterministic and statistical points of view Ill-posed and ill-conditioned problems An illustrative example: The deconvolution problem Truncated Fourier decomposition (TFD); Tikhonov regularization Generalized Tikhonov regularization; Bayesian perspective. Iterative optimization. IP, José Bioucas Dias, 2007, IST 1

2 Direct/Inverse problems Direct (forward) problem Causes Inverse problem Effects Example: Direct problem: the computation of the trajectories of bodies from the knowledge of the forces. Inverse problem: determination of the forces from the knowledge of the trajectories Newton solved the first direct/inverse problem: the determintion of the gravitation force from the Kepler laws describing the trajectories of planets 2

3 An example: a linear time invariant (LTI) system Direct problem: Fourier domain Inverse problem: Source of difficulties: is unbounded A perturbation on leads to a perturbation on given by high frequencies of the perturbation are amplified, degrading the estimate of f 3

4 Image deblurring Observation model in (linear) image restoration/reconstruction observed image noise Linear operator original image (e.g., blur, tomography, MRI,...) Goal: estimate f from g 4

5 Image deblurring via regularization original Blurred, 9x9 uniform restored 5

6 MRI example Hydrogen density 2D frequency samples (9.4%) 6

7 Compressed Sensing (sparse representation) Sparse vector f Random matrix Observed data y Compressed Sensing N=1000 M=

8 Classes of direct problems Deterministic observation mechanism Original data (image) Operator + Observed data (image) perturbation 8

9 Classes of direct problems (deterministic) Linear space-invariant imaging systems Blur (motion, out-of-focus, Diffraction-limited imaging atmospheric) Near field acoustic holography Channel equalization Parameter identification Linear space-variant imaging systems (first kind Fredholm equation) X-ray tomography MR imaging Radar imaging Sonar imaging Inverse diffraction Inverse source Linear regression 9

10 Classes of direct problems Statistical observation mechanism Original data (image) Observed data (image) Ex: Linear/nonlinear observations in additive Gaussian noise + 10

11 Classes of direct problems (statistic) Linear/nonlinear observation driven by non-additive noise Parameters of a distribution Random signal/ image Rayleigh noise in coherent imaging Poisson noise in photo-electric conversion SPET (single photon emission tomography) PET (positron emission tomography) Ex: Amplitude in a coherent imaging system (radar, ultrasound) Terrain reflectance Inphase/quadrature backscattered signal 11

12 Well-posed/ill-posed inverse problems [Hadamard, 1923] Definition: Let be a (possible nonlinear) operator The inverse problem of solving Hadamard sense if: is well-posed in the 1) A solution exists for any in the observed data space 2) The solution is unique 3) The inverse mapping is continuous An inverse problem that is not well-posed is termed ill-posed The operator A of an inverse well/ill-posed problem is termed well/ill-posed 12

13 Finite/Infinite dimensional linear operators Linear Operators: The linear inverse problem equivalently, is well-posed if 1) and 2) holds or, and If is finite-dimensional, the corresponding inverse problem is well-posed iif either one of the properties 1) and 2) holds Example: In infinite-dimensional spaces Consider A defined on, If a solution of exists, it is unique since However, there are elements not in Thus, A is ill-posed (point 1 of the Hadamard conditions does not hold) Stability is also lacking: Take Then, does not converge when 13

14 Ill-conditioned inverse problems lll-posed lll-conditioned Many well-posed inverse problems are ill-conditioned, in the sense that For linear operators (tight bound) 14

15 Example: Discrete deconvolution Cyclic convolution Matrix notation N-periodic funtions A is cyclic Toeplitz 15

16 Example: Discrete deconvolution 16

17 Eigen-decomposition of cyclic matrices (unitary) Eigenvector (Fourier) matrix Eigenvalue matrix (diagonal) is the DFT of at frequency 17

18 Example: Discrete deconvolution (geometric viewpoint) Action of A on f Note: 18

19 Example: Discrete deconvolution (inferring f) Assume that Then is invertible and Thus, assuming the direct model We have error 19

20 Example: cyclic convolution with a Gaussian kernel What went wrong? 20

21 Example: Discrete deconvolution (estimation error) Size of the error Assume that Thus Which is a set enclosed by an ellipsoid with radii 21

22 Example: Discrete deconvolution (estimation error) The estimation error is the vector The components satisfy 22

23 Cyclic convolution with a Gaussian kernel (cont.) (unit impulse function) Noise dominates at high frequencies and is amplified by 23

24 Example: Discrete deconvolution (A is ill-posed) Assume now that is not invertible and it may happen that i.e, some are zero Least-squares solution Projection error 24

25 Example: Discrete deconvolution (A is ill-posed) Least-squares approach Orthogonal components 25

26 Example: Discrete deconvolution (A is ill-posed) Invisible objects is the minimum norm solution (related to the Moore-Penrose inverse) 26

27 Example: Discrete deconvolution (Regularization) A is ill-conditioned A is ill-posed In both cases small eigenvalues are sources of instabilities Often, the smaller the eigenvalue the more oscilating the corresponding eigenvector (high frequences) Regularization by filtereing: shrink/threshold large values of i.e, multiply the eigenvalues by a regularizer filter such that as 27

28 Example: Discrete deconvolution Regularization by filtering (frequency multiplication time convolution) Such that 1) as 2 ) The larger eigenvalues are retained Truncated Fourier Decomposition (TFD) Tikhonov (Wiener) filter 28

29 Example: Discrete deconvolution (Regularization by filtering) TFD Tikhonov Tikhonov regularization Thus Solution of the variational problem 29

30 Example: Discrete deconvolution (1D example) Gaussian shaped of standard deviation = f g frequency

31 Example: Discrete deconvolution (1D example -TFD) f f f f 1 f f f f 1 f f 1 f f

32 Example: Discrete deconvolution (2D example-fd) uniform 32

33 Example: Discrete deconvolution (2D example-tfd) 33

34 Curing Ill-posed/Ill-conditioned inverse problems Golden rule for solving ill-posed/ill-conditioned inverse problems Search for solutions which: are compatible with the observed data satisfy additional constraints (a priori or prior information) coming from the (physics) problem 34

35 Generalized Tikhonov regularization Tikhonov and TFD regularization are not well suited to deal with data Nonhomogeneities, such as edges Generalized Tikhonov regularization Bayesian viewpoint Data Discrepancy term Penalty/ Regularization term Negative loglikelihood Negative logprior 35

36 Dominating approaches to regularization 1) 2) 3) 4) In given circumstances 2), 3), and 4) are equivalent 36

37 Example: Discrete deconvolution (Nonquadratic regularization) penalize oscillatory solutions discontinuity preserving (robust) regularization is nonconvex hard optimization problem non-discontinuity preserving regularization is convex treatable optimization problem 37

38 Optimization - Quadratic Linear system of equations Large systems require iterative methods - Non-quadratic and smooth Methods: Steepest descent, nonlinear conjugate gradient, Newton, trust regions, - Non-quadratic and nonsmooth Constrained optimization (Linear, quadratic, second-order cone programs) Methods: Iterative Shrinkage/Thesholding; Coordinate Subspace Optimization; forward-backward splitting; Primal-dual Newton Majorization Minimizaton (MM) class 38

39 Majorization Minorization (MM) Framework Let Majorization Minorization algorithm:...with equality if and only if Easy to prove monotonicity: Notes: should be easy to maximize EM is an algorithm of this type. 39

40 Example: Discrete deconvolution (1D example NQ Regula.) Tikhonov f f f f f f f f f f f f

41 Example: Discrete deconvolution (2D example-total Variation) Total variation regularization (TV) where TV regularizer penalizes highly oscilatory solutions, while it preserves the edges 41

42 Bibliography [Ch1. RB2, Ch1. L1] Important topics Euclidian and Hilbert spaces of functions [App. A, RB2] Linear operators in function spaces [App. B, RB2] Euclidian vector spaces and matrices [App. C, RB2] Properties of the DFT and the FFT algorithm [App. B, RB2] Matlab scripts TFD_regularization_1D.m TFD_regularization_2D.m TFD_Error_1D.m TV_regulatization_1D.m 42

### ADVANCED LINEAR ALGEBRA FOR ENGINEERS WITH MATLAB. Sohail A. Dianat. Rochester Institute of Technology, New York, U.S.A. Eli S.

ADVANCED LINEAR ALGEBRA FOR ENGINEERS WITH MATLAB Sohail A. Dianat Rochester Institute of Technology, New York, U.S.A. Eli S. Saber Rochester Institute of Technology, New York, U.S.A. (g) CRC Press Taylor

### Numerical Methods For Image Restoration

Numerical Methods For Image Restoration CIRAM Alessandro Lanza University of Bologna, Italy Faculty of Engineering CIRAM Outline 1. Image Restoration as an inverse problem 2. Image degradation models:

### Computational Optical Imaging - Optique Numerique. -- Deconvolution --

Computational Optical Imaging - Optique Numerique -- Deconvolution -- Winter 2014 Ivo Ihrke Deconvolution Ivo Ihrke Outline Deconvolution Theory example 1D deconvolution Fourier method Algebraic method

### Another Example: the Hubble Space Telescope

296 DIP Chapter and 2: Introduction and Integral Equations Motivation: Why Inverse Problems? A large-scale example, coming from a collaboration with Università degli Studi di Napoli Federico II in Naples.

### Sparse recovery and compressed sensing in inverse problems

Gerd Teschke (7. Juni 2010) 1/68 Sparse recovery and compressed sensing in inverse problems Gerd Teschke (joint work with Evelyn Herrholz) Institute for Computational Mathematics in Science and Technology

### Regularized Matrix Computations

Regularized Matrix Computations Andrew E. Yagle Department of EECS, The University of Michigan, Ann Arbor, MI 489- Abstract We review the basic results on: () the singular value decomposition (SVD); ()

### Advanced Signal Processing and Digital Noise Reduction

Advanced Signal Processing and Digital Noise Reduction Saeed V. Vaseghi Queen's University of Belfast UK WILEY HTEUBNER A Partnership between John Wiley & Sons and B. G. Teubner Publishers Chichester New

### ROBERTO BATTITI, MAURO BRUNATO. The LION Way: Machine Learning plus Intelligent Optimization. LIONlab, University of Trento, Italy, Apr 2015

ROBERTO BATTITI, MAURO BRUNATO. The LION Way: Machine Learning plus Intelligent Optimization. LIONlab, University of Trento, Italy, Apr 2015 http://intelligentoptimization.org/lionbook Roberto Battiti

### Eigenvalues of C SMS (16) 1 A. Eigenvalues of A (32) 1 A (16,32) 1 A

Negative Results for Multilevel Preconditioners in Image Deblurring C. R. Vogel Department of Mathematical Sciences Montana State University Bozeman, MT 59717-0240 USA Abstract. A one-dimensional deconvolution

### COMPUTATIONAL INTELLIGENCE (INTRODUCTION TO MACHINE LEARNING) SS16. Lecture 2: Linear Regression Gradient Descent Non-linear basis functions

COMPUTATIONAL INTELLIGENCE (INTRODUCTION TO MACHINE LEARNING) SS16 Lecture 2: Linear Regression Gradient Descent Non-linear basis functions LINEAR REGRESSION MOTIVATION Why Linear Regression? Regression

### Inverse problems. Nikolai Piskunov 2014. Regularization: Example Lecture 4

Inverse problems Nikolai Piskunov 2014 Regularization: Example Lecture 4 Now that we know the theory, let s try an application: Problem: Optimal filtering of 1D and 2D data Solution: formulate an inverse

### AN INTRODUCTION TO NUMERICAL METHODS AND ANALYSIS

AN INTRODUCTION TO NUMERICAL METHODS AND ANALYSIS Revised Edition James Epperson Mathematical Reviews BICENTENNIAL 0, 1 8 0 7 z ewiley wu 2007 r71 BICENTENNIAL WILEY-INTERSCIENCE A John Wiley & Sons, Inc.,

### 1. Introduction. Consider the computation of an approximate solution of the minimization problem

A NEW TIKHONOV REGULARIZATION METHOD MARTIN FUHRY AND LOTHAR REICHEL Abstract. The numerical solution of linear discrete ill-posed problems typically requires regularization, i.e., replacement of the available

### Numerical Recipes in C

2008 AGI-Information Management Consultants May be used for personal purporses only or by libraries associated to dandelon.com network. Numerical Recipes in C The Art of Scientific Computing Second Edition

### Lecture 14. Point Spread Function (PSF)

Lecture 14 Point Spread Function (PSF), Modulation Transfer Function (MTF), Signal-to-noise Ratio (SNR), Contrast-to-noise Ratio (CNR), and Receiver Operating Curves (ROC) Point Spread Function (PSF) Recollect

### P164 Tomographic Velocity Model Building Using Iterative Eigendecomposition

P164 Tomographic Velocity Model Building Using Iterative Eigendecomposition K. Osypov* (WesternGeco), D. Nichols (WesternGeco), M. Woodward (WesternGeco) & C.E. Yarman (WesternGeco) SUMMARY Tomographic

### STA 4273H: Statistical Machine Learning

STA 4273H: Statistical Machine Learning Russ Salakhutdinov Department of Statistics! rsalakhu@utstat.toronto.edu! http://www.cs.toronto.edu/~rsalakhu/ Lecture 6 Three Approaches to Classification Construct

### Convolution. 1D Formula: 2D Formula: Example on the web: http://www.jhu.edu/~signals/convolve/

Basic Filters (7) Convolution/correlation/Linear filtering Gaussian filters Smoothing and noise reduction First derivatives of Gaussian Second derivative of Gaussian: Laplacian Oriented Gaussian filters

### Least-Squares Intersection of Lines

Least-Squares Intersection of Lines Johannes Traa - UIUC 2013 This write-up derives the least-squares solution for the intersection of lines. In the general case, a set of lines will not intersect at a

### Bindel, Fall 2012 Matrix Computations (CS 6210) Week 8: Friday, Oct 12

Why eigenvalues? Week 8: Friday, Oct 12 I spend a lot of time thinking about eigenvalue problems. In part, this is because I look for problems that can be solved via eigenvalues. But I might have fewer

### Applications to Data Smoothing and Image Processing I

Applications to Data Smoothing and Image Processing I MA 348 Kurt Bryan Signals and Images Let t denote time and consider a signal a(t) on some time interval, say t. We ll assume that the signal a(t) is

### Section for Cognitive Systems DTU Informatics, Technical University of Denmark

Transformation Invariant Sparse Coding Morten Mørup & Mikkel N Schmidt Morten Mørup & Mikkel N. Schmidt Section for Cognitive Systems DTU Informatics, Technical University of Denmark Redundancy Reduction

### ALGEBRAIC EIGENVALUE PROBLEM

ALGEBRAIC EIGENVALUE PROBLEM BY J. H. WILKINSON, M.A. (Cantab.), Sc.D. Technische Universes! Dsrmstedt FACHBEREICH (NFORMATiK BIBL1OTHEK Sachgebieto:. Standort: CLARENDON PRESS OXFORD 1965 Contents 1.

### Modern methods for solving inverse problems

Modern methods for solving inverse problems Jürgen Hesser Experimental Radiation Oncology University Medical Center Mannheim University of Heidelberg 1 Motivation Image Guided Radiation Therapy (IGRT)

### Contents. Gbur, Gregory J. Mathematical methods for optical physics and engineering digitalisiert durch: IDS Basel Bern

Preface page xv 1 Vector algebra 1 1.1 Preliminaries 1 1.2 Coordinate System invariance 4 1.3 Vector multiplication 9 1.4 Useful products of vectors 12 1.5 Linear vector Spaces 13 1.6 Focus: periodic media

### Chapter 3. Inverse Problems

20 Chapter 3 Inverse Problems Mathematicians often cannot help but cringe when physicists are doing math, and even more so whenever physicists claim to be rigorous with their math. This chapter is not

### Blind Deconvolution of Corrupted Barcode Signals

Blind Deconvolution of Corrupted Barcode Signals Everardo Uribe and Yifan Zhang Advisors: Ernie Esser and Yifei Lou Interdisciplinary Computational and Applied Mathematics Program University of California,

### System Identification for Acoustic Comms.:

System Identification for Acoustic Comms.: New Insights and Approaches for Tracking Sparse and Rapidly Fluctuating Channels Weichang Li and James Preisig Woods Hole Oceanographic Institution The demodulation

### PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 4: LINEAR MODELS FOR CLASSIFICATION

PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 4: LINEAR MODELS FOR CLASSIFICATION Introduction In the previous chapter, we explored a class of regression models having particularly simple analytical

### Numerical Methods I Eigenvalue Problems

Numerical Methods I Eigenvalue Problems Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 Course G63.2010.001 / G22.2420-001, Fall 2010 September 30th, 2010 A. Donev (Courant Institute)

### Linear Algebra. Introduction

Linear Algebra Caren Diefenderfer, Hollins University, chair David Hill, Temple University, lead writer Sheldon Axler, San Francisco State University Nancy Neudauer, Pacific University David Strong, Pepperdine

### Blind Deconvolution of Barcodes via Dictionary Analysis and Wiener Filter of Barcode Subsections

Blind Deconvolution of Barcodes via Dictionary Analysis and Wiener Filter of Barcode Subsections Maximilian Hung, Bohyun B. Kim, Xiling Zhang August 17, 2013 Abstract While current systems already provide

### 2.5 Complex Eigenvalues

1 25 Complex Eigenvalues Real Canonical Form A semisimple matrix with complex conjugate eigenvalues can be diagonalized using the procedure previously described However, the eigenvectors corresponding

### By choosing to view this document, you agree to all provisions of the copyright laws protecting it.

This material is posted here with permission of the IEEE Such permission of the IEEE does not in any way imply IEEE endorsement of any of Helsinki University of Technology's products or services Internal

### ELEC-E8104 Stochastics models and estimation, Lecture 3b: Linear Estimation in Static Systems

Stochastics models and estimation, Lecture 3b: Linear Estimation in Static Systems Minimum Mean Square Error (MMSE) MMSE estimation of Gaussian random vectors Linear MMSE estimator for arbitrarily distributed

### 1646 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 6, NO. 12, DECEMBER 1997

1646 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL 6, NO 12, DECEMBER 1997 Restoration of a Single Superresolution Image from Several Blurred, Noisy, and Undersampled Measured Images Michael Elad and Arie

### Latest Results on High-Resolution Reconstruction from Video Sequences

Latest Results on High-Resolution Reconstruction from Video Sequences S. Lertrattanapanich and N. K. Bose The Spatial and Temporal Signal Processing Center Department of Electrical Engineering The Pennsylvania

### Need for Speed in MRI & Parallel Imaging I

G16.4428 Practical Magnetic Resonance Imaging II Sackler Institute of Biomedical Sciences New York University School of Medicine Need for Speed in MRI & Parallel Imaging I Ricardo Otazo, PhD ricardo.otazo@nyumc.org

### Change Point Estimation of Bar Code Signals

1 Change Point Estimation of Bar Code Signals Leming Qu, and Yi-Cheng Tu Abstract Existing methods for bar code signal reconstruction is based on either the local approach or the regularization approach

### Numerical Analysis Introduction. Student Audience. Prerequisites. Technology.

Numerical Analysis Douglas Faires, Youngstown State University, (Chair, 2012-2013) Elizabeth Yanik, Emporia State University, (Chair, 2013-2015) Graeme Fairweather, Executive Editor, Mathematical Reviews,

### Markov chains and Markov Random Fields (MRFs)

Markov chains and Markov Random Fields (MRFs) 1 Why Markov Models We discuss Markov models now. This is the simplest statistical model in which we don t assume that all variables are independent; we assume

### Computational Methods CMSC/AMSC/MAPL 460. Eigenvalues and Eigenvectors. Ramani Duraiswami, Dept. of Computer Science

Computational Methods CMSC/AMSC/MAPL 460 Eigenvalues and Eigenvectors Ramani Duraiswami, Dept. of Computer Science Eigen Values of a Matrix Definition: A N N matrix A has an eigenvector x (non-zero) with

### Principal Component Analysis Application to images

Principal Component Analysis Application to images Václav Hlaváč Czech Technical University in Prague Faculty of Electrical Engineering, Department of Cybernetics Center for Machine Perception http://cmp.felk.cvut.cz/

### Signal Processing for Intelligent Sensor Systems with MATLAB 9

Signal Processing for Intelligent Sensor Systems with MATLAB 9 Second Edition David С Swanson @ CRC Press Taylor & Francis Gro jroup Boca Raton London New York CRC Press is an imprint of the Taylor & Francis

### Compressive Acquisition of Sparse Deflectometric Maps

Compressive Acquisition of Sparse Deflectometric Maps Prasad Sudhakar*, Laurent Jacques*, Adriana Gonzalez Gonzalez* Xavier Dubois +, Philippe Antoine +, Luc Joannes + *: Louvain University (UCL), Louvain-la-Neuve,

### ORTHOGONAL PROJECTION REGULARIZATION OPERATORS

ORTHOGONAL PROJECTION REGULARIZATION OPERATORS S. MORIGI, L. REICHEL, AND F. SGALLARI Abstract. Tikhonov regularization often is applied with a finite difference regularization operator that approximates

### Bildverarbeitung und Mustererkennung Image Processing and Pattern Recognition

Bildverarbeitung und Mustererkennung Image Processing and Pattern Recognition 710.080 2VO 710.081 1KU 1 Optical Flow (I) Content Introduction Local approach (Lucas Kanade) Global approaches (Horn-Schunck,

### Digital Image Processing

Digital Image Processing Using MATLAB Second Edition Rafael C. Gonzalez University of Tennessee Richard E. Woods MedData Interactive Steven L. Eddins The MathWorks, Inc. Gatesmark Publishing A Division

### International Journal of Advanced Research in Computer Science and Software Engineering

Volume 4, Issue 8, August 2014 ISSN: 2277 128X International Journal of Advanced Research in Computer Science and Software Engineering Research Paper Available online at: www.ijarcsse.com Comparative Study

### Lecture 9: Continuous

CSC2515 Fall 2007 Introduction to Machine Learning Lecture 9: Continuous Latent Variable Models 1 Example: continuous underlying variables What are the intrinsic latent dimensions in these two datasets?

### PTE505: Inverse Modeling for Subsurface Flow Data Integration (3 Units)

PTE505: Inverse Modeling for Subsurface Flow Data Integration (3 Units) Instructor: Behnam Jafarpour, Mork Family Department of Chemical Engineering and Material Science Petroleum Engineering, HED 313,

### Curve Fitting. Next: Numerical Differentiation and Integration Up: Numerical Analysis for Chemical Previous: Optimization.

Next: Numerical Differentiation and Integration Up: Numerical Analysis for Chemical Previous: Optimization Subsections Least-Squares Regression Linear Regression General Linear Least-Squares Nonlinear

### Linear Algebra Review. Vectors

Linear Algebra Review By Tim K. Marks UCSD Borrows heavily from: Jana Kosecka kosecka@cs.gmu.edu http://cs.gmu.edu/~kosecka/cs682.html Virginia de Sa Cogsci 8F Linear Algebra review UCSD Vectors The length

### New Method for Deconvolving Adaptive Optics Images

New Method for Deconvolving Adaptive Optics Images or: (All you ever wanted to know about...) Myopic maximum-likelihood deconvolution with a space-varying kernel Ralf Flicker, François Rigaut Gemini Observatory,

### Dual Methods for Total Variation-Based Image Restoration

Dual Methods for Total Variation-Based Image Restoration Jamylle Carter Institute for Mathematics and its Applications University of Minnesota, Twin Cities Ph.D. (Mathematics), UCLA, 2001 Advisor: Tony

### The Power Method for Eigenvalues and Eigenvectors

Numerical Analysis Massoud Malek The Power Method for Eigenvalues and Eigenvectors The spectrum of a square matrix A, denoted by σ(a) is the set of all eigenvalues of A. The spectral radius of A, denoted

### Linear Regression CS434. Supervised learning

Linear Regression CS434 A regression problem We want to learn to predict a person s height based on his/her knee height and/or arm span This is useful for patients who are bed bound and cannot stand to

### Principles of Digital Communication

Principles of Digital Communication Robert G. Gallager January 5, 2008 ii Preface: introduction and objectives The digital communication industry is an enormous and rapidly growing industry, roughly comparable

### Journal of Computational and Applied Mathematics

Journal of Computational and Applied Mathematics 226 (2009) 92 102 Contents lists available at ScienceDirect Journal of Computational and Applied Mathematics journal homepage: www.elsevier.com/locate/cam

### Advanced Computational Techniques in Engineering

Course Details Course Code: 5MWCI432 Level: M Tech CSE Semester: Semester 4 Instructor: Prof Dr S P Ghrera Advanced Computational Techniques in Engineering Outline Course Description: Data Distributions,

### Computational Methods. Least Squares Approximation/Optimization

Computational Methods Least Squares Approximation/Optimization Manfred Huber 2010 1 Least Squares Least squares methods are aimed at finding approximate solutions when no precise solution exists Find the

### Optical Flow. Thomas Pock

Optical Flow Thomas Pock 1 Optical Flow (II) Content Global approaches (Horn-Schunck, TV-L1) Coarse-to-fine warping 2 The Horn and Schunck (HS) Method 3 The Horn and Schunck (HS) Method Global energy to

### Regression Using Support Vector Machines: Basic Foundations

Regression Using Support Vector Machines: Basic Foundations Technical Report December 2004 Aly Farag and Refaat M Mohamed Computer Vision and Image Processing Laboratory Electrical and Computer Engineering

### Numerical Recipes in C++

Numerical Recipes in C++ The Art of Scientific Computing Second Edition William H. Press Los Alamos National Laboratory Saul A. Teukolsky Department of Physics, Cornell University William T. Vetterling

### Nonlinear Iterative Partial Least Squares Method

Numerical Methods for Determining Principal Component Analysis Abstract Factors Béchu, S., Richard-Plouet, M., Fernandez, V., Walton, J., and Fairley, N. (2016) Developments in numerical treatments for

Vector Derivatives, Gradients, and Generalized Gradient Descent Algorithms ECE 275A Statistical Parameter Estimation Ken Kreutz-Delgado ECE Department, UC San Diego November 1, 2013 Ken Kreutz-Delgado

### Linear Threshold Units

Linear Threshold Units w x hx (... w n x n w We assume that each feature x j and each weight w j is a real number (we will relax this later) We will study three different algorithms for learning linear

### Signal Detection. Outline. Detection Theory. Example Applications of Detection Theory

Outline Signal Detection M. Sami Fadali Professor of lectrical ngineering University of Nevada, Reno Hypothesis testing. Neyman-Pearson (NP) detector for a known signal in white Gaussian noise (WGN). Matched

### Numerical Methods I Solving Linear Systems: Sparse Matrices, Iterative Methods and Non-Square Systems

Numerical Methods I Solving Linear Systems: Sparse Matrices, Iterative Methods and Non-Square Systems Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 Course G63.2010.001 / G22.2420-001,

### CS 591.03 Introduction to Data Mining Instructor: Abdullah Mueen

CS 591.03 Introduction to Data Mining Instructor: Abdullah Mueen LECTURE 3: DATA TRANSFORMATION AND DIMENSIONALITY REDUCTION Chapter 3: Data Preprocessing Data Preprocessing: An Overview Data Quality Major

### LECTURE NOTES: FINITE ELEMENT METHOD

LECTURE NOTES: FINITE ELEMENT METHOD AXEL MÅLQVIST. Motivation The finite element method has two main strengths... Geometry. Very complex geometries can be used. This is probably the main reason why finite

### Review Jeopardy. Blue vs. Orange. Review Jeopardy

Review Jeopardy Blue vs. Orange Review Jeopardy Jeopardy Round Lectures 0-3 Jeopardy Round \$200 How could I measure how far apart (i.e. how different) two observations, y 1 and y 2, are from each other?

### Inverse Problems and Regularization An Introduction

Inverse Problems and Regularization An Introduction Stefan Kindermann Industrial Mathematics Institute University of Linz, Austria What are Inverse Problems? One possible definition [Engl, Hanke, Neubauer

### NMR Measurement of T1-T2 Spectra with Partial Measurements using Compressive Sensing

NMR Measurement of T1-T2 Spectra with Partial Measurements using Compressive Sensing Alex Cloninger Norbert Wiener Center Department of Mathematics University of Maryland, College Park http://www.norbertwiener.umd.edu

### Mean value theorem, Taylors Theorem, Maxima and Minima.

MA 001 Preparatory Mathematics I. Complex numbers as ordered pairs. Argand s diagram. Triangle inequality. De Moivre s Theorem. Algebra: Quadratic equations and express-ions. Permutations and Combinations.

### Applied Linear Algebra I Review page 1

Applied Linear Algebra Review 1 I. Determinants A. Definition of a determinant 1. Using sum a. Permutations i. Sign of a permutation ii. Cycle 2. Uniqueness of the determinant function in terms of properties

### Understanding and Applying Kalman Filtering

Understanding and Applying Kalman Filtering Lindsay Kleeman Department of Electrical and Computer Systems Engineering Monash University, Clayton 1 Introduction Objectives: 1. Provide a basic understanding

### 4F7 Adaptive Filters (and Spectrum Estimation) Least Mean Square (LMS) Algorithm Sumeetpal Singh Engineering Department Email : sss40@eng.cam.ac.

4F7 Adaptive Filters (and Spectrum Estimation) Least Mean Square (LMS) Algorithm Sumeetpal Singh Engineering Department Email : sss40@eng.cam.ac.uk 1 1 Outline The LMS algorithm Overview of LMS issues

### Total Variation Regularization in PET Reconstruction

Total ariation Regularization in PET Reconstruction Milán Magdics, Balázs Tóth, Balázs Kovács, and László Szirmay-Kalos BME IIT, Hungary magdics@iit.bme.hu Abstract. Positron Emission Tomography reconstruction

### 2094 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 44, NO. 6, OCTOBER Information-Theoretic Image Formation. (Invited Paper)

2094 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 44, NO. 6, OCTOBER 1998 Information-Theoretic Image Formation Joseph A. O Sullivan, Senior Member, IEEE, Richard E. Blahut, Fellow, IEEE, and Donald L.

### ADVANCED LINEAR ALGEBRA FOR ENGINEERS WITH MATLAB

ADVANCED LINEAR ALGEBRA FOR ENGINEERS WITH MATLAB ADVANCED LINEAR ALGEBRA FOR ENGINEERS WITH MATLAB Sohail A. Dianat Rochester Institute of Technology, New York, U.S.A. Eli S. Saber Rochester Institute

### WAVES AND FIELDS IN INHOMOGENEOUS MEDIA

WAVES AND FIELDS IN INHOMOGENEOUS MEDIA WENG CHO CHEW UNIVERSITY OF ILLINOIS URBANA-CHAMPAIGN IEEE PRESS Series on Electromagnetic Waves Donald G. Dudley, Series Editor IEEE Antennas and Propagation Society,

### THE PROBLEM OF finding localized energy solutions

600 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 45, NO. 3, MARCH 1997 Sparse Signal Reconstruction from Limited Data Using FOCUSS: A Re-weighted Minimum Norm Algorithm Irina F. Gorodnitsky, Member, IEEE,

### Mean-Shift Tracking with Random Sampling

1 Mean-Shift Tracking with Random Sampling Alex Po Leung, Shaogang Gong Department of Computer Science Queen Mary, University of London, London, E1 4NS Abstract In this work, boosting the efficiency of

### Lecture 5: Variants of the LMS algorithm

1 Standard LMS Algorithm FIR filters: Lecture 5: Variants of the LMS algorithm y(n) = w 0 (n)u(n)+w 1 (n)u(n 1) +...+ w M 1 (n)u(n M +1) = M 1 k=0 w k (n)u(n k) =w(n) T u(n), Error between filter output

### Cortical Source Localization of Human Scalp EEG. Kaushik Majumdar Indian Statistical Institute Bangalore Center

Cortical Source Localization of Human Scalp EEG Kaushik Majumdar Indian Statistical Institute Bangalore Center Cortical Basis of Scalp EEG Baillet et al, IEEE Sig Proc Mag, Nov 2001, p 16 Mountcastle,

### Lecture 3: Linear methods for classification

Lecture 3: Linear methods for classification Rafael A. Irizarry and Hector Corrada Bravo February, 2010 Today we describe four specific algorithms useful for classification problems: linear regression,

### Lecture Topic: Low-Rank Approximations

Lecture Topic: Low-Rank Approximations Low-Rank Approximations We have seen principal component analysis. The extraction of the first principle eigenvalue could be seen as an approximation of the original

### Finite Dimensional Hilbert Spaces and Linear Inverse Problems

Finite Dimensional Hilbert Spaces and Linear Inverse Problems ECE 174 Lecture Supplement Spring 2009 Ken Kreutz-Delgado Electrical and Computer Engineering Jacobs School of Engineering University of California,

### arxiv:math/ v2 [math.fa] 2 Nov 2003

An iterative thresholding algorithm for linear inverse problems with a sparsity constraint Ingrid Daubechies Michel Defrise Christine De Mol February 1, 2008 arxiv:math/0307152v2 [math.fa] 2 Nov 2003 Abstract

### Discrete Fourier Series & Discrete Fourier Transform Chapter Intended Learning Outcomes

Discrete Fourier Series & Discrete Fourier Transform Chapter Intended Learning Outcomes (i) Understanding the relationships between the transform, discrete-time Fourier transform (DTFT), discrete Fourier

### Introduction and message of the book

1 Introduction and message of the book 1.1 Why polynomial optimization? Consider the global optimization problem: P : for some feasible set f := inf x { f(x) : x K } (1.1) K := { x R n : g j (x) 0, j =

### Bayesian Image Super-Resolution

Bayesian Image Super-Resolution Michael E. Tipping and Christopher M. Bishop Microsoft Research, Cambridge, U.K..................................................................... Published as: Bayesian

### Chapter 6. Orthogonality

6.3 Orthogonal Matrices 1 Chapter 6. Orthogonality 6.3 Orthogonal Matrices Definition 6.4. An n n matrix A is orthogonal if A T A = I. Note. We will see that the columns of an orthogonal matrix must be

### 168 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 1, JANUARY Yonina C. Eldar, Member, IEEE, Aharon Ben-Tal, and Arkadi Nemirovski

168 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 1, JANUARY 2005 Robust Mean-Squared Error Estimation in the Presence of Model Uncertainties Yonina C. Eldar, Member, IEEE, Aharon Ben-Tal, and Arkadi

### Time Domain and Frequency Domain Techniques For Multi Shaker Time Waveform Replication

Time Domain and Frequency Domain Techniques For Multi Shaker Time Waveform Replication Thomas Reilly Data Physics Corporation 1741 Technology Drive, Suite 260 San Jose, CA 95110 (408) 216-8440 This paper

### Outline. Random Variables. Examples. Random Variable

Outline Random Variables M. Sami Fadali Professor of Electrical Engineering University of Nevada, Reno Random variables. CDF and pdf. Joint random variables. Correlated, independent, orthogonal. Correlation,