Splines: A unifying framework for image processing. Michael Unser Biomedical Imaging Group EPFL, Lausanne Switzerland

Transcription

1 Splines: A unifying framework for image processing Michael Unser Biomedical Imaging Group EPL, Lausanne Switzerland Plenary talk, ICIP 2005, Genoa, September 2005

2 Splines: A unifying framework Linking the continuous and the discrete "Sampling and acquisition 2 1 Splines

3 Splines: A unifying framework Linking the discrete and the continuous "Sampling and acquisition "Algorithm design Think analog, act digital Geometric processing eature extraction PDEs... Multiresolution Splines 2 1 Wavelets "Multi-scale approaches 3

4 OUTLINE The basic atoms: B-splines Spline-based image processing Interpolation vs. approximation ast algorithms Applications urther perspectives Splines and wavelet theory Splines and fractals 4

5 Splines: definition Splines: definition Definition: A function s(x) is a polynomial spline of degree n with knots < x k < x k+1 < iff it satisfies the following two properties: Piecewise polynomial: s(x) is a polynomial of degree n within each interval [x k, x k+1 ); Higher-order continuity: s(x), s (1) (x),, s (n 1) (x) are continuous at the knots x k Effective degrees of freedom per segment: n+1 n = 1 (polynomial coefficients) (constraints) Cardinal splines = unit spacing and infinite number of knots The right framework for signal processing 5

6 THE BASIC ATOMS: B-SPLINES Polynomial B-splines B-spline representation Differential properties Dilation properties Multidimensional B-splines 6

7 Polynomial B-splines B-spline basis functions Causal B-spline of degree B-spline of degree n n 1 B-spline basis functions β+(x) n = β+ 0 β+ 0 B-spline β+ 0 (x) basis functions β 0 1, x [0, 1) +(x) = Causal B-spline } B-spline {{ of degree basis } n functions 0, otherwise. Causal B-spline (n + of 1) degree n times Causal B-spline of β+(x) n = β+ 0 degree n β+ 0 β+ 0 (x), β+(x) 0 1, x [0, 1) Key properties = } {{ β+(x) n = β+ 0 β+ 0 β+ 0 (x), } β 0 1, x [0, 1) +(x) = 0, otherwise. Compact support: shortest polynomial spline of degree n β+(x) n = β+ 0 } β+ 0 {{ (n + 1) β+ 0 } times (x) β+(x) 0 1, x [0, 1) = 0, otherwise. Positivity } (n{{ + 1) times } 0, otherwise. Key properties Key Piecewise properties (n polynomial + 1) times Compact support: shortest polynomial spline of degree Key properties n Compact Smoothness: support: Hölder shortest continuous polynomial of order spline n of degree n Positivity Compact Positivity support: shortest polynomial spline of degree n Symmetric PiecewiseB-spline polynomial of degree n Positivity Piecewise ( ) β n polynomial (x) Smoothness: = β n Hölder continuous of order n + x + n+1 Piecewise Smoothness: polynomial Hölder continuous 2 of order n Smoothness: Hölder continuous of order n Symmetric B-spline of degree n Symmetric B-splines ( of degree ) n β n (x) = β+ n x + n+1 Symmetric B-spline ( of degree ) n 1 β n (x) = 2 ( β ) β n (x) = β+ n + n x + n+1 2 x + n

8 Theorem: Every cardinal spline, s(x) V, has a unique and stable representation in terms of its B-spline expansion B-spline representation s(x) = c[k] β n +(x k) Theorem (Schoenberg, 1946) Every cardinal polynomial spline, s(x), has a unique and stable representation in terms of its B-spline expansion s(x) = c[k] β n +(x k) V = { s(x) 1: D n+1 {s(x)} = a[k]δ(x k) 0.8 B-spline representation Basis functions } 4 3 analog signal Cubic spline (n=3) 0.6 Derivative operator: D = d discrete signal 8 (B-spline coefficients) dx Theorem: Every cardinal spline, s(x) V, has a unique and sta representation in terms of its B-spline expansion s(x) = c[k] β n +(x k) In modern terminology: {β n +(x k)} forms a Riesz basis. 8 8

9 The lego revisited Differential operators "Continous operator D{ } = d dx D m { } Construction of the B-spline of degree 0 Discrete operator Step function: x 0 + = D 1 {δ(x)} + { } 1 e jω The lego Step function: x 0 + = D 1 {δ(x)} β+(x) 0 = x 0 + (x 1) 0 + = 1 +x 0 + m β+(x) 0 + { } (1 e jω ) m = x 0 + (x 1) 0 + = 1 +D 1 {δ(x)} β+(x) n = n+1 + D (n+1) {δ(x)} = n + D{ } = d jω 4 β+(x) n = n+1 + D (n+1) {δ(x)} = n+1 + xdx n ( ) + 1 e n! ˆβ n jω n+1 +(ω) = = (1 e ( ) jω (jω 1 e Discrete operator (finite difference) ˆβ +(ω) n jω n+1 = = (1 + { } e jω ) n+1 1 e jω f S, m jω (jω) n+1 + f(x) = β+ m 1 D m f 4 ˆβ +(ω) 0 = 1 e jω f S, m + f(x) = jωβ + m 1 D m D m β n f(x) +(x) = m + β+ n m (x) D m β n +(x) = m + β n m + (x) jω (jω) m "ourier domain formula D{ } = d dx + { } m + { } D m { } Continuous operator (derivative) jω (jω) m 1 e jω (1 e jω ) m 9

10 3 B-splines: differential interpretation Differential operators "Continuous operators D{ } = d dx D m { } Derivatives D{ } = d dx jω + { } Differential operators (jω) m m + { } D m { } Discrete operators x 0 ( ) 1 e + { } 1 e jω ˆβ n jω n+1 + = D 1 {δ(x)} (Step function) +(ω) = = (1 e jω ) n+1 D{ } = d jω B-spline β+(x) 0 = construction 1 +D 1 {δ(x)} = x 0 dx jω (jω) n+1 m + { } (1 e jω ) D m + (x 1) m { } (jω) m jω ) β n+1 +(x) n = n+1 = (1 + D (n+1) {δ(x)} = n+1 + x n + e jω ) n+1 n! ( ) (jω) n+1 1 e + { } 1 e jω 4 ˆβ n jω n+1 +(ω) = = (1 e jω ) n+1 "ourier domain jω formula m + { } (jω) n+1 (1 e jω ) m f S, ( m + f(x) = ) 1 e ˆβ +(ω) n jω n+1 β+ m 1 = = (1 D m f(x) e jω ) n+1 D m β+(x) n = 4 m + jω β+ n m x n + = (x) (jω) n+1 inite differences jω (jω) m 1 e jω (1 e jω ) m Green function of D n+1 D n+1 {ρ(x)} = δ(x) = ρ(x) = D One-sided powerfunction: x n, x 0 0, x < 0 (impulse respo 10

11 B-splines: Dilation properties m-scale relation Dilation by a factor m m-scale relation m-scale relation Dilation by a factor m Dilation by a factor m ( k) with Hm(z) n = 1 m 1 ) n+1 β m n z +(x/m) n = h n m[k]β+(x n k) with k β k=0 +(x/m) n = h n m[k]β n (x k) with Hm(z) n = 1 ( Piecewise constant case (n = 0) Hm(z) n = 1 m 1 ) n+1 Hm(z) 0 m = z k 1 + z 1 + z (m 1) Piecewise constant case (n = 0) k=0 1 (Moving sum filter) Piecewise constant case (n H= m(z) 0 0) = 1 + z 1 + z (m 1) (Moving sum filter) Dyadic case (m = 2) Hm(z) 0 = 1 ( + z 1 + ) 1 + z H2 n 1 n+1 z (m 1) (z) = 2 = 1 (Moving n+1 ( sum filter) ) n + 1 Applications: fast2 spline-based2 n algorithms ( k k=0 1 + z H2 n 1 Zooming (z) = 2 2 Smoothing Multi-scale processing Wavelet transform Dyadic case (m = 2) 5 z k (Binomial filter) ) n+1 = 1 2 n n+1 ( ) n + 1 k 5 k=0 m n ( z k (Bin 5 11

12 Dyadic case: Wavelets Dilation by a factor ofdyadic 2 case: wavelets Dilation β+(x/2) n by= a factor h n 2 [k]β of 2+(x n k) Dyadic case: wavelets β+(x/2) n = h n 2 [k]β+(x n k) Binomial filter Binomial H filter 2 n (z) = 1 ( 1 + z 1 ) n+1 n+1 1 ( ) n + 1 = 2 n ( ) 1 + z H2 n 1 n+1 2 n (z) = 2 = 1 n+1 ( k ) k=0 n + 1 Example: piecewise 2 linear splines 2 n k Example: piecewise linear splines k=0 z k z k

13 B-spline representation of images B-spline representation of images Symmetric, tensor-product B-splines Symmetric, tensor-product B-splines β n (x 1,, x d ) = β n (x 1 ) β n (x d ) Multidimensional spline functions B-spline representation of images s(x 1,, x d ) = c[k 1,, k d ] β n (x 1 k 1,, x d k d ) Symmetric, tensor-product (k 1, k d ) Z d B-splines β n (x 1,, x d ) = β n +(x 1 ) β n +(x d ) Multidimensional spline function Multidimensional spline functions s(x 1,, x d ) = c[k 1,, k d ] β n (x 1 k 1,, x d k d ) (k 1, k d ) Z d continuous-space image 15 image array (B-spline coefficients) Compactly supported basis functions 13

14 SPLINE-BASED IMAGE PROCESSING Spline fitting: overview interpolation approximation Designing simple, efficient algorithms B-spline interpolation ast multi-scale processing Applications 14

15 Spline fitting: Overview s(x) = c[k]β n (x k) "B-spline representation: Spline fitting s(x) = c[k]β n (x k) Spline fitting Spline fitting s(x) = Spline fitting c[k]β s(x) n (x = Goal: Determine c[k] such that s(x) is a good representation k) c[k]β n n (x (x k) k) Discrete input f[k] c[k] such that s(x) x=k = Goal: Determine c[k] such that s(x) is a good representation of our signal Discrete input f[k] c[k] such that s(x) x=k = f[k] "Exact fit: interpolation (reversible) Analog input f(x) c[k] such that min f s 2 L s V 2 a Goal: Determine Goal: c[k] Determine such that Analog s(x) c[k] such is input a good that f(x) s(x) representation is isa a good c[k] of representation our signal such that of of our min our signal f s s V a Discrete input f[k] Discrete input c[k] f[k] Noisy discrete such c[k] input that f[k] s(x) such = x=k s(k) that = f[k] + s(x) n[k] x=k = f[k] such that Interpolation { such that Analog input f(x) Analog input algorithm c[k] f(x) such c[k] that min f s 2 such that Lmin s V 2 f f s s 2 L 2 } { + } L a s V 2 2 min + f[k] s(k) 2 + λ a a D m s(x) 2 dx f[k] Noisy discrete input f[k] s(k) = s(k) 2 s W + + λ2 Noisy discrete input f[k] m n[k] = s(k) D m + such s(x) n[k] 2 dx that such that { { } + } + f[k] min s(k) 2 f[k] + λ s(k) 2 D 2 m + λs(x) λ 2 dx D D m s(x) 2 dx 2 dx Noisy discrete input f[k] = s(k) + n[k] min s W "Regularized 2 m fit: smoothing splines min s W m 2 s W2 m 2 m "Least squares approximation: spline projectors "Generalized sampling theory "Multi-scale approximation (resizing, pyramids, wavelets)

16 s(x) = c[k]β n (x k) Regularized fit: Smoothing splines Spline fitt Goal: Determine c[k] such that s(x) is a good representation s(x) = Spline fitting c[k]β n of our signal (x k) "B-spline representation: s(x) = Discrete input f[k] c[k] c[k]β such n (x that s(x) k) x=k = f[k] Analog input f(x) c[k] = f, ϕ( /a Goal: k) Determine c[k] such such that that s(x) is a goo min f s 2 Goal: Determine c[k] such that s(x) is a good representation L s V 2 a Discrete, noisy input: Discrete input f[k] c[k] such Discrete input f[k] c[k] Smoothing such that s(x) x=k = Noisy discrete input f[k] = s(k) + n[k] Analog such input algorithm that f(x) c[k] such t { Analog input f(x) c[k] such that min f s } + s V Noisy discrete input f[k] = s(k) + a min f[k] s(k) 2 + λ D m s(x) n[k] Noisy discrete input f[k] { 2 dx s W2 m = s(k) + n[k] such that { { + min f[k] + s(k) 2 } } + λ D m + + s W min min f[k] min s(k) 2 + λ f[k] D 2s(k) m m s(x) λ dx D m s(x) 2 dx "Smoothing splines Theorem: Theorem: The Thesolution (among (amongall all functions) functions) of ofthe thesmoothing smoothingspline splineproblem problem s Ws(x) 2 m s W m 2 is a cardinal spline of degree 2m 1. In addition, 13 its coefficients c[k] = h λ f[k] is a cardinal spline of degree 2m 1. Its coefficients c[k] = h λ f[k] can be obtained can be obtained by suitable by digital suitable filtering digitalof filtering the input of the samples input samples f[k]. f[k]. "Special case: the draftman s spline spline spline The Theminimum minimumcurvature curvatureinterpolant interpolantisisobtained obtainedby bysetting m = 22and andλ λ It Itisisa acubic cubicspline!!

17 Smoothing splines: Example Spline s(x) = c[k]β n (x k) Goal: Determine c[k] such that s(x) is a Discrete input f[k] c[k] s Smoothing with increasing values of λ "Efficient implementation: separable, recursive filtering 17 1

18 Spline fitting s(x) = Spline fitting s(x) c[k]β = n Spline space at scale a (x c[k]β k) n Rescaled basis function: βa n ( (x k) V Dual B-spline: β a n a = s(x) = c[k]βa n (x (x) ak) such : c[k th Least squares fit: Multi-scale { approximation Goal: Determine Goal: c[k] Determine such that c[k] s(x) suchisthat a good s(x) representation is a good representation of our signal of our signal Spline space at scale a a = 1 a = 2 Discrete { input Discrete f[k] input f[k] c[k] c[k] such that s(x) such x=k that= s(x) f[k] x=k = f[k] V a Spline = s(x) space Spline = Rescaled } basis function: βa n (x) := β ( ) n x a Minimum error spline app at scale c[k]β space a n a(x at scale ak) : ac[k] l 2 Dual B-spline: { V a = s(x) } β 1 a n (x) 2 such 3 that 4 β a 5 n (x), βa n { V c[k]βa n a = s(x) = Continuous-space } input f(x) Rescaled basis function: β (x c[k]β ak) : c[k] l a n (x ak) 2 : c[k] l a n (x) := β ( ) n x a = 1 a = 2 2 such that min f s 2 a L Spline fitting Rescaled basis function: s(x) = βa n (x) := ( s V Dual B-spline: β ) x Rescaled basis function: n c[k]β n a (x) a := β ( ) a n (x) such that β a n (x), β n Minimum error spline approximation a at a (x ak) n = x δ[k] f[k] = 2s(k) + n[k] 4 Dual B-spline: β n (x k) a (x) such that β Continuous-space a input f(x) c[k] = f Minimum error spline approximation Dual B-spline: β n a n (x), βa n (x ak) = δ[k] a (x) at scale such that β a n (x), a βa n (x ak) = δ[k] Minimum error approximation Continuous-space at scale a Minimum Minimum error input f(x) spline approximation at scale a Goal: Determine error spline c[k] = c[k] approximation f, β n such that min f s 2 L a ( ak) s V 2 such that s(x) isat a scale good a representation of our signal Continuous-space such that Continuous-space min input f f(x) s 2 f[k] L input c[k] = f, Discrete input f[k] f(x) Sampling c[k] c[k] β a n = s(k) + ( = f, ak) β n[k] n such that s V 2 such that s(x) x=k = f[k] min f s 2 a ( ak) such that a min L f[k] s V = 2 f s 2 a s(k) + L s V n[k] 2 algorithm Analog input f(x) c[k] such that min f s 2 a L s V 2 a Spline space at scale a 19 Noisy discrete input f[k] = s(k) 19 + n[k] { f[k] 20 s(k) 2 + λ min s W m 2 + such that D m s(x) 2 dx } 20 a

19 Rescaled basis function: βa n (x) := ( ) n x Rescaled basis function: β a Dual B-spline: β a n (x) such that β a n (x) := β a n (x), βa n (x Dual B-spline: β a n (x) such that β a n ( Spline approximation: LS resizing a = 1 a = a 1 = 2 a = 2 Minimum error Minimum spline error approximation spline approximat at sca Continuous-space input f(x) c[k] = f, β a n Continuous-space input f(x) c[k such that min such that min f s 2 L s V 2 a f s 2 L s V 2 a f[k] = s(k) f[k] + n[k] = s(k) + n[k] Orthogonal projection onto V a (cubic spline) Orthogonal projection onto V a (cubic splin a = 1 10 a = 1 10 a 20 19

20 Even though splines are quite sophisticated mathematically,... they can be implemented simply and efficiently! 20

21 B-spline interpolation made simple B-spline interpolation Discrete B-spline kernels b n 1 [k] = β n (x) x=k n/2 z B-spline B1 n (z) interpolation = β n (k)z k k= n/2 f(x) x=k = b n ( ) 1 [k] = β n z (x) x=k B-spline c[l]β n B (k l) = 1 n (z) interpolation (b n = β n (k)z k 1 c) [k] c[k] = (b n 1 ) 1 f [k] k= n/2 f[k] = b n 1 [k] = β n z (x) c[l] β x=k n B (x l) (b n 1 ) 1 z 6 [k] z z 1 = (1 α) 2 x=k = (b1 n n (z) = β 1 c) [k] c[k] n (k)z = k B-spline B-spline interpolation interpolation (b n 1 ) 1 f[k] k= n/2 (1 αz)(1 αz 1 ) f[k] (b n ) 1 = c[l] β n z (x l) 6 [k] 1 αz 1 1 αz z z 1 = (1 α) 2 x=k = (b n n/2 1 c) [k] c[k] = (b n 1 ) 1 n/2 b n f[k] 1 [k] = β n z b n (x) x=k B1 n (z) = β n (k)z k 1 [k] = β n z (x) x=k B1 n (z) = β n (k)z k (1 αz)(1 αzk= n/2 1 ) (b n ) 1 z 6 [k] 1 αz 1 1 αz z z 1 = (1 α) 2 f[k] = k= n/2 c[l] β n (x l) (symmetric exponential) f[k] = c[l] β n (x x=k = (b (1 l) αz)(1 n 1 c) [k] αz 1 c[k] = (b ) n 1 ) 1 x=k = (b n 1 c) [k] c[k] = (b n 1 f[k] ) 1 f[k αz 1 Cascade (b 1 n 1 ) 1 of first z 6 αz[k] 7 z z 1 = (1 α) 2 (b n 1 ) 1 order recursive z filters 6 [k] z z(1 1 = (1 α) 2 αz)(1 αz)(1 1 ) αz 1 ) αz1 1 αz 1 1 αz1 7 αz B-spline interpolation: inverse filter solution Efficient recursive implementation 4/6 1/6 1/6 causal anti-causal 7 21

22 Generic C-code (splines of any degree n) Main recursion void ConvertToInterpolationCoefficients ( double c[ ], long DataLength, double z[ ], long NbPoles, double Tolerance) {double Lambda = 1.0; long n, k; if (DataLength == 1L) return; for (k = 0L; k < NbPoles; k++) Lambda = Lambda * (1.0 - z[k]) * ( / z[k]); for (n = 0L; n < DataLength; n++) c[n] *= Lambda; for (k = 0L; k < NbPoles; k++) { c[0] = InitialCausalCoefficient(c, DataLength, z[k], Tolerance); for (n = 1L; n < DataLength; n++) c[n] += z[k] * c[n - 1L]; c[datalength - 1L] = (z[k] / (z[k] * z[k] - 1.0)) * (z[k] * c[datalength - 2L] + c[datalength - 1L]); for (n = DataLength - 2L; 0 <= n; n--) c[n] = z[k] * (c[n + 1L]- c[n]); } } Initialization double InitialCausalCoefficient ( double c[ ], long DataLength, double z, double Tolerance) { double Sum, zn, z2n, iz; long n, Horizon; Horizon = (long)ceil(log(tolerance) / log(fabs(z))); if (DataLength < Horizon) Horizon = DataLength; zn = z; Sum = c[0]; for (n = 1L; n < Horizon; n++) {Sum += zn * c[n]; zn *= z;} return(sum); } 22

23 Spline interpolation Equivalent forms of spline representation s(x) = c[k]β n (x k) = ( ( s(k) (b n 1 ) 1 [k] ) β n (x k) s(k) (b n 1 ) 1 [k] ) β n (x k) c[k]β n (x k) = = s(k)ϕ n int(x k) = s(k)ϕ n int(x k) Cardinal (or fundamental) spline 1 ϕ n int(x) = (b n 1 ) 1 [k] β n (x k) ϕ n int(x) = (b n 1 ) 1 [k] β n (x k) inite-cost implementation of an infinite impulse response interpolator! 7 23

24 Limiting behavior Spline interpolator Impulse response requency response ϕ n int(x) H n (ω) = ( ) n+1 sin(ω/2) 1 ω/2 B1 n(ejω ) 0.5 The cardinal splineϕ interpolators n int(x) converge to the H n sinc-interpolator (ω) = (ideal filter) as the Asymptotic property degree goes to infinity: lim n ϕn int(x) = sinc(x), (in all L p degree -norms) goes to infinity: ( ω ) lim n Hn (ω) = rect 2π ( sin(ω/2) ω/2 ) n+1 1 B n 1 (ejω ) The cardinal spline interpolators converge to the sinc-interpolator (ideal filter) as the lim n ϕn int(x) = sinc(x), (in all L p -norms) ( ω ) lim n Hn (ω) = rect 2π (in all L p -norms ) (Aldroubi et al., Sig. Proc., 1992) 7 Includes Shannon s theory as a particular case! 7 24

25 Geometric transformation of images Geometric transformation Geometric transformation Geometric transformation 2D separable model k 1 +n+1 l 1 +n+1 k 1 +n+1 l 1 +n+1 f(x, y) = c[k, l] β n (x l) β n (y l) f(x, y) = c[k, l] β n (x k l) β n 1 +n+1 l 1 +n+1 k=k 1 l=l 1 (y l) k=k 1 l=l 1 f(x, y) = c[k, l] β n (x l) β n (y l) k=k 1 l=l 1 f[k, l] c[k, l] (x, y) l] c[k, l] (x, y) f[k, l] c[k, l] (x, y) 2D filtering (separable) 2D re-sampling Applications 5 5 zooming, rotation, re-sizing, re-formatting, warping 5 25

26 Geometric Cubic spline coefficients in 2D f(x, y) = Geometric transformation Digital filter (recursive, separable) k 1 +n+1 k=k 1 l 1 +n+1 l=l 1 c[k, l] β n (x l) β n (y l) f(x, y) = k 1 +n+1 k=k 1 f[k, l] c[k, l] l 1 +n l=l f[k, l] c[k, l] (x, y) Pixel values f[k, l] Pixel values f[k, l] B-spline coefficients c[k, l] B-spline coefficients c[k, l] 26

27 Interpolation benchmark Cumulative rotation experiment: the best algorithm wins! Bilinear Windowed-sinc Cubic spline Truncated sinc Cubic Cubic spline spline 27

28 High-quality image interpolation Splines: best cost-performance tradeoff 35 Lena 256 x 256, rotation 15 x 24, central 128 x 128 SNR (db) Bspline(6) Bspline(5) Bspline(4) Bspline(3) Bspline(2) Schaum(2) [1993] Meijering(7) [1999] Keys [1981] Meijering(5) [1999] Schaum(3) [1993] Dodgson [1997] Linear Sinc Hamming(4) Nearest-neighbor German [1997] Demo Execution time (s rot -1 ) Thévenaz et al., Handbook of Medical Image Processing,

29 ast multi-scale filtering Three alternative methods for the fast computation of: f(x) β n (x/a) Three O(N) complexity Three alternative alternative methods methods Three for Three the alternative for the alternative fast fast computation methods computation methods for of: the for f(x) of: the fast f(x) fast computation β n computation (x/a) β n (x/a) of: f(x) of: f(x) β n ( β O(N) complexity Three alternative Three alternative methods methods for the for the fast fast evaluation computationof of: f(x) β n (x/a) 1) Pyramid O(N) O(N) complexity or and complexity tree algorithms O(N) O(N) complexity complexity Pyramid and tree algorithms a = 2 i Pyramid Pyramid and and tree tree algorithms Pyramid algorithms Pyramid and and tree tree algorithms algorithms tion of: f(x) a = 2 i H2 n (z) a β= n (x/a) Three alternative a 2= i 2 i methods 2 for a the = a 2fast = i 2computation i of: f(x) β n (x/a) O(N) Recursive H HH n 2 n 2 n complexity (z) (z) filtering (iterative 2 2 (z) 2 2H 2 n H(z) 2 n (z) moving average) 2 2 a = m Z + s filtering (iterative moving average) a = m Z + m Pyramid [k] Recursive = Recursive s m [k 1] filtering + filtering f[k] (iterative Recursive (iterative f[k Recursive moving m] filtering moving filtering average) (iterative average) (iterative a = moving a m= moving m Z average) + Z + a = a m= Binomial and filter tree algorithms (Unser et al., IEEE Trans. PAMI, 1993) m a = s2 m i [k] s m [k] = s= m [k s m [k 1] + 1] sf[k] m + sf[k] m = f[k s = m f[k s m m] [k1] m] + 1] f[k] + f[k] f[k f[k m] m] Differential: s m [k] integration = s m [k and 1] + weighted f[k] f[k differences m] 2) Recursive a R Differential: filtering + Differential: integration (iterated integration and moving and weightedaverage) H differences 2 n (z) 2 differences Differential: integration and weighted differences Differential: Differential: integration integration and and weighted weighted differences differences verage) a = am Integral ar Z Recursive (or + primitive): R + a + filtering R + (iterative (x) a = a R x + moving R + f(t)dt = D 1 {f(x)} Integral inite difference Integral (or primitive): (or with primitive): (x) step a: Integral (or primitive): (x) = a (x) {f(x} = average) a = x Integral = x Integral (or primitive): (or= f(t)dt primitive): f(x) f(t)dt x = f(t)dt f(x (x) D 1 = (x) D 1 = {f(x)} m1 1Z x a) = {f(x)} x s f(t)dt = f(t)dt = m [k] = s m [k 1] + f[k] f[k m] D 1 D 1 {f(x)} = 1 D 1 {f(x)} 1 {f(x)} differences inite f(x) Differential: inite β+(x/a) 0 difference difference integration with = (x) with step inite step and a: inite (x a: difference weighted a difference {f(x} a) a {f(x} with = = differences 1 with f(x) step ad = 1 f(x) step a: {f(x)} f(x a: a {f(x} a a) {f(x} a) = f(x) = f(x) f(x f(x a) a) f(x) inite difference with step a: a {f(x} = f(x) f(x a) f(x) β +(x/a) 0 β+(x/a) 0 = = f(x) f(x) β +(x/a) 0 β +(x/a) 0 = = 1 ad (x) 1 (Unser et al., IEEE Trans. 1 ad {f(x)} 1 {f(x)} a) = a) = 1 ad 1 Sig. Proc, 1994) a R + 1 ad {f(x)} 1 29 {f(x)}

30 Recursive filtering (iterative moving average) a = m Z Pyramid and tree algorithms s Pyramid ast m [k] = s and i multi-scale m [k tree 1] + algorithms f[k] f[k H filtering 2 n (z) m] 2 (Cont d) a = Differential: 2 i integration and weighted Recursive differences filtering (iterative moving average) a = 2 n (z) H a 2 n (z) R + 2 s m [k] = s m [k 1] + f[k] f[k m] Recursive filtering (iterative moving average) Recursive filtering + Integral (or primitive): (iterative (x) = xmoving Differential: mplexity m [k] m [k 1] f[k] f[k f(t)dt average) m] = D 1 integration {f(x)} a = m and Z + weighted differences 3) sdifferential inite m [k] = s difference m [k approach 1] + f[k] with step a: f[k a {f(x} a R m] + = f(x) f(x a) id and tree Differential: algorithms integration and weighted differences f(x) β+(x/a) 0 = (x) (xintegral a) = (or primitive): 1 ad 1 {f(x)} (x) = x Differential: integration and weighted differences + f(t)dt = D 1 {f(x) a R + Integral (or primitive): (x) inite difference with step a: x a {f(x} = f(x) f(x a 2 f(t)dt 17 D 1 {f(x)} Integral (or primitive): (x) = x f(x) f(t)dt β+(x/a) 0 = D 1 = {f(x)} (x a) = 1 ad 1 {f(x)} inite difference with step a: a {f(x} = f(x) f(x a) inite-difference with step a: a {f(x)} = f(x) f(x a) f(x) 0 (x) (x a) 1 ad 1 {f(x)} 17 f(x) β+(x/a) 0 = (x) (x a) = 1 ad 1 {f(x)} ernative Challenge: methods foro(n) the fast evaluation computation of: f(x) β n (x/a) sive filtering (iterative moving average) a = m Z + s m [k 1] + f[k] f[k m] Generalization 17 ntial: integration and weighted differences 26 f(x) β n +(x/a) = 1 a n+1 n a f(x) Principle: β The integral of a spline of degree n is a spline of degree n + 1. or primitive): +(x/a) n = 1 a n+1 n a D (n+1) {f(x)} (x) = x f(t)dt = D 1 {f(x)} erence with step a: a {f(x} = f(x) f(x a) D (n+1) {f(x)} multiple integration Running sums Principle: The integral of a spline of degree n is a spline of degree n + 1. weighted differences Interpolation + filtering 30

31 Application: Image resizing "Resizing algorithm "Interpolation "n=1 "scaling= 70% Text SNR=22.94 db Spline projector

32 Application: Image resizing (LS) "Resizing algorithm "Orthogonal projector "n=1 "scaling= 70% Text SNR= db db (Munoz et al., IEEE Trans. Imag. Proc, 2001)

33 Splines: More applications Sampling and interpolation Interpolation, re-sampling, grid conversion Image reconstruction Geometric correction eature extraction Contours, ridges Differential geometry Image pyramids Shape and active contour models Image matching Stereo Image registration (multi-modal, rigid body or elastic) Motion analysis Optical flow 33

34 SPLINES: URTHER PERSPECTIVES ractional B-splines ional B-splines ractional B-splines β 0 +(x) = + x 0 + M β 0 +(x) = + x e jω jω M 1 e jω jω β+(x) α = α+1 + x α + Γ(α + 1) ed power function: x α + = β+(x) α = α+1 + x α + Γ(α + 1) x α, x 0 One-sided power function: x α + = 0, x < 0 ( 1 e jω jω x α, x 0 0, x < 0 ) α+1 ( 1 e jω jω ) α+1 erties Properties α (x k) } is a valid Riez basis for α < 1 2 volution property: β α 1 { β α + (x k) } is a valid Riez basis for α < 1 2 Convolution property: β α 1 + β α 2 + = β α 1+α β α 2 + = β α 1+α 2 + (Unser & Blu, SIAM Rev, 2000)

35 URTHER PERSPECTIVES Splines and wavelet theory (Unser and Blu, IEEE-SP, 2003) actorization of any scaling function (or wavelet) of order γ: ϕ(x) = ( β γ 1 + ϕ 0 ) (x) where β+ α with α = γ 1: regular, B-spline part actorization of any scaling distribution function (or wavelet) of order γ: B-spline: ϕ 0 explains S :( irregular. all distributional ) part fundamental ϕ(x) properties = β γ 1 + ϕ 0 (x) ractional splines are the optimal scaling functions function for the estimation regular part of fractal irregular processes part with where 1/ωβ + 2H+1 α withspectral α = γ decay 1: regular, (fractional B-spline Brownian partmotion) ϕ 0 S : irregular. distributional part Splines and fractals Splines are the optimal functions for the estimation of fractal processes with 1/ω 2H+1 spectral decay (fractional Brownian motion) 35

36 Splines: The key to wavelet theory Sobolev smoothness s ϕ L 2 B-spline factorization: γ s Approximation order: Multi-scale differentiator γ ϕ = β 1 + ϕ 0 f P a f L2 = O(a γ ) ψ ˆ (ω) ( jω) γ, ω 0 general case: compact support: n < γ n +1 γ = n +1 (Strang-ix, 1971) Polynomial reproduction degree: n = γ 1 c Vanishing moments: x p ψ (x)dx = 0, p = 0,, n (Unser and Blu, IEEE-SP, 2003) 36

37 CONCLUSION Distinctive features of splines Simple to manipulate Smooth and well-behaved Excellent approximation properties Multiresolution properties undamental nature (Green functions of derivative operators) Splines and image processing A story of avoidance and, more recently, love Best cost/performance tradeoff Many applications Unifying signal processing formulation Tools: digital filters, convolution operators Efficient recursive-filtering solutions lexibility: piecewise-constant to bandlimited 37

38 Acknowledgments Many thanks to Dr. Thierry Blu Prof. Akram Aldroubi Prof. Murray Eden Dr. Philippe Thévenaz Annette Unser, Artist + many other researchers and graduate students 38

39 The end: Thank you! Spline tutorial M. Unser, "Splines: A Perfect it for Signal and Image Processing," IEEE Signal Processing Magazine, vol. 16, no. 6, pp , Spline and wavelets M. Unser, T. Blu, "Wavelet Theory Demystified," IEEE Trans. on Signal Processing, vol. 51, no. 2, pp , Smoothing splines and stochastic formulation M. Unser, T. Blu, "Generalized Smoothing Splines and the Optimal Discretization of the Wiener ilter," IEEE Trans. Signal Processing, vol. 53, no. 6, pp , June Preprints and demos: 39