Patch Complexity, Finite Pixel Correlations and Optimal Denoising

Transcription

1 Patch Complexity, Finite Pixel Correlations an Optimal Denoising Anat Levin Boaz Naler Freo Duran William T. Freeman Weizmann Institute MIT CSAIL Abstract. Image restoration tasks are ill-pose problems, typically solve with priors. Since the optimal prior is the exact unknown ensity of natural images, actual priors are only approximate an typically restricte to small patches. This raises several questions: How much may we hope to improve current restoration results with future sophisticate algorithms? An more funamentally, even with perfect knowlege of natural image statistics, what is the inherent ambiguity of the problem? In aition, since most current methos are limite to finite support patches or kernels, what is the relation between the patch complexity of natural images, patch size, an restoration errors? Focusing on image enoising, we make several contributions. First, in light of computational constraints, we stuy the relation between enoising gain an sample size requirements in a non parametric approach. We present a law of iminishing return, namely that with increasing patch size, rare patches not only require a much larger ataset, but also gain little from it. This result suggests novel aaptive variable-size patch schemes for enoising. Secon, we stuy absolute enoising limits, regarless of the algorithm use, an the converge rate to them as a function of patch size. Scale invariance of natural images plays a key role here an implies both a strictly positive lower boun on enoising an a power law convergence. Extrapolating this parametric law gives a ballpark estimate of the best achievable enoising, suggesting that some improvement, although moest, is still possible. Introuction Characterizing the properties of natural images is critical for computer an human vision [0, 3,, 8, 6, 6]. In particular, low level vision tasks such as enoising, super resolution, eblurring an completion, are funamentally ill-pose since an infinite number of images x can explain an observe egrae image y. Image priors are crucial in reucing this ambiguity, as even approximate knowlege of the probability p(x) of natural images can rule out unlikely solutions. This raises several funamental questions. First, at the most basic level, what is the inherent ambiguity of low level image restoration problems? i.e., can they be solve with zero error given perfect knowlege of the ensity p(x)? More practically, how much can we hope to improve current restoration results with future avances in algorithms an image priors? Clearly, more accurate priors improve restoration results. However, while most image priors (parametric, non-parametric, learning-base) [, 4,, 8, 6] as well as stuies

2 on image statistics [3, 6] are restricte to local image patches or kernels, little is known about their epenence on patch size. Hence another question of practical importance is the following: What is the potential restoration gain from an increase in patch size? an, how is it relate to the patch complexity of natural images, namely their geometry, ensity an internal correlations. In this paper we stuy these questions in the context of the simplest restoration task: image enoising [0,, 5, 0, 8, 6, 9, 6]. We buil on prior attempts to stuy the limits of natural image enoising [9, 3, 7]. In particular, on the non-parametric approach of [4], which estimate the optimal error for the class of patch base algorithms that enoise each pixel using only a finite support of noisy pixels aroun it. A major limitation of [4], is that computational constraints restricte it to relatively small patches. Thus, [4] was unable to preict the best achievable enoising of algorithms that are allowe to utilize the entire image support. In other wors, an absolute PSNR boun, inepenent of patch size restrictions, is still unknown. We make several theoretical contributions with practical implications, towars answering these questions. First we consier non-parametric enoising with a finite external atabase an finite patch size. We stuy the epenence of enoising error on patch size. Our main result is a law of iminishing return: when the winow size is increase, the ifficulty of fining enough training ata for an input noisy patch irectly correlates with iminishing returns in enoising performance. That is, not only is it easier to increase winow size for smooth patches, they also benefit more from such an increase. In contrast, texture regions require a significantly larger sample size to increase the patch size, while gaining very little from such an increase. From a practical viewpoint, this analysis suggests an aaptive strategy where each pixel is enoise with a variable winow size that epens on its local patch complexity. Next, we put computational issues asie, an stuy the funamental limit of enoising, with an infinite winow size an a perfectly known p(x) (i.e., an infinite training atabase). Uner a simplifie image formation moel we stuy the following question: What is the absolute lower boun on enoising error, an how fast o we converge to it, as a function of winow size. We show that the scale invariance of natural images plays a key role an yiels a power law convergence curve. Remarkably, espite the moel s simplicity, its preictions agree well with empirical observations. Extrapolating this parametric law provies a ballpark preiction on the best possible enoising, suggesting that current algorithms may still be improve by about 0.5 B. Optimal Mean Square Error Denoising In image enoising, given a noisy version y = x + n of a clean image x, corrupte by aitive noise n, the aim is to estimate a cleaner version ˆx. The common quality measure of enoising algorithms is their mean square error, average over all possible clean an noisy x, y pairs, where x is sample from the ensity p(x) of natural images Z Z MSE = E[ ˆx x ] = p(x) p(y x) x ˆx yx ()

3 It is known, e.g. [4], that for a single pixel of interest x c the estimator minimizing Eq. () is the conitional mean: p(y x) ˆx c = µ(y) = E[x c y] = p(y) p(x)x cx. () Inserting Eq. () into Eq. () yiels that the minimum mean square error (MMSE) per pixel is the conitional variance Z Z MMSE = E y[v[x c y]] = p(y) p(x y)(x c µ(y)) xy. (3) The MMSE measures the inherent ambiguity of the enoising problem an the statistics of natural images, as any natural image x within the noise level of y may have generate y. Since Eq. () epens on the exact unknown ensity p(x) of natural images (with full image support), it is unfortunately not possible to compute. Nonetheless, by efinition it is the theoretically optimal enoising algorithm, an in particular outperforms all other algorithms, even those that etect the class of a picture an then use class-specific priors [3], or those which leverage internal patch repetition [5, 5]. That sai, such approaches can yiel significant practical benefits when using a finite ata. Finally, note that the ensity p(x) plays a ual role. Accoring to Eq. (), it is neee for evaluating any enoising algorithm, since the MSE is the average over natural images. Aitionally, it etermines the optimal estimator µ(y) in Eq. (). Finite support: First, we consier algorithms that only use information in a winow of noisy pixels aroun the pixel to be enoise. When neee, we enote by x w, y w the restriction of the clean an noisy images to a -pixel winow an by x c, y c the pixel of interest, usually the central one with c =. As in Eq. (3), the optimal MMSE of any enoising algorithm restricte to a pixels support is also the conitional variance, but compute over the space of natural patches of size rather than on full-size images. By efinition, the optimal enoising error may only ecrease with winow size, since the best algorithm seeing + pixels can ignore the last pixel an provie the answer of the pixels algorithm. This raises two critical questions: how oes MMSE ecrease with, an what is MMSE, namely the best achievable enoising error of any algorithm (not necessarily patch base)? Non-Parametric approach with a finite training set: The challenge in evaluating MMSE is that the ensity p(x) of natural images is unknown. To bypass it, a nonparametric stuy of MMSE for small values of was mae in [4], by approximating Eq. () with a iscrete sum over a large ataset of clean -imensional patches {x i } N i=. P ˆµ (y) = N N i p(yw xi,w )xi,c Pi p(yw xi,w ) (4) where, for ii zero-mean aitive Gaussian noise n with variance σ, p(y w x w ) = xw yw (πσ e σ ). (5) / 3

4 An interesting conclusion of [4] was that for small patches or high noise levels, existing enoising algorithms are close to the optimal MMSE. For Eq. (4) to be an accurate estimate of µ (y), the given ataset must contain many clean patches at istance (σ ) / from y w, which is the expecte istance between the original an noisy patches, E[ x w y w ] = σ. As a result, non-parametric enoising requires a larger training set at low noise levels σ where the istance σ is smaller, or at larger patch sizes where clean patch samples are sprea further apart. This curse of imensionality restricte [4] to small values of. In contrast, in this paper we are intereste in the best achievable enoising of any algorithm, without restrictions on support size, namely MMSE. We thus generalize [4] by stuying how MMSE ecreases as a function of, an as a result provie a novel preiction of MMSE (see Section 4). Note that MMSE correspons to an infinite atabase of all clean images, which in particular also inclues the original image x. However, this oes not imply that MMSE = 0, since this atabase also inclues many slight variants of x, with small spatial shifts or illumination changes. Any of these variants may have generate the noisy image y, making it is impossible to ientify the correct one with zero error. 3 Patch Size, Complexity an PSNR Gain Increasing the winow size provies a more accurate prior as it consiers the information of istant pixels on the pixel of interest. However, in a non-parametric approach, this requires a much larger training set an it is unclear how substantial the PSNR gain might be. This section shows that this traeoff epens on patch complexity, an presents a law of iminishing return: patches that require a large increase in atabase size also benefit little from a larger winow. This gain is governe by the statistical epenency of peripheral pixels an the central one: weakly correlate pixels provie little information while leaing to a much larger sprea in patch space, an thus require a significantly larger training ata. 3. Empirical stuy To unerstan the epenence of PSNR on winow size, we present an empirical stuy with M = 0 4 clean an noisy pairs {(x j, y j )} M j= an N = 08 samples taken from the LabelMe ataset, as in [4]. We compute the non-parametric mean (Eq. (4)) at varying winow sizes. For each noisy patch we etermine the largest at which estimation is still reliable by comparing the results with ifferent clean subsets. We ivie the N clean samples into 0 groups, compute the non-parametric estimator ˆµ (y j) on each group separately, an check if the variance of these 0 estimators is much smaller than σ. For small, samples are ense enough an all these estimators provie consistent results. For large, sample ensity is insufficient, an each estimator gives a very ifferent result. 4

5 35 G 7 G 30 PSNR G 4 G 7 G G 5 G 30 5 G G 69 G 3 0 G 49 G 97 5 G 40 G Number of pixels Number of pixels D signals, σ = 0 D signals, σ = 50 PSNR Fig. : For patch groups G l of varying complexity, we present PSNR vs. number of pixels in winow w, where =,..., l. Higher curves correspon to smooth regions, which flatten at larger patch imensions. Texture regions correspon to lower curves which not only run out of samples sooner, but also their curves flatten earlier. y x x y y y y x 3 x x x x x 3 y unconitional y y = 0 y y = 40 y y = 80 (a) (b) (c) () (e) x 3 y x 3 y x 3 Fig. : (a) A clean an noisy D signal. (b) Unconitional joint istribution p(x, x 3). (c-e) Conitional istributions p(x, x 3 y, y ) for a few observe graients y y, at noise s.t.. σ=0. The original epenence of x, x 3 is broken if a high graient is observe, y y σ. We ivie the M test patches into groups G l base on the largest winow size l at which the estimate is still reliable. For each group, Fig. isplays the empirical PSNR average over the group s patches as a function of winow size, for =,...,l (that is, up to the maximal winow size = l at which estimation is reliable), where: ( ) PSNR(G l w ) = 0 log 0 G l j G l (x j,c ˆµ (y j )) We further compute for each group its mean graient magnitue, y wl, an observe that groups with smaller support size l, which run more quickly out of training ata, inclue mostly patches with large graients (texture). These patches correspon to PSNR curves that are lower an also flatten earlier (Fig. ). In contrast, smoother patches are in groups that run out of examples later (higher l) an also gain more from an increase in patch with: the higher curves in Fig. flatten later. The ata in Fig. emonstrates an important principle: When an increase in patch with requires many more training samples, the performance gain ue to these aitional samples is relatively small. To unerstan the relation between patch complexity, enoising gain, an require number of samples, we show that the statistical epenency between ajacent pixels is broken when large graients are observe. We sample rows of 3 consecutive pixels from 5

6 x 4 3 (y,y ) x 4 3 (y,y ) p.5 p(x y )=p(x y,y ) p.5 p(x y ) p(x y,y ) x x 0 0 x x 3 4 (a) Inepenent (b) Fully Depenent (c) Inepenent () Fully Depenent Samples from joint istributions Conitional istributions Fig. 3: A toy example of D sample ensities. clean x an noisy y natural images (Fig. (a)), iscretize them into 00 intensity bins, an estimate the conitional probability p(x, x 3 y, y ) by counting occurrences in each bin. When the graient y y is high with respect to the noise level, x, x 3 are approximately inepenent, p(x = i, x 3 = j y y σ) p (i)p 3 (j), see Fig. (e). In contrast, small graients on t break the epenency, an we observe a much more elongate structure, see Fig. (c,). For reference, Fig. (b) shows the unconitional joint istribution p(x, x 3 ), without seeing any y. Its iagonal structure implies that while the pixels (x, x 3 ) are by efault epenent, the epenency is broken in the presence of a strong ege between them. From a practical perspective, if y y σ, aing the pixel y 3 oes not contribute much to the estimation of x. If the graient y y is small there is still epenency between x 3 an x, so aing y 3 oes further reuce the reconstruction error. A simple explanation for this phenomenon is to think of ajacent objects in an image. As objects can have inepenent colors, the color of one object tells us nothing about its neighbor on the other sie of the ege. 3. Theoretical Analysis Motivate by Fig. an Fig., we stuy the implications of partial statistical epenence between pixels, both on the performance gain expecte by increasing the winow size, an on the requirements on sample size. D Gaussian case: To gain intuition, we first consier a trivial scenario where patch size is increase from to pixels an istributions are Gaussians. In Fig. 3(a), x an x are inepenent, while in Fig. 3(b) they are fully epenent an x = x. Both cases have the same marginal istribution p(x ) with equal enoising performance for a -pixel winow. We raw N = 00 samples from p(x, x ) an see how many of them fall within a raius σ aroun a noisy observation (y, y ). In the uncorrelate case (Fig. 3(a)), the samples are sprea in the D plane an therefore only a small portion of them fall near (y, y ). In the secon case, since the samples are concentrate in a significantly smaller region (a -D line), there are many more samples near (y, y ). Hence, in the fully correlate case a non parametric estimator requires a significantly smaller ataset to have a sufficient number of clean samples in the vicinity of y. 6

7 To stuy the accuracy of restoration, Fig. 3(c,) shows the conitional istributions p(x y, y ). When x, x are inepenent, increasing winow size to take y into account provies no information about x, an p(x y ) = p(x y, y ). Worse, enoising performance ecreases when the winow size is increase because we now have fewer training patches insie the relevant neighborhoo. In contrast, in the fully correlate case, aing y provies valuable information about x, an the variance of p(x y, y ) is half of the variance given y alone. This illustrates how high correlation between pixels yiels a significant ecrease in error without requiring a large increase in sample size. Conversely, weak correlation gives only limite gain while requiring a large increase in training ata. General erivation: We exten our D analysis to imensions. The following claim, prove in [5], provies the leaing error term of the non-parametric estimator ˆµ (y) of Eq.(4) as a function of training set size N an winow size. It is similar to results in the statistics literature on the MSE of the Naaraya-Watson estimator. Claim. Asymptotically, as N, the expecte non-parametric MSE with a winow of size pixels is E N [MSE (y)] = MMSE (y) + N V (y) + o ( ) N (6) V V[x y w ] Φ σ, (7) with V[x y w ] the conitional variance of the central pixel x given a winow w from y, an Φ is the eterminant of the local covariance matrix of p(y), Φ = log p(y w ) y w. (8) The expecte error is the sum of the funamental limit MMSE (y) an a variance term that accounts for the finite number of samples N in the ataset. As in Monte-Carlo sampling, it ecreases as N. When winow size increases, MMSE (y) ecreases, but the variance V (y) might increase. The tension between these two terms etermines whether for a constant training size N increasing winow size is beneficial. The variance V is proportional to the volume of p(y w ), as measure by the eterminant Φ of the local covariance matrix. When the volume of the istribution is larger, the N samples are sprea over a wier area an there are fewer clean patches near each noisy patch y. This is precisely the ifference between Fig. 3(a) an Fig. 3(b). For the error to be close to the optimal MMSE, the term V /N in Eq. (6) must be small. Eq. (7) shows that V epens on the volume Φ an we expect this term to grow with imension, thus requiring many more samples N. Both our empirical ata an our D analysis show that the require increase in sample size is a function of the statistical epenency of the central pixel with the ae one. To unerstan the require increase in training size N when winow size is increase by one pixel from to, we analyze the ratio of variances V /V. Let g (y) be 7

8 the gain in performance (for an infinite ataset), which accoring to Eq. (3) is given by: g (y) = MMSE (y) MMSE (y) = V[x y,... y ] V[x y,... y ] We also enote by g (y) the ieal gain if x an x were perfectly correlate, i.e. r = cor(x, x y,...,y ) =. Assuming for simplicity a Gaussian istribution, the following claim shows that when MMSE (y) is most improve, sampling is not harer since the volume an variance V o not grow. Claim. Let p(y) be Gaussian. When increasing the patch size from to, the variance ratio an the performance gain of the estimators are relate by: (9) V V = g g. (0) That is, the ratio of variances equals the ratio of optimal enoising gain to the achievable gain. When x, x are perfectly correlate, g = g, we get V /V =, an a larger winow gives improve restoration results without increasing the require ataset size. In contrast, if x, x are weakly correlate, increasing winow size requires a bigger ataset to keep V /N small, an yet the PSNR gain is small. Proof. Let C be the covariance of x, x given y,...,y (before seeing y ) «c c C = Cov(x, x y,... y ) = () c c an let r = c / c c be the correlation between x, x. Uner the Gaussian assumption, upon observing y, the marginal variance of x ecreases from c to the following expression (see Eq..73 in [4]), «V[x y,..., y ] = c c c + σ = c c /c c ( r ) + σ = c c + σ. () c + σ Hence the contribution to performance gain of the aitional pixel y is g = V[x y,... y ] V[x y,... y ] = c + σ c ( r ) + σ. (3) When r =, the largest possible gain from y is g = (c + σ )/σ. The ratio of best possible gain to achieve gain is g = c( r ) + σ. (4) g σ Next, let us compute the ratio V /V. For Gaussian istributions, accoring to Eq..8 in [4], the conitional variance of y given y,..., y is inepenent of the specific observe values. Further, since p(y,..., y ) = p(y,..., y )p(y y,...y ), we obtain that Φ = V(y y,... y ) Φ (5) 8

9 σ Optimal Fixe Aaptive BM3D Table : Aaptive an fixe winow enoising results in PSNR. This implies that V = V(y y,... y ) V V[x y,... y ] σ V[x y,... y ] Next, since y = x + n with n N(0, σ ) inepenent of y,..., y, then V(y y,...y ) = c + σ. Thus, V = c + σ c ( r ) + σ V σ c + σ = g g. (6) In [5] we compute V an g for several cases. For a signal whose pixels are all inepenent with equal variance, V an the require number of samples N both grow exponentially with imension. In contrast, for a fully correlate signal, V is constant. 3.3 Aaptive Denoising Our analysis suggests that patch base enoising can be improve mostly in flat areas an less in texture ones. In [5] we show that this is implicit in several recent works, consistent with [7]. This motivates an aaptive enoising scheme [] where each pixel is enoise with a variable patch size that epens on its local image complexity. To test this iea, we evise the following scheme. Given a noisy image, we enoise each pixel using several patch withs an multiple isjoint clean samples. As before, we compute the variance of all these ifferent estimates, an select the largest with for which the variance is still below a threshol. Table compares the PSNR of this aaptive scheme to fixe winow size non-parametric enoising using the optimal winow size at each noise level, an to BM3D [8], a state-of-the-art algorithm. We use M = 000 test pixels an N = clean samples. At all consiere noise levels, the aaptive approach significantly improves the fixe patch approach, by about B. At low noise levels, sample size N is too small, an aaptive enoising is worse than BM3D. At higher noise levels it increasingly outperforms BM3D. Fig. 4 visualizes the ifference between the aaptive an fixe patch size approaches, at noise level σ = 50. When patch size is small, noise resiuals are highly visible in The reason is that at this finite N, with σ = 0 our non-parametric approach uses 5 5 patches at texture regions. In contrast, BM3D uses 8 8 ones, with aitional algorithmic operations which allow it to better generalize from a limite number of samples. 9

10 (a)original (b)noisy input (c)aaptive ()Fixe k = 5 (e)fixe k = 6 (f)fixe k = 0 Fig. 4: Visual comparison of aaptive vs. fixe patch size non parametric enoising (optimal fixe size results obtaine with k = 6). A fixe patch has noise resiuals either in flat areas(,e), or in texture areas(f). the flat regions. With a large patch size, one cannot fin goo matches in the texture regions, an as a result noise is visible aroun eges. Both eges an flat regions are hanle properly by the aaptive approach. Moreover, uner perceptual error metrics such as SSIM [4], ecreasing the error in the smooth regions is more important, thus unerscoring the potential benefits of an aaptive approach. Note that this aaptive non-parametric enoising is not a practical algorithm, as Fig. 4 require several ays of computation. Nonetheless, these results suggest that aaptive versions to existing enoising algorithms such as [0, 8, 6, 9, 6] an other low-level vision tasks are a promising irection for future research. 4 The Convergence an Limits of Optimal Denoising In this section, we put computational an atabase size issues asie, an stuy the behavior of optimal enoising error as winow size increases to infinity. Fig. shows that optimal enoising yiels a iminishing return beyon a winow size that varies with patches. Moreover, patches that plateau at larger winow sizes also reach a higher PSNR. Fig. shows that strong eges break statistical correlation between pixels. Combining the two suggests that each image pixel has a finite compact region of informative neighboring pixels. Intuitively, the size istribution of these regions must irectly impact both enoising error vs. winow size an its limit with an infinite winow. We make two contributions towars eluciating this question. First we show that a combination of the simplifie ea leaves image formation moel, together with scale invariance of natural images implies both a power-law convergence, MMSE e+c/, as well as a strictly positive lower boun on the optimal enoising with infinite winow, MMSE =e>0. Next, we present empirical results showing that espite the simplicity of this moel, its conclusions match well the behavior of real images. 4. Scale-invariance an Denoising Convergence We consier a ea leaves image formation moel, e.g. [], whereby an image is a ranom collection of piecewise constant segments, whose size is rawn from a scale- 0

11 Inverse Histogram x 0 3 Data 0.8 Polynomial fit Segment size T = 5, h(). Inverse Histogram 5 x 0 4 Data 4 Polynomial fit Segment size T = 0, h() Fig. 5: Inverse histograms of segment lengths follow a scale invariant istribution. invariant istribution an whose intensity is rawn i.i.. from a uniform istribution. This yiels perfect correlation between pixels in the same region, as in Fig. 3(b). To further simplify the analysis, we conservatively assume an ege oracle which gives the exact locations of eges in the image. The optimal enoising is then to average all observations in a segment. For a pixel belonging to segment of size s pixels, the MMSE is σ /s. Overall the expecte reconstruction error with infinite-size winows is Z MMSE = p(s) σ s (7) s where p(s) is the probability that a pixel belongs to a segment with s pixels. The optimal error is strictly larger than zero if the probability of finite segments is larger than zero. Without the ege-oracle, the error is even higher. Scale invariance: Scale invariance is a funamental property of natural images. As shown in numerous stuies, own-sampling natural images retains many of their statistical properties, from graient istributions to segment (region) areas, e.g. [, 3,, 7, ]. A simple argument [, 5] shows that scale-invariance implies that the probability that a ranom image pixel belongs to a segment of size s is of the form p(s) /s. In a Markov moel, in contrast, p(s) ecays exponentially fast with s [, 5]. To get a sense of the empirical size istribution of nearly-constant-intensity regions in natural images, we perform a simple experiment inspire by []. For a ranom set of pixels {x i }, we compute the size (i) of the connecte region whose pixel values iffer from x i by at most a threshol T : (i) = #{x j x j x i T }. The empirical histogram h() of region sizes follows a power law behavior h() α with α, as shown in Fig. 5(a,b), which plots /h(). Optimal enoising as a function of winow size: We now compute the optimal enoising for the ea leaves moel with the scale invariance property. Since /s is not integrable, scale invariance cannot hol at infinitely large scales. Assuming it hols up to a maximal size D, gives the normalize probability p D(s) = s R D s s = ln D s. (8)

12 PSNR Data Power law fit Exponential fit PSNR Data Power law fit Exponential fit log(mmse e) log(mmse e) log() (a) σ = 50 (b) σ = 00 (c) Log plot () Log log plot Fig. 6: PSNR vs. patch imension. A power law fits the ata well, whereas an exponential law fits poorly. Panels (c) an () show log(mmse e) v.s. or log(). An exponential law shoul be linear in the first plot, a power law linear in the secon. We compute the optimal error with a winow of size D pixels. Given the ege oracle, every segment of size s attains its optimal enoising error of σ /s, whereas if s > we obtain only σ /. Splitting the integral in (7) into these two cases gives MMSE = D σ s p σ D(s)s + p D(s)s (9) = MMSE D + σ ( ln + ln D ) + σ D ln D MMSE D + σ For this moel, MMSE = MMSE D. Thus, the ea leaves moel with scale invariance property implies a power law / convergence to a strictly positive MMSE. 4. Empirical valiation an optimal PSNR While ea leaves is clearly an over-simplifie moel, it captures the salient properties of natural images. Even though real images are not mae of piecewise constant segments, the results of Sec. 3, an Fig. 5 suggest that each image pixel has a finite informative region, whose pixel values are most relevant for enoising it. While for real images, correlations may not be perfect insie this region an might not fully rop to zero outsie it, we now show that empirically, optimal enoising in natural images inee follows a power law similar to that of the ea-leaves moel. To this en, we apply the metho of [4] an compute the optimal patch base MMSE for several small winow sizes. Fig. 6(a-b) show that consistent with the ea leaves moel, we obtain an excellent fit to a power law MMSE = e + c with α. In α contrast, we get a poor fit to an exponential law, MMSE = e + cr, implie by the common Markovian assumption [, 5]. In aition, Fig. 6(c,) show log an log-log plots of (MMSE e), with the best fitte e in each case. The linear behavior in the log-log plot (Fig. 6()) further supports the power law. Fitting etails appear in [5]. Preicting Optimal PSNR: For small winow sizes, using a large atabase an Eq. (4), we can estimate the optimal patch-base enoising MMSE. Fig. 6 shows that the curve of MMSE is accurately fitte by a power law MMSE = e + c/ α, with α. Extrapolating this curve, we can preict the value of MMSE, which is the best possible

13 σ Extrapolate boun (PSNR ) KSVD [0] BM3D [8] EPLL [6] Table : Extrapolate optimal enoising in PSNR, an the results of recent algorithms. A moest room for improvement exists. error of any enoising algorithm (not necessarily patch base). Since the power law is only approximate, this extrapolation shoul be taken with a grain of salt. Nonetheless, it gives an interesting ballpark estimate on the amount of further achievable gain by any future algorithmic improvements. Table compares the PSNR of existing algorithms to the preicte PSNR, over M = 0, 000 patches using the power law fit base on N = 0 8 clean samples 3. The comparison suggests that epening on noise level σ, current methos may still be improve by 0.5 B. While the extrapolate value may not be exact, our analysis oes suggest that there are inherent limits impose by the statistics of natural images, which cannot be broken, no matter how sophisticate future enoising algorithms will be. 5 Discussion In this paper we stuie both computational an information aspects of image enoising. Our analysis reveale an intimate relation between enoising performance an the scale invariance of natural image statistics. Yet, only few approaches account for it [0]. Our finings suggest that scale invariance can be an important cue to explore in the evelopment of future natural image priors. In aition, aaptive patch size approaches are a promising irection to improve current algorithms, such as [0, 8, 6, 9, 6]. Our work also highlights the relation between the frequency of occurrence of a patch, local pixel correlations, an potential enoising gains. This concept is not restricte to the enoising problem, an may have implications in other fiels. Acknowlagments: We thank ISF, BSF, ERC, Intel, Quanta an NSF for funing. References. L. Alvarez, Y. Gousseau, an J. Morel. The size of objects in natural images, S. Arietta an J. Lawrence. Builing an using a atabase of one trillion natural-image patches. IEEE Computer Graphics an Applications, S. Baker an T. Kanae. Limits on super-resolution an how to break them. PAMI, C. M. Bishop. Pattern recognition an machine learning. Springer, The numerical results in Tables, are not irectly comparable, since Table was compute on a small subset of only M =, 000 test examples, but with a larger sample size N. 3

14 5. A. Buaes, B. Coll, an J. Morel. A review of image enoising methos, with a new one. Multiscale Moel. Simul., D. Chanler an D. Fiel. Estimates of the information content an imensionality of natural scenes from proximity istributions. J. Opt. Soc. Am., P. Chatterjee an P. Milanfar. Is enoising ea? IEEE Trans Image Processing, K. Dabov, A. Foi, V. Katkovnik, an K. Egiazarian. Image enoising by sparse 3- transformomain collaborative filtering. IEEE Trans Image Processing, W. Dong, X. Li, L. Zhang, an G. Shi. Sparsity-base image enoising vis ictionary learning an structural clustering. In CVPR, M. Ela an M. Aharon. Image enoising via sparse an reunant representations over learne ictionaries. IEEE Trans Image Processing, D. Fiel. Relations between the statistics of natural images an the response properties of cortical cells. JOSA, C. Kervrann an J. Boulanger. Optimal spatial aaptation for patch-base image enoising. ITIP, A. Lee, K. Peersen, an D. Mumfor. The nonlinear statistics of high-contrast patches in natural images. IJCV, A. Levin an B. Naler. Natural image enoising: optimality an inherent bouns. In CVPR, A. Levin, B. Naler, F. Duran, an W. Freeman. Patch complexity, finite pixel correlations an optimal enoising. MIT CSAIL TR J. Mairal, F. Bach, J. Ponce, G. Sapiro, an A. Zisserman. Non-local sparse moels for image restoration. In ICCV, D. Mumfor an B. Gias. Stochastic moels for generic images. Quarterly of applie mathematics, S. Osinero an G. Hinton. Moeling image patches with a irecte hierarchy of markov ranom fiels. NIPS, J. Polzehl an V. Spokoiny. Image enoising: Pointwise aaptive approach. Annals of Statistics, 3:30 57, J. Portilla, V. Strela, M. Wainwright, an E. Simoncelli. Image enoising using scale mixtures of gaussians in the wavelet omain. IEEE Trans Image Processing, X. Ren an J. Malik. A probabilistic multi-scale moel for contour completion base on image statistics. In ECCV, S. Roth an M.J. Black. Fiels of experts: A framework for learning image priors. In CVPR, D. Ruerman. Origins of scaling in natural images. Vision Research, Z. Wang, A. C. Bovik, H. R. Sheikh, an E. P. Simoncelli. Image quality assessment: From error visibility to structural similarity. IEEE Trans. on Image Processing, M. Zontak an M. Irani. Internal statistics of a single natural image. In CVPR, D. Zoran an Y. Weiss. From learning moels of natural image patches to whole image restoration. In ICCV, 0. 4