Assignment of Multiplicative Mixtures in Natural Images

Transcription

1 Assignment of Mutipicative Mixtures in Natura Images Odeia Schwartz HHMI and Sak Institute La Joa, CA 94 Terrence J. Sejnowski HHMI and Sak Institute La Joa, CA 94 Abstract Peter Dayan GCNU, UCL 7 Queen Square, London dayan@gatsby.uc.ac.uk In the anaysis of natura images, Gaussian scae mixtures (GSM have been used to account for the statistics of fiter responses, and to inspire hierarchica cortica representationa earning schemes. GSMs pose a critica assignment probem, working out which fiter responses were generated by a common mutipicative factor. We present a new approach to soving this assignment probem through a probabiistic extension to the basic GSM, and show how to perform inference in the mode using Gibbs samping. We demonstrate the efficacy of the approach on both synthetic and image data. Understanding the statistica structure of natura images is an important goa for visua neuroscience. Neura representations in eary cortica areas decompose images (and ikey other sensory inputs in a way that is sensitive to sophisticated aspects of their probabiistic structure. This structure aso pays a key roe in methods for image processing and coding. A striking aspect of natura images that has refections in both top-down and bottom-up modeing is coordination across nearby ocations, scaes, and orientations. From a topdown perspective, this structure has been modeed using what is known as a Gaussian Scae Mixture mode (GSM. 3 GSMs invove a muti-dimensiona Gaussian (each dimension of which captures oca structure as in a inear fiter, mutipied by a spatiaized coection of common hidden scae variabes or mixer variabes (which capture the coordination. GSMs have wide impications in theories of cortica receptive fied deveopment, eg the comprehensive bubbes framework of Hyvärinen. 4 The mixer variabes provide the top-down account of two bottom-up characteristics of natura image statistics, namey the bowtie statistica dependency, 5, 6 and the fact that the margina distributions of receptive fied-ike fiters have high kurtosis. 7, 8 In hindsight, these ideas aso bear a cose reationship with Ruderman and Biaek s mutipicative bottom-up image anaysis framework 9 and statistica modes for divisive gain contro. 6 Coordinated structure has aso been addressed in other image work, 4 and in other domains such as speech 5 and finance. 6 Many approaches to the unsupervised specification of representations in eary cortica areas rey on the coordinated structure. 7 The idea is to earn inear fiters (eg modeing simpe ces as in, 3, and then, based on the coordination, to find combinations of these (perhaps non-ineary transformed as a way of finding higher order fiters (eg compex ces. One critica facet whose specification from data is not obvious is the neighborhood arrangement, ie which inear fiters share which mixer variabes. Mixer variabes are aso caed mutipiers, but are unreated to the scaes of a waveet.

2 Here, we suggest a method for finding the neighborhood based on Bayesian inference of the GSM random variabes. In section, we consider estimating these components based on information from different-sized neighborhoods and show the modes of faiure when inference is too oca or too goba. Based on these observations, in section we propose an extension to the GSM generative mode, in which the mixer variabes can overap probabiisticay. We sove the neighborhood assignment probem using Gibbs samping, and demonstrate the technique on synthetic data. In section 3, we appy the technique to image data. GSM inference of Gaussian and mixer variabes In a simpe, n-dimensiona, version of a GSM, fiter responses are synthesized by mutipying an n-dimensiona Gaussian with vaues g = {g... g n }, by a common mixer variabe v. = vg ( We assume g are uncorreated ( aong diagona of the covariance matrix. For the anaytica cacuations, we assume that v has a Rayeigh distribution: p[v] [v exp v /] a where < a parameterizes the strength of the prior ( For ease, we deveop the theory for a =. As is we known, and repeated in figure (B, the margina distribution of the resuting GSM is sparse and highy kurtotic. The joint conditiona distribution of two eements and, foows a bowtie shape, with the width of the distribution of one dimension increasing for arger vaues (both positive and negative of the other dimension. The inverse probem is to estimate the n+ variabes g... g n, v from the n fiter responses... n. It is formay i-posed, though reguarized through the prior distributions. Four posterior distributions are particuary reevant, and can be derived anayticay from the mode: rv distribution posterior mean ( p[v ] exp v B, B v B ( p[v ] p[ g ] p[ g ] ( (n B( n, v (n exp B( n, B ( n, exp g g (n 3 exp v v ( g g ( g g,, B( 3 n, B( n, B, B, B( n, B( n, where B(n, x is the modified Besse function of the second kind (see aso 4, = i i and g i is forced to have the same sign as i, since the mixer variabes are aways positive. Note that p[v ] and p[g ] (rows,3 are oca estimates, whie p[v ] and p[g ] (rows,4 are estimates according to fiter outputs {... n }. The posterior p[v ] has aso been estimated numericay in noise remova for other mixer priors, by Portia et a 5 The fu GSM specifies a hierarchy of mixer variabes. Wainwright considered a prespecified tree-based hierarhica arrangement. In practice, for natura sensory data, given a heterogeneous coection of i, it is advantageous to earn the hierachica arrangement from exampes. In an approach reated to that of the GSM, Karkin and Lewicki 9 suggested We describe the as being fiter responses even in the synthetic case, to faciitate comparison with images.

3 A v g... g g... g v β Mutipy Mutipy 4 B C Mixer.6 Actua fiter, too oca fiters v E(v E(v fiters, too goba.6 5 E(v.. 4 D Gaussian g E(g E(g E(g.. 4 E Gaussian joint conditiona g E(g E(g.. E(g.. 4 g E(g E(g.. E(g.. 4 Figure : A Generative mode: each fiter response is generated by mutipying its Gaussian variabe by either mixer variabe v, or mixer variabe v β. B Margina and joint conditiona statistics (bowties of sampe synthetic fiter responses. For the joint conditiona statistics, intensity is proportiona to the bin counts, except that each coumn is independenty re-scaed to fi the range of intensities. C-E Left: actua distributions of mixer and Gaussian variabes; other coumns: estimates based on different numbers of fiter responses. C of estimate of the mixer variabe v. Note that mixer variabe vaues are by definition positive. D of estimate of one of the Gaussian variabes, g. E Joint conditiona statistics of the estimates of Gaussian variabes g and g. generating og mixer vaues for a the fiters and earning the inear combinations of a smaer coection of underying vaues. Here, we consider the probem in terms of mutipe mixer variabes, with the inear fiters being custered into groups that share a singe mixer. This poses a critica assignment probem of working out which fiter responses share which mixer variabes. We first study this issue using synthetic data in which two groups of fiter responses... and... 4 are generated by two mixer variabes v and v β (figure. We attempt to infer the components of the GSM mode from the synthetic data. Figure C;D shows the empirica distributions of estimates of the conditiona means of a mixer variabe E(v {} and one of the Gaussian variabes E(g {} based on different assumed assignments. For estimation based on too few fiter responses, the estimates do not we match the actua distributions. For exampe, for a oca estimate based on a singe fiter response, the Gaussian estimate peaks away from zero. For assignments incuding more fiter responses, the estimates become good. However, inference is aso compromised if the estimates for v are too goba, incuding fiter responses actuay generated from v β (C and D, ast coumn. In (E, we consider the joint conditiona statistics of two components, each

4 A Generative mode B Actua v v β v γ v v β v γ g... g Mutipy Fiter number Fiter number Fiter number Inferred v v v β γ... Fiter number Fiter number Fiter number C Pixe Gaussian Mixer. -. Gibbs fit assumed -4 4 E(g E(g.5 Gibbs fit assumed E(v β 5 E(g E(v E(v Figure : A Generative mode in which each fiter response is generated by mutipication of its Gaussian variabe by a mixer variabe. The mixer variabe, v, v β, or v γ, is chosen probabiisticay upon each fiter response sampe, from a Rayeigh distribution with a =.. B Top: actua probabiity of fiter associations with v, v β, and v γ; Bottom: Gibbs estimates of probabiity of fiter associations corresponding to v, v β, and v γ. C Statistics of generated fiter responses, and of Gaussian and mixer estimates from Gibbs samping. estimating their respective g and g. Again, as the number of fiter responses increases, the estimates improve, provided that they are taken from the right group of fiter responses with the same mixer variabe. Specificay, the mean estimates of g and g become more independent (E, third coumn. Note that for estimations based on a singe fiter response, the joint conditiona distribution of the Gaussian appears correated rather than independent (E, second coumn; for estimation based on too many fiter responses (4 in this exampe, the joint conditiona distribution of the Gaussian estimates shows a dependent (rather than independent bowtie shape (E, ast coumn. Mixer variabe joint statistics aso deviate from the actua when the estimations are too oca or goba (not shown. We have observed quaitativey simiar statistics for estimation based on coefficients in natura images. Neighborhood size has aso been discussed in the context of the quaity of noise remova, assuming a GSM mode. 6 Neighborhood inference: soving the assignment probem The pots in figure suggest that it shoud be possibe to infer the assignments, ie work out which fiter responses share common mixers, by earning from the statistics of the resuting joint dependencies. Hard assignment probems (in which each fiter response pays aegiance to just one mixer are notoriousy computationay britte. Soft assignment probems (in which there is a probabiistic reationship between fiter responses and mixers are computationay better behaved. Further, rea word stimui are ikey better captured by the possibiity that fiter responses are coordinated in somewhat different coections in different images. We consider a richer, mixture GSM as a generative mode (Figure. To mode the generation of fiter responses i for a singe image patch, we mutipy each Gaussian variabe g i by a singe mixer variabe from the set v... v m. We assume that g i has association probabi-

5 ity p ij (satisfying j p ij =, i of being assigned to mixer variabe v j. The assignments are assumed to be made independenty for each patch. We use s i {,,... m} for the assignments: i = g i v si (3 Inference and earning in this mode proceeds in two stages, according to the expectation maximization agorithm. First, given a fiter response i, we use Gibbs samping for the E phase to find possibe appropriate (posterior assignments. Wiiams et a. 7 suggested using Gibbs samping to sove a simiar assignment probem in the context of dynamic tree modes. Second, for the M phase, given the coection of assignments across mutipe fiter responses, we update the association probabiities p ij. Given sampe mixer assignments, we can estimate the Gaussian and mixer components of the GSM using the tabe of section, but restricting the fiter response sampes just to those associated with each mixer variabe. We tested the abiity of this inference method to find the associations in the probabiistic mixer variabe synthetic exampe shown in figure, (A,B. The true generative mode specifies probabiistic overap of 3 mixer variabes. We generated 5 sampes for each fiter according to the generative mode. We ran the Gibbs samping procedure, setting the number of possibe neighborhoods to 5 (e.g., > 3; after 5 iterations the weights converged near to the proper probabiities. In (B, top, we pot the actua probabiity distributions for the fiter associations with each of the mixer variabes. In (B, bottom, we show the estimated associations: the three non-zero estimates cosey match the actua distributions; the other two estimates are zero (not shown. The procedure consistenty finds correct associations even in arger exampes of data generated with up to mixer variabes. In (C we show an exampe of the actua and estimated distributions of the mixer and Gaussian components of the GSM. Note that the joint conditiona statistics of both mixer and Gaussian are independent, since the variabes were generated as such in the synthetic exampe. The Gibbs procedure can be adjusted for data generated with different parameters a of equation, and for reated mixers, aowing for a range of image coefficient behaviors. 3 Image data Having vaidated the inference mode using synthetic data, we turned to natura images. We derived inear fiters from a muti-scae oriented steerabe pyramid, 8 with fiters, at preferred orientations, 5 non-overapping spatia positions (with spatia subsamping of 8 pixes, and two phases (quadrature pairs, and a singe spatia frequency peaked at /6 cyces/pixe. The image ensembe is 4 images from a standard image compression database (boats, godhi, pant eaves, and mountain and 4 sampes. We ran our method with the same parameters as for synthetic data, with 7 possibe neighborhoods and Rayeigh parameter a =. (as in figure. Figure 3 depicts the association weights p ij of the coefficients for each of the obtained mixer variabes. In (A, we show a schematic (tempate of the association representation that wi foow in (B, C for the actua data. Each mixer variabe neighborhood is shown for coefficients of two phases and two orientations aong a spatia grid (one grid for each phase. The neighborhood is iustrated via the probabiity of each coefficient to be generated from a given mixer variabe. For the first two neighborhoods (B, we aso show the image patches that yieded the maximum og ikeihood of P (v patch. The first neighborhood (in B prefers vertica patterns across most of its receptive fied, whie the second has a more ocaized region of horizonta preference. This can aso be seen by averaging the image patches with the maximum og ikeihood. Strikingy, a the mixer variabes group together two phases of quadrature pair (B, C. Quadrature pairs have aso been extracted from cortica data, and are the components of idea compex ce modes. Another tendency is to group

6 A Y position Phase X position Y position Phase X position Phase Phase B Neighborhood Exampe max patches Average C Neighborhood Neighborhood Exampe max patches Average D Coefficient Gaussian Mixer Gibbs fit assumed -5 5 E(g E(g E(g.5 Gibbs fit assumed 5 E(v E(v β E(v Figure 3: A Schematic of the mixer variabe neighborhood representation. The probabiity that each coefficient is associated with the mixer variabe ranges from (back to (white. Left: Vertica and horizonta fiters, at two orientations, and two phases. Each phase is potted separatey, on a 38 by 38 pixe spatia grid. Right: summary of representation, with fiter shapes repaced by oriented ines. Fiters are approximatey 6 pixes in diameter, with the spacing between fiters 8 pixes. B First two image ensembe neighborhoods obtained from Gibbs samping. Aso shown, are four pixe patches that had the maximum og ikeihood of P (v patch, and the average of the first maxima patches. C Other image ensembe neighborhoods. D Statistics of representative coefficients of two spatiay dispaced vertica fiters, and of inferred Gaussian and mixer variabes. orientations across space. The phase and iso-orientation grouping bear some interesting simiarity to other recent suggestions; 7, 8 as do the maxima patches. 9 Waveet fiters have the advantage that they can span a wider spatia extent than is possibe with current ICA techniques, and the anaysis of parameters such as phase grouping is more controed. We are comparing the anaysis with an ICA first-stage representation, which has other obvious advantages. We are aso extending the anaysis to correated waveet fiters; 5 and to simuations with a arger number of neighborhoods. From the obtained associations, we estimated the mixer and Gaussian variabes according to our mode. In (D we show representative statistics of the coefficients and of the inferred variabes. The earned distributions of Gaussian and mixer variabes are quite cose to our assumptions. The Gaussian estimates exhibit joint conditiona statistics that are roughy independent, and the mixer variabes are weaky dependent. We have thus far demonstrated neighborhood inference for an image ensembe, but it is aso interesting and perhaps more intuitive to consider inference for particuar images or image casses. In figure 4 (A-B we demonstrate exampe mixer variabe neighborhoods derived from earning patches of a zebra image (Core CD-ROM. As before, the neighborhoods are composed of quadrature pairs; however, the spatia configurations are richer and have

7 A Neighborhood Average B Neighborhood Average Exampe max patches Top 5 max patches Exampe max patches Top 5 max patches Figure 4: Exampe of Gibbs on Zebra image. Image is 5 5 pixes, and each spatia neighborhood spans pixes. A, B Exampe mixer variabe neighborhoods. Left: exampe mixer variabe neighborhood, and average of patches that yieded the maximum ikeihood of P (v patch. Right: Image and marked on top of it exampe patches that yieded the maximum ikeihood of P (v patch. not been previousy reported with unsupervised hierarchica methods: for exampe, in (A, the mixture neighborhood captures a horizonta-bottom/vertica-top spatia configuration. This appears particuary reevant in segmenting regions of the front zebra, as shown by marking in the image the patches i that yieded the maximum og ikeihood of P (v patch. In (B, the mixture neighborhood captures a horizonta configuration, more focused on the horizonta stripes of the front zebra. This exampe demonstrates the ogic behind a probabiistic mixture: coefficients corresponding to the bottom horizonta stripes might be inked with top vertica stripes (A or to more horizonta stripes (B. 4 Discussion Work on the study of natura image statistics has recenty evoved from issues about scaespace hierarchies, waveets, and their ready induction through unsupervised earning modes (oosey based on cortica deveopment towards the coordinated statistica structure of the waveet components. This incudes bottom-up (eg bowties, hierarchica representations such as compex ces and top-down (eg GSM viewpoints. The resuting new insights inform a weath of modes and ideas and form the essentia backdrop for the work in this paper. They aso ink to impressive engineering resuts in image coding and processing. A most critica aspect of an hierarchica representationa mode is the way that the structure of the hierarchy is induced. We addressed the hierarchy question using a nove extension to the GSM generative mode in which mixer variabes (at one eve of the hierarchy enjoy probabiistic assignments to fiter responses (at a ower eve. We showed how these assignments can be earned (using Gibbs samping, and iustrated some of their attractive properties using both synthetic and a variety of image data. We grounded our method firmy in Bayesian inference of the posterior distributions over the two casses of random variabes in a GSM (mixer and Gaussian, pacing particuar emphasis on the interpay between the generative mode and the statistica properties of its components. An obvious question raised by our work is the neura correate of the two different posterior variabes. The Gaussian variabe has characteristics resembing those of the output of divisivey normaized simpe ces; 6 the mixer variabe is more obviousy reated to the output of quadrature pair neurons (such as orientation energy or motion energy ces, which may aso be divisivey normaized. How these different information sources may subsequenty be used is of great interest.

8 Acknowedgements This work was funded by the HHMI (OS, TJS and the Gatsby Charitabe Foundation (PD. We are very gratefu to Patrik Hoyer, Mike Lewicki, Zhaoping Li, Simon Osindero, Javier Portia and Eero Simoncei for discussion. References [] D Andrews and C Maows. Scae mixtures of norma distributions. J. Roya Stat. Soc., 36:99, 974. [] M J Wainwright and E P Simoncei. Scae mixtures of Gaussians and the statistics of natura images. In S. A. Soa, T. K. Leen, and K.-R. Müer, editors, Adv. Neura Information Processing Systems, voume, pages , Cambridge, MA, May. MIT Press. [3] M J Wainwright, E P Simoncei, and A S Wisky. Random cascades on waveet trees and their use in modeing and anayzing natura imagery. Appied and Computationa Harmonic Anaysis, (:89 3, Juy. Specia issue on waveet appications. [4] A Hyvärinen, J Hurri, and J Vayrynen. Bubbes: a unifying framework for ow-eve statistica properties of natura image sequences. Journa of the Optica Society of America A, :37 5, May 3. [5] R W Buccigrossi and E P Simoncei. Image compression via joint statistica characterization in the waveet domain. IEEE Trans Image Proc, 8(:688 7, December 999. [6] O Schwartz and E P Simoncei. Natura signa statistics and sensory gain contro. Nature Neuroscience, 4(8:89 85, August. [7] D J Fied. Reations between the statistics of natura images and the response properties of cortica ces. J. Opt. Soc. Am. A, 4(: , 987. [8] H Attias and C E Schreiner. Tempora ow-order statistics of natura sounds. In M Jordan, M Kearns, and S Soa, editors, Adv in Neura Info Processing Systems, voume 9, pages MIT Press, 997. [9] D L Ruderman and W Biaek. Statistics of natura images: Scaing in the woods. Phys. Rev. Letters, 73(6:84 87, 994. [] C Zetzsche, B Wegmann, and E Barth. Noninear aspects of primary vision: Entropy reduction beyond decorreation. In Int Symposium, Society for Information Dispay, voume XXIV, pages , 993. [] J Huang and D Mumford. Statistics of natura images and modes. In CVPR, page 547, 999. [] J. Romberg, H. Choi, and R. Baraniuk. Bayesian waveet domain image modeing using hidden Markov trees. In Proc. IEEE Int Conf on Image Proc, Kobe, Japan, October 999. [3] A Turie, G Mato, N Parga, and J P Nada. The sef-simiarity properties of natura images resembe those of turbuent fows. Phys. Rev. Lett., 8:98, 998. [4] J Portia and E P Simoncei. A parametric texture mode based on joint statistics of compex waveet coefficients. Int Journa of Computer Vision, 4(:49 7,. [5] Hemut Brehm and Water Stammer. Description and generation of sphericay invariant speech-mode signas. Signa Processing, :9 4, 987. [6] T Boersey, K Enge, and D Neson. ARCH modes. In B Enge and D McFadden, editors, Handbook of Econometrics V [7] A Hyvärinen and P Hoyer. Emergence of topography and compex ce properties from natura images using extensions of ICA. In S. A. Soa, T. K. Leen, and K.-R. Müer, editors, Adv. Neura Information Processing Systems, voume, pages , Cambridge, MA, May. MIT Press. [8] P Hoyer and A Hyvärinen. A muti-ayer sparse coding network earns contour coding from natura images. Vision Research, 4(:593 65,. [9] Y Karkin and M S Lewicki. Learning higher-order structures in natura images. Network: Computation in Neura Systems, 4: , 3. [] W Laurenz and T Sejnowski. Sow feature anaysis: Unsupervised earning of invariances. Neura Computation, 4(4:75 77,. [] C Kayser, W Einhäuser, O Dümmer, P König, and K P Körding. Extracting sow subspaces from natura videos eads to compex ces. In G Dorffner, H Bischof, and K Hornik, editors, Proc. Int Conf. on Artificia Neura Networks (ICANN-, pages 75 8, Vienna, Aug. Springer-Verag, Heideberg. [] B A Oshausen and D J Fied. Emergence of simpe-ce receptive fied properties by earning a sparse factoria code. Nature, 38:67 69, 996. [3] A J Be and T J Sejnowski. The independent components of natura scenes are edge fiters. Vision Research, 37(3: , 997. [4] U Grenander and A Srivastava. Probabibiity modes for cutter in natura images. IEEE Trans. on Patt. Ana. and Mach. Inte., 3:43 49,. [5] J Portia, V Strea, M Wainwright, and E Simoncei. Adaptive Wiener denoising using a Gaussian scae mixture mode in the waveet domain. In Proc 8th IEEE Int Conf on Image Proc, pages 37 4, Thessaoniki, Greece, Oct 7-. IEEE Computer Society. [6] J Portia, V Strea, M Wainwright, and E P Simoncei. Image denoising using a scae mixture of Gaussians in the waveet domain. IEEE Trans Image Processing, (:338 35, November 3. [7] C K I Wiiams and N J Adams. Dynamic trees. In M. S. Kearns, S. A. Soa, and D. A. Cohn, editors, Adv. Neura Information Processing Systems, voume, pages , Cambridge, MA, 999. MIT Press. [8] E P Simoncei, W T Freeman, E H Adeson, and D J Heeger. Shiftabe muti-scae transforms. IEEE Trans Information Theory, 38(:587 67, March 99. Specia Issue on Waveets.