RealTime Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations


 Rodger Small
 1 years ago
 Views:
Transcription
1 RealTime Computig Without Stable States: A New Framework for Neural Computatio Based o Perturbatios Wolfgag aass+, Thomas Natschläger+ & Hery arkram* + Istitute for Theoretical Computer Sciece, Techische Uiversität Graz; A8010 Graz, Austria * Brai id Istitute, Ecole Polytechique Federale de Lausae, CH1015 Lausae, Switzerlad Wolfgag aass & Thomas Natschlaeger Istitute for Theoretical Computer Sciece Techische Uiversitaet Graz Iffeldgasse 16b, A8010 Graz, Austria Tel: Fax: Hery arkram Brai id Istitute Ecole Polytechique Federale de Lausae CEEcubles, CH1015 Lausae, Switzerlad Tel: Fax: Address for Correspodece: Wolfgag aass 1
2 A key challege for eural modelig is to explai how a cotiuous stream of multimodal iput from a rapidly chagig eviromet ca be processed by stereotypical recurret circuits of itegrateadfire euros i realtime. We propose a ew computatioal model for realtime computig o timevaryig iput that provides a alterative to paradigms based o Turig machies or attractor eural etworks. It does ot require a taskdepedet costructio of eural circuits. Istead it is based o priciples of high dimesioal dyamical systems i combiatio with statistical learig theory, ad ca be implemeted o geeric evolved or foud recurret circuitry. It is show that the iheret trasiet dyamics of the highdimesioal dyamical system formed by a sufficietly large ad heterogeeous eural circuit may serve as uiversal aalog fadig memory. Readout euros ca lear to extract i realtime from the curret state of such recurret eural circuit iformatio about curret ad past iputs that may be eeded for diverse tasks. Stable iteral states are ot required for givig a stable output, sice trasiet iteral states ca be trasformed by readout euros ito stable target outputs due to the high dimesioality of the dyamical system. Our approach is based o a rigorous computatioal model, the liquid state machie, that ulike Turig machies, does ot require sequetial trasitios betwee welldefied discrete iteral states. It is supported, like the Turig machie, by rigorous mathematical results that predict uiversal computatioal power uder idealized coditios, but for the biologically more realistic sceario of realtime processig of timevaryig iputs. Our approach provides ew perspectives for the iterpretatio of eural codig, for the desig of experimets ad dataaalysis i europhysiology, ad for the solutio of problems i robotics ad eurotechology. 2
3 Itroductio Itricate topographically orgaized feedforward pathways project rapidly chagig spatiotemporal iformatio about the eviromet ito the eocortex. This iformatio is processed by extremely complex but surprisigly stereotypic microcircuits that ca perform a wide spectrum of tasks (Shepherd, 1988, Douglas et al., 1998, vo elcher et al., 2000). The microcircuit features that eable this seemigly uiversal computatioal power, is a mystery. Oe particular feature, the multiple recurret loops that form a immesely complicated etwork usig as may as 80% of all the syapses withi a fuctioal eocortical colum, has preseted a itractable problem both for computatioal models ispired by curret artificial computig machiery (Savage, 1998), ad for attractor eural etwork models. The difficulty to uderstad computatios withi recurret etworks of itegrateadfire euros comes from the fact that their dyamics takes o a life of its ow whe challeged with rapidly chagig iputs. This is particularly true for the very high dimesioal dyamical system formed by a eural microcircuit, whose compoets are highly heterogeeous ad where each euro ad each syapse adds degrees of freedom to the dyamics of the system. The most commo approach for modelig computig i recurret eural circuits has bee to try to take cotrol of their high dimesioal dyamics. ethods for cotrollig the dyamics of recurret eural etworks through adaptive mechaisms are reviewed i (Pearlmutter, 1995). So far oe has ot bee able to apply these to the case of etworks of spikig euros. Other approaches towards modelig computatio i biological eural systems are based o costructios of artificial eural etworks that simulate Turig machies or other models for digital computatio, see for example (Pollack, 1991), (Giles et al., 1992), (Siegelma et al., 1994), (Hyoetyiemi, 1996), (oore, 1998). Amog these there are models, such as dyamical recogizers, which are capable of realtime computig o olie iput (i discrete time). Noe of these approaches has bee demostrated to work for etworks of spikig euros, or ay more realistic models for eural microcircuits. It was show i (aass, 1996) that oe also ca costruct recurret circuits of spikig euros that ca simulate arbitrary Turig machies. But all of these approaches require sychroizatio of all euros by a cetral clock, a feature that appears to missig i eural microcircuits. I additio they require the costructio of particular recurret circuits, ad caot be implemeted by evolvig or adaptig a give circuit. Furthermore the results of (aass et al., 1999) o the impact of oise o the computatioal power of recurret eural etworks suggest that all these approaches break dow as soo as oe assumes that the uderlyig aalog computatioal uits are subject to Gaussia or other realistic oise distributios. Attractor eural etworks o the other had allow oise robust computatio, but their attractor ladscape is i geeral hard to cotrol, ad they eed to have a very large set of attractors i order to store saliet iformatio o past iputs (for example 1024 attractors i order to store 10 bits). I additio they are less suitable for realtime computig o rapidly varyig iput streams because of the time required for covergece to a attractor. Fially, oe of these approaches allows that several realtime computatios are carried out i parallel withi the same circuitry, which appears to be a geeric feature of eural microcircuits. I this article we aalyze the dyamics of eural microcircuits from the poit of view of a readout euro, whose task is to extract iformatio ad report results from a eural microcircuit to other circuits. A huma observer of the dyamics i a eural microcircuit would be lookig for clearly distict ad temporally stable features, such as covergece to attractors. We show that a readout euro, that receives iputs from hudreds or thousads of euros i a eural 3
4 microcircuit, ca lear to extract saliet iformatio from the high dimesioal trasiet states of the circuit, ad ca trasform trasiet circuit states ito stable readouts. I particular each readout ca lear to defie its ow otio of equivalece of dyamical states withi the eural microcircuit, ad ca the perform its task o ovel iputs. This uexpected fidig of readoutassiged equivalet states of a dyamical system explais how ivariat readout is possible despite the fact that the eural microcircuit may ever revisit the same state. Furthermore we show that multiple readout modules ca be traied to perform differet tasks o the same state trajectories of a recurret eural circuit, thereby eablig parallel realtime computig. We preset the mathematical framework for a computatioal model that does ot require covergece to stable iteral states or attractors (eve if they do occur), sice iformatio about past iputs is automatically captured i the perturbatios of a dyamical system, i.e. i the cotiuous trajectory of trasiet iteral states. Special cases of this mechaism were already reported i (Buoomao et al., 1995) ad (Domiey et al., 1995). Similar ideas have bee discovered idepedetly by Herbert Jaeger (Jaeger, 2001) i the cotext of artificial eural etworks. Computig without Attractors As a illustratio for our geeral approach towards realtime computig cosider a series of trasiet perturbatios caused i a excitable medium (see (Holde et al., 1991)), for example a liquid, by a sequece of exteral disturbaces ("iputs") such as wid, soud, or sequeces of pebbles dropped ito the liquid. Viewed as a attractor eural etwork, the liquid has oly oe attractor state the restig state ad may therefore seem useless for computatioal purposes. However, the perturbed state of the liquid, at ay momet i time, represets preset as well as past iputs, potetially providig the iformatio eeded for a aalysis of various dyamic aspects of the eviromet. I order for such a liquid to serve as a source of saliet iformatio about preset ad past stimuli without relyig o stable states, the perturbatios must be sesitive to salietly differet iputs but ochaotic. The maer i which perturbatios are formed ad maitaied would vary for differet types of liquids ad would determie how useful the perturbatios are for such retrograde aalysis. Limitatios o the computatioal capabilities of liquids are imposed by their timecostat for relaxatio, ad the strictly local iteractios ad homogeeity of the elemets of a liquid. Neural microcircuits, however, appear to be ideal liquids for computig o perturbatios because of the large diversity of their elemets, euros ad syapses (see (Gupta et al., 2000)), ad the large variety of mechaisms ad time costats characterizig their iteractios, ivolvig recurret coectios o multiple spatial scales ("loops withi loops"). The foudatio for our aalysis of computatios without stable states is a rigorous computatioal model: the liquid state machie. Two macroscopic properties emerge from our theoretical aalysis ad computer simulatios as ecessary ad sufficiet coditios for powerful realtime computig o perturbatios: a separatio property, SP, ad a approximatio property, AP. SP addresses the amout of separatio betwee the trajectories of iteral states of the system that are caused by two differet iput streams (i the case of a physical liquid, SP could reflect the differece betwee the wave patters resultig from differet sequeces of disturbaces). 4
5 AP addresses the resolutio ad recodig capabilities of the readout mechaisms  more precisely its capability to distiguish ad trasform differet iteral states of the liquid ito give target outputs (whereas SP depeds mostly o the complexity of the liquid, AP depeds mostly o the adaptability of the readout mechaism to the required task). Liquid State achies Like the Turig machie (Savage, 1998), the model of a liquid state machie (LS) is based o a rigorous mathematical framework that guaratees, uder idealized coditios, uiversal computatioal power. Turig machies, however, have uiversal computatioal power for offlie computatio o (static) discrete iputs, while LSs have i a very specific sese uiversal computatioal power for realtime computig with fadig memory o aalog fuctios i cotiuous time. The iput fuctio u ( ) ca be a cotiuous sequece of disturbaces, ad the target output ca be some chose fuctio y ( ) of time that provides a realtime aalysis of this sequece. I order for a machie to map iput fuctios of time u ( ) to output fuctios y( ) of time, we assume that it geerates, at every time t, a iteral liquid state x (t), which costitutes its curret respose to precedig perturbatios, i.e., to precedig iputs u (s) for s t (Figure 1). I cotrast to the fiite state of a fiite state machie (or fiite automato) this liquid state cosists of aalog values that may chage cotiuously over time. Whereas the state set ad the state trasitio fuctio of a fiite state machie is i geeral costructed for a Figure 1: A: Architecture of a LS. A fuctio of time (time series) u( ) is ijected as iput ito the L liquid filter, creatig at time t the liquid state x (t), which is trasformed by a memoryless readout map f to geerate a output y(t). specific task, the liquid states ad the trasitios betwee them eed ot be customized for a specific task. I a physical implemetatio this liquid state cosists of all iformatio about the curret iteral state of a dyamical system that is accessible to the readout modules. I 5
6 mathematical terms, this liquid state is simply the curret output of some operator or filter 1 that maps iput fuctios u ( ) oto fuctios x (t) : x ( t) = ( L u)( t). L I the followig we will refer to this filter L ofte as liquid filter, or liquid circuit if implemeted by a circuit. If it is implemeted by a eural circuit, we refer to the euros i that circuit as liquid euros. The secod compoet of a LS is a memoryless readout map f that trasforms, at every time t, the curret liquid state x (t) ito the output y( t) = f ( x ( t)). I cotrast to the liquid filter L, this readout map f is i geeral chose i a taskspecific maer (ad there may be may differet readout maps, that extract differet taskspecific iformatio i parallel from the curret output of L ). Note that i a fiite state machie there exists o aalogo to such taskspecific readout maps, sice there the iteral fiite states are already costructed i a taskspecific maer. Accordig to the precedig defiitio readout maps are i geeral memoryless 2. Hece all iformatio about iputs u (s) from precedig time poits s t that is eeded to produce a target output y(t) at time t has to be cotaied i the curret liquid state x (t). odels for computatio that have origiated i computer sciece store such iformatio about the past i stable states (for example i memory buffers or tapped delay lies). We argue, however, that this is ot ecessary sice large computatioal power o fuctios of time ca also be realized eve if all memory traces are cotiuously decayig. Istead of worryig about the code ad locatio where iformatio about past iputs is stored, ad how this iformatio decays, it is eough to address the separatio questio: For which later time poits t will ay two sigificatly differet iput fuctios of time u( ) ad v( ) cause sigificatly differet liquid states x u (t) ad (t). Good separatio capability, i combiatio x v 1 Fuctios F that map iput fuctios of time u ( ) o output fuctios y ( ) of time are usually called operators i mathematics, but are commoly referred to as filters i egieerig ad eurosciece. We use the term filter i the followig, ad we write ( Fu )( t) for the output of the filter F at time t whe F is applied to the iput fuctio u ( ). Formally, such filter F is a map from U ito ( R R ) k, where R R is the set of all realvalued fuctios of time, vectors cosistig of k such fuctios of time, U is some subset of vectors cosistig of fuctios of time i U. 2 The term "memoryless" refers to the fact that the readout map R R, ad ( R k R ) is the set of U is the set of f is ot required to retai ay memory of previous states x ( s), s < t, of the liquid. However, i a biological cotext, the readout map will i geeral be subject to plasticity, ad may also cotribute to the memory capability of the system. We do ot explore this issues i this article because the differetiatio ito a memoryless readout map ad a liquid that serves as a memory device is made for coceptual clarificatio, ad is ot essetial to the model. 6
7 with a adequate readout map f, allows us to discard the requiremet of storig bits "util further otice" i stable states of the computatioal system. Uiversal Computatioal Power of LSs for Time Varyig Iputs We say that a class of machies has uiversal power for computatios with fadig memory o fuctios of time if ay filter F, i.e., ay map from fuctios of time u ( ) to fuctios of time y ( ), that is time ivariat 3 ad has fadig memory 4, ca be approximated by machies from this class, to ay degree of precisio. Arguably, these filters F are approximated accordig to this defiitio iclude all maps from iput fuctios of time to output fuctios of time that a behavig orgaism might eed to compute. A mathematical theorem (see Appedix A) guaratees that LSs have this uiversal computatioal power regardless of specific structure or implemetatio, provided that two abstract properties are met: the class of basis filters from which the liquid filters L are composed satisfies the poitwise separatio property ad the class of fuctios from which the readout maps f are draw, satisfies the approximatio property. These two properties provide the mathematical basis for the separatio property SP ad the approximatio property AP that were previously discussed. Theorem 1 i Appedix A implies that there are o serious a priori limits for the computatioal power of LSs o cotiuous fuctios of time, ad thereby provides a theoretical foudatio for our approach towards modelig eural computatio. I particular, sice this theorem makes o specific requiremet regardig the exact ature or behaviour of the basis filters, as log as they satisfy the separatio property (for the iputs i questio), it provides theoretical support for employig istead of circuits that were costructed 3 A filter F is called time ivariat if ay temporal shift of the iput fuctio u ( ) by some amout t 0 causes a temporal shift of the output fuctio y = Fu by the same amout t 0, i.e., ( Fu t 0 )( t) = ( Fu)( t + t 0) for all t,t 0 R, where u t 0 ( t) : = u( t + t 0). Note that if U is closed uder R k temporal shifts, the a time ivariat filter F : U ( R ) ca be idetified uiquely by the values y ( 0) = ( Fu)(0) of its output fuctios y ( ) at time 0. 4 Fadig memory (Boyd at al., 1985) is a cotiuity property of filters F which demads that for ay iput fuctio u ) U ay other iput fuctios ( the output )(0) v ) U [ T,0]. Formally oe defies that (Fu ca be approximated by the outputs (Fv)(0) for ( that approximate ( ) F : U ) u o a sufficietly log time iterval R k ( R has fadig memory if for every u U ad every ε > 0 there exist δ > 0 ad T > 0 so that ( Fv )(0) ( Fu)(0) < ε for all v U with u ( t) v( t) <δ for all t [ T,0]. Iformally a filter F has fadig memory if the most sigificat bits of its curret output value (Fu)(0) deped just o the most sigificat bits of the values of its iput fuctio ( ) T,0 ito the past. Thus, i order to compute the most sigificat bits of (Fu)(0) it is ot ecessary to kow the precise value of the iput fuctio u (s) for ay time s, ad it is also ot ecessary to kow aythig about the values of u ( ) for more tha a fiite time iterval back ito the past. Note that a filter that has fadig memory is automatically causal. u from some fiite time widow [ ] 7
8 for a specific task, partially evolved or eve rather arbitrary foud computatioal modules for purposeful computatios. This feature highlights a importat differece to computatioal theories based o Turig machies or fiite state machies, which are ofte used as coceptual basis for modelig eural computatio. The mathematical theory of LSs ca also be exteded to cover computatio o spike u of the iput u( ) trais (discrete evets i cotiuous time) as iputs. Here the i th compoet ( ) is a fuctio that assumes oly the values 0 ad 1, with ( t) = 1 at time t. Thus ( ) u i if the i th precedig euro fires u is ot a cotiuous fuctio but a sequece of poit evets. Theorem 2 i i Appedix A provides a theoretical foudatio for approximatig ay biologically relevat computatio o spike trais by LSs. i Neural icrocircuits as Implemetatios of LSs I order to test the applicability of this coceptual framework to modelig computatio i eural microcircuits, we carried out computer simulatios where a geeric recurret circuit of itegrateadfire euros (see Appedix B for details) was employed as liquid filter. I other words: computer models for eural microcircuits were viewed as implemetatio of the liquid filter L of a LS. I order to test the theoretically predicted uiversal realtime computig capabilities of these eural implemetatios of LSs, we evaluated their performace o a wide variety of challegig bechmark tasks. The iput to the eural circuit was via oe or several iput spike trais, which diverged to iject curret ito 30% radomly chose liquid euros. The amplitudes of the iput syapses were chose from a Gaussia distributio, so that each euro i the liquid circuit received a slightly differet iput (a form of topographic ijectio). The liquid state of the eural microcircuit at time t was defied as all iformatio that a readout euro could extract at time t from the circuit, i.e. the output at time t of all the liquid euros represeted the curret liquid state of this istatiatio of a LS. ore precisely, sice the readout euros were modeled as I&F euros with a biologically realistic membrae time costat of 30 ms, the liquid state x (t) at time t cosisted of the vector of cotributios of all the liquid euros to the membrae potetial at time t of a geeric readout euro (with uit syaptic weights). athematically this liquid state x (t) ca be defied as the vector of output values at time t of liear filters with expoetial decay (time costat 30 ms) applied to the spike trais emitted by the liquid euros. Each readout map f was implemeted by a separate populatio P of itegrateadfire euros (referred to as "readout euros") that received iput from all the "liquid euros", but had o lateral or recurret coectios 5. The curret firig activity p(t) of the populatio P, that is the fractio of euros i P firig durig a time bi of 20ms, was iterpreted as the aalog output of f at time t (oe ofte refers to such represetatio of aalog values by the curret firig activity i a pool of euros as space rate codig). Theoretically the class of readout maps that ca be implemeted i this fashio satisfies the approximatio property AP (aass, 2000, Auer et al., 2001), ad is accordig to Theorem 1 i priciple sufficiet for approximatig arbitrary give fadig memory filters F. I cases where a readout with discrete values 1 ad 0 5 For coceptual purposes we separate the liquid ad readout elemets i this paper, although dual liquidreadout fuctios ca also be implemeted. 8
9 suffices, oe ca implemet a readout map eve by a sigle I&F euro that represets these discrete output values by firig/ofirig at time t. I cases where the target output cosists of slowly varyig aalog values, a sigle readout euro ca be traied to represet these values through its timevaryig firig rate. I ay case the readout euros ca be traied to perform a specific task by adjustig the stregths of syapses projected oto them from the liquid euros usig a perceptrolike local learig rule (Auer et al., 2001). The fial leared state of the readout euros eables them to take particular weighted sums of the curret outputs x (t) of the liquid euros ad geerate a respose f ( x (t) ) that approximates the target value y(t). As a first test of these eural implemetatios of LSs we evaluated the separatio property SP of computer models for eural microcircuits o spike trai iputs. A large set of pairs of Poisso spike trais u( ) ad v( ) were radomly geerated ad ijected (i separate trials) as iput to the recurret eural circuit. The resultig trajectories x ( ) ad Figure 2: Average distace of liquid states for two differet iput spike trais u ad v (give as iput to the eural circuit i separate trials, each time with a idepedetly chose radom iitial state of the eural circuit, see Appedix B) plotted as a fuctio of time t. The state distace icreases with the distace d(u,v) betwee the two iput spike trais u ad v. Plotted o the yaxis is the average value of x ( t) x ( t) u v, where. deotes the Euclidea orm, ad x u (t), (t) x v deote the liquid states at time t for iput spike trais u ad v. The plotted results for the values 0.1, 0.2, 0.4 of the iput differece d' represet the average over 200 radomly geerated pairs u ad v of spike trais such that d ' d( u, v) < Parameters of the liquid: 1 colum, degree of coectivity λ = 2 (see Appedix B for details). x v ( ) of liquid states of the recurret circuit were recorded for each of these timevaryig iputs u( ) ad v( ). The average distace x ( t) x ( t) betwee these liquid states was plotted i Figure 2 as a fuctio of the time t after the oset of the iput, for various fixed values of the distace d(u,v) 6 betwee the two spike trai iputs u ad v. These curves show that the distace betwee these liquids states is well above the oise level, i.e. above the average liquid state differeces caused by the same spike trai applied with two differet radomly chose iitial coditios of the circuit (idicated by the solid curve). Furthermore these curves show that the differece i liquid states is after the first 30 ms roughly proportioal to the distace betwee the correspodig iput 6 I order to defie the distace d ( u, v) betwee two spike trais u ad v we replaced each 2 spike by a Gaussia exp( ( t / τ) ) for τ = 5ms (to be precise, u ad v are covolved with the 2 Gaussia kerel exp( ( t / τ) ) ) ad defied d ( u, v) as the distace of the resultig two cotiuous fuctios i the L2orm (divided by the maximal legths 0.5 s of the spike trais u ad v). u u v 9
10 spike trais. Note i particular the absece of chaotic effects for these geeric eural microcircuit models with biologically realistic itermediate coectio legths. Explorig the Computatioal Power of odels for Neural icrocircuit As a first test of its computatioal power this simple geeric circuit was applied to a previously cosidered classificatio task (Hopfield & Brody, 2001), where spoke words were represeted by oisecorrupted spatiotemporal spike patters over a rather log time iterval (40chael spike patters over 0.5s). This classificatio task had bee solved i (Hopfield & Brody, 2001) by a etwork of euros desiged for this task (relyig o ukow mechaisms that could provide smooth decays of firig activity over loger time periods, ad apparetly requirig substatially larger etworks of I&F euros if fully implemeted with I&F euros). The architecture of that etwork, which had bee customized for this task, limited its classificatio power to spike trais cosistig of a sigle spike per chael. We foud that the same, but also a more geeral versio of this spatiotemporal patter recogitio task that allowed several spikes per iput chael, ca be solved by a geeric recurret circuit as described i the previous sectio. Furthermore the output of this etwork was available at ay time, ad was usually correct as soo as the liquid state of the eural circuit had absorbed eough iformatio about the iput (the iitial value of the correctess just reflects the iitial guess of the readout). Formally we defied the correctess of the eural readout at time s by the term 1 target output y(s) readout activity p(s), where the target output y(s) cosisted i this case of the costat values 1 or 0, depedig o the iput patter. Plotted i Fig. 3 is for ay time t durig the presetatio of the iput patters i additio to the correctess as a fuctio of t also the certaity of the output at time t, which is defied as the average correctess up to that time t. Whereas the etwork costructed by Hopfield ad Brody was costructed to be ivariat with regard to liear time warpig of iputs (provided that oly oe spike arrives i each chael), the readouts of the geeric recurret circuit that we cosidered could be traied to be ivariat with regard to a large class of differet types of oises. The results show i Fig. 3 are for a oise where each iput spike is moved idepedetly by a amout draw from a Gaussia distributio with mea 0 ad SD 32 ms. 10
11 Figure 3: Applicatio of a geeric recurret etwork of I&F euros modeled as LS to a more difficult versio of a wellstudied classificatio task (Hopfield & Brody, 2001). Five radomly draw patters (called zero, "oe ", "two",..), each cosistig of 40 parallel Poisso spike trais over 0.5s, were chose. Five readout modules, each cosistig of 50 itegrateadfire euros, were traied with 20 oisy versios of each iput patter to respod selectively to oisy versios of just oe of these patters (oise was ijected by radomly movig each spike by a amout draw idepedetly from a Gaussia distributio with mea 0 ad variace 32ms; i additio the iitial state of the liquid euros was chose radomly at the begiig of each trial). The resposes of the readout which had bee traied * to detect the patter "zero" is show for a ew, previously ot show, oisy versios of two of the iput patters. The correctess ad certaity (= average correctess so far) are show as fuctios of time from the oset of the stimulus at the bottom. The correctess is calculated as 1 p(t) y(t) where p(t) is the ormalized firig activity i the readout pool (ormalized to the rage [0 1]; 1 correspodig to a activity of 180Hz; biwidth 20ms) ad y(t) is the target output. (Correctess starts at a level of 0 for patter zero where this readout pool is supposed to become active, ad at a value of 1 for patter oe, because the readout pool starts i a iactive state). I cotrast to most circuits of spikig euros that have bee costructed for specific computatioal task, the spike trais of liquid ad readout euros show i this figure look rather realistic. * The familiar deltarule was applied or ot applied to each readout euro, depedig o whether the curret firig activity i the readout pool was too high, too low, or about right, thus requirig at most two bits of global commuicatio. The precise versio of the learig rule was the pdelta rule that is discussed i Auer et al., (2001). 11
12 Givig a costat output for a timevaryig liquid state (caused by a timevaryig iput) is a serious challege for a LS, sice it caot rely o attractor states, ad the memoryless readout has to trasform the trasiet ad cotiuously chagig states of the liquid ito a stable output (see the discussio below ad Fig. 9 for details). I order to explore the limits of this simple eural implemetatio of a LS for computig o timevaryig iput, we chose aother classificatio task where all iformatio of the iput is cotaied i its temporal evolutio, more precisely i the iterspike itervals of a sigle iput spike trai. I this test, 8 radomly geerated Poisso spike trais over 250 ms, or equivaletly 2 Poisso spike trais over 1000 ms partitioed Figure 4: Evaluatig the fadig memory of a geeric eural microcircuit: the task. I this more challegig classificatio task all spike trais are of legth 1000 ms ad cosist of 4 segmets of legth 250 ms each. For each segmet 2 templates were geerated radomly (Poisso spike trai with a frequecy of 20 Hz); see upper traces. The actual iput spike trais of legth 1000 ms used for traiig ad testig were geerated by choosig for each segmet oe of the two associated templates, ad the geeratig a oisy versio by movig each spike by a amout draw from a Gaussia distributio with mea 0 ad a SD that we refer to as jitter (see lower trace for a visualizatio of the jitter with a SD of 4 ms). The task is to output with 4 differet readouts at time t = 1000 ms for each of the precedig 4 iput segmets the umber of the template from which the correspodig segmet of the iput was geerated. Results are summarized i Figures 5 ad 6. ito 4 segmets each (see top of Figure 4), were chose as template patters. Other spike trais over 1000 ms were geerated by choosig for each 250 ms segmet oe of the two templates for this segmet, ad by jitterig each spike i the templates (more precisely: each spike was moved by a amout draw from a Gaussia distributio with mea 0 ad a SD that we refer to as jitter, see bottom of Figure 4). A typical spike trai geerated i this way is show i the middle of Figure 4. Because of the oisy dislocatio of spikes it was impossible to recogize a specific template from a sigle iterspike iterval (ad there were o spatial cues cotaied i this sigle chael iput). Istead, a patter formed by several iterspike itervals had to be recogized ad classified retrospectively. Furthermore readouts were ot oly traied to classify at time t = 1000 ms (i.e., at after the iput spike trai had etered the circuit) the template from which the last 250 ms segmet of this iput spike trai had bee geerated, but other readouts were traied to classify simultaeously also the templates from which precedig segmets of the iput (which had etered the circuit several hudred ms earlier) had bee geerated. Obviously 12
13 the latter classificatio task is substatially more demadig, sice the correspodig earlier segmets of the iput spike trai may have left a clear trace i the curret firig activity of the recurret circuit just after they had etered the circuit, but this trace was subsequetly overwritte by the ext segmets of the iput spike trai (which had o correlatio with the choice of the earlier segmets). Altogether there were i this experimet 4 readouts f 1 to f 4, where f i had bee traied to classify at time t = 1000 ms the ith idepedetly chose 250 ms segmet of the precedig iput spike trai. The performace of the LS, with a geeric recurret etwork of 135 I&F euros as liquid filter (see Appedix B), was evaluated after traiig of the readout pools o iputs from the same distributio (for jitter = 4 ms), but with a example that the LS had ot see before. The accuracy of the 4 readouts is plotted i pael A of Figure 5. It demostrates the fadig memory of a geeric recurret circuit of I&F euros, where iformatio about iputs that occurred several hudred ms ago ca be recovered eve after that iput segmet was subsequetly overwritte. Sice readout euros (ad euros withi the liquid circuit) were modeled with a realistic time costat of just 30 ms, the questio arises where this iformatio about earlier iputs had bee stored for Figure 5: Evaluatig the fadig memory of a geeric eural microcircuit: results. 4 readout modules f 1 to f 4, each cosistig of a sigle perceptro, were traied for their task by liear regressio. The readout module f i was traied to output 1 at time t=1000 ms if the ith segmet of the previously preseted iput spike trai had bee costructed from the correspodig template 1, ad to output 0 at time t=1000 ms otherwise. Correctess (percetage of correct classificatio o a idepedet set of 500 iputs ot used for traiig) is calculated as average over 50 trials. I each trial ew Poisso spike trais were draw as templates, a ew radomly coected circuit was costructed (1 colum, λ=2; see Appedix B), ad the readout modules f 1 to f 4 were traied with1000 traiig examples geerated by the distributio described i Figure 4. A: Average correctess of the 4 readouts for ovel test iputs draw from the same distributio. B: Firig activity i the liquid circuit (time iterval [0.5 s, 0.8 s]) for a typical iput spike trai. C: Results of a cotrol experimet where all dyamic syapses i the liquid circuit had bee replaced by static syapses (the mea values of the syaptic stregths were uiformly rescaled so that the average liquid activity is approximately the same as for dyamic syapses). The liquid state of this circuit cotaied substatially less iformatio about earlier iput segmets. D: Firig activity i the liquid circuit with static syapses used for the classificatio results reported i pael C. The circuit respose to each of the 4 iput spikes that etered the circuit durig the observed time iterval [0.5 s, 0.8 s] is quite stereotypical without dyamic syapses (except for the secod iput spike that arrives just 20 ms after the first oe). I cotrast the firig respose of the liquid circuit with dyamic syapses (pael B) is differet for each of the 4 iput spikes, showig that dyamic syapses edow these circuits with the capability to process ew iput differetly depedig o the cotext set by precedig iput, eve if that precedig iput occurred several hudred ms before. 13
14 several hudred ms. As a cotrol we repeated the same experimet with a liquid circuit where the dyamic syapses had bee replaced by static syapses (with syaptic weights that achieved about the same level of firig activity as the circuit with dyamic syapses). Pael C of Fig. 5 shows that this results i a sigificat loss i performace for the classificatio of all except for the last iput segmet. A possible explaatio is provided by the raster plots of firig activity i the liquid circuit with (pael B) ad without dyamic syapses (pael D), show here with high temporal resolutio. I the circuit with dyamic syapses the recurret activity differs for each of the 4 spikes that etered the circuit durig the time period show, demostratig that each ew spike is processed by the circuit i a idividual maer that depeds o the cotext defied by precedig iput spikes. I cotrast, the firig respose is very stereotypical for the same 4 iput spikes i the circuit without dyamic syapses, except for the respose to the secod spike that arrives withi 20 ms of the first oe (see the period betwee 500 ad 600 ms i pael D). This idicates Figure 6: Average correctess depeds o the parameter λ that cotrols the distributio of radom coectios withi the liquid circuit. Plotted is the average correctess (at time t=1000 ms, calculated as average over 50 trials as i Figure 5; same umber of traiig ad test examples) of the readout module f 3 (which is traied to classify retroactively the secod to last segmet of the precedig spike trai) as a fuctio of λ. The bad performace for λ=0 (o recurret coectios withi the circuit) shows that recurret coectios are essetial for achievig a satisfactory separatio property i eural microcircuits. Too large values of λ also decrease the performace because they support a chaotic respose. that the short term dyamics of syapses may play a essetial role i the itegratio of iformatio for realtime processig i eural microcircuits. Figure 6 examies aother aspect of eural microcircuits that appears to be importat for their separatio property: the statistical distributio of coectio legths withi the recurret circuit. Six types of liquid circuits, each cosistig of 135 I&F euros but with differet values of the parameter λ which regulated the average umber of coectios ad the average spatial legth of coectios (see Appedix B), were traied ad evaluated accordig to the same protocol ad for the same task as i Fig. 5. Show i Fig. 6 is for each of these 6 types of liquid circuits the average correctess of the readout f 3 o ovel iputs, after it had bee traied to classify the secod to last segmet of the iput spike trai. The performace was fairly low for circuits without recurret coectios (λ = 0). It also was fairly low for recurret circuits with large values of λ, whose largely legthidepedet distributio of coectios homogeized the microcircuit ad facilitated chaotic behavior. Hece for this classificatio task the ideal liquid circuit is a microcircuit that has i additio to local coectios to eighborig euros also a few lograge coectios, thereby iterpolatig betwee the customarily cosidered extremes of strictly total coectivity (like i a cellular automato) o oe had, ad the localityigorig global coectivity of a Hopfield et o the other had. 14
15 The performace results of eural implemetatios of LSs that were reported i this sectio should ot be viewed as absolute data o the computatioal power of recurret eural circuits. Rather the geeral theory suggests that their computatioal power icreases with ay improvemet i their separatio or approximatio property. Sice the approximatio property AP was already close to optimal for these etworks (icreasig the umber of euros i the readout module did ot icrease the performace sigificatly; ot show), the primary limitatio i performace lay i the separatio property SP. Ituitively it is clear that the liquid circuit eeds to be sufficietly complex to hold the details required for the particular task, but should reduce iformatio that is ot relevat to the task (for example spike time jitter). SP ca be egieered i may ways such as icorporatig euro diversity, implemetig specific syaptic architectures, alterig microcircuit coectivity, or simply recruitig more colums. The last optio is of particular iterest because it is ot available i most computatioal models. It will be explored i the ext sectio. Addig Computatioal Power A iterestig structural differece betwee eural systems ad our curret geeratio of artificial computig machiery is that the computatioal power of eural systems ca apparetly be elarged by recruitig more circuitry (without the eed to rewire old or ew circuits). We explored the cosequeces of recruitig additioal colums for eural implemetatios of LSs (see pael B of Fig. 7), ad compared it with the optio of just addig further coectios to the Figure 7: Separatio property ad performace of liquid circuits with larger umbers of coectios or euros. A ad B: Schematic drawigs of LSs cosistig of oe colum (A) ad four colums (B). Each colum cosists of = 135 I&F euros. C: Separatio property depeds o the structure of the liquid. Average state distace (at time t = 100 ms) calculated as described i Figure 2. A colum with high iteral coectivity (high λ) achieves higher separatio as a sigle colum with lower coectivity, but teds to chaotic behavior where it becomes equally sesitive to small ad large iput differeces d(u,v). O the other had the characteristic curve for a liquid cosistig of 4 colums with small λ is lower for values of d(u,v) lyig i the rage of jittered versios u ad v of the same spike trai patter (d(u,v) 0.1 for jitter 8 ms) ad higher for values of d(u,v) i the rage typical for spike trais u ad v from differet classes (mea: 0.22). D: Evaluatio of the same three types of liquid circuits for oise robust classificatio. Plotted is the average performace for the same task as i Fig. 6, but for various values of the jitter i iput spike times. Several colums (ot itercoected) with low iteral coectivity yield a better performig implemetatio of a LS for this computatioal task, as predicted by the aalysis of their separatio property. 15
16 primary oecolumliquid that we used so far (135 I&F euros with λ = 2, see pael A of Fig. 7). Pael C of Fig. 7 demostrates that the recruitmet of additioal colums icreases the separatio property of the liquid circuit i a desirable maer, where the distace betwee subsequet liquid states (always recorded at time t = 1000 ms i this experimet) is proportioal to the distace betwee the spike trai iputs that had previously etered the liquid circuit (spike trai distace measured i the same way as for Fig. 2). I cotrast the additio of more coectios to a sigle colum (λ = 8, see Appedix B) also icreases the separatio betwee subsequet liquid states, but i a quasichaotic maer where small iput distaces cause about the same distaces betwee subsequet liquid states as small iput differeces. I particular the subsequet liquid state distace is about equally large for two jittered versios of the iput spike trai state (yieldig typically a value of d(u,v) aroud 0.1) as for sigificatly differet iput spike trais that require differet outputs of the readouts. Thus improvig SP by alterig the itrisic microcircuitry of a sigle colum icreases sesitivity for the task, but also icreases sesitivity to oise. The performace of these differet types of liquid circuits for the same classificatio task as i Fig. 6 is cosistet with this aalysis of their characteristic separatio property. Show i pael D of Fig. 7 is their performace for various values of the spike time jitter i the iput spike trais. The optimizatio of SP for a specific distributio of iputs ad a specific group of readout modules is likely to arrive at a specific balace betwee the itrisic complexity of the microcircuitry ad the umber of repeatig colums. Parallel Computig i RealTime o Novel Iputs Sice the liquid of the LS does ot have to be traied for a particular task, it supports parallel computig i realtime. This was demostrated by a test i which multiple spike trais were ijected ito the liquid ad multiple readout euros were traied to perform differet tasks i parallel. We added 6 readout modules to a liquid cosistig of 2 colums with differet values of λ 7. Each of the 6 readout modules was traied idepedetly for a completely differet olie task that required a output value at ay time t. We focused here o tasks that require diverse ad rapidly chagig aalog output resposes y(t). Figure 8 shows that after traiig each of these 6 tasks ca be performed i realtime with high accuracy. The performace show is for a ovel iput that was ot draw from the same distributio as the traiig examples, ad differs i several aspects from the traiig examples (thereby demostratig the possibility of extrageeralizatio i eural microcircuits, due to their iheret bias, that goes beyod the usual defiitio of geeralizatio i statistical learig theory). ReadoutAssiged Equivalet States of a Dyamical System Realtime computatio o ovel iputs implies that the readout must be able to geerate a ivariat or appropriately scaled respose for ay iput eve though the liquid state may ever repeat. Ideed, Figure 3 showed already that the dyamics of readout pools ca become quite idepedet from the dyamics of the liquid eve though the liquid euros are the oly source 7 I order to combie high sesitivity with good geeralizatio performace we chose here a liquid cosistig of two colums as before, oe with λ=2, the other with λ=8 ad the iterval [ ] for the uiform distributio of the ospecific backgroud curret I b. 16
17 of iput. To examie the uderlyig mechaism for this relatively idepedet readout respose, we reexamied the readout pool from Figure 3. Whereas the firig activity withi the liquid circuit was highly dyamic, the firig activity i the readout pool was almost costat after traiig. The stability of the readout respose does ot simply come about because the readout oly samples a few uusual liquid euros as show by the distributio of syaptic weights oto a sample readout euro (Figure 9F). Sice the syaptic weights do ot chage after learig, this idicates that the readout euros have leared to defie a otio of equivalece for dyamic states of the liquid. Ideed, equivalece classes are a ievitable cosequece of collapsig the high dimesioal space of liquid states ito a sigle dimesio, but what is surprisig is that the equivalece classes are meaigful i terms of the task, allowig ivariat ad appropriately scaled readout resposes ad therefore realtime computatio o ovel iputs. Furthermore, while the iput rate may cotai saliet iformatio that is costat for a particular readout elemet, it may ot be for aother (see for example Fig. 8), idicatig that equivalece classes ad dyamic stability exist purely from the perspective of the readout elemets. 17
18 Figure 8: ultitaskig i realtime. 4 iput spike trais of legth 2 s (show at the top) are ijected ito a liquid module cosistig of 2 colums (radomly costructed with the same parameters; see Appedix B), which is coected to multiple readout modules. Each readout module is traied to extract iformatio for a differet realtime computig task. The target fuctios are plotted as dashed lie, ad populatio respose of the correspodig readout module as solid lie. The tasks assiged to the 6 readout modules were the followig: Represet the sum of rates: at time t, output the sum of firig rates of all 4 iput spike trais withi the last 30ms. Represet the itegral of the sum of rates: at time t, output the total activity i all 4 iputs itegrated over the last 200ms. Patter detectio: output a high value if a specific spatio temporal spike patter appears. Represet a switch i spatial distributio of rates: output a high value if a specific iput patter occurs where the rate of iput spike trais 1 ad 2 goes up ad simultaeously the rate of iput spike trais 3 ad 4 goes dow, otherwise remai low. Represet the firig correlatio: at time t, output the umber of spike coicideces (ormalized ito the rage [0 1]) durig the last 75 ms for iputs 1 ad 3 ad separately for iputs 1 ad 2. Target readout values are plotted as dashed lies, actual outputs of the readout modules as solid lies, all i the same time scale as the 4 spike trais show at the top that eter the liquid circuit durig this 2 s time iterval. Results show are for a ovel iput that was ot draw from the same distributio as the traiig examples. 150 traiig examples were draw radomly from the followig distributio. Each iput spike trai was a idepedetly draw Possio spike trai with a time varyig rate of r(t) = A+B si (2 π f t + α). The parameters A, B, ad f where draw radomly from the followig itervals (the phase was fixed at α=0 deg): A [0Hz, 30Hz] ad [70Hz, 100Hz], B [0Hz, 30Hz] ad [70Hz, 100Hz], f [0.5Hz, 1Hz] ad [3Hz, 5Hz]. O this backgroud activity 4 differet patters had bee superimposed (always i the same order durig traiig): rate switch to iputs 1 ad 3, a burst patter, rate switch to iputs 1 ad 2, ad fially a spatio temporal spike patter. The results show are for a test iput that could ot be geerated by the same distributio as the traiig examples, because its base level (A=50Hz), as well as the amplitude (B=50Hz), frequecy (f=2hz) ad phase (α=180 deg) of the uderlyig time varyig firig rate of the Poisso iput spike trais were chose to lie i the middle of the gaps betwee the two itervals that were used for these parameters durig traiig. Furthermore the spatiotemporal patters (a burst patter, rate switch to iputs 1 ad 3, ad rate switch to iputs 1 ad 2), that were superimposed to achieve more iput variatio withi the observed 2 s, ever occured i this order ad at these time poits for ay traiig iput. Hece the accurate performace for this ovel iput demostrates substatial geeralizatio capabilities of the readouts after traiig. 18
19 Discussio We itroduce the liquid state machie, a ew paradigm for realtime computig o timevaryig iput streams. I cotrast to most computatioal models it does ot require the costructio of a circuit or program for a specific computatioal task. Rather, it relies o priciples of highdimesioal dyamical systems ad learig theory that allow it to adapt uspecific evolved or foud recurret circuitry for a give computatioal task. Sice oly the readouts, ot the recurret circuit itself, have to be adapted for specific computatioal tasks, the same recurret circuit ca support completely differet realtime computatios i parallel. The uderlyig abstract computatioal model of a liquid state machie (LS) emphasizes the importace of perturbatios i dyamical systems for realtime computig, sice eve without stable states or attractors the separatio property ad the approximatio property may edow a dyamical system with virtually ulimited computatioal power o timevaryig iputs. I particular we have demostrated the computatioal uiversality of geeric recurret circuits of itegrateadfire euros (eve with quite arbitrary coectio structure), if viewed as special cases of LSs. Apparetly this is the first stable ad geerally applicable method for usig geeric recurret etworks of itegrateadfire euros to carry out a wide family of complex realtime computatios o spike trais as iputs. Hece this approach provides a platform for explorig the computatioal role of specific aspects of biological eural microcircuits. The computer simulatios reported i this article provide possible explaatios ot oly for the computatioal role of the highly recurret coectivity structure of eural circuits, but also for their characteristic distributio of coectio legths, which places their coectivity structure betwee the extremes of strictly local coectivity (cellular automata or coupled map lattices) ad uiform global coectivity (Hopfield ets) that are usually addressed i theoretical studies. Furthermore our computer simulatios suggest a importat computatioal role of dyamic syapses for realtime computig o timevaryig iputs. Fially, we reveal a most uexpected ad remarkable priciple that readout elemets ca establish their ow equivalece relatioships o highdimesioal trasiet states of a dyamical system, makig it possible to geerate stable ad appropriately scaled output resposes eve if the iteral state ever coverges to a attractor state. I cotrast to virtually all computatioal models from computer sciece or artificial eural etworks, this computatioal model is ehaced rather tha hampered by the presece of diverse computatioal uits. Hece it may also provide isight ito the computatioal role of the complexity ad diversity of euros ad syapses (see for example (Gupta et al., 2000)). While there are may plausible models for spatial aspects of eural computatio, a biologically realistic framework for modelig temporal aspects of eural computatio has bee missig. I cotrast to models ispired by computer sciece, the liquid state machie does ot try to reduce these temporal aspects to trasitios betwee stable states or limit cycles, ad it does ot require delay lies or buffers. Istead it proposes that the trajectory of iteral states of a recurret eural circuit provides a raw, ubiased, ad uiversal source of temporally itegrated iformatio, from which specific readout elemets ca extract specific iformatio about past iputs for their idividual task. Hece the otorious trialtotrial stimulus respose variatios i sigle ad populatios of euros observed experimetally, may reflect a accumulatio of iformatio from previous iputs i the trajectory of iteral states, rather tha oise (see also (Arieli et al., 1996)). This would imply that averagig over trials or biig, peels out most of the iformatio processed by recurret microcircuits ad leaves mostly topographic iformatio. 19
20 This approach also offers ew ideas for models of the computatioal orgaisatio of cogitio. It suggests that it may ot be ecessary to scatter all iformatio about sesory iput by recodig it through feedforward processig as output vector of a esemble of feature Figure 9: Readout assiged equivalet states of a dyamical system. A LS (liquid circuit as i Figure 3) was traied for the classificatio task as described i Figure 3. Results show are for a ovel test iput (draw from the same distributio as the traiig examples). A: The test iput cosists of 40 Poisso spike trais, each with a costat rate of 5 Hz. B: Raster plot of the 135 liquid euros i respose to this iput. Note the large variety of liquid states that arise durig this time period. C: Populatio rate of the liquid (bisize 20 ms). Note that this populatio rate chages quit a bit over time. D: Readout respose (solid lie) ad target respose (dashed lie). The target respose had a costat value of 1 for this iput. The output of the traied readout module is also almost costat for this test example (except for the begiig), although its iput, the liquid states of the recurret circuit, varied quit a bit durig this time period. F: Weight distributio of a sigle readout euro. detectors with fixed receptive fields (thereby creatig the "bidig problem"). It proposes that at the same time more global iformatio about precedig iputs ca be preserved i the trajectories of very high dimesioal dyamical systems, from which multiple readout modules extract ad combie the iformatio eeded for their specific tasks. This approach is evertheless compatible with experimetal data that cofirm the existece of special maps of feature detectors. These 20
SUPPORT UNION RECOVERY IN HIGHDIMENSIONAL MULTIVARIATE REGRESSION 1
The Aals of Statistics 2011, Vol. 39, No. 1, 1 47 DOI: 10.1214/09AOS776 Istitute of Mathematical Statistics, 2011 SUPPORT UNION RECOVERY IN HIGHDIMENSIONAL MULTIVARIATE REGRESSION 1 BY GUILLAUME OBOZINSKI,
More informationCounterfactual Reasoning and Learning Systems: The Example of Computational Advertising
Joural of Machie Learig Research 14 (2013) 32073260 Submitted 9/12; Revised 3/13; Published 11/13 Couterfactual Reasoig ad Learig Systems: The Example of Computatioal Advertisig Léo Bottou Microsoft 1
More informationMAXIMUM LIKELIHOODESTIMATION OF DISCRETELY SAMPLED DIFFUSIONS: A CLOSEDFORM APPROXIMATION APPROACH. By Yacine AïtSahalia 1
Ecoometrica, Vol. 7, No. 1 (Jauary, 22), 223 262 MAXIMUM LIKELIHOODESTIMATION OF DISCRETEL SAMPLED DIFFUSIONS: A CLOSEDFORM APPROXIMATION APPROACH By acie AïtSahalia 1 Whe a cotiuoustime diffusio is
More informationNoisy mean field stochastic games with network applications
Noisy mea field stochastic games with etwork applicatios Hamidou Tembie LSS, CNRSSupélecUiv. Paris Sud, Frace Email: tembie@ieee.org Pedro Vilaova AMCS, KAUST, Saudi Arabia Email:pedro.guerra@kaust.edu.sa
More informationCrowds: Anonymity for Web Transactions
Crowds: Aoymity for Web Trasactios Michael K. Reiter ad Aviel D. Rubi AT&T Labs Research I this paper we itroduce a system called Crowds for protectig users aoymity o the worldwideweb. Crowds, amed for
More informationSOME GEOMETRY IN HIGHDIMENSIONAL SPACES
SOME GEOMETRY IN HIGHDIMENSIONAL SPACES MATH 57A. Itroductio Our geometric ituitio is derived from threedimesioal space. Three coordiates suffice. May objects of iterest i aalysis, however, require far
More informationA Direct Approach to Inference in Nonparametric and Semiparametric Quantile Models
A Direct Approach to Iferece i Noparametric ad Semiparametric Quatile Models Yaqi Fa ad Ruixua Liu Uiversity of Washigto, Seattle Workig Paper o. 40 Ceter for Statistics ad the Social Scieces Uiversity
More informationCahier technique no. 194
Collectio Techique... Cahier techique o. 194 Curret trasformers: how to specify them P. Foti "Cahiers Techiques" is a collectio of documets iteded for egieers ad techicias, people i the idustry who are
More informationDryad: Distributed DataParallel Programs from Sequential Building Blocks
Dryad: Distributed DataParallel Programs from Sequetial uildig locks Michael Isard Microsoft esearch, Silico Valley drew irrell Microsoft esearch, Silico Valley Mihai udiu Microsoft esearch, Silico Valley
More informationConsistency of Random Forests and Other Averaging Classifiers
Joural of Machie Learig Research 9 (2008) 20152033 Submitted 1/08; Revised 5/08; Published 9/08 Cosistecy of Radom Forests ad Other Averagig Classifiers Gérard Biau LSTA & LPMA Uiversité Pierre et Marie
More informationSystemic Risk and Stability in Financial Networks
America Ecoomic Review 2015, 105(2): 564 608 http://dx.doi.org/10.1257/aer.20130456 Systemic Risk ad Stability i Fiacial Networks By Daro Acemoglu, Asuma Ozdaglar, ad Alireza TahbazSalehi * This paper
More informationHOW MANY TIMES SHOULD YOU SHUFFLE A DECK OF CARDS? 1
1 HOW MANY TIMES SHOULD YOU SHUFFLE A DECK OF CARDS? 1 Brad Ma Departmet of Mathematics Harvard Uiversity ABSTRACT I this paper a mathematical model of card shufflig is costructed, ad used to determie
More informationKernel Mean Estimation and Stein Effect
Krikamol Muadet KRIKAMOL@TUEBINGEN.MPG.DE Empirical Iferece Departmet, Max Plack Istitute for Itelliget Systems, Tübige, Germay Keji Fukumizu FUKUMIZU@ISM.AC.JP The Istitute of Statistical Mathematics,
More informationWork Placement in ThirdLevel Programmes. Edited by Irene Sheridan and Dr Margaret Linehan
Work Placemet i ThirdLevel Programmes Edited by Iree Sherida ad Dr Margaret Lieha Work Placemet i ThirdLevel Programmes Edited by Iree Sherida ad Dr Margaret Lieha The REAP Project is a Strategic Iovatio
More informationA General Multilevel SEM Framework for Assessing Multilevel Mediation
Psychological Methods 1, Vol. 15, No. 3, 9 33 1 America Psychological Associatio 18989X/1/$1. DOI: 1.137/a141 A Geeral Multilevel SEM Framework for Assessig Multilevel Mediatio Kristopher J. Preacher
More informationTeaching Bayesian Reasoning in Less Than Two Hours
Joural of Experimetal Psychology: Geeral 21, Vol., No. 3, 4 Copyright 21 by the America Psychological Associatio, Ic. 963445/1/S5. DOI: 1.7//963445..3. Teachig Bayesia Reasoig i Less Tha Two Hours Peter
More informationStéphane Boucheron 1, Olivier Bousquet 2 and Gábor Lugosi 3
ESAIM: Probability ad Statistics URL: http://wwwemathfr/ps/ Will be set by the publisher THEORY OF CLASSIFICATION: A SURVEY OF SOME RECENT ADVANCES Stéphae Bouchero 1, Olivier Bousquet 2 ad Gábor Lugosi
More informationEverything You Always Wanted to Know about Copula Modeling but Were Afraid to Ask
Everythig You Always Wated to Kow about Copula Modelig but Were Afraid to Ask Christia Geest ad AeCatherie Favre 2 Abstract: This paper presets a itroductio to iferece for copula models, based o rak methods.
More informationLeadership Can Be Learned, But How Is It Measured?
Maagemet Scieces for Health NO. 8 (2008) O C C A S I O N A L PA P E R S Leadership Ca Be Leared, But How Is It Measured? How does leadership developmet cotribute to measurable chages i orgaizatioal performace,
More informationSoftware Reliability via RuTime ResultCheckig Hal Wasserma Uiversity of Califoria, Berkeley ad Mauel Blum City Uiversity of Hog Kog ad Uiversity of Califoria, Berkeley We review the eld of resultcheckig,
More informationType Less, Find More: Fast Autocompletion Search with a Succinct Index
Type Less, Fid More: Fast Autocompletio Search with a Succict Idex Holger Bast MaxPlackIstitut für Iformatik Saarbrücke, Germay bast@mpiif.mpg.de Igmar Weber MaxPlackIstitut für Iformatik Saarbrücke,
More informationMeanSemivariance Optimization: A Heuristic Approach
57 MeaSemivariace Optimizatio: A Heuristic Approach Javier Estrada Academics ad practitioers optimize portfolios usig the meavariace approach far more ofte tha the measemivariace approach, despite the
More informationThe Nonparanormal: Semiparametric Estimation of High Dimensional Undirected Graphs
Joural of Machie Learig Research 0 2009 22952328 Submitted 3/09; Revised 5/09; ublished 0/09 The Noparaormal: Semiparametric Estimatio of High Dimesioal Udirected Graphs Ha Liu Joh Lafferty Larry Wasserma
More informationTHE PROBABLE ERROR OF A MEAN. Introduction
THE PROBABLE ERROR OF A MEAN By STUDENT Itroductio Ay experimet may he regarded as formig a idividual of a populatio of experimets which might he performed uder the same coditios. A series of experimets
More informationWHICH MEAN DO YOU MEAN? AN EXPOSITION ON MEANS
WHICH MEAN DO YOU MEAN? AN EXPOSITION ON MEANS A Thesis Submitted to the Graduate Faculty of the Louisiaa State Uiversity ad Agricultural ad Mechaical College i partial fulfillmet of the requiremets for
More informationThe Unicorn, The Normal Curve, and Other Improbable Creatures
Psychological Bulleti 1989, Vol. 105. No.1, 156166 The Uicor, The Normal Curve, ad Other Improbable Creatures Theodore Micceri 1 Departmet of Educatioal Leadership Uiversity of South Florida A ivestigatio
More informationJ. J. Kennedy, 1 N. A. Rayner, 1 R. O. Smith, 2 D. E. Parker, 1 and M. Saunby 1. 1. Introduction
Reassessig biases ad other ucertaities i seasurface temperature observatios measured i situ sice 85, part : measuremet ad samplig ucertaities J. J. Keedy, N. A. Rayer, R. O. Smith, D. E. Parker, ad M.
More informationG r a d e. 5 M a t h e M a t i c s. Number
G r a d e 5 M a t h e M a t i c s Number Grade 5: Number (5.N.1) edurig uderstadigs: the positio of a digit i a umber determies its value. each place value positio is 10 times greater tha the place value
More informationFederal Reserve Bank of New York Staff Reports
Federal Reserve Bak of New York Staff Reports Crime, House Prices, ad Iequality: The Effect of UPPs i Rio Claudio Frischtak Bejami R. Madel Staff Report o. 542 Jauary 2012 This paper presets prelimiary
More informationChild: Family: Place: Radical efficiency to improve outcomes for young children. Croydon
Child: Family: Place: Radical efficiecy to improve outcomes for youg childre Croydo 2 cotets Foreword...5 Executive summary...7 1 Cotext...13 The public sector eeds ew models to improve public services...14
More information