Neural Networks. Hervé Abdi. The University of Texas at Dallas

Transcription

1 Neural Netorks. Hervé Abd The Unversty o Texas at Dallas Introducton Neural netorks are adaptve statstcal models based on an analogy th the structure o the bran. They are adaptve because they can learn to estmate the parameters o some populaton usng a small number o exemplars (one or a e) at a tme. They do not der essentally rom standard statstcal models. For example, one can nd neural netork archtectures akn to dscrmnant analyss, prncpal component analyss, logstc regresson, and other technques. In act, the same mathematcal tools can be used to analyze standard statstcal models and neural netorks. Neural netorks are used as statstcal tools n a varety o elds, ncludng psychology, statstcs, engneerng, econometrcs, and even physcs. They are used also as models o cogntve processes by neuro- and cogntve scentsts. Bascally, neural netorks are bult rom smple unts, sometmes called neurons or cells by analogy th the real thng. These unts are lnked by a set o eghted connectons. Learnng s usually accomplshed by modcaton o the connecton eghts. Each unt codes or corresponds to a eature or a characterstc o a pattern that e ant to analyze or that e ant to use as a predctor. These netorks usually organze ther unts nto several layers. The rst layer s called the nput layer, the last one the output layer. The ntermedate layers ( any) are called the hdden layers. The normaton to be analyzed s ed to the neurons o the rst layer and then propagated to the neurons o the second layer or urther processng. The result o ths processng s then propagated to the next layer and so on untl the last layer. Each unt receves some normaton rom other unts (or rom the external orld through some devces) and processes ths normaton, hch ll be converted nto the output o the unt. The goal o the netork s to learn or to dscover some assocaton beteen nput and output patterns, or to analyze, or to nd the structure o the nput patterns. The learnng process s acheved through the modcaton o the connecton eghts beteen unts. In statstcal terms, ths s equvalent to In: Les-Beck M., Bryman, A., Futng T. (Eds.) (2003). Encyclopeda o Socal Scences Research Methods. Thousand Oaks (CA): Sage. Address correspondence to Hervé Abd Program n Cognton and Neuroscences, MS: Gr.4., The Unversty o Texas at Dallas, Rchardson, TX , USA E-mal: herve@utdallas.edu herve

2 nterpretng the value o the connectons beteen unts as parameters (e.g., lke the values o a and b n the regresson equaton ŷ = a + bx) to be estmated. The learnng process speces the algorthm used to estmate the parameters. The buldng blocks o neural netorks Neural netorks are made o basc unts (see Fgure ) arranged n layers. A unt collects normaton provded by other unts (or by the external orld) to hch t s connected th eghted connectons called synapses. These eghts, called synaptc eghts multply (.e., amply or attenuate) the nput normaton: A postve eght s consdered exctatory, a negatve eght nhbtory. x Bas cell x 0 = Input x I a =Σ 0 x = θ a (a) Output x I Computaton o the actvaton Transormaton o the actvaton The Basc Neural Unt Fgure : The basc neural unt processes the nput normaton nto the output normaton. Each o these unts s a smpled model o a neuron and transorms ts nput normaton nto an output response. Ths transormaton nvolves to steps: Frst, the actvaton o the neuron s computed as the eghted sum o t nputs, and second ths actvaton s transormed nto a response by usng a transer uncton. Formally, each nput s denoted x, and each eght, then the actvaton s equal to a = x, and the output denoted o s obtaned as o = (a). Any uncton hose doman s the real numbers can be used as a transer uncton. The most popular ones are the lnear uncton (o a), the step uncton (actvaton values less than a gven threshold [ are set to 0 or to ] and the other values are set to +), the logstc uncton (x) = + exp{ x} hch maps the real numbers nto the nterval [ + ] and hose dervatve, needed or learnng, s easly computed { (x) = (x)[ (x)]}, and the normal or Gaussan uncton [o = (σ 2π) exp{ 2 (a/σ)2 }]. Some o these unctons can nclude probablstc varatons; or example, a neuron can transorm ts actvaton nto the response + th a probablty o 2 hen the actvaton s larger than a gven threshold. 2

3 The archtecture (.e., the pattern o connectvty) o the netork, along th the transer unctons used by the neurons and the synaptc eghts, completely specy the behavor o the netork. Learnng rules Neural netorks are adaptve statstcal devces. Ths means that they can change teratvely the values o ther parameters (.e., the synaptc eghts) as a uncton o ther perormance. These changes are made accordng to learnng rules hch can be characterzed as supervsed (hen a desred output s knon and used to compute an error sgnal) or unsupervsed (hen no such error sgnal s used). The Wdro-Ho (a.k.a., gradent descent or Delta rule) s the most dely knon supervsed learnng rule. It uses the derence beteen the actual nput o the cell and the desred output as an error sgnal or unts n the output layer. Unts n the hdden layers cannot compute drectly ther error sgnal but estmate t as a uncton (e.g., a eghted average) o the error o the unts n the ollong layer. Ths adaptaton o the Wdro-Ho learnng rule s knon as error backpropagaton. Wth Wdro-Ho learnng, the correcton to the synaptc eghts s proportonal to the error sgnal multpled by the value o the actvaton gven by the dervatve o the transer uncton. Usng the dervatve has the eect o makng nely tuned correctons hen the actvaton s near ts extreme values (mnmum or maxmum) and larger correctons hen the actvaton s n ts mddle range. Each correcton has the mmedate eect o makng the error sgnal smaller a smlar nput s appled to the unt. In general, supervsed learnng rules mplement optmzaton algorthms akn to descent technques because they search or a set o values or the ree parameters (.e., the synaptc eghts) o the system such that some error uncton computed or the hole netork s mnmzed. The Hebban rule s the most dely knon unsupervsed learnng rule, t s based on ork by the Canadan neuropsychologst Donald Hebb, ho theorzed that neuronal learnng (.e., synaptc change) s a local phenomenon expressble n terms o the temporal correlaton beteen the actvaton values o neurons. Speccally, the synaptc change depends on both presynaptc and postsynaptc actvtes and states that the change n a synaptc eght s a uncton o the temporal correlaton beteen the presynaptc and postsynaptc actvtes. Speccally, the value o the synaptc eght beteen to neurons ncreases henever they are n the same state and decreases hen they are n derent states. Some mportant neural netork archtecture One the most popular archtectures n neural netorks s the mult-layer perceptron (see Fgure 2). Most o the netorks th ths archtecture use the Wdro-Ho rule as ther learnng algorthm and the logstc uncton as the transer uncton o the unts o the hdden layer (the transer uncton s n general non-lnear or these neurons). These netorks are very popular because they can approxmate any multvarate uncton relatng the nput to the 3

4 output. In a statstcal rameork, these netorks are akn to multvarate non-lnear regresson. When the nput patterns are the same are the output patterns, these netorks are called auto-assocators. They are closely related to lnear ( the hdden unts are lnear) or non-lnear ( not) prncpal component analyss and other statstcal technques lnked to the general lnear model (see Abd et al., 996), such as dscrmnant analyss or correspondence analyss. Input Output pattern pattern Input layer Hdden layer Output layer Fgure 2: A mult-layer perceptron. A recent development generalzes the radal bass uncton netorks (rb) (see Abd, Valentn, & Edelman, 999) and ntegrates them th statstcal learnng theory (see Vapnk, 999) under the name o support vector machne or SVM (see Schölkop & Smola, 2003). In these netorks, the hdden unts (called the support vectors) represent possble (or even real) nput patterns and ther response s a uncton to ther smlarty to the nput pattern under consderaton. The smlarty s evaluated by a kernel uncton (e.g., dot product; n the radal bass uncton the kernel s the Gaussan transormaton o the Eucldean dstance beteen the support vector and the nput). In the specc case o rb netorks that e ll use as an example o SVM the output o the unts o the hdden layers are connected to an output layer composed o lnear unts. In act, these netorks ork by breakng the dcult problem o a nonlnear approxmaton nto to more smple ones. The rst step s a smple nonlnear mappng (the Gaussan transormaton o the dstance rom the kernel to the nput pattern), the second step corresponds to a lnear transormaton rom the hdden layer to the output layer. Learnng occurs at the level o the output layer. The man dculty th these archtectures resdes n the choce o the support vectors and the specc kernels to use. These netorks are used or pattern recognton, classcaton, and or clusterng data. Valdaton From a statstcal pont a ve, neural netorks represent a class o nonparametrc adaptve models. In ths rameork, an mportant ssue s to evaluate the perormance o the model. Ths s done by separatng the data nto to sets: the tranng set and the testng set. The parameters (.e., the value o the synaptc eghts) o the netork are computed usng the tranng set. Then 4

5 learnng s stopped and the netork s evaluated th the data rom the testng set. Ths cross-valdaton approach s akn to the bootstrap or the jackkne. Useul reerences Neural netorks theory connects several domans rom the neuroscences to engneerng ncludng statstcal theory. Ths dversty o sources creates also a real heterogenety n the presentaton o the materal as textbooks oten try to address only one porton o the nterested readershp. The ollong reerences should be o nterest or the reader nterested n the statstcal propertes o neural netorks: Abd et al. (999), Bshop (995), Cherkassky and Muler (998), Duda, Hart & Stork (200), Haste, Tbshran, & Fredman (2002), Looney (997), Rpley (996), and Vapnk (999). *Reerences [] Abd, H., Valentn, D., & Edelman, B. (999). Neural netorks. Thousand Oaks (CA): Sage. [2] Abd, H., Valentn, D., Edelman, B., O Toole. A.J. (996). A Wdro-Ho learnng rule or a generalzaton o the lnear auto-assocator. Journal o Mathematcal Psychology, 40, [3] Bshop, C. M. (995) Neural netorks or pattern recognton. Oxord, UK: Oxord Unversty Press. [4] Cherkassky, V., & Muler, F. (998). Learnng rom data. Ne York: Wley. [5] Duda, R., Hart, P.E., Stork, D.G. (200) Pattern classcaton. Ne York: Wley. [6] Haste T., Tbshran R., Fredman J. (200). The elements o statstcal learnng. Ne-Yrok: Sprnger-Verlag [7] Rpley, B.D. (996) Pattern recognton and neural netorks. Cambrdge, MA: Cambrdge Unversty Press. [8] Schölkop B., Smola, A.J. (2003). learnng th kernels. Cambrdge (MA): MIT Press. [9] Vapnk, V. N. (999) Statstcal learnng theory. Ne York: Wley. 5