A Statistical Perspective on Data Mining

Transcription

1 A Statstal Persetve on Data Mnng Ranjan Matra Abstrat Tehnologal advanes have led to new and automated data olleton methods. Datasets one at a remum are often lentful nowadays and sometmes ndeed massve. A new breed of hallenges are thus resented rmary among them s the need for methodology to analyze suh masses of data wth a vew to understandng omlex henomena and relatonshs. Suh aablty s rovded by data mnng whh ombnes ore statstal tehnques wth those from mahne ntellgene. Ths artle revews the urrent state of the dslne from a statstan s ersetve, llustrates ssues wth real-lfe examles, dsusses the onnetons wth statsts, the dfferenes, the falngs and the hallenges ahead. 1 Introduton The nformaton age has been mathed by an exloson of data. Ths surfet has been a result of modern, mroved and, n many ases, automated methods for both data olleton and storage. For nstane, many stores tag ther tems wth a rodut-sef bar ode, whh s sanned n when the orresondng tem s bought. Ths automatally reates a ggant reostory of nformaton on roduts and rodut ombnatons sold. Smlar databases are also reated by automated book-keeng, dgtal ommunaton tools or by remote sensng satelltes, and aded by the avalablty of affordable and effetve storage mehansms magnet taes, data warehouses and so on. Ths has reated a stuaton of lentful data and the otental for new and deeer understandng of omlex henomena. The very sze of these databases however means that any sgnal or attern may be overshadowed by nose. New methodology for the areful analyss of suh datasets s therefore alled for. Consder for nstane the database reated by the sannng of rodut bar odes at sales hekouts. Orgnally adoted for reasons of onvenene, ths now forms the bass for ggant databases as large stores mantan reords of roduts bought by ustomers n any Ranjan Matra s Assstant Professor of Statsts n the Deartment of Mathemats and Statsts at the Unversty of Maryland, Baltmore County, Baltmore, MD 21250, USA. The author thanks Surajt Chaudhur for dsussons on the ratal asets of data mnng from the ont of vew of a researher n databases and for hel wth Fgure 4, Rouben Rostaman for rovdng me wth the enrolment data of Table 1 and Devass Bassu for hel wth the examle n Seton 6 of ths aer. 1

2 transaton. Some busnesses have gone further: by rovdng ustomers wth an nentve to use a magnet-stred frequent shoer ard, they have reated a database not just of rodut ombnatons but also tme-sequened nformaton on suh transatons. The goal behnd olletng suh data s the ablty to answer questons suh as If otato hs and kethu are urhased together, what s the tem that s most lkely to be also bought?, or If shamoo s urhased, what s the most ommon tem also bought n that same transaton?. Answers to suh questons result n what are alled assoaton rules. Suh rules an be used, for nstane, n dedng on store layout or on romotons of ertan brands of roduts by offerng dsounts on selet ombnatons. Alatons of assoaton rules transend sales transatons data ndeed, I llustrate the onets n Seton 2 through a small-sale lass-shedulng roblem n my home deartment but wth mllons of daly transatons on mllons of roduts, that alaton best reresents the omlextes and hallenges n dervng meanngful and useful assoaton rules and s art of folklore. An oft-stated goal of data mnng s the dsovery of atterns and relatonshs among dfferent varables n the database. Ths s no dfferent from some of the goals of statstal nferene: onsder for nstane, smle lnear regresson. Smlarly, the ar-wse relatonsh between the roduts sold above an be nely reresented by means of an undreted weghted grah, wth roduts as the nodes and weghted edges for the resene of the artular rodut ar n as many transatons as roortonal to the weghts. Whle undreted grahs rovde a grahal dslay, dreted ayl grahs are erhas more nterestng they rovde understandng of the henomena drvng the relatonshs between the varables. The nature of these relatonshs an be analyzed usng lassal and modern statstal tools suh as regresson, neural networks and so on. Seton 3 llustrates ths onet. Closely related to ths noton of knowledge dsovery s that of ausal deendene models whh an be studed usng Bayesan belef networks. Hekerman (1996) ntrodues an examle of a ar that does not start and roeeds to develo the goal of fndng the most lkely ause for the malfunton as an llustraton for ths onet. The buldng bloks are elementary statstal tools suh as Bayes theorem and ondtonal robablty statements, but as we shall see n Seton 3, the use of these onets to make a ass at exlanng ausalty s unque. One agan, the roblem beomes more aute wth large numbers of varables as n many omlex systems or roesses. Another aset of knowledge dsovery s suervsed learnng. Statstal tools suh as dsrmnant analyss or lassfaton trees often need to be refned for these roblems. Some addtonal methods to be nvestgated here are k-nearest neghbor methods, bootstra aggregaton or baggng, and boostng whh orgnally evolved n the mahne learnng lterature, but whose statstal roertes have been analyzed n reent years by statstans. Boostng s artularly useful n the ontext of data streams when we have rad data flowng nto the system and real-tme lassfaton rules are needed. Suh aablty s eseally desrable n the ontext of fnanal data, to guard aganst redt ard and allng ard fraud, when transatons are streamng n from several soures and an automated slt-seond determnaton of fraudulent or genune use has to be made, based on ast exerene. Modern lassfaton tools suh as these are surveyed n Seton 4. 2

3 Another mortant aset of knowledge dsovery s unsuervsed learnng or lusterng, whh s the ategorzaton of the observatons n a dataset nto an a ror unknown number of grous, based on some haraterst of the observatons. Ths s a very dffult roblem, and s only omounded when the database s massve. Herarhal lusterng, robabltybased methods, as well as otmzaton arttonng algorthms are all dffult to aly here. Matra (2001) develos, under restrtve Gaussan equal-dserson assumtons, a multass sheme whh lusters an ntal samle, flters out observatons that an be reasonably lassfed by these lusters, and terates the above roedure on the remander. Ths method s salable, whh means that t an be used on datasets of any sze. Ths aroah, along wth several unsurmounted hallenges, are revewed n detal n Seton 5. Fnally, we address the ssue of text retreval. Wth ts ready aessblty, the World Wde Web s a treasure-trove of nformaton. An user wshng to obtan douments on a artular to an do so by tyng the word usng one of the ubl-doman searh engnes. However when an user tyes the word ar he lkely wants not just all douments ontanng the word ar but also relevant douments nludng ones on automoble. In Seton 6, we dsuss a tehnque, smlar to dmenson reduton whh makes ths ossble. Here also, the roblem s omounded by databases that are ggant n sze. Note therefore, that a large number of the goals of data mnng overla wth those n statsts. Indeed, n some ases, they are exatly the same and sound statstal solutons exst, but are often omutatonally mratal to mlement. In the sequel, I revew some of the faets of the onneton between data mnng and statsts as ndated above. I llustrate eah aset through a real-lfe examle, dsuss urrent methodology and outlne suggested solutons. Exet for art of the lusterng examle where I used some of my own re-wrtten software routnes n C and Fortran I was able to use the ublly avalable and free statstal software akage R avalable at htt:// to erform neessary data analyss. Whle not exhaustve, I hoe that ths artle rovdes a broad revew of the emergng area from a statstal vew-ont and surs nterest n the many unsolved or unsatsfatorly-solved roblems n these areas. 2 Assoaton Rules Assoaton rules (Patetsky-Sharo, 1991) are statements of the form, 93% of ustomers who urhase ant also buy ant brushes. The mortane of suh a rule from a ommeral vewont s that a store sellng ant wll also stok ant brushes, for ustomer onvenene, translatng nto more buyer vsts and and store sales. Further, he an offer romotons on brands wth hgher roft margns to ustomers buyng ant and subtly dtate ustomer referene. Suh nformaton an addtonally be used to dede on storage layout and sheduled dsath from the warehouse. Transatons data are used to derve rules suh as the one mentoned above. Formally, let D = {T 1, T 2,..., T n } be the transatons database, where eah transaton T j s a member of the set of tems n I = { 1, 2,..., m }. (The transatons data may, for nstane, be reresented as an n m matrx of 0 s and 1 s alled D wth the (k, l) th entry as an ndator 3

4 varable for the resene of tem k n transaton T l.) Further, wrte X T l to denote that a set of tems X s ontaned n T l. An assoaton rule s a statement of the form X Y where X, Y I and X and Y are dsjont sets. The rule X Y has onfdene γ f 100γ% of the transatons n the database D ontanng X also ontan the set of tems Y. It s sad to have suort δ f 100δ% of the transatons n D ontan all elements of the set X Y. The mrovement ι of a rule X Y s the rato of the onfdene and the roorton of transatons only nvolvng Y. Note that whle onfdene rovdes us wth a measure of how onfdent we are n the gven rule, suort assesses the roorton of transatons on whh the rule s based. In general, rules wth greater suort are more desrable from a busness ersetve, though rules wth lower suort often reresent small nhes who, beause of ther sze are not neessarly wooed by other busnesses and may be very loyal ustomers. Further, mrovement omares how muh better a rule X Y s at redtng Y than n usng no ondtonal nformaton n dedng on Y. When mrovement s greater than unty, t s better to redt Y usng X than to smly use the margnal robablty dstrbuton of Y. On the other hand, f the mrovement of X Y s less than unty, then t s better to redt Y usng the margnal robablty dstrbuton than n usng t dstrbuton ondtonal on X. At equalty, there s no dfferene n usng ether X for redtng Y or n smly dedng on Y by hane. Although, assoaton rules have been rmarly aled to sales data, they are llustrated here n the ontext of shedulng mathemats and statsts graduate lasses. 2.1 An Alaton Lke several deartments, the Deartment of Mathemats and Statsts at the Unversty of Maryland, Baltmore County offers several graduate ourses n mathemats and statsts. Most students n these lasses ursue Master of Sene (M. S.) and dotoral (Ph. D.) degrees offered by the deartment. Shedulng these lasses has always been an onerous task. Pratal ssues do not ermt all lasses to be sheduled at dfferent tmes, and ndeed, four artular tme-slots are most desrable for students. Addtonally, t s desrable for ommutng and art-tme students to have ther lasses sheduled lose to one another. Htherto, lass tmes have been deded on the bass of emral eretons of the shedulng authortes. Dedng on obvous anddates for onurrently sheduled lasses s sometmes rather straghtforward. For nstane, students should have a roer gras of the elementary robablty ourse (Stat 651) to be ready for the hgher-level ourse (Stat 611) so that these an run smultaneously. Suh rules are not always obvous however. Further, the other queston whh lasses to shedule lose to eah other s not always easy to determne. Here I use assoaton rules to suggest lasses that should not be offered onurrently, as well as lasses that may be sheduled lose to eah other to maxmze student onvenene. The database on the enrollment of graduate students n the fall semester of 2001 s small wth 11 tems (the lasses offered) and 38 transatons (students enrollment n the dfferent lasses) see Table 1. Assoaton rules were bult usng the above fve rules wth the hghest suort (for both one- and two-lass ondtonng rules) are reorted n Table 2. Sne eah 4

5 Table 1: Enrollment data of deartmental graduate students n mathemats and statsts at the Unversty of Maryland, Baltmore County, Fall semester, Course Math 600 Math 617 Math 630 Math 700 Math 710 Stat 601 Stat 611 Stat 615 Stat 621 Stat 651 Stat 710 Class lst by student s last name Gavrea, Korostyshevskaya, Lu, Musedere, Shevhenko, Soane, Vdovna Challou, Gavrea, Korostyshevskaya, Shevhenko, Vdovna, Waldt Du, Feldman, Foster, Gavrea, Korostyshevskaya, Shevhenko, Sngh, Smth, Soane, Taff, Vdovna,Waldt Challou, Hanhart, He, Sabaka, Soane, Tymofyeyev, Wang, Webster, Zhu Korolev, Korostyshevskaya, Maura, Osmoukhna Cameron, L, Paul, Pone, Sddan, Zhang Cameron, L, Lu, Sddan, Zhang Benamat, Kelly, L, Lu, Pone, Sddan, Wang, Webb, Zhang Hang, Osmoukhna, Tymofyeyev Waldt, Wang Hang, Lu, Paul, Pone, Safronov student enrolls n at most three lasses, note that all ondtonng rules nvolve only u to two lasses. To suggest a lass shedule usng the above, t s farly lear that Math 600, Math 617 and Math 630 should be held at dfferent tme-slots but lose to eah other. Note that whle the rule If Math 600 and Math 617, then Math 630 has onfdene 100%, both (Math 600 & Math 630) Math 617 and (Math 617 & Math 630) Math 600 have onfdene of 80%. Based on ths, t s referable to suggest an orderng of lass tmes suh that the Math 630 tme-slot s n between those of Math 600 and Math 617. Smlar rules are used to suggest the shedule n Table 3 for future semesters that have a smlar mx of lasses. The suggested shedule requres at most four dfferent tme-slots, whle the urrent shedule derved emrally and gnorng enrollment data has no less than eght tme-slots. Table 2: Some assoaton rules obtaned from Table 1 and ther onfdene (γ),suort (δ) and mrovement (ι). Only sngle- and double-lass ondtonng rules wth the fve hghest mrovement measures are reorted here. Rule γ δ ι Rule γ δ ι Stat 601 Stat (Math 700 & Stat 615) Stat Stat 611 Stat (Math 630 & Stat 651) Math Math 600 Math (Math 600 & Math 630) Math Math 617 Math (Stat 611 & Stat 615) Stat Stat 611 Stat (Math 617 & Math 630) Math

6 Table 3: Suggested shedule for graduate lasses n mathemats and statsts at the Unversty of Maryland, Baltmore County based on the assoaton rules obtaned n Table 2. Slot Classes Slot Classes Slot Classes 1 Stat Math Math 600, Stat Stat 651, Stat Math Math 630, Math 700, Stat 601, Math Math Math 700, Stat Math 617, Stat 615, Stat Stat Math 630, Stat Stat 651, Stat 710 The above s an nterestng and somewhat straghtforward llustraton of assoaton rules. It s made easer by the lmtaton that all graduate students regster for at most three lasses. For nstane, the onsequenes of searhng for suh rules for all undergraduate and graduate lasses taught at the unversty an well be magned. There are hundreds of suh lasses and over 12,000 students, eah of who take any number between one and eght lasses a semester. Obtanng assoaton rules from suh large databases s qute dauntng. Ths roblem of ourse, ales n omarson to the searh for suh rules n sales databases, where there are several mllons of tems and hundreds of mllons of transatons. 2.2 Alaton to Large Databases There has been substantal work on fndng assoaton rules n large databases. Agarwal et al. (1993) fous on dsoverng assoaton rules of suffent suort and onfdene. The bas dea s to frst sefy large tem-sets, desrbed by those sets of tems that our have a re-sefed transaton suort (. e. roorton of transatons n whh they our together) s greater than the desred mnmum suort δ +. By restrtng attenton to these tem-sets, one an develo rules X Y of onfdene gven by the rato of the suort of X Y to the suort of X. Ths rule holds f and only f ths rato s greater than the mnmum desred onfdene γ +. It may be noted that sne X Y s a large tem-set, the gven rule X Y trvally has re-sefed mnmum suort δ +. The algorthms for dsoverng all large tem-sets n a transatons database are teratve n srt. The dea s to make multle asses over the transatons database startng wth a seed set of large tem-sets and use ths to generate anddate tem-sets for onsderaton. The suort of eah of these anddate tem-sets s also alulated at eah ass. The anddate tem-sets satsfyng our defnton of large tem-sets at eah ass form the seed for the next ass, and the roess ontnues tll onvergene of the tem-sets, whh s defned as the stage when no new anddate tem-set s found. The dea s onetually very smle and hnges on the fat that an tem-set of j elements an be large only f every subset of t s large. Consequently, t s enough for a mult-ass algorthm to buld the rules nrementally n terms of the ardnalty of the large tem-sets. For nstane, f at any stage, there are k large tem-sets of j elements eah, then the only ossble large tem-sets wth more than j elements ertan to transatons whh have members from these k tem-sets. So, the searh-sae for large tem-sets an be onsderably whttled down, even n the ontext of huge databases. 6

7 Ths forms the bass of algorthms n the database lterature. 2.3 Addtonal Issues The solutons for large tem-sets gnore transatons wth small suort. In some ases, these reresent nhes whh may be worth aterng to, eseally from a busness ont of vew. Redung γ + and δ + would allow for these rules but result n a large number of surous assoaton rules defeatng the very urose of data mnng. Brn et al. (1997a, 1997b) therefore develos the noton of both orrelaton and mlaton rules. Instead of onfdene measures, ther rules have mlaton strengths rangng from 0 to an mlaton strength of 1 means that the rule s as useful as under the framework of statstal ndeendene. An mlaton strength greater than 1 ndates a greater than exeted resene, under statstal ndeendene, of the temset. Another noton (Aggarwal and Yu, 1998) alled olletve strength of an temset I nvolves defnng a volaton rate v(i) as the fraton of volatons (transatons n whh only some members of an temset are resent) of the temset over all transatons. The olletve strength s then defned as (C(I)) = [{1 v(i)}/{1 IEv(I)}].[{IEv(I)}{IEv(I)] where IE denotes exeted value under the assumton of statstal ndeendene. Ths measure regards both absene and resene of tem ombnatons n an temset symmetrally so that t an be used f the absene of ertan tem ombnatons rather than ther resene s of nterest. It addtonally nororates the orrelaton measure beause for erfetly ostvely orrelated tems, C(I) = whle for erfetly negatvely orrelated tems, C(I) = 0. Fnally, the olletve strength uses the relatve ourrenes of an temset n the database: temsets wth nsgnfant suort an then be runed off at a later stage. DuMouhel and Pregbon (2001) ont out however, that the above advantages arue at onsderable loss of nterretablty. They roose a measure of nterestngness for all temsets and ft an emral Bayes model to the temset ounts. All lower 95% redble lmts for the nterestngness measure of an temset are ranked. Ths aroah has the added advantage of dsoverng omlex mehansms among mult-tem assoatons that are more frequent than ther orresondng arwse assoatons suggest. One of the ssues n a mult-ass algorthm as of Agarwal et al. (1993) s that t s not suted for data streams suh as web alatons. Understandng the struture and nature of suh transatons s mortant and to ths extent, some work has been done reently (Babok et al., 2002). Moreover, none of the above formulatons use any tme seres nformaton, whh s very often resent n the database. For examle, a ustomer may urhase tea and sugar searately n suessve transatons, resumably falng to reall that he needed to urhase the seond rodut also. Suh assoatons would not show u above, and ndeed t would erhas be desrable for the store to rovde a frendly remnder to suh ustomers. Addtonally, no onsderaton s gven to multle taxonomes (or herarhes) n ostulatng these rules. For examle, a refrgerator s a kthen alane s a heavy eletral alane. Ths s a taxonomy. Gven suh a taxonomy, t may be ossble to nfer a rule that eole who buy kthen alanes tend also to buy dshware. Ths rule may hold even f rules suh as eole who buy refrgerators also tend to buy dshware or eole who buy kthen 7

8 Toner Comuter Workstaton Prnt Server Prnter Outut (Prntout) Paer Fgure 1: A dreted grah reresentaton for the outut from a laser rnter. Dreted edges are drawn n the dreton of ause to effet. alanes also tend to buy dshware do not hold. It would be useful to onstrut rules usng suh herarhes. These are some of the asets of assoaton rules requrng attenton of both the statstal and the database ommunty. 3 Grahal Models and Belef Networks The ar-wse assoatons from the enrollment data n Table 1 an be reresented usng an undreted grah wth nodes reresentng the lasses. Weghts on the edges between a ar of nodes would ndate the frequenes wth whh the two lasses are taken by the same student. Grahs are frequently used to rovde toral reresentatons of jont arwse relatonshs between varables. From a statstal ont of vew they addtonally hel us to learn and sefy the jont multvarate dstrbuton of the varables. For suose we are nterested n sefyng the jont dstrbuton of varables X 1, X 2,..., X. The smlest aroah s to sefy that the jont dstrbuton of the varables s the rodut of the margnals of eah,. e. the varables are all ndeendent. In many omlex henomena and real-lfe stuatons, suh an assumton s not tenable. On the other hand, multvarate dstrbutons are dffult to sefy and subsequent nferene s often ntratable, eseally when s large. Reresentng the jont dstrbuton through an arorate sequene of margnal and ondtonal dstrbutons allevates the roblem somewhat. I rovde an llustraton through a smlfed day-to-day examle, also grahally reresented n Fgure 1. When a rnt job s submtted from a deskto omuter/workstaton, ontrol of the job s transferred to the rnt server whh queues t arorately and sends t to the 8

9 rntng deve. The rnter needs toner and aer to rnt ts outut. The outut may be reresented by a varable whh may be bnary or a ontnuous varable measurng the degree of satsfaton wth the result. For nstane, 0% may ndate no outut, 100% a erfet result, whle 25% may mean faded outut as a onsequene of an over-used toner. The varables here are the bnary varable X 1 reresentng the orretness of the rnt ommand, X 2 for the number of ages transmtted from the rnt server to the rnter, X 3 denotng the qualty of the toner, X 4 for the number of ages n the rnter aer tray and X 5 ndatng the status of the rnter (off-lne, on-lne or jammed) whle nteratng wth the toner, the rnt server and the aer tray. The outut varable s gven by X 6. The dstrbuton of X 1, X 2,..., X 6 an be reresented by means of the dreted grah n Fgure 1. The arrows ndate ondtonal deendene of the desendants to ther arents. For nstane, X 2 s deendent on X 1, whle X 5 s deendent on ts anteedents X 1, X 2,..., X 4 only through ts arents X 2, X 3, X 4. Fnally X 6 s deendent on ts anteedents X 1, X 2,..., X 5 only through X 5. The grah of Fgure 1 ndates that the jont dstrbuton of X 1, X 2..., X 6 s: Pr(X 1, X 2..., X 6 ) = Pr(X 1 )Pr(X 2 X 1 )Pr(X 3 )Pr(X 4 )Pr(X 5 X 2, X 3, X 4 )Pr(X 6 X 5 ). (1) Note that ondtonal and margnal robablty dstrbutons may have addtonal arameters: for nstane, both aer and toner may be modeled n terms of the number of ages sent through the rnt server and the date last relenshed. Now suose that gven X 6 = 0, we want to dagnose the ossble ause. If all the robablty dstrbutons were omletely sefed, by usng Bayes Theorem and ondtonal robablty statements one an alulate the robabltes of the other omonents malfuntonng, gven X 6 = 0 and use ths to dentfy the most lkely auses, together wth a statement on the relablty of our assessments. The grahal struture desrbed above s a Bayesan belef network. In realty of ourse, we would not be handed down a omlete sefaton of the ondtonal and margnal robablty dstrbutons, even wth a omletely known network. I address the alternatve stuaton a lttle later but for now address the ase for a senaro wth knowledge of network struture suh as outlned, but unknown robabltes. Data on the network (suh as several observatons on the workng of the rnter n the examle above) are used to learn the dfferent robabltes). For eah varable X, let reresent the set of ts anteedents, and θ ω (x ) be the ondtonal densty of X at the ont x gven that = ω. For dsrete statesaes, one an ut Drhlet dstrbutons on eah robablty whle for ontnuous statesaes a Drhlet Proess ror may be used. Assumng arameter ndeendene. e. ndeendene of the arameters underlyng the unknown robabltes, the robablty dstrbutons are udated wth newer data. I refer to Hekerman (1996) for a detaled desrton of the related mehans but note that unertanty n our knowledge of the robablty dstrbutons governng the network s addressed through these rors. The use of grahal and ondtonal strutures to sefy jont dstrbutons has a long hstory n statsts. The smlest ase s that of a Markov Chan. If X 1, X 2,..., X n are suessve realzatons of a frst-order Markov Chan the deendene struture may be reresented usng the dreted grah X 1 X 2... X n. Note that the above struture 9

10 mles that the ondtonal dstrbuton of any nteror X gven both the ast and the future nvolves only ts mmedate anteedent X 1 and ts mmedate desendant X +1. In other words the ondtonal dstrbuton of X gven X 1 and X +1 s ndeendent of the other varables. Thus an equvalent struture may be sefed by an undreted ayl grah wth the only edges nvolvng X beng those onnetng t to X 1 and X +1 for all = 2, 3,..., n 1. X 1 and X n have only one onnetng edge, to X 2 and X n 1 resetvely. The undreted grah reresentaton s mortant beause a Markov Chan runnng bakwards and therefore havng the same dreted grah X 1 X 2... X n has the same ondtonal deendene struture as the one above. Sefyng jont dstrbutons through ondtonal dstrbutons s farly ommon and wellestablshed n statsts. For nstane, t an be shown that under ondtons of ostvty for the statesae (whh may be relaxed somewhat), omlete knowledge of the loal haratersts or the full ondtonal dstrbutons Pr(X X j : j, j = 1, 2,..., n) s enough to sefy the jont dstrbuton of X 1, X 2,..., X n (Brook, 1964; Besag, 1974). Indeed a system sefed through ts ondtonal robablty struture s alled a Markov Random Feld (MRF) and s ommonly used n statstal hyss, Bayesan satal statsts, to name a few areas. Of ourse, a ondtonal ndeendene model as desrbed above usng a grah s handy here beause the loal haraterst at a artular X s deendent only on those X j s whh share a ommon edge wth X. Ths roerty s alled the loally Markov roerty. Those X j s sharng an edge wth X n the grahal reresentaton of deendene are alled neghbors of the latter. Defnng the envronment of a set of random varables to be the unon of neghbors of eah element n the set, a globally Markov roerty s defned to be the ase when the ondtonal dstrbuton of the subset of random varables gven everythng else deends only through the envronment. Under ondtons of ostvty, the loally Markov roerty s equvalent to the globally Markov roerty. Further, f we defne a lquo to be a set onsstng ether of a sngle random varable or a set of random varables all of whh are neghbors to eah other and a lque to be a maxmal lquo (suh that there s no suerset of the lquo whh s also a lquo), t an be shown (Hammersley and Clfford, 1971) that the underlyng robablty densty/mass funton an be deomosed nto a rodut of funtons, eah defned on a searate lque. In most ases, t s mratal to know ausal relatonshs between the dfferent omonents of the network. Indeed, learnng the underlyng ausal struture s desrable and a muh-sought goal of the knowledge dsovery roess. Ths an be done by extendng the methodology from the ase where the network s omletely known but the governng robablty dstrbutons are not. We sefy a ror robablty on eah ossble network and then for eah network, roeed as above. The osteror robablty of eah network s margnalzed over the arameter robablty dstrbutons and the maxmum a osteror (MAP) estmate omuted. Note that the omutatonal burden s often severe so that stohast methods suh as Markov Chan Monte Carlo (MCMC) are used n omutaton. Markov grahs are artularly amenable to suh methods. Also, I refer bak to the dsusson n the revous aragrah and menton that n the termnology of Verma and Pearl (1990), two network strutures are equvalent f and only f they have the same struture, gnorng ar 10

11 dretons and the same v-strutures. (A v-struture s an ordered trle (x, y, z) wth ars between x to y and z to y, but no ar between x and z.) Equvalent network strutures have the same robablty and therefore, our learnng methods atually learn about equvalene lasses n the network. Even wth these smlfatons, the number of networks to evaluate grows exonentally wth the number of nodes n. Of nterest therefore s to see whether a few network-strutures an be used n the learnng roedure. Some researhers have shown that even a sngle good network struture often rovdes an exellent aroxmaton to several networks (Cooer and Hersovts, 1992; Alfers and Cooer, 1994, Hekerman et al., 1995). The fous then s on how to dentfy a few of these good networks. One aroah s to sore networks n terms of the goodness of ft and rank them avalable methods nlude Madgan and Raftery s (1994) Oam s Razor, Rsannen s (1987) Mnmum Desrton Length (MDL), the more tradtonal Akake s (1974) An Informaton Crteron (AIC) or Shwarz s (1978) Bayes Informaton Crteron (BIC). Fnally, onsder the ssue of learnng new varables n the network whose exstene and relevane s not known a ror they are therefore hdden. Suh varables (both ther number and statesae) an be nororated n the ror model durng the network-learnng roess. I refer to Cheeseman and Stutz (1996) for a detaled desrton of a network wth hdden varables. Fnally, belef networks have generated a lot of nterest n the knowledge dsovery ommunty a few referenes n ths regard are Charnak (1991), Segelhalter et al. (1993), Hekerman et al. (1995), Laurtzen (1982), Verma and Pearl (1990), Frydenberg (1990), Buntne (1994, 1996). I onlude ths dsusson of grahal models and belef networks by notng that most of the networklearnng methods dsussed here are omuter- and memory-ntensve, eseally n the ase of very omlex roesses wth lots of varables. Develong methodology n ths ontext s therefore desrable and useful. 4 Classfaton and Suervsed Learnng Suervsed learnng or lassfaton has long been an mortant area n statsts, and an arse wth regard to a number of alatons. For nstane, a bank may lke to dentfy long-term deostors by ther attrbutes and rovde them wth nentves for ondutng busness at that bank. Other alatons also exst: here we dsuss a ase study n the emergng area of bonformats. 4.1 Proten Loalzaton Identfyng roten loalzaton stes s an mortant early ste for fndng remedes (Naka and Kanehsa, 1991). The E. ol batera, has eght suh stes: or the ytolasm, m or the nner membrane wthout sgnal sequene, or erlasm, mu or nner membrane wth an unleavable sgnal sequene, om or the outer membrane, oml or the outer membrane loroten, ml or the nner membrane loroten, ms or the nner membrane wth a leavable sgnal sequene. Eah roten sequene has a number of numer attrbutes: these are mg or measurements obtaned usng MGeoh s method 11

12 for sgnal sequene reognton, gvh or measurements va von Hejne s method for sgnal sequene reognton, ad or the sore of a dsrmnant analyss of the amno ad ontent of outer membrane and erlasm rotens, alm1 or the sore of the ALOM membrane sannng regon redton rogram, and alm2 or the sore usng ALOM rogram after exludng utatve leavable sgnal regons from the sequene. In addton, there are two bnary attrbutes l or the von Hejne s Sgnal Petdase II onsensus sequene sore and the hg ndatng the resene of a harge on the N-termnus of redted lorotens. (See Horton and Naka, 1996 and the referenes theren for a detaled desrton of these attrbutes.) Data on 336 suh roten sequenes for E. ol are avalable for analyss at the Unversty of Calforna Irvne s Mahne Learnng Reostory (Horton and Naka, 1996). The l and hg are bnary attrbutes. These attrbutes are gnored for lassfaton beause some of the methods dsussed here do not admt bnary attrbutes. As a onsequene, roten sequenes from the stes ml and ms an no longer be dstngushed from eah other. Sequenes from these stes are also elmnated. Two ases from the ste oml are dsarded beause the number of ases s too few for quadrat dsrmnant analyss, leavng for analyss a subset of 324 sequenes from fve roten loalzaton stes. Further, a set of 162 random sequenes was ket asde as a test set the remanng 162 formed the tranng data. 4.2 Deson-Theoret Aroahes The dataset s n the form {(X, G ):=1,2...,n}, where 1 G K reresents the grou to whh the th observaton X belongs. The densty of an arbtrary observaton X evaluated at a ont x may be wrtten as f(x) = K k=1 π kf k (x), where π k s the ror robablty that an observaton belongs to the k th grou, and f k ( ) s the densty of an observaton from the k th grou. Our goal s to sefy the grou G(x), gven an observaton x. Alyng Bayes theorem rovdes osteror membersh robabltes of x for eah grou: Pr(G(x) = k X = x) = π k f k (x) K k=1 π kf k (x). The grou wth the hghest osteror robablty s dentfed as the redted lass for x. Note that havng the ndvdual lass denstes s equvalent to obtanng the osteror lass robabltes, and hene the redted lassfaton (assumng that the π k s are known). When the denstes f k s are multvarate Gaussan, eah wth dfferent mean vetors µ k s but wth ommon dserson matrx Σ, the logarthm of the rato between the osteror lass robabltes of lasses k and j s a lnear funton n x. Ths s beause log Pr(G(x) = k X = x) Pr(G(x) = j X = x) = log π k π j 1 2 (µ k + µ j ) Σ 1 (µ k + µ j ) + x Σ 1 (µ k µ j ). Ths s the bass for Fsher s (1936) lnear dsrmnant analyss (LDA). Lnear dsrmnant funtons δ k (x) = x Σ 1 µ k 1 2 µ k Σ 1 µ k +log π k are defned for eah grou. Then the deson 12

13 rule s equvalent to G(x) = argmax k δ k (x). In rate, the π s, µ s and Σ are unknown and need to be estmated. The lass roortons of the tranng data are used to estmate the π k s, the ndvdual grou means are used to estmate the µ k s, whle the ommon estmate of Σ s estmated usng the ooled wthn-grous dserson matrx. Oeratonal mlementaton of LDA s very straghtforward: Usng the setral deomoston ˆΣ = V DV and X = D 1 2 V X as the rojeton of the shered data-ont X nto the transformed sae, a new observaton X s lassfed to losest lass entrod n the transformed sae, after salng loseness arorately to aount for the effet of the lass ror robabltes π k s. For arbtrary Gaussan denstes, the dsrmnant funton δ k (x) = 1 2 log Σ k 1 2 (x µ k ) Σ 1 (x µ k ) + log π k nludes a quadrat term n x, so s alled quadrat dsrmnant analyss (QDA). The estmaton of the arameters s smlar as before, wth the kth grou dsersons from the tranng data used to estmate Σ k. For the roten sequene data, the tranng dataset was used to devse lassfaton rules usng both LDA and QDA. Wth LDA, the test set had a mslassfaton rate of 30.5% whle wth QDA, t was only 9.3%. Clearly, LDA was not very suessful here. In general, both methods are reuted to have good erformane deste beng farly smle. Ths s erhas beause most deson boundares are no more omlated than lnear or quadrat. Fredman (1989) suggested a omromse between LDA and QDA by roosng regularzed dsrmnant analyss (RDA). The bas dea s to have a regularzed estmate for the ovarane matrx ˆΣ k (γ) = γ ˆΣ k + (1 γ) ˆΣ, where ˆΣ s the ooled dserson matrx from LDA and ˆΣ k s the k th grou varaneovarane matrx obtaned as er QDA. Other regularzatons exst, and we address these shortly. But frst, a few addtonal words on the oularty of LDA. As dsussed n Haste et al (2001), one attraton of LDA s an addtonal restrton that allows substantal reduton n dmensonalty through nformatve low-dmensonal rojetons. The K-dmensonal entrods obtaned usng LDA san at most a (K 1)-dmensonal subsae, so there s substantal reduton n dmensonalty f s muh more than K. So, X may as well be rojeted on to a (K 1)-dmensonal subsae, and the lasses searated there wthout any loss of nformaton. A further reduton n dmensonalty s ossble by rojetng X onto a smaller subsae n some otmal way. Fsher s otmalty rteron was to fnd rojetons whh sread the entrods as muh as ossble n terms of varane. The roblem s smlar to fndng rnal omonent subsaes of the entrods themselves: formally, the roblem s to defne the j th dsrmnant oordnate by the orresondng rnal omonent of the matrx of entrods (after salng wth ˆΣ). The soluton s equvalent to that for the generalzed egenvalue roblem and for Fsher s roblem of fndng the rojeton of the data that maxmzes the between-lass varane (hene searates the dfferent lasses as wdely as ossble) relatve to the wthn-lass ovarane and s orthogonal to the revous rojeton oordnates already found. As wth rnal omonents, a few dsrmnant oordnates are often enough to rovde well-searated lasses. We dslay LDA usng only the frst two dsrmnant varables the rojeted dataont usng the orresondng dsrmnant oordnates for the E. ol roten sequene dataset n Fgure 2. 13

14 frst dsrmnant oordnate seond dsrmnant oordnate o U o U o U o U U o U o U U o U U U o U U U U o Fgure 2: Deson boundares for the lassfer based on the frst two lnear dsrmnants. The larger font,, o, u and reresent the rojeted means n the frst two dsrmnant oordnates for the,, om, m and mu ategores. Smaller font haraters reresent the rojeted observatons for eah grou. One drawbak of LDA arses from the fat that lnear deson boundares are not always adequate to searate the lasses. QDA allevates the roblem somewhat by usng quadrat boundares. More general deson boundares are rovded by flexble dsrmnant analyss (FDA) (Haste et al., 1994). The bas dea s to reast the lassfaton roblem as a regresson roblem and to allow for nonarametr aroahes. To do ths, defne a sore funton ϕ : G IR 1 whh assgns sores to the dfferent lasses and regress these sores on a transformed verson of the redtor varables, erhas allowng for regularzaton and other onstrants. In other words, the roblem redues to fndng sorngs ϕ s and transformatons ζ( ) s that mnmze the dfferent lasses and regress these sores on a transformed verson of the redtor varables, erhas allowng for regularzaton and other onstrants. In other words, the roblem redues to fndng sorngs ϕ s and transformatons ζ( ) s that mnmze 14

15 the onstraned mean squared error MSE({ϕ m, ζ m } M m=1) = 1 n [ M n ] {ϕ m (G ) ζ m (x )} + λj(ζ m ). m=1 =1 Here J s a regularzaton onstrant orresondng to the desred nonarametr regresson methodology suh as smoothng and addtve slnes, kernel estmaton and so on. Comutatons are artularly smlfed when the nonarametr regresson roedure an be reresented as a lnear oerator ϖ h wth smoothng arameter h. Then the roblem an be reformulated n terms of the multvarate adatve regresson of Y on X. Here Y s the (n K) ndator resonse matrx wth th row orresondng to the th tranng data-ar (X, G ) and havng exatly one non-zero entry of 1 for the olumn orresondng to G. Denote ϖ h as the lnear oerator that fts the fnal hosen model wth ftted values Ŷ and let ˆζ(x) be the vetor of ftted funtons. The otmal sores are obtaned by omutng the egen-deomoston of Y Ŷ and then rojetng the ˆζ(x) on to ths normalzed egen-sae. These otmal sores are then used to udate the ft. The above aroah redues to LDA when ϖ h s the lnear regresson rojeton oerator. FDA an also be vewed dretly as a form of enalzed dsrmnant analyss (PDA) where the regresson roedure s wrtten as a generalzed rdge regresson roedure (Haste et al., 1995). For the roten sequene dataset, PDA rovded us wth a mslassfaton rate of 13.59% on the test set. FDA used wth multvarate adatve regresson slnes (MARS) (Fredman, 1991) gave us a test set mslassfaton rate of 16.05%. Wth BRUTO, the rate was 16.67% whle wth a quadrat olynomal regresson sorng method, the test set mslassfaton error was 14.81%. Another generalzaton s mxture dsrmnant analyss (MDA). Ths s an mmedate extenson of LDA and QDA and s bult on the assumton that the densty for eah lass s a mxture of Gaussan dstrbutons wth unknown mxng roortons, means and dsersons (Haste and Tbshran, 1996). The number of mxtures n eah lass s assumed to be known. The arameters are estmated for eah lass by the Exetaton-Maxmzaton (E-M) algorthm. MDA s more flexble than LDA or QDA n that t allows for more rototyes than the mean and varane to desrbe a lass: here the rototyes are the mxng roortons, means and dsersons that make u the mxture dserson n eah lass. Ths allows for more general deson boundares beyond the lnear or the quadrat. One may ombne FDA or PDA wth MDA models to allow for even more generalzatons. More flexble otons for deson-theoret aroahes to the lassfaton roblem nvolve the use of nonarametr densty estmates for eah lass, or the seal ase of nave Bayesan aroahes whh assume that the nuts are ondtonally ndeendent wthn eah lass. The full array of tools n densty estmaton an then be ut to work here. 4.3 Dstrbuton-free Predtve Aroahes The methods dsussed n the revous seton are essentally model-based. Model-free aroahes suh as tree-based lassfaton also exst and are oular for ther ntutve aeal. 15

16 In ths seton, we dsuss n detal a few suh redtve aroahes these are the nearestneghbor methods, suort vetor mahnes, neural networks and lassfaton trees Nearest-neghbor Aroahes Perhas the smlest and most ntutve of all redtve aroahes s k-nearest-neghbor lassfaton. Deendng on k, the strategy for redtng the lass of an observaton s to dentfy the k losest neghbors from among the tranng dataset and then to assgn the lass whh has the most suort among ts neghbors. Tes may be broken at random or usng some other aroah. Deste the smle ntuton behnd the k-nearest neghbor method, there s some smlarty between ths and regresson. To see ths, note that the regresson funton f(x) = IE(Y X = x) mnmzes the exeted (squared) redton error. Relaxng the ondtonng at a ont to nlude a regon lose to the ont, and wth a 0-1 loss funton leads to the nearest-neghbor aroah. The hoe of k or the number of neghbors to be nluded n the lassfaton at a new ont s mortant. A ommon hoe s to take k = 1 but ths an gve rse to very rregular and jagged regons wth hgh varanes n redton. Larger hoes of k lead to smoother regons and less varable lassfatons, but do not ature loal detals and an have larger bases. Sne ths s a redtve roblem, a hoe of k may be made usng ross-valdaton. Cross-valdaton was used on the roten sequene tranng dataset to obtan k = 9 as the most otmal hoe n terms of mnmzng redted mslassfaton error. The dstane funton used here was Euldean. The 9-nearest neghbor lassfaton redted the roten sequenes n the test set wth a mslassfaton rate of 15.1%. Wth k = 1, the mslassfaton rate was 17.3%. On the fae of t, k-nearest neghbors have only one arameter the number of neghbors to be nluded n dedng on the majorty-vote redted lassfaton. However, the effetve number of arameters to be ft s not really k but more lke n/k. Ths s beause the aroah effetvely dvdes the regon sanned by the tranng set nto aroxmately n/k arts and eah art s assgned to the majorty vote lassfer. Another ont to be noted s that the urse of dmensonalty (Bellman, 1961) also mats erformane. As dmensonalty nreases, the data-onts n the tranng set beome loser to the boundary of the samle sae than to any other observaton. Consequently, redton s muh more dffult at the edges of the tranng samle, sne some extraolaton n redton may be needed. Furthermore, for a -dmensonal nut roblem, the samlng densty s roortonal to n 1. Hene the samle sze requred to mantan the same samlng densty as n lower dmensons grows exonentally wth the number of dmensons. The k-nearest-neghbors aroah s therefore not mmune to the henomenon of degraded erformane n hgher dmensons Suort Vetor Classfers and Mahnes Consder a two-lass roblem wth two nut dmensons n the feature sae as reresented n Fgure 3. Wthout loss of generalty, let the lass ndators G be -1 (reresented by flled s) and +1 (denoted by flled s). Fgure 3a shows the ase where the two lasses 16

17 are lnearly searable (the lne ω t x = γ s one ossble lne). Note that for onts lassfed orretly, G (ω t x γ) 0. Rosenblatt (1958) devsed the eretron learnng algorthm wth a vew to fndng a searatng hyerlane (lne n two dmensons) mnmzng the dstane of mslassfed onts to the deson boundary. However Fgure 3a shows that there an be an nfnte number of suh solutons. Vank (1996) rovded an elegant aroah to ths roblem by ntrodung an addtonal requrement: the revsed objetve s to fnd the otmal searatng hyerlane searatng the two lasses whle also maxmzng the shortest dstane of the onts n eah lass to the hyerlane. Assumng wthout loss of generalty that ω = 1, the otmzaton roblem s to maxmze δ over ω = 1, γ subjet to the onstrant that G (ω t x γ) δ for all = 1, 2,... n. The roblem an be wrtten n the equvalent but more onvenent form as: mnmze ω over all ω and γ subjet to the onstrant G (ω t x γ ) 1 for all = 1, 2,... n. Here, we have droed the unt-norm requrement on ω and set δ = ω 1. Standard solutons to ths roblem exst and nvolve quadrat rogrammng. The lassfer for a new ont x s gven by sgn( ˆω t x ˆγ) where ˆω, ˆγ are the otmal values. Fgure 3b shows the ase where the two lasses overla n the nut sae so that no hyerlane an omletely searate the two grous. A reasonable omromse s to allow for some varables to be on the wrong sde of the margn. Let us defne addtonal nonnegatve slak varables as ϑ = {ϑ, = 1, 2,..., n}. The orgnal onstrant n the roblem of fndng the otmal searatng hyerlane s then modfed to be G (ω t x γ) δ(1 ϑ ) for all = 1, 2,..., n wth ϑ bounded above by a onstant. The slak varables ϑ n the onstrant denote the roortonal amount by whh any redted observaton s on the wrong sde of the lassfaton hyerlane. Slang the uer bound on ther sum lmts the total roortonal amount by whh redtons are made on the wrong sde of the margn. Note that mslassfatons our only when ϑ > 1 so that ϑ M automatally lmts the number of tranng set mslassfatons to at most M. One agan, the otmzaton roblem an be rehrased as that of mnmzng ω subjet to the onstrants G (ω t x γ) (1 ϑ ) wth δ = ω 1, ϑ M, ϑ 0 for all = 1, 2,..., n so that smlar to the searable ase, quadrat rogrammng tehnques an agan be used. It an be shown (Haste et al., 2001) that the soluton ˆω s n the form ˆω = n =1 ˆα G x, wth non-zero values of ˆα only when G (ω t x γ) = (1 ϑ ). The soluton s alled the suort vetor lassfer. Note that onts wth non-zero values for the slak varables ϑ lay a major role these are alled the suort vetors. Ths s a major dstnton wth LDA where the soluton deends on all data onts, nludng those far away from the deson boundary. It does ths through alulatons on the lass ovarane-matres and the lass entrods. The suort vetor lassfer on the other hand uses all data-onts to dentfy the suort vetors, but fouses on the observatons near the boundary for the lassfaton. Of ourse, f the underlyng lasses are really Gaussan, LDA wll erform better than SVM beause of the latter s heavy relane on the (noser) observatons near the lass boundares. Suort vetor mahnes (Crstann and Shawe-Taylor, 2001) generalze the above senaro n a srt smlar to the extenson of FDA over LDA. The suort vetor lassfer fnds lnear boundares n the nut feature sae. The aroah s made more flexble by 17

18 ω t x = γ δ ω t x = γ + δ ω t x = γ θ 1 θ 2 θ 3 θ 4 θ 8 θ 6 θ 7 ω t x = γ ω t x = γ ζ θ 5 θ 11 θ 10 θ θ 13 9 ω t x = γ + ζ θ (a) (b) Fgure 3: The suort vetor lassfer for a two-lass roblem n a two-dmensonal nut sae. (a) In the searable ase, the deson boundary s the sold lne whle broken lnes bound the maxmal searatng margn of wdth 2δ. (b) In the overlang ase, onts on the wrong sde of ther margn are labeled θ j = δϑ j, whh the dstane from ther margn. All unlabeled onts have θ j = 0. The maxmal margn tself s obtaned wthn the onstrant that ϑ j does not exeed a ertan allowable budget. enlargng the feature sae to nlude bass exansons suh as olynomals or slnes. One the bass funtons are seleted to be, say, ζ(x) = {ζ (x), = 1, 2,..., m}, the roblem s reast n the same frame-work to obtan the lassfer G(x) = sgn( ˆω t ζ(x) ˆγ). Smlar to the ase of lnear boundares, the soluton s now ˆω = n =1 ˆα G ζ(x ). Ths reresentaton makes omutatons ratal. Ths s beause the ftted funton Ĝ(x) now has the form n =1 ˆα G < ζ(x), ζ(x ) > whh means that knowledge of the kernel funton Ψ(x, y) =< ζ(x), ζ(x ) > rather than the exat transformaton ζ(x) s enough to solve the roblem. Poular hoes for Ψ(x, y) nlude the th degree olynomal Ψ(x, y) = (1+x t y), the radal bass funton Ψ h (x, y) = ex ( x y 2 /h) or the sgmodal funton denoted by Ψ(x, y) = tanh(β 1 x t y + β 0 ). Suort vetor mahnes wth dfferent kernels were used on the roten sequene dataset. Suort vetor lassfaton usng 35 suort vetors, yelded a test set mslassfaton rate of 62.34%. A slghtly hgher mslassfaton rate of 63.5% was obtaned wth the sgmodal 18

19 kernel wth β 1 = 0.2 and β 0 = 0. Ths last alaton used 33 suort vetors. Usng a olynomal kernel of degree 6 mroved mslassfaton a bt redton errors on the test set were on the order of 56.8%. The number of suort vetors was then 45. Fnally usng the radal bass funton wth h = 5 gave a test set mslassfaton rate of 17.9%. The number of suort vetors was dentfed to be 51. Clearly, the erformane s the best wth the radal kernel, even though t s worse than that usng ether QDA, FDA, PDA or k-nearest neghbors. I onlude wth a bref dsusson on reent work n ths fast-develong area. As wth several smlar roblems, suort vetor mahnes are also hallenged by massve databases. One aroah, develoed by Platt (1998, 1999) s to break the quadrat rogrammng nto a seres of smallest ossble roblems. Ths s alled hunkng. The smallest ossble hunk sze s 2. Sne the omonent roblems are small, analyt solutons may be obtaned, seedng u omutatons. Another area of nterest s to formulate the roblem n a robablst framework and to obtan not a redton but the osteror robablty of a lass gven an nut. Ths s done, for examle, by formulatng the lassfaton model through a logst lnk funton Prob(G = 1 x) = (1 + ex { f(x)}) 1 (Wahba, 1999). Other aroahes also exst see, for examle, Haste and Tbshran (1998), Platt (1999), Vank (1996) and the referenes theren Artfal Neural Networks Consder a lassfaton roblem wth two nut varables X 1 and X 2. A sngle-hddenlayer artfal neural network s obtaned by defnng a layer Z wth omonents Z l = σ(α 0l + α t lx), l = 1, 2,... L. The outut G s then assumed to be a erturbed verson of a funton of a lnear ombnaton of ths layer. Here, σ( ) s the atvaton funton and G = F j (X) ɛ wth F (X) = g j (T j ) where T j = β 0j + β t jx and reresents the degradaton oerator on the sgnal F (X) wth ɛ and j = 0, 1 for the two-lass roblem (more generally 0, 1,..., K 1 for the K-lass roblem). Ths s llustrated n Fgure 4, wth m = 3. The X s form the nut layer, the Z s are the hdden layer and the Y s are the outut layer. The omonents α 0 s and β 0 s are the bas unts for eah layer. In general there an be several hdden layers. The most ommon hoe for the atvaton funton s the sgmod σ(u) = (1 + ex ( u)) 1 though a Gaussan radal bass funton σ(u) = ex { u 2 } has also been used n what s alled a radal bass funton network. The funton g j ( ) s alled the outut funton and s usually the softmax funton g j (v) = ex (T j )/ j ex (T j). Ths s of ourse the transformaton also used n logst regresson whh lke other regresson models suh as multle lnear regresson or rojeton ursut (Fredman and Stuetzle, 1984) are seal ases wthn ths framework: hoosng the dentty funton for σ( ), m = 1, β 0 = 0 and the bnomal dstrbuton for nstane, takes us to the logst model. Note that there may be any number of hdden layers n the model. Fttng the neural network means estmatng the unknown arameters α s and β s to obtan the lassfer G(x) = argmax j F j (X) otmzng the error funton - usually hosen to be least-squares or entroy error. Beause of the strong ossblty of over-fttng, gven the 19

20 Z 1 X 1 X 2 Z 2 F(Z) Y Z 3 Outut Layer Inut Layer Hdden Layer Fgure 4: A two-varables-nut, one-hdden layer, feed-forward neural network. model s enormous flexblty, a enalty omonent s usually nororated: early stong s also emloyed. The most ommon method used to ft a neural network model s bakroagaton roosed by Hofeld (1982, 1984) essentally a gradent desent aroah whh works by settng the arameters to some ntal values, buldng F from these values n the network and then omutng the error n the ftted F and the observatons. Ths error s then roagated bak to the revous hdden layer where t s aortoned to eah omonent. The roess s n turn reeated down to the revous layer and on down to the nut layer. The nut arameters are adjusted to mnmze these aortoned errors and fed forward to form a new udated verson of F. Ths roedure s terated tll onvergene. Bak-roagaton an be qute slow oeratonally and s sed u usng onjugate gradents and varable metr methods. Artfal neural networks have been wldly oular n the engneerng and artfal ntellgene ommuntes, artly beause of an effetve namng strategy that has lnked the methodology to an mtaton of the bran. The enormous flexblty rovded by any number of hdden layers s often mentoned as ts great vrtue. However, from a statstal ersetve, they are really nothng more than non-lnear statstal models wth the nherent 20