An Introductory Study on Time Series Modeling and Forecasting

Transcription

1 An Inroducory Sudy on Tme Seres Modelng and Forecasng Ranadp Adhkar R. K. Agrawal

2 ACKNOWLEDGEMENT The mely and successful compleon of he book could hardly be possble whou he helps and suppors from a lo of ndvduals. I wll ake hs opporuny o hank all of hem who helped me eher drecly or ndrecly durng hs mporan work. Frs of all I wsh o express my sncere graude and due respec o my supervsor Dr. R.K. Agrawal, Assocae Professor SC & SS, JNU. I am mmensely graeful o hm for hs valuable gudance, connuous encouragemens and posve suppors whch helped me a lo durng he perod of my work. I would lke o apprecae hm for always showng keen neres n my queres and provdng mporan suggesons. I also express whole heared hanks o my frends and classmaes for her care and moral suppors. The momens, I enjoyed wh hem durng my M.Tech course wll always reman as a happy memory hroughou my lfe. I owe a lo o my moher for her consan love and suppor. She always encouraged me o have posve and ndependen hnkng, whch really maer n my lfe. I would lke o hank her very much and share hs momen of happness wh her. Las bu no he leas I am also hankful o enre faculy and saff of SC & SS for her unselfsh help, I go whenever needed durng he course of my work. RATNADIP ADHIKARI - 3 -

3 ABSTRACT Tme seres modelng and forecasng has fundamenal mporance o varous praccal domans. Thus a lo of acve research works s gong on n hs subjec durng several years. Many mporan models have been proposed n leraure for mprovng he accuracy and effecency of me seres modelng and forecasng. The am of hs book s o presen a concse descrpon of some popular me seres forecasng models used n pracce, wh her salen feaures. In hs book, we have descrbed hree mporan classes of me seres models, vz. he sochasc, neural neworks and SVM based models, ogeher wh her nheren forecasng srenghs and weaknesses. We have also dscussed abou he basc ssues relaed o me seres modelng, such as saonary, parsmony, overfng, ec. Our dscusson abou dfferen me seres models s suppored by gvng he expermenal forecas resuls, performed on sx real me seres daases. Whle fng a model o a daase, specal care s aken o selec he mos parsmonous one. To evaluae forecas accuracy as well as o compare among dfferen models fed o a me seres, we have used he fve performance measures, vz. MSE, MAD, RMSE, MAPE and Thel s U-sascs. For each of he sx daases, we have shown he obaned forecas dagram whch graphcally depcs he closeness beween he orgnal and forecased observaons. To have auhency as well as clary n our dscusson abou me seres modelng and forecasng, we have aken he help of varous publshed research works from repued journals and some sandard books

4 CONTENTS Declaraon Cerfcae 2 Acknowledgemen 3 Absrac 4 Ls of Fgures 7 Chaper : Inroducon 9 Chaper 2: Basc Conceps of Tme Seres Modelng 2 2. Defnon of A Tme Seres Componens of A Tme Seres Examples of Tme Seres Daa Inroducon o Tme Seres Analyss Tme Seres and Sochasc Process Concep of Saonary Model Parsmony 6 Chaper 3: Tme Seres Forecasng Usng Sochasc Models 8 3. Inroducon The Auoregressve Movng Average (ARMA) Models Saonary Analyss Auocorrelaon and Paral Auocorrelaon Funcons Auoregressve Inegraed Movng Average (ARIMA) Models Seasonal Auoregressve Inegraed Movng Average (SARIMA) Models Some Nonlnear Tme Seres Models Box-Jenkns Mehodology 23 Chaper 4: Tme Seres Forecasng Usng Arfcal Neural Neworks Arfcal Neural Neworks (ANNs) The ANN Archecure Tme Lagged Neural Neworks (TLNN) Seasonal Arfcal Neural Neworks (SANN) Selecon of A Proper Nework Archecure

5 Chaper 5: Tme Seres Forecasng Usng Suppor Vecor Machnes 3 5. Concep of Suppor Vecor Machnes Inroducon o Sascal Learnng Theory Emprcal Rsk Mnmzaon (ERM) Vapnk-Chervonenks (VC) Dmenson Srucural Rsk Mnmzaon (SRM) Suppor Vecor Machnes (SVMs) Suppor Vecor Kernels SVM for Regresson (SVR) The LS-SVM Mehod The DLS-SVM Mehod 40 Chaper 6: Forecas Performance Measures Makng Real Tme Forecass: A Few Pons Descrpon of Varous Forecas Performance Measures The Mean Forecas Error (MFE) The Mean Absolue Error (MAE) The Mean Absolue Percenage Error (MAPE) The Mean Percenage Error (MPE) The Mean Squared Error (MSE) The Sum of Squared Error (SSE) The Sgned Mean Squared Error (SMSE) The Roo Mean Squared Error (RMSE) The Normalzed Mean Squared Error (NMSE) The Thel s U-sascs 45 Chaper 7: Expermenal Resuls A Bref Overvew The Canadan Lynx Daase The Wolf s Sunspo Daase The Arlne Passenger Daase The Quarerly Sales Daase The Quarerly U.S. Beer Producon Daase The Monhly USA Accdenal Deahs Daase 6 Concluson 64 References 65 Daases Sources

6 Ls of Fgures Fg. 2.: A four phase busness cycle 3 Fg. 2.2: Weekly BP/USD exchange rae seres ( ) 4 Fg. 2.3: Monhly nernaonal arlne passenger seres (Jan. 949-Dec. 960) 4 Fg. 3.: The Box-Jenkns mehodology for opmal model selecon 24 Fg. 4.: The hree-layer feed forward ANN archecure 26 Fg. 4.2: A ypcal TLNN archecure for monhly daa 27 Fg. 4.3: SANN archecure for seasonal me seres 29 Fg. 5.: Probablsc mappng of npu and oupu pons 32 Fg. 5.2: The wo-dmensonal XOR problem 34 Fg. 5.3: Suppor vecors for lnearly separable daa pons 35 Fg. 5.4: Non-lnear mappng of npu space o he feaure space 37 Fg. 7.2.: Canadan lynx daa seres (82-934) 46 Fg : Sample ACF plo for lynx seres 47 Fg : Sample PACF plo for lynx Seres 47 Fg : Forecas dagrams for lynx seres 49 Fg. 7.3.: Wolf s sunspo daa seres ( ) 49 Fg : Sample ACF plo for sunspo seres 50 Fg : Sample PACF plo for sunspo seres

7 Fg : Forecas dagrams for sunspo seres 52 Fg. 7.4.: Arlne passenger daa seres (Jan. 949-Dec. 960) 53 Fg : Sample ACF plo for arlne seres 53 Fg : Sample PACF plo for arlne seres 54 Fg : Forecas dagrams for arlne passenger seres 56 Fg. 7.5.: Quarerly sales me seres (for 6 years) 56 Fg : Forecas dagrams for quarerly sales seres 58 Fg. 7.6.: Quarerly U.S. beer producon me seres ( ) 58 Fg : Forecas dagrams for quarerly U.S. beer producon seres 60 Fg. 7.7.: Monhly USA accdenal deahs me seres ( ) 6 Fg : Forecas dagrams for monhly USA accdenal deahs seres

8 Inroducon Chaper- Tme seres modelng s a dynamc research area whch has araced aenons of researchers communy over las few decades. The man am of me seres modelng s o carefully collec and rgorously sudy he pas observaons of a me seres o develop an approprae model whch descrbes he nheren srucure of he seres. Ths model s hen used o generae fuure values for he seres,.e. o make forecass. Tme seres forecasng hus can be ermed as he ac of predcng he fuure by undersandng he pas [3]. Due o he ndspensable mporance of me seres forecasng n numerous praccal felds such as busness, economcs, fnance, scence and engneerng, ec. [7, 8, 0], proper care should be aken o f an adequae model o he underlyng me seres. I s obvous ha a successful me seres forecasng depends on an approprae model fng. A lo of effors have been done by researchers over many years for he developmen of effcen models o mprove he forecasng accuracy. As a resul, varous mporan me seres forecasng models have been evolved n leraure. One of he mos popular and frequenly used sochasc me seres models s he Auoregressve Inegraed Movng Average (ARIMA) [6, 8, 2, 23] model. The basc assumpon made o mplemen hs model s ha he consdered me seres s lnear and follows a parcular known sascal dsrbuon, such as he normal dsrbuon. ARIMA model has subclasses of oher models, such as he Auoregressve (AR) [6, 2, 23], Movng Average (MA) [6, 23] and Auoregressve Movng Average (ARMA) [6, 2, 23] models. For seasonal me seres forecasng, Box and Jenkns [6] had proposed a que successful varaon of ARIMA model, vz. he Seasonal ARIMA (SARIMA) [3, 6, 23]. The populary of he ARIMA model s manly due o s flexbly o represen several varees of me seres wh smplcy as well as he assocaed Box-Jenkns mehodology [3, 6, 8, 23] for opmal model buldng process. Bu he severe lmaon of hese models s he pre-assumed lnear form of he assocaed me seres whch becomes nadequae n many praccal suaons. To overcome hs drawback, varous non-lnear sochasc models have been proposed n leraure [7, 8, 28]; however from mplemenaon pon of vew hese are no so sragh-forward and smple as he ARIMA models. Recenly, arfcal neural neworks (ANNs) have araced ncreasng aenons n he doman of me seres forecasng [8, 3, 20]. Alhough nally bologcally nspred, bu laer on ANNs have been successfully appled n many dfferen areas, especally for forecasng - 9 -

9 and classfcaon purposes [3, 20]. The excellen feaure of ANNs, when appled o me seres forecasng problems s her nheren capably of non-lnear modelng, whou any presumpon abou he sascal dsrbuon followed by he observaons. The approprae model s adapvely formed based on he gven daa. Due o hs reason, ANNs are daa-drven and self-adapve by naure [5, 8, 20]. Durng he pas few years a subsanal amoun of research works have been carred ou owards he applcaon of neural neworks for me seres modelng and forecasng. A sae-of-he-ar dscusson abou he recen works n neural neworks for ne seres forecasng has been presened by Zhang e al. n 998 [5]. There are varous ANN forecasng models n leraure. The mos common and popular among hem are he mul-layer perceprons (MLPs), whch are characerzed by a sngle hdden layer Feed Forward Nework (FNN) [5,8]. Anoher wdely used varaon of FNN s he Tme Lagged Neural Nework (TLNN) [, 3]. In 2008, C. Hamzaceb [3] had presened a new ANN model, vz. he Seasonal Arfcal Neural Nework (SANN) model for seasonal me seres forecasng. Hs proposed model s surprsngly smple and also has been expermenally verfed o be que successful and effcen n forecasng seasonal me seres. Offcourse, here are many oher exsng neural nework srucures n leraure due o he connuous ongong research works n hs feld. However, n he presen book we shall manly concenrae on he above menoned ANN forecasng models. A major breakhrough n he area of me seres forecasng occurred wh he developmen of Vapnk s suppor vecor machne (SVM) concep [8, 24, 30, 3]. Vapnk and hs co-workers desgned SVM a he AT & T Bell laboraores n 995 [24, 29, 33]. The nal am of SVM was o solve paern classfcaon problems bu aferwards hey have been wdely appled n many oher felds such as funcon esmaon, regresson, sgnal processng and me seres predcon problems [24, 3, 34]. The remarkable characersc of SVM s ha s no only desned for good classfcaon bu also nended for a beer generalzaon of he ranng daa. For hs reason he SVM mehodology has become one of he well-known echnques, especally for me seres forecasng problems n recen years. The objecve of SVM s o use he srucural rsk mnmzaon (SRM) [24, 29, 30] prncple o fnd a decson rule wh good generalzaon capacy. In SVM, he soluon o a parcular problem only depends upon a subse of he ranng daa pons, whch are ermed as he suppor vecors [24, 29, 33]. Anoher mporan feaure of SVM s ha here he ranng s equvalen o solvng a lnearly consraned quadrac opmzaon problem. So he soluon obaned by applyng SVM mehod s always unque and globally opmal, unlke he oher radonal sochasc or neural nework mehods [24]. Perhaps he mos amazng propery of SVM s ha he qualy and complexy of he soluon can be ndependenly conrolled, rrespecve of he dmenson of - 0 -

10 he npu space [9, 29]. Usually n SVM applcaons, he npu pons are mapped o a hgh dmensonal feaure space, wh he help of some specal funcons, known as suppor vecor kernels [8, 29, 34], whch ofen yelds good generalzaon even n hgh dmensons. Durng he pas few years numerous SVM forecasng models have been developed by researchers. In hs book, we shall presen an overvew of he mporan fundamenal conceps of SVM and hen dscuss abou he Leas-square SVM (LS-SVM) [9] and Dynamc Leas-square SVM (LS- SVM) [34] whch are wo popular SVM models for me seres forecasng. The objecve of hs book s o presen a comprehensve dscusson abou he hree wdely popular approaches for me seres forecasng, vz. he sochasc, neural neworks and SVM approaches. Ths book conans seven chapers, whch are organzed as follows: Chaper 2 gves an nroducon o he basc conceps of me seres modelng, ogeher wh some assocaed deas such as saonary, parsmony, ec. Chaper 3 s desgned o dscuss abou he varous sochasc me seres models. These nclude he Box-Jenkns or ARIMA models, he generalzed ARFIMA models and he SARIMA model for lnear me seres forecasng as well as some non-lnear models such as ARCH, NMA, ec. In Chaper 4 we have descrbed he applcaon of neural neworks n me seres forecasng, ogeher wh wo recenly developed models, vz. TLNN [, 3] and SANN [3]. Chaper 5 presens a dscusson abou he SVM conceps and s usefulness n me seres forecasng problems. In hs chaper we have also brefly dscussed abou wo newly proposed models, vz. LS-SVM [9] and DLS-SVM [34] whch have ganed mmense populares n me seres forecasng applcaons. In Chaper 6, we have nroduced abou en mporan forecas performance measures, ofen used n leraure, ogeher wh her salen feaures. Chaper 7 presens our expermenal forecasng resuls n erms of fve performance measures, obaned on sx real me seres daases, ogeher wh he assocaed forecas dagrams. Afer compleon of hese seven chapers, we have gven a bref concluson of our work as well as he prospecve fuure am n hs feld. - -

11 Basc Conceps of Tme Seres Modelng Chaper-2 2. Defnon of A Tme Seres A me seres s a sequenal se of daa pons, measured ypcally over successve mes. I s mahemacally defned as a se of vecors x ( ), = 0,,2,... where represens he me elapsed [2, 23, 3]. The varable x () s reaed as a random varable. The measuremens aken durng an even n a me seres are arranged n a proper chronologcal order. A me seres conanng records of a sngle varable s ermed as unvarae. Bu f records of more han one varable are consdered, s ermed as mulvarae. A me seres can be connuous or dscree. In a connuous me seres observaons are measured a every nsance of me, whereas a dscree me seres conans observaons measured a dscree pons of me. For example emperaure readngs, flow of a rver, concenraon of a chemcal process ec. can be recorded as a connuous me seres. On he oher hand populaon of a parcular cy, producon of a company, exchange raes beween wo dfferen currences may represen dscree me seres. Usually n a dscree me seres he consecuve observaons are recorded a equally spaced me nervals such as hourly, daly, weekly, monhly or yearly me separaons. As menoned n [23], he varable beng observed n a dscree me seres s assumed o be measured as a connuous varable usng he real number scale. Furhermore a connuous me seres can be easly ransformed o a dscree one by mergng daa ogeher over a specfed me nerval. 2.2 Componens of a Tme Seres A me seres n general s supposed o be affeced by four man componens, whch can be separaed from he observed daa. These componens are: Trend, Cyclcal, Seasonal and Irregular componens. A bref descrpon of hese four componens s gven here. The general endency of a me seres o ncrease, decrease or sagnae over a long perod of me s ermed as Secular Trend or smply Trend. Thus, can be sad ha rend s a long erm movemen n a me seres. For example, seres relang o populaon growh, number of houses n a cy ec. show upward rend, whereas downward rend can be observed n seres relang o moraly raes, epdemcs, ec. Seasonal varaons n a me seres are flucuaons whn a year durng he season. The mporan facors causng seasonal varaons are: clmae and weaher condons, cusoms, radonal habs, ec. For example sales of ce-cream ncrease n summer, sales of woolen clohs ncrease n wner. Seasonal varaon s an mporan facor for busnessmen, shopkeeper and producers for makng proper fuure plans

12 The cyclcal varaon n a me seres descrbes he medum-erm changes n he seres, caused by crcumsances, whch repea n cycles. The duraon of a cycle exends over longer perod of me, usually wo or more years. Mos of he economc and fnancal me seres show some knd of cyclcal varaon. For example a busness cycle consss of four phases, vz. ) Prospery, ) Declne, ) Depresson and v) Recovery. Schemacally a ypcal busness cycle can be shown as below: Fg. 2.: A four phase busness cycle Irregular or random varaons n a me seres are caused by unpredcable nfluences, whch are no regular and also do no repea n a parcular paern. These varaons are caused by ncdences such as war, srke, earhquake, flood, revoluon, ec. There s no defned sascal echnque for measurng random flucuaons n a me seres. Consderng he effecs of hese four componens, wo dfferen ypes of models are generally used for a me seres vz. Mulplcave and Addve models. Mulplcave Model: Y ( ) = T ( ) S( ) C( ) I( ). Addve Model: Y ( ) = T ( ) + S( ) + C( ) + I( ). Here Y () s he observaon and T (), S (), C () and I() are respecvely he rend, seasonal, cyclcal and rregular varaon a me. Mulplcave model s based on he assumpon ha he four componens of a me seres are no necessarly ndependen and hey can affec one anoher; whereas n he addve model s assumed ha he four componens are ndependen of each oher. 2.3 Examples of Tme Seres Daa Tme seres observaons are frequenly encounered n many domans such as busness, economcs, ndusry, engneerng and scence, ec [7, 8, 0]. Dependng on he naure of analyss and praccal need, here can be varous dfferen knds of me seres. To vsualze he - 3 -

13 basc paern of he daa, usually a me seres s represened by a graph, where he observaons are ploed agans correspondng me. Below we show wo me seres plos: Fg. 2.2: Weekly BP/USD exchange rae seres ( ) Fg. 2.3: Monhly nernaonal arlne passenger seres (Jan. 949-Dec. 960) The frs me seres s aken from [8] and represens he weekly exchange rae beween Brsh pound and US dollar from 980 o 933. The second one s a seasonal me seres, consdered n [3, 6, ] and shows he number of nernaonal arlne passengers (n housands) beween Jan. 949 o Dec. 960 on a monhly bass

14 2.4 Inroducon o Tme Seres Analyss In pracce a suable model s fed o a gven me seres and he correspondng parameers are esmaed usng he known daa values. The procedure of fng a me seres o a proper model s ermed as Tme Seres Analyss [23]. I comprses mehods ha aemp o undersand he naure of he seres and s ofen useful for fuure forecasng and smulaon. In me seres forecasng, pas observaons are colleced and analyzed o develop a suable mahemacal model whch capures he underlyng daa generang process for he seres [7, 8]. The fuure evens are hen predced usng he model. Ths approach s parcularly useful when here s no much knowledge abou he sascal paern followed by he successve observaons or when here s a lack of a sasfacory explanaory model. Tme seres forecasng has mporan applcaons n varous felds. Ofen valuable sraegc decsons and precauonary measures are aken based on he forecas resuls. Thus makng a good forecas,.e. fng an adequae model o a me seres s vary mporan. Over he pas several decades many effors have been made by researchers for he developmen and mprovemen of suable me seres forecasng models. 2.5 Tme Seres and Sochasc Process A me seres s non-deermnsc n naure,.e. we canno predc wh cerany wha wll occur n fuure. Generally a me seres { ( ), = 0,, 2,... } x s assumed o follow ceran probably model [2] whch descrbes he jon dsrbuon of he random varable x. The mahemacal expresson descrbng he probably srucure of a me seres s ermed as a sochasc process [23]. Thus he sequence of observaons of he seres s acually a sample realzaon of he sochasc process ha produced. A usual assumpon s ha he me seres varables x are ndependen and dencally dsrbued (..d) followng he normal dsrbuon. However as menoned n [2], an neresng pon s ha me seres are n fac no exacly..d; hey follow more or less some regular paern n long erm. For example f he emperaure oday of a parcular cy s exremely hgh, hen can be reasonably presumed ha omorrow s emperaure wll also lkely o be hgh. Ths s he reason why me seres forecasng usng a proper echnque, yelds resul close o he acual value. 2.6 Concep of Saonary The concep of saonary of a sochasc process can be vsualzed as a form of sascal equlbrum [23]. The sascal properes such as mean and varance of a saonary process do no depend upon me. I s a necessary condon for buldng a me seres model ha s useful for fuure forecasng. Furher, he mahemacal complexy of he fed model reduces wh hs assumpon. There are wo ypes of saonary processes whch are defned below: - 5 -

15 A process { ( ), = 0,, 2,... } x s Srongly Saonary or Srcly Saonary f he jon probably dsrbuon funcon of { x,..., x,... x x } x s, s + + s, + s s ndependen of for all s. Thus for a srong saonary process he jon dsrbuon of any possble se of random varables from he process s ndependen of me [2, 23]. However for praccal applcaons, he assumpon of srong saonary s no always needed and so a somewha weaker form s consdered. A sochasc process s sad o be Weakly Saonary of order k f he sascal momens of he process up o ha order depend only on me dfferences and no upon he me of occurrences of he daa beng used o esmae he momens [2, 2, 23]. For example a sochasc process { ( ), = 0,, 2,... } x s second order saonary [2, 23] f has me ndependen mean and varance and he covarance values Cov( x, x s ) depend only on s. I s mporan o noe ha neher srong nor weak saonary mples he oher. However, a weakly saonary process followng normal dsrbuon s also srongly saonary [2]. Some mahemacal ess lke he one gven by Dckey and Fuller [2] are generally used o deec saonary n a me seres daa. As menoned n [6, 23], he concep of saonary s a mahemacal dea consruced o smplfy he heorecal and praccal developmen of sochasc processes. To desgn a proper model, adequae for fuure forecasng, he underlyng me seres s expeced o be saonary. Unforunaely s no always he case. As saed by Hpel and McLeod [23], he greaer he me span of hsorcal observaons, he greaer s he chance ha he me seres wll exhb non-saonary characerscs. However for relavely shor me span, one can reasonably model he seres usng a saonary sochasc process. Usually me seres, showng rend or seasonal paerns are non-saonary n naure. In such cases, dfferencng and power ransformaons are ofen used o remove he rend and o make he seres saonary. In he nex chaper we shall dscuss abou he seasonal dfferencng echnque appled o make a seasonal me seres saonary. 2.7 Model Parsmony Whle buldng a proper me seres model we have o consder he prncple of parsmony [2, 7, 8, 23]. Accordng o hs prncple, always he model wh smalles possble number of parameers s o be seleced so as o provde an adequae represenaon of he underlyng me seres daa [2]. Ou of a number of suable models, one should consder he smples one, sll mananng an accurae descrpon of nheren properes of he me seres. The dea of model parsmony s smlar o he famous Occam s razor prncple [23]. As dscussed by Hpel and McLeod [23], one aspec of hs prncple s ha when face wh a number of compeng and - 6 -

16 adequae explanaons, pck he mos smple one. The Occam s razor provdes consderable nheren nformaons, when appled o logcal analyss. Moreover, he more complcaed he model, he more possbles wll arse for deparure from he acual model assumpons. Wh he ncrease of model parameers, he rsk of overfng also subsequenly ncreases. An over fed me seres model may descrbe he ranng daa very well, bu may no be suable for fuure forecasng. As poenal overfng affecs he ably of a model o forecas well, parsmony s ofen used as a gudng prncple o overcome hs ssue. Thus n summary can be sad ha, whle makng me seres forecass, genune aenon should be gven o selec he mos parsmonous model among all oher possbles

17 Tme Seres Forecasng Usng Sochasc Models Chaper-3 3. Inroducon In he prevous chaper we have dscussed abou he fundamenals of me seres modelng and forecasng. The selecon of a proper model s exremely mporan as reflecs he underlyng srucure of he seres and hs fed model n urn s used for fuure forecasng. A me seres model s sad o be lnear or non-lnear dependng on wheher he curren value of he seres s a lnear or non-lnear funcon of pas observaons. In general models for me seres daa can have many forms and represen dfferen sochasc processes. There are wo wdely used lnear me seres models n leraure, vz. Auoregressve (AR) [6, 2, 23] and Movng Average (MA) [6, 23] models. Combnng hese wo, he Auoregressve Movng Average (ARMA) [6, 2, 2, 23] and Auoregressve Inegraed Movng Average (ARIMA) [6, 2, 23] models have been proposed n leraure. The Auoregressve Fraconally Inegraed Movng Average (ARFIMA) [9, 7] model generalzes ARMA and ARIMA models. For seasonal me seres forecasng, a varaon of ARIMA, vz. he Seasonal Auoregressve Inegraed Movng Average (SARIMA) [3, 6, 23] model s used. ARIMA model and s dfferen varaons are based on he famous Box-Jenkns prncple [6, 8, 2, 23] and so hese are also broadly known as he Box-Jenkns models. Lnear models have drawn much aenon due o her relave smplcy n undersandng and mplemenaon. However many praccal me seres show non-lnear paerns. For example, as menoned by R. Parrell n [28], non-lnear models are approprae for predcng volaly changes n economc and fnancal me seres. Consderng hese facs, varous nonlnear models have been suggesed n leraure. Some of hem are he famous Auoregressve Condonal Heeroskedascy (ARCH) [9, 28] model and s varaons lke Generalzed ARCH (GARCH) [9, 28], Exponenal Generalzed ARCH (EGARCH) [9] ec., he Threshold Auoregressve (TAR) [8, 0] model, he Non-lnear Auoregressve (NAR) [7] model, he Nonlnear Movng Average (NMA) [28] model, ec. In hs chaper we shall dscuss abou he mporan lnear and non-lnear sochasc me seres models wh her dfferen properes. Ths chaper wll provde a background for he upcomng chapers, n whch we shall sudy oher models used for me seres forecasng. 3.2 The Auoregressve Movng Average (ARMA) Models An ARMA(p, q) model s a combnaon of AR(p) and MA(q) models and s suable for unvarae me seres modelng. In an AR(p) model he fuure value of a varable s assumed o - 8 -

18 be a lnear combnaon of p pas observaons and a random error ogeher wh a consan erm. Mahemacally he AR(p) model can be expressed as [2, 23]: Here y p = c + y + ε = c + ϕ y + ϕ 2 y ϕ p y p = ϕ + ε (3.) y and ε are respecvely he acual value and random error (or random shock) a me perod, ϕ (,2,..., p) are model parameers and c s a consan. The neger consan p s = known as he order of he model. Somemes he consan erm s omed for smplcy. Usually For esmang parameers of an AR process usng he gven me seres, he Yule- Walker equaons [23] are used. Jus as an AR(p) model regress agans pas values of he seres, an MA(q) model uses pas errors as he explanaory varables. The MA(q) model s gven by [2, 2, 23]: y q = + θ jε j + ε = μ + θε + θ 2ε θ qε q j= μ + ε Here μ s he mean of he seres, θ ( j,2,..., q) are he model parameers and q s he j = order of he model. The random shocks are assumed o be a whe nose [2, 23] process,.e. a sequence of ndependen and dencally dsrbued (..d) random varables wh zero mean 2 and a consan varance σ. Generally, he random shocks are assumed o follow he ypcal normal dsrbuon. Thus concepually a movng average model s a lnear regresson of he curren observaon of he me seres agans he random shocks of one or more pror observaons. Fng an MA model o a me seres s more complcaed han fng an AR model because n he former one he random error erms are no fore-seeable. (3.2) Auoregressve (AR) and movng average (MA) models can be effecvely combned ogeher o form a general and useful class of me seres models, known as he ARMA models. Mahemacally an ARMA(p, q) model s represened as [2, 2, 23]: y p q = c ϕ y = j= Here he model orders ε θ ε (3.3) j j p, q refer o p auoregressve and q movng average erms. Usually ARMA models are manpulaed usng he lag operaor [2, 23] noaon. The lag or backshf operaor s defned as Ly = y. Polynomals of lag operaor or lag polynomals are used o represen ARMA models as follows [2]: AR(p) model: ε = ϕ L) y. ( - 9 -

19 MA(q) model: y = θ L) ε. ( ARMA(p, q) model: ϕ L) y = θ ( L) ε. p = ( q Here ϕ ( L) = ϕ L and θ ( L) = + θ L. j= j j I s shown n [23] ha an mporan propery of AR(p) process s nverbly,.e. an AR(p) process can always be wren n erms of an MA( ) process. Whereas for an MA(q) process o be nverble, all he roos of he equaon θ ( L) = 0 mus le ousde he un crcle. Ths condon s known as he Inverbly Condon for an MA process. 3.3 Saonary Analyss When an AR(p) process s represened as ε = ϕ(l) y, hen ϕ ( L) = 0 s known as he characersc equaon for he process. I s proved by Box and Jenkns [6] ha a necessary and suffcen condon for he AR(p) process o be saonary s ha all he roos of he characersc equaon mus fall ousde he un crcle. Hpel and McLeod [23] menoned anoher smple algorhm (by Schur and Pagano) for deermnng saonary of an AR process. For example as shown n [2] he AR() model 2 c σ wh a consan mean μ = and consan varance γ 0 =. 2 ϕ ϕ y ϕ, = c + ϕ y + ε s saonary when < An MA(q) process s always saonary, rrespecve of he values he MA parameers [23]. The condons regardng saonary and nverbly of AR and MA processes also hold for an ARMA process. An ARMA(p, q) process s saonary f all he roos of he characersc equaon ϕ ( L) = 0 le ousde he un crcle. Smlarly, f all he roos of he lag equaon θ ( L) = 0 le ousde he un crcle, hen he ARMA(p, q) process s nverble and can be expressed as a pure AR process. 3.4 Auocorrelaon and Paral Auocorrelaon Funcons (ACF and PACF) To deermne a proper model for a gven me seres daa, s necessary o carry ou he ACF and PACF analyss. These sascal measures reflec how he observaons n a me seres are relaed o each oher. For modelng and forecasng purpose s ofen useful o plo he ACF and PACF agans consecuve me lags. These plos help n deermnng he order of AR and MA erms. Below we gve her mahemacal defnons: For a me seres{ x ( ), = 0,, 2,... } he Auocovarance [2, 23] a lag k s defned as: γ Cov ( x, x ) = E[( x μ)( x μ)] (3.4) k = + k + k

20 The Auocorrelaon Coeffen [2, 23] a lag k s defned as: γ k ρ = (3.5) k γ 0 Here μ s he mean of he me seres,.e. μ = E[ x ]. The auocovarance a lag zero.e. γ 0 s he varance of he me seres. From he defnon s clear ha he auocorrelaon coeffcen ρ k s dmensonless and so s ndependen of he scale of measuremen. Also, clearly ρ k. Sascans Box and Jenkns [6] ermed γ k as he heorecal Auocovarance Funcon (ACVF) and ρ k as he heorecal Auocorrelaon Funcon (ACF). Anoher measure, known as he Paral Auucorrelaon Funcon (PACF) s used o measure he correlaon beween an observaon k perod ago and he curren observaon, afer conrollng for observaons a nermedae lags (.e. a lags < k ) [2]. A lag, PACF() s same as ACF(). The dealed formulae for calculang PACF are gven n [6, 23]. Normally, he sochasc process governng a me seres s unknown and so s no possble o deermne he acual or heorecal ACF and PACF values. Raher hese values are o be esmaed from he ranng daa,.e. he known me seres a hand. The esmaed ACF and PACF values from he ranng daa are respecvely ermed as sample ACF and PACF [6, 23]. As gven n [23], he mos approprae sample esmae for he ACVF a lag k s c k = n n k = ( x μ )( x μ) (3.6) + k Then he esmae for he sample ACF a lag k s gven by ck rk = (3.7) c 0 Here { ( ), = 0,,2,... } x s he ranng seres of sze n wh mean μ. As explaned by Box and Jenkns [6], he sample ACF plo s useful n deermnng he ype of model o f o a me seres of lengh N. Snce ACF s symmercal abou lag zero, s only requred o plo he sample ACF for posve lags, from lag one onwards o a maxmum lag of abou N/4. The sample PACF plo helps n denfyng he maxmum order of an AR process. The mehods for calculang ACF and PACF for ARMA models are descrbed n [23]. We shall demonsrae he use of hese plos for our praccal daases n Chaper Auoregressve Inegraed Movng Average (ARIMA) Models The ARMA models, descrbed above can only be used for saonary me seres daa. However n pracce many me seres such as hose relaed o soco-economc [23] and - 2 -

21 busness show non-saonary behavor. Tme seres, whch conan rend and seasonal paerns, are also non-saonary n naure [3, ]. Thus from applcaon vew pon ARMA models are nadequae o properly descrbe non-saonary me seres, whch are frequenly encounered n pracce. For hs reason he ARIMA model [6, 23, 27] s proposed, whch s a generalzaon of an ARMA model o nclude he case of non-saonary as well. In ARIMA models a non-saonary me seres s made saonary by applyng fne dfferencng of he daa pons. The mahemacal formulaon of he ARIMA(p,d,q) model usng lag polynomals s gven below [23, 27]: d ϕ( L)( L) y = θ ( L) ε,. e. p q ϕ L = j= d j ( L) y = + θ j L ε Here, p, d and q are negers greaer han or equal o zero and refer o he order of he auoregressve, negraed, and movng average pars of he model respecvely. The neger d conrols he level of dfferencng. Generally d= s enough n mos cases. When d=0, hen reduces o an ARMA(p,q) model. An ARIMA(p,0,0) s nohng bu he AR(p) model and ARIMA(0,0,q) s he MA(q) model. ARIMA(0,,0),.e. y = y + ε s a specal one and known as he Random Walk model [8, 2, 2]. I s wdely used for non-saonary daa, lke economc and sock prce seres. (3.8) A useful generalzaon of ARIMA models s he Auoregressve Fraconally Inegraed Movng Average (ARFIMA) model, whch allows non-neger values of he dfferencng parameer d. ARFIMA has useful applcaon n modelng me seres wh long memory [7]. In hs model he expanson of he erm ( L) d s o be done by usng he general bnomal heorem. Varous conrbuons have been made by researchers owards he esmaon of he general ARFIMA parameers. 3.6 Seasonal Auoregressve Inegraed Movng Average (SARIMA) Models The ARIMA model (3.8) s for non-seasonal non-saonary daa. Box and Jenkns [6] have generalzed hs model o deal wh seasonaly. Ther proposed model s known as he Seasonal ARIMA (SARIMA) model. In hs model seasonal dfferencng of approprae order s used o remove non-saonary from he seres. A frs order seasonal dfference s he dfference beween an observaon and he correspondng observaon from he prevous year and s calculaed as z = y y. For monhly me seres s = 2 and for quarerly me seres s s = 4. Ths model s generally ermed as he s SARIMA ( p, d, q) ( P, D, Q) model

22 The mahemacal formulaon of a polynomals s gven below [3]: s SARIMA ( p, d, q) ( P, D, Q) model n erms of lag Φ P. e. Φ s ( L ) ϕ ( L)( L) P p s ( L ) ϕ ( L) z p d s ( L ) = Θ Q D y s ( L ) θ ( L) ε. q = Θ Q s ( L ) θ ( L) ε, q (3.9) Here z s he seasonally dfferenced seres. 3.7 Some Nonlnear Tme Seres Models So far we have dscussed abou lnear me seres models. As menoned earler, nonlnear models should also be consdered for beer me seres analyss and forecasng. Campbell, Lo and McKnley (997) made mporan conrbuons owards hs drecon. Accordng o hem almos all non-lnear me seres can be dvded no wo branches: one ncludes models nonlnear n mean and oher ncludes models non-lnear n varance (heeroskedasc). As an llusrave example, here we presen wo nonlnear me seres models from [28]: 2 Nonlnear Movng Average (NMA) Model: y ε + αε. Ths model s non-lnear n mean bu no n varance. = 2 Eagle s (982) ARCH Model: y = ε + α ε. Ths model s heeroskedasc,.e. nonlnear n varance, bu lnear n mean. Ths model has several oher varaons, lke GARCH, EGARCH ec. 3.8 Box-Jenkns Mehodology Afer descrbng varous me seres models, he nex ssue o our concern s how o selec an approprae model ha can produce accurae forecas based on a descrpon of hsorcal paern n he daa and how o deermne he opmal model orders. Sascans George Box and Gwlym Jenkns [6] developed a praccal approach o buld ARIMA model, whch bes f o a gven me seres and also sasfy he parsmony prncple. Ther concep has fundamenal mporance on he area of me seres analyss and forecasng [8, 27]. The Box-Jenkns mehodology does no assume any parcular paern n he hsorcal daa of he seres o be forecased. Raher, uses a hree sep erave approach of model denfcaon, parameer esmaon and dagnosc checkng o deermne he bes parsmonous model from a general class of ARIMA models [6, 8, 2, 27]. Ths hree-sep process s repeaed several mes unl a sasfacory model s fnally seleced. Then hs model can be used for forecasng fuure values of he me seres. The Box-Jenkns forecas mehod s schemacally shown n Fg. 3.:

23 Posulae a general class of ARIMA model Idenfy he model, whch can be enavely eneraned No Esmae parameers n he enavely eneraned model Dagnoss Checkng Is he model adequae? Yes Use hs model o generae forecas Fg. 3.: The Box-Jenkns mehodology for opmal model selecon A crucal sep n an approprae model selecon s he deermnaon of opmal model parameers. One creron s ha he sample ACF and PACF, calculaed from he ranng daa should mach wh he correspondng heorecal or acual values [, 3, 23]. Oher wdely used measures for model denfcaon are Akake Informaon Creron (AIC) [, 3] and Bayesan Informaon Creron (BIC) [, 3] whch are defned below []: 2 AIC( p) = n ln( ˆ σ e n) + 2 p 2 BIC( p) = n ln( ˆ σ e n) + p + p ln( n) Here n s he number of effecve observaons, used o f he model, p s he number of 2 parameers n he model and ˆ σ e s he sum of sample squared resduals. The opmal model order s chosen by he number of model parameers, whch mnmzes eher AIC or BIC. Oher smlar crera have also been proposed n leraure for opmal model denfcaon

24 Chaper-4 Tme Seres Forecasng Usng Arfcal Neural Neworks 4. Arfcal Neural Neworks (ANNs) In he prevous Chaper we have dscussed he mporan sochasc mehods for me seres modelng and forecasng. Arfcal neural neworks (ANNs) approach has been suggesed as an alernave echnque o me seres forecasng and ganed mmense populary n las few years. The basc objecve of ANNs was o consruc a model for mmckng he nellgence of human bran no machne [3, 20]. Smlar o he work of a human bran, ANNs ry o recognze regulares and paerns n he npu daa, learn from experence and hen provde generalzed resuls based on her known prevous knowledge. Alhough he developmen of ANNs was manly bologcally movaed, bu aferwards hey have been appled n many dfferen areas, especally for forecasng and classfcaon purposes [3, 20]. Below we shall menon he salen feaures of ANNs, whch make hem que favore for me seres analyss and forecasng. Frs, ANNs are daa-drven and self-adapve n naure [5, 20]. There s no need o specfy a parcular model form or o make any a pror assumpon abou he sascal dsrbuon of he daa; he desred model s adapvely formed based on he feaures presened from he daa. Ths approach s que useful for many praccal suaons, where no heorecal gudance s avalable for an approprae daa generaon process. Second, ANNs are nherenly non-lnear, whch makes hem more praccal and accurae n modelng complex daa paerns, as opposed o varous radonal lnear approaches, such as ARIMA mehods [5, 8, 20]. There are many nsances, whch sugges ha ANNs made que beer analyss and forecasng han varous lnear models. Fnally, as suggesed by Hornk and Snchcombe [22], ANNs are unversal funconal approxmaors. They have shown ha a nework can approxmae any connuous funcon o any desred accuracy [5, 22]. ANNs use parallel processng of he nformaon from he daa o approxmae a large class of funcons wh a hgh degree of accuracy. Furher, hey can deal wh suaon, where he npu daa are erroneous, ncomplee or fuzzy [20]. 4.2 The ANN Archecure The mos wdely used ANNs n forecasng problems are mul-layer perceprons (MLPs), whch use a sngle hdden layer feed forward nework (FNN) [5,8]. The model s characerzed by a nework of hree layers, vz. npu, hdden and oupu layer, conneced by acyclc lnks. There may be more han one hdden layer. The nodes n varous layers are also known as

25 processng elemens. The hree-layer feed forward archecure of ANN models can be dagrammacally depced as below: Fg. 4.: The hree-layer feed forward ANN archecure The oupu of he model s compued usng he followng mahemacal expresson [7]: q p y = α 0 + α j g β 0 j + β j y + ε, (4.) j= = Here y ( =,2,..., p) are he p npus and y s he oupu. The negers p, q are he number of npu and hdden nodes respecvely. α ( j 0,,2,..., q) and β ( = 0,,2,..., p; j 0,,2,..., q) j = are he connecon weghs and ε s he random shock; α 0 and j = β 0 j are he bas erms. Usually, he logsc sgmod funcon g( x) = s appled as he nonlnear acvaon funcon. Oher x + e acvaon funcons, such as lnear, hyperbolc angen, Gaussan, ec. can also be used [20]. The feed forward ANN model (4.) n fac performs a non-lnear funconal mappng from he pas observaons of he me seres o he fuure value,.e. = f ( y, y 2,... y p, ) where w s a vecor of all parameers and f s a funcon deermned by he nework srucure and connecon weghs [5, 8] y w + ε,

26 To esmae he connecon weghs, non-lnear leas square procedures are used, whch are based on he mnmzaon of he error funcon [3]: 2 2 F( Ψ) = e = ( y yˆ ) (4.2) Here Ψ s he space of all connecon weghs. The opmzaon echnques used for mnmzng he error funcon (4.2) are referred as Learnng Rules. The bes-known learnng rule n leraure s he Backpropagaon or Generalzed Dela Rule [3, 20]. 4.3 Tme Lagged Neural Neworks (TLNN) In he FNN formulaon, descrbed above, he npu nodes are he successve observaons of he me seres,.e. he arge x s a funcon of he values x,( =,2,..., p) where p s he number of npu nodes. Anoher varaon of FNN, vz. he TLNN archecure [, 3] s also wdely used. In TLNN, he npu nodes are he me seres values a some parcular lags. For example, a ypcal TLNN for a me seres, wh seasonal perod s = 2 can conan he npu nodes as he lagged values a me, 2 and 2. The value a me s o be forecased usng he values a lags, 2 and 2. x x 2 xˆ x 2 Inpu Layer Hdden Layer Oupu Layer Fg. 4.2: A ypcal TLNN archecure for monhly daa

27 In addon, here s a consan npu erm, whch may be convenenly aken as and hs s conneced o every neuron n he hdden and oupu layer. The nroducon of hs consan npu un avods he necessy of separaely nroducng a bas erm. For a TLNN wh one hdden level, he general predcon equaon for compung a forecas may be wren as []: xˆ = φ0 + wc0 + wh0φh wch + wh x j (4.3) h Here, he seleced pas observaons x j, x j,..., x j are he npu erms, { w } 2 k ch are he weghs for he connecons beween he consan npu and hdden neurons and w c0 s he wegh of he drec connecon beween he consan npu and he oupu. Also { w h} and { w } h0 denoe he weghs for oher connecons beween he npu and hdden neurons and beween he hdden and oupu neurons respecvely. φ h and φ 0 are he hdden and oupu layer acvaon funcons respecvely. Faraway and Chafeld [] used he noaon NN j, j,..., jk ; ) o denoe he TLNN wh npus a lags j, ( 2 h, j2,... jk and h hdden neurons. We shall also adop hs noaon n our upcomng expermens. Thus Fg. 4.2 represens an NN (, 2, 2; 3) model. 4.4 Seasonal Arfcal Neural Neworks (SANN) The SANN srucure s proposed by C. Hamzaceb [3] o mprove he forecasng performance of ANNs for seasonal me seres daa. The proposed SANN model does no requre any preprocessng of raw daa. Also SANN can learn he seasonal paern n he seres, whou removng hem, conrary o some oher radonal approaches, such as SARIMA, dscussed n Chaper 3. The auhor has emprcally verfed he good forecasng ably of SANN on four praccal me daa ses. We have also used hs model n our curren work on wo new seasonal me seres and obaned que sasfacory resuls. Here we presen a bref overvew of SANN model as proposed n [3]. In hs model, he seasonal parameer s s used o deermne he number of npu and oupu neurons. Ths consderaon makes he model surprsngly smple for undersandng and mplemenaon. The h and (+) h seasonal perod observaons are respecvely used as he values of npu and oupu neurons n hs nework srucure. Each seasonal perod s composed of a number of observaons. Dagrammacally an SANN srucure can be shown as [3]:

28 Fg. 4.3: SANN archecure for seasonal me seres Mahemacal expresson for he oupu of he model s [3]: Y m s α l + w jl f j + vjy ; l,2,3,..., s. (4.4) j= = 0 = + l θ = Here Y + l ( l =,2,3,..., s) are he predcons for he fuure s perods and Y ( = 0,,2,..., s ) are he observaons of he prevous s perods; v j ( = 0,,2,..., s ; j =,2,3,..., m) are weghs of connecons from npu nodes o hdden nodes and w jl ( j =,2,3,..., m; l =,2,3,..., s) are weghs of connecons from hdden nodes o oupu nodes. Also α ( l,2,3,..., s) and l = θ ( j,2,3,..., m) are weghs of bas connecon and f s he acvaon funcon. j = Thus whle forecasng wh SANN, he number of npu and oupu neurons should be aken as 2 for monhly and 4 for quarerly me seres. The approprae number of hdden nodes can be deermned by performng suable expermens on he ranng daa. 4.5 Selecon of A Proper Nework Archecure So far we have dscussed abou hree mporan nework archecures, vz. he FNN, TLNN and SANN, whch are exensvely used n forecasng problems. Some oher ypes of neural models are also proposed n leraure, such as he Probablsc Neural Nework (PNN) [20] for classfcaon problem and Generalzed Regresson Neural Nework (GRNN) [20] for regresson problem. Afer specfyng a parcular nework srucure, he nex mos mporan ssue s he

29 deermnaon of he opmal nework parameers. The number of nework parameers s equal o he oal number of connecons beween he neurons and he bas erms [3, ]. A desred nework model should produce reasonably small error no only on whn sample (ranng) daa bu also on ou of sample (es) daa [20]. Due o hs reason mmense care s requred whle choosng he number of npu and hdden neurons. However, s a dffcul ask as here s no heorecal gudance avalable for he selecon of hese parameers and ofen expermens, such as cross-valdaon are conduced for hs purpose [3, 8]. Anoher major problem s ha an nadequae or large number of nework parameers may lead o he overranng of daa [2, ]. Overranng produces spurously good whn-sample f, whch does no generae beer forecass. To penalze he addon of exra parameers some model comparson crera, such as AIC and BIC can be used [, 3]. Nework Prunng [3] and MacKay s Bayesan Regularzaon Algorhm [, 20] are also que popular n hs regard. In summary we can say ha NNs are amazngly smple hough powerful echnques for me seres forecasng. The selecon of approprae nework parameers s crucal, whle usng NN for forecasng purpose. Also a suable ransformaon or rescalng of he ranng daa s ofen necessary o oban bes resuls

30 Chaper-5 Tme Seres Forecasng Usng Suppor Vecor Machnes 5. Concep of Suppor Vecor Machnes Tll now, we have suded abou varous sochasc and neural nework mehods for me seres modelng and forecasng. Despe of her own srenghs and weaknesses, hese mehods are que successful n forecasng applcaons. Recenly, a new sascal learnng heory, vz. he Suppor Vecor Machne (SVM) has been recevng ncreasng aenon for classfcaon and forecasng [8, 24, 30, 3]. SVM was developed by Vapnk and hs co-workers a he AT & T Bell laboraores n 995 [24, 29, 33]. Inally SVMs were desgned o solve paern classfcaon problems, such as opmal characer recognon, face denfcaon and ex classfcaon, ec. Bu soon hey found wde applcaons n oher domans, such as funcon approxmaon, regresson esmaon and me seres predcon problems [24, 3, 34]. Vapnk s SVM echnque s based on he Srucural Rsk Mnmzaon (SRM) prncple [24, 29, 30]. The objecve of SVM s o fnd a decson rule wh good generalzaon ably hrough selecng some parcular subse of ranng daa, called suppor vecors [29, 3, 33]. In hs mehod, an opmal separang hyperplane s consruced, afer nonlnearly mappng he npu space no a hgher dmensonal feaure space. Thus, he qualy and complexy of SVM soluon does no depend drecly on he npu space [8, 9]. Anoher mporan characersc of SVM s ha here he ranng process s equvalen o solvng a lnearly consraned quadrac programmng problem. So, conrary o oher neworks ranng, he SVM soluon s always unque and globally opmal. However a major dsadvanage of SVM s ha when he ranng sze s large, requres an enormous amoun of compuaon whch ncreases he me complexy of he soluon [24]. Now we are gong o presen a bref mahemacal dscusson abou SVM concep. 5.2 Inroducon o Sascal Learnng Theory Vapnk s sascal learnng heory s developed n order o derve a learnng echnque whch wll provde good generalzaon. Accordng o Vapnk [33] here are hree man problems n machne learnng, vz. Classfcaon, Regresson and Densy Esmaon. In all hese cases he man goal s o learn a funcon (or hypohess) from he ranng daa usng a learnng machne and hen nfer general resuls based on hs knowledge. In case of supervsed learnng he ranng daa s composed of pars of npu and oupu varables. The npu vecors n x X R and he oupu pons D R. y The wo ses X and D are respecvely ermed as he npu space and oupu space [29, 33]. = {,} or { 0,} bnary classfcaon problem and D = R for regresson problem D for

31 In case of unsupervsed learnng he ranng daa s composed of only he npu vecors. Here he man goal s o nfer he nheren srucure of he daa hrough densy esmaon and cluserng echnque. The ranng daa s supposed o be generaed from an..d process followng an unknown dsrbuon P( x, y) defned on he se X D. An npu vecor s drawn from X wh he margnal probably P(x) condonal probably P ( y x). and he correspondng oupu pon s observed n D wh he Inpu Space X Oupu Space D P (x) P ( y x) Fg. 5.: Probablsc mappng of npu and oupu pons Afer hese descrpons, he learnng problem can be vsualzed as searchng for he approprae esmaor funcon f : X D whch wll represen he process of oupu generaons from he npu vecors [29, 33]. Ths funcon hen can be used for generalzaon,.e. o produce an oupu value n response o an unseen npu vecor. 5.3 Emprcal Rsk Mnmzaon (ERM) We have seen ha he man am of sascal learnng heory s o search for he mos approprae esmaor funcon f : X D whch maps he pons of he npu space X o he oupu space D. Followng Vapnk and Chervonenks (97) frs a Rsk Funconal s defned on X D o measure he average error occurred among he acual and predced (or classfed) oupus due o usng an esmaor funcon f. Then he mos suable esmaor funcon s chosen o be ha funcon whch mnmzes hs rsk [29, 30, 33]. F = f ( x, w) ha map he pons from he npu space X Le us consder he se of funcons { } n R no he oupu space R D where w denoes he parameers defnng f. Also

32 suppose ha y be he acual oupu pon correspondng o he npu vecor x. Now f L( y, f ( x, w)) measures he error beween he acual value y and he predced value f ( x, w) for usng he predcon funcon f hen he Expeced Rsk s defned as [29, 33]: R( f ) = L( y, f ( x, w)) dp( x, y) (5.) Here P( x, y) s he probably dsrbuon followed by he ranng daa. L ( y, f ( x, w)) s known as he Loss Funcon and can be defned n a number of ways [24, 29, 30]. The mos suable predcon funcon s he one whch mnmzes he expeced rsk R ( f ) and s denoed by f 0. Ths s known as he Targe Funcon. The man ask of learnng problem s now o fnd ou hs arge funcon, whch s he deal esmaor. Unforunaely hs s no possble because he probably dsrbuon P( x, y) of he gven daa s unknown and so he expeced rsk (5.) canno be compued. Ths crcal problem movaes Vapnk o sugges he Emprcal Rsk Mnmzaon (ERM) prncple [33]. The concep of ERM s o esmae he expeced rsk R( f ) by usng he ranng se. Ths approxmaon of R ( f ) s called he emprcal rsk. For a gven ranng se{ x, }, where n x X R, y D R(,2,3,..., N) he emprcal rsk s defned as [29, 33]: = R emp N ( f ) = L( y, f ( x, w)) (5.2) N = The emprcal rsk R emp ( f ) has s own mnmzer n F, whch can be aken as ˆf. The goal of ERM prncple s o approxmae he arge funcon f 0 by ˆf. Ths s possble due o he resul ha R ( f ) nfac converges o R emp ( f ) when he ranng sze N s nfnely large [33]. y 5.4 Vapnk-Chervonenks (VC) Dmenson The VC dmenson h of a class of funcons F s defned as he maxmum number of pons ha can be exacly classfed (.e. shaered) by F [29, 33]. So mahemacally [, 33]: n X h = max{ X, X R,such ha b {,}, f F such ha x X ( N), f(x ) = b }. The VC dmenson s nfac a measure of he nrnsc capacy of a class of funcons F. I s proved by Burges n 998 [] ha he VC dmenson of he se of orened hyperplanes n s ( n +). Thus hree pons labeled n egh dfferen ways can always be classfed by a lnear orened decson boundary n n R 2 R bu four pons canno. Thus VC dmenson of he se of 2 orened sragh lnes n R s hree. For example he XOR problem [29] canno be realzed usng a lnear decson boundary. However a quadrac decson boundary can correcly classfy he pons n hs problem. Ths s shown n he fgures below: