Generalization Dynamics in LMS Trained Linear Networks
|
|
- Beverley Andrews
- 8 years ago
- Views:
Transcription
1 Geeralizatio Dyamics i LMS Traied Liear Networks Yves Chauvi Psychology Departmet Staford Uiversity Staford, CA Abstract For a simple liear case, a mathematical aalysis of the traiig ad geeralizatio (validatio) performace of etworks traied by gradiet descet o a Least Mea Square cost fuctio is provided as a fuctio of the learig parameters ad of the statistics of the traiig data base. The aalysis predicts that geeralizatio error dyamics are very depedet o a priori iitial weights. I particular, the geeralizatio error might sometimes weave withi a computable rage durig exteded traiig. I some cases, the aalysis provides bouds o the optimal umber of traiig cycles for miimal validatio error. For a speech labelig task, predicted weavig effects were qualitatively tested ad observed by computer simulatios i etworks traied by the liear ad o-liear back-propagatio algorithm. 1 INTRODUCTION Recet progress i etwork desig demostrates that o-liear feedforward eural etworks ca perform impressive patter classificatio for a variety of real-world applicatios (e.g., Le Cu et al., 1990; Waibel et al., 1989). Various simulatios ad relatioships betwee the eural etwork ad machie learig theoretical literatures also suggest that too large a umber of free parameters ("weight overfittig") could substatially reduce geeralizatio performace. (e.g., Baum, ). A umber of solutios have recetly bee proposed to decrease or elimiate the overfittig problem i specific situatios. They rage from ad hoc heuristics to i theoretical cosideratios (e.g., Le Cu et al., 1990; Chauvi, 1990a; Weiged et al., Also with Thomso-CSF, Ic., 630 Hase Way, Suite 250, Palo Alto, CA
2 Geeralizatio Dyamics i LMS Traied Liear Networks 891 I Press). For a phoeme labelig applicatio, Chauvi showed that the overfittig pheomeo was actually observed oly whe etworks were overtraied far beyod their "optimal" performace poit (Chauvi, 1990b). Furthermore, geeralizatio performace of etworks seemed to be idepedet of the size of the etwork durig early traiig but the rate of decrease i performace with overtraiig was ideed related the umber of weights. The goal of this paper is to better uderstad traiig ad geeralizatio error dyamics i Least-Mea-Square traied liear etworks. As we will see, gradiet descet traiig o liear etworks ca actually geerate surprisigly rich ad isightful validatio dyamics. Furthermore, i umerous applicatios, eve o-liear etworks ted to fuctio i their liear rage, as if the etworks were makig use of o-liearities oly whe ecessary ('Veiged et al., I Press; Chauvi, 1990a). I Sectio 2, I preset a theoretical illustratio yieldig a better uderstadig of traiig ad validatio error dyamics. I Sectio 3, umerical solutios to obtaied aalytical results make iterestig predictios for validatio dyamics uder overtraiig. These predictios are tested for a phoemic labelig task. The obtaied simulatios suggest that the results of the aalysis obtaied with the simple theoretical framework of Sectio 2 might remai qualitatively valid for o-liear complex architectures. 2 THEORETICAL ILLUSTRATION 2.1 ASSUMPTIONS Let us cosider a liear etwork composed of iput uits ad output uits fully coected by a. weight matrix W. Let us suppose the etwork is traied to reproduce a oiseless output "sigal" from a oisy iput "sigal" (the etwork ca be see as a liear filter). 'Ve write F as the "sigal", N the oise, X the iput, Y the output, ad D the desired output. For the cosidered case, we have X = F+N, Y = W X ad D = F. The statistical properties of the data base are the followig. The sigal is zero-mea with covariace matrix CF. 'Ve write Ai ad ei as the eigevalues ad eigevectors of C F (ei are the so-called pricipal compoets; we will call Ai the "sigal ~ower spectrum"). The oise is assumed to be zero-mea, with covariace matrix CN = v.i where I is the idetity matrix. We assume the oise is ucorrelated with the sigal: CFN = O. We suppose two sets of patters have bee sampled for traiig ad for validatio. We write CF, CN ad CFN the resultig covariace matrices for the traiig set ad CF, CN ~d CF N the corresp_odig matrices for the validatio set. We assume CF ~ Cp ~ CF, CFN ~ CPN ~ CFN = 0, CN = v.i ad CN = v'.i with v' > v. (N umerous of these assumptios are made for the sake of clarity of explaatio: they ca be relaxed without chagig the resultig implicatios.) The problem cosidered is much simpler tha typical realistic applicatios. However, we will see below that (i) a formal aalysis becomes complex very quickly (ii) the validatio dyamics are rich, isightful ad ca be mapped to a umber of results observed i simulatios of realistic applicatios ad (iii) a iterestig umber of predictios ca be obtaied.
3 892 Chauvi 2.2 LEARNING The etwork is traied by gradiet descet o the Least Mea Square (LMS) error: dw = -1JV'wE where 1J is the usual learig rate ad, i the case cosidered, E = E; (Fp - Yp)T(Fp - Yp). We ca write the gradiet as a fuctio of the various covariace matrices: V' we = (I - W)C F + (I - 2W)C F N - W C N. From the geeral assumptios, we get: V'wE ~ CF - WCF - WCN (1) We assume ow that the pricipal compoets ei are also eigevectors of the weight matrix W at iteratio k with correspodig eigevalue Qik: Wk.ei = Qikei. We ca the compute the image of each eigevector ei at iteratio k + 1: Wk+l.ei = 1JAi.ei + Qik[I-1J(Ai + v)).ei (2) Therefore, ei is also a eigevector of Wk+l ad Qi,k+l satisfies the iductio: Qi,k+l = 1JAi + Qik[l - 1J(Ai + v)] (3) Assumig Wo = 0, we ca compute the alpha-dyamics of the weight matrix W: A Qik= A ' [1-(I-1J(Ai+ v ))k] (4),+v < 1/ AM + v, Qi approaches Ai/(A, + Vi), which As k goes to ifiity, provided 1J correspods to the optimal (Wieer) value of the liear filter implemeted by the etwork. We will write the covergece rates ai = I-1JA, -1JV. These rates deped o the sigal "power spectrum", o the oise power ad o the learig rate 1J. If we ow assume WO.ei = QiO.ei with QiO #- 0 (this assumptio ca be made more geeral), we get: where bi = 1 - QiO - QiOV / Ai. Figure 1 represets possible alpha dyamics for arbitrary values of Ai with QiD = Qo #- O. We ca ow compute the learig error dyamics by expadig the LMS error term E at time k. Usig the geeral assumptios o the covariace matrices, we fid: Ek = E Eik = E Ai(1 - Qik)2 + VQ~k (6) Therefore, traiig error is a sum of error compoets, each of them beig a quadratic fuctio of Qi. Figure 2 represets a traiig error compoet Ei as a fuctio of Q. Kowig the alpha-dyamics, we ca write these error compoets as a fuctio of k: A, ( \ b2 2k) E... = V+A a h; Ai + V ' It is easy to see that E is a mootoic decreasig fuctio (geerated by gradiet descet) which coverges to the bottom of the quadratic error surface, yieldig the residual asymptotic error: (5) (7) (8)
4 Geeralizatio Dyamics i LMS Traied Liear Networks , o.~ -~ >.. =.2 ~ , O.O;---~--~I--~ ~~I--~--~I--~--~I--~---,I o N umber of Cycles Figure 1: Alpha dyamics for differet values of >'i with 'T1 =.01 ad aio = ao =j:. O. The solid lies represet the optimal values of ai for the traiig data set. The dashed lies represet correspodig optimal values for the validatio data set. LMS v!, o ~~ A;+V J A.+V aik 1 Figure 2: Traiig ad validatio error dyamics as a fuctio of ai. The dashed curved lies represet the error dyamics for the iitial coditios aiq. Each traiig error compoet follows the gradiet of a quadratic learig curve (bottom). Note the overtraiig pheomeo (top curve) betwee at (optimal for validatio) ad aioo (optimal for traiig).
5 894 Chauvi 2.3 GENERALIZATION Cosiderig the geeral assumptios o the statistics of the data base, we ca compute the validatio error E' (N ote that "validatio error" strictly applies to the validatio data set. "Geeralizatio error" ca qualify the validatio data set or the whole populatio, depedig o cotext.): Ek = ~E:k = ~Ai(l- aik)2 + v'a;k (9) where the alpha-dyamics are imposed by gradiet descet learig o the traiig data set. Agai, the validatio error is a sum of error compoets Ei, quadratic fuctios of ai. However, because the alpha-dyamics are adapted to the traiig sample, they might geerate complex dyamics which will strogly deped o the iital values aio (Figure 1). Cosequetly, the resultig error compoets E: are ot mootoic decreasig fuctios aymore. As see i Figure 2, each of the validatio error compoets might (i) decrease (ii) decrease the icrease (overtraiig) or (iii) icrease as a fuctio of aio. For each of these compoets, i the case of overtraiig, it is possible to compute the value of aik at which traiig should be stopped to get miimal validatio error: L 2L-+L v'-v og >.;+v' og >';-aio(>'.+v') Log(1-7JAi - 7Jv) (10) However, the validatio error dyamics become much more complex whe we cosider sums of these compoets. If we assume aiq = 0, the miimum (or miima) of E' ca be foud to correspod to possible itersectios of hyper-ellipsoids ad power curves. I geeral, it is possible to show that there exists at least oe such miimum. It is also possible to fid simple bouds o the optimal traiig time for miimal validatio error: These bouds are tight whe the oise power is small compared to the sigal "power spectrum". For aio =f. 0, a formal aalysis of the validatio error dyamics becomes itractable. Because some error compoets might icrease while others decrease, it is possible to imagie multiple miima ad maxima for the total validatio error (see simulatios below). Cosiderig each compoet's dyamics, it is oetheless possible to compute bouds withi which E' might vary durig traiig: ~ AW' '2:" Ai(V2 + v' Ai) -:---- < Ek <,. Ai + v' - -,. (Ai + v)2 Because of the "expoetial" ature of traiig (Figure 1), it is possible to imagie that this "weavig" effect might still be observed after a log traiig period, whe the traiig error itself has become stable. Furthermore, whereas the traiig error will qualitatively show the same dyamics, validatio error will very much deped o aio: for sufficietly large iitial weights, validatio dyamics might be very depedet o particular simulatio "rus". (11) (12)
6 Geeralizatio Dyamics i LMS Traied Liear Networks " o Figure 3: Traiig (bottom curves) ad validatio (top curves) error dyamics i a two-dimesioal case for ).1 = 17,).2 = 1.7, v = 2, v' = 10, l: 10 = 0 as l: 20 varies from 0 to 1.6 (bottom-up) i.2 icremets. 3 SIMULATIONS 3.1 CASE STUDY Equatios 7 ad 9 were simulated for a two-dimesioal case ( = 2) with ).1 17,).2 = 1.7, v = 2, v' = 10 ad l: 10 = O. The values of l: 20 determied the relative domiace of the two error compoets durig traiig. Figure 3 represets traiig ad validatio dyamics as a fuctio of k for a rage of values of l: 20. As show aalytically, traiig dyamics are basically uaffected by the iitial coditios of the weight matrix Woo However, a variety of validatio dyamics ca be observed as l: 20 varies from 0 to 1.6. For 1.6 ~ l: 20 ~ 1.4, the validatio error is mootically decreasig ad looks like a typical "gradiet descet" traiig error. For 1.2 ~ l: 20 ~ 1.0, each error compoet i tur imposes a descet rate: the validatio error looks like two "coected descets". For.8 ~ 0'20 ~.6, E~ is mootically decreasig with a slow covergece rate, forcig the validatio error to decrease log after E~ has become stable. This creates a miimum, followed by a maximum, followed by a miimum for E'. Fially, for.4 ~ l: 20 ~ 0, both error compoets have a sigle miimum durig traiig ad geerate a sigle miimum for the total validatio error E'. 3.2 PHONEMIC LABELING Oe of the mai predictios obtaied from the aalytical results ad from the previous case study is that validatio dyamics ca demostrate multiple local miima ad maxima. To my kowledge, this pheomeo has ot bee described i the literature. However, the theory also predicts that the pheomeo will probably appear very late i traiig, well after the traiig error has become stable, which might explai the absece of such observatios. The predictios were tested for a phoemic labelig task with spectrograms as iput patters ad phoemes as output
7 896 Chauvi patters. Various architectures were tested (direct coectios or back-propagatio etworks with liear or o-liear hidde layers). Due to the limited legth of this article, the complete simulatios will be reported elsewhere. I all cases, as predicted, multiple mimia/maxima were observed for the validatio dyamics, provided the etworks were traied way beyod usual traiig times. Furthermore, these geeralizatio dyamics were very depedet o the iitial weights (provided sufficiet variace o the iitial weight distributio). 4 DISCUSSION It is sometimes assumed that optimal learig is obtaied whe validatio error starts to icrease durig the course of traiig. Although for the theoretical study preseted, the first miimum of E' is probably always a global miimum, idepedetly of aw, simulatios of the speech labelig task show it is ot always the case with more complex architectures: late validatio miima ca sometimes (albeit rarely) be deeper tha the first "local" miimum. These observatios ad a lack of theoretical uderstadig of statistical iferece uder limited data set raise the questio of the sigificace of a validatio data set. As a fial commet, we are ot ready iterested i miimal validatio error (E') but i miimal geeralizatio error (E'). Uderstadig the dyamics of the "populatio" error as a fuctio of traiig ad validatio errors ecessitates, at least, a evaluatio of the sample statistics as a fuctio of the umber of traiig ad validatio patters. This is beyod the scope of this paper. Ackowledgemets Thaks to Pierre Baldi ad Julie Holmes for their helpful commets. Refereces Baum, E. B. & Haussler, D. (1989). 'ivhat size et gives valid geeralizatio? Neural Computatio, 1, Chauvi, Y. (1990a). Dyamic behavior of costraied back-propagatio etworks. I D. S. Touretzky (Ed.), Neural Iformatio Processig Systems (Vol. 2) (pp ). Sa Mateo, CA: Morga Kaufma. Chauvi, Y. (1990b). Geeralizatio performace of overtraied back-propagatio etworks. I L. B. Almeida & C. J. 'ivellekes (Eds.), Lecture Notes i Computer Sciece (Vo1. 412) (pp ). Berli: Germay: Spriger-Verlag. Cu, Y. 1., Boser, B., Deker, J. S., Hederso, D., Howard, R. E., Hubbard, 'iv., & Jackel, 1. D. (1990). Hadwritte digit recogitio with a back-propagatio etwork. I D. S. Touretzky (Ed.), Neural Iformatio Processig Systems (Vo1. 2) (pp ). Sa Mateo, CA: Morga Kaufma. 'ivaibel, A., Sawai, H., & Shikao, K. (1989). Modularity ad scalig i large phoemic eural etworks. IEEE Trasactios o Acoustics, Speech ad Sigal Processig, ASSP-37, 'iveiged, A. S., Huberma, B. A., & Rumelhart, D. E. (I Press). Predictig the future: a coectioist approach. Iteratioal Joural of Neural Systems.
Modified Line Search Method for Global Optimization
Modified Lie Search Method for Global Optimizatio Cria Grosa ad Ajith Abraham Ceter of Excellece for Quatifiable Quality of Service Norwegia Uiversity of Sciece ad Techology Trodheim, Norway {cria, ajith}@q2s.tu.o
More informationI. Chi-squared Distributions
1 M 358K Supplemet to Chapter 23: CHI-SQUARED DISTRIBUTIONS, T-DISTRIBUTIONS, AND DEGREES OF FREEDOM To uderstad t-distributios, we first eed to look at aother family of distributios, the chi-squared distributios.
More information1 Correlation and Regression Analysis
1 Correlatio ad Regressio Aalysis I this sectio we will be ivestigatig the relatioship betwee two cotiuous variable, such as height ad weight, the cocetratio of a ijected drug ad heart rate, or the cosumptio
More informationNEW HIGH PERFORMANCE COMPUTATIONAL METHODS FOR MORTGAGES AND ANNUITIES. Yuri Shestopaloff,
NEW HIGH PERFORMNCE COMPUTTIONL METHODS FOR MORTGGES ND NNUITIES Yuri Shestopaloff, Geerally, mortgage ad auity equatios do ot have aalytical solutios for ukow iterest rate, which has to be foud usig umerical
More informationLECTURE 13: Cross-validation
LECTURE 3: Cross-validatio Resampli methods Cross Validatio Bootstrap Bias ad variace estimatio with the Bootstrap Three-way data partitioi Itroductio to Patter Aalysis Ricardo Gutierrez-Osua Texas A&M
More informationHypothesis testing. Null and alternative hypotheses
Hypothesis testig Aother importat use of samplig distributios is to test hypotheses about populatio parameters, e.g. mea, proportio, regressio coefficiets, etc. For example, it is possible to stipulate
More informationResearch Article Sign Data Derivative Recovery
Iteratioal Scholarly Research Network ISRN Applied Mathematics Volume 0, Article ID 63070, 7 pages doi:0.540/0/63070 Research Article Sig Data Derivative Recovery L. M. Housto, G. A. Glass, ad A. D. Dymikov
More information1 Computing the Standard Deviation of Sample Means
Computig the Stadard Deviatio of Sample Meas Quality cotrol charts are based o sample meas ot o idividual values withi a sample. A sample is a group of items, which are cosidered all together for our aalysis.
More informationReview: Classification Outline
Data Miig CS 341, Sprig 2007 Decisio Trees Neural etworks Review: Lecture 6: Classificatio issues, regressio, bayesia classificatio Pretice Hall 2 Data Miig Core Techiques Classificatio Clusterig Associatio
More informationIrreducible polynomials with consecutive zero coefficients
Irreducible polyomials with cosecutive zero coefficiets Theodoulos Garefalakis Departmet of Mathematics, Uiversity of Crete, 71409 Heraklio, Greece Abstract Let q be a prime power. We cosider the problem
More informationChapter 7 Methods of Finding Estimators
Chapter 7 for BST 695: Special Topics i Statistical Theory. Kui Zhag, 011 Chapter 7 Methods of Fidig Estimators Sectio 7.1 Itroductio Defiitio 7.1.1 A poit estimator is ay fuctio W( X) W( X1, X,, X ) of
More informationDepartment of Computer Science, University of Otago
Departmet of Computer Sciece, Uiversity of Otago Techical Report OUCS-2006-09 Permutatios Cotaiig May Patters Authors: M.H. Albert Departmet of Computer Sciece, Uiversity of Otago Micah Colema, Rya Fly
More informationwhere: T = number of years of cash flow in investment's life n = the year in which the cash flow X n i = IRR = the internal rate of return
EVALUATING ALTERNATIVE CAPITAL INVESTMENT PROGRAMS By Ke D. Duft, Extesio Ecoomist I the March 98 issue of this publicatio we reviewed the procedure by which a capital ivestmet project was assessed. The
More informationCase Study. Normal and t Distributions. Density Plot. Normal Distributions
Case Study Normal ad t Distributios Bret Halo ad Bret Larget Departmet of Statistics Uiversity of Wiscosi Madiso October 11 13, 2011 Case Study Body temperature varies withi idividuals over time (it ca
More informationMaximum Likelihood Estimators.
Lecture 2 Maximum Likelihood Estimators. Matlab example. As a motivatio, let us look at oe Matlab example. Let us geerate a radom sample of size 00 from beta distributio Beta(5, 2). We will lear the defiitio
More informationPROCEEDINGS OF THE YEREVAN STATE UNIVERSITY AN ALTERNATIVE MODEL FOR BONUS-MALUS SYSTEM
PROCEEDINGS OF THE YEREVAN STATE UNIVERSITY Physical ad Mathematical Scieces 2015, 1, p. 15 19 M a t h e m a t i c s AN ALTERNATIVE MODEL FOR BONUS-MALUS SYSTEM A. G. GULYAN Chair of Actuarial Mathematics
More informationChapter 6: Variance, the law of large numbers and the Monte-Carlo method
Chapter 6: Variace, the law of large umbers ad the Mote-Carlo method Expected value, variace, ad Chebyshev iequality. If X is a radom variable recall that the expected value of X, E[X] is the average value
More informationIn nite Sequences. Dr. Philippe B. Laval Kennesaw State University. October 9, 2008
I ite Sequeces Dr. Philippe B. Laval Keesaw State Uiversity October 9, 2008 Abstract This had out is a itroductio to i ite sequeces. mai de itios ad presets some elemetary results. It gives the I ite Sequeces
More informationInstitute of Actuaries of India Subject CT1 Financial Mathematics
Istitute of Actuaries of Idia Subject CT1 Fiacial Mathematics For 2014 Examiatios Subject CT1 Fiacial Mathematics Core Techical Aim The aim of the Fiacial Mathematics subject is to provide a groudig i
More informationTHE REGRESSION MODEL IN MATRIX FORM. For simple linear regression, meaning one predictor, the model is. for i = 1, 2, 3,, n
We will cosider the liear regressio model i matrix form. For simple liear regressio, meaig oe predictor, the model is i = + x i + ε i for i =,,,, This model icludes the assumptio that the ε i s are a sample
More informationEkkehart Schlicht: Economic Surplus and Derived Demand
Ekkehart Schlicht: Ecoomic Surplus ad Derived Demad Muich Discussio Paper No. 2006-17 Departmet of Ecoomics Uiversity of Muich Volkswirtschaftliche Fakultät Ludwig-Maximilias-Uiversität Müche Olie at http://epub.ub.ui-mueche.de/940/
More informationHow To Solve The Homewor Problem Beautifully
Egieerig 33 eautiful Homewor et 3 of 7 Kuszmar roblem.5.5 large departmet store sells sport shirts i three sizes small, medium, ad large, three patters plaid, prit, ad stripe, ad two sleeve legths log
More informationChair for Network Architectures and Services Institute of Informatics TU München Prof. Carle. Network Security. Chapter 2 Basics
Chair for Network Architectures ad Services Istitute of Iformatics TU Müche Prof. Carle Network Security Chapter 2 Basics 2.4 Radom Number Geeratio for Cryptographic Protocols Motivatio It is crucial to
More information5: Introduction to Estimation
5: Itroductio to Estimatio Cotets Acroyms ad symbols... 1 Statistical iferece... Estimatig µ with cofidece... 3 Samplig distributio of the mea... 3 Cofidece Iterval for μ whe σ is kow before had... 4 Sample
More information0.7 0.6 0.2 0 0 96 96.5 97 97.5 98 98.5 99 99.5 100 100.5 96.5 97 97.5 98 98.5 99 99.5 100 100.5
Sectio 13 Kolmogorov-Smirov test. Suppose that we have a i.i.d. sample X 1,..., X with some ukow distributio P ad we would like to test the hypothesis that P is equal to a particular distributio P 0, i.e.
More informationCOMPARISON OF THE EFFICIENCY OF S-CONTROL CHART AND EWMA-S 2 CONTROL CHART FOR THE CHANGES IN A PROCESS
COMPARISON OF THE EFFICIENCY OF S-CONTROL CHART AND EWMA-S CONTROL CHART FOR THE CHANGES IN A PROCESS Supraee Lisawadi Departmet of Mathematics ad Statistics, Faculty of Sciece ad Techoology, Thammasat
More informationConfidence Intervals for One Mean
Chapter 420 Cofidece Itervals for Oe Mea Itroductio This routie calculates the sample size ecessary to achieve a specified distace from the mea to the cofidece limit(s) at a stated cofidece level for a
More informationCHAPTER 3 DIGITAL CODING OF SIGNALS
CHAPTER 3 DIGITAL CODING OF SIGNALS Computers are ofte used to automate the recordig of measuremets. The trasducers ad sigal coditioig circuits produce a voltage sigal that is proportioal to a quatity
More informationSystems Design Project: Indoor Location of Wireless Devices
Systems Desig Project: Idoor Locatio of Wireless Devices Prepared By: Bria Murphy Seior Systems Sciece ad Egieerig Washigto Uiversity i St. Louis Phoe: (805) 698-5295 Email: bcm1@cec.wustl.edu Supervised
More informationSoving Recurrence Relations
Sovig Recurrece Relatios Part 1. Homogeeous liear 2d degree relatios with costat coefficiets. Cosider the recurrece relatio ( ) T () + at ( 1) + bt ( 2) = 0 This is called a homogeeous liear 2d degree
More informationCenter, Spread, and Shape in Inference: Claims, Caveats, and Insights
Ceter, Spread, ad Shape i Iferece: Claims, Caveats, ad Isights Dr. Nacy Pfeig (Uiversity of Pittsburgh) AMATYC November 2008 Prelimiary Activities 1. I would like to produce a iterval estimate for the
More informationCONTROL CHART BASED ON A MULTIPLICATIVE-BINOMIAL DISTRIBUTION
www.arpapress.com/volumes/vol8issue2/ijrras_8_2_04.pdf CONTROL CHART BASED ON A MULTIPLICATIVE-BINOMIAL DISTRIBUTION Elsayed A. E. Habib Departmet of Statistics ad Mathematics, Faculty of Commerce, Beha
More informationA probabilistic proof of a binomial identity
A probabilistic proof of a biomial idetity Joatho Peterso Abstract We give a elemetary probabilistic proof of a biomial idetity. The proof is obtaied by computig the probability of a certai evet i two
More informationSection 11.3: The Integral Test
Sectio.3: The Itegral Test Most of the series we have looked at have either diverged or have coverged ad we have bee able to fid what they coverge to. I geeral however, the problem is much more difficult
More informationCME 302: NUMERICAL LINEAR ALGEBRA FALL 2005/06 LECTURE 8
CME 30: NUMERICAL LINEAR ALGEBRA FALL 005/06 LECTURE 8 GENE H GOLUB 1 Positive Defiite Matrices A matrix A is positive defiite if x Ax > 0 for all ozero x A positive defiite matrix has real ad positive
More informationDetermining the sample size
Determiig the sample size Oe of the most commo questios ay statisticia gets asked is How large a sample size do I eed? Researchers are ofte surprised to fid out that the aswer depeds o a umber of factors
More information, a Wishart distribution with n -1 degrees of freedom and scale matrix.
UMEÅ UNIVERSITET Matematisk-statistiska istitutioe Multivariat dataaalys D MSTD79 PA TENTAMEN 004-0-9 LÖSNINGSFÖRSLAG TILL TENTAMEN I MATEMATISK STATISTIK Multivariat dataaalys D, 5 poäg.. Assume that
More informationData-Enhanced Predictive Modeling for Sales Targeting
Data-Ehaced Predictive Modelig for Sales Targetig Saharo Rosset Richard D. Lawrece Abstract We describe ad aalyze the idea of data-ehaced predictive modelig (DEM). The term ehaced here refers to the case
More informationJJMIE Jordan Journal of Mechanical and Industrial Engineering
JJMIE Jorda Joural of Mechaical ad Idustrial Egieerig Volume 5, Number 5, Oct. 2011 ISSN 1995-6665 Pages 439-446 Modelig Stock Market Exchage Prices Usig Artificial Neural Network: A Study of Amma Stock
More information*The most important feature of MRP as compared with ordinary inventory control analysis is its time phasing feature.
Itegrated Productio ad Ivetory Cotrol System MRP ad MRP II Framework of Maufacturig System Ivetory cotrol, productio schedulig, capacity plaig ad fiacial ad busiess decisios i a productio system are iterrelated.
More informationMulti-server Optimal Bandwidth Monitoring for QoS based Multimedia Delivery Anup Basu, Irene Cheng and Yinzhe Yu
Multi-server Optimal Badwidth Moitorig for QoS based Multimedia Delivery Aup Basu, Iree Cheg ad Yizhe Yu Departmet of Computig Sciece U. of Alberta Architecture Applicatio Layer Request receptio -coectio
More informationOutput Analysis (2, Chapters 10 &11 Law)
B. Maddah ENMG 6 Simulatio 05/0/07 Output Aalysis (, Chapters 10 &11 Law) Comparig alterative system cofiguratio Sice the output of a simulatio is radom, the comparig differet systems via simulatio should
More informationComparative Study On Estimate House Price Using Statistical And Neural Network Model
Comparative Study O Estimate House Price Usig Statistical Ad Neural Network Model Azme Bi Khamis, Nur Khalidah Khalilah Biti Kamarudi Abstract: This study was coducted to compare the performace betwee
More informationCHAPTER 3 THE TIME VALUE OF MONEY
CHAPTER 3 THE TIME VALUE OF MONEY OVERVIEW A dollar i the had today is worth more tha a dollar to be received i the future because, if you had it ow, you could ivest that dollar ad ear iterest. Of all
More informationThe analysis of the Cournot oligopoly model considering the subjective motive in the strategy selection
The aalysis of the Courot oligopoly model cosiderig the subjective motive i the strategy selectio Shigehito Furuyama Teruhisa Nakai Departmet of Systems Maagemet Egieerig Faculty of Egieerig Kasai Uiversity
More informationTaking DCOP to the Real World: Efficient Complete Solutions for Distributed Multi-Event Scheduling
Taig DCOP to the Real World: Efficiet Complete Solutios for Distributed Multi-Evet Schedulig Rajiv T. Maheswara, Milid Tambe, Emma Bowrig, Joatha P. Pearce, ad Pradeep araatham Uiversity of Souther Califoria
More informationBuilding Blocks Problem Related to Harmonic Series
TMME, vol3, o, p.76 Buildig Blocks Problem Related to Harmoic Series Yutaka Nishiyama Osaka Uiversity of Ecoomics, Japa Abstract: I this discussio I give a eplaatio of the divergece ad covergece of ifiite
More informationVladimir N. Burkov, Dmitri A. Novikov MODELS AND METHODS OF MULTIPROJECTS MANAGEMENT
Keywords: project maagemet, resource allocatio, etwork plaig Vladimir N Burkov, Dmitri A Novikov MODELS AND METHODS OF MULTIPROJECTS MANAGEMENT The paper deals with the problems of resource allocatio betwee
More informationA Combined Continuous/Binary Genetic Algorithm for Microstrip Antenna Design
A Combied Cotiuous/Biary Geetic Algorithm for Microstrip Atea Desig Rady L. Haupt The Pesylvaia State Uiversity Applied Research Laboratory P. O. Box 30 State College, PA 16804-0030 haupt@ieee.org Abstract:
More informationNon-life insurance mathematics. Nils F. Haavardsson, University of Oslo and DNB Skadeforsikring
No-life isurace mathematics Nils F. Haavardsso, Uiversity of Oslo ad DNB Skadeforsikrig Mai issues so far Why does isurace work? How is risk premium defied ad why is it importat? How ca claim frequecy
More informationProject Deliverables. CS 361, Lecture 28. Outline. Project Deliverables. Administrative. Project Comments
Project Deliverables CS 361, Lecture 28 Jared Saia Uiversity of New Mexico Each Group should tur i oe group project cosistig of: About 6-12 pages of text (ca be loger with appedix) 6-12 figures (please
More informationCoordinating Principal Component Analyzers
Coordiatig Pricipal Compoet Aalyzers J.J. Verbeek ad N. Vlassis ad B. Kröse Iformatics Istitute, Uiversity of Amsterdam Kruislaa 403, 1098 SJ Amsterdam, The Netherlads Abstract. Mixtures of Pricipal Compoet
More informationGroups of diverse problem solvers can outperform groups of high-ability problem solvers
Groups of diverse problem solvers ca outperform groups of high-ability problem solvers Lu Hog ad Scott E. Page Michiga Busiess School ad Complex Systems, Uiversity of Michiga, A Arbor, MI 48109-1234; ad
More informationReal-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations
Real-Time Computig Without Stable States: A New Framework for Neural Computatio Based o Perturbatios Wolfgag aass+, Thomas Natschläger+ & Hery arkram* + Istitute for Theoretical Computer Sciece, Techische
More informationChapter 7: Confidence Interval and Sample Size
Chapter 7: Cofidece Iterval ad Sample Size Learig Objectives Upo successful completio of Chapter 7, you will be able to: Fid the cofidece iterval for the mea, proportio, ad variace. Determie the miimum
More informationIncremental calculation of weighted mean and variance
Icremetal calculatio of weighted mea ad variace Toy Fich faf@cam.ac.uk dot@dotat.at Uiversity of Cambridge Computig Service February 009 Abstract I these otes I eplai how to derive formulae for umerically
More informationEstimating Probability Distributions by Observing Betting Practices
5th Iteratioal Symposium o Imprecise Probability: Theories ad Applicatios, Prague, Czech Republic, 007 Estimatig Probability Distributios by Observig Bettig Practices Dr C Lych Natioal Uiversity of Irelad,
More informationProperties of MLE: consistency, asymptotic normality. Fisher information.
Lecture 3 Properties of MLE: cosistecy, asymptotic ormality. Fisher iformatio. I this sectio we will try to uderstad why MLEs are good. Let us recall two facts from probability that we be used ofte throughout
More informationChapter 5 Unit 1. IET 350 Engineering Economics. Learning Objectives Chapter 5. Learning Objectives Unit 1. Annual Amount and Gradient Functions
Chapter 5 Uit Aual Amout ad Gradiet Fuctios IET 350 Egieerig Ecoomics Learig Objectives Chapter 5 Upo completio of this chapter you should uderstad: Calculatig future values from aual amouts. Calculatig
More informationUniversity of California, Los Angeles Department of Statistics. Distributions related to the normal distribution
Uiversity of Califoria, Los Ageles Departmet of Statistics Statistics 100B Istructor: Nicolas Christou Three importat distributios: Distributios related to the ormal distributio Chi-square (χ ) distributio.
More informationHigh-dimensional support union recovery in multivariate regression
High-dimesioal support uio recovery i multivariate regressio Guillaume Oboziski Departmet of Statistics UC Berkeley gobo@stat.berkeley.edu Marti J. Waiwright Departmet of Statistics Dept. of Electrical
More informationCS103A Handout 23 Winter 2002 February 22, 2002 Solving Recurrence Relations
CS3A Hadout 3 Witer 00 February, 00 Solvig Recurrece Relatios Itroductio A wide variety of recurrece problems occur i models. Some of these recurrece relatios ca be solved usig iteratio or some other ad
More informationBasic Elements of Arithmetic Sequences and Series
MA40S PRE-CALCULUS UNIT G GEOMETRIC SEQUENCES CLASS NOTES (COMPLETED NO NEED TO COPY NOTES FROM OVERHEAD) Basic Elemets of Arithmetic Sequeces ad Series Objective: To establish basic elemets of arithmetic
More informationBiology 171L Environment and Ecology Lab Lab 2: Descriptive Statistics, Presenting Data and Graphing Relationships
Biology 171L Eviromet ad Ecology Lab Lab : Descriptive Statistics, Presetig Data ad Graphig Relatioships Itroductio Log lists of data are ofte ot very useful for idetifyig geeral treds i the data or the
More informationA Fuzzy Model of Software Project Effort Estimation
TJFS: Turkish Joural of Fuzzy Systems (eissn: 309 90) A Official Joural of Turkish Fuzzy Systems Associatio Vol.4, No.2, pp. 68-76, 203 A Fuzzy Model of Software Project Effort Estimatio Oumout Chouseioglou
More informationAP Calculus AB 2006 Scoring Guidelines Form B
AP Calculus AB 6 Scorig Guidelies Form B The College Board: Coectig Studets to College Success The College Board is a ot-for-profit membership associatio whose missio is to coect studets to college success
More informationAutomatic Tuning for FOREX Trading System Using Fuzzy Time Series
utomatic Tuig for FOREX Tradig System Usig Fuzzy Time Series Kraimo Maeesilp ad Pitihate Soorasa bstract Efficiecy of the automatic currecy tradig system is time depedet due to usig fixed parameters which
More informationOverview on S-Box Design Principles
Overview o S-Box Desig Priciples Debdeep Mukhopadhyay Assistat Professor Departmet of Computer Sciece ad Egieerig Idia Istitute of Techology Kharagpur INDIA -721302 What is a S-Box? S-Boxes are Boolea
More informationStudy on the application of the software phase-locked loop in tracking and filtering of pulse signal
Advaced Sciece ad Techology Letters, pp.31-35 http://dx.doi.org/10.14257/astl.2014.78.06 Study o the applicatio of the software phase-locked loop i trackig ad filterig of pulse sigal Sog Wei Xia 1 (College
More informationSolutions to Selected Problems In: Pattern Classification by Duda, Hart, Stork
Solutios to Selected Problems I: Patter Classificatio by Duda, Hart, Stork Joh L. Weatherwax February 4, 008 Problem Solutios Chapter Bayesia Decisio Theory Problem radomized rules Part a: Let Rx be the
More informationAsymptotic Growth of Functions
CMPS Itroductio to Aalysis of Algorithms Fall 3 Asymptotic Growth of Fuctios We itroduce several types of asymptotic otatio which are used to compare the performace ad efficiecy of algorithms As we ll
More informationFOUNDATIONS OF MATHEMATICS AND PRE-CALCULUS GRADE 10
FOUNDATIONS OF MATHEMATICS AND PRE-CALCULUS GRADE 10 [C] Commuicatio Measuremet A1. Solve problems that ivolve liear measuremet, usig: SI ad imperial uits of measure estimatio strategies measuremet strategies.
More informationResearch Method (I) --Knowledge on Sampling (Simple Random Sampling)
Research Method (I) --Kowledge o Samplig (Simple Radom Samplig) 1. Itroductio to samplig 1.1 Defiitio of samplig Samplig ca be defied as selectig part of the elemets i a populatio. It results i the fact
More informationAnalyzing Longitudinal Data from Complex Surveys Using SUDAAN
Aalyzig Logitudial Data from Complex Surveys Usig SUDAAN Darryl Creel Statistics ad Epidemiology, RTI Iteratioal, 312 Trotter Farm Drive, Rockville, MD, 20850 Abstract SUDAAN: Software for the Statistical
More informationNATIONAL SENIOR CERTIFICATE GRADE 12
NATIONAL SENIOR CERTIFICATE GRADE MATHEMATICS P EXEMPLAR 04 MARKS: 50 TIME: 3 hours This questio paper cosists of 8 pages ad iformatio sheet. Please tur over Mathematics/P DBE/04 NSC Grade Eemplar INSTRUCTIONS
More informationSubject CT5 Contingencies Core Technical Syllabus
Subject CT5 Cotigecies Core Techical Syllabus for the 2015 exams 1 Jue 2014 Aim The aim of the Cotigecies subject is to provide a groudig i the mathematical techiques which ca be used to model ad value
More informationTrading rule extraction in stock market using the rough set approach
Tradig rule extractio i stock market usig the rough set approach Kyoug-jae Kim *, Ji-youg Huh * ad Igoo Ha Abstract I this paper, we propose the rough set approach to extract tradig rules able to discrimiate
More informationUC Berkeley Department of Electrical Engineering and Computer Science. EE 126: Probablity and Random Processes. Solutions 9 Spring 2006
Exam format UC Bereley Departmet of Electrical Egieerig ad Computer Sciece EE 6: Probablity ad Radom Processes Solutios 9 Sprig 006 The secod midterm will be held o Wedesday May 7; CHECK the fial exam
More informationAP Calculus BC 2003 Scoring Guidelines Form B
AP Calculus BC Scorig Guidelies Form B The materials icluded i these files are iteded for use by AP teachers for course ad exam preparatio; permissio for ay other use must be sought from the Advaced Placemet
More informationPSYCHOLOGICAL STATISTICS
UNIVERSITY OF CALICUT SCHOOL OF DISTANCE EDUCATION B Sc. Cousellig Psychology (0 Adm.) IV SEMESTER COMPLEMENTARY COURSE PSYCHOLOGICAL STATISTICS QUESTION BANK. Iferetial statistics is the brach of statistics
More informationModeling of Ship Propulsion Performance
odelig of Ship Propulsio Performace Bejami Pjedsted Pederse (FORCE Techology, Techical Uiversity of Demark) Ja Larse (Departmet of Iformatics ad athematical odelig, Techical Uiversity of Demark) Full scale
More informationHIGH-DIMENSIONAL REGRESSION WITH NOISY AND MISSING DATA: PROVABLE GUARANTEES WITH NONCONVEXITY
The Aals of Statistics 2012, Vol. 40, No. 3, 1637 1664 DOI: 10.1214/12-AOS1018 Istitute of Mathematical Statistics, 2012 HIGH-DIMENSIONAL REGRESSION WITH NOISY AND MISSING DATA: PROVABLE GUARANTEES WITH
More informationLesson 17 Pearson s Correlation Coefficient
Outlie Measures of Relatioships Pearso s Correlatio Coefficiet (r) -types of data -scatter plots -measure of directio -measure of stregth Computatio -covariatio of X ad Y -uique variatio i X ad Y -measurig
More informationMathematical goals. Starting points. Materials required. Time needed
Level A1 of challege: C A1 Mathematical goals Startig poits Materials required Time eeded Iterpretig algebraic expressios To help learers to: traslate betwee words, symbols, tables, ad area represetatios
More informationThe Stable Marriage Problem
The Stable Marriage Problem William Hut Lae Departmet of Computer Sciece ad Electrical Egieerig, West Virgiia Uiversity, Morgatow, WV William.Hut@mail.wvu.edu 1 Itroductio Imagie you are a matchmaker,
More informationEvaluation of Different Fitness Functions for the Evolutionary Testing of an Autonomous Parking System
Evaluatio of Differet Fitess Fuctios for the Evolutioary Testig of a Autoomous Parkig System Joachim Wegeer 1, Oliver Bühler 2 1 DaimlerChrysler AG, Research ad Techology, Alt-Moabit 96 a, D-1559 Berli,
More informationData Analysis and Statistical Behaviors of Stock Market Fluctuations
44 JOURNAL OF COMPUTERS, VOL. 3, NO. 0, OCTOBER 2008 Data Aalysis ad Statistical Behaviors of Stock Market Fluctuatios Ju Wag Departmet of Mathematics, Beijig Jiaotog Uiversity, Beijig 00044, Chia Email:
More informationSequences and Series
CHAPTER 9 Sequeces ad Series 9.. Covergece: Defiitio ad Examples Sequeces The purpose of this chapter is to itroduce a particular way of geeratig algorithms for fidig the values of fuctios defied by their
More informationTIGHT BOUNDS ON EXPECTED ORDER STATISTICS
Probability i the Egieerig ad Iformatioal Scieces, 20, 2006, 667 686+ Prited i the U+S+A+ TIGHT BOUNDS ON EXPECTED ORDER STATISTICS DIMITRIS BERTSIMAS Sloa School of Maagemet ad Operatios Research Ceter
More informationA Model Based Mixture Supervised Classification Approach in Hyperspectral Data Analysis
A Model Based Mixture Supervised Classificatio Approach i Hyperspectral Data Aalysis M. Murat Dudar ad David Ladgrebe, Life Fellow, IEEE School of Electrical ad Computer Egieerig Purdue Uiversity Copyright
More informationMeasures of Spread and Boxplots Discrete Math, Section 9.4
Measures of Spread ad Boxplots Discrete Math, Sectio 9.4 We start with a example: Example 1: Comparig Mea ad Media Compute the mea ad media of each data set: S 1 = {4, 6, 8, 10, 1, 14, 16} S = {4, 7, 9,
More informationAnnuities Under Random Rates of Interest II By Abraham Zaks. Technion I.I.T. Haifa ISRAEL and Haifa University Haifa ISRAEL.
Auities Uder Radom Rates of Iterest II By Abraham Zas Techio I.I.T. Haifa ISRAEL ad Haifa Uiversity Haifa ISRAEL Departmet of Mathematics, Techio - Israel Istitute of Techology, 3000, Haifa, Israel I memory
More informationRanking Irregularities When Evaluating Alternatives by Using Some ELECTRE Methods
Please use the followig referece regardig this paper: Wag, X., ad E. Triataphyllou, Rakig Irregularities Whe Evaluatig Alteratives by Usig Some ELECTRE Methods, Omega, Vol. 36, No. 1, pp. 45-63, February
More informationCooley-Tukey. Tukey FFT Algorithms. FFT Algorithms. Cooley
Cooley Cooley-Tuey Tuey FFT Algorithms FFT Algorithms Cosider a legth- sequece x[ with a -poit DFT X[ where Represet the idices ad as +, +, Cooley Cooley-Tuey Tuey FFT Algorithms FFT Algorithms Usig these
More informationChapter 5 O A Cojecture Of Erdíos Proceedigs NCUR VIII è1994è, Vol II, pp 794í798 Jeærey F Gold Departmet of Mathematics, Departmet of Physics Uiversity of Utah Do H Tucker Departmet of Mathematics Uiversity
More information(VCP-310) 1-800-418-6789
Maual VMware Lesso 1: Uderstadig the VMware Product Lie I this lesso, you will first lear what virtualizatio is. Next, you ll explore the products offered by VMware that provide virtualizatio services.
More informationHypergeometric Distributions
7.4 Hypergeometric Distributios Whe choosig the startig lie-up for a game, a coach obviously has to choose a differet player for each positio. Similarly, whe a uio elects delegates for a covetio or you
More informationWHEN IS THE (CO)SINE OF A RATIONAL ANGLE EQUAL TO A RATIONAL NUMBER?
WHEN IS THE (CO)SINE OF A RATIONAL ANGLE EQUAL TO A RATIONAL NUMBER? JÖRG JAHNEL 1. My Motivatio Some Sort of a Itroductio Last term I tought Topological Groups at the Göttige Georg August Uiversity. This
More informationApplication and research of fuzzy clustering analysis algorithm under micro-lecture English teaching mode
SHS Web of Cofereces 25, shscof/20162501018 Applicatio ad research of fuzzy clusterig aalysis algorithm uder micro-lecture Eglish teachig mode Yig Shi, Wei Dog, Chuyi Lou & Ya Dig Qihuagdao Istitute of
More informationConvention Paper 6764
Audio Egieerig Society Covetio Paper 6764 Preseted at the 10th Covetio 006 May 0 3 Paris, Frace This covetio paper has bee reproduced from the author's advace mauscript, without editig, correctios, or
More information