Methods and Data Analysis

Fudametal Numercal Methods ad Data Aalyss by George W. Colls, II George W. Colls, II

Table of Cotets Lst of Fgures...v Lst of Tables... Preface... Notes to the Iteret Edto...v. Itroducto ad Fudametal Cocepts.... Basc Propertes of Sets ad Groups.... Scalars, Vectors, ad Matrces... 5. Coordate Systems ad Coordate Trasformatos... 8.4 Tesors ad Trasformatos....5 Operators... 8 Chapter Eercses... Chapter Refereces ad Addtoal Readg.... The Numercal Methods for Lear Equatos ad Matrces... 5. Errors ad Ther Propagato... 6. Drect Methods for the Soluto of Lear Algebrac Equatos... 8 a. Soluto by Cramer's Rule... 8 b. Soluto by Gaussa Elmato... c. Soluto by Gauss Jorda Elmato... d. Soluto by Matr Factorzato: The Crout Method... 4 e. The Soluto of Tr-dagoal Systems of Lear Equatos... 8. Soluto of Lear Equatos by Iteratve Methods... 9 a. Soluto by The Gauss ad Gauss-Sedel Iterato Methods... 9 b. The Method of Hotellg ad Bodewg... 4 c. Relaato Methods for the Soluto of Lear Equatos... 44 d. Covergece ad Fed-pot Iterato Theory... 46.4 The Smlarty Trasformatos ad the Egevalues ad Vectors of a Matr... 48

Chapter Eercses... 5 Chapter Refereces ad Supplemetal Readg... 54. Polyomal Appromato, Iterpolato, ad Orthogoal Polyomals... 55. Polyomals ad Ther Roots... 56 a. Some Costrats o the Roots of Polyomals... 57 b. Sythetc Dvso... 58 c. The Graffe Root-Squarg Process... 6 d. Iteratve Methods... 6. Curve Fttg ad Iterpolato... 64 a. Lagrage Iterpolato... 65 b. Hermte Iterpolato... 7 c. Sples... 75 d. Etrapolato ad Iterpolato Crtera... 79. Orthogoal Polyomals... 85 a. The Legedre Polyomals... 87 b. The Laguerre Polyomals... 88 c. The Hermte Polyomals... 89 d. Addtoal Orthogoal Polyomals... 9 e. The Orthogoalty of the Trgoometrc Fuctos... 9 Chapter Eercses...9 Chapter Refereces ad Supplemetal Readg...95 4. Numercal Evaluato of Dervatves ad Itegrals...97 4. Numercal Dfferetato...98 a. Classcal Dfferece Formulae...98 b. Rchardso Etrapolato for Dervatves... 4. Numercal Evaluato of Itegrals: Quadrature... a. The Trapezod Rule... b. Smpso's Rule... c. Quadrature Schemes for Arbtrarly Spaced Fuctos...5 d. Gaussa Quadrature Schemes...7 e. Romberg Quadrature ad Rchardso Etrapolato... f. Multple Itegrals...

4. Mote Carlo Itegrato Schemes ad Other Trcks...5 a. Mote Carlo Evaluato of Itegrals...5 b. The Geeral Applcato of Quadrature Formulae to Itegrals...7 Chapter 4 Eercses...9 Chapter 4 Refereces ad Supplemetal Readg... 5. Numercal Soluto of Dfferetal ad Itegral Equatos... 5. The Numercal Itegrato of Dfferetal Equatos... a. Oe Step Methods of the Numercal Soluto of Dfferetal Equatos... b. Error Estmate ad Step Sze Cotrol... c. Mult-Step ad Predctor-Corrector Methods...4 d. Systems of Dfferetal Equatos ad Boudary Value Problems...8 e. Partal Dfferetal Equatos...46 5. The Numercal Soluto of Itegral Equatos...47 a. Types of Lear Itegral Equatos...48 b. The Numercal Soluto of Fredholm Equatos...48 c. The Numercal Soluto of Volterra Equatos...5 d. The Ifluece of the Kerel o the Soluto...54 Chapter 5 Eercses...56 Chapter 5 Refereces ad Supplemetal Readg...58 6. Least Squares, Fourer Aalyss, ad Related Appromato Norms...59 6. Legedre's Prcple of Least Squares...6 a. The Normal Equatos of Least Squares...6 b. Lear Least Squares...6 c. The Legedre Appromato...64 6. Least Squares, Fourer Seres, ad Fourer Trasforms...65 a. Least Squares, the Legedre Appromato, ad Fourer Seres...65 b. The Fourer Itegral...66 c. The Fourer Trasform...67 d. The Fast Fourer Trasform Algorthm...69

6. Error Aalyss for Lear Least-Squares...76 a. Errors of the Least Square Coeffcets...76 b. The Relato of the Weghted Mea Square Observatoal Error to the Weghted Mea Square Resdual...78 c. Determg the Weghted Mea Square Resdual...79 d. The Effects of Errors the Idepedet Varable...8 6.4 No-lear Least Squares...8 a. The Method of Steepest Descet...8 b. Lear appromato of f(a,)...84 c. Errors of the Least Squares Coeffcets...86 6.5 Other Appromato Norms...87 a. The Chebyschev Norm ad Polyomal Appromato...88 b. The Chebyschev Norm, Lear Programmg, ad the Smple Method...89 c. The Chebyschev Norm ad Least Squares...9 Chapter 6 Eercses...9 Chapter 6 Refereces ad Supplemetary Readg...94 7. Probablty Theory ad Statstcs...97 7. Basc Aspects of Probablty Theory... a. The Probablty of Combatos of Evets... b. Probabltes ad Radom Varables... c. Dstrbutos of Radom Varables... 7. Commo Dstrbuto Fuctos...4 a. Permutatos ad Combatos...4 b. The Bomal Probablty Dstrbuto...5 c. The Posso Dstrbuto...6 d. The Normal Curve...7 e. Some Dstrbuto Fuctos of the Physcal World... 7. Momets of Dstrbuto Fuctos... 7.4 The Foudatos of Statstcal Aalyss...7 a. Momets of the Bomal Dstrbuto...8 b. Multple Varables, Varace, ad Covarace...9 c. Mamum Lkelhood... v

Chapter 7 Eercses... Chapter 7 Refereces ad Supplemetal Readg...4 8. Samplg Dstrbutos of Momets, Statstcal Tests, ad Procedures...5 8. The t, χ, ad F Statstcal Dstrbuto Fuctos...6 a. The t-desty Dstrbuto Fucto...6 b. The χ -Desty Dstrbuto Fucto...7 c. The F-Desty Dstrbuto Fucto...9 8. The Level of Sgfcace ad Statstcal Tests... a. The "Studets" t-test... b. The χ -test... c. The F-test...4 d. Kolmogorov-Smrov Tests...5 8. Lear Regresso, ad Correlato Aalyss...7 a. The Separato of Varaces ad the Two-Varable Correlato Coeffcet...8 b. The Meag ad Sgfcace of the Correlato Coeffcet...4 c. Correlatos of May Varables ad Lear Regresso...4 d Aalyss of Varace...4 8.4 The Desg of Epermets...46 a. The Termology of Epermet Desg...49 b. Blocked Desgs...5 c. Factoral Desgs...5 Chapter 8 Eercses...55 Chapter 8 Refereces ad Supplemetal Readg...57 Ide...57 v

Lst of Fgures Fgure. shows two coordate frames related by the trasformato agles φ. Four coordates are ecessary f the frames are ot orthogoal... Fgure. shows two eghborg pots P ad Q two adacet coordate systems X ad X' The dfferetal dstace betwee the two s d. The vectoral dstace to the two pots s X (P) or X '(P) ad X (Q) or X '(Q) respectvely... 5 Fgure. schematcally shows the dvergece of a vector feld. I the rego where the arrows of the vector feld coverge, the dvergece s postve, mplyg a crease the source of the vector feld. The opposte s true for the rego where the feld vectors dverge.... 9 Fgure.4 schematcally shows the curl of a vector feld. The drecto of the curl s determed by the "rght had rule" whle the magtude depeds o the rate of chage of the - ad y-compoets of the vector feld wth respect to y ad.... 9 Fgure.5 schematcally shows the gradet of the scalar dot-desty the form of a umber of vectors at radomly chose pots the scalar feld. The drecto of the gradet pots the drecto of mamum crease of the dot-desty, whle the magtude of the vector dcates the rate of chage of that desty..... Fgure. depcts a typcal polyomal wth real roots. Costruct the taget to the curve at the pot k ad eted ths taget to the -as. The crossg pot k+ represets a mproved value for the root the Newto-Raphso algorthm. The pot k- ca be used to costruct a secat provdg a secod method for fdg a mproved value of.... 6 Fgure. shows the behavor of the data from Table.. The results of varous forms of terpolato are show. The appromatg polyomals for the lear ad parabolc Lagraga terpolato are specfcally dsplayed. The specfc results for cubc Lagraga terpolato, weghted Lagraga terpolato ad terpolato by ratoal frst degree polyomals are also dcated.... 69 Fgure 4. shows a fucto whose tegral from a to b s beg evaluated by the trapezod rule. I each terval the fucto s appromated by a straght le... Fgure 4. shows the varato of a partcularly complcated tegrad. Clearly t s ot a polyomal ad so could ot be evaluated easly usg stadard quadrature formulae. However, we may use Mote Carlo methods to determe the rato area uder the curve compared to the area of the rectagle....7 v

Fgure 5. show the soluto space for the dfferetal equato y' g(,y). Sce the tal value s dfferet for dfferet solutos, the space surroudg the soluto of choce ca be vewed as beg full of alterate solutos. The two dmesoal Taylor epaso of the Ruge-Kutta method eplores ths soluto space to obta a hgher order value for the specfc soluto ust oe step...7 Fgure 5. shows the stablty of a smple predctor scheme that systematcally uderestmates the soluto leadg to a cumulatve buld up of trucato error...5 Fgure 6. compares the dscrete Fourer trasform of the fucto e - wth the cotuous trasform for the full fte terval. The oscllatory ature of the dscrete trasform largely results from the small umber of pots used to represet the fucto ad the trucato of the fucto at t ±. The oly pots the dscrete trasform that are eve defed are deoted by...7 Fgure 6. shows the parameter space defed by the φ ()'s. Each f(a, ) ca be represeted as a lear combato of the φ ( ) where the a are the coeffcets of the bass fuctos. Sce the observed varables Y caot be epressed terms of the φ ( ), they le out of the space....8 Fgure 6. shows the χ hypersurface defed o the a space. The o-lear least square seeks the mmum regos of that hypersurface. The gradet method moves the terato the drecto of steepest decet based o local values of the dervatve, whle surface fttg tres to locally appromate the fucto some smple way ad determes the local aalytc mmum as the et guess for the soluto....84 Fgure 6.4 shows the Chebyschev ft to a fte set of data pots. I pael a the ft s wth a costat a whle pael b the ft s wth a straght le of the form f() a + a. I both cases, the adustmet of the parameters of the fucto ca oly produce + mamum errors for the (+) free parameters....88 Fgure 6.5 shows the parameter space for fttg three pots wth a straght le uder the Chebyschev orm. The equatos of codto deote half-plaes whch satsfy the costrat for oe partcular pot...89 Fgure 7. shows a sample space gvg rse to evets E ad F. I the case of the de, E s the probablty of the result beg less tha three ad F s the probablty of the result beg eve. The tersecto of crcle E wth crcle F represets the probablty of E ad F [.e. P(EF)]. The uo of crcles E ad F represets the probablty of E or F. If we were to smply sum the area of crcle E ad that of F we would double cout the tersecto.... v

Fgure 7. shows the ormal curve appromato to the bomal probablty dstrbuto fucto. We have chose the co tosses so that p.5. Here µ ad σ ca be see as the most lkely value of the radom varable ad the 'wdth' of the curve respectvely. The tal ed of the curve represets the rego appromated by the Posso dstrbuto...9 Fgure 7. shows the mea of a fucto f() as <>. Note ths s ot the same as the most lkely value of as was the case fgure 7.. However, some real sese σ s stll a measure of the wdth of the fucto. The skewess s a measure of the asymmetry of f() whle the kurtoss represets the degree to whch the f() s 'flatteed' wth respect to a ormal curve. We have also marked the locato of the values for the upper ad lower quartles, meda ad mode...4 Fgure. shows a comparso betwee the ormal curve ad the t-dstrbuto fucto for N 8. The symmetrc ature of the t-dstrbuto meas that the mea, meda, mode, ad skewess wll all be zero whle the varace ad kurtoss wll be slghtly larger tha ther ormal couterparts. As N, the t-dstrbuto approaches the ormal curve wth ut varace....7 Fgure 8. compares the χ -dstrbuto wth the ormal curve. For N the curve s qute skewed ear the org wth the mea occurrg past the mode (χ 8). The Normal curve has µ 8 ad σ. For large N, the mode of the χ -dstrbuto approaches half the varace ad the dstrbuto fucto approaches a ormal curve wth the mea equal the mode...8 Fgure 8. shows the probablty desty dstrbuto fucto for the F-statstc wth values of N ad N 5 respectvely. Also plotted are the lmtg dstrbuto fuctos f(χ /N ) ad f(t ). The frst of these s obtaed from f(f) the lmt of N. The secod arses whe N. Oe ca see the tal of the f(t ) dstrbuto approachg that of f(f) as the value of the depedet varable creases. Fally, the ormal curve whch all dstrbutos approach for large values of N s show wth a mea equal to F ad a varace equal to the varace for f(f).... Fgure 8.4 shows a hstogram of the sampled pots ad the cumulatve probablty of obtag those pots. The Kolmogorov-Smrov tests compare that probablty wth aother kow cumulatve probablty ad ascerta the odds that the dffereces occurred by chace...7 Fgure 8.5 shows the regresso les for the two cases where the varable X s regarded as the depedet varable (pael a) ad the varable X s regarded as the depedet varable (pael b)....4 v

Lst of Tables Table. Covergece of Gauss ad Gauss-Sedel Iterato Schemes... 4 Table. Sample Iteratve Soluto for the Relaato Method... 46 Table. Sample Data ad Results for Lagraga Iterpolato Formulae... 67 Table. Parameters for the Polyomals Geerated by Nevlle's Algorthm... 7 Table. A Comparso of Dfferet Types of Iterpolato Formulae... 79 Table.4 Parameters for Quotet Polyomal Iterpolato... 8 Table.5 The Frst Fve Members of the Commo Orthogoal Polyomals... 9 Table.6 Classcal Orthogoal Polyomals of the Fte Iterval... 9 Table 4. A Typcal Fte Dfferece Table for f()...99 Table 4. Types of Polyomals for Gaussa Quadrature... Table 4. Table 4.4 Sample Results for Romberg Quadrature... Test Results for Varous Quadrature Formulae... Table 5. Results for Pcard's Method...5 Table 5. Table 5. Table 5.4 Sample Ruge-Kutta Solutos... Solutos of a Sample Boudary Value Problem for Varous Orders of Appromato...45 Solutos of a Sample Boudary Value Problem Treated as a Ital Value Problem...45 Table 5.5 Sample Solutos for a Type Volterra Equato...5 Table 6. Summary Results for a Sample Dscrete Fourer Trasform...7 Table 6. Calculatos for a Sample Fast Fourer Trasform...75 Table 7. Grade Dstrbuto for Sample Test Results...5

Table 7. Eamato Statstcs for the Sample Test...5 Table 8. Sample Beach Statstcs for Correlato Eample...4 Table 8. Factoral Combatos for Two-level Epermets wth -4...5

Preface The orgs of ths book ca be foud years ago whe I was a doctoral caddate workg o my thess ad fdg that I eeded umercal tools that I should have bee taught years before. I the terveg decades, lttle has chaged ecept for the worse. All felds of scece have udergoe a formato eploso whle the computer revoluto has steadly ad rrevocablty bee chagg our lves. Although the crystal ball of the future s at best "see through a glass darkly", most would declare that the advet of the dgtal electroc computer wll chage cvlzato to a etet ot see sce the comg of the steam ege. Computers wth the power that could be offered oly by large sttutos a decade ago ow st o the desks of dvduals. Methods of aalyss that were oly dreamed of three decades ago are ow used by studets to do homework eercses. Etrely ew methods of aalyss have appeared that take advatage of computers to perform logcal ad arthmetc operatos at great speed. Perhaps studets of the future may regard the multplcato of two two-dgt umbers wthout the ad of a calculator the same ve that we regard the formal etracto of a square root. The whole approach to scetfc aalyss may chage wth the advet of maches that commucate orally. However, I hope the day ever arrves whe the vestgator o loger uderstads the ature of the aalyss doe by the mache. Ufortuately structo the uses ad applcablty of ew methods of aalyss rarely appears the currculum. Ths s o surprse as such courses ay dscple always are the last to be developed. I rapdly chagg dscples ths meas that actve studets must fed for themselves. Wth umercal aalyss ths has meat that may smply take the tools developed by others ad apply them to problems wth lttle kowledge as to the applcablty or accuracy of the methods. Numercal algorthms appear as eatly packaged computer programs that are regarded by the user as "black boes" to whch they feed ther data ad from whch come the publshable results. The complety of may of the problems dealt wth ths maer makes determg the valdty of the results early mpossble. Ths book s a attempt to correct some of these problems. Some may regard ths effort as a survey ad to that I would plead gulty. But I do ot regard the word survey as peoratve for to survey, codese, ad collate, the kowledge of ma s oe of the resposbltes of the scholar. There s a mplcato heret ths resposblty that the formato be made more comprehesble so that t may more readly be assmlated. The etet to whch I have succeeded ths goal I wll leave to the reader. The dscusso of so may topcs may be regarded by some to be a mpossble task. However, the subects I have selected have all bee requred of me durg my professoal career ad I suspect most research scetsts would make a smlar clam.

Ufortuately few of these subects were ever covered eve the troductory level of treatmet gve here durg my formal educato ad certaly they were ever placed wth a coheret cotet of umercal aalyss. The basc format of the frst chapter s a very wde ragg vew of some cocepts of mathematcs based loosely o aomatc set theory ad lear algebra. The tet here s ot so much to provde the specfc mathematcal foudato for what follows, whch s doe as eeded throughout the tet, but rather to establsh, what I call for lack of a better term, "mathematcal sophstcato". There s a geeral acquatace wth mathematcs that a studet should have before embarkg o the study of umercal methods. The studet should realze that there s a subect called mathematcs whch s artfcally broke to sub-dscples such a lear algebra, arthmetc, calculus, topology, set theory, etc. All of these dscples are related ad the sooer the studet realzes that ad becomes aware of the relatos, the sooer mathematcs wll become a coveet ad useful laguage of scetfc epresso. The ablty to use mathematcs such a fasho s largely what I mea by "mathematcal sophstcato". However, ths book s prmarly teded for scetsts ad egeers so whle there s a certa famlarty wth mathematcs that s assumed, the rgor that oe epects wth a formal mathematcal presetato s lackg. Very lttle s proved the tradtoal mathematcal sese of the word. Ideed, dervatos are resorted to maly to emphasze the assumptos that uderle the results. However, whe dervatos are called for, I wll ofte wrte several forms of the same epresso o the same le. Ths s doe smply to gude the reader the drecto of a mathematcal developmet. I wll ofte gve "rules of thumb" for whch there s o formal proof. However, eperece has show that these "rules of thumb" almost always apply. Ths s doe the sprt of provdg the researcher wth practcal ways to evaluate the valdty of hs or her results. The basc premse of ths book s that t ca serve as the bass for a wde rage of courses that dscuss umercal methods used scece. It s meat to support a seres of lectures, ot replace them. To reflect ths, the subect matter s wde ragg ad perhaps too broad for a sgle course. It s epected that the structor wll eglect some sectos ad epad o others. For eample, the socal scetst may choose to emphasze the chapters o terpolato, curve-fttg ad statstcs, whle the physcal scetst would stress those chapters dealg wth umercal quadrature ad the soluto of dfferetal ad tegral equatos. Others mght choose to sped a large amout of tme o the prcple of least squares ad ts ramfcatos. All these approaches are vald ad I hope all wll be served by ths book. Whle t s customary to drect a book of ths sort at a specfc pedagogc audece, I fd that task somewhat dffcult. Certaly advaced udergraduate scece ad egeerg studets wll have o dffculty dealg wth the cocepts ad level of ths book. However, t s ot at all obvous that secod year studets could't cope wth the materal. Some mght suggest that they have ot yet had a formal course dfferetal equatos at that pot ther career ad are therefore ot adequately prepared. However, t s far from obvous to me that a studet s frst ecouter wth dfferetal equatos should be a formal mathematcs course. Ideed, sce most equatos they are lable to ecouter wll requre a umercal soluto, I feel the case ca be made that t s more practcal for them to be troduced to the subect from a graphcal ad umercal pot of vew. Thus, f the structor eercses some care the presetato of materal, I see o real barrer to usg ths tet at the secod year level some areas. I ay case I hope that the studet wll at least be eposed to the wde rage of the materal the book lest he feel that umercal aalyss s lmted oly to those topcs of mmedate terest to hs partcular specalty.

Nowhere s ths phlosophy better llustrated that the frst chapter where I deal wth a wde rage of mathematcal subects. The prmary obectve of ths chapter s to show that mathematcs s "all of a pece". Here the structor may choose to gore much of the materal ad ump drectly to the soluto of lear equatos ad the secod chapter. However, I hope that some cosderato would be gve to dscussg the materal o matrces preseted the frst chapter before embarkg o ther umercal mapulato. May wll feel the materal o tesors s rrelevat ad wll skp t. Certaly t s ot ecessary to uderstad covarace ad cotravarace or the oto of tesor ad vector destes order to umercally terpolate a table of umbers. But those the physcal sceces wll geerally recogze that they ecoutered tesors for the frst tme too late ther educatoal eperece ad that they form the fudametal bass for uderstadg vector algebra ad calculus. Whle the otos of set ad group theory are ot drectly requred for the uderstadg of cubc sples, they do form a ufyg bass for much of mathematcs. Thus, whle I epect most structors wll heavly select the materal from the frst chapter, I hope they wll ecourage the studets to at least read through the materal so as to reduce ther surprse whe the see t aga. The et four chapters deal wth fudametal subects basc umercal aalyss. Here, ad throughout the book, I have avoded gvg specfc programs that carry out the algorthms that are dscussed. There are may useful ad broadly based programs avalable from dverse sources. To pck specfc packages or eve specfc computer laguages would be to uduly lmt the studet's rage ad selecto. Ecellet packages are cota the IMSL lbrary ad oe should ot overlook the ecellet collecto provded alog wth the book by Press et al. (see referece 4 at the ed of Chapter ). I geeral collectos compled by users should be preferred for they have at least bee screeed tally for effcacy. Chapter 6 s a legthy treatmet of the prcple of least squares ad assocated topcs. I have foud that algorthms based o least squares are amog the most wdely used ad poorest uderstood of all algorthms the lterature. Vrtually all studets have ecoutered the cocept, but very few see ad uderstad ts relatoshp to the rest of umercal aalyss ad statstcs. Least squares also provdes a logcal brdge to the last chapters of the book. Here the huge feld of statstcs s surveyed wth the hope of provdg a basc uderstadg of the ature of statstcal ferece ad how to beg to use statstcal aalyss correctly ad wth cofdece. The foudato lad Chapter 7 ad the tests preseted Chapter 8 are ot meat to be a substtute for a proper course of study the subect. However, t s hoped that the studet uable to ft such a course a already crowded currculum wll at least be able to avod the ptfalls that trap so may who use statstcal aalyss wthout the approprate care. Throughout the book I have tred to provde eamples tegrated to the tet of the more dffcult algorthms. I testg a earler verso of the book, I foud myself spedg most of my tme wth studets gvg eamples of the varous techques ad algorthms. Hopefully ths tal shortcomg has bee overcome. It s almost always approprate to carry out a short umercal eample of a ew method so as to test the logc beg used for the more geeral case. The problems at the ed of each chapter are meat to be geerc ature so that the studet s ot left wth the mpresso that ths algorthm or that s oly used astroomy or bology. It s a farly smple matter for a structor to fd eamples dverse dscples that utlze the techques dscussed each chapter. Ideed, the studet should be ecouraged to udertake problems dscples other tha hs/her ow f for o other reaso tha to fd out about the types of problems that cocer those dscples.

Here ad there throughout the book, I have edeavored to covey somethg of the phlosophy of umercal aalyss alog wth a lttle of the phlosophy of scece. Whle ths s certaly ot the cetral theme of the book, I feel that some acquatace wth the cocepts s essetal to ayoe asprg to a career scece. Thus I hope those deas wll ot be gored by the studet o hs/her way to fd some tool to solve a mmedate problem. The phlosophy of ay subect s the bass of that subect ad to gore t whle utlzg the products of that subect s to vte dsaster. There are may people who kowgly ad ukowgly had a had geeratg ths book. Those at the Numercal Aalyss Departmet of the Uversty of Wscos who took a youg astroomy studet ad showed hm the beauty of ths subect whle remag patet wth hs bumblg uderstadg have my perpetual grattude. My colleagues at The Oho State Uversty who years ago also saw the eed for the presetato of ths materal ad provded the evromet for the developmet of a formal course the subect. Specal thaks are due Professor Phlp C. Keea who ecouraged me to clude the sectos o statstcal methods spte of my shortcomgs ths area. Peter Stoychoeff has eared my grattude by turg my crude sketches to clear ad structve drawgs. Certaly the studets who suffered through ths book as a epermetal tet have my admrato ad well as my thaks. George W. Colls, II September, 99 A Note Added for the Iteret Edto A sgfcat amout of tme has passed sce I frst put ths effort together. Much has chaged Numercal Aalyss. Researchers ow seem ofte cotet to rely o packages prepared by others eve more tha they dd a decade ago. Perhaps ths s the prce to be pad by tacklg creasgly ambtous problems. Also the advet of very fast ad cheap computers has eabled vestgators to use effcet methods ad stll obta aswers a tmely fasho. However, wth the avalache of data about to desced o more ad more felds, t does ot seem ureasoable to suppose that umercal tasks wll overtake computg power ad there wll aga be a eed for effcet ad accurate algorthms to solve problems. I suspect that may of the techques descrbed here wll be redscovered before the ew cetury cocludes. Perhaps efforts such as ths wll stll fd favor wth those who wsh to kow f umercal results ca be beleved. George W. Colls, II Jauary, v

A Further Note for the Iteret Edto Sce I put up a verso of ths book two years ago, I have foud umerous errors whch largely resulted from the geeratos of word processors through whch the tet evolved. Durg the last effort, ot all the fots used by the tet were avalable the word processor ad PDF traslator. Ths led to errors that were more wde spread that I realzed. Thus, the ma force of ths effort s to brg some uformty to the varous software codes requred to geerate the verso that wll be avalable o the teret. Havg spet some tme covertg Fudametals of Stellar Astrophyscs ad The Vral Theorem Stellar Astrophyscs to Iteret compatblty, I have leared to better uderstad the problems of takg old mauscrpts ad settg the the cotemporary format. Thus I hope ths verso of my Numercal Aalyss book wll be more error free ad therefore useable. Wll I have foud all the errors? That s most ulkely, but I ca assure the reader that the umber of those errors s sgfcatly reduced from the earler verso. I addto, I have attempted to mprove the presetato of the equatos ad other aspects of the book so as to make t more attractve to the reader. All of the software codg for the de was lost durg the travels through varous word processors. Therefore, the curret verso was prepared by meas of a page comparso betwee a earler correct verso ad the curret presetato. Such a table has a trsc error of at least ± page ad the de should be used wth that md. However, t should be good eough to gude the reader to geeral area of the desred subect. Havg re-read the earler preface ad ote I wrote, I fd I stll share the setmets epressed there. Ideed, I fd the flght of the studet to black-bo computer programs to obta solutos to problems has proceeded eve faster tha I thought t would. May of these programs such as MATHCAD are ecellet ad provde quck ad geerally accurate frst looks at problems. However, the researcher would be well advsed to uderstad the methods used by the black-boes to solve ther problems. Ths effort stll provdes the bass for may of the operatos cotaed those commercal packages ad t s hoped wll provde the researcher wth the kowledge of ther applcablty to hs/her partcular problem. However, t has occurred to me that there s a addtoal vew provded by ths book. Perhaps, the future, a hstora may woder what sort of umercal sklls were epected of a researcher the md tweteth cetury. I my opo, the cotets of ths book represet what I feel scetsts ad egeers of the md tweteth cetury should have kow ad may dd. I am cofdet that the kowledge-base of the md twety frst cetury scetst wll be qute dfferet. Oe ca hope that the dfferece wll represet a mprovemet. Fally, I would lke to thak Joh Mart ad Charles Ko who helped me adapt ths verso for the Iteret ad the Astroomy Departmet at the Case Wester Reserve Uversty for makg the server-space avalable for the PDF fles. As s the case wth other books I have put o the Iteret, I ecourage ayoe who s terested to dow load the PDF fles as they may be of use to them. I would oly request that they observe the courtesy of proper attrbuto should they fd my efforts to be of use. George W. Colls, II Aprl, Case Wester Reserve Uversty v

Itroducto ad Fudametal Cocepts The umercal epresso of a scetfc statemet has tradtoally bee the maer by whch scetsts have verfed a theoretcal descrpto of the physcal world. Durg ths cetury there has bee a revoluto both the ature ad etet to whch ths umercal comparso ca be made. Ideed, t seems lkely that whe the hstory of ths cetury s deftvely wrtte, t wll be the developmet of the computer, whch wll be regarded as ts greatest techologcal achevemet - ot uclear power. Whle t s true that the orgs of the dgtal computer ca be traced through the work of Isaac Babbtt, Herma Hollerth, ad others the eteeth cetury, the real advace came after the Secod World War whe maches were developed that were able to carry out a eteded sequece of structos at a rate that was very much greater tha a huma could maage. We call such maches programmable. The electroc dgtal computer of the sort developed by Joh vo Neuma ad others the 95s really ushered the preset computer revoluto. Whle t s stll to soo to deleate the form ad cosequeces of ths revoluto, t s already clear that t has forever chaged the way whch scece ad egeerg wll be doe. The etre approach to umercal aalyss has chaged the past two decades ad that chage wll most certaly cotue rapdly to the future. Pror to the advet of the electroc dgtal computer, the emphass computg was o short cuts ad methods of verfcato whch sured that computatoal errors could be caught before they propagated through the soluto. Lttle atteto was pad to "roud off error" sce the "huma computer" could easly cotrol such problems whe they were ecoutered. Now the relablty of electroc maches has early elmated cocers of radom error, but roud off error ca be a persstet problem.

Numercal Methods ad Data Aalyss The etreme speed of cotemporary maches has tremedously epaded the scope of umercal problems that may be cosdered as well as the maer whch such computatoal problems may eve be approached. However, ths epaso of the degree ad type of problem that may be umercally solved has removed the scetst from the detals of the computato. For ths, most would shout "Hooray"! But ths removal of the vestgator from the detals of computato may permt the propagato of errors of varous types to trude ad rema udetected. Moder computers wll almost always produce umbers, but whether they represet the soluto to the problem or the result of error propagato may ot be obvous. Ths stuato s made worse by the presece of programs desged for the soluto of broad classes of problems. Almost every class of problems has ts pathologcal eample for whch the stadard techques wll fal. Geerally lttle atteto s pad to the recogto of these pathologcal cases whch have a ucomfortable habt of turg up whe they are least epected. Thus the cotemporary scetst or egeer should be skeptcal of the aswers preseted by the moder computer uless he or she s completely famlar wth the umercal methods employed obtag that soluto. I addto, the soluto should always be subected to varous tests for "reasoableess". There s ofte a tedecy to regard the computer ad the programs whch they ru as "black boes" from whch come fallble aswers. Such a atttude ca lead to catastrophc results ad beles the atttude of "healthy skeptcsm" that should pervade all scece. It s ecessary to uderstad, at least at some level, what the "Black Boes" do. That uderstadg s oe of the prmary ams of ths book. It s ot my teto to teach the techques of programmg a computer. There are may ecellet tets o the multtudous laguages that est for commucatg wth a computer. I wll assume that the reader has suffcet capablty ths area to at least coceptualze the maer by whch certa processes could be commucated to the computer or at least recogze a computer program that does so. However, the programmg of a computer does represet a cocept that s ot foud most scetfc or mathematcal presetatos. We wll call that cocept a algorthm. A algorthm s smply a sequece of mathematcal operatos whch, whe preformed sequece, lead to the umercal aswer to some specfed problem. Much tme ad effort s devoted to ascertag the codtos uder whch a partcular algorthm wll work. I geeral, we wll omt the proof ad gve oly the results whe they are kow. The use of algorthms ad the ablty of computers to carry out vastly more operatos a short terval of tme tha the huma programmer could do several lfetmes leads to some usettlg dffereces betwee umercal aalyss ad other braches of mathematcs ad scece. Much as the scetst may be uwllg to admt t, some aspects of art creep to umercal aalyss. Kowg whe a partcular algorthm wll produce correct aswers to a gve problem ofte volves a otrval amout of eperece as well as a broad based kowledge of maches ad computatoal procedures. The studet wll acheve some feelg for ths aspect of umercal aalyss by cosderg problems for whch a gve algorthm should work, but does't. I addto, we shall gve some "rules of thumb" whch dcate whe a partcular umercal method s falg. Such "rules of thumb" are ot guaratees of ether success or falure of a specfc procedure, but represet staces whe a greater heght of skeptcsm o the part of the vestgator may be warrated. As already dcated, a broad base of eperece s useful whe tryg to ascerta the valdty of the results of ay computer program. I addto, whe tryg to uderstad the utlty of ay algorthm for calculato, t s useful to have as broad a rage of mathematcal kowledge as possble. Mathematcs s

@ Fudametal Cocepts deed the laguage of scece ad the more profcet oe s the laguage the better. So a studet should realze as soo as possble that there s essetally oe subect called mathematcs, whch for reasos of coveece we break dow to specfc areas such as arthmetc, algebra, calculus, tesors, group theory, etc. The more areas that the scetst s famlar wth, the more he/she may see the relatos betwee them. The more the relatos are apparet, the more useful mathematcs wll be. Ideed, t s all too commo for the moder scetst to flee to a computer for a aswer. I caot emphasze too strogly the eed to aalyze a problem thoroughly before ay umercal soluto s attempted. Very ofte a better umercal approach wll suggest tself durg the aalyses ad occasoally oe may fd that the aswer has a closed form aalytc soluto ad a umercal soluto s uecessary. However, t s too easy to say "I do't have the backgroud for ths subect" ad thereby ever attempt to lear t. The complete study of mathematcs s too vast for ayoe to acqure hs or her lfetme. Scetsts smply develop a base ad the cotue to add to t for the rest of ther professoal lves. To be a successful scetst oe caot kow too much mathematcs. I that sprt, we shall "revew" some mathematcal cocepts that are useful to uderstadg umercal methods ad aalyss. The word revew should be take to mea a superfcal summary of the area maly doe to dcate the relato to other areas. Vrtually every area metoed has tself bee a subect for may books ad has occuped the study of some vestgators for a lfetme. Ths short treatmet should ot be costrued ay sese as beg complete. Some of ths materal wll deed be vewed as elemetary ad f thoroughly uderstood may be skmmed. However may wll fd some of these cocepts as beg far from elemetary. Nevertheless they wll sooer or later be useful uderstadg umercal methods ad provdg a bass for the kowledge that mathematcs s "all of a pece".. Basc Propertes of Sets ad Groups Most studets are troduced to the oto of a set very early ther educatoal eperece. However, the cocept s ofte preseted a vacuum wthout showg ts relato to ay other area of mathematcs ad thus t s promptly forgotte. Bascally a set s a collecto of elemets. The oto of a elemet s left delberately vague so that t may represet aythg from cows to the real umbers. The umber of elemets the set s also left uspecfed ad may or may ot be fte. Just over a cetury ago Georg Cator bascally fouded set theory ad dog so clarfed our oto of fty by showg that there are dfferet types of fte sets. He dd ths by geeralzg what we mea whe we say that two sets have the same umber of elemets. Certaly f we ca detfy each elemet oe set wth a uque elemet the secod set ad there are oe left over whe the detfcato s completed, the we would be ettled sayg that the two sets had the same umber of elemets. Cator dd ths formally wth the fte set composed of the postve tegers ad the fte set of the real umbers. He showed that t s ot possble to detfy each real umber wth a teger so that there are more real umbers tha tegers ad thus dfferet degrees of fty whch he called cardalty. He used the frst letter of the Hebrew alphabet to deote the cardalty of a fte set so that the tegers had cardalty ℵ ad the set of real umbers had cardalty of ℵ. Some of the brghtest mds of the tweteth cetury have bee cocered wth the propertes of fte sets. Our ma terest wll ceter o those sets whch have costrats placed o ther elemets for t wll be possble to make some very geeral statemets about these restrcted sets. For eample, cosder a set

Numercal Methods ad Data Aalyss where the elemets are related by some "law". Let us deote the "law" by the symbol. If two elemets are combed uder the "law" so as to yeld aother elemet the set, the set s sad to be closed wth respect to that law. Thus f a, b, ad c are elemets of the set ad a b c, (..) the the set s sad to be closed wth respect to. We geerally cosder to be some operato lke + or, but we should't feel that the cocept s lmted to such arthmetc operatos aloe. Ideed, oe mght cosder operatos such as b 'follows' a to be a eample of a law operatg o a ad b. If we place some addtoal codtos of the elemets of the set, we ca create a somewhat more restrcted collecto of elemets called a group. Let us suppose that oe of the elemets of the set s what we call a ut elemet. Such a elemet s oe whch, whe combed wth ay other elemet of the set uder the law, produces that same elemet. Thus a a. (..) Ths suggests aother useful costrat, amely that there are elemets the set that ca be desgated "verses". A verse of a elemet s oe that whe combed wth ts elemet uder the law produces the ut elemet or a - a. (..) Now wth oe further restrcto o the law tself, we wll have all the codtos requred to produce a group. The restrcto s kow as assocatvty. A law s sad to be assocatve f the order whch t s appled to three elemets does ot determe the outcome of the applcato. Thus (a b) c a (b c). (..4) If a set possess a ut elemet ad verse elemets ad s closed uder a assocatve law, that set s called a group uder the law. Therefore the ormal tegers form a group uder addto. The ut s zero ad the verse operato s clearly subtracto ad certaly the addto of ay two tegers produces aother teger. The law of addto s also assocatve. However, t s worth otg that the tegers do ot form a group uder multplcato as the verse operato (recprocal) does ot produce a member of the group (a teger). Oe mght thk that these very smple costrats would ot be suffcet to tell us much that s ew about the set, but the oto of a group s so powerful that a etre area of mathematcs kow as group theory has developed. It s sad that Eugee Wger oce descrbed all of the essetal aspects of the thermodyamcs of heat trasfer o oe sheet of paper usg the results of group theory. Whle the restrctos that eable the elemets of a set to form a group are useful, they are ot the oly restrctos that frequetly apply. The oto of commutvty s certaly preset for the laws of addto ad scalar multplcato ad, f preset, may eable us to say eve more about the propertes of our set. A law s sad to be commutatve f a b b a. (..5) A further restrcto that may be appled volves two laws say ad. These laws are sad to be dstrbutve wth respect to oe aother f a (b c) (a b) (a c). (..6) Although the laws of addto ad scalar multplcato satsfy all three restrctos, we wll ecouter commo laws the et secto that do ot. Subsets that form a group uder addto ad scalar 4

@ Fudametal Cocepts multplcato are called felds. The oto of a feld s very useful scece as most theoretcal descrptos of the physcal world are made terms of felds. Oe talks of gravtatoal, electrc, ad magetc felds physcs. Here oe s descrbg scalars ad vectors whose elemets are real umbers ad for whch there are laws of addto ad multplcato whch cause these quattes to form ot ust groups, but felds. Thus all the abstract mathematcal kowledge of groups ad felds s avalable to the scetst to ad uderstadg physcal felds.. Scalars, Vectors, ad Matrces I the last secto we metoed specfc sets of elemets called scalars ad vectors wthout beg too specfc about what they are. I ths secto we wll defe the elemets of these sets ad the varous laws that operate o them. I the sceces t s commo to descrbe pheomea terms of specfc quattes whch may take o umercal values from tme to tme. For eample, we may descrbe the atmosphere of the plaet at ay pot terms of the temperature, pressure, humdty, ozoe cotet or perhaps a polluto de. Each of these tems has a sgle value at ay stat ad locato ad we would call them scalars. The commo laws of arthmetc that operate o scalars are addto ad multplcato. As log as oe s a lttle careful ot to allow dvso by zero (ofte kow as the cacellato law) such scalars form ot oly groups, but also felds. Although oe ca geerally descrbe the codto of the atmosphere locally terms of scalar felds, the locato tself requres more tha a sgle scalar for ts specfcato. Now we eed two (three f we clude alttude) umbers, say the lattude ad logtude, whch locate that part of the atmosphere for further descrpto by scalar felds. A quatty that requres more tha oe umber for ts specfcato may be called a vector. Ideed, some have defed a vector as a "ordered -tuple of umbers". Whle may may ot fd ths too helpful, t s essetally a correct statemet, whch emphaszes the mult-compoet sde of the oto of a vector. The umber of compoets that are requred for the vector's specfcato s usually called the dmesoalty of the vector. We most commoly thk of vectors terms of spatal vectors, that s, vectors that locate thgs some coordate system. However, as suggested the prevous secto, vectors may represet such thgs as a electrc or magetc feld where the quatty ot oly has a magtude or scalar legth assocated wth t at every pot space, but also has a drecto. As log as such quattes obey laws of addto ad some sort of multplcato, they may deed be sad to form vector felds. Ideed, there are varous types of products that are assocated wth vectors. The most commo of these ad the oe used to establsh the feld ature of most physcal vector felds s called the "scalar product" or er product, or sometmes smply the dot product from the maer whch t s usually wrtte. Here the result s a scalar ad we ca operatoally defe what we mea by such a product by A B c A B. (..) Oe mght say that as the result of the operato s a scalar ot a vector, but that would be to put to restrctve a terpretato o what we mea by a vector. Specfcally, ay scalar ca be vewed as vector havg oly oe compoet (.e. a -dmesoal vector). Thus scalars become a subgroup of vectors ad sce the vector scalar product degeerates to the ordary scalar product for -dmesoal vectors, they are actually a subfeld of the more geeral oto of a vector feld. 5

Numercal Methods ad Data Aalyss It s possble to place addtoal costrats (laws) o a feld wthout destroyg the feld ature of the elemets. We most certaly do ths wth vectors. Thus we ca defe a addtoal type of product kow as the "vector product" or smply cross product aga from the way t s commoly wrtte. Thus Cartesa coordates the cross product ca be wrtte as î ĵ kˆ A B A A A î(a B A B ) ĵ(a B A B ) + kˆ(a B A B ). (..) B B B k k k k The result of ths operato s a vector, but we shall see later that t wll be useful to sharpe our defto of vectors so that ths result s a specal kd of vector. Fally, there s the "tesor product" or vector outer product that s defed as AB C. (..) C A B Here the result of applyg the "law" s a ordered array of ( m) umbers where ad m are the dmesos of the vectors A ad B respectvely. Aga, here the result of applyg the law s ot a vector ay sese of the ormal defto, but s a member of a larger class of obects we wll call tesors. But before dscussg tesors geeral, let us cosder a specal class of them kow as matrces. The result of equato (..) whle eedg more tha oe compoet for ts specfcato s clearly ot smply a vector wth dmeso ( m). The values of ad m are separately specfed ad to specfy oly the product would be to throw away formato that was tally specfed. Thus, order to keep ths formato, we ca represet the result as a array of umbers havg colums ad m rows. Such a array ca be called a matr. For matrces, the products already defed have o smple terpretato. However, there s a addtoal product kow as a matr product, whch wll allow us to at least defe a matr group. Cosder the product defed by AB C C A kb. (..4) k k Wth ths defto of a product, the ut matr deoted by wll have elemets δ specfed for m by δ. (..5) The quatty δ s called the Kroecker delta ad may be geeralzed to -dmesos. Thus the verse elemets of the group wll have to satsfy the relato k k AA -, (..6) ad we shall sped some tme the et chapter dscussg how these members of the group may be calculated. Sce matr addto ca smply be defed as the scalar addto of the elemets of the matr, 6

@ Fudametal Cocepts ad the 'ut' matr uder addto s smply a matr wth zero elemets, t s temptg to thk that the group of matrces also form a feld. However, the matr product as defed by equato (..4), whle beg dstrbutve wth respect to addto, s ot commutatve. Thus we shall have to be cotet wth matrces formg a group uder both addto ad matr multplcato but ot a feld. There s much more that ca be sad about matrces as was the case wth other subects of ths chapter, but we wll lmt ourselves to a few propertes of matrces whch wll be partcularly useful later. For eample, the traspose of a matr wth elemets A s defed as T A A. (..7) We shall see that there s a mportat class of matrces (.e. the orthoormal matrces) whose verse s ther traspose. Ths makes the calculato of the verse trval. Aother mportat scalar quatty s the trace of a matr defed as TrA A. (..8) A matr s sad to be symmetrc f A A. If, addto, the elemets are themselves comple umbers, the should the elemets of the traspose be the comple cougates of the orgal matr, the matr s sad to be Hermta or self-adot. The cougate traspose of a matr A s usually deoted by A. If the Hermta cougate of A s also A -, the the matr s sad to be utary. Should the matr A commute wth t Hermta cougate so that AA A A, (..9) the the matr s sad to be ormal. For matrces wth oly real elemets, Hermta s the same as symmetrc, utary meas the same as orthoormal ad both classes would be cosdered to be ormal. Fally, a most mportat characterstc of a matr s ts determat. It may be calculated by epaso of the matr by "mors" so that a a a det A a a a a(a a a a ) a (a a a a ) + a (a a a a ). (..) a a a Fortuately there are more straghtforward ways of calculatg the determat whch we wll cosder the et chapter. There are several theorems cocerg determats that are useful for the mapulato of determats ad whch we wll gve wthout proof.. If each elemet a row or colum of a matr s zero, the determat of the matr s zero.. If each elemet a row or colum of a matr s multpled by a scalar q, the determat s multpled by q.. If each elemet of a row or colum s a sum of two terms, the determat equals the sum of the two correspodg determats. 7

Numercal Methods ad Data Aalyss 4. If two rows or two colums are proportoal, the determat s zero. Ths clearly follows from theorems, ad. 5. If two rows or two colums are terchaged, the determat chages sg. 6. If rows ad colums of a matr are terchaged, the determat of the matr s uchaged. 7. The value of a determat of a matr s uchaged f a multple of oe row or colum s added to aother. 8. The determat of the product of two matrces s the product of the determats of the two matrces. Oe of the mportat aspects of the determat s that t s a sgle parameter that ca be used to characterze the matr. Ay such sgle parameter (.e. the sum of the absolute value of the elemets) ca be so used ad s ofte called a matr orm. We shall see that varous matr orms are useful determg whch umercal procedures wll be useful operatg o the matr. Let us ow cosder a broader class of obects that clude scalars, vectors, ad to some etet matrces.. Coordate Systems ad Coordate Trasformatos There s a area of mathematcs kow as topology, whch deals wth the descrpto of spaces. To most studets the oto of a space s tutvely obvous ad s restrcted to the three dmesoal Euclda space of every day eperece. A lttle reflecto mght persuade that studet to clude the flat plae as a allowed space. However, a lttle further geeralzato would suggest that ay tme oe has several depedet varables that they could be used to form a space for the descrpto of some pheomea. I the area of topology the oto of a space s far more geeral tha that ad may of the more eotc spaces have o kow couterpart the physcal world. We shall restrct ourselves to spaces of depedet varables, whch geerally have some physcal terpretato. These varables ca be sad to costtute a coordate frame, whch descrbes the space ad are farly hgh up the herarchy of spaces catalogued by topology. To uderstad what s meat by a coordate frame, mage a set of rgd rods or vectors all coected at a pot. We shall call such a collecto of rods a referece frame. If every pot space ca be proected oto the rods so that a uque set of rod-pots represet the space pot, the vectors are sad to spa the space. If the vectors that defe the space are locally perpedcular, they are sad to form a orthogoal coordate frame. If the vectors defg the referece frame are also ut vectors say eˆ the the codto for orthogoalty ca be wrtte as ê δ, (..) ê where δ s the Kroecker delta. Such a set of vectors wll spa a space of dmesoalty equal to the 8

umber of vectors ê @ Fudametal Cocepts. Such a space eed ot be Euclda, but f t s the the coordate frame s sad to be a Cartesa coordate frame. The covetoal yz-coordate frame s Cartesa, but oe could mage such a coordate system draw o a rubber sheet, ad the dstorted so that locally the orthogoalty codtos are stll met, but the space would o loger be Euclda or Cartesa. Of the orthogoal coordate systems, there are several that are partcularly useful for the descrpto of the physcal world. Certaly the most commo s the rectagular or Cartesa coordate frame where coordates are ofte deoted by, y, z or,,. Other commo three dmesoal frames clude sphercal polar coordates (r,θ, ϕ) ad cyldrcal coordates (ρ,ϑ,z). Ofte the most mportat part of solvg a umercal problem s choosg the proper coordate system to descrbe the problem. For eample, there are a total of thrtee orthogoal coordate frames whch Laplace's equato s separable (see Morse ad Feshbach ). I order for coordate frames to be really useful t s ecessary to kow how to get from oe to aother. That s, f we have a problem descrbed oe coordate frame, how do we epress that same problem aother coordate frame? For quattes that descrbe the physcal world, we wsh ther meag to be depedet of the coordate frame that we happe to choose. Therefore we should epect the process to have lttle to do wth the problem, but rather volve relatoshps betwee the coordate frames themselves. These relatoshps are called coordate trasformatos. Whle there are may such trasformatos mathematcs, for the purposes of ths summary we shall cocer ourselves wth lear trasformatos. Such coordate trasformatos relate the coordates oe frame to those a secod frame by meas of a system of lear algebrac equatos. Thus f a vector oe coordate system has compoets, a prmed-coordate system a vector ' to the same pot wll have compoets ' I vector otato we could wrte ths as A + B. (..) ' A + B. (..) Ths defes the geeral class of lear trasformato where A s some matr ad B s a vector. Ths geeral lear form may be dvded to two costtuets, the matr A ad the vector. It s clear that the vector B may be terpreted as a shft the org of the coordate system, whle the elemets A are the coses of the agles betwee the aes X ad X ', ad are called the drectos coses (see Fgure.). Ideed, the vector B s othg more tha a vector from the org of the u-prmed coordate frame to the org of the prmed coordate frame. Now f we cosder two pots that are fed space ad a vector coectg them, the the legth ad oretato of that vector wll be depedet of the org of the coordate frame whch the measuremets are made. That places a addtoal costrat o the types of lear trasformatos that we may cosder. For stace, trasformatos that scaled each coordate by a costat amout, whle lear, would chage the legth of the vector as measured the two coordate systems. Sce we are oly usg the coordate system as a coveet way to descrbe the vector, the coordate system ca play o role cotrollg the legth of the vector. Thus we shall restrct our vestgatos of lear trasformatos to those that trasform orthogoal coordate systems whle preservg the legth of the vector. 9

Numercal Methods ad Data Aalyss Thus the matr A must satsfy the followg codto ' ' ( A ) ( A), (..4) whch compoet form becomes A A k k A A k k k. (..5) Ths must be true for all vectors the coordate system so that A A δ A A. (..6) k k Now remember that the Kroecker delta δ s the ut matr ad ay elemet of a group that multples aother ad produces that group's ut elemet s defed as the verse of that elemet. Therefore A [A ] -. (..7) Iterchagg the rows wth the colums of a matr produces a ew matr whch we have called the traspose of the matr. Thus orthogoal trasformatos that preserve the legth of vectors have verses that are smply the traspose of the orgal matr so that A - A T. (..8) Ths meas that gve the trasformato A the lear system of equatos (..), we may vert the trasformato, or solve the lear equatos, by multplyg those equatos by the traspose of the orgal matr or T T A ' A B. (..9) Such trasformatos are called orthogoal utary trasformatos, or orthoormal trasformatos, ad the result gve equato (..9) greatly smplfes the process of carryg out a trasformato from oe coordate system to aother ad back aga. We ca further dvde orthoormal trasformatos to two categores. These are most easly descrbed by vsualzg the relatve oretato betwee the two coordate systems. Cosder a trasformato that carres oe coordate to the egatve of ts couterpart the ew coordate system whle leavg the others uchaged. If the chaged coordate s, say, the -coordate, the trasformato matr would be A, (..) whch s equvalet to vewg the frst coordate system a mrror. Such trasformatos are kow as reflecto trasformatos ad wll take a rght haded coordate system to a left haded coordate system. The legth of ay vectors wll rema uchaged. The -compoet of these vectors wll smply be replaced by ts egatve the ew coordate system. However, ths wll ot be true of "vectors" that result from the vector cross product. The values of the compoets of such a vector wll rema uchaged mplyg that a reflecto trasformato of such a vector wll result the oretato of that vector beg chaged. If you wll, ths s the org of the "rght had rule" for vector cross products. A left had rule results a vector potg the opposte drecto. Thus such vectors are ot varat to reflecto k

@ Fudametal Cocepts trasformatos because ther oretato chages ad ths s the reaso for puttg them a separate class, amely the aal (pseudo) vectors. It s worth otg that a orthoormal reflecto trasformato wll have a determat of -. The utary magtude of the determat s a result of the magtude of the vector beg uchaged by the trasformato, whle the sg shows that some combato of coordates has udergoe a reflecto. Fgure. shows two coordate frames related by the trasformato agles ϕ. Four coordates are ecessary f the frames are ot orthogoal As oe mght epect, the elemets of the secod class of orthoormal trasformatos have determats of +. These represet trasformatos that ca be vewed as a rotato of the coordate system about some as. Cosder a trasformato betwee the two coordate systems dsplayed Fgure.. The compoets of ay vector C the prmed coordate system wll be gve by C ' cosϕ cosϕ C C y' cosϕ cosϕ C y. (..) Cz' C z If we requre the trasformato to be orthoormal, the the drecto coses of the trasformato wll ot be learly depedet sce the agles betwee the aes must be π/ both coordate systems. Thus the agles must be related by ϕ ϕ ϕ ϕ ϕ + π / ϕ + π /. (..) (π ϕ ) π / ϕ π / ϕ Usg the addto dettes for trgoometrc fuctos, equato (..) ca be gve terms of the sgle agle φ by

Numercal Methods ad Data Aalyss C ' cosϕ s ϕ C C y' s ϕ cosϕ C y. (..) Cz' C z Ths trasformato ca be vewed as a smple rotato of the coordate system about the Z-as through a agle ϕ. Thus, cosϕ s ϕ Det s ϕ cosϕ cos ϕ + s ϕ +. (..4) I geeral, the rotato of ay Cartesa coordate system about oe of ts prcpal aes ca be wrtte terms of a matr whose elemets ca be epressed terms of the rotato agle. Sce these trasformatos are about oe of the coordate aes, the compoets alog that as rema uchaged. The rotato matrces for each of the three aes are P ( φ) P ( φ) y s φ cosφ s φ cosφ s φ cosφ s φ cosφ. (..5) cosφ s φ Pz ( φ) s φ cosφ It s relatvely easy to remember the form of these matrces for the row ad colum of the matr correspodg to the rotato as always cotas the elemets of the ut matr sce that compoet s ot affected by the trasformato. The dagoal elemets always cota the cose of the rotato agle whle the remag off dagoal elemets always cota the se of the agle modulo a sg. For rotatos about the - or z-aes, the sg of the upper rght off dagoal elemet s postve ad the other egatve. The stuato s ust reversed for rotatos about the y-as. So mportat are these rotato matrces that t s worth rememberg ther form so that they eed ot be re-derved every tme they are eeded. Oe ca show that t s possble to get from ay gve orthogoal coordate system to aother through a seres of three successve coordate rotatos. Thus a geeral orthoormal trasformato ca always be wrtte as the product of three coordate rotatos about the orthogoal aes of the coordate systems. It s mportat to remember that the matr product s ot commutatve so that the order of the rotatos s mportat.

@ Fudametal Cocepts.4 Tesors ad Trasformatos May studets fd the oto of tesors to be tmdatg ad therefore avod them as much as possble. After all Este was oce quoted as sayg that there were ot more tha te people the world that would uderstad what he had doe whe he publshed Geeral Theory of Relatvty. Sce tesors are the foudato of geeral relatvty that must mea that they are so esoterc that oly te people could maage them. Wrog! Ths s a beautful eample of msterpretato of a quote take out of cotet. What Este meat was that the otato he used to epress the Geeral Theory of Relatvty was suffcetly obscure that there were ulkely to be more tha te people who were famlar wth t ad could therefore uderstad what he had doe. So ufortuately, tesors have geerally bee represeted as beg far more comple tha they really are. Thus, whle readers of ths book may ot have ecoutered them before, t s hgh tme they dd. Perhaps they wll be somewhat less tmdated the et tme, for f they have ay ambto of really uderstadg scece, they wll have to come to a uderstadg of them sooer or later. I geeral a tesor has N compoets or elemets. N s kow as the dmesoalty of the tesor by aalogy wth vectors, whle s called the rak of the tesor. Thus scalars are tesors of rak zero ad vectors of ay dmeso are rak oe. So scalars ad vectors are subsets of tesors. We ca defe the law of addto the usual way by the addto of the tesor elemets. Thus the ull tesor (.e. oe whose elemets are all zero) forms the ut uder addto ad arthmetc subtracto s the verse operato. Clearly tesors form a commutatve group uder addto. Furthermore, the scalar or dot product ca be geeralzed for tesors so that the result s a tesor of rak m. I a smlar maer the outer product ca be defed so that the result s a tesor of rak m +. It s clear that all of these operatos are closed; that s, the results rema tesors. However, whle these products are geeral dstrbutve, they are ot commutatve ad thus tesors wll ot form a feld uless some addtoal restrctos are made. Oe obvous way of represetg tesors of rak s as N N square matrces Thus, the scalar product of a tesor of rak wth a vector would be wrtte as A B C C A B, (.4.) whle the tesor outer product of the same tesor ad vector could be wrtte as AB C. (.4.) C k A Bk It s clear from the defto ad specfcally from equato (.4.) that tesors may frequetly have

Numercal Methods ad Data Aalyss a rak of more tha two. However, t becomes more dffcult to dsplay all the elemets a smple geometrcal fasho so they are geerally ust lsted or descrbed. A partcularly mportat tesor of rak three s kow as the Lev-Cvta Tesor (or correctly the Lev-Cvta Tesor Desty). It plays a role that s somewhat complmetary to that of the Kroecker delta that whe ay two dces are equal the tesor elemet s zero. Whe the dces are all dfferet the tesor elemet s + or - depedg o whether the de sequece ca be obtaed as a eve or odd permutato from the sequece,, respectvely. If we try to represet the tesor ε k as a successo of matrces we would get εk + ε k. (.4.) + ε + k Ths somewhat awkward lookg thrd rak tesor allows us to wrte the equally awkward vector cross product summato otato as A B ε :(AB) ε A B C. (.4.4) k k Here the symbol : deotes the double dot product whch s eplctly specfed by the double sum of the rght had term. The quatty ε k s sometmes called the permutato symbol as t chages sg wth every permutato of ts dces. Ths, ad the detty εk εpq δ pδ kq δ qδ kp, (.4.5) makes the evaluato of some complcated vector dettes much smpler (see eercse ). I secto. we added a codto to what we meat by a vector, amely we requred that the legth of a vector be varat to a coordate trasformato. Here we see the way whch addtoal costrats of what we mea by vectors ca be specfed by the way whch they trasform. We further lmted what we meat by a vector by otg that some vectors behave stragely uder a reflecto trasformato ad callg these pseudo-vectors. Sce the Lev-Cvta tesor geerates the vector cross product from the elemets of ordary (polar) vectors, t must share ths strage trasformato property. Tesors that share ths trasformato property are, geeral, kow as tesor destes or pseudo-tesors. Therefore we should call ε k defed equato (.4.) the Lev-Cvta tesor desty. Ideed, t s the varace of tesors, vectors, ad scalars to orthoormal trasformatos that s most correctly used to defe the elemets of the group called tesors. k 4

@ Fudametal Cocepts Fgure. shows two eghborg pots P ad Q two adacet coordate systems X ad X'. The dfferetal dstace betwee the two s d. The vectoral dstace to the two pots s (P) or X X ' (P) ad X(Q) or?x '(Q) respectvely. Sce vectors are ust a specal case of the broader class of obects called tesors, we should epect these trasformato costrats to eted to the geeral class. Ideed the oly fully approprate way to defe tesors s to defe the way whch they trasform from oe coordate system to aother. To further refe the oto of tesor trasformato, we wll look more closely at the way vectors trasform. We have wrtte a geeral lear trasformato for vectors equato (..). However, ecept for rotatoal ad reflecto trasformatos, we have sad lttle about the ature of the trasformato matr A. So let us cosder how we would epress a coordate trasformato from some pot P a space to a earby eghborg pot Q. Each pot ca be represeted ay coordate system we choose. Therefore, let us cosder two coordate systems havg a commo org where the coordates are deoted by ad ' respectvely. Sce P ad Q are ear each other, we ca represet the coordates of Q to those of P ether coordate system by (Q) (P) + d. (.4.6) ' (Q) ' (P) + d' Now the coordates of the vector from P to Q wll be d ad d, the u-prmed ad prmed coordate systems respectvely. By the cha rule the two coordates wll be related by ' d ' d. (.4.7) 5

Numercal Methods ad Data Aalyss Note that equato (.4.7) does ot volve the specfc locato of pot Q but rather s a geeral epresso of the local relatoshp betwee the two coordate frames. Sce equato (.4.7) bascally descrbes how the coordates of P or Q wll chage from oe coordate system to aother, we ca detfy the elemets A from equato (..) wth the partal dervatves equato (.4.6). Thus we could epect ay vector? to trasform accordg to ' '. (.4.8) Vectors that trasform ths fasho are called cotravarat vectors. I order to dstgush them from covarat vectors, whch we shall shortly dscuss, we shall deote the compoets of the vector wth superscrpts stead of subscrpts. Thus the correct form for the trasformato of a cotravarat vector s ' '. (.4.9) We ca geeralze ths trasformato law to cotravarat tesors of rak two by kl ' ' T ' T. (.4.) l k k l Hgher rak cotravarat tesors trasform as oe would epect wth addtoal coordate chages. Oe mght thk that the use of superscrpts to represet cotravarat dces would be cofused wth epoets, but such s geerally ot the case ad the dstcto betwee ths sort of vector trasformato ad covarace s suffcetly mportat physcal scece to make the accommodato. The sorts of obects that trasform a cotravarat maer are those assocated wth, but ot lmted to, geometrcal obects. For eample, the ftesmal dsplacemets of coordates that makes up the taget vector to a curve show that t s a cotravarat vector. Whle we have used vectors to develop the oto of cotravarace, t s clear that the cocept ca be eteded to tesors of ay rak cludg rak zero. The trasformato rule for such a tesor would smply be T ' T. (.4.) I other words scalars wll be varat to cotravarat coordate trasformatos. Now stead of cosderg vector represetatos of geometrcal obects mbedded the space ad ther trasformatos, let us cosder a scalar fucto of the coordates themselves. Let such a fucto be Φ( ). Now cosder compoets of the gradet of Φ the ' -coordate frame. Aga by the cha rule Φ ' Φ '. (.4.) If we call Φ / ' a vector wth compoets V, the the trasformato law gve by equato (.4.) appears very lke equato (.4.8), but wth the partal dervatves verted. Thus we would detfy the elemets A of the lear vector trasformato represeted by equato (..) as / ', (.4.) A ad the vector trasformato would have the form V A V. (.4.4) 6

@ Fudametal Cocepts Vectors that trasform ths maer are called covarat vectors. I order to dstgush them from cotravarat vectors, the compoet dces are wrtte as subscrpts. Aga, t s ot dffcult to see how the cocept of covarace would be eteded to tesors of hgher rak ad specfcally for a secod rak covarat tesor we would have l k T ' Tlk. (.4.5) ' ' k l The use of the scalar varat Φ to defe what s meat by a covarat vector s a clue as to the types of vectors that behave as covarat vectors. Specfcally the gradet of physcal scalar quattes such as temperature ad pressure would behave as a covarat vector whle coordate vectors themselves are cotravarat. Bascally equatos (.4.5) ad (.4.) defe what s meat by a covarat or cotravarat tesor of secod rak. It s possble to have a med tesor where oe de represets covarat trasformato whle the other s cotravarat so that l k T ' Tl. (.4.6) ' k l k Ideed the Kroecker delta ca be regarded as a tesor as t s a two de symbol ad partcular t s a med tesor of rak two ad whe covarace ad cotravarace are mportat should be wrtte as. Remember that both cotravarat ad covarat trasformatos are locally lear trasformatos of the form gve by equato (..). That s, they both preserve the legth of vectors ad leave scalars uchaged. The troducto of the terms cotravarace ad covarace smply geerate two subgroups of what we earler called tesors ad defed the members of those groups by meas of ther detaled trasformato propertes. Oe ca geerally tell the dfferece betwee the two types of trasformatos by otg how the compoets deped o the coordates. If the compoets deote 'dstaces' or deped drectly o the coordates, the they wll trasform as cotravarat tesor compoets. However, should the compoets represet quattes that chage wth the coordates such as gradets, dvergeces, ad curls, the dmesoally the compoets wll deped versely o the coordates ad the wll trasform covaratly. The use of subscrpts ad superscrpts to keep these trasformato propertes straght s partcularly useful the developmet of tesor calculus as t allows for the developmet of rules for the mapulato of tesors accord wth ther specfc trasformato characterstcs. Whle coordate systems have bee used to defe the tesor characterstcs, those characterstcs are propertes of the tesors themselves ad do ot deped o ay specfc coordate frame. Ths s of cosderable mportace whe developg theores of the physcal world as aythg that s fudametal about the uverse should be depedet of ma made coordate frames. Ths s ot to say that the choce of coordate frames s umportat whe actually solvg a problem. Qute the reverse s true. Ideed, as the propertes of the physcal world represeted by tesors are depedet of coordates ad ther eplct represetato ad trasformato propertes from oe coordate system to aother are well defed, they may be qute useful reformulatg umercal problems dfferet coordate systems. δ 7

Numercal Methods ad Data Aalyss.5 Operators The oto of a mathematcal operator s etremely mportat mathematcal physcs ad there are etre books wrtte o the subect. Most studets frst ecouter operators calculus whe the otato [d/d] s troduced to deote the operatos volved fdg the dervatve of a fucto. I ths stace the operator stads for takg the lmt of the dfferece betwee adacet values of some fucto of dvded by the dfferece betwee the adacet values of as that dfferece teds toward zero. Ths s a farly complcated set of structos represeted by a relatvely smple set of symbols. The desgato of some symbol to represet a collecto of operatos s sad to represet the defto of a operator. Depedg o the detals of the defto, the operators ca ofte be treated as f they were quattes ad subect to algebrac mapulatos. The etet to whch ths s possble s determed by how well the operators satsfy the codtos for the group o whch the algebra or mathematcal system questo s defed. The operator [d/d] s a scalar operator. That s, t provdes a sgle result after operatg o some fucto defed a approprate coordate space. It ad the operator represet the fudametal operators of the ftesmal calculus. Sce [d/d] ad carry out verse operatos o fuctos, oe ca defe a detty operator by [d/d] so that cotuous dfferetable fuctos wll form a group uder the acto of these operators. I umercal aalyss there are aalogous operators ad Σ that perform smlar fuctos but wthout takg the lmt to vashgly small values of the depedet varable. Thus we could defe the forward fte dfferece operator by ts operato o some fucto f() so that f() f(+h) - f(),.(.5.) where the problem s usually scaled so that h. I a smlar maer Σ ca be defed as f ( ) f () + f ( + h) + f ( + h) + f ( + h) + f ( + h). (.5.) Such operators are most useful epressg formulae umercal aalyss. Ideed, t s possble to buld up a etre calculus of fte dffereces. Here the base for such a calculus s stead of e.7888... as the ftesmal calculus. Other operators that are useful the fte dfferece calculus are the shft operator E[f()] ad the Idetty operator I[f()] whch are defed as E[f()] f( + h). (.5.) I[f()] f() These operators are ot learly depedet as we ca wrte the forward dfferece operator as E - I. (.5.4) The fte dfferece ad summato calculus are etremely powerful whe summg seres or evaluatg covergece tests for seres. Before attemptg to evaluate a fte seres, t s useful to kow f the seres coverges. If possble, the studet should sped some tme studyg the calculus of fte dffereces. I addto to scalar operators, t s possble to defe vector ad tesor operators. Oe of the most commo vector operators s the "del" operator or "abla". It s usually deoted by the symbol ad s defed Cartesa coordates as 8

@ Fudametal Cocepts î + ĵ + kˆ. (.5.5) y z Ths sgle operator, whe combed wth the some of the products defed above, costtutes the foudato of vector calculus. Thus the dvergece, gradet, ad curl are defed as A b a B, (.5.6) A C respectvely. If we cosder A to be a cotuous vector fucto of the depedet varables that make up the space whch t s defed, the we may gve a physcal terpretato for both the dvergece ad curl. The dvergece of a vector feld s a measure of the amout that the feld spreads or cotracts at some gve pot the space (see Fgure.)..Fgure. schematcally shows the dvergece of a vector feld. I the rego where the arrows of the vector feld coverge, the dvergece s postve, mplyg a crease the source of the vector feld. The opposte s true for the rego where the feld vectors dverge. Fgure.4 schematcally shows the curl of a vector feld. The drecto of the curl s determed by the "rght had rule" whle the magtude depeds o the rate of chage of the - ad y- compoets of the vector feld wth respect to y ad.. 9

Numercal Methods ad Data Aalyss The curl s somewhat harder to vsualze. I some sese t represets the amout that the feld rotates about a gve pot. Some have called t a measure of the "swrless" of the feld. If the vcty of some pot the feld, the vectors ted to veer to the left rather tha to the rght, the the curl wll be a vector potg up ormal to the et rotato wth a magtude that measures the degree of rotato (see Fgure.4). Fally, the gradet of a scalar feld s smply a measure of the drecto ad magtude of the mamum rate of chage of that scalar feld (see Fgure.5). Fgure.5 schematcally shows the gradet of the scalar dot-desty the form of a umber of vectors at radomly chose pots the scalar feld. The drecto of the gradet pots the drecto of mamum crease of the dot-desty whle the magtude of the vector dcates the rate of chage of that desty. Wth these smple pctures md ad what we developed secto.4 t s possble to geeralze the oto of the Del-operator to other quattes. Cosder the gradet of a vector feld. Ths represets the outer product of the Del-operator wth a vector. Whle oe does't see such a thg ofte freshma physcs, t does occur more advaced descrptos of flud mechacs (ad may other places). We ow kow eough to uderstad that the result of ths operato wll be a tesor of rak two whch we ca represet as a matr. What do the compoets mea? Geeralze from the scalar case. The e elemets of the vector gradet ca be vewed as three vectors deotg the drecto of the mamum rate of chage of each of the compoets of the orgal vector. The e elemets represet a perfectly well defed quatty ad t has a useful purpose descrbg may physcal stuatos. Oe ca also cosder the dvergece of a secod rak tesor, whch s clearly a vector. I hydrodyamcs, the dvergece of the pressure tesor may reduce to the gradet of the scalar gas pressure f the macroscopc flow of the materal s small compared to the teral speed of the partcles that make up the materal. Wth some care the defto of a collecto of operators, ther acto o the elemets of a feld or group wll preserve the feld or group ature of the orgal elemets. These are the operators that are of the greatest use mathematcal physcs.

@ Fudametal Cocepts By combg the varous products defed ths chapter wth the famlar otos of vector calculus, we ca formulate a much rcher descrpto of the physcal world. Ths revew of scalar ad vector mathematcs alog wth the all-too-bref troducto to tesors ad matrces wll be useful settg up problems for ther evetual umercal soluto. Ideed, t s clear from the trasformatos descrbed the last sectos that a prme aspect umercally solvg problems wll be dealg wth lear equatos ad matrces ad that wll be the subect of the et chapter

Numercal Methods ad Data Aalyss Chapter Eercses. Show that the ratoal umbers (ot cludg zero) form a group uder addto ad multplcato. Do they also form a scalar feld?. Show that t s ot possble to put the ratoal umbers to a oe to oe correspodece wth the real umbers.. Show that the vector cross product s ot commutatve. 4. Show that the matr product s ot commutatve. 5. Is the scalar product of two secod rak tesors commutatve? If so show how you kow. If ot, gve a couter eample. 6. Gve the ecessary ad suffcet codtos for a tesor feld. 7. Show that the Kroecker delta δ s deed a med tesor. 8. Determe the ature (.e. cotravarat, covarat, or med) of the Lev-Cvta tesor desty. 9. Show that the vector cross product does deed gve rse to a pseudo-vector.. Use the forward fte dfferece operator to defe a secod order fte dfferece operator ad use t to evaluate [f()], where f() + 5 +.. If g () () (-)(-)(-) (-+), show that [g ()] g - (). g () s kow as the factoral fucto.. Show that f f() s a polyomal of degree, the t ca be epressed as a sum of factoral fuctos (see problem ).. Show that Σε k ε p q δ p δ k q - δ q δ k p, ad use the result to prove ( F) ( F) F.

@ Fudametal Cocepts Chapter Refereces ad Addtoal Readg Oe of the great books theoretcal physcs, ad the oly oe I kow that gves a complete lst of the coordate frames for whch Laplace's equato s separable s. Morse, P.M., ad Feshbach, H., "Methods of Theoretcal Physcs" (95) McGraw-Hll Book Co., Ic. New York, Toroto, Lodo, pp. 665-666. It s a rather formdable book o theoretcal physcs, but ay who aspre to a career the area should be famlar wth ts cotets. Whle may books gve ecellet troductos to moder set ad group theory, I have foud. Adree, R.V., "Selectos from Moder Abstract Algebra" (958) Hery Holt & Co. New York,to be clear ad cocse. A farly complete ad cocse dscusso of determats ca be foud. Sokolkoff, I.S., ad Redheffer, R.M., "Mathematcs of Physcs ad Moder Egeerg" (958) McGraw-Hll Book Co., Ic. New York, Toroto, Lodo, pp. 74-75. A partcularly clear ad approachable book o Tesor Calculus whch has bee reprted by Dover s 4. Syge, J.L., ad Schld, A., "Tesor Calculus" (949) Uversty of Toroto Press, Toroto. I would strogly advse ay studet of mathematcal physcs to become famlar wth ths book before attemptg books o relatvty theory that rely heavly o tesor otato. Whle there are may books o operator calculus, a veerable book o the calculus of fte dffereces s 5. Mle-Thomso, L.M., "The Calculus of Fte Dffereces" (9) Macmlla ad Co., LTD, Lodo. A more elemetary book o the use of fte dfferece equatos the socal sceces s 6. Goldberg, S., "Itroducto to Dfferece Equatos", (958) Joh Wley & Sos, Ic., Lodo. There are may fe books o umercal aalyss ad I wll refer to may of them later chapters. However, there are certa books that are vrtually uque the area ad foremost s 7. Abramowtz, M. ad Stegu, I.A., "Hadbook of Mathematcal Fuctos" Natoal Bureau of Stadards Appled Mathematcs Seres 55 (964) U.S. Govermet Prtg Offce, Washgto D.C.

Numercal Methods ad Data Aalyss Whle ths book has also bee reprted, t s stll avalable from the Govermet Prtg Offce ad represets a eceptoal buy. Appromato schemes ad may umercal results have bee collected ad are clearly preseted ths book. Oe of the more obscure seres of books are collectvely kow as the Batema mauscrpts, or 8. Batema, H., "The Batema Mauscrpt Proect" (954) Ed. A. Erde ' ly, 5 Volums, McGraw-Hll Book Co., Ic. New York, Toroto, Lodo. Harry Batema was a mathematca of cosderable skll who eoyed collectg obscure fuctoal relatoshps. Whe he ded, ths collecto was orgazed, catalogued, ad publshed as the Batema Mauscrpts. It s a truly amazg collecto of relatos. Whe all else fals a aalyss of a problem, before fleeg to the computer for a soluto, oe should cosult the Batema Mauscrpts to see f the problem could ot be trasformed to a dfferet more tractable problem by meas of oe of the remarkable relatos collected by Harry Batema. A book of smlar utlty but easer to obta ad use s 9. Lebedev, N.N., "Specal Fuctos ad Ther Applcatos" (97), Tras. R.A.Slverma. Dover Publcatos, Ic. New York. 4

The Numercal Methods for Lear Equatos ad Matrces We saw the prevous chapter that lear equatos play a mportat role trasformato theory ad that these equatos could be smply epressed terms of matrces. However, ths s oly a small segmet of the mportace of lear equatos ad matr theory to the mathematcal descrpto of the physcal world. Thus we should beg our study of umercal methods wth a descrpto of methods for mapulatg matrces ad solvg systems of lear equatos. However, before we beg ay dscusso of umercal methods, we must say somethg about the accuracy to whch those calculatos ca be made. 5

Numercal Methods ad Data Aalyss. Errors ad Ther Propagato Oe of the most relable aspects of umercal aalyss programs for the electroc dgtal computer s that they almost always produce umbers. As a result of the cosderable relablty of the maches, t s commo to regard the results of ther calculatos wth a certa ar of fallblty. However, the results ca be o better tha the method of aalyss ad mplemetato program utlzed by the computer ad these are the works of hghly fallble ma. Ths s the org of the aphorsm "garbage garbage out". Because of the large umber of calculatos carred out by these maches, small errors at ay gve stage ca rapdly propagate to large oes that destroy the valdty of the result. We ca dvde computatoal errors to two geeral categores: the frst of these we wll call roud off error, ad the secod trucato error. Roud off error s perhaps the more sdous of the two ad s always preset at some level. Ideed, ts ompresece dcates the frst problem facg us. How accurate a aswer do we requre? Dgtal computers utlze a certa umber of dgts ther calculatos ad ths base umber of dgts s kow as the precso of the mache. Ofte t s possble to double or trple the umber of dgts ad hece the phrase "double" or "trple" precso s commoly used to descrbe a calculato carred out usg ths epaded umber of dgts. It s a commo practce to use more dgts tha are ustfed by the problem smply to be sure that oe has "got t rght". For the scetst, there s a subtle dager ths that the temptato to publsh all the dgts preseted by the computer s usually overwhelmg. Thus publshed artcles ofte cota umercal results cosstg of may more decmal places tha are ustfed by the calculato or the physcs that wet to the problem. Ths ca lead to some reader ukowgly usg the results at a uustfed level of precesso thereby obtag meagless coclusos. Certaly the full mache precesso s ever ustfed, as after the frst arthmetcal calculato, there wll usually be some ucertaty the value of the last dgt. Ths s the result of the frst kd of error we called roud off error. As a etreme eample, cosder a mache that keeps oly oe sgfcat fgure ad the epoet of the calculato so that 6+ wll yeld 9. However, 6+4, 6+5, ad 6+8 wll all yeld the same aswer amely. Sce the mache oly carres oe dgt, all the other formato wll be lost. It s ot mmedately obvous what the result of 6+9, or 7+9 wll be. If the result s, the the mache s sad to roud off the calculato to the earest sgfcat dgt. However, f the result remas, the the mache s sad to trucate the addto to the earest sgfcat dgt. Whch s actually doe by the computer wll deped o both the physcal archtecture (hardware) of the mache ad the programs (software) whch struct t to carry out the operato. Should a huma operator be carryg out the calculato, t would usually be possble to see whe ths s happeg ad allow for t by keepg addtoal sgfcat fgures, but ths s geerally ot the case wth maches. Therefore, we must be careful about the propagato of roud off error to the fal computatoal result. It s temptg to say that the above eample s oly for a -dgt mache ad therefore urealstc. However, cosder the commo 6-dgt mache. It wll be uable to dstgush betwee mllo dollars ad mllo ad e dollars. Subtracto of those two umbers would yeld zero. Ths would be sgfcat to ay accoutat at a bak. Repeated operatos of ths sort ca lead to a completely meagless result the frst dgt. Ths emphaszes the questo of 'how accurate a aswer do we eed?'. For the accoutat, we clearly eed eough dgts to accout for all the moey at a level decded by the bak. For eample, the Iteral Reveue Servce allows tapayers to roud all calculatos to the earest dollar. Ths sets a lower 6

@ Lear Equatos ad Matrces boud for the umber of sgfcat dgts. Oe's come usually sets the upper boud. I the physcal world very few costats of ature are kow to more tha four dgts (the speed of lght s a otable ecepto). Thus the results of physcal modelg are rarely mportat beyod four fgures. Aga there are eceptos such as ull epermets, but geeral, scetsts should ot deceve themselves to belevg ther aswers are better aswers tha they are. How do we detect the effects of roud off error? Etre studes have bee devoted to ths subect by cosderg that roud off errors occurs bascally a radom fasho. Although computers are bascally determstc (.e. gve the same tal state, the computer wll always arrve at the same aswer), a large collecto of arthmetc operatos ca be cosdered as producg a radom collecto of roud-ups ad roud-dows. However, the umber of dgts that are affected wll also be varable, ad ths makes the problem far more dffcult to study geeral. Thus practce, whe the effects of roud off error are of great cocer, the problem ca be ru double precesso. Should both calculatos yeld the same result at the acceptable level of precesso, the roud off error s probably ot a problem. A addtoal "rule of thumb" for detectg the presece of roud off error s the appearace of a large umber of zeros at the rghthad sde of the aswers. Should the umber of zeros deped o parameters of the problem that determe the sze or umercal etet of the problem, the oe should be cocered about roud off error. Certaly oe ca thk of eceptos to ths rule, but geeral, they are ust that - eceptos. The secod form of error we called trucato error ad t should ot be cofused wth errors troduced by the "trucato" process that happes half the tme the case of roud off errors. Ths type of error results from the ablty of the appromato method to properly represet the soluto to the problem. The magtude of ths kd of error depeds o both the ature of the problem ad the type of appromato techque. For eample, cosder a umercal appromato techque that wll gve eact aswers should the soluto to the problem of terest be a polyomal (we shall show chapter that the maorty of methods of umercal aalyss are deed of ths form). Sce the soluto s eact for polyomals, the etet that the correct soluto dffers from a polyomal wll yeld a error. However, there are may dfferet kds of polyomals ad t may be that a polyomal of hgher degree appromates the soluto more accurately tha oe of lower degree. Ths provdes a ht for the practcal evaluato of trucato errors. If the calculato s repeated at dfferet levels of appromato (.e. for appromato methods that are correct for dfferet degree polyomals) ad the aswers chage by a uacceptable amout, the t s lkely that the trucato error s larger tha the acceptable amout. There are formal ways of estmatg the trucato error ad some 'blackbo' programs do ths. Ideed, there are geeral programs for fdg the solutos to dfferetal equatos that use estmates of the trucato error to adust parameters of the soluto process to optmze effcecy. However, oe should remember that these estmates are ust that - estmates subect to all the errors of calculato we have bee dscussg. It may cases the correct calculato of the trucato error s a more formdable problem tha the oe of terest. I geeral, t s useful for the aalyst to have some pror kowledge of the behavor of the soluto to the problem of terest before attemptg a detaled umercal soluto. Such kowledge wll geerally provde a 'feelg' for the form of the trucato error ad the etet to whch a partcular umercal techque wll maage t. We must keep md that both roud-off ad trucato errors wll be preset at some level ay calculato ad be wary lest they destroy the accuracy of the soluto. The acceptable level of accuracy s 7

Numercal Methods ad Data Aalyss determed by the aalyst ad he must be careful ot to am too hgh ad carry out grossly effcet calculatos, or too low ad obta meagless results. We ow tur to the soluto of lear algebrac equatos ad problems volvg matrces assocated wth those solutos. I geeral we ca dvde the approaches to the soluto of lear algebrac equatos to two broad areas. The frst of these volve algorthms that lead drectly to a soluto of the problem after a fte umber of steps whle the secod class volves a tal "guess" whch the s mproved by a successo of fte steps, each set of whch we wll call a terato. If the process s applcable ad properly formulated, a fte umber of teratos wll lead to a soluto.. Drect Methods for the Soluto of Lear Algebrac Equatos I geeral, we may wrte a system of lear algebrac equatos the form a + a + a + + a c a + a + a + + a c a + + + + a a a c, (..) a + + + + a a a c whch vector otato s A c. (..) Here s a -dmesoal vector the elemets of whch represet the soluto of the equatos. c s the costat vector of the system of equatos ad A s the matr of the system's coeffcets. We ca wrte the soluto to these equatos as A - c, (..) thereby reducg the soluto of ay algebrac system of lear equatos to fdg the verse of the coeffcet matr. We shall sped some tme descrbg a umber of methods for dog ust that. However, there are a umber of methods that eable oe to fd the soluto wthout fdg the verse of the matr. Probably the best kow of these s Cramer's Rule a. Soluto by Cramer's Rule It s ufortuate that usually the oly method for the soluto of lear equatos that studets remember from secodary educato s Cramer's rule or epaso by mors. As we shall see, ths method s rather effcet ad relatvely dffcult to program for a computer. However, as t forms sort of a stadard by whch other methods ca by udged, we wll revew t here. I Chapter [equato (..)] we gave the form for the determat of a matr. The more geeral defto s ductve so that the determat of the matr A would be gve by 8

@ Lear Equatos ad Matrces + Det A ( ) a M,. (..4) Here the summato may be take over ether or, or deed, ay mootocally creasg sequece of both. The quatty M s the determat of the matr A wth the th row ad th colum removed ad, wth the sg carred by (-) (+) s called the cofactor of the mor elemet a. Wth all ths termology, we ca smply wrte the determat as Det A C a,, a C,. (.5) By makg use of theorems ad 7 secto., we ca wrte the soluto terms of the determat of A as a a a a a a (a + a + + a ) a a a a c c c a a a a a a a a a a a a a a a a (a (a + a + a whch meas that the geeral soluto of equato (..) s gve by a a c a a a + + + + [Det ] a a a c c + a a + + a a a a ) ) a a a a, (..6) A. (..7) Ths requres evaluatg the determat of the matr A as well as a augmeted matr where the th colum has bee replaced by the elemets of the costat vector c. Evaluato of the determat of a matr requres about operatos ad ths must be repeated for each ukow, thus soluto by Cramer's rule wll requre at least operatos. I addto, to mata accuracy, a optmum path through the matr (fdg the least umercally sestve cofactors) wll requre a sgfcat amout of logc. Thus, soluto by Cramer's rule s ot a partcularly desrable approach to the umercal soluto of lear equatos ether for a computer or a had calculato. Let us cosder a smpler algorthm, whch forms the bass for oe of the most relable ad stable drect methods for the soluto of lear equatos. It also provdes a method for the verso of matrces. Let beg by descrbg the method ad the tryg to uderstad why t works. 9

Numercal Methods ad Data Aalyss b. Soluto by Gaussa Elmato Cosder represetg the set of lear equatos gve equato (..) by a a a c a a a c. (..8) a a a c Here we have suppressed the presece of the elemets of the soluto vector. Now we wll perform a seres of operatos o the rows ad colums of the coeffcet matr A ad we shall carry through the row operatos to clude the elemets of the costat vector c. I other words, we shall treat the rows as f they were deed the equatos so that aythg doe to oe elemet s doe to all. Oe begs by dvdg each row cludg the costat elemet by the lead elemet the row. The frst row s the subtracted from all the lower rows. Thus all rows but the frst wll have zero the frst colum. Now repeat these operatos for all but the frst equato startg wth the secod elemet of the secod equato producg oes the secod colum of the remag equatos. Subtractg the resultg secod le from all below wll yeld zeros the frst two colums of equato three ad below. Ths process ca be repeated utl oe has arrved at the last le represetg the last equato. Whe the dagoal coeffcet there s uty, the last term of the costat vector cotas the value of. Ths ca be used the (-)th equato represeted by the secod to the last le to obta - ad so o rght up to the frst le whch wll yeld the value of. The ame of ths method smply derves from the elmato of each ukow from the equatos below t producg a tragular system of equatos represeted by a' a' c' a' c', (..9) c' whch ca the be easly solved by back substtuto where c'. (..) c' a' + Oe of the dsadvatages of ths approach s that errors (prcpally roud off errors) from the successve subtractos buld up through the process ad accumulate the last equato for. The errors thus curred are further magfed by the process of back substtuto forcg the mamum effects of the roud-off error to. A smple modfcato to ths process allows us to more evely dstrbute the effects of roud off error yeldg a soluto of more uform accuracy. I addto, t wll provde us wth a effcet mechasm for calculato of the verse of the matr A.

@ Lear Equatos ad Matrces c. Soluto by Gauss Jorda Elmato Let us beg by wrtg the system of lear equatos as we dd equato (..8), but ow clude a ut matr wth elemets δ o the rght had sde of the epresso. Thus, a a a c a a a c. (..) a a a c We wll treat the elemets of ths matr as we do the elemets of the costat vector c. Now proceed as we dd wth the Gauss elmato method producg zeros the colums below ad to the left of the dagoal elemet. However, addto to subtractg the le whose dagoal elemet has bee made uty from all those below t, also subtract from the equatos above t as well. Ths wll requre that these equatos be ormalzed so that the correspodg elemets are made equal to oe ad the dagoal elemet wll o loger be uty. I addto to operatg o the rows of the matr A ad the elemets of C, we wll operate o the elemets of the addtoal matr whch s tally a ut matr. Carryg out these operatos row by row utl the last row s completed wll leave us wth a system of equatos that resemble a ' c' b b b a' c' b b b. (..) a' c' b b b If oe eames the operatos we have performed lght of theorems ad 7 from secto., t s clear that so far we have doe othg to chage the determat of the orgal matr A so that epaso by mors of the modfed matr represet by the elemets a' s smply accomplshed by multplyg the dagoal elemets a together. A fal step of dvdg each row by a wll yeld the ut matr o the left had sde ad elemets of the soluto vector wll be foud where the C' s were. The fal elemets of B wll be the elemets of the verse matr of A. Thus we have both solved the system of equatos ad foud the verse of the orgal matr by performg the same steps o the costat vector as well as a addtoal ut matr. Perhaps the smplest way to see why ths works s to cosder the system of lear equatos ad what the operatos mea to them. Sce all the operatos are performed o etre rows cludg the costat vector, t s clear that they costtute legal algebrac operatos that wo't chage the ature of the soluto ay way. Ideed these are othg more tha the operatos that oe would preform by had f he/she were solvg the system by elmatg the approprate varables. We have smply formalzed that procedure so that t may be carred out a systematc fasho. Such a procedure leds tself to computato by mache ad may be relatvely easly programmed. The reaso for the algorthm yeldg the matr verse s somewhat less easy to see. However, the product of A ad B wll be the ut matr, ad the operatos that go to that matr-multply are the verse of those used to geerate B.

Numercal Methods ad Data Aalyss To see specfcally how the Gauss-Jorda route works, cosder the followg system of equatos: + + + + + + 6 4. (..) If we put ths the form requred by epresso (..) we have 6 4. (..4) Now ormalze the all rows by factorg out the lead elemets of the frst colum so that 8 8 )()() (. (..5) The frst row ca the be subtracted from the remag rows (.e. rows ad ) to yeld + 8 4 6 4 6) (. (..6) Now repeat the cycle ormalzg by factorg out the elemets of the secod colum gettg + + 4 4 4 6 () 4 6) (. (..7) Subtractg the secod row from the remag rows (.e. rows ad ) gves + + 4 4 4 4 4 7 4) (. (..8) Aga repeat the cycle ormalzg by the elemets of the thrd colum so + 4 8 8 7 6 ) )()( / 4)( (, (..9) ad subtract from the remag rows to yeld + 4 8 4 7 4 5 7 4) (. (..)

@ Lear Equatos ad Matrces Fally ormalze by the remag elemets so as to produce the ut matr o the left had sde so that + 5 4 ( 4)( )(/ )( + ) 7 4 8. (..) + 7 4 The soluto to the equatos s ow cotaed the ceter vector whle the rght had matr cotas the verse of the orgal matr that was o the left had sde of epresso (..4). The scalar quatty accumulatg at the frot of the matr s the determat as t represets factors of dvdual rows of the orgal matr. Here we have repeatedly use theorem ad 7 gve secto (.) chapter. Theorem allows us to buld up the determat by factorg out elemets of the rows, whle theorem 7 guaratees that the row subtracto show epressos (..6), (..8), ad (..) wll ot chage the value of the determat. Sce the determat of the ut matr o left sde of epresso (..) s oe, the determat of the orgal matr s ust the product of the factored elemets. Thus our complete soluto s [,, + 7] Det A. (..) 5 4 A 7 4 4 I carryg out ths procedure, we have bee careful to mata full accuracy by keepg the fractos that eplctly appear as a result of the dvso. I geeral, ths wll ot be practcal ad the perceptve studet wll have otce that there s the potetal for great dffculty as a result of the dvso. Should ay of the elemets of the matr A be zero whe they are to play the role of dvsor, the a umercal sgularty wll result. Ideed, should the dagoal elemets be small, dvso would produce such large row elemets that subtracto of them from the remag rows would geerate sgfcat roudoff error. However, terchagg two rows or two colums of a system of equatos does't alter the soluto of these equatos ad, by theorem 5 of chapter (sec.), oly the sg of the determat s chaged. Sce the equatos at each step represet a system of equatos, whch have the same soluto as the orgal set, we may terchage rows ad colums at ay step the procedure wthout alterg the soluto. Thus, most Gauss-Jorda programs clude a search of the matr to place the largest elemet o the dagoal pror to dvso by that elemet so as to mmze the effects of roud off error. Should t be mpossble to remove a zero from the dvso part of ths algorthm, the oe colum of the matr ca be made to be completely zero. Such a matr has a determat, whch s zero ad the matr s sad to be sgular. Systems of equatos that are characterzed by sgular matrces have o uque soluto. It s clear that oe could approach the sgular state wthout actually reachg t. The result of ths would be to produce a soluto of oly margal accuracy. I such crcumstaces the tal matr mght have coeffcets wth s sgfcat fgures ad the soluto have oe or less. Whle there s o a pror way of kowg how early sgular the matr may be, there are several "rules of thumb" whch whle ot guarateed to resolve the stuato, geerally work. Frst cosder some characterstc of the matr that measures the typcal sze of ts elemets. Most ay reasoable crtero wll do such as the absolute value of

Numercal Methods ad Data Aalyss the largest elemet, the sum of the absolute values of the elemets, or possbly the trace. Dvde ths characterstc by the absolute value of the determat ad f the result eceeds the mache precso, the result of the soluto should be regarded wth suspco. Thus f we deote ths characterstc of the matr by M, the N log M/d, (..) where d s the determat of the orgal matr. Ths should be regarded as a ecessary, but ot suffcet, codto for the soluto to be accurate. Ideed a rough guess as to the umber of sgfcat fgures the resultat soluto s N s ~ N log M/d. (..4) Sce most Gauss-Jorda routes retur the determat as a byproduct of the soluto, t s rresposble to fal to check to see f the soluto passes ths test. A addtoal test would be the substtuto of the soluto back to the orgal equatos to see how accurately the elemets of the costat vector are reproduced. For the verse matr, oe ca always multply the orgal matr by the verse ad see to what etet the ut matr results. Ths rases a terestg questo. What do we mea whe we say that a soluto to a system of equatos s accurate. Oe could mea that each elemet of the soluto vector cotas a certa umber of sgfcat fgures, or oe mght mea that the soluto vector satsfes the equatos at some acceptable level of accuracy (.e. all elemets of the costat vector are reproduced to some predetermed umber of sgfcat fgures). It s worth otg that these two codtos are ot ecessarly the same. Cosder the stuato of a poorly codtoed system of equatos where the costat vector s oly weakly specfed by oe of the ukows. Large chages ts value wll make lttle chage the elemets of the costat vector so that tght toleraces o the costat vector wll ot yeld values of the that partcular ukow wth commesurate accuracy. Ths system would ot pass the test gve by equato (..). I geeral, there should always be a a pror specfcato of the requred accuracy of the soluto ad a effort must be made to ascerta f that level of accuracy has bee reached. d. Soluto by Matr Factorzato: The Crout Method Cosder two tragular matrces U ad V wth the followg propertes u U >. (..5) < V v Further assume that A ca be wrtte terms of these tragular matrces so that A VU. (..6) The our lear system of equatos [equato (..) ] could be wrtte as A c V( U). (..7) Multplyg by V - we have that the soluto wll be gve by a dfferet set of equatos U V c c', (..8) 4

@ Lear Equatos ad Matrces where c Vc'. (..9) If the vector c ' ca be determed, the equato (..8) has the form of the result of the Gauss elmato ad would resemble epresso (..9) ad have a soluto smlar to equato (..). I addto, equato (..9) s tragular ad has a smlarly smple soluto for the vector c '. Thus, we have replaced the geeral system of lear equatos by two tragular systems. Now the costrats o U ad V oly deped o the matr A ad the tragular costrats. I o way do they deped o the costat vector c. Thus, f oe has a large umber of equatos dfferg oly the costat vector, the matrces U ad V eed oly be foud oce. \ The matrces U ad V ca be foud from the matr A a farly smple way by u a vku k k, (..) v a vku k u k whch s ustfed by Hldebradt. The soluto of the resultg tragular equatos s the ust c' c vkc' k v k. (..) c' u k k u k + Both equatos (..) ad (..) are recursve ature that the ukow reles o prevously determed values of the same set of ukows. Thus roud-off error wll propagate systematcally throughout the soluto. So t s useful f oe attempts to arrage the tal equatos a maer whch mmzes the error propagato. However, the method volves a mmum of readly detfable dvsos ad so teds to be eceptoally stable. The stablty wll clearly be mproved as log as the system of equatos cotas large dagoal elemets. Therefore the Crout method provdes a method of smlar or greater stablty to Gauss-Jorda method ad cosderable effcecy dealg wth systems dfferg oly the costat vector. I staces where the matr A s symmetrc the equatos for u smplfy to u v /u. (..) As we shall see the ormal equatos for the least squares formalsm always have ths form so that the Crout method provdes a good bass for ther soluto. Whle equatos (..) ad (..) specfcally deleate the elemets of the factored matrces U ad V, t s useful to see the maer whch they are obtaed. Therefore let us cosder the same equatos that served as a eample for the Gauss-Jorda method [.e. equatos (..)]. I order to mplemet the Crout method we wsh to be able to epress the coeffcet matr as 5

Numercal Methods ad Data Aalyss v u u u A VU v v u u. (..) v v v u The costat vector c that appears equato (..) s c (, 4, 6 ). (..4) To factor the matr A to the matrces U ad V accordace wth equato (..), we proceed colum by colum through the matr so that the ecessary elemets of U ad V requred by equato (..) are avalable whe they are eeded. Carryg out the factorg process specfed by equatos (..) sequetally colum by colum yelds u a v v v u u v v u u u v ( a ) ( a ) ( a ) a a [ a (v u ) ] ( ) 4 [ a (v u )] [ ( ) ] [ a (v u )] [ ( ) ] a a (v (v u u u u u u u ) ( ) 8 + v u ) 4 4 4 [( ) + ( 8) ] [ a (vu + vu )] [ ( ) ( 8 )] u 4 4 /. (..5) 6

@ Lear Equatos ad Matrces Therefore we ca wrte the orgal matr A accordace wth equato (..) as A 4 8 (6 4) (9 8). (..6) 4 (4 ) (6 6 + ) Here the eplct multplcato of the two factored matrces U ad V demostrates that the factorg has bee doe correctly. Now we eed to obta the augmeted costat vector c ' specfed by equatos (..). These equatos must be solved recursvely so that the results appear the order whch they are eeded. Thus c' (c ) / v / c' [c (v c' )]/ v [4 ( )]/. (..7) c' [c (v c' + v c' )]/ v [6 ( ) + ( )]/ 4 Fally the complete soluto ca be obtaed by back-solvg the secod set of equatos (..) so that c' / u / 7 (c' u ) / u [ + (8 7)]/( 4). (..8) (c' u u ) / u [ ( ) ( 7)]/ As atcpated, we have obtaed the same soluto as equato (..). The stregth of the Crout method resdes the mmal umber of operatos requred to solve a secod set of equatos dfferg oly the costat vector. The factorg of the matr remas the same ad oly the steps specfed by equatos (..7) ad (..8) eed be repeated. I addto, the method s partcularly stable. 7

Numercal Methods ad Data Aalyss 8 e. The Soluto of Tr-dagoal Systems of Lear Equatos All the methods descrbed so far geerally requre about operatos to obta the soluto. However, there s oe frequetly occurrg system of equatos for whch etremely effcet soluto algorthms est. Ths system of equatos s called tr-dagoal because there are ever more tha three ukows ay equato ad they ca be arraged so that the coeffcet matr s composed of o-zero elemets o the ma dagoal ad the dagoal mmedately adacet to ether sde. Thus such a system would have the form + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 4 4 c a a c a a a c a a a c a a a c a a. (..9) Equatos of ths type ofte occur as a result of usg a fte dfferece operator to replace a dfferetal operator for the soluto of dfferetal equatos (see chapter 5). A route that performed straght Gauss elmato would oly be volved oe subtracto below the dagoal ormalzato elemet ad so would reach ts 'tragular' form after steps. Sce the resultg equatos would oly cota two terms, the back substtuto would also oly requre two steps meag that the etre process would requre somethg of the order of steps for the etre soluto. Ths s so very much more effcet tha the geeral soluto ad equatos of ths form occur suffcetly frequetly that the studet should be aware of ths specalzed soluto.

. Soluto of Lear Equatos by Iteratve Methods @ Lear Equatos ad Matrces So far we have dealt wth methods that wll provde a soluto to a set of lear equatos after a fte umber of steps (geerally of the order of ). The accuracy of the soluto at the ed of ths sequece of steps s fed by the ature of the equatos ad to a lesser etet by the specfc algorthm that s used. We wll ow cosder a seres of algorthms that provde aswers to a lear system of equatos cosderably fewer steps, but at a level of accuracy that wll deped o the umber of tmes the algorthm s appled. Such methods are geerally referred to as teratve methods ad they usually requre of the order of steps for each terato. Clearly for very large systems of equatos, these methods may prove very much faster tha drect methods provdg they coverge quckly to a accurate soluto. a. Soluto by the Gauss ad Gauss-Sedel Iterato Methods All teratve schemes beg by assumg that a appromate aswer s kow ad the the scheme proceeds to mprove that aswer. Thus we wll have a soluto vector that s costatly chagg from terato to terato. I geeral, we wll deote ths by a superscrpt paretheses so that () wll deote the value of at the th terato. Therefore order to beg, we wll eed a tal value of the soluto vector (). The cocept of the Gauss terato scheme s etremely smple. Take the system of lear equatos as epressed equatos (..) ad solve each oe for the dagoal value of so that c a. (..) a Now use the compoets of the tal value of o the rght had sde of equato (..) to obta a mproved value for the elemets. Ths procedure ca be repeated utl a soluto of the desred accuracy s obtaed. Thus the geeral terato formula would have the form (k ) c a (k). (..) a It s clear, that should ay of the dagoal elemets be zero, there wll be a problem wth the stablty of the method. Thus the order whch the equatos are arraged wll make a dfferece to the maer whch ths scheme proceeds. Oe mght suppose that the value of the tal guess mght fluece whether or ot the method would fd the correct aswer, but as we shall see secto.4 that s ot the case. However, the choce of the tal guess wll determe the umber of teratos requred to arrve at a acceptable aswer. The Gauss-Sedel scheme s a mprovemet o the basc method of Gauss. Let us rewrte equatos (..) as follows: (k ) (k ) c a a (k) +. (..) a Whe usg ths as a bass for a terato scheme, we ca ote that all the values of the frst 9

Numercal Methods ad Data Aalyss summato for the kth terato wll have bee determed before the value of (k) so that we could wrte the terato scheme as (k) (k ) c a a (k) +. (..4) a Here the mproved values of are utlzed as soo as they are obtaed. As oe mght epect, ths ca lead to a faster rate of covergece, but there ca be a prce for the mproved speed. The Gauss-Sedel scheme may ot be as stable as the smple Gauss method. I geeral, there seems to be a trade off betwee speed of covergece ad the stablty of terato schemes. Ideed, f we were to apply ether f the Gauss teratve methods to equatos (..) that served as a eample for the drect method, we would fd that the teratve solutos would ot coverge. We shall see later (sectos.d ad.4) that those equatos fal to satsfy the smple suffcet covergece crtera gve secto.d ad the ecessary ad suffcet codto of secto.4. Wth that md, let us cosder aother system of equatos whch does satsfy those codtos. These equatos are much more strogly dagoal tha those of equato (..) so + + + 4 + + + 5 8 5 9. (..5) For these equatos, the soluto uder the Gauss-terato scheme represeted by equatos (..) takes the form (k) (k) (k+ ) [ 8 ] (k) (k) (k+ ) [ 5 ] 4. (..6) (k+ ) (k) (k) [ 9 ] However, f we were to solve equatos (..5) by meas of the Gauss-Sedel method the teratve equatos for the soluto would be 5 4

(k+ ) (k+ ) (k+ ) (k) (k) [ 8 ] (k+ ) (k) [ 5 ] 4 (k+ ) (k+ ) [ 9 ] @ Lear Equatos ad Matrces 5. (..7) If we take the tal guess to be () () (), (..8) the repettve use of equatos (..6) ad (..7) yeld the results gve Table.. Table. Covergece of Gauss ad Gauss-Sedel Iterato Schemes k 4 5 G GS G GS G GS G GS G GS G GS G GS.....6.9.9.9.7.98.8..9.....75.65.9.64..66.99...9.....45.9.97...5..8..95. As s clear from the results labeled "G" table., the Gauss-terato scheme coverges very slowly. The correct soluto whch would evetually be obtaed s ( ) [,, ]. (..9) There s a tedecy for the soluto to oscllate about the correct soluto wth the ampltude slowly dampg out toward covergece. However, the Gauss-Sedel terato method damps ths oscllato very rapdly by employg the mproved values of the soluto as soo as they are obtaed. As a result, the Gauss-Sedel scheme has coverged o ths problem about 5 teratos whle the straght Gauss scheme stll shows sgfcat error after teratos. b. The Method of Hotellg ad Bodewg Assume that the correct soluto to equato (..) ca be wrtte as A c, (..) but that the actual soluto that s obtaed by matr verso s really (k) (k) ( A ) c. (..) Substtuto of ths soluto to the orgal equatos would yeld a slghtly dfferet costat vector, amely c 4

Numercal Methods ad Data Aalyss c Let us defe a resdual vector terms of the costat vector we started wth ad the oe that results from the substtuto of the correct soluto to the orgal equatos so that (k) (k) A. (..) ( k) (k) (k) (k) R c c A A c A( c ). (..) Solvg ths for the true soluto we get c (k) c (k) (k) (k) (k) (k) (k) (k) (k) [ A ] R [ A ] c + [ A ] c [ A ] [c c ]. (..4) The soluto of equato (..) wll volve bascally the same steps as requred to solve equato (..). Thus the quatty ( k) (k) ) wll be foud wth the same accuracy as (k) provdg R s ot too large. ( c Now we ca wrte (k) c terms c of by usg equatos (.., ) ad get c (k) (k) A A[ A ] c. (..5) ( k) Usg ths result to elmate c (k ) the appromate matr verse [A - ] (k) as from equato (..4) we ca wrte the "correct" soluto c c terms of (k) (k) [ A ] { A[ A ] }c. (..6) Here deotes the ut matr wth elemets equal to the Kroecker delta δ. Roud-off error ad other problems that gave rse to the tally accurate aswer wll realty keep from beg the correct aswer, but t may be regarded as a mprovemet over the orgal soluto. It s temptg to use equato (..6) as the bass for a cotuous terato scheme, but practce very lttle mprovemet ca be made over a sgle applcato as the errors that prevet equato (..6) from producg the correct aswer wll prevet ay further mprovemet over a sgle terato. If we compare equatos (..) ad (..6), we see that ths method provdes us wth a mechasm for mprovg the verse of a matr sce A - [A - ] (k) { A[A - ] (k) }. (..7) All of the problems of usg equato (..6) as a terato formula are preset equato (..7). However, the matr verse as obtaed from equato (..7) should be a mprovemet over [A - ] (k). To see how ths method works, cosder the equatos used to demostrate the Gauss-Jorda ad Crout methods. The eact matr verse s gve equatos (..) so we wll be able to compare the terated matr wth the correct value to udge the mprovemet. For demostrato purposes, assume that the verse equato (..) s kow oly to two sgfcat fgures so that c 4

@ Lear Equatos ad Matrces.4.5. (k) ( A ).58.5.67. (..8).8.5. Takg the costat vector to be the same as equato (..), the soluto obtaed from the mperfect matr verse would be.4.5..84 ( k) (k) ( A ) c.58.5.67 4.6. (..9).8.5. 6 6.84 ad substtuto of ths soluto to the orgal equatos [.e. equato (..)] wll yeld the costat vector c k wth the elemets....84.4 ( k) (k) A c....6.4, (..)... 6.84 5.4 that are used to obta the resdual vector equato (..). The method of Hotellg ad Bodewg operates by bascally fdg a mproved value for the matr verse ad the usg that wth the orgal costat vector to obta a mproved soluto. Therefore, usg equato (..7) to mprove the matr verse we get or for eample A - [A - ] (k) { -A[A - ] (k) }, A [A [A [A ] (k) ] (k) ] (k)..........98...4.58.8.5......98.5.67.5... (..) ad performg the fal matr multplcato we have 4

Numercal Methods ad Data Aalyss.4.5.....468.5. A.58.5.67....58.5.6668. (..).8.5.....8.5. Ths ca be compared wth the s fgure verso of the eact verse from equato (..) whch s.46667.5. A.58.5.666667. (..).8.5. Every elemet epereced a sgfcat mprovemet over the two fgure value [equato(..8)]. It s terestg that the elemets of the orgal verse for whch two fgures yeld a eact result (.e a,a, ) rema uchaged. Ths result ca be traced back to the augmetato matr [.e. the rght a had matr equato (..) thrd le]. The secod colum s detcal to the ut matr so that the secod colum of the tal verse wll be left uchaged. ad get We may ow use ths mproved verse to re-calculate the soluto from the tal costat vector c.468.5..99 A c.58.5.6668 4.. (..4).8.5. 6 6.994 As oe would epect from the mproved matr verse, the soluto represets a sgfcat mprovemet over the tal values gve by equato (..9). Ideed the dfferece betwee ths soluto ad the eact soluto gve by equato (..) s the ffth sgfcat whch s smaller tha the calculato accuracy used to obta the mproved verse. Thus we see that the method of Hotellg ad Bodewg s a powerful algorthm for mprovg a matr verse ad hece a soluto to a system of lear algebrac equatos. c. Relaato Methods for the Soluto of Lear Equatos The Method of Hotellg ad Bodewg s bascally a specalzed relaato techque ad such techques ca be used wth vrtually ay terato scheme. I geeral, relaato methods ted to play off speed of covergece for stablty. Rather tha deal wth the geeral theory of relaato techques, we wll llustrate them by ther applcato to lear equatos. (k) As equato (..8) we ca defe a resdual vector R as ( k) (k) R A c. (..5) Let us assume that each elemet of the soluto vector (k) s subect to a mprovemet δ (k ) so that 44

(k+ ) (k) + δ @ Lear Equatos ad Matrces. (..6) Sce each elemet of the soluto vector may appear each equato, a sgle correcto to a elemet ca chage the etre resdual vector. The elemets of the ew resdual vector wll dffer from the tal resdual vector by a amout δr m a m δ m. (..7) Now search the elemets of the matr δr m over the de m for the largest value of δr m ad reduce the correspodg resdual by that mamum value so that (k) (k) Ma ρ R / ( δr ). (..8) The parameter ρ s kow as the relaato parameter for the th equato ad may chage from terato to terato. The terato formula the takes the form (k+ ) (k) (k) + ρ δ. (..9) Clearly the smaller ρ s, the smaller the correcto to wll be ad the loger the terato wll take to coverge. The advatage of ths techque s that t treats each ukow a dvdual maer ad thus teds to be etremely stable. Provdg a specfc eample of a relaato process rus the rsk of appearg to lmt the cocept. Ulke the other teratve procedures we have descrbed, relaato schemes leave the choce of the correcto to the elemets of the soluto vector completely arbtrary. However, havg pcked the correctos, the scheme descrbes how much of them to apply by calculatg a relaato parameter for each elemet of the soluto vector. Whle covergece of these methods s geerally slow, ther stablty s ofte qute good. We shall demostrate that by applyg the method to the same system of equatos used to demostrate the other teratve processes [.e. equatos (..5)]. We beg by choosg the same tal soluto that we used for the tal guess of the teratve schemes [.e. (,, )]. Isertg that tal guess to equato (..5), we obta the appromate costat vector (k, whch yelds a resdual vector c ) m m R 4 5 8 5 9 5 7 8 8 5 9 8. (..) It should be emphaszed that the tal guesses are somewhat arbtrary as they oly defe a place from whch to start the terato scheme. However, we wll be able to compare the results gve Table. wth the other teratve methods. We wll further arbtrarly choose to vary all the ukows by the same amout so that δ m.. (..) Now calculate the varatoal resdual matr specfed by equato (..7) ad get 45

Numercal Methods ad Data Aalyss.9.. δr m 4....6. (..) 5.6..5 The elemet of the matr wth the largest magtude s δr.5. We may ow calculate the elemets of the relaato vector accordace wth equato (..8) ad modfy the soluto vector as equato (..9). Repeatg ths process we get the results Table. Table. Sample Iteratve Soluto for the Relaato Method k 4 7 \ ρ ρ ρ ρ ρ ρ...6 -.7...6-.7.998.6... 4.44. -.55.7 +.4.7 -.4..... 7.. -7.. -.9. -.9.99... We see that the soluto does deed coverge at a rate that s termedate betwee that obta for the Gauss method ad that of the Gauss-Sedel method. Ths applcato of relaato techques allows the relaato vector to chage approachg zero as the soluto coverges. Aother approach s to use the relaato parameter to chage the correcto gve by aother type of terato scheme such as Gauss-Sedel. Uder these codtos, t s the relaato parameter that s chose ad usually held costat whle the correctos approach zero. There are may ways to arrve at sutable values for the relaato parameter but the result wll usually be the rage ½ ρ ½. For values of ρ<½, the rate of covergece s so slow that oe s ot sure whe the soluto has bee obtaed. O rare occasos oe may choose a relaato parameter greater tha uty. Such a procedure s sad to be over relaed ad s lkely to become ustable. If ρ, the stablty s almost guarateed. We have sad a great deal about covergece, but lttle that s quattatve so let us tur to a bref dscusso of covergece wth the cofes of fed-pot terato theory. d. Covergece ad Fed-pot Iterato Theory The problems of decdg whe a correct umercal soluto to a system of equatos has bee reached are somewhat more complcated for the teratve methods tha wth the drect methods. Not oly does the practcal problem of what costtutes a suffcetly accurate soluto have to be dealt wth, but the problem of whether or ot the terato method s approachg that soluto has to be solved. The terato method wll most certaly produce a ew soluto set, but whether that set s ay closer to the 46

@ Lear Equatos ad Matrces correct set s ot mmedately obvous. However, we may look to fed-pot terato theory for some help wth ths problem. Just as there s a large body of kowledge coected wth relaato theory, there s a equally large body of kowledge relatg to f-pot terato theory. Before lookg at terato methods of may varables such as the Gauss terato scheme, let us cosder a much smpler terato scheme of oly oe varable. We could wrte such a scheme as (k) Φ[ (k-) ]. (..) Here Φ[ (k-) ] s ay fucto or algorthm that produces a ew value of based o a prevous value. Such a fucto s sad to posses a fed-pot f Φ( ). (..4) If Φ() provdes a steady successo of values of that approach the fed-pot, the t ca be sad to be a coverget teratve fucto. There s a lttle-kow theorem whch states that a ecessary ad suffcet codto for Φ() to be a coverget teratve fucto s dφ() d < ε (k). (..5) For multdmesoal teratve fuctos of the form the theorem becomes dφ ( d (k+ ) Φ ) <, ( (k) ), (..6) ε (k). (..7) However, t o loger provdes ecessary codtos, oly suffcet oes. If we apply ths to the Gauss terato scheme as descrbed by equato (..) we have a a <,. (..8) It s clear that the covergece process s strogly flueced by the sze of the dagoal elemets preset the system of equatos. Thus the equatos should be tally arraged so that the largest possble elemets are preset o the ma dagoal of the coeffcet matr. Sce the equatos are lear, the suffcet codto gve equato (..) meas that the covergece of a system of equatos uder the Gauss terato scheme s depedet of the soluto ad hece the tal guess. If equato (..) s satsfed the the Gauss terato method s guarateed to coverge. However, the umber of teratos requred to acheve that covergece wll stll deped o the accuracy of the tal guess. If we apply these codtos to equatos (..) whch we used to demostrate the drect methods 47

Numercal Methods ad Data Aalyss of soluto, we fd that a a 5. (..9) Each equato fals to satsfy the suffcet covergece crtera gve equato (..8). Thus t s ulkely that these equatos ca be solved by most teratve techques. The fact that the method of Hotellg ad Bodewg gave a sgfcatly mproved soluto s a testamet to the stablty of that method. However, t must be remembered that the method of Hotellg ad Bodewg s ot meat to be used a teratve fasho so comparso of teratve techques wth t s ot completely ustfed. The suffcet covergece crtera gve by equato (..8) essetally says that f the sum of the absolute values of the off-dagoal elemets of every row s less tha the absolute value of the dagoal elemet, the the terato sequece wll coverge. The ecessary ad suffcet codto for covergece of ths ad the Gauss Sedel Scheme s that the egevalues of the matr all be postve ad less tha oe. Thus t s approprate that we sped a lttle tme to defe what egevalues are, ther mportace to scece, ad how they may be obtaed..4 The Smlarty Trasformatos ad the Egevalues ad Vectors of a Matr I Chapter (secto.) we saw that t s ofte possble to represet oe vector terms of aother by meas of a system of lear algebrac equatos whch we called a coordate trasformato. If ths trasformato preserved the legth of the vector, t was called a orthoormal trasformato ad the matr of the trasformato coeffcets had some specal propertes. May problems scece ca be represeted terms of lear equatos of the form y A. (.4.) I geeral, these problems could be made much smpler by fdg a coordate frame so that each elemet of the trasformed vector s proportoal to the correspodg elemet of the orgal vector. I other words, does there est a space where the bass vectors are arraged so that the trasformato s a dagoal matr of the form y' S', (.4.) where ' ad y ' represet the vectors ad y ths ew space where the trasformato matr becomes dagoal. Such a trasformato s called a smlarty trasformato as each elemet of y ' would be smlar (proportoal) to the correspodg elemet of '. Now the space whch we epress ad s defed by a set of bass vectors e ad the space whch ' ad y ' are epressed s spaed by e '. If we let the trasformato that relates the uprmed ad prmed coordate frames be D, the the bass vectors are related by 48

@ Lear Equatos ad Matrces ê ' d ê. (.4.) e' De Ay lear trasformato that relates the bass vectors of two coordate frames wll trasform ay vector from oe frame to the other. Therefore D '. (.4.4) e' De If we use the results of equato (.4.4) to elmate ad y from equato (.4.) favor of ' ad y ' we get y' [ DAD ]' S'. (.4.5) Comparg ths result wth equato (.4.) we see that the codtos for S to be dagoal are DAD - S, (.4.6) whch we ca rewrte as AD T D T S. (.4.7) Here we have made use of a mplct assumpto that the trasformatos are orthoormal ad so preserve the legth of vectors. Thus the codtos that lead to equato (..8) are met ad D - D T. We ca wrte these equatos compoet form as a d d s k k k k These are systems of lear homogeeous equatos of the form whch have a soluto f ad oly f k d k δ k s,,.(.4.8) ( a δ s )d,,, (.4.9) k k Det a k k δ s,. (.4.) Now the ature of D ad S deped oly o the matr A ad o way o the values of or y. Thus they may be regarded as propertes of the matr A. The elemets s are kow as the egevalues (also as the proper values or characterstc values) of A, whle the colums that make up D are called the egevectors (or proper vectors or characterstc vectors) of A. I addto, equato (.4.) s kow as the ege (or characterstc) equato of the matr A. It s ot obvous that a smlarty trasformato ests for all matrces ad deed, geeral they do ot. However, should the matr be symmetrc, the such a trasformato s guarateed to est. Equato (.4.) suggests the maer by whch we ca fd the egevalues of a matr. The epaso of equato (.4.) by mors as equato (..), or more geerally equato (..5), makes t clear that the resultg epresso wll be a polyomal of degree s whch wll have roots whch are the egevalues. Thus oe approach to fdg the egevalues of a matr s equvalet to fdg the roots of the ege-equato (.4.9). We shall say more about fdg the roots of a polyomal the et chapter so for the momet we wll restrct ourselves to some specal techques for fdg the egevalues ad egevectors of a matr. k 49

Numercal Methods ad Data Aalyss We saw secto (.c) that dagoalzato of a matr wll ot chage the value of ts determat. Sce the applcato of the trasformato matr D ad ts verse effectvely accomplshes a dagoalzato of A to the matr S we should epect the determat to rema uchaged. Sce the determat of S wll ust be the product of the dagoal elemets we ca wrte The trace of a matr s also varat to a smlarty trasformato so Det A Πs. (.4.) Tr A Σs. (.4.) These two costrats are always eough to eable oe to fd the egevalues of a matr ad may be used to reduce the ege-equato by two ts degree. However, for the more terestg case where s large, we shall have to fd a more geeral method. Sce ay such method wll be equvalet to fdg the roots of a polyomal, we may epect such methods to be farly complcated as fdg the roots of polyomals s oe of the trckest problems umercal aalyss. So t s wth fdg the egevalues of a matr. Whle we oted that the trasformato that gves rse to S s a smlarty trasformato [equato (.4.6)], ot all smlarty trasformatos eed dagoalze a matr, but smply have the form B - AB Q. (.4.) The varace of the egevalues to smlarty trasformatos provde the bass for the geeral strategy employed by most "caed" egevalue programs. The basc dea s to force the matr A toward dagoal form by employg a seres of smlarty trasformatos. The detals of such procedures are well beyod the scope of ths book but ca be foud the refereces suggested at the ed of ths chapter, 4. However, whatever approach s selected, the prudet vestgator wll see how well the costrats gve by equatos (.4., ) are met before beg satsfed that the "caed" package has actually foud the correct egevalues of the matr. Havg foud the egevalues, the correspodg egevectors ca be foud by appealg to equato (.4.9). However, these equatos are stll homogeeous, mplyg that the elemets of the egevectors are ot uquely determed. Ideed, t s the magtude of the egevector that s usually cosdered to be uspecfed so that all that s mssg s a scale factor to be appled to each egevector. A commo approach s to smply defe oe of the elemets of the egevector to be uty thereby makg the system of equatos (.4.9) ohomogeeous ad of the form k ( a δ s )d / d a. (.4.4) I ths form the elemets of the egevector wll be foud relatve to the elemet d. k k Let us coclude our dscusso of egevalues ad ege-vectors by aga cosderg the matr of the equatos (..) used to llustrate the drect soluto schemes. We have already see from equato (..9) that these equatos faled the suffcet codtos for the estece of Gauss-Sedel teratve soluto. By evaluatg the egevalues for the matr we ca evaluate the ecessary ad suffcet codtos for covergece, amely that the egevalues all be postve ad less tha uty. k 5

@ Lear Equatos ad Matrces The matr for equatos (..) s A, (.4.4) so that the ege-equato deleated by equato (.4.) becomes (-s) Det A Det ( s) s + 6s + s. (.4.5) ( s) The cubc polyomal that results has three roots whch are the egevalues of the matr. However before solvg for the egevalues we ca evaluate the costrats gve by equatos (.4.) ad (.4.) ad get Det Tr A A s s + 6. (.4.6) The determat tells us that the egevalues caot all be postve so that the ecessary ad suffcet codtos for the covergece of Gauss-Sedel are ot fulflled cofrmg the result of suffcet codto gve by equato (..9). The costrats gve by equato (.4.6) ca also ad us fdg roots for the ege-equato (.4.5). The fact that the product of the roots s the egatve of twce ther sum suggests that two of the roots occur as a par wth opposte sg. Ths coecture s supported by Descarte's "rule of sgs" dscussed the et chapter (secto.a). Wth that kowledge coupled wth the values for the trace ad determat we fd that the roots are 6 s +. (.4.7) Thus, ot oly does oe of the egevalues volate the ecessary ad suffcet covergece crtera by beg egatve, they all do as they all have a magtude greater tha uty. We may complete the study of ths matr by fdg the ege-vectors wth the ad of equato (.4.9) so that (-s) d ( s) d. (.4.8) ( s) d As we oted earler, these equatos are homogeeous so that they have o uque soluto. Ths meas that the legth of the ege-vectors s determat. May authors ormalze them so that they are of ut legth thereby costtutg a set of ut bass vectors for further aalyss. However, we shall smply take oe compoet d to be uty thereby reducg the system of homogeeous equatos (.4.8) to a system of homogeeous equatos, 5

Numercal Methods ad Data Aalyss ( s )d d + d + ( s )d, (.4.9) whch have a uque soluto for the remag elemets of the ege-vectors. For our eample the soluto s s + 6 : D [.,.,. ] s + : D [., (7 + ) /(7 5 ), + ( ) /(7 5 ). (.4.) s : D [., (7 + ) /(7 + 5 ), ( + ) /(7 + 5 ) Should oe wsh to re-ormalze these vectors to be ut vectors, oe eed oly dvde each elemet by the magtude of the vectors. Each egevalue has ts ow assocated ege-vector so that equato (.4.) completes the aalyss of the matr A. We troduced the oto of a egevalue tally to provde a ecessary ad suffcet codto for the covergece of the Gauss-Sedel terato method for a system of lear equatos. Clearly, ths s a ecellet eample of the case where the error or covergece crtera pose a more dffcult problem tha the orgal problem. There s far more to the detaled determato of the egevalues of a matr tha merely the verso of a matr. All the dfferet classes of matrces descrbed secto. pose specal problems eve the case where dstct egevalues est. The soluto of the ege-equato (.4.) volves fdg the roots of polyomals. We shall see the et chapter that ths s a trcky problem deed. 5

@ Lear Equatos ad Matrces Chapter Eercses. Fd the verse, egevalues, ad egevectors for a (+-) - for 5, 5. Descrbe the accuracy of your aswer ad how you kow.. Solve the followg set of equatos both by meas of a drect method ad teratve method. Descrbe the methods used ad why you chose them. X + 5X - 7X 4 + X 5 - X 6 + 7X 7 + 8X 8 + X 9-5X 7X - 4X - 75X 4 +X 5-8X 6 + X 7-8X 8 + 9X 9-5X -4 X - X + 5X - 78X 5-9X 6-7X 7 +8X 8-75X 9 + X -7 5X + 5X - X - 7X 5 - X 6 + 8X 7 - X 8 +X 9-8X 4 X - 4X - 75X - 8X 4 + 8X 6 - X 7-75X 8 + X 9-8X -5 7X + 85X - 4X - 9X 4 + X 5 + X 7-7X 8 - X 9 - X X + 5X +X - 4X 4 - X 5 + X 6 + 7X 8 - X 9 +7X -6 6X + X - 7X + 89X 4-7X 5 + X 6-7X 7-8X 9 - X 5X + 47X - X + 5X 4 - X 5 + 8X 6-99X 7-8X 8 +X X + X + X + X 4 + X 5 + X 6 + X 7 + X 8 + X 9. Solve the equatos A c where a (+-) -, ad c for 5, ad 5. Use both Gauss-Jorda ad Gauss-Sedel methods ad commet o whch gves the better aswer. 4. Solve the followg system of equatos by Gauss-Jorda ad Gauss-Sedel terato startg wth a tal guess of XYZ. 8X + Y + Z. 6X + 6Y + 4.Z 4. 4X +.5Y + Z.. Commet o the accuracy of your soluto ad the relatve effcecy of the two methods. 5. Show that f A s a orthoormal matr, the A - A T. 6. If A' where A cosφ s φ s φ cosφ, fd the compoets of terms of the compoets of for φ π/6. 5

Numercal Methods ad Data Aalyss Chapter Refereces ad Supplemetal Readg A reasoable complete descrpto of the Crout factorzato method s gve by. Hldebrad, F.B., "Itroducto to Numercal Aalyss" (956) McGraw-Hll Book Co., Ic., New York, Toroto, Lodo. A very ce troducto to fed-pot terato theory s gve by. Moursud, D.G., ad Durs, C.S., "Elemetary Theory ad Applcatos of Numercal Aalyss" (988) Dover Publcatos, Ic. New York. The et two refereces provde a ecellet troducto to the determato of egevalues ad egevectors. Householder's dscusso s hghly theoretcal, but provdes the uderpgs for cotemporary methods. The work ttled "Numercal Recpes" s ust that wth some descrpto o how the recpes work. It represets probably the most complete ad useful complato of cotemporary umercal algorthms curretly avalable.. Householder, A.S., "Prcples of Numercal Aalyss" (95) McGraw-Hll Book Co., Ic., New York, Toroto, Lodo, pp.4-84. 4. Press, W.H., Flaery, B.P., Teukolsky, S.A., Vetterlg, W.T., "Numercal Recpes The Art of Scetfc Computg" (986), Cambrdge Uversty Press, Cambrdge, New York, Melboure, pp. 5-8. Rchard Hammg's most recet umercal aalyss provdes a good troducto to the methods for hadlg error aalyss, whle referece 6 s a ecellet eample of the type of effort oe may fd the Russa lterature o umercal methods. Ther approach teds to be fudametally dfferet tha the typcal wester approach ad s ofte superor as they rely o aalyss to a far greater degree tha s commo the west. 5. Hammg, R.W., "Itroducto to Appled Numercal Aalyss" (97) McGraw-Hll Book Co., Ic., New York, Sa Fracsco, Toroto, Lodo. 6. Faddeeva, V.N., "Computatoal Methods of Lear Algebra",(959), Tras. C.D. Bester, Dover Publcatos, Ic. New York. 54

Polyomal Appromato, Iterpolato, ad Orthogoal Polyomals I the last chapter we saw that the ege-equato for a matr was a polyomal whose roots were the egevalues of the matr. However, polyomals play a much larger role umercal aalyss tha provdg ust egevalues. Ideed, the foudato of most umercal aalyss methods rests o the uderstadg of polyomals. As we shall see, umercal methods are usually talored to produce eact aswers for polyomals. Thus, f the soluto to a problem s a polyomal, t s ofte possble to fd a method of aalyss, whch has zero formal trucato error. So the etet to whch a problem's soluto resembles a polyomal wll geerally determe the accuracy of the soluto. Therefore we shall sped some tme uderstadg polyomals themselves so that we may better uderstad the methods that rely o them. 55

Numercal Methods ad Data Aalyss. Polyomals ad Ther Roots Whe the term polyomal s metoed, oe geerally thks of a fucto made up of a sum of terms of the form a. However, t s possble to have a much broader defto where stead of the smple fucto we may use ay geeral fucto φ () so that a geeral defto of a polyomal would have the form P() a φ (). (..) Here the quatty s kow as the degree of the polyomal ad s usually oe less tha the umber of terms the polyomal. Whle most of what we develop ths chapter wll be correct for geeral polyomals such as those equato (..), we wll use the more commo represetato of the polyomal so that φ (). (..) Thus the commo form for a polyomal would be P() a + a + a + + a. (..) Famlar as ths form may be, t s ot the most coveet form for evaluatg the polyomal. Cosder the last term equato (..). It wll take + multplcatos to evaluate that term aloe ad multplcatos for the et lowest order term. If oe sums the seres, t s clear that t wll take (+)/ multplcatos ad addtos to evaluate P(). However, f we wrte equato (..) as P() a + (a + (a + a )) ), (..4) the, whle there are stll addtos requred for the evaluato of P(), the umber of multplcatos has bee reduced to. Sce the tme requred for a computer to carry out a multplcato s usually a order of magtude greater tha that requred for addto, equato (..4) s a cosderably more effcet way to evaluate P() tha the stadard form gve by equato (..). Equato (..4) s sometmes called the "factored form" of the polyomal ad ca be mmedately wrtte dow for ay polyomal. However, there s aother way of represetg the polyomal terms of factors, amely P() a ( )( )( ) ( ). (..5) Here the last coeffcets of the polyomal have bee replaced by quattes kow as the roots of the polyomal. It s mportat to ote that, geeral, there are (+) parameters specfyg a polyomal of degree. These parameters ca be ether the (+) coeffcets or the roots ad a multplcatve scale factor a. I order to fully specfy a polyomal ths may parameters must be specfed. We shall see that ths requremet sets costrats for terpolato. The quattes kow as the roots are ot related to the coeffcets a smple way. Ideed, t s ot obvous that the polyomal should be able to be wrtte the form of equato (..5). The fact that a 56

@ Polyomal Appromato polyomal of degree has eactly such roots s kow as the fudametal theorem of algebra ad ts proof s ot smple. As we shall see, smply fdg the roots s ot smple ad costtutes oe of the more dffcult problems umercal aalyss. Sce the roots may be ether real or comple, the most geeral approach wll have to utlze comple arthmetc. Some polyomals may have multple roots (.e. more tha oe root wth the same umercal value). Ths causes trouble for some root fdg methods. I geeral, t s useful to remove a root (or a par f they are comple) oce t s foud thereby reducg the polyomal to a lower degree. Oce t has bee reduced to a quadratc or eve a cubc, the aalytc formulae for these roots maybe used. There s a aalytc form for the geeral soluto of a quartc (.e. polyomal of 4th degree), but t s so cumbersome that t s rarely used. Sce t has bee show that there s o geeral form for the roots of polyomals of degree 5 or hgher, oe wll usually have to resort to umercal methods order to fd the roots of such polyomals. The absece of a geeral scheme for fdg the roots terms of the coeffcets meas that we shall have to lear as much about the polyomal as possble before lookg for the roots. a. Some Costrats o the Roots of Polyomals Ths subect has bee studed by some of the greatest mathematcal mds of the last several cetures ad there are umerous theorems that ca be helpful descrbg the roots. For eample, f we remultply equato (..5) the coeffcet of - s ust a tmes the egatve summato of the roots so that I a smlar maer we fd that a - a Σ. (..6) a a. (..7) We wll see that t s possble to use these relatos to obta estmates of the magtude of the roots. I addto, the magtude of the roots s bouded by ( a ) ( a ) ma ma + +. (..8) Fally there s Descarte's rule of sgs whch we all leared at oe tme but usually forgot. If we reverse the order of equato (..) so that the terms appear descedg powers of as P () a + a + a + + a, (..9) the ay chage of sg betwee two successve terms s called a varato sg. Coeffcets that are zero are gored. Wth that defto of a sg varato we ca state Descarte's rule of sgs as The umber of postve roots of P() caot eceed the umber of varatos of sg P() ad, ay case, dffers from the umber of varatos by a eve teger. A useful ad easly proved corollary to ths s The umber of egatve roots of P() caot eceed the umber of varatos sg P(-) ad, ay case, dffers from the umber of varatos by a eve teger. 57

Numercal Methods ad Data Aalyss The phrasg cocerg the "eve teger" results from the possblty of the estece of comple roots whch occur pars (provdg the coeffcets are real) where oe s the comple cougate of the other. Wth these tools, t s ofte possble to say a good deal about the propertes of the roots of the polyomal questo. Sce most of the methods for fdg roots are sequetal ad requre the removal of the roots leadg to a ew polyomal of lower degree, we should say somethg about how ths s accomplshed. b. Sythetc Dvso If we wsh to remove a factor from a polyomal we may proceed as f we were dog log dvso wth the added provso that we keep track of the approprate powers of. Thus f (-r) s to be factored out of P() we could proceed eactly the same fasho as log dvso. Cosder the specfc case where r ad P() 4 + 7 + 6 8. (..) The log dvso would the look lke ( ) + 5 + 7 7 7 7 7 + 8 + + 4 8 + 4 Thus we ca wrte P() as 4 4 5 5. (..) or geeral as P() (-)( +5-7-) 4/(-), (..) So f we evaluate the polyomal for r we get P() (-r)q() + R. (..) P(r) R. (..4) 58

@ Polyomal Appromato Now f R(r) s zero, the r s a root by defto. Ideed, oe method for mprovg roots s to carry out repeated dvso, varyg r utl the remader R s acceptably close to zero. A cursory specto of the log dvso epresso (..) shows that much more s beg wrtte dow tha s ecessary. I order for the dvso to proceed a orderly fasho, there s o freedom what s to be doe wth the lead coeffcets of the largest powers of. Ideed, the coeffcets of the resultat polyomal Q() are repeated below. Also, whe searchg for a root, the lead coeffcet of the dvsor s always oe ad therefore eed ot be wrtte dow. Thus f we wrte dow oly the coeffcets ad r-value for the dvso process, we ca compress the otato so that r + + 7 + 8 P() + + 4. (..5) Q() + + 5 7 4 R Ths shorthad form of keepg track of the dvso s kow as sythetc dvso. Eve ths otato ca be formulated terms of a recursve procedure. If we let the coeffcets of the quotet polyomal Q() be b so that Q() b + b + b + + b - -, (..6) the the process of fdg the b 's terms of the coeffcets a of the orgal polyomal P() ca be wrtte as b a b rb + a. (..7) R b Here the remader R s gve by b - ad should t be zero, the r s a root. Therefore, oce a root has bee foud, t ca be removed by sythetc dvso leadg to a ew polyomal Q(). Oe ca the beg aga to fd the roots of Q() utl the orgal polyomal has bee reduced to a cubc or less. Because of the complety of the geeral cubc, oe usually uses the quadratc formula. However, eve here Press et al suggest cauto ad recommed the use of both forms of the formula, amely b ± b 4ac a. (..8) c b ± b 4ac Should a or c be small the dscrmate wll be early the same as b ad the resultat soluto wll suffer from roud-off error. They suggest the followg smple soluto to ths problem. Defe The the two roots wll be gve by q [b + sg(b) b 4ac]/. (..9) 59

Numercal Methods ad Data Aalyss q / a c / q. (..) Let us see how oe mght aalyze our specfc polyomal equato (..). Descartes rule of sgs for P() tells us that we wll have o more tha three real postve roots whle for P(-) t states that we wll have o more tha oe real egatve root. The degree of the polyomal tself dcates that there wll be four roots all. Whe the coeffcets of a polyomal are teger, t s temptg to look for teger roots. A lttle eplorg wth sythetc dvso shows that we ca fd two roots so that P() (-)(+6)( +), (..) ad clearly the last two roots are comple. For polyomals wth real coeffcets, oe ca eve use sythetc dvso to remove comple roots. Sce the roots wll appear cougate pars, smply form the quadratc polyomal (-r)(-r * ) (r+r * ) + r r *, (..) whch wll have real coeffcets as the magary part of r cacels out of (r+r * ) ad rr * s real by defto. Oe the uses sythetc dvso to dvde out the quadratc form of equato (..). A geeral recurrece relato smlar to equato (..7) ca be developed for the purposes of mache computato. Normally the coeffcets of terestg polyomals are ot tegers ad the roots are ot smple umbers. Therefore the sythetc dvso wll have a certa roud off error so that R(r) wll ot be zero. Ths pots out oe of the greatest dffcultes to be ecoutered fdg the roots of a polyomal. The roud off error R(r) accumulates from root to root ad wll geerally deped o the order whch the roots are foud. Thus the fal quadratc polyomal that yelds the last two roots may be sgfcatly dfferet tha the correct polyomal that would result the absece of roud off error. Oe may get a feelg for the etet of ths problem by redog the calculato but fdg the roots a dfferet order. If the values are depedet of the order whch they are foud, the they are probably accurate. If ot, the they are ot. c. The Graffe Root-Squarg Process We dscuss ths process ot so much for ts practcal utlty as to show the effcacy of the costrats gve equatos (..6,7). Cosder evaluatg a polyomal for values of where are the roots so that k k+ P ( ) a a + a. (..) We may separate the terms of the polyomal to eve ad odd powers of ad sce P( ), we may arrage the odd powers so that they are all o oe sde of the equato as k k k+ k a k k+ k a k k +. (..4) 6

@ Polyomal Appromato Squarg both sdes produces epoets wth eve powers ad a polyomal wth ew coeffcets a (p) ad havg the form (p) p (p) p (p) S () a + a + + a. (..5) These ew coeffcets ca be geerated by the recurrece relato from a a (p+ ) (p) (p) (p) a a a, > k (p) k a (p) k + ( ) (a (p) ). (..6) If we cotue to repeat ths process t s clear that the largest root wll domate the sum equato (..6) so that (p) p p ma Lm Lm a (p) p p. (..7) a Sce the product of the largest two roots wll domate the sums of equato (..7), we may geeralze the result of eq (..7) so that each root wll be gve by (p) p Lm a (p) p a. (..8) Whle ths method wll prcple yeld all the roots of the polyomal, the coeffcets grow so fast that roudoff error quckly begs to domate the polyomal. However, some stace t may yeld appromate roots that wll suffce for tal guesses requred by more sophstcated methods. Impressve as ths method s theoretcally, t s rarely used. Whle the algorthm s reasoably smple, the large umber of dgts requred by eve a few steps makes the programmg of the method eceedgly dffcult. d. Iteratve Methods Most of the stadard algorthms used to fd the roots of polyomals sca the polyomal a orderly fasho searchg for the root. Ay such scheme requres a tal guess, a method for predctg a better guess, ad a system for decdg whe a root has bee foud. It s possble to cast ay such method the form of a fed-pot teratve fucto such as was dscussed secto.d. Methods havg ths form are lego so we wll dscuss oly the smplest ad most wdely used. Puttg asde the problem of establshg the tal guess, we wll tur to the cetral problem of predctg a mproved value for the root. Cosder the smple case of a polyomal wth real roots ad havg a value P( k ) for some value of the depedet varable k (see Fgure.). 6

Numercal Methods ad Data Aalyss Fgure. depcts a typcal polyomal wth real roots. Costruct the taget to the curve at the pot k ad eted ths taget to the -as. The crossg pot k+ represets a mproved value for the root the Newto-Raphso algorthm. The pot k- ca be used to costruct a secat provdg a secod method for fdg a mproved value of. May teratve techques use a straght le eteso of the fucto P() to the -as as a meas of determg a mproved value for. I the case where the straght-le appromato to the fucto s obtaed from the local taget to the curve at the pot k, we call the method the Newto-Raphso method. We ca cast ths the form of a fed-pot teratve fucto sce we are lookg for the place where P(). I order to fd the teratve fucto that wll accomplsh ths let us assume that a mproved value of the root (k) wll be gve by (k+) (k) + [ (k+) - (k) ] (k) + (k). (..9) Now sce we are appromatg the fucto locally by a straght le, we may wrte (k) (k) P[ ] α + β (k+ ) (k+ ). (..) P[ ] α + β Subtractg these two equatos we get P[ (k) ] α[ (k) (k+) ] α (k). (..) 6

@ Polyomal Appromato However the slope of the taget le α s gve by the dervatve so that α dp[ (k) ]/d. (..) Thus the Newto-Raphso terato scheme ca be wrtte as (k+) (k) P[ (k) ]/P'[ (k) ]. (..) By comparg equato (..) to equato (..8) t s clear that the fed-pot teratve fucto for Newto-Raphso terato s Φ() P()/P'(). (..4) We ca also apply the covergece crtero gve by equato (..) ad fd that the ecessary ad suffcet codto for the covergece of the Newto-Raphso terato scheme s P()P"() (k) <, ε. (..5) [P'()] Sce ths volves oly oe more dervatve tha s requred for the mplemetato of the scheme, t provdes a qute reasoable covergece crtero ad t should be used coucto wth the terato scheme. The Newto-Raphso terato scheme s far more geeral tha s mpled by ts use polyomal root fdg. Ideed, may o-lear equatos ca be dealt wth by meas of equatos (..4, 5). From equato (..), t s clear that the scheme wll yeld 'eact' aswers for frst degree polyomals or straght les. Thus we ca epect that the error at ay step wll deped o [ (k) ]. Such schemes are sad to be secod order schemes ad coverge qute rapdly. I geeral, f the error at ay step ca be wrtte as E() K ( ), (..6) where K s appromately costat throughout the rage of appromato, the appromato scheme s sad to be of (order) O( ). It s also clear that problems ca occur for ths method the evet that the root of terest s a multple root. Ay multple root of P() wll also be a root of P'(). Geometrcally ths mples that the root wll occur at a pot where the polyomal becomes taget to the -as. Sce the deomator of equato (..5) wll approach zero at least quadratcally whle the umerator may approach zero learly the vcty of the root(s), t s ulkely that the covergece crtero wll be met. I practce, the shallow slope of the taget wll cause a large correcto to (k) movg the terato scheme far from the root. A modest varato of ths approach yelds a rather more stable terato scheme. If stead of usg the local value of the dervatve to obta the slope of our appromatg le, we use a pror pot from the terato sequece, we ca costruct a secat through the pror pot ad the preset pot stead of the local taget. The straght le appromato through these two pots wll have the form (k) (k) P[ ] α + β (k ) (k ), (..7) P[ ] α + β whch, the same maer as was doe wth equato (..) yelds a value for the slope of the le of 6

Numercal Methods ad Data Aalyss (k) (k ) P[ ] P[ ] α. (..8) (k) (k ) So the teratve form of the secat terato scheme s (k) (k) (k ) P[ ][ ] (k+ ) (k). (..9) (k) (k P[ ] P[ ) ] Useful as these methods are for fdg real roots, as preseted, they wll be effectve locatg comple roots. There are umerous methods that are more sophstcated ad amout to searchg the comple plae for roots. For eample Barstow's method sythetcally dvdes the polyomal of terest by a tal quadratc factor whch yelds a remader of the form R α + β, (..4) where α ad β deped o the coeffcets of the tral quadratc form. For that form to cota two roots of the polyomal both α ad β must be zero. These two costrats allow for a two-dmesoal search the comple plae to be made usually usg a scheme such as Newto-Raphso or versos of the secat method. Press et al strogly suggest the use of the Jeks-Taub method or the Lehmer-Schur method. These rather sophstcated schemes are well beyod the scope of ths book, but may be studed Acto. Before leavg ths subect, we should say somethg about the determato of the tal guess. The lmts set by equato (..8) are useful choosg a tal value of the root. They also allow for us to devse a orderly progresso of fdg the roots - say from large to small. Whle most geeral root fdg programs wll do ths automatcally, t s worth spedg a lttle tme to see f the procedure actually follows a orderly scheme. Followg ths le, t s worth repeatg the cautos rased earler cocerg the dffcultes of fdg the roots of polyomals. The bld applcato of geeral programs s almost certa to lead to dsaster. At the very least, oe should check to see how well ay gve root satsfes the orgal polyomal. That s, to what etet s P( ). Whle eve ths does't guaratee the accuracy of the root, t s ofte suffcet to ustfy ts use some other problem.. Curve Fttg ad Iterpolato The very processes of terpolato ad curve fttg are bascally attempts to get "somethg for othg". I geeral, oe has a fucto defed at a dscrete set of pots ad desres formato about the fucto at some other pot. Well that formato smply does't est. Oe must make some assumptos about the behavor of the fucto. Ths s where some of the "art of computg" eters the pcture. Oe eeds some kowledge of what the dscrete etres of the table represet. I pckg a terpolato scheme to geerate the mssg formato, oe makes some assumptos cocerg the fuctoal ature of the tabular etres. That assumpto s that they behave as polyomals. All terpolato theory s based o polyomal appromato. To be sure the polyomals eed ot be of the smple form of equato (..), but evertheless they wll be polyomals of some form such as equato (..). 64

@ Polyomal Appromato Havg detfed that mssg formato wll be geerated o the bass that the tabular fucto s represeted by a polyomal, the problem s reduced to determg the coeffcets of that polyomal. Actually some thought should be gve to the form of the fuctos φ () whch determes the basc form of the polyomal. Ufortuately, more ofte tha ot, the fuctos are take to be ad ay dffcultes represetg the fucto are offset by creasg the order of the polyomal. As we shall see, ths s a dagerous procedure at best ad ca lead to absurd results. It s far better to see f the basc data s - say epoetal or perodc form ad use bass fuctos of the form e, s( π ), or some other approprate fuctoal form. Oe wll be able to use terpolatve fuctos of lower order whch are subect to fewer large ad uepected fluctuatos betwee the tabular pots thereby producg a more reasoable result. Havg pcked the bass fuctos of the polyomal, oe the proceeds to determe the coeffcets. We have already observed that a th degree polyomal has (+) coeffcets whch may be regarded as (+) degrees of freedom, or + free parameters to adust so as to provde the best ft to the tabular etry pots. However, oe stll has the choce of how much of the table to ft at ay gve tme. For terpolato or curve-fttg, oe assumes that the tabular data are kow wth absolute precso. Thus we epect the appromatg polyomal to reproduce the data pots eactly, but the umber of data pots for whch we wll make ths demad at ay partcular part of the table remas at the dscreto of the vestgator. We shall develop our terpolato formulae tally wthout regard to the degree of the polyomal that wll be used. I addto, although there s a great deal of lterature developed aroud terpolatg equally spaced data, we wll allow the spacg to be arbtrary. Whle we wll forgo the elegace of the fte dfferece operator our dervatos, we wll be more tha compesated by the geeralty of the results. These more geeral formulae ca always be used for equally spaced data. However, we shall lmt our geeralty to the etet that, for eamples, we shall cofe ourselves to bass fuctos of the form. The geeralzato to more eotc bass fuctos s usually straghtforward. Fally, some authors make a dstcto betwee terpolato ad curve fttg wth the latter beg eteded to a sgle fuctoal relato, whch fts a etre tabular rage. However, the approaches are bascally the same so we shall treat the two subects as oe. Let us the beg by developg Lagrage Iterpolato formulae. a. Lagrage Iterpolato Let us assume that we have a set of data pots Y( ) ad that we wsh to appromate the behavor of the fucto betwee the data pots by a polyomal of the form Φ() Σ a. (..) Now we requre eact coformty betwee the terpolatve fucto Φ( ) ad the data pots Y( ) so that Y( ) Φ( ) a,. (..) Equato (..) represets + homogeeous equatos the + coeffcets a whch we could solve usg the techques chapter. However, we would the have a sgle terpolato formula that would have to be chaged every tme we chaged the values of the depedet varable Y( ). Istead, let us combe equatos (..) ad (..) to form + homogeeous equatos of the form 65

Numercal Methods ad Data Aalyss 66. (..) () a ) Y( a Φ These equatos wll have a soluto f ad oly f () Y Y Y Det Φ. (..4) Now let ad subtract the last row of the determat from the th row so that epaso by mors alog that row wll yeld [Φ( ) Y ] k. (..5) Sce k, the value of Φ( ) must be Y( ) satsfyg the requremets gve by equato (..). Now epad equato (..4) by mors about the last colum so that Φ k ) ( )A Y( Y Y Y (). (..) Here the A () are the mors that arse from the epaso dow the last colum ad they are depedet of the Y 's. They are smply lear combatos of the ' s ad the coeffcets of the lear combato deped oly o the 's. Thus t s possble to calculate them oce for ay set of depedet varables ad use the results for ay set of Y 's. The determat k depeds oly o the spacg of the tabular values of the depedet varable ad s called the Vadermode determat ad s gve by > k d ) ( V. (..7) Therefore dvdg A () equato (..6) by the Vadermode determat we ca wrte the terpolato formula gve by equato (..6) as Φ ) ( )L Y( ) (, (..8) where L () s kow as the Lagrage Iterpolatve Polyomal ad s gve by

() ( ) @ Polyomal Appromato ( ) L. (..9) Ths s a polyomal of degree wth roots for sce oe term s skpped (.e. whe ) a product of + terms. It has some terestg propertes. For eample ( k ) L ( k ) ( ) δ k, (..) where δ k s Kroecker's delta. It s clear that for values of the depedet varable equally separated by a amout h the Lagrage polyomals become ( ) L () ( ). (..) ( )!!h The use of the Lagraga terpolato polyomals as descrbed by equatos (..8) ad (..9) suggest that etre rage of tabular etres be used for the terpolato. Ths s ot geerally the case. Oe pcks a subset of tabular pots ad uses them for the terpolato. The use of all avalable tabular data wll geerally result a polyomal of a very hgh degree possessg rapd varatos betwee the data pots that are ulkely to represet the behavor of the tabular data. Here we cofrot specfcally oe of the "artstc" aspects of umercal aalyss. We kow oly the values of the tabular data. The scheme we choose for represetg the tabular data at other values of the depedet varable must oly satsfy some aesthetc sese that we have cocerg that behavor. That sese caot be quatfed for the obectve formato o whch to evaluate t smply does ot est. To llustrate ths ad quatfy the use of the Lagraga polyomals, cosder the fuctoal values for ad Y gve Table.. We wsh to obta a value for the depedet varable Y whe the depedet varable 4. As show fgure., the varato of the tabular values Y s rapd, partcularly the vcty of 4. We must pck some set of pots to determe the terpolatve polyomals. Table. Sample Data ad Results for Lagraga Iterpolato Formulae L (4) L (4) L (4) L (4) Y Φ (4) Φ (4) Φ (4) Φ (4) -/ -/9 +/ + +/5 +4/5 8 4 6 5/ 86/5 /5 4 5 +/ +/ +/ +4/9 4 5 8 -/5 -/45 6 67

Numercal Methods ad Data Aalyss The umber of pots wll determe the order ad we must decde whch pots wll be used. The pots are usually pcked for ther promty to the desred value of the depedet varable. Let us pck them cosecutvely begg wth tabular etry k. The the th degree Lagraga polyomals wll be ( ) L. (..) + k () k ( ) k Should we choose to appromate the tabular etres by a straght le passg through pots bracketg the desred value of 4, we would get ( ) L() ( ) ( ) L () ( ) for 4 for 4. (..) Thus the terpolatve value Φ(4) gve table. s smply the average of the adacet values of Y. As ca be see fgure., ths stace of lear terpolato yelds a reasoably pleasg result. However, should we wsh to be somewhat more sophstcated ad appromate the behavor of the tabular fucto wth a parabola, we are faced wth the problem of whch three pots to pck. If we bracket the desred pot wth two pots o the left ad oe o the rght we get Lagraga polyomals of the form ( )( ) L(), ( )( ) ( )( ) L (), ( )( ) ( L () ( )( ) + )( ), 4 4 4. (..4) 68

@ Polyomal Appromato Fgure. shows the behavor of the data from Table.. The results of varous forms of terpolato are show. The appromatg polyomals for the lear ad parabolc Lagraga terpolato are specfcally dsplayed. The specfc results for cubc Lagraga terpolato, weghted Lagraga terpolato ad terpolato by ratoal frst degree polyomals are also dcated. Substtutg these polyomals to equato (..8) ad usg the values for Y from Table., we get a terpolatve polyomal of the form P () L () + 8 L () + 4 L () (7-5+6)/. (..5) Had we chose the bracketg pots to clude two o the left ad oly oe o the rght the polyomal would have the form P () 8 L () + 4 L () + L () ( -+5)/5. (..6) However, t s ot ecessary to fuctoally evaluate these polyomals to obta the terpolated value. Oly the umercal value of the Lagraga polyomals for the specfc value of the depedet varable gve 69

Numercal Methods ad Data Aalyss o the rght had sde of equatos (..4) eed be substtuted drectly to equato (..8) alog wth the approprate values of Y. Ths leads to the values for Φ (4) ad Φ(4) gve Table.. The values are qute dfferet, but bracket the result of the lear terpolato. Whle equatos (.) - (.6) provde a acceptable method of carryg out the terpolato, there are more effcetly ad readly programmed methods. Oe of the most drect of these s a recursve procedure where values of a terpolatve polyomal of degree k are ft to successve sets of the data pots. I ths method the polyomal's behavor wth s ot foud, ust ts value for a specfc choce of. Ths value s gve by ( + k )P, +,,+ k + ( )P +,+,,+ k () P, +,,+ k () ( + k ). (..7) P () Y, for k, For our test data gve table. the recursve formula gve by equato (..7) yelds (4 )Y + ( 4)Y (4 ) + ( 4) 8 P, (4) + ( ) ( ) (4 + + )Y ( 4)Y (4 5) 8 ( 4) 4 P, (4) + 6. (..8) ( ) ( 5) (4 )Y + ( 4)Y (4 5) 4 + (5 4) 4 4 4 P,4 (4) + ( 4 ) (5 8) for k. Here we see that P, (4) correspods to the lear terpolatve value obtaed usg pots ad gve table. as Φ (4). I geeral, the values of P,+ () correspod to the value of the straght le passg through pots ad + evaluated at. The et geerato of recursve polyomal-values wll correspod to parabolas passg through the pots, +, ad + evaluated at. For ths eample they are P P,,,,4 (4 (4) )P (4 4)P (4), (4) + ( ( ), (4) + ( ( 4 4)P ), 4)P (4) (4 ) + ( 4) 8 5 + ( 5),4 (4) (4 8) 6 + ( 4) ( ( 8) 4 ) 86 + 5, (..) whch correspod to the values for Φ (4) ad Φ (4) table. respectvely. The cubc whch passes through pots,,, ad 4 s the last geerato of the polyomals calculated here by ths recursve procedure ad s (4 )P (4) ( 4)P (4) (4 8) ( 5 ) ( 4) ( 86,, +,,4 + 5) P,,,4 (4) +. (..) ( ) ( 8) 5 4 The procedure descrbed by equato (..7) s kow as Nevlle's algorthm ad ca cely be summarzed by a Table.. 7

@ Polyomal Appromato The fact that these results eactly replcate those of table. s o surprse as the polyomal of a partcular degree k that passes through a set of k+ pots s uque. Thus ths algorthm descrbes a partcularly effcet method for carryg out Lagraga terpolato ad, lke most recursve proceedures, s easly mplemeted o a computer. How are we to decde whch of the parabolas s "better". I some real sese, both are equally lkely. The large value of Φ(4) results because of the rapd varato of the tabular fucto through the three chose pots (see fgure.) ad most would reect the result as beg too hgh. However, we must remember that ths s a purely subectve udgmet. Perhaps oe would be well advsed to always have the same umber of pots o ether sde so as to sure the tabular varato o ether sde s equally weghted. Ths would lead to terpolato by polyomals of a odd degree. If we chose two pots ether sde of the desred value of the depedet varable, we ft a cubc through the local pots ad obta Φ(4) whch s rather close to Φ (4). It s clear that the rapd tabular varato of the pots precedg 4 domate the terpolatve polyomals. So whch oe s correct? We must emphass that there s o obectvely "correct" aswer to ths questo. Geerally oe prefers a terpolatve fucto that vares o more rapdly that the tabular values themselves, but whe those values are sparse ths crtero s dffcult to mpose. We shall cosder addtoal terpolatve forms that ted to meet ths subectve oto over a wde rage of codtos. Let us ow tur to methods of ther costructo. Table. Parameters for the Polyomals Geerated by Nevlle's Algorthm Y P, P, + P, +, + P, +, +, + + 8 8 +5/ 4 +6 /5 5 4 4 +86/5 +4/ 4 8 5 It s possble to place addtoal costrats o the terpolatve fucto whch wll make the approprate terpolatve polyomals somewhat more dffcult to obta, but t wll always be possble to obta them through cosderato of the determatal equato smlar to equato (..6). For eample, let 7

Numercal Methods ad Data Aalyss us cosder the case where costrats are placed o the dervatve of the fucto at a gve umber of values for the depedet varable. b. Hermte Iterpolato Whle we wll use the Hermte terpolato formula to obta some hghly effcet quadrature formulae later, the prmary reaso for dscussg ths form of terpolato s to show a powerful approach to geeratg terpolato formulae from the propertes of the Lagrage polyomals. I addto to the fuctoal costrats of Lagrage terpolato gve by equato (..), let us assume that the fuctoal values of the dervatve Y'( ) are also specfed at the pots. Ths represets a addtoal (+) costrats. However, sce we have assumed that the terpolatve fucto wll be a polyomal, the relatoshp betwee a polyomal ad ts dervatve meas we shall have to be careful order that these + costrats rema learly depedet. Whle a polyomal of degree has (+) coeffcets, ts dervatve wll have oly coeffcets. Thus the specfcato of the dervatve at the varous values of the depedet varable allow for a polyomal wth + coeffcets to be used whch s a polyomal of degree +. Rather tha obta the determatal equato for the + costrats ad the fuctoal form of the terpolatve fucto, let us derve the terpolatve fucto from what we kow of the propertes of L (). For the terpolatve fucto to be depedet of the values of the depedet varable ad ts dervatve, t must have a form smlar to equato (..8) so that Φ( ) Y( )h () + Y' ( )H (). (..) As before we shall requre that the terpolatve fucto yeld the eact values of the fucto at the tabular values of the depedet varable. Thus, ) Y( ) Y( )h ( ) + Y' ( )H ( ) Φ(. (..) Now the beauty of a terpolato formula s that t s depedet of the values of the depedet varable ad, ths case, ts dervatve. Thus equato (..) must hold for ay set of data pots Y ad ther dervatves Y. So lets cosder a very specfc set of data pots gve by Y( ) Y( ), Y' ( ),. (..) Ths certaly mples that h ( ) must be oe. A dfferet set of data pots that have the propertes that Y( ), k Y( k ),, (..4) Y' ( ), wll requre that h k ( ) be zero. However, the codtos o h ( ) must be depedet of the values of the depedet varable so that both codtos must hold. Therefore 7

@ Polyomal Appromato h ( ) δ. (..5) where δ s Kroecker's delta. Fally oe ca cosder a data set where Y( )...6) Y'( ), Substtuto of ths set of data to equato (..) clearly requres that H ( ). (..7) Now let us dfferetate equato (..) wth respect to ad evaluate at the tabular values of the depedet varable. Ths yelds ) Y( ) Y( )h' ( ) + Y' ( )H' ( ) Φ' (. (..8) By choosg our data sets to have the same propertes as equatos (..,4) ad (..6), but wth the roles of the fucto ad ts dervatve reversed, we ca show that h' ( ) H' ( ) δ. (..9) We have ow place costrats o the terpolatve fuctos h (), H () ad ther dervatves at each of the + values of the depedet varable. Sce we kow that both h () ad H () are polyomals, we eed oly epress them terms of polyomals of the correct degree whch have the same propertes at the pots to uquely determe ther form. We have already show that the terpolatve polyomals wll have a degree of (+). Thus we eed oly fd a polyomal that has the form specfed by equatos (..5) ad (..9). From equato (..) we ca costruct such a polyomal to have the form h () v ()L (), (..) where v () s a lear polyomal whch wll have oly two arbtrary costats. We ca use the costrat o the ampltude ad dervatve of h ( ) to determe those two costats. Makg use of the costrats equatos (..5) ad (..9) we ca wrte that h ( ) v ( h' ( ) v' ( )L ( ) )L ( ) + v ( )L' ( )L ( ). (..) Sce v () s a lear polyomal, we ca wrte v () a + b. (..) 7

Numercal Methods ad Data Aalyss Specfcally puttg the lear form for v () to equato (..) we get v ( ) a + b v' ( ) a (a + b )L' ( ), (..) whch ca be solved for a ad b to get a L' ( ) b + L' ( ). (..4) Therefore the lear polyomal v () wll have the partcular form v () (- )L' ( ). (..5) We must follow the same procedure to specfy H (). Lke h (), t wll also be a polyomal of degree + so let us try the same form for t as we dd for h (). So H () u ()L (), (..6) where u () s also a lear polyomal whose coeffcets must be determed from H ( ) u ( )L ( ). (..7) H' ( ) u' ( )L ( ) + u ( )L' ( )L ( ) Sce L ( ) s uty, these costrats clearly lmt the values of u() ad ts dervatve at the tabular pots to be u ( ). (..8) u' ( ) Sce u () s lear ad must have the form u () α + β, (..9) we ca use equato (..8) to fe the costats α ad β as α β u () ( ), (..4) thereby completely specfyg u (). Therefore, the two fuctos h () ad H () wll have the specfc form h () [ ( )L' ( )]L () H () ( )L (). (..4) All that remas s to fd L' ( ). By dfferetatg equato (..9) wth respect to ad settg to, we get L ' ( ) (, (..4) whch meas that v () wll smplfy to k ) k 74

k Therefore the Hermte terpolatve fucto wll take the form @ Polyomal Appromato ( ) v (). (..4) ( ) k Φ( ) [Y v () + Y' u ()] ( ) ( ). (..44) Ths fucto wll match the orgal fucto Y ad ts dervatve at each of the tabular pots. Ths fucto s a polyomal of degree - wth coeffcets. These coeffcets are specfed by the costrats o the fucto ad ts dervatve. Therefore ths polyomal s uque ad whether t s obtaed the above maer, or by epaso of the determatal equato s rrelevat to the result. Whle such a specfcato s rarely eeded, ths procedure does dcate how the form of the Lagrage polyomals ca be used to specfy terpolatve fuctos that meet more complcated costrats. We wll ow cosder the mposto of a dfferet set of costrats that lead to a class of terpolatve fuctos that have foud wde applcato. c. Sples Sples are terpolatve polyomals that volve formato cocerg the dervatve of the fucto at certa pots. Ulke Hermte terpolato that eplctly vokes kowledge of the dervatve, sples utlze that formato mplctly so that specfc kowledge of the dervatve ot requred. Ulke geeral terpolato formulae of the Lagraga type, whch maybe used a small secto of a table, sples are costructed to ft a etre ru of tabular etres of the depedet varable. Whle oe ca costruct sples of ay order, the most commo oes are cubc sples as they geerate tr-dagoal equatos for the coeffcets of the polyomals. As we saw chapter, tr-dagoal equatos led themselves to rapd soluto volvg about N steps. I ths case N would be the umber of tabular etres of the depedet varable. Thus for relatvely few arthmetc operatos, oe ca costruct a set of cubc polyomals whch wll represet the fucto over ts etre tabular rage. If oe were to make a dstcto betwee terpolato ad curve fttg, that would be t. That s, oe may obta a local value of a fucto by terpolato, but f oe desres to descrbe the etre rage of a tabular fucto, oe would call that curve fttg. Because of the commo occurrece of cubc sples, we shall use them as the bass for our dscusso. Geeralzato to hgher orders s ot dffcult, but wll geerate systems of equatos for ther coeffcets that are larger tha tr-dagoal. That removes much of the attractveess of the sples for terpolato. To uderstad how sples ca be costructed, cosder a fucto wth tabular pots whose depedet varable we wll deote as ad depedet values as Y. We wll appromate the fuctoal values betwee ay two adacet pots ad + by a cubc polyomal deoted by Ψ (). Also let the terval betwee + ad be called +. (..45) Sce the cubc terpolatve polyomals Ψ () cover each of the - tervals betwee the tabular 75

Numercal Methods ad Data Aalyss pots, there wll be 4(-) costats to be determed to specfy the terpolatve fuctos. As wth Lagrage terpolato theory we wll requre that the terpolatve fucto reproduce the tabular etres so that Ψ ( ) Y. (..46) Ψ ( + ) Y + Requrg that a sgle polyomal match two successve pots meas that two adacet polyomals wll have the same value where they meet, or Ψ ( + ) Ψ+ ( + ). (..47) The requremet to match tabular pots represets learly depedet costrats o the 4-4 coeffcets of the polyomals. The remag costrats come from codtos placed o the fuctoal dervatves. Specfcally we shall requre that Ψ' ( ) Ψ' ( ). (..48) Ψ" ( ) Ψ" ( ) Ulke Hermte terpolato, we have ot specfed the magtude of the dervatves at the tabular pots, but oly that they are the same for two adacet fuctos Ψ - ( ) ad Ψ ( ) at the pots all across the tabular rage. Oly at the ed pots have we made o restrctos. Requrg the frst two dervatves of adacet polyomals to be equal where they overlap wll guaratee that the overall effect of the sples wll be to geerate a smoothly varyg fucto over the etre tabular rage. Sce all the terpolatve polyomals are cubcs, ther thrd dervatves are costats throughout the terval so that ''' ''' Ψ ( ) Ψ ( + ) cost.,. (..49) Thus the specfcato of the fuctoal value ad equalty of the frst two dervatves of adacet fuctos essetally forces the value of the thrd dervatve o each of the fuctos Ψ (). Ths represets - costrats. However, the partcular value of that costat for all polyomals s ot specfed so that ths really represets oly - costrats. I a smlar maer, the specfcato of the equalty of the dervatve of two adacet polyomals for - pots represets aother - costrats. Sce two dervatves are volved we have a addtoal -4 costrats brgg the total to 4-6. However, there were 4-4 costats to be determed order that all the cubc sples be specfed. Thus the system as specfed so far s uder-determed. Sce we have sad othg about the ed pots t seems clear that that s where the added costrats must be made. Ideed, we shall see that addtoal costrats must be placed ether o the frst or secod dervatve of the fucto at the ed pots order that the problem have a uque soluto. However, we shall leave the dscusso of the specfcato of the fal two costrats utl we have eplored the cosequeces of the 4-6 costrats we have already developed. Sce the value of the thrd dervatve of ay cubc s a costat, the costrats o the equalty of the secod dervatves of adacet sples requre that the costat be the same for all sples. Thus the secod dervatve for all sples wll have the form Ψ"() a + b. (..5) If we apply ths form to two successve tabular pots, we ca wrte Ψ" ( ) a + b Y" Ψ" + ( + ) a + + b Y" +. (..5) Here we have troduced the otato that Ψ" ( )Y". The fact of the matter s that Y" does't est. We 76

@ Polyomal Appromato have o kowledge of the real values of the dervatves of the tabular fucto aywhere. All our costrats are appled to the terpolatve polyomals Ψ () otherwse kow as the cubc sples. However, the otato s clear, ad as log as we keep the phlosophcal dstcto clear, there should be o cofuso about what Y" meas. I ay evet they are ukow ad must evetually be foud. Let us press o ad solve equatos (..5) for a ad b gettg a (Y" + Y" + ) /( + ) (Y" + Y" + ) /. (..5) b Y" (Y" + Y" ) / Substtutg these values to equato (..5) we obta the form of the secod dervatve of the cubc sple as Ψ" () [Y" + (- ) Y" (- + )]/. (..5) Now we may tegrate ths epresso twce makg use of the requremet that the fucto ad ts frst dervatve are cotuous across a tabular etry pot, ad evaluate the costats of tegrato to get Ψ () {Y Y" [( ) -( + -) ]/6}[( + -)/ ] {Y + Y" + [( ) -( -) ]/6}[( -)/ ]. (..54) Ths farly formdable epresso for the cubc sple has o quadratc term ad depeds o those ukow costats Y". To get equato (..54) we dd ot eplctly use the costrats o Ψ' () so we ca use them ow to get a set of equatos that the costats Y must satsfy. If we dfferetate equato (..54) ad make use of the codto o the frst dervatve that Ψ' -( ) Ψ' ( ), (..55) we get after some algebra that Y" - - +Y" ( - + )+Y" + 6[(Y + -Y )/ + (Y -Y - )/ - ] -. (..56) Everythg o the rght had sde s kow from the tabular etres whle the left had sde cotas three of the ukow costats Y". Thus we see that the equatos have the form of a tr-dagoal system of equatos ameable to fast soluto algorthms. Equato (..56) represets - equatos ukows clearly potg out that the problem s stll uder determed by two costats. If we arbtrarly take Y" Y", the the sples that arse from the soluto of equato (..56) are called atural sples. Keepg the secod dervatve zero wll reduce the varato the fucto ear the ed pots ad ths s usually cosdered desrable. Whle ths arbtrary choce may troduce some error ear the ed pots, the effect of that error wll dmsh as oe moves toward the mddle of the tabular rage. If oe s gve othg more tha the tabular etres Y ad, the there s lttle more that oe ca do ad the atural sples are as good as ay other assumpto. However, should aythg be kow about the frst or secod dervatves at the ed pots oe ca make a more ratoal choce for the last two costats of the problem? For eample, f the values of the frst dervatves are kow at the ed pots the dfferetatg equato (..56) ad evaluatg t at the ed pots yelds two more equatos of codto whch deped o the ed pot frst dervatves as Y Y " " + Y Y " [(Y Y ) / Y ]/ " ' / 6 [(Y Y ) / Y ]/ '. (..57) 77

Numercal Methods ad Data Aalyss These two added codtos complete the system of equatos wthout destroyg ther tr-dagoal form ad pose a sgfcat alteratve to atural sples should some kowledge of the edpot dervatves est. It s clear that ay such formato at ay pot the tabular rage could be used to further costra the system so that a uque soluto ests. I the absece of such formato oe has lttle choce but to use the aesthetcally pleasg atural sples. Oe may be somewhat dsechated that t s ecessary to appeal to esthetcs to ustfy a soluto to a problem, but aga remember that we are tryg to get "somethg for othg" ay terpolato or curve fttg problem. The "true" ature of the soluto betwee the tabular pots smply does ot est. Thus we have aother eample of where the "art of computg" s utlzed umercal aalyss. I order to see the effcacy of sples, cosder the same tabular data gve Table. ad vestgate how sples would yeld a value for the table at 4. Ulke Lagraga terpolato, the costrats that determe the values for the sples wll volve the etre table. Thus we shall have to set up the equatos specfed by equato (..56). We shall assume that atural sples wll be approprate for the eample so that Y " Y 5 ". (..58) For, equato (..56) ad the tabular values from table. yeld 4Y " + Y " 6[(8-)/ + (-)/] 4,, (..59) ad the etre system of lear equatos for the Y"'s ca be wrtte as " 4 Y 4 " 6 Y 8 " Y 6 " 7 Y 4. (..6) The soluto for ths tr-dagoal system ca be foud by ay of the methods descrbed Chapter, but t s worth otg the crease effcecy afforded by the tr-dagoal ature of the system. The soluto s gve Table.. The frst tem to ote s that the assumpto of atural sples may ot be the best, for the value of Y " s sgfcatly dfferet from the zero assumed for Y ". The value of Y" the proceeds to drop smoothly toward the other boudary mplyg that the assumpto of Y 5 " s pretty good. Substtutg the soluto for Y " to equato (..54) we get that Ψ (4) {8 -.9876[4-(5-4) ]/6}(4-)/ {4 - (-.964)[4-(-4) ]/6}(-4)/ 5.994. (..6) As ca be see from Table., the results for the atural cubc sples are early detcal to the lear terpolato, ad are smlar to that of the secod parabolc Lagraga terpolato. However, the most approprate comparso would be wth the cubc Lagraga terpolato Φ(4) as both appromatg fuctos are cubc polyomals. Here the results are qute dfferet llustratg the mportace of the costrats o the dervatves of the cubc sples. The Lagraga cubc terpolato utlzes tabular formato for 8 order to specfy the terpolatg polyomal. The sples rely o the more local 78

@ Polyomal Appromato formato volvg the fucto ad ts dervatves specfed the rage 5. Ths mmzes the large tabular varatos elsewhere the table that affect the Lagraga cubc polyomal ad make for a smoother fuctoal varato. The egatve aspect of the sple approach s that t requres a soluto throughout the table. If the umber of tabular etres s large ad the requred umber of terpolated values s small, the addtoal umercal effort maybe dffcult to ustfy. I the et secto we shall fd esthetcs ad effcecy playg a eve larger role choosg the approprate appromatg method. Table. A Comparso of Dfferet Types of Iterpolato Formulae " Y Φ (4) Φ (4) Φ (4) Φ(4) Y Ψ (4 ) R,,, 4, w ).. 8.988 4 6. 8. 5.7 7.467 5.994 5.4 6. 5 4 -.965 4 8 -. 5 -- -. d. Etrapolato ad Iterpolato Crtera So far we have obtaed several dfferet types of terpolato schemes, but sad lttle about choosg the degree of the polyomal to be used, or the codtos uder whch oe uses Lagrage terpolato or sples to obta the formato mssg from the table. The reaso for ths was alluded to the prevous paragraph - there s o correct aswer to the questo. Oe ca dodge the phlosophcal questo of the "correctess" of the terpolated umber by appealg to the foudatos of polyomal appromato - amely that to the etet that the fucto represeted by the tabular pots ca be represeted by a polyomal, the aswer s correct. But ths s deed a dodge. For f t were true that the tabular fucto was deed a polyomal, oe would smply use the terpolato scheme to determe the polyomal that ft the etre table ad use t. I scece, oe geerally does kow somethg about the ature of a tabular fucto. For eample, may such tables result from a complcated computato of such legth that t s ot practcal to repeat the calculato to obta addtoal tabular values. Oe ca usually guaratee that the results of such calculatos are at least cotuous dfferetable fuctos. Or f there are dscotutes, ther locato s kow ad ca be avoded. Ths may ot seem lke much kowledge, but t guaratees that oe ca locally appromate the table by a polyomal. The et ssue s what sort of polyomal should be used ad over what part of the tabular rage. I secto. we poted out that a polyomal ca have a very geeral form [see equato (..)]. 79

Numercal Methods ad Data Aalyss Whle we have chose our bass fuctos φ () to be for most of the dscusso, ths eed ot have bee the case. Iterpolato formulae of the type developed here for ca be developed for ay set of bass fuctos φ (). For eample, should the table ehbt epoetal growth wth the depedet varable, t mght be advsable to choose φ () e α [e α ] z. (..6) The smple trasformato of z e α allows all prevously geerated formulae to be mmedately carred over to the epoetal polyomals. The choce of α wll be made to sut the partcular table. I geeral, t s far better to use bass fuctos φ () that characterze the table tha to use some set of fuctos such as the coveet ad a larger degree for terpolato. Oe must always make the choce betwee fttg the tabular form ad usg the lowest degree polyomal possble. The choce of bass fuctos that have the proper form wll allow the use of a lower degree polyomal. Why s t so desrable to choose the lowest degree polyomal for terpolato? There s the obvous reaso that the lower the degree the faster the computato ad there are some cases where ths may be a overrdg cocer. However, plausblty of the result s usually the determg factor. Whe oe fts a polyomal to a fte set of pots, the value of the polyomal teds to oscllate betwee the pots of costrat. The hgher the degree of the polyomal, the larger s the ampltude ad frequecy of these oscllatos. These cosderatos become eve more mportat whe oe cosders the use of the terpolatve polyomal outsde the rage specfed by the tabular etres. We call such use etrapolato ad t should be doe wth oly the greatest care ad crcumspecto. It s a farly geeral characterstc of polyomals to vary qute rapdly outsde the tabular rage to whch they have bee costraed. The varato s usually characterzed by the largest epoet of the polyomal. Thus f oe s usg polyomals of the forth degree, he/she s lkely to fd the terpolatve polyomal varyg as 4 mmedately outsde the tabular rage. Ths s lkely to be uacceptable. Ideed, there are some who regard ay etrapolato beyod the tabular rage that vares more tha learly to be uustfed. There are, of course, eceptos to such a hard ad fast rule. Occasoally asymptotc propertes of the fucto that yeld the tabular etres are kow, the etrapolatve fuctos that mmc the asymptotc behavor maybe ustfable. There s oe form of etrapolato that reduces the stabltes assocated wth polyomals. It s a form of appromato that abados the classcal bass for polyomal appromato ad that s appromato by ratoal fuctos or more specfcally quotet polyomals. Let us ft such a fucto through the (k +) pots k. The we ca defe a quotet polyomal as a P() R,+,,+ k (). (..6) Q() b Ths fucto would appear to have (m++) free parameters, but we ca factor a from the umerator ad b from the deomator so that oly ther rato s a free parameter. Therefore there are oly (m++) free parameters so we must have k+ m++, (..64) fuctoal pots to determe them. However, the values of ad m must also be specfed separately. Normally the determato of the coeffcets of such a fucto s rather dffcult, but Stoer ad Bulrsch 8

have obtaed a recurrece formula for the value of the fucto tself, whch s @ Polyomal Appromato R +,,+ k () R,,+ k () R,+,,+ k () R +,,+ k () ( R + + () R + + () ),, k,,, k ( + k ) R +,,+ k () R +,,+ k (). (..65) R, f ( ) R,k, k < Ths recurrece relato produces a fucto where m f the umber of pots used s odd, but where m + should the umber of pots be eve. However, ts use elmates the eed for actually kowg the values of the coeffcets as the relatoshp gves the value of the appromated fucto tself. That s f () R, +,,+ k. (..66) Equato (..65) coceals most of the dffcultes of usg ratoal fuctos or quotet polyomals. Whle the great utlty of such appromatg fuctos are ther stablty for etrapolato, we shall demostrate ther use for terpolato so as to compare the results wth the other forms of terpolato we have vestgated. Sce the bulk of the other methods have four parameters avalable for the specfcato of the terpolatg polyomal (.e. they are cubc polyomals), we shall cosder a quotet polyomal wth four free parameters. Ths wll requre that we use four tabular etres whch we shall choose to bracket the pot 4 symmetrcally. Such a appromatg fucto would have the form R,,,4 () (a+b)/(α+β). (..67) However, the recursve form of equato (..65) meas that we wll ever determe the values of a, b, α, ad β. The subscrpt otato used equatos (..6) (..66) s desged to eplctly covey the recursve ature of the determato of the terpolatve fucto. Each addtoal subscrpt deotes a successve "geerato" the developmet of the fal result. Oe begs wth the tabular data ad the secod of equatos (..65). Takg the data from table. so that f( ) Y, we get for the secod geerato that represets the terpolatve value at 4 8

Numercal Methods ad Data Aalyss R, () R, () 8 R, () R, () + 8 + ( R () R () 4 8 ),, ( 4 8 ) R, () R, () R, () 4 8 + 6 R, () R, () + 4 + ( R () R () 4 4 8 ),,. (..68) ( 4 5 4 ) R, () R 4,4 () R, () 4 + + 6 R () R () +,4 4,4 ( ) R () R () 4 4 5 4,4, ( ) 4 5 R () 4 4 4,4 The thrd geerato wll cota oly two terms so that R R,,,,4 () R () R,,4 () + ( ( () + ( ( R ) R, () R ) R, () R R () R () 4,,4 ) R ) R () R,4,4,, () () R () R,,,, () () () (). (..69) Fally the last geerato wll have the sgle result. R,,4 () R,, () R,,,4 () R,,4 () +. (..7) ( R () R () ),,4,, ( 4 ) R,,4 () R,, () We ca summarze ths process eatly the form of a "dfferece" Table (smlar to Table. ad Table 4.) below. Note how the recurso process drves the successve 'geeratos' of R toward the fal result. Ths s a clear demostrato of the stablty of ths sort of scheme. It s ths type of stablty that makes the method desrable for etrapolato. I addto, such recursve procedures are very easy to program ad qute fast eecuto. The fal result s gve equato (..7), tabulated for comparso wth other methods Table., ad dsplayed Fgure.. Ths result s the smallest of the s results lsted dcatg that the 8

@ Polyomal Appromato rapd tabular varato of the mddle four pots has bee mmzed. However, t stll compares favorably wth the secod parabolc Lagraga terpolato. Whle there s ot a great deal of dfferetato betwee these methods for terpolato, there s for etrapolato. The use of quotet polyomals for etrapolato s vastly superor to the use of polyomals, but oe should always remember that oe s bascally after "freeluch" ad that more sophstcated s ot ecessarly better. Geerally, t s rsky to etrapolate ay fucto far beyod oe typcal tabular spacg. We have see that the degree of the polyomal that s used for terpolato should be as low as possble to avod urealstc rapd varato of the terpolatve fucto. Ths oto of provdg a geeral "smoothess" to the fucto was also mplct the choce of costrats for cubc sples. The costrats at the teror tabular pots guaratee cotuty up through the secod dervatve of the terpolatve fucto throughout the full tabular rage. The choce of Y" Y" that produces "atural" sples meas that the terpolatve fucto wll vary o faster tha learly ear the edpots. I geeral, whe oe has to make a assumpto cocerg uspecfed costats a terpolato scheme, oe chooses them so as to provde a slowly varyg fucto. The eteso of ths cocept to more complcated terpolato schemes s llustrated the followg hghly successful terpolato algorthm. Table.4 Parameters for Quotet Polyomal Iterpolato Y R, R, +, R, +, + R, +, +, + - 8 8 6.574 4 +6/ 5.47 5 4 4 5.4 +6/5 4 8 5 Oe of the most commoly chose polyomals to be used for terpolato s the parabola. It teds ot to vary rapdly ad yet s slghtly more sophstcated tha lear terpolato. It wll clearly requre three tabular pots to specfy the three arbtrary costats of the parabola. Oe s the cofroted wth the problem of whch of the two tervals betwee the tabular pots should the pot to be terpolated be placed. A scheme that removes ths problem whle far more mportatly provdg a getly varyg fucto 8

Numercal Methods ad Data Aalyss proceeds as follows: Use four pots symmetrcally placed about the pot to be terpolated. But stead of fttg a cubc to these four pots, ft two parabolas, oe utlzg the frst three pots ad oe utlzg the last three pots. At ths pot oe eercses a artstc udgmet. Oe may choose to use the parabola wth that ehbts the least curvature (.e. the smallest value of the quadratc coeffcet). However, oe may combe both polyomals to form a sgle quadratc polyomal where the cotrbuto of each s weghted versely by ts curvature. Specfcally, oe could wrte ths as kφ w () {a k+ [ kφ()] + a k [ k+ Φ()]}/(a k +a k+ ), (..7) where a k s are the verse of the coeffcet of the term of the two polyomals ad are gve by k+ Y( ) a k k ( ), (..7) ad are ust twce the verse of the curvature of that polyomal. The k Φ() are the Lagrage polyomals of secod degree ad are k+ kφ() Σ Y( )L (). (..7) k Sce each of the k Φ()s wll produce the value of Y( ) whe, t s clear that equato (..7) wll produce the values of Y( ) ad Y( ) at the pots ad adacet to the terpolatve pot. The fuctoal behavor betwee these two pots wll reflect the curvature of both polyomals gvg hgher weght to the flatter, or more slowly varyg polyomal. Ths scheme was developed the 96s by researchers at Harvard Uversty who eeded a fast ad relable terpolato scheme for the costructo of model stellar atmospheres. Whle the ustfcato of ths algorthm s strctly aesthetc, t has bee foud to fucto well a wde varety of stuatos. We may compare t to the other terpolato formulae by applyg t to the same data from tables. ad. that we have used throughout ths secto. I developg the parabolc Lagraga formulae secto., we obtaed the actual terpolatve polyomals equatos (..5) ad (..6). By dfferetatg these epressos twce, we obta the a k s requred by equato (..7) so that " a P (4) / 7. (..74) " a P (4) 5 / 4 Substtuto of these values to equato (..7) yelds a weghted Lagraga terpolated value of, Φ w {[P (4)/7] + [5P (4)/4]}/[(/7)+(5/4)] 6. (..75) We have evaluated equato (..75) by usg the ratoal fracto values for P (4) ad P (4) whch are detcal to the terpolatve values gve table.. The values for the relatve weghts gve equato (..74) show that the frst parabola wll oly cotrbute about 5% to the fal aswer do to ts rapd varato. The more getly chagg secod parabola cotrbutes the overwhelmg maorty of the fal result reflectg our aesthetc udgmet that slowly varyg fuctos are more plausble for terpolatg 84

@ Polyomal Appromato fuctos. The fact that the result s detcal to the result for lear terpolato s a umercal accdet. Ideed, had roud-off error ot bee a factor, t s lkely that the result for the cubc sples would have also bee eactly 6. However, ths cocdece pots up a commo truth: "more sophstcated s ot ecessarly better". Although slghtly more complcated tha quadratc Lagraga terpolato, ths scheme s rather more stable agast rapd varato ad s certaly more sophstcated tha lear terpolato. I my opo, ts oly real competto s the use of cubc sples ad the oly whe the etre rage of the table s to be used as curve fttg. Eve here there s o clear dstcto as to whch produces the more approprate terpolatve values, but a edge mght be gve to cubc sples o the bass of speed depedg o the table sze ad umber of requred terpolatve values. It s worth takg a last look at the results Table.. We used the accuracy mpled by the tables to provde a bass for the comparso of dfferet terpolatve methods. Ideed, some of the calculatos were carred out as ratoal fractos to elmate roud-off error as the possble source of the dfferece betwee methods. The plausble values rage from about 5. to 6.. However, based o the tabular data, there s o real reaso to prefer oe value over aother. The approprate choce should revolve aroud the etet that oe should epect a aswer of a partcular accuracy. Noe of the tabular data cota more tha two sgfcat fgures. There would have to be some compellg reaso to clude more the fal result. Gve the data spacg ad the tabular varablty, eve two sgfcat fgures are dffcult to ustfy. Wth that md, oe could argue persuasvely that lear terpolato s really all that s ustfed by ths problem. Ths s a mportat lesso to be leared for t les at the root of all umercal aalyss. There s o eed to use umercal methods that are vastly superor to the basc data of the problem.. Orthogoal Polyomals Before leavg ths chapter o polyomals, t s approprate that we dscuss a specal, but very mportat class of polyomals kow as the orthogoal polyomals. Orthogoal polyomals are defed terms of ther behavor wth respect to each other ad throughout some predetermed rage of the depedet varable. Therefore the orthogoalty of a specfc polyomal s ot a mportat oto. Ideed, by tself that statemet does ot make ay sese. The oto of orthogoalty mples the estece of somethg to whch the obect questo s orthogoal. I the case of polyomals, that somethg happes to be other polyomals. I secto. we dscussed the oto of orthogoalty for vectors ad foud that for a set of vectors to be orthogoal, o elemet of the set could be epressed term of the other members of the set. Ths wll also be true for orthogoal polyomals. I the case of vectors, f the set was complete t was sad to spa a vector space ad ay vector that space could be epressed as a lear combato of the orthogoal bass vectors. Sce the oto of orthogoalty seems to hge o two thgs beg perpedcular to each other, t seems reasoable to say that two fuctos f () ad f () are orthogoal f they are everywhere perpedcular to each other. If we mage taget vectors t () ad t () defed at every pot of each fucto, the f 85

Numercal Methods ad Data Aalyss () t (), (..) t oe could coclude from equato (..) that f () ad f () were mutually perpedcular at each value of. If oe cosders the rage of to represet a fte dmeso vector space wth each value of represetg a dmeso so that the vectors t () represeted bass vectors that space, the orthogoalty could be epressed as b t a ()t ()d Thus, t s ot ureasoable to geeralze orthogoalty of the fuctos themselves by b. (..) f ()f ()d,. (..) a Aga, by aalogy to the case of vectors ad lear trasformatos dscussed chapter we ca defe two fuctos as beg orthoormal f b w () f ()f () d δ. (..4) a Here we have cluded a addtoal fucto w() whch s called a weght fucto. Thus the proper statemet s that two fuctos are sad to be orthoormal the terval a b, relatve to a weght fucto w(), f they satsfy equato (..4). I ths secto we shall cosder the subset of fuctos kow as polyomals. It s clear from equato (..4) that orthoormal polyomals come sets defed by the weght fucto ad rage of. These parameters provde for a fte umber of such sets, but we wll dscuss oly a few of the more mportat oes. Whle we wll fd t relatvely easy to characterze the rage of the depedet varable by three dstct categores, the codtos for the weght fucto are far less strget. Ideed the oly costrat o w() s w() > a b. (..5) Whle oe ca fd orthogoal fuctos for o-postve weght fuctos, t turs out that they are ot uque ad therefore ot well defed. Smply lmtg the weght fucto to postve defte fuctos the terval a-b, stll allows for a fte umber of such weght fuctos ad hece a fte umber of sets of orthogoal polyomals. Let us beg our search for orthogoal polyomals by usg the orthogoalty codtos to see how such polyomals ca be geerated. For smplcty, let us cosder a fte terval from a to b. Now a orthogoal polyomal φ () wll be orthogoal to every member of the set of polyomals other tha tself. I addto, we wll assume (t ca be prove) that the polyomals wll form a complete set so that ay polyomal ca be geerated from a lear combato of the orthogoal polyomals of the same degree or less. Thus, f q () s a arbtrary polyomal of degree, we ca wrte Now let b w() φ ()q () d. (..6) a 86

@ Polyomal Appromato d U () () w() φ () U (). (..7) d The fucto U () s called the geeratg fucto of the polyomals φ () ad s tself a polyomal of () degree so that the th dervatve U s a th degree polyomal. Now tegrate equato (..7) by parts -tmes to get a b ( ) ( ) ' ( ) [ U ()q () U ()q () + + ( ) U ()q ( ] () ()q () d ) U. (..8) Sce q () s a arbtrary polyomal each term equato (..8) must hold separately so that ' ( ) U (a) U (a) U (a). (..9) ' ( ) U (b) U (b) U (b) Sce φ () s a polyomal of degree we may dfferetate t + tmes to get + d d U (). (..) d w() d Ths costtutes a dfferetal equato of order + subect to the boudary codtos gve by equato (..9). The remag codto requred to uquely specfy the soluto comes from the ormalzato costat requred to make the tegral of φ () uty. So at ths pot we ca leave U () ucerta by a scale factor. Let us ow tur to the soluto of equato (..) subect to the boudary codtos gve by equato (..9) for some specfc weght fuctos w(). a b a. The Legedre Polyomals Let us beg by restrctg the rage to - ad takg the smplest possble weght fucto, amely w(), (..) so that equato (..9) becomes + d [ U ()]. (..) d Sce U () s a polyomal of degree, a obvous soluto whch satsfes the boudary codtos s U () C ( -). (..) Therefore the polyomals that satsfy the orthogoalty codtos wll be gve by d ( ) φ () C. (..4) d If we apply the ormalzato crtero we get + + d ( ) φ ( ) d C d, (..5) d so that 87

Numercal Methods ad Data Aalyss C [!] -. (..6) We call the orthoormal polyomals wth that ormalzato costat ad satsfyg equato (..4) the Legedre polyomals ad deote them by P () [!] - d ( -) /d. (..7) Oe ca use equato (..7) to verfy that these polyomals wll satsfy the recurrece relato + P + () P () P () + + P (), (..8) P () The set of orthogoal polyomals that covers the fte terval from - to + ad whose members are orthogoal relatve to the weght fucto w() are clearly the smplest of the orthogoal polyomals. Oe mght be tempted to say that we have bee uduly restrctve to lmt ourselves to such a specfc terval, but such s ot the case. We may trasform equato (..5) to ay fte terval by meas of a lear trasformato of the form y() [(b-a)/] +(a+b)/, (..9) so that we obta a tegral b φ (y) φ (y) dy δ b a, (..) a that resembles equato (..4). Thus the Legedre polyomals form a orthoormal set that spas ay fte terval relatve to the ut weght fucto. b. The Laguerre Polyomals Whle we oted that the Legedre polyomals could be defed over ay fte terval sce the lear trasformato requred to reach such as terval dd't affect the polyomals, we had earler metoed that there are three dstct tervals that would have to be treated dfferetly. Here we move to the secod of these - the sem-fte terval where. Clearly the lmts of ths terval caot be reached from ay fte terval by a lear trasformato. A o-lear trasformato that would accomplsh that result would destroy the polyomc ature of ay polyomals obtaed the fte terval. I addto, we shall have to cosder a weght fucto that asymptotcally approaches zero as as ay polyomals wll dverge makg t mpossble to satsfy the ormalzato codto. Perhaps the smplest weght fucto that wll force a dvergg polyomal to zero as s e -α. Therefore our orthogoal polyomals wll take the form α d U () φ I () e, (..) d where the geeratg fucto wll satsfy the dfferetal equato 88

+ α @ Polyomal Appromato d d U () e d d, (..) ad be subect to the boudary codtos ' ( ) U () U () U () ' ( ) U ( ) U ( ) U ( ). (..) Whe subected to those boudary codtos, the geeral soluto to equato (..) wll be U () C e -α, (..4) so that the polyomals ca be obtaed from α α e d ( e ) φ (),! d (..5) If we set α, the the resultat polyomals are called the Laguerre polyomals ad whe ormalzed have the form e d ( e ) L,! d (,,6) ad wll satsfy the recurrece relato + L + () L () L () + + L () L(). (..7) These polyomals form a orthoormal set the sem-fte terval relatve to the weght fucto e -. c. The Hermte Polyomals Clearly the remag terval that caot be reached from ether a fte terval or semfte terval by meas of a lear trasformato s the full fte terval - +. Aga we wll eed a weght fucto that wll drve the polyomal to zero at both ed pots so that t must be symmetrc. Thus the weght fucto for the sem-fte terval wll ot do. Istead, we pck the smplest symmetrc epoetal e α, whch leads to polyomals of the form α d U () φ () e, (..8) d that satsfy the dfferetal equato + d d U () e α, (..9) d d subect to the boudary codtos 89

Numercal Methods ad Data Aalyss ' ( ) U ( ± ) U ( ± ) U ( ± ). (..) Ths has a geeral soluto satsfyg the boudary codtos that look lke U () C e -α, (..) whch whe ormalzed ad wth α, leads to the Hermte polyomals that satsfy H () d e ( ) e. (..) Table.5 The Frst Fve Members of the Commo Orthogoal Polyomals P () L () H () - ( -)/ (-4+ )/ ( -) (5 -)/ (6-8+9 - )/6 4( -) 4 (5 4 - +)/8 (4-96+7-6 + 4 )/4 4(4 4-6 +) Lke the other polyomals, the Hermte polyomals ca be obtaed from a recurrece relato. For the Hermte polyomals that relato s H + () H () H () H (). (..) H ().We have ow developed sets of orthoormal polyomals that spa the three fudametal rages of the real varable. May other polyomals ca be developed whch are orthogoal relatve to other weght fuctos, but these polyomals are the most mportat ad they appear frequetly all aspects of scece. d d. Addtoal Orthogoal Polyomals There are as may addtoal orthogoal polyomals as there are postve defte weght fuctos. Below we lst some of those that are cosdered to be classcal orthogoal polyomals as they tur up frequetly mathematcal physcs. A lttle specto of Table.6 shows that the Chebyschev polyomals are specal cases of the more geeral Gegebauer or Jacob polyomals. However, they tur up suffcetly frequetly that t s worth sayg more about them. They ca be derved from the geeratg fucto the same maer that the other orthogoal polyomals were, so we wll oly quote the results. The Chebyschev polyomals of the frst kd ca be obtaed from the reasoably smple trgoometrc formula T () cos[ cos - ()]. (..4) 9

@ Polyomal Appromato Table.6 Classcal Orthogoal Polyomals of the Fte Iterval NAME WEIGHT FUNCTION W(X) Legedre Gegebauer or Ultrasphercal λ ( ) Jacob or Hypergeometrc ( α β ) ( ) Chebyschev of the frst kd ( ) Chebyschev of the secod kd + ( ) However, practce they are usually obtaed from a recurrece formula smlar to those for the other polyomals. Specfcally T + () T () T () T (). (..5) T () The Chebyschev polyomals of the secod kd are represeted by the somewhat more complcated trgoometrc formula V () s[(+)cos - ()]/s[cos - ()], (..6) ad obey the same recurrece formula as Chebyschev polyomals of the frst kd so V + () V () V () V (). (..7) V () Oly the startg values are slghtly dfferet. Sce they may be obtaed from a more geeral class of polyomals, we should ot be surprsed f there are relatos betwee them. There are, ad they take the form T () V () V (). (..8) ( )V () T () T + () Sce the orthogoal polyomals form a complete set eablg oe to epress a arbtrary polyomal terms of a lear combato of the elemets of the set, they make ecellet bass fuctos for terpolato formulae. We shall see later chapters that they provde a bass for curve fttg that provdes great umercal stablty ad ease of soluto. I the et chapter, they wll eable us to geerate formulae to evaluate tegrals that yeld great precso for a mmum of effort. The utlty of these 9

Numercal Methods ad Data Aalyss fuctos s of cetral mportace to umercal aalyss. However, all of the polyomals that we have dscussed so far form orthogoal sets over a cotuous rage of. Before we leave the subect of orthogoalty, let us cosder a set of fuctos, whch form a complete orthogoal set wth respect to a dscrete set of pots the fte terval. e. The Orthogoalty of the Trgoometrc Fuctos At the begg of the chapter where we defed polyomals, we represeted the most geeral polyomal terms of bass fuctos φ (). Cosder for a momet the case where φ () s(π). (..9) Now tegrato by parts twce, recoverg the tal tegral but wth a sg chage, or perusal of ay good table of tegrals 4 wll covce oe that + s( kπ) s( π) d + cos(kπ) cos(π) d δ k. (..4) Thus ses ad coses form orthogoal sets of fuctos of the real varable the fte terval. Ths wll come as o surprse to the studet wth some famlarty wth Fourer trasforms ad we wll make much of t chapters to come. But what s less well kow s that N N s(kπ / N) s(π / N) N N cos(kπ / N) cos(π / N) δ k, < (k + ) < N, (..4) whch mples that these fuctos also form a orthogoal set o the fte terval for a dscrete set of pots. The proof of ths result ca be obtaed much the same way as the tegral, but t requres some kowledge of the fte dfferece calculus (see Hammg 5 page 44, 45). We shall see that t s ths dscrete orthogoalty that allows for the developmet of Fourer seres ad the umercal methods for the calculato of Power Spectra ad "Fast Fourer Trasforms". Thus the cocept of orthogoal fuctos ad polyomals wll play a role much of what follows ths book. 9

@ Polyomal Appromato Chapter Eercses. Fd the roots of the followg polyomal 5 5 4 + 6 56 +744 44 P(), a. by the Graffe Root-squarg method, b. ay teratve method, c. the compare the accuracy of the two methods.. Fd the roots of the followg polyomals: a. P() 4 7 + 7 + b. P() 4 5 + 4 5 + 4 c. P() 4 4 9 5 8 d. P() +.( +) +. +.. Commet of the accuracy of your soluto.. Fd Lagraga terpolato formulae for the cases where the bass fuctos are a. φ () e b. φ () s(π/h), where h s a costat terval spacg betwee the pots. 4. Use the results from problem to obta values for f() at.5,.9 ad. the followg table: f( )...4..8.. 5... 5.. 8. 8.. Compare wth ordary Lagraga terpolato for the same degree polyomals ad cubc sples. Commet o the result. 9

Numercal Methods ad Data Aalyss 5. Gve the followg table, appromate f() by f() Σ a s(). Determe the "best" value of for fttg the table. Dscuss your reasog for makg the choce you made. f( ). +.4546. -.784. -.97 4. +.4947 5. -.7 6. -.68 7. +.495 8. -.49 6. Fd the ormalzato costats for a. Hermte polyomals b. Laguerre polyomals c. Legedre polyomals that are defed the terval - +. 7. Use the rules for the mapulato of determats gve chapter (page 8) to show how the Vadermode determat takes the form gve by equato (..7) 8. I a maer smlar to problem 7, show how the Lagraga polyomals take the form gve by equato (..9). 9. Eplctly show how equato (..9) s obtaed from equatos (..), (..4), ad (..6).. Itegrate equato (..5) to obta the tr-dagoal equatos (..54). Show eplctly how the costrats of the dervatves of Y eter to the problem.. By obtag equato (..8) from equato (..7) show that oe ca obta the recurrece relatos for orthogoal polyomals from the defg dfferetal equato.. Fd the geeratg fucto for Gegebauer polyomals ad obta the recurrece relato for them.. Show that equato (..4) s deed correct. 94

Chapter Refereces ad Supplemetal Readg @ Polyomal Appromato. Press, W.H., Flaery, B.P., Teukolsky, S.A., ad Vetterlg, W.T., "Numercal Recpes the Art of Scetfc Computg" (986), Cambrdge Uversty Press Cambrdge, New York, New Rochelle, Melboure, Sydey.. Acto, Forma S., "Numercal Methods That Work", (97) Harper ad Row, New York.. Stoer, J. ad Bulrsch, R., "Itroducto to Numercal Aalyss" (98), Sprger-Verlag, New York,.. 4. Gradshtey, I.S. ad Ryzhk,I.M., "Table of Itegrals, Seres, ad Products : corrected ad elarged edto" (98), (ed. A. Jeffrey), Academc Press, New York, Lodo, Toroto, Sydey, Sa Fracsco, pp 9-4. 5. Hammg, R.W., "Numercal Methods for Scetsts ad Egeers" (96) McGraw-Hll Book Co., Ic., New York, Sa Fracsco, Toroto, Lodo. For a ecellet geeral dscusso of polyomals oe should read 6. Moursud, D.G., ad Durs, C.S., "Elemetary Theory ad Applcatos of Numercal Aalyss" (988) Dover Publcatos, Ic. New York, pp 8-4. A very complete dscusso of classcal orthogoal polyomals ca be foud 7. Batema, H., The Batema Mauscrpt Proect, "Hgher Trascedetal Fuctos" (954) Ed. A. Erde ' ly, Vol., McGraw-Hll Book Co., Ic. New York, Toroto, Lodo, pp 5-8. 95

Numercal Methods ad Data Aalyss 96

4 Numercal Evaluato of Dervatves ad Itegrals The mathematcs of the Greeks was suffcet to hadle the cocept of tme. Perhaps the clearest demostrato of ths s Zeo's Parado regardg the flght of arrows. Zeo reasoed that sce a arrow must cover half the dstace betwee the bow ad the target before travelg all the dstace ad half of that dstace (.e. a quarter of the whole) before that, etc., that the total umber of steps the arrow must cover was fte. Clearly the arrow could ot accomplsh that a fte amout of tme so that ts flght to the target was mpossble. Ths oto of a lmtg process of a ftesmal dstace beg crossed a ftesmal tme producg a costat velocty seems obvous to us ow, but t was a fudametal barrer to the developmet of Greek scece. The calculus developed the 7th cetury by Newto ad Lebtz has permtted, ot oly a proper hadlg of tme ad the lmtg process, but the mathematcal represetato of the world of pheomea whch scece seeks to descrbe. Whle the aalytc represetato of the calculus s essetal ths descrpto, ultmately we must umercally evaluate the aalytc epressos that we may develop order to compare them wth the real world. 97

Numercal Methods ad Data Aalyss Aga we cofrot a seres of subects about whch books have bee wrtte ad etre courses of study developed. We caot hope to provde a ehaustve survey of these areas of umercal aalyss, but oly develop the bass for the approach to each. The dfferetal ad tegral operators revewed chapter appear early all aspects of the scetfc lterature. They represet mathematcal processes or operatos to be carred out o cotuous fuctos ad therefore ca oly be appromated by a seres of dscrete umercal operatos. So, as wth ay umercal method, we must establsh crtera for whch the dscrete operatos wll accurately represet the cotuous operatos of dfferetato ad tegrato. As the case of terpolato, we shall fd the crtera the realm of polyomal appromato. 4. Numercal Dfferetato Compared wth other subects to be covered the study of umercal methods, lttle s usually taught about umercal dfferetato. Perhaps that s because the processes should be avoded wheever possble. The reaso for ths ca be see the ature of polyomals. As was poted out the last chapter o terpolato, hgh degree polyomals ted to oscllate betwee the pots of costrat. Sce the dervatve of a polyomal s tself a polyomal, t too wll oscllate betwee the pots of costrat, but perhaps ot qute so wldly. To mmze ths oscllato, oe must use low degree polyomals whch the ted to reduce the accuracy of the appromato. Aother way to see the dagers of umercal dfferetato s to cosder the ature of the operator tself. Remember that df () d f ( + ) f () Lm. (4..) Sce there are always computatoal errors assocated wth the calculato of f(), they wll ted to be preset as, whle smlar errors wll ot be preset the calculato of tself. Thus the rato wll ed up beg largely determed by the computatoal error f(). Therefore umercal dfferetato should oly be doe f o other method for the soluto of the problem ca be foud, ad the oly wth cosderable crcumspecto. a. Classcal Dfferece Formulae Wth these caveats clearly md, let us develop the formalsms for umercally dfferetatg a fucto f(). We have to appromate the cotuous operator wth a fte operator ad the fte dfferece operators descrbed chapter are the obvous choce. Specfcally, let us take the fte dfferece operator to be defed as t was equato (.5.). The we may appromate the dervatve of a fucto f() by df () f (). (4..) d The fte dfferece operators are lear so that repeated operatos wth the operator lead to f() [ - f()]. (4..) 98

4 @ Dervatves ad Itegrals Ths leads to the Fudametal Theorem of the Fte Dfferece Calculus whch s The th dfferece of a polyomal of degree s a costat ( a! h ), ad the (+) st dfferece s zero. Clearly the etet to whch equato (4..) s satsfed wll deped partly o the value of h. Also the ablty to repeat the fte dfferece operato wll deped o the amout of formato avalable. To fd a otrval th order fte dfferece wll requre that the fucto be appromated by a th degree polyomal whch has + learly depedet coeffcets. Thus oe wll have to have kowledge of the fucto for at least + pots. For eample, f oe were to calculate fte dffereces for the fucto at a fte set of pots, the oe could costruct a fte dfferece table of the form: Table 4. A Typcal Fte Dfferece Table for f() X I F(X I ) F(X) F(X) F(X) f()4 f()5 f()9 f() f()7 f() 4 f(4)6 f() f(4)9 f() 5 f(5)5 f(4) f(5) 6 f(6)6 Ths table cely demostrates the fudametal theorem of the fte dfferece calculus whle potg out a addtoal problem wth repeated dffereces. Whle we have chose f() to be a polyomal so that the dffereces are eact ad the fudametal theorem of the fte dfferece calculus s satsfed eactly, oe ca mage the stuato that would preval should f() oly appromately be a polyomal. The trucato error that arses from the appromato would be qute sgfcat for f( ) ad compouded for f( ). The propagato of the trucato error gets progressvely worse as oe proceeds to hgher ad hgher dffereces. The table llustrates a addtoal problem wth fte dffereces. Cosder the values of f( ). They are ot equal to the values of the dervatve at mpled by the defto of the forward dfferece operator at whch they are meat to apply. For eample f()7 ad wth h for ths table would suggest that f '()7, but smple dfferetato of the polyomal wll show that f '()6. Oe mght thk that ths could be corrected by averagg f () ad f (), or by re-defg the dfferece operator so that t dd't always refer backward. Such a operator s kow as the cetral dfferece operator whch s defed as δf() f(+½h) f(-½h). (4..4) 99

Numercal Methods ad Data Aalyss However, ths does ot remove the problem that the value of the th dfferece, beg derved from formato spag a large rage the depedet varable, may ot refer to the th dervatve at the pot specfed by the dfferece operator. I Chapter we metoed other fte dfferece operators, specfcally the shft operator E ad the detty operator I (see equato.5.). We may use these operators ad the relato betwee them gve by equato (.5.4), ad the bomal theorem to see that k k k k k k k k [ f ()] [E I] [f ()] ( ) E [f ()] ( ) f ( + ), (4..5) where ( k ) s the bomal coeffcet whch ca be wrtte as k k!. (4..6) (k )!! Oe ca use equato (4..5) to fd the kth dfferece for equally spaced data wthout costructg the etre dfferece table for the fucto. If a specfc value of f( ) s mssg from the table, ad oe assumes that the fucto ca be represeted by a polyomal of degree k-, the, sce k f ( ), equato (4..5) ca be solved for the mssg value of f( ). Whle equato (4..5) ca be used to fd the dffereces of ay equally spaced fucto f( ) ad hece s a estmate of the kth dervatve, the procedure s equvalet to fdg the value of a polyomal of degree -k at a specfc value of. Therefore, we may use ay terpolato formula to obta a epresso for the dervatve at some specfc pot by dfferetato of the approprate formula. If we do ths for Lagraga terpolato, we obta where ' Φ' () f ( )L (, (4..7) ) ( ) L. (4..8) '() ( k ) k Hgher order formulae ca be derved by successve dfferetato, but oe must always use umercal dfferetato wth great care. b. Rchardso Etrapolato for Dervatves We wll ow cosder a "clever trck" that eables the mprovemet of early all formulae that we have dscussed so far ths book ad a umber yet to come. It s kow as Rchardso etrapolato, but dffers from what s usually meat by etrapolato. I chapter we descrbed etrapolato terms of etedg some appromato formula beyod the rage of the data whch costraed that formula. Here we use t to descrbe a process that attempts to appromate the results of ay dfferece or dfferece based formula to lmt where the spacg h approaches zero. Sce h s usually a small umber, the eteso, or etrapolato, to zero does't seem so ureasoable. Ideed, t may ot seem very mportat, but remember the lmt of the accuracy o early all appromato formulae s set by the fluece of roud-off error the case where a appromatg terval becomes small. Ths wll be

4 @ Dervatves ad Itegrals partcularly true for problems of the umercal soluto of dfferetal equatos dscussed the et chapter. However, we ca develop ad use t here to obta epressos for dervatves that have greater accuracy ad are obtaed wth greater effcecy tha the classcal dfferece formulae. Let us cosder the specal case where a fucto f() ca be represeted by a Taylor seres so that f + kh, (4..9) the f ( + kh) f ( ) + khf '( () () (kh) f"( ) (kh) f ( ) (kh) f ( ) ) + + +. (4..) +!!! Now let us make use of the fact that h appears to a odd power eve terms of equato (4..). Thus f we subtract the a Taylor seres for -k from oe for +k, the eve terms wll vash leavg () + (+ ) (kh) f ( ) (kh) f ( ) f ( + kh) f ( kh) khf '( ) + + +. (4..)! ( + )! The fuctoal relatoshp o the left had sde of equato (4..) s cosdered to be some mathematcal fucto whose value s precsely kow, whle the rght had sde s the appromate relatoshp for that fucto. That relatoshp ow oly volves odd powers of h so that t coverges much faster tha the orgal Taylor seres. Now evaluate equato (4..) for k ad eplctly keepg ust the frst two terms o the rght had sde so that () 5 f ( + h) f ( h) hf '( ) + h f ( ) / 6 + + R(h ) ~. (4..) () 5 f ( + h) f ( h) 4hf '( ) + 6h f ( ) / 6 + + R(h ) We ow have two equatos from whch the term volvg the thrd dervatve may be elmated yeldg f( -h)-8f( -h)+8f( +h)-f( +h) hf'( )+R(h 5 )- ~ R(h 5 ), (4..) ad solvg for f'( ) we get. f'( ) [f( -h) 8f( -h) + 8f( +h) f( +h)]/(h) + O(h 4 ). (4..4) It s ot hard to show that the error term equato (4..) dvded by h s O(h 4 ). Thus we have a epresso for the dervatve of the fucto f() evaluated at some value of whch requres four values of the fucto ad s eact for cubc polyomals. Ths s ot too surprsg as we have four free parameters wth whch to ft a Taylor seres or alterately a cubc polyomal ad such polyomals wll be uque. What s surprsg s the rapd rate of covergece wth decreasg terval h. But what s eve more amazg s that ths method ca be geeralzed to ay appromato formulae that ca be wrtte as m f () Φ(, αh) + Ch + O(h ). (4..5) m >, α >, α so that α Φ(,h) Φ(, αh) m f () + O(h ). (4..6) α Ideed, t could be used to obta a eve hgher order appromato for the dervatve utlzg more tabular pots. We shall revst ths method whe we cosder the soluto to dfferetal equatos Chapter 5.

Numercal Methods ad Data Aalyss 4. Numercal Evaluato of Itegrals: Quadrature Whle the term quadrature s a old oe, t s the correct term to use for descrbg the umercal evaluato of tegrals. The term umercal tegrato should be reserved for descrbg the umercal soluto of dfferetal equatos (see chapter 5). There s a geue ecessty for the dstcto because the very ature of the two problems s qute dfferet. Numercally evaluatg a tegral s a rather commo ad usually stable task. Oe s bascally assemblg a sgle umber from a seres of depedet evaluatos of a fucto. Ulke umercal dfferetato, umercal quadrature teds to average out radom computatoal errors. Because of the heret stablty of umercal quadrature, studets are geerally taught oly the smplest of techques ad thereby fal to lear the more sophstcated, hghly effcet techques that ca be so mportat whe the tegrad of the tegral s etremely complcated or occasoally the result of a separate legthy study. Vrtually all umercal quadrature schemes are based o the oto of polyomal appromato. Specfcally, the quadrature scheme wll gve the eact value of the tegral f the tegrad s a polyomal of some degree. The scheme s the sad to have a degree of precso equal to. I geeral, sce a th degree polyomal has + learly depedet coeffcets, a quadrature scheme wll have to have + adustable parameters order to accurately represet the polyomal ad ts tegral. Occasoally, oe comes across a quadrature scheme that has a degree of precso that s greater tha the umber of adustable parameters. Such a scheme s sad to be hyper-effcet ad there are a umber of such schemes kow for multple tegrals. For sgle, or oe dmesoal, tegrals, there s oly oe whch we wll dscuss later. a. The Trapezod Rule The oto of evaluatg a tegral s bascally the oto of evaluatg a sum. After all the tegral sg s a stylzed S that stads for a cotuous "sum". The symbol Σ as troduced equato (.5.) stads for a dscrete or fte sum, whch, f the terval s take small eough, wll appromate the value for the tegral. Such s the motvato for the Trapezod rule whch ca be stated as b f ( + ) + f ( ) f () d. (4..) a The formula takes the form of the sum of a dscrete set of average values of the fucto each of whch s multpled by some sort of weght W. Here the weghts play the role of the adustable parameters of the quadrature formula ad the case of the trapezod rule the weghts are smply the tervals betwee fuctoal evaluatos. A graphcal represetato of ths ca be see below Fgure 4. The meag of the rule epressed by equato (4..) s that the tegral s appromated by a seres of trapezods whose upper boudares the terval are straght les. I each terval ths formula would have a degree of precso equal to (.e. equal to the umber of free parameters the terval mus oe). The other "adustable" parameter s the used obtag the average of f( ) the terval. If we dvde the terval a b equally the the 's have the partcularly smple form (b-a)/(-). (4..)

4 @ Dervatves ad Itegrals I Chapter, we showed that the polyomc form of the tegrad of a tegral was uaffected by a lear trasformato [see equatos (..9) ad (..)]. Therefore, we ca rewrte equato (4..) as b + (b a) (b a) f[(y+ )] + f[(y )] f () d f (y)dy W', (4..) a where the weghts for a equally spaced terval are W' /(-). (4..4) If we absorb the factor of (b-a)/ to the weghts we see that for both represetatos of the tegral [.e. equato (4..) ad equato (4..)] we get W b a. (4..5) Notce that the fucto f() plays absolutely o role determg the weghts so that oce they are determed; they ca be used for the quadrature of ay fucto. Sce ay quadrature formula that s eact for polyomals of some degree greater tha zero must be eact for f(), the sum of the weghts of ay quadrature scheme must be equal to the total terval for whch the formula holds. Fgure 4. shows a fucto whose tegral from a to b s beg evaluated by the trapezod rule. I each terval a straght le appromates the fucto. b. Smpso's Rule The trapezod rule has a degree of precso of as t fts straght les to the fucto the terval. It would seem that we should be able to do better tha ths by fttg a hgher order polyomal to the fucto. So stead of usg the fuctoal values at the edpots of the terval to represet the fucto by a straght le, let us try three equally spaced pots. That should allow us to ft a polyomal wth three adustable parameters (.e. a parabola) ad obta a quadrature formula wth a degree of precso

Numercal Methods ad Data Aalyss of. However, we shall see that ths quadrature formula actually has a degree of precso of makg t a hyper-effcet quadrature formula ad the oly oe kow for tegrals oe dmeso. I geeral, we ca costruct a quadrature formula from a terpolato formula by drect tegrato. I chapter we developed terpolato formulae that were eact for polyomals of a arbtrary degree. Oe of the more geeral forms of these terpolato formulae was the Lagrage terpolato formula gve by equato (..8). I that equato Φ() was a polyomal of degree ad was made up of a lear combato of the Lagrage polyomals L (). Sce we are terested usg three equally spaced pots, wll be. Also, we have see that ay fte terval s equvalet to ay other for the purposes of fttg polyomals, so let us take the terval to be h so that our formula wll take the form h () d f ( )W h f f ( ) L ()d. (4..6) Here we see that the quadrature weghts W are gve by h h ( ) W L ()d d. (4..7) ( ) Now the three equally spaced pots the terval h wll have, h, ad h. For equal tervals we ca use equato (..) to evaluate the Lagrage polyomals to get ( h)( h) ( h + h ) L () h h ( )( h) ( h) L() h h ( )( h) ( h) L () h h. (4..8) Therefore the weghts for Smpso's rule become W W W h h h (8h / h / + 4h L ()d h (8h / 8h / ) 4h L()d h (8h / 4h / ) h L ()d h ) h. (4..9) Actually we eed oly to have calculated two of the weghts sce we kow that the sum of the weghts had to be h. Now sce h s oly half the terval we ca wrte so that the appromato formula for Smpso's quadrature becomes h /, (4..) 4

4 @ Dervatves ad Itegrals () d f ( )W 6 [ f ( ) + 4f ( ) + f ( )] f. (4..) Now let us cofrm the asserto that Smpso's rule s hyper-effcet. We kow that the quadrature formula wll yeld eact aswers for quadratc polyomals, so cosder the evaluato of a quartc. We pck the etra power of atcpato of the result. Thus we ca wrte ( α + β 4 α )d 4 4 β + 5 5 4α 6 + α( ) 4 + 4β + β( ) 4 + R( ) 4 5 α( ) 5β( ) + + R( ). (4.. ) 4 4 Here R( ) s the error term for the quadrature formula. Completg the algebra equato (4..) we get R( ) β( ) 5 /. (4..) Clearly the error the tegral goes as the terval to the ffth power ad ot the fourth power. So the quadrature formula wll have o error for cubc terms the tegrad ad the formula s deed hypereffcet. Therefore Smpso's rule s a surprsgly good quadrature scheme havg a degree of precso of over the terval. Should oe wsh to spa a larger terval (or reduce the spacg for a gve terval), oe could wrte h f () d f ( )d [ f () 4f ( ) f ( ) 4f ( 4 ) 4f ( ) f ( )] + + + + + +. ( ) 6 (4..4) By breakg the tegral up to sub-tervals, the fucto eed oly be well appromated locally by a cubc. Ideed, the fucto eed ot eve be cotuous across the separate boudares of the subtervals. Ths form of Smpso's rule s sometmes called a rug Smpso's rule ad s qute easy to mplemet o a computer. The hyper-effcecy of ths quadrature scheme makes ths a good "all purpose" equal terval quadrature algorthm. c. Quadrature Schemes for Arbtrarly Spaced Fuctos As we saw above, t s possble to obta a quadrature formula from a terpolato formula ad mata the same degree of precso as the terpolato formula. Ths provdes the bass for obtag quadrature formula for fuctos that are specfed at arbtrarly spaced values of the depedet varable. For eample, smply evaluatg equato (4..6) for a arbtrary terval yelds b () d a a b f f ( ) L ()d, (4..5) whch meas that the weghts assocated wth the arbtrarly spaced pots are b a W L () d. (4..6) However, the aalytc tegrato of L () ca become tedous whe becomes large so we gve a alteratve strategy for obtag the weghts for such a quadrature scheme. Remember that the scheme s to 5

Numercal Methods ad Data Aalyss have a degree of precso of so that t must gve the eact aswers for ay polyomal of degree. But there ca oly be oe set of weghts, so we specfy the codtos that must be met for a set of polyomals for whch we kow the aswer - amely. Therefore we ca wrte a b + + b a d W,. (4..7) + The tegral o the left s easly evaluated to yeld the ceter term whch must be equal to the sum o the rght f the formula s to have the requred degree of precso. Equatos (4..7) represet + lear equatos the + weghts W. Sce we have already dscussed the soluto of lear equatos some detal chapter, we ca cosder the problem of fdg the weghts to be solved. Whle the spacg of the pots gve equatos (4..7) s completely arbtrary, we ca use these equatos to determe the weghts for Smpso's rule as a eample. Assume that we are to evaluate a tegral the terval h. The the equatos (4..7) for the weghts would be For [,h,h], the equatos specfcally take the form (h) h h W (h) 8h h W whch upo removal of the commo powers of h are These have the soluto h + (h) d W,. (4..8) + h W + W h W h W 8h W + W + W + W + 4W + h W + 4h W + W. (4..9). (4..) W [/, 4/, /]h. (4..) The weghts gve equato (4..) are detcal to those foud for Smpso's rule equato (4..9) whch lead to the appromato formula gve by equato (4..). The detals of fdg the weghts by ths method are suffcetly smple that t s geerally preferred over the method dscussed the prevous secto (secto 4.b). 6

4 @ Dervatves ad Itegrals There are stll other alteratves for determg the weghts. For eample, the tegral equato (4..6) s tself the tegral of a polyomal of degree ad as such ca be evaluated eactly by ay quadrature scheme wth that degree of precso. It eed ot have the spacg of the desred scheme at all. Ideed, the tegral could be evaluated at a suffcet level of accuracy by usg a rug Smpso's rule wth a suffcet total umber of pots. Or the weghts could be obtaed usg the hghly effcet Gaussa type quadrature schemes descrbed below. I ay evet, a quadrature scheme ca be talored to ft early ay problem by wrtg dow the equatos of codto that the weghts must satsfy order to have the desred degree of precso. There are, of course, some potetal ptfalls wth ths approach. If very hgh degrees of precso formulae are sought, the equatos (4..7) may become early sgular ad be qute dffcult to solve wth the accuracy requred for relable quadrature schemes. If such hgh degrees of precso formulae are really requred, the oe should cosder Gaussa quadrature schemes. d. Gaussa Quadrature Schemes We tur ow to a class of quadrature schemes frst suggested by that brllat 9th cetury mathematca Karl Fredrch Gauss. Gauss oted that oe could obta a much hgher degree of precso for a quadrature scheme desged for a fucto specfed at a gve umber of pots, f the locato of those pots were regarded as addtoal free parameters. So, f addto to the N weghts oe also had N locatos to specfy, oe could obta a formula wth a degree of precso of N- for a fucto specfed at oly N pots. However, they would have to be the proper N pots. That s, ther locato would o loger be arbtrary so that the fucto would have to be kow at a partcular set of values of the depedet varable. Such a formula would ot be cosdered a hyper-effcet formula sce the degree of precso does ot eceed the umber of adustable parameters. Oe has smply elarged the umber of such parameters avalable a gve problem. The questo the becomes how to locate the proper places for the evaluato of the fucto gve the fact that oe wshes to obta a quadrature formula wth ths hgh degree of precso. Oce more we may appeal to the oto of obtag a quadrature formula from a terpolato formula. I secto (.b) we developed Hermte terpolato whch had a degree of precso of N-. (Note: that dscusso the actual umberg f the pots bega wth zero so that N+ where s the lmt of the sums the dscusso.) Sce equato (..) has the requred degree of precso, we kow that ts tegral wll provde a quadrature formula of the approprate degree. Specfcally b b Φ ) d f ( ) h ()d + a a ( f '( ) H ()d. (4..) Now equato (4..) would resemble the desred quadrature formula f the secod sum o the rght had sde could be made to vash. Whle the weght fuctos H () themselves wll ot always be zero, we ca ask uder what codtos ther tegral wll be zero so that b H a a b () d. (4..) 7

Numercal Methods ad Data Aalyss Here the secret s to remember that those weght fuctos are polyomals [see equato (..)] of degree + (.e. N-) ad partcular H () ca be wrtte as where ()L () H (), (4..4) ( ) ( ) ( ). (4..5) Here the addtoal multplcatve lear polyomal u () that appears equato has bee cluded oe of the Lagrage polyomals L () to produce the + degree polyomal Π(). Therefore the codto for the weghts of f'( ) to vash [equato(4..)] becomes b a ()L ()d. (4..6) ( ) The product the deomator s smply a costat whch s ot zero so t may be elmated from the equato. The remag tegral looks remarkably lke the tegral for the defto of orthogoal polyomals [equato (..6)]. Ideed, sce L () s a polyomal of degree [or (N-)] ad Π() s a polyomal of degree + (also N), the codtos requred for equato (4..6) to hold wll be met f Π() s a member of the set of polyomals whch are orthogoal the terval a b. But we have ot completely specfed Π() for we have ot chose the values where the fucto f() ad hece Π() are to be evaluated. Now t s clear from the defto of Π() [equato (4..5)] that the values of are the roots of a polyomal of degree + (or N) that Π() represets. Thus, we ow kow how to choose the 's so that the weghts of f'() wll vash. Smply choose them to be the roots of the (+)th degree polyomal whch s a member o a orthogoal set o the terval a b. Ths wll sure that the secod sum equato (4..) wll always vash ad the codto becomes b Φ ) d a a b ( f ( ) h ()d. (4..7) Ths epresso s eact as log as Φ() s a polyomal of degree + (or N-) or less. Thus, Gaussa quadrature schemes have the form b () d a f f ( )W, (4..8) where the 's are the roots of the Nth degree orthogoal polyomal whch s orthogoal the terval a b, ad the weghts W ca be wrtte wth the ad of equato (..) as b b W h ()d [ ( )L' ()L ()] d. (4..9) a a 8

4 @ Dervatves ad Itegrals Now these weghts ca be evaluated aalytcally should oe have the determato, or they ca be evaluated from the equatos of codto [equato (4..7)] whch ay quadrature weghts must satsfy. Sce the etet of the fte terval ca always be trasformed to the terval + where the approprate orthoormal polyomals are the Legedre polyomals, ad the weghts are depedet of the fucto f(), they wll be specfed by the value of N aloe ad may be tabulated oce ad for all. Probably the most complete tables of the roots ad weghts for Gaussa quadrature ca be foud Abramowtz ad Stegu ad uless a partcularly uusual quadrature scheme s eeded these tables wll suffce. Before cotug wth our dscusso of Gaussa quadrature, t s perhaps worth cosderg a specfc eample of such a formula. Sce the Gaussa formulae make use of orthogoal polyomals, we should frst epress the tegral the terval over whch the polyomals form a orthogoal set. To that ed, let us eame a tegral wth a fte rage so that b b a + f ()d f{[(b a)y + (a + b)]/ }dy. (4..) a Here we have trasformed the tegral to the terval +. The approprate trasformato ca be obtaed by evaluatg a lear fucto at the respectve ed pots of the two tegrals. Ths wll specfy the slope ad tercept of the straght le terms of the lmts ad yelds y [ (a + b)]/(b a). (4..) dy [ /(b a)]d We have o complcatg weght fucto the tegrad so that the approprate polyomals are the Legedre polyomals. For smplcty, let us take. We gave the frst few Legedre polyomals Table.4 ad for we have P (y) (y -)/. (4..) The pots at whch the tegrad s to be evaluated are smply the roots of that polyomal whch we ca fe from the quadratc formula to be (y ) /. (4..) y ± Quadrature formulae of larger wll requre the roots of much larger degree polyomals whch have bee tabulated by Abramowtz ad Stegu. The weghts of the quadrature formula are yet to be determed, but havg already specfed where the fucto s to be evaluated, we may use equatos (4..7) to fd them. Alteratvely, for ths smple case we eed oly remember that the weghts sum to the terval so that W + W. (4..4) Sce the weghts must be symmetrc the terval, they must both be uty. Substtutg the values for y ad W to equato (4..8), we get b (b f ()d a) (b a) {f[( ) (a b) + (a + b)] + f[( ) + (a + b)]}. (4..5) a 9

Numercal Methods ad Data Aalyss Whle equato (4..5) cotas oly two terms, t has a degree of precso of three (-) or the same as the three term hyper-effcet Smpso's rule. Ths cely llustrates the effcecy of the Gaussa schemes. They rapdly pass the fed abscssa formulae ther degree of precso as [(-)/]. So far we have restrcted our dscusso of Gaussa quadrature to the fte terval. However, there s othg the etre dscusso that would affect geeral tegrals of the form β α I w()f () d. (4..6) Here w() s a weght fucto whch may ot be polyomc ad should ot be cofused wth the quadrature weghts W. Such tegrals ca be evaluated eactly as log as f() s a polyomal of degree N-. Oe smply uses a Gaussa scheme where the pots are chose so that the values of are the roots of the Nth degree polyomal that s orthogoal the terval α β relatve to the weght fucto w(). We have already studed such polyomals secto. so that we may use Gaussa schemes to evaluate tegrals the sem-fte terval [ + ] ad full fte terval [ + ] as well as the fte terval [ +] as log as the approprate weght fucto s used. Below s a table of the tervals ad weght fuctos that ca be used for some commo types of Gaussa quadrature. Table 4. Types of Polyomals for Gaussa Quadrature Iterval Weght Fucto w() Type of Polyomal - + (- ) -½ Chebyschev: st kd - + (- ) +½ Chebyschev: d kd + e - Laguerre - + e - Hermte It s worth otg from the etres Table 4. that there are cosderable opportutes for creatvty avalable for the evaluato of tegrals by a clever choce of the weght fucto. Remember that t s oly f() of the product w()f() makg up the tegrad that eed be well appromated by a polyomal order for the quadrature formula to yeld accurate aswers. Ideed the weght fucto for Gaussa- Chebyschev quadrature of the frst kd has sgulartes at the ed pots of the terval. Thus f oe's tegral has smlar sgulartes, t would be a good dea to use Gauss-Chebyschev quadrature stead of Gauss-Legedre quadrature for evaluatg the tegral. Proper choce of the weght fucto may smply be used to mprove the polyomc behavor of the remag part of the tegrad. Ths wll certaly mprove the accuracy of the soluto. I ay evet, the quadrature formulae ca always be wrtte to have the form β ()f () d α w f ( ), (4..7) W

4 @ Dervatves ad Itegrals where the weghts, whch may clude the weght fucto w() ca be foud from β α w w()h () d. (4..8) Here h () s the approprate orthogoal polyomal for the weght fucto ad terval. e. Romberg Quadrature ad Rchardso Etrapolato So far we have gve eplct formulae for the umercal evaluato of a defte tegral. I realty, we wsh the result of the applcato of such formulae to specfc problems. Romberg quadrature produces ths result wthout obtag the actual form for the quadrature formula. The basc approach s to use the geeral propertes of the equal-terval formulae such as the Trapezod rule ad Smpso's rule to geerate the results for formulae successvely appled wth smaller ad smaller step sze. The results ca be further mproved by meas of Rchardso's etrapolato to yeld results for formulae of greater accuracy [.e. hgher order O(h m )]. Sce the Romberg algorthm geerates these results recursvely, the applcato s etremely effcet, readly programmable, ad allows a o-gog estmate of the error. Let us defe a step sze that wll always yeld equal tervals throughout the terval a b as h (b-a)/. (4..9) The geeral Trapezod rule for a tegral over ths rage ca wrtte as b h F (b a) f ()d f (a) + f (b) + f (a + h ) a. (4..4) The Romberg recursve quadrature algorthm states that the results of applyg ths formula for successve values of (.e. smaller ad smaller step szes h ) ca be obtaed from F Q F (F h ( + Q ) f[b + ( )h (b a)[f (a) + f (b)]/ ) ]. (4..4) Each estmate of the tegral wll requre (-) evaluatos of the fucto ad should yeld a value for the tegral, but ca have a degree of precesso o greater tha (-). Sce a sequece of steps must be eecute to reach ths level, the effcecy of the method s poor compared to Gaussa quadrature. However the dfferece (F F-) does provde a cotuous estmate of the error the tegral. We ca sgfcatly mprove the effcecy of the scheme by usg Romberg etrapolato to mprove the ature of the quadrature formulae that the terato scheme s usg. Remember that successve values of h dffer by a factor of two. Ths s eactly the form that we used to develop the Rchardso formula for the dervatve of a fucto [equato (4..5)]. Thus we ca use the geeralzato of the Rchardso algorthm gve by equato (4..5) ad utlzg two successve values of F to "etrapolate" to the result

Numercal Methods ad Data Aalyss for a hgher order formula. Each value of tegral correspodg to the hgher order quadrature formula ca, tur, serve as the bass for a addtoal etrapolato. Ths procedure also ca be cast as a recurrece formula where k k k F F k + F. (4..4) k There s a trade off betwee the results geerated by equato (4..4) ad equato (4..4). Larger values of produce values for F k whch correspod to decreasg values of h (see table 4.). However, creasg values of k yeld values for F k whch correspod to quadrature formulae smaller error terms, but wth larger values of h. Thus t s ot obvous whch sequece, equato (4..4) or equato (4..4) wll yeld the better value for the tegral. I order to see how ths method works, cosder applyg t to the aalytc tegral 5 5 e + e d 9.4868. (4..4) 5 F Table 4. Sample Results for Romberg Quadrature F 74.766.8 9.649 9.487 9.487 4.4445 9.886 9.4856 9.486.5 9.564 9.487.46 9.484 4 9.7 Here t s clear that mprovg the order of the quadrature formula rapdly leads to a coverged soluto. The covergece of the o-etrapolated quadrature s ot mpressve cosderg the umber of evaluatos requred to reach, say, F 4. Table 4.4 gves the results of applyg some of the other quadrature methods we have developed to the tegral equato (4..4). We obta the results for the Trapezod rule by applyg equato (4..) to the tegral gve by equato (4..4). The results for Smpso's rule ad the two-pot Gaussa quadrature come from equatos (4..) ad (4..5) respectvely. I the last two colums of Table 4.4 we have gve the percetage error of the method ad the umber of evaluatos of the fucto requred for the determato of the tegral. Whle the Romberg etrapolated tegral s fve tmes more accurate that t earest compettor, t takes twce the umber of evaluatos. Ths stuato gets rapdly worse so that the Gaussa quadrature becomes the most effcet ad accurate scheme whe eceeds about fve. The trapezod rule ad Romberg F yeld detcal results as they are the same appromato. Smlarly Romberg F yelds the same results as Smpso's rule. Ths s to be epected as the Rchardso etrapolato of the Romberg quadrature equvalet to the Trapezod rule should lead to the et hgher order quadrature formula whch s Smpso's rule. F F 4 F

4 @ Dervatves ad Itegrals Table 4.4 Test Results for Varous Quadrature Formulae TYPE F(X) F(%) N[F(X)] Aalytc Result 9.4868. Trapezod Rule 74.7658 5.9 Smpso's Rule.86. -pot Gauss Quad. 7.454 7.6 Romberg Quadrature F 74.7658 5.9 Romberg Quadrature F 9.886.4 4 f. Multple Itegrals Most of the work o the umercal evaluato of multple tegrals has bee doe the mddle of ths cetury at the Uversty of Wscos by Presto C. Hammer ad hs studets. A reasoably complete summary of much of ths work ca be foud the book by Stroud. Ufortuately the work s ot wdely kow sce problems assocated wth multple tegrals occur frequetly the sceces partcularly the area of the modelg of physcal systems. From what we have already developed for quadrature schemes oe ca see some of the problems. For eample, should t take N pots to accurately represet a tegral oe dmeso, the t wll take N m pots to calculate a m-dmesoal tegral. Should the tegrad be dffcult to calculate, the computato volved evaluatg t at N m pots ca be prohbtve. Thus we shall cosder oly those quadrature formulae that are the most effcet - the Gaussa formulae. The frst problem umercally evaluatg multple tegrals s to decde what wll costtute a appromato crtero. Lke tegrals of oe dmeso, we shall appeal to polyomal appromato. That s, some sese, we shall look for schemes that are eact for polyomals of the multple varables that descrbe the multple dmesos. However, there are may dstct types of such polyomals so we shall choose a subset. Followg Stroud let us look for quadrature schemes that wll be eact for polyomals that ca be wrtte as smple products of polyomals of a sgle varable. Thus the appromatg polyomal wll be a product polyomal m-dmesos. Now we wll ot attempt to derve the geeral theory for multple Gaussa quadrature, but rather pck a specfc space. Let the space be m-dmesoal ad of the full fte terval. Ths allows us, for the momet, to avod the problem of boudares. Thus we ca represet our tegral by V + + + ( + + + m ) e f (,,, m )dd d m. (4..44) Now we have see that we lose o geeralty by assumg that our th order polyomal s a moomal of the form α so let us cotue wth ths assumpto that f(,, m ) has the form f (). (4..45) α

Numercal Methods ad Data Aalyss We ca the wrte equato (4..44) as + + + m m α V e d e d. (4..46) The rght had sde has the relatvely smple form due to the learty of the tegral operator. Now make a coordate trasformato to geeral sphercal coordates by meas of r cosθm cosθm cosθ cosθ θ θ θ θ r cos m cos m cos s, (4..47) θ θ m r cos m s m m r s θm whch has a Jacoba of the trasformato equal to m J( r,θ ) r m- cos m- (θ m- )cos m- (θ m- ) + α cos(θ). (4..48) Ths allows the epresso of the tegral to take the form m m ( ) ( ) α α m + r m +π / α+ V e r r dr (cosθ ) (cosθ ) (s θ ) dθ. (4..49) π / Cosder how we could represet a quadrature scheme for ay sgle tegral the rug product. For eample +π / N α α+ α θ ) (cosθ ) (s θ ) dθ B(cosθ ) π / (cos. (4..5) Here we have chose the quadrature pots for θ to be at θ ad we have let α Σα. (4..5) Now make oe last trasformato of the form y cosθ, (4..5) whch leads to + N + ( ) / α α y ) y dy By w(y )y dy, (m ) (. (4..5) The tegral o the rght had sde ca be evaluated eactly f we take the y 's to be the roots of a polyomal of degree (α+)/ whch s a member of a orthogoal set the terval +, relatve to the weght fucto w(y ) whch s ( ) / 4 ( ) / 4 w(y ) ( y ) ( + y. (4..54) ) By cosderg Table. we see that the approprate polyomals wll be members of the Jacob 4

4 @ Dervatves ad Itegrals polyomals for α β ( - )/4. The remag tegral over the radal coordate has the form + r α' e r dr, (4..55) whch ca be evaluated usg Gauss-Hermte quadrature. Thus we see that multple dmesoal quadratures ca be carred out wth a Gaussa degree of precso for product polyomals by cosderg each tegral separately ad usg the approprate Gaussa scheme for that dmeso. For eample, f oe desres to tegrate over the sold sphere, oe would choose Gauss-Hermte quadrature for the radal quadrature, Gauss-Legedre quadrature for the polar agle θ, ad Gauss-Chebyschev quadrature for the azmuthal agle φ. Such a scheme ca be used for tegratg over the surface of spheres or surfaces that ca be dstorted from a sphere by a polyomal the agular varables wth good accuracy. The use of Gaussa quadrature schemes ca save o the order of N m/ evaluatos of the fuctos whch s usually sgfcat. For mult-dmesoal tegrals, there are a umber of hyper-effcet quadrature formulae that are kow. However, they deped o the boudares of the tegrato ad are geerally of rather low order. Nevertheless such schemes should be cosdered whe the boudares are smple ad the fucto well behaved. Whe the boudares are ot smple, oe may have to resort to a modelg scheme such a Mote Carlo method. It s clear that the umber of pots requred to evaluate a tegral m-dmesos wll crease as N m. It does ot take may dmesos for ths to requre a eormous umber of pots ad hece, evaluatos of the tegrad. Thus for multple tegrals, effcecy may dctate aother approach. 4. Mote Carlo Itegrato Schemes ad Other Trcks The Mote Carlo approach to quadrature s a phlosophy as much as t s a algorthm. It s a applcato of a much more wdely used method due to Joh vo Neuma. The method was developed durg the Secod World War to facltate the soluto to some problems cocerg the desg of the atomc bomb. The basc phlosophy s to descrbe the problem as a sequece of causally related physcal pheomea. The by determg the probablty that each separate pheomeo ca occur, the ot probablty that all ca occur s a smple product. The procedure ca be fashoed sequetally so that eve probabltes that deped o pror evets ca be hadled. Oe ca coceptualze the etre process by followg a seres of radomly chose tal states each of whch tates a causal sequece of evets leadg to the desred fal state. The probablty dstrbuto of the fal state cotas the aswer to the problem. Whle the method derves t ame from the caso at Mote Carlo order to emphasze the probablstc ature of the method, t s most easly uderstood by eample. Oe of the smplest eamples of Mote Carlo modelg techques volves the umercal evaluato of tegrals. a. Mote Carlo Evaluato of Itegrals Let us cosder a oe dmesoal tegral defed over a fte terval. The graph of the tegrad mght look lke that Fgure 4.. Now the area uder the curve s related to the tegral of the fucto. Therefore we ca replace the problem of fdg the tegral of the fucto to that of fdg the area uder the curve. However, we must place some uts o the tegral ad we do that by fdg the relatve area 5

Numercal Methods ad Data Aalyss uder the curve. For eample, cosder the tegral a b f d (b a). (4..) ma f ma The graphcal represetato of ths tegral s ust the area of the rectagle bouded by y, a, b, ad y f ma. Now f we were to radomly select values of ad y, oe could ask f y f ( ). (4..) If we let rato of the umber of successful trals to the total umber of trals be R, the a b f ()d R(b a)f ma. (4..) Clearly the accuracy of the tegral wll deped o the accuracy of R ad ths wll mprove wth the umber N of trals. I geeral, the value of R wll approach ts actual value as N. Ths emphaszes the maor dfferece betwee Mote Carlo quadrature ad the other types of quadrature. I the case of the quadrature formulae that deped o a drect calculato of the tegral, the error of the result s determed by the etet to whch the tegrad ca be appromated by a polyomal (eglectg roud-off error). If oe s suffcetly determed he/she ca determe the magtude of the error term ad thereby place a absolute lmt o the magtude of the error. However, Mote Carlo schemes are ot based o polyomal appromato so such a absolute error estmate caot be made eve prcple. The best we ca hope for s that there s a certa probablty that the value of the tegral les wth ε of the correct aswer. Very ofte ths s suffcet, but t should always remembered that the certaty of the calculato rests o a statstcal bass ad that the appromato crtero s dfferet from that used most areas of umercal aalyss. If the calculato of f() s volved, the tme requred to evaluate the tegral may be very great deed. Ths s oe of the maor drawbacks to the use of Mote Carlo methods geeral. Aother lesser problem cocers the choce of the radom varables ad y. Ths ca become a problem whe very large umbers of radom umbers are requred. Most radom umber geerators are subect to perodctes ad other o-radom behavor after a certa umber of selectos have bee made. Ay o-radom behavor wll destroy the probablstc ature of the Mote Carlo scheme ad thereby lmt the accuracy of the aswer. Thus, oe may be deceved to belevg the aswer s better tha t s. Oe should use Mote Carlo methods wth great care. It should usually be the method of last choce. However, there are problems that ca be solved by Mote Carlo methods that defy soluto by ay other method. Ths moder method of modelg the tegral s remscet of a method used before the advet of moder computers. Oe smply graphed the tegrad o a pece of graph paper ad the cut out the area that represeted the tegral. By comparg the carefully measured weght of the cutout wth that of a kow area of graph paper, oe obtaed a crude estmate of the tegral. Whle we have dscussed Mote Carlo schemes for oe-dmesoal tegrals oly, the techque ca easly be geeralzed to multple dmesos. Here the accuracy s bascally govered by the umber of pots requred to sample the "volume" represeted by the tegrad ad lmts. Ths samplg ca geerally be doe more effcetly tha the N m pots requred by the drect multple dmeso quadrature schemes. Thus, the Mote-Carlo scheme s lkely to effcetly compete wth those schemes as the umber of dmesos creases. Ideed, should m >, ths s lkely to be the case. 6

4 @ Dervatves ad Itegrals Fgure 4. shows the varato of a partcularly complcated tegrad. Clearly t s ot a polyomal ad so could ot be evaluated easly usg stadard quadrature formulae. However, we may use Mote Carlo methods to determe the rato area uder the curve compared to the area of the rectagle. Oe should ot be left wth the mpresso that other quadrature formulae are wthout ther problems. We caot leave ths subect wthout descrbg some methods that ca be employed to mprove the accuracy of the umercal evaluato of tegrals. b. The Geeral Applcato of Quadrature Formulae to Itegrals Addtoal trcks that ca be employed to produce more accurate aswers volve the proper choce of the terval. Occasoally the tegrad wll dsplay pathologcal behavor at some pot the terval. It s geerally a good dea to break the terval at that pot ad represet the tegral by two (or more) separate tegrals each of whch may separately be well represeted by a polyomal. Ths s partcularly useful dealg wth tegrals o the sem-fte terval, whch have pathologcal tegrads the vcty of zero. Oe ca separate such a tegral to two parts so that + a + a f ()d f ()d + f ()d. (4..4) The frst of these ca be trasformed to the terval - + ad evaluated by meas of ay combato of the fte terval quadrature schemes show table 4.. The secod of these tegrals ca be trasformed back to the sem-fte terval by meas of the lear trasformato 7

Numercal Methods ad Data Aalyss so that + a y a, (4..5) y + y f ()d e [e f (y + a)dy. (4..6) + Gauss-Laguerre quadrature ca be used to determe the value of the secod tegral. By udcously choosg places to break a tegral that correspod to locatos where the tegrad s ot well appromated by a polyomal, oe ca sgfcatly crease the accuracy ad ease wth whch tegrals may be evaluated. Havg decded o the rage over whch to evaluate the tegral, oe has to pck the order of the quadrature formula to be used. Ulke the case for umercal dfferetato, the hgher the degree of precso of the quadrature formula, the better. However, there does come a pot where the roud-off error volved the computato of the tegrad eceeds the cremetal mprovemet from the creased degree of precso. Ths pot s usually dffcult to determe. However, f oe evaluates a tegral wth formulae of creasg degree of precso, the value of the tegral wll steadly chage, reach a plateau, ad the chage slowly reflectg the fluece of roud-off error. As a rule of thumb 8 to pot Gauss- Legedre quadrature s suffcet to evaluate ay tegral over a fte rage. If ths s ot the case, the the tegral s somewhat pathologcal ad other approaches should be cosdered. I some staces, oe may use very hgh order quadrature (roots ad weghts for Legedre polyomals ca be foud up to N ), but these staces are rare. There are may other quadrature formulae that have utlty specfc crcumstaces. However, should the quadrature preset specal problems, or requre hghly effcet evaluato, these formulae should be cosdered. 8

4 @ Dervatves ad Itegrals Chapter 4 Eercses. Numercally dfferetate the fucto f() e -, at the pots,.5,, 5,. Descrbe the umercal method you used ad why you chose t. Dscuss the accuracy by comparg your results wth the aalytc closed form dervatves.. Numercally evaluate f e - d. Carry out ths evaluato usg a. 5-pot Gaussa quadrature b. a 5-pot equal terval formula that you choose c. 5 pot trapezod rule d. aalytcally. Compare ad dscuss your results.. Repeat the aalyss of problem for the tegral + d. Commet o your results 4. What method would you use to evaluate + ( -4 + - ) Tah() d? Epla your choce. 5. Use the techques descrbed secto (4.e) to fd the volume of a sphere. Dscuss all the choces you make regardg the type of quadrature use ad the accuracy of the result. 9

Numercal Methods ad Data Aalyss Chapter 4 Refereces ad Supplemetal Readg. Abramowtz, M. ad Stegu, I.A., "Hadbook of Mathematcal Fuctos" Natoal Bureau of Stadards Appled Mathematcs Seres 55 (964) U.S. Govermet Prtg Offce, Washgto D.C.. Stroud, A.H., "Appromate Calculato of Multple Itegrals", (97), Pretce-Hall Ic. Eglewood Clffs. Because to the umercal stabltes ecoutered wth most approaches to umercal dfferetato, there s ot a great deal of accessble lterature beyod the troductory level that s avalable. For eample. Abramowtz, M. ad Stegu, I.A., "Hadbook of Mathematcal Fuctos" Natoal Bureau of Stadards Appled Mathematcs Seres 55 (964) U.S. Govermet Prtg Offce, Washgto D.C., p. 877, devote less tha a page to the subect quotg a varety of dfferece formulae. The stuato wth regard to quadrature s ot much better. Most of the results are techcal papers varous ourals related to computato. However, there are three books Eglsh o the subect: 4. Davs, P.J., ad Rabowtz,P., "Numercal Itegrato", Blasdell, 5. Krylov, V.I., "Appromate Calculato of Itegrals" (96) (tras. A.H.Stroud), The Macmlla Compay 6. Stroud, A.H., ad Secrest, D. "Gaussa Quadrature Formulas", (966), Pretce-Hall Ic., Eglewood Clffs. Ufortuately they are all out of prt ad are to be foud oly the better lbrares. A very good summary of varous quadrature schemes ca be foud 7. Abramowtz, M. ad Stegu, I.A., "Hadbook of Mathematcal Fuctos" Natoal Bureau of Stadards Appled Mathematcs Seres 55 (964) U.S. Govermet Prtg Offce, Washgto D.C., pp. 885-899. Ths s also probably the referece for the most complete set of Gaussa quadrature tables for the roots ad weghts wth the possble ecepto of the referece by Stroud ad Secrest (.e. ref 4). They also gve some hyper-effcet formulae for multple tegrals wth regular boudares. The book by Art Stroud o the evaluato of multple tegrals 6. Stroud, A.H., "Appromate Calculato of Multple Itegrals", (97), Pretce-Hall Ic., Eglewood Clffs. represets largely the preset state of work o multple tegrals, but t s also dffcult to fd.

5 Numercal Soluto of Dfferetal ad Itegral Equatos The aspect of the calculus of Newto ad Lebtz that allowed the mathematcal descrpto of the physcal world s the ablty to corporate dervatves ad tegrals to equatos that relate varous propertes of the world to oe aother. Thus, much of the theory that descrbes the world whch we lve s cotaed what are kow as dfferetal ad tegral equatos. Such equatos appear ot oly the physcal sceces, but bology, socology, ad all scetfc dscples that attempt to uderstad the world whch we lve. Iumerable books ad etre courses of study are devoted to the study of the soluto of such equatos ad most college maors scece ad egeerg requre at least oe such course of ther studets. These courses geerally cover the aalytc closed form soluto of such equatos. But may of the equatos that gover the physcal world have o soluto closed form. Therefore, to fd the aswer to questos about the world whch we lve, we must resort to solvg these equatos umercally. Aga, the lterature o ths subect s volumous, so we ca oly hope to provde a bref troducto to some of the basc methods wdely employed fdg these solutos. Also, the subect s by o meas closed so the studet should be o the lookout for ew techques that prove creasgly effcet ad accurate.

Numercal Methods ad Data Aalyss 5. The Numercal Itegrato of Dfferetal Equatos Whe we speak of a dfferetal equato, we smply mea ay equato where the depedet varable appears as well as oe or more of ts dervatves. The hghest dervatve that s preset determes the order of the dfferetal equato whle the hghest power of the depedet varable or ts dervatve appearg the equato sets ts degree. Theores whch employ dfferetal equatos usually wll ot be lmted to sgle equatos, but may clude sets of smultaeous equatos represetg the pheomea they descrbe. Thus, we must say somethg about the solutos of sets of such equatos. Ideed, chagg a hgh order dfferetal equato to a system of frst order dfferetal equatos s a stadard approach to fdg the soluto to such equatos. Bascally, oe smply replaces the hgher order terms wth ew varables ad cludes the equatos that defe the ew varables to form a set of frst order smultaeous dfferetal equatos that replace the orgal equato. Thus a thrd order dfferetal equato that had the form f '''() + αf"() + βf'() + γf() g(), (5..) could be replaced wth a system of frst order dfferetal equatos that looked lke y'() + αz'() + βf '() + γf () g() z'() y(). (5..) f '() z() Ths smplfcato meas that we ca lmt our dscusso to the soluto of sets of frst order dfferetal equatos wth o loss of geeralty. Oe remembers from begg calculus that the dervatve of a costat s zero. Ths meas that t s always possble to add a costat to the geeral soluto of a frst order dfferetal equato uless some addtoal costrat s mposed o the problem. These are geerally called the costats of tegrato. These costats wll be preset eve f the equatos are homogeeous ad ths respect dfferetal equatos dffer sgfcatly from fuctoal algebrac equatos. Thus, for a problem volvg dfferetal equatos to be fully specfed, the costats correspodg to the dervatve preset must be gve advace. The ature of the costats (.e. the fact that ther dervatves are zero) mples that there s some value of the depedet varable for whch the depedet varable has the value of the costat. Thus, costats of tegrato ot oly have a value, but they have a "place" where the soluto has that value. If all the costats of tegrato are specfed at the same place, they are called tal values ad the problem of fdg a soluto s called a tal value problem. I addto, to fd a umercal soluto, the rage of the depedet varable for whch the soluto s desred must also be specfed. Ths rage must cota the tal value of the depedet varable (.e. that value of the depedet varable correspodg to the locato where the costats of tegrato are specfed). O occaso, the costats of tegrato are specfed at dfferet locatos. Such problems are kow as boudary value problems ad, as we shall see, these requre a specal approach. So let us beg our dscusso of the umercal soluto of ordary dfferetal equatos by cosderg the soluto of frst order tal value dfferetal equatos. The geeral approach to fdg a soluto to a dfferetal equato (or a set of dfferetal equatos) s to beg the soluto at the value of the depedet varable for whch the soluto s equal to the tal values. Oe the proceeds a step by step maer to chage the depedet varable ad move across the requred rage. Most methods for dog ths rely o the local polyomal appromato of the

5 @ Dfferetal ad Itegral Equatos soluto ad all the stablty problems that were a cocer for terpolato wll be a cocer for the umercal soluto of dfferetal equatos. However, ulke terpolato, we are ot lmted our choce of the values of the depedet varable to where we ca evaluate the depedet varable ad ts dervatves. Thus, the spacg betwee soluto pots wll be a free parameter. We shall use ths varable to cotrol the process of fdg the soluto ad estmatg ths error. Sce the soluto s to be locally appromated by a polyomal, we wll have costraed the soluto ad the values of the coeffcets of the appromatg polyomal. Ths would seem to mply that before we ca take a ew step fdg the soluto, we must have pror formato about the soluto order to provde those costrats. Ths "chcke or egg" aspect to solvg dfferetal equatos would be removed f we could fd a method that oly depeded o the soluto at the prevous step. The we could start wth the tal value(s) ad geerate the soluto at as may addtoal values of the depedet varable as we eeded. Therefore let us beg by cosderg oe-step methods. a. Oe Step Methods of the Numercal Soluto of Dfferetal Equatos Probably the most coceptually smple method of umercally tegratg dfferetal equatos s Pcard's method. Cosder the frst order dfferetal equato y'() g(,y). (5..) Let us drectly tegrate ths over the small but fte rage h so that whch becomes y y + h dy g(, y)d, (5..4) + + h y () y g(, y)d, (5..5) Now to evaluate the tegral ad obta the soluto, oe must kow the aswer to evaluate g(,y). Ths ca be doe teratvely by turg eq (5..5) to a fed-pot terato formula so that + h (k) (k ) y ( + + h) y g[, y ()]d. (5..6) (k ) (k ) y () y ( + h) A more spred choce of the teratve value for y ( k-) () mght be y (k-) () ½[y +y (k-) ( +h)]. (5..7) However, a eve better approach would be to admt that the best polyomal ft to the soluto that ca be acheved for two pots s a straght le, whch ca be wrtte as y() y + a(- ) {[y (k-) ( +h)](- ) + [y ( )]( +h-)}/h. (5..8) Whle the rght had sde of equato (5..8) ca be used as the bass for a fed pot terato scheme, the terato process ca be completely avoded by takg advatage of the fuctoal form of g(,y). The lear

Numercal Methods ad Data Aalyss form of y ca be substtuted drectly to g(,y) to fd the best value of a. The equato that costras a s the smply + h ah g[,(a + y )] d. (5..9) Ths value of a may the be substtuted drectly to the ceter term of equato (5..8) whch tur s evaluated at +h. Eve should t be mpossble to evaluate the rght had sde of equato (5..9) closed form ay of the quadrature formulae of chapter 4 ca be used to drectly obta a value for a. However, oe should use a formula wth a degree of precso cosstet wth the lear appromato of y. To see how these varous forms of Pcard's method actually work, cosder the dfferetal equato y'() y, (5..) subect to the tal codtos y(). (5..) Drect tegrato yelds the closed form soluto y e /. (5..) The rapdly varyg ature of ths soluto wll provde a formdable test of ay tegrato scheme partcularly f the step sze s large. But ths s eactly what we wat f we are to test the relatve accuracy of dfferet methods. I geeral, we ca cast Pcard's method as z y () + zy(z) dz, (5..) where equatos (5..6) - (5..8) represet varous methods of specfyg the behavor of y(z) for purposes of evaluatg the tegrad. For purposes of demostrato, let us choose h whch we kow s ureasoably large. However, such a large choce wll serve to demostrate the relatve accuracy of our varous choces qute clearly. Further, let us obta the soluto at, ad. The ave choce of equato (5..6) yelds a terato formula of the form + h (k ) (k ) + h) + zy ( + h)dz + + [h( + h) / ]y ( + h) y (. (5..4) Ths may be terated drectly to yeld the results colum (a) of table 5., but the fed pot ca be foud drectly by smply solvg equato (5..4) for y ( ) ( +h) to get y ( ) ( +h) (-h -h /) -. (5..5) For the frst step whe, the lmtg value for the soluto s. However, as the soluto proceeds, the terato scheme clearly becomes ustable. 4

5 @ Dfferetal ad Itegral Equatos Table 5. Results for Pcard's Method (a) (b) (c) (d) y() y() y() y c ()...5.5.75.65.875.656 4.98.664 5.969.666. 5/ 7/4.6487 y() y() y() y c () 4..6666 7...5 4.5 8.5 5.65 4 8.75 6.4688 5 4.56 7.5 9. 7.5 7.89 Estmatg the approprate value of y() by averagg the values at the lmts of the tegral as dcated by equato (5..7) teds to stablze the procedure yeldg the terato formula + h (k) (k ) (k ) y ( + h) + z[y( ) + y ( + h)dz + [h( + h) / ][y( ) + y ( + h)]/, (5..6) the applcato of whch s cotaed colum (b) of Table 5.. The lmtg value of ths terato formula ca also be foud aalytcally to be + [h( +h/)y( )]/ y ( ) ( +h) (5..7) [ h( +h/)/], whch clearly demostrates the stablzg fluece of the averagg process for ths rapdly creasg soluto. 5

Numercal Methods ad Data Aalyss Fally, we ca vestgate the mpact of a lear appromato for y() as gve by equato (5..8). Let us assume that the soluto behaves learly as suggested by the ceter term of equato (5..8). Ths ca be substtuted drectly to the eplct form for the soluto gve by equato (5..) ad the value for the slope, a, obtaed as equato (5..9). Ths process yelds a y( )( +h/)/[-( h/)-(h /)], (5..8) whch wth the lear form for the soluto gves the soluto wthout terato. The results are lsted table 5. colum (c). It s temptg to thk that a combato of the rght had sde of equato (5..7) tegrated closed form equato (5..) would gve a more eact aswer tha that obtaed wth the help of equato (5..8), but such s ot the case. A terato formula developed such a maer ca be terated aalytcally as was doe wth equatos (5..5) ad (5..7) to yeld eactly the results colum (c) of table 5.. Thus the best oe ca hope for wth a lear Pcard's method s gve by equato (5..8) wth the slope, a, specfed by equato (5..9). However, there s aother approach to fdg oe-step methods. The dfferetal equato (5..) has a full famly of solutos depedg o the tal value (.e. the soluto at the begg of the step). That famly of solutos s restrcted by the ature of g(,y). The behavor of that famly the eghborhood of +h ca shed some lght o the ature of the soluto at +h. Ths s the fudametal bass for oe of the more successful ad wdely used oe-step methods kow as the Ruge-Kutta method. The Ruge- Kutta method s also oe of the few methods umercal aalyss that does ot rely drectly o polyomal appromato for, whle t s certaly correct for polyomals, the basc method assumes that the soluto ca be represeted by a Taylor seres. So let us beg our dscusso of Ruge-Kutta formulae by assumg that the soluto ca be represeted by a fte taylor seres of the form k (k) y + y + hy' + (h /!)y" + + (h / k!) y. (5..9) Now assume that the soluto ca also be represeted by a fucto of the form y + y + h{α g(,y )+α g[( +µ h),(y +b h)] +α g[( +µ h),(y +b h)]+ +α k g[( +µ k h),(y +b k h)]}. (5..) Ths rather covoluted epresso, whle appearg to deped oly o the value of y at the tal step (.e. y ) volves evaluatg the fucto g(,y) all about the soluto pot, y (see Fgure 5.). By settg equatos (5..9) ad (5..) equal to each other, we see that we ca wrte the soluto the from y + y + α t + α t + + α k t k, (5..) where the t s ca be epressed recursvely by t hg(, y ) t hg[( + µ h),(y + λ,t )] t hg[( + µ h),(y + λ,t + λ,t )] t hg[( + µ h),(y + λ t + λ t + + λ k k k, k, k,k t k )]. (5..) 6

5 @ Dfferetal ad Itegral Equatos Now we must determe k+ values of α, k values of µ ad k (k+)/ values of λ,. But we oly have k+ terms of the Taylor seres to act as costrats. Thus, the problem s hopelessly uder-determed. Thus determecy wll gve rse to etre famles of Ruge-Kutta formulae for ay order k. I addto, the algebra to elmate as may of the ukows as possble s qute formdable ad ot uque due to the udetermed ature of the problem. Thus we wll cotet ourselves wth dealg oly wth low order formulae whch demostrate the basc approach ad ature of the problem. Let us cosder the lowest order that provdes some sght to the geeral aspects of the Ruge-Kutta method. That s k. Wth k equatos (5..) ad (5..) become y + y + α t + αt t hg( y ). (5..) t hg[( + µ h),(y + λt )] Here we have dropped the subscrpt o λ as there wll oly be oe of them. However, there are stll four free parameters ad we really oly have three equatos of costrat. Fgure 5. show the soluto space for the dfferetal equato y' g(,y). Sce the tal value s dfferet for dfferet solutos, the space surroudg the soluto of choce ca be vewed as beg full of alterate solutos. The two dmesoal Taylor epaso of the Ruge- Kutta method eplores ths soluto space to obta a hgher order value for the specfc soluto ust oe step. 7

Numercal Methods ad Data Aalyss If we epad g(,y) about, y, a two dmesoal taylor seres, we ca wrte g(, y ) g(, y ) g[( + µ h),(y + λt )] g(, y ) + µ h + λt + µ y g(, y ) g(, y ) + λ t + µλt + + y y Makg use of the thrd of equatos (5..), we ca eplctly wrte t as g(, y ) g(, y ) t hg(, y ) + h µ + λg(, y ) y g(, y ) g(, y h µ + λ g (, y ) y Drect substtuto to the frst of equatos (5..) gves ) + µλg(, y h g( g(, y ) y, y ). ) (5..4). (5..5) g(, y ) g(, y ) y + y + h( α + α)g(, y ) + h µ + λg(, y ) y. (5..6) g(, y ) g(, y ) g(, y ) h α µ + λ g (, y ) + µλg(, y ) y y We ca also epad y' a two dmesoal taylor seres makg use of the orgal dfferetal equato (5..) to get y' g(, y) g(, y) g(, y) g(, y) g(, y) y" + y' + g(, y) y y y" y" g(, y) g(, y) g(, y) g(, y) + + +. (5..7) y''' y' g(, y) y y y g(, y) g(, y) g(, y) + g(, y) + g(, y) + g(, y) y y y Substtutg ths to the stadard form of the Taylor seres as gve by equato (5..9) yelds g(, y) g(, y) h g(, y) g(, y) y + y + hg(, y) + h + λg(, y) + + g (, y) y 6 y. g(, y) g(, y) g(, y) g(, y) + g(, y) + + g(, y) y y y (5..8) Now by comparg ths term by term wth the epaso show equato (5..6) we ca coclude that the free parameters α, α, µ, ad λ must be costraed by 8

5 @ Dfferetal ad Itegral Equatos ( α + α) αµ. (5..9) α λ As we suggested earler, the formula s uder-determed by oe costrat. However, we may use the costrat equatos as represeted by equato (5..9) to epress the free parameters terms of a sgle costat c. Thus the parameters are α c ad the appromato formula becomes α c µ λ c. (5..) g(, y) g(, y) h g(, y) g(, y) y + y + hg(, y) + h + λg(, y) + + g (, y) y 8c y. g(, y) + g(, y) y (5..) We ca match the frst two terms of the Taylor seres wth ay choce of c. The error term wll tha be of order O(h ) ad specfcally has the form h ''' g(, y ) " R + [ 4c]y y. (5..) 4c y Clearly the most effectve choce of c wll deped o the soluto so that there s o geeral "best" choce. However, a umber of authors recommed c ½ as a geeral purpose value. If we crease the umber of terms the seres, the uder-determato of the costats gets rapdly worse. More ad more parameters must be chose arbtrarly. Whe these formulae are gve, the arbtraress has ofte bee removed by fat. Thus oe may fd varous Ruge-Kutta formulae of the same order. For eample, a commo such fourth order formula s y + y + (t + t + t + t ) / 6 t hg(, y ) t hg[( + h),(y + t )]. (5..) t hg[( + h),(y + t )] t hg[( + h),(y + t )] Here the "best" choce for the uder-determed parameters has already bee made largely o the bass of eperece. If we apply these formulae to our test dfferetal equato (5..), we eed frst specfy whch Ruge-Kutta formula we pla to use. Let us try the secod order (.e. eact for quadratc polyomals) formula gve by equato (5..) wth the choce of costats gve by equato (5..9) whe c ½. The 9

Numercal Methods ad Data Aalyss formula the becomes y + y + t + t t hg(, y ). (5..4) t hg[( + h),(y + t )] So that we may readly compare to the frst order Pcard formula, we wll take h ad y(). The takg g(,y) from equato (5..) we get for the frst step that t h y ()()() The secod step yelds t y( t t y( h( + h)(y + h) y h y h( + h) y + h)(y + t () + ( ()()( ( + t ) ()( + )( + ) ) ) + ( )() + ( ) + ( )() ) ()( + )( + Table 5. )( ) 5 )(5) Sample Ruge-Kutta Solutos 9 4. (5..5). (5..6) Secod Order Soluto Fourth Order Soluto Step h h/ y c h t t t. [, 9/].. [/4, 45/64].5 ----------- -----------.65 ----------- -----------.65 y.5.67.64587.6558 δy.7 h'.85* Step t t t.5 [.886,.984].6458 5. [.89, 5.96].7 ----------- ----------- 5.469 ----------- -----------.7884 y 4.75 6.595 7.896 7.5 δy.845 h'.65 * Ths value assumes that δy.

5 @ Dfferetal ad Itegral Equatos The Ruge-Kutta formula teds to uder-estmate the soluto a systematc fasho. If we reduce the step sze to h ½ the agreemet s much better as the error term ths formula s of O(h ). The results for h ½ are gve table 5. alog wth the results for h. I addto we have tabulated the results for the fourth order formula gve by equato (5..) For our eample, the frst step would requre that equato (5..) take the form t t t t y( h y h( h( h( + + + + h) y ()()() h)(y h)(y h)(y + + + t ) ()( + () + [() + ( t ) ()( + t ) ()( + )[ + ( ) + ( )( + ) )[ + ( 5 8 ) + ( )( )( 8 5 8 )] )] 5 8 8 )]/ 6) 79 48. (5..7) The error term for ths formula s of O(h 5 ) so we would epect t to be superor to the secod order formula for h ½ ad deed t s. These results demostrate that usually t s preferable to crease the accuracy of a soluto by creasg the accuracy of the tegrato formula rather tha decreasg the step sze. The calculatos leadg to Table 5. were largely carred out usg fractoal arthmetc so as to elmate the roud-off error. The effects of roud-off error are usually such that they are more serous for a dmshed step sze tha for a tegrato formula yeldg sutably creased accuracy to match the decreased step sze. Ths smply accetuates the ecessty to mprove soluto accuracy by mprovg the appromato accuracy of the tegrato formula. The Ruge-Kutta type schemes eoy great popularty as ther applcato s qute straght forward ad they ted to be qute stable. Ther greatest appeal comes from the fact that they are oe-step methods. Oly the formato about the fucto at the prevous step s ecessary to predct the soluto at the et step. Thus they are etremely useful tatg a soluto startg wth the tal value at the boudary of the rage. The greatest drawback of the methods s ther relatve effcecy. For eample, the forth order scheme requres four evaluatos of the fucto at each step. We shall see that there are other methods that requre far fewer evaluatos of the fucto at each step ad yet have a hgher order. b. Error Estmate ad Step Sze Cotrol A umercal soluto to a dfferetal equato s of lttle use f there s o estmate of ts accuracy. However, as s clear from equato (5..), the formal estmate of the trucato error s ofte more dffcult tha fdg the soluto. Ufortuately, the trucato error for most problems volvg dfferetal equatos teds to mmc the soluto. That s, should the soluto be mootocally creasg, the the absolute trucato error wll also crease. Eve mootocally decreasg solutos wll ted to have trucato errors that keep the same sg ad accumulate as the soluto progresses. The commo effect of trucato errors o oscllatory solutos s to troduce a "phase shft" the soluto. Sce the effect of trucato error teds to be systematc, there must be some method for estmatg ts magtude.

Numercal Methods ad Data Aalyss Although the formal epresso of the trucato error [say equato (5..)] s usually rather formdable, such epressos always deped o the step sze. Thus we may use the step sze h tself to estmate the magtude of the error. We ca the use ths estmate ad a a pror value of the largest acceptable error to adust the step sze. Vrtually all geeral algorthms for the soluto of dfferetal equatos cota a secto for the estmate of the trucato error ad the subsequet adustmet of the step sze h so that predetermed toleraces ca be met. Ufortuately, these methods of error estmate wll rely o the varato of the step sze at each step. Ths wll geerally trple the amout of tme requred to effect the soluto. However, the crease tme spet makg a sgle step may be offset by beg able to use much larger steps resultg a over all savgs tme. The geeral accuracy caot be arbtrarly creased by decreasg the step sze. Whle ths wll reduce the trucato error, t wll crease the effects of roud-off error due to the creased amout of calculato requred to cover the same rage. Thus oe does ot wat to set the a pror error tolerace to low or the roud-off error may destroy the valdty of the soluto. Ideally, the, we would lke our soluto to proceed wth rather large step szes (.e. values of h) whe the soluto s slowly varyg ad automatcally decrease the step sze whe the soluto begs to chage rapdly. Wth ths md, let us see how we may cotrol the step sze from toleraces set o the trucato error. Gve ether the oe step methods dscussed above or the mult-step methods that follow, assume that we have determed the soluto y at some pot. We are about to take the et step the soluto to + by a amout h ad wsh to estmate the trucato error y +. Calculate ths value of the soluto two ways. Frst, arrvg at + by takg a sgle step h, the repeat the calculato takg two steps of (h/). Let us call the frst soluto y,+ ad the secod y,+. Now the eact soluto (eglectg earler accumulated error) at + could be wrtte each case as k+ y e y, + + αh + + +, (5..8) k y y + + α( e, h ) + + where k s the order of the appromato scheme. Now α ca be regarded as a costat throughout the terval h sce t s ust the coeffcet of the Taylor seres ft for the (k+)th term. Now let us defe δ as a measure of the error so that δ(y + ) y,+ y,+ αh k+ /(- k ). (5..9) Clearly, δ(y + ) ~ h k+, (5..4) so that the step sze h ca be adusted at each step order that the trucato error remas uform by h + h δ(y )/δ(y + ) k+. (5..4) Itally, oe must set the tolerace at some pre-assged level ε so that δy ε. (5..4) If we use ths procedure to vestgate the step szes used our test of the Ruge-Kutta method, we see that we certaly chose the step sze to be too large. We ca verfy ths wth the secod order soluto for we carred out the calculato for step szes of h ad h½. Followg the prescrpto of equato (5..9) ad (5..4) we have, that for the results specfed Table 5.,

5 @ Dfferetal ad Itegral Equatos δy y, y,.67.5.7 δy. (5..4) h h ()(./.7).85 δy Here we have tactly assumed a tal tolerace of δy.. Whle ths s arbtrary ad rather large for a tolerace o a soluto, t s llustratve ad cosstet wth the sprt of the soluto. We see that to mata the accuracy of the soluto wth. we should decrease the step sze slghtly for the tal step. The error at the ed of the frst step s.6 for h, whle t s oly about.4 for h ½. By comparg the umercal aswers wth the aalytc aswer, y c, we see that factor of two chage the step sze reduces the error by about a factor of four. Our stated tolerace of. requres oly a reducto the error of about % whch mples a reducto of about 6% the step sze or a ew step sze h '.84h. Ths s amazgly close to the recommeded chage, whch was determed wthout kowledge of the aalytc soluto. The amout of the step sze adustmet at the secod step s made to mata the accuracy that ests at the ed of the frst step. Thus, δy y, y, 6.595 4.75.845 δ h h y. (5..44) ()(.7 /.845).65 δy Normally these adustmets would be made cumulatvely order to mata the tal tolerace. However, the coveet values for the step szes were useful for the earler comparsos of tegrato methods. The rapd crease of the soluto after causes the Ruge-Kutta method to have a creasgly dffcult tme matag accuracy. Ths s abudatly clear the drastc reducto the step sze suggested at the ed of the secod step. At the ed of the frst step, the relatve errors where 9% ad % for the h ad h½ step sze solutos respectvely. At the ed of the secod step those errors, resultg from comparso wth the aalytc soluto, had umped to 55% ad % respectvely (see table 5.). Whle a factor of two-chage the step sze stll produces about a factor of four chage the soluto, to arrve at a relatve error of 9%, we wll eed more lke a factor of 6 chage the soluto. Ths would suggest a chage the step sze of a about a factor of three, but the recommeded chage s more lke a factor of 6. Ths dfferece ca be uderstood by otcg that equato (5..4) attempts to mata the absolute error less tha δy. For our problem ths s about. at the ed of step oe. To keep the error wth those toleraces, the accuracy at step two would have to be wth about.5% of the correct aswer. To get there from 55% meas a reducto the error of a factor of 6, whch correspods to a reducto the step sze of a factor of about 8, s close to that gve by the estmate. Thus we see that the equato (5..4) s desged to mata a absolute accuracy the soluto by adustg the step sze. Should oe wsh to adust the step sze so as to mata a relatve or percetage accuracy, the oe could adust the step sze accordg to h + h {[δ(y )y + ]/[δ(y + )y ]} k+. (5..45) Whle these procedures vary the step sze so as to mata costat trucato error, a sgfcat prce the amout of computg must be pad at each step. However, the amout of etra effort eed ot be used oly to estmate the error ad thereby cotrol t. Oe ca solve equatos (5..8) (eglectg terms of order greater tha k) to provde a mproved estmate of y +. Specfcally

Numercal Methods ad Data Aalyss y e y,+ + δ(y + )/( k -). (5..46) However, sce oe caot smultaeously clude ths mprovemet drectly the error estmate, t s advsable that t be regarded as a "safety factor" ad proceeds wth the error estmate as f the mprovemet had ot bee made. Whle ths may seem uduly coservatve, the umercal soluto of dfferetal equatos coservatsm s a vrtue. c. Mult-Step ad Predctor-Corrector Methods The hgh order oe step methods acheve ther accuracy by eplorg the soluto space the eghborhood of the specfc soluto. I prcple, we could use pror formato about the soluto to costra our etrapolato to the et step. Sce ths formato s the drect result of pror calculato, far greater levels of effcecy ca be acheved tha by methods such as Ruge-Kutta that eplore the soluto space the vcty of the requred soluto. By usg the soluto at pots we could, prcple, ft a (-) degree polyomal to the soluto at those pots ad use t to obta the soluto at the (+)st pot. Such methods are called mult-step methods. However, oe should remember the caveats at the ed of chapter where we poted out that polyomal etrapolato s etremely ustable. Thus such a procedure by tself wll geerally ot provde a sutable method for the soluto of dfferetal equatos. But whe combed wth algorthms that compesate for the stablty such schemes ca provde very stable soluto algorthms. Algorthms of ths type are called predctor-corrector methods ad there are umerous forms of them. So rather tha attempt to cover them all, we shall say a few thgs about the geeral theory of such schemes ad gve some eamples. A predctor-corrector algorthm, as the ame mples, cossts of bascally two parts. The predctor etrapolates the soluto over some fte rage h based o the formato at pror pots ad s heretly ustable. The corrector allows for ths local stablty ad makes a correcto to the soluto at the ed of the terval also based o pror formato as well as the etrapolated soluto. Coceptually, the oto of a predctor s qute smple. I ts smplest form, such a scheme s the oe-step predctor where y + y + hy'. (5..47) By usg the value of the dervatve at the scheme wll systematcally uder estmate the proper value requred for etrapolato of ay mootocally creasg soluto (see fgure 5.). The error wll buld up cumulatvely ad hece t s ustable. A better strategy would be to use the value of the dervatve mdway betwee the two soluto pots, or alteratvely to use the formato from the pror two pots to predct y +. Thus a two pot predctor could take the form y + y - +hy'. (5..48) Although ths s a two-pot scheme, the etrapolatg polyomal s stll a straght le. We could have used the value of y drectly to ft a parabola through the two pots, but we dd't due to the stabltes to be assocated wth a hgher degree polyomal etrapolato. Ths delberate reecto of the some of the formatoal costrats favor of creased stablty s what makes predctor-corrector schemes o-trval ad effectve. I the geeral case, we have great freedom to use the formato we have regardg y ad y'. If we were to clude all the avalable formato, a geeral predctor would have the form 4

5 @ Dfferetal ad Itegral Equatos y + Σ a y + h Σ b y' + R, (5..49) where the a s ad b s are chose by mposg the approprate costrats at the pots ad R s a error term. Whe we have decded o the form of the predctor, we must mplemet some sort of corrector scheme to reduce the trucato error troduced by the predctor. As wth the predctor, let us take a smple case of a corrector as a eample. Havg produced a soluto at + we ca calculate the value of the dervatve y' + at +. Ths represets ew formato ad ca be used to modfy the results of the predcto. For eample, we could wrte a corrector as y (k) (k-) + y + ½h[y' + + y' ]. (5..5) Therefore, f we were to wrte a geeral epresso for a corrector based o the avalable formato we would get Fgure 5. shows the stablty of a smple predctor scheme that systematcally uderestmates the soluto leadg to a cumulatve buld up of trucato error. (k) + α y + h β y' + hβ+ y y'. (5..5) Equatos (5..5) ad (5..5) both are wrtte the form of terato formulae, but t s ot at all clear that (k+ ) + 5

Numercal Methods ad Data Aalyss the fed-pot for these formulae s ay better represetato of the soluto tha sgle terato. So order to mmze the computatoal demads of the method, correctors are geerally appled oly oce. Let us ow cosder certa specfc types of predctor corrector schemes that have bee foud to be successful. Hammg gves a umber of popular predctor-corrector schemes, the best kow of whch s the Adams-Bashforth-Moulto Predctor-Corrector. Predctor schemes of the Adams-Bashforth type emphasze the formato cotaed pror values of the dervatve as opposed to the fucto tself. Ths s presumably because the dervatve s usually a more slowly varyg fucto tha the soluto ad so ca be more accurately etrapolated. Ths phlosophy s carred over to the Adams-Moulto Corrector. A classcal fourth-order formula of ths type s () ' ' ' ' 5 y + + + + y h(55y 59y 7y 9y ) / 4 O(h ). (5..5) ' ' ' 5 y + y + h(9y + + 9y 5y ) / 4 + O(h ) Legthy study of predctor-corrector schemes has evolved some specal forms such as ths oe ' ' ' ' z + (y + y ) / + h(9y 7y + 9y 5y ) / 75 u + z + 77(z c ) / 75. (5..5) c + (y + y ) / + h(5u' + + 9y' + 4y' + 9y' ) / 7 6 y + + + c + 4(z + c + ) / 75 O(h ) where the etrapolato formula has bee epressed terms of some recursve parameters u ad c. The dervatve of these termedate parameters are obtaed by usg the orgal dfferetal equato so that u' g(,u). (5..54) By good chace, ths formula [equato (5..5)] has a error term that vares as O(h 6 ) ad so s a ffth-order formula. Fally a classcal predctor-corrector scheme whch combes Adams-Bashforth ad Mle predctors ad s qute stable s parametrcally (.e. Hammg p6) ' ' ' ' z (y + y ) + h(9y 99y + 69y 7y ) / 48 u c y + + + + z c + (y + 6(z + y + 9(z ) + c + h(7u' c + ) /7 + ) /7 + 5y' + y' 6 + O(h ) + y' ) / 48. (5..55) Press et al are of the opo that predctor-corrector schemes have see ther day ad are made obsolete by the Bulrsch-Stoer method whch they dscuss at some legth. They qute properly pot out that the predctor-corrector schemes are somewhat fleble whe t comes to varyg the step sze. The step sze ca be reduced by terpolatg the ecessary mssg formato from earler steps ad t ca be epaded tegral multples by skppg earler pots ad takg the requred formato from eve earler the soluto. However, the Bulrsch-Stoer method, as descrbed by Press et. al. utlzes a predctor scheme wth some specal propertes. It may be parameterzed as 6

z z z y y( ) z k+ () z + hz' k (z z' g(z, ) + hz' + z k 5 @ Dfferetal ad Itegral Equatos k,,,,. (5..56) 5 + hz' ) + O(h ) It s a odd characterstc of the thrd of equatos (5..56) that the error term oly cotas eve powers of the step sze. Thus, we may use the same trck that was used equato (5..46) of utlzg the formato geerated estmatg the error term to mprove the appromato order. But sce oly eve powers of h appear the error term, ths sgle step wll ga us two powers of h resultg a predctor of order seve. y h {4y () () (+h) y / [+(/)(h)]}/ + O(h 7 ). (5..57) Ths yelds a predctor that requres somethg o the order of ½ evaluatos of the fucto per step compared to four for a Ruge-Kutta formula of feror order. Now we come to the aspect of the Bulrsch-Stoer method that begs to dfferetate t from classcal predctor-correctors. A predctor that operates over some fte terval ca use a successvely creasg umber of steps order to make ts predcto. Presumably the predcto wll get better ad better as the step sze decreases so that the umber of steps to make the oe predcto creases. Of course practcal aspects of the problem such as roudoff error ad fte computg resources prevet us from usg arbtrarly small step szes, but we ca appromate what would happe a deal world wthout roud-off error ad utlzg ulmted computers. Smply cosder the predcto at the ed of the fte terval H where H αh. (5..58) Thus y α (+H) ca be take to be a fucto of the step sze h so that, y α (+H) y(+αh) f(h). (5..59) Now we ca phrase our problem to estmate the value of that fucto the lmt Lm f(h) Y (+H). (5..6) h α We ca accomplsh ths by carryg out the calculato for successvely smaller ad smaller values of h ad, o the bass of these values, etrapolatg the result to h. I spte of the admotos rased chapter regardg etrapolato, the rage here s small. But to produce a truly powerful umercal tegrato algorthm, Bulrsch ad Stoer carry out the etrapolato usg ratoal fuctos the maer descrbed secto. [equato (..65)]. The superorty of ratoal fuctos to polyomals represetg most aalytc fuctos meas that the step sze ca be qute large deed ad the covetoal meag of the 'order' of the appromato s rrelevat descrbg the accuracy of the method. 7

Numercal Methods ad Data Aalyss I ay case, remember that accuracy ad order are ot syoymous! Should the soluto be descrbed by a slowly varyg fucto ad the umercal tegrato scheme operate by fttg hgh order polyomals to pror formato for the purposes of etrapolato, the hgh-order formula ca gve very accurate results. Ths smply says that the tegrato scheme ca be ustable eve for well behaved solutos. Press et. al. 4 suggest that all oe eeds to solve ordary dfferetal equatos s ether a Ruge- Kutta or Bulrsch-Stoer method ad t would seem that for most problems that may well be the case. However, there are a large umber of commercal dfferetal equato solvg algorthms ad the maorty of them utlze predctor-corrector schemes. These schemes are geerally very fast ad the more sophstcated oes carry out very volved error checkg algorthms. They are geerally qute stable ad ca volve a very hgh order whe requred. I ay evet, the user should kow how they work ad be wary of the results. It s far too easy to smply take the results of such programs at face value wthout ever questog the accuracy of the results. Certaly oe should always ask the questo "Are these results reasoable?" at the ed of a umercal tegrato. If oe s geuely skeptcal, t s ot a bad dea to take the fal value of the calculato as a tal value ad tegrate back over the rage. Should oe recover the orgal tal value wth the acceptable toleraces, oe ca be reasoably cofdet that the results are accurate. If ot, the dfferece betwee the begg tal value ad what s calculated by the reverse tegrato over the rage ca be used to place lmts o the accuracy of the tal tegrato. d. Systems of Dfferetal Equatos ad Boudary Value Problems All the methods we have developed for the soluto of sgle frst order dfferetal equatos may be appled to the case where we have a coupled system of dfferetal equatos. We saw earler that such systems arose wheever we dealt wth ordary dfferetal equatos of order greater tha oe. However, there are may scetfc problems whch are trscally descrbed by coupled systems of dfferetal equatos ad so we should say somethg about ther soluto. The smplest way to see the applcablty of the sgle equato algorthms to a system of dfferetal equatos s to wrte a system lke y' g(, y, y, y ) y' g (, y, y, y ) y' g (, y, y, y ), (5..6) as a vector where each elemet represets oe of the depedet varables or ukows of the system. The the system becomes y' g(, y), (5..6) whch looks ust lke equato (5..) so that everythg applcable to that equato wll apply to the system of equatos. Of course some care must be take wth the termology. For eample, equato (5..4) would have to be uderstood as stadg for a etre system of equatos volvg far more complcated tegrals, but prcple, the deas carry over. Some care must also be eteded to the error aalyss that the error term s also a vector R (). I geeral, oe should worry about the magtude of the error vector, but 8

5 @ Dfferetal ad Itegral Equatos practce, t s usually the largest elemet that s take as characterzg the accuracy of the soluto. To geerate a umercal tegrato method for a specfc algorthm, oe smply apples t to each of the equatos that make up the system. By way of a specfc eample, let's cosder a forth order Ruge- Kutta algorthm as gve by equato (5..) ad apply t to a system of two equatos. We get y y t t t t u u, +,+ y y hg [( hg [( hg [( hg ( u hg [( hg ( hg [( u hg [(,, + (t + (u, y + + + h),(y, y + +,, + t, y + u, h),(y h),(y, y h),(y h),(y + h),(y ),, + t ),,, + u + + + + + t t t + t ),(y,, t + t ),(y t + u t t ) / 6 ),(y ),(y, + u ),(y ),(y, ) / 6,,,, + u + + )] + + )] u u u u )] )] )] )]. (5..6) We ca geeralze equato (5..6) to a arbtrary system of equatos by wrtg t vector form as y + A(y ). (5..64) The vector A(y ) cossts of elemets whch are fuctos of depedet varables y, ad, but whch all have the same geeral form varyg oly wth g (, y ). Sce a th order dfferetal equato ca always be reduced to a system of frst order dfferetal equatos, a epresso of the form of equato (5..6) could be used to solve a secod order dfferetal equato. The estece of coupled systems of dfferetal equatos admts the terestg possblty that the costats of tegrato requred to uquely specfy a soluto are ot all gve at the same locato. Thus we do ot have a full complmet of y, 's wth whch to beg the tegrato. Such problems are called boudary value problems. A comprehesve dscusso of boudary value problems s well beyod the scope of ths book, but we wll eame the smpler problem of lear two pot boudary value problems. Ths subclass of boudary value problems s qute commo scece ad etremely well studed. It cossts of a system of lear dfferetal equatos (.e. dfferetal equatos of the frst degree oly) where part of the tegrato costats are specfed at oe locato ad the remader are specfed at some other value of the depedet varable. These pots are kow as the boudares of the problem ad we seek a soluto to the problem wth these boudares. Clearly the soluto ca be eteded beyod the boudares as the soluto at the boudares ca serve as tal values for a stadard umercal tegrato. The geeral approach to such problems s to take advatage of the learty of the equatos, whch guaratees that ay soluto to the system ca be epressed as a lear combato of a set of bass 9

Numercal Methods ad Data Aalyss solutos. A set of bass solutos s smply a set of solutos, whch are learly depedet. Let us cosder a set of m lear frst order dfferetal equatos where k values of the depedet varables are specfed at ad (m-k) values correspodg to the remag depedet varables are specfed at. We could solve (m-k) tal value problems startg at ad specfyg (m-k) depedet, sets of mssg tal values so that the tal value problems are uquely determed. Let us deote the mssg set of tal values at by () ( ) whch we kow ca be determed from tal sets of learly depedet tral tal values y y (t) ( ) by () (t) y ( ) Ay ( ), (5..65) The colums of y (t) ( ) are ust the dvdual vectors (t) ( ). Clearly the matr A wll have to be y () y dagoal to always produce ( ). Sce the tral tal values are arbtrary, we wll choose the elemets of the (m-k) sets to be so that the mssg tal values wll be () y ( ) δ, (5..66) y ( ) A A. (5..67) Itegratg across the terval wth these tal values wll yeld (m-k) soluto y ( ) at the other boudary. Sce the equatos are lear each tral soluto wll be related to the kow boudary values y (t) ( ) by y ( ) (t) ( ) A[ y ( )], (5..68) so that for the complete set of tral solutos we may wrte () y ( (t) ) Ay (t) ( ), (5..69) where by aalogy to equato (5..65), the colum vectors of y (t) ( ) are y (t) ( ). We may solve these equatos for the ukow trasformato matr A so that the mssg tal values are () y ( () ) A y - y ( ). (5..7) If oe employs a oe step method such as Ruge-Kutta, t s possble to collapse ths etre operato to the pot where oe ca represet the complete boudary codtos at oe boudary terms of the values at the other boudary y a system of lear algebrac equatos such as y( ) By( ). 5..7) The matr B wll deped oly o the detals of the tegrato scheme ad the fuctoal form of the equatos themselves, ot o the boudary values. Therefore t may be calculated for ay set of boudary values ad used repeatedly for problems dfferg oly the values at the boudary (see Day ad Colls 5 ). To demostrate methods of soluto for systems of dfferetal equatos or boudary value 4

5 @ Dfferetal ad Itegral Equatos problems, we shall eed more tha the frst order equato (5..) that we used for earler eamples. However, that equato was qute llustratve as t had a rapdly creasg soluto that emphaszed the shortcomgs of the varous umercal methods. Thus we shall keep the soluto, but chage the equato. Smply dfferetate equato (5..) so that Y" (+ )e (+ )y. (5..7) Let us keep the same tal codto gve by equato (5..) ad add a codto of the dervatve at so that y(). (5..7) y'() e 5.4656 Ths sures that the closed form soluto s the same as equato (5..) so that we wll be able to compare the results of solvg ths problem wth earler methods. We should ot epect the soluto to be as accurate for we have made the problem more dffcult by creasg the order of the dfferetal equato addto to separatg the locato of the costats of tegrato. Ths s o loger a tal value problem sce the soluto value s gve at, whle the other costrat o the dervatve s specfed at. Ths s typcal of the classcal two-pot boudary value problem. We may also use ths eample to dcate the method for solvg hgher order dfferetal equatos gve at the start of ths chapter by equatos (5..) ad (5..). Wth those equatos md, let us replace equato (5..7) by system of frst order equatos y' () y (), (5..74) y' () ( + )y() whch we ca wrte vector form as y' A()y, (5..75) where A ( ). (5..76) ( + ) The compoets of the soluto vector y are ust the soluto we seek (.e.) ad ts dervatve. However, the form of equato (5..75) emphaszes ts lear form ad were t a scalar equato, we should kow how to proceed. For purposes of llustrato, let us apply the fourth order Ruge-Kutta scheme gve by equato (5..6). Here we ca take specfc advatage of the lear ature of our problem ad the fact that the depedece o the depedet varable factors out of the rght had sde. To llustrate the utlty of ths fact, let g(,y) [f()]y, (5..77) equato (5..6). The we ca wrte the fourth order Ruge-Kutta parameters as 4

Numercal Methods ad Data Aalyss 4 + + + + + + + + + + 4 4 4 )y f f f h f f h f f h (hf ) t (y hf t )y f f h f h (hf ) t (y hf t )y f f h (hf y hf (y hf ) t (y hf t y hf t. (5..78) where + + h) f ( f h) f ( f ) f ( f, (5..79) so that the formula becomes 4 y ) f f f 4 h ) f f f (f h ) f f f f (f 6 h ) f 4f (f 6 h ) t t t (t y y + + + + + + + + + + + + +. (5..8) Here we see that the learty of the dfferetal equato allows the soluto at step to be factored out of the formula so that the soluto at step appears eplctly the formula. Ideed, equato (5..8) represets a power seres h for the soluto at step (+) terms of the soluto at step. Sce we have bee careful about the order whch the fuctos f multpled each other, we may apply equato (5..8) drectly to equato (5..75) ad obta a smlar formula for systems of lear frst order dfferetal equatos that has the form 4 y 4 h ) ( h ) 4 ( 6 h ) 4 ( 6 h y + + + + + + + + + + A A A A A A A A A A A A A A A. (5..8) Here the meag of A s the same as f that the subscrpt dcates the value of the depedet varable for whch the matr s to be evaluated. If we take h, the matrces for our specfc problem become 4 A A A. (5..8) Keepg md that the order of matr multplcato s mportat, the products appearg the secod order term are

5 @ Dfferetal ad Itegral Equatos 4 6 A A A A A. (5..8) The two products appearg the thrd order term ca be easly geerated from equatos (5..8) ad (5..8) ad are. (5..84) 8 9 A A A A Fally the sgle matr of the frst order term ca be obta by successve multplcato usg equatos(5..8) ad (5..84) yeldg 8 9 A A A. (5..85) Lke equato (5..8), we ca regard equato (5..8) as a seres soluto h that yelds a system of lear equatos for the soluto at step + terms of the soluto at step. It s worth otg that the coeffcets of the varous terms of order h k are smlar to those developed for equal terval quadrature formulae chapter 4. For eample the lead term beg the ut matr geerates the coeffcets of the trapezod rule whle the h(+, +4, +)/6 coeffcets of the secod term are the famlar progresso characterstc of Smpso's rule. The hgher order terms the formula are less recogzable sce they deped o the parameters chose the uder-determed Ruge-Kutta formula. If we defe a matr P(h k ) so that k k y ) (h y P P +, (5..86) the seres ature of equato (5..8) ca be eplctly represeted terms of the varous values of k P. For our problem they are:

Numercal Methods ad Data Aalyss P P 6 7 P 6 7 P 49 65 4 4. (5..87) P 49 5 4 The boudary value problem ow s reduced to solvg the lear system of equatos specfed by equato (5..86) where the kow values at the respectve boudares are specfed. Usg the values gve equato (5..7) the lear equatos for the mssg boudary values become k P k y () P k y () + P k y () + P (5.4656) (5.4656). (5..88) The frst of these yelds the mssg soluto value at [.e. y ()]. Wth that value the remag value ca be obtaed from the secod equato. The results of these solutos cludg addtoal terms of order h k are gve table 5.. We have take h to be uty, whch s ureasoably large, but t serves to demostrate the relatve accuracy of cludg hgher order terms ad smplfes the arthmetc. The results for the mssg values y () ad y () (.e. the ceter two rows) coverge slowly, ad ot uformly, toward ther aalytc values gve the colum labeled k. Had we chose the step sze h to be smaller so that a umber of steps were requred to cross the terval, the each step would have produced a matr k P ad the soluto at each step would have bee related to the soluto at the followg step by equato (5..86). Repeated applcato of that equato would yeld the soluto at oe boudary terms of the soluto at the other so that 44

y 5 @ Dfferetal ad Itegral Equatos k k k k ( P P P P )y Q y. (5..89) Table 5. Solutos of a Sample Boudary Value Problem for Varous Orders of Appromato \ k 4 y ()...... y () 5.47.6..45.69. y (). 4.6.5..5.788 y () 5.47 5.47 5.47 5.47 5.47 e Thus oe arrves at a smlar set of lear equatos to those mpled by equato (5..86) ad eplctly gve equato (5..88) relatg the soluto at oe boudary terms of the soluto at the other boudary. These ca be solved for the mssg boudary values the same maer as our eample. Clearly the decrease the step sze wll mprove the accuracy as dramatcally as creasg the order k of the appromato formula. Ideed the step sze ca be varable at each step allowg for the use of the error correctg procedures descrbed secto 5.b. Table 5.4 Solutos of a Sample Boudary Value Problem \ k 4 y ()...... y ()...... y ().....78.78 y ()..8.8 4.8 4.8 5.47 Ay set of boudary values could have bee used wth equatos (5..8) to yeld the soluto elsewhere. Thus, we could treat our sample problem as a tal value problem for comparso. If we take the aalytc values for y () ad y () ad solve the resultg lear equatos, we get the results gve Table 5.4. Here the fal soluto s more accurate ad ehbts a covergece sequece more lke we would epect from Ruge-Kutta. Namely, the soluto systematcally les below the rapdly creasg aalytc soluto. For the boudary value problem, the reverse was true ad the fal result less accurate. Ths s ot a ucommo result for two-pot boudary value problems sce the error of the appromato scheme s drectly reflected the determato of the mssg boudary values. I a tal value problem, there s assumed to be o error the tal values. 45

Numercal Methods ad Data Aalyss Ths smple eample s ot meat to provde a deftve dscusso of eve the restrcted subset of lear two-pot boudary value problems, but smply to dcate a way to proceed wth ther soluto. Ayoe wshg to pursue the subect of two-pot boudary value problems further should beg wth the veerable tet by Fo 6. e. Partal Dfferetal Equatos The subect of partal dfferetal equatos has a lterature at least as large as that for ordary dfferetal equatos. It s beyod the scope of ths book to provde a dscusso of partal dfferetal equatos eve at the level chose for ordary dfferetal equatos. Ideed, may troductory books o umercal aalyss do ot treat them at all. Thus we wll oly sketch a geeral approach to problems volvg such equatos. Partal dfferetal equatos form the bass for so may problems scece, that to lmt the choce of eamples. Most of the fudametal laws of physcal scece are wrtte terms of partal dfferetal equatos. Thus oe fds them preset computer modelg from the hydrodyamc calculatos eeded for arplae desg, weather forecastg, ad the flow of fluds the huma body to the dyamcal teractos of the elemets that make up a model ecoomy. A partal dervatve smply refers to the rate of chage of a fucto of may varables, wth respect to ust oe of those varables. I terms of the famlar lmtg process for defg dfferetals we would wrte F( +,,, ) F(,,,, ) F(,,,, ) Lm. (5..9) Partal dfferetal equatos usually relate dervatves of some fucto wth respect to oe varable to dervatves of the same fucto wth respect to aother. The oto of order ad degree are the same as wth ordary dfferetal equatos. Although a partal dfferetal equato may be epressed multple dmesos, the smallest umber for llustrato s two, oe of whch may be tme. May of these equatos, whch descrbe so may aspects of the physcal world, have the form z(, y) z(, y) z(, y) z z a(, y) + b(, y) + c(, y) F, y,z,. y y y (5..9) ad as such ca be classfed to three dstct groups by the dscrmate so that [b (, y) a(, y)c(, y)] < Ellptc [b (, y) a(, y)c(, y)] Parabolc [b (, y) a(, y)c(, y)] > Hyperbolc. (5..9) Should the equato of terest fall to oe of these three categores, oe should search for soluto 46

5 @ Dfferetal ad Itegral Equatos algorthms desged to be effectve for that class. Some methods that wll be effectve at solvg equatos of oe class wll fal mserably for aother. Whle there are may dfferet techques for dealg wth partal dfferetal equatos, the most wde-spread method s to replace the dfferetal operator by a fte dfferece operator thereby turg the dfferetal equato to a fte dfferece equato at least two varables. Just as a umercal tegrato scheme fds the soluto to a dfferetal equato at dscrete pots alog the real le, so a two dmesoal tegrato scheme wll specfy the soluto at a set of dscrete pots, y. These pots ca be vewed as the tersectos o a grd. Thus the soluto the -y space s represeted by the soluto o a fte grd. The locato of the grd pots wll be specfed by the fte dfferece operators for the two coordates. Ulke problems volvg ordary dfferetal equatos, the tal values for partal dfferetal equatos are ot smply costats. Specfyg the partal dervatve of a fucto at some partcular value of oe of the depedet varables stll allows t to be a fucto of the remag depedet varables of the problem. Thus the fuctoal behavor of the soluto s ofte specfed at some boudary ad the soluto proceeds from there. Usually the fte dfferece scheme wll take advatage of ay symmetry that may result for the choce of the boudary. For eample, as was poted out secto. there are thrtee orthogoal coordate systems whch Laplace's equato s separable. Should the boudares of a problem match oe of those coordate systems, the the fte dfferece scheme would be totally separable the depedet varables greatly smplfyg the umercal soluto. I geeral, oe pcks a coordate system that wll match the local boudares ad that wll determe the geometry of the grd. The soluto ca the proceed from the tal values at a partcular boudary ad move across the grd utl the etre space has bee covered. Of course the soluto should be depedet of the path take fllg the grd ad that ca be used to estmate the accuracy of the fte dfferece scheme that s beg used. The detals of settg up varous types of schemes are beyod the scope of ths book ad could serve as the subect of a book by themselves. For a further troducto to the soluto of partal dfferetal equatos the reader s referred to Sokolkoff ad Redheffer 7 ad for the umercal mplemetato of some methods the studet should cosult Press et.al. 8. Let us ow tur to the umercal soluto of tegral equatos. 5. The Numercal Soluto of Itegral Equatos For reasos that I have ever fully uderstood, the mathematcal lterature s crowded wth books, artcles, ad papers o the subect of dfferetal equatos. Most uverstes have several courses of study the subect, but lttle atteto s pad to the subect of tegral equatos. The dfferetal operator s lear ad so s the tegral operator. Ideed, oe s ust the verse of the other. Equatos ca be wrtte where the depedet varable appears uder a tegral as well as aloe. Such equatos are the aalogue of the dfferetal equatos ad are called tegral equatos. It s ofte possble to tur a dfferetal equato to a tegral equato whch may make the problem easer to umercally solve. Ideed may physcal pheomea led themselves to descrpto by tegral equatos. So oe would thk that they mght form as large a area for aalyss are do the dfferetal equatos. Such s ot the case. Ideed, we wll ot be able to devote as much tme to the dscusso of these terestg equatos as we should, but we shall sped eough tme so that the studet s at least famlar wth some of ther basc propertes. Of ecessty, we wll restrct our dscusso to those tegral equatos where the ukow appears learly. Such lear equatos are more tractable ad yet descrbe much that s of terest scece. 47

Numercal Methods ad Data Aalyss a. Types of Lear Itegral Equatos We wll follow the stadard classfcato scheme for tegral equatos whch, whle ot ehaustve, does clude most of the commo types. There are bascally two ma classes kow as Fredholm ad Volterra after the mathematcas who frst studed them detal. Fredholm equatos volve defte tegrals, whle Volterra equatos have the depedet varable as oe of the lmts. Each of these categores ca be further subdvded as to whether or ot the depedet varable appears outsde the tegral sg as well as uder t. Thus the two types of Fredholm equatos for the ukow φ are b F() K(, t) φ(t)dt Fredholm Type I a b, (5..) φ() F() + λ K(, t) φ(t)dt Fredholm Type II a whle the correspodg two types of Volterra equatos for φ take the form F() K(, t) φ(t)dt Volterra Type I a. (5..) φ() F() + λ K(, t) φ(t)dt Volterra Type II a The parameter K(,t) appearg the tegrad s kow as the kerel of the tegral equato. Its form s crucal determg the ature of the soluto. Certaly oe ca have homogeeous or homogeeous tegral equatos depedg o whether or ot F() s zero. Of the two classes, the Fredholm are geerally easer to solve. b. The Numercal Soluto of Fredholm Equatos Itegral equatos are ofte easer to solve tha a correspodg dfferetal equato. Oe of the reasos s that the trucato errors of the soluto ted to be averaged out by the process of quadrature whle they ted to accumulate durg the process of umercal tegrato employed the soluto of dfferetal equatos. The most straght-forward approach s to smply replace the tegral wth a quadrature sum. I the case of Fredholm equatos of type oe, ths results a fuctoal equato for the ukow φ() at a dscrete set of pots t used by the quadrature scheme. Specfcally F() Σ K(,t )φ(t )W +R (). (5..) Sce equato (5..) must hold for all values of, t must hold for values of equal to those chose for the quadrature pots so that t,,,,. (5..4) By pckg those partcular pots we ca geerate a lear system of equatos from the fuctoal equato (5..) ad, eglectg the quadrature error term, they are F( ) Σ K(,t )φ(t )W Σ A φ( ),,,,, (5..5) whch ca be solved by ay of the methods dscussed Chapter yeldg 48

k k 5 @ Dfferetal ad Itegral Equatos φ( ) A F( ),,,,. (5..6) The soluto wll be obtaed at the quadrature pots so that oe mght wsh to be careful the selecto of a quadrature scheme ad pck oe that cotaed the pots of terest. However, oe ca use the soluto set φ( ) to terpolate for mssg pots ad mata the same degree of precesso that geerated the soluto set. For Fredholm equatos of type, oe ca perform the same trck of replacg the tegral wth a quadrature scheme. Thus k φ( ) F() + λ K(, t ) φ(t )W + R (). (5..7) Here we must be a lttle careful as the ukow φ() appears outsde the tegral. Thus equato (5..7) s a fuctoal equato for φ() tself. However, by evaluatg ths fuctoal equato as we dd for Fredholm equatos of type we get φ( ) F( ) + λ K(, t ) φ(t ) W, (5..8) whch, after a lttle algebra, ca be put the stadard form for lear equatos that have a soluto F( ) [ δ λk(, t )W ] φ(t ) B φ( ),,,,, (5..9) k k φ( ) B F( ),,,,. (5..) k Here the soluto set φ( ) ca be substtuted to equato (5..7) to drectly obta a terpolato formula for φ() whch wll have the same degree of precso as the quadrature scheme ad s vald for all values of. Such equatos ca be solved effcetly by usg the approprate Gaussa quadrature scheme that s requred by the lmts. I addto, the form of the kerel K(,t) may fluece the choce of the quadrature scheme ad t s useful to clude as much of the behavor of the kerel the quadrature weght fuctos as possble. We could also choose to break the terval a b several peces depedg o the ature of the kerel ad what ca be guessed about the soluto tself. The subsequet quadrature schemes for the subtervals wll ot the deped o the cotuty of polyomals from oe sub-terval to aother ad may allow for more accurate appromato the sub-terval. For a specfc eample of the soluto to Fredholm equatos, let us cosder a smple equato of the secod type amely y () + ty dt. (5..) Comparg ths to equato (5..7), we see that F(), ad that the kerel s separable whch leads us mmedately to a aalytc soluto. Sce the tegral s a defte tegral, t may be regarded as some costat α ad the soluto wll be lear of the form Ths leads to a value for α of y () + α t( + αt)dt + ( + α ). (5..) 49

Numercal Methods ad Data Aalyss α /4. (5..) However, had the equato requred a umercal soluto, the we would have proceeded by replacg the tegral by a quadrature sum ad evaluatg the resultg fuctoal equato at the pots of the quadrature. Kowg that the soluto s lear, let us choose the quadrature to be Smpso's rule whch has a degree of precso hgh eough to provde a eact aswer. The lear equatos for the soluto become y() + ()[()y() + 4( y( ) + ( )[()y() + 4( y() + ()[()y() + 4( )y( )y( )y( ) + ) + ) + y()]/ 6 y()]/ 6 y()]/ 6 + y( + y( ) / 6 + y() / ) / + y() / 6, (5..4) whch have the mmedate soluto y() y( ). (5..5) 8 7 y() 4 Clearly ths soluto s eact agreemet wth the aalytc form correspodg to α/4, y() + /4. (5..6) Whle there are varatos o a theme for the soluto of these type of equatos, the basc approach s cely llustrated by ths approach. Now let us tur to the geerally more formdable Volterra equatos. c. The Numercal Soluto of Volterra Equatos We may approach Volterra equatos much the same way as we dd Fredholm equatos, but there s the problem that the upper lmt of the tegral s the depedet varable of the equato. Thus we must choose a quadrature scheme that utlzes the edpots of the terval; otherwse we wll ot be able to evaluate the fuctoal equato at the relevat quadrature pots. Oe could adopt the vew that Volterra equatos are, geeral, ust specal cases of Fredholm equatos where the kerel s K(,t), t >. (5..7) but ths would usually requre the kerel to be o-aalytc However, f we choose such a quadrature formula the, for Volterra equatos of type, we ca wrte F( ) K(, ) φ( )W k a + kh,,,,. (5..8) Not oly must the quadrature scheme volve the edpots, t must be a equal terval formula so that successve evaluatos of the fuctoal equato volve the pots where the fucto has bee prevously 5

5 @ Dfferetal ad Itegral Equatos evaluated. However, by dog that we obta a system of lear equatos (+) ukows. The value of φ(a) s ot clearly specfed by the equato ad must be obtaed from the fuctoal behavor of F(). Oe costrat that supples the mssg value of φ() s df() φ (a) K(a, a). (5..9) d The value of φ(a) reduces equatos (5..8) to a tragular system that ca be solved quckly by successve substtuto (see secto.). The same method ca be used for Volterra equatos of type yeldg F( ) φ( ) + K(, ) φ( )W,,,, a + kh k. (5..) Here the dffculty wth φ(a) s removed sce the lmt as a φ(a) F(a). (5..) Thus t would appear that type equatos are more well behaved that type equatos. To the etet that ths s true, we may replace ay type equato wth a type equato of the form K(, t) F '() K(, ) φ() + φ(t)dt. a (5..) Ufortuately we must stll obta F'() whch may have to be accomplshed umercally. Cosder how these drect soluto methods ca be appled practce. Let us choose equato (5..), whch served so well as a test case for dfferetal equatos. I settg that equato up for Pcard's method, we tured t to a type Volterra tegral equato of the form a y () + ty dt. (5..) If we put ths the form suggested by equato (5..7) where the kerel vashes for t >, we could wrte ty dt y( ) y() t y(t )W, W, >. (5..4) Here we have sured that the kerel vashes for t> by choosg the quadrature weghts to be zero whe that codto s satsfed. The resultg lear equatos for the soluto become y() [()y() + 4()y( ) + ()y()]/ 6 y(), y( ) [()y() + 4( )y( ) + ()y()]/ 6 y( ) /,. (5..5) y() [()y() + 4( )y( ) + y()]/ 6 y( ) / + 5y() / 6, The method of usg equal terval quadrature formulae of varyg degrees of precso as creases s epresses by equato (5..8), whch for our eample takes the form Ths results lear equatos for the soluto that are ty dt y( ) y() t y(t )W. (5..6) 5

Numercal Methods ad Data Aalyss y() () y( ) [()y() + ( )y( )]/ y( ) / 4,. (5..7) y() [()y() + 4( )y( ) + y()]/ 6 y( ) / + 5y() / 6 The solutos to the two sets of lear equatos (5..5) ad (5..7) that represet these two dfferet approaches are gve table 5.5 Table 5.5 Sample Solutos for a Type Volterra Equato FREDHOLM SOLN. TRIANGULAR SOLN. ANALYTIC SOLN. y()... % Error.%.% --------- y(½).5..84 % Error 6.8%.8% --------- y().8.7.78 % Error -.8% -6.% --------- As wth the other eamples, we have take a large step sze so as to emphasze the relatve accuracy. Wth the step sze aga beg uty, we get a rather poor result for the rapdly creasg soluto. Whle both method gve aswers that are slghtly larger tha the correct aswer at ½, they rapdly fall behd the epoetally creasg soluto by. As was suggested, the tragular soluto s over all slghtly better that the Fredholm soluto wth the dscotuous kerel. Whe applyg quadrature schemes drectly to Volterra equatos, we geerate a soluto wth varable accuracy. The quadrature scheme ca tally have a degree of precso o greater tha oe. Whle ths mproves as oe crosses the terval the trucato error curred the frst several pots accumulates the soluto. Ths was ot a problem wth Fredholm equatos as the trucato error was spread across the terval perhaps weghted to some degree by the behavor of the kerel. I addto, there s o opportuty to use the hghly effcet Gaussa schemes drectly as the pots of the quadrature must be equally spaced. Thus we wll cosder a drect applcato of quadrature schemes to the soluto of both types of tegral equatos. By usg a quadrature scheme, we are tactly assumg that the tegrad s well appromated by a polyomal. Let us stead assume that the soluto tself ca be appromated by a polyomal of the form φ( ) Σ α ξ (). (5..8) Substtuto of ths polyomal to the tegral of ether Fredholm or Volterra equatos yelds K(, t) φ(t)dt α K(, t) ξ (t)dt + R α H () + R. 5..9) Now the etre tegrad of the tegral s kow ad may be evaluated to geerate the fuctos H (). It 5

5 @ Dfferetal ad Itegral Equatos should be oted that the fucto H () wll deped o the lmts for both classes of equatos, but ts evaluato poses a separate problem from the soluto of the tegral equato. I some cases t may be evaluated aalytcally ad others t wll have to be computed umercally for ay chose value of. However, oce that s doe, type oe equatos of both classes ca be wrtte as F( ) Σ α H ( ),,,,, (5..) whch costtute a lear system of (+) algebrac equatos the (+) ukows α. These, ad equato (5..8) supply the desred soluto φ(). Soluto for the type equatos s oly slghtly more complcated as equato (5..8) must be drectly serted to the tegral equato a evaluated at. However, the resultg lear equatos ca stll be put to stadard form so that the α s ca be solved for to geerate the soluto φ(). We have sad othg about the fuctos ξ () that appear the appromato equato (5..8). For omal polyomal appromato these mght be, but for large such a choce teds to develop stabltes. Thus the same sort of care that was used developg terpolato formulae should be employed here. Oe mght eve wsh to employ a ratoal fucto approach to appromatg φ() as was doe secto.. Such care s ustfed as we have troduced a addtoal source of trucato error wth ths approach. Not oly wll there be trucato error resultg from the quadrature appromato for the etre tegral, but there wll be trucato error from the appromato of the soluto tself [.e. equato (5..8)]. Whle each of these trucato errors s separately subect to cotrol, ther combed effect s less predctable. Fally, we should cosder the feasblty of teratve approaches coucto wth quadrature schemes for fdg solutos to these equatos. The type equatos mmedately suggest a teratve fucto of the form φ b (k) (k ) ( ) F() + λ K(, t) φ (t)dt. (5..) a Rememberg that t s φ() that we are after, we ca use equato (..) ad the learty of the tegral equatos wth respect to φ() to establsh that the teratve fucto wll coverge to a fed pot as log as λ b K(, t)dt <. (5..) a Equato (5..7) shows us that a Volterra equato s more lkely to coverge by terato tha a Fredholm equato wth a smlar kerel. If λ s small, the ot oly s the teratve sequece lkely to coverge, but a tal guess of φ () () F(). (5..) suggests tself. I all cases tegrato requred for the terato ca be accomplshed by ay desreable quadrature scheme as the prelmary value for the soluto φ (k-) () s kow. 5

Numercal Methods ad Data Aalyss d. The Ifluece of the Kerel o the Soluto Although the learty of the tegral operator ad ts verse relatoshp to the dfferetal operator teds to make oe thk that tegral equatos are o more dffcult tha dfferetal equatos, there are some subtle dffereces. For eample, oe would ever attempt a umercal soluto of a dfferetal equato that could be show to have o soluto, but that ca happe wth tegral equatos f oe s ot careful. The presece of the kerel uder the operator makes the behavor of these equatos less trasparet tha dfferetal equatos. Cosder the apparetly beg kerel K(,t) cos() cos(t), (5..4) ad a assocated Fredholm equato of the frst type + a a F() cos()cos(t)φ(t)dt cos()z(a). (5..5) Clearly ths equato has a soluto f ad oly f F() has the form gve by the rght had sde. Ideed, ay kerel that s separable the depedet varables so as to have the form K(,t) P()Q(t), (5..6) places costrats o the form of F() for whch the equato has a soluto. Nevertheless, t s cocevable that someoe could try to solve equato (5..5) for fuctoal forms of F() other tha the those whch allow for a value of φ() to est. Udoubtedly the umercal method would provde some sort of aswer. Ths probably prompted Baker 9, as reported by Crag ad Brow, to remark 'wthout care we may well fd ourselves computg appromate solutos to problems that have o true solutos'. Clearly the form of the kerel s crucal to ature of the soluto, deed, to ts very estece. Should eve the codtos mposed o F() by equato (5..5) be met, ay soluto of the form φ() φ () + ζ(), (5..7) where φ () s the tal soluto ad ζ() s ay at-symmetrc fucto wll also satsfy the equato. Not oly are we ot guarateed estece, we are ot eve guarateed uqueess whe estece ca be show. Fortuately, these are ofte ust mathematcal cocers ad equatos that arse from scetfc argumets wll geerally have uque solutos f they are properly formulated. However, there s always the rsk that the formulato wll sert the problem a class wth may solutos oly oe of whch s physcal. The vestgator s the faced wth the added problem of fdg all the solutos ad decdg whch oes are physcal. That may prove to be more dffcult tha the umercal problem of fdg the solutos. There are other ways whch the kerel ca fluece the soluto. Crag ad Brow devote most of ther book to vestgatg the soluto of a class of tegral equatos whch represet verso problems astroomy. They show repeatedly that the presece of a approprate kerel ca cause the umercal methods for the soluto to become wldly ustable. Most of ther atteto s drected to the effects of radom error F() o the subsequet soluto. However, the trucato error equato (5..) ca combe wth F() to smulate such errors. The mplcatos are devastatg. I Fredholm equatos of Type, f λ s large ad the kerel a weak fucto of t, the the soluto s lable to be etremely ustable. The reaso for ths ca be see the role of the kerel determg the soluto φ(). K(,t) behaves lke a 54

5 @ Dfferetal ad Itegral Equatos flter o the cotrbuto of the soluto at all pots to the local value of the soluto. If K(,t) s large oly for t the the cotrbuto of the rest of the tegral s reduced ad φ() s largely determed by the local value of [.e. F()]. If the Kerel s broad the dstat values of φ(t) play a maor role determg the local value of φ(). If λ s large, the the role of F() s reduced ad the equato becomes more early homogeeous. Uder these codtos φ() wll be poorly determed ad the effect of the trucato error o F() wll be dsproportoately large. Thus oe should hope for o-separable Kerels that are strogly peaked at t. What happes at the other etreme whe the kerel s so strogly peaked at t that t ehbts a sgularty. Uder may codtos ths ca be accommodated wth the quadrature approaches we have already developed. Cosder the ultmately peaked kerel K(,t) δ(-t), (5..8) where δ() s the Drac delta fucto. Ths reduces all of the tegral equatos dscussed here to have solutos φ() F() type. (5..9) φ() F()( λ) type Thus, eve though the Drac delta fucto s "udefed" for zero argumet, the tegrals are well defed ad the subsequet solutos smple. For kerels that have sgulartes at t, but are defed elsewhere we ca remove the sgularty by the smple epedet of addg ad subtractg the aswer from the tegrad so that φ (k) b ( ) F() + λ K(, t)[ φ(t) φ()]dt + λφ() K(, t)dt. (5..4) a We may use the stadard quadrature techques o ths equato f the followg codtos are met: b K(, t)dt <, a. (5..4) Lm{K(, t)[ φ(t) φ()]} t The frst of these s a reasoable costrat of the kerel. If that s ot met t s ulkely that the soluto ca be fte. The secod codto wll be met f the kerel does ot approach the sgularty faster tha learly ad the soluto satsfes a Lpshtz codto. Sce ths s true of all cotuous fuctos, t s lkely to be true for ay equato that arses from modelg the real world. If ths codto s met the the cotrbuto to the quadrature sum from the terms where ( ) ca be omtted (or assged weght zero). Wth that slght modfcato all the prevously descrbed schemes ca be utlzed to solve the resultg equato. Although some addtoal algebra s requred, the resultg lear algebrac equatos ca be put to stadard form ad solved usg the formalsms from Chapter. I ths chapter we have cosdered the soluto of dfferetal ad tegral equatos that arse so ofte the world of scece. What we have doe s but a bref survey. Oe could devote hs or her lfe to the study of these subects. However, these techques wll serve the studet of scece who wshes smply to use them as tools to arrve at a aswer. As problems become more dffcult, algorthms may eed to become more sophstcated, but these fudametals always provde a good begg. a b 55

Numercal Methods ad Data Aalyss Chapter 5 Eercses. Fd the soluto to the followg dfferetal equato y' y, the rage. Let the tal value be y(). Use the followg methods for your soluto: a. a secod order Ruge-Kutta b. a -pot predctor-corrector. c. Pcard's method wth steps. d. Compare your aswer to the aalytc soluto.. Fd the soluto for the dfferetal equato y" + y' + ( -6)y, the rage wth tal values of y'()y(). Use ay method you lke,but epla why t was chose.. Fd the umercal soluto to the tegral equato y() + y(t)( t+t +5t 5 )dt,. Commet o the accuracy of your soluto ad the reaso for usg the umercal method you chose. 4. Fd a closed form soluto to the equato problem of the form y() a + b + c, ad specfcally obta the values for a,b, ad c. 5. How would you have umercally obtaed the values for a, b, ad c of problem 4 had you oly kow the umercal soluto to problem? How would the compare to the values obtaed from the closed form soluto? 56

6. We wsh to fd a appromate soluto to the followg tegral equato: 5 @ Dfferetal ad Itegral Equatos y() + + t y(t) dt. a. Frst assume we shall use a quadrature formula wth a degree of precso of two where the pots of evaluato are specfed to be.5,.5, ad.75. Use Lagrage terpolato to fd the weghts for the quadrature formula ad use the results to fd a system of lear algebrac equatos that represet the soluto at the quadrature pots. b. Solve the resultg lear equatos by meas of Gauss-Jorda elmato ad use the results to fd a terpolatve soluto for the tegral equato. Commet o the accuracy of the resultg soluto over the rage. 7. Solve the followg tegral equato: B() / B(t)E t- dt, where E () e -t dt/t. a. Frst solve the equato by treatg the tegral as a Gaussa sum. Note that Lm E, b. Solve the equato by epadg B(t) a Taylor seres about ad thereby chagg the tegral equato to a th order lear dfferetal equato. Covert ths equato to a system of frst order lear dfferetal equatos ad solve the system subect to the boudary codtos B() B, B'( ) B"( ) B () ( ). Note that the tegral equato s a homogeeous equato. Dscuss how that flueced your approach to the problem. 57

Numercal Methods ad Data Aalyss Chapter 5 Refereces ad Supplemetal Readg. Hammg, R.W., "Numercal Methods for Scetsts ad Egeers" (96) McGraw-Hll Book Co., Ic., New York, Sa Fracsco, Toroto, Lodo, pp. 4-7.. Press, W.H., Flaery, B.P., Teukolsky, S.A., ad Vetterlg, W.T., "Numercal Recpes the Art of Scetfc Computg" (986), Cambrdge Uversty Press, Cambrdge, pp. 569.. Press, W.H., Flaery, B.P., Teukolsky, S.A., ad Vetterlg, W.T., "Numercal Recpes the Art of Scetfc Computg" (986), Cambrdge Uversty Press, Cambrdge, pp. 56-568. 4. Press, W.H., Flaery, B.P., Teukolsky, S.A., ad Vetterlg, W.T., "Numercal Recpes the Art of Scetfc Computg" (986), Cambrdge Uversty Press, Cambrdge, pp. 56. 5. Day, J.T., ad Colls, G.W.,II, "O the Numercal Soluto of Boudary Value Problems for Lear Ordary Dfferetal Equatos", (964), Comm. A.C.M. 7, pp -. 6. Fo, L., "The Numercal Soluto of Two-pot Boudary Value Problems Ordary Dfferetal Equatos", (957), Oford Uversty Press, Oford. 7. Sokolkoff, I.S., ad Redheffer, R.M., "Mathematcs of Physcs ad Moder Egeerg" (958) McGraw-Hll Book Co., Ic. New York, Toroto, Lodo, pp. 45-5. 8. Press, W.H., Flaery, B.P., Teukolsky, S.A., ad Vetterlg, W.T., "Numercal Recpes the art of scetfc computg" (986), Cambrdge Uversty Press, Cambrdge, pp. 65-657. 9. Baker, C.T.N., "The Numercal Treatmet of Itegral Equatos", (977), Oford Uversty Press, Oford.. Crag, I.J.D., ad Brow, J.C., (986), "Iverse Problems Astroomy -A Gude to Iverso Strateges for Remotely Sesed Data", Adam Hlger Ltd. Brstol ad Bosto, pp. 5. Crag, I.J.D., ad Brow, J.C., (986), "Iverse Problems Astroomy -A Gude to Iverso Strateges for Remotely Sesed Data", Adam Hlger Ltd. Brstol ad Bosto. 58

6 Least Squares, Fourer Aalyss, ad Related Appromato Norms Up to ths pot we have requred that ay fucto we use to represet our 'data' pots pass through those pots eactly. Ideed, ecept for the predctor-corrector schemes for dfferetal equatos, we have used all the formato avalable to determe the appromatg fucto. I the etreme case of the Ruge-Kutta method, we eve made demads that eceeded the avalable formato. Ths led to appromato formulae that were uder-determed. Now we wll cosder approaches for determg the appromatg fucto where some of the formato s delberately gored. Oe mght woder why such a course would ever be followed. The aswer ca be foud by lookg two rather dfferet drectos. 59

Numercal Methods ad Data Aalyss Remember, that cosderg predctor-corrector schemes the last chapter, we delberately gored some of the fuctoal values whe determg the parameters that specfed the fucto. That was doe to avod the rapd fluctuatos characterstc of hgh degree polyomals. I short, we felt that we kew somethg about etrapolatg our appromatg fucto that trasceded the kow values of specfc pots. Oe ca mage a umber of stuatos where that mght be true. Therefore we ask f there s a geeral approach whereby some of the fuctoal values ca be delberately gored whe determg the parameters that represet the appromatg fucto. Clearly, aytme the form of the fucto s kow ths ca be doe. Ths leads drectly to the secod drecto where such a approach wll be useful. So far we have treated the fuctoal values that costra the appromatg fucto as f they were kow wth absolute precso. What should we do f ths s ot the case? Cosder the stuato where the fuctoal values resulted from observato or epermetato ad are characterzed by a certa amout of error. There would be o reaso to demad eact agreemet of the fuctoal form at each of the data pots. Ideed, such cases the fuctoal form s geerally cosdered to be kow a pror ad we wsh to test some hypothess by seeg to what etet the mprecse data are represeted by the theory. Thus the two dfferet cases for ths approach to appromato ca be summarzed as: a. the data s eact but we desre to represet t by a appromatg fucto wth fewer parameters tha the data. b. the appromatg fucto ca be cosdered to be "eact" ad the data whch represets that fucto s mprecse. There s a thrd stuato that occasoally arses where oe wshes to appromate a table of emprcally determed umbers whch are heretly mprecse ad the form of the fucto must also be assumed. The use of ay method ths stace must be cosdered suspect as there s o way to separate the errors of observato or epermetato from the falure of the assumed fucto to represet the data. However, all three cases have oe thg commo. They wll geerate systems that wll be overdetermed sce there wll, geeral, be more costrag data tha there are free parameters the appromatg fucto. We must the develop some crtero that wll eable us to reduce the problem to oe that s eactly determed. Sce the fucto s ot requred to match the data at every pot, we must specfy by how much t should mss. That crtero s what s kow as a appromato orm ad we shall cosder two popular oes, but devote most of our effort to the oe kow as the Least Square Norm. 6. Legedre's Prcple of Least Squares Legedre suggested that a approprate crtero for fttg data pots wth a fucto havg fewer parameters tha the data would be to mmze the square of the amout by whch the fucto msses the data pots. However, the oto of a "mss" must be quatfed. For least squares, the "mss" wll be cosdered to result from a error the depedet varable aloe. Thus, we assume that there s o error the depedet varable. I the evet that each pot s as mportat as ay other pot, we ca do ths by mmzg the sum-square of those errors. The use of the square of the error s mportat for t elmates the fluece of ts sg. Ths s the lowest power depedece of the error ε betwee the data pot ad the 6

6 @ Least Squares appromatg fucto that eglects the sg. Of course oe could appeal to the absolute value fucto of the error, but that fucto s ot cotuous ad so may produce dffcultes as oe tres to develop a algorthm for determg the adustable free parameters of the appromatg fucto. Least Squares s a very broad prcple ad has specal eamples may areas of mathematcs. For eample, we shall see that f the appromatg fuctos are ses ad coses that the Prcple of Least Squares leads to the determato of the coeffcets of a Fourer seres. Thus Fourer aalyss s a specal case of Least Squares. The relatoshp betwee Least Squares ad Fourer aalyss suggests a broad appromato algorthm volvg orthogoal polyomals kow as the Legedre Appromato that s etremely stable ad applcable to very large data bases. Wth ths md, we shall cosder the developmet of the Prcple of Least Squares from several dfferet vatage pots. There are those who feel that there s somethg profoud about mathematcs that makes ths the "correct" crtero for appromato. Others feel that there s somethg about ature that makes ths the approprate crtero for aalyzg data. I the et two chapters we shall see that there are codtos where the Prcple of Least Squares does provde the most probable estmate of adustable parameters of a fucto. However, geeral, least squares s ust oe of may possble appromato orms. As we shall see, t s a partcularly coveet oe that leads to a straghtforward determato of the adustable free parameters of the appromatg fucto. a. The Normal Equatos of Least Squares Let us beg by cosderg a collecto of N data pots (,Y ) whch are to be represeted by a appromatg fucto f(a,) so that f(a, ) Y. (6..) Here the (+) a 's are the parameters to be determed so that the sum-square of the devatos from Y are a mmum. We ca wrte the devato as ε Y f(a, ). (6..) The codtos that the sum-square error be a mmum are ust N a ε N [Y f (a f (a, ), )], a,,,,. (6..) There s oe of these equatos for each of the adustable parameters a so that the resultat system s uquely determed as log as (+) N. These equatos are kow as the ormal equatos for the problem. The ature of the ormal equatos wll be determed by the ature of f(a,). That s, should f(a,) be o-lear the adustable parameters a, the the ormal equatos wll be o-lear. However, f f(a,) s lear the a 's as s the case wth polyomals, the the resultat equatos wll be lear the a 's. The ease of soluto of such equatos ad the great body of lterature relatg to them make ths a most mportat aspect of least squares ad oe o whch we shall sped some tme. 6

Numercal Methods ad Data Aalyss b. Lear Least Squares Cosder the appromatg fucto to have the form of a geeral polyomal as descrbed chapter [equato (..)]. Namely, ) a kφk () k f (a φ. (6..4) Here the φ k () are the bass fuctos whch for commo polyomals are ust k. Ths fucto, whle hghly o-lear the depedet varable s lear the adustable free parameters a k. Thus the partal dervatve equato (6..) s ust ad the ormal equatos themselves become N f (a, ) φ ( a k φk ( ) φ ( ) k N ), (6..5) a Y φ ( ),,,,. (6..6) These are a set of lear algebrac equatos, whch we ca wrte compoet or vector form as a ka k C k. (6..7) a A C Sce the φ () are kow, the matr A( ) s kow ad depeds oly o the specfc values,, of the depedet varable. Thus the ormal equatos ca be solved by ay of the methods descrbed chapter ad the set of adustable parameters ca be determed. There are a umber of aspects of the lear ormal equatos that are worth otg. Frst, they form a symmetrc system of equatos sce the matr elemets are Σφ k φ. Sce φ () s presumed to be real, the matr wll be a ormal matr (see secto.). Ths s the org of the ame ormal equatos for the equatos of codto for least squares. Secod, f we wrte the appromatg fucto f(a,) vector form as f (a, ) a φ(), (6..8) the the ormal equatos ca be wrtte as N N a φ( ) φ( ) Y φ(. (6..9) ) Here we have defed a vector φ () whose compoets are the bass fuctos φ (). Thus the matr elemets of the ormal equatos ca be geerated smply by takg the outer (tesor) product of the bass vector wth tself ad summg over the values of the vector for each data pot. A thrd way to develop the ormal equatos s to defe a o-square matr from the bass fuctos evaluated at the data pots as 6

6 @ Least Squares φ () φ() φ () φ ( ) φ( ) φ ( ) φk. (6..) φ ( ) φ( ) φ ( ) Now we could wrte a over determed system of equatos whch we would lke to hold as φ a Y. (6..) The ormal equatos ca the be descrbed by T T [ φ φ]a φ Y, (6..) where we take advatage of the matr product to perform the summato over the data pots. Equatos (6..9) ad (6..) are smply dfferet mathematcal ways of epressg the same formalsm ad are useful developg a detaled program for the geerato of the ormal equatos. So far we have regarded all of the data pots to be of equal value determg the soluto for the free parameters a. Ofte ths s ot the case ad we would lke to cout a specfc pot (,Y ) to be of more or less value tha the others. We could smply clude t more tha oce the summatos that lead to the ormal equatos (6..6) or add t to the lst of observatoal pots defg the matr φ gve by equato (6..). Ths smplstc approach oly yelds tegral weghts for the data pots. A far more geeral approach would smply assg the epresso [equato (6..) or equato (6..8)] represetg the data pot a weght ω. the equato (6..) would have the form f (a, ) ϖa φ( ) ϖy. (6..) However, the partal dervatve of f wll also cota the weght so that f (a, ) ϖ ĵ φ( ) ϖ φ ( ). (6..4) a Thus the weght wll appear quadratcally the ormal equatos as N k ϖ φk ( ) φ ( ) k N a ϖ Y φ ( ),,,,. (6..5) I order to cotually epress the weght as a quadratc form, may authors defe w ϖ, (6..6) so that the ormal equatos are wrtte as N k w φk ( ) φ ( ) k N a w Y φ ( ),,,,. (6..7) Ths smple substtuto s ofte a source of cosderable cofuso. The weght w s the square of the weght assged to the observato ad s of ecessty a postve umber. Oe caot detract from the mportace of a data pot by assgg a egatve weght ϖ. The geerato of the ormal equatos would force the square-weght w to be postve thereby ehacg the role of that pot determg the soluto. Throughout the remader of ths chapter we shall cosstetly use w as the square-weght deoted by equato (6..6). However, we shall also use ϖ as the dvdual weght of a gve observato. The reader should be careful ot to cofuse the two. 6

Numercal Methods ad Data Aalyss Oce geerated, these lear algebrac equatos ca be solved for the adustable free parameters by ay of the techques gve chapter. However, uder some crcumstaces, t may be possble to produce ormal equatos whch are more stable tha others. c. The Legedre Appromato I the stace where we are appromatg data, ether tabular or epermetal, wth a fucto of our choce, we ca mprove the umercal stablty by choosg the bass fuctos φ () to be members of orthogoal set. Now the maorty of orthogoal fuctos we have dscussed have bee polyomals (see secto.) so we wll base our dscusso o orthogoal polyomals. But t should rema clear that ths s a coveece, ot a requremet. Let φ () be a orthogoal polyomal relatve to the weght fucto w() over the rage of the depedet varable. The elemets of the ormal equatos (6..7) the take the form N k w φk ( ) φ ( ) A. (6..8) If we weght the pots accordace wth the weght fucto of the polyomal, the the weghts are w w( ). (6..9) If the data pots are truly depedet ad radomly selected throughout the rage of, the as the umber of them creases, the sum wll approach the value of the tegral so that N A k Lm w( ) φk ( ) φ ( ) N w() φk () φ ()d Nδ k. (6..) N Ths certaly smplfes the soluto of the ormal equatos (6..7) as equato (6..) states that the off dagoal elemets wll ted to vash. If the bass fuctos φ () are chose from a orthoormal set, the the soluto becomes N a w( ) φ ( )Y,,,,. (6..) N Should they be merely orthogoal, the the soluto wll have to be ormalzed by the dagoal elemets leadg to a soluto of the form N N a w( ) ( )Y w( ) ( ),, φ φ,,. (6..) The process of usg a orthogoal set of fuctos φ () to descrbe the data so as to acheve the smple result of equatos (6..) ad (6..) s kow as the Legedre appromato. It s of cosderable utlty whe the amout of data s vast ad the process of formg ad solvg the full set of ormal equatos would be too tme cosumg. It s eve possble that some cases, the soluto of a large system of ormal equatos could troduce greater roud-off error tha s curred the use of the Legedre appromato. Certaly the umber of operatos requred for the evaluato of equatos (6..) or (6..) are of the order of (+)N where for the formato ad soluto of the ormal equatos (6..7) themselves somethg of the order of (+) (N++) operatos are requred. 64

6 @ Least Squares Oe should always be wary of the tme requred to carry out a Least Squares soluto. It has the habt of growg rapdly ad gettg out of had for eve the fastest computers. There are may problems where may be of the order of whle N ca easly reach 6. Eve the Legedre appromato would mply 8 operatos for the completo of the soluto, whle for a full soluto of the ormal equatos operatos would eed to be performed. For curret megaflop maches the Legedre appromato would oly take several mutes, whle the full soluto would requre several hours. There are problems that are cosderably larger tha ths eample. Icreasg ether or N by a order of magtude could lead to computatoally prohbtve problems uless a faster approach ca be used. To uderstad the org of oe of the most effcet appromato algorthms, let us cosder the relato of least squares to Fourer aalyss. 6. Least Squares, Fourer Seres, ad Fourer Trasforms I ths secto we shall eplctly eplore the relatoshp betwee the Prcple of least Squares ad Fourer seres. The we eted the oto of Fourer seres to the Fourer tegral ad fally to the Fourer trasform of a fucto. Lastly, we shall descrbe the bass for a etremely effcet algorthm for umercally evaluatg a dscrete Fourer trasform. a. Least Squares, the Legedre Appromato, ad Fourer Seres I secto.e we oted that the trgoometrc fuctos se ad cose formed orthoormal sets the terval +, ot oly for the cotuous rage of but also for a dscrete set of values as log as the values were equally spaced. Equato (..4) states that N N s(kπ )s(π ) cos(kπ )cos(π ) Nδ k. (6..) ( N) / N,,, N Here we have trasformed to the more famlar terval - +. Now cosder the ormal equatos that wll be geerated should the bass fuctos be ether cos(π) or s(π) ad the data pots are spaced accord wth the secod of equatos (6..). Sce the fuctoal sets are orthoormal we may employ the Legedre appromato ad go mmedately to the soluto gve by equato (6..) so that the coeffcets of the se ad cose seres are N a f ( )cos(π ) N + N. (6..) b π f ( )s( ) N + Sce these trgoometrc fuctos are strctly orthogoal the terval, as log as the data pots are equally spaced, the Legedre appromato s ot a appromato. Therefore the equal sgs equatos (6..) are strctly correct. The orthogoalty of the trgoometrc fuctos wth respect to equally spaced data ad the cotuous varable meas that we ca replace the summatos equato (6..) wth tegral 65

Numercal Methods ad Data Aalyss sgs wthout passg to the lmt gve equato (6..) ad wrte whch are the coeffcets of the Fourer seres a b + + f ()cos(π)d f ()s(π)d, (6..) a + a k cos(kπ) + b k s(kπ ) k f (). (6..4) Let us pause for a momet to reflect o the meag of the seres gve by equato (6..4). The fucto f() s represeted terms of a lear combato of perodc fuctos. The coeffcets of these fuctos are themselves determed by the perodcally weghted behavor of the fucto over the terval. The coeffcets a k ad b k smply measure the perodc behavor of the fucto tself at the perod (/πk). Thus, a Fourer seres represets a fucto terms of ts ow perodc behavor. It s as f the fucto were broke to peces that ehbt a specfc perodc behavor ad the re-assembled as a lear combato of the relatve stregth of each pece. The coeffcets are the ust the weghts of ther respectve cotrbuto. Ths s all accomplshed as a result of the orthogoalty of the trgoometrc fuctos for both the dscrete ad cotuous fte terval. We have see that Least Squares ad the Legedre appromato lead drectly to the coeffcets of a fte Fourer seres. Ths result suggests a mmedate soluto for the seres appromato whe the data s ot equally spaced. Namely, do ot use the Legedre appromato, but keep the off-dagoal terms of the ormal equatos ad solve the complete system. As log as N ad are ot so large as to pose computatoal lmts, ths s a perfectly acceptable ad rgorous algorthm for dealg wth the problem of uequally spaced data. However, the evet that the amout of data (N) s large there s a further developmet that ca lead to effcet data aalyss. b. The Fourer Itegral The fuctos that we dscussed above were cofed to the terval +. However, f the fuctos meet some farly geeral codtos, the we ca eted the seres appromato beyod that terval. Those codtos are kow as the Drchlet codtos whch are that the fucto satsfy Drchlet's theorem. That theorem states: Suppose that f() s well defed ad bouded wth a fte umber of mama, mma, ad dscotutes the terval -π +π. Let f() be defed beyod ths rego by f(+π) f(). The the Fourer seres for f() coverges absolutely for all. It should be oted that these are suffcet codtos, but ot ecessary codtos for the covergece of a Fourer seres. However, they are suffcetly geeral eough to clude a very wde rage of fuctos whch embrace vrtually all the fuctos oe would epect to arse scece. We may use these codtos to eted the oto of a Fourer seres beyod the terval +. 66

6 @ Least Squares where Let us defe z ξ /, (6..5) ξ >. (6..6) Usg Drchlet's theorem we develop a Fourer seres for f() terms of z so that mples whch wll have Fourer coeffcets gve by + a k f (z)cos(kπz)dz ξ a + a k cos(kπz) + b k s(kπz ) k f (zξ), (6..7) +ξ f ()cos(kπ / ξ)d ξ. (6..8) + +ξ b π π ξ k f (z)s(k z)dz ξ f ()s(k / )d ξ Makg use of the addto formula for trgoometrc fuctos cos(α-β) cosα cosβ + sα sβ, (6..9) we ca wrte the Fourer seres as +ξ +ξ f () f (z)dz + ξ f (z)cos[kπ(z ) / ξ]dz. (6..) ξ ξ k ξ Here we have doe two thgs at oce. Frst, we have passed from a fte Fourer seres to a fte seres, whch s assumed to be coverget. (.e. the Drchlet codtos are satsfed). Secod, we have eplctly cluded the a k 's ad b k 's the seres terms. Thus we have represeted the fucto terms of tself, or more properly, terms of ts perodc behavor. Now we wsh to let the fte summato seres pass to ts lmtg form of a tegral. But here we must be careful to remember what the terms of the seres represet. Each term the Fourer seres costtutes the cotrbuto to the fucto of ts perodc behavor at some dscrete perod or frequecy. Thus, whe we pass to the tegral lmt for the seres, the tegrad wll measure the frequecy depedece of the fucto. The tegrad wll tself cota a tegral of the fucto tself over space. Thus ths process wll trasform the represetato of the fucto from ts behavor frequecy to ts behavor space. Such a trasformato s kow as a Fourer Trasformato. c. The Fourer Trasform Let us see eplctly how we ca pass from the dscrete summato of the Fourer seres to the tegral lmt. To do ths, we wll have to represet the frequecy depedece a cotuous way. Ths ca be accomplshed by allowg the rage of the fucto (.e. ξ +ξ) to be varable. Let δα /ξ, (6..) so that each term the seres becomes +ξ +ξ f (z)cos[kπ(z ) / ξ]dz δα ξ f (z)cos[(kδα) π(z ) / ξ]dz ξ ξ. (6..) Now as we pass to the lmt of lettg δα, or ξ, each term the seres wll be multpled by a 67

Numercal Methods ad Data Aalyss ftesmal dα, ad the lmts o the term wll eted to fty. The product kδα wll approach the varable of tegrato α so that Lm δα ξ k +ξ ξ f (z)cos[(kδα) π(z ) / ξ]dz + ξ ξ f (z)cos[(kδα) π(z ) / ξ]dz dα. (6..) The rght had sde of equato 6.. s kow as the Fourer tegral whch allows a fucto f() to be epressed terms of ts frequecy depedece f(z). If we use the trgoometrc detty (6..9) to re-epress the Fourer tegrals eplctly terms of ther se ad cose depedece o z we get + + f () f (z)s( απz) s( απ)dz + +. (6..4) f () f (z)cos( απz) cos( απ)dz The separate forms of the tegrals deped o the symmetry of f(). Should f() be a odd fucto, the t wll cacel from all the cose terms ad produce oly the frst of equatos (6..4). The secod wll result whe f() s eve ad the se terms cacel. Clearly to produce a represetato of a geeral fucto f() we shall have to clude both the se ad cose seres. There s a otatoal form that wll allow us to do that usg comple umbers kow as Euler's formula e cos() + s(). (6..5) Ths yelds a fte Fourer seres of the form + k f () C ke k, (6..6) + π k t C k f (t)e dt where the comple costats C k are related to the a k 's ad b k 's of the cose ad se seres by C a / C + k a k / b k /. (6..7) C + k a k / b k / We ca eted ths represetato beyod the terval + the same way we dd for the Fourer Itegral. Replacg the fte summato by a tegral allows us to pass to the lmt ad get where + π z f () e F(z) dz, (6..8) F(z) + f (t)e π z t dt T(f ). (6..9) The tegral T(f) s kow as the Fourer Trasform of the fucto f(). It s worth cosderg the trasform of the fucto f(t) to smply be a dfferet represetato of the same fucto sce 68

6 @ Least Squares + π zt F(z) f (t)e dt T(f ) +. (6..) + π zt f (t) F(z)e dt T(F) T (f ) The secod of equatos (6..) reverses the effect of the frst, [.e.t(f) T - (f) ] so the secod equato s kow as the verse Fourer trasform. The Fourer trasform s oly oe of a large umber of tegrals that trasform a fucto from oe space to aother ad whose repeated applcato regeerates the fucto. Ay such tegral s kow as a tegral trasform. Net to the Fourer trasform, the best kow ad most wdely used tegral trasform s the Laplace trasform L(f) whch s defed as L (f) f (t)e pt dt. (6..) For may forms of f(t) the tegral trasforms as defed both equatos (6..) ad (6..) ca be epressed closed form whch greatly ehaces ther utlty. That s, gve a aalytc closed-form epresso for f(t), oe ca fd aalytc closed-form epresso for T(f) or L(f). Ufortuately the epresso of such tegrals s usually ot obvous. Perhaps the largest collecto of tegral trasforms, ot lmted to ust Fourer ad Laplace trasforms, ca be foud amog the Batema Mauscrpts where two full volumes are devoted to the subect. Ideed, oe must be careful to show that the trasform actually ests. For eample, oe mght beleve from the etremely geerous codtos for the covergece of a Fourer seres, that the Fourer trasform must always est ad there are those the sceces that take ts estece as a aom. However, equato (6..) we passed from a fte terval to the full ope fte terval. Ths may result a falure to satsfy the Drchlet codtos. Ths s the case for the bass fuctos of the Fourer trasform themselves, the ses ad coses. Thus s() or cos() wll ot have a dscrete Fourer trasform ad that should gve the healthy skeptc pause for thought. However, the evet that a closed form represetato of the tegral trasform caot be foud, oe must resort to a umercal approach whch wll yeld a dscrete Fourer trasform. After establshg the estece of the trasform, oe may use the very effcet method for calculatg t kow as the Fast Fourer Trasform Algorthm. d. The Fast Fourer Trasform Algorthm Because of the large umber of fuctos that satsfy Drchlet's codtos, the Fourer trasform s oe of the most powerful aalytc tools scece ad cosderable effort has bee devoted to ts evaluato. Clearly the evaluato of the Fourer trasform of a fucto f(t) wll geerally be accomplshed by appromatg the fucto by a Fourer seres that covers some fte terval. Therefore, let us cosder a fte terval of rage t so that we ca wrte the trasform as + + / N π t t π t π t F (z zk zk zk k ) f (t)e dt f (t)e dt f (t )e W. (6..) t / I order to take advatage of the orthogoalty of the ses ad coses over a dscrete set of equally spaced data the quadrature weghts W equato (6..) wll all be take to be equal ad to sum to the rage of the tegral so that 69

Numercal Methods ad Data Aalyss W t / N t(n) / N δ. (6..) Ths meas that our dscrete Fourer trasform ca be wrtte as F(z k ) δ N π z ( δ) f (t ) e. (6..4) I order for the uts to yeld a dmesoless epoet equato (6..4), z~t -. Sce we are determg a dscrete Fourer trasform, we wll choose a dscrete set of pot z k so that z k ±k/t(n) ± k/(nδ), (6..5) ad the dscrete trasform becomes F(z k ) N π (k / N) δf k δ f (t ) e. (6..6) To determe the Fourer trasform of f() s to fd N values of F k. If we wrte equato (6..6) vector otato so that F E f π (k / N). (6..7) E k e It would appear that to fd the N compoets of the vector F () we would have to evaluate a matr E havg N comple compoets. The resultg matr multplcato would requre N operatos. However, there s a approach that yelds a Fourer Trasform about Nlog N steps kow as the Fast Fourer Trasform algorthm or FFT for short. Ths trcky algorthm reles o otcg that we ca wrte the dscrete Fourer trasform of equato (6..6) as the sum of two smaller dscrete trasform volvg the eve ad odd pots of the summato. Thus F k N N / f (t f (t )e π(k / N) )e π(k / N) N / + e f (t )e π(k/ N) N / π(k/ N) f (t + N / + )e f (t )e π(k/ N) π(k/ N) F () k + Q k F () k. (6..8) If we follow the argumet of Press et. al., we ote that each of the trasforms volvg half the pots ca themselves be subdvded to two more. We ca cotue ths process utl we arrve at subtrasforms cotag but a sgle term. There s o summato for a oe-pot trasform so that t s smply equal to a partcular value of f( t k ). Oe eed oly detfy whch sub-trasform s to be assocated wth whch pot. The aswer, whch s what makes the algorthm practcal, s cotaed the order whch a sub-trasform s geerated. If we deote a eve sub-trasform at a gve level of subdvso by a superscrpt ad a odd oe by a superscrpt of, the sequetal geerato of sub-trasforms wll geerate a seres of bary dgts uque to that sub-trasform. The bary umber represeted by the reverse order of those dgts s the bary represetato of deotg the fuctoal value f( t ). Now re-sort the pots so that they are ordered sequetally o ths ew bary subscrpt say p. Each f( t p ) represets a oe pot subtrasform whch we ca combe va equato (6..8) wth ts adacet eghbor to form a two pot subtrasform. There wll of course be N of these. These ca be combed to form N four-pot sub-trasforms ad so o utl the N values of the fal trasform are geerated. Each step of combg trasforms wll take o the order of N operatos. The process of breakg the orgal trasform dow to oe-pot 7

6 @ Least Squares trasforms wll double the umber of trasforms at each dvso. Thus there wll be m sub-dvsos where m N, (6..9) so that m Log N. (6..) Therefore the total umber of operatos ths algorthm wll be of the order of Nlog N. Ths clearly suggests that N had better be a power of eve f t s ecessary to terpolate some addtoal data. There wll be some addtoal computato volved the calculato order to obta the Q k 's, carry out the addtos mpled by equato (6..46), ad perform the sortg operato. However, t s worth otg that at each subdvso, the values of Q k are related to ther values from the prevous subdvso e kπ/n for oly the legth of the sub-trasform, ad hece N, has chaged. Wth moder effcet sortg algorthms these addtoal tasks ca be regarded as eglgble addtos to the etre operato. Whe oe compares N to Nlog N for N ~ 6, the the savg s of the order of 5 4. Ideed, most of the algorthm ca be regarded as a bookkeepg eercse. There are etremely effcet packages that perform FFTs. The great speed of FFTs has lead to ther wde spread use may areas of aalyss ad has focused a great deal of atteto o Fourer aalyss. However, oe should always remember the codtos for the valdty of the dscrete Fourer aalyss. The most mportat of these s the estece of equally space data. The speed of the FFT algorthm s largely derved from the repettve ature of the Fourer Trasform. The fucto s assumed to be represeted by a Fourer Seres whch cotas oly terms that repeat outsde the terval whch the fucto s defed. Ths s the essece of the Drchlet codtos ad ca be see by spectg equato (6..8) ad otcg what happes whe k creases beyod N. The quatty e πk/n smply revolves through aother cycle yeldg the perodc behavor of F k. Thus whe values of a sub-trasform F k o are eeded for values of k beyod N, they eed ot be recalculated. Therefore the bass for the FFT algorthm s a systematc way of keepg track f the bookg assocated wth the geerato of the shorter sub-trasforms. By way of a eample, let us cosder the dscrete Fourer trasform of the fucto f(t) e - t. (6..) We shall cosder represetg the fucto over the fte rage (-½t +½t ) where t 4. Sce the FFT algorthm requres that the calculato be carred out over a fte umber of pots, let us take pots to sure a suffcet umber of geeratos to adequately demostrate the subdvso process. Wth these costrats md the equato (6..) defg the dscrete Fourer Trasform becomes + t / + + π tz + π tz t f (t)e dt e dt t / 7 t π t z F(z) e e W. (6..) We may compare the dscrete trasform wth the Fourer Trasform for the full fte terval (.e. - + ) as the tegral equato (6..) may be epressed closed form so that F[f(t)] F(z) /[+(π z )]. (6..) The results of both calculatos are summarzed table 6.. We have delberately chose a eve fucto of t as the Fourer trasform wll be real ad eve. Ths property s shared by both the dscrete ad cotuous trasforms. However, there are some sgfcat dffereces betwee the cotuous trasform 7

Numercal Methods ad Data Aalyss for the full fte terval ad the dscrete trasform. Whle the mamum ampltude s smlar, the dscrete trasform oscllates whle the cotuous trasform s mootoc. The oscllato of the dscrete trasform results from the trucato of the fucto at ½t. To properly descrbe ths dscotuty the fucto a larger ampltude for the hgh frequecy compoets wll be requred. The small umber of pots the trasform eacerbates ths. The absece of the hgher frequecy compoets that would be specfed by a larger umber of pots forces ther fluece to the lower order terms leadg to the oscllato. I spte of ths, the magtude of the trasform s roughly accord wth the cotuous trasform. Fgure 6. shows the comparso of the dscrete trasform wth the full terval cotuous trasform. We have cluded a dotted le coectg the pots of the dscrete trasform to emphasze the oscllatory ature of the trasform, but t should be remembered that the trasform s oly defed for the dscrete set of pots z k. Table 6. Summary Results for a Sample Dscrete Fourer Trasform I 4 5 6 7 t -. -.5 -. -.5. +.5 +. +.5 f(t ).5..678.665..665.678. k 4 5 6 7 z k. +.5 +.5 +.75 +. -.75 -.5 -.5 F(z k ) +.7648 -.7 +. -.6 +.56 -.6 +. -.7 F c (z k ) +. +.5768 +.84 +.86 +.494 +.86.84 +.5768 Whle the fucto we have chose s a eve fucto of t, we have ot chose the pots represetg that fucto symmetrcally the terval ( ½t +½t ). To do so would have cluded the each ed pot, but sce the fucto s regarded to be perodc over the terval, the edpots would ot be learly depedet ad we would ot have a addtoally dstct pot. I addto, t s mportat to clude the pot t the calculato of the dscrete trasform ad ths would be mpossble wth m pots symmetrcally spaced about zero. Let us proceed wth the detaled mplemetato of the FFT. Frst we must calculate the weghts W that appear equato (6..) by meas of equato (6..) so that W δ 4/ /. (6..4) The frst sub-dvso to sub-trasforms volvg the eve ad odd terms the seres specfed by equato (6..) s F k δ(f k + Q k F k ). (6..5) The sub-trasforms specfed by equato (6..5) ca be further subdvded so that F F k k ( F ( F k k + Q F k + Q F k k k ) ). (6..6) 7

6 @ Least Squares Fgure 6. compares the dscrete Fourer trasform of the fucto e - wth the cotuous trasform for the full fte terval. The oscllatory ature of the dscrete trasform largely results from the small umber of pots used to represet the fucto ad the trucato of the fucto at t ±. The oly pots the dscrete trasform that are eve defed are deoted by, the dashed le s oly provded to gude the reader's eye to the et pot. The fal geerato of sub-dvso yelds F ( F where F F F k k k k ( F ( F ( F k k k k Q k k k + Q F N + Q F + Q F + Q F k (e k k k k f f (t ) k ) f ) f ) f ) f πk / N N / ( ) ) + Q f k + Q f k + Q f k + Q f k 5 7 4 6, (6..7). (6..8) 7

Numercal Methods ad Data Aalyss Here we have used the "bt-reversal" of the bary superscrpt of the fal sub-trasforms to detfy whch of the data pots f(t ) correspod to the respectve oe-pot trasforms. The umercal detals of the calculatos specfed by equatos (6..5) - (6..8) are summarzed Table 6.. Here we have allowed k to rage from 8 geeratg a odd umber of resultat aswers. However, the values for k ad k 8 are detcal due to the perodcty of the fucto. Whle the symmetry of the tal fucto f(t ) demads that the resultat trasform be real ad symmetrc, some of the sub-trasforms may be comple. Ths ca be see table 6. the values of F y,,5,7. They subsequetly cacel, as they must, the fal trasform F k, but ther presece ca affect the values for the real part of the trasform. Therefore, comple arthmetc must be used throughout the calculato. As was already metoed, the sub-trasforms become more rapdly perodc as a fucto of k so that fewer ad fewer terms eed be eplctly kept as the subdvso process proceeds. We have dcated ths by hghlghtg the umbers table 6. that must be calculated. Whle the tabular umbers represet values that would be requred to evaluate equato (6..) for ay specfc value of k, we may use the repettve ature of the sub-trasforms whe calculatg the Fourer trasform for all values of k. The hghlghted umbers of table 6. are clearly far fewer that N cofrmg the result mpled by equato (6..) that Nlog N operatos wll be requred to calculate that dscrete Fourer trasform. Whle the savg s qute otceable for N 8, t becomes moumetal for large N. The curous wll have otced that the sequece of values for z k does ot correspod wth the values of t. The reaso s that the partcular values of k that are used are somewhat arbtrary as the Fourer trasform ca always be shfted by e πm/n correspodg to a shft k by +m. Ths smply moves o to a dfferet phase of the perodc fucto F(z). Thus, our tabular values beg wth the ceter pot z, ad moves to the ed value of + before startg over at the egatve ed value of -.75 (ote that - s to be detfed wth + due to the perodcty of F k ). Whle ths cyclcal ragg of k seems to provde a edless set of values of F k, there are oly N dstctly dfferet values because of the perodc behavor of F k. Thus our orgal statemet about the ature of the dscrete Fourer trasform - that t s defed oly at a dscrete set of pots - remas true. As wth most subects ths book, there s much more to Fourer aalyss tha we have developed here. We have ot dscussed the accuracy of such aalyss ad ts depedece o the samplg or amout of the tal data. The oly suggesto for dealg wth data mssg from a equally spaced set was to terpolate the data. Aother popular approach s to add a "fake" pece of data wth f(t ) o the grouds that t makes o drect cotrbuto to the sums equato (6..8). Ths s a deceptvely dagerous argumet as there s a mplct assumpto as to the form of the fucto at that pot. Iterpolato, as log as t s ot ecessve, would appear to be a better approach. 74

6 @ Least Squares Table 6. Calculatos for a Sample Fast Fourer Trasform K f k F k f Fk f4 Fk f F k f6 Fk f F k f5 Fk f Fk f7.5.5..678.678..665.665..5.5..678.678..665.665..5.5..678.678..665.665..5.5..678.678..665.665. 4.5.5..678.678..665.665. 5.5.5..678.678..665.665. 6.5.5..678.678..665.665. 7.5.5..678.678..665.665. 8.5.5..678.678..665.665. k Q k F k F k F k F k Q k F k F k Q k F k z k +.5.75.896.896 +.87.659 +.7648. - -.8647. -.84 -.84 + -.8647 -.84 ( + ) / -.7.5. +.84 +.5.75.896.896 -.4. +..5 - -.8647. -.84 -.84 - -.8647 -.84 ( ) / -.6.75. -.84 4 +.5.75.896.896 +.87.659 -.56. 5 - -.8647. -.84 -.84 + -.8647 -.84 ( + ) / -.6 -.75. +.84 6 +.5.75.896.896 -.4. -. -.5 7 - -.8647. -.84 -.84 - -.8647 -.84 ( ) / -.7 -.5. -.84 8 +.5.75.896.896 +.87.659 +.7648. 75

Numercal Methods ad Data Aalyss 6. Error Aalyss for Lear Least-Squares Whle Fourer aalyss ca be used for basc umercal aalyss, t s most ofte used for observatoal data aalyss. Ideed, the wdest area of applcato of least squares s probably the aalyss of observatoal data. Such data s trscally flawed. All data, whether t results from drect observato of the atural world or from the observato of a carefully cotrolled epermet, wll cota errors of observato. The equpmet used to gather the formato wll have characterstcs that lmt the accuracy of that formato. Ths s ot smply poor egeerg, but at a very fudametal level, the observg equpmet s part of the pheomeo ad wll dstort the epermet or observato. Ths, at least, s the vew of moder quatum theory. The ablty to carry out precse observatos s a lmt mposed by the very ature of the physcal world. Sce moder quatum theory s the most successful theory ever devsed by ma, we should be mdful of the lmts t mposes o observato. However, few epermets ad observatoal equpmet approach the error lmts set by quatum theory. They geerally have ther accuracy set by more practcal aspects of the research. Nevertheless observatoal ad epermetal errors are always wth us so we should uderstad ther mpact o the results of epermet ad observato. Much of the remag chapters of the book wll deal wth ths questo greater detal, but for ow we shall estmate the mpact of observatoal errors o the parameters of least square aalyss. We shall gve ths developmet some detal for t should be uderstood completely f the formalsm of least squares s to be used at all. a. Errors of the Least Square Coeffcets Let us beg by assumg that the appromatg fucto has the geeral lear form of equato (6..4). Now we wll assume that each observato Y has a uspecfed error E assocated wth t whch, f kow, could be corrected for, yeldg a set of least square coeffcets a. However, these are ukow so that our least square aalyss actually yelds the set of coeffcets a. If we kew both sets of coeffcets we could wrte E Y a φ ( ). (6..) ε φ Y a ( ) Here ε s the ormal resdual error resultg from the stadard least square soluto. I performg the least square aalyss we weghted the data by a amout ω so that N ( ω ε ) Mmum. (6..) We are terested the error a resultg from the errors E Y so let us defe δa a a. (6..) 76

6 @ Least Squares We ca multply the frst of equatos (6..) by ω φ k ( ), sum over, ad get N N ω φ ( ) φk ( ) ω Yφ k ( ) a ω φ ( )E, k,,,, (6..4) whle the stadard ormal equatos of the problem yeld N N N ω φ ( ) φk ( ) N N k a ω Y φ ( ), k,,,. (6..5) If we subtract equato (6..4) from equato (6..5) we get a epresso for δa. N N δ w φ ( ) φk ( ) δa A k N a w φ ( )E, k,,,. (6..6) Here we have replace ω wth w as secto [equato (6..6)]. These lear equatos are bascally the ormal equatos where the errors of the coeffcets δa have replaced the least square coeffcets a, ad the observatoal errors E have replace the depedet varable Y. If we kew the dvdual observatoal errors E, we could solve them eplctly to get N [A k ] k k k δa w φ ( ) E, (6..7) ad we would kow precsely how to correct our stadard aswers a to get the "true" aswers a. Sce we do ot kow the errors E, we shall have to estmate them terms of ε, whch at least s kowable. Ufortuately, relatg E to ε t wll be ecessary to lose the sg formato o δa. Ths s a small prce to pay for determg the magtude of the error. For smplcty let We ca the square equato (6..7) ad wrte ( δa ) k k C C N C w φ ( N N k p k p q k k C A -. (6..8) )E w w q p p q φ k C N ( ) φ ( p w q q φ p )E E ( q q )E q. (6..9) Here we have eplctly wrtte out the product as we wll edeavor to get rd of some of the terms by makg reasoable assumptos. For eample, let us specfy the maer whch the weghts should be chose so that ω E cost. (6..) Whle we do ot kow the value of E, practce, oe usually kows somethg about the epected error dstrbuto. The value of the costat equato (6..) does't matter sce t wll drop out of the ormal equatos. Oly the dstrbuto of E matters ad the data should be weghted accordgly. We shall further assume that the error dstrbuto of E s at-symmetrc about zero. Ths s a less ustfable assumpto ad should be carefully eamed all cases where the error aalyss for least squares 77

Numercal Methods ad Data Aalyss s used. However, ote that the dstrbuto eed oly be at-symmetrc about zero, t eed ot be dstrbuted lke a Gaussa or ormal error curve, sce both the weghts ad the product φ( ) φ( q ) are symmetrc ad q. Thus f we chose a egatve error, say, E q to be pared wth a postve error, say, E we get N N q q w w q φ ( ) φ ( k p q )E E q, k,,,, p,,,. (6..) Therefore oly terms where q survve equato (6..9) ad we may wrte t as N ( δa ) ( ωe) C k C p w φk ( ) φp ( ) ( ωe) C k C pa pk. (6..) k p k p Sce CA - [.e. equato (6..8)], the term large brackets o the far rght-had-sde s the Kroecker delta δ k ad the epresso for (δa ) smplfes to ( δa ) ( ωe) C kδ k ( ω k E) C. (6..) The elemets C are ust the dagoal elemets of the verse of the ormal equato matr ad ca be foud as a by product of solvg the ormal equatos. Thus the square error a s ust the mea weghted square error of the data multpled by the approprate dagoal elemet of the verse of the ormal equato matr. To produce a useful result, we must estmate (ω E). b. The Relato of the Weghted Mea Square Observatoal Error to the Weghted Mea Square Resdual If we subtract the secod of equatos (6..) from the frst, we get δa φ ( ) φ ( ) C k E ε w φ ( ) E. (6..4) k q Now multply by w ε ad sum over all. Re-arragg the summatos we ca wrte N N N N N w εe ε w ε δa φ ( ) C k w qφk ( q )E q w εφ ( ). (6..5) k q But the last term brackets ca be obtaed from the defto of least squares to be so that N w ε a N ε N N w ε a N φ N q ( )w ε k q q, (6..6) w E ε w ε. (6..7) Now multply equato (6..4) by w E ad sum over all. Aga rearragg the order of summato we get 78

6 @ Least Squares N w E N w E ε N N k q C k N w q w E φ δa ( ) φ ( k q φ )E ( q E ) C N k k w E φ ( ) φ ( k ), (6.,) where we have used equato (6..) to arrve at the last epresso for the rght had sde. Makg use of equato (6..) we ca further smplfy equato (6..8) to get N N ( ωe) w E ε ( ωe) C A ( ωe). (6..9) k Combg ths wth equato (6..7) we ca wrte N ( ωe) N ( ω ε ), N (6..) ad fally epress the error a [see equato (6..)] as C N ( δa ) ( ωε ). N (6..) Here everythg o the rght had sde s kow ad s a product of the least square soluto. However, to obta the ε 's we would have to recalculate each resdual after the soluto has bee foud. For problems volvg large quattes of data, ths would double the effort. c. Determg the Weghted Mea Square Resdual To epress the weghted mea square resdual equato (6..) terms of parameters geerated durg the tal soluto, cosder the followg geometrcal argumet. The φ ()'s are all learly depedet so they ca form the bass of a vector space whch the f(a, )'s ca be epressed (see fgure 6.). The values of f(a, ) that result from the least square soluto are a lear combato of the φ ( )'s where the costats of proportoalty are the a 's. However, the values of the depedet varable are also depedet of each other so that the legth of ay vector s totally ucorrelated wth the legth of ay other ad ts locato the vector space wll be radom [ote: the space s lear the a 's, but the compoet legths deped o φ ()]. Therefore the magtude of the square of the vector sum of the s wll grow as the square of the dvdual vectors. Thus, f F s the vector sum of all the dvdual vectors magtude s ust N k k f f the ts F f (a, ). (6..) The observed values for the depedet varable Y are geeral ot equal to the correspodg f(a, ) so they caot be embedded the vector space formed by the φ ( )'s. Therefore fgure 6. depcts them lyg above (or out of) the vector space. Ideed the dfferece betwee them s ust ε. Aga, the Y 's are 79

Numercal Methods ad Data Aalyss depedet so the magtude of the vector sum of the Y s ad the ε s s Y ε N N ε Y. (6..) Fgure 6. shows the parameter space defed by the φ ()'s. Each f(a, ) ca be represeted as a lear combato of the φ ( ) where the a are the coeffcets of the bass fuctos. Sce the observed varables Y caot be epressed terms of the φ ( ), they le out of the space. 8

6 @ Least Squares Sce least squares seeks to mmze Σε, that wll be accomplshed whe the tp of Y les over the tp of F so that ε s perpedcular to the φ() vector space. Thus we may apply the theorem of Pythagoras ( -dmesos f ecessary) to wrte N N N w f (a, ) ε w Y w. (6..4) Here we have cluded the square weghts w as ther cluso o way chages the result. From the defto of the mea square resdual we have N N N N N w f (a, ) w ε ) w [Y f (a, )] w Y w Yf (a, ) + (, (6..5) whch f we combe wth equato (6..4) wll allow us to elmate the quadratc term f so that equato (6..) fally becomes C N N ( δa φ ) w Y a k w Y k ( ). (6..6) N k The term the square brackets o the far rght had sde s the costat vector of the ormal equatos. The the oly ukow term the epresso for δa s the scalar term [Σw Y ], whch ca easly be geerated durg the formato of the ormal equatos. Thus t s possble to estmate the effect of errors the data o the soluto set of least square coeffcets usg othg more tha the costat vector of the ormal equatos, the dagoal elemets of the verse matr of the ormal equatos, the soluto tself, ad the weghted sum squares of the depedet varables. Ths amouts to a trval calculato compared to the soluto of the tal problem ad should be part of ay geeral least square program. d. The Effects of Errors the Idepedet Varable Throughout the dscusso ths secto we have vestgated the effects of errors the depedet varable. We have assumed that there s o error the depedet varable. Ideed the least square orm tself makes that assumpto. The "best" soluto the least square sese s that whch mmzes the sum square of the resduals. Kowledge of the depedet varable s assumed to be precse. If ths s ot true, the real problems emerge for the least square algorthm. The geeral problem of ucorrelated ad ukow errors both ad Y has ever bee solved. There do est algorthms that deal wth the problem where the rato of the errors Y to those s kow to be a costat. They bascally volve a coordate rotato through a agle α ta(/y) followed by the regular aalyss. If the appromatg fucto s partcularly smple (e.g. a straght le), t may be possble to vert the defg equato ad solve the problem wth the role of depedet ad depedet varable terchaged. If the soluto s the same (allowg for the trasformato of varables) wth the formal errors of the soluto, the some cofdece may be gaed that a meagful soluto has bee foud. Should they dffer by more tha the formal error the the aalyss s approprate ad o weght should be attached to the soluto. Ufortuately, verso of all but the smplest problems wll geerally result a o-lear system of equatos f the verso ca be foud at all. So the et secto we wll dscuss how oe ca approach a least square problem where the ormal equatos are o-lear. 8

Numercal Methods ad Data Aalyss 6.4 No-lear Least Squares I geeral, the problem of o-lear least squares s fraught wth all the complcatos to be foud wth ay o-lear problem. Oe must be cocered wth the uqueess of the soluto ad the o-lear propagato of errors. Both of these basc problems ca cause great dffculty wth ay soluto. The smplest approach to the problem s to use the defto of least squares to geerate the ormal equatos so that N f (a, ) w [Y f (a, )],,,,. (6.4.) a These + o-lear equatos must the be solved by whatever meas oe ca fd for the soluto of o-lear systems of equatos. Usually some sort of fed-pot terato scheme, such as Newto- Raphso, s used. However, the error aalyss may become as bg a problem as the tal least square problem tself. Oly whe the basc equatos of codto wll gve rse to stable equatos should the drect method be tred. Sce oe wll probably have to resort to teratve schemes at some pot the soluto, a far more commo approach s to learze the o-lear equatos of codto ad solve them teratvely. Ths s geerally accomplshed by learzg the equatos the vcty of the aswer ad the solvg the lear equatos for a soluto that s closer to the aswer. The process s repeated utl a suffcetly accurate soluto s acheved. Ths ca be vewed as a specal case of a fed-pot terato scheme where oe s requred to be relatvely ear the soluto. I order to fd approprate startg values t s useful to uderstad precsely what we are tryg to accomplsh. Let us regard the sum square of the resduals as a fucto of the regresso coeffcets a so that N N [Y f (a, )] w ε χ (a ) w. (6.4.) For the momet, we shall use the short had otato of χ to represet the sum square of the resduals. Whle the fucto f(a,) s o loger lear the a 's they may be stll regarded as depedet ad therefore ca serve to defe a space whch χ s defed. Our o-lear least square problem ca be geometrcally terpreted to be fdg the mmum the χ hypersurface (see fgure 6.). If oe has o pror kowledge of the locato of the mma of the χ surface, t s best to search the space wth a coarse multdmesoal grd. If the umber of varables a s large, ths ca be a costly search, for f oe pcks m values of each varable a, oe has m fuctoal evaluatos of equato (6.4.) to make. Such a search may ot locate all the mma ad t s ulkely to deftvely locate the deepest ad therefore most desrable mmum. However, t should detfy a set(s) of parameters wll fd the true mmum. a k from whch oe of the followg schemes We wll cosder two basc approaches to the problem of locatg these mma. There are others, but they are ether logcally equvalet to those gve here or very closely related to them. Bascally we shall assume that we are ear the true mmum so that frst order chages to the soluto set a k wll lead us to that mmum. The prmary dffereces the methods are the maer by whch the equatos are formulated. 8

6 @ Least Squares a. The Method of Steepest Descet A reasoable way to approach the problem of fdg a mmum χ -space would be to chage the values of a so that oe s movg the drecto, whch yelds the largest chage the value of χ. Ths wll occur the drecto of the gradet of the surface so that χ χ a N χ a χ (a â + a a ) χ (a ). (6.4.) We ca calculate ths by makg small chages a the parameters ad evaluatg the compoets of the gradet accordace wth the secod of equatos (6.4.). Alterately, we ca use the defto of least squares ad calculate N χ f (a, ) χ w [Y f (a, )]. (6.4.4) a a If the fucto f(a,) s ot too complcated ad has closed form dervatves, ths s by far the preferable maer to obta the compoets of χ. However, we must eercse some care as the compoets of χ are ot dmesoless. I geeral, oe should formulate a umercal problem so that the uts do't get the way. Ths meas ormalzg the compoets of the gradet some fasho. For eample we could defe [a χ / χ ] a χ ξ, (6.4.5) a χ / χ a χ whch s a sort of ormalzed gradet wth ut magtude. The et problem s how far to apply the gradet obtag the et guess, A coservatve possblty s to use a from equato (6.4.) so that δa a /ξ. (6.4.6) I order to mmze computatoal tme, the drecto of the gradet s usually mataed utl χ begs to crease. The t s tme to re-evaluate the gradet. Oe of the dffcultes of the method of steepest descet s that the values of the gradet of χ vash as oe approaches a mmum. Therefore the method becomes ustable as oe approaches the aswer the same maer ad for the same reasos that Newto- Raphso fed-pot terato became ustable the vcty of multple roots. Thus we shall have to fd aother approach. 8

Numercal Methods ad Data Aalyss Fgure 6. shows the χ hypersurface defed o the a space. The o-lear least square seeks the mmum regos of that hypersurface. The gradet method moves the terato the drecto of steepest decet based o local values of the dervatve, whle surface fttg tres to locally appromate the fucto some smple way ad determes the local aalytc mmum as the et guess for the soluto. b. Lear appromato of f(a,) Let us cosder appromatg the o-lear fucto f(a,) by a Taylor seres a. To the etet that we are ear the soluto, ths should yeld good results. A mult-varable epaso of f(a,) aroud the preset values a of the least square coeffcets s f (a k, ) f (a, ) f (a, ) + δa k. (6.4.7) a If we substtute ths epresso for f(a,) to the defto for the sum-square resdual χ, we get χ k N N f (a, ) w δa k. (6.4.8) k a k [Y f (a, )] w Y f (a, ) k Ths epresso s lear δa so we ca use the regular methods of lear least squares to wrte the ormal 84

6 @ Least Squares equatos as N χ f (a, ) f (a, ) w Y f (a, ) δa k, p,,, δa p k a k a p whch ca be put the stadard form of a set of lear algebrac equatos for δa k so that, (6.4.9) k A B δa A kp p k N N kp B, p p,,, f (a, ) f (a, ) w, k,,,, p,,, a a k w [Y f (a p f (a, ), )], p,,, a p. (6.4.) The dervatve of f(a,) that appears equatos (6.4.9) ad (6.4.) ca ether be foud aalytcally or umercally by fte dffereces where f (a, ) f[a,(a p + a p ), ] f (a,a p, ). (6.4.) a a p Whle the equatos (6.4.) are lear δa k, they ca be vewed as beg quadratc a k. Cosder ay epaso of a k terms of χ such as a k q + q χ + q χ 4. (6.4.) The varato of a k wll the have the form δa k q + q χ, (6.4.) whch s clearly lear χ. Ths result therefore represets a parabolc ft to the hypersurface χ wth the codto that δa k s zero at the mmum value of χ. The soluto of equatos (6.4.) provdes the locato of the mmum of the χ hypersurface to the etet that the mmum ca locally be well appromated by a parabolc hypersurface. Ths wll certaly be the case whe we are ear the soluto whch s precsely where the method of steepest descet fals. It s worth otg that the costat vector of the ormal equatos s ust half of the compoets of the gradet gve equato (6.4.4). Thus t seems reasoable that we could combe ths approach wth the method of steepest descet. Oe approach to ths s gve by Marquardt 4. Sce we were somewhat arbtrary about the dstace we would follow the gradet a sgle step we could modfy the dagoal elemets of equatos (6.4.) so that A' kk A kk ( + λ), k,,,. (6.4.4) A' kp A kp, k p p Clearly as λ creases, the soluto approaches the method of steepest descet sce 85

Numercal Methods ad Data Aalyss Lm δa k B k /λa kk. (6.4.5) λ All that remas s to fd a algorthm for choosg λ. For small values of λ, the method approaches the frst order method for δa k. Therefore we wll choose λ small (say about - ) so that the δa k 's are gve by the soluto to equatos (6.4.). We ca use that soluto to re-compute χ. If χ (a + δa) > χ (a), (6.4.6) the crease λ by a factor of ad repeat the step. However, f codto (6.4.6) fals ad the value of χ s decreasg, the decrease λ by a factor of, adopt the ew values of a k ad cotue. Ths allows the aalytc fttg procedure to be employed where t works the best - ear the soluto, ad utlzes the method of steepest descet where t wll gve a more relable aswer - well away from the mmum. We stll must determe the accuracy of our soluto. c. Errors of the Least Squares Coeffcets The error aalyss for the o-lear case turs out to be credbly smple. True, we wll have to make some addtoal assumptos to those we made secto 6., but they are reasoable assumptos. Frst, we must assume that we have reached a mmum. Sometmes t s ot clear what costtutes a mmum. For eample, f the mmum χ hyperspace s descrbed by a valley of uform depth, the the soluto s ot uque, as a wde rage of oe varable wll mmze χ. The error ths varable s large ad equal at least to the legth of the valley. Whle the method we are suggestg wll gve relable aswers to the formal errors for a whe the appromato accurately matches the χ hypersurface, whe t does ot the errors wll be urelable. The error estmate reles o the learty of the appromatg fucto δa. I the vcty of the χ mmum δa a a. (6.4.7) For the purposes of the lear least squares soluto that produces δa, the tal value a s a costat devod of ay error. Thus whe we arrve at the correct soluto, the error estmates for δa wll provde the estmate for the error a tself sce (δa ) a [a ] a. 6.4.8) Thus the error aalyss we developed for lear least squares secto 6. wll apply here to fdg the error estmates for δa ad hece for a tself. Ths s oe of the vrtues of teratve approaches. All past ss are forgotte at the ed of each terato. Ay terato scheme that coverges to a fed-pot s some real sese a good oe. To the etet that the appromatg fucto at the last step s a accurate represetato of the χ hypersurface, the error aalyss of the lear least squares s equvalet to dog a frst order perturbato aalyss about the soluto for the purposes of estmatg the errors the coeffcets represetg the coordates of the hyperspace fucto. As we saw secto 6., we ca carry out that error aalyss for almost o addtoal computg cost. Oe should keep md all the caveats that apply to the error estmates for o-lear least squares. They are accurate oly as log as the appromatg fucto fts the hyperspace. The error dstrbuto of the depedet varable s assumed to be at-symmetrc. I the evet that all the codtos are met, the 86

6 @ Least Squares errors are ust what are kow as the formal errors ad should be take to represet the mmum errors of the parameters. 6.5 Other Appromato Norms Up to ths pot we have used the Legedre Prcple of Least Squares to appromate or "ft" our data pots. As log as ths dealt wth epermetal data or other forms of data whch cotaed trsc errors, oe could ustfy the Least Square orm o statstcal grouds (as log as the error dstrbuto met certa crtera). However, cosder the stuato where oe desres a computer algorthm to geerate, say, s() over some rage of such as π/4. If oe ca maage ths, the from multple agle formulae, t s possble to geerate s() for ay value of. Sce at a very basc level, dgtal computers oly carry out arthmetc, oe would eed to fd some appromatg fucto that ca be computed arthmetcally to represet the fucto s() accurately over that terval. A crtero that requred the average error of computato to be less tha ε s ot acceptable. Istead, oe would lke to be able to guaratee that the computatoal error would always be less tha ε ma. A appromatg orm that wll accomplsh ths s kow as the Chebyschev orm ad s sometmes called the "m-ma" orm. Let us defe the mamum value of a fucto h() over some rage of to be h ma Ma h() allowed. (6.5.) Now assume that we have a fucto Y() whch we wsh to appromate by f(a,) where a represets a set of free parameters that may be adusted to provde the "best" appromato some sese. Let h() be the dfferece betwee those two fuctos so that h() ε() Y() f(a,). (6.5.) The least square appromato orm would say that the "best" set of a 's s foud from M ε ()d. (6.5.) However, a appromatg fucto that wll be the best fucto for computatoal appromato wll be better gve by M h ma M ε ma M Ma Y()-f(a,). (6.5.4) A set of adustable parameters a that are obtaed by applyg ths orm wll guaratee that ε() ε ma, (6.5.5) ad that ε ma s the smallest possble value that ca be foud for the gve fucto f(a,). Ths guaratees the vestgator that ay umercal evaluato of f() wll represet Y() wth a amout ε ma. Thus, by mmzg the mamum error, oe has obtaed a appromato algorthm of kow accuracy throughout the etre rage. Therefore ths s the appromato orm used by those who geerate hgh qualty fuctoal subroutes for computers. Ratoal fuctos are usually employed for such computer algorthms stead of ordary polyomals. However, the detaled mplemetato of the orm for determg the free parameters appromatg ratoal fuctos s well beyod the scope of ths book. Sce we have emphaszed polyomal appromato throughout ths book, we wll dscuss the mplemetato of ths orm wth polyomals. 87

Numercal Methods ad Data Aalyss a. The Chebyschev Norm ad Polyomal Appromato Let our appromatg fucto f(a,) be of the form gve by equato (..) so that, ) a φ () f (a. (6.5.6) The choce of f(a,) to be a polyomal meas that the free parameters a wll appear learly ay aalyss. So as to facltate comparso wth our earler approaches to polyomal appromato ad least squares, let us choose φ to be ad we wll attempt to mmze ε ma () over a dscrete set of pots. Thus we wsh to fd a set of a so that M( ε ) ma M Y a ma. (6.5.7) Sce we have (+) free parameters, a, we wll eed at least N + pots our dscrete set. Ideed, f + N the we ca ft the data eactly so that ε ma wll be zero ad the a 's could be foud by ay of the methods chapter. Cosder the more terestg case where N >> +. For the purposes of a eample let us cosder the cases where, ad. For the appromatg fucto s a costat, represeted by a horzotal le Fgure 6.4 Fgure 6.4 shows the Chebyschev ft to a fte set of data pots. I pael a the ft s wth a costat a whle pael b the ft s wth a straght le of the form f() a +a. I both cases, the adustmet of the parameters of the fucto ca oly produce (+) mamum errors for the (+) free parameters. By adustg the horzotal le up or dow fgure 6.a we wll be able to get two pots to have the same largest value of ε wth oe chage sg betwee them. For the straght le Fgure 6.b, we wll be able to adust both the slope ad tercept of the le thereby makg the three largest values of ε the same. Amog the etreme values of ε there wll be at least two chages sg. I geeral, as log as N > (+), oe ca adust the parameters a so that there are + etreme values of ε all equal to ε ma ad there 88

6 @ Least Squares wll be (+) chages of sg alog the appromatg fucto. I addto, t ca be show that the a 's wll be uque. All that remas s to fd them. b. The Chebyschev Norm, Lear Programmg, ad the Smple Method Let us beg our search for the "best" set of free-parameters a by cosderg a eample. Sce we wll try to show graphcally the costrats of the problem, cosder a appromatg fucto of the frst degree whch s to appromate three pots (see fgure 6.b). We the desre Y (a + a) ε ma Y + ε (a a ) ma. (6.5.8) Y (a + a ) ε ma ε ε ma M ma Fgure 6.5 shows the parameter space for fttg three pots wth a straght le uder the Chebyschev orm. The equatos of codto deote half-plaes whch satsfy the costrat for oe partcular pot. These costrats costtute the basc mmum requremets of the problem. If they were to be plotted parameter space (see Fgure 6.4), they would costtute sem-plaes bouded by the le for ε. The half of the sem-plae that s permtted would be determed by the sg of ε. However, we have used the result 89

Numercal Methods ad Data Aalyss from above that there wll be three etreme values for ε all equal to ε ma ad havg opposte sg. Sce the value of ε ma s ukow ad the equato ( geeral) to whch t s attached s also ukow, let us regard t as a varable to be optmzed as well. The sem-plaes represetg the costrats are ow eteded out of the a -a plae the drecto of creasg ε ma wth the sem-plaes of the costrats formg a verted rregular pyramd. The varato of the sg of ε ma guaratees that the plaes wll tersect to form a cove sold. The soluto to our problem s trval, as the lower verte of the pyramd represets the mmum value of the mamum error, whch wll be the same for each costrat. However, t s ce that the method wll tell us that wthout t beg cluded the specfcato of the problem. Sce the umber of etrema for ths problem s +, ths s a epected result. The cluso of a ew pot produces a addtoal sem-costrat plae whch wll tersect the pyramd producg a tragular upper base. The mmum value of the mamum error wll be foud at oe of the vertces of ths tragle. However sce the verte wll be defed by the tersecto of three les, there wll stll be three etrema as s requred by the degree of the appromatg polyomal. Addtoal pots wll crease the umber of sdes as they wll cut the tal pyramd formg a mult-sded polygo. The vertces of the polygo that s defed parameterε ma space wll stll hold the optmal soluto. I ths stace the search s smple as we smply wsh to kow whch ε ma s the smallest magtude. Thus we look for the verte earest the plae of the parameters. A crease the umber of ukows a 's wll produce fgures hgher dmesos, but the aalyss remas essetally the same. The area of mathematcs that deals wth problems that ca be formulated term of lear costrats (cludg equaltes) s kow as Lear Programmg ad t has othg to do wth computer programmg. It was the outgrowth of a group of mathematcas workg a broader area of mathematcs kow as operatos research. The sprato for ts developmet was the fdg of solutos to certa optmzato problems such as the effcet allocato of scarce resources (see Blad 4 ). Lke may of the subects we have troduced ths book, lear programmg s a large feld of study havg may ramfcatos far beyod the scope of ths book. However, a problem that s formulated terms of costrat equaltes wll cosst of a collecto of sem-spaces that defe a polytope (a fgure where each sde s a polygo) multdmesoal parameter space. It ca be show that the optmum soluto les at oe of the vertces of the polytope. A method for sequetally testg each verte so that the optmal oe wll be foud a determstc way s kow as the smple method. Startg at a arbtrary verte oe vestgates the adacet vertces fdg the oe whch best satsfes the optmal codtos. The remag vertces are gored ad oe moves to the ew "optmal" verte ad repeats the process. Whe oe ca fd o adacet vertces that better satsfy the optmal codto that verte s the most optmal of the etre polytope ad represets the optmal soluto to the problem. I practce, the smple method has bee foud to be far more effcet tha geeral theoretcal cosderatos would lead oe to epect. So, whle there are other approaches to lear programmg problems, the oe that stll attracts most atteto s the smple method. c. The Chebyschev Norm ad Least Squares At the begg of ths chapter, we ustfed the choce of the Least Square appromato orm o the grouds that t yelded lear equatos of codto ad was the lowest power of the devato ε that was guarateed to be postve. What about hgher powers? The desre to keep the error costrats postve should lmt us to eve powers of ε. Thus cosder a orm of the form 9

6 @ Least Squares M Σ ε M Σ [Y -f(a, )], (6.5.9) whch lead to the o-lear equatos f (a, ) [ y f (a, )] [ y f (a, )]. (6.5.) a a Now oe could solve these o-lear equatos, but there s o reaso to epect that the soluto would be "better" ay real sese tha the least square soluto. However, cosder the lmt of equato (6.5.9) as. Lm( M Σ ε ) M( Lm Σ ε ) M ε ma. (6.5.) The soluto that s foud subect to the costrat that ε ma s a mmum wll be the same soluto that s obtaed whe ε ma s a mmum. Thus the lmt of the th orm as goes to fty s the Chebyschev orm. I ths chapter we have made a trasto from dscussg umercal aalyss where the basc puts to a problem are kow wth arbtrary accuracy tp those where the basc data cotaed errors. I earler chapters the oly errors that occur the calculato result from roud-off of arthmetc processes or trucato of the appromato formulae. However, secto 6. we allowed for the troducto of "flawed" puts, wth heret errors resultg from epermet or observato. Sce ay teracto wth the real world wll volve errors of observato, we shall sped most of the remader of the book dscussg the mplcato of these errors ad the maer by whch they ca be maaged. 9

Numercal Methods ad Data Aalyss Chapter 6 Eercses. Develop ormal equatos for the fuctos: a. f() a e a b. f() a + a s(a π + a ). Whch epressos could be replaced wth a lear fucto wth o loss of accuracy? What would the error aalyss of that fucto ft to observatoal data say about the errors of the orgal coeffcets a?. Usg least squares fd the "best" straght-le ft ad the error estmates for the slope ad tercept of that le for the followg set of data. Y.5..8 4 4. 5 4.9 6 6. 7 5. 8.5. Ft the followg table wth a polyomal of the form f(a,) Σ k φ k (), where φ k () cos(kπ) f(a, )...745.7.497.9.4888.757.689.4755.7854.4997..4494.8.445.95.6496.5.6959 How may terms are requred to ft the table accurately? Dscuss what you mea by "accurately" ad why you have chose that meag. 9

6 @ Least Squares 4. Gve the followg two sets of data to be ft by straght les., Y,, Y, 9..5 8.5. 7.6.5 4.5 4 4.6 5 4. 5 5. 6. 6 6.9 7. 7 6.8 fd the "best" value for the tersecto of the straght les ad a estmate for the error Y. How would you cofrm the assumpto that there s o error? 5. Determe the comple Fourer trasform of a. e -t - < t < +. b. e -t cos(t), < t < +. 6. Fd the FFT for the fuctos problem 5 where the fucto s sampled every. t ad the total umber of pots s 4. Calculate the verse trasform of the result ad compare the accuracy of the process. 9

Numercal Methods ad Data Aalyss Chapter 6 Refereces ad Supplemetary Readg. Batema, H., "Tables of Itegral Trasforms" (954) Ed. A. Erde ' ly, Volumes,, McGraw-Hll Book Co., Ic. New York, Toroto, Lodo.. Press, W.H., Flaery, B.P., Teukolsky, S.A., ad Vetterlg, W.T., "Numercal Recpes the Art of Scetfc Computg" (986), Cambrdge Uversty Press, Cambrdge, pp. 9-94.. Marquardt, D.W., "A Algorthm for Least-Squares Estmato of Nolear Parameters", (96), J. Soc. Id. Appl. Math., Vol., No., pp.4-44. 4. Blad, R.G., "The Allocato of Resources by Lear Programmg", (98) Sc. Amer. Vol. 44, #6, pp.6-44. Most books o umercal aalyss cota some referece to least squares. Ideed most freshme calculus courses deal wth the subect at some level. Ufortuately o sgle tet cotas a detaled descrpto of the subect ad ts ramfcatos.. Hldebrad, F.B., "Itroducto to Numercal Aalyss" (956) McGraw-Hll Book Co., Ic., New York, Toroto, Lodo, pp. 58-, Ths book presets a classcal dscusso ad much of my dscusso secto 6. s based o hs presetato. The error aalyss for o-lear least squares secto 6.4 s dealt wth cosderable detal. Bevgto, P.R., "Data Reducto ad Error Aalyss for the Physcal Sceces", (969), McGraw-Hll Book Co. Ic., New York, Sa Fracsco, St. Lous, Toroto, Lodo, Sydey, pp. 4-46. Nearly ay book that dscusses Fourer seres ad trasforms cotas useful formato elaboratg o the uses ad eteded theory of the subect. A eample would be. Sokolkoff, I.S., ad Redheffer, R.M., "Mathematcs of Physcs ad Moder Egeerg", (958) McGraw-Hll Book Co., Ic. New York, Toroto, Lodo, pp. 75-. Two books completely devoted to Fourer aalyss ad the trasforms partcularly are: 4. Brgham, E.O., "The Fast Fourer Trasform", (974) Pretce-Hall, Ic. Eglewood Clffs, N.J., ad 5. Bracewell, R.N., "The Fourer Trasform ad ts Applcatos", d Ed., (978), McGraw-Hll Book Compay, New York N.Y. 94

6 @ Least Squares A very compressed dscusso, of Lear Programmg, whch covers much more that we ca, s to be foud 6. Press, W.H., Flaery, B.P., Teukolsky, S.A., ad Vetterlg, W.T., "Numercal Recpes the Art of Scetfc Computg" (986), Cambrdge Uversty Press, Cambrdge. pp. 74-4, but a more basc dscusso s gve by 7. Gass, S.T., "Lear Programmg" (969), rd ed. McGraw-Hll, New York. 95

Numercal Methods ad Data Aalyss 96

7 Probablty Theory ad Statstcs I the last chapter we made the trasto from dscussg formato whch s cosdered to be error free to dealg wth data that cotaed trsc errors. I the case of the former, ucertates the results of our aalyss resulted from the falure of the appromato formula to match the gve data ad from roud-off error curred durg calculato. Ucertates resultg from these sources wll always be preset, but addto, the basc data tself may also cota errors. Sce all data relatg to the real world wll have such errors, ths s by far the more commo stuato. I ths chapter we wll cosder the mplcatos of dealg wth data from the real world more detal. 97

Numercal Methods ad Data Aalyss Phlosophers dvde data to at least two dfferet categores, observatoal, hstorcal, or emprcal data ad epermetal data. Observatoal or hstorcal data s, by ts very ature, o-repeatable. Epermetal data results from processes that, prcple, ca be repeated. Some have troduced a thrd type of data labeled hypothetcal-observatoal data, whch s based o a combato of observato ad formato suppled by theory. A eample of such data mght be the dstace to the Adromeda galay sce a drect measuremet of that quatty has yet to be made ad must be deduced from other aspects of the physcal world. However, the last aalyss, ths s true of all observatos of the world. Eve the determato of repeatable, epermetal data reles o agreed covetos of measuremet for ts uque terpretato. I addto, oe may valdly ask to what etet a epermet s precsely repeatable. Is there a fudametal dfferece betwee a epermet, whch ca be repeated ad successve observatos of a pheomeo that apparetly does't chage? The oly dfferece would appear to be that the scetst has the opto the case of the former repeatg the epermet, whle the latter case he or she s at the mercy of ature. Does ths costtute a fudametal dfferece betwee the sceces? The hard sceces such as physcs ad chemstry have the luury of beg able to repeat epermets holdg mportat varables costat, thereby ledg a certa level of certaty to the outcome. Dscples such as Socology, Ecoomcs ad Poltcs that deal wth the huma codto geerally preclude epermet ad thus must rely upo observato ad "hstorcal epermets" ot geerally desged to test scetfc hypotheses. Betwee these two etremes are sceces such as Geology ad Astroomy whch rely largely upo observato but are fouded drectly upo the epermetal sceces. However, all sceces have commo the gatherg of data about the real world. To the aalyst, there s lttle dfferece ths data. Both epermetal ad observatoal data cota trsc errors whose effect o the sought for descrpto of the world must be uderstood. However, there s a maor dfferece betwee the physcal sceces ad may of the socal sceces ad that has to do wth the oto of cause ad effect. Perhaps the most mportat cocept drvg the physcal sceces s the oto of causalty. That s the physcal bologcal, ad to some etet the behavoral sceces, have a clear oto that evet A causes evet B. Thus, testg a hypothess, t s always clear whch varables are to be regarded as the depedat varables ad whch are to be cosdered the depedet varables. However, there are may problems the socal sceces where ths luury s ot preset. Ideed, t may ofte be the case that t s ot clear whch varables used to descrbe a comple pheomeo are eve related. We shall see the fal chapter that eve here there are some aalytcal techques that ca be useful decdg whch varables are possbly related. However, we shall also see that these tests do ot prove cause ad effect, rather they smply suggest where the vestgator should look for causal relatoshps. I geeral data aalyss may gude a vestgator, but caot substtute for hs or her sght ad uderstadg of the pheomea uder vestgato. Durg the last two cetures a steadly creasg terest has developed the treatmet of large quattes of data all represetg or relatg to a much smaller set of parameters. How should these data be combed to yeld the "best" value of the smaller set of parameters? I the tweteth cetury our ablty to collect data has grow eormously, to the pot where collatg ad sytheszg that data has become a scholarly dscple tself. May academc sttutos ow have a etre departmet or a academc ut devoted to ths study kow as statstcs. The term statstcs has become almost geerc the laguage as t ca stad for a umber of rather dfferet cocepts. Occasoally the collected data tself ca be referred to as statstcs. Most have heard the referece to reckless operato of a motor vehcle leadg to the operator "becomg a statstc". As we shall see, some of the quattes that we wll develop to represet large 98

7 @ Probablty Theory ad Statstcs amouts of data or characterstcs of that data are also called statstcs. Fally, the etre study of the aalyss of large quattes of data s referred to as the study of statstcs. The dscple of statstcs has occasoally bee defed as provdg a bass for decso-makg o the bass of complete or mperfect data. The defto s ot a bad oe for t hghlghts the breadth of the dscple whle emphaszg t prmary fucto. Nearly all scetfc eterprses requre the vestgator to make some sort of decsos ad as ay epermeter kows, the data s always less tha perfect. The subect has ts orgs the late 8th ad early 9th cetury astroomcal problems studed by Gauss ad Legedre. Now statstcal aalyss has spread to early every aspect of scholarly actvty. The developg tools of statstcs are used the epermetal ad observatoal sceces to combe ad aalyze data to test theores of the physcal world. The socal ad bologcal sceces have used statstcs to collate formato about the habtats of the physcal world wth a eye to uderstadg ther future behavor terms of ther past performace. The samplg of publc opo has become a drvg fluece for publc polcy the coutry. Whle the market ecoomes of the world are largely self-regulatg, cosderable effort s employed to "gude" these ecoomes based o ecoomc theory ad data cocerg the performace of the ecoomes. The commercal world allocates resources ad develops plas for growth based o the statstcal aalyss of past sales ad surveys of possble future demad. Moder medce uses statstcs to ascerta the effcacy of drugs ad other treatmet procedures. Such methods have bee used, ot wthout cotroversy, to dcate ma made hazards our evromet. Eve the study of laguage, statstcal aalyss has bee used to decde the authorshp of documets based o the frequecy of word use as a characterstc of dfferet authors. The hstorcal developmet of statstcs has see the use of statstcal tools may dfferet felds log before the bass of the subect were codfed the aomatc foudatos to whch all scece aspres. The result s that smlar mathematcal techques ad methods took o dfferet desgatos. The multdscple developmet of statstcs has lead to a ucommoly large amout of argo. Ths argo has actually become a maor mpedmet to uderstadg. There seems to have bee a predlecto, certaly the eteeth cetury, to dgfy shaky cocepts wth gradose labels. Thus the argo statstcs teds to have a ecessvely pretetous soud ofte stemmg from the dscple where the partcular form of aalyss was used. For eample, durg the latter quarter of the eteeth cetury, Sr Fracs Galto aalyzed the heght of chldre terms of the heght of ther parets. He foud that f the average heght of the parets departed from the geeral average of the populato by a amout, the the average heght of the chldre would depart by, say, / from the average for the populato. Whle the specfc value of the fracto (/) may be dsputed all ow agree that t s less tha oe. Thus we have the observato that departures from the populato average of ay sub group wll regress toward the populato average subsequet geeratos. Sr Fracs Galto used Legedre's Prcple of Least Squares to aalyze hs data ad determe the coeffcet of regresso for hs study. The use of least squares ths fasho has become popularly kow as regresso aalyss ad the term s eteded to problems where the term regresso has absolutely o applcablty. However, so wde spread has the use of the term become, that falure to use t costtutes a barrer to effectve commucato. Statstcs ad statstcal aalyss are ubqutous the moder world ad o educated perso should veture to that world wthout some kowledge of the subect, ts stregths ad lmtatos. Aga we touch upo a subect that trasceds eve addtoal courses of qury to ecompass a lfetme of study. Sce we may preset oly a bare revew of some aspects of the subect, we shall ot attempt a hstorcal developmet. 99

Numercal Methods ad Data Aalyss Rather we wll beg by gvg some of the cocepts upo whch most of statstcs rest ad the developg some of the tools whch the aalyst eeds. 7. Basc Aspects of Probablty Theory We ca fd the coceptual orgs of statstcs probablty theory. Whle t s possble to place probablty theory o a secure mathematcal aomatc bass, we shall rely o the commoplace oto of probablty. Everyoe has heard the phrase "the probablty of sow for tomorrow 5%". Whle ths souds very quattatve, t s ot mmedately clear what the statemet meas. Geerally t s terpreted to mea that o days that have codtos lke those epected for tomorrow, sow wll fall o half of them. Cosder the case where studet A atteds a partcular class about three quarters of the tme. O ay gve day the professor could clam that the probablty of studet A attedg the class s 75%. However, the studet kows whether or ot he s gog to atted class so that he would state that the probablty of hs attedg class o ay partcular day s ether % or %. Clearly the probablty of the evet happeg s depedet o the pror kowledge of the dvdual makg the statemet. There are those who defe probablty as a measure of gorace. Thus we ca defe two evets to be equally lkely f we have o reaso to epect oe evet over the other. I geeral we ca say that f we have equally lkely cases ad ay m of them wll geerate a evet E, the the probablty of E occurrg s P(E) m/. (7..) Cosder the probablty of selectg a damod card from a deck of 5 playg cards. Sce there are damods the deck, the probablty s ust /5 ¼. Ths result dd ot deped o there beg 4 suts the stadard deck, but oly o the rato of 'correct' selectos to the total umber of possble selectos. It s always assumed that the evet wll take place f all cases are selected so that the probablty that a evet E wll ot happe s ust Q( ~ E) P(E). (7..) I order to use equato (7..) to calculate the probablty of evet E takg place, t s ecessary that we correctly eumerate all the possble cases that ca gve rse to the evet. I the case of the deck of cards, ths seems farly smple. However, cosder the tossg of two cos where we wsh to kow the probablty of two 'heads' occurrg. The dfferet possbltes would appear to be each co comg up 'heads', each co comg up 'tals', ad oe co comg up 'heads' whle the other s 'tals'. Thus aïvely oe would thk that the probablty of obtag two 'heads' would be /. However, sce the cos are truly depedet evets, each co ca be ether 'heads' or 'tals'. Therefore there are two separate cases where oe co ca be 'head' ad the other 'tals' yeldg four possble cases. Thus the correct probablty of obtag two 'heads' s /4. The set of all possble cases s kow as the sample set, or sample space, ad statstcs s sometmes referred to as the paret populato.

7 @ Probablty Theory ad Statstcs a. The Probablty of Combatos of Evets It s possble to vew our co tossg eve as two separate ad depedet evets where each co s tossed separately. Clearly the result of tossg each co ad obtag a specfc result s /. Thus the result of tossg two cos ad obtag a specfc result (two heads) wll be /4, or (/) (/). I geeral, the probablty of obtag evet E ad evet F, [P(EF)], wll be P(EF) P(E) P(F). (7..) Requrg of the occurrece of evet E ad evet F costtutes the use of the logcal ad whch always results a multplcatve acto. We ca ask what wll be the total, or ot, probablty of evet E or evet F occurrg. Should evets E ad F be mutually eclusve (.e. there are o cases the sample set that result both E ad F), the P(E or F) s gve by P(E or F) P(E) + P(F). (7..4) Ths use of addto represets the logcal 'or'. I our co tossg eercse obtag oe 'head' ad oe 'tal' could be epressed as the probablty of the frst co beg 'heads' ad the secod co beg 'tals' or the frst co beg 'tals' ad the secod co beg 'heads' so that P(HT) P(H)P(T) + P(T)P(H) (/) (/) + (/) (/) /. (7..5) We could obta ths drectly from cosderato of the sample set tself ad equato (7..) sce m, ad 4. However, more complcated stuatos the laws of combg probabltes eable oe to calculate the combed probablty of evets a clear ad uambguous way. I calculatg P(E or F) we requred that the evets E ad F be mutually eclusve ad the co eercse, we guarateed ths by usg separate cos. What ca be doe f that s ot the case? Cosder the stuato where oe rolls a de wth the covetoal s faces umbered through 6. The probablty of ay partcular face comg up s /6. However, we ca ask the questo what s the probablty of a umber less tha three appearg or a eve umber appearg. The cases where the result s less tha three are ad, whle the cases where the result s eve are, 4, ad 6. Naïvely oe mght thk that the correct aswer 5/6. However, these are ot mutually eclusve cases for the umber s both a eve umber ad t s also less tha three. Therefore we have couted twce for the oly dstct cases are,, 4, ad 6 so that the correct result s 4/6. I geeral, ths result ca be epressed as or the case of the de P(E or F) P(E) + P(F) P(EF), (7..6) P(< or eve) [(/6)+(/6)] + [(/6)+(/6)+(/6)] [(/) (/)] /. (7..7) We ca epress these laws graphcally by meas of a Ve dagram as fgure 7.. The smple sum of the depedet probabltes couts the tersecto o the Ve dagram twce ad therefore t must be removed from the sum.

Numercal Methods ad Data Aalyss Fgure 7. shows a sample space gvg rse to evets E ad F. I the case of the de, E s the probablty of the result beg less tha three ad F s the probablty of the result beg eve. The tersecto of crcle E wth crcle F represets the probablty of E ad F [.e. P(EF)]. The uo of crcles E ad F represets the probablty of E or F. If we were to smply sum the area of crcle E ad that of F we would double cout the tersecto. b. Probabltes ad Radom Varables We ca defe a radom process as oe where the result of the process caot be predcted. For eample, the toss of a co wll produce ether 'heads' or 'tals', but whch wll occur as the result of flppg the cos caot be predcted wth complete certaty. If we assg a to 'heads' ad a to 'tals', the a successo of co flps wll geerate a seres of 's ad 's havg o predctable order. If we regard a fte sequece of flps as a bary umber, the we ca call t a radom umber sce ts value wll ot be predctable. Ay secesso of fte sequeces of the same legth wll produce a secesso of radom bary umbers where o umber ca be predcted from the earler umbers. We could carry out the same epermet wth the de where the results would rage from to 6 ad the sequeces would form base s radom umbers. Now the sequece that produces our radom umber could be of arbtrary legth eve though the sample set s fte, but t wll always have some umercal value. We ca defe a radom varable as ay umercally valued fucto that s defed o the sample set. I the case we have pcked, t could be, say, all umbers wth fve dgts or less. Let us defe the elemets of the sample set to have umercal values. I the case of the co these would be the 's ad 's we assged to 'heads' ad 'tals'. For the de, they are smply the values of the faces. The ay radom varable, whch would appear through ts defto as a

7 @ Probablty Theory ad Statstcs radom process, would have a result X ( ) X that depeds o the values of the sample set. The probablty P that ay partcular value of X wll appear wll deped o the probabltes p assocated wth the values that produce the umercal value of the radom varable X. We could the ask "If we geerate values of the radom varable X from the sample set, what s the most lkely value of X that we should epect?". We wll call that value of X the epected or epectato value of X ad t wll be gve by N E (X) P X. (7..8) Cosder the smple case of tossg cos ad ask "What s the epectato value for obtag oe 'head' ay gve tral of tossg the two cos?". The possbltes are that both cos could tur up 'tals' yeldg o 'heads', or oe co could be 'heads' ad the other 'tals', or both could be 'heads'. The probabltes of the frst ad last occurrg s ¼, but sce ether co ca be 'heads' whle the other s 'tals' the mddle possblty represets two separate cases. Thus the epected value for the umber of 'heads' s ust E(H) (¼) + (¼) + (¼) + (¼). (7..9) The frst term s made up of the umber of heads that result for each tral tmes the probablty of that tral whle the other represetato of that sum show the dstctly dfferet values of X multpled by the combed probablty of those values occurrg. The result s that we may epect oe 'head' wth the toss of two cos. The epectato value of a radom varable s sort of a average value or more properly the most lkely value of that varable. c. Dstrbutos of Radom Varables It s clear from our aalyss of the co tossg epermet that ot all values of the radom varable (eg. the umber of 'heads') are equally lkely to occur. Epermets that yeld oe 'head' are twce as lkely to occur as ether o 'heads' or two 'heads'. The frequecy of occurrece wll smply be determed by the total probablty of the radom varable. The depedece of the probablty of occurrece o the value of the radom varable s called a probablty dstrbuto. I ths stace there s a rather lmted umber of possbltes for the value of the radom varable. Such cases are called dscrete probablty dstrbutos. If we were to defe our radom varable to be the value epected from the roll of two dce, the the values could rage from -, ad we would have a more etesve dscrete probablty dstrbuto. I geeral, measured values cota a fte set of dgts for the radom varables ad ther probablty dstrbutos are always dscrete. However, t s useful to cosder cotuous radom varables as they are easer to use aalytcally. We must be careful our defto of probablty. We ca follow the stadard practce of lmts used the dfferetal calculus ad defe the dfferetal probablty of the cotuous radom varable occurrg wth the terval betwee ad + to be dp() Lmt [f(+ )-f()]/. (7..)

Numercal Methods ad Data Aalyss Thus the probablty that the value of the radom varable wll le betwee a ad b wll be P(a, b) b a f ()d. (7..) The fucto f() s kow as the probablty desty dstrbuto fucto whle P(a,b) s called the probablty dstrbuto fucto. The use of probablty desty fuctos ad ther assocated probablty dstrbuto fuctos costtute a cetral tool of aalyss scece. 7. Commo Dstrbuto Fuctos From our dscusso of radom varables, let us cosder how certa wdely used dstrbuto fuctos arse. Most dstrbuto fuctos are determed for the dscrete case before geeralzg to ther cotuous couterparts ad we wll follow ths practce. Cosder a sample space where each evet has a costat probablty p of occurrg. We wll let the radom varable be represeted by a sequece of samplg evets. We the wsh to kow what the probablty of obtag a partcular sequece mght be. If we assg each sequece a umercal value, the the probablty values of the sequeces form a probablty dstrbuto fucto. Let us sample the set of equally probable evets tmes wth m occurreces of a evet that has probablty p so that we obta the sequece wth total probablty where P(S) ppqqq pqqqppqp p q -m, (7..) q p, (7..) s the probablty that the samplg dd ot result the evet. Oe ca thk of a evet as gettg a head from a co toss. Sce the samplg evets are cosdered depedet, oe s rarely terested the probablty of the occurrece of a partcular sequece. That s, a sequece ppq wll have the same probablty as the sequece pqp, but oe geerally wshes to kow the probablty that oe or the other or some equvalet (.e. oe havg the same umber of p's ad q's) sequece wll occur. Oe could add all the dvdual probabltes to obta the probablty of all equvalet sequeces occurrg, or, sce each sequece has the same probablty, we may smply fd the umber of such sequeces ad multply by the probablty assocated wth the sequece. a. Permutatos ad Combatos The term permutato s a specal way of descrbg a arragemet of tems. The letters the word cat represet a sequece or permutato, but so do act, tac, tca, atc, ad cta. All of these represet permutatos of the same letters. By eumerato we see that there are 6 such permutatos the case of the word cat. However, f there are N elemets the sequece, the there wll be N! dfferet permutatos that ca be formed. A smple way to see ths s to go about costructg the most geeral permutato possble. We ca beg by selectg the frst elemet of the sequece from ay of the -elemets. That meas that we would have at least permutatos that beg wth oe of the frst elemets. However, havg selected a 4

7 @ Probablty Theory ad Statstcs frst elemet, there are oly (-) elemets left. Thus we wll have oly (-) ew permutatos for each of our tal permutatos. Havg chose twce oly (-) elemets wll rema. each of the (-) permutatos geerated by the frst two choces wll yeld (-) ew permutatos. Ths process ca be cotued utl there are o more elemets to select at whch pot we wll have costructed! dstct permutatos. Now let us geeralze ths argumet where we wll pck a sequece of m elemets from the orgal set of. How may dfferet permutatos of m-elemets ca we buld out of -elemets? Aga, there are -ways to select the frst elemet the permutato leavg (-) remag elemets. However, ow we do ot pck all -elemets, we repeat ths process oly m-tmes. Therefore the umber of permutatos, P m, of -elemets take m at a tme s P m (-)(-) (-m+)!/(-m)!. (7..) A combato s a very dfferet thg tha a permutato. Whe oe selects a combato of thgs, the order of selecto s umportat. If we select a combato of four elemets out of twety, we do't care what order they are selected oly that we eded up wth four elemets. However, we ca ask a questo smlar to that whch we asked for permutatos. How may combatos wth m-elemets ca we make from -elemets? Now t s clear why we troduced the oto of a permutato. We may use the result of equato (7..) to aswer the questo about combatos. Each permutato that s geerated accordace wth equato (7..) s a combato. However, sce the order whch elemets of the combato are selected s umportat, all permutatos wth those elemets ca be cosdered the same combato. But havg pcked the m elemets, we have already establshed that there wll be m! such permutatos. Thus the umber of combatos C m of -elemets take m at a tme ca be wrtte terms of the umber of permutatos as C m P m /m!!/[(-m)!m!] ( m). (7..4) These are ofte kow as the bomal coeffcets sce they are the coeffcets of the bomal seres (+y) C + C - y + + C - y - + C y. (7..5) As mpled by the last term equato (7..4), the bomal coeffcets are ofte deoted by the symbol ( m ). b. The Bomal Probablty Dstrbuto Let us retur to the problem of fdg the probablty of equvalet sequeces. Each sequece represets a permutato of the samplgs producg evets m-tmes. However, sce we are ot terested the order of the samplg, the dstctly dfferet umber of sequeces s the umber of combatos of -samplgs producg m-evets. Thus the probablty of achevg m-evets - samplgs s P B (m) C m p m q -m C m p m (-p) -m, (7..6) ad s kow as the bomal frequecy fucto. The probablty of havg at least m-evets -tres s 5

Numercal Methods ad Data Aalyss F(m) m ad s kow as the bomal dstrbuto. P() C (-p) + C p(-p) - + + C m p m (-p) -m. (7..7) Equatos (7..6) ad (7..7) are dscrete probablty fuctos. Sce a great deal of statstcal aalyss s related to samplg populatos where the samples are assumed to be depedet of oe aother, a great deal of emphass s placed o the bomal dstrbuto. Ufortuately, t s clear from equato (7..4) that there wll be some dffcultes ecoutered whe s large. Aga sce may problems volve samplg very large populatos, we should pay some atteto to ths case. I realty, the case whe s large should be cosdered as two cases; oe where the total sample,, ad the product of the sample sze ad the probablty of a sgle evet, p, are both large, ad oe where s large but p s ot. Let us cosder the latter. c. The Posso Dstrbuto By assumg that s large but p s ot we are cosderg the case where the probablty of obtag a successful evet from ay partcular samplg s very small (.e. p<<). A good eample of ths s the decay of radoactve sotopes. If oe focuses o a partcular atom ay sample, the probablty of decay s early zero for ay reasoable tme. Whle p s cosdered small, we wll assume both ad m to be large. If m s large, the the terval betwee m ad m+ (.e. ) wll be small compared to m ad we ca replace m wth a cotuous varable. Now! (-)(-) (-+), >>, >>. ( )! (7..8) Wth ths appromato we ca wrte equato (7..6) as! PB () p q p ( p)!( )!!. (7..9) The last term ca be wrtte as (-p) (-p) µ/p [(-p) /p ] µ,.) where µ p. (7..) The meag of the parameter µ wll become apparet later. For the momet t s suffcet to ote that t results from the product of a very large umber ad a very small umber. If epad the quatty o the rght brackets by meas of the bomal theorem ad take the lmt as p, we get Lm[( p p p p p! p p p p! / p p) Lm p e p + + +. Therefore the lmt of vashg p equato (7..9) becomes (7..) Lm P B () P P (,µ) µ e -µ /!. (7..) p 6

7 @ Probablty Theory ad Statstcs P P (,µ) s kow as the Posso probablty desty dstrbuto fucto. From equato (7..8) ad equato (7..) oe ca show that µ s the epected value of. However, oe ca see that tutvely from the defto equato (7..). Surely f oe has a large umber of samples ad the probablty p that ay oe of them wll produce a evet, the the epected umber of evets wll smply be p µ. The Posso dstrbuto fucto s etremely useful descrbg the behavor of ulkely evets large populatos. However, the case where the evet s much more lkely so that p s large, the stuato s somewhat more complcated. d. The Normal Curve By assumg that both ad p are large, we move to the realm where all the elemets of the bomal coeffcets are large. So although the varables are stll techcally dscrete, the ut terval betwee them remas small compared to ther value ad we ca aga replace m by the cotuous varable ad p by the cotuous varable µ. We ca summarze the stuato by >> p µ >>. (7..4) Now we may use Sterlg's appromato formula, k k k! e k πk, (7..5) for large factorals to smplfy the bomal coeffcets equato (7..9) to get P () B p q ( ) ( ) ( ) πk( ) p q πk( ), (7..6) Now we add the further restrcto that < p <. (7..7) As the case of the Posso dstrbuto, p wll be the epectato value of ad t s ear that value that we wll be most terested the probablty dstrbuto. Thus let us descrbe the vcty of p by defg a small quatty δ so that δ p, (7..8) ad - (-p) δ q-δ. (7..9) Epressg the bomal dstrbuto fucto gve by equato (7..6) terms of δ, we get ( δ+ p) + ( δ p) δ δ Pb () + p p π(q δ)(p + δ), (7..) whch terms of logarthms ca be wrtte as l [P B ()Q] - (δ+p)l(+δ/p) (q-δ)l(-δ/q), (7..) where Q πpq( δ )( + δ ). (7..) q p 7

Numercal Methods ad Data Aalyss Now we choose to vestgate the rego the mmedate vcty of the epected value of, amely ear p. Therefore δ wll rema small so that δ < pq. (7..) Ths mples that δ < p, (7..4) δ < q ad the terms equatos (7..) ad (7..) ca be appromated by Q πpq δ δ δ l + + +. (7..5) p p p δ δ δ l + + + q q q Keepg all terms through secod order δ for the logarthmc epasos, equato (7..) becomes l[p B ()Q] -(δ+p)(δ/p)(-δ/p)+(q-δ)(δ/q)(-δ/q) -δ /pq, (7..6) so that the bomal dstrbuto fucto becomes f B () e δ pq πpq. (7..7) Replacg p by µ as we dd wth the Posso dstrbuto ad defg a ew quatty σ by σ pq p( p), (7..8) δ µ we ca wrte equato (7..7) as ( µ ) σ e f N (). (7..9) πσ Ths dstrbuto fucto s kow as the ormal dstrbuto fucto or ust the ormal curve. Some tets refer to t as the "Bell-shaped" curve. I realty t s a probablty desty dstrbuto fucto sce, cosderg large, we have passed to the lmt of the cotuous radom varable. Whle the ormal curve s a fucto of the cotuous radom varable, the curve also depeds o the epectato value of (that s µ) ad the probablty p of a sgle samplg yeldg a evet. The sample set s assumed to be very 8

7 @ Probablty Theory ad Statstcs much larger tha the radom varable whch tself s assumed to be very much greater tha. The meag of the parameters µ ad σ ca be see from Fgure 7.. Although the ormal curve s usually attrbuted to Laplace, t s ts use by Gauss for descrbg the dstrbuto of epermetal or observatoal error that brought the curve to promece. It s smply the large umber lmt of the dscrete bomal probablty fucto. If oe makes a seres of depedet measuremets where the error of measuremet s radomly dstrbuted about the "true" value, oe wll obta a epected value of equal to µ ad the errors wll produce a rage of values of havg a characterstc wdth of σ. Used ths cotet the ormal curve s ofte called the Gaussa error curve. Fgure 7. shows the ormal curve appromato to the bomal probablty dstrbuto fucto. We have chose the co tosses so that p.5. Here µ ad σ ca be see as the most lkely value of the radom varable ad the 'wdth' of the curve respectvely. The tal ed of the curve represets the rego appromated by the Posso dstrbuto. Because of the basc ature of the samplg assumptos o whch t s based, the ormal curve plays a maor role testg. Ths s the curve that studets hope wll be used whe they ask "Wll the course be curved?". Of course there are may reasos why a test sample wll depart from the ormal curve ad we wll eplore some of them the et chapter. Oe of the most obvous s that the sample sze s small. It should always be remembered that the cotuous dstrbuto fuctos such as the ormal curve ad the 9

Numercal Methods ad Data Aalyss Posso dstrbuto are appromatos whch oly approach valdty whe the sample set s very large. Also, these are ot the oly dstrbuto fuctos that arse from probablty theory. To demostrate ths pot, let us cosder some mportat oes that occur the physcal world. e. Some Dstrbuto Fuctos of the Physcal World The foudatos of statstcal mechacs devote cosderable effort to descrbg the dstrbuto fuctos for partcles that make up our physcal world. The radom varable that s used turs out to be the total eergy of the partcles. Most of the detals of the dervatos are related to the maer by whch epermet effectvely samples the set of avalable partcles. I the realm of the quatum, the ature of the partcles also plays a maor role determg the resultg probablty dstrbuto fuctos. Sce the physcal world ca be vewed as beg made up of atomc, or f ecessary uclear, partcles, the umber of partcles the sample set s usually huge. Therefore the derved dstrbuto fuctos are usually epressed terms of fuctos of the cotuous radom varable. Cosult a book o statstcal mechacs, ad you wll mmedately ecouter the terms mcrostate, ad macrostate. A macrostate s bascally a physcal dstrbuto of partcles wth respect to the radom varable. A mcrostate s a artfcal cocept developed to ad eumeratg the varous possble macrostates the same sprt that permutatos aded the calculato of combatos. The cocept of a mcrostate specfcally assumes that the partcles are dstgushable. The detaled arragemet of whch partcles have whch values of the radom varable determes the mcrostate. Based o the samplg assumptos, oe attempts to fd the most probable macrostate whch correspods to the epectato value of the system of partcles. I addto, oe searches for the umber of mcrostates wth a partcular macrostate. Sce the relatve probablty of a partcular macrostate occurrg wll be proportoal to the umber of mcrostates yeldg that macrostate, fdg that umber s equvalet to fdg the probablty dstrbuto of macrostates. The most probable macrostate s the oe most lkely to occur ature. The basc dffereces of the dstrbuto fuctos (.e. most probable macrostates) that occur ca be traced to propertes attrbuted to the partcles themselves ad to the ature of the space whch they occur. Cosder the total umber of partcles (N) to be arraged sequetally amog m volumes of some space. The total umber of sequeces or permutatos s smply N!. However, wth each volume (say the th volume), there wll be N partcles whch yeld N! dstgushable sequeces whch must be removed. If we take the 'volumes' whch we are arragg the partcles to be eergy w the we get the dstrbuto fucto to be N a e -w /kt. (7..) Here T s the temperature of the gas, w s the eergy of the partcles, the costat a depeds o the detaled physcal makeup of the gas, ad k s the Boltzma costat. The statstcal dstrbuto of partcles wth the m 'spatal' volumes gve by equato (7..) s kow as Mawell-Boltzma statstcs ad gves ecellet results for a classcal gas where the partcles ca be regarded as dstgushable. I the world of classcal physcs, the posto ad mometum of a partcle are suffcet to make t dstgushable from all other partcles. However, the quatum-mechacal pcture of the physcal world s qute dfferet ad results dfferet dstrbuto fuctos. I the world of the

7 @ Probablty Theory ad Statstcs quatum, as a cosequece of the Heseberg ucertaty prcple, there s a small volume of 'space' wth whch partcles are dstgushable. Thus, oe may loose ay umber of partcles to oe of these 'volumes' ad they would all be cosdered the same kd of partcle. Earler, the samplg order produced permutatos that were dfferet from combatos where the samplg order dd't matter. Ths affected the probablty dstrbutos through the dfferece betwee P m ad C m. I a smlar maer we would epect the dstgushablty of partcles to affect the ature of the most probable macrostate. I ths case the resultat dstrbuto fucto has the form N a (e w /kt ), (7..) where the parameter a ca be determed terms of the eergy of the partcles N. Ths s the dstrbuto fucto that s sutable for the partcles of lght called photos ad ay partcles that behave lke photos. The dstrbuto fucto s kow as the Bose-Este dstrbuto fucto. Fally f oe vokes the Paul Ecluso Prcple that says you ca put o more tha two of certa kds of uclear partcles the mmum volume desgated by the Heseberg ucertaty prcple, the the partcle dstrbuto fucto has the form N a (e w /kt + ), (7..) Ths s kow as the Ferm-Drac dstrbuto fucto ad aga a s determed by the detaled ature of the partcles. Equatos (7.. - ) are ust eamples of the kds of probablty dstrbuto fuctos that occur ature. There are may more. Clearly the kowledge of the etre dstrbuto fucto provdes all the avalable formato about the sample set. However, much of the mportat formato ca be obtaed from smpler propertes of the dstrbuto fucto. 7. Momets of Dstrbuto Fuctos Let us beg by defg what s meat by the momet of a fucto. The momet of a fucto s the tegral of some property of terest, weghted by ts probablty desty dstrbuto fucto, over the space for whch the dstrbuto fucto s defed. Commo eamples of such momets ca be foud statstcs. The mea, or average of a dstrbuto fucto s smply the frst momet of the dstrbuto fucto ad what s called the varace ca be smply related to the secod momet. I geeral, f the dstrbuto fucto s aalytc, all the formato cotaed the fucto s also cotaed the momets of that fucto. Oe of the most dffcult problems ay type of aalyss s to kow what formato s uecessary for the uderstadg of the basc ature of a partcular pheomeo. I other words, what formato ca be safely throw away? The complete probablty desty dstrbuto fucto represetg some pheomeo cotas much more formato about the pheomeo tha we usually wsh to kow. The process of tegratg the fucto over ts defed space order to obta a specfc momet removes or averages out much of the formato about the fucto. However, t results parameters whch are much easer to terpret. Thus oe trades off formato for the ablty to utlze the result ad obta some eplct propertes of the pheomeo. Ths s a stadard 'trck' of mathematcal aalyss.

Numercal Methods ad Data Aalyss We shall defe the kth momet of a fucto f() as M k k f()d / / f()d, k. (7..) / The kth momet the s the kth power of the depedet varable averaged over all allowed values of the that varable ad weghted by the probablty desty dstrbuto fucto. Clearly M s uty as we have chose to ormalze the momet by f()d. Ths has the practcal advatage of makg the uts of M k the same as the uts ad magtude of a average of k the occasoal stuato where f() s ot a ormalzed probablty desty fucto. If the fucto f() s defed for a rage of the depedet varable a b, the the momets ca be wrtte as b < > f ()d M a b f ()d a b < > f ()d M a b f ()d. (7..) a b k k < > f ()d M a k b f ()d a I equatos (7..) ad (7..) we have chose to defe momets of the cotuous radom varable whch s represeted by a probablty desty dstrbuto fucto f(). However, we could ust as easly defe a set of dscrete momets where the tegral s replaced by a sum ad the probablty desty dstrbuto fucto s replaced by the probablty of obtag the partcular value of the radom varable tself. Such momets would the be wrtte as N k P( ) k N. (7..) If the case where the probablty of obtag the radom varable s uform (whch t should be f s really a radom varable), equato (7..) becomes N k P( ) k. (7..4) N As we shall see, much of statstcal aalyss s cocered wth decdg whe the fte or dscrete momet ca be take to represet the cotuous momet (.e. whe P( ) k < k >). Whle a complete kowledge of the momets of a aalytc fucto wll eable oe to specfy the fucto ad hece all the formato t cotas, t s usually suffcet to specfy oly a few of the

7 @ Probablty Theory ad Statstcs momets order to obta most of that formato. Ideed, ths s the stregth ad utlty of the cocept of momets. Four parameters whch characterze a probablty desty dstrbuto fucto, ad are commoly used statstcs are the mea, varace, skewess, ad kurtoss. Fgure 7. shows a graphcal represetato of these parameters for a arbtrary dstrbuto fucto. These four parameters provde a great deal of formato about the probablty desty dstrbuto fucto f() ad they are related to the frst four momets of the dstrbuto fucto. Ideed, the mea of a fucto s smply defed as the frst momet ad s ofte deoted by the symbol µ. We have already used the symbol σ to deote the 'wdth' of the ormal curve ad t s called the stadard devato [see equato (7..9) ad fgure 7.]. I that stace, the 'wdth' was a measure of the root-mea-square of the departure of the radom varable from the mea. The quatty σ s formally called the varace of the fucto ad s defed as σ ( µ ) f ()d f ()d µ f ()d + µ f ()d < > µ. (7..5) Thus the varace clearly cotas the formato suppled by the secod momet of f() ad s ust the mea-square mus the square of the mea. We ca defe a dmesoless parameter, the skewess of a fucto, as a measure of the cube of the departure of f() from ts mea value so that ( µ ) f ()d s [ < > µ ( < > µ )]/ σ [ < > µ ( < > σ )]/ σ. σ (7..6) The ame skewess gve s descrbes what t measures about the fucto f(). If the dstrbuto fucto s symmetrc about µ, the the tegrad of the tegral equato (7..6) s at-symmetrc ad s. If the skewess s postve the o average f() > f(-), ad the dstrbuto fucto s 'skewed' to the rght. The stuato s reversed for s <. Sce ths parameter descrbes a aspect of the relatve shape of the dstrbuto fucto, t should be ormalzed so that t carres o uts. Ths s the reaso for the presece of σ the deomator of equato (7..6). As oe would epect, the kurtoss volves formato from the fourth momet of the probablty desty dstrbuto fucto. Lke the skewess, the kurtoss s dmesoless as t s ormalzed by the square of the varace. Therefore the kurtoss of a fucto s defed as β (-µ) 4 f()d / / (σ ) f()d [< 4 >-4µ< >+6µ < >-µ 4 ]. (7..7) / For the ormal curve gve by equato (7..9), β. Thus f β < the dstrbuto fucto f() s 'flatter' the vcty of the mamum tha the ormal curve whle β > mples a dstrbuto fucto that s more sharply peaked. Sce a great deal of statstcal aalyss deals wth ascertag to what etet a sample of evets represets a ormal probablty dstrbuto fucto, these last two parameters are very helpful tools.

Numercal Methods ad Data Aalyss Fgure 7. shows the mea of a fucto f() as <>. Note ths s ot the same as the most lkely value of as was the case Fgure 7.. However, some real sese σ s stll a measure of the wdth of the fucto. The skewess s a measure of the asymmetry of f() whle the kurtoss represets the degree to whch the f() s 'flatteed' wth respect to a ormal curve. We have also marked the locato of the values for the upper ad lower quartles, meda ad mode. There are two other quattes that are ofte used to characterze a dstrbuto fucto. These are the meda ad mode. To uderstad the oto of meda, let us cosder the more geeral cocept of a percetle. Cosder a probablty desty fucto defed for values of the radom varable the terval a b. Now let α represet that fracto of the terval correspodg to α so that α ( α -a)/(b-a). (7..8) Now we ca defe the αth percetle by α a a f ()d a b f ()d. (7..9) The value of α s ofte gve terms of the percetage of the terval a b, hece the ame for α. α s a measure of the probablty that the evet wll occur α-percet of the sample tres. Whe α s gve as a fracto /4 or /4, α s kow as a quartle Q α. Specfcally ¼ s called the lower quartle, whle /4 s called the upper quartle. The parameter ½ acqures the specal ame of meda. Thus the meda s that value of the radom varable for whch t s equally probable that a evet wll occur wth greater or less tha ½. Thus the meda s defed by 4

7 @ Probablty Theory ad Statstcs f ()d a b a f ()d. (7..) Fally, the term mode s reserved for the most frequetly occurrg value of. Ths parameter s smlar to the epectato value of dscussed secto 7. [see equato (7..8)]. For cotuous dstrbuto fuctos, ths wll clearly occur where the curve has a mamum. Thus we may defe the mode of a fucto as df (). (7..) d m I ths secto we have made all of the deftos terms of the cotuous probablty desty dstrbuto fucto f(). The reaso for geeratg these specfc parameters s to provde ways of characterzg that fucto wthout eumeratg t for all values of. These parameters allow us to compare f() to other dstrbuto fuctos wth certa lmts ad thereby to ascerta the etet to whch the codtos that gve rse to f() correspod to the codtos that yeld kow probablty desty dstrbuto fuctos. Usually oe does ot have a complete cotuous probablty desty dstrbuto fucto avalable for aalyss. Istead, oe deals wth fte samples ad attempts to ascerta the ature of the dstrbuto fucto that govers the results of the samplg. All the parameters defed ths secto ca be defed for fte samples. Usually the trasformato s obvous for those parameters based o momets. Equatos (7..) ad (7..4) gve sutable deftos of ther dscrete deftos. However, the case of the mode, o smple mathematcal formula ca be gve. It wll smply be the most frequetly occurrg value of the sampled evets. Whe dealg wth fte samples, t s commo to defe skewess terms of other more easly calculated parameters of the sample dstrbuto. Some of these deftos are s ( µ m ) / σ s ( µ ) / σ. (7..) s ( + ) /( + ) 4 4 4 4 There are practcal reasos for pckg ay partcular oe of these deftos, but they are ot equvalet so that the user should be careful ad cosstet whe usg them. Let us close ths secto by cosderg a hypothetcal case of a set of grades gve a course. Suppose that there s a class of te studets who take a twety-questo test wth the results gve Table 7.. Here we ecouter a commo problem wth the use of statstcs o small samples. The values for the percetles do ot come out to be teger values so that t s ecessary to smply assg them to the earest teger value. At frst look, we fd that the meda ad mode are the same whch s requred f the scores are to follow the ormal curve. However, we mght suspect that the curve departs somewhat from the statstcally desred result as there are a umber of grades that equal the mamum allowed. Therefore let us cosder the momets of the grade dstrbuto as gve Table 7. 5

Numercal Methods ad Data Aalyss Table 7. Grade Dstrbuto for Sample Test Results Studet No. Percetage Grade Percetle Scores 95 Upper Quartle 4 9 5 85 Meda 6 85 Mode 7 85 8 7 Lower Quartle 9 6 4 Table 7. Eamato Statstcs for the Sample Test STATISTIC VALUE Mode 85 8 689 6575 4 5495 Stadard Devato σ 8.8 Skewess s -.4 s -. s. s -.6 Kurtoss β.87 Here we see that the mea s somewhat below the meda ad mode dcatg that there are more etreme egatve scores tha there are postve oes. Or coversely that a larger fracto of the class has scores above the mea tha below the mea. Ths s supported by the value for the skewess. However, here we have four dfferet choces to choose from. The values s are ofte used to allow for the small umber statstcs. 6

7 @ Probablty Theory ad Statstcs Whle they would ted to mply that the curve s skewed somewhat toward egatve umbers the sese suggested by the relatve values of the meda ad mea, the magtude s ot serous. The value of the Kurtoss s obtaed from equato (7..7) ad suggests that the curve s very smlar to a ormal curve ts flatess. Thus the structor resposble for ths test could feel cofdet that the test grades represet a sample of the paret populato. I the et chapter we wll vestgate quattatvely how secure he or she may be that regard. However, ths begs the ssue as to whether or ot ths s a good test. Wth the mea at 8, oe fds 7% of the class wth grades betwee the mea ad the top possble grade of. Thus % of the gradg rage has bee used to evaluate 7% of the class. Ecellet dscrmato has bee obtaed for the lower % of the class as ther grades are spread over 8% of the possble test rage. If the goal of the test s to evaluate the relatve performace of the class, the spread scores dcates that ths was ot doe a very effcet way. Ideed, for the two studets who scored, o upper lmt o ther ablty has bee establshed. The eamer whe establshg the degree of dffculty of the eamato so that uform dscrmato s obtaed for all segmets of the class should cosder such factors. 7.4 The Foudatos of Statstcal Aalyss I makg the trasto to fte sample szes we also make the trasto from the theoretcal realm of probablty theory to the more practcal world of statstcal aalyss. Thus we should sped some tme uderstadg the basc teets of statstcs before we use the results. I scece we ever prove a theory or hypothess correct, we smple add cofrmatory evdece to a estg body of evdece that supports the theory or hypothess. However, we may prove a theory or hypothess to be correct or at least vald for a partcular set of crcumstaces. We vestgate the valdty of a hypothess by carryg out epermets or observatos. I ts purest form, the act of epermetato ca be vewed as the measuremet of the values of two supposedly related quattes. The relatoshp s sad to be a fuctoal relatoshp whe the quattes are theoretcally related [for eample yf()] where the relatoshp volves parameters that are to be determed by the epermet. The etre pot of the dual measuremet of y ad s to determe those parameters ad thereby test the valdty of the statemet yf(). I the physcal world o measuremet ca be carred out wth arbtrary precso ad therefore there wll be errors heret both y ad. Oe of the mportat roles of statstcs s to obectvely establsh the etet to whch the errors affect the determato of the parameters f() ad thereby place lmts o the etet to whch the epermet cofrms or reects the hypothess. Most statstcal aalyss s focused o aswerg the questo "To what etet s ths epermetal result a matter of chace?". I geeral, we assume that epermets sample some aspect of the real world producg values of y ad. We further assume that ths samplg could prcple be carred out forever yeldg a arbtrarly large set of values of y ad. I other words there ests a fte sample space or set whch s ofte called the paret populato. As a result of samplg error, our sample values wll devate from those of the paret populato by a amout, say ε. Each measured value of departs from ts 'true' value by some ukow value ε. However, we have already see that f the errors ε are ot correlated wth each other, the ε wll be dstrbuted accordace wth the bomal dstrbuto. The oto that we are ubasedly samplg the paret populato bascally assumes that our error sample wll follow the bomal dstrbuto 7

Numercal Methods ad Data Aalyss ad ths s a cetral assumpto of most statstcal aalyss. To be sure there are ways we may check the valdty of ths assumpto, but most of the tests comprsg statstcal ferece rely o the assumpto beg true. It s essetally what we mea whe we address the questo "To what etet s ths epermetal result a matter of chace?". May studets fd the termology of statstcs to be a maor barrer to uderstadg the subect. As wth ay dscple, the specfc argo of the dscple must be uderstood before ay real compreheso ca take place. Ths s partcularly true wth statstcs where the termology has arse from may dverse scetfc dscples. We have already oted how a study populato geetcs gave rse to the term "regresso aalyss" to descrbe the use of Legedre's prcple of least squares. Ofte properly phrased statstcal statemets wll appear awkward ther effort to be precse. Ths s mportat for there are multtudous ways to deceve usg statstcs badly. Ths ofte results from a lack of precso makg a statstcal statemet or falure to properly address the questo "To what etet s ths epermetal result a matter of chace?". a. Momets of the Bomal Dstrbuto Sce the bomal dstrbuto, ad ts assocated large sample lmt, the ormal curve, play such a cetral role statstcal aalyss, we should cosder the meag of the momets of ths dstrbuto. As s clear from fgure 7., the bomal dstrbuto s a symmetrc fucto about ts peak value. Thus the mea of the dstrbuto [as gve by the frst of equatos (7..)] wll be the peak value of the dstrbuto. From the symmetrc ature of the curve, the meda wll also be the peak value whch, tur, s the mode by defto. Therefore, for the ormal curve the meda, mea ad mode are all equal or µ N <> N ( ½ ) N ( m ) N. (7.4.) Smlarly the varous percetles wll be symmetrcally placed about the mea. We have already see that the fourth momet about the mea called the kurtoss takes o the partcular value of for the ormal curve ad t s clear from the symmetry of the ormal curve that the skewess wll be zero. The varace σ, s smply the square of a characterstc half-wdth of the curve called the stadard devato σ. Sce ay area uder a ormalzed probablty desty dstrbuto fucto represets the probablty that a observato wll have a value of defed by the lmts of the area, σ correspods to the probablty that wll le wth σ of µ N. We may obta that probablty by tegratg equato 7..9 so that ( µ N ) µ+σ + σ y PN ( σ) e d e dy erf ().6869 µ σ πσ π. (7.4.) Thus the probablty that a partcular radomly sampled value of wll fall wth σ of the mea value µ, s about 68%. Sce ths argumet apples to the error dstrbuto ε, σ s sometme called the stadard error of estmate. Oe could ask What s the rage correspodg to a 5% probablty of beg wth that value of the mea? Ths wll clearly be a smaller umber tha σ sce we wsh 8

( µ N ) 7 @ Probablty Theory ad Statstcs P ( ) p σ N p e d µ πσ p. (7.4.) The quatty p s usually called the probable error. p.6745σ. (7.4.4) The use of the probable error s dscouraged sce t has become assocated wth statstcal argumets here the author chooses the smaller probable error over the more commo stadard error smply for ts psychologcal effect. b. Multple Varables, Varace, ad Covarace We have dscussed the behavor of evets that ca be characterzed by a sgle radom varable dstrbuted accordg to f(). What are we to do whe the evet s the result of two or more varables each characterzed by ther ow probablty desty dstrbuto fuctos? Say the evet y s related to two varables v ad w by y g(v,w). (7.4.5) If oly two varables are volved y s sad to have a bvarat dstrbuto. Should the evet deped o more tha two varables, t has a multvarat dstrbuto. Such a stuato ca result from a epermet where more tha oe varable must be measured smultaeously order to characterze the result. Cosder the Hall effect physcs where a curret flowg perpedcular to a magetc feld wll geerate a voltage the drecto of the feld. I order to vestgate ths effect oe must smultaeously measure the stregth of the feld ad the curret as well as the resultg voltage. Each of the depedet varables v ad w wll be characterzed by probablty desty dstrbuto fuctos that reflect the errors of measuremet. Each dstrbuto fucto wll be characterzed by the momets we developed for the sgle radom varable. Measuremet error wll affect the values of both the curret ad magetc feld ad t s a far questo to ask how those errors of measuremet affect the epected value of the voltage through the fucto g(v,w). Let ay varato from the meas of y, v, ad w be deoted by δ. The the cha rule of calculus guaratees that Therefore y y g g g g ( y) δv + δw ( δv) + δvδw + ( δw) δ v w v v w. (7.4.6) w g g g g σ y σ v + σ vw + σ w v v w w. (7.4.7) Here we have troduced the parameter σ vw whch s called the coeffcet of covarace, or ust the covarace, as t measures the combed varatos from the mea of the varables v ad w. For cotuous radom varables v ad w, the coeffcet of covarace s defed by 9

Numercal Methods ad Data Aalyss v µ v )(w µ σ vw ( w )f (v)h(w) dvdw. (7.4.8) Here f(v) ad h(v) are the ormalzed probablty desty dstrbuto fuctos of v ad w respectvely. The coeffcet of covarace ca be defed over a fte data set as N (v µ v )(w µ w ) σ vw. (7.4.9) N Ulke the varace, whch some sese measures the varato of a sgle y varable agast tself, the terms that make up the covarace ca be ether postve or egatve. Ideed, f the probablty desty dstrbuto fuctos that gover v ad w are symmetrc about the mea, the σvw. If ths s true for a multvarat dstrbuto fucto, the all the covaraces wll be zero ad N g σ σ y. (7.4.) k k Ths s a result smlar to that obtaed secto 6. [see equatos (6..9) - (6..)] for the errors of the least square coeffcets ad rests o the same assumpto of error symmetry. Ideed, we shall see the et chapter that there s a very close relato betwee lear least squares, ad the statstcal methods of regresso aalyss ad aalyss of varace. Whe oe s dscussg the momets ad propertes of the ormal curve, there s o questo as to the ther value. Ths s a result of the fte sample sze ad therefore s ot realzed for actual cases where the sample s fte. Thus there wll be a ucertaty resultg from the error of the sampled tems the mea as well as other momets ad t s a far questo to ask how that ucertaty ca be estmated. Let us regard the determato of the mea from a fte sample to be the result of a multvarat aalyss where N µ g( ). (7.4.) N The partal dervatve requred by equato (7.4.) wll the yeld g, (7.4.) N ad takg y µ we get the varace of the mea to be N σ k σ σµ N N ; (7.4.) the dfferet observatos are all of the same parameter, ad the values of σ k wll all be equal. I order to evaluate the varace of the mea σ µ drectly, we requre a epresso for the varace of a sgle observato for a fte sample of data. Equato (7..5) assumes that the value of the mea s kow wth absolute precso ad so ts geeralzato to a fte data set wll uderestmate the actual spread the fte dstrbuto fucto. Say we were to use oe of our observatos to specfy the value of the mea. That observato would o loger be avalable to determe other statstcal parameters as t could o loger be regarded as depedet. So the total umber of depedet observatos would ow be N- ad we could wrte the varace of a sgle observato as k

Therefore, the varace of the mea becomes σ ( µ ) ) N (N 7 @ Probablty Theory ad Statstcs. (7.4.4) N ( µ ) σµ. (7.4.5) N(N ) The factor of (N-) the deomator results from the ucertaty of the mea tself. The umber of depedet observatos that go to a statstcal aalyss are ofte referred to as the umber of degrees of freedom of the aalyss. Sce the equvalet of oe observato s requred to specfy the mea, oe degree of freedom s removed from further aalyss. It s that degree of freedom requred to specfy the value of the mea. At ay pot a statstcal aalyss oe should always be cocered wth the umber of degrees of freedom avalable to specfy the soluto to the problem. I some real sese, the umber of degrees of freedom represets the etet to whch the problem s over-determed the absece of error. Thus a least square problem wth coeffcets to be determed from N data pots, there are oly (N-) degrees of freedom. Ths s the statstcal org of the factor of (N-) equato (6..6) that specfes the error the least square coeffcets. c. Mamum Lkelhood Most of statstcs s devoted to determg the etet to whch a sample populato represets the paret populato. A corollary to ths task s the problem of determg the etet to whch the paret populato s represeted by a ormal dstrbuto. We have already see that the mea, mode, ad meda are all equal for a ormal dstrbuto. Ths meas that the most probable value (.e. the epectato value) of s obtaed from the mea, meda, or mode. For a fte populato, these three wll ot, geeral be equal. Is there some way to decde f the dffereces result smply from chace ad a fte radom sample, or whether the paret populato s ot represeted by the ormal curve? Oe approach s to reverse the questo ad ask, "What s the lkelhood that the fte sample wll result a partcular value for the mea, meda, mode or ay other statstc?". To aswer ths questo assumes that the probablty desty dstrbuto for the paret populato s kow. If ths s the case, the oe ca calculate the probablty that a sample of kow sze (ad characterstcs) wll result from samplg that dstrbuto. Ideed the logarthm of that probablty s kow as the lkelhood of the statstc. The value of the lkelhood wll deped o the partcular value of the statstc, whch should ot be regarded as a varable, as well as the ature of the probablty dstrbuto of the paret populato. Mamum lkelhood algorthms are those that adust the samplg procedure wth the costrats mposed by the defto of the statstc so as to mamze the lkelhood of obtag a partcular statstc whe samplg the paret populato. Assume that we are terested determg the most probable value of a evet from a sample of a paret populato, whch does ot follow the ormal curve. If the dstrbuto fucto s ot symmetrc about the mea, the the arthmetc mea wll ot, geeral, be the most probable result (see fgure 7.). However, f we kew the ature of the dstrbuto fucto of the paret populato (.e. ts shape, ot ts eact values) we could devse a samplg procedure that yelded a accurate value for the mode, whch the would be the most probable value for the sampled evet. If the probablty desty fucto of the paret populato s the ormal curve, the the mea s that value. I the case of multvarat aalyss, least-squares

Numercal Methods ad Data Aalyss yelds the mamum lkelhood values for the coeffcets whe the paret populatos of the varous varables are represeted by the ormal curve. I the et chapter we wll cosder some specfc ways of determg the ature of the paret populato ad the etet to whch we ca beleve that the values of the momets accurately sample the paret populato. I addto, we wll also deal wth the problem of multvarat aalyss, small sample sze ad other practcal problems of statstcal aalyss.

7 @ Probablty Theory ad Statstcs Chapter 7 Eercses. Fd the probablty that, from a deck of 5 playg cards, a perso ca draw eactly: a. a par, b. three of a kd, c. four of a kd.. Calculate the probablty that a perso sttg thrd from the dealer a four perso game wll be dealt fve cards cotag: a. a par, b. three of a kd, c. four of a kd. What s the effect of havg addtoal players the game? Does t matter where the player s located wth respect to the other players? If so, why?. What s the probablty that a sgle perso ca draw a fve-card straght or a flush from a sgle deck of cards? 4. Calculate the bomal probablty dstrbuto fucto of obtag "heads" for te throws of a ubased co. 5. Show eplctly how the skewess ad the kurtoss are related to the thrd ad fourth momets of the dstrbuto fucto. Epress them terms of these momets ad the mea ad varace. Reepress the kurtoss terms of the fourth momet, the mea varace ad skewess. 6. Show that the value for the kurtoss of the ormal curve s. 7. Obta epressos for: a. the varace of the skewess of a fte sample, b. the varace of the kurtoss of a fte sample.

Numercal Methods ad Data Aalyss Chapter 7 Refereces ad Supplemetal Readg. Eddgto, Sr A.S. "The Phlosophy of Physcal Scece" (99). Smth, J.G., ad Duca, A.J. "Elemetary Statstcs ad Applcatos: Fudametals of the Theory of Statstcs", (944), Mc Graw-Hll Book Compay Ic., New York, Lodo, pp.. The bascs of probablty theory ad statstcs ca be foud s a very large umber of books. The studet should try to fd oe that s slated to hs/her partcular area of terest. Below are a few that he/she may fd useful.. DeGroot, M.H., "Probablty ad Statstcs" (975), Addso-Wesley Pub. Co. Ic., Readg, Mass.. Mller, I.R., Freud, J.E., ad Johso,R., "Probablty ad Statstcs for Egeers", 4th ed., (99), Pretce-Hall, Ic. Eglewood Clffs, N.J.. Rce, J.A. "Mathematcal Statstcs ad Data Aalyss", (988), Wadsworth ad Brooks/Cole Advaced Books ad Software, Pacfc Grove Cal. 4. Devore, J.L., "Probablty ad Statstcs for Egeerg ad the Sceces", d ed., (987), Brooks/Cole Publshg Co. Ic. Moterey Cal. 5. Larse, R.J., ad Mar, M.L., A Itroducto to Mathematcal Statstcs ad Its Applcatos", d ed., (986) Pretce-Hall, Eglewood Clffs, N.J. 4

8 Samplg Dstrbutos of Momets, Statstcal Tests, ad Procedures The basc fucto of statstcal aalyss s to make udgmets about the real world o the bass of complete formato. Specfcally, we wsh to determe the ature of some pheomeo based o a fte samplg of that pheomeo. The samplg procedure wll produce a dstrbuto of values, whch ca be characterzed by varous momets of that dstrbuto. I the last chapter we saw that the dstrbuto of a radom varable s gve by the bomal dstrbuto fucto, whch uder certa lmtg codtos ca be represeted by the ormal probablty desty dstrbuto fucto ad the Posso dstrbuto fucto. I addto, certa physcal pheomea wll follow dstrbuto fuctos that are o-ormal ature. We shall see that the characterstcs, or statstcs, of the dstrbuto fuctos themselves ca be characterzed by samplg probablty desty dstrbuto fuctos. Geerally these dstrbuto fuctos are also o-ormal partcularly the small sample lmt. 5

Numercal Methods ad Data Aalyss I secto 7.4 we determed the varace of the mea whch mpled that the momets of ay samplg could themselves be regarded as sample that would be characterzed by a dstrbuto. However, the act of formg the momet s a decdedly o-radom process so that the dstrbuto of the momets may ot be represeted by the ormal dstrbuto. Let us cosder several dstrbutos that commoly occur statstcal aalyss. 8. The t, χ, ad F Statstcal Dstrbuto Fuctos I practce, the momets of ay samplg dstrbuto have values that deped o the sample sze. If we were to repeat a fte sample havg N values a large umber of tmes, the the varous momets of that sample wll vary. Sce samplg the same paret populato geerates them all, we mght epect the samplg dstrbuto of the momets to approach that of the paret populato as the sample sze creases. If the paret populato s represeted by a radom varable, ts momets wll approach those of the ormal curve ad ther dstrbutos wll also approach that of the ormal curve. However, whe the sample sze N s small, the dstrbuto fuctos for the mea, varace ad other statstcs that characterze the dstrbuto wll depart from the ormal curve. It s these dstrbuto fuctos that we wsh to cosder. a. The t-desty Dstrbuto Fucto Let us beg by cosderg the rage of values for the mea that we ca epect from a small samplg of the paret populato N. Let us defe the amout that the mea of ay partcular sample departs from the mea of the paret populato p as t ( ) σ. (8..) Here we have ormalzed our varable t by the best u-based estmate of the stadard devato of the mea σ so as to produce a dmesoless quatty whose dstrbuto fucto we ca dscuss wthout worryg about ts uts. Clearly the dstrbuto fucto of t wll deped o the sample sze N. The dffereces from the ormal curve are represeted Fgure 8.. The fucto s symmetrc wth a mea, mode, ad skewess equal to zero. However, the fucto s rather flatter tha the ormal curve so the kurtoss s greater tha three, but wll approach three as N creases. The specfc form of the t-dstrbuto s whch has a varace of p / (N+ ) / Γ[ (N + )] t f (t) +, (8..) πnγ( N N) σ t N/(N-). (8..) Geerally, the dffereces betwee the t-dstrbuto fucto ad the ormal curve are eglgble for N >, but eve ths dfferece ca be reduced by usg a ormal curve wth a varace gve by equato (8..) stead of uty. At the out set we should be clear about the dfferece betwee the umber of samples N ad the umber of degrees of freedom v cotaed the sample. I Chapter 7 (secto 7.4) we troduced the cocept of "degrees of freedom" whe determg the varace. The varace of both a sgle observato ad the mea was epressed terms of the mea tself. The determato of the mea 6

8 Momets ad Statstcal Tests reduced the umber of depedet formato pots represeted by the data by oe. Thus the factor of (N- ) represeted the remag depedet peces of formato, kow as the degrees of freedom, avalable for the statstc of terest. The presece of the mea the epresso for the t-statstc [equato ( 8..)] reduces the umber of degrees of freedom avalable for t by oe. Fgure 8. shows a comparso betwee the ormal curve ad the t-dstrbuto fucto for N8. The symmetrc ature of the t-dstrbuto meas that the mea, meda, mode, ad skewess wll all be zero whle the varace ad kurtoss wll be slghtly larger tha ther ormal couterparts. As N, the t-dstrbuto approaches the ormal curve wth ut varace. b. The χ -Desty Dstrbuto Fucto Just as we qured to the dstrbuto of meas that could result from varous samples, so we could ask what the dstrbuto of varaces mght be. I chapter 6 (secto 6.4) we troduced the parameter χ as a measure of the mea square error of a least square ft to some data. We chose that symbol wth the curret use md. Defe χ N ( ) / σ, (8..4) where σ s the varace of a sgle observato. The quatty χ s the sort of a ormalzed square error. Ideed, the case where the varace of a sgle observato s costat for all observatos we ca wrte 7

Numercal Methods ad Data Aalyss χ Nε / σ, (8..5) where ε s the mea square error. However, the value of χ wll cotue to grow wth N so that some authors further ormalze χ so that χ ν χ / ν. (8..6) Fgure 8. compares the χ - dstrbuto wth the ormal curve. For N the curve s qute skewed ear the org wth the mea occurrg past the mode (χ 8). The Normal curve has µ 8 ad σ. For large N, the mode of the χ - dstrbuto approaches half the varace ad the dstrbuto fucto approaches a ormal curve wth the mea equal the mode. Here the umber of degrees of freedom (.e. the sample sze N reduced by the umber of depedet momets preset the epresso) does ot appear eplctly the result. Sce χ s trscally postve, ts dstrbuto fucto caot be epected to be symmetrc. Fgure 8. compares the probablty desty dstrbuto fucto for χ, as gve by wth the ormal dstrbuto fucto. f(χ ) [ N/ Γ(½N)] - e -χ / (χ ) ½ (N-), (8..7) 8

8 Momets ad Statstcal Tests The momets of the χ desty dstrbuto fucto yeld values of the varace, mode, ad skewess of σ χ N χ m N. (8..8) s N As N creases, the mode creases approachg half the varace whle the skewess approaches zero. Thus, ths dstrbuto fucto wll also approach the ormal curve as N becomes large. c. The F-Desty Dstrbuto Fucto So far we have cosdered cases where the momets geerated by the samplg process are all geerated from samples of the same sze (.e. the same value of N). We ca ask how the sample sze could affect the probablty of obtag a partcular value of the varace. For eample, the χ dstrbuto fucto descrbes how values of the varace wll be dstrbuted for a partcular value of N. How could we epect ths dstrbuto fucto to chage relatvely f we chaged N? Let us qure to the ature of the probablty desty dstrbuto of the rato of two varaces, or more specfcally defe F to be ( χ ν χ / ) ν F χ ν ). (8..9) ( / χ ν Ths ca be show to have the rather complcated desty dstrbuto fucto of the form f(f) ) / N N (N ν ( ν ) / Γ[ (N + N)] N N F Γ[ ( ν + ν)] ν F N ( ) / ) (N +, (8..) ν + ν Γ( ( ) ( N ) ( F / ) ) Γ( N)(NF + N) Γ ν Γ ν ν + ν ν where the degrees of freedom ν ad ν are N ad N respectvely. The shape of ths desty dstrbuto fucto s dsplayed Fgure 8.. The mea, mode ad varace of F-probablty desty dstrbuto fucto are F N /(N ) N (N ) Fm. (8..) N(N ) + (N N )N σ F N(N 4)(N ) 9

Numercal Methods ad Data Aalyss As oe would epect, the F-statstc behaves very much lke a χ ecept that there s a addtoal parameter volved. However, as N ad N both become large, the F-dstrbuto fucto becomes dstgushable from the ormal curve. Whle N ad N have bee preseted as the sample szes for two dfferet samplgs of the paret populato, they really represet the umber of depedet peces of formato (.e. the umber of degrees of freedom gve or take some momets) eterg to the determato of the varaceσ or alterately, the value of χ. As we saw chapter 6, should the statstcal aalyss volve a more complcated fucto of the form g(,a ), the umber of degrees of freedom wll deped o the umber of values of a. Thus the F-statstc ca be used to provde the dstrbuto of varaces resultg from a chage the umber of values of a thereby chagg the umber of degrees of freedom as well as a chage the sample sze N. We shall fd ths very useful the et secto. Fgure 8. shows the probablty desty dstrbuto fucto for the F-statstc wth values of N ad N 5 respectvely. Also plotted are the lmtg dstrbuto fuctos f(χ /N ) ad f(t ). The frst of these s obtaed from f(f) the lmt of N. The secod arses whe N. Oe ca see the tal of the f(t ) dstrbuto approachg that of f(f) as the value of the depedet varable creases. Fally, the ormal curve whch all dstrbutos approach for large values of N s show wth a mea equal to F ad a varace equal to the varace for f(f). Sce the t, χ, ad F desty dstrbuto fuctos all approach the ormal dstrbuto fucto as N, the ormal curve may be cosdered a specal case of the three curves. What s less obvous s that the t- ad χ desty dstrbuto fuctos are specal cases of the F desty dstrbuto. From the defg

8 Momets ad Statstcal Tests equatos for t [equato (8..)] ad χ [equato(8..4)] we see that Lmt t χ, (8..) N From equatos (8..5) ad (8..6) the lmtg value of the ormalzed or reduced χ s gve by Lmt χ v, (8..) v so that Lmt F χ /N. (8..4) N N N Fally t ca be related to F the specal case where Lmt F t. (8..5) N N N Thus we see that the F probably desty dstrbuto fucto s the geeral geerator for the desty dstrbuto fuctos for t ad χ ad hece for the ormal desty dstrbuto fucto tself. 8. The Level of Sgfcace ad Statstcal Tests Much of statstcal aalyss s cocered wth determg the etet to whch the propertes of a sample reflect the propertes of the paret populato. Ths could be re-stated by obtag the probablty that the partcular result dffers from the correspodg property of the paret populato by a amout ε. These probabltes may be obtaed by tegratg the approprate probablty desty dstrbuto fucto over the approprate rage. Problems formulated ths fasho costtute a statstcal test. Such tests geerally test hypotheses such as "ths statstc does ot dffer from the value of the paret populato". Such a hypothess s ofte called ull hypothess for t postulates o dfferece betwee the sample ad the value for the paret populato. We test ths hypothess by ascertag the probablty that the statemet s true or possbly the probablty that the statemet s false. Statstcally, oe ever "proves" or "dsproves" a hypothess. Oe smply establshes the probablty that a partcular statemet (usually a ull hypothess) s true or false. If a hypothess s sustaed or reected wth a certa probablty p the statemet s ofte sad to be sgfcat at a percet level correspodg to the probablty multpled by. That s, a partcular statemet could be sad to be sgfcat at the 5% level f the probablty that the evet descrbed could occur by chace s.5.

Numercal Methods ad Data Aalyss a. The "Studets" t-test Say we wsh to establsh the etet to whch a partcular mea value obtaed from a samplg of N tems from some paret populato actually represets the mea of the paret populato. To do ths we must establsh some toleraces that we wll accept as allowg the statemet that s deed "the same" as p. We ca do ths by frst decdg how ofte we are wllg to be wrog. That s, what s the acceptable probablty that the statemet s false? For the sake of the argumet, lets us take that value to be 5%. We ca re-wrte equato (8..) as ± σ t, (8..) ad thereby establsh a rage δ gve by or for the 5% level as p δ σ t, (8..) p δ, (8..) ( 5%) σ t 5% Now we have already establshed that the t-dstrbuto depeds oly o the sample sze N so that we may fd t 5% by tegratg that dstrbuto fucto over that rage of t that would allow for t to dffer from the epected value wth a probablty of 5%. That s t5%.5 f (t)dt t 5% f (t)dt. (8..4) The value of t wll deped o N ad the values of δ that result ad are kow as the cofdece lmts of the 5% level. There are umerous books that provde tables of t for dfferet levels of cofdece for varous values of N (e.g. Croto et al ). For eample f N s 5, the the value of t correspodg to the 5% level s.57. Thus we could say that there s oly a 5% chace that dffers from p by more tha.57σ. I the case where the umber of samples creases to, the same cofdece lmts drop to.96 σ. We ca p obta the latter result smply by tegratg the 'tals' of the ormal curve utl we have eclosed 5% of the total area of the curve. Thus t s mportat to use the proper desty dstrbuto fucto whe dealg wth small to moderate sample szes. There tegrals set the cofdece lmt approprate for the small sample szes. We may also use ths test to eame addtoal hypotheses about the ature of the mea. Cosder the followg two hypotheses: ad a. The measured mea s greater tha the mea of the paret populato (.e > p ), b. The measured mea s less tha the mea of the paret populato (.e < p ). Whle these hypotheses resemble the ull hypothess, they dffer subtly. I each case the probablty of meetg the hypothess volves the frequecy dstrbuto of t o ust oe sde of the mea. Thus the factor of two that s preset equato (8..4) allowg for both "tals" of the t-dstrbuto establshg the probablty of occurrece s abset. Therefore the cofdece lmts at the p-percetle are set by

8 Momets ad Statstcal Tests tp p a f (t) dt f (t) dt t p. (8..5) tp p b f (t) dt f (t) dt tp Aga oe should be careful to remember that oe ever "proves" a hypothess to be correct, oe smply fds that t s ot ecessarly false. Oe ca say that the data are cosstet wth the hypothess at the p- percet level. As the sample sze becomes large ad the t desty dstrbuto fucto approaches the ormal curve, the tegrals equatos (8..4) ad (8..5) ca be replaced wth p erfc(t p ) [ erf (t p )], (8..6) p a,b erfc( ± t p ) erf ( ± t p ) where erf() s called the error fucto ad erfc() s kow as the complmetary error fucto of respectvely. The effect of sample szes o the cofdece lmts, or alterately the levels of sgfcace, whe estmatg the accuracy of the mea was frst poted out by W.S. Gossett who used the pseudoym "Studet" whe wrtg about t. It has bee kow as "Studets's t-test" ever sce. There are may other uses to whch the t-test may be put ad some wll be dscussed later ths book, but these serve to llustrate ts basc propertes. b. The χ -test Sce χ s a measure of the varace of the sample mea compared wth what oe mght epect, we ca use t as a measure of how closely the sampled data approach what oe would epect from the sample of a ormally dstrbuted paret populato. As wth the t-test, there are a umber of dfferet ways of epressg ths, but perhaps the smplest s to aga calculate cofdece lmts o the value of χ that ca be epected from ay partcular samplg. If we sample the etre paret populato we would epect a χ v of uty. For ay fte samplg we ca establsh the probablty that the actual value of χ should occur by chace. Lke the t-test, we must decde what probablty s acceptable. For the purposes of demostrato, let us say that a 5% probablty that χ dd occur by chace s a suffcet crtera. The value of χ that represets the upper lmt o the value that could occur by chace 5% of the tme s whch for a geeral percetage s χ 5% χ 5%.5 f ( χ, N) dχ N f ( χ, N) dχ, (8..7) χ χ χ p f (, N) d p, (8..8) Thus a observed value of χ that s greater tha χ p would suggest that the paret populato s ot represeted by the ormal curve or that the samplg procedure s systematcally flawed. The dffculty wth the χ -test s that the dvdual values of σ must be kow before the calculatos mpled by equato (8..4) ca be carred out. Usually there s a depedet way of

Numercal Methods ad Data Aalyss estmatg them. However, there s usually also a tedecy to uder estmate them. Epermeters ted beleve ther epermetal apparatus performs better tha t actually does. Ths wll result too large a value of a observed ch-squared (.e. χ o). Both the t-test ad the χ -test as descrbed here test specfc propertes of a sgle sample dstrbuto agast those epected for a radomly dstrbuted paret populato. How may we compare two dfferet samples of the paret populato where the varace of a sgle observato may be dfferet for each sample? c. The F-test I secto 8. we foud that the rato of two dfferet χ 's would have a samplg dstrbuto gve by equato (8..). Thus f we have two dfferet epermets that sample the paret populato dfferetly ad obta two dfferet values of χ, we ca ask to what etet are the two epermets dfferet. Of course the epected value of F would be uty, but we ca ask what s the probablty that the actual value occurred by chace? Aga we establsh the cofdece lmts o F by tegratg the probablty desty dstrbuto fucto so that (p) F p f (F) df. (8..9) Thus f the observed value of F eceeds F (p), the we may suspect that oe of the two epermets dd ot sample the paret populato a ubased maer. However, satsfyg the codto that F < F (p) s ot suffcet to establsh that the two epermets dd sample the paret populato the same way. F mght be too small. Note that from equato (8..9) we ca wrte F /F. (8..) Oe must the compare F to ts epected value F (p) gve by (p) F p f (F) df. (8..) Equatos (8..9) ad (8..) are ot eactly symmetrc so that oly the lmt of large ν ad ν ca we wrte F > F > /F. (8..) So far we have dscussed the cases where the sampled value s a drect measure of some quatty foud the paret populato. However, more ofte tha ot the observed value may be some complcated fucto of the radom varable. Ths was certaly the case wth our dscusso of least squares chapter 6. Uder these codtos, the parameters that relate y ad must be determed by removg degrees of freedom eeded to determe other parameters of the ft from the statstcal aalyss. If we were to ft N data pots wth a fucto havg depedet coeffcets, the we could, prcple, ft of the data pots eactly leavg oly (N-) pots to determe, say, ε. Thus there would oly be (N-) degrees of freedom left for statstcal aalyss. Ths s the org of the (N-) term the deomator of equato (6..6) for the errors (varaces) of the least square coeffcets that we foud chapter 6. Should the mea be requred subsequet aalyss, oly (N--) degrees of freedom would rema. Thus we must be careful determg the umber of degrees of freedom whe dealg wth a problem havg multple parameters. Ths cludes the use of the t-test ad the χ -test. However, such problems suggest a very powerful applcato of the F- test. Assume that we have ft some data wth a fucto of parameters. The χ -test ad perhaps other cosderatos suggest that we have ot acheved the best ft to the data so that we cosder a fucto wth a addtoal parameter so that there are ow a total of (+) depedet parameters. Now we kow that cludg a addtoal parameter wll remove oe more degree of freedom from the aalyss ad that the 4

8 Momets ad Statstcal Tests mea square error ε should decrease. The questo the becomes, whether or ot the decrease ε represets a amout that we would epect to happe by chace, or by cludg the addtoal parameter have we matched some systematc behavor of the paret populato. Here the F-test ca provde a very useful aswer. Both samples of the data are "observatoally" detcal so that the σ 's for the two χ 's are detcal. The oly dfferece betwee the two χ's s the loss o oe degree of freedom. Uder the codtos that σ 's are all equal, the F-statstc takes o the farly smple form of ( ) ε F. (8..) (N ) ε However, ow we wsh to kow f F s greater that what would be epected by chace (.e. s F > F (p) ). Or aswerg the questo "What s the value of p for whch F F (p)?" s aother way of addressg the problem. Ths s a partcularly smple method of determg whe the addto of a parameter a appromatg fucto produces a mprovemet whch s greater tha that to be epected by chace. It s equvalet to settg cofdece lmts for the value of F ad thereby establshg the sgfcace of the addtoal parameter. Values of the probablty tegrals that appear equatos (8..5), (8..6), (8..8), (8..9), ad (8..) ca be foud the appedces of most elemetary statstcs books or the CRC Hadbook of tables for Probablty ad Statstcs. Therefore the F-test provdes a ecellet crtero for decdg whe a partcular appromato formula, lackg a prmary theoretcal ustfcato, cotas a suffcet umber of terms. d. Kolmogorov-Smrov Tests Vrtually all aspects of the statstcal tests we have dscussed so far have bee based o ascertag to what etet a partcular property or statstc of a sample populato ca be compared to the epected statstc for the paret populato. Oe establshes the "goodess of ft" of the sample to the paret populato o the bass of whether or ot these statstcs fall wth the epected rages for a radom samplg. The parameters such as skewess, kurtoss, t, χ, or F, all represet specfc propertes of the dstrbuto fucto ad thus such tests are ofte called parametrc tests of the sample. Such tests ca be deftve whe the sample sze s large so that the actual value of the parameter represets the correspodg value of the paret populato. Whe the sample sze s small, eve whe the departure of the samplg dstrbuto fucto from a ormal dstrbuto s allowed for, the persuasveess of the statstcal argumet s reduced. Oe would prefer tests that eamed the etre dstrbuto lght of the epected paret dstrbuto. Eamples of such tests are the Kolmogorov-Smrov tests. Let us cosder a stuato smlar to that whch we used for the t-test ad χ -test where the radom varable s sampled drectly. For these tests we shall used the observed data pots,, to estmate the cumulatve probablty of the probablty desty dstrbuto that characterzes the paret populato. Say we costruct a hstogram of the values of that are obtaed from the samplg procedure (see fgure 8.4). Now we smply sum the umber of pots wth <, ormalzed by the total umber of pots the sample. Ths umber s smply the probablty of obtag < ad s kow as the cumulatve probablty dstrbuto S( ). It s remscet of the probablty tegrals we had to evaluate for the parametrc tests [eg. equatos (8..5),(8..8), ad (8..9)] ecept that ow we are usg the sampled probablty dstrbuto tself stead of oe obtaed from a assumed bomal dstrbuto. Therefore we ca defe S( ) by 5

Numercal Methods ad Data Aalyss ) ( < ) N S (. (8..4) Ths s to be compared wth the cumulatve probablty dstrbuto of the paret populato, whch s p () f (z) dz. (8..5) The statstc whch s used to compare the two cumulatve probablty dstrbutos s the largest departure D betwee the two cumulatve probablty dstrbutos, or D Ma S( ) p( ),. (8..6) If we ask what s the probablty that the two probablty desty dstrbuto fuctos are dfferet (.e. dsproof of the ull hypothess), the P D Q(D N), (8..7) PD Q[D NN /(N + N ) ] where Press et al gve Q () ( ) e. (8..8) Equatos (8..7) smply state that f the measured value of D D the p s the probablty that the ull hypothess s false. The frst of equatos (8..7) apples to the case where the probablty desty dstrbuto fucto of the paret populato s kow so that the cumulatve probablty requred to compute D from equatos (8..5) ad (8..6) s kow a pror. Ths s kow as the Kolmogorov- Smrov Type test. If oe has two dfferet dstrbutos S ( ) ad S ( ) ad wshes to kow f they orgate from the same dstrbuto, the oe uses the secod of equatos (8..7) ad obtas D from Ma S ( )-S ( ). Ths s usually called the Kolmogorov-Smrov Type test. Note that ether test assumes that the paret populato s gve by the bomal dstrbuto or the ormal curve. Ths s a maor stregth of the test as t s relatvely depedet of the ature of the actual probablty desty dstrbuto fucto of the paret populato. All of the parametrc tests descrbed earler compared the sample dstrbuto wth a ormal dstrbuto whch may be a qute lmtg assumpto. I addto, the cumulatve probablty dstrbuto s bascally a tegral of the probablty desty dstrbuto fucto whch s tself a probablty that les the rage of the tegral. Itegrato teds to smooth out local fluctuatos the samplg fucto. However, by cosderg the etre rage of the sampled varable, the propertes of the whole desty dstrbuto fucto go to determg the D -statstc. The combato of these two aspects of the statstc makes t partcularly useful dealg wth small samples. Ths teds to be a basc property of the o-parametrc statstcal tests such as the Kolmogorov- Smrov tests. We have assumed throughout ths dscusso of statstcal tests that a sgle choce of the radom varable results a specfc sample pot. I some cases ths s ot true. The data pots or samples could themselves be averages or collectos of data. Ths data may be treated as beg collected groups or bs. The treatmet of such data becomes more complcated as the umber of degrees of freedom s o loger calculated as smply as for the cases we have cosdered. Therefore we wll leave the statstcal aalyss of grouped or bed data to a more advaced course of study statstcs. 6

8 Momets ad Statstcal Tests. Fgure 8.4 shows a hstogram of the sampled pots ad the cumulatve probablty of obtag those pots. The Kolmogorov-Smrov tests compare that probablty wth aother kow cumulatve probablty ad ascerta the odds that the dffereces occurred by chace. 8. Lear Regresso, ad Correlato Aalyss I Chapter 6 we showed how oe could use the prcple of least squares to ft a fucto of several varables ad obta a mamum lkelhood or most probable ft uder a specfc set of assumptos. We also oted chapter 7 that the use of smlar procedures statstcs was referred to as regresso aalyss. However, may statstcal problems t s ot clear whch varable should be regarded as the depedet varable ad whch should be cosdered as the depedet varable. I ths secto we shall descrbe some of the techques for approachg problems where cause ad effect caot be determed. Let us beg by cosderg a smple problem volvg ust two varables, whch we wll call X ad X. We have reaso to beleve that these varables are related, but have o a pror reaso to beleve that ether should be regarded as causally depedet o the other. However, wrtg ay algebrac formalsm t s ecessary to decde whch varables wll be regarded as fuctos of others. For eample, we could wrte X a. + X b., (8..) or 7

Numercal Methods ad Data Aalyss X a. + X b.. (8..) Here we have troduced a otato commoly used statstcs to dstgush the two dfferet sets of a's ad b's. The subscrpt m. dcates whch varable s regarded as beg depedet (.e. the m) ad whch s to be regarded as beg depedet (.e. the ). a. The Separato of Varaces ad the Two-Varable Correlato Coeffcet I developg the prcple of least squares chapter 6, we regarded the ucertates to be cofed to the depedet varable aloe. We also dcated some smple techques to deal wth the case where there was error each varable. Here where the very ature of depedecy s ucerta, we must eted these otos. To do so, let us aga cosder the case of ust two varables X ad X. If we were to cosder these varables dvdually, the the dstrbuto represeted by the sample of each would be characterzed by momets such as X, σ, X, σ, etc. However, these varables are suspected to be related. Sce the smplest relatoshp s lear, let us vestgate the lear least square solutos where the roles of depedece are terchaged. Such aalyss wll produce solutos of the form c X a. + Xb. c. (8..) X a. + X b. Here we have deoted the values of the depedet varable resultg from the soluto by the superscrpt c. The les descrbed by equatos (8..) resultg from a least square aalyss are kow statstcs as regresso les. We wll further defe the departure of ay data value X from ts mea value as a devato. I a smlar maer let c be the calculated devato of the th varable. Ths varable measures the spread the th varable as gve by the regresso equato. Aga the subscrpt deotes the depedet varable. Thus, for a regresso le of the form of the frst of equatos (8..), ( - c ) would be the same as the error ε that was troduced chapter 6 (see fgure 8.5). We may ow cosder the statstcs of the devatos. The mea of the devatos s zero sce a m. X, but the varaces of the devatos wll ot be. Ideed they are ust related to what we called the mea square error chapter 6. However, the value of these varaces wll deped o what varable s take to be the depedet varable. For our stuato, we may wrte the varaces of as σ ( ). X a. X b. XX / N ( ). (8..4) σ X a X b X X / N... Some authors 4 refer to these varaces as frst-order varaces. Whle the org of equatos (8..4) s ot mmedately obvous, t ca be obtaed from the aalyss we dd chapter 6 (secto 6.). Ideed, the rght had sde of the frst of equatos (8..4) ca be obtaed by combg equatos (6..4) ad (6..5) to get the term the large paretheses o the rght had sde of equato (6..6). From that epresso t s clear that. σ wε. (8..5) The secod of equatos (8..4) ca be obtaed from the frst by symmetry. Aga, the mea of c s clearly zero but ts varace wll ot be. It s smple a measure the spread of the computed values of the 8

8 Momets ad Statstcal Tests depedet varable. Thus the total varace σ wll be the sum of the varace resultg from the relato betwee X ad X (.e. σ c ) ad the varace resultg from the falure of the lear regresso le to accurately represet the data. Thus σ σ + σ c.. (8..6) σ σ c + σ. The dvso of the total varace σ to parts resultg from the relatoshp betwee the varables X ad X ad the falure of the relatoshp to ft the data allow us to test the etet to whch the two varables are related. Let us defe X c c X σ σ σ. σ. r ± ± r ± ± Nσ σ σ σ σ σ. (8..7) The quatty r s kow as the Pearso correlato coeffcet after Karl Pearso who made wde use of t. Ths smple correlato coeffcet r measures the way the varables X ad X chage wth respect to ther meas ad s ormalzed by the stadard devatos of each varable. However, the meag s perhaps more clearly see from the form o the far rght had sde of equato (8..7). Remember σ smply measures the scatter of X about the mea X, whle σ. measures the scatter of X about the regresso le. Thus, f the varace σ. accouts for the etre varace of the depedet varable X, the the correlato coeffcet s zero ad a plot of X agast X would smply show a radom scatter dagram. It would mea that the varace σ c would be zero meag that oe of the total varace resulted from the regresso relato. Such varables are sad to be ucorrelated. However, f the magtude of the correlato coeffcet s ear uty the σ. must be early zero mplyg that the total varace of X s a result of the regresso relato. The defto of r as gve by the frst term equato (8..7) cotas a sg whch s lost the subsequet represetatos. If a crease X results a decrease X the the product of the devatos wll be egatve yeldg a egatve value for r. Varables whch have a correlato coeffcet wth a large magtude are sad to be hghly correlated or at-correlated depedg o the sg of r. It s worth otg that r r, whch mples that t makes o dfferece whch of the two varables s regarded as the depedet varable. 9

Numercal Methods ad Data Aalyss Fgure 8.5 shows the regresso les for the two cases where the varable X s regarded as the depedet varable (pael a) ad the varable X s regarded as the depedet varable (pael b). b. The Meag ad Sgfcace of the Correlato Coeffcet There s a early rresstble tedecy to use the correlato coeffcet to mply a causal relatoshp betwee the two varables X ad X. The symmetry of r r shows that ths s completely uustfed. The correlato statstc r does ot dstgush whch varable s to be cosdered the depedet varable ad whch s to be cosdered the depedet varable. But ths s the very bass of causalty. Oe says that A causes B, whch s very dfferet tha B causg A. The correlato coeffcet smply measures the relato betwee the two. That relato could be drect, or result from relatos that est betwee each varable ad addtoal varables, or smply be a matter of the chace samplg of the data. Cosder the followg epermet. A scetst sets out to fd out how people get from where they lve to a popular beach. Researchers are employed to motor all the approaches to the beach ad cout the total umber of people that arrve o each of a umber of days. Say they fd the umbers gve Table 8.. Table 8. Sample Beach Statstcs for Correlato Eample Day Total # Gog to the # Takg the Ferry # Takg the Bus Beach 5 5 5 4 4 4 5 4

8 Momets ad Statstcal Tests If oe carres out the calculato of the correlato coeffcet betwee the umber takg the Ferry ad the umber of people gog to the beach oe would get r. If the researcher dd't uderstad the meag of the correlato coeffcet he mght be tempted to coclude that all the people who go to the beach take the Ferry. That, of course, s absurd sce hs ow research shows some people takg the bus. However, a correlato betwee the umber takg the bus ad the total umber of people o the beach would be egatve. Should oe coclude that people oly take the bus whe they kow obody else s gog to the beach? Of course ot. Perhaps most people drve to the beach so that large beach populatos cause such cogesto so that busses fd t more dffcult to get there. Perhaps there s o causal coecto at all. Ca we at least rule out the possblty that the correlato coeffcet resulted from the chace samplg? The aswer to ths questo s yes ad t makes the correlato coeffcet a powerful tool for ascertag relatoshps. We ca quatfy the terpretato of the correlato coeffcet by formg hypotheses as we dd wth the moo-varat statstcal tests ad the testg whether the data supports or reects the hypotheses. Let us frst cosder the ull hypothess that there s o correlato the paret populato. If ths hypothess s dscredted, the the correlato coeffcet may be cosdered sgfcat. We may approach ths problem by meas of a t-test. Here we are testg the probablty of the occurrece of a correlato coeffcet r that s sgfcatly dfferet from zero ad ( ) t r. (8..8) r The factor of (N-) the umerator arses because we have lost two degrees of freedom to the costats of the lear regresso le. We ca the use equatos (8..5) to determe the probablty that ths value of t (ad hece r ) would result from chace. Ths wll of course deped o the umber of degrees of freedom ( ths case N-) that are volved the sample. Coversely, oe ca tur the problem aroud ad fd a value of t for a gve p ad v that oe cosders sgfcat ad that sets a lower lmt to the value for r that would support the hypothess that r occurred by chace. For eample, say we had pars of data pots whch we beleved to be related, but we would oly accept the probablty of a chace occurrece of.% as beg sgfcat. The solvg equato (8..8) for r we get r t(v+t ) ½. (8..9) Cosultg tables that solve equatos (8..5) we fd the boudary value for t s 4.587 whch leads to a mmum value of r.85. Thus, small sample szes ca produce rather large values for the correlato coeffcet smply from the chace samplg. Most scetsts are very crcumspect about moderate values of the correlato coeffcet. Ths probably results from the fact that causalty s ot guarateed by the correlato coeffcet ad the falure of the ull hypothess s ot geerally take as strog evdece of sgfcace. A secod hypothess, whch s useful to test, s apprasg the etet to whch a gve correlato coeffcet represets the value preset the paret populato. Here we desre to set some cofdece lmts as we dd for the mea secto 8.. If we make the trasformato z ½ [(+r)/(-r )] tah - (r ), (8..) 4

Numercal Methods ad Data Aalyss the the cofdece lmts o z are gve by where δz t p σ z, (8..) σ z [N-(8/)] ½. (8..) If for our eample of pars of pots we ask what are the cofdece lmts o a observed value of r.85 at the 5% level, we fd that t.8 ad that δz.87. Thus we ca epect the value of the paret populato correlato coeffcet to le betwee.4<r <.969. The mea of the z dstrbuto s z ½{ [(+rp)/(-r p )] + r p /(N-)}. (8..) For our eample ths leads to the best ubased estmator of r p.87. Ths cely llustrates the reaso for the cosderable skeptcsm that most scetsts have for small data samples. To sgfcatly reduce these lmts, σ z should be reduced at least a factor of three whch mples a crease the sample sze of a factor of te. I geeral, may scetsts place lttle fath a correlato aalyss cotag less tha data pots for reasos demostrated by ths eample. The problem s two-fold. Frst small sample correlato coeffcets must ehbt a magtude ear uty order for t to represet a statstcally sgfcat relatoshp betwee the varables uder cosderato. Secodly, the probablty that the correlato coeffcet les ear the correlato coeffcet of the paret populato s small for a small sample. For the correlato coeffcet to be meagful, t must ot oly represet a relatoshp the sample, but also a relatoshp for the paret populato. c. Correlatos of May Varables ad Lear Regresso Our dscusso of correlato has so far bee lmted to two varables ad the smple Pearso correlato coeffcet. I order to dscuss systems of may varables, we shall be terested the relatoshps that may est betwee ay two varables. We may cotue to use the defto gve equato (8..7) order to defe a correlato coeffcet betwee ay two varables X ad X as r Σ X X / Nσ σ. (8..4) Certaly the correlato coeffcets may be evaluated by brute force after the ormal equatos of the least square soluto have bee solved. Gve the complete mult-dmesoal regresso le, the devatos requred by equato (8..4) could be calculated ad the stadard devatos of the dvdual varables obtaed. However, as fdg the error of the least square coeffcets chapter 6 (see secto 6.), most of the requre work has bee doe by the tme the ormal equatos have bee solved. I equato (6..6) we estmated the error of the least square coeffcets terms of parameters geerated durg the establshmet ad soluto of the ormal equatos. If we choose to weght the data by the verse of the epermetal errors ε, the the errors ca be wrtte terms of the varace of a as σ (a ) C σ. (8..5) Here C s the dagoal elemet of the verse matr of the ormal equatos. Thus t should ot be surprsg that the off-dagoal elemets of the verse matr of the ormal equatos are the covaraces 4

8 Momets ad Statstcal Tests σ C. (8..6) of the coeffcets a ad a as defed secto 7.4 [see equato (7.4.9)]. A specto of the form of equato (7.4.9) wll show that much of what we eed for the geeral correlato coeffcet s cotaed the defto of the covarace. Thus we ca wrte r σ /σ σ. (8..7) Ths allows us to solve the multvarat problems of statstcs that arse may felds of scece ad vestgate the relatoshps betwee the varous parameters that characterze the problem. Remember that the matr of the ormal equatos s symmetrc so that the verse s also symmetrc. Therefore we fd that r r. (8..8) Equato (8..8) geeralzes the result of the smple two varable correlato coeffcet that o cause ad effect result s mpled by the value of the coeffcet. A large value of the magtude of the coeffcet smply mples a relatoshp may est betwee the two varables questo. Thus correlato coeffcets oly test the relatos betwee each set of varables. But we may go further by determg the statstcal sgfcace of those correlato coeffcets usg the t-test ad cofdece lmts gve earler by equatos (8..8)-(8..). d Aalyss of Varace We shall coclude our dscusso of the correlato betwee varables by brefly dscussg a dscple kow as the aalyss of varace. Ths cocept was developed by R.A. Fsher the 9's ad s wdely used to search for varables that are correlated wth oe aother ad to evaluate the relablty of testg procedures. Ufortuately there are those who frequetly make the leap betwee correlato ad causalty ad ths s beyod what the method provdes. However, t does form the bass o whch to search for causal relatoshps ad for that reaso aloe t s of cosderable mportace as a aalyss techque. Sce ts troducto by Fsher, the techque has bee epaded may dverse drectos that are well beyod the scope of our vestgato so we wll oly treat the smplest of cases a attempt to covey the flavor of the method. The ame aalyss the varace s derved from the eamato of the varaces of collectos of dfferet sets of observed data values. It s geerally assumed from the outset that the observatos are all obtaed from a paret populato havg a ormal dstrbuto ad that they are all depedet of oe aother. I addto, we assume that the dvdual varaces of each sgle observato are equal. We wll use the method of least squares descrbg the formalsm of the aalyss, but as wth may other statstcal methods dfferet termology s ofte used to epress ths veerable approach. The smplest case volves oe varable or "factor", say y. Let there be m epermets that each collect a set of values of y. Thus we could form m average values of y for each set of values that we shall label y. It s a far questo to ask f the varous meas y dffer from oe aother by more tha chace. The geeral approach s ot to compare the dvdual meas wth oe aother, but rather to cosder the meas as a group ad determe ther varace. We ca the compare the varace of the meas wth the estmated varaces of each member wth the group to see f that varace departs from the overall varace of the group by more tha we would epect from chace aloe. 4

Numercal Methods ad Data Aalyss 44 Frst we wsh to fd the mamum lkelhood values of these estmates of y so we shall use the formalsm of least squares to carry out the averagg. Lets us follow the otato used chapter 6 ad deote the values of y that we seek as a. We ca the descrbe our problem by statg the equatos we would lke to hold usg equatos (6..) ad (6..) so that a y φ, (8..9) where the o-square matr φ has the rather specal ad restrcted form φ k. (8..) Ths matr s ofte called the desg matr for aalyss of varace. Now we ca use equato (6..) to geerate the ormal equatos, whch for ths problem wth oe varable wll have the smple soluto y a. (8..) The over all varace of y wll smply be σ m ) y (y y) (, (8..) by defto, ad m. (8..) We kow from least squares that uder the assumptos made regardg the dstrbuto of the y 's that the a 's are the best estmate of the value of y (.e.y ), but ca we decde f the varous values of y are all equal? Ths s a typcal statstcal hypothess that we would lke to cofrm or reect. We shall do ths by vestgatg the varaces of a ad comparg them to the over-all varace. Ths procedure s the source of the ame of the method of aalyss.

8 Momets ad Statstcal Tests that Let us beg by dvdg up the over-all varace much the same way we dd secto 8.a so m m (y y ) (y y ) (y y ) + σ σ σ. (8..4) The term o the left s ust the sum of square of depedet observatos ormalzed by σ ad so wll follow a χ dstrbuto havg degrees of freedom. Ths term s othg more tha the total varato of the observatos of each epermet set about ther true meas of the paret populatos (.e. the varace f the true mea weghted by the verse of the varace of the observed mea). The two terms of the rght wll also follow the χ dstrbuto fucto but have -m ad m degree of freedom respectvely. The frst of these terms s the total varato of the data about the observed sample meas whle the last term represets the varato of the sample meas themselves about ther true meas. Now defe the overall meas for the observed data ad paret populatos to be respectvely. Fally defe y y m m y a y m y. (8..5) y y, (8..6) whch s usually called the effect of the factor y ad s estmated by the least square procedure to be We ca ow wrte the last term o the rght had sde of equato (8..4) as m ad the frst term o the rght here s m a σ σ σ y y. (8..7) m (y y ) (y y a ) ) ad the defto of α allows us to wrte that m (y y a ) (a a ) σ σ m (y y +, (8..8), (8..9) a. (8..) However, should ay of the α 's ot be zero, the the results of equato (8..9) wll ot be zero ad the assumptos of ths dervato wll be volated. That bascally meas that oe of the observato sets does ot sample a ormal dstrbuto or that the samplg procedure s flawed. We may determe f ths s the case by cosderg the dstrbuto of the frst term o the rght had sde of equato (8..8). Equato (8..8) represets the further dvso of the varato of the frst term o the rght of equato (8..4) to two ew terms. Ths term was the total varato of the 45

Numercal Methods ad Data Aalyss observatos about ther sample meas ad so would follow a χ -dstrbuto havg -m degrees of freedom. As ca be see from equato (8..9), the frst term o the rght of equato (8..8) represets the varato of the sample effects about ther true value ad therefore should also follow a χ -dstrbuto wth m- degrees of freedom. Thus, f we are lookg for a sgle statstc to test the assumptos of the aalyss, we ca cosder the statstc m (y y) (m ) Q, (8..) m (y y ) ( m) whch, by vrtue of beg the rato of two terms havg χ -dstrbutos, wll follow the dstrbuto of the F- statstc ad ca be wrtte as Q ( m) m m ( y y y) m (m ) y. (8..) Thus we ca test the hypothess that all the effects α are zero by comparg the results of calculatg Q[(m),(m-)] wth the value of F epected for ay specfed level of sgfcace. That s, f Q>F c, where F c s the value of F determed for a partcular level of sgfcace, the oe kows that the α 's are ot all zero ad at least oe of the sets of observatos s flawed. I developmet of the method for a sgle factor or varable, we have repeatedly made use of the addtve ature of the varaces of ormal dstrbutos [.e. equatos (8..4) ad (8..8)]. Ths s the prmary reaso for the assumpto of "ormalty" o the paret populato ad forms the foudato for aalyss of varace. Whle ths eample of a aalyss of varace s for the smplest possble case where the umber of "factors" s oe, we may use the techque for much more complcated problems employg may factors. The phlosophy of the approach s bascally the same as for oe factor, but the specfc formulato s legthy ad beyod the scope of ths book. Ths ust begs the study of correlato aalyss ad the aalyss of varace. We have ot dealt wth multple correlato, partal correlato coeffcets, or the aalyss of covarace. All are of cosderable use eplorg the relatoshp betwee varables. We have aga sad othg about the aalyss of grouped or bed data. The bass for aalyss of varace has oly bee touched o ad the testg of olear relatoshps has ot bee dealt wth at all. We wll leave further study these areas to courses specalzg statstcs. Whle we have dscussed may of the basc topcs ad tests of statstcal aalyss, there remas oe area to whch we should gve at least a cursory look. 8.4 The Desg of Epermets I the last secto we saw how oe could use correlato techques to search for relatoshps betwee varables. We dealt wth stuatos where t was eve uclear whch varable should be regarded as the depedet varable ad whch were the depedet varables. Ths s a stuato ufamlar to the 46

8 Momets ad Statstcal Tests physcal scetst, but ot ucommo the socal sceces. It s the stuato that prevals wheever a ew pheomeology s approached where the mportace of the varables ad relatoshps betwee them are totally ukow. I such stuatos statstcal aalyss provdes the oly reasoable hope of sortg out ad detfyg the varables ad ascertag the relatoshps betwee them. Oly after that has bee doe ca oe beg the search for the causal relatoshps whch lead to a uderstadg upo whch theory ca be bult. Geerally, physcal epermetato sets out to test some theoretcal predcto ad whle the equpmet desg of the epermet may be etremely sophstcated ad the terpretato of the results subtle ad dffcult, the phlosophcal foudatos of such epermets are geerally straghtforward. Where there ests lttle or o theory to gude oe, epermetal procedures become more dffcult to desg. Egeers ofte tread ths area. They may kow that classcal physcs could predct how ther epermets should behave, but the stuato may be so comple or subect to chaotc behavor, that actual predcto of the outcome s mpossble. At ths pot the egeer wll fd t ecessary to search for relatoshps much the same maer as the socal scetst. Some gudace may come from the physcal sceces, but the fal desg of the epermet wll rely o the skll ad wsdom of the epermeter. I the realm of medce ad bology theoretcal descrpto of pheomea may be so vague that oe should eve rela the term varable whch mples a specfc relato to the result ad use the term "factor" mplyg a parameter that may, or may ot, be relevat to the result. Such s the case the epermets we wll be descrbg. Eve the physcal sceces, ad frequetly the socal ad bologcal sceces udertake surveys of pheomea of terest to ther dscples. A survey, by ts very ature, s vestgatg factors wth suspected but ukow relatoshps ad so the proper layout of the survey should be subect to cosderable care. Ideed, Cochra ad Co 5 have observed "Partcpato the tal stages of a epermet dfferet areas of research leads to the strog covcto that too lttle tme ad effort s put to the plag of epermets. The statstca who epects that hs cotrbuto to the plag wll volve some techcal matter statstcal theory fds repeatedly that he makes a much more valuable cotrbuto smply by gettg the vestgator to epla clearly why he s dog the epermet, to ustfy epermetal treatmets whose effects he epects to compare ad to defed hs clam that the completed epermet wll eable hs obectves to be realzed...." Therefore, t s approprate that we sped a lttle tme dscussg the laguage ad ature of epermetal desg. At the begg of chapter 7, we drew the dstcto betwee data that were obtaed by observato ad those obtaed by epermetato. Both processes are essetally samplg a paret populato. Oly the latter case, does the scetst have the opportuty to partake the specfc outcome. However, eve the observer ca arrage to carry out a well desged survey or a badly desged survey by choosg the ature ad rage of varables or factors to be observed ad the equpmet wth whch to do the observg. The term epermet has bee defed as "a cosdered course of acto amed at aswerg oe or 47

Numercal Methods ad Data Aalyss more carefully framed questos". Therefore ay epermet should meet certa crtera. It should have a specfc ad well defed msso or obectve. The lst of relevat varables, or factors, should be complete. Ofte ths latter codto s dffcult to maage. I the absece of some theoretcal descrpto of the pheomea oe ca mage that a sequece of epermets may be ecessary smply to establsh what are the relevat factors. As a corollary to ths codto, every attempt should be made to eclude or mmze the effect of varables beyod the scope or cotrol of the epermet. Ths cludes the bas of the epermeters themselves. Ths latter cosderato s the source of the famous "double-bld" epermets so commo medce where the admsters of the treatmet are uaware of the specfc ature of the treatmet they are admstratg at the tme of the epermet. Whch patets receved whch medces s revealed at a later tme. Astroomers developed the oto of the "persoal equato" to attempt to allow for the bas advertetly troduced by observers where persoal udgemet s requred makg observatos. Fally the epermet should have the teral precso ecessary to measure the pheomea t s vestgatg. All these codtos soud lke "commo sese", but t s easy to fal to meet them specfc staces. For eample, we have already see that the statstcal valdty of ay epermet s strogly depedet o the umber of degrees of freedom ehbted by the sample. Whe may varables are volved, ad the cost of samplg the paret populato s hgh, t s easy to short cut o the sample sze usually wth dsastrous results. Whle we have emphaszed the two etremes of scetfc vestgato where the hypothess s fully specfed to the case where the depedecy of the varables s ot kow, the maorty of epermetal vestgatos le somewhere betwee. For eample, the qualty of mlk the market place could deped o such factors as the dares that produce the mlk, the types of cows selected by the farmers that supply the dares, the tme of year whe the mlk s produced, supplemets used by the farmers, etc. Here causalty s ot frmly establshed, but the order of evets s so there s o questo that the qualty of the mlk determes the tme of year, but the relevace of the factors s certaly ot kow. It s also lkely that there are other uspecfed factors that may fluece the qualty of the mlk that are accessble to the vestgator. Yet, assumg the cocept of mlk qualty ca be clearly defed, t s reasoable to ask f there s ot some way to determe whch of the kow factors affect the mlk qualty ad desg a epermet to fd out. It s these mddle areas that epermetal desg ad techques such as aalyss of varace are of cosderable use. The desg of a epermet bascally s a program or pla for the maer whch the data wll be sampled so as to meet the obectves of the epermet. There are three geeral techques that are of use producg a well desged epermet. Frst, data may be grouped so that ukow or accessble varables wll be commo to the group ad therefore affect all the data wth the group the same maer. Cosder a epermet where the oe wshes to determe the factors that fluece the bakg of a type of bread. Let us assume that there ests a obectve measure of the qualty of the resultat loaf. We suspect that the ove temperature ad durato of bakg are relevat factors determg the qualty of the loaf. It s also lkely that the qualty depeds o the baker mg ad keadg the loaf. We could have all the loaves produced by all the bakers at the dfferet temperatures ad bakg tmes measured for qualty wthout keepg track of whch baker produced whch loaf. I our subsequet aalyss the varatos troduced by the dfferet bakers would appear as varatos attrbuted to temperature ad bakg tme reducg the accuracy of our test. But the smple epedet of groupg the data accordg to each baker ad separately aalyzg the group would solate the effect of varatos amog bakers ad crease the accuracy of the epermet regardg the prmary factors of terest. 48

8 Momets ad Statstcal Tests Secod, varables whch caot be cotrolled or "blocked out" by groupg the data should be reduced sgfcace by radomly selectg the sampled data so that the effects of these remag varables ted to cacel out of the fal aalyss. Such radomzato procedures are cetral to the desg of a well-coceved epermet. Here t s ot eve ecessary to kow what the factors may be, oly that ther effect ca be reduced by radomzato. Aga, cosder the eample of the bakg of bread. Each baker s gog to be asked to bake loaves at dfferet temperatures ad for varyg tmes. Perhaps as the baker bakes more ad more bread fatgue sets affectg the qualty of the dough he produces. If each baker follows the same patter of bakg the loaves (.e. all bake the frst loaves at temperature T for a tme t etc.) the systematc errors resultg from fatgue wll appear as dffereces attrbutable to the factors of the epermet. Ths ca be avoded by assgg radom sequeces of tme ad temperature to each baker. Whle fatgue may stll affect the results, t wll ot be a systematc fasho. Fally, order to establsh that the epermet has the precso ecessary to aswer the questos t poses, t may be ecessary to repeat the samplg procedure a umber of tmes. I the parlace of statstcal epermet desg the oto of repeatg the epermet s called replcato ad ca be used to help acheve proper radomzato ad well as establsh the epermetal accuracy. Thus the cocepts of data groupg, radomzato ad repeatablty or replcato are the basc tools oe has to work wth desgg a epermet. As other areas of statstcs, a partcular argo has bee developed assocated wth epermet desg ad we should detfy these terms ad dscuss some of the basc assumptos assocated wth epermet desg. a. The Termology of Epermet Desg Lke may subects statstcs, the termology of epermet desg has ts org a subect where statstcal aalyss was developed for the specfc aalyss of the subect. As the term regresso aalyss arose form studes geetcs, so much of epermetal desg formalsm was developed for agrculture. The term epermetal area used to descrbe the scope or evromet of the epermet was tally a area of lad o whch a agrcultural epermet was to be carred out. The terms block ad plot meat subdvsos of ths area. Smlarly the oto of a treatmet s kow as a factor the epermet ad s usually the same as what we have prevously meat by a varable. A treatmet level would the refer to the value of the varable. (However, remember the caveats metoed above relatg to the relatve role of varables ad factors.) Fally the term yeld was ust that for a agrcultural epermet. It was the results of a treatmet beg appled to some plot. Notce that here there s a strog causal bas the use of the term yeld. For may epermets ths eed ot be the case. Oe factor may be chose as the yeld, but ts role as depedet varable ca be chaged durg the aalyss. Perhaps a somewhat less preudcal term mght be result. All these terms have survved ad have take o very geeral meags for epermet desg. Much of the mystery of epermet desg s smply relatg the terms of agrcultural org to epermets set far dfferet cotets. For eample, the term factoral epermet refers to ay epermet desg where the levels (values) of several factors (.e. varables) are cotrolled at two or more levels so as to vestgate ther effects o oe aother. Such a aalyss wll result the presece of terms volvg each factor combato wth the remag factors. The epresso of the umber of combatos of thg 49

Numercal Methods ad Data Aalyss take m at a tme does volve factorals [see equato (7..4)] but ths s a slm ecuse for callg such systems "factoral desgs". Nevertheless, we shall follow tradto ad do so. Before delvg to the specfcs of epermet desgs, let us cosder some of the assumptos upo whch ther costructo rests. Uderlyg ay epermet there s a model whch descrbes how the factors are assumed to fluece the result or yeld. Ths s ot a full blow detaled equato such as the physcal scetst s used to usg to frame a hypothess. Rather, t s a statemet of addtvty ad learty. All the factors are assumed to have a smple proportoal effect o the result ad the cotrbuto of all factors s smply addtve. Whle ths may seem, ad some cases may be, a etremely restrctve assumpto, t s the smplest o-trval behavor ad the absece of other formato provdes a good place to beg ay vestgato. I the last secto we dvded up the data for a aalyss of varace to sets of epermets each of whch cotaed dvdual data etres. For the purposes of costructg a model for epermet desg we wll smlarly dvde the observed data so that represets the treatmet level, ad represets the block cotag the factor, ad we may eed a thrd subscrpt to deote the order of the treatmet wth the block. We could the wrte the mathematcal model for such a epermet as y k <y> + f + b + ε k. (8.4.) Here y k s the yeld or results of the th treatmet or factor-value cotaed the th block subect to a epermetal error ε k. The auusmpto of addtvty meas that the block effect b wll be the same for all treatmets wth the same block so that y k y k f f + ε k ε k. (8.4.) I addto, as was the case wth the aalyss of varace t s further assumed that the errors ε k are ormally dstrbuted. By postulatg a lear relato betwee the factors of terest ad the result, we ca see that oly two values of the factors would be ecessary to establsh the depedece of the result o that factor. Usg the termology of epermet desg we would say that oly two treatmet levels are ecessary to establsh the effect of the factor o the yeld. However, we have already establshed that the order whch the treatmets are appled should be radomzed ad that the factors should be grouped or blocked some ratoal way order for the epermet to be well desged. Let us brefly cosder some plas for the acqusto of data whch costtute a epermet desg. b. Blocked Desgs So far we have studously avoded dscussg data that s grouped bs or raks etc. However, the oto s cetral to epermet desg so we wll say ust eough about the cocept to dcate the reasos for volvg t ad dcate some of the completes that result. However, we shall cotue to avod dscussg the statstcal aalyss that results from such groupgs of the data ad refer the studet to more complete courses o statstcs. To uderstad the oto of grouped or blocked data, t s useful to retur to the agrcultural orgs of epermet desg. 5

8 Momets ad Statstcal Tests If we were to desg a epermet to vestgate the effects of varous fertlzers ad sectcdes o the yeld of a partcular speces of plat, we would be foolsh to treat oly oe plat wth a partcular combato of products. Istead, we would set out a block or plot of lad wth the epermetal area ad treat all the plats wth that block the same way. Presumably the average for the block s a more relable measure of the behavor of plats to the combato of products tha the results from a sgle plat. The data obtaed from a sgle block would the be called grouped data or blocked data. If we ca completely solate a o-epermetal factor wth a block, the data ca be sad to be completely blocked wth respect to that data. If the factor caot be completely solated by the groupg, the data s sad to be completely blocked. The subsequet statstcal aalyss for these dfferet types of blockg wll be dfferet ad s beyod the scope of ths dscusso. Now we must pla the arragemets of blocks so that we cover all combatos of the factors. I addto, we would lke to arrage the blocks so that varables that we ca't allow for have a mmal fluece o our result. For eample, sol codtos our epermetal area are lable to be smlar for blocks that are close together tha for blocks that are wdely separated. We would lke to arrage the blocks so that varatos the feld codtos wll affect all trals a radom maer. Ths s smlar to our approach wth the bread where havg the bakers follow a radom sequece of allowed factors (,e, T, ad t ) was used to average out fatgue factors. Thus radomzato ca take place a tme sequece as well as a spatal layout. Ths wll ted to mmze the effects of these ukow varables. The reaso ths works s that f we ca group our treatmets (levels or factor values) so that each factor s eposed to the same uspecfed fluece a radom order, the the effects of that fluece should ted to cacel out over the etre ru of the epermet. Ufortuately oe pays a prce for the groupg or blockg of the epermetal data. The arragemet of the blocks may troduce a effect that appears as a teracto betwee the factors. Usually t s a hgh level teracto ad t s predctable from the ature of the desg. A teracto that s lable to be cofused wth a effect arsg strctly from the arragemet of the blocks s sad to be cofouded ad thus ca ever be cosdered as sgfcat. Should that teracto be the oe of terest, the oe must chage the desg of the epermet. Stadard statstcal tables gve the arragemets of factors wth blocks ad the specfc teractos that are cofouded for a wde rage of the umber of blocks ad factors for two treatmet-level epermets. However, there are other ways of arragg the blocks or the takg of the data so that the fluece of accessble factors or sources of varato are reduced by radomzato. By way of eample cosder the agrcultural stuato where we try to mmze the systematc effects of the locato of the blocks. Oe possble arragemet s kow as a Lat square sce t s a square of Lat letters arraged a specfc way. The rule s that o row or colum shall cota ay partcular letter more tha oce. Thus a Lat square would have the form: ABC BCA. CAB Let the Lat letters A, B, ad C represet three treatmets to be vestgated. Each row ad each colum represets a complete epermet (.e. replcato). Thus the square symbolcally represets a way of radomzg the order of the treatmets wth each replcato so that varables depedg o the order are 5

Numercal Methods ad Data Aalyss averaged out. I geeral, the rows ad colums represet two varables that oe hopes to elmate by radomzato. I the case of the feld, they are the -y locato wth the feld ad the assocated sol varatos etc. I the case of the bakg of bread, the two varables could have bee the batch of flour ad tme. The latter would the elmate the fatgue factor whch was a cocer. Should there have bee a thrd factor, we mght have used a Greco-Lat square where a thrd dmeso s added to the square by the use of Greek subscrpts so that the arragemet becomes: A αbδcβ B βcαa δ. CδAβBα Here the three treatmets are grouped to replcates three dfferet ways wth the result three sources of varato ca be averaged out. A Lat or Greco-Lat square desg s restrctve that t requres that the umber of "rows" ad "colums" correspodg to the two uspecfed systematc parameters, be the same. I addto, the umber of levels or treatmets must equal the umber of rows ad colums. The procedure for use of such a desg s to specfy a tral by assgg the levels to the letters radomly ad the permutg the rows ad colums of the square utl all trals are completed. Oe ca fd larger squares that allow for the use of more treatmets or factors books o epermet desg 6 or hadbooks of statstcs 7. These squares smply provde radom arragemets for the applcato of treatmets or the takg of data whch wll ted to mmze the effects of pheomea or sources of systematc error whch caot be measures, but of whch the epermeter s aware. Whle ther use may crease the amout of replcato above the mmum requred by the model, the addtoal effort s usually more tha compesated by the mprovemet the accuracy of the result. Whle the Lat ad Greco-Lat squares provde a fe desg for radomzg the replcatos of the epermet, they are by o meas the oly method for dog so. Ay reasoable moder computer wll provde a mechasm for geeratg radom umbers whch ca be used to desg the pla for a epermet. However, oe must be careful about the cofoudg betwee blocked data that ca result ay epermet ad be sure to detfy those regos of the epermet whch t s lkely to occur. c. Factoral Desgs As wth all epermetal desgs, the prmary purpose of the factoral desg s to specfy how the epermet s to be ru ad the data samplg carred out. The ma purpose of ths protocol s to sure that all combatos of the factors (varables) are tested at the requred treatmet levels (values). Thus the basc model for the epermet s somewhat dfferet from that suggested by equatos (8.4.) ad (8.4.). Oe looks for effects whch are dvded to ma effects o the yeld (assumed depedet varable) resultg from chages the level of a specfc factor, ad teracto effects whch are chages the yeld that result from the smultaeous chage of two or more factors. I short, oe looks for correlatos betwee the factors ad the yeld ad betwee the factors themselves. A epermet that has factors each of whch s allowed to have m levels wll be requred to have m trals or replcatos. Sce most of the statstcal aalyss that s doe o such epermetal data wll assume that the relatoshps are lear, m s usually take to be two. Such a epermet would be called a factoral epermet. Ths smply meas that t s a epermet wth -factors requres trals. 5

8 Momets ad Statstcal Tests A partcularly cofusg otato s used to deote the order ad values of the factors the epermet. Whle the factors themselves are deoted by captal letters wth subscrpts startg at zero to deote ther level (.e. A, B, C, etc.), a partcular tral s gve a combato of lower case letters. If the letter s preset t mples that the correspodg factor has the value wth the subscrpt. Thus a tral where the factors A,B, ad C have the values A, B, ad C would be labeled smply bc. A specal represetato s reserved for the case A, B, C, where by coveto othg would appear. The symbology s that ths case s represeted by (). Thus all the possble combatos of factors whch gve rse to the teracto effects requrg the trals for a factoral epermet are gve Table 8. Table 8. Factoral Combatos for Two-level Epermets wth 4 No. of Levels Combatos of factors stadard otato factors (), a, b, ab factors (), a, b, ab, c, ac, bc, abc 4 factors (), a, b, ab, c, ac, bc, abc, d, ad, bd, cd, acd, bcd, abcd. Tables est of the possble combatos of the teracto terms for ay umber of factors ad reasoable umbers of treatmet-levels. As a eample, let us cosder the model for two factors each havg the two treatmets (.e. values) requred for the evaluato of lear effects y <y> + a + b + a b + ε. (8.4.) The subscrpt wll take o values of ad for the two treatmets gve to a ad b. Here we see that the cross term ab appears as a addtoal ukow. Each of the factors A ad B wll have a ma effect o y. I addto the cross term AB whch s kow as the teracto term, wll produce a teracto effect. These represet three ukows that wll requre three depedet peces of formato (.e. trals, replcatos, or repettos) for ther specfcato. If we also requre the determato of the grad mea <y> the a addtoal depedet pece of formato wll be eeded brgg the total to. I order to determe all the cross terms arsg from a creased umber of factors may more depedet peces of formato are eeded. Ths s the source of the requred umber of trals or replcatos gve above. I carryg out the trals or replcatos requred by the factoral desg, t may be useful to make use of the blocked data desgs cludg the Lat ad Greco-lat squares to provde the approprate radomzato whch reduces the effect of accessble varables. There are addtoal desgs whch further mmze the effects of suspected flueces ad allow more fleblty the umber of factors ad levels to be used, but they are beyod the scope of ths book. 5

Numercal Methods ad Data Aalyss The statstcal desg of a epermet s etremely mportat whe dealg wth a array of factors or varables whose teracto s upredctable from theoretcal cosderatos. There are may ptfalls to be ecoutered ths area of study whch s why t has become the doma of specalsts. However, there s o substtute for the sght ad geuty of the researcher detfyg the varables to be vestgated. Ay statstcal study s lmted practce by the sample sze ad the systematc ad ukow effects that may plague the study. Oly the kowledgeable researcher wll be able to detfy the possble areas of dffculty. Statstcal aalyss may be able to cofrm those suspcos, but wll rarely fd them wthout the foresght of the vestgator. Statstcal aalyss s a valuable tool of research, but t s ot meat to be a substtute for wsdom ad geuty. The user must also always be aware that t s easy to phrase statstcal ferece so that the resultg statemet says more tha s ustfed by the aalyss. Always remember that oe does ot "prove" hypotheses by meas of statstcal aalyss. At best oe may reect a hypothess or add cofrmatory evdece to support t. But the sample populato s ot the paret populato ad there s always the chace that the vestgator has bee ulucky. 54

8 Momets ad Statstcal Tests Chapter 8 Eercses. Show that the varace of the t-probablty desty dstrbuto fucto gve by equato (8..) s deed σ t as gve by equato (8..).. Use equato (8..7) to fd the varace, mode, ad skewess of the χ -dstrbuto fucto. Compare your results to equato (8..8).. Fd the mea, mode ad varace of the F-dstrbuto fucto gve by equato (8..). 4. Show that the lmtg relatos gve by equatos (8..) - (8..5) are deed correct. 5. Use the umercal quadrature methods dscussed chapter 4 to evaluate the probablty tegral for the t-test gve by equato (8..5) for values of p.,.,., ad N,,. Obta values for t p ad compare wth the results you would obta from equato (8..6). 6. Use the umercal quadrature methods dscussed chapter 4 to evaluate the probablty tegral for the χ -test gve by equato (8..8) for values of p.,.,., ad N,,. Obta values for χ p ad compare wth the results you would obta from usg the ormal curve for the χ - probablty desty dstrbuto fucto. 7. Use the umercal quadrature methods dscussed chapter 4 to evaluate the probablty tegral for the F-test gve by equato (8..9) for values of p.,.,., N,,, ad N,,. Obta values for F p. 8. Show how the varous forms of the correlato coeffcet gve by equato (8..7) ca be obtaed from the defto gve by the secod term o the left. 9. Fd the varous values of the.% margally sgfcat correlato coeffcets whe 5,,,,.. Fd the correlato coeffcet betwee X ad Y, ad Y ad Y problem 4 of chapter 6.. Use the F-test to decde whe you have added eough terms to represet the table gve problem of chapter 6.. Use aalyss of varace to show that the data Table 8. mply that takg the bus ad takg the ferry are mportat factors populatg the beach.. Use aalyss of varace to determe f the eamato represeted by the data Table 7. sampled a ormal paret populato ad at what level of cofdece o ca be sure of the result. 55

Numercal Methods ad Data Aalyss 4. Assume that you are to desg a epermet to fd the factors that determe the qualty of bread baked at dfferet bakeres. Idcate what would be your cetral cocers ad how you would go about addressg them. Idetfy four factors that are lable to be of cetral sgfcace determg the qualty of bread. Idcate how you would desg a epermet to fd out f the factors are deed mportat. 56

Chapter 8 Refereces ad Supplemetal Readg 8 Momets ad Statstcal Tests Croto, F.E., Cowde, D.J., ad Kle, S., "Appled Geeral Statstcs", (967), Pretce-Hall, Ic., Eglewood Clffs, N.J.. Weast, R.C., "CRC Hadbook of Tables for Probablty ad Statstcs", (966), (Ed. W.H.Beyer), The Chemcal Rubber Co. Clevelad.. Press, W.H., Flaery, B.P., Teukolsky, S.A., ad Vetterlg, W.T., "Numercal Recpes the art of scetfc computg" (986), Cambrdge Uversty Press, Cambrdge. 4. Smth, J.G., ad Duca, A.J., "Samplg Statstcs ad Applcatos: Fudemetals of the Theory of Statstcs", (944), McGraw-Hll Book Compay Ic., New York, Lodo, pp.8. 5. Cochra, W.G., ad Co, G.M., "Epermetal Desgs" (957) Joh Wley ad Sos, Ic., New York, pp. 6. Cochra, W.G., ad Co, G.M., "Epermetal Desgs" (957) Joh Wley ad Sos, Ic., New York, pp 45-47. 7. Weast, R.C., "CRC Hadbook of Tables for Probablty ad Statstcs", (966), (Ed. W.H.Beyer), The Chemcal Rubber Co. Clevelad, pp6-65. 57

Numercal Methods ad Data Aalyss 58

Ide Numercal Methods ad Data Aalyss A Adams-Bashforth-Moulto Predctor-Corrector.. Characterstc values... 49...6 of a matr... 49 Aalyss of varace..., 45 Characterstc vectors... 49 desg matr for...4 of a matr... 49 for oe factor...4 Chebyschev polyomals... 9 At-correlato: meag of...9 of the frst kd... 9 Appromato orm...74 of the secod kd... 9 Arthmetc mea... recurrece relato... 9 Assocatvty defed... Average... Aal vectors... relatos betwee frst ad secod kd... 9 Chebyshev orm ad least squares... 9 B defed... 86 Babbtt... Ch square Back substtuto... defed... 7 Barstow's method for polyomals dstrbuto ad aalyss of varace. 44...6 ormalzed... 7 Bell-shaped curve ad the ormal curve 9 statstc for large N... Bomal coeffcet....99, 4 Ch-square test Bomal dstrbuto fucto...4, 7 cofdece lmts for... Bomal seres...4 defed... Bomal theorem...5 meag of... Bvarat dstrbuto...9 Cofactor of a matr... 8 Blocked data ad epermet desg 7 Combato defed... 4 Bodewg...4 Commutatve law... Bose-Este dstrbuto fucto... Complmetary error fucto... Boudary value problem... Cofdece level a sample soluto...4 defed... compared to a tal value problem45 defed...9 ad percetles... for correlato Bulrsch-Stoer method...6 coeffcets... 4, 4 for the F-test... 4 C Cofouded teractos Cator, G... defed... 5 Cartesa coordates...8, Costats of tegrato for ordary dfferetal Causal relatoshp ad correlato 9, 4 equatos... Cetral dfferece operator Cotravarat vector... 6 defed...99 Covergece of Gauss-Sedel terato... 47 Characterstc equato...49 of a matr...49 Coverget teratve fucto crtero for... 46 59

Ide Degrees of freedom ad correlato... 4 defed... for bed data... 6 for the F-statstc... for the F-test... for the t-dstrbuto... 7 aalyss of varace... 44 Del operator... 9 (see Nabula) Dervatve from Rchardso etrapolato... Descartes's rule of sgs... 57 Desg matr for aalyss of varace... 4 Determat calculato by Gauss-Jorda Method... of a matr... 7 trasformatoal varace of 47 Devato from the mea... 8 statstcs of... 7 Dfferece operator defto... 9 Dfferetal equatos ad lear -pot boudary value problems... 9 Bulrsch-Stoer method... 6 error estmate for... ordary, defed... partal... 45 soluto by oe-step methods... soluto by predctor-corrector D methods... 4 soluto by Ruga-Kutta method... 6 Coordate trasformato...8 Corrector Adams-Moulto...6 Correlato coeffcet ad causalty...4 ad covarace...4 ad least squares...4 defed...9 for may varables...4 for the paret populato...4 meag of...9, 4 symmetry of...4 Covarace...9 ad the correlato coeffcet...4 coeffcet of...9 of a symmetrc fucto... Covarat vectors defto...7 Cramer's rule...8 Cross Product... Crout Method...4 eample of...5 Cubc sples costrats for...75 Cumulatve probablty ad KS tests...5 Cumulatve probablty dstrbuto of the paret populato...5 Curl...9 defto of...9 Curve fttg defed...64 wth sples...75 Degree of a partal dfferetal equato...46 of a ordary dfferetal equato.. Degree of precso defed... for Gaussa quadrature...6 for Smpso's rule...4 for the Trapezod rule... step sze cotrol... systems of... 7 Dmesoalty of a vector... 4 Drac delta fucto as a kerel for a tegral equato... 55 Drectos coses... 9 6

Numercal Methods ad Data Aalyss Drchlet codtos F-test for Fourer seres...66 ad least squares... 4 Drchlet's theorem...66 defed... Dscrete Fourer trasform...69 for a addtoal parameter... 4 Dstrbuto fucto meag of... 4 for ch-square...7 Factor for the t-statstc...6 aalyss of varace... 4 of the F-statstc...9 of a epermet... 49 Dvergece...9 defto of...9 Double-bld epermets...46 Factored form of a polyomal... 56 Factoral desg... 49 Fast Fourer Trasform... 9, 68 Ferm-Drac dstrbuto fucto... E Feld Effect defto... 5 defed for aalyss of varace...44 scalar... 5 Ege equato...49 vector... 5 of a matr...49 Fte dfferece calculus Ege-vectors...49 fudemetal theorem of... 98 of a matr...49 Fte dfferece operator sample soluto for...5 use for umercal dfferetato... 98 Egevalues Frst-order varaces of a matr...48, 49 defed... 7 sample soluto for...5 Fed-pot Equal terval quadrature... defed... 46 Equatos of codto Fed-pot terato theory... 46 for quadrature weghts...6 ad tegral equatos... 5 Error aalyss ad o-lear least squares... 8, 86 for o-lear least squares...86 ad Pcard's method... Error fucto... for the corrector ODEs... 6 Euler formula for comple umbers...68 Fourer aalyss... 64 Epectato value... defed... Fourer tegral... 67 Fourer seres... 9, 6 Epermet desg...45 termology for...49 usg a Lat square...5 ad the dscrete Fourer trasform... 69 coeffcets for... 65 covergece of... 66 Epermetal area...49 Fourer trasform... 9, 64 Etrapolato...77, 78 defed... 67 for a dscrete fucto... 69 F verse of... 68 F-dstrbuto fucto Fredholm equato defed...7 defed... 46 F-statstc... soluto by terato... 5 ad aalyss of varace...44 soluto of Type... 47 for large N... soluto of Type... 48 6

Ide Freedom Hermta matr degrees of... defto... 6 Fudametal theorem of algebra...56 Hgher order dfferetal equatos as systems of frst order equatos...4 G Hldebradt... Galto, Sr Fracs...99 Hollerth... Gauss, C.F....6, 98 Hotellg... 4 Gauss elmato Hotellg ad Bodewg method ad tr-dagoal eample of... 4 equatos 8 Hyper-effcet quadrature formula Gauss Jorda Elmato... for oe dmeso... Gauss-Chebyschev quadrature multple dmesos... 5 ad mult-dmeso quadrature...4 Hypothess testg ad aalyss of varace.. 45 Gauss-Hermte quadrature...4 Gauss-terato scheme I eample of...4 Idetty operator... 99 Gauss-Jorda matr verso Ital values for dfferetal equatos... eample of... Itegral equatos Gauss-Laguerre quadrature...7 defed... 46 Gauss-Legedre quadrature... homogeeous ad homogeeous... 47 ad mult-dmeso quadrature...5 lear types... 47 Gauss-Sedel Iterato...9 Itegral trasforms... 68 eample of...4 Iteracto effects ad epermetal desg.. 5 Gaussa Elmato...9 Iterpolato Gaussa error curve... by a polyomal... 64 Gaussa quadrature...6 geeral theory... 6 compared to other quadrature formulae Iterpolato formula as a bass for quadrature compared wth Romberg quadrature. formulae 4 degree of precso for...7 Iterpolatve polyomal multple dmesos... eample of... 68 specfc eample of...8 Iverse... Gaussa-Chebyschev quadrature... of a Fourer Trasform... 68 Gegebauer polyomals...9 Iteratve fucto Geeratg fucto for orthogoal polyomals87 covergece of... 46 Gossett... defed... 46 Gradet...9 defto of...9 multdmesoal... 46 Iteratve Methods of the Ch-squared surface..8 ad lear equatos... 9 H J Heseberg Ucertaty Prcple... Jacob polyomals... 9 Hermte terpolato...7 ad mult-dmeso Gaussa as a bass for Gaussa quadrature.6 quadrature... 4 Hermte Polyomals...89 Jacoba... recurrece relato...89 Jeks-Taub method for polyomals... 6 6

Numercal Methods ad Data Aalyss K Kerel of a tegral equato...48 Legedre Polyomals... 87 ad uqueess of the soluto for Gaussa quadrature... 8.. 54 recurrece relato... 87 effect o the soluto...54 Lehmer-Schur method for polyomals... 6 Kolmogorov-Smrov tests...5 Lebtz... 97 Type...6 Type...6 Kroecker delta...9, 4, 66 Levels of cofdece defed... Lev-Cvta Tesor... 4 defto...6 defto... 4 Kurtoss... Lkelhood... of a fucto of the ormal curve...8 defed... mamum value for... of the t-dstrbuto...6 Lear correlato... 6 Lear equatos L formal soluto for... 8 Lagrage Iterpolato...64 Lear Programmg... 9 ad quadrature formulae... ad the Chebyshev orm... 9 Lagrage polyomals Lear trasformatos... 8 for equal tervals...66 Logcal 'or'... relato to Gaussa quadrature..7 Logcal 'ad'... specfc eamples of...66 Lagraga terpolato M ad umercal dffereto 99 Macrostate... weghted form...84 Ma effects ad epermetal desg..5 Laguerre Polyomals...88 recurrece relato...89 Matr defto... 6 Laplace trasform defed...68 factorzato... 4 Matr verse Lat square defed...5 Least square coeffcets mprovemet of... 4 Matr product defto... 6 errors of...76, Mamum lkelhood Least Square Norm defed...6 Least squares ad aalyss of varace... 4 of a fucto... Mawell-Boltzma statstcs... ad aalyss of varace...4 Mea..., ad correlato dstrbuto of... 5 coeffcets 6 of a fucto..., ad mamum lkelhood... of the F-statstc... ad regresso aalyss...99 of the ormal curve... 8 ad the Chebyshev orm...9 of the t-dstrbuto... 6 for lear fuctos...6 Mea square error for o-lear problems...8 ad Ch-square... 7 wth errors the depedet varable8 statstcal terpretato of..8 Legedre, A....6, 98 Mea square resdual (see mea square error) Legedre Appromato...6, 64 determato of... 79 6

Ide for uequally spaced data... 65 Meda defed...4 of the ormal curve...8 matr developmet for tesor product..6 for weghted... 6 Mcrostate... for Normal matrces Mle predctor...6 defed... 7 M-ma orm...86 for least squares... 76 (see also Chebyshev orm) Null hypothess... Mor of a matr...8 for correlato... 4 Mode... defed... for the K-S tests... 5 Numercal dfferetato... 97 of a fucto...4 Numercal tegrato... of ch-square...7 of the F-statstc... O of the ormal curve...8 Operatos research... 9 of the t-dstrbuto...6 Operator... 8 Momet of a fucto... cetral dfferece... 99 Mote Carlo methods...5 quadrature...5 dfferece... 9 dfferetal... 8 Mult-step methods for the soluto of ODEs....4 fte dfferece... 98 fte dfferece detty...99 Multple correlato...45 detty... 9 Multple tegrals... tegral... 8 Multvarat dstrbuto...9 shft... 9, 99 summato... 9 N vector... 9 Nabula...9 Optmzato problems... 99 Natural sples...77 Order Nevlle's algorthm for polyomals...7 Newto, Sr I....97 Newto-Raphso ad o-lear least squares...8 for polyomals...6 for a ordary dfferetal equato... of a partal dfferetal equato.46 of a appromato... 6 No-lear least squares of covergece... 64 errors for...86 Orthogoal polyomals No-parametrc statstcal tests ad Gaussa quadrature... 7 (see Kolmogorov-Smrov as bass fuctos for terpolato... 9 tests)...6 some specfc forms for... 9 Normal curve...9 Orthogoal utary trasformatos... ad the t-,f-statstcs... Orthoormal Normal dstrbuto... fuctos..86 ad aalyss of varace...45 Orthoormal polyomals Normal dstrbuto fucto...9 defed... 86 Normal equatos...6 Orthoormal trasformatos..., 48 for o-lear least squares...8 Over relaato for lear equatos... 46 for orthogoal fuctos...64 P for the errors of the coeffcets...75 64

Numercal Methods ad Data Aalyss Parabolc hypersurface ad o-lear least Polytope... 9 squares...84 Power Spectra... 9 Parametrc tests...5 Precso of a computer... 5 (see t-,f-,ad ch-square tests) Predctor Paret populato...7,, ad statstcs... correlato coeffcets...9 Adams-Bashforth... 6 stablty of... 4 Predctor-corrector Partal correlato...45 for soluto of ODEs... 4 Partal dervatve defed...46 Probabltly defto of... 99 Partal dfferetal equato...45 ad hydrodyamcs...45 classfcato of...46 Paul ecluso prcple... Pearso correlato coeffcet...9 Pearso, K....9 Percet level... Percetle defed... for the ormal curve...8 Permutato defed...4 Probablty desty dstrbuto fucto... defed... Probable error... 8 Product polyomal defed... Proper values... 49 of a matr... 49 Proper vectors... 49 of a matr... 49 Protocol for a factoral desg... 5 Pseudo vectors... Pseudo-tesor... 4 (see tesor desty) Persoal equato...46 Photos...9 Pythagoras theorem ad least squares... 79 Pcard's method... Posso dstrbuto...7 Q Polyomal Quadrature... factored form for...56 geeral defto...55 ad tegral equatos... 48 for multple tegrals... roots of...56 Mote Carlo... 5 Polyomal appromato...97 Quadrature weghts ad terpolato theory...6 determato of... 5 ad multple quadrature... Quartle ad the Chebyshev orm...87 defed... 4 Polyomals upper ad lower... 4 Chebyschev...9 for sples...76 Quotet polyomal... 8 terpolato wth... 8 Gegebauer...9 (see ratoal fucto)... 8 Hermte...9 Jacob...9 R Lagrage...66 Radom varable Laguerre...89 Legedre...87 orthoormal...86 Ultrasphercal...9 defed... momets for... Ratoal fucto... 8 ad the soluto of ODEs... 7 65

Ide Recurrece relato Sgfcace for Chebyschev polyomals...9 level of... for Hermte polyomals...9 meag of... for Laguerre polyomals...89 of a correlato coeffcet... 4 for Legedre polyomals...87 Smlarty trasformato... 48 for quotet polyomals...8 defto of... 5 for ratoal terpolatve fuctos...8 Recursve formula for Lagraga polyomals68 Smple method... 9 Smpso's rule ad Ruge-Kutta... 4 Reflecto trasformato... as a hyper-effcet quadrature Regresso aalyss...7,, 6 formula.4 ad least squares...99 compared to other quadrature... Regresso le...7 formulae... degrees of freedom for...4 degree of precso for... 4 Relaato Methods derved... 4 for lear equatos...4 rug form of... 5 Relaato parameter Sgular matrces... defed...44 eample of...44 Resdual error Skewess... of a fucto... of ch-square... 7 least squares...76 of the ormal curve... 8 Rchardso etrapolato...99 of the t-dstrbuto... 6 or Romberg quadrature... Sples... 75 Rght had rule... specfc eample of... 77 Romberg quadrature... Stadard devato compared to other formulae... ad the correlato coeffcet... 9 cludg Rchardso etrapolato... defed... Roots of a polyomal...56 of the mea... 5 Rotato matrces... of the ormal curve... 8 Rotatoal Trasformato... Stadard error of estmate... 8 Roudoff error...5 Statstcs Rule of sgs...57 Bose-Este... Ruga-Kutta algorthm for systems of ODEs 8 Ferm-Drac... Ruga-Kutta method...6 Mawell-Boltzma... appled to boudary value problems.4 Steepest descet for o-lear least squares. 84 Step sze S cotrol of for ODE... Sample set ad probablty Sterlg's formula for factorals... 7 theory.. Studets's t-test... Sample space... (see t-test) Scalar product defto...5 Secat terato scheme for polyomals...6 Self-adot...6 Shft operator...99 Symmetrc matr... 6 Sythetc Dvso... 57 recurrece relatos for... 58 66

Numercal Methods ad Data Aalyss T Ut matr... 4 t-statstc Utary matr... 6 defed...5 for large N... V t-test Vadermode determat... 65 defed... Varace...,, for correlato coeffcets...4 aalyss of... 4 for large N... for a sgle observato... 7 Taylor seres of the t-dstrbuto... 6 ad o-lear least squares...8 of a fucto... ad Rchardso etrapolato...99 of a sgle observato... ad Ruga-Kutta method...6 of ch-square... 7 Tesor destes...4 of the ormal curve... 8 Tesor product of the F-statstc... for least square ormal equatos...6 of the mea..., 5 Topology...7 Varaces Trace ad Ch-squared... 7 of a matr...6 frst order... 8 trasformatoal varece of...49 of devatos from the mea... 8 Trasformato- rotatoal... Vector operators... 9 Traspose of the matr... Vector product Trapezod rule... ad Ruge-Kutta...4 defto... 6 Vector space compared to other quadrature formulae geeral form... for least squares... 79 Vectors Treatmet ad epermetal desg...49 cotravarat... 6 Treatmet level Ve dagram for combed probablty... for a epermet...49 Volterra equatos Tr-dagoal equatos...8 as Fredholm equatos... 5 for cubc sples...77 defed... 46 Trals soluto by terato... 5 ad epermatal desg...5 symbology for 5 soluto of Type... 5 soluto of Type... 5 Tragular matrces for factorzato...4 W Tragular system Weght fucto... 86 of lear equatos... for Chebyschev polyomals... 9 for Gaussa quadrature... 9 Trgoometrc fuctos orthogoalty of...9 for Gegebauer polyomals... 9 for Hermte polyomals... 89 Trucato error...6 for Laguerre polyomals... 88 estmate ad reducto for ODE... for Legedre polyomals... 87 estmate for dfferetal equatos... Jacob polyomals... 9 for umercal dfferetato...99 Weghts for Gaussa quadrature... 8 U 67

Ide Y Yeld for a epermet...49 Z Zeo's Parado...97 68