Method of least squares J. M. Powers University of Notre Dame February 28, 2003

Transcription

1 Mehod of leas squares J. M. Powers Universiy of Nore Dame February 28, 200 One imporan applicaion of daa analysis is he mehod of leas squares. This mehod is ofen used o fi daa o a given funcional form. The form is mos ofen in erms of polynomials, bu here is absoluely no resricion; rigonomeric funcions, logarihmic funcions, Bessel funcions can all serve as well. Here we will resric ourselves o sricly scalar funcions of he form x = f(; a j, j =,...,M, where x is a dependen variable, is an independen variable, f is an assumed funcional form, and a j is a se of M consan parameers in he funcional form. The analysis can easily be exended for funcions of many variables. Mahemaically, he fundamenal problem is given aseofn discree daa poins, x i, i, i =,...,N, an assumed funcional form for he curve fi f(, a j which has M parameers a j, j =,...,M, find he bes se of parameer values a j so as o minimize he leas squares error beween he curve fi and he acual daa poins. Tha is, he problem is o find a j, j =,...,M,such ha l 2 = x i f( i,a j 2 N (x i f( i,a j 2, is minimized. Here l 2 represens a oal error of he approximaion. I is someimes called a norm of he approximaion or an L-wo norm. The noaion 2 represens he L-wo norm of a vecor represened by. In he leas squares mehod, one examines he daa, makes a non-unique judgmen of wha he funcional form migh be, subsiues each daa poin ino he assumed form so as o form an overconsrained sysem of equaions, uses sraighforward echniques from linear algebra o solve for he coefficiens which bes represen he given daa if he problem is linear in he coefficiens a j, uses echniques from opimizaion heory o solve for he coefficiens which bes represen he given daa if he problem is non-linear in a j. i=

2 The mos general problem, in which he dependency a j is non-linear, is difficul, and someimes impossible. For cases in which he funcional form is linear in he coefficiens a j or can be rendered linear via simple ransformaion, i is possible o ge a unique represenaion of he bes se of parameers a j. This is ofen he case for common curve fis such as sraigh line, polynomial, or logarihmic fis. Le us firs consider polynomial curve fis. Now if one has say, en daa poins, one can in principle, find a ninh order polynomial which will pass hrough all he daa poins. Ofen imes, especially when here is much experimenal error in he daa, such a funcion may be subjec o wild oscillaions, which are unwarraned by he underlying physics, and hus is no useful as a predicive ool. In such cases, i may be more useful o choose a lower order curve which does no exacly pass hrough all experimenal poins, bu which does minimize he error. Unweighed leas squares This is he mos common mehod used when one has equal confidence in all he daa. Example 0. Find he bes sraigh line o approximae he measured daa relaing x o. x A sraigh line fi will have he form x = a + a 2, where a and a 2 are he erms o be deermined. Subsiuing each daa poin o he assumed form, we ge five equaions in wo unknowns: 5 = a +0a 2, 7 = a +a 2, 0 = a +2a 2, 2 = a +a 2, 5 = a +a 2. This is an overconsrained problem, and here is no unique soluion ha saisfies all of he equaions! If a unique soluion exised, hen he curve fi would be perfec. However, here does exis a soluion which minimizes he error, as is ofen proved in linear algebra exbooks (and will no be proved here. The procedure is sraighforward. Rearranging, we ge 0 2 ( a a =

3 This is of he form A a = b. We hen find Subsiuing, we find ha ( a a 2 0 = A T A a = A T b, a = ( A T A AT b. ( = Sohebesfiesimaeis x = The leas squares error is A a b 2 =.920. This represens wha is known as he l 2 error norm of he predicion. In malab, his is found by he command norm(a a b wherea, a, andb are he coefficien marix A, he soluion a and he inpu vecor b, respecively. If he curve fi were perfec he error norm would be zero. A plo of he raw daa and he bes fi sraigh line is shown in Figure Daa Poins 2 x 0 8 x = Figure : Plo of x daa and bes leas squares sraigh line fi. Weighed leas squares If one has more confidence in some daa poins han ohers, one can define a weighing funcion o give more prioriy o hose paricular daa poins.

4 Example 0.2 Find he bes sraigh line fi for he daa in he previous example. Now however, assume ha we have five imes he confidence in he accuracy of he final wo daa poins, relaive o he oher poins. Define a square weighing marix W: Now we perform he following operaions: W = A a = b, W A a = W b, (W A T W A a = (W A T W b, ( a = (W A T W A (W AT W b. Wih he above values of W, direc subsiuion leads o a a = =..972 So he bes weighed leas squares fi is a 2 x = A plo of he raw daa and he bes fi sraigh line is shown in Figure weighed daa poins 4 2 x 0 8 x = Figure 2: Plo of x daa and bes weighed leas squares sraigh line fi. 4

5 When he measuremens are independen and equally reliable, W is he ideniy marix. If he measuremens are independen bu no equally reliable, W is a mos diagonal. If he measuremens are no independen, hen non-zero erms can appear off he diagonal in W. I is ofen advanageous, for insance in problems in which one wans o conrol a process in real ime, o give prioriy o recen daa esimaes over old daa esimaes and o coninually employ a leas squares echnique o esimae fuure sysem behavior. The previous example does jus ha. A famous fas algorihm for such problems is known as a Kalman Filer. Power law/logrihmic curve fis I is exremely common and useful a imes o fi daa o eiher a power law form, especially when he daa range over wide orders of magniude. For clean unis, i is highly advisable o scale boh x and by characerisic values. Someimes his is obvious, and someimes i is no. Whaever he case, he following form can usually be found x( a2 = a. x c c Here x is a dependen variable, is an independen variable, x c is a characerisic value of x (perhaps is maximum, and c is a characerisic value of (perhaps is maximum, and a and a 2 are curve fi parameers. This fi is no linear in he coefficiens, bu can be rendered so by aking he logarihm of boh sides o ge x( a2 ln =ln (a =ln(a +a 2 ln. c c x c Ofen imes one mus no include values a = 0 because of he logrihmic singulariy here. Example 0. An experimen yields he following daa: (s x(nm A plo of he raw daa is shown in Figure. Noice ha he linear plo obscures he daa a very small ime, while he log-log plo makes he rends more clear. Now o ge a curve fi for he log-log plo, we assume a power law form. We firs eliminae he poin a he origin, hen scale he daa, in his case by he maximum values of and x, and ake appropriae logarihms o ge ( ( x (s x(nm / max x/x max ln ln max x max

6 8 x a 0 0 b x (nm 4 x (nm (s (s Figure : Plo of x daa in a linear and b log-log plos. Now we prepare he sysem of linear equaions o solve ( x ln = lna + a 2 ln x max max, = ln a + a 2 (.529, 2.9 = ln a + a 2 ( 9.20, = ln a + a 2 ( 4.052, 2.4 = ln a + a 2 ( 2.02, = ln a + a 2 (0.0000, In marix form, his becomes ln a = a This is of he form A a = b.

7 As before, we muliply boh sides by A T andhensolvefora, wege a = ( A T A AT b. Solving, we find So ha or So he power law curve fi is a = ln a =0.420, a 2 = a = x( nm =.5228, 00 s or x( = (.28 0 nm s A plo of he raw daa and curve fi is shown in Figure x (nm (s Figure 4: Plo of x daa and power law curve fi: x( =(.28 0 nm ( s. 7

8 Higher order curve fis As long as he assumed form for he curve fi is linear in he coefficiens, i is sraighforward o exend o high order curve fis as demonsraed in he following example. Example 0.4 An experimen yields he following daa: x Find he leas squares bes fi coefficiens a, a 2,anda if he assumed funcional form is. x = a + a 2 + a 2 2. x = a + a 2 sin + a sin Plo on a single graph he daa poins and he wo bes fi esimaes. Which bes fi esimae has he smalles leas squares error? x = a + a 2 + a 2 We subsiue each daa poin ino he assumed form and ge he following se of linear equaions.0 = a + a 2 (0.0 + a (0.0 2,. = a + a 2 (0.7 + a (0.7 2,.8 = a + a 2 (0.9 + a (0.9 2, 2.0 = a + a 2 (.5 + a (.5 2,.5 = a + a 2 (2. + a (2. 2,. = a + a 2 ( a (.0 2, This can be rewrien as a. a = 2.0 a This is of he form A a = b. As before, we muliply boh sides by A T and hen solve for a, wege a = ( A T A AT b. Solving, we find a =

9 So he bes quadraic curve fi o he daa is The leas squares error norm is x = a + a 2 sin + a sin x( A a x 2 = This form has applied a bi of inuiion. The curve looks like a sine wave of wavelengh which has been ransposed. So we suppose i is of he form. The erm a is he ransposiion; he erm on a 2 is he fundamenal frequency which fis in he domain; he erm on a is he firs harmonic, which we have hrown in for good measure. We subsiue each daa poin ino he assumed form and ge he following se of linear equaions 0.0 (0.0.0 = a + a 2 sin + a sin, 0.7 (0.7. = a + a 2 sin + a sin, 0.9 (0.9.8 = a + a 2 sin + a sin,.5 ( = a + a 2 sin + a sin, 2. (2..5 = a + a 2 sin + a sin, (.0 (.0. = a + a 2 sin + a sin This can be rewrien as a. a = 2.0 a This is of he form A a = b. As before, we muliply boh sides by A T and hen solve for a, wege a = ( A T A AT b. Solving, we find a = So he bes curve fi of he form is x( sin sin The leas squares error norm is A a x 2 =0.5. A plo of he raw daa and he wo bes fi curves is shown in Figure 5. 9

10 2.2 2 wo-erm sinusoidal curve fi.8 x..4 quadraic polynomial curve fi Figure 5: Plo of x daa and wo leas squares curve fis x( , and x( sin ( sin. 0