Data Modeling and Least Squares Fitting COS 323

Transcription

1 Data Modeln and Least Squares Fttn COS 33

2 Data Modeln or Reresson Gven: data ponts, unctonal orm, nd constants n uncton Eample: ven,, nd lne throuh them;.e., nd a and n a+ 3, 3 5, 5 6, 6 a+ 1, 1 7, 7, 4, 4

3 Data Modeln You mht do ths ecause ou actuall care aout those numers Eample: measure poston o alln oject, t paraola tme z 1 / t poston data ponts t, z known constant unknown Estmate rom t

4 Data Modeln or ecause some aspect o ehavor s unknown and ou want to nore t Measurn relatve resonant requenc o two ons, want to nore manetc eld drt

5 Data Modeln or to compare model tpes to ure out what knd o dependence ests Is happness lnear w.r.t. ncome?

6 Data Modeln or to make predctons Core Booker s current lead

7 Whch model s est?

8 Best-t lnes under derent metrcs Sum o resduals Sum o asolute values o resduals Mamum error o an pont

9 Least Squares Nearl unversal ut prolematc! ormulaton: mnmze squares o derences etween data and uncton Eample: to t a lne to ponts,, mnmze χ wth respect to a and a +

10 Lnear Least Squares Important specal case lso called Ordnar least squares General pattern: a + + c h + Gven,, solve or a,, c, Dependence on unknowns a,, c s lnear, ut,, etc. mht not e!

11 Lnear Least Squares Eamples General orm: a + + c h + Gven,, solve or a,, c, Lnear reresson:, 1 a * + Multple lnear reresson: a * 1 + * + c Polnomal reresson: a * + * + c

12 Lnear Least Squares Pros and Cons + Relatvel smple to compute + Eas to analze stalt / adequac o data + Gven sucent data, eactl one soluton Senstve to outlers emptaton to model nonlnear dependenc as lnear

13 How do we compute the model parameters?

14 Solvn Lnear Least Squares Prolem one smple approach ake partal dervatves: a χ + + a a 0 0 a a + + a

15 Solvn Lnear Least Squares Prolem For convenence, rewrte as matr: Factor: a a

16 lternatve Perspectve: Overconstraned ppromate Lnear Sstem here s a derent dervaton o ths: overconstraned lnear sstem has n rows and m<n columns: more equatons than unknowns Notaton: Rows o are ass unctons computed on oservatons,, s column o model parameters a,, c s column o

17 Geometrc Interpretaton or Over-determned Sstem Fnd the that comes closest to satsn.e., mnmze

18 Geometrc Interpretaton Interpretaton: nd that comes closest to satsn.e., mnmze.e., mnmze Equvalentl, nd such that r s orthoonal to span 0 r

19 Formn the equaton What are and? Row o s ass unctons computed on Row o s,, 1 1 1

20 Mnmzn Sum o Squares Fndn Closest n span Compare two epressons we ve derved: equal! a a Startn rom oal o mnmzn sum o squares Startn rom oal o ndn n span closest to outsde span

21 Great, ut how do we solve t?

22 Was o Solvn Lnear Least Squares Opton 1: or each, compute,, etc. store n row o store n compute -1-1 s known as pseudonverse o a

23 Was o Solvn Lnear Least Squares Opton : or each, compute,, etc. store n row o store n compute, solve Known as normal equatons or least squares Inecent, snce tpcall larer than and

24 Was o Solvn Lnear Least Squares Opton 3: or each, compute,, etc. accumulate outer product n U accumulate product wth n v solve Uv a U v

25 he Prolem wth Normal Equatons Involves solvn hs can e naccurate Independent o soluton method Rememer: cond cond [cond] Net week: computn pseudonverse stal More epensve, ut more accurate lso allows danosn nsucent data

26 Specal Cases

27 Specal Case: Constant Let s tr to model a uncton o the orm a

28 Specal Case: Constant Let s tr to model a uncton o the orm a Comparn to eneral orm a + + ch + we have 1 and we are solvn [ 1][ a] [ ] n a

29 Specal Case: Lne Ft to a+ 1,. So, solve: [ ] a ,, n n n a n n n Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ

30 Varant: Wehted Least Squares

31 Wehted Least Squares Common case: the, have derent uncertantes assocated wth them Want to ve more weht to measurements o whch ou are more certan Wehted least squares mnmzaton mn χ w I uncertant stdev s σ, est to take w 1 σ

32 Wehted Least Squares Dene weht matr W as hen solve wehted least squares va w w w w W W W

33 Understandn Error and Uncertant

34 Error Estmates rom Lnear Least Squares For man applcatons, ndn model s useless wthout estmate o ts accurac Resdual s Can compute χ How do we tell whether answer s ood? Lots o measurements χ s small χ ncreases quckl wth perturatons to standard varance o estmate s small

35 Error Estmates rom Lnear Least Squares Let s look at ncrease n χ : So, the er s, the aster error ncreases as we move awa rom current δ δ χ χ δ δ δ χ δ δ δ δ δ δ δ δ So,

36 Error Estmates rom Lnear Least Squares C 1 s called covarance o the data he standard varance n our estmate o s χ σ C n m hs s a matr: Daonal entres ve varance o estmates o components o : e.., vara 0 O-daonal entres eplan mutual dependence: e.., cova 0, a 1 n m s # o samples mnus # o derees o reedom n the t: consult a statstcan

37 standard devaton o mean Specal Case: Error n Constant Model standard error o a standard devaton o a : σ data : σ a n χ a 1 C n 1 n a a 1 n

38 Coecent o Determnaton R 1 R : Proporton o oserved varalt that s eplaned the model vs. just the mean e.., R 0.7 means 70% varalt eplaned For lnear reresson, R s Pearson s correlaton. χ

39 Keep n mnd In eneral, uncertant n estmated parameters oes down slowl: lke 1/sqrt# samples Formulas or specal cases lke ttn a lne are mess: smpler to thnk o orm Normal equatons method oten not numercall stale: orthoonal decomposton methods used nstead Lnear least squares s not alwas the most approprate modeln technque

40 Net tme Non-lnear models Includn lostc reresson Dealn wth outlers and ad data Practcal consderatons Is least squares an approprate method or m data?