General Linear Least-Squares and Nonlinear Regression

Transcription

1 General Lnear Least-Squares and Nonlnear Regresson Berln Chen Department of Computer Scence & Informaton Engneerng Natonal Tawan Normal Unversty Reference: 1. Appled Numercal Methods wth MATLAB for Engneers, Chapter 15 & Teachng materal

2 Chapter Objectves Knowng how to mplement polynomal regresson Knowng how to mplement multple lnear regresson Understandng the formulaton of the general lnear leastsquares model Understandng how the general lnear least-squares model can be solved wth MATLAB usng ether the normal equatons or left dvson Understandng how to mplement nonlnear regresson wth optmzaton technques NM Berln Chen

3 Polynomal Regresson The least-squares procedure from Chapter 14 can be readly etended to ft data to a hgher-order polynomal. Agan, the dea s to mnmze the sum of the squares of the estmate resduals The fgure shows the same data ft wth: a) A frst order polynomal b) A second order polynomal NM Berln Chen 3

4 Process and Measures of Ft For a second order polynomal, the best ft would mean mnmzng: n n S r e y a 0 a 1 a 1 1 In general, for an m th order polynomal, ths would mean mnmzng : n n S r e y a 0 a 1 a m a m 1 1 The standard error for fttng an m th order polynomal to n data ponts s: S s y/ r n m 1 because the m th order polynomal has (m+1) coeffcents The coeffcent of determnaton r s stll found usng: r S t S r S t NM Berln Chen 4

5 Polynomal Regresson: An Eample Second Order Polynomal NM Berln Chen 5 y y y a a a n

6 Multple Lnear Regresson (1/) Another useful etenson of lnear regresson s the case where y s a lnear functon of two or more ndependent varables: y a a a a m m Agan, the best ft s obtaned by mnmzng the sum of the squares of the estmate resduals: n S r e 1 n 1 y a 0 a 1 1, a, a m m, For two dmensonal case, the regresson lne becomes a plane NM Berln Chen 6

7 Multple Lnear Regresson (/) NM Berln Chen 7

8 Multple Lnear Regresson: An Eample Eample 15. NM Berln Chen 8

9 General Lnear Least Squares Lnear, polynomal, and multple lnear regresson all belong to the general lnear least-squares model: y a 0 z 0 a 1 z 1 a z a m z m e where z 0, z 1,, z m are a set of m+1 bass functons and e s the error of the ft The bass functons can be any functon data but cannot contan any of the coeffcents a 0, a 1, etc. E.g., y a a a sn However, the followng smple-lookng model s truly nonlnear a y a 1 e 1 0 cos 0 1 NM Berln Chen 9

10 Solvng General Lnear Least Squares Coeffcents (1/) The equaton: y a 0 z 0 a 1 z 1 a z a m z m e can be re-wrtten for each data pont as a matr equaton: y Z ae where {y} contans the dependent data, {a} contans the coeffcents of the equaton, {e} contans the error at each pont, and [Z] s: z 01 z 11 z m1 Z z 0 z 1 z m z 0n z 1n z mn wth z j representng the the value of the j th bass functon calculated at the I th pont NM Berln Chen 10

11 Solvng General Lnear Least Squares Coeffcents (/) Generally, [Z] s not a square matr, so smple nverson cannot be used to solve for {a}. Instead the sum of the squares of the estmate resduals s mnmzed: n n m S r e y a j z j 1 1 j0 The outcome of ths mnmzaton process s the normal equatons that can epressed concsely n a matr form as: Z T Z a Z T y NM Berln Chen 11

12 MATLAB Eample Gven and y data n columns, solve for the coeffcents of the best ft lne for y=a 0 +a 1 +a Z = [ones(sze().^] a = (Z *Z)\(Z *y) Note also that MATLAB s left-dvde wll automatcally nclude the [Z] T terms f the matr s not square, so a = Z\y would work as well To calculate measures of ft: St = sum((y-mean(y)).^) Sr = sum((y-z*a).^) coeffcent of determnaton standard error r = 1-Sr/St sy = sqrt(sr/(length()-length(a))) NM Berln Chen 1

13 Nonlnear Regresson As seen n the prevous chapter, not all fts are lnear equatons of coeffcents and bass functons, e.g., y a 0 a 1 e 1 e One method to handle ths s to transform the varables and solve for the best ft of the transformed varables. There are two problems wth ths method Not all equatons can be transformed easly or at all The best ft lne represents the best ft for the transformed varables, not the orgnal varables Another method s to perform nonlnear regresson to drectly determne the least-squares ft, e.g., n a a y [ y a (1 a f 0, e 1 1 )] Usng the MATLAB fmnsearch functon NM Berln Chen 13

14 Nonlnear Regresson n MATLAB To perform nonlnear regresson n MATLAB, wrte a functon that returns the sum of the squares of the estmate resduals for a ft and then use MATLAB s fmnsearch functon to fnd the values of the coeffcents where a mnmum occurs The arguments to the functon to compute S r should be the coeffcents, the ndependent varables, and the dependent varables NM Berln Chen 14

15 Nonlnear Regresson n MATLAB Eample Gven dependent force data F for ndependent velocty data v, determne the coeffcents for the ft: F a 0 v a 1 Frst - wrte a functon called fssr.m contanng the followng: functon f = fssr(a, m, ym) yp = a(1)*m.^a(); f = sum((ym-yp).^); Then, use fmnsearch n the command wndow to obtan the values of a that mnmze fssr: a = fmnsearch(@fssr, [1, 1], [], v, F) where [1, 1] s an ntal guess for the [a0, a1] vector, [] s a placeholder for the optons NM Berln Chen 15