Chapter 13 Introduction to Nonlinear Regression( 非 線 性 迴 歸 )

Transcription

1 Chapter 13 Introduction to Nonlinear Regression( 非 線 性 迴 歸 ) and Neural Networks( 類 神 經 網 路 ) 許 湘 伶 Applied Linear Regression Models (Kutner, Nachtsheim, Neter, Li) hsuhl (NUK) LR Chap 10 1 / 35

2 13 Examples of nonlinear regression model: growth from birth to maturity in human: nonlinear in nature dose-response relationships( 劑 量 反 應 關 係 ) Purpose: obtain estimates of parameters Neural network model: data mining applications ( 略 過 ) logistic regression models: Chap. 14 hsuhl (NUK) LR Chap 10 2 / 35

3 13.1 Linear and Nonlinear Regression Models Linear and Nonlinear Regression Models the general linear regression model (6.7): Y i = β 0 + β 1 X i1 + β 2 X i β p 1 X i,p 1 + ε i A polynomial regression model in one or more predictor variables is linear in the parameters. Ex: Y i = β 0 + β 1 X i1 + β 2 X 2 i1 + β 3 X i2 + β 4 X 2 i2 + β 5 X i1 X i2 + ε i log 10 Y i = β 0 + β 1 X i1 + β 2 exp(x i2 ) + ε i (transformed) hsuhl (NUK) LR Chap 10 3 / 35

4 13.1 Linear and Nonlinear Regression Models Linear and Nonlinear Regression Models (cont.) In general, we can state a linear regression model as: nonlinear regression models Y i = f (X i, β) + ε i = X iβ + ε i where X i = [1 X i1 X i,p 1 ] the same basic form as that in (13.4): Y i = f (X i, γ) + ε i f (X i, γ): mean response given by the nonlinear response function f (X, γ) ε i : error term; assumed to have E{ε i } = 0, constant variance and to be uncorrelated γ: parameter vector hsuhl (NUK) LR Chap 10 4 / 35

5 13.1 Linear and Nonlinear Regression Models Linear and Nonlinear Regression Models (cont.) exponential regression model: in growth( 生 長 ) studies; concentration( 濃 度 ) Y i = γ 0 exp(γ 1 X i ) + ε i, ε i indep. N(0, σ 2 ) hsuhl (NUK) LR Chap 10 5 / 35

6 13.1 Linear and Nonlinear Regression Models Linear and Nonlinear Regression Models (cont.) logistic regression models( 邏 輯 式 迴 歸 模 型 ): in population studies( 人 口 研 究 ); Y i = γ γ 1 exp(γ 2 X i ) + ε i, ε i indep. N(0, σ 2 ) hsuhl (NUK) LR Chap 10 6 / 35

7 13.1 Linear and Nonlinear Regression Models Linear and Nonlinear Regression Models (cont.) logistic regression models( 邏 輯 式 迴 歸 模 型 ): the response variable is qualitative (0,1): purchase a new car the error terms are not normally distributed with constant variance (Chap. 14) hsuhl (NUK) LR Chap 10 7 / 35

8 13.1 Linear and Nonlinear Regression Models Linear and Nonlinear Regression Models (cont.) General Form of Nonlinear Regression Models: The error terms ε i are often assumed to be independent normal random variables with constant variance. Important difference: #{β i } is not necessarily directly related to #{X i } in the model linear regression: (p 1) X variables p regression coefficients nonlinear regression: p 1 Y i = β j X ji + ε i j=0 Y i = γ 0 exp(γ 1 X i ) + ε i (p = 2, q = 1) hsuhl (NUK) LR Chap 10 8 / 35

9 13.1 Linear and Nonlinear Regression Models Linear and Nonlinear Regression Models (cont.) Nonlinear regression model q: #{X variables} p: #{regression parameters} X i : the observations on the X variables without the initial element 1 The general form of a nonlinear regression model: Y i = f (X i, γ) + ε i X i = q 1 X i1 X i2. γ = p 1 γ 0 γ 1. X iq γ p 1 hsuhl (NUK) LR Chap 10 9 / 35

10 13.1 Linear and Nonlinear Regression Models Linear and Nonlinear Regression Models (cont.) intrinsically linear( 實 質 線 性 的 ) response functions: nonlinear response functions can be linearized by a transformation Ex: f (X, γ) = γ 0 [exp(γ 1 X)] g(x, γ) = log e f (X, γ) = β 0 + β 1 X β 0 = log γ 0, β 1 = γ 1 Just because a nonlinear response function is intrinsically linear does not necessarily imply that linear regression is appropriate. ( the error term in the linearized model will no longer be normal with constant variance) hsuhl (NUK) LR Chap / 35

11 13.1 Linear and Nonlinear Regression Models Linear and Nonlinear Regression Models (cont.) Estimation of regression parameters: 1 least squares method 2 maximum likelihood method Also as in linear regression, both of these methods of estimation yield the same parameter estimates when the error terms in (13.12) are independent normal with constant variance. It is usually not possible to find analytical expression for LSE and MLE for nonlinear regression models. numerical search procedures must be used: require intensive computations hsuhl (NUK) LR Chap / 35

12 13.2 Least Squares Estimation in Nonlinear Regression LSE in nonlinear regression The concepts of LSE for linear regression also extend directly to nonlinear regression models. The least squares criterion: n Q = [Y i f (X i, γ)] 2 i=1 Q must be minimized with respect to γ 0,..., γ p 1 A difference from linear regression is that the solution of the normal equations usually requires an iterative numerical search procedure because analytical solutions generally cannot be found. hsuhl (NUK) LR Chap / 35

13 13.2 Least Squares Estimation in Nonlinear Regression Solution of Normal Equations Y i = f (X i, γ) + ε i n Q = [Y i f (X i, γ)] 2 i=1 g = arg min γ Q (g : the vector of the LSE g k ) Q γ k = (partial derivative of Q with respect to γ k ) [ n f (X i, γ) 2[Y i f (X i, γ)] γ k i=1 ] set = 0 γk =g k hsuhl (NUK) LR Chap / 35

14 13.2 Least Squares Estimation in Nonlinear Regression Solution of Normal Equations (cont.) The p normal equations: [ ] [ ] n f (X i, γ) n f (X i, γ) Y i f (X i, g) i=1 γ k γ=g i=1 γ k γ=g k = 0, 1,..., p 1 = 0, g: the vector of the least squares estimates g k g = [g 0,..., g p 1 ] p 1 nonlinear in the parameter estimates g k numerical search procedure are required multiple solution may be possible hsuhl (NUK) LR Chap / 35

15 13.2 Least Squares Estimation in Nonlinear Regression Solution of Normal Equations (cont.) Example: Severely injured patients Y : Prognostic( 預 後 的 ) Index X: Days Hospitalized ( 住 院 治 療 ) Related earlier studies: the relationship between Y and X is exponential Y i = γ 0 exp(γ 1 X i ) + ε i hsuhl (NUK) LR Chap / 35

16 13.2 Least Squares Estimation in Nonlinear Regression Solution of Normal Equations (cont.) f (X i, γ) γ 0 = exp(γ 1 X i ) f (X i, γ) γ 1 = γ 0 X i exp(γ 1 X i ) Q γ k g = 0 Y i exp(g 1 X i ) g 0 exp(2g1 X i ) = 0 Yi X i exp(g 1 X i ) g 0 Xi exp(2g 1 X i ) = 0 No closed-form solution exists for g = (g 0, g 1 ) T hsuhl (NUK) LR Chap / 35

17 13.2 Least Squares Estimation in Nonlinear Regression Gauss-Newton Method( 高 斯 - 牛 頓 法 ) linearization method: 1 use a Taylor series expansion to approximate the nonlinear regression model ( f (x) = f (a) + n=1 2 employ OLS to estimate the parameters f (n) (x a)n) n! hsuhl (NUK) LR Chap / 35

18 13.2 Least Squares Estimation in Nonlinear Regression Gauss-Newton Method( 高 斯 - 牛 頓 法 ) (cont.) 圖 片 來 源 :Wiki: Taylor s theorem hsuhl (NUK) LR Chap / 35

19 13.2 Least Squares Estimation in Nonlinear Regression Gauss-Newton Method( 高 斯 - 牛 頓 法 ) (cont.) Gauss-Newton Method: 1 initial parameters γ 0,..., γ p 1 : g (0) 0,..., g (0) p 1 2 approximate the mean responses f (X i, γ) in the Taylor series expansion around g (0) k : f (X i, γ) f (X i, g (0) ) + p 1 k=0 [ ] f (X i, γ) γ k γ=g (0) (γ k g (0) k ) 3 obtain revised estimated regression coefficients g (1) k : (later) g (1) k = g (0) k + b (0) k hsuhl (NUK) LR Chap / 35

20 13.2 Least Squares Estimation in Nonlinear Regression Gauss-Newton Method( 高 斯 - 牛 頓 法 ) (cont.) (Y (0) i =Y i f (0) i ) p 1 f (X i, γ) f (X i, g (0) [ ] f (Xi, γ) ) + (γ k g (0) k γ ) k γ=g (0) = f (0) p 1 i + k=0 k=0 D (0) ik β(0) k Y i f (0) p 1 i + D (0) ik β(0) k + ε i = Y (0) i p 1 k=0 k=0 D (0) ik β(0) k + ε i (no intercept) (13.24) The purpose of fitting the linear regression model approximation (13.24) is therefore to estimate β (0) k and use these estimates to adjust the initial starting estimates of the regression parameters. hsuhl (NUK) LR Chap / 35

21 13.2 Least Squares Estimation in Nonlinear Regression Gauss-Newton Method( 高 斯 - 牛 頓 法 ) (cont.) Matrix Form: Y (0) i p 1 k=0 D(0) ik β(0) k + ε i hsuhl (NUK) LR Chap / 35

22 13.2 Least Squares Estimation in Nonlinear Regression Gauss-Newton Method( 高 斯 - 牛 頓 法 ) (cont.) the D matrix of partial derivative play the role of the X matrix (without a column of 1s for the intercept) Estimate β (0) by OLS: b (0) = (D (0) D (0) ) 1 D (0) Y (0) obtain revised estimated regression coefficients g (1) k : g (1) k = g (0) k + b (0) k hsuhl (NUK) LR Chap / 35

23 13.2 Least Squares Estimation in Nonlinear Regression Gauss-Newton Method( 高 斯 - 牛 頓 法 ) (cont.) Evaluated for g (0) by SSE (0) : n n SSE (0) = [Y i f (X i, g (0) )] 2 = [Y i f (0) i ] 2 i=1 i=1 After the end of the first iteration: n n SSE (1) = [Y i f (X i, g (1) )] 2 = [Y i f (1) i ] 2 i=1 i=1 If the Gauss-Newton method is working effectively in the first iteration, SSE (1) should be smaller than SSE (0). ( g (1) should be better estimates) hsuhl (NUK) LR Chap / 35

24 13.2 Least Squares Estimation in Nonlinear Regression Gauss-Newton Method( 高 斯 - 牛 頓 法 ) (cont.) The Gauss-Newton method repeats the procedure with g (1) now used for the new starting values. Until g (s+1) g (s) and/or SSE (s+1) SSE (s) become negligible The Gauss-Newton method works effectively in many nonlinear regression applications. (Sometimes may require numerous iterations before converging.) hsuhl (NUK) LR Chap / 35

25 13.2 Least Squares Estimation in Nonlinear Regression Gauss-Newton Method( 高 斯 - 牛 頓 法 ) (cont.) Example: Severely injured patients Initial: Transformed Y (log γ 0 exp(γ 1 X) = log γ 0 + γ 1 X) Y i = β 0 + β 1 X i + ε i OLS b 0 = , b 1 = g (0) 0 = exp(b 0 ) = , g (0) 1 = b 1 = hsuhl (NUK) LR Chap / 35

26 13.2 Least Squares Estimation in Nonlinear Regression Gauss-Newton Method( 高 斯 - 牛 頓 法 ) (cont.) Ŷ = ( ) exp( X) hsuhl (NUK) LR Chap / 35

27 13.2 Least Squares Estimation in Nonlinear Regression Gauss-Newton Method( 高 斯 - 牛 頓 法 ) (cont.) The choice of initial starting values: a poor choice may result in slow convergence, convergence to a local minimum, or even divergence Good starting values: result in faster convergence, will lead to a solution that is the global minimum rather than a local minimum A variety of methods for obtaining starting values: 1 related earlier studies 2 select p representative observations solve for p parameters, then used as the starting values 3 do a grid search in the parameter space using as the starting values that g for which Q is smallest hsuhl (NUK) LR Chap / 35

28 13.2 Least Squares Estimation in Nonlinear Regression Gauss-Newton Method( 高 斯 - 牛 頓 法 ) (cont.) Some properties that exist for linear regression least squares do not hold for nonlinear regression least squares. Ex: e i 0; SSR + SSE SSTO; R 2 is not a meaningful descriptive statistic for nonlinear regression. Two other direct search procedures: 1 The method of steepest descent searches 2 The Marquardt algorithm: seeks to utilize the best feature of the Gauss-Newton method and the method of steepest descent a middle ground( 折 衷 ; 妥 協 ) between these two method hsuhl (NUK) LR Chap / 35

29 13.3 Model Building and Diagnostics Model building and diagnostics The model-building process for nonlinear regression models often differs somewhat from that for linear regression models Validation of the selected nonlinear regression model can be performed in the same fashion as for linear regression models. Use of diagnostics tools to examine the appropriateness of a fitted model plays an important role in the process of building a nonlinear regression model. hsuhl (NUK) LR Chap / 35

30 13.3 Model Building and Diagnostics Model building and diagnostics (cont.) When replicate observations are available and the sample size is reasonably large, the appropriate of a nonlinear regression function can be tested formally by means of the lack of fit test for linear regression models. (the test is an approximate one) Plots: e i vs. t i, Ŷ i, X ik can be helpful in diagnosing departures from the assumed model Unequal variances WLS; transformations hsuhl (NUK) LR Chap / 35

31 13.4 Inferences about Nonlinear Regression Parameters Inferences Inferences about the regression parameters in nonlinear regression are usually based on large-sample theory. When n is large, LSE and MLE for nonlinear regression models with normal error terms are approximately normally distributed and almost unbiased and have almost minimum variance Estimate of Error Term Variance: MSE = SSE n p = [Yi f (X i, g)] 2 n p (Not unbiased estimator of σ 2 but the bias is small when n is large) hsuhl (NUK) LR Chap / 35

32 13.4 Inferences about Nonlinear Regression Parameters Inferences (cont.) Large-Sample Theory When the error terms ε i are independent N(0, σ 2 ) and the sample size n is reasonably large, the sampling distribution of g is approximately normal. The expected value of the mean vector is approximately: E{g} γ The approximate variance-covariance matrix of the regression coefficients is estimated by: s 2 {g} = MSE(D D) 1 hsuhl (NUK) LR Chap / 35

33 13.4 Inferences about Nonlinear Regression Parameters Inferences (cont.) No simple rule exists that tells us when it is appropriate to use the large-sample inference methods and when it is not appropriate. However, a number of guidelines have been developed that are helpful in assessing the appropriateness of using the large-sample inference procedures in a given application. When the diagnostics suggest that large-sample inference procedures are not appropriate in a particular instance, remedial measures should be explored. reparameterize the nonlinear regression model bootstrap estimated of precision and confidence intervals instead of the large-sample inferences hsuhl (NUK) LR Chap / 35

34 13.4 Inferences about Nonlinear Regression Parameters Interval Estimation Large-sample theorem: approximate result for a single γ k g k γ k t(n p), k = 0, 1,..., p 1 s{g k } g k ± t(1 α/2; n p)s{g k } Several γ k : m parameters to be estimated with approximate family confidence coefficient 1 α the Bonferroni confidence limits: g k ± Bs{g k } B = t(1 α/2m; n p) hsuhl (NUK) LR Chap / 35

35 13.4 Inferences about Nonlinear Regression Parameters Test Concerning a single γ k A large-sample test: H 0 : γ k = γ k0 vs. H a : γ k γ k0 t = g k γ k0 s{g k } If t t(1 α/2; n p), conclude H 0 If t > t(1 α/2; n p), conclude H a Test concerning several γ k F = approx SSE(R) SSE(F ) df R df F MSE(F ) F (df R df F, df F ) when H 0 holds hsuhl (NUK) LR Chap / 35