INTRODUCTION TO VECTOR AND MATRIX DIFFERENTIATION Econometrics 2 Heino Bohn Nielsen Septemer 2, 2005 This note expands on appendix A7 in Vereek (2004) on matrix differentiation We first present the conventions for derivatives of scalar and vector functions; then we present the derivatives of a numer of special functions particularly useful in econometrics, and, finally, we apply the ideas to derive the ordinary least squares (OLS) estimator in the linear regression model We should emphasize that this note is cursory reading; the rules for specific functions needed in this course are indicated with a ( ) Conventions for Scalar Functions Let β (β,,β k ) 0 e a k vector and let f(β) f(β,,β k ) e a real-valued function that depends on β, ie f( ) :R k 7 R maps the vector β into a single numer, f(β) Then the derivative of f( ) with respect to β is defined as k () This is a k column vector with typical elements given y the partial derivative i Sometimes this vector is referred to as the gradient It is useful to rememer that the derivative of a scalar function with respect to a column vector gives a column vector as the result We can note that Wooldridge (2003, p783) does not follow this convention, and let row vector e a k
Similarly, the derivative of a scalar function with respect to a row vector yields the k row vector 0 k 2 Conventions for Vector Functions Now let g (β) g(β) g n (β) e a vector function depending on β (β,,β k ) 0,ie g( ) :R k 7 R n maps the k vector into a n vector, where g i (β) g i (β,,β k ), i, 2,,n, is a real-valued function Since g( ) is a column vector it is natural to consider the derivatives with respect to a row vector, β 0,ie g (β) g (β) g(β) k 0, (2) g n (β) g n (β) k where each row, i, 2,,n, contains the derivative of the scalar function g i ( ) with respect to the elements in β The result is therefore a n k matrix of derivatives with typical element (i, j) given y g i(β) If the vector function is defined as a row vector, it j is natural to take the derivative with respect to the column vector, β We can note that it holds in general that (g(β) 0 ) µ g(β) 0 0, (3) whichinthecaseaoveisak n matrix Applying the conventions in () and (2) we can define the Hessian matrix of second derivatives of a scalar function f(β) as 0 2 f(β) 0 k k k k, which is a k k matrix with typical elements (i, j) given y the second derivative 2 f(β) i j Note that it does not matter if we first take the derivative with respect to the column or the row 2
3 Some Special Functions First, let c e a k vector and let β e a k vector of parameters Next define the scalar function f(β) c 0 β, which maps the k parameters into a single numer It holds that (c 0 β) c ( ) To see this, we can write the function as f(β) c 0 β c β + c 2 β 2 + + c k β k Taking the derivative with respect to β yields (c β +c 2 β 2 ++c k β k ) (c β +c 2 β 2 ++c k β k ) k c c k c, which is a k vector as expected Also note that since β 0 c c 0 β,itholdsthat β 0 c c ( ) Now, let A e a n k matrix and let β e a k vector of parameters Furthermore define the vector function g(β) Aβ, which maps the k parameters into n function values g(β) is an n vector and the derivative with respect to β 0 is a n k matrix given y (Aβ) 0 A ( ) To see this, write the function as A β + A 2 β 2 + + A k β k g(β) Aβ, A n β + A n2 β 2 + + A nk β k and find the derivative (A β ++A k β k ) g(β) 0 (A n β ++A nk β k ) (A β ++A k β k ) k (A n β ++A nk β k ) k A A k A n A nk A Similarly, if we consider the transposed function, g(β) β 0 A 0,whichisa n row vector, we can find the k n matrix of derivatives as β 0 A 0 A 0 ( ) This is just an application of the result in (3) 3
Now consider a quadratic function f(β) β 0 Vβfor some k k matrix V This function maps the k parameters into a single numer Here we find the derivatives as the k column vector (V + V 0 )β, ( ) or the row variant 0 β 0 (V + V 0 ) ( ) If V is symmetric this reduces to 2Vβ and 2β 0 V, respectively To see how this works, consider the simple case k 3and write the function as β 0 Vβ β β 2 β 3 V V 2 V 3 V 2 V 22 V 23 V 3 V 32 V 33 V β 2 + V 22 β 2 2 + V 33 β 2 3 +(V 2 + V 2 )β β 2 +(V 3 + V 3 )β β 3 +(V 23 + V 32 )β 2 β 3 Taking the derivative with respect to β, weget 2 3 β β 2 β 3 2V β +(V 2 + V 2 )β 2 +(V 3 + V 3 )β 3 2V 22 β 2 +(V 2 + V 2 )β +(V 23 + V 32 )β 3 2V 33 β 3 +(V 3 + V 3 )β +(V 23 + V 32 )β 2 2V V 2 + V 2 V 3 + V 3 β V 2 + V 2 2V 22 V 23 + V 32 β 2 V 3 + V 3 V 23 + V 32 2V 33 β 3 V V 2 V 3 V V 2 V 3 V 2 V 22 V 23 + V 2 V 22 V 32 V 3 V 32 V 33 V 3 V 23 V 33 (V + V 0 )β β β 2 β 3 4 The Linear Regression Model To illustrate the use of matrix differentiation consider the linear regression model in matrix notation, Y Xβ +, where Y is a T vector of stacked left-hand-side variales, X is a T k matrix of explanatory variales, β is a k vector of parameters to e estimated, and is a T vector of error terms Here k is the numer of explanatory variales and T is the numer of oservations 4
One way to motivate the ordinary least squares (OLS) principle is to choose the estimator, β OLS of β, as the value that minimizes the sum of squared residuals, ie β OLS argmin β TX t 2 t argmin 0 β Looking at the function to e minimized, we find that 0 Y Xβ 0 Y Xβ Y 0 β 0 0 X Y Xβ Y 0 Y Y 0 Xβ β 0 X 0 Y + β 0 X 0 Xβ Y 0 Y 2Y 0 Xβ + β 0 X 0 Xβ, where the last line uses the fact that Y 0 Xβ and β 0 X 0 Y are identical scalar variales Note that 0 is a scalar function and taking the first derivative with respect to β yields the k vector 0 Y 0 Y 2Y 0 Xβ + β 0 X 0 X β 2X 0 Y +2X 0 Xβ Solving the k equations, ( 0 ) β 0, yields the OLS estimator β OLS X 0 X X 0 Y, provided that X 0 X is non-singular To make sure that β OLS is a minimum of 0 and not a maximum, we should formally take the second derivative and make sure that it is positive definite The k k Hessian matrix of second derivatives is given y 2 0 2X 0 Y +2X 0 X β β 0 0 2X 0 X, which is a positive definite matrix y construction References [] Vereek, Marno (2004): A Guide to Modern Econometrics, Second edition, John Wiley and Sons [2] Wooldridge, Jeffrey M (2003): Introductory Econometrics: A Modern Approach, 2nd edition, South Western College Pulishing 5