Orthogonal Projection in Teaching Regression and Financial Mathematics

Journal of Statistics Education Volume 8 Number () Orthogonal Projection in Teaching Regression and Financial Mathematics Farida Kachapoa Auckland Uniersity of Technology Ilias Kachapo Journal of Statistics Education Volume 8 Number () wwwamstatorg/publications/jse/8n/kachapoapdf Copyright by Farida Kachapoa and Ilias Kachapo all rights resered This tet may be freely shared among indiiduals but it may not be republished in any medium without epress written consent from the authors and adance notification of the editor Key Words: Population model of regression; Regression in finance; Optimal portfolio; Minimizing risk Abstract Two improements in teaching linear regression are suggested The first is to include the population regression model at the beginning of the topic The second is to use a geometric approach: to interpret the regression estimate as an orthogonal projection and the estimation error as the distance (which is minimized by the projection) Linear regression in finance is described as an eample of practical applications of the population regression model The paper also describes a geometric approach to teaching the topic of finding an optimal portfolio in financial mathematics The approach is to epress the optimal portfolio through an orthogonal projection in Euclidean space This allows replacing the traditional solution of the problem with a geometric solution so the proof of the result is merely a reference to the basic properties of orthogonal projection This method improes the teaching of the topic by aoiding tedious technical details of the traditional solution such as Lagrange multipliers and partial deriaties The described method is illustrated by two numerical eamples Introduction In this paper we demonstrate how the concepts of ector space and orthogonal projection are used in teaching linear regression and some topics in financial mathematics Through geometric interpretations the proofs are made shorter and clearer

Journal of Statistics Education Volume 8 Number () In Section we remind the reader of some basic facts from linear algebra about orthogonal projection In Section we describe the Euclidean space of random ariables and discuss meanings of the term independence in different contets Sections 6 and 7 describe some improements in teaching linear regression Many statistics courses consider regression as fitting lines to data Modern tetbooks for eample Wild and Seber () Chatterjee () Moore and McCabe (6) teach applied statistics without referring to probability and mathematical foundations of the statistical methods Computer software is widely used for data analysis which makes the analysis easier but also turns it into a mysterious process The population model of regression is sometimes taught in courses on probability theory for eample Hsu (997) Grimmett and Stirzaker () But this model is rarely considered in statistics courses These courses teach regression only for samples or briefly mention the population model in a descriptie way They introduce formulas for estimates of the regression coefficients without considering formulas for the coefficients themseles This leads to long definitions and tedious proofs (or lack of the proofs) So instead of focussing on the idea of regression the students concentrate on long calculations or specifications of particular computer software There are at least two benefits of discussing the population model of regression The first is that it makes some of the ideas in regression clear because sample estimates are natural analogs of features in the population The second is that this facilitates discussion of an interesting application in financial mathematics which will be discussed in Section 5 This paper describes a geometric approach in teaching regression when the regression estimate is interpreted as an orthogonal projection and the residual is interpreted as the distance from the projection The geometric approach is used in tetbooks on regression to some etent but the projection there is completely different: it relates to samples and n- dimensional space R n while in this paper the projection relates to the population and linear space of random ariables In Section 8 we describe another application of orthogonal projection in financial mathematics One of the problems of portfolio analysis is finding an optimal portfolio the portfolio with the lowest risk for a targeted return This problem is included in tetbooks on financial modelling (Benninga ; Francis and Taylor ) often without mathematical justification of the result When a mathematical solution is proided the problem is treated as a minimization problem and the solution is found in coordinate form using Lagrange multipliers and partial deriaties (Teall and Hasan ; Cheang and Zhao ; Kachapoa and Kachapo 5; Kachapoa and Kachapo 6) Here we suggest an inariant solution that uses geometric approach to random ariables and orthogonal projection in particular Geometric Facts about Orthogonal Projection In this section we will fi a Euclidean space L Definition Suppose is a ector in L and W is a linear subspace of L A ector z is called the orthogonal projection of onto W if ) zw and ) ( z) W Then z is denoted Pr oj W Pr oj W W

Journal of Statistics Education Volume 8 Number () The following is a well-known fact in linear algebra Theorem ) Pr oj W is the ector in W closest to and it is the only ector with this property ) If n is an orthogonal basis in W then ProjW n n where (u ) denotes the scalar product of ectors u and Definition A subset Q of a linear space B is called an affine subspace of B if there is qq and a linear subspace W of B such that Q = {q + w ww } Then W is called the corresponding linear subspace It is easy to check that any ector in Q can be taken as q Theorem Let Q = {q + w ww } be an affine subspace of L Then the ector in Q with the smallest length is unique and is gien by the formula n n Proof of Theorem min = q Pr oj W q Q q min W O Pr oj W q Denote z = Pr oj W q then min = q z Consider any ector yq For some ww y = q + w By Theorem ) z is the ector in W closest to q and ww so we hae y = q ( w) q z = min The equality holds only when w = z that is when y = q + w = q z = min Since min is unique it does not depend on the choice of q

Journal of Statistics Education Volume 8 Number () Geometric Approach to Random Variables The Vector Space of Random Variables We will consider random ariables on the same probability space The set H of all random ariables with finite ariances is a linear space (with obious operations of addition and multiplication by a number) We denote = E() the epectation of a random ariable = Var () the ariance of and Co ( Y) the coariance of random ariables and Y A scalar product gien by ( Y) = E ( Y) makes H a Euclidean space In this space the length of a ector is gien by = ectors and Y is gien by d ( Y) = Y = E and the distance between A similar approach is used by Grimmett and Stirzaker ( pg -7) but they do not introduce scalar product on random ariables Howeer the scalar product is ery releant to orthogonal projections and makes proofs shorter In simple cases we can construct a basis of the space H The following eample illustrates that Eample Consider a finite sample space = { n } with the probabilities of the outcomes p i = P( i ) > i = n In this case we can introduce a finite orthogonal basis in the Euclidean space H if For each i define a random ariable F i as follows: F i ( j ) = if Then for any random ariable in H n j i j i = i F i () i where i = ( i ) For any i j F i F j = and (F i F j ) = E(F i F j ) = so F i F j () () and () mean that F F n make an orthogonal basis in H and the dimension of H is n For any Y in H their scalar product equals ( Y) = p i i y i where y i = Y( i ) The Concepts of Independence The authors think it is worthwhile to discuss the concepts of independence and dependence that students encounter in different contets There is independence of eents which we do not use here For the current topics it is important that the students distinguish between independence/ dependence of random ariables their linear relation and their linear dependence as ectors in H We will remind the definitions for only two ariables for breity Definition Suppose and Y are elements of H n i

Journal of Statistics Education Volume 8 Number () ) The random ariables and Y are called independent if for any numbers y P( Y y) = P( ) P(Y y) Otherwise the random ariables are called dependent ) The random ariables and Y are said to hae a linear relation if Y = + or = + Y for some numbers Equialently it means that there are numbers a b not both zero and a number c for which a + by = c ) The ectors and Y in a linear space are called linearly dependent if there are numbers a b not both zero for which a + by = Otherwise the ectors are called linearly independent Obiously linear dependence ) implies linear relation ) and linear relation ) implies dependence ) Eample Let us look at the ariables F F n described in Eample They are dependent as random ariables (Definition ) because P(F = F = ) = and P(F = ) P(F = ) = P( ) P( ) = p p Net the random ariables F F n hae a linear relation (Definition ) because F ++ F n = Indeed for any i (F ++ F n )( i) = F i ( i) = Finally F F n are linearly independent as ectors (Definition ) because if a F ++ a n F n = then for any i = n = (a F ++ a n F n)( i) = a i F i ( i) = a i To add to the students confusion there are also terms independent ariable and dependent ariable in statistics which are not strictly defined but intuitiely understood In statistics we also talk about linear dependence of random ariables measured by their correlation coefficient This term linear dependence is neer properly defined but the Y theory states that the linear dependence between and Y gets weaker as Y approaches and does not eist when Y = This linear dependence has an interesting geometric analogy for ariables with epectations If and Y are such ariables in H then ( Y) = Co ( Y) = Y = Y and for the angle between ectors and Y Cos = Y Co Y Y Y Y It is easy to proe the following facts using properties of the correlation coefficient as a measure of linear dependence between random ariables 5

Journal of Statistics Education Volume 8 Number () Theorem Suppose and Y are ariables in H with epectations and is the angle between them as ectors Then ) Cos = Y ) the random ariables and Y hae a linear relation if and only if the angle equals or 8 ; ) if and Y are independent then they are orthogonal; ) there is no linear dependence between random ariables and Y if they are orthogonal ectors; 5) with angle getting closer to 9 there is less linear dependence remaining between and Y The Population Model of Regression and a Geometric Approach to Teaching Regression The idea of regression is clear and simple when it is applied to random ariables and epressed in terms of a population not samples Regression apparently means estimating an inaccessible random ariable Y in terms of an accessible random ariable (Hsu 997) that is finding a function f () closest to Y We call this the population model of regression f () can be restricted to a certain class of functions the most common being the class of linear functions We describe closest in terms of the distance d defined in Section Theorem ) states that Pr oj W Y is the ector in W closest to Y Choosing different W s we can get different types of regression: simple linear multiple linear quadratic polynomial etc Theorem The conditional epectation E(Y ) is the function of closest to Y This is based on the following fact: E(Y ) = Pr oj W Y for W = { f () f: R R and f () H} Grimmett & Stirzaker () on pg 6 proe the fact by showing that E(Y )W and that for any h () W E[(Y E(Y )) h ()] = that is Y E(Y ) h () Theorem 5 (simple linear regression) If then the linear function of closest to Y is gien by Co Y + where () Y Corollary If Ŷ = + is the best linear estimator of Y from Theorem 5 then the residual = Y Ŷ has the following properties: ) = ) Co ( ) = 6

Journal of Statistics Education Volume 8 Number () Thus according to the Corollary the residuals (estimation errors) equal on aerage and are uncorrelated with the predictor ; this is another eidence that Ŷ is the best linear estimator of Y Geometric proofs of Theorem 5 and its Corollary will be gien in Section 6 5 Application in Financial Mathematics Regression is often taught only for samples without een considering the population model for random ariables Perhaps some people beliee that the population model is not used in real life We use the following application of regression in finance to demonstrate to students that the population regression model is a practical concept Portfolio analysis is the part of financial mathematics that studies the world of N fied assets A A A N and their combinations called portfolios The % return (we will omit % for breity) from an inestment is treated as a random ariable We will identify any portfolio with its return and also denote the return by For a portfolio = E () is called the epected return and the ariance = Var () is used as a measure of risk Despite the tradition to use capital letters for random ariables in portfolio analysis it is more suitable to use small letters for portfolio returns All portfolios are regressed to the market portfolio m the portfolio containing eery asset with the weight proportional to its market alue Thus for any portfolio the following is true ) = + m + The regression line + m is the linear function of m closest to So the coefficient is the aerage rate of change of s return with respect to the market return ) For the residual = and Co ( m) = ) The ariance represents the total risk of portfolio ; m is the systematic risk (or market risk) of the risk that affects most inestments; is the unsystematic risk of the risk that affects only a small number of inestments The total risk of is a sum of the systematic risk and unsystematic risk: = m + Indeed = Var () = Var ( + m + ) = Var ( m + ) = = Var (m) + Var () + Co ( m) = m + since Co ( m) = ) The coefficient m of correlation between the s return and the market return has the following interpretation 7

Journal of Statistics Education Volume 8 Number () Theorem 6 m is the proportion of the systematic risk of portfolio Proof of Theorem 6 systematic risk of total risk of m m (by Theorem 5) = = Co m m m Co m m m Clearly β = m m systematic risk of So the beta coefficient is used risk of the market portfolio for ordinal ranking of assets according to their systematic risk In particular an asset with > is called an aggressie asset (it is more olatile than the market portfolio) and an asset with < is called a defensie asset (it is less olatile than the market portfolio) The following table shows the estimated beta coefficients of some companies in June 6: Company Name Beta coefficient Coca-Cola Company 6 Honda Motor Company 65 Toyota Motor Corporation 7 Telecom Corporation NZ 7 Vodafone Group PLC Harley-Daidson Sony Corporation 9 Microsoft Corporation Boeing Hilton Hotels McDonald s Corporation 9 General Motors Apple Computer 5 Nokia 79 Ford Motor Company 8 ero Corporation 9 Yahoo! Inc 5 5) All this leads to a ery important model in finance the capital asset pricing model: = f + ( m f ) Here f is the risk-free return and m is the epected return of the market portfolio For more details on regression in finance see Kachapoa and Kachapo (6) 8

Journal of Statistics Education Volume 8 Number () 6 Geometric Proofs Adanced tetbooks on regression (Seber 98; Saille and Wood 996; Freund 998; Seber and Lee ; Chiang ; Dowdy ) contain mathematical proofs of the regression model that use the geometric approach with samples rather than the population Such proofs inole coordinates matrices and partial deriaties and either are ery long or contain significant gaps So it would be beneficial for the students to proide shorter proofs The authors beliee that the geometric proof below for the coefficients of simple linear regression is shorter and conceptually clearer than the usual proofs minimising mean-square error Proof of Theorem 5 Denote W = { a + b a br} Proj W Y W so Pr oj W Y = + for some R We just need to show that and are gien by the formula () For = Y Pr oj W Y = Y ( + ) we hae and since W So ( ) = and ( ) = ( + ) = (Y ) and ( + ) = (Y ) which leads to a system of linear equations: E E Y E Y E and Y E E Y The solution of the system is gien by () The following corollary of Theorem 5 was stated in Section Corollary If Ŷ = + is the best linear estimator of Y from Theorem 5 then the residual = Y Ŷ has the following properties: Proof of Corollary ) so E() = ) = ) Co ( ) = ) so E() = and Co ( ) = E() E() E() = The Corollary leads to a different interpretation of regression It is useful in teaching students who are not familiar with the concept of orthogonal projection The random ariable Y is diided into two parts: Y = f () + and it is required that = and Co ( ) = The part f () is called the regression estimate of Y 9

Journal of Statistics Education Volume 8 Number () Theorem 5a The regression estimate of Y in the class of all linear functions of is gien by the formula () Proof of Theorem 5a Y = + + Co (Y ) = Co ( ) + Co ( ) + Co ( ) = = + + = So Co Y Y = + + = + so = Y Unlike Theorem 5 Theorem 5a is not stated in terms of closest object But it has a ery easy proof 7 Linear Regression for Samples After the population regression model is introduced we create the statistics regression model as a sample estimate We follow the common pattern in estimation theory when a population object is estimated from a sample For eample the population mean is estimated by a n i i sample mean Similarly the equation Y = + + of the simple linear n regression is estimated from a sample by the equation Y = a + b + e where a b and e are sample estimates of and respectiely Substituting the corresponding sample estimates for the parameters in () we get formulas for the coefficients a and b: b a s s y y b where y s and s y are the sample estimates of Y and Co Y respectiely 8 Optimal Portfolio of Financial Assets We fi N financial assets A A A N 8 Modelling Portfolios as Random Variables Notations r k denotes the return of asset A k ; k = E(r k ) k = Var (r k ) and k j = Co (r k r j ) U = is the column of ones of length N;

Journal of Statistics Education Volume 8 Number () M = N is the ector of the epected asset returns where not all k are the same; S N N N N NN is the coariance matri of the asset returns We will assume that all epected returns μ μ N are defined and the coariance matri S eists with det S > The positiity of the determinant of S is equialent to the fact that the random ariables r r N do not hae a linear relation (recall Definition ) This also means that r r N are linearly independent as ectors For a portfolio k denotes the proportion of the alue of asset A k in the portfolio s total alue (negatie k means short sales) Theorem 7 For a portfolio N ) = r k k ; k ) + + N = Let us consider the set K of all linear combinations of r r N Apparently K is a linear subspace of the Euclidean space H of random ariables described in Section Theorem 8 ) r r N is a basis in K ) The dimension of K equals N According to Theorems 7 and 8 any portfolio is a ector in the N-dimensional Euclidean space K and can be represented as a column of its coordinates in the basis r r N : = N y Theorem 9 For any portfolios = and y = the following hold: N y N ) = E () = + + N N = T M; ) = Var () = T S ; ) Co ( y ) = T S y

Journal of Statistics Education Volume 8 Number () 8 The Problem of Optimal Portfolio Suppose an inestor wants a certain return c from a portfolio Usually one can choose between many portfolios with the same epected return Clearly the inestor wants to pick the portfolio with the lowest risk We will call it the optimal portfolio Risk is measured with ariance Thus we get the following definition Definition The optimal portfolio for targeted return c is the portfolio with the smallest ariance among the portfolios with the same epected return c Thus the optimal portfolio minimizes risk for a targeted return A ector = is a portfolio with epected return c if it satisfies the following two N conditions: N c These conditions can be written in matri form: T T U M Thus in mathematical form the problem of optimal portfolio can be written as follows: Var min T U T M c c () 8 Geometric Solution of the Problem of Optimal Portfolio Denote Q the set of all solutions of the system of linear equations () and W the set of all solutions of the corresponding homogeneous system: T T U M (5) Clearly W is a linear subspace of K of dimension N (note that U and M are not proportional) and Q is an affine subspace of K with W as the corresponding linear subspace That is if q is any solution of () then Q = {q + w ww } Here we assume N The case N = is triial: in this case there is only one portfolio with the epected return c Theorem The optimal portfolio for the targeted return c is unique and is gien by the formula min = q Pr oj W q where q is any solution of () and W is the set of all solutions of (5)

Journal of Statistics Education Volume 8 Number () Proof of Theorem For any solution of () we hae = ( ) = E ( ) = + = Var () + c Since = c is fied the solution of () with the smallest ariance is the same as the solution of () with the smallest length So Theorem can be applied Theorem Suppose q is a solution of () and N is an orthogonal system of ectors each of which is a solution of (5) Then ) the optimal portfolio for the targeted return c is gien by the formula min = q q N k k k k ) for any ector y (y k) = Co (y k) k = N Proof of Theorem ) The dimension of W is N so the orthogonal system N makes a basis in W and the formula follows from Theorem and Theorem ) ) Since each k is a solution of (5) E( k) = k T M = (y k) = E (y k) = Co (y k) + E (y) E ( k) = Co (y k) + E (y) = Co (y k) Eample Three assets hae epected returns of % % and % respectiely and coariance matri S 5 If the targeted return is % find a) the portfolio of these assets with the lowest risk and b) its ariance k ; Solution a) N = M = c = The system () has the form q = is a solution of this system (we assign an arbitrary alue to one of the ariables eg take = and sole for the other two ariables) Similarly we find a solution of the homogeneous system = :

Journal of Statistics Education Volume 8 Number () By Theorem min = q q (q ) = Co (q ) and ( ) = Var () So (q ) = q T S = 5 = ; ( ) = T S = 5 = 8; min = q q = 8 = 5 5 min = 5 8 b) The ariance of min equals mint S min = 5 Eample Four assets hae epected returns of % % % and % respectiely and coariance matri S If the targeted return is % find a) the portfolio of these assets with the lowest risk and b) its ariance Solution a) N = M = c = The system () has the form q = is a solution of this system (we assign arbitrary alues to two ariables eg take = = and sole for the other two ariables) Similarly we find a solution of the homogeneous system :

Journal of Statistics Education Volume 8 Number () 5 = Net we need to find a solution = of the same system that is orthogonal to Thus = ( ) = Co ( ) = T S = = = + So is found as a solution of the system : = By Theorem min = q q q (q ) = Co (q ) = q T S = = ; (q ) = Co (q ) = q T S = = ; ( ) = Var ( ) = T S = = ;

Journal of Statistics Education Volume 8 Number () ( ) = Var ( ) = T S = = q So min = q 6 min = 5 q = = 5 75 5 5 b) The ariance of min equals mint S min = 5 9 Discussion Most students want to know reasons for the formulas and equations that they study so omitting all the proofs can be frustrating to students with sufficient mathematical background to understand them When proofs are presented the authors suggest making them short and clear Using the approach described we justify basic formulas for regression and at the same time aoid lengthy and tedious proofs Clearly this is not applicable to the statistical inference for regression where tedious proofs are hard to simplify When considering the population regression model and applying orthogonal projection we clarify the main idea of regression as the estimation of Y in terms of This is a logical way to teach regression that we beliee improes the students critical thinking and conceptual knowledge of regression as a complement to the procedural knowledge proided in traditional statistics courses The suggested teaching approach requires the students to hae some mathematical background Firstly they need to hae some intuition about such concepts of linear algebra as scalar product orthogonal projection length and distance (the last three concepts are intuitiely clear for two-dimensional and three-dimensional spaces) Secondly the students need to be familiar with such concepts of probability theory as random ariables ariance and coariance and hae basic skills in applying them Therefore the suggested approach can be effectie only in the statistics courses which are part of uniersity courses in quantitatie areas such as mathematical studies physics and engineering The authors beliee that these students are capable of abstract thinking and will appreciate the logic and structure of this approach The authors hae used the approach described to teach regression in courses on statistics probability theory and financial mathematics at the Auckland Uniersity of Technology (New Zealand) and the Moscow Technological Uniersity (Russia) Though a formal statistical analysis of the results is yet to be done our case studies show that the students gained a better understanding of the concept of regression regression formulas and their logical connections 6

Journal of Statistics Education Volume 8 Number () In particular they demonstrated an understanding that a regression line constructed from empirical data is only an approimation of the true relationship between two ariables and that a different set of data may lead to a different approimation of the same relationship The second part of the paper (Section 8) describes a geometric approach to teaching the problem of optimal portfolio in a uniersity course on financial mathematics The courses on financial mathematics deelop financial theories using mathematical techniques of calculus probability theory stochastic differentiation and integration Here we apply the geometric technique of orthogonal projection to the problem of optimal portfolio The use of the orthogonal projection makes the reasoning for the problem short and inariant while the traditional solution for the optimal portfolio is long and inoles coordinates and partial deriaties The new method helps the students to concentrate on meaningful modelling instead of tedious technical details This is especially helpful for the students whose calculus technique is weak Clearly the described approach is suitable mostly in the uniersity mathematical courses since the students need to hae a reasonable mathematical background Case studies at the same uniersities show that the students understand the topic better and learn it faster with the new approach The approach described also demonstrates links between different areas of mathematics and helps the students to see practical applications of the abstract concepts of ector space and orthogonal projection References Benninga S () Financial modelling Massachusetts Institute of Technology Chatterjee S () Regression analysis by eample (rd ed) New York: Wiley Cheang GH & Zhao Y () Calculus and matri algebra for finance (nd ed) McGraw-Hill Chiang CL () Statistical methods of analysis Rier Edge NJ: World Scientific Dowdy SM () Statistics for research (rd ed) Hoboken NJ: Wiley-Interscience Francis JC & Taylor RW () Inestments (nd ed) New York: McGraw-Hill Freund RJ (998) Regression analysis: statistical modelling of a response ariable San Diego: Academic Press Grimmett G & Stirzaker D () Probability and random processes (rd ed) New York: Oford Uniersity Press Hsu HP (997) Probability random ariables and random processes New York: McGraw-Hill Kachapoa F & Kachapo I (6) Mathematical approach to portfolio analysis Auckland: Maths Ken Kachapoa F & Kachapo I (5) Calculating optimal portfolios of three financial assets Mathematics and Informatics Quarterly No / Vol 5 pp -7 7

Journal of Statistics Education Volume 8 Number () Moore DS & McCabe GP (6) Introduction to the practice of statistics (5th ed) New York: WHFreeman and Company Saille DJ & Wood GR (996) Statistical methods A geometric primer New York: Springer-Verlag Scheffé H (959) The analysis of ariance John Wiley & Sons Seber GAF (98) The linear hypothesis: a general theory (th ed) Charles Griffin & Company Ltd Seber GAF & Lee AJ () Linear regression analysis (th ed) John Wiley & Sons Teall JL & Hasan I () Quantitatie methods for finance and inestment Oford: Blackwell Wild CJ & Seber GAF () Chance encounters John Wiley & Sons Dr Farida Kachapoa Auckland Uniersity of Technology School of Computing and Mathematical Sciences Priate Bag 96 Auckland New Zealand faridakachapoa@autacnz Ph +6 9 99999 et 88 Dr Ilias Kachapo Fairlands Ae Wateriew Auckland 6 New Zealand kachapo@traconz Ph +6 9 887 Volume 8 () Archie Inde Data Archie Resources Editorial Board Guidelines for Authors Guidelines for Data Contributors Home Page Contact JSE ASA Publications 8