Linear Algebra Methods for Data Mining

Transcription

1 Linear Algebra Methods for Data Mining Saara Hyvönen, Spring 2007 Lecture 3: QR, least squares, linear regression Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki

2 QR decomposition Any matrix A R m n, m n, can be transformed to upper triangular form by an orthogonal matrix: A = Q ( R 0 ) If the columns of A are linearly independent, then R is non-singular. Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki 1

3 How? By using the Givens rotation: Let x be a vector. The parameters c and s, c 2 + s 2 = 1, can be chosen so that multiplication of x by G = c 0 s s 0 c will zero the element 4 in vector x by a rotation in plane (2,4). How? Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki 2

4 x θ Gx Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki 3

5 Skinny QR decomposition Partition Q = (Q 1 Q 2 ), where Q 1 R m n : A = (Q 1 Q 2 ) ( R1 0 ) = Q 1 R 1. Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki 4

6 A = [Q,R]=qr(A) %the QR decomposition Q = R = Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki 5

7 [Q,R]=qr(A,0) %the skinny version of QR Q = R = Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki 6

8 Uniqueness? Suppose A R m n has full column rank (i.e. the column vectors are linearly independent). The skinny QR factorization A = Q 1 R 1 is unique where Q 1 has orthonormal columns and R 1 is upper triangular with positive diagonal entries. Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki 7

9 What is QR good for? It will come up when we discuss the computation of some matrix decompositions It can also be used for solving the least squares problem. Other applications exist. Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki 8

10 Least squares for linear regression Consider problems of the following form: given a number of measurements X = [x 1...x n ] and an outcome y, build a linear model ŷ = b 0 + n j=1 x j b j, which uses the measurements X to predict the outcome y. X = clinical measures of patients, y = level of cancer specific antigen. X = atmospheric measurements of each day, y = occurence of spontaneous particle formation. Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki 9

11 Least squares for linear regression The model ŷ = b 0 + n j=1 x jb j can be written in the form ŷ = X T b, b = (b 1... b n b 0 ) T. To do this, one must append X = [x 1... x n x n+1 ], where x n+1 is a vector of ones. Fitting a linear model to data is usually done using the method of least squares. Note: X need not be a square matrix! Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki 10

12 We have measurement data: Example x y Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki 11

13 Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki 12

14 Example (continued) x y We wish to find α and β such that αx + β = y. Thus α + β = α + β = α + β = α + β = α + β = 21.2 Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki 13

15 In matrix form: Example (continued) ( ) α β = Overdetermined! (More equations than unknowns.) Solve using the least squares method. Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki 14

16 Example X = atmospheric measurements of each day: temperature, UV-A: X = ( ) t1 t 2... t n r 1 r 2... r n = ( temperatures UV-A measurements ) y = which month each day belongs to. Choices are: August (y = 0) or September (y = 1). Looking for b such that ŷ = X T b and ŷ y. Use method of least squares. Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki 15

17 1.5 August days (b) and September days (r) UV A temperature T Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki 16

18 The least squares problem Let A R m n, m n. The system Ax = b is called overdetermined: more equations than unknown. Usually such a system has no solution. Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki 17

19 Example m = 3, n = 2. Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki 18

20 What to do? make the residual vector r = b Ax as small as possible. But how? Make r orthogonal to the columns of A: r T ( ) a 1 a 2... a n = r T A = 0. Now write r = b Ax to get the normal equations A T Ax = A T b; solve for x. Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki 19

21 Example A= 1 1 b= C=A *A %Normal equations C= x=c\(a *b) x= Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki 20

22 If the column vectors of A are linearly independent, then the normal equations A T Ax = A T b are non-singular and have a unique solution BUT: the normal equations have two significant drawbacks: forming A T A leads to loss of information. the condition number of A T A is the square of that of A: κ(a T A) = (κ(a)) 2 Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki 21

23 Example A = cond(a) = cond(a *A) = Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki 22

24 Worse example A = cond(a) = cond(a *A) = e e+07 Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki 23

25 Solving least squares problem using QR r 2 = b Ax 2 = b Q = Q T b ( ) R x 2 0 ( ) R x 2 = Q(Q T b 0 ( ) R x) 2 0 Partition Q = (Q 1 Q 2 ), where Q 1 R m n, and denote Q T b = ( b1 b 2 ) := ( ) Q T 1 b Q T. 2 b Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki 24

26 Then r 2 = ( b1 b 2 ) ( ) R x) 2 = b 1 Rx 2 + b Minimize r by making the first term equal to zero: i.e. solve Rx = b 1. Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki 25

27 LS by QR Theorem. Let A R m n have full column rank and a thin QRdecomposition A = Q 1 R. Then the least squares problem min x Ax b 2 has the unique solution x = R 1 Q T 1 b. Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki 26

28 Example A= 1 1 b= [Q1,R]=qr(A,0) %skinny QR Q1 = Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki 27

29 R = x=r\(q1 *b) x = Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki 28

30 Back to this example: X = atmospheric measurements of each day: temperature, UV-A: X = ( ) t1 t 2... t n r 1 r 2... r n = ( temperatures UV-A measurements ) y = which month each day belongs to. Choices are: August (y = 0) or September (y = 1). Looking for b such that ŷ = X T b and ŷ y. Use method of least squares. Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki 29

31 1.5 August days (b) and September days (r) UV A temperature T Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki 30

32 Example %X (transpose of) data matrix, 1st col: temperature, 2nd col: UV-A %y month vector, y(j)=0 if August and 1 if September [length(find(y==0)) length(find(y==1))] = %number of August and September days size(x) = %size of data matrix Xap=[X ones(215,1)]; [Q1,R]=qr(Xap,0); b=r\(q1 *y) %this is the same as b=inv(r)*(q1 *y) b= Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki 31

33 yhat=xap*b; %least squares solution [min(yhat) max(yhat)] = alpha=0.5; %day classified as September if x y>alpha length(find(1.0*(yhat>alpha)-y)) = 34 %misclassifications x=-1:0.01:1;db=(alpha-b(3)-b(1)*x)/b(2);%decision boundary x b=alpha %attn: in reality, choose alpha with more care! scatter(xnorm(:,1),xnorm(:,2),20,y, filled ) hold;plot(x,db, k ) Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki 32

34 1.5 August days (b) and September days (r) UV A temperature T Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki 33

35 Updating the solution of the LS problem Assume we have reduced the matrix and the right hand side (A b) Q T (A b) = ( ) R Q T 1 b 0 Q T. 2 b From this the solution of the LS problem is readily available. Assume we have not saved Q. We then get a new observation (a b), a R n, b R. Do we have to recompute the whole solution? Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki 34

36 Updating the solution of the LS problem No. Instead, write the reduction on the previous slide as ( A ) b a T b ( ) ( ) Q T 0 A b 0 1 a T b = R Q T 1 b 0 Q T 2 b. a T b And reduce this to triangular form using plane rotations. Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki 35

37 References [1] Lars Eldén: Matrix Methods in Data Mining and Pattern Recognition, SIAM [2] G. H. Golub and C. F. Van Loan. Matrix Computations. 3rd ed. Johns Hopkins Press, Baltimore, MD., [3] T. Hastie, R. Tibshirani, J. Friedman: The Elements of Statistical Learning. Data mining, Inference and Prediction, Springer Verlag, New York, Linear Algebra Methods for Data Mining, Spring 2007, University of Helsinki 36