Linear Algebra Review (with a Small Dose of Optimization) Hristo Paskov CS246

Transcription

1 Linear Algebra Review (with a Small Dose of Optimization) Hristo Paskov CS246

2 Outline Basic definitions Subspaces and Dimensionality Matrix functions: inverses and eigenvalue decompositions Convex optimization

3 Vectors and Matrices Vector R = May also write =

4 Vectors and Matrices Matrix R = Written in terms of rows or columns = = = =

5 Multiplication Vector-vector:, R R = Matrix-vector: R, R R = =

6 Multiplication Matrix-matrix: R, R R = 5

7 Multiplication Matrix-matrix: R, R R rows of, cols of = = =

8 Multiplication Properties Associative = Distributive + =+ NOT commutative Dimensions may not even be conformable

9 Useful Matrices Identity matrix R =,= Diagonal matrix R =diag,, = = 0 1 = 0 0

10 Useful Matrices Symmetric R := Orthogonal R : = = Columns/ rows are orthonormal Positive semidefinite R : 0 for all R Equivalently, there exists R =

11 Outline Basic definitions Subspaces and Dimensionality Matrix functions: inverses and eigenvalue decompositions Convex optimization

12 Norms Quantify size of a vector Given R, a norm satisfies 1. = 2. =0 = Common norms: 1. Euclidean -norm: = norm: = norm: =max

13 Linear Subspaces

14 Linear Subspaces Subspace R satisfies If, and R, then + Vectors,, span if = R

15 Linear Independence and Dimension Vectors,, are linearlyindependent if =0 =0 Every linear combination of the is unique Dim =if,, span and are linearly independent If,, span then If >then are NOT linearly independent

16 Linear Independence and Dimension

17 Matrix Subspaces Matrix R defines two subspaces Column space col = R R Row space row = R R Nullspaceof : null = R =0 null row dim null +dim row = Analog for column space

18 Matrix Rank rank gives dimensionality of row and column spaces If R has rank, can decompose into product of and matrices = rank=

19 Properties of Rank For, R 1. rank min, 2. rank =rank 3. rank min rank,rank 4. rank + rank +rank has fullrankif rank =min, If >rank rows not linearly independent Same for columns if >rank

20 Outline Basic definitions Subspaces and Dimensionality Matrix functions: inverses and eigenvalue decompositions Convex optimization

21 Matrix Inverse R is invertible iffrank = Inverse is unique and satisfies 1. = = 2. = 3. = 4. If is invertible then is invertible and =

22 Systems of Equations Given R, R wish to solve = Exists only if col Possibly infinite number of solutions If is invertible then = Notational device, do not actually invert matrices Computationally, use solving routines like Gaussian elimination

23 Systems of Equations What if col? Find that gives = closest to is projection of onto col Also known as regression Assume rank =< = = Invertible Projection matrix

24 Systems of Equations = =.5 1.5

25 Eigenvalue Decomposition Eigenvalue decomposition of symmetric R is =Σ = Σ=diag,, contains eigenvalues of is orthogonal and contains eigenvectors of If is not symmetric but diagonalizable =Σ Σis diagonal by possibly complex not necessarily orthogonal

26 Characterizations of Eigenvalues Traditional formulation = Leads to characteristic polynomial det =0 Rayleigh quotient (symmetric ) max

27 Eigenvalue Properties For R with eigenvalues 1. tr = 2. det = 3. rank =# 0 When is symmetric Eigenvalue decomposition is singular value decomposition Eigenvectors for nonzero eigenvalues give orthogonal basis for row =col

28 Simple Eigenvalue Proof Why det =0? Assume is symmetric and full rank 1. =Σ = 2. =Σ = Σ 3. If =, eigenvalue of is 0 4. Since det is product of eigenvalues, one of the terms is 0, so product is 0

29 Outline Basic definitions Subspaces and Dimensionality Matrix functions: inverses and eigenvalue decompositions Convex optimization

30 Convex Optimization Find minimum of a function subject to solution constraints Business/economics/ game theory Resource allocation Optimal planning and strategies Statistics and Machine Learning All forms of regression and classification Unsupervised learning Control theory Keeping planes in the air!

31 Convex Sets A set is convex if, and 0,1 + 1 Ex Line segment between points in also lies in Intersection of halfspaces balls Intersection of convex sets

32 Convex Functions A real-valued function is convex if dom is convex and, dom and 0, Graph of upper bounded by line segment between points on graph,,

33 Gradients Differentiable convex with dom=r Gradient at gives linear approximation = +

34 Gradients Differentiable convex with dom=r Gradient at gives linear approximation = +

35 Gradient Descent To minimize move down gradient But not too far! Optimum when =0 Given, learning rate, starting point = Do until =0 =

36 Stochastic Gradient Descent Many learning problems have extra structure = ; Computing gradient requires iterating over all points, can be too costly Instead, compute gradient at single training example

37 Stochastic Gradient Descent Given = ; starting point, learning rate, = Do until nearly optimal For =1 to in random order = ; Finds nearly optimal

38 Minimize

39 Learning Parameter