ECS289: Scalable Machine Learning

Transcription

1 ECS289: Scalable Machine Learning Big-O Notations Cho-Jui Hsieh UC Davis Oct 20, 2015

2 Outline Time complexity and Big-O notations Time complexity for basic linear algebra operators

3 Time Complexity From Wikipedia: The time complexity of an algorithm quantifies the amount of time taken by an algorithm to run as a function of the length of the input The time complexity of an algorithm is commonly expressed using big O notation, which excludes coefficients and lower order terms.

4 Time Complexity From Wikipedia: The time complexity of an algorithm quantifies the amount of time taken by an algorithm to run as a function of the length of the input The time complexity of an algorithm is commonly expressed using big O notation, which excludes coefficients and lower order terms. Although time complexity is a good indication of efficiency, in practical numerical computation sometimes constants are important: For example, time for running 1 billion operations: exp : secs * : 1.84 secs / : 7.31 secs + : 1.77 secs In this course we will ignore these constants

5 Big-O Definition of O( ): Let f and g be two functions, we write f (x) = O(g(x)) as x if and only if there exists a positive constant M and x 0 such that f (x) M g(x) for all x x 0 In short, f (x) = O(g(x)) means f is upper bounded by g up to a constant factor

6 Big-O Definition of O( ): Let f and g be two functions, we write f (x) = O(g(x)) as x if and only if there exists a positive constant M and x 0 such that f (x) M g(x) for all x x 0 In short, f (x) = O(g(x)) means f is upper bounded by g up to a constant factor How to show the time complexity O(g(x))? Show there exists a way to implement the algorithm and the implementation requires Cg(x) operations for some C

7 Big-Omega Definition of Ω( ): Let f and g be two functions, we write f (x) = Ω(g(x)) as x if and only if there exists a positive constant m and x 0 such that f (x) m g(x) for all x x 0 In short, f (x) = Ω(g(x)) means f is lower bounded by g up to a constant factor

8 Big-Omega Definition of Ω( ): Let f and g be two functions, we write f (x) = Ω(g(x)) as x if and only if there exists a positive constant m and x 0 such that f (x) m g(x) for all x x 0 In short, f (x) = Ω(g(x)) means f is lower bounded by g up to a constant factor How to show the time complexity Ω(g(x))? Prove any implementation requires at least Cg(x) operations for some constant C.

9 Big-Theta Definition of Θ( ): Let f and g be two functions, we write f (x) = Θ(g(x)) as x if and only if there exists positive constant m, M and x 0 such that M g(x) f (x) m g(x) for all x x 0 In short, f (x) = Θ(g(x)) means f has the same order with g up to a constant factor

10 Big-Theta Definition of Θ( ): Let f and g be two functions, we write f (x) = Θ(g(x)) as x if and only if there exists positive constant m, M and x 0 such that M g(x) f (x) m g(x) for all x x 0 In short, f (x) = Θ(g(x)) means f has the same order with g up to a constant factor How to show the time complexity Θ(g(x))? Show both Big-O and Big-Omega

11 Count number of operations Count the total number of operations (+,,, /, exp, log, if,... ) Only need to count the order of operations, and then use the big-o notation

12 Dense vector and sparse vector If x, y R m are dense: x + y, x y, x T y: O(m) operations If x, y R m, x is dense and y is sparse: x + y, x y, x T y: O(nnz(y)) operations If x, y R m and both of them are sparse: x + y, x y, x T y: O(nnz(y) + nnz(x)) operations

13 Dense Matrix vs Sparse Matrix Any matrix X R m n can be dense or sparse Dense Matrix: most entries in X are nonzero (mn space) Sparse Matrix: only few entries in X are nonzero (O(nnz) space)

14 Dense Matrix Operations Let A R m n, B R m n, s R A + B, sa, A T : O(mn) operations Let A R m n, b R n 1 Ab: O(mn) operations

15 Dense Matrix Operations Matrix-matrix multiplication: let A R m k, B R k n, what is the time complexity of computing AB?

16 Dense Matrix Operations Assume A, B R n n, what is the time complexity of computing AB? Naive implementation: O(n 3 ) Theoretical best: O(n 2.xxx ) (but slower than naive implementation in practice) Best way: using BLAS (Basic Linear Algebra Subprograms)

17 Dense Matrix Operations BLAS matrix product: O(mnk) for computing AB where A R m k, B R k n Compute matrix product block by block to minimize cache miss rate Can be called from C, Fortran; can be used in MATLAB, R, Python,...

18 Sparse Matrix Operations Widely-used format: Compressed Sparse Column (CSC), Compressed Sparse Row (CSR),... CSR: three arrays for storing an m n matrix with nnz nonzeroes 1 val (nnz real numbers): the values of each nonzero elements 2 row ind (nnz integers): the column indices corresponding to the values 3 col ptr (m + 1 integers): the list of value indexes where each column starts

19 Sparse Matrix Operations If A R m n (sparse), B R m n (sparse or dense), s R A + B, sa, A T : O(nnz) operations If A R m n, b R n 1 Ab: O(nnz) operations If A R m k (sparse), B R k n (dense): AB: O((nnz)n) operations (use sparse BLAS) If A R m k (sparse), B R k n (sparse): AB: O(nnz(A)nnz(B)/k) in average AB: O(nnz(A)n) worst case The resulting matrix will be much denser