CSE 190, Great ideas in algorithms: Matrix multiplication

Transcription

1 CSE 190, Great ideas in algorithms: Matrix multiplication 1 Matrix multiplication Given two n n matrices A, B, compute their product C = AB using as few additions and multiplications as possible. That is, we want to compute c i,j = k a i,kb k,j. For concreteness, consider computations over the reals, although any other field would do. Definition 1.1. The matrix multiplication exponent is the minimal ω such that n n matrices can be multiplied using O(n ω ) operations. Open Problem 1.2. What is the matrix multiplication exponent? Trivially, 2 ω 3. The first nontrivial algorithm was by Strassen, who showed that ω log The starting point of Strassen s algorithm is the following algorithm for multiplying 2 2 matrices: 1. m 1 = (a 1,1 + a 2,2 )(b 1,1 + b 2,2 ) 2. m 2 = (a 2,1 + a 2,2 )b 1,1 3. m 3 = a 1,1 (b 1,2 b 2,2 ) 4. m 4 = a 2,2 (b 2,1 b 1,1 ) 5. m 5 = (a 1,1 + a 1,2 )b 2,2 6. m 6 = (a 2,1 a 1,1 )(b 1,1 + b 1,2 ) 7. m 7 = (a 1,2 a 2,2 )(b 2,1 + b 2,2 ) 8. c 1,1 = m 1 + m 4 m 5 + m 7 9. c 1,2 = m 3 + m c 2,1 = m 2 + m c 2,2 = m 1 m 2 + m 3 + m 6 1

2 The computation model is a straight line program: each internal computation is a sum or a product of two previous computed values. We can expand Strassen s algorithm in this way: it will contain 7 multiplications and 18 additions. Moreover, we can first compute the linear forms, then multiply them, and then take a linear combination of the results. We first show that any straight line program for matrix multiplication can be put in such a form, which we call a normal form. Lemma 1.3 (normal form). Any straight-line program for computing matrix multiplication, which uses M multiplications (and any number of additions), can be converted to the following form: (i) For 1 i 2M, compute linear combinations α i of the entries of A. (ii) For 1 i 2M, compute linear combinations β i of the entries of B. (iii) For 1 i 2M, Compute p i = α i β i. (iv) For 1 i, j n, compute c i,j as a linear combination of p 1,..., p 2M. Note that in the normal form, the program computes 2M multiplications and O(Mn 2 ) additions. Proof. Let z 1,..., z N be the intermediate variables of the straight line program, where for simplicity assume that the firsn 2 are the inputs, ie the entries of A, B. Each z t (other than the firsn 2 ) is a sum or a product of two previous variables. Each z t is some polynomial in the inputs. As the required result is quadratic, we will show that only linear and quadratic computations are necessary. For simplicity, let x, y be a vectors of length n 2 each, containing the elements of A and B, respectively. Then z t (x, y) is some polynomial. We can decompose it as z t (x, y) = c t + l t(x) + l t (y) + q t (x, y) + r t (x, y), where c t is a constant, l t, l t are linear functions, q t (x, y) is a bilinear function, that is q t (x, y) = γ i,j x i y j for some coefficients γ i,j, and r t (x, y) are the remaining terms. Note that inputs have only a linear part; and that outputs have only a bilinear part. The main point is that we can compute all of the linear and bilinear parts directly, without computing r t at all. via a straight line program. If z t = z t1 + z t2 with, < t, then c t = c t1 + c t2, l t(x) = l (x) + l (x), l t (y) = l (y) + l (y) and q t (x, y) = q t1 (x, y) + q t2 (x, y). Same for general linear combinations. If z t = z t1 z t2 with, < t then c t = c t1 c t2, l t(x) = c t2 l (x), l t (y) = c t1 l t2 (y) and q t (x, y) = c t,1 q t2 (x, y) + c t,2 q t1 (x, y) + l (y) + l (y). Note that c t are constants independent of the inputs. So, the only actual multiplications we do is in computing l (y) and l (y). Instead, we can compute: l t(x), l t (y) for all 1 t N. 2

3 If z t is a multiplication gate, compute l (y) and l (y). Answers are linear combinations of q t (x, y). Note that we only need the linear combinations which enter the multiplication gates, which gives the lemma. Theorem 1.4. If two m m matrices can be computed using M = m α multiplications (and any number of additions) in a normal form, then for any n 1, any two n n matrices can be multiplied using only O((mn) α log(mn)) operations. So for example, Strassen s algorithm is an algorithm in normal form which uses 7 multiplications to multiply two 2 2 matrices. So, any two n n matrices can be multiplied using O(n log 2 7 ) O(n 2.81 ) operations. So, ω log The best known algorithms give ω Proof. Let T (n) denote the number of operations required to compute the product of two n n matrices. We assume that n is a power of m, by possible increasing it to the smallest power of m larger than it. This might increase n to at most nm. Now, the main idea is to compute it recursively. We partition an n n matrix as an m m matrix, whose entries are (n/m) (n/m) matrices. Let C = AB and let A i,j, B i,j, C i,j be these sub-matrices of A, B, C, respectively, where 1 i, j m. Then, observe that (as matrices) we have C i,j = m A i,k B k,j. k=1 We can apply any algorithm for m m matrix multiplication in normal form to compute {C i,j }, as the algorithm never assumes that the inputs commute. So, to compute {C i,j }, we: (i) For 1 i M, compute linear combinations α i of the A i,j. (ii) For 1 i M, compute linear combinations β i of the B i,j. (iii) For 1 i M, Compute p i = α i β i. (iv) For 1 i, j m, compute C i,j as a linear combination of P 1,..., P M. Note that α i, β i, p i are all (n/m) (n/m) matrices. How many operations do we do? steps (i),(ii),(iv) each require Mm 2 additions of (n/m) (n/m) matrices, so in total require O(Mn 2 ) additions. Step (iii) requires M multiplications of matrices of size (n/m) (n/m). So, we get the recursion formula T (n) = m α T (n/m) + O(m α n 2 ). This solves to O((mn) α ) if α > 2 and to O((mn) 2 log n) if α = 2. Lets see explicitly the first case, the second being similar. 3

4 Let n = m s. This recursion solves to a tree of depth s, where each node has m α children. The number of nodes at depth i is m αi, and the amount of computation that each makes is O(m α (n/m i ) 2 ). Hence, the total amount of computation at depth i is O(m α m (α 2)i n 2 ). As long as α > 2, this grows exponentially fast in the depth, and hence controlled by the last level (at depth s) which takes O(m α m (α 2)s m 2s ) = O((mn) α ). 1.1 Verifying matrix multiplication Assume that someone gives you a magical algorithm that is supposed to multiply two matrices quickly. How would you verify it? one way is to compute matrix multiplication yourself, and compare the results. This will take time O(n ω ). Can you do better? the answer is yes, if we allow for randomization. In the following, our goal is to verify that AB = C where A, B, C are n n matrices over an arbitrary field. Function MatrixMultVerify Input : n n matrices A, B, C. Output : I s i s true that AB = C? 1. Choose x {0, 1} n randomly. 2. Return TRUE i f A(Bx) = Cx, and FALSE otherwise. Clearly, if AB = C then the algorithm always returns true. Moreover, as all the algorithm does is iteratively multiply an n n matrix with a vector, it runs in time O(n 2 ). The main question is: can we find matrices A, B, C where AB C, but where the algorithm returns TRUE with high probability? The answer is no, and is provided by the following lemma, applied to M = AB C. Lemma 1.5. Let M be a nonzero n n matrix. Then Pr x {0,1} n[mx = 0] 1/2. In particular, if we repeat this t times, the error probability will reduce to 2 t. Proof. The matrix M has some nonzero row, lets say it is a 1,..., a n. Then, [ ] Pr x {0,1} n[mx = 0] Pr ai x i = 0. Let i be minimal such that a i 0. Then, a i x i = 0 iff x i = j>i ( a j/a i )x j. Hence, for any fixing of {x j : j > i}, there is at most one value for x i which would make this hold. [ ] [ [ Pr ai x i = 0 = E xj,...,x n {0,1} Pr ai x i = 0] ] x i {0,1} [ [ = E xj,...,x n {0,1} Pr x i = ]] ( a j /a i )x j x i {0,1} j>i 1/2. 4

5 1.2 Finding triangles in graphs Let G = (V, E) be a graph. Our goal is to find whether G contains a triangle, and more generally, enumerate the triangles in G. Trivially, this takes n 3 time. We will show how to improve it using fast matrix multiplication. Let V = n, A be the n n adjacency matrix of G, A i,j = 1 (i,j) E. Observe that (A 2 ) i,j = k A i,k A k,j = number of pathes of length two between i, j So, to check G contains a triangle, we can first compute A 2, and then use it to detect if there is a triangle. Function TriangleExists(A) Input : An n n adjacency matrix A Output : I s t h e r e a t r i a n g l e in the graph? 1. Compute A 2 2. Check i f t h e r e i s 1 i, j n with A i,j = 1 and (A 2 ) i,j 1. The running time of step 1 is O(n ω ), and of step 2 is O(n 2 ). The enumeration algorithm will be recursive. At each step, we partition the vertices to two sets and recurse over the possible 8 configurations. To this end, we will need to check if a triangle i, j, k exists in G with i I, j J, k K for some I, J, K V. The same algorithm works. Function TriangleExists(A; I,J,K) Input : An n n adjacency matrix A, I, J, K {1,..., n} Output : I s t h e r e a t r i a n g l e in the graph? 1. Let A 1, A 2, A 3 be I J, J K, I K sub matrices o f A, r e s p e c t i v e l y. 2. Compute A 1 A Check i f t h e r e i s i I, k K with (A 1 A 2 ) i,k = 1 and (A 3 ) i,k = 1. We next describe the triangle listing algorithm. For simplicity, we assume n is a power of two. 5

6 Function TrianglesList(A; I,J,K) Input : An n n adjacency matrix A, I, J, K {1,..., n} Output : L i s t i n g o f a l l t r i a n g l e s in the graph 1. I f n = 1 check i f the s i n g l e p o s s i b l e t r i a n g l e e x i s t s, and i f so, output i t. 2. I f CheckTriangle ( I, J,K)==False return. 3. P a r t i t i o n I = I 1 I 2, J = J 1 J 2, K = K 1 K 2, each o f s i z e n/2. 4. Run T r i a n g l e s L i s t (I a, J b, K c ) f o r a l l 1 a, b, c 2. We will run TrianglesList(A,V,V,V) to enumerate all triangles in the graph. Lemma 1.6. If G has m triangles, then TrianglesList outputs all triangles, and runs in time O(n ω m 1 ω/3 ). In particular, if ω = 2, the algorithm runs in time O(n 2 m 1/3 ). Proof. It is clear that the algorithm lists all triangles, and every triangle is listed once. To analyze its running time, consider the tree defined by the execution of the algorithm. A node at depth d corresponds to three matrices of size n/2 d n/2 d. It either has no children (if there is no triangle in the corresponding sets of vertices), or has 8 children. Let l i denote the number of nodes at depth i, then we know that l i min(8 i, 8m). The first bound is obvious, the second follows because for any node at depth i, its parent at depth i 1 must contain a triangle, and all the triangles at a given depth are disjoint. The computation time at level i is given by T i = l i O((n/2 i ) ω ) Let i be the level at which 8 i = 8m. If i i then If i i then T i 8 i O((n/2 i ) ω ) = O(n ω 2 i(3 ω) ). T i 8m O((n/2 i ) ω ) = O(m n ω 2 iω ). So the total running time is controlled by that of level i, and hence Ti = O(T i ) = O(n ω m 1 ω/3 ). Open Problem 1.7. How fast can we find one triangle in a graph? how about m triangles? 6