Sequential Communication Bounds for Fast Linear Algebra

Transcription

1 Sequetial Commuicatio Bouds for Fast Liear Algebra Grey Ballard James Demmel Olga Holtz Oded Schwartz Electrical Egieerig ad Computer Scieces Uiversity of Califoria at Berkeley Techical Report No. UCB/EECS arch 30, 01

2 Copyright 01, by the authors. All rights reserved. Permissio to make digital or hard copies of all or part of this work for persoal or classroom use is grated without fee provided that copies are ot made or distributed for profit or commercial advatage ad that copies bear this otice ad the full citatio o the first page. To copy otherwise, to republish, to post o servers or to redistribute to lists, requires prior specific permissio. Ackowledgemet This work is supported by icrosoft Award #0463 ad Itel Award #04894 fudig ad by matchig fudig by U.C. Discovery Award #DIG07-107; additioal support from Par Lab affiliates Natioal Istrumets, NEC, Nokia, NVIDIA, ad Samsug. Research supported by U.S. Departmet of Eergy grats uder Grat Numbers DE-SC , DE-SC , ad DE-FC0-06-ER5786, as well as Lawrece Berkeley Natioal Laboratory Cotract DE-AC0-05CH1131. Research supported by the Sofja Kovalevskaja programme of Alexader vo Humboldt Foudatio ad by the Natioal Sciece Foudatio uder agreemet DS Research supported by ERC Startig Grat Number

3 Sequetial Commuicatio Bouds for Fast Liear Algebra Grey Ballard, James Demmel, Olga Holtz, Oded Schwartz Abstract I this ote we obtai commuicatio cost lower ad upper bouds o the algorithms for LU ad QR give i Demmel, Dumitriu, ad Holtz 007. The algorithms there use fast, stable matrix multiplicatio as a subroutie ad are show to be as stable ad as computatioally efficiet as the matrix multiplicatio subroutie. We show here that they are also as commuicatio-efficiet i the sequetial, two-level memory model as the matrix multiplicatio algorithm. The aalysis for LU ad QR exteds to all the algorithms i Demmel, Dumitriu, ad Holtz 007. Further, we prove that i the case of usig Strasse-like matrix multiplicatio, these algorithms are commuicatio optimal. This work is supported by icrosoft Award #0463 ad Itel Award #04894 fudig ad by matchig fudig by U.C. Discovery Award #DIG07-107; additioal support from Par Lab affiliates Natioal Istrumets, NEC, Nokia, NVIDIA, ad Samsug. Research supported by U.S. Departmet of Eergy grats uder Grat Numbers DE-SC , DE-SC , ad DE-FC0-06-ER5786, as well as Lawrece Berkeley Natioal Laboratory Cotract DE-AC0-05CH1131. Research supported by the Sofja Kovalevskaja programme of Alexader vo Humboldt Foudatio ad by the Natioal Sciece Foudatio uder agreemet DS Research supported by ERC Startig Grat Number

4 1 Itroductio The ruig time of a umerical computatio depeds both o the umber of floatig poit operatios flops ad o the amout of data it moves, which we call commuicatio. I order to determie the commuicatio cost of a sequetial algorithm, we use a simple machie model which cosists of a fast memory of size words where computatio takes place coected to a slow memory with ubouded size. We assume the data ivolved i the computatio is too large to fit etirely i the fast memory, ad we measure the commuicatio i terms of floatig poit umbers words moved betwee the two memory levels. We will refer to the umber of words moved by a algorithm its badwidth cost. The aïve approach of multiplyig two matrices usig three ested loops performs 3 flops ad moves O 3 words of data. The 3 flops ca be re-ordered ito a blocked or recursive algorithm which moves O 3 words, a sigificat improvemet over the aïve approach. Hog ad Kug [1] proved that this commuicatio reductio is asymptotically optimal, that performig 3 flops requires movig at least Ω 3 words. Strasse [13] showed that matrix multiplicatio ca be performed with O log 7 flops, asymptotically fewer tha the classical O 3 approach, ad may further improvemets o the expoet have bee made. By reducig the umber of flops performed, it is also possible to reduce the commuicatio below the boud proved by Hog ad Kug. I [4], we prove a commuicatio lower boud for fast matrix multiplicatio algorithms: executig a matrix ω0 multiplicatio algorithm with O flops requires movig Ω words. 1 This lower boud is attaied by the atural recursive algorithm. Demmel, Dumitriu, ad Holtz [6] showed that early all of the fudametal algorithms i dese liear algebra ca be executed with asymptotically the same umber of flops as matrix multiplicatio. To obtai practical umerical algorithms, oe must also cosider stability ad commuicatio issues. Although the stability properties of fast matrix multiplicatio are slightly weaker tha those of classical matrix multiplicatio, the authors show i [7] that fast matrix multiplicatio is stable. Further, i [6] they show that fast liear algebra ca be made stable at the expese of oly a polylogarithmic i.e., polyomial i log factor icrease i cost. That is, to maitai stability, oe ca use polylogarithmically more bits to represet each floatig poit umber ad to compute each flop. While this icreases the time to perform oe flop or move oe word, it does ot chage the umber of flops computed or words moved by the algorithm. The mai cotributio of this ote is the extesio of both upper ad lower commuicatio bouds for the algorithms preseted i [6]. Stability ad computatioal complexity were the mai cocers i [6]. Here, we show that the commuicatio cost of the algorithms also matches that of the matrix multiplicatio subrouties they employ ad that the recursive implemetatios yield a commuicatio-optimal orderig of the computatio. We summarize the previous results ad those show here i Table 1. 1 Uder certai techical assumptios. See Sectio for details.

5 Algorithm Flops Words Lower Boud Classical matrix multiplicatio Classical liear algebra O 3 O 3 + Ω 3 + Fast matrix multiplicatio O + O / 1 Ω + Fast liear algebra O + log / 1 / 1 Table 1: Summary of cost comparisos betwee classical ad fast algorithms. The lower bouds give correspod to the umber of words moved. Lower Bouds I this sectio we obtai commuicatio lower bouds for the algorithms i [6] usig the results from [4]. The applicability of these bouds are similar to those for classical algorithms [5] ad for Strasse-like matrix multiplicatio algorithms [4]: they apply to ay re-orderig of the computatio which respects the depedecies ad ay use of commutativity ad associativity of additio. The lower bouds here deped directly o the choice of fast matrix multiplicatio algorithm used as a subroutie, ad they apply oly to Strasse-like algorithms which are defied below. Note that for all dese matrix algorithms there exists a trivial lower boud of Ω words sice every matrix elemet must be accessed at least oce. We iclude this term i the statemets of the lower bouds i Table 1..1 Strasse-like atrix ultiplicatio Commuicatio lower bouds for Strasse s ad Strasse-like matrix multiplicatio algorithms are proved i [4] usig aalysis of the computatio directed acyclic graph CDAG. We repeat the defiitio of Strasse-like ad the mai result from [4] here, as our results for Strasse-like liear algebra will be direct extesios. Defiitio.1. A Strasse-like matrix multiplicatio algorithm is a recursive algorithm for multiplyig square matrices which is costructed from a base case of multiplyig 0 0 matrices usig m 0 < 3 0 scalar multiplicatios, resultig i a algorithm for multiplyig matrices requirig O flops where = log 0 m 0. I order to be Strasse-like, the base case decodig graph, which gives the depedecies betwee the m 0 scalar multiplicatio results ad the 0 etries of the output matrix, must be coected. Theorem. [4]. The umber of words moved by a Strasse-like matrix multiplicatio algorithm with O flops o a machie with fast memory of size, assumig o iter- See [4] for further discussio. 3

6 mediate value is computed twice, is B = Ω ω0.. Liear Algebra with Strasse-like atrix ultiplicatio The followig lower boud applies to all the algorithms i [6] assumig the fast matrix multiplicatio subroutie is Strasse-like. I particular, alog with the results of Sectios 4 ad 5, it implies that the LU ad QR algorithms are commuicatio-optimal. Corollary.3. Suppose a algorithm has a CDAG cotaiig as a subgraph the CDAG of a Strasse-like matrix multiplicatio algorithm with iput size Θ which performs Θ flops. The, assumig that o itermediate value is computed twice, the umber of words moved durig the computatio o a machie with fast memory of size is B = Ω ω0 Proof. The proof of Theorem. is based o a aalysis of the CDAG of a Strasse-like matrix multiplicatio algorithm. If the CDAG of a computatio icludes as a subgraph a CDAG which correspods to Θ Θ Strasse-like matrix multiplicatio, the the aalysis yields the same commuicatio lower boud for that subset of the computatio ad therefore the etire computatio. Note that there may be may differet CDAGs which correspod to computig a LU decompositio usig a Strasse-like matrix multiplicatio as a subroutie. For example, the algorithms of [6] split the matrix ito equal-sized left ad right halves, but aother algorithm may split the matrix ito a tall-skiy pael ad a larger trailig matrix. Corollary.3 applies to all such algorithms that cotai a sufficietly large subgraph correspodig to a Strasse-like matrix multiplicatio. This result implies that give the CDAG that a recursive algorithm of [6] produces, o reorderig of the computatio ca improve the commuicatio costs by more tha a costat factor compared to the depth-first orderig give by the recursive algorithm. The result does ot apply to algorithms which restructure the CDAG beyod the freedom allowed by commutativity ad associativity of additio. Ulike the case of classical matrix multiplicatio ad other O 3 algorithms, the freedom to exploit commutativity ad associativity withi fast matrix multiplicatio ad fast liear algebra is quite limited ad ca affect commuicatio costs oly by a costat factor. I the case of classical algorithms, exploitig commutativity ad associativity is a importat meas of blockig algorithms to improve commuicatio costs ad ca alter the badwidth cost by up to a factor of O. I the case of fast algorithms, the freedom to order the computatio allows much greater flexibility tha just exploitig commutativity ad associativity: a pessimal orderig results i O badwidth cost while the optimal orderig decreases that cost by a factor of O / 1. I this sese, the applicability of the lower bouds for fast liear algebra are as geeral as those for fast matrix multiplicatio. 4.

7 3 Rectagular Recursive O 3 Algorithms I this sectio we review the commuicatio cost upper bouds for rectagular recursive algorithms for Cholesky, LU, ad QR which use classical matrix multiplicatio as a subroutie. We provide the detailed aalysis here because it does ot appear elsewhere i the literature i full geerality, ad because the aalysis for fast liear algebra i.e., usig fast matrix multiplicatio as a subroutie is very similar. We will assume the use of classical matrix multiplicatio ad triagular solve with multiple right had sides subrouties which attai their commuicatio lower bouds. For upper boud aalysis of recursive versios of these algorithms, see [3]. The badwidth cost recurrece for rectagular recursive algorithms for Cholesky [3], LU [11, 14], ad QR [8, 9] o m matrices where m is give by Bm, = B m, + B m, + Θ m + m if > 1 ad m > Θm if = 1 or m. The base case occurs either whe the recursio stops i order to factor oe colum = 1 or the problem fits etirely ito fast memory m <. The fact that either base case ca occur is differet tha i may other commuicatio cost recurreces ad is the mai reaso the aalysis is more complicated. I either base case, the badwidth cost is oly a costat factor more tha the cost of readig the matrix oce. A equivalet recurrece appears i [8] see equatio 3.5 for the case of QR, but the cost of the update of the right half of the matrix is writte as O m 1 i the otatio here, which igores the m term. The m term domiates m whe < ; ote that this case may occur simultaeously with m > for sufficietly large m. I this case, the problem is too large to fit ito fast memory but the umber of colums is too small to attai Θ re-use withi matrix multiplicatio. Thus, the m term caot be igored. A simplified recurrece for LU appears i [14], where the base case m < is ot cosidered. This implies the recurrece provided i [14] is accurate oly i certai cases, but the recurrece provides a valid upper boud i all cases oly upper bouds are cosidered i [14]. Tighter aalysis, icludig cosideratio of lower bouds, is provided i [3] for Cholesky, ad the two cases m ad m > are treated separately. However, i the case m >, the error i the statemet of the recurrece is propagated from [14]. The recurrece assumes that if m >, the o subproblem will fit i fast memory throughout the recursio; however, the umber of rows i oe of the subproblems is reduced, ad therefore both base cases are possible. The recurrece is valid oly if m >. That is, the smallest umber of rows amog all subproblems is still greater tha the size of the fast memory. Here we show both upper ad lower bouds 3 for equatio 1, thus showig that each term is iheret to the algorithm ad ot due to lax aalysis. Sice the cost of factorig 3 Note that this is a lower boud for the recurrece correspodig to these particular rectagular recursive algorithms, ot for a class of algorithms for computig these decompositios. Lower bouds for a class of algorithms is the subject of [4, 5] ad Sectio. 5

8 the left half of the matrix is greater tha the cost of factorig the right half, we ca upper boud the right had side of equatio 1 as doe i [8] with B m, + O m Bm, + m if > 1 ad m > Om if = 1 or m. Thus, Bm, O t m t 1 + t i=0 m = O + mt i O m i + m i where t is the height of the recursio tree. I the case m >, the base case m < ever occurs because the umber of rows i ay subproblem is always too large to fit oe colum i fast memory. Thus, the oly relevat base case is = 1, ad so t = log. I the case m <, the base case = 1 ever occurs because each subproblem will fit i fast memory before the umber of colums is reduced to 1. Thus, the oly relevat base case is m <, ad so t = log m. Similarly, we ca lower boud the right had side of equatio 1 with B m Bm,, + Ω m + m if > 1 ad m > Ωm if = 1 or m. Thus, sice the umber of rows i ay subproblem is at least m, we have Bm, Ω t m t 1 m i + i j=1 Ω j i i + m t i=0 j i j=1 t 1 m Ωm + i Ω i + m i = Ω i=0 m + m t where t = log i the case m >, or t = log m i the case m <. Thus, for m c for some costat c > 1 ad m >, we have m Bm, = Θ + m log, ad for m c for some costat c > 1 ad m, we have m Bm, = Θ + m log m. 6

9 Note the m log ad m log m terms: they arise because of the extra m term i the badwidth cost of matrix multiplicatio ad triagular solve, possibly at each iteral ode of the recursio tree. This is why the solutios give here differ from [8]. Also, as argued i [14], pivotig i the case of LU adds aother O m log term to the total badwidth cost. While the coditio m c with c > 1 does ot hold for square matrices, we may focus attetio o factorig the left half of the matrix i which case the umber of rows is twice the umber of colums ad obtai both upper ad lower bouds. Thus, for >, we have B = Θ 3 + log, ad for <, we have B = Θ 3 + log. For each of these recurreces, the base cases together cost oly Θ words the size of the iput/output. Sice the cost of the iteral odes domiates, usig a faster matrix multiplicatio subroutie that also commuicates less improves the total commuicatio costs of these algorithms. However, usig a faster matrix multiplicatio caot reduce the extra log or log term that accumulates from the Θ badwidth cost required for each recursive level of iteral odes. 4 LU with Fast atrix ultiplicatio The LU decompositio algorithm of [6] icludes computig ad multiplyig by explicit iverses of diagoal blocks of L which are triagular. Thus, we first establish the badwidth costs of fast matrix multiplicatio ad triagular matrix iversio usig fast matrix multiplicatio. Fast atrix ultiplicatio The commuicatio cost of fast matrix multiplicatio implemeted with the atural recursive algorithm is give by B = Θ + / 1 see equatio 1 i [4] where is the expoet correspodig to the computatioal cost. Triagular atrix Iversio with Fast atrix ultiplicatio. Followig Sectio 3.1 of [6], the recursive algorithm for triagular matrix iversio 4 ivolves two recursive calls ad two matrix multiplicatios which igore triagular sparsity. Agai, the cost recurrece give for arithmetic also applies to the badwidth cost. The badwidth cost is B T RT RI + Θ / + / if > B T RT RI = / 1 Θ if. The solutio to this recurrece is B T RT RI = Θ + / 1. 4 We use the LAPACK acroym TRTRI to refer to triagular matrix iversio. 7

10 Note that this algorithm is commuicatio optimal. I order to re-use the cost fuctio for LU decompositio give i Sectio 4. of [6], we compute the badwidth cost of LU without pivotig first ad the add i the costs of pivotig. With a slight chage of otatio from [6] iterchagig m ad, we obtai a similar recursive boud o the badwidth cost: B LU m, B LU m, + 4 m B + B T RI + O m. Note that the base cases of the recurrece are differet tha i [6] ad match those i Sectio 3. Sice the badwidth cost of triagular iversio matches that of matrix multiplicatio, we obtai B LU m, = B LU m, + O m/ 1 + m/ / 1 if > 1 ad m > Θm if = 1 or m. Sice the pivotig is doe the same way as i the classical O 3 recursive algorithm, the badwidth cost due to pivotig is Om log as show i [14]. Followig the same aalysis as i Sectio 3 ad icludig the cost of pivotig, we have that the badwidth cost of LU decompositio with partial pivotig is m 1 B LU m, = Θ + / 1 m log. Note that sice m, m log domiates m log ad m log m Thus, for a square matrix, the badwidth cost is B LU = Θ + / 1 log ad by Corollary.3 the algorithm is commuicatio-optimal for = O 5 QR with Fast atrix ultiplicatio for m <. Sectio 4.1 of [6] presets a recursive QR decompositio algorithm based o that of Elmroth ad Gustavso [9] which exploits fast matrix multiplicatio ad has the same asymptotic computatioal cost as the matrix multiplicatio subroutie. The recursive cost fuctio, give i [6] to represet arithmetic cost, also represets the badwidth cost. Boudig it i the same way agai with a slight chage i otatio ad differet base cases, we obtai B QR m, B QR m, + 8 m B + O m. log Thus, we have B QR m, + O m/ 1 + m/ if > 1 ad m > B QR m, = / 1 Θm if = 1 or m. 8.

11 Followig the aalysis from Sectio 3, i the case m >, the depth of the recursio tree is log, ad therefore m 1 B QR m, = Θ + m log. / 1 For smaller values of m, whe oe or more colums of the matrix fit i fast memory, the depth of the recursio tree is log m, ad the badwidth cost becomes m 1 m B QR m, = Θ + m log. / 1 Note that the cost is domiated by the iteral odes of the recursio tree the leaves cotribute Θm words total. Also ote that log m < log whe m <, which implies that equatio is a valid upper boud i all cases. Thus, for a square matrix, the badwidth cost is B QR = Θ + / 1 log ad by Corollary.3 the algorithm is commuicatio-optimal for = O. 6 Discussio ad Extesios I this ote we provide the commuicatio cost aalysis for recursive LU ad QR algorithms which use fast matrix multiplicatio subrouties as preseted i [6], ad we show that these algorithms are commuicatio optimal. Several more algorithms are give i [6], icludig radomized rak-revealig URV decompositio, eigevalue ad sigular value decompositios, solvig the Sylvester equatio, ad computig eigevectors from Schur form. Corollary.3 applies to each of these algorithms, ad the commuicatio costs of these algorithms attai the lower bouds up to additive log terms, though we omit the proofs here. There are several extesios to this work. First, all of the results here are for the sequetial, two-level memory model. The lower boud here ca be easily applied to every pair of adjacet levels i the sequetial, hierarchical memory model discussed i [5]; the recursive structure of these algorithms leads to cache-obliviousess [10] ad optimality i this hierarchical model. The lower boud ca also be exteded to the parallel, distributed-memory model as is doe i [4] ad [1]. However, it is uclear if there exist commuicatio optimal parallel algorithms for LU ad QR i the parallel case. A optimal algorithm for Strasse s matrix multiplicatio was obtaied oly recetly []. It seems likely that the parallelizatio approach employed there ca be used for other liear algebra algorithms. Aother useful commuicatio metric is latecy cost. The latecy cost of a algorithm is the umber of messages commuicated betwee fast ad slow memory, where a message 9 log

12 cosists of may words stored cotiguously i slow memory which are commuicated as a uit. The badwidth cost lower boud easily traslates to a latecy cost lower boud. I order to cosider the latecy cost of a algorithm, we must defie the data layout used for the iput ad output matrices. As argued i [3], rectagular recursive algorithms which miimize badwidth cost do ot ecessarily also miimize latecy cost. Refereces [1] G. Ballard, J. Demmel, O. Holtz, B. Lipshitz, ad O. Schwartz. Brief aoucemet: Strog scalig of matrix multiplicatio algorithms ad memory-idepedet commuicatio lower bouds, ar. 01. Submitted to SPAA, available as EECS Tech. Report No. UCB/EECS [] G. Ballard, J. Demmel, O. Holtz, B. Lipshitz, ad O. Schwartz. Commuicatio-optimal parallel algorithm for Strasse s matrix multiplicatio, ar. 01. Submitted to SPAA, available as EECS Tech. Report No. UCB/EECS [3] G. Ballard, J. Demmel, O. Holtz, ad O. Schwartz. Commuicatio-optimal parallel ad sequetial Cholesky decompositio. SIA Joural o Scietific Computig, 36: , 010. [4] G. Ballard, J. Demmel, O. Holtz, ad O. Schwartz. Graph expasio ad commuicatio costs of fast matrix multiplicatio: regular submissio. I Proceedigs of the 3rd AC Symposium o Parallelism i Algorithms ad Architectures, SPAA 11, pages 1 1, New York, NY, USA, 011. AC. [5] G. Ballard, J. Demmel, O. Holtz, ad O. Schwartz. iimizig commuicatio i umerical liear algebra. SIA Joural o atrix Aalysis ad Applicatios, 33: , 011. [6] J. Demmel, I. Dumitriu, ad O. Holtz. Fast liear algebra is stable. Numerische athematik, 1081:59 91, Oct [7] J. Demmel, I. Dumitriu, O. Holtz, ad R. Kleiberg. Fast matrix multiplicatio is stable. Numerische athematik, 106:199 4, ar [8] J. Demmel, L. Grigori,. Hoemme, ad J. Lagou. Commuicatio-optimal parallel ad sequetial QR ad LU factorizatios. SIA Joural o Scietific Computig, 341:A06 A39, 01. [9] E. Elmroth ad F. Gustavso. New serial ad parallel recursive QR factorizatio algorithms for SP systems. I B. K. et al., editor, Applied Parallel Computig. Large Scale Scietific ad Idustrial Problems., volume 1541 of Lecture Notes i Computer Sciece, pages Spriger,

13 [10]. Frigo, C. Leiserso, H. Prokop, ad S. Ramachadra. Cache-oblivious algorithms. I FOCS 99: Proceedigs of the 40th Aual Symposium o Foudatios of Computer Sciece, page 85, Washigto, DC, USA, IEEE Computer Society. [11] F. Gustavso. Recursio leads to automatic variable blockig for dese liear-algebra algorithms. IB J. Res. Dev., 416: , [1] J. W. Hog ad H. T. Kug. I/O complexity: The red-blue pebble game. I STOC 81: Proceedigs of the Thirteeth Aual AC Symposium o Theory of Computig, pages , New York, NY, USA, AC. [13] V. Strasse. Gaussia elimiatio is ot optimal. Numerische athematik, 13: , /BF [14] S. Toledo. Locality of referece i LU decompositio with partial pivotig. SIA Joural o atrix Aalysis ad Applicatios, 184: ,