An Analysis of Data Distribution Methods for Gaussian Elimination in Distributed-Memory Multicomputers

Transcription

1 A Aalysis of Data Distributio Methods for Gaussia Elimiatio i Distributed-Memory Multicomuters Be Lee Orego tate Uiversity Deartmet of Electrical ad Comuter Egieerig Abstract I multicomuters, a aroriate data distributio is crucial for reducig commuicatio overhead ad therefore the overall erformace. For this reaso, data arallel laguages rovide rogrammers with rimitives, such as BLOCK ad CYCLIC that ca be used to distribute data across the distributed memory. However, the laguages do ot aid the rogrammer as to how the distributio should be erformed to maximize the erformace. Therefore, this aer resets a aalysis of data distributio methods for overlaig comutatio ad commuicatio i the Gaussia elimiatio algorithm. The aalysis idicates that both BLOCK ad CYCLIC distributios have their ow merit; however, BLOCK_CYCLIC with its hybrid characteristic cosistetly out erforms its couterarts. 1 Itroductio Multicomuters with distributed memory have become revalet i recet years for rovidig high levels of erformace i scietific alicatios. Moreover, arallel comutig o these machies usig data-arallel or PMD rogrammig model has made it easier to exress arallelism available withi a rogram ideedet of the architectural characteristics of a give multicomuter [1, 3, 9]. However, oe of the difficulties i writig data arallel code is the maig of virtual rocessors oto hysical rocessors. This is due to the fact that the maig requiremet is greatly iflueced by the data commuicatio atter of the rogram. To deal with this roblem effectively, data arallel laguages cotai rimitives that allow the rogrammer to secify how virtual rocessors should be maed oto hysical rocessors [1, 7]. However, the roblem is that the laguages do ot rovide ay aid to the rogrammer as to how the maig should be erformed. I light of these difficulties, this aer resets the roblem of maig arallel imlemetatio of the Gaussia elimiatio algorithm oto hyercube-based multicomuters, ad comares the erformaces of differet data distributio methods BLOCK, CYCLIC, ad BLOCK_CYCLIC. I articular, we are iterested i the erformaces of these data distributio methods whe comutatio ad commuicatio are overlaed usig o-blockig commuicatio rimitives. The advatage of BLOCK distributio is that the commuicatio requiremet ca be reduced by allowig comutatio art to roceed as soo as messages are disatched. O the other had, CYCLIC distributio icurs more overhead i terms of commuicatio comared to BLOCK distributio because successive iteratios that reside o differet rocessors must wait for messages to arrive. However, CYCLIC distributio offers better comutatio time due to its load balacig effect. BLOCK_CYCLIC distributio is a hybrid method that rovides a tradeoff betwee comutatio ad commuicatio. This tradeoff is drive by a umber of arameters such as the size of the matrix, the umber of rocessors, ad the characteristics of the commuicatio latecy. To better uderstad this relatioshi, a aalytical exressio is develoed to characterizes the relative erformaces of BLOCK, CYCLIC, ad BLOCK_CYCLIC distributios. The aalysis ca aid the rogrammer i determiig which distributio method will erform better uder differet circumstaces. The results of the study idicate that whe comutatio ad commuicatio are carefully balaced, BLOCK_CYCLIC method cosistetly rovides better erformace tha its couterarts. 2 Oe-to-All Broadcast I this sectio, the mai commuicatio atter used i the arallel imlemetatio of the Gaussia elimiatio algorithm, oe-to-all broadcast, is described. The commuicatio model for oe-to-all broadcast ca be develoed from a simle message trasfer betwee two rocessors based o cut-through routig, which is give as T oe to oe = + ht h + mt w, where is the startu latecy, t h is the er-ho time, h reresets the umber of hos the message travels, m is the legth of the message, ad t w = 1 BW, where BW is the chael badwidth. The startu latecy cosists of This aer aears i the Proceedigs of the 6 th IEEE ymosium o Parallel ad Distributed Processig, Dallas, Texas, 1994.

2 te 3 te 2 te Gaussia-Elimiatio(A) { for (k=0;k=-1;k++){ for (j=k+1;j=-1;j++) A[k,j]:=A[k,j]/A[k,k]; /*Divisio ste*/ for (i=k+1;i=-1;i++) for(j=k+1;j=-1;j++) A[i,j]:=A[i,j]-A[i,k] A[k,j] /*Elimiatio ste*/} } Figure 2: A serial Gaussia elimiatio algorithm. 0 1 Figure 1: Oe-to-all broadcast. the time to reare the message, the time to execute the routig algorithm, ad the time to establish a iterface betwee the local rocessor ad the router. Based o this, oe-to-all broadcast ca be erformed o a hyercube with 2 d rocessors i d stes. Figure 1 shows a examle of oe-to-all broadcast i a hyercube with d=3. ice each of the log stes requires + mt w time, the total time take to erform a oe-to-all broadcast o a -rocessor hyercube is T oe to all = ( + t w m)log. 3 Gaussia Elimiatio Gaussia Elimiatio is a imortate i solvig a system of liear equatios [7, 8, 10]. A serial versio of the Gaussia elimiatio algorithm show i Figure 2 cosists of three ested loos. For each iteratio of the outer loo, there is a divisio ste ad a elimiatio ste. As the comutatio roceeds, oly the lower-right k k sub-matrix of A becomes active. Therefore, the amout of comutatio icreases for elemets i the directio of the lower-right corer of the matrix causig a o-uiform comutatioal load. Assumig divisio, multilicatio, ad subtractio each takes a uit time [2], the total sequetial executio time is give by 1 1 T = ( k 1) + 2 ( k 1) 2 k =0 k =0 = (1) 6. Although there are may arallel imlemetatios of the Gaussia elimiatio algorithm [7], we reset two straight forward data distributio methods: BLOCK ad CYCLIC distributios. For BLOCK distributio, the coefficiet matrix A is block-stried amog rocessors such that each rocessor is assiged / cotiguous rows of the matrix. Figure 3 shows a examle of BLOCK distributio for a 8 8 matrix distributed amog 4 rocessors. For each k th iteratio, the comutatio art of the algorithm erforms k 1 divisios ad the a elimiatio ste o the ( k 1) ( k 1) sub-matrix. Therefore, the total time set o the comutatio art is (1 + 2 ) j =1( j ( )i) for i from 0 to -1. Before the elimiatio ste, the rocessor P i cotaiig the k th row must commuicate k 1 elemets to rocessors P i+1,l,p 1. This requires a oe-to-all broadcast that takes at worst case log stes i a hyercube. For BLOCK distributio, the otatio i=0 1 j =1 is 1 used i lace of k =0 throughout the aer for clarity. This is doe by relacig the idex k by j 1 + i. The overall arallel ru time of the algorithm is the give as T Block P = j i (2) t w j i log. For a sufficietly large, we have T Block P 3 + log t w ( 1) log. (3) For CYCLIC distributio, rows of matrix A are distributed amog the rocessors i a roud-robi fashio usig the fuctio mod(k ), where 0 k 1 ad is the umber of rocessors. Figure 4 shows a examle of CYCLIC distributio for a 8 8 matrix distributed amog 4 rocessors. The comutatio art of the algorithm roceeds from to P 1, ad this rocess is reeated times that results i a comutatio time of 1 i=0 ((1+ 2(( ) i)) j =1( j i) ). As i BLOCK distributio, the time required by the commuicatio art

3 1 (0,1) (0,2) (0,3) (0,4) (0,5) (0,6) (0,7) 1 (0,1) (0,2) (0,3) (0,4) (0,5) (0,6) (0,7) 0 1 (1,2) (1,3) (1,4) (1,5) (1,6) (1,7) 0 0 (4,2) (4,3) (4,4) (4,5) (4,6) (4,7) (2,3) (2,4) (2,5) (2,6) (2,7) 0 1 (1,2) (1,3) (1,4) (1,5) (1,6) (1,7) 0 0 (3,2) (3,3) (3,4) (3,5) (3,6) (3,7) 0 0 (5,2) (5,3) (5,4) (5,5) (5,6) (5,7) 0 0 (4,2) (4,3) (4,4) (4,5) (4,6) (4,7) 0 0 (5,2) (5,3) (5,4) (5,5) (5,6) (5,7) (2,3) (2,4) (2,5) (2,6) (2,7) 0 0 (6,2) (6,3) (6,4) (6,5) (6,6) (6,7) 0 0 (6,2) (6,3) (6,4) (6,5) (6,6) (6,7) 0 0 (7,2) (7,3) (7,4) (7,5) (7,6) (7,7) 0 0 (3,2) (3,3) (3,4) (3,5) (3,6) (3,7) 0 0 (7,2) (7,3) (7,4) (7,5) (7,6) (7,7) Figure 3: Row-wise BLOCK distributio. Figure 4: Row-wise CYCLIC distributio. is i=0 j =1( + t w ( j i)) log. For CYCLIC distributio, the otatio 1 j =1 is used i lace of 1 k =0 1 i=0 throughout the aer for clarity. This is doe by relacig the idex k by j 1 + i. This gives the overall arallel ru time of T Cyclic 1 1 P = ( j i) + 2 i ( j i) 1 ( ( )) log + + t w j i. Whe is sufficietly large, we have 3 ( ) log T Cyclic P log t w 1. (5) Equatios 3 ad 5 idicate that the CYCLIC maig rovides a suerior erformace due to the load balacig effect. Note that it is ossible to reduce the commuicatio art for BLOCK distributio. ice the umber of rocessors ivolved i the commuicatio decreases as the comutatio roceeds, commuicatio art ca be erformed i log( - i) stes. For examle, after 2 iteratios there are 2 rocessors ivolved i the broadcast, after 3 4 iteratios there are 4 rocessors ivolved i the broadcast ad so o. Therefore, the total time required for the commuicatio art is j =1 ( + t w ( - j ( )i)) log( - i) for i from 0 to -1. This reductio i commuicatio ca rovide a sigificat imrovemet whe is large. Thus, i effect Equatio 3 reresets the worst-case arallel ru-time. imilar imrovemet ca also be made for CYCLIC distributio. For examle, whe i = ( ) 1 the commuicatio art ca be comleted i log( - j) stes rather tha log (4) stes. However, this has a egligible effect o the overall aalysis ad therefore we ca igore it. 4 Asychroous Commuicatio I the revious sectio, the arallel ru-times of BLOCK ad CYCLIC maigs for the Gaussia elimiatio algorithm usig blockig or sychroous commuicatio rimitives were comared. The erformace of the algorithm ca be sigificatly imroved by usig oblockig or asychroous commuicatio. I a o-block commuicatio scheme, a rocessor ca roceed to execute the comutatio art as soo as a message is disatched. This allows the comutatio ad the commuicatio to be overlaed i a ielie fashio. To illustrate this effect, cosider the block-stried executio o 4 rocessors show i Figure 5. First, cosider the executio of a sigle iteratio (k=0) through the rocessors. Neglectig the iitial divisio oeratio, erforms the followig oeratios: (i) seds messages from to ad icurrig a startu delay of 2, (ii) executes the elimiatio ste, ad (iii) executes the divisio ste for the subsequet iteratio (i.e., k=1). For the o-blockig scheme, while is erformig stes (ii) ad (iii), the commuicatio of the message t w ( 1) ca occur at the same time. imilarly, the itermediate rocessor receives the aroriate message from after a delay of + t w ( 1), seds the same message to, ad erforms the elimiatio ste. Therefore, both ad receive the message ad comlete the iteratio after a delay 2 + 2t w ( 1) + 2 ( 1). Now cosider the overlaig of successive iteratios. To determie how much overlaig ca occur, we aalyze the geeral timig as to whe rocessors comlete its assiged art of the executio. I articular we are

4 δ 0 t w ( -1) t w ( -1) k = 0 + 2( 1)( 1) 2 ( 1) 2 ( 1) t w ( -1) δ ( 1) t w ( -2) t w ( -2) k = 1 2( - 2)( 2) t w ( -2) 2 ( 2) 2 ( 2) δ ( 2) t w ( -3) t w ( -3) k = 2 2( 3)( - 3) 2 ( - 3) t w ( -3) 2 ( - 3) 3 2 ( - 3) Figure 5: Overlaig of successive iteratios i BLOCK distributio. iterested i the delay δ k (or δ j 1+ ) betwee iteratios, i where δ k reresets the differece betwee the time whe the laset of rocessors (e.g., ad ) i the ielie fiish their resective oeratios for iteratio k ad whe the same set of rocessors is ready to start the ext iteratio k+1. This is doe by cosiderig the rocessor P i s fiish time T F k (P i ) for iteratio k ad the start time T k +1 (P i ) for iteratio k+1. Neglectig the iitial divisio ste, the time whe fiishes iteratio 0 is give as T F 0 ( )= 2 + 2t w ( 1)+2 ( 1). The logest ath that leads to T 1 ( ) deeds o the values of,, t w ad. Let P k i deote rocessor i for iteratio k. If 1 comletes its oeratio i sufficiet time, it will ot affect the roagatio of a message from 0 to 3 via 1. T 1 ( ) will the be dictated by the ath 0 0 1, which is give as T 1 ( )= ( 1)+ ( 1)+2 + 2t w ( 2). If 1 does ot comlete its oeratio i time, T 1 ( ) will be dictated by the ath 0 1 1, which is give as T 1 ( )= + t w ( 1)+ + 2 ( 1)+ + t ( w 2). The differece T 1 ( ) T F 0 ( )=δ 0, or simly T 1 T F 0 = δ 0, idicates the amout of delay icurred betwee iteratios 0 ad 1, which is either 2( t w ) ( 1) or t w. I geeral, the worst-case timig delay betwee ay two iteratios k ad k+1 ca be exressed as 1 T F k 1 k = log + 2 l 1 ( l 1)+ ( l 1) l =0 + ( + t w ( k 1) )log +2 ( k 1) (6a) 1 As discussed i ectio 3, after every iteratios, the umber of active rocessors icremetally decreases. Therefore, as the iteratios roceeed, the commuicatio requiremet also decreases slightly. However, for simlicity, this effect is igored i the aalysis.

5 δ k + σ k t w ( 4 ) t w ( 4 ) k = 3 t w ( 4 ) 2 ( 4) 2 ( 4) 4 2 ( 4) t w ( 5) t w ( 5) k = 4 2( -1)( 5) t w ( 5) 2 ( 5) 2 ( 5) 5 Figure 6: A examle of commuicatio that occurs every iteratios. ad T k +1 or k = log + 2 l 1 ( l 1)+ ( l 1) l =0 +( + t w ( k 2))log (6b) T F k = ( + t w ( 1) ) log 1 ad T k +1 k 1 ( )+ + 2 ( l 1) l=0 + + t w ( k 1)+2 ( k 1) = ( + t w ( 1) ) log t w ( k 2) (7a) k ( )+ + 2 ( l 1) (7b) Therefore, the amout of delay icurred betwee iteratios is give as l=0 { } δ k = max ( t w )log ( 2k + 1)( k 1), t w or δ j 1+ i = max ( t w )log 2 j + i 1 ( j i),. t w (8) Equatio 8 idicates that amout of delay betwee successive iteratios deeds o which ath is loger (give by either Equatio 6b or 7b). This is determied by evaluatig the differece betwee Equatios 6b ad 7b, which leads to k ( ) log 1 l =0 ( ) log 1 ( k + 1) t w ( ) ( 2l + 1) ( k 1) = 0. Therefore, if t w ( ) ( k + 1) ( k 1), the delay betwee successive iteratios is give by t w, otherwise, a delay of ( t w )log (2k + 1)( k 1) occurs betwee successive iteratios. I additio to the delay that exists betwee successive iteratios, a commuicatio betwee rocessors occurs every iteratios. Therefore, additioal delay σ k exists for every iteratio k, where k = 1 + i, that causes commuicatio betwee rocessors. A examle of such a situatio is deicted i Figure 6. If ( t w )( log 1) ( k + 1) ( k 1), the commuicatio betwee successive iteratios causes the logest ath to shift from roagatig through the rocessors associated with iteratio k to rocessors associated with iteratio k+1. I geeral, the timig for these two aths is give as = ( + t w ( k 1) )( log 1)+ T k +1 or +2 ( k 1)+ + t ( w k 2) (9a) T k +1 = + t w ( k 1)+ + 2 ( k 1) + ( + t w ( k 2) )( log ), (9b) where T k +1 idicates the shift i ath due to commuicatio betwee successive iteratios. Therefore, the additioal delay ca be obtaied by takig the differece betwee the two aths, which is + t w ( k log ). O the other had if ( t w )( log 1)> ( k + 1) ( k 1), the delay ca be evaluated by cosiderig the differece

6 betwee whe the two rocessors ivolved i the commuicatio (e.g., ad ) fiish their resective oeratios withi the same iteratio k. I geeral this timig is exressed as T F k (P i ) = log + 2( k 1)( k 1) + ( k 1) ad (10a) T F k (P i+1 ) = + t w ( k 1) ( k 1), (10b) which leads to ( 2 log ) + ( t w + 2k + 1) ( k 1). Therefore, the additioal delay betwee successive iteratios residig o searate rocessors is give as ( 2 log ) + ( t w + 2k + 1) ( k 1), σ k = mi + t w ( k log ) or σ 1+ i = ( 2 log ) + t w + 2 ( i + 1) 1 ( i + 1), mi + t w ( i + 1 )+ 1 log (11) Based o Equatios 8 ad 11, it is imortat to ote that regardless of which ath determies the delay betwee successive iteratios, the delay betwee successive iteratios residig o differet rocessor is δ 1+ i + σ 1+ i = 2 + t w ( ( i + 1 ) log ). Usig the above aalysis, the overall arallel ru time is give as Block T Pielied = 2 1 j i + + ( + t w ( 1) )log (12) 1 + σ 1+ i + δ j 1+ j =1 i, i=0 where + ( + t w ( 1))log terms rereset the iitial startu delay. Equatio 12 ca be simlified if the coditio ( t w )( log 1) ( k + 1) ( k 1) is satisfied by selectig a aroriate values of ad. The delay betwee successive iteratios will the be dictated by terms t w ad + t w ( ( i + 1 )+ 1 log ). Eve if this is ot the case, the coditio will quickly be satisfied after few iteratios resultig i a small error. Based o this assumtio, the simlified arallel ru-time is give by Block T Pielied = 2 1 j i + + ( + t w ( 1) )log + + t w 1 ( i + 1 )+ 1 log + ( t w ) j =1. i=0 (13) For CYCLIC distributio, we erform a similar timig aalysis by evaluatig whe fiishes ad whe ca start. A examle of CYCLIC distributio for 4 rocessors is show i Figure 7. I cotrast to BLOCK distributio, CYCLIC distributio icurs a larger delay betwee successive iteratios. This is because successive iteratios residig o differet rocessors must wait for the messages. I geeral, the timig delay betwee ay two iteratios k ad k+1 ca be exressed as T F k = ( + t w ( k 1))log +2 ( k 1) (14a) ad T k +1 = + t w ( k 1)+ + 2 ( k 1) (14b) + ( + t w ( k 2) )log. Therefore, the amout of delay icurred betwee iteratios is give as δ k = 2 + t w ( k 1 log ) or δ j 1+ i = 2 + t w ( j i log ). (15) The overall arallel ru-time is the give as Cyclic T Pielied 1 = 2 i j i + + ( + t w ( 1) )log 1 ( ) ( ( )) t w j i log. i=0 j =1 (16) Both Equatios 13 ad 16 show that, comared to usig sychroous commuicatio rimitives, overlaig comutatio ad commuicatio ca sigificatly imrove the overall erformace. BLOCK distributio has less commuicatio overhead comared to CYCLIC distributio; however, due to its load balacig effect, CYCLIC distributio has a better comutatioal erformace tha BLOCK distributio method. BLOCK_CYCLIC(α) distributio method has a hybrid characteristic ad therefore ca rovide a balace betwee the advatages offered by the two methods. I BLOCK_CYCLIC distributio, the rows are divided ito cotiguous chuks of size α ad the they are distributed i the same fashio as CYCLIC distributio. A examle of a BLOCK_CYCLIC distributio is show i Figure 8. Based o this, the overall arallel ru-time is give as

7 δ 0 k = 0 t w ( -1) t w ( -1) + 2( 1)( 1) 2 ( 1) 2 ( 1) t w ( -1) δ ( 1) t w ( -2) t w ( -2) k = 1 2( 1)( - 2) t w ( -2) 2 ( 2) 2 ( 2) δ 2 2 2( 1)( - 2) t w ( -3) t w ( -3) k = 2 2( 1)( - 3) t w ( -3) 2 ( - 3) 2( 1)( - 3) 3 2( 1)( - 3) Figure 7: Overlaig of successive iteratios i CYCLIC distributio. Block T _Cyclic Pielied = α 1 α 2 αi ( j αi) + + ( + t w ( 1) )log α 1 α + + t w ( ( j + i)α + 1 log )+ ( t w ). i=0 j =1 l =1 (17) Note that whe α =, the distributio becomes BLOCK ad therefore equivalet to Equatio 13. O the other had, whe α = 1, the distributio becomes CYCLIC ad thus becomes equivalet to Equatio Performace Predictio I the revious sectio, a exressio was develoed that characterizes the overlaig of comutatio ad commuicatio usig BLOCK_CYCLIC distributio, which is a useful tool i redictig erformaces of differet data distributio methods. Therefore, by kowig the system s etwork latecy arameters, we ca comare the exected erformace of the three data distributio methods. These studies were coducted usig arameters for CUBE-2. The CUBE-2 message-assig system has A(0,-) A(1,-) A(8,-) A(9,-) A(2,-) A(3,-) A(10,-) A(11,-) A(4,-) A(5,-) A(12,-) A(13,-) A(6,-) A(7,-) A(14,-) A(15,-) Figure 8: A examle of BLOCK_CYCLIC(2) distributio for =16 ad =4.

8 eedu =256 =512 =1024 eedu =256 =512 = log 2 α Figure 9: eedu versus α for =16. a start-u latecy of aroud 6,800 rocessor cycles ad a t w of aroud 23 rocessor cycles [6, 2]. Figures 9-10 show the erformace as fuctio of α for =16, ad 32, resectively. From these articular curves, some iterestig observatios ca be made about overlaig comutatio ad commuicatio. Note that for give the umber of rocessors, as icreases so does the seedu. The large erformace ga betwee the various is maily due to the relatively large etwork latecy. Therefore, as the etwork latecy decreases, we ca exect the erformace ga to decrease. More imortatly, excet whe is small, BLOCK_CYCLIC distributio cosistetly rovides better erformace tha BLOCK or CYCLIC distributio. imilarly, whe the etwork latecy overhead is low, the α required will also be smaller. 6 Coclusio I this aer, we aalyzed the trade-off betwee comutatio ad commuicatio i Gaussia elimiatio algorithm by overlaig successive iteratios i a ielie fashio. Both BLOCK ad CYCLIC distributios have their advatage. CYCLIC distributio has lower comutatio requiremet due to its load balacig effect; however, BLOCK distributio icurs lower commuicatio overhead sice successive iteratio ca be iitiated without havig to wait for messages to be set. BLOCK_CYCLIC distributio, which has a hybrid characteristic, offers the rogrammer the ability to merge both advatages. Our fidigs idicate that by carefully choosig α, the data ca be distributed i such a way that the erformace will be cosistetly better tha usig ay oe method. 0.0 Refereces log 2 α Figure 10: eedu versus α for =32. [1] Choudhary, A. et al., Comilig FORTRAN 77D ad 90D for MIMD Distributed-Memory Machies, Fourth ymosium o the Frotiers of Massively Parallel Comutatio, 1992, [2] Duzett, B. ad Buck, R., A Overview of the CUBE 3 uercomuter, Fourth ymosium o the Frotiers of Massively Parallel Comutatio, 1992, [3] Guta, G. ad Baerjee, P., Demostratio of automatic data artitioig techiques for arallelizig comilers o multicomuters, IEEE Trasactios o Parallel ad Distributed ystems, Vol. 3, No. 2, , March [4] Hovlad, P. D. ad Ni, L. M., A Model for Automatic Data Partitioig, Proc. of the 1993 Iteratioal Coferece o Parallel Processig. [5] Hwag, K., Advaced Comuter Architecture: Parallelism, calability, Programmability, McGraw Hill, Ic., [6] Kals, E. T., Xu, H., ad Ni, L. M., Evaluatio of Data Distributio Patters i Distributed-Memory Machies, Proc. of the 1993 Iteratioal Coferece o Parallel Processig. [7] Kumar et al., Itroductio to Parallel Comutig: Desig ad Aalysis of Algorithms, The Bejami/Cummigs Publishig Comay, Ic., Redwood City, CA, [8] Leighto, F. T., Itroductio to Parallel Algorithms ad Architectures, Morga Kaufma, [9] Li, J. ad Che, M., The data aligmet hase i comilig rograms for distributed-memory machies, Joural of Parallel ad Distributed Comutig, Vol. 13, , Oct [10] aad, Y., Commuicatio comlexity of Gaussia elimiatio algorithm o multirocessors, Liear Algebra ad Its Alicatios, Vol. 77, , 1986.