Dynamic Load Balancing of Parallel Computational Iterative Routines on Platforms with Memory Heterogeneity

Transcription

1 Dynamc Load Balancng of Parallel Comutatonal Iteratve Routnes on Platforms wth Memory Heterogenety Davd Clare, Alexey Lastovetsy, Vladmr Rychov School of Comuter Scence and Informatcs, Unversty College Dubln, Belfeld, Dubln 4, Ireland {Alexey.Lastovetsy, Abstract. Tradtonal load balancng algorthms for data-ntensve teratve routnes can successfully load balance relatvely small roblems. We demonstrate that they may fal for large roblem szes on comutatonal clusters wth memory heterogenety. Tradtonal algorthms use too smlstc models of rocessors erformance whch cannot reflect many asects of heterogenety. Ths aer resents a new dynamc load balancng algorthm based on the advanced functonal erformance model. The model conssts of seed functons of roblem sze, whch are bult adatvely from a hstory of load measurements. Exermental results demonstrate that our algorthm can successfully balance data-ntensve teratve routnes on arallel latforms wth memory heterogenety. Keywords: teratve algorthms; dedcated heterogeneous latforms; dynamc load balancng; data arttonng; functonal erformance models of heterogeneous rocessors. Introducton In ths aer we study load balancng of data-ntensve arallel teratve routnes on heterogeneous latforms. These routnes are charactersed by a hgh data-tocomutaton rato n a sngle teraton. The comutaton load of a sngle teraton can be broen nto any number of equal ndeendent comutatonal unts []. Each teraton s deendent on the revous one. The generalsed scheme of these routnes can be summarsed as follows: () data s arttoned over the rocessors, () at each teraton some ndeendent calculatons are carred out n arallel, and () some data synchronsaton taes lace. Tycally comutatonal worload s drectly roortonal to the sze of data. Examles of scentfc comutatonal routnes nclude Jacob method, mesh-based solvers, sgnal rocessng and mage rocessng. Our target archtecture s a dedcated cluster wth heterogeneous rocessors and heterogeneous dstrbuted memory. Hgh erformance of teratve routnes on ths latform can be acheved when all rocessors comlete ther wor wthn the same tme. Ths s acheved by arttonng the comutatonal worload and, hence, data

2 unevenly across all rocessors. Worload should be dstrbuted wth resect to the rocessor seed, memory herarchy and communcaton networ [2]. Load balancng of arallel alcatons on heterogeneous latforms has been wdely studed for dfferent tyes of alcatons and n varous asects of heterogenety. Many load balancng algorthms are not arorate to ether the alcatons or latforms consdered n ths aer. Alcable algorthms use models of rocessors erformance whch are too smlstc. These tradtonal algorthms are sutable for roblem szes, whch are small relatve to the latform, but can fal for larger roblems. Ths aer resents a new dynamc load balancng algorthm for data-ntensve teratve routnes on comutatonal clusters wth memory heterogenety. In contrast to the tradtonal algorthms, our algorthm s adatve and taes nto account heterogenety of rocessors and memory. Load balancng decsons are based on functonal erformance models whch are constantly mroved wth each teraton [3]. Use of the functonal erformance models remove restrctons on the roblem sze whch can be comuted. Ths allows a comutatonal scentst to utlse the maxmum avalable resources on a gven cluster. We demonstrate that our algorthm succeeds n balancng the load even n stuatons when tradtonal algorthms fal. Ths aer s structured as follows. In Secton 2, related wor s dscussed. In Secton 3, we descrbe the target class of teratve routnes and the tradtonal load balancng algorthm. Then we analyse the shortcomngs of the tradtonal algorthm and resent exermental results. In Secton 4, we descrbe our algorthm and demonstrate that t can successfully balance data-ntensve teratve routnes wth large roblem szes. 2 Related Wor In ths secton, we classfy load balancng algorthms and dscuss ther alcablty to data-ntensve teratve routnes and dedcated comutatonal clusters wth memory heterogenety. Load balancng algorthms can be ether statc or dynamc. Statc algorthms [4, 5, 6] use a ror nformaton about the arallel alcaton and latform. Ths nformaton can be gathered ether at comle-tme or run-tme. These strateges are restrcted to alcatons wth re-determned worload and cannot be aled to such teratve routnes as adatve mesh refnement [7], for whch the amount of comutaton data grows unredctably. Dynamc algorthms [8, 9,,, 2] do not requre a ror nformaton and can be used wth a wder class of arallel alcatons. In addton, dynamc algorthms can be deloyed on non-dedcated latforms. The algorthm we resent n ths aer s dynamc. Another classfcaton s based on how load balancng decsons are made: n a centralsed or non-centralsed manner. In non-centralsed algorthms [, 2], load s mgrated locally between neghbourng rocessors, whle n centralsed ones [4, 5, 6, 8, 9, ], load s dstrbuted based on global load nformaton. Non-centralzed algorthms are slower to converge. At the same tme, centralzed algorthms tycally have hgher overhead. Our algorthm belongs to the class of centralsed algorthms.

3 Centralsed algorthms can be subdvded nto two grous: tas queue and redctng the future [2]. Tas queue algorthms [9, ] dstrbute tass. They target arallel routnes consstng of ndeendent tass and schedule them on sharedmemory latforms. Predctng-the-future algorthms [4, 5, 6, 8] can dstrbute both tass and data by redctng future erformance based on ast nformaton. They are sutable for data-ntensve teratve routnes and any arallel comutatonal latform. A tradtonal aroach taen for load balancng of data-ntensve teratve routnes belongs to statc/dynamc centralsed redctng-the-future algorthms. In these tradtonal algorthms, comutaton load s evaluated ether n the frst few teratons [6] or at each teraton [8] and globally redstrbuted among the rocessors. Current load measurements are used for redcton of future erformance. Nether memory structure nor memory constrants are taen nto account. As t wll be demonstrated n Secton 3, when aled to large scentfc roblems and arallel latforms wth memory heterogenety, ths strategy may never balance the load, because t uses smlstc models of rocessors erformance. It has been shown n [3] that t s more accurate to reresent erformance as a functon of roblem sze, whch reflects contrbutons from both rocessor and memory. In ths aer, we roose a new dynamc load balancng algorthm based on artal functonal erformance models of rocessors [3]. Unle tradtonal algorthms, our algorthm moses no restrcton on roblem szes. We would also le to menton some advanced load balancng strateges whch are not drectly alcable to data-ntensve teratve routnes on heterogeneous clusters. It has been shown that the tas queue model mlemented n [] can outerform the model [9] because decsons are based on adatve seed measurements rather then sngle seed measurements. The algorthm resented n ths aer also ales an adatve erformance model, but n such a way that t s alcable to scentfc comutatonal teratve routnes. In ths aer, we focus on dynamc load balancng wth resect to rocessor erformance and memory herarchy, and to ths end we do not tae nto account communcaton heterogenety. Future wor could be the develoment of a hybrd aroach, smlar to [5], n whch our algorthm s combned wth one of the many exstng communcaton models. 3 Tradtonal Load Balancng Algorthm of Iteratve Routnes Iteratve routnes have the followng structure: x + = f( x ), =,,... wth x gven, n where each x s an n-dmensonal vector, and f s some functon from nto tself [2]. The teratve routne can be arallelzed on a cluster of rocessors by lettng x and f be arttoned nto bloc-comonents. In an teraton, each rocessor calculates ts assgned elements of x +. Therefore, each teraton s deendent on the revous one. The objectve of load balancng algorthms for teratve routnes s to dstrbute comutatons across a cluster of heterogeneous rocessors n such a way that all rocessors wll fnsh ther comutaton wthn the same tme and thereby mnmsng

4 the overall comutaton tme: t t,, j. The comutaton s sread across a j cluster of rocessors P,,P such that n. Processor P contans d elements of x and f, such that n= d. = Tradtonal load balancng algorthms wor by measurng the comutaton tme of one teraton, calculatng the new dstrbuton and redstrbutng the worload, f necessary, for the next teraton. The algorthm s as follows: Intally. The comutaton worload s dstrbuted evenly between all rocessors, d = n/. All rocessors execute n/ comutatonal unts n arallel. At each teraton. ) The comutaton executon tmes t ( d ),..., t ( d ) for ths teraton s measured on 2) If each rocessor and gathered to the root rocessor. t( d ) tj( d j ) max ε then the current dstrbuton s consdered balanced t ( d ), j and redstrbuton s not needed. 3) Otherwse, the root rocessor calculates the new dstrbuton of comutatons d,..., d as gven by s = d / t ( d ). j= d = n s / sj where s s the seed of the 'th rocessor 4) The new dstrbuton + + d,..., d necessary data s redstrbuted accordngly. s broadcast to all rocessors and where 3. Analyss of Tradtonal Load Balancng The tradtonal load balancng algorthm s based on the assumton that the absolute seed of a rocessor deends on roblem sze but the seed s reresented by a constant at each teraton. Ths s true for small roblem szes as dected n Fg. (a). The roblem s ntally dvded evenly between two rocessors for the frst teraton and then redstrbuted to the otmal dstrbuton n the second teraton. Consder the stuaton n whch the roblem can stll ft wthn the total man memory of the cluster but the roblem sze s such that the memory requrement of n/ s close to the avalable memory of one of the rocessors. In ths case agng can occur. If agng does occur, the tradtonal load balancng algorthm s no longer adequate. Ths s llustrated for two rocessors n Fg. (b, c). Let the real erformance of rocessors P and P 2 be reresented by the seed functons s ( x ) and s ( x ) resectvely. Processor P 2 s a faster rocessor but wth less man memory than P 2. The seed functon dros radly at the ont where man memory s full and agng s requred. Frst, n ndeendent unts of comutatons are evenly dstrbuted, d = d2 = n/2, between the two rocessors and the seeds of the rocessors, s, s 2, are measured. Then at the second teraton the comutatonal unts are dvded

5 d s accordng to =, where d + d2 = n. Therefore n the second teraton, P wll d2 s2 execute less comutatonal unts than P 2. However P wll erform much faster and P 2 wll erform much slower than the model redcts, Fg. (b). Moreover the seed of P 2 at the second teraton s slower then P at the frst teraton. Based on the seeds of the rocessors demonstrated at the second teraton, ther constant erformance models are changed accordngly, Fg. (c), and the 2 d s comutatonal unts are redstrbuted agan for the thrd teraton as: =, where 2 d s d + d2 = n. Now the stuaton s reversed, P2 erforms much faster than P. Ths stuaton wll contnue n subsequent teratons wth the majorty of the comutatonal unts oscllatng between rocessors. (a) (b) Fg.. Predcted results from dynamc load balancng on two rocessors usng constant erformance models. In (a) the roblem sze s small relatve to avalable man memory and balance s acheved. In (b, c) the roblem sze s large and may requre agng, the balancng algorthm causes further unbalance. (b) shows frst and second teratons, (c) shows second and thrd teratons. Outlned onts reresent erformance redcted by constant erformance model. (c)

6 3.2 Exermental Results of the Tradtonal Load Balancng Algorthm The tradtonal load balancng algorthm was aled to the Jacob method, whch s reresentatve of the class of teratve routnes we study. The rogram was tested successfully on a cluster of 6 rocessors. For clarty the results resented here are from two confguratons of 4 rocessors, Table. The essental dfference s that cluster has one rocessor wth 256MB RAM and cluster 2 has two rocessors wth 256MB RAM. Table. Secfcatons of test nodes. Cluster conssts of nodes: P, P 3, P 4, P 5. Cluster 2 conssts of nodes: P, P 2, P 3, P 4. P P 2 P 3 P 4 P 5 Processor 3.6 Xeon 3. Xeon 3.4 P4 3.4 Xeon 3.4 Xeon Ram (MB) The memory requrement of the arttoned routne s a n d bloc of a matrx, three n dmensonal vectors and some addtonal arrays of sze. For 4 rocessors wth an even dstrbuton, roblem szes of n=8 and n= wll have a memory requrement whch les ether sde of the avalable memory on the 256MB RAM machnes, and hence wll be good values for benchmarng. Tme (s) Tme (s) Iteratons Iteratons (a) Cluster wth n = 8 (b) Cluster wth n = Tme (s) Tme (s) Iteratons Iteratons (c) Cluster 2 wth n = 8 (d) Cluster 2 wth n = Fg. 2. Tme taen for each of the 4 rocessors to comlete ther assgned comutatonal unts for each teraton,2,3,. In (a) and (c) the roblem fts n man memory the load converges to a balanced soluton. In (b) and (d) agng occurs on some machnes and the load remans unbalanced.

7 The tradtonal load balancng algorthm wored effcently for small roblem szes, Fg. 2(a, c). For roblem szes suffcently large to otentally cause agng on some machnes the load balancng algorthm caused dvergence as the theory, n secton 2., redcted, Fg. 2 (b,d). st Iteraton 2nd Iteraton sze of roblem, x sze of roblem, x 3rd Iteraton 4th Iteraton sze of roblem, x sze of roblem, x 2 5th Iteraton sze of roblem, x Fg. 3. Tradtonal load balancng algorthm for four rocessors on cluster 2 wth n=. Showng ntal dstrbuton at n/4 and four subsequent teratons. The x axs reresents the number of rows of the matrx are held n memory, and the number of elements of x' comuted by each rocessor. The full functonal erformance models are dotted n to ad vsualsaton. A lot of roblem sze vs. absolute seed can hel llustrate why the tradtonal load balancng algorthm s falng for large roblems. Fg. 3 shows the absolute seed of each of the rocessors for the frst fve teratons. The exermentally bult full functonal model for each rocessor are dotted n to ad vsualsaton but ths nformaton was not avalable to the load balancng algorthm. Intally each rocessor

8 has n/4 rows of the matrx. In the second teraton, P and P 2 are gven very few rows as they both erformed slowly n the frst teraton, however they now comute these few rows qucly. In the thrd teraton, P s gven suffcent rows to cause agng and hence a cycle of oscllatng row allocaton ensues. Snce data arttonng s emloyed n our teratve routne, t s necessary to do data redstrbuton wth each rebalancng. When the balancng algorthm converges qucly to an otmum dstrbuton the networ load from data redstrbuton s accetable. However as the dstrbuton oscllates not only s the comutaton tme affected but so too s the networ load. On cluster 2 wth n= aroxmately 3MB s been assed bac and forth between P and P 2 wth each teraton. 4 Dynamc Load Balancng Based on Accurate Evaluaton of Comutaton Load and Memory Herarchy Our dynamc load balancng algorthm s based on functonal erformance models [3], whch are alcaton centrc and hardware secfc. Functonal erformance models reflect both rocessor and memory heterogenety. In ths secton, we descrbe how the load can be balanced wth hel of these models. The functonal erformance models of the rocessors are reresented by ther seed functons s (d),,s (d), wth s( d) = d/ t( d), where t( d) s the executon tme for rocessng of d elements on the rocessor P. As n tradtonal algorthms, load balancng s acheved when t t,, j. Ths can be exressed as 2 2 j d d d 2, where d+ d d = n. These equatons can be solved s ( d ) s ( d ) s ( d ) geometrcally by ntersecton of the seed functons wth a lne assng through the orgn of the coordnate system (Fg. 4). Ths aroach can be used for statc load balancng. Fg. 4. Otmal dstrbuton of comutatonal unts showng the geometrc roortonalty of the number of chuns to the seed of the rocessor.

9 Functonal erformance models are bult exermentally. Ther accuracy deends on the number of exermental onts. Unfortunately, generatng these seed functons s comutatonally exensve, esecally n the resence of agng. To create just 2 onts of a functon n Fg. 3 too aroxmately 473seconds, 4 tmes longer then the actual calculaton wth a homogeneous dstrbuton for 2 teratons. Ths forbds buldng full functonal erformance models at run tme. However, n ths aer, we aly artal functonal erformance models to dynamc load balancng of teratve routnes. The artally bult erformance models are ecewse lnear aroxmatons of the real seed functons, s ( d) s( d), whch estmate the real functons n detal only n the relevant regons [3]. The low cost of artally buldng the models maes t deal for emloyment n self-adatve arallel alcatons. The artal models can be bult durng the executon of the comutatonal teratve routne. We modfed the tradtonal dynamc load balancng algorthm, resented n Secton 2, usng artal seed functons nstead of sngle seed values. The artal functons s ( d) are bult by addng an exermental ont ( d, s ) after each teraton of the routne. The more onts are added, the closer the artal functon aroxmates the real seed functon n the relevant regon. At each teraton, we aly the balance crtera to fnd a new dstrbuton d +,..., d + by solvng the system of equatons: d + + +, d + d d = n. In few teratons, our s ( d ) s ( d ) s ( d ) d d algorthm wll adatvely converge to the otmal data dstrbuton, snce s ( d) s ( d). Let us outlne how the artal functons s ( d) are constructed. n/ The frst teraton. The seed of each rocessor s calculated as s =. The t ( n/ ) frst aroxmaton of the artal seed functon, s ( d), s created as a constant s ( d) = s, Fg. 5(a). Subsequent teratons. The seed of each rocessor s calculated as s / ( = d t d ). The ecewse lnear aroxmatons s ( d) are mroved by addng the onts ( d, s ), Fg. 5(b). Namely, let ( j) ( j) m {(, )} j () ( ) d s =, d < < d m, be the exermentally obtaned onts of s ( d) used to buld ts current ecewse lnear aroxmaton, then () () () () If d < d, then the lne segment (, s ) ( d, s ) of the s ( d) aroxmaton wll be relaced by two connected lne segments (, s )) ( d, s ) and () () ( d, s ) ( d, s ); ( m) ( ) ( ) ( ) If d > d, then the lne ( d m, s m ) (, s m ) of ths aroxmaton wll be ( ) ( ) relaced by the lne segment ( d m, m ) (, s d s ) and the lne ( d, ) (, s s ); ( ) ( ) If d j d d j+ ( ) ( ) ( ) ( ) < <, the lne segment ( j, j ) ( j +, j + d s d s ) of s ( d) wll be relaced by two connected lne segments ( ) ( ) (, ) ( j +, j + d s d s ). ( ) ( ) ( j, j ) ( d s d, s ) and

10 (a) (b) Fg. 5. Dynamc load balancng usng artal estmaton of the functonal erformance model. 4. Exermental Results For small roblem szes (n = 8, = 4), our algorthm erformed n much the same way as the tradtonal algorthm. For larger roblem szes (n = ), our algorthm was able to successfully balance the comutatonal load wthn a few teratons (Fg. 6). As n the tradtonal algorthm, agng also occurred but our algorthm exermentally ft the roblem to the avalable RAM. Pagng at the 8 th teraton on P demonstrates how the algorthm exermentally fnds the memory lmt of P. The 9 th teraton reresents a near otmum dstrbuton for the comutaton on ths hardware Tme (s) Iteratons Fg. 6. Tme taen for each of the 4 rocessors to comlete ther tas for each teraton. These results are from the same exerment as fg. 5 wth roblem sze n=. A lot of seed vs. roblem sze, Fg. 7, shows how the comutatonal dstrbuton aroaches an otmum dstrbuton wthn 9 teratons. We can see why P erforms slowly at the 8 th teraton. At the 9 th teraton n Fg. 7, we can see that the maxmum erformance of rocessors P and P 2 has been acheved.

11 st Iteraton 2nd Iteraton sze of roblem, x sze of roblem, x 3rd Iteraton 7th Iteraton sze of roblem, x sze of roblem, x 8th Iteraton 9th Iteraton sze of roblem, x sze of roblem, x Fg. 7. Exermental results from load balancng usng artal estmaton of the functonal erformance model wth n=. Showng the st, 2 nd, 3 rd, 7 th, 8 th and 9 th teratons. The lne ntersectng the orgn reresents the otmum soluton and onts converge towards ths lne. 5 Concluson In ths aer, we have shown that tradtonal dynamc load balancng algorthms can fal for large roblem szes on arallel latforms wth memory heterogenety. They do not tae nto account memory herarchy and use smlfed models of rocessors erformance. We have shown that our dynamc load balancng algorthm, n whch erformance s reresented by a functon of roblem sze, can be used successfully wth any roblem sze and on a wde class of heterogeneous latforms. Ths ublcaton has emanated from research conducted wth the fnancal suort of Scence Foundaton Ireland under Grant Number 8/IN./I254.

12 References. Bharadwaj, V., Ghose, D., Robertazz, T.G.: Dvsble Load Theory: A New Paradgm for Load Schedulng n Dstrbuted Systems. Cluster Comut. 6, 7--7 (23) 2. Cerna, M., Za, M.J., L, W.: Comle-Tme Schedulng Algorthms for Heterogeneous Networ of Worstatons. Comuter J. 4, (997) 3. Lastovetsy, A., Reddy, R.: Dstrbuted Data Parttonng for Heterogeneous Processors Based on Partal Estmaton of ther Functonal Performance Models. In: HeteroPar 29. LNCS, vol. 643, Srnger (2) 4. Ichawa, S., Yamashta, S.: Statc Load Balancng of Parallel PDE Solver for Dstrbuted Comutng Envronment. In: PDCS-2, ISCA (2) 5. Legrand, A., Renard, H., Robert, Y., Vven, F.: Mang and load-balancng teratve comutatons. IEEE T. Parall. Dstr. 5, (24) 6. Martínez, J.A., Garzón, E.M., Plaza, A., García, I.: Automatc tunng of teratve comutaton on heterogeneous multrocessors wth ADITHE. J. Suercomut. (to aear) 7. L, X.-Y., Teng, S.-H.: Dynamc Load Balancng for Parallel Adatve Mesh Refnement. In: IRREGULAR'98, Srnger (998) 8. Galndo, I., Almeda, F., Badía-Contelles, J. M.: Dynamc Load Balancng on Dedcated Heterogeneous Systems. In: EuroPVM/MPI 28, Srnger (28) 9. Hummel, S.F., Schmdt, J., Uma, R. N., Wen, J.: Load-sharng n heterogeneous systems va weghted factorng. In: SPAA 96, ACM (996). Carño, R.L., Bancescu, I.: Dynamc load balancng wth adatve factorng methods n scentfc alcatons. J. Suercomut. 44, (28). Cybeno, G.: Dynamc load balancng for dstrbuted memory mult-rocessors. J. Parallel Dstr. Com. 7, (989) 2. Bah, J.M., Contassot-Vver, S., Couturer, R.: Dynamc Load Balancng and Effcent Load Estmators for Asynchronous Iteratve Algorthms. IEEE T. Parall. Dstr. 6, (25) 3. Lastovetsy, A., Reddy, R.: Data Parttonng wth a Functonal Performance Model of Heterogeneous Processors. Int. J. Hgh Perform. Comut. Al. 2, (27)