Multi-level Visuliztion of Conurrent nd Distriuted Computtion in Erlng Roert Bker r440@kent..uk Peter Rodgers P.J.Rodgers@kent..uk Simon Thompson S.J.Thompson@kent..uk Huiqing Li H.Li@kent..uk Astrt This pper desries prototype visuliztion system for onurrent nd distriuted pplitions progrmmed using Erlng, providing two levels of grnulrity of view. Both visuliztions re nimted to show the dynmis of spets of the omputtion. At the low level, we show the onurrent ehviour of the Erlng shedulers on single instne of the Erlng virtul mhine, whih we ll n Erlng node. Typilly there will e one sheduler per ore on multiore system. Eh sheduler mintins run queue of proesses to exeute, nd we visulize the migrtion of Erlng onurrent proesses from one run queue to nother s work is redistriuted to fully exploit the hrdwre. The shedulers re shown s grph with irulr lyout. Next to eh sheduler we drw vrile length r inditing the urrent size of the run queue for the sheduler. At the high level, we visulize the distriuted spets of the system, showing intertions etween Erlng nodes s dynmi grph drwn with fore model. Speifilly we show messge pssing etween nodes s edges nd ly out nodes ording to their urrent onnetions. In ddition, we lso show the grouping of nodes into s groups using n Euler digrm drwn with irles. I. INTRODUCTION Erlng pplitions re typilly onurrent nd distriuted, nd run on multiore pltforms. By visulizing the onurrent proesses involved we n onvey how individul proesses re performing nd ommuniting. This mens the progrmmer n evlute performne, verify progrm orretness nd gin insight into progrm ehviour [10]. This pper desries prototype visuliztion system for Erlng pplitions, giving two (low nd high level) views of the system. Erlng [2], [6] is progrmming lnguge with uilt-in support for onurreny nd distriution. Erlng models the world s sets of prllel proesses tht n intert only y messge pssing, tht is without ny shred memory. Erlng onurreny is diretly supported in the implementtion of the Erlng Virtul Mhine (VM), rther thn indiretly y operting system threds. Messges etween proesses n onsist of dt of ny Erlng type: s well s the si types (integers, oolens, strings nd it strings, or inries) these n e tuples (onsisting of numer of vlues of different types), lists or funtions. Proess identifiers nd symoli nmes n lso e ommunited s (prts of) messges. Erlng supports multi-ore progrmming out of the ox. At strtup, the Erlng VM utomtilly detets the CPU topology on whih it is running nd retes proess sheduler for eh ville CPU ore. Eh proess sheduler mintins its own run queue of proesses to exeute. In order to lne the worklod etween shedulers, proess migrtion (work steling) mehnism is pplied periodilly, so tht shedulers with less work pull proesses from shedulers with more work, while shedulers with more work push proesses to others. A distriuted Erlng system onsists of numer of Erlng VMs ommuniting with eh other. Eh VM is lled n Erlng node. Messge pssing etween proesses on different nodes is indistinguishle from ommunition on the sme node (t lest when proess identifiers (pids) re used for messge ddressing). Whilst different Erlng nodes in distriuted Erlng system n run on physilly seprted omputers, it is lso possile for multiple nodes to run on the sme omputer. The defult model for n Erlng distriuted system is for ll nodes to e onneted to ll other nodes, ut this hs negtive onsequenes for the slility of systems. Erlng nodes in distriuted Erlng system n e prtitioned into groups, where every element in group is onneted to ll others, ut the ft tht the groups re disjoint mens tht ertin kinds of rhitetures re not possile. One of the outomes of the ongoing FP7 RELEASE projet [3] is the notion of s groups. These extend Erlng s originl prtition-sed grouping in numer of wys. As fr s the work desried in this pper is onerned, the mjor differene etween s groups nd groups is tht s groups re not neessrily disjoint, nd so one Erlng node my e memer of multiple s groups, or indeed memer of none. This pper desries prototype visuliztion system for onurrent nd distriuted Erlng pplitions. In the system we provide two grnulrities of view: low level nd high level. The low level view shows the onurrent ehviour of Erlng proesses on single Erlng node, foussing in prtiulr on sheduling spets. The shedulers re lid out in irle: we indite the size of their run queues y the length of rs whih re drwn next to the shedulers. We lso visulize the migrtion of Erlng onurrent proesses from one sheduler to nother, s work is redistriuted to fully exploit the hrdwre: this migrtion is represented y nimted edges etween shedulers. The high level view visulizes the distriuted spets of the system, with eh grph node representing n Erlng node. We
show messge pssing etween nodes s edges, lying nodes out ording to their urrent onnetions. The grouping of nodes into s groups is shown y surrounding eh s group y n interlinking irle. Dynmi ommunition ehviour is nimted using fore model [7]. The high level lyout is produed y first drwing n Euler digrm for the interseting s groups of Erlng nodes [19]. The nodes re then pled in the pproprite zone of the digrm, nd fore-direted lyout is pplied to the ommunition grph. The fores in the lyout inlude the stndrd spring emedder fores [7], with n dditionl repulsion from the edges of the irles to keep the nodes within the orret regions [16]. Whilst the numer of nodes nd their grouping into s groups is fixed nd does not hnge during exeution (t lest in this prototype), the ommunition etween nodes does hnge over time, nd the use of fore sed pproh llows the lyout of the grph to lter dynmilly in n nimted wy. The reserh ontriution of this pper is to desrie the first visuliztion for onurrent nd distriuted Erlng exeution. The visuliztion hs novel fetures: The use of Euler digrms to show groups of virtul mhine nodes in onurrent progrms is new. Whilst inspired y previous pprohes [16], [7], our method of integrting Euler digrms with grphs represents new omintion of fores. The two different levels of view with disrete visuliztions hs not een seen, to our knowledge, in previous systems for onurrent nd distriuted visuliztion. The omintion of r hrts nd irulr grph lyout ppers to e new in the ontext of onurrent proess visuliztion. In the reminder of this pper, we first disuss relted work in Setion II. Setion III then provides n overview of the low level visuliztion nd nimtion method, whilst Setion IV disusses the high level method. Finlly, Setion V presents our onlusions nd outlines possile future work. II. RELATED WORK Cirulr lyout hs een used previously for proesses on ore [10]. However, in this erlier work, ommunition ws shown y dots moving from one proess to nother. This requires the users ttention during the movement of the dot to identify the strt nd ending proesses, whih is prtiulrly prolemti when severl proesses re ommuniting t the sme time. We improve on this y using urved edges to show the ommunition etween two proesses. These edges fde over time, to prevent the disply from eoming overly luttered. We note tht the use of urved edges follow the Gestlt priniple of good ontinution [17], nd so n e justified y pereptul theories. Justifition for grouping proesses y surrounding groups with ontours whih overlp in the high level view, nd do not overlp in the low level view is provided y the prettentive priniple of losure [20]. Visuliztion of ommunition grphs in onurrent proesses hs een prevlent for some time. Visputer [22] hs multi-level pproh to visulizing suh grphs, where single node n ontin either proess or ommunition sugrph. This onsistent pproh to the different levels, where lower levels hve the sme syntx s higher levels, ontrsts with our pproh where the visul disply of the two levels is tuned to the different digrm semntis. The Mender projet [21] not only visulized ommunition grphs, it used them s the sis for visul onurrent progrmming lnguge. The use of grph visuliztion for the presenttion of ommunition grphs is widespred, see for instne 1992 survey [11]. More reent work hs pplied vriety of grph drwing methods to utomtilly generte lyouts for vrious grphs ssoited with onurrent nd distriuted progrms [4], [14], [18]. Another pproh to visulising Erlng progrms is provided y Perept2 [12]. Perept2 is tool for offline visulistion of Erlng pplition level onurreny nd identifying onurreny ottleneks. It utilises Erlng s uilt-in support for tring to monitor events from proess sttes. Tre events re olleted nd stored in file, whih n then e nlysed offline. One the nlysis is done, the dt n e viewed in hrt form through we-sed interfe. Perept2 gives piture of pplition-level prllelism, s well s how muh time proesses spend witing for messges. It lso exposes sheduler-relted tivities to end-users, nd provides finer-grined profiling informtion out the existing prllelism of n pplition. It is lso designed to sle well to prllel pplitions running on multiple ores. VmpirTre/Vmpir [15] is tool set nd runtime lirry for instrumenttion nd tring of softwre pplitions. It llows vriety of detiled performne properties to e olleted nd reorded during runtime, inluding funtion enter nd leve events, MPI ommunition, OpenMP [1] events, nd performne ounters. VmpirTre provides the input dt for the Vmpir nlysis nd visuliztion tool. Sls [8] is nother representtive tre-sed tool tht supports performne optimiztion of prllel progrms y mesuring nd nlysing their runtime ehviours. The nlysis identifies potentil performne ottleneks, in prtiulr those onerning ommunition nd synhroniztion, nd offers guidne in exploring their uses. Finlly, ThredSope [9] is used for performne profiling of prllel Hskell progrms. ThredSope reds GHC-generted tring events from log file, nd displys the thred profile informtion in we-sed grphil viewer. III. LOW LEVEL VISUALIZATION The low level visuliztion shows the onurrent ehviour of Erlng proesses on single Erlng node. We visulize the migrtion of Erlng onurrent proesses from one sheduler to nother nd the size of the run queue on eh sheduler. The visuliztion system (oth low level nd high level omponents) is developed in JvSript nd rendered in SVG using the Dt-Driven Douments lirry (D3) [5]. The visuliztion is re-drwn on every itertion nd this gives the effet of nimting the stte of the shedulers to reflet the urrent
p0 10725 p0 0 10719 0 p0 10673 0 Fig. 1. p0 10708 0 p0 10707 0 p0 10705 0 Low Level Visuliztion p0 10433 0 sitution. This onstnt re-drwing is required s the system is dynmi: proesses n e reted, terminte or migrte etween shedulers t ny time. The dt used in this pper ws generted from running Dilyzer [13], the Erlng stti type nlysis tool, on seletion of Erlng pplitions nd oserving the omputtion. During the running of the pplition, every proess migrtion etween shedulers is reorded using Erlng s uilt-in support for tring, nd the size of run queues is smpled every milliseond. The tre dt re reorded in n Erlng file, with eh tre dtum represented y n Erlng tuple (tht is olletion of Erlng vlues), nd dt nlysis is performed y n Erlng progrm. The pplition ws run on n Intel mhine, running CentOS 6, with two proessors, eh hving six physil CPU ores. Through hyper-threding tehnology, the OS is le to ddress two logil ores for eh physil CPU ore, therefore there re 24 logil ores in totl for n Erlng node to exploit. An exmple frme in the low level visuliztion n e seen in Figure 1. The prllel mhine is split into two six-ore proessors, represented y the two pink semiirles. Erlng shedulers re shown s numer tht represents the unique ID of the relted threds. Pirs of Erlng shedulers tht run on single rel ore re shown grouped y the lue ellipses. Eh Erlng sheduler hs n individul run queue, nd this is represented y r rditing from the visuliztion. The height nd olour re used to indite the size of the run queue: lrger queues (i.e. usier Erlng shedulers), re shown with longer nd redder rs, wheres smller queues re represented y shorter nd greener rs. The run queue size n e displyed on sreen s vlue y seleting the relevnt option in the interfe, or y hovering ursor over the r or Erlng sheduler in question. It is possile tht the queues n e so usy tht they re not fully displyed on sreen, if this is the se, the user is le to tke dvntge of the slility of SVG grphis nd lter the zoom level on their rowser. Proesses get spwned on prtiulr sheduler. The list of numers rditing from Erlng sheduler shows the unique ID for eh proess spwned on tht sheduler. As n e seen in Figure 1, Erlng sheduler 5 hs just spwned numer of new proesses. Proesses my migrte from one Erlng sheduler to nother nd this is shown y edges etween vrious shedulers. These re urved to void olusion with other shedulers (e.g. migrtion from 10 to 18 would ross shedulers 20 nd 8). The edges re oloured to indite the three types of migrtion: within single ore (green), etween different ores on the sme proessor (grey), nd from ore on one proessor to ore on the other (lue), lthough these migrtions n lso e dedued from the strt nd end position of the edge. These edges fde wy fter few seonds to prevent the visuliztion from eoming oluded. The entire visuliztion displys the urrent stte of the tre nd is nimted suh tht new time frme (itertion) is shown every 200 milliseonds nd eh time frme is 1 milliseond smple of the originl tre, tht is, the nimtion runs 200 times more slowly thn the originl pplition. This llows the user to fous on the hnging stte without eing overwhelmed with the mount of nimtion. In ddition, it llows enough proessing time for the nimtions to disply orretly on slower mhines. The nimtion n e pused nd resumed using the interfe. A slider llows the seletion of speifi time frme nd permits the nimtion of sequene of time frmes in either diretion. Figure 2 shows three snpshots of the low level view t vrious stges of exeution of n Erlng progrm. The top digrm shows the stte t itertion 890, where the none of the shedulers re prtiulrly usy, nd the mount of ommunition etween different shedulers is orrespondingly low. The middle digrm, t itertion 967, shows the Erlng node in muh more tive stte, here the run queues re extremely uneven. The ottom digrm shows the stte t itertion 3804, where the disprity in run queue size is now extreme, with three outliers tht hve full run queues, nd others with reltively short queues. A numer of interesting phenomen re indited y the visuliztions. A ler onsequene of hyper-threding is tht there is n symmetry etween the two shedulers on single physil ore. This is evident in the visulistion in the different lengths of the two run queues on single ore (e.g. ores 0 nd 12), resulting in the lternting pttern seen in the figure. Setting side this intr-ore issue, migrtion does not ensure tht run queues hve the sme size: however this is not neessry, sine ll tht is needed is tht eh ore hs enough work to ensure tht it is fully oupied.
Fig. 3. High level view. p0 9152 0 p0 10890 0 p0 9501 0 Fig. 4. Initil plement of Erlng nodes efore fore pplition. Fig. 2. Three steps in the low level view IV. HIGH LEVEL VISUALIZATION The high level visuliztion shows the grouping of Erlng nodes within prtiulr s groups nd edges visulizing the ommunition etween the nodes. Eh node on this high level visuliztion orresponds to n instne of the low level visuliztion. An exmple of the high level visuliztion is shown in Figure 3. The s groups re drwn first, with eh s group eing represented y irle. As node my e the memer of more thn one s group, the irles typilly interset. The s group memership is derived nd is pssed to ode desried y [19] whih returns the irle entres nd rdii tht will then e drwn in SVG. The interseting irles form onneted regions nd the lrgest retngle within eh region is lulted (shown in green in Figure 4). Nodes tht re
ontined in the relevnt s groups re drwn in grid pttern within the retngle. The method for drwing Euler digrms often retes extr regions, in whih no node is pled. In some visuliztion systems (prtiulrly those using Venn digrms) these empty regions n e shded to indite tht they hve no elements inside. However, we regrd it s self-evident tht there re no elements in empty regions nd so we do not dd the extr syntti omplexity of shding. A fore-direted lyout is then pplied to the nodes to improve the lyout. The fores re the stndrd spring emedder ttrtion nd repulsion fores [7], with n dditionl fore. This third fore is pplied to node to keep it wy from the orders of the region, nd is simplified version of the fore introdued in [16]. It is proportionl to the node s distne from the irumferene every irle, pushing the node towrd the entre of irles in whih it is inside, whilst pushing the node wy from the entre of irles whih it is outside of. Additionlly, test is mde to ensure tht nodes do not leve the region to whih they elong y disregrding ny node movement tht moves it outside the orret region (lthough nodes re llowed to move outside their strting retngle). Communition etween nodes is represented y edges nd these my hnge during the progrm s exeution. The shding of the edges is relted to the frequeny of messges within the timefrme. The drker the edge, the more frequently messges re sent. Edges re removed if no ommunition exists etween them nd new edges re introdued when ommunition is initited etween nodes. The use of foredireted lyout method llows the visuliztion to e dynmi, so tht the repositioning of the nodes n e shown in n nimted mnner s the digrm rehes new equilirium. The dt used in this pper ws generted y running distriuted Erlng enhmrk pplition on olletion of Erlng nodes, whih were onfigured to form the s groups (these s groups were mnully defined, s no s group dt is urrently present in the tre dt). The totl wlllok runtime of the pplition ws 222 seonds. On eh prtiipting Erlng node, Erlng s uilt-in support for tring ws used to reord messge sending events. Eh suh event reorded ontins the nme of the sending node, the nme of the reeiving node s well s the tul messge sent. Tre dt from eh node were written to file, nd the dt files were then olleted, nlyzed nd synthesized to provide the visuliztion tool with the umulted ommunition dt on eh itertion, whih ours every 200 milliseonds. Figure 5 shows some steps in the nimtion proess. The top digrm shows the visuliztion fter 100 itertions sine the strt of the tre. At this point there re reltively few ommunition edges etween Erlng nodes. In the middle digrm, n itertion lter, the numer of ommunition edges suddenly inreses drmtilly, inditing tht the sheduling hs initited numer of ommunitions etween nodes. In terms of the visuliztion, the middle digrm hs not hd time for the fores to e pplied to the nodes, nd so the nodes re shown in the sme position s in the top digrm. After nother 99 itertions, the fores find n new equilirium, d 100 itertions d 101 itertions d 200 itertions Fig. 5. Three steps in the nimtion of the high level view.
with the nodes in the positions s shown in the lower digrm. Here, the nodes hve moved further wy from the region entres, so inditing stronger ffinity to nodes outside their regions. However, the only node in s group (top irle) hs een reltively unffeted y this, s the numer of nodes it is ommuniting with hs not hnged. V. CONCLUSION This pper hs desried two level visuliztion for Erlng distriuted nd onurrent proesses. The omplexity of suh systems requires mixed pproh where onurreny nd distriution re visulized y distint methods. Conurreny in Erlng nodes hs een represented in low level visuliztion y nimting ommunition edges etween Erlng shedulers whih re shown in irulr lyout. How usy eh Erlng sheduler is n e seen y the length of the r ssoited with tht sheduler. The distriuted nture of Erlng omputtion is shown in high level visuliztion using dynmi grph of ommunition etween Erlng nodes. Nodes re pled in s groups whih is shown y drwing the nodes in regions of Euler digrm drwn with irles. The hnges in ommunition etween nodes is nimted y fore-direted lyout, whih lso mintins the nodes in their s groups. Informl feedk from RELEASE projet memers hs een soliited. This hs resulted in rodly positive retions, with suggestions for lterntive dt tht might e represented y the edges etween proesses or nodes. This leds to the first spet of further work: one the system is more stle, there is potentil for forml evlution of the system to see if it is suessful in its ims of: ssisting system performne evlution; iding the verifition of progrm orretness; nd improving insight into progrm ehviour. Given the limited numer of distriuted Erlng experts, nd the onsiderle mount of time tht would need to e devoted to using the visuliztions to get useful informtion, longitudinl study with hndful of prtiipnts might e the est mehnism for this. Before suh n evlution, the user interfe will hve to e improved, nd the onnetion etween the two levels of visuliztion needs to integrted more losely. Other further work reltes to modifying the system to del with n inresing mount of distriution. The urrent high level visuliztion shows eh Erlng node s grphil node. Whilst this hs the dvntge of showing ll Erlng nodes, it n eome unwieldy when their numer grows. To llow the system to sle we expet to luster Erlng nodes, using mehnisms similr to those developed in [22]. When pplied to distriuted Erlng, grphil node might represent n s group, nd edge ffinity ould e omposite mesure of overlp nd ommunition ndwidth etween groups. The system so fr supports offline visulistion, ut the sme grphil pproh n e used to provide n online tht is, rel time view. The hllenge of online visulistion in this se is to provide strems of suitle ggregte dt without dversely ffeting the performne of the system under oservtion. These strems of dt ould e provided y the uilt-in Erlng tring mehnism, or using Unix tehing frmework suh s DTre / SystemTp. Aknowledgements This reserh is supported y EU FP7 ollortive projet RELEASE (http://www.releseprojet.eu/), grnt numer 287510. REFERENCES [1] The OpenMP API speifition for prllel progrmming. http://openmp.org/wp/. [2] Erlng/OTP. http://www.erlng.org, My 2013. [3] RELEASE. http://www.relese-projet.eu/, My 2013. [4] Benjmin Alreht, Philip Effinger, Mrkus Held, Mihel Kufmnn, nd Stephn Kottler. Visuliztion of omplex pel models. In Grph Drwing, pges 421 423. Springer, 2010. [5] Mihel Bostok, Vdim Ogievetsky, nd Jeffrey Heer. D3 dtdriven douments. IEEE Trnstions on Visuliztion nd Computer Grphis, 17(12):2301 2309, Deemer 2011. [6] Frneso Cesrini nd Simon Thompson. Erlng Progrmming. O Reilly Medi, In., 2009. [7] Peter Edes. A heuristi for grph drwing. Congressus Numerntium, 42:149 160, 1984. [8] Mrkus Geimer, Felix Wolf, Brin J. N. Wylie, Erik rhm, Dniel Beker, Bernd Mohr, nd Forshungszentrum Jlih. The SCALASCA performne toolset rhiteture. Conurr. Comput. : Prt. Exper., 22(6):702 719, April 2010. [9] Don Jones Jr., Simon Mrlow, nd Stnm Singh. Prllel performne tuning for Hskell. In Hskell Symposium 2009, Edinurgh, Sotlnd, Septemer 2009. ACM Press. [10] Eileen Kremer nd John T Stsko. Creting n urte portryl of onurrent exeutions. Conurreny, IEEE, 6(1):36 46, 1998. [11] Eileen T Kremer nd John T Stsko. The visuliztion of prllel systems: An overview. 1992. [12] H. Li nd S. Thompson. Multiore Profiling for Erlng Progrms Using Perept2. In Erlng Workshop 2013, 2013. [13] Tois Lindhl nd Konstntinos Sgons. Prtil type inferene sed on suess typings. In Proeedings of the 8th ACM SIGPLAN Symposium on Priniples nd Prtie of Delrtive Progrmming, pges 167 178, New York, NY, USA, 2006. ACM Press. [14] J. Lonnerg, M. Ben-Ari, nd L. Mlmi. Visulising onurrent progrms with dynmi dependene grphs. In Visulizing Softwre for Understnding nd Anlysis (VISSOFT), 2011 6th IEEE Interntionl Workshop on, pges 1 4, 2011. [15] Mtthis S. Mller, Andres Knpfer, Mtthis Jurenz, Mtthis Lieer, Holger Brunst, Hrtmut Mix, nd Wolfgng E. Ngel. Developing Slle Applitions with Vmpir, VmpirServer nd VmpirTre. In PARCO, volume 15 of Advnes in Prllel Computing, pges 637 644. IOS Press, 2007. [16] P. Mutton, P. Rodgers, nd J. Flower. Drwing grphs in Euler digrms. In Proeedings of 3rd Interntionl Conferene on the Theory nd Applition of Digrms, volume 2980 of LNAI, pges 66 81, Cmridge, UK, Mrh. Springer. [17] A. Rusu, A.J. Fin, R. Jinu, nd A. Rusu. Using the gestlt priniple of losure to llevite the edge rossing prolem in grph drwings. In Informtion Visulistion (IV), 2011 15th Interntionl Conferene on, pges 488 493, 2011. [18] Lus Mello Shnorr, Arnud Legrnd, Jen-Mr Vinent, et l. Intertive nlysis of lrge distriuted systems with topology-sed visuliztion. 2012. [19] Gem Stpleton, Leishi Zhng, John Howse, nd Peter Rodgers. Drwing euler digrms with irles: The theory of pierings. Visuliztion nd Computer Grphis, IEEE Trnstions on, 17(7):1020 1032, 2011. [20] Anne Treismn nd Jnet Souther. Serh symmetry: dignosti for prettentive proessing of seprle fetures. Journl of Experimentl Psyhology: Generl, 114(3):285, 1985. [21] Guido Wirtz. A visul pproh for developing, understnding nd nlyzing prllel progrms. In Visul Lnguges, 1993., Proeedings 1993 IEEE Symposium on, pges 261 266. IEEE, 1993. [22] Kng Zhng nd Gurv Mrwh. Visputer grphil visuliztion tool for prllel progrmming. The Computer Journl, 38(8):658 669, 1995.