REMO: Resource-Aware Application State Monitoring for Large-Scale Distributed Systems

Transcription

1 : Resoure-Awre Applition Stte Monitoring for Lrge-Sle Distriuted Systems Shiong Meng Srinivs R. Kshyp Chitr Venktrmni Ling Liu College of Computing, Georgi Institute of Tehnology, Atlnt, GA 332, USA {smeng, IBM Reserh T.J. Wtson, Hwthorne, NY 532, USA {srkshy, Astrt To oserve, nlyze nd ontrol lrge sle distriuted systems nd the pplitions hosted on them, there is n inresing need to ontinuously monitor performne ttriutes of distriuted system nd pplition sttes. This results in pplition stte monitoring tsks tht require finegrined ttriute informtion to e olleted from relevnt nodes effiiently. Existing pprohes either tret multiple pplition stte monitoring tsks independently nd uild d-ho monitoring trees for eh tsk, or onstrut single stti monitoring tree for multiple tsks. We rgue tht reful plnning of multiple pplition stte monitoring tsks y jointly onsidering multi-tsk optimiztion nd node level resoure onstrints n provide signifint gins in performne nd slility. In this pper, we present, REsoure-wre pplition stte MOnitoring system. produes forest of optimized monitoring trees through itertions of two phses, one phse exploring ost shring opportunities vi estimtion nd the other refining the monitoring pln through resoure-sensitive tree onstrution. Our experimentl results inlude those gthered y deploying on BlueGene/P rk running IBM s lrge-sle distriuted streming system - System S. Using in the ontext of running over monitoring tsks for n pplition deployed ross nodes results in 35%-45% derese in the perentge error of olleted ttriutes ompred to existing shemes. 1. Introdution Lrge-sle distriuted systems suh s loud omputing systems [1] nd strem-proessing systems [2], [3] llow multiple pplitions to e deployed over different sets of hosts. These systems pose interesting hllenges for monitoring the funtioning of oth systems nd pplitions running on them. We refer to the tsk of ontinuously monitoring the stte of pplitions deployed on suh systems s pplition stte monitoring, whih is essentil for the oservtion, nlysis nd ontrol of these systems nd the long running pplitions they host. Applition stte monitoring requires ontinuously monitoring nd olleting performne ttriute vlues, suh s CPU-usge, memory-usge, dt rte, pket size distriution, nd ny numer of ustom ttriutes t the pplition level. In mny ses, it is useful to ollet detiled performne ttriutes t ontrolled olletion frequeny. For instne, fine-grined performne hrteriztion informtion is required to onstrut vrious system models nd to test hypotheses on system ehvior. Similrly, the dtrte nd uffer oupny in eh element of distriuted pplition my e required for dignosis purposes when there is pereived ottlenek. As the sle of the system, pplition nd monitoring tsks inreses, the overhed of the stte monitoring nd ttriute olletion n eome prohiitive. A lrge ody of reserh ddresses the design of distriuted monitoring systems nd hs minly foused on exeuting single monitoring tsks effiiently (TAG [4], SDIMS [5], PIER [6], pproximte quntiles [7], [8], join ggregtions [9], REED [], opertor plement [11]). These pprohes use either stti monitoring trees [12], [11], where pre-onstruted monitoring tree is used for ll monitoring tsks ross ll pplitions, or onstrut monitoring trees for individul monitoring tsks independently [6]. We rgue tht these pprohes suffer from two mjor drwks. Firstly, performing multiple pplition stte monitoring tsks using either pre-onstruted single monitoring tree or onstruting one stte monitoring tree for eh pplition my result in su-optiml monitoring pln. For exmple, if two tsks re performed over the sme set of nodes, using one monitoring tree for monitoring dt trnsmission is more effiient thn using two, s nodes n merge updtes for oth tsks nd redues per-messge proessing overhed. While finding n optiml monitoring network eomes muh more omplited s the size of the system, the numer of monitoring tsks, nd the durtion of monitoring inrese, it is ritil for the monitoring slility. Seondly, these pprohes fil to onsider node level resoure onstrints in the onstrution of stte monitoring trees, nd thus, suffer from node overlod nd onsequently serious monitoring dt loss. This prolem gets worse in pplition stte monitoring, s the olleting of individul ttriute vlues from different nodes in lrge sle distriuted systems demnds onsiderle resoures nd nodes my dedite most of their pities for pplition hosting.

2 Without onsidering resoure onstrints in stte monitoring plnning, these pprohes my use ertin nodes to e ssigned with exessive monitoring worklod, leding undesirle performne degrdtion. Thus, we envision tht pplition stte monitoring in lrge sle distriuted systems should e slle, effiient, nd resoure-wre. In this pper, we rgue tht reful plnning of multiple pplition stte monitoring tsks y jointly onsidering multi-tsk optimiztion nd resoure-onstrined monitoring tree onstrution n provide signifint gins in performne nd slility. We present, resoure-wre pplition stte monitoring system, whih produes forest of optimized monitoring trees through itertions of two proedures. One proedure explores opportunities for shring per-messge proessing overhed sed on performne estimtion. The other proedure refines the monitoring pln produed in the previous proedure y uilding monitoring trees with resoure sensitive tree onstrution lgorithm. We undertke n experimentl study of our system nd present results inluding those gthered y deploying on BlueGene/P rk (using 256 nodes ooted into Linux) running IBMs lrge-sle distriuted streming system - System S. The results show tht our resoure-wre pproh for pplition stte monitoring onsistently outperforms the urrent est known shemes for multiple pplition stte monitoring. For instne, in our experiments with rel pplition tht spnned up to nodes nd out s mny monitoring tsks, using to ollet ttriutes resulted in 35%-45% redution in the perentge error of the ttriutes tht were olleted. This pper mkes two unique ontriutions: We hrterize nd identify two importnt requirements for effiient pplition stte monitoring in lrge sle distriuted systems: the shring of messge proessing ost mong ttriutes nd onsidering node level resoure onstrints. These two requirements re essentil for the slility of monitoring nd existing pprohes filed to ddress them. We develop frmework for ommunition-effiient pplition stte monitoring, whih jointly solves prolems of multiple monitoring tsk optimiztion nd resoure-wre monitoring network onstrution. To our est knowledge, is the first system tht promotes resoure-wre methodology to support nd sle multiple pplition stte monitoring tsks in lrge sle distriuted systems. 2. System Overview Figure 1 illustrtes the rhiteture of our system. The tsk mnger tkes stte monitoring tsks nd removes duplition mong pplition stte monitoring tsks. These de-duplited tsks re then feed to the mngement ore, where the monitoring plnner rries out the resoure wre multi-tsk optimiztion y orgnizing monitoring nodes into forest of optimized monitoring trees. The mngement ore lso provides support for reliility enhnement nd filure hndling. The dt olletor module provides lirry Monitoring Tsks Tsk Mnger De-duplited tsks Reliility Hndler Root Sme node Filure Hndler Monitoring Plnner Dt Colletor Sme node Cost Estimtor Root Mngement Core Result Proessor Monitoring Nodes Applition Sttes App CPU Usge Throughput xxx xx;xx;xx;xx; xx;xxx;x xx x;xx;x;xx; x;xxx;x xxx xx;xx;xx;xx; xx;xxx;x Figure 1: A Simplified System Arhiteture for effiient olleting ttriute vlues from the monitoring network. The result proessor exeutes the onrete monitoring opertions inluding olleting nd ggregting ttriute vlues, triggering wrnings, nd et. In this pper, we fous on the design of the monitoring plnner. Applition stte monitoring tsks re long running tivities used for oservtion, nlysis, nd ontrol of lrge sle distriuted systems or pplitions they host. Eh tsk periodilly ollets vlues of ertin ttriutes from the set of nodes over whih n pplition is running. Formlly, we define n pplition stte monitoring tsk t s follows: Definition 1: A monitoring tsk t =(A t,n t ) is pir of sets, where A t i N t A i is set of ttriutes nd N t N is set of nodes. In ddition, t n lso e represented s list of node-ttriute pirs (i, j), wherei N t,j A t. As suh systems run multiple pplitions simultneously nd my monitor mny fets of n pplition, more often thn not, different monitoring tsks my shre duplited monitored ttriutes. For instne, monitoring tsks t 1 = ({pu utiliztion}, {, }) nd t 2 = ({pu utiliztion}, {, }) hve duplited monitored ttriute pu utiliztion on node. With suh duplition, node hs to send pu utiliztion informtion twie for eh updte, whih is lerly unneessry. Therefore, given set of monitoring tsks, the tsk mnger trnsforms this set of tsks into list of node-ttriute pirs nd elimintes ll duplited nodettriute pirs. For instne, t 1 nd t 2 re equivlent to the list {-pu utiliztion, -pu utiliztion} nd {pu utiliztion, -pu utiliztion} respetively. In this se, node-ttriute pir {-pu utiliztion} is duplited, nd thus, is eliminted from the output of the tsk mnger. Given list of node-ttriute pirs, the monitoring plnner orgnizes ssoited nodes into forest of monitoring trees, eh of whih ollets vlues for set of ttriutes. One monitoring node my onnet to multiple trees (s shown in Figure 1 nd 3()). Within monitoring tree T, eh node i periodilly sends n updte messge to its prent. Suh updte messges inlude oth vlues lolly oserved y node i nd vlues sent y i s hildren, for ttriutes monitored y T. This proess ontinues upwrds in the tree until the messge rehes the entrl dt olletor node. Eh monitoring node hs set of oservle ttriutes A i = { j j [1,m]}. Attriutes t different nodes ut with the sme susription re onsidered s ttriutes of the sme type. For instne, monitored nodes my ll hve lo-

3 CPU usge (logsle) Num. of nodes = Num. of messges t str-root nd Messge Size [x ''] num-messges messge-size Figure 2: CPU Usge vs Inresing Messge Numer/Size lly oservle CPU utiliztion. We onsider n ttriute s ontinuously hnging vrile whih outputs new vlue in every unit time. For simpliity, we ssume ll ttriutes re of the sme size nd it is strightforwrd to extend our work to support ttriutes with different sizes. Eh node i, the entrl node or monitoring node, hs pity i (lso referred to s the resoure onstrint of node i) for reeiving nd trnsmitting monitoring dt. In ddition, eh messge trnsmitted in the system is ssoited with per-messge overhed C, nd, orrespondingly, the ost of trnsmitting messge with x vlues is C + x. In our setting, we hrterize the per-messge overhed s proessing overhed. Figure 2 shows how signifint the per-messge proessing overhed is. The mesurements were performed on BlueGene/P node whih hs 4-ore 8MHz PowerPC proessor. The figure shows n exmple monitoring tsk where nodes re onfigured in str network where eh node periodilly trnsmits single fixed smll messge to root node. The CPU utiliztion of the root node grows roughly linerly from round 6% for 16 nodes (the root reeives 16 messges periodilly) to round 68% for 256 nodes (the root reeives 256 messges periodilly). Note tht this inresed overhed is due to the inresed numer of messges t the root node nd not due to the inrese in the totl size of messges. The ost inurred to reeive single smll messge of size is round.2% while the ost inurred to reeive single messge of size 256 is still round 1.4% (see Figure 2). In other senrios, the per-messge overhed ould e trnsmission or protool overhed. For instne, typil monitoring messge delivered vi TCP/IP protool hs messge heder of t lest 78 ytes not inluding pplition-speifi heders, while n integer monitoring dt is just 4 ytes Chllenges in Monitoring Plnning From the users perspetive, monitoring results should e s urte s possile, suggesting tht the underlying monitoring network should mximize the numer of nodettriute pirs reeived t the entrl node. In ddition, suh monitoring network should not use the exessive use of resoure t ny node. Aordingly, we define the monitoring plnning prolem (MP) s follows: Prolem Sttement 1: Given set of node-ttriute pirs for monitoring Ω={ω 1,ω 2,...,ω p } where ω q =(i, j), i N, j A, q [1,p], nd resoure onstrint i for eh ssoited node, find prent f(i, j), i, j, wherej A i 1.3 suh tht node i forwrds ttriute j to node f(i, j) tht mximizes the totl numer of node-ttriute pirs reeived t the entrl node nd the resoure demnd of node i, d i, stisfies d i i, i N. NP-ompleteness. When restriting ll nodes to only monitor the sme ttriute j, we otin speil se of the monitoring plnning prolem where eh node hs t most one ttriute to monitor. As shown y [13], this speil se is NP-omplete prolem. Consequently, the MP is NP-Complete prolem, sine eh instne of MP n e restrited to this speil se. Therefore, in, we primrily fous on effiient pprohes tht n deliver resonly good monitoring plns. We now use some intuitive exmples to illustrte the hllenges nd key questions tht need to e ddressed in designing resoure-wre monitoring plnner. Consider exmples in Figure 3, where we hve 6 monitoring nodes in the system nd eh monitors set of ttriutes indited y lphets on nodes. Exmple () shows widely used topology in whih every nodes send their updtes diretly to the entrl node. This topology, however, hs poor slility, euse it requires the entrl node to hve lrge mount of resoures to ount for per-messge overhed. Exmple () orgnizes ll nodes in single tree whih delivers updtes for ll ttriutes. While this topology redues the resoure onsumption t the entrl node, the root node now hs to rely updtes for ll node-ttriute pirs, nd gin fes slility issues due to limited resoures. These two exmples suggest tht hieving ertin degree of lod lning is ritil. Unfortuntely, lod lne lone does not led to good monitoring pln. In exmple (), to lne the trffi mong nodes, the entrl node uses three trees, eh of whih delivers only one ttriute, nd thus hieves more lned worklod mong nodes ompred with exmple (). However, sine eh node monitors t lest two ttriutes, nodes hve to send out multiple updte messges insted of one s in exmple (). Due to per-messge overhed, this pln leds to higher resoure onsumption t lmost every node. As result, ertin nodes my still fil to deliver ll updtes nd less resoures will e left over for dditionl monitoring tsks. The ove exmples revel two fundmentl spets of the monitoring plnning prolem: How to determine the numer of monitoring trees nd the set of ttriutes on eh? This is non-trivil prolem. Exmple (d) shows topology whih uses one tree to deliver ttriute, nd nother tree to deliver ttriute. It introdues less per-messge overhed ompred with exmple () nd is more lod-lned solution ompred with exmple (). How to determine the topology of eh monitoring tree under node level resoure onstrints? Construting monitoring trees sujet to resoure onstrints t nodes is lso non-trivil prolem nd the hoie of topology n signifintly impt the resoure usge on the prtiipting nodes. Exmple (e) shows three different

4 C entrlnode, C entrlnode, C entrlnode C entrlnode,,,,, STAR MIXED CHAIN () Str olletion () One-Set olletion () Singleton-Set olletion (d) Optiml olletion (e) Different Trees Figure 3: Motivting exmples for the topology plnning prolem. trees. The str topology (upper left), while introdues the lest relying ost, uses signifint per-messge overhed t its root. The hin topology (upper right), on the ontrry, distriutes the per-messge overhed mong ll nodes, ut uses the most relying ost. A mixed tree (ottom) might hieve good trde-off etween relying ost nd per-messge overhed, ut it is diffiult to determine its optiml topology. 3. The Approh The pproh promotes the resoure wre multitsk optimiztion frmework, onsisting of two phse itertive proess nd suite of multi-tsk optimiztion tehniques. At high level, opertes s guided lol serh pproh, whih strts with n initil monitoring network omposed of multiple independently onstruted monitoring trees, nd itertively optimizes the monitoring network until no further improvements re possile. When exploring vrious optimiztion diretions, employs ost estimtion to guides susequent improvement so tht the serh spe n e restrited to smll size. This guiding feture is essentil for the slility of. Conretely, during eh itertion, first runs prtition ugmenttion proedure whih genertes list of most promising ndidte ugmenttions for improving the urrent distriution of monitoring worklod mong monitoring trees. While the totl numer of ndidte ugmenttions is very lrge, this proedure n trim down the size of the ndidte list y seleting the most promising ones through ost estimtion. Given the generted ndidte ugmenttion list, the resoure-wre evlution proedure further refines ndidte ugmenttions y uilding monitoring trees ordingly with resoure-wre tree onstrution lgorithm Prtition Augmenttion The prtition ugmenttion proedure is designed to produe the ttriute prtitions tht n potentilly redue permessge proessing ost through guided itertive proess. These ttriute prtitions determine the numer of monitoring trees in the forest nd the set of ttriutes eh tree delivers. To etter understnd the design priniples of our pproh, we first riefly desrie two simple ut essentilly the-stte-of-the-rt shemes in multi-tsk monitoring. Rell tht mong exmple shemes in Figure 3, one sheme (exmple ()) delivers eh ttriute in seprte tree, nd the other sheme (exmple ()) uses single tree to deliver updtes for ll ttriutes. We refer to these two shemes s the Singleton-set prtition sheme (SP) nd the One-set prtition (OP) sheme respetively. We use the term prtition euse these shemes prtition the set of monitored ttriutes into numer of non-overlpping susets nd ssign eh susets to monitoring tree. Singleton-Set Prtition (SP). Speifilly, given set of ttriutes A for olleting, singleton-set prtition sheme divides A into A susets, eh of whih ontins distint ttriute in A. Thus, if node monitors m ttriutes, it is ssoited with m trees. This sheme is widely used in previous work, e.g. PIER[6], whih onstruts routing tree for eh ttriute olletion. While this sheme provides the most lned lod mong trees, it is not effiient s nodes pys high ost in per-messge overhed. One-Set Prtition (OP). The one-set prtition sheme uses the set A s the only prtitioned set. This sheme is lso used in numer of previous work[12], [11]. Using OP, eh node n send just one messge whih inludes ll the ttriute vlues, nd thus, sves per-messge overhed. Nevertheless, sine the size of eh messge is lrger ompred with messges ssoited with SP, the orresponding olleting tree n not grow very lrge Exploring Prtition Augmenttions seeks middle ground etween these extreme solutions - one where nodes py lower per-messge over ompred to SP while eing more lod-lned nd onsequently more slle thn OP. Our prtition ugmenttion sheme explores possile ugmenttions to given ttriute prtition P y serhing for ll prtitions tht re lose to P in the sense tht the resulting prtition n e reted y modifying P with ertin predefined opertions. We first define two si opertions for modifying ttriute set prtitions. Given two ttriute sets A P i nd A P j in prtition P,merge opertion over A P i nd A P j, denoted s A P i A P j, yields new set AP k = AP i A P j. Given one ttriute set A P i nd n ttriute α, split opertion on A P i with regrding to α, denoted s A P i α, yields two new sets A P k = AP i α. Bsed on the definition of merge nd split opertions, we now define neighoring solution s follows: Definition 2: For n ttriute set prtition P,wesyprtition P is neighoring solution of P if nd only if either A P i,ap j P so tht P = P/{A P i,ap j } (AP i A P j ),or A P i P, α A P i so tht P = P/A P i (A P i α) {α}.

5 Guided Prtition Augmenttion. Exploring nd evlute ll neighoring ugmenttions of given prtition is prtilly infesile, sine the evlution involves onstruting resoure-onstrined monitoring trees. To mitigte this prolem, we use guided prtition ugmenttion sheme whih gretly redues serhing spe. The si ide is to rnk ndidte prtitions ording to the estimted ost redution tht would result from using the new prtition, s prtition tht provides lrge ost redution will likely free up more pity for ggregting ttriute vlue pirs. Following this, we evlute neighoring prtitions in the deresed order of their estimted pity-redution so tht we n find good ugmenttion without evluting ll ndidtes. To estimte the gin of ndidte ugmenttion, we first need to understnd how the resoure onsumption would hnge fter pplying n ugmenttion m. Chnge in the totl resoure onsumption resulting from n ugmenttion m n e ontriuted y the hnge in the rely ost nd tht in the per-messge overhed ost, s m my hnge the numer of trees nd the struture of trees. Let g(m) e the overll redution in resoure onsumption of n ugmenttion m, Δ p (m) e the estimted differene in overhed ost due to m nd Δ r (m) e the estimted differene in rely ost due to m. Wehveg(m) =Δ p (m)+δ r (m). We estimte g(m) ssuming tht following n ugmenttion, the tree onstrution proedure is le to ssign ll the ttriute-vlue pirs tht were in the ffeted prtitions using str topology. Assuming topology is neessry to e le to estimte Δ r (m). Rell tht C is the per-messge overhed nd is the ost of messge of unit size. Also let N Ai denote the numer of nodes ssoited with ttriute set A i.wethenhve: { ( 1).C. NAi N Aj m : A i A j = A k Δ p(m) = C. N Aj N Ak m : A i A j = A k {. NAi A Δ r(m) = j N Ai A j m : A i A j = A k ( 1).. N Ai N Ai A j m : A i A j = A k Intuitively, when we merge two ttriute sets, the permessge overhed redues s nodes ssoited with oth sets send fewer messges for n updte. However, the orresponding relying ost my inrese s the merged tree my e higher thn the previous two trees, whih mkes messges trvel more hops to reh the root. On the ontrry, when we split n ttriute set, the per-messge overhed inreses nd the relying ost dereses. The ove equtions pture these two hnges nd mke the estimtion possile. This guided lol-serh heuristi is essentil to ensure the prtility of our sheme Resoure-wre Evlution To evlute the ojetive funtion for given ndidte prtition ugmenttion m, the resoure-wre evlution exmines m y onstruting trees for nodes ffeted y m nd mesures the the numer of node-ttriute pirs tht n e olleted using these trees. This proedure primrily involves two tsks. One is onstruting tree for given set of nodes without exeeding resoure onstrints t ny node. The other is for node onneted to multiple trees to llote its resoures to different trees Tree Constrution The tree onstrution proedure onstruts olletion tree for given set of nodes D suh tht no node exeeds its resoure onstrints while trying to inlude s mny nodes s possile into the onstruted tree. Formlly, we define the tree onstrution prolem s follows: Prolem Sttement 2: Given set of n nodes, eh hs x i ttriutes to monitor, nd resoure onstrint i, find prent p(i), i, so tht the numer of nodes in the onstruted tree is mximized sujet to the following onstrints where u i is the resoure onsumed t node i for sending updte messges to its prent: 1) For ny node i in the tree, p(j)=i u j + u i i 2) Let y i e the numer of ll ttriute vlues trnsmitted y node i. Wehvey i = x i + p(j)=i x j. 3) Aording to our definition, u i C + y i The first onstrint requires tht the resoure spent on node i for sending nd reeiving updtes should not exeed its resoure onstrint 1 i. The seond onstrint requires node to deliver its lol monitored vlues s well s vlues reeived from its hildren. The lst onstrint sttes tht the ost of proessing n outgoing messge is the omintion of per-messge overhed nd vlue proessing ost. The tree onstrution prolem, however, is lso NP-Complete[13] nd we present heuristis for the tree-onstrution prolem. We first disuss two simple tree onstrution heuristis: Str. This sheme forms str -like trees y giving priority to inresing the redth of the tree. Speifilly, it dds nodes into the onstruted tree in the order of deresed ville pity, nd tthes new node to the node with the lowest height nd suffiient ville pity, until no suh nodes exist. STAR retes ushy trees nd onsequently pys low rely ost. However, due to lrge node degrees, the root suffers from high per-messge overhed, nd thus, the tree n not grow very lrge. Chin. This sheme gives priority to inresing the height of the tree, nd onstruts hin -like trees. CHAIN dds nodes to the tree in the sme wy s STAR does exept tht it tries to tth nodes to the node with the highest height nd suffiient ville pity. CHAIN retes long trees tht hieve good lod lne, ut due to the numer of hops eh messge hs to trvel to reh the root, most nodes py high rely ost. STAR nd CHAIN revel two onfliting ftors in tree onstrution effiieny nd slility. Minimizing tree height hieves effiieny, i.e. minimum rely ost, ut uses poor slility, i.e. smll tree size. On the other hnd, mximizing tree height hieves good slility, ut degrdes effiieny. The dptive tree onstrution sheme seeks middle-ground etween the STAR nd CHAIN proedures. It tries to minimize the totl resoure onsumption, nd n trde off per-messge overhed for rely ost, nd vie vers, if it ommodtes more nodes y doing so. The dptive tree onstrution sheme frequently uses the following two onepts: Given set of nodes N for tree 1. In our setting, proessing n outgoing messge onsumes the sme mount of resoure s proessing the sme messge s n inoming one

6 onstrution nd the orresponding tree T whih ontins nodes set N N, wesyt is sturted if no more nodes d (N N ) n e dded to T without using violtion of resoure onstrint for t lest one node in T. We refer to nodes whose resoure onstrint would e violted if d (N N ) is dded to T s ongested nodes. The dptive tree onstrution sheme itertively invokes two proedures, the onstrution proedure nd the djusting proedure. The onstrution proedure runs the STAR sheme whih tthes new nodes to low level existing tree nodes. STAR uses the pity onsumption t low level nodes to e muh hevier thn tht t other nodes. Thus, s low level tree nodes eome ongested ( sturted tree), the onstrution proedure termintes nd returns ll ongested nodes. The sheme then invokes the djusting proedure, whih tries to relieve the worklod of low level nodes y reduing the degree of these nodes nd inresing the height of the tree(similr to CHAIN). As result, the djusting proedure redues ongested nodes nd mkes sturted tree unsturted. The sheme then repets the onstrutingdjusting itertion until no more nodes n e dded to the tree. For detils of the djusting proedure nd resoure llotion for nodes prtiipted in multiple trees, we refer interested reders to our tehnil report[14]. 4. Experimentl Evlution We undertke n experimentl study of nd present results inluding those gthered y deploying on BlueGene/P rk running IBM s lrge-sle distriuted streming system - System S. We first hrterize our prtition ugmenttion nd tree onstrution shemes using syntheti dt, nd then vlidte the performne of our system through rel system experiments. Syntheti Dtset Experiments. For our experiments on syntheti dt, we ssign rndom suset of ttriutes to eh node in the system. We generte monitoring tsks y rndomly seleting A t ttriutes nd N t nodes with uniform distriution, for given size of ttriute set A nd node set N. We lso lssify monitoring tsks into two tegories - 1) smll-sle ones tht re for smll set of ttriutes from smll set of nodes, nd 2) lrge-sle ones tht either involves mny nodes or mny ttriutes. Rel System Experiments. Through experiments in rel system deployment, we lso show tht the error in ttriute vlue oservtions (due to either stle or dropped ttriute vlues used y insuffiient node resoures) introdued y is smll. Note tht this error n e mesured in meningful wy only for rel system. System S is lrge-sle distriuted strem proessing middlewre. Applitions re expressed s dtflow grphs tht speify nlyti opertors interonneted y dt strems. These pplitions re deployed in System S s proesses exeuting on distriuted set of hosts, nd interonneted y strem onnetions using trnsports suh s TCP/IP. Eh node tht runs pplition proesses n oserve ttriutes t vrious levels suh s t the nlyti opertor level, System S middlewre level nd the OS Perentge of Colleted Vlues Perentge of Colleted Vlues 1 1 Averge Numer of Attriutes Per Tsk () Inresing A t Numer of Tsks () Inresing Smll-sle Tsks Perentge of Colleted Vlues Perentge of Colleted Vlues Averge Numer of Nodes Per Tsk () Inresing N t Numer of Tsks (d) Inresing Lrge-sle Tsks Figure 4: Comprison of Attriute Set Prtition Shemes under Diff. Worklod Chrteristis level. For these experiments, we deployed one suh System S pplition lled YieldMonitor [15], tht monitors hip mnufturing test proess nd uses sttistil strem proessing to predit the yield of individul hips ross different eletril tests. This pplition onsisted of over proesses deployed ross nodes, with - ttriutes to e monitored on eh node, on BlueGene/P luster. The BlueGene is very ommunition rih nd ll ompute nodes re interonneted y 3D Torus mesh network. Consequently, for ll prtil purposes, we hve fully onneted network where ll pirs of nodes n ommunite with eh other t lmost equl ost. We generte worklod for these experiments using the sme proedure tht we used for the syntheti dtset experiments Result Anlysis We present smll suset of our experimentl results to highlight the following oservtions mongst others: n ollet lrger frtion of node-ttriute pirs to server tsks presented to the system ompred to simple heuristis (whih re essentilly the stte-ofthe-rt). dpts to tsk nd system hrteristis, nd outperforms eh of these simple heuristis. In rel pplition senrio, lso signifintly redues perentge error in the oserved vlues of the node-ttriute pirs required y monitoring tsks when ompred to simple heuristis. Vrying the sle nd numer of monitoring tsks. Figure 4 ompres the performne of different ttriute set prtition shemes under different worklod hrteristis. In Figure 4(), where we inrese the numer of ttriutes A t in monitoring tsks, our prtition ugmenttion sheme() performs onsistently etter thn singletonset() nd one-set() shemes. In ddition, outperforms when A t is reltively smll. As eh node sends only one

7 Perentge of Colleted Vlues Numer of Nodes Perentge of Colleted Vlues Numer of Nodes Perentge of Colleted Vlues STAR CHAIN MAX_AVB ADAPTIVE 1 1 Averge Numer of Attriutes Per Tsk Perentge of Colleted Vlues STAR CHAIN MAX_AVB ADAPTIVE Numer of Tsks () Inresing Nodes(Smll-sle) () Inresing Nodes(Lrge-sle) () Inresing A t () Inresing Smll-sle Tsks Perentge of Colleted Vlues C/ () Inresing C/(Smll-sle) Perentge of Colleted Vlues C/ (d) Inresing C/(Lrge-sle) Figure 5: Comprison of Attriute Set Prtition Shemes under Diff. System Chrteristis messge whih inludes ll its own ttriutes nd those reeived from its hildren, uses the minimum per-messge overhed. However, when A t inreses, the pity demnd of low level nodes, i.e. nodes lose to the root, inreses signifintly, whih limits the size of the tree nd uses poor performne. In Figure 4(), where we set A t = nd inrese N t to rete extremely hevy worklods, grdully onverges to SINGLETON- SET, s provides the est lod lne whih results in the est performne in this se. We oserve similr results in Figure 4() nd 4(d), where we inrese the totl numer of smll-sle nd lrge-sle monitoring tsks respetively. Vrying nodes in the system. Figure 5 shows the performne of different ttriute set prtition shemes under different system hrteristis. In Figure 5() nd 5(), where we inrese the numer of nodes in the system given smll nd lrge sle monitoring tsks respetively, we n see is etter for lrge-sle tsks while is etter for smll-sle tsks, nd performs muh etter thn them in oth ses, round % extr olleted node-ttriute pirs. Vrying per-messge proessing overhed. To study the impt of per-messge overhed, we vry the C/ rtio under oth smll nd lrge sle monitoring tsks in Figure 5() nd 5(d). As expeted, inresed per-messge overhed hits the sheme hrd sine it onstruts lrge numer of trees nd, onsequently, inurs the lrgest overhed ost while the performne of the ONE- SET sheme whih onstruts just single tree degrdes more grefully. However, hving single tree is not the est solution s shown y whih outperforms oth shemes s C/ inreses. Comprison of tree-onstrution shemes. In Figure 6, we study the performne of different tree onstrution shemes under different worklods nd system hr- Figure 6: Comprison of Tree Constrution Shemes under Different Worklod nd System Chrteristis teristis. Our omprison lso inludes new sheme, nmely MAX AVB, heuristi sheme used in TMON[13] whih tthes new nodes to the existing node with the most ville pity. While we vry different worklods nd system hrteristis in the four figures, our dptive tree onstrution sheme(adaptive) lwys performs the est. Among ll the other tree onstrution shemes, STAR performs well when worklod is hevy, s suggested y Figure 6() nd 6(). This is euse STAR uilds trees with minimum height, nd thus, voids pying onsiderle ost for relying given hevy worklods. CHAIN performs the worst in lmost ll ses. While CHAIN provides good lod lne y distriuting per-messge overhed in CHAIN-like trees, nodes hve to py high ost for relying, espeilly when worklods re hevy. MAX AVB sheme outperforms oth STAR nd CHAIN given smll worklod, s it voids over strething tree in redth or height y growing trees from nodes with the most ville pity. However, its performne quikly degrdes with inresing worklod due to relying ost. Averge Perentge Error Numer of Nodes () Comprison etween Prtition Shemes Averge Perentge Error Numer of Tsks () Comprison etween Tree Constrution Shemes Figure 7: Comprison of Shemes on Averge Perentge Error Rel-world performne. To evlute the performne of in rel world pplition. we mesure the verge perentge error of reeived ttriute vlues for synthetilly generted monitoring tsks. Speifilly, we mesure verge perentge error etween the snpshot of vlues oserved y our sheme nd ompre it to the snpshot of tul vlues (tht n e otined y omining lol log files t the end of the experiment). Figures 7() ompres the hieved perentge error etween different prtition shemes given inresing nodes. The figure shows tht our prtition ugmenttion sheme in outperforms the other prtition shemes. The perentge error hieved y is round %-% less thn tht hieved y

8 nd. Interestingly, the perentge error hieved y lerly redues when the numer of nodes in the system inreses. The reson is tht s nodes inreses, monitoring tsks re more sprsely distriuted mong nodes. Thus, eh messge is reltively smller nd eh node n hve more hildren, whih redues the perentge error used y lteny. Similrly, gins signifint error redution ompred with the other two shemes in Figure 7() where we ompre tree onstrution shemes under inresing monitoring tsks. 5. Relted Work Muh erly work on distriuted query systems minly fouses on exeuting single queries effiiently. As our work study multiple queries, we omit disussing these work. In ddition, reserh on proessing multiple queries on entrlized dt strem [16], [17], [18], [19] is not diretly relted with our work either, s we study distriuted streming where inter-node ommunition is of onern. A lrge ody of work studies query optimiztion nd proessing for distriuted dtses (see [] for survey). Our prolem is fundmentlly different s nodes in our prolem re pity onstrined. There re lso muh work on multiquery optimiztion for ontinuous ggregtion queries over physilly distriuted dt strems [19], [21], [22], [23], [24], [25], [16], [17], [18]. These shemes ssume routing trees re provided s prt of the input, whih my not e true for ertin senrios. is le to hoose optiml routing trees from mny possile ones sed on multi-tsk optimiztion nd resoure onstrints. Reently, severl work studies effiient dt olletion mehnisms. CONCH [26] uilds spnning forest with miniml monitoring osts for ontinuously olleting redings from sensor network y utilizing temporl nd sptil suppression. However, it does not onsider node level resoure limittion nd per-messge overhed s we did, whih my limit its ppliility. PIER [6] suggests using distint routing trees for eh query in the system, whih is essentilly the sheme we disussed. This sheme, though hieves good lod lne, my use signifint ost on per-messge overhed. 6. Conlusion nd Future Work We hve presented, resoure-wre multi-tsk optimiztion frmework for sling pplition stte monitoring in lrge sle distriuted systems. Our pproh optimizes monitoring y lning ommunition effiieny nd slility of the monitoring network. The unique ontriution of is the tehniques for generting monitoring forests tht optimize multiple tsks nd, t the sme time, lnes node level resoure onsumption. We evlute through extensive experiments inluding deploying in rel-world strem proessing pplition hosted on BlueGene/P. The results show tht signifintly nd onsistently outperforms existing pprohes. The development ontinues long severl dimensions. First we pln on implementing extensions suh s in-network ggregtion nd lternte routing. Seond, we re onsidering extending our frmework to hndle oth short-term nd long-term monitoring tsks. Furthermore, we re interested in studying the runtime dpttion ost of trnsforming from one monitoring network to nother. Aknowledgment The first nd lst uthors re prtilly supported y grnts from NSF CISE CyerTrust progrm, IBM fulty prtnership wrd, nd n IBM SUR grnt. Referenes [1] B. Hyes, Cloud omputing, Commun. ACM 8, vol. 51. [2] N. Jin nd et l., Design, implementtion, nd evlution of the liner rod enhmrk on the strem proessing ore, in SIGMOD Conferene, 6, pp [3] D. Adi nd et l, The design of the orelis strem proessing engine, in Proeedings of CIDR, 5. [4] S. Mdden nd et l., Tg: A tiny ggregtion servie for d-ho sensor networks, in OSDI, 2. [5] P. Ylgndul nd M. Dhlin, A slle distriuted informtion mngement system, in SIGCOMM4, pp [6] R. H. et l., The rhiteture of pier: n internet-sle query proessor, in CIDR, 5, pp [7] G. Cormode, M. N. Groflkis, S. Muthukrishnn, nd R. Rstogi, Holisti ggregtes in networked world: Distriuted trking of pproximte quntiles, in SIGMOD5. [8] M. Greenwld nd S. Khnn, Power-onserving omputtion of order-sttistis over sensor networks, in PODS4. [9] G. Cormode nd et l., Skething strems through the net: Distriuted pproximte query trking, in VLDB5. [] D. J. Adi nd et l., Reed: Roust, effiient filtering nd event detetion in sensor networks, in VLDB, 5. [11] U. Srivstv, K. Mungl, nd J. Widom, Opertor plement for in-network strem query proessing, in PODS5. [12] A. Deliginnkis nd et l., Hierrhil in-network dt ggregtion with qulity gurntees, in EDBT4. [13] S. R. Kshyp, D. Turg, nd C. Venktrmni, Effiient trees for ontinuous monitoring, 8. [14] S. Meng, S. R. Kshyp, C. Venktrmni, nd L. Liu, Resoure-wre pplition stte monitoring for lrge-sle distriuted systems, Tehnil Report, 8. [15] D. S. Turg nd et l., Rel-time monitoring nd preditive nlysis of hip mnufturing dt, 8. [16] R. Z. 3, N. Kouds, B. C. Ooi, nd D. Srivstv, Multiple ggregtions over dt strems, in SIGMOD5. [17] S. Krishnmurthy, C. Wu, nd M. J. Frnklin, On-the-fly shring for stremed ggregtion, in SIGMOD6. [18] J. Li nd et l., Effiient evlution of sliding-window ggregtes over dt strems, SIGMOD Reord 5. [19] S. Mdden nd et l., Continuously dptive ontinuous queries over strems, in SIGMOD, 2, pp. 49. [] D. Kossmnn, The stte of the rt in distriuted query proessing, ACM Comput. Surv., vol. 32, no. 4,. [21] R. Huesh nd et l., Shring ggregte omputtion for distriuted queries, in SIGMOD, 7, pp [22] R. Gupt nd K. Rmmrithm, Optimized query plnning of ontinuous ggregtion queries in dynmi dt dissemintion networks, in WWW7. New York, NY, USA: ACM. [23] N. Trigoni nd et l., Multi-query optimiztion for sensor networks, in DCOSS, 5, pp [24] A. Silerstein nd J. Yng, Mny-to-mny ggregtion for sensor networks, in ICDE, 7, pp [25] S. Xing, H.-B. Lim, K.-L. Tn, nd Y. Zhou, Two-tier multiple query optimiztion for sensor networks, in ICDCS7. [26] A. Silerstein, R. Brynrd, nd J. Yng, Constrint hining: on energy-effiient ontinuous monitoring in sensor networks, in SIGMOD Conferene, 6, pp