Computer Networks 56 (2012) Contents lists available at SciVerse ScienceDirect. Computer Networks

Transcription

1 Computer etworks 56 (01) Cotets lists available at SciVerse ScieceDirect Computer etworks joural homepage: DPillar: Dual-port server itercoectio etwork for large scale data ceters q Yog Liao a, Jiagtao Yi a, Dog Yi b, Lii Gao a, a Departmet of Electrical ad Computer Egieerig, Uiversit of Massachusetts at Amherst, Amherst, MA 01003, Uited States b Automatio Departmet, orthwester Poltechical Uiversit, Xi a, ShaXi 71007, Chia article ifo abstract Article histor: Received Ma 011 Received i revised form 5 Februar 01 Accepted 4 Februar 01 Available olie 6 March 01 Kewords: Data ceter etwork Multi-path routig etwork topolog To meet the huge demads of computatio power ad storage space, a future data ceter ma have to iclude up to millios of servers. The covetioal hierarchical tree-based data ceter etwork architecture faces several challeges i scalig a data ceter to that size. Previous research effort has show that a server-cetric architecture, where servers are ot ol computatio ad storage workstatios but also itermediate odes relaig traffic for other servers, performs well i scalig a data ceter to a huge umber of servers. This paper presets a server-cetric data ceter etwork called DPillar, whose topolog is ispired b the classic butterfl etwork. DPillar provides several ice properties ad achieves the balace betwee topological scalabilit, etwork performace, ad cost efficiec, which make it suitable for buildig large scale future data ceters. Usig ol commodit hardware, a DPillar etwork ca easil accommodate millios of servers. The structure of a DPillar etwork is smmetric so that a etwork bottleeck is elimiated at the architectural level. With each server havig ol two ports, DPillar is able to provide the badwidth to support commuicatio itesive distributed applicatios. This paper studies the itercoectio features of DPillar, how to compute routes i DPillar, ad how to forward packets i DPillar. Etesive simulatio eperimets have bee performed to evaluate the performace of DPillar. The results show that DPillar performs well eve i the presece of a large umber of server ad switch failures. Ó 01 Elsevier B.V. All rights reserved. 1. Itroductio The prevalece of cloud computig is drivig the deplomet of data ceters to host various etwork applicatios ad services [1,3,3]. I order to better support computatio ad storage demadig large scale distributed applicatios, as well as provide the multi-teac abilit for efficiet resource utilizatio ad elastic resource allocatio, data ceters have to accommodate a large umber of itercoected servers. A tpical data ceter toda q Part of this work has bee published i Proceedigs of ICCC 010: the 19th Iteratioal Coferece o Computer Commuicatios ad etworks [1]. Correspodig author. Tel.: ; fa: addresses: [email protected] (Y. Liao), [email protected] (J. Yi), [email protected] (D. Yi), [email protected] (L. Gao). has thousads of servers; a data ceter i the future ca have hudreds of thousads or eve millios of servers [17,15]. Although the availabilit of iepesive commodit PCs has made it possible to epad a data ceter to a huge umber of servers, efficietl itercoectig the servers is still a challegig task. The itercoectio etwork has to provide high badwidth to facilitate distributed applicatios requirig frequet data accessig ad shufflig [11,9,13]. It is also ofte desired to use cost efficiet commodit hardware i order to meet the budget costraits. The itercoectio etwork should provide high availabilit as well because more ad more eterprise-class missio critical applicatios are migratig ito data ceters. The covetioal wa of coectig multiple levels of switches ito a tree ad attachig servers as leaves of the tree [4] faces difficulties i scalig a data ceter to more tha a few thousads servers [5]. The high /$ - see frot matter Ó 01 Elsevier B.V. All rights reserved. doi: /j.comet

2 Y. Liao et al. / Computer etworks 56 (01) speed core switches close to the tree s root will soo become the cost ad performace bottleeck as the umber of servers grows. Researchers are activel searchig for ew etwork architectures to build cost efficiet ad high performace data ceters. The VL etwork [15] uses a tree-based switch fabric to coect servers ad provides a desig with better agilit propert tha covetioal data ceter etworks. The fat-tree etwork [5,] is also a tree structure etwork. Istead of havig epesive high-ed switches at the tree s root, fat-tree uses idetical switches at all levels of the tree. The umber of servers coected b a fattree is determied b the umber of ports o each switch. To scale a data ceter without relig o switches with more ports, a secod thread of research work proposes movig the etworkig itelligece from switches to servers, i.e., each server ca forward traffic for other servers. Such a data ceter etwork is ofte referred to as a server-cetric etwork. Besides the abilit to easil scale the etwork size, a server-cetric etwork offers other advatages i buildig large scale data ceters. As the etworkig itelligece is moved to servers, switches ca be simple laer- plug-ad-pla devices. Usig dumm low-ed switches ca greatl reduce the cost of buildig a data ceter. Secodl, as servers are easier to program tha switches, it is more coveiet to develop ad deplo routig ad maagemet mechaisms i server-cetric data ceter etworks. The DCell etwork [17] uses small switches to coect multi-port servers ito cells ad scales the etwork b recursivel coectig smaller cells ito larger cells. The etwork topolog of DCell is ot smmetric. Some liks are more likel to be saturated ad failures of those liks are of greater impact to the etwork performace. The FiCo etwork [0] adopts similar idea as DCell to recursivel epad the etwork size. Each server i FiCo eeds to have ol two ports. Because most off-the-shelf servers alread itegrate two ports, oe primar ad oe backup, FiCo ca use commodit servers i a as-is maer. FiCo is ot a smmetric structure either ad it has similar issues as DCell, e.g., some liks will be more loaded tha others. The BCube etwork [16] has smmetric etwork topolog ad it performs ver well i terms of etwork badwidth. I order to scale the etwork size, each server i BCube eeds to have more ports. I this paper, we propose a ew server-cetric data ceter etwork architecture called DPillar. DPillar uses ol commodit hardware ad ca easil scale to a huge umber of servers. o matter how ma servers eist i a DPillar etwork, the umber of ports i each server is alwas fied. More specificall, with each server havig ol two ports, we ca build a DPillar etwork to accommodate a umber of servers. I cotrast, both DCell ad BCube require additioal ports from servers i order to coect more servers ito the etworks, which could be a practical issue because off-the-shelf commodit servers alwas have a fied umber of ports. DPillar has a smmetric structure ad therefore it elimiates a etwork bottleeck at the architectural level. The smmetric structure of DPillar facilitates the developmet of high performace routig mechaisms. I DPillar, servers forwardig traffic does ot icur much overhead to them because there is o routig table lookup i packet forwardig. Istead, a server computes the et hop i O(1) time based o its ow address ad destiatio address of the packet beig forwarded. Eve though each server has ol two ports, a DPillar etwork offers rich coectios betwee servers, which we have leveraged i desigig a efficiet multi-path routig scheme. This scheme produces multiple paths betwee a source destiatio pair i a DPillar etwork. The disjoitess of the ielded paths is formall proved i this paper. We also provide a desig to utilize the disjoit paths to tolerate failures ad balace load i a DPillar etwork. DPillar is closel related to the classic wrapped butterfl etwork [19]. The desig of our routig schemes is ispired b previous work o thewrapped butterfl etwork as well. It is worth to highlight here a few essetial differeces betwee DPillar ad thewrapped butterfl etwork. DPillar requires ol two ports for each server but a ode i a wrapped butterfl etwork eeds to have four ports. More importatl, because of usig switches istead of direct server-to-server liks, a DPillar etwork has much smaller etwork diameter tha a wrapped butterfl etwork of the same size. A DPillar etwork also provides much better badwidth tha a wrapped butterfl etwork of the same size. More details o comparig DPillar to a wrapped butterfl etwork will be preseted i Sectio.6. I summar, i this paper we make the followig techical cotributios i data ceter etwork desig: (1) We propose a ew server-cetric data ceter etwork architecture with smmetric topological structure. A DPillar etwork ca scale to a huge umber of servers with each server havig a fi umber of ports. () We provide a comprehesive stud of DPillar s itercoectio properties. (3) We desig a simple et high performace multi-path routig scheme, ad have provable path disjoitess result applicable to a large set of related etwork topologies. The rest of this paper is orgaized as follows. Sectio presets the etwork structure of DPillar i detail ad studies its topological properties. Sectios 3 ad 4 are devoted to the discussio of routig ad packet forwardig i DPillar etworks. Sectio 5 presets performace evaluatio results. The backgroud of itercoectio etworks ad a discussio of related work o data ceter etworks are preseted i Sectio 6. Sectio 7 cocludes this paper.. DPillar itercoectio This sectio presets the itercoectio of DPillar. We first preset how servers i DPillar are addressed ad coected. The we stud the mathematical priciples behid DPillar s structure ad its topological properties, which serve as the foudatio of desigig packet routig ad forwardig mechaisms for DPillar. A compariso betwee DPillar ad the closed related wrapped butterfl etwork [19] is also preseted i this sectio..1. Logical represetatio of DPillar etwork A DPillar etwork is built from two kids of devices, dual-port servers ad -port switches. The servers ad

3 134 Y. Liao et al. / Computer etworks 56 (01) switches are logicall arraged ito k equal-size server colums ad k equal-size switch colums. Each server colum k k1 has servers ad each switch colum has switches. The server colums ad switch colums are umbered as H 0 H k1 ad S 0 S k1, respectivel. The server colums ad switch colums are alteratel placed ito a logical circle, as show i Fig. 1. Visuall, the k colums of servers ad switches are attached to the clidrical surface of a pillar. For ease of epositio, i the rest of this paper we call server colum H (i+1)%k a clockwise eighborig colum of H i ad H (i+k1)%k a couter-clockwise eighborig colum of H i. As we ca see, a DPillar etwork is uiquel defied b two parameters,, the umber of ports i each switch, ad k, the umber of server colums. We call such a DPillar etwork a (,k) DPillar etwork... Addressig servers i DPillar Fig. 1. A vertical view of the pillar. The blocks represet switch colums ad the circles represet server colums. The server colums ad switch colums are alteratel placed ito a large circle. It looks like the are attached to the clidrical surface of a pillar. For the k servers i a server colum H i i a (,k) DPillar etwork, each of them is assiged with a k-smbol label (m k1...m 0 ), where each smbol m i is a iteger umber betwee 0 ad 1. Uder this labelig scheme, oe server i DPillar ca be uiquel idetified as (C,m k1... m 0 ), meaig a server with label (m k1...m 0 ) at server colum H C. We call (C,m k1...m 0 ) the address of the server. Fig. shows a (8,) DPillar etwork i a two-dimesioal view. There are two server colums (H 0 ad H 1 ) ad two switch colums (S 0 ad S 1 ). The label of a server has two digits, ad each digit ca be of value [0,3]. For Fig.. Two-dimesioal view of a (8,) DPillar etwork. The umber i each circle is the label of that server. ote that server colum H 0 is duplicated i this figure to better show the coectio betwee server colum H 0 ad switch colum S 1.

4 Y. Liao et al. / Computer etworks 56 (01) istace, the top left circle is a server at colum H 0 ad its label is (00), therefore its address is (0,00); the bottom left circle is a server of address (0,33)..3. Coectig servers via switches Let us cosider server colums H i ad H (i+1)%k. There are totall k servers i those two colums. Uder the addressig scheme described i Sectio., those k servers ca be divided ito k1 equal-size groups so that for those servers i the same group, their labels are the same ecept the ith smbol (i.e., smbol m i ). The servers i the same group are coected to the same switch i switch colum S i. It is eas to see that amog the servers i a group, half of them are i server colum H i ad the other half are i H (i+1)%k. For eample, Fig. shows the coectio of servers ad switches i a (8,) DPillar etwork. For ease of illustratio, we duplicate server colum H 0 ad plot the clidrical surface of the logical pillar as a two-dimesioal view. I this eample, each server colum has 8 ¼ 16 servers ad the label of each server has two smbols. The address of each server ca be represeted as (C,m 1 m 0 ). The first switch i S 1 is coected with eight servers, icludig four servers i H 1 ad four servers i H 0. The labels of those servers are (00), (10), (0), ad (30), respectivel. That is, their labels are of form (m 1 0), with 0 6 m ote that those labels are the same if m 1 is removed. Similarl, the labels of those eight servers coected to the first switch at S 0 are (00), (01), (0), ad (03). The are of form (0m 0 ) 0 6 m 0 6 3, ad the are the same if m 0 is removed. The rule of coectig servers via switches ca be summarized as followig. For a label (m k1...m i...m 0 ), there are servers i H i whose labels are ðm k1...m i...m0 Þ, where 0 6 m i 6 1; there are such servers i H (i+1)%k too. Those servers are coected to the same -port switch i switch colum S i. Give the wa of how servers ad switches are coected, the followig Propositio.1 is obvious. Propositio.1. For two servers i a server colum H i, the are coected to the same switch if their labels differ at the ith smbol ol (i.e., smbol m i ) or differ at the ((i 1)%k)th smbol ol (i.e., smbol m (i1)%k )..4. Topological properties For buildig large scale data ceters, the itercoectio etwork must accommodate a huge umber of servers. The etwork should also provide sufficiet badwidth to support traffic itesive applicatios ruig i a data ceter. DPillar s topological structure does ot impose a limit o the umber of servers coected ito the etwork. Hece, it is possible to scale a DPillar etwork ito a huge umber of servers. DPillar s topological structure also has good bisectio width because there are rich coectios amog servers umber of servers For a (, k) DPillar etwork, sice each server colum has k servers, there are k k servers i total. I the rest of this paper, we use to represet the total umber of servers i a (,k) DPillar etwork ad we have Propositio.. Propositio.. A (,k) DPillar etwork has ¼ k k servers. Because the total umber of servers,, grows epoetiall as the umber of server colums, k, grows, DPillar structure scales well i terms of accommodatig a large umber of servers. Here we provide some eamples o how ma servers a (, k) DPillar etwork ca support. Cosiderig that 48-port Gbit Etheret switches are widel available ow ad relativel iepesive, we assume 48- port switches are used i buildig DPillar etworks. A (48, 3) DPillar etwork has 41,47 servers. The umber of servers ca be about 1.3 millio i a (48,4) DPillar etwork. If we build a (48,5) DPillar etwork, it has aroud 40 millio servers..4.. Bisectio width Bisectio width is a importat factor to quatif a itercoectio etwork s badwidth. It is defied as the smallest umber of edges removal of which divides the odes i the etwork ito two parts of equal size. Larger bisectio width meas that the etwork ca sustai more commuicatios betwee odes i the etwork. Because servers i a data ceter usuall have lots of iteractios amog them whe ruig distributed applicatios, it is desirable for a data ceter etwork to have large bisectio width. We use the eample show i Fig. to stud how to cut a DPillar etwork ito two halves. I Fig., the servers i this (8,) DPillar etwork ca be divided ito a top half ad a bottom half so that for all servers i the top half, the m 1 smbol i their labels satisfies 0 6 m 1 6 1; for servers i the bottom half, the m 1 smbol i their labels satisfies 6 m This scheme cuts four liks for each switch i S 1 ad the size of the cut is 16. I geeral, we ca cut a (,k) DPillar etwork ito a top half ad a bottom half so that for all servers i the top half, the m k1 smbol i their labels satisfies 0 6 m k1 6 1; for servers i the bottom 4 half, the m k1 smbol i their labels satisfies 6 4 mk This scheme cuts liks for each -port switch i switch colum S k1. As there are k1 switches i S k1, the size of the cut is k. Therefore, the upper boud of the bisectio badwidth of a (, k) DPillar etwork is k k. We ca prove that is also the lower boud. This is stated as Propositio.3. The proof of Propositio.3 is preseted i Appedi A. Propositio.3. The bisectio width of a (,k) DPillar k. etwork is.5. Cost efficiec DPillar etwork is cost-efficiet as it uses commodit hardware. Here we provide some budget eamples of buildig DPillar etworks. We igore the cost of servers ad focus o the etworkig devices, icludig switches

5 136 Y. Liao et al. / Computer etworks 56 (01) Table 1 Eample budget for etworkig cost whe buildig a DPillar etwork with four colums of servers. Switch tpe 8-Port 16-Port 4-Port 48-Port Switch uit price ($) umber of servers ,384 8,944 1,37,104 Total etworkig 14, ,968 1,410,048 35,831,808 cost ($) Per server cost ($) ad Etheret cables. As most off-the-shelf servers alread itegrate dual-port etwork iterfaces, there is o eed to ivest o etwork iterfaces for servers. A (,k) DPillar etwork has k switch colums ad each oe icludes k1 -port switches. The total umber of switches is k k1. There are k k servers ad each oe has two ports, so the total umber of cables used i a (,k) DPillar etwork is k k. Let U s be the uit price of a -port switch ad U c be the uit price of a Etheret cable, the total etworkig cost of a (,k) DPillar etwork is U s k k1 þ U c k k. The average cost of coectig oe server i a DPillar etwork is (U s / + U c ). 1 The uit prices we get from a olie retailig store are $150 for a 16-port Gbit Etheret switch ad $1 for a Etheret cable. We epect the wholesale price of the switches ad cables would be eve lower. To build a (16, 4) DPillar etwork, the cost of all switches is $307,00. The cost of cables is $3,768, as we eed two cables for each server. The total cost of etworkig devices is about 0.34 millio USD. O average it costs about $0 to coect oe server. Table 1 shows the total cost ad the per server cost of buildig DPillar etworks with four colums of servers, whe differet tpes of switches are used..6. Cotrastig DPillar with wrapped butterfl etwork DPillar is closel related to the wrapped butterfl etwork [19], which is oe of the multi-stage itercoectio etworks studied before. The colum i DPillar is equivalet to the stage i the wrapped butterfl etwork. DPillar ca be viewed as a etesio of the wrapped butterfl etwork, but DPillar has a set of uique properties which make it more suitable i buildig large scale data ceters. Here we highlight the importat differeces betwee DPillar ad the wrapped butterfl etwork umber of ports i servers I order to coect servers via a wrapped butterfl etwork, the servers must have four ports. Because most commodit servers ad servers i eistig data ceters itegrate ol two ports, we have to phsicall upgrade 1 If we igore the cost of the cables, the average cost of coectig a server i a DPillar etwork is two times the per-port cost of the switches used i this DPillar etwork. the servers if usig a wrapped butterfl etwork to itercoect them. Although the cost of additioal etwork iterfaces is ot a issue, istallig those iterfaces i a large umber of servers ca be quite time ad mapower cosumig. B coectig the servers via switches, DPillar ca use off-the-shelf ad eistig dual-port servers to build a scalable data ceter etwork..6.. etwork diameter A wrapped butterfl etwork with k 0 colum ca accommodate 0 ¼ k 0 k0 servers, while the umber is ¼ k k for a (,k) DPillar etwork. Assumig 0 =, i.e., coectig the same amout of servers i a wrapped butterfl etwork ad a DPillar etwork, we have k 0 k log logðþ ¼ log ðþ1: We will show later i Sectio 3 that the diameter of DPillar (the ma path legth betwee two servers i DPillar) is a liear fuctio of k, the umber of server colums i the etwork. The liear relatioship betwee k 0 ad the etwork diameter also holds for a wrapped butterfl etwork. Therefore, to itercoect the same amout of servers, the diameter of wrapped butterfl is about (log () 1) times the diameter of DPillar. We see that because of usig switches, a DPillar etwork ca have much more servers i oe server colum tha the wrapped butterfl etwork. Hece, DPillar etwork scales the umber of servers with much less server colums tha the wrapped butterfl etwork does ad results i much shorter paths whe forwardig traffic betwee servers i the etwork etwork bisectio width Usig switches to coect servers also ields larger bisectio width i DPillar, as compared to wrapped butterfl. The bisectio width of a (,k) DPillar etwork is equal to the umber of servers i each server colum. A wrapped butterfl etwork also has this propert [19]. Each server colum has k0 servers i a wrapped butterfl; while each k server colum has servers i DPillar. If coectig the same amout of servers i a wrapped butterfl etwork ad a DPillar etwork, i.e., 0 =, we have ð Þk 0 ¼ k k0 k log ðþ1: I other words, DPillar s bisectio width is about (log () 1) times the bisectio width of a wrapped butterfl. For eample, if usig 48-port switches, a DPillar etwork s bisectio width is about 4.5 times the bisectio width of a wrapped butterfl etwork accommodatig the same amout of servers. 3. Sigle-path routig B cosiderig the etwork structure ad addressig the odes i a itelliget wa, oe ca desig efficiet routig i a etwork with smmetric structure. We will show i this sectio that routig ad packet forwardig i DPillar ca be quite straightforward because of the

6 Y. Liao et al. / Computer etworks 56 (01) wa of itercoectig servers. A server ca determie the ethop b simpl replacig some bits i its address with the correspodig bits i the destiatio address of the packet beig forwarded. Therefore, servers ca avoid epesive table lookup operatios whe forwardig packets. The routig scheme described i this sectio computes a uique path from a source server to a destiatio server i DPillar. This sigle-path scheme serves as the basis for our DPillar multi-path routig scheme preseted later i Sectio Two-phase packet forwardig As discussed i Sectio.3, each switch i a (,k) DPillar directl coects servers i its two eighborig server colums, ad the labels of those servers are the same if oe of the smbols is removed. This propert esures that a server u i DPillar ca alwas directl reach aother server v i its eighborig server colum, where the label of v has 1 smbols i commo with u s label ad the other smbols of v ca be of a value i 0; 1. Assumig u is at server colum H i, u ca forward a packet to server v at H (i+1)%k, where the ith smbol of server v s label ad the destiatio server s label are the same. Similarl, v ca forward the packet to aother server w at H (i+)%k, where the (i + 1)%kth smbol of w is the same as that of the destiatio server s label. B keepig doig this, the packet ca be forwarded to a server whose label is the same as the destiatio s label withi hops. ote that i two eighborig colums, servers of the same label are also directl coected b switches, the packet ca be set to its destiatio b alwas forwardig to a ethop server with the same label ad whose colum is closer to the destiatio server s colum. Based o the ratioale described above, packet forwardig i DPillar ca be divided ito two phases. I the first phase, the packet is forwarded from the source server to a itermediate server whose label is the same as the destiatio server s label. I the secod phase, the packet is forwarded from that itermediate server to the destiatio. I the followig, we cosider a sceario where a source server s seds a packet to a destiatio server d. The addresses of those two servers are (C s, L s ) ad (C d, L d ), where L s ad L d are the k-smbol labels of server s ad d, respectivel. Let the labels of those two servers be L s ¼ ms k1...m 0 s, Ld ¼ m k1 d...m 0 d, ad Ls L d. (1) Phase oe heli phase: From server s, which is i colum H Cs, the packet is set to a server s 1 i colum H ðcsþ1þ%k. The label of s 1 is the same as the label of s ecept the C s th smbol i s 1 s label ca be a umber from 0 to 1. If server s 1 seds the packet to server s i colum H ðcsþþ%k, s s label is the same as s 1 s ecept for that the ((C s + 1)%k)th smbol of s s label, which ca be a umber from 0 to 1. We see that whe a packet is forwarded for oe hop, we ca chage oe smbol i the label of the server which receives that packet. Whe a packet is alwas forwarded from oe server colum to its clockwise eighborig server colum withi k hops, the packet ca reach a server with a give label. For eample, i a (,k) DPillar etwork, the trace of a packet forwarded from (0, 0...0) to (k 1, 1...1) ca be (0, 0...0)? (1, 0...1)? (, )? (k 1, 1...1). As this path resembles a heli amog the clidrical surface of the virtual pillar, we call the first phase of the packet forwardig process the heli phase. ote that i the heli phase, we ca alwas sed the packet to either a server i the clockwise eighborig colum or a server i the couter-clockwise eighborig colum. However, the directio of forwardig a packet should ot be chaged back ad forth i order to avoid loops. Hece, oe field i the packet header is used to record the forwardig directio, ad the source server decides the forwardig directio as clockwise or couterclockwise. () Phase two rig phase: After the packet is forwarded to a itermediate server d whose label is the same as the label of destiatio server d, oe ca forward the packet to d b alwas sedig it to the server i the clockwise eighborig colum whose label is L d too, or sedig alog the couter-clockwise directio. We choose the same directio as the heli phase i our forwardig process. We see that i this phase of packet forwardig, the packet forwardig path is like a segmet i a rig amog the clidrical surface of the virtual pillar. We call the secod phase the rig phase. Algorithm 1. DPillarSP((C c,l c ), (C d,l d ), D) The pseudo-code of this sigle-path routig algorithm, deoted as DPillarSP, is show i Algorithm 1. This algorithm takes the address of the curret server (C c,l c ), the destiatio server s address (C d,l d ), ad the forwardig directio D as iput parameters. D = 1 meas the directio is clockwise; D = 1 idicates the couter-clockwise directio. The output is the address of the ethop server (C P,L P ). DPillarSP is loop free, ad there is o eed to maitai a routig table i the servers. Each server ca determie the ethop of a packet i costat time, idepedet of how ma servers there are i a DPillar etwork. I the rest of this paper, we call a path geerated b DPillarSP the DPillarSP path. 3.. Legth of the path computed b DPillarSP A path derived from DPillarSP is obviousl ot the shortest oe. However, DPillarSP alwas ields paths with

7 138 Y. Liao et al. / Computer etworks 56 (01) bouded legth. We cosider the worst case sceario where the labels of two servers have o commo smbols, i which case a packet eeds to be forwarded k times i order to reach a server d whose label is the same as server d s label. After that, from server d, the packet still eeds to be forwarded (k 1) hops i order to reach server d i the worst case. Therefore, the legth of the logest path computed b DPillarSP i a (, k) DPillar etwork is (k 1). Propositio 3.1. Usig DPillarSP, a two servers i a (, k) DPillar etwork ca reach each other i at most k 1 hops. Fig. 3. I a (,k) DPillar etwork, b pairig clockwise eighborig servers of the source s ad the couter-clockwise eighborig servers of the destiatio d, we ca ield disjoit paths betwee s ad d. 4. Multi-path routig The migratio of missio critical applicatios ito data ceters imposes stricter requiremets to a data ceter etwork i terms of reliabilit, robustess, ad performace. Multi-path routig has bee show to be a efficiet mechaism to tolerate failures ad balace traffic load i a etwork [18,1,6]. The rich coectios iside a DPillar etwork ad the fleibilit of DPillar s server-cetric architecture facilitate the desig of efficiet multi-path routig scheme. This sectio presets DPillarMP, a multi-path routig scheme for DPillar. I a (,k) DPillar etwork, DPillarMP computes disjoit paths betwee a pair of source destiatio servers. We will formall prove the correctess of our mechaism i computig disjoit paths, ad provide a desig to utilize the disjoit paths i bpassig failures ad balacig traffic load i a DPillar etwork The basic idea I DPillar, we defie two paths from a source server s to a destiatio server d to be ode-disjoit if the do ot have a commo servers or switches ecept the switch coected to s ad the switch coected to d. As alread show i Sectio 3, a source server s ca alwas sed packets to a server at its clockwise eighborig colum. I a (,k) DPillar etwork, because server s has directl coected servers (via the same switch) at its clockwise eigh- borig colum, s has possible et hops to forward packets. For the destiatio server d, there alwas eists some server at d s couter-clockwise eighborig colum forwardig the packets to d, ad d directl coects to servers at its couter clockwise eighborig colum as well. We show a eample i Fig. 3. The source s has et hops to forward the packet; similarl, there are previous hops which ca deliver the packets to destiatio d. Let s 0 i be the et hops of source s where 0 6 i 6 1, ad d 0 j,06 j 6 1 be the previous hops of destiatio d. If there is a algorithm to ield pairs of s0 i ; d0 j, where 0 6 i; j 6 1 so that the paths betwee a two pairs do ot have commo itermediate servers or switches, we ca have disjoit paths betwee source s ad destiatio d. et we stud the propert of disjoit paths i DPillar, ad the preset how to pair s 0 i ad d0 j to costruct the ode-disjoit paths. 4.. Disjoit paths i DPillar We use a eample i Fig. 4 to demostrate the disjoitess of two paths. Here the source server is (0,00), ad the destiatio server is (0,33), deoted as s ad d i Fig. 4. We eed to pick two et hop servers (deoted b 1 ad ) for source s ad two previous hop servers (deoted b 1 ad ) for destiatio d so that the path betwee 1 ad 1 does ot share commo itermediate odes with the path betwee ad. I this eample, we select server (1,00) ad (1,01) as the two et hops of source s; ad select (1,13) ad (1,3) as the two previous hops of destiatio d. The DPillarSP path from 1 to 1 is (1,00)? (0,10)? (1,13); the DPillarSP path from to is (1,01)? (0,1)? (1,3). Oe ca see that these two paths do ot have a commo itermediate servers. Let us have a closer look at the addresses of 1,, 1, ad. Because 1 ad have differet smbol m 0, their et hops accordig to Algorithm 1 should ot be the same sice the m 0 smbols of those two servers labels are differet. I geeral, the addresses of 1,, 1, ad should be chose i a wa that whe the paths are computed b Algorithm 1 ad two itermediate servers are at the same colum, their labels are differet. Based o the ituitio described above, we have idetified a sufficiet coditio for two DPillarSP paths betwee two et hops of the source ad two previous hops of the destiatio to be disjoit. Without loss of geeralit we cosider a sceario where source server s is at colum H s ad destiatio server is at colum H d. Let 1 ad be two clockwise eighborig servers of s at colum H s+1 ; 1 ad be two couter-clockwise eighborig servers of d at colum H d1 (the modulo operatios are omitted here for simplicit of epressio). Sice 1 ad are coected to the same switch at switch colum S s, their labels are the same ecept for the sth smbol. Their addresses ca be represeted as s þ 1; m k1...m s 1...m 0, ad s þ 1; m k1... m s...m 0 Þ, respectivel, where ms 1 m s. Similarl, 1 ad are coected to the same switch at switch colum S d1, so their labels are the same ecept for the (d 1)th smbol. We use ðd 1; m k1...m d1 d 1...m 0 dþ ad ðd 1; m k1...m d1 d...m 0 dþ to represet their addresses, where m d1 d 1 m d1 d. The followig Corollar 4.1 describes a sufficiet coditio for the DPillarSP paths betwee those two pairs of servers to share o commo servers or

8 Y. Liao et al. / Computer etworks 56 (01) Fig. 4. Two paths with o commo servers or switches i a (8,) DPillar etwork. switches, assumig that the DPillarSP paths follow the clockwise directio. Corollar 4.1. Let P 1 be the DPillarSP path from server s þ 1; m k1...m s 1...m d1...m 0 to server d 1; m k1... m s...md1 1...m 0 ; ad P be the DPillarSP path from server s þ 1; m k1...m s...m d1...m 0 to server d 1; m k1... m s...md1...m 0.P 1 ad P do ot share a commo servers or switches, if m s 1 m s m d1 1. or md1 m d1, ad m s m s or md1 Proof. For each path, we divide it ito two segmets, Seg h ad Seg r. Seg h icludes the odes (servers ad switches) o the path from colum H s+1 to colum H s, ad Seg r icludes remaiig odes o the path (from colum H s+1 to colum H d1 ). ote that we reuse the first letter of heli ad rig here as the subscripts because Seg h ad Seg r resemble the heli phase ad the rig phase, respectivel. Whe d 6 s: I the first segmet of P 1, deoted b Seg h (P 1 ), the labels of all servers have the sth smbol to be m s 1. Similarl, the labels of all servers i Seg h (P ) have sth smbol to be m s. Because m s 1 m s, these two segmets have o commo servers. Moreover, as the switch colums i those two segmets are S s+1, S s+,..., S s1, accordig to Propositio.1, we kow servers i these two segmets alwas coect to differet switches whe the are i the same server colum. So these two segmets have o commo switches either. I Seg r (P 1 ), all servers have the (d 1)th smbol to be m d1 1 ; all servers i Seg r (P ) have the (d 1)th smbol to be m d1. Similarl, because m d1 1 m d1, these two segmets also have o commo servers or switches. For Seg h (P 1 ) ad Seg r ðp Þ; m s 1 m s or m d1 m d1 esures that the have o commo servers or switches. Similarl, m s m s or md1 m d1 1 esures that Seg h (P ) ad Seg r (P 1 ) have o commo servers or switches. Hece, the first segmet of oe path shares o commo servers or switches with the secod segmet of aother path. Whe d > s: The proof is similar. h et we will show how to pair the clockwise eighborig servers of source s with the couter-clockwise eighborig servers of destiatio d, so as to ield ode-disjoit paths betwee s ad d.

9 140 Y. Liao et al. / Computer etworks 56 (01) Path costructio Let source server s be at colum H s, destiatio server d be at colum H d, ad their addresses be ðc s ; m k1... m s...m0 Þ; C d; m k1...m d...m0, respectivel. The addresses of the clockwise eighbors of the source are ðs þ 1; m k1...m s i...m d1...m 0 Þ, where i ½0; 1Š ad m s i ¼ i. The destiatio server has couter-clockwise eighbors at colum H d1, whose addresses are ðd 1; m k1...m s...md1 j...m 0 Þ, where j ½0; 1Š ad md1 j ¼ j. For those servers at H s+1 ad servers at H d1, we first pair server s þ 1; m k1...m s...md1...m 0 with server ðd 1; m k1...m s...md1...m 0 Þ, where ms ¼ ms ad md1 ¼ m d1, the arbitraril pair the rest servers at H s+1 with the other servers at H d1. This pairig scheme ields clockwise DPillarSP paths betwee the servers at H s+1 ad servers at H d1. ow we show that those paths betwee servers i H s+1 ad i H d1 do ot share a commo servers or switches. Let P be the path betwee server ðs þ 1; m k1... m s...md1...m 0 Þ ad server ðd 1; mk1...m s...md1...m 0 Þ, where m s ¼ ms ad md1 ¼ m d1. Let P 0 be aother path betwee server ðs þ 1; m k1...m s i...m d1...m 0 Þ ad server ðd 1; m k1...m s...md1 j...m 0 Þ. Sice ms ms i ad m d1 m d1 j, we have m s ms i ad m d1 m d1 j. Accordig to Corollar 4.1, P ad P 0 caot have a commo servers or switches. Similarl, we ca derive that P 0 must be disjoit to a other path P 00 too, where P 00 P. Algorithm. CostructPairSet((C s,l s ), (C d,l d )) remaiig 1 servers i SETs ad 1 servers i SETd accordig to oe of the smbols ad alwas pairs the first elemet i each set together, so as to ield determiistic pairig of all servers. 3 Algorithm geerates server pairs. All the clockwise DPillarSP paths betwee two servers of these server pairs do ot share a commo servers or switches. As we alread discussed i Sectio 4.1, usig those server pairs, we ca have ode-disjoit paths betwee source s ad destiatio d. The correctess of Algorithm i producig ode-disjoit paths is formall stated as Theorem 4. ad its detailed proof is provided i Appedi B. Theorem 4.. For a two servers s ad d i a (,k) DPillar etwork, we pair the clockwise eighbors of s with the couter-clockwise eighbors of d b followig Algorithm to obtai clockwise DPillarSP paths betwee these eighbors. Those paths betwee s ad d are ode-disjoit. ote that accordig to Propositio 3.1, o DPillarSP path i a (,k) DPillar etwork is loger tha k 1, so the legth of these ode-disjoit paths betwee the source server ad the destiatio server is bouded b k +1. We have show how to costruct ode-disjoit paths betwee a source ad a destiatio i the clockwise directio. There are also couter-clockwise ode-disjoit paths betwee the source ad destiatio servers. That is, we pair the couter-clockwise eighbors of source with the destiatio s clockwise eighbors, ad build ode-disjoit couter-clockwise paths. The detailed scheme is omitted here as it is quite similar to the techique i buildig clockwise ode-disjoit paths Packet forwardig The idea described above leads to a straightforward algorithm i computig ode-disjoit paths i DPillar. We show the pseudo code i Algorithm. This algorithm first pairs a server i SET s with a server i SET d, where their sth ad (d 1)th smbols are the same. The it sorts the After costructig the multiple ode-disjoit paths, the et questio is how to eforce a packet to follow oe of those paths. For a packet to follow a path from server s to server d, the packet traverses a path s? s 0...d 0? d, where s 0 ad d 0 are two itermediate servers paired b the scheme described i Sectio 4.3. We use tuelig to embed the iformatio of s 0 ad d 0 ito a packet header. The source selects a path amog the set of ode-disjoit paths ad adds aother header i frot of the origial packet, where the source ad destiatio fields i the outer header are set to s 0 ad d 0, respectivel. We call the source address i the outer header a pro source, ad the destiatio address i the outer header a pro destiatio. As the source server is coected directl with the pro source server, the source simpl forwards the packet to the pro source server. The the packet is forwarded accordig to the scheme specified i Algorithm 1 to the pro destiatio server, which will decapsulate the packet ad sed it to the destiatio. Algorithm 3 shows the pseudo-code of how a server computes the ethop i forwardig a packet based o the ier ad outer packet headers. 3 ote that this is purel for geeratig determiistic pairig of the servers. Oe ca arbitraril the remaiig 1 servers i SETs with the 1 servers i SETd, ad it does ot affect the disjoitess of those paths.

10 Y. Liao et al. / Computer etworks 56 (01) Algorithm 3. ethop(c, pkt) applicatio to coect with the destiatio usig TCP, it radoml picks oe of the ode-disjoit paths to establish the coectio. If the first tr of TCP coectio fails due to failure or cogestio i the etwork ad the applicatio issues a re-tr request, the source server radoml picks a path agai. As there are multiple ode-disjoit paths, it is ver likel that the source server ca fid a path to bpass the failure or cogestio poit after a few tries Applicatios of multiple ode-disjoit paths i DPillar The multiple ode-disjoit paths i DPillar ca be utilized to improve the performace of a DPillar etwork i various aspects. Here we provide the sample desigs o how to tolerate failures ad balace traffic b usig the multiple disjoit paths i DPillar Fault-tolerat multi-path routig The multiple disjoit paths betwee two servers i DPillar ca be utilized to route traffic i a fault-tolerat maer. Whe a applicatio ruig i a source server requires fault-tolerat routig from DPillar, the source server builds multiple ode-disjoit paths betwee itself ad the destiatio server before the applicatio starts to sed traffic. After costructig the paths, the source server ca pick oe of them to sed packets to the destiatio server. I order to tolerate failures, the source server should also proactivel moitor the status of those ode-disjoit paths b periodicall sedig probig messages o each path. After the destiatio server receives a probig message, it replies a probig respose message to the source. Upo receivig the probig respose message, the source kows that the path is workig. If a itermediate server receives a probig message ad its ethop is ureachable, it returs a path failure message to the source. Upo receivig the path failure message, the source ca switch the traffic to a alterative ode-disjoit path. We should ote that moitorig the status of multiple paths icurs certai overhead to the etwork because of usig probig messages. Probig paths also takes time as a message eeds to do a roud trip betwee source ad destiatio servers. Therefore, this proactive probig scheme is more suitable for applicatios geeratig lots of bulk flows or log-haul flows. For small ad short-lived flows, we ca use a more light-weight scheme i utilizig the multiple ode-disjoit paths. Each time a applicatio i a source server geerates traffic to a destiatio, it tries a few times i establishig a TCP coectio to the destiatio. 4 Whe the source server gets the request from a 4 Eistig measuremet work shows that most of the traffic i data ceters is TCP [7] Traffic-aware multi-path routig The eistig of multiple disjoit paths betwee two servers also opes the desig dimesio of balacig traffic load i DPillar. A source geeratig a bulk or log-haul flow ca periodicall sed messages o the multiple odedisjoit paths to the destiatio, to probe the available badwidth o each path, ad schedule its flow to the path with the largest available badwidth. Because the multiple paths are ode-disjoit, we ca easil avoid a hot-spot i the etwork. 5. Evaluatio We evaluate DPillar from two differet aspects. First, we implemet the DPillar packet forwardig mechaism i Click software router [] ad stud the microscopic behavior of DPillar b measurig the packet forwardig overhead of oe DPillar server. Our measuremets show that packet forwardig does ot cause too much CPU usage overhead to servers. Secod, we stud the macroscopic behavior of DPillar b simulatig the packet routig ad forwardig i DPillar etworks, usig a simulatio tool we developed. Our simulatio results show that our routig scheme performs well eve i the presece of a large umber of server ad switch failures Implemetatio ad testbed We have implemeted the DPillar routig algorithms preseted i Sectios 3 ad 4 as a elemet i kerel mode Click software router []. Our implemetatio uses a IP address to ecode the colum ad label iformatio of oe server, so as to provide backward compatibilit to upper laer applicatios. The most sigificat 8 bits of the 3-bit IPv4 address are reserved to represet the colum umber of a server. As each host has two ICs, the least sigificat bit of the 3-bit IPv4 address is used to represet the ICs. The k-smbol label cosumes kd log () 1e bits. The 3-bit IPv4 address is allocated as show i Fig. 5. For eample, whe usig 48-port switches to build a DPillar etwork with 4 colums of hosts (the total umber of hosts is about 1.3 millio), we eed to use bits i the most sigificat 8 bits of the IP header to represet the colums. I the remaiig 4 bits, we use 4dlog (48) 1e + 1 = 1 bits to represet the label of servers ad the iterfaces. Fig. 6 shows the testbed used i our eperimets to evaluate DPillar routig ad forwardig elemet implemeted i kerel mode Click. Server P i Fig. 6 is a commodit PC with a.4 GHz dual-core CPU ad 1 GB

11 14 Y. Liao et al. / Computer etworks 56 (01) Fig. 5. Reusig bits of a IP address to represet the addresses of servers i DPillar. is commol supported i commodit Etheret switches, 5 we ca take advatage of this feature to reduce the CPU usage of DPillar servers i traffic forwardig. Fig. 6. The testbed etwork. Server P rus DPillar packet routig ad forwardig Click elemet ad forwards packets betwee server A ad server B. memor. Server A ad B are similar machies pumpig traffic ito server P. 5.. Overhead of forwardig packets We measure its CPU usage whe a DPillar server forwards traffic for other servers. I this eperimet, server A ad B i Fig. 6 use iperf to sed out UDP traffic to each other at various speed, ad server P forwards the traffic betwee A ad B. We record the CPU usage of the Click kerel thread ad plot the results i Fig. 7. The results i Fig. 7 show that eve whe the DPillar server forwards traffic at full load, i.e., 1 Gbps each directio, Gbps i total, the CPU usage of the DPillar server is less tha 50%. Our server is a dual-core machie ad the less tha 50% CPU usage is for oe CPU core; the other core is almost 100% idle. As commodit PCs with multicore CPUs are becomig commo, we epect the traffic forwardig overhead ca be amortized so that ol a small portio of the total processig power of a multi-core server is used i traffic forwardig. It is well kow that the amout of CPU ccles used to forward a packet does ot deped o the packet legth. Hece, i a server-cetric data ceter etwork, oe ca reduce the CPU load ad still maitai the throughput b usig larger packets [16]. As jumbo frame packet trasfer 5.3. Path legth We have developed a evet-drive simulatio tool to stud packet forwardig i DPillar etworks. Give the umber of server colums k ad the switch port umber, our simulatio tool builds a (, k) DPillar etwork topolog ad simulates how each server routes packets Without failure We first stud the legth of paths computed b DPillarSP ad DPillarMP. We simulate the scearios where the DPillar etwork is built from 16-port switches, i.e., = 16, ad the umber of server colums k varies. For each etwork, we radoml select 10,000 source destiatio pairs ad simulate the paths ielded b DPillarSP ad DPillarMP. Fig. 8 plots the results of the average legth of the paths betwee those 10,000 source destiatio pairs. The results show that the path legth is proportioal to the umber of server colums i a DPillar etwork, ad it is alwas about 0% shorter tha the maimum path legth (k 1) for DPillarSP ad about 5% shorter tha the maimum path legth (k + 1) for DPillarMP. I additio, compared with DPillarSP, DPillarMP does ot iflate the path legth too much. The iflatio is alwas withi hops With failure We also stud the average legth of paths computed b DPillarMP scheme whe there are server failures. Our simulatio icludes four DPillar etworks of differet sizes, i.e., (16,3) etwork, (4,3) etwork, (16,4) etwork ad (4, 4) etwork. I each simulatio istace, we radoml fail a certai percetage of the total servers. We var the ratio of failed servers from 0 to 0. (e.g., failure ratio of 0.1 meas 10% of the servers have failed). The we radoml select 10,000 source destiatio pairs from those servers without failures ad compute the average legth of the radoml picked workig paths (if a) for each pair ielded b DPillarMP. Fig. 9 plots the average path legth vs. the server failure ratio. We observe that although there are failures, the path legth is still proportioal to the umber of server colums ad it slightl decreases as there are more failed servers. This is because the legths of the ode-disjoit paths for a source destiatio pair ca be differet. As we radoml Fig. 7. The server CPU usage uder various UDP traffic load. The packet size is 104 btes. 5 The 16-port Gbit switch metioed i Sectio.5 supports up to 1. k btes jumbo frame.

12 Y. Liao et al. / Computer etworks 56 (01) Average path legth (# of hops) DPillarSP DPillarMP k: # of server colums Avg # of available paths per pair server failure switch failure # of failed servers (switches) Fig. 8. Average path legth for DPillarSP ad DPillarMP. Fig. 10. Average umber of available paths for oe pair. Average path legth (# of hops) geerate failed servers, a shorter path is less likel to fail as compared with a loger path, because the short path has fewer servers Fault tolerace (16,3) etwork (4,3) etwork (16,4) etwork (4,4) etwork Server failure ratio Fig. 9. Average path legth vs. server failure ratio. We also stud the performace of DPillarMP i toleratig failures i the etwork. I each simulatio istace, a certai umber of servers or switches i a (16,3) DPillar etwork are radoml chose as failed oes. We the radoml select 10,000 source destiatio pairs from those servers without failures. For each source destiatio pair, we compute how ma paths amog the clockwise disjoit paths ielded b DPillarMP are still available. The average umber of available paths amog all 10,000 source destiatio pairs is plotted i Fig. 10. From Fig. 10, we see that as there are more server failures or switch failures, the average umber of the available paths decreases. Switch failures have much larger impact tha server failures, sice a switch coects much more liks tha a server ( vs. ). We further stud how the server ad switch failures ca impact reachabilit i DPillar. For a source destiatio pair, if DPillarSP (or DPillarMP ) caot ield a workig path, we sa DPillarSP (or DPillarMP ) has a routig failure. I each simulatio istace, we radoml select M = 10,000 source destiatio pairs. If M of them have routig failures, we sa the routig failure ratio of this scheme is M. M Figs. 11 ad 1 plot the routig failure ratio of DPillarSP ad DPillarMP, vs. the umber of server failures ad the switch failures, respectivel. For both DPillarSP ad DPillarMP, the routig failure ratio icreases as there are more server failures or switch failures. However, the routig failure ratio of DPillarMP is much lower tha that of DPillarSP i all cases. DPillarMP does ot have a routig failures eve whe the umber of server failures reaches 300 (0% of total servers), while DPillarSP has 45% routig failures at this poit. Whe the umber of server failures is as high as 500 (3% of total servers), DPillarMP has ol about 0.1% routig failures. Eve whe the umber of switch failures reaches 0 (10% of total switches), the routig failure ratio is about 1% ol for DPillarMP Server forwardig load uder tpical traffic patters We stud the forwardig load of servers uder two tpical traffic patters, i.e., oe-to-oe commuicatio patter ad all-to-all commuicatio patter. I the former patter, we divide all servers i a DPillar ito sources ad destiatios. Each source server seds a flow to a radoml selected destiatio server. For the latter patter, we radoml select 1% servers from a DPillar etwork Routig failure ratio DPillarMP DPillarSP # of server failures Fig. 11. Routig failure ratio vs. umber of server failures.

13 144 Y. Liao et al. / Computer etworks 56 (01) Routig failure ratio DPillarMP DPillarSP # of switch failures Server forwardig load Percetage of servers (%) Fig. 1. Routig failure ratio vs. umber of switch failures. ad let each of those servers sed flows to the other servers. I our eperimets, we use the umber of flows forwarded b a server as the measure for its traffic forwardig load. The distributio of all servers traffic forwardig load is plotted i Fig. 13. We ca see i both etworks we tested (a (16,3) etwork ad a (4,3) etwork), most servers forward a small umber of flows. For eample, i oe-to-oe commuicatio patter aroud 95% of servers i both (16,3) etwork ad (4,3) etwork forward o more tha CDF CDF (16,3) etwork w/ SP 0.3 (4,3) etwork w/ SP 0. (16,3) etwork w/ MP (4,3) etwork w/ MP # of flows oe server forwardig (a) oe-to-oe commuicatio patter (16,3) etwork w/ SP (4,3) etwork w/ SP 0.5 (16,3) etwork w/ MP (4,3) etwork w/ MP # of flows oe server forwardig (b) all-to-all commuicatio patter Fig. 13. Server traffic forwardig load i tpical traffic patters. Here w/ SP meas with DPillarSP routig scheme; w/mp meas with DPillarMP routig scheme. I DPillarMP routig scheme, the source server radoml selects oe route amog all routes. Fig. 14. Average server forwardig load i all-to-all commuicatio patter whe the percetage of participatig servers varies. The etwork is (16,3) DPillar etwork. A source server radoml selects a path amog all paths to a destiatio. 4 flows usig DPillarMP. The most loaded servers forward o more tha 11 flows. Although the total umber of flows i (4,3) etwork is much more tha the total umber of flows i (16, 3) etwork, the server forwardig load does ot have oticeable icrease i oe-to-oe commuicatio patter. I all-to-all commuicatio patter, servers i (4,3) etwork eed to forward more flows as compared to servers i (16,3) etwork, because there are more flows i (4,3) etwork as we select 1% of the servers to sed out traffic. To stud how the server forwardig load chages whe more servers participate i all-to-all commuicatio, we simulate DPillarMP i a (16, 3) DPillar where the percetage of servers participatig i all-to-all commuicatio rages from 1% to 9%. We compute the average umber of flows forwarded b servers i the DPillar etwork ad plot the results i Fig. 14. As epected, whe more servers participate i all-to-all commuicatio, each server eeds to forward more flows. The forwardig load of each server does ot icrease faster tha the umber of flows i the etwork. For eample, whe 3% of the servers participate i all-to-all commuicatio, each server forwards 5 flows o average; whe 9% of the servers participate i all-toall commuicatio, the total umber of flows icreases roughl 0:09 ¼ 9 times, ad the average umber of flows 0:03 forwarded b each server is 45, which icreases 9 times as well. 6. Backgroud ad related work 6.1. Itercoectio etworks Server itercoectio etwork has log bee a active research topic. Two categories of itercoectio etworks are broadl studied. The first oe has a clear boudar betwee etwork ad ed hosts, where multiple levels of switches are coected ito a switchig fabric, ad servers are attached as leaves of the switchig fabric [4]. Servers are pure ed-hosts, which perform computatio ad storage tasks ol. Oe etwork iterface is eough for each server to be coected with other servers. I the secod categor of itercoectio etwork, servers are ot ol computatio/storage workstatios

14 Y. Liao et al. / Computer etworks 56 (01) but also itermediate odes relaig traffic for other servers. Classic itercoectio topologies iclude full mesh, hpercube, butterfl, de Bruij, etc. [19,4,10]. Compared to coectig servers b a switchig fabric, usig servers as rela odes is ofte believed to be more fleible i buildig the itercoectio etwork, because servers are much easier to program tha switchig devices. 6.. Related work i data ceter etworks A thread of recet research activities o data ceter etworks have proposed several itercoectio architectures. The Mosoo etwork preseted i [14] uses a hierarchical tree-structure switchig fabric with two levels of switches, the top-of-rack switches ad the core switches. The fat-tree etwork preseted i [5] also uses a switchig fabric to coect servers. The switchig fabric of fat-tree etwork is built from idetical switches, so there is o eed to use epesive high-speed core switches. The switches used i fat-tree should have laer-3 switchig capabilit ad eed to be slightl upgraded i order to make full use of its uderlig topolog. A secod fat-tree based data ceter etwork structure, the PortLad etwork, is proposed i []. B usig hierarchical pseudo MAC addresses, switches i a PortLad etwork ca forward traffic as laer- packets. Removig laer-3 switchig capabilit from switches ca sigificatl reduce the cost of buildig large scale data ceter etworks. DCell [17] is a server-cetric etwork where a higher level DCell etwork is recursivel costructed from lower level DCell etworks, ad the umber of accommodated servers grows double epoetiall as the level icreases. As the umber of levels i a DCell etwork icreases, servers eed to istall more iterfaces. The liks i DCell etwork are ot evel loaded. Those liks coectig lower level DCells are usuall more loaded tha the liks coectig higher level DCells. The FiCo etwork proposed i [0] uses similar recursive costructio techique as DCell, but requires ol two etwork iterfaces i each server. FiCo also has the uevel loaded lik issue. BCube [16] is aother server-cetric etwork, whose topolog is closel related to the Hpercube etwork [19]. BCube has rich coectios to support badwidth demadig applicatios ruig i data ceters. But servers i a BCube etwork eed to istall more etwork iterfaces i order to scale the etwork size to accommodate more servers. Table summarizes the ke features of differet data ceter itercoectio etworks. 7. Coclusio This paper presets DPillar, a data ceter etwork architecture built from commodit hardware. DPillar is a server-cetric architecture ad the etworkig itelligece is placed i servers. Switches i DPillar are laer- plugad-pla devices, which are cost efficiet ad widel available. DPillar ca easil scale to a huge umber of servers without imposig a additioal requiremets to servers, such as istallig more etwork iterfaces. The topolog of DPillar is smmetric ad a DPillar etwork provides rich coectios betwee servers. We have desiged efficiet routig schemes for DPillar. Prototpig implemetatio ad simulatio studies show that our routig schemes are lightweight, high-performace, ad efficiet i bpassig failures i the etwork. Ackowledgmets The authors are grateful to the editor, Dr. Luigi Iaoe, ad the aomous reviewers for ma isightful commets ad costructive suggestios. This work is partiall supported b SF Grats CS ad CS Appedi A. Proof of Propositio.3 Our proof is ispired b previous work [8] i studig the bisectio width of butterfl etworks. Clearl, if we cut a (, k) DPillar etwork horizotall, i.e., each server colum is cut ito halves, we ca alwas cut a DPillar etwork ito a top half ad a bottom half b cuttig the coectios amog H k1, S k1, ad H 0. For eample, we ca divide the DPillar etwork show i Fig. ito top ad bottom halves b cuttig the liks cross a virtual lie betwee row (13) ad row (0). Because each switch i S k1 has liks crossig the virtual cuttig lie, the total umber of liks crossig the virtual cuttig lie is k1 ð Þ¼ð Þk. k Hece, we have a upper boud,, for the bisectio width of a (,k) DPillar etwork. Table Compariso betwee differet data ceter itercoectio etworks. Parameter is the umber of ports i each switch; is the total umber of servers; U s is the uit price of a -port switch. For DCell, FiCo, ad BCube, l is depth of recursio i buildig the etwork. For DPillar, k is the umber of server colums. DCell FiCo BCube FatTree DPillar Server degree l +1 l +1 1 Bisectio width umber of servers 4log 4 l k ð þ 1Þ l lþ l l k k umber of switches ðl þ 1Þ 6 k ðl þ 1Þ Us 5 Us Us Cost of coectig Us Us Oe server Switch upgrade o o o Yes o Traffic balace o o Yes Yes Yes Disjoit paths l +1 1 l +1 1 ot icludig the cost of ICs ad cables.

15 146 Y. Liao et al. / Computer etworks 56 (01) et we fid the lower boud of the bisectio width. Let G be the umber of servers i each colum of a (,k) DPillar etwork. We cosider bisectig the G servers i server colums S 0 ad S k1 b embeddig a complete bipartite graph K G, G ito a (,k) DPillar etwork so that the left side odes ad right side odes of K G,G are mapped to the servers i S 0 ad servers i S k1 of the (,k) DPillar etwork, respectivel. If each of the G servers i S 0 has a path to ever server i S k1, because DPillar etwork is smmetr, there are at most G/ paths use the same server-to-switch lik. Also because the bisectio width of a complete bipartite graph K G,G is G /, the size of the cut that bisects the G servers i S 0 ad S k1 should be at least G. ow we cosider a miimal cut C that bisects all servers i a (,k) DPillar etwork ito Set 1 ad Set. If there eist two eighborig server colums, e.g., S i ad S (i+1)%k, where the G servers are bisected b cut C, we kow that the size of cut C is at least G. Otherwise, we fid two eighborig server colums S j ad S (j+1)%k so that amog the G servers i those two server colums, more servers are i Set 1 tha i Set. The we move some servers (amog those G servers i server colums S j ad S (j+1)%k ) from Set 1 to Set so that half of those G servers are i Set 1. ote that movig the servers from Set 1 to Set does ot icrease the size of cut C. We alread kow that bisectig the G servers i S j ad S (j+1)%k requires cuttig at least G liks. Hece, the size of cut C has lower boud G ¼ k. Because both the upper boud ad the lower boud is k, the bisectio width of a (,k) DPillar etwork is k. Appedi B. Proof of Theorem 4. Without loss of geeralit, the addresses of s s clockwise eighbors ca be represeted as ðs þ 1; m k1... m s i...m d1...m 0 Þ, where i 0; 1 ad m s i ¼ i. The addresses of d s couter-clockwise eighbors are deoted as d 1; m k1...m s...md1 j...m 0, where j ½0; 1Š ad m d1 j ¼ j. et we divide the theorem ito two cases ad prove them oe b oe. Case 1: s = d 1: We have m s i m d1 ¼ m d1 i ¼ i ad m s i ¼ i ¼ i, ad thus Algorithm will pair server ðs þ 1; m k1...m s i...m 0 Þ with server d 1; mk1...m s i...m 0 for i 0; 1. Let P be the DPillarSP path betwee server s þ 1; m k1...m s i...m 0 ad server ðd 1; m k1...m s i...m 0 Þ, ad P0 be the DPillarSP path betwee server s þ 1; m k1...m s j...m 0 ad server d 1; m k1...m s j...m 0 Þ, where i j. It is eas to show both P ad P 0 have legth k 1. Furthermore, all servers o P have the sth smbol to be m s i ad all servers o P 0 have sth smbol to be m s j. Because m s i m s j, these two paths have o commo servers. Moreover, for the servers o these two paths, s does ot equate to the umber of the server colum or the umber miuses oe. B Propositio.1, we kow servers of these two paths alwas coect to differet switches whe the are i the same server colum. Therefore, P ad P 0 have o commo servers or switches. Case : s d 1: I this case, the (d 1)th smbol m d1 of the labels of s s clockwise eighbors is the same, ad the sth smbol m s of the labels of d s couter-clockwise eighbors is the same as well. We let a ¼ m d1 ad b ¼ m s. The, server s þ 1; mk1...m s b... m d1...m 0 Þ ad server d 1; mk1...m s...md1 a...m 0 have the same sth ad (d 1)th smbols. Let P 00 be the DPillarSP path from server s þ 1; m k1 m d1...m 0 Þ to server d 1; mk1...m s b... )....m s...md1 a...m 0 Assume P be the DPillarSP path betwee server s þ 1; m k1...m s e...md1...m 0 ad server ðd 1; m k1...m s...md1 g...m 0 Þ, ad P0 be the DPillarSP path betwee server s þ 1; m k1...m s f...m d1...m 0 ad server d 1; m k1...m s...md1 h...m 0, where b e f ad a g h. We have m s e ms, ad ms f m s. Accordig to Corollar 4.1, P ad P 0 have o commo servers or switches. We also have m d1 m d1 g, ad m s f m s. Accordig to Corollar 4.1, P 00 ad P have o commo servers or switches either. Similarl, we ca prove P 00 ad P 0 have o commo servers or switches. We have show that all the DPillarSP paths betwee two servers of the server pairs costructed b Algorithm have o commo servers or switches. Therefore, those paths (each path via oe server pair) betwee s ad d are ode-disjoit. The proof completes. Refereces [1] Amazo elastic compute cloud. < [] The click modular router project. < [3] Microsoft widows azure platform. < widowsazure>. [4] Cisco data ceter ifrastructure.5 desig guide, December 007. < c649/ccmigratio_09186a d.pdf>. [5] M. Al-Fares, A. Loukissas, A. Vahdat, A scalable, commodit data ceter etwork architecture, i: Proceedigs of SIGCOMM 08, 008. [6] R. Baer, A. Orda, Multipath routig algorithms for cogestio miimizatio, IEEE/ACM Tras. etw. 15 (007) [7] T. Beso, A. Akella, ad D. A. Maltz, etwork traffic characteristics of data ceters i the wild, i: IMC 10: Proceedigs of Iteret Measuremet Coferece, 010. [8] C. Borstei, A. Litma, B. Maggs, R. Sitarama, T. Yatzkar, O the bisectio width ad epasio of butterfl etworks, i: Parallel Processig Smposium, IPPS/SPDP Proceedigs of the First Merged Iteratioal Smposium o Parallel ad Distributed Processig, 1998, pp [9] F. Chag, J. Dea, S. Ghemawat, W.C. Hsieh, D.A. Wallach, M. Burrows, T. Chadra, A. Fikes, R.E. Gruber. Bigtable: a distributed storage sstem for structured data, i: OSDI 06: Proceedigs of the 7th Smposium o Operatig Sstems Desig ad Implemetatio, Berkele, CA, USA, 006. USEIX Associatio, pp [10] W.J. Dall, B.P. Towles, Priciples ad Practices of Itercoectio etworks, Morga Kaufma, 004. [11] J. Dea, S. Ghemawat, MapReduce: simplified data processig o large clusters, i: OSDI 04: Proceedigs of the 5th Smposium o Operatig Sstems Desig ad Implemetatio, 004, pp [1] D. Gaesa, R. Govida, S. Sheker, D. Estri, Highl-resiliet, eerg-efficiet multipath routig i wireless sesor etworks, SIGMOBILE Mob. Comput. Commu. Rev. 5 (001) 11 5.

16 Y. Liao et al. / Computer etworks 56 (01) [13] S. Ghemawat, H. Gobioff, S.-T. Leug, The google file sstem, SOSP 03: Proceedigs of the ieteeth ACM Smposium o Operatig Sstems Priciples, ACM Press, ew York, Y, USA, 003, pp [14] A. Greeberg, P. Lahiri, D.A. Maltz, P. Patel, S. Segupta, Towards a et geeratio data ceter architecture: scalabilit ad commoditizatio, i: PRESTO 08: Proceedigs of the ACM Workshop o Programmable Routers for Etesible Services of Tomorrow, ew York, Y, USA, 008. ACM, pp [15] A.G. Greeberg, J.R. Hamilto,. Jai, S. Kadula, C. Kim, P. Lahiri, D.A. Maltz, P. Patel, S. Segupta, Vl: a scalable ad fleible data ceter etwork, i: ACM SIGCOMM Coferece, 009, pp [16] C. Guo, G. Lu, D. Li, H. Wu, X. Zhag, Y. Shi, T. Che, Y. Zhag, S. Lu, BCube: a high performace, server-cetric etwork architecture for modular data ceters, i: Proceedigs of SIGCOMM 09, 009. [17] C. Guo, H. Wu, K. Ta, L. Shi, Y. Zhag, S. Lu, Dcell: a scalable ad fault-tolerat etwork structure for data ceters, i: Proceedigs of SIGCOMM 08, 008. [18] J. He, J. Reford, Toward iteret-wide multipath routig, etwork, IEEE () (008) [19] F.T. Leighto, Itroductio to Parallel Algorithms ad Architectures: Arras. Trees. Hpercubes, Morga Kaufma, 199. [0] D. Li, C. Guo, H. Wu, K. Ta, Y. Zhag, S. Lu, FiCo: Usig backup port for server itercoectio i data ceters, i: Proceedigs of IFOCOM 09, 009. [1] Y. Liao, D. Yi, L. Gao, Dpillar: Scalable dual-port server itercoectio for data ceter etworks, i: Proceedigs of ICCC 010: the 19th Iteratioal Coferece o Computer Commuicatios ad etworks. [] R.. Msore, A. Pamboris,. Farrigto,. Huag, P. Miri, S. Radhakrisha, V. Subramaa, A. Vahdat, Portlad: a scalable faulttolerat laer data ceter etwork fabric, i: Proceedigs of SIGCOMM 09, 009. [3] L. Rabbe, Powerig the Yahoo! etwork, ovember 006. < corpblog.com/006/11/7/powerig-the-ahoo-etwork>. [4] M.R. Samatham, D.K. Pradha, The de bruij multiprocessor etwork: a versatile parallel processig ad sortig etwork for VLSI, IEEE Tras. Comput. 38 (4) (1989) Yog Liao graduated with a BS degree i 001 from Uiversit of Sciece ad Techolog of Chia. I 004, he received his MS degree from the Graduate School of Chiese Academ of Scieces. Sice fall 004, he has bee workig as research assistat i Uiversit of Massachusetts at Amherst, where ow he is a PhD cadidate i the Electrical ad Computer Egieerig departmet. His curret research iterests iclude iter-domai routig, data ceter etwork, ad etwork virtualizatio. Jiagtao Yi received a BE degree from Beijig Istitute of Techolog, Chia, i 006, ad his ME degree from Beijig Uiversit of Posts ad Telecommuicatios, Chia, i 009. Sice fall 009, he has bee workig as research assistat i Uiversit of Massachusetts at Amherst, where he is curretl pursuig the PhD degree i electrical ad computer egieerig departmet. His curret research iterests iclude data ceter etwork ad data-itesive computig. Dog Yi is a PhD studet at orthwester Poltechical Uiversit (WPU), Xi a, Shai, Chia, whose major is Cotrol Theor ad Egieerig. He received the bachelor degree i Iformatio Securit ad the master degree i Automatic Cotrol ad Egieerig from WPU i 004 ad 007, respectivel. From October 009 to October 010, he had bee a visitig studet i departmet of Electrical ad Computer Egieerig at Uiversit of Massachusetts at Amherst, supported b Chia state scholarship fud. His research iterests iclude etwork virtualizatio ad iformatio securit. Lii Gao is a professor of Electrical ad Computer Egieerig at the Uiversit of Massachusetts at Amherst. She received her PhD degree i computer sciece from the Uiversit of Massachusetts at Amherst i Her research iterests iclude multimedia etworkig, Iteret routig ad securit. Betwee Ma 1999 ad Jauar 000, she was a visitig researcher at AT&T Research Labs ad DIMACS. She is a Alfred P. Sloa Fellow, a IEEE Fellow ad received a SF CAREER Award i She has served o umber of techical program committees icludig SIGCOMM006, SIGCOMM004, SIGMETRICS003, ad IFOCOM004, ad is o the Editorial Board of IEEE Trasactios o etworkig.