5 Interconnection Networks

Transcription

1 5 Itercoectio Networks 5. INTRODUCTION Networkig strategy was origially employed i the 950's by the telephoe idustry as a meas of reducig the time required for a call to go through. Similarly, the computer idustry employs etworkig strategy to provide fast commuicatio betwee computer subparts, particularly with regard to parallel machies. The performace requiremets of may applicatios, such as weather predictio, sigal processig, radar trackig, ad image processig, far eceed the capabilities of sigle-processor architectures. Parallel machies break a sigle problem dow ito parallel tasks that are performed cocurretly, reducig sigificatly the applicatio processig time. Ay parallel system that employs more tha oe processor per applicatio program must be desiged to allow its processors to commuicate efficietly; otherwise, the advatages of parallel processig may be egated by iefficiet commuicatio. This fact emphasizes the importace of itercoectio etworks to overall parallel system performace. I may proposed or eistig parallel processig architectures, a itercoectio etwork is used to realize trasportatio of data betwee processors or betwee processors ad memory modules. This chapter deals with several aspects of the etworks used i moder (ad theoretical) computers. After classifyig various etwork structures, some of the most well kow etworks are discussed, alog with a list of advatages ad disadvatages associated with their use. Some of the elemets of etwork desig are also eplored to give the reader a uderstadig of the compleity of such desigs. 5.2 NETWORK TOPOLOGY Network topology refers to the layouts of liks ad switch boes that establish itercoectios. The liks are essetially physical wires (or chaels); the switch boes are devices that coect a set of iput liks to a set of output liks. There are two groups of etwork topologies: static ad dyamic. Static etworks provide fied coectios betwee odes. (A ode ca be a processig uit, a memory module, a I/O module, or ay combiatio thereof.) With a static etwork, liks betwee odes are uchageable ad caot be easily recofigured. Dyamic etworks provide recofigurable coectios betwee odes. The switch bo is the basic compoet of the dyamic etwork. With a dyamic etwork the coectios betwee odes are established by the settig of a set of itercoected switch boes. I the followig sectios, eamples of static ad dyamic etworks are discussed i detail Static Networks There are various types of static etworks, all of which are characterized by their ode degree; ode degree is the umber of liks (edges) coected to the ode. Some well-kow static etworks are the followig: Degree : Degree 2: Degree 3: Degree 4: Varyig degree: shared bus liear array, rig biary tree, fat tree, shuffle-echage two-dimesioal mesh (Illiac, torus) -cube, -dimesioal mesh, k-ary -cube

2 A measuremet uit, called diameter, ca be used to compare the relative performace characteristics of differet etworks. More specifically, the diameter of a etwork is defied as the largest miimum distace betwee ay pair of odes. The miimum distace betwee a pair of odes is the miimum umber of commuicatio liks (hops) that data from oe of the odes must traverse i order to reach the other ode. I the followig sectios, the listed static etworks are discussed i detail. Shared bus. The shared bus, also called commo bus, is the simplest type of static etwork. The shared bus has a degree of. I a shared bus architecture, all the odes share a commo commuicatio lik, as show i Figure 5.. The shared bus is the least epesive etwork to implemet. Also, odes (uits) ca be easily added or deleted from this etwork. However, it requires a mechaism for hadlig coflict whe several odes request the bus simultaeously. This mechaism ca be achieved through a bus cotroller, which gives access to the bus either o a first-come, first-served basis or through a priority scheme. (The structure of a bus cotroller is eplaied i the Chapter 6.) The shared bus has a diameter of sice each ode ca access the other odes through the shared bus. Figure 5. Shared bus. Liear array. The liear array (degree of 2) has each ode coected with two eighbors (ecept the fareds odes). The liear quality of this structure comes from the fact that the first ad last odes are ot coected, as illustrated i Figure 5.2. Although the liear array has a simple structure, its desig ca mea log commuicatio delays, especially betwee far-ed odes. This is because ay data eterig the etwork from oe ed must pass through a umber of odes i order to reach the other ed of the etwork. A liear array, with N odes, has a diameter of N-. Figure 5.2 Liear array. Rig. Aother etworkig cofiguratio with a simple desig is the rig structure. A rig etwork has a degree of 2. Similar to the liear array, each ode is coected to two of its eighbors, but i this case the first ad last odes are also coected to form a rig. Figure 5.3 shows a rig etwork. A rig ca be uidirectioal or bidirectioal. I a uidirectioal rig the data ca travel i oly oe directio, clockwise or couterclockwise. Such a rig has a diameter of N-, like the liear array. However, a bidirectioal rig, i which data travel i both directios, reduces the diameter by a factor of 2, or less if N is eve. A bidirectioal rig with N odes has a diameter of N / 2. Although this rig's diameter is much better tha that of the liear array, its cofiguratio ca still cause log commuicatio delays betwee distat odes for large N. A bidirectioal rig etwork s reliability, as compared to the liear array, is also improved. If a ode should fail, effectively cuttig off the coectio i oe directio, the other directio ca be used to complete a message trasmissio. Oce the coectio is lost betwee ay two adjacet odes, the rig becomes a liear array, however. Figure 5.3 Rig. Biary tree. Figure 5.4 represets the structure of a biary tree with seve odes. The top ode is called

3 the root, the four odes at the bottom are called leaf (or termial) odes, ad the rest of the odes are called itermediate odes. I such a etwork, each itermediate ode has two childre. The root has ode address. The addresses of the childre of a ode are obtaied by appedig 0 ad to the ode's address that is, the childre of ode are labeled 2 ad 2+. A biary tree with N odes has diameter 2(h-), where h log 2 N is the height of the tree. The biary tree has the advatages of beig epadable ad havig a simple implemetatio. Noetheless, it ca still cause log commuicatio delays betwee faraway leaf odes. Leaf odes farthest away from each other must ultimately pass their message through the root. Sice traffic icreases as the root is approached, leaf odes farthest away from each other will sped the most amout of time waitig for a message to traverse the tree from source to destiatio. Oe desirable characteristic for a itercoectio etwork is that data ca be routed betwee the odes i a simple maer (remember, a ode may represet a processor). The biary tree has a simple routig algorithm. Let a packet deote a uit of iformatio that a ode eeds to sed to aother ode. Each packet has a header that cotais routig iformatio, such as source address ad destiatio address. A packet is routed upward toward the root ode util it reaches a ode that is either the destiatio or acestor of the destiatio ode. If the curret ode is a acestor of the destiatio ode, the packet is routed dowward toward the destiatio. Figure 5.4 Biary tree. Fat tree. Oe problem with the biary tree is that there ca be heavy traffic toward the root ode. Cosider that the root ode acts as the sigle coectio poit betwee the left ad right subtrees. As ca be observed i Figure 5.4, all messages from odes N 2, N 4, ad N 5 to odes N 3, N 6, ad N 7 have o choice but to pass through the root. To reduce the effect of such a problem, the fat tree was proposed by Leiserso [LEI 85]. Fat trees are more like real trees i which the braches get thicker ear the truk. Proceedig up from the leaf odes of a fat tree to the root, the umber of commuicatio liks icreases, ad therefore the commuicatio badwidth icreases. The commuicatio badwidth of a itercoectio etwork is the epected umber of requests that ca be accepted per uit of time. The structure of the fat tree is based o a biary tree. Each edge of the biary tree correspods to two chaels of the fat tree. Oe of the chaels is from paret to child, ad the other is from child to paret. The umber of commuicatio liks i each chael icreases as we go up the tree from the leaves ad is determied by the amout of hardware available. For eample, Figure 5.5 represets a fat tree i which the umber of commuicatio liks i each chael is icreased by from oe level of the tree to the et. The fat tree ca be used to itercoect the processors of a geeral-purpose parallel machie. Sice its commuicatio badwidth ca be scaled idepedetly from the umber of processors, it provides great fleibility i desig. Figure 5.5 Fat tree.

4 Shuffle-echage. Aother method for establishig etworks is the shuffle-echage coectio. The shuffle-echage etwork is a combiatio of two fuctios: shuffle ad echage. Each is a simple bijectio fuctio i which each iput is mapped oto oe ad oly oe output. Let s - s s 0 be the biary represetatio of a ode address; the the shuffle fuctio ca be described as shuffle(s - s s 0 ) = s -2 s s 0 s -. For eample, usig the shuffle fuctio for N=8 (i.e. 2 3 odes) the followig coectios ca be established betwee the odes. Source Destiatio Source Destiatio The reaso that the fuctio is called shuffle is that it reflects the process of shufflig cards. Give that there are eight cards, the shuffle fuctio performs a perfect playig card shuffle as follows. First, the deck is cut i half, betwee cards 3 ad 4. The the two half decks are merged by selectig cards from each half i a alterative order. Figure 5.6 represets how the cards are shuffled. Figure 5.6 Card shufflig. Aother way to defie shuffle coectio is through the decimal represetatio of the addresses of the odes. Let N=2 be the umber of odes ad i represet the decimal address of a ode. For 0 i N / 2, ode i is coected to ode 2i. For N / 2 i N, ode i is coected to ode 2i+-N. The echage fuctio is also a simple bijectio fuctio. It maps a biary address to aother biary address that differs oly i the rightmost bit. It ca be described as echage(s - s s s 0 ) = s - s s s 0. Figure 5.7 shows the shuffle-echage coectios betwee odes whe N = 8. Figure 5.7 Shuffle-echage coectios.

5 The shuffle-echage etwork provides suitable itercoectio patters for implemetig certai parallel algorithms, such as polyomial evaluatio, fast Fourier trasform (FFT), sortig, ad matri traspositio [STO 7]. For eample, polyomial evaluatio ca be easily implemeted o a parallel machie i which the odes (processors) are coected through a shuffle-echage etwork. I geeral, a polyomial of degree N ca be represeted as a 0 + a + a a N- N- + a N N, where a 0, a,... a N are the coefficiets ad is a variable. As a eample, cosider the evaluatio of a polyomial of degree 7. Oe way to evaluate such a polyomial is to use the architecture give i Figure 5.7. I this figure, assume that each ode represets a processor havig three registers: oe to hold the coefficiet, oe to hold the variable, ad the third to hold a bit called the mask bit. Figure 5.8 illustrates the three registers of a ode. Figure 5.8 A shuffle-echage ode's registers. The evaluatio of the polyomial is doe i two phases. First, each term a i i is computed at ode i for i=0 to 7. The the terms a i i, for i= to 7, are added to produce the fial result. Figure 5.9 represets the steps ivolved i the computatio of a i i. Figure 5.9a shows the iitial values of the registers of each ode. The coefficiet a i, for i=0 to 7, is stored i ode i. The value of the variable is stored i each ode. The mask register of ode i, for i=, 3, 5, ad 7, is set to ; others are set to 0. I each step of computatio, every ode checks the cotet of its mask register. Whe the cotet of the mask register is, the cotet of the coefficiet register is multiplied with the cotet of the variable register, ad the result is stored i the coefficiet register. Whe the cotet of the mask register is zero, the cotet of the coefficiet register remais uchaged. The cotet of the variable register is multiplied with itself. The cotets of the mask registers are shuffled betwee the odes usig the shuffle etwork. Figures 5.9b, c, ad d show the values of the registers after the first step, secod step, ad third step, respectively. At the ed of the third step, each registers cotais a i i.

6 Figure 5.9 Steps for the computatio of the a i i. (a) Iitial values after step. (c) Values after step 2. (d) Values after step 3. At this poit, the terms a i i for i=0 to 7 are added to produce the fial result. To perform such a summatio, echage coectios are used i additio to shuffle coectios. Figure 5.0 shows all the coectios ad the iitial values of the coefficiet registers.

7 Figure 5.0 Required coectios for addig the terms a i i. I each step of computatio the cotets of the coefficiet registers are shuffled betwee the odes usig the shuffle coectios. The copies of the cotets of the coefficiet registers are echaged betwee the odes usig the echage coectios. After the echage is performed, each ode adds the cotet of its coefficiet register to the value that the copy of the curret cotet is echaged with. After three shuffle 7 ad echages, the cotet of each coefficiet register will be the desired ai i. The followig shows the three steps required to obtai result. i0 As you ca see i the chart, after the third step, the value ai i is stored i each coefficiet register. 7 i0

8 From this eample, it should be apparet that the shuffle-echage etwork provides the desired coectios for maipulatig the values of certai problems efficietly. Two-dimesioal mesh. A two-dimesioal mesh cosists of k * k 0 odes, where k i 2 deotes the umber of odes alog dimesio i. Figure 5. represets a two-dimesioal mesh for k 0 =4 ad k =2. There are four odes alog dimesio 0, ad two odes alog dimesio. As show i Figure 5., i a two-dimesioal mesh etwork each ode is coected to its orth, south, east, ad west eighbors. I geeral, a ode at row i ad colum j is coected to the odes at locatios (i-, j), (i+, j), (i, j-), ad (i, j+). The odes o the edge of the etwork have oly two or three immediate eighbors. The diameter of a mesh etwork is equal to the distace betwee odes at opposite corers. Thus, a twodimesioal mesh with k * k 0 odes has a diameter (k -) + (k 0 -). Figure 5. A two-dimesioal mesh with k 0 =4 ad k =2. I practice, two-dimesioal meshes with a equal umber of odes alog each dimesio are ofte used for coectig a set of processig odes. For this reaso i most literature the otio of two-dimesioal mesh is used without idicatig the values for k ad k 0 ; rather, the total umber of odes is defied. A twodimesioal mesh with k =k 0 = is usually referred to as a mesh with N odes, where N = 2. For eample, Figure 5.2 shows a mesh with 6 odes. From this poit forward, the term mesh will idicated a twodimesioal mesh with a equal umber of odes alog each dimesio. Figure 5.2 A two-dimesioal mesh with k 0 =k =4. The routig of data through a mesh ca be accomplished i a straightforward maer. The followig simple routig algorithm routes a packet from source S to destiatio D i a mesh with 2 odes.. Compute the row distace R as R D / S /. 2. Compute the colum distace C as C D(mod ) S(mod ). 3. Add the values R ad C to the packet header at the source ode. 4. Startig from the source, sed the packet for R rows ad the for C colums. The values R ad C determie the umber of rows ad colums that the packet eeds to travel. The directio the message takes at each ode is determied by the sig of the values R ad C. Whe R (C) is positive, the packet travels dowward (right); otherwise, the packet travels upward (left). Each time that the packet travels from oe ode to the adjacet ode dowward, the value R is decremeted by, ad whe it travels upward, R is icremeted by. Oce R becomes 0, the packet starts travelig i the horizotal

9 directio. Each time that the packet travels from oe ode to the adjacet ode i the right directio, the value C is decremeted by, ad whe it travels i the left directio, C is icremeted by. Whe C becomes 0, the packet has arrived at the destiatio. For eample, to route a packet from ode 6 (i.e., S=6) to ode 2 (i.e., D= 2), the packet goes through two paths, as show i Figure 5.3. I this eample, 2 / 4 6 / 4 2, R C Figure 5.3 Routig path from ode 6 to ode 2. It should be oted that i the case just described the odes o the edge of the mesh etwork have o coectios to their far eighbors. Whe there are such coectios, the etwork is called a wraparoud two-dimesioal mesh, or a Illiac etwork. A Illiac etwork is illustrated i Figure 5.4 for N = 6. Figure 5.4 A 6-ode Illiac etwork. I geeral, the coectios of a Illiac etwork ca be defied by the followig four fuctios: Illiac + (j) = j+ (mod N), Illiac - (j) = j- (mod N), Illiac + (j) = j+ (mod N), Illiac - (j) = j- (mod N), where N is the umber of odes, 0 j < N, is the umber of odes alog ay dimesio, ad N= 2. For eample, i Figure 5.4, ode 4 is coected to odes 5, 3, 8, ad 0, sice Illiac + (4) = ( 4 + ) (mod 6) = 5, Illiac - (4) = ( 4 - ) (mod 6) = 3, Illiac +4 (4) = ( ) (mod 6) = 8, Illiac -4 (4) = ( 4-4 ) (mod 6) = 0. The diameter of a Illiac with N= 2 odes is -, which is shorter tha a mesh. Although the etra wraparoud coectios i Illiac allow the diameter to decrease, they icrease the compleity of the desig.

10 Figure 5.5 shows the coectivity of the odes i a differet form. This graph shows that four odes ca be reached from ay ode i oe step, seve odes i two steps, ad four odes i three steps. I geeral, the umber of steps (recirculatios) to route data from a ode to ay other ode is upper bouded by the diameter (i.e., ). Figure 5.5 Alterative represetatio of a 6-ode Illiac etwork. To reduce the diameter of a mesh etwork, aother variatio of this etwork, called torus (or twodimesioal tours), has also bee proposed. As show i Figure 5.6a, a torus is a combiatio of rig ad mesh etworks. To make the wire legth betwee the adjacet odes equal, the torus may be folded as show i Figure 5.6b. I this way the commuicatio delay betwee the adjacet odes becomes equal. Note that both Figures 5.6a ad b provide the same coectios betwee the odes; i fact, Figure 5.6b is derived from Figure 5.6a by switchig the positio of the rightmost two colums ad the bottom two rows of odes. The diameter of a torus with N= 2 odes is 2 / 2, which is the distace betwee the corer ad the ceter ode. Note that the diameter is further decreased from the mesh etwork. Figure 5.6 Differet types of torus etwork. (a) A 4-by-4 torus etwork. (b) A 4-by-4 torus etwork with folded coectio.

11 The mesh etwork provides suitable itercoectio patters for problems whose solutios require the computatio of a set of values o a grid of poits, for which the value at each poit is determied based o the values of the eighborig poits. Here we cosider oe of these class of problems: the problem of fidig a steady-state temperature over the surface of a square slab of material whose four edges are held at differet temperatures. This problem requires the solutio of the followig partial differetial equatio, kow as Laplace's equatio: U / U / y 0, where U is the temperature at a give poit specified by the coordiates ad y o the slab. The followig describes a method, give by Slotick [SLO 7], to solve this problem. Eve if ufamiliar with Laplace's equatio, the reader should still be able to follow the descriptio. The method is based o the fact that the temperature at ay poit o the slab teds to become the average of the temperatures of eighborig poits. Assume that the slab is covered with a mesh ad that each square of the mesh has h uits o each side. The the temperature of a iterior ode at coordiates ad y is the average of the temperatures of the four eighbor odes. That is, the temperature at ode (, y), deoted as U(, y), equals the sum of the four eighborig temperatures divided by 4. For eample, as show i Figure 5.7, assume that the slab ca be covered with a 6-ode mesh. Here the value of U(, y) is epressed as U(,y)=[U(,y+h) + U(+h,y) + U(,y-h) + U(-h,y)]/4. Figure 5.7 Coverig a slab with a 6-ode mesh. Figure 5.8 illustrates a alterative represetatio of Figure 5.7. Here the positio of the odes is more coveietly idicated by the itegers i ad j. I this case, the temperature equatio ca be epressed as U(i,j)=[U(i,j+) + U(i+,j) + U(i,j-) + U(i-,j)]/4. Assume that each ode represets a processor havig oe register to hold the ode's temperature. The odes o the boudary are arbitrarily held at certai fied temperatures. Let the odes o the bottom of the mesh ad o the right edge be held at zero degrees. The odes alog the top ad left edges are set accordig to their positios. The temperatures of these 2 boudary odes do ot chage durig the computatio. The temperatures at the 4 iterior odes are the ukows. Iitially, the temperatures at these 4 odes are set to zero. I the first iteratio of computatio, the 4 iterior ode processors simultaeously calculate the ew temperature values usig the values iitially give.

12 Figure 5.8 Iitial values of the odes. Figure 5.9 represets the ew values of the iterior odes after the first iteratio. These values are calculated as follows: U(,2)=[U(,3)+U(2,2)+U(,)+U(0,2)]/4 = [ ]/4 = 4; U(2,2)=[U(2,3)+U(3,2)+U(2,)+U(,2)]/4 = [ ]/4 = ; U(,)=[U(,2)+U(2,)+U(,0)+U(0,)]/4 = [ ]/4 = ; U(2,)=[U(2,2)+U(3,)+U(2,0)+U(,)]/4 = [ ]/4 = 0. I the secod iteratio, the values of U(,2), U(2,2), U(,), ad U(2,) are calculated usig the ew values just obtaied: U(,2) = [8+++8]/4 = 4.5; U(2,2) = [ ]/4 = 2; U(,) = [ ]/4 = 2; U(2,) = [+0+0+]/4 = 0.5. This process cotiues util a steady-state solutio is obtaied. As more iteratios are performed, the values of the iterior odes coverge to the eact solutio. Whe values for two successive iteratios are close to each other (withi a specified error tolerace), the process ca be stopped, ad it ca be said that a steady-state solutio has bee reached. Figure 5.20 represets a solutio obtaied after iteratios. Figure 5.9 Values of the odes after the first iteratio.

13 Figure 5.20 Values of the odes after the eleveth iteratio. -cube or hypercube. A -cube etwork, also called hypercube, cosists of N=2 odes; is called the dimesio of the -cube etwork. Whe the ode addresses are cosidered as the corers of a - dimesioal cube, the etwork coects each ode to its eighbors. I a -cube, idividual odes are uiquely idetified by -bit addresses ragig from 0 to N-. Give a ode with biary address d, this ode is coected to all odes whose biary addresses differ from d i eactly bit. For eample, i a 3- cube, i which there are eight odes, ode 7 () is coected to odes 6 (0), 5 (0), ad 3 (0). Figure 5.2 demostrates all the coectios betwee the odes. Figure 5.2 A three-dimesioal cube. As ca be see i the 3-cube, two odes are directly coected if their biary addresses differ by bit. This method of coectio is used to cotrol the routig of data through the etwork i a simple maer. The followig simple routig algorithm routes a packet from its source S = (s -... s 0 ) to destiatio D = (d -... d 0 ).. Tag T S D t -... t 0 is added to the packet header at the source ode ( deotes a XOR gate). 2. If t i 0 for some 0 i, the use ith-dimesio lik to sed the packet to a ew ode with the same address as the curret ode ecept the ith bit, ad chage t i to 0 i the packet header. 3. Repeat step 2 util t i = 0 for all 0 i. For eample, as show i Figure 5.22, to route a packet from ode 0 to ode 5, the packet could go through two differet paths, P ad P 2. Here T=000 0 = 0. If we first cosider the bit t 0 ad the t 2, the packet goes through the path P. Sice t 0 =, the packet is set through the 0th-dimesio lik to ode.

14 At ode, t 0 is set to 0; thus T ow becomes equal to 00. Now, sice t 2 =, the packet is set through the secod-dimesio lik to ode 5. If, istead of t 0, bit t 2 is cosidered first, the packet goes through P 2. Figure 5.22 Differet paths for routig a packet from ode 0 to ode 5. I the etwork of Figure 5.22, the maimum distace betwee odes is 3. This is because the distace betwee odes is equal to the umber of bit positios i which their biary addresses differ. Sice each address cosists of 3 bits, the differece betwee two addresses ca be at most 3 whe every bit at the same positio differs. I geeral, i a -cube the maimum distace betwee odes is, makig the diameter equal to. The -cube etwork has several features that make it very attractive for parallel computatio. It appears the same from every ode, ad o ode eeds special treatmet. It also provides disjoit paths betwee a source ad a destiatio. Let the source be represeted as S = (s - s s 0 ) ad the destiatio by D = (d - d d 0 ). The shortest paths ca be symbolically represeted as Path : s - s s 0 s - s d 0 s - s d d 0 d - d d 0 Path 2: s - s s 0 s - s d s 0 s - s d 2 d s 0 d - d d s 0 d - d d d 0.. Path : s - s s 0 d - s s 0 d - s s d 0 d - s d d 0 d - d d d 0 For eample, cosider the 3-cube of Figure 5.2. Sice =3, there are three paths from a source, say 000, to a destiatio, say. The paths are path : ; path 2: ; path 3: This ability to have alterative paths betwee ay two odes makes the -cube etwork highly reliable if ay oe (or more) paths become uusable. Differet etworks, such as two-dimesioal meshes ad trees, ca be embedded i a -cube i such a way that the coectivity betwee eighborig odes remais cosistet with their defiitio. Figure 5.23 shows how a 4-by-4 mesh ca be embedded i a 4-cube (four-dimesioal hypercube). The 4-cube s itegrity is ot compromised ad is well-suited for uses like this, where a great deal of fleibility is required. All defiitioal cosideratios for both the 4-cube ad the 4-by-4 mesh, as stated earlier, are cosistet.

15 Figure 5.23 Embeddig a 4-by-4 mesh i a 4-cube. The itercoectio supported by the -cube provides a atural eviromet for implemetig highly parallel algorithms, such as sortig, mergig, fast Fourier trasform (FFT), ad matri operatios. For eample, Batcher's bitoic merge algorithm ca easily be implemeted o a -cube. This algorithm sorts a bitoic sequece (a bitoic sequece is a sequece of odecreasig umbers followed by a sequece of oicreasig umbers). Figure 5.24 presets the steps ivolved i mergig a odecreasig sequece [0,4,6,9] ad a oicreasig sequece [8,5,3,]. This algorithm performs a sequece of comparisos o pairs of data that are successively 2 2, 2, ad 2 0 locatios apart. Figure 5.24 Mergig two sorted lists of data. At each stage of the merge each pair of data elemets is compared ad switched if they are ot i ascedig order. This rearragig cotiues util the fial merge with a distace of puts the data ito ascedig order. Figure 5.24 requires the followig coectios betwee odes: Node 0 should be coected to odes:,2,4; Node should be coected to odes: 0,3,5;

16 Node 2 should be coected to odes: 0,3,6; Node 3 should be coected to odes:,2,7; Node 4 should be coected to odes: 0,5,6; Node 5 should be coected to odes:,4,7; Node 6 should be coected to odes: 2,4,7; Node 7 should be coected to odes: 3,5,6. These are eactly the same as 3-cube coectios. That is, the -cube provides the ecessary coectios for the Batcher's algorithm. Thus, applyig Batcher's algorithm to a -cube etwork is straightforward. I geeral, the -cube provides the ecessary coectios for ascedig ad descedig classes of parallel algorithms. To defie each of these classes, assume that there are 2 iput data items stored i 2 locatios (or processors) 0,, 2,..., 2 -. A algorithm is said to be i the descedig class if it performs a sequece of basic operatios o pairs of data that are successively 2 -, 2-2,..., ad 2 0 locatios apart. (Therefore, Batcher's algorithm belogs to this class.) I compariso, a ascedig algorithm performs successively o pairs that are 2 0, 2,..., ad 2 - locatios apart. Whe =3 Figures 5.25 ad 5.26 show the required coectios for each stage of operatio i this class of algorithms. As show, the -cube is able to efficietly implemet algorithms i descedig or ascedig classes. Figure 5.25 Descedig class. Figure 5.26 Ascedig class. Although the -cube ca implemet this class of algorithms i parallel steps, it requires coectios for each ode, which makes the desig ad epasio difficult. I other words, the -cube provides poor scalability ad has a iefficiet structure for packagig ad therefore does ot facilitate the icreasigly importat property of modular desig. -Dimesioal mesh. A -dimesioal mesh cosists of k -* k -2*... * k 0 odes, where k i 2 deotes the umber of odes alog dimesio i. Each ode X is idetified by coordiates -, -2,..., 0, where 0 i k i - for 0 i -. Two odes X=( -, -2,..., 0 ) ad Y=(y -,y -2,...,y 0 ) are said to be eighbors if ad oly if y i = i for all i, 0 i -, ecept oe, j, where y j = j + or y j = j -. That is, a ode may have from to 2 eighbors, depedig o its locatio i the mesh. The corers of the mesh have eighbors, ad the iteral odes have 2 eighbors, while other odes have b eighbors, where < b <2. The diameter of a -dimesioal mesh is ( k ). A -cube is a special case of -dimesioal meshes; it is i fact a - i0 i dimesioal mesh i which k i =2 for 0 i -. Figure 5.27 represets the structure of two three-

17 dimesioal meshes: oe for k 2 = k = k 0 = 3 ad the other for k 2 =4, k =3, ad k 0 =2. (a) k 2 = k = k 0 = 3. (b) k 2 =4, k =3, ad k 0 =2. Figure 5.27 Three-dimesioal meshes. k-ary -cube. A k-ary -cube cosists of k odes such that there are k odes alog each dimesio. Each ode X is idetified by coordiates, -, -2,..., 0, where 0 i k - for 0 i -. Two odes X=( -, -2,..., 0 ) ad Y=(y -,y -2,...,y 0 ) are said to be eighbors if ad oly if y i = i for all i, 0 i -, ecept oe, j, where y j =( j + ) mod k, or y j =( j -) mod k. That is, i cotrast to the -dimesioal mesh, a k-ary - cube has a symmetrical topology i which each ode has a equal umber of eighbors. A ode has eighbors whe k=2 ad 2 eighbors whe k>2. The k-ary -cube has a diameter of k / 2. A -cube is a special case of k-ary -cubes; it is i fact a 2-ary -cube. Figure 5.28 represets the structure of two k- ary -cubes: oe for k=4, =2 ad the other for k==3. Note that a 4-ary 2-cube is actually a torus etwork. Figure 5.28 (a) 4-Ary 2-cube ad (b) 3-ary 3-cube etworks.

18 Routig i -dimesioal meshes ad k-ary -cubes. Oe of the routig algorithms that ca be used for routig the packets withi a -dimesioal mesh or a k-ary -cube is called store-ad-forward routig [TAN 8]. Each ode of the etwork cotais a buffer equal to the size of a packet. I store-ad-forward routig, a packet is trasmitted from a source ode to a destiatio ode through a sequece of itermediate odes. Each itermediate ode of the etwork receives a packet i its etirety before trasmittig it to the et ode. Whe a itermediate ode receives a packet, it first stores the packet i its buffer; the it forwards the packet to the et ode whe the receivig ode's buffer is empty. Store-ad-forward routig is easy to uderstad ad simple to implemet. However, it requires a trasmissio time proportioal to the distace (the umber of hops or chaels) betwee the source ad the destiatio odes to deliver a packet. (Chaels are actually electrical coectios betwee odes ad are arraged based o the etwork topology.) To reduce trasmissio time ad make this task almost idepedet of the distace betwee source ad destiatio odes, a hardware-supported routig protocol, called wormhole routig (also direct-coect routig ) is ofte used. I wormhole routig, a packet is divided ito several smaller data uits, called flits. Oly the first flit (the leadig flit) carries the routig iformatio (such as the destiatio address), ad the remaiig flits follow this leader. Oce a leader flit arrives at a ode, the ode selects the outgoig chael based o the flit's routig iformatio ad begis forwardig the flits through that chael. Sice the remaiig flits carry o routig iformatio, they must ecessarily follow the chaels established by the header for the trasmissio to be successful. Therefore, they caot be iterleaved (alterated or mied) with the flits of other packets. Whe a leader flit arrives at a ode that has o output chael available, all the flits remai i their curret positio util a suitable chael becomes available. Each ode cotais a few small buffers for storig such flits. At each ode, the selectio of a outgoig chael for a particular leadig flit depeds o the icomig chael (the chael that was used by the flit to eter the ode) ad the destiatio ode. This type of depedecy ca be represeted by a chael depedecy graph. A chael depedecy graph for a give itercoectio etwork together with a routig algorithm is a directed graph such as show i Figure 5.29b. The vertices of the graph i Figure 5.29b are the chaels of the etwork, ad the edges are the pairs of chaels coected by the routig algorithm. For eample, cosider Figure 5.29a, where a etwork with odes, 0,..., ad 00 ad uidirectioal chaels c, c 0,..., ad c 00, is show. The chaels are labeled by the idetificatio umber (id) of their source ode. A routig algorithm for such a etwork could advace the flits o c to c 0, o c 0 to c 0, ad so o. Based o this routig algorithm, Figure 5.29b represets the depedecy graph for such a etwork. Notice that the depedecy graph cosists of a cycle that may cause a deadlock i the etwork. A deadlock ca occur wheever o flits ca proceed toward their destiatios because the buffers o the route are full. Figure 5.29c presets a deadlock cofiguratio i the case whe there are two buffers i each ode.

19 Figure 5.29 A simple etwork with four odes. (a) Network. (b) Depedecy graph. (c) Deadlock. To have reliable ad efficiet commuicatio betwee odes, a deadlock-free routig algorithm is eeded. Dally ad Seitz [DAL 87] have show that a routig algorithm for a itercoectio etwork is deadlock free if ad oly if there are o cycles (a route that recoects with itself) i the chael depedecy graph. Their proposal is to avoid deadlock by elimiatig cycles through the use of virtual chaels. A virtual chael is a logical lik betwee two odes formed by a physical chael ad a flit buffer i each of the two odes. Each physical chael is shared amog a group of virtual chaels. Although several virtual chaels share a physical chael, each virtual chael has its ow buffer. With may (virtual) chaels to choose from, cycles, ad therefore deadlock, ca be avoided. Figure 5.30a represets the virtual chaels for a etwork whe each physical chael is split ito two virtual chaels: lower virtual chaels ad upper virtual chaels. The lower virtual chael of c (where is idetified as the source ode) is labeled c 0, ad the upper virtual chael is labeled c. For eample, the lower virtual chael of c is umbered as c 0. Dally ad Seitz's routig algorithm routes packets at a ode with a label value less tha the destiatio ode o the upper virtual chaels ad routes packets at a ode labeled greater tha their destiatio ode o the lower chaels. This routig algorithm restricts the packets' routig to the order of decreasig virtual chael labels. Thus there is o cycle i the depedecy graph ad the etwork is deadlock free (see Figure 5.30b).

20 Figure 5.30 (a) Virtual chaels ad (b) depedecy graph for a simple etwork with four odes. Wormhole routig is based o a method of dividig packets ito smaller trasmissio uits called flits. Trasmittig flits rather tha packets reduces the average time required to deliver a packet i the etwork, as show i Figure 5.3. Figure 5.3 Comparig (a) store-ad-forward routig with (b) wormhole routig. T sf ad T wh are average trasmissio time over three chaels whe usig store-ad-forward routig ad wormhole routig, respectively. For eample, assume that each packet cosists of q flits, ad T f is the amout of time required for each flit to be trasmitted across a sigle chael. The amout of time required to trasmit a packet over a sigle chael is therefore q * T f. With store-ad-forward routig, the average time required to trasmit a packet over D chaels will be D * q * T f. However, with wormhole routig, i which the flits are forwarded i a

21 pipelie fashio, the average trasmissio time over D chaels becomes (q+d-) * T f. This meas that wormhole routig is much faster tha store-ad-forward routig. Furthermore, wormhole routig requires very little storage, resultig i a small ad fast commuicatio cotroller. I geeral, it is a efficiet routig techique for k-ary -cubes ad -dimesioal meshes. I literature, several deadlock-free routig algorithms, based o the wormhole routig cocept, have bee proposed. These algorithms ca be classified ito two groups: determiistic (or static) routig ad adaptive (or dyamic) routig. I determiistic routig, the routig path, which is traversed by flits, is fied ad is determied by the source ad destiatio addresses. Although, these routigs usually select oe of the shortest paths betwee the source ad destiatio odes, they limit the ability of the itercoectio etwork to adapt itself to failures or heavy traffic (cogestio) alog the iteded routes. It is i this case that adaptive routig becomes importat. Adaptive routig algorithms allow the path take by flits to deped o dyamic etwork coditios (such as the presece of faulty or cogested chaels), rather tha source ad destiatio addresses. The descriptio of these algorithms is beyod the scope of this book. The reader ca refer to [NI 93] for a survey o determiistic ad adaptive wormhole routig i k-ary -cubes ad -dimesioal meshes. Network Latecy. Here, based o the work of Agarwal [AGA 9] ad Dally ad Seitz[DAL 87], we focus o derivig a equatio for the average time required to trasmit a packet i k-ary -cubes that uses wormhole routig. A similar aalysis ca also be carried out for -dimetioal etworks. We assume that the etworks are embedded i a plae ad have uidirectioal chaels. The etwork latecy, T b, refers to the elapsed time from the time that the first flit of the packet leaves the source to the time the last flit arrives at the destiatio. Hece, igorig the etwork load, T b ca be epressed as T b = (q+d-) * T f, where D deotes the umber of chaels (hops) that a packet traverses. Let T f be represeted as the sum of the wire delay T w () ad ode delay T s, that is, T f = T w () + T s. Hece, T b = (q+d-)[ T w () + T s ]. The umber of chaels, D, ca be determied by the product of the etwork dimesio ad the average distace (k d ) that a packet must travel i each dimesio of the etwork. Assumig that the packet destiatios are radomly chose, the average distace a packet must travel is give by Hece k d =(k-)/2. T b = [q + (k-)/2 -][ T w () + T s ]. To determie T w (), we must fid the legth of the logest wire of a -dimesioal etwork embedded i a plae. The embeddig of a -dimesioal etwork i a plae ca be achieved by mappig /2 dimesios of the etwork i each of the two physical dimesios. That is, the umber of odes i each physical dimesio is k /2. Thus each additioal dimesio of the etwork icreases the umber of odes i each physical dimesio by k /2. Assumig that the distace betwee the physically adjacet odes remais fied, each additioal dimesio also icreases the legth of the logest wire legth by a factor of k /2. Assume that the wire delay depeds liearly o the wire legth. If we cosider the delay of a wire i a twodimesioal etwork [i.e., T w (2)] as a base time period, the delay of the logest wire is give by Hece T w () = (k /2 ) -2 T w (2) = k (/2 - ).

22 T b = [q + (k-)/2 -][ k (/2 - ) +T s ]. Agarwal [AGA 9] has eteded this result to the aalysis of a k-ary -cube uder differet load parameters, such as packet size, degree of local commuicatio, ad etwork request rate. [The degree of local commuicatio icreases as the probability of commuicatio with (or access to) various odes decreases as a fuctio of physical distace.] Agarwal has show that two-dimesioal etworks have the lowest latecy whe ode delays ad etwork cotetio are igored. Otherwise, three or four dimesios are preferred. However, whe the degree of local commuicatios becomes high, two-dimesioal etworks outperform three- ad four-dimesioal etworks. Local commuicatio depeds o several factors, such as machie architecture, type of applicatios, ad compiler. If these factors are ehaced, twodimesioal etworks ca be used without icurrig the high cost of higher dimesios. Aother alterative for ehacig local commuicatio is to provide short paths for olocal packets. The k-ary -cube etwork ca be augmeted by oe or more levels of epress chaels that allow olocal messages to bypass odes [DAL 9]. The augmeted etwork, called epress cube, reduces the etwork diameter ad icreases the wire legth. This arragemet allows the etwork to operate with latecies that approach the physical speed-of-light limitatio, rather tha beig limited by ode delays. Figure 5.32 illustrates the additio of epress chaels to a k-ary -cube etwork. I epress cubes the wire legth of epress chaels ca be icreased to the poit that wire delays domiate ode delay, makig lowdimesioal etworks more attractive. Figure 5.32 Epress cube. (a) Regular k-ary -cube etwork. (b) k-ary -cube etwork with epress chaels Dyamic Networks Dyamic etworks provide recofigurable coectios betwee odes. The topology of a dyamic etwork is the physical structure of the etwork as determied by the switch boes ad the itercoectig liks. Sice the switch bo is the basic compoet of the etwork, the cost of the etwork (i hardware terms) is measured by the umber of switch boes required. Therefore, the topology of the etwork is the prime determiat of the cost. To clarify the precedig termiology, let us cosider the desig of a dyamic etwork usig simple switch boes. Figure 5.33 represets a simple switch with two iputs ( ad y) ad two outputs (z 0 ad z ). A cotrol lie, s, determies whether the iput lies should be coected to the output lies i straight state or echage state. For eample, whe the cotrol lie s=0, the iputs are coected to the outputs i a straight state; that is, is coected to z 0 ad y is coected to z. Whe the cotrol lie s=, the iputs are coected to outputs i a echage state; that is, is coected to z ad y is coected to z 0.

23 Figure 5.33 A simple two-iput switch. Now let's use this switch to desig a etwork that ca coect a source to oe of eight possible destiatios 0 to 7. A solutio for such a etwork is show i Figure I this desig, there are three stages (colums), stages 2,, ad 0. The destiatio address is deoted bit-wise d 2 d d 0. The switch i stage 2 is cotrolled by the most sigificat bit of the destiatio address (i.e., d 2 ). This bit is used because, whe d 2 =0, the source is coected to oe of the destiatios 0 to 3 (000 to 0); otherwise, is coected to oe of the destiatios 4 to 7 (00 to ). I a similar way, the switches i stages ad 0 are cotrolled by d ad d 0, respectively. Figure 5.34 A simple -to-8 itercoectio etwork. Now let's epad our etwork to have eight sources istead of oe. Figure 5.35 represets a solutio to such a etwork costructed i the same maer as the desig i Figure 5.34.

24 Figure 5.35 A simple 8-to-8 itercoectio etwork. Note that, i this etwork, the destiatio address bits caot be used to cotrol switches for some coectios, such as, coectig source to destiatio 5. Therefore, at this poit, let s assume there is some kid of mechaism for cotrollig switches. Based o this assumptio, the etwork is able to coect ay sigle source to ay sigle destiatio. However, it is ot able to establish certai coectios with multiple sources ad multiple destiatios. Describig such multiple coectios requires the use of the term permutatio. A permutatio refers to the coectio of a set of sources to a set of destiatios such that each source is coected to a sigle destiatio. A permutatio [(s 0,d 0 ),(s,d ),...,(s 7,d 7 )] meas that source s 0 is coected to d 0, s to d, ad so o. The etwork of Figure 5.35 caot establish particular permutatios. For eample, a permutatio that requires sources 0 ad to be coected to destiatios 0 ad, respectively, caot be established at the same time. However, by chagig the positio of some of the switches, such a permutatio becomes possible. Figure 5.36 represets the same etwork after switchig the positio of the coectios of iputs ad 4 to the switches 0 ad 2 of stage 2. This ew etwork is able to coect 0 to 0 ad to, simultaeously, ad establish the permutatio. Nevertheless, there are may permutatios, such as [(0,),(,2),(2,3),(3,4),(4,5),(5,6),(6,7),(7,0)], that caot be established by this ew etwork. Later i this chapter, better etworks that ca provide the ecessary permutatios for may applicatios are represeted. Figure 5.36 A alterative desig for a 8-to-8 itercoectio etwork. To provide a perspective o the various dyamic etwork topologies ad to aid i orgaizig the later sectios, a dyamic etworks taoomy is preseted i Figure At the first level of the hierarchy are the crossbar switch, sigle-stage, ad multistage etworks.

25 Figure 5.37 Classificatio of dyamic etworks. The crossbar switch ca be used for coectig a set of iput odes to a set of output odes. I this etwork every iput ode ca be coected to ay output ode. The crossbar switch provides all possible permutatios, as well as support for high system performace. It ca be viewed as a umber of vertical ad horizotal liks itercoected by a switch at each itersectio. Figure 5.38 represets a crossbar for coectig N odes to N odes. The coectio betwee each pair of odes is established by a crosspoit switch. The crosspoit switch ca be set o or off i respose to applicatio eeds. There are N 2 crosspoit switches for providig complete coectios betwee all the odes. The crossbar switch is a ideal etwork to use for small N. However, for large N, the implemetatio of the crosspoit switches makes this desig comple ad epesive ad thus less attractive to use. Figure 5.38 Crossbar switch. Sigle-stage etworks, also called recirculatig etworks, require routig algorithms to direct the flow of data several times through the etwork so that various coectios ad permutatios ca be costructed. Each time that the data traverse the etwork is called a pass. As a eample, Figure 5.39 represets a sigle-stage etwork based o the shuffle-echage coectio. Multistage etworks, such as the oe i Figure 5.36, are more comple from a hardware poit of view, but the routig of data is made simpler by virtue of permaet coectios betwee the stages of the etwork. Because there are more switches i a multistage etwork, the umber of possible permutatios o a sigle pass icreases; however, there is a higher ivestmet i hardware. There is also a possible reductio i the compleity of routig fuctios ad the time it takes to geerate the ecessary permutatios.

26 Figure 5.39 A sigle-stage etwork. Multistage etworks are further divided ito cocetrators ad coectors. Both of these techologies were established i the 950s by Bell Labs. A cocetrator itercoects a specific idle iput to a arbitrary idle output. Oe way to specify a cocetrator is by a triplet of itegers (I, O, C), where I>O C, ad where I is the umber of iputs, O is the umber of outputs, ad C is the capacity of the cocetrator. The capacity of a cocetrator is the maimum umber of coectios that ca be made simultaeously I through the etwork. Thus a cocetrator (I, O, C) is capable of itercoectig ay of the choices of K I I! iputs (K C) to some K of the outputs, where. For eample, Figure 5.40 represets K K!( I K)! a (6,4,4) cocetrator, called Masso's biomial cocetrator. I this etwork, the crosspoit switches 4 coectig the iputs to the outputs cosist of all the possible choices of two switches per iput lie. 2 There are si possible differet matchigs betwee si iput ad four output lies with two switches per iput lie. Ofte cocetrators are used for coectig several termials to a computer. Figure 5.40 A cocetrator with si iputs ad four outputs. A coector establishes a path from a specific iput to a specific output. I geeral, coector etworks ca be grouped ito three differet classes: oblockig, rearrageable, ad blockig etworks. I a oblockig etwork, it is always possible to coect a idle pair of termials (iput/output odes) without

27 disturbig coectios (calls) already i progress. This is called "oblockig i the strict sese" simply because such a etwork has o blockig states whatsoever. These type of etworks are said to be uiversal etworks sice they ca provide all possible permutatios. The rearrageable etworks are also uiversal etworks; however, i this type of etwork it may ot always be possible to coect a idle pair of termials without disturbig established coectios. I a rearrageable etwork, give ay set of coectios i progress ad ay pair of idle termials, the eistig coectios ca be reassiged ew routes (if ecessary) so as to make it possible to coect the idle pair at ay time. I cotrast, i a blockig etwork, depedig o what state the etwork may be i, it may ot be possible to coect a idle pair of termials i ay way. For each group of coectors, a class of dyamic etworks is show i the followig discussio. Noblockig etworks. Clos has proposed a class of etworks with iterestig properties [CLO 53]. Figure 5.4 shows oe eample of such a etwork. This particular etwork is called a three-stage Clos etwork. It cosists of a iput stage of m crossbar switches, a output stage of m crossbar switches, ad a middle stage of r r crossbar switches. This class of etworks is deoted by the triple N(m,,r), which determies the switches' dimesios. Figure 5.4 A oblockig etwork Clos has show that for m 2-, the etwork N(m,,r), is a oblockig etwork. For eample, the etwork N(3,2,2) i Figure 5.42 is a oblockig etwork. This etwork requires 2 crosspoit switches i every stage, or 36 switches i all. Note that a crossbar switch with the same umber of iputs ad outputs (i.e., 4) requires 6 switches. Thus i this case it is more ecoomical to desig a crossbar switch tha a Clos etwork. However, whe the umber of iputs, N, icreases, the umber of switches becomes much less tha N 2, as i the case of the crossbar. For eample, for N=36, oly 88 switches are ecessary i a Clos etwork, whereas i the case of a crossbar etwork 36 2, or 296, switches are required. Figure 5.42 A Clos etwork with =2, r=2, ad m=3. There should be at least 2- switches i the middle stage of the Clos etwork i order to become a oblockig etwork. To demostrate the ecessity of this coditio, let's cosider the followig eample. Figure 5.43 represets a sectio of a Clos etwork i which each iput (output) switch has three iputs