5 Interconnection Networks

Transcription

1 5 Itercoectio Networks 5. INTRODUCTION Networkig strategy was origially employed i the 950's by the telephoe idustry as a meas of reducig the time required for a call to go through. Similarly, the computer idustry employs etworkig strategy to provide fast commuicatio betwee computer subparts, particularly with regard to parallel machies. The performace requiremets of may applicatios, such as weather predictio, sigal processig, radar trackig, ad image processig, far eceed the capabilities of sigle-processor architectures. Parallel machies break a sigle problem dow ito parallel tasks that are performed cocurretly, reducig sigificatly the applicatio processig time. Ay parallel system that employs more tha oe processor per applicatio program must be desiged to allow its processors to commuicate efficietly; otherwise, the advatages of parallel processig may be egated by iefficiet commuicatio. This fact emphasizes the importace of itercoectio etworks to overall parallel system performace. I may proposed or eistig parallel processig architectures, a itercoectio etwork is used to realize trasportatio of data betwee processors or betwee processors ad memory modules. This chapter deals with several aspects of the etworks used i moder (ad theoretical) computers. After classifyig various etwork structures, some of the most well kow etworks are discussed, alog with a list of advatages ad disadvatages associated with their use. Some of the elemets of etwork desig are also eplored to give the reader a uderstadig of the compleity of such desigs. 5.2 NETWORK TOPOLOGY Network topology refers to the layouts of liks ad switch boes that establish itercoectios. The liks are essetially physical wires (or chaels); the switch boes are devices that coect a set of iput liks to a set of output liks. There are two groups of etwork topologies: static ad dyamic. Static etworks provide fied coectios betwee odes. (A ode ca be a processig uit, a memory module, a I/O module, or ay combiatio thereof.) With a static etwork, liks betwee odes are uchageable ad caot be easily recofigured. Dyamic etworks provide recofigurable coectios betwee odes. The switch bo is the basic compoet of the dyamic etwork. With a dyamic etwork the coectios betwee odes are established by the settig of a set of itercoected switch boes. I the followig sectios, eamples of static ad dyamic etworks are discussed i detail Static Networks There are various types of static etworks, all of which are characterized by their ode degree; ode degree is the umber of liks (edges) coected to the ode. Some well-kow static etworks are the followig: Degree : Degree 2: Degree 3: Degree 4: Varyig degree: shared bus liear array, rig biary tree, fat tree, shuffle-echage two-dimesioal mesh (Illiac, torus) -cube, -dimesioal mesh, k-ary -cube

2 A measuremet uit, called diameter, ca be used to compare the relative performace characteristics of differet etworks. More specifically, the diameter of a etwork is defied as the largest miimum distace betwee ay pair of odes. The miimum distace betwee a pair of odes is the miimum umber of commuicatio liks (hops) that data from oe of the odes must traverse i order to reach the other ode. I the followig sectios, the listed static etworks are discussed i detail. Shared bus. The shared bus, also called commo bus, is the simplest type of static etwork. The shared bus has a degree of. I a shared bus architecture, all the odes share a commo commuicatio lik, as show i Figure 5.. The shared bus is the least epesive etwork to implemet. Also, odes (uits) ca be easily added or deleted from this etwork. However, it requires a mechaism for hadlig coflict whe several odes request the bus simultaeously. This mechaism ca be achieved through a bus cotroller, which gives access to the bus either o a first-come, first-served basis or through a priority scheme. (The structure of a bus cotroller is eplaied i the Chapter 6.) The shared bus has a diameter of sice each ode ca access the other odes through the shared bus. Figure 5. Shared bus. Liear array. The liear array (degree of 2) has each ode coected with two eighbors (ecept the fareds odes). The liear quality of this structure comes from the fact that the first ad last odes are ot coected, as illustrated i Figure 5.2. Although the liear array has a simple structure, its desig ca mea log commuicatio delays, especially betwee far-ed odes. This is because ay data eterig the etwork from oe ed must pass through a umber of odes i order to reach the other ed of the etwork. A liear array, with N odes, has a diameter of N-. Figure 5.2 Liear array. Rig. Aother etworkig cofiguratio with a simple desig is the rig structure. A rig etwork has a degree of 2. Similar to the liear array, each ode is coected to two of its eighbors, but i this case the first ad last odes are also coected to form a rig. Figure 5.3 shows a rig etwork. A rig ca be uidirectioal or bidirectioal. I a uidirectioal rig the data ca travel i oly oe directio, clockwise or couterclockwise. Such a rig has a diameter of N-, like the liear array. However, a bidirectioal rig, i which data travel i both directios, reduces the diameter by a factor of 2, or less if N is eve. A bidirectioal rig with N odes has a diameter of N / 2. Although this rig's diameter is much better tha that of the liear array, its cofiguratio ca still cause log commuicatio delays betwee distat odes for large N. A bidirectioal rig etwork s reliability, as compared to the liear array, is also improved. If a ode should fail, effectively cuttig off the coectio i oe directio, the other directio ca be used to complete a message trasmissio. Oce the coectio is lost betwee ay two adjacet odes, the rig becomes a liear array, however. Figure 5.3 Rig. Biary tree. Figure 5.4 represets the structure of a biary tree with seve odes. The top ode is called

3 the root, the four odes at the bottom are called leaf (or termial) odes, ad the rest of the odes are called itermediate odes. I such a etwork, each itermediate ode has two childre. The root has ode address. The addresses of the childre of a ode are obtaied by appedig 0 ad to the ode's address that is, the childre of ode are labeled 2 ad 2+. A biary tree with N odes has diameter 2(h-), where h log 2 N is the height of the tree. The biary tree has the advatages of beig epadable ad havig a simple implemetatio. Noetheless, it ca still cause log commuicatio delays betwee faraway leaf odes. Leaf odes farthest away from each other must ultimately pass their message through the root. Sice traffic icreases as the root is approached, leaf odes farthest away from each other will sped the most amout of time waitig for a message to traverse the tree from source to destiatio. Oe desirable characteristic for a itercoectio etwork is that data ca be routed betwee the odes i a simple maer (remember, a ode may represet a processor). The biary tree has a simple routig algorithm. Let a packet deote a uit of iformatio that a ode eeds to sed to aother ode. Each packet has a header that cotais routig iformatio, such as source address ad destiatio address. A packet is routed upward toward the root ode util it reaches a ode that is either the destiatio or acestor of the destiatio ode. If the curret ode is a acestor of the destiatio ode, the packet is routed dowward toward the destiatio. Figure 5.4 Biary tree. Fat tree. Oe problem with the biary tree is that there ca be heavy traffic toward the root ode. Cosider that the root ode acts as the sigle coectio poit betwee the left ad right subtrees. As ca be observed i Figure 5.4, all messages from odes N 2, N 4, ad N 5 to odes N 3, N 6, ad N 7 have o choice but to pass through the root. To reduce the effect of such a problem, the fat tree was proposed by Leiserso [LEI 85]. Fat trees are more like real trees i which the braches get thicker ear the truk. Proceedig up from the leaf odes of a fat tree to the root, the umber of commuicatio liks icreases, ad therefore the commuicatio badwidth icreases. The commuicatio badwidth of a itercoectio etwork is the epected umber of requests that ca be accepted per uit of time. The structure of the fat tree is based o a biary tree. Each edge of the biary tree correspods to two chaels of the fat tree. Oe of the chaels is from paret to child, ad the other is from child to paret. The umber of commuicatio liks i each chael icreases as we go up the tree from the leaves ad is determied by the amout of hardware available. For eample, Figure 5.5 represets a fat tree i which the umber of commuicatio liks i each chael is icreased by from oe level of the tree to the et. The fat tree ca be used to itercoect the processors of a geeral-purpose parallel machie. Sice its commuicatio badwidth ca be scaled idepedetly from the umber of processors, it provides great fleibility i desig. Figure 5.5 Fat tree.

4 Shuffle-echage. Aother method for establishig etworks is the shuffle-echage coectio. The shuffle-echage etwork is a combiatio of two fuctios: shuffle ad echage. Each is a simple bijectio fuctio i which each iput is mapped oto oe ad oly oe output. Let s - s s 0 be the biary represetatio of a ode address; the the shuffle fuctio ca be described as shuffle(s - s s 0 ) = s -2 s s 0 s -. For eample, usig the shuffle fuctio for N=8 (i.e. 2 3 odes) the followig coectios ca be established betwee the odes. Source Destiatio Source Destiatio The reaso that the fuctio is called shuffle is that it reflects the process of shufflig cards. Give that there are eight cards, the shuffle fuctio performs a perfect playig card shuffle as follows. First, the deck is cut i half, betwee cards 3 ad 4. The the two half decks are merged by selectig cards from each half i a alterative order. Figure 5.6 represets how the cards are shuffled. Figure 5.6 Card shufflig. Aother way to defie shuffle coectio is through the decimal represetatio of the addresses of the odes. Let N=2 be the umber of odes ad i represet the decimal address of a ode. For 0 i N / 2, ode i is coected to ode 2i. For N / 2 i N, ode i is coected to ode 2i+-N. The echage fuctio is also a simple bijectio fuctio. It maps a biary address to aother biary address that differs oly i the rightmost bit. It ca be described as echage(s - s s s 0 ) = s - s s s 0. Figure 5.7 shows the shuffle-echage coectios betwee odes whe N = 8. Figure 5.7 Shuffle-echage coectios.

5 The shuffle-echage etwork provides suitable itercoectio patters for implemetig certai parallel algorithms, such as polyomial evaluatio, fast Fourier trasform (FFT), sortig, ad matri traspositio [STO 7]. For eample, polyomial evaluatio ca be easily implemeted o a parallel machie i which the odes (processors) are coected through a shuffle-echage etwork. I geeral, a polyomial of degree N ca be represeted as a 0 + a + a a N- N- + a N N, where a 0, a,... a N are the coefficiets ad is a variable. As a eample, cosider the evaluatio of a polyomial of degree 7. Oe way to evaluate such a polyomial is to use the architecture give i Figure 5.7. I this figure, assume that each ode represets a processor havig three registers: oe to hold the coefficiet, oe to hold the variable, ad the third to hold a bit called the mask bit. Figure 5.8 illustrates the three registers of a ode. Figure 5.8 A shuffle-echage ode's registers. The evaluatio of the polyomial is doe i two phases. First, each term a i i is computed at ode i for i=0 to 7. The the terms a i i, for i= to 7, are added to produce the fial result. Figure 5.9 represets the steps ivolved i the computatio of a i i. Figure 5.9a shows the iitial values of the registers of each ode. The coefficiet a i, for i=0 to 7, is stored i ode i. The value of the variable is stored i each ode. The mask register of ode i, for i=, 3, 5, ad 7, is set to ; others are set to 0. I each step of computatio, every ode checks the cotet of its mask register. Whe the cotet of the mask register is, the cotet of the coefficiet register is multiplied with the cotet of the variable register, ad the result is stored i the coefficiet register. Whe the cotet of the mask register is zero, the cotet of the coefficiet register remais uchaged. The cotet of the variable register is multiplied with itself. The cotets of the mask registers are shuffled betwee the odes usig the shuffle etwork. Figures 5.9b, c, ad d show the values of the registers after the first step, secod step, ad third step, respectively. At the ed of the third step, each registers cotais a i i.

6 Figure 5.9 Steps for the computatio of the a i i. (a) Iitial values after step. (c) Values after step 2. (d) Values after step 3. At this poit, the terms a i i for i=0 to 7 are added to produce the fial result. To perform such a summatio, echage coectios are used i additio to shuffle coectios. Figure 5.0 shows all the coectios ad the iitial values of the coefficiet registers.

7 Figure 5.0 Required coectios for addig the terms a i i. I each step of computatio the cotets of the coefficiet registers are shuffled betwee the odes usig the shuffle coectios. The copies of the cotets of the coefficiet registers are echaged betwee the odes usig the echage coectios. After the echage is performed, each ode adds the cotet of its coefficiet register to the value that the copy of the curret cotet is echaged with. After three shuffle 7 ad echages, the cotet of each coefficiet register will be the desired ai i. The followig shows the three steps required to obtai result. i0 As you ca see i the chart, after the third step, the value ai i is stored i each coefficiet register. 7 i0

8 From this eample, it should be apparet that the shuffle-echage etwork provides the desired coectios for maipulatig the values of certai problems efficietly. Two-dimesioal mesh. A two-dimesioal mesh cosists of k * k 0 odes, where k i 2 deotes the umber of odes alog dimesio i. Figure 5. represets a two-dimesioal mesh for k 0 =4 ad k =2. There are four odes alog dimesio 0, ad two odes alog dimesio. As show i Figure 5., i a two-dimesioal mesh etwork each ode is coected to its orth, south, east, ad west eighbors. I geeral, a ode at row i ad colum j is coected to the odes at locatios (i-, j), (i+, j), (i, j-), ad (i, j+). The odes o the edge of the etwork have oly two or three immediate eighbors. The diameter of a mesh etwork is equal to the distace betwee odes at opposite corers. Thus, a twodimesioal mesh with k * k 0 odes has a diameter (k -) + (k 0 -). Figure 5. A two-dimesioal mesh with k 0 =4 ad k =2. I practice, two-dimesioal meshes with a equal umber of odes alog each dimesio are ofte used for coectig a set of processig odes. For this reaso i most literature the otio of two-dimesioal mesh is used without idicatig the values for k ad k 0 ; rather, the total umber of odes is defied. A twodimesioal mesh with k =k 0 = is usually referred to as a mesh with N odes, where N = 2. For eample, Figure 5.2 shows a mesh with 6 odes. From this poit forward, the term mesh will idicated a twodimesioal mesh with a equal umber of odes alog each dimesio. Figure 5.2 A two-dimesioal mesh with k 0 =k =4. The routig of data through a mesh ca be accomplished i a straightforward maer. The followig simple routig algorithm routes a packet from source S to destiatio D i a mesh with 2 odes.. Compute the row distace R as R D / S /. 2. Compute the colum distace C as C D(mod ) S(mod ). 3. Add the values R ad C to the packet header at the source ode. 4. Startig from the source, sed the packet for R rows ad the for C colums. The values R ad C determie the umber of rows ad colums that the packet eeds to travel. The directio the message takes at each ode is determied by the sig of the values R ad C. Whe R (C) is positive, the packet travels dowward (right); otherwise, the packet travels upward (left). Each time that the packet travels from oe ode to the adjacet ode dowward, the value R is decremeted by, ad whe it travels upward, R is icremeted by. Oce R becomes 0, the packet starts travelig i the horizotal

9 directio. Each time that the packet travels from oe ode to the adjacet ode i the right directio, the value C is decremeted by, ad whe it travels i the left directio, C is icremeted by. Whe C becomes 0, the packet has arrived at the destiatio. For eample, to route a packet from ode 6 (i.e., S=6) to ode 2 (i.e., D= 2), the packet goes through two paths, as show i Figure 5.3. I this eample, 2 / 4 6 / 4 2, R C Figure 5.3 Routig path from ode 6 to ode 2. It should be oted that i the case just described the odes o the edge of the mesh etwork have o coectios to their far eighbors. Whe there are such coectios, the etwork is called a wraparoud two-dimesioal mesh, or a Illiac etwork. A Illiac etwork is illustrated i Figure 5.4 for N = 6. Figure 5.4 A 6-ode Illiac etwork. I geeral, the coectios of a Illiac etwork ca be defied by the followig four fuctios: Illiac + (j) = j+ (mod N), Illiac - (j) = j- (mod N), Illiac + (j) = j+ (mod N), Illiac - (j) = j- (mod N), where N is the umber of odes, 0 j < N, is the umber of odes alog ay dimesio, ad N= 2. For eample, i Figure 5.4, ode 4 is coected to odes 5, 3, 8, ad 0, sice Illiac + (4) = ( 4 + ) (mod 6) = 5, Illiac - (4) = ( 4 - ) (mod 6) = 3, Illiac +4 (4) = ( ) (mod 6) = 8, Illiac -4 (4) = ( 4-4 ) (mod 6) = 0. The diameter of a Illiac with N= 2 odes is -, which is shorter tha a mesh. Although the etra wraparoud coectios i Illiac allow the diameter to decrease, they icrease the compleity of the desig.

10 Figure 5.5 shows the coectivity of the odes i a differet form. This graph shows that four odes ca be reached from ay ode i oe step, seve odes i two steps, ad four odes i three steps. I geeral, the umber of steps (recirculatios) to route data from a ode to ay other ode is upper bouded by the diameter (i.e., ). Figure 5.5 Alterative represetatio of a 6-ode Illiac etwork. To reduce the diameter of a mesh etwork, aother variatio of this etwork, called torus (or twodimesioal tours), has also bee proposed. As show i Figure 5.6a, a torus is a combiatio of rig ad mesh etworks. To make the wire legth betwee the adjacet odes equal, the torus may be folded as show i Figure 5.6b. I this way the commuicatio delay betwee the adjacet odes becomes equal. Note that both Figures 5.6a ad b provide the same coectios betwee the odes; i fact, Figure 5.6b is derived from Figure 5.6a by switchig the positio of the rightmost two colums ad the bottom two rows of odes. The diameter of a torus with N= 2 odes is 2 / 2, which is the distace betwee the corer ad the ceter ode. Note that the diameter is further decreased from the mesh etwork. Figure 5.6 Differet types of torus etwork. (a) A 4-by-4 torus etwork. (b) A 4-by-4 torus etwork with folded coectio.

11 The mesh etwork provides suitable itercoectio patters for problems whose solutios require the computatio of a set of values o a grid of poits, for which the value at each poit is determied based o the values of the eighborig poits. Here we cosider oe of these class of problems: the problem of fidig a steady-state temperature over the surface of a square slab of material whose four edges are held at differet temperatures. This problem requires the solutio of the followig partial differetial equatio, kow as Laplace's equatio: U / U / y 0, where U is the temperature at a give poit specified by the coordiates ad y o the slab. The followig describes a method, give by Slotick [SLO 7], to solve this problem. Eve if ufamiliar with Laplace's equatio, the reader should still be able to follow the descriptio. The method is based o the fact that the temperature at ay poit o the slab teds to become the average of the temperatures of eighborig poits. Assume that the slab is covered with a mesh ad that each square of the mesh has h uits o each side. The the temperature of a iterior ode at coordiates ad y is the average of the temperatures of the four eighbor odes. That is, the temperature at ode (, y), deoted as U(, y), equals the sum of the four eighborig temperatures divided by 4. For eample, as show i Figure 5.7, assume that the slab ca be covered with a 6-ode mesh. Here the value of U(, y) is epressed as U(,y)=[U(,y+h) + U(+h,y) + U(,y-h) + U(-h,y)]/4. Figure 5.7 Coverig a slab with a 6-ode mesh. Figure 5.8 illustrates a alterative represetatio of Figure 5.7. Here the positio of the odes is more coveietly idicated by the itegers i ad j. I this case, the temperature equatio ca be epressed as U(i,j)=[U(i,j+) + U(i+,j) + U(i,j-) + U(i-,j)]/4. Assume that each ode represets a processor havig oe register to hold the ode's temperature. The odes o the boudary are arbitrarily held at certai fied temperatures. Let the odes o the bottom of the mesh ad o the right edge be held at zero degrees. The odes alog the top ad left edges are set accordig to their positios. The temperatures of these 2 boudary odes do ot chage durig the computatio. The temperatures at the 4 iterior odes are the ukows. Iitially, the temperatures at these 4 odes are set to zero. I the first iteratio of computatio, the 4 iterior ode processors simultaeously calculate the ew temperature values usig the values iitially give.

12 Figure 5.8 Iitial values of the odes. Figure 5.9 represets the ew values of the iterior odes after the first iteratio. These values are calculated as follows: U(,2)=[U(,3)+U(2,2)+U(,)+U(0,2)]/4 = [ ]/4 = 4; U(2,2)=[U(2,3)+U(3,2)+U(2,)+U(,2)]/4 = [ ]/4 = ; U(,)=[U(,2)+U(2,)+U(,0)+U(0,)]/4 = [ ]/4 = ; U(2,)=[U(2,2)+U(3,)+U(2,0)+U(,)]/4 = [ ]/4 = 0. I the secod iteratio, the values of U(,2), U(2,2), U(,), ad U(2,) are calculated usig the ew values just obtaied: U(,2) = [8+++8]/4 = 4.5; U(2,2) = [ ]/4 = 2; U(,) = [ ]/4 = 2; U(2,) = [+0+0+]/4 = 0.5. This process cotiues util a steady-state solutio is obtaied. As more iteratios are performed, the values of the iterior odes coverge to the eact solutio. Whe values for two successive iteratios are close to each other (withi a specified error tolerace), the process ca be stopped, ad it ca be said that a steady-state solutio has bee reached. Figure 5.20 represets a solutio obtaied after iteratios. Figure 5.9 Values of the odes after the first iteratio.

13 Figure 5.20 Values of the odes after the eleveth iteratio. -cube or hypercube. A -cube etwork, also called hypercube, cosists of N=2 odes; is called the dimesio of the -cube etwork. Whe the ode addresses are cosidered as the corers of a - dimesioal cube, the etwork coects each ode to its eighbors. I a -cube, idividual odes are uiquely idetified by -bit addresses ragig from 0 to N-. Give a ode with biary address d, this ode is coected to all odes whose biary addresses differ from d i eactly bit. For eample, i a 3- cube, i which there are eight odes, ode 7 () is coected to odes 6 (0), 5 (0), ad 3 (0). Figure 5.2 demostrates all the coectios betwee the odes. Figure 5.2 A three-dimesioal cube. As ca be see i the 3-cube, two odes are directly coected if their biary addresses differ by bit. This method of coectio is used to cotrol the routig of data through the etwork i a simple maer. The followig simple routig algorithm routes a packet from its source S = (s -... s 0 ) to destiatio D = (d -... d 0 ).. Tag T S D t -... t 0 is added to the packet header at the source ode ( deotes a XOR gate). 2. If t i 0 for some 0 i, the use ith-dimesio lik to sed the packet to a ew ode with the same address as the curret ode ecept the ith bit, ad chage t i to 0 i the packet header. 3. Repeat step 2 util t i = 0 for all 0 i. For eample, as show i Figure 5.22, to route a packet from ode 0 to ode 5, the packet could go through two differet paths, P ad P 2. Here T=000 0 = 0. If we first cosider the bit t 0 ad the t 2, the packet goes through the path P. Sice t 0 =, the packet is set through the 0th-dimesio lik to ode.

14 At ode, t 0 is set to 0; thus T ow becomes equal to 00. Now, sice t 2 =, the packet is set through the secod-dimesio lik to ode 5. If, istead of t 0, bit t 2 is cosidered first, the packet goes through P 2. Figure 5.22 Differet paths for routig a packet from ode 0 to ode 5. I the etwork of Figure 5.22, the maimum distace betwee odes is 3. This is because the distace betwee odes is equal to the umber of bit positios i which their biary addresses differ. Sice each address cosists of 3 bits, the differece betwee two addresses ca be at most 3 whe every bit at the same positio differs. I geeral, i a -cube the maimum distace betwee odes is, makig the diameter equal to. The -cube etwork has several features that make it very attractive for parallel computatio. It appears the same from every ode, ad o ode eeds special treatmet. It also provides disjoit paths betwee a source ad a destiatio. Let the source be represeted as S = (s - s s 0 ) ad the destiatio by D = (d - d d 0 ). The shortest paths ca be symbolically represeted as Path : s - s s 0 s - s d 0 s - s d d 0 d - d d 0 Path 2: s - s s 0 s - s d s 0 s - s d 2 d s 0 d - d d s 0 d - d d d 0.. Path : s - s s 0 d - s s 0 d - s s d 0 d - s d d 0 d - d d d 0 For eample, cosider the 3-cube of Figure 5.2. Sice =3, there are three paths from a source, say 000, to a destiatio, say. The paths are path : ; path 2: ; path 3: This ability to have alterative paths betwee ay two odes makes the -cube etwork highly reliable if ay oe (or more) paths become uusable. Differet etworks, such as two-dimesioal meshes ad trees, ca be embedded i a -cube i such a way that the coectivity betwee eighborig odes remais cosistet with their defiitio. Figure 5.23 shows how a 4-by-4 mesh ca be embedded i a 4-cube (four-dimesioal hypercube). The 4-cube s itegrity is ot compromised ad is well-suited for uses like this, where a great deal of fleibility is required. All defiitioal cosideratios for both the 4-cube ad the 4-by-4 mesh, as stated earlier, are cosistet.

15 Figure 5.23 Embeddig a 4-by-4 mesh i a 4-cube. The itercoectio supported by the -cube provides a atural eviromet for implemetig highly parallel algorithms, such as sortig, mergig, fast Fourier trasform (FFT), ad matri operatios. For eample, Batcher's bitoic merge algorithm ca easily be implemeted o a -cube. This algorithm sorts a bitoic sequece (a bitoic sequece is a sequece of odecreasig umbers followed by a sequece of oicreasig umbers). Figure 5.24 presets the steps ivolved i mergig a odecreasig sequece [0,4,6,9] ad a oicreasig sequece [8,5,3,]. This algorithm performs a sequece of comparisos o pairs of data that are successively 2 2, 2, ad 2 0 locatios apart. Figure 5.24 Mergig two sorted lists of data. At each stage of the merge each pair of data elemets is compared ad switched if they are ot i ascedig order. This rearragig cotiues util the fial merge with a distace of puts the data ito ascedig order. Figure 5.24 requires the followig coectios betwee odes: Node 0 should be coected to odes:,2,4; Node should be coected to odes: 0,3,5;

16 Node 2 should be coected to odes: 0,3,6; Node 3 should be coected to odes:,2,7; Node 4 should be coected to odes: 0,5,6; Node 5 should be coected to odes:,4,7; Node 6 should be coected to odes: 2,4,7; Node 7 should be coected to odes: 3,5,6. These are eactly the same as 3-cube coectios. That is, the -cube provides the ecessary coectios for the Batcher's algorithm. Thus, applyig Batcher's algorithm to a -cube etwork is straightforward. I geeral, the -cube provides the ecessary coectios for ascedig ad descedig classes of parallel algorithms. To defie each of these classes, assume that there are 2 iput data items stored i 2 locatios (or processors) 0,, 2,..., 2 -. A algorithm is said to be i the descedig class if it performs a sequece of basic operatios o pairs of data that are successively 2 -, 2-2,..., ad 2 0 locatios apart. (Therefore, Batcher's algorithm belogs to this class.) I compariso, a ascedig algorithm performs successively o pairs that are 2 0, 2,..., ad 2 - locatios apart. Whe =3 Figures 5.25 ad 5.26 show the required coectios for each stage of operatio i this class of algorithms. As show, the -cube is able to efficietly implemet algorithms i descedig or ascedig classes. Figure 5.25 Descedig class. Figure 5.26 Ascedig class. Although the -cube ca implemet this class of algorithms i parallel steps, it requires coectios for each ode, which makes the desig ad epasio difficult. I other words, the -cube provides poor scalability ad has a iefficiet structure for packagig ad therefore does ot facilitate the icreasigly importat property of modular desig. -Dimesioal mesh. A -dimesioal mesh cosists of k -* k -2*... * k 0 odes, where k i 2 deotes the umber of odes alog dimesio i. Each ode X is idetified by coordiates -, -2,..., 0, where 0 i k i - for 0 i -. Two odes X=( -, -2,..., 0 ) ad Y=(y -,y -2,...,y 0 ) are said to be eighbors if ad oly if y i = i for all i, 0 i -, ecept oe, j, where y j = j + or y j = j -. That is, a ode may have from to 2 eighbors, depedig o its locatio i the mesh. The corers of the mesh have eighbors, ad the iteral odes have 2 eighbors, while other odes have b eighbors, where < b <2. The diameter of a -dimesioal mesh is ( k ). A -cube is a special case of -dimesioal meshes; it is i fact a - i0 i dimesioal mesh i which k i =2 for 0 i -. Figure 5.27 represets the structure of two three-

17 dimesioal meshes: oe for k 2 = k = k 0 = 3 ad the other for k 2 =4, k =3, ad k 0 =2. (a) k 2 = k = k 0 = 3. (b) k 2 =4, k =3, ad k 0 =2. Figure 5.27 Three-dimesioal meshes. k-ary -cube. A k-ary -cube cosists of k odes such that there are k odes alog each dimesio. Each ode X is idetified by coordiates, -, -2,..., 0, where 0 i k - for 0 i -. Two odes X=( -, -2,..., 0 ) ad Y=(y -,y -2,...,y 0 ) are said to be eighbors if ad oly if y i = i for all i, 0 i -, ecept oe, j, where y j =( j + ) mod k, or y j =( j -) mod k. That is, i cotrast to the -dimesioal mesh, a k-ary - cube has a symmetrical topology i which each ode has a equal umber of eighbors. A ode has eighbors whe k=2 ad 2 eighbors whe k>2. The k-ary -cube has a diameter of k / 2. A -cube is a special case of k-ary -cubes; it is i fact a 2-ary -cube. Figure 5.28 represets the structure of two k- ary -cubes: oe for k=4, =2 ad the other for k==3. Note that a 4-ary 2-cube is actually a torus etwork. Figure 5.28 (a) 4-Ary 2-cube ad (b) 3-ary 3-cube etworks.

18 Routig i -dimesioal meshes ad k-ary -cubes. Oe of the routig algorithms that ca be used for routig the packets withi a -dimesioal mesh or a k-ary -cube is called store-ad-forward routig [TAN 8]. Each ode of the etwork cotais a buffer equal to the size of a packet. I store-ad-forward routig, a packet is trasmitted from a source ode to a destiatio ode through a sequece of itermediate odes. Each itermediate ode of the etwork receives a packet i its etirety before trasmittig it to the et ode. Whe a itermediate ode receives a packet, it first stores the packet i its buffer; the it forwards the packet to the et ode whe the receivig ode's buffer is empty. Store-ad-forward routig is easy to uderstad ad simple to implemet. However, it requires a trasmissio time proportioal to the distace (the umber of hops or chaels) betwee the source ad the destiatio odes to deliver a packet. (Chaels are actually electrical coectios betwee odes ad are arraged based o the etwork topology.) To reduce trasmissio time ad make this task almost idepedet of the distace betwee source ad destiatio odes, a hardware-supported routig protocol, called wormhole routig (also direct-coect routig ) is ofte used. I wormhole routig, a packet is divided ito several smaller data uits, called flits. Oly the first flit (the leadig flit) carries the routig iformatio (such as the destiatio address), ad the remaiig flits follow this leader. Oce a leader flit arrives at a ode, the ode selects the outgoig chael based o the flit's routig iformatio ad begis forwardig the flits through that chael. Sice the remaiig flits carry o routig iformatio, they must ecessarily follow the chaels established by the header for the trasmissio to be successful. Therefore, they caot be iterleaved (alterated or mied) with the flits of other packets. Whe a leader flit arrives at a ode that has o output chael available, all the flits remai i their curret positio util a suitable chael becomes available. Each ode cotais a few small buffers for storig such flits. At each ode, the selectio of a outgoig chael for a particular leadig flit depeds o the icomig chael (the chael that was used by the flit to eter the ode) ad the destiatio ode. This type of depedecy ca be represeted by a chael depedecy graph. A chael depedecy graph for a give itercoectio etwork together with a routig algorithm is a directed graph such as show i Figure 5.29b. The vertices of the graph i Figure 5.29b are the chaels of the etwork, ad the edges are the pairs of chaels coected by the routig algorithm. For eample, cosider Figure 5.29a, where a etwork with odes, 0,..., ad 00 ad uidirectioal chaels c, c 0,..., ad c 00, is show. The chaels are labeled by the idetificatio umber (id) of their source ode. A routig algorithm for such a etwork could advace the flits o c to c 0, o c 0 to c 0, ad so o. Based o this routig algorithm, Figure 5.29b represets the depedecy graph for such a etwork. Notice that the depedecy graph cosists of a cycle that may cause a deadlock i the etwork. A deadlock ca occur wheever o flits ca proceed toward their destiatios because the buffers o the route are full. Figure 5.29c presets a deadlock cofiguratio i the case whe there are two buffers i each ode.

19 Figure 5.29 A simple etwork with four odes. (a) Network. (b) Depedecy graph. (c) Deadlock. To have reliable ad efficiet commuicatio betwee odes, a deadlock-free routig algorithm is eeded. Dally ad Seitz [DAL 87] have show that a routig algorithm for a itercoectio etwork is deadlock free if ad oly if there are o cycles (a route that recoects with itself) i the chael depedecy graph. Their proposal is to avoid deadlock by elimiatig cycles through the use of virtual chaels. A virtual chael is a logical lik betwee two odes formed by a physical chael ad a flit buffer i each of the two odes. Each physical chael is shared amog a group of virtual chaels. Although several virtual chaels share a physical chael, each virtual chael has its ow buffer. With may (virtual) chaels to choose from, cycles, ad therefore deadlock, ca be avoided. Figure 5.30a represets the virtual chaels for a etwork whe each physical chael is split ito two virtual chaels: lower virtual chaels ad upper virtual chaels. The lower virtual chael of c (where is idetified as the source ode) is labeled c 0, ad the upper virtual chael is labeled c. For eample, the lower virtual chael of c is umbered as c 0. Dally ad Seitz's routig algorithm routes packets at a ode with a label value less tha the destiatio ode o the upper virtual chaels ad routes packets at a ode labeled greater tha their destiatio ode o the lower chaels. This routig algorithm restricts the packets' routig to the order of decreasig virtual chael labels. Thus there is o cycle i the depedecy graph ad the etwork is deadlock free (see Figure 5.30b).

20 Figure 5.30 (a) Virtual chaels ad (b) depedecy graph for a simple etwork with four odes. Wormhole routig is based o a method of dividig packets ito smaller trasmissio uits called flits. Trasmittig flits rather tha packets reduces the average time required to deliver a packet i the etwork, as show i Figure 5.3. Figure 5.3 Comparig (a) store-ad-forward routig with (b) wormhole routig. T sf ad T wh are average trasmissio time over three chaels whe usig store-ad-forward routig ad wormhole routig, respectively. For eample, assume that each packet cosists of q flits, ad T f is the amout of time required for each flit to be trasmitted across a sigle chael. The amout of time required to trasmit a packet over a sigle chael is therefore q * T f. With store-ad-forward routig, the average time required to trasmit a packet over D chaels will be D * q * T f. However, with wormhole routig, i which the flits are forwarded i a

21 pipelie fashio, the average trasmissio time over D chaels becomes (q+d-) * T f. This meas that wormhole routig is much faster tha store-ad-forward routig. Furthermore, wormhole routig requires very little storage, resultig i a small ad fast commuicatio cotroller. I geeral, it is a efficiet routig techique for k-ary -cubes ad -dimesioal meshes. I literature, several deadlock-free routig algorithms, based o the wormhole routig cocept, have bee proposed. These algorithms ca be classified ito two groups: determiistic (or static) routig ad adaptive (or dyamic) routig. I determiistic routig, the routig path, which is traversed by flits, is fied ad is determied by the source ad destiatio addresses. Although, these routigs usually select oe of the shortest paths betwee the source ad destiatio odes, they limit the ability of the itercoectio etwork to adapt itself to failures or heavy traffic (cogestio) alog the iteded routes. It is i this case that adaptive routig becomes importat. Adaptive routig algorithms allow the path take by flits to deped o dyamic etwork coditios (such as the presece of faulty or cogested chaels), rather tha source ad destiatio addresses. The descriptio of these algorithms is beyod the scope of this book. The reader ca refer to [NI 93] for a survey o determiistic ad adaptive wormhole routig i k-ary -cubes ad -dimesioal meshes. Network Latecy. Here, based o the work of Agarwal [AGA 9] ad Dally ad Seitz[DAL 87], we focus o derivig a equatio for the average time required to trasmit a packet i k-ary -cubes that uses wormhole routig. A similar aalysis ca also be carried out for -dimetioal etworks. We assume that the etworks are embedded i a plae ad have uidirectioal chaels. The etwork latecy, T b, refers to the elapsed time from the time that the first flit of the packet leaves the source to the time the last flit arrives at the destiatio. Hece, igorig the etwork load, T b ca be epressed as T b = (q+d-) * T f, where D deotes the umber of chaels (hops) that a packet traverses. Let T f be represeted as the sum of the wire delay T w () ad ode delay T s, that is, T f = T w () + T s. Hece, T b = (q+d-)[ T w () + T s ]. The umber of chaels, D, ca be determied by the product of the etwork dimesio ad the average distace (k d ) that a packet must travel i each dimesio of the etwork. Assumig that the packet destiatios are radomly chose, the average distace a packet must travel is give by Hece k d =(k-)/2. T b = [q + (k-)/2 -][ T w () + T s ]. To determie T w (), we must fid the legth of the logest wire of a -dimesioal etwork embedded i a plae. The embeddig of a -dimesioal etwork i a plae ca be achieved by mappig /2 dimesios of the etwork i each of the two physical dimesios. That is, the umber of odes i each physical dimesio is k /2. Thus each additioal dimesio of the etwork icreases the umber of odes i each physical dimesio by k /2. Assumig that the distace betwee the physically adjacet odes remais fied, each additioal dimesio also icreases the legth of the logest wire legth by a factor of k /2. Assume that the wire delay depeds liearly o the wire legth. If we cosider the delay of a wire i a twodimesioal etwork [i.e., T w (2)] as a base time period, the delay of the logest wire is give by Hece T w () = (k /2 ) -2 T w (2) = k (/2 - ).

22 T b = [q + (k-)/2 -][ k (/2 - ) +T s ]. Agarwal [AGA 9] has eteded this result to the aalysis of a k-ary -cube uder differet load parameters, such as packet size, degree of local commuicatio, ad etwork request rate. [The degree of local commuicatio icreases as the probability of commuicatio with (or access to) various odes decreases as a fuctio of physical distace.] Agarwal has show that two-dimesioal etworks have the lowest latecy whe ode delays ad etwork cotetio are igored. Otherwise, three or four dimesios are preferred. However, whe the degree of local commuicatios becomes high, two-dimesioal etworks outperform three- ad four-dimesioal etworks. Local commuicatio depeds o several factors, such as machie architecture, type of applicatios, ad compiler. If these factors are ehaced, twodimesioal etworks ca be used without icurrig the high cost of higher dimesios. Aother alterative for ehacig local commuicatio is to provide short paths for olocal packets. The k-ary -cube etwork ca be augmeted by oe or more levels of epress chaels that allow olocal messages to bypass odes [DAL 9]. The augmeted etwork, called epress cube, reduces the etwork diameter ad icreases the wire legth. This arragemet allows the etwork to operate with latecies that approach the physical speed-of-light limitatio, rather tha beig limited by ode delays. Figure 5.32 illustrates the additio of epress chaels to a k-ary -cube etwork. I epress cubes the wire legth of epress chaels ca be icreased to the poit that wire delays domiate ode delay, makig lowdimesioal etworks more attractive. Figure 5.32 Epress cube. (a) Regular k-ary -cube etwork. (b) k-ary -cube etwork with epress chaels Dyamic Networks Dyamic etworks provide recofigurable coectios betwee odes. The topology of a dyamic etwork is the physical structure of the etwork as determied by the switch boes ad the itercoectig liks. Sice the switch bo is the basic compoet of the etwork, the cost of the etwork (i hardware terms) is measured by the umber of switch boes required. Therefore, the topology of the etwork is the prime determiat of the cost. To clarify the precedig termiology, let us cosider the desig of a dyamic etwork usig simple switch boes. Figure 5.33 represets a simple switch with two iputs ( ad y) ad two outputs (z 0 ad z ). A cotrol lie, s, determies whether the iput lies should be coected to the output lies i straight state or echage state. For eample, whe the cotrol lie s=0, the iputs are coected to the outputs i a straight state; that is, is coected to z 0 ad y is coected to z. Whe the cotrol lie s=, the iputs are coected to outputs i a echage state; that is, is coected to z ad y is coected to z 0.

23 Figure 5.33 A simple two-iput switch. Now let's use this switch to desig a etwork that ca coect a source to oe of eight possible destiatios 0 to 7. A solutio for such a etwork is show i Figure I this desig, there are three stages (colums), stages 2,, ad 0. The destiatio address is deoted bit-wise d 2 d d 0. The switch i stage 2 is cotrolled by the most sigificat bit of the destiatio address (i.e., d 2 ). This bit is used because, whe d 2 =0, the source is coected to oe of the destiatios 0 to 3 (000 to 0); otherwise, is coected to oe of the destiatios 4 to 7 (00 to ). I a similar way, the switches i stages ad 0 are cotrolled by d ad d 0, respectively. Figure 5.34 A simple -to-8 itercoectio etwork. Now let's epad our etwork to have eight sources istead of oe. Figure 5.35 represets a solutio to such a etwork costructed i the same maer as the desig i Figure 5.34.

24 Figure 5.35 A simple 8-to-8 itercoectio etwork. Note that, i this etwork, the destiatio address bits caot be used to cotrol switches for some coectios, such as, coectig source to destiatio 5. Therefore, at this poit, let s assume there is some kid of mechaism for cotrollig switches. Based o this assumptio, the etwork is able to coect ay sigle source to ay sigle destiatio. However, it is ot able to establish certai coectios with multiple sources ad multiple destiatios. Describig such multiple coectios requires the use of the term permutatio. A permutatio refers to the coectio of a set of sources to a set of destiatios such that each source is coected to a sigle destiatio. A permutatio [(s 0,d 0 ),(s,d ),...,(s 7,d 7 )] meas that source s 0 is coected to d 0, s to d, ad so o. The etwork of Figure 5.35 caot establish particular permutatios. For eample, a permutatio that requires sources 0 ad to be coected to destiatios 0 ad, respectively, caot be established at the same time. However, by chagig the positio of some of the switches, such a permutatio becomes possible. Figure 5.36 represets the same etwork after switchig the positio of the coectios of iputs ad 4 to the switches 0 ad 2 of stage 2. This ew etwork is able to coect 0 to 0 ad to, simultaeously, ad establish the permutatio. Nevertheless, there are may permutatios, such as [(0,),(,2),(2,3),(3,4),(4,5),(5,6),(6,7),(7,0)], that caot be established by this ew etwork. Later i this chapter, better etworks that ca provide the ecessary permutatios for may applicatios are represeted. Figure 5.36 A alterative desig for a 8-to-8 itercoectio etwork. To provide a perspective o the various dyamic etwork topologies ad to aid i orgaizig the later sectios, a dyamic etworks taoomy is preseted i Figure At the first level of the hierarchy are the crossbar switch, sigle-stage, ad multistage etworks.

25 Figure 5.37 Classificatio of dyamic etworks. The crossbar switch ca be used for coectig a set of iput odes to a set of output odes. I this etwork every iput ode ca be coected to ay output ode. The crossbar switch provides all possible permutatios, as well as support for high system performace. It ca be viewed as a umber of vertical ad horizotal liks itercoected by a switch at each itersectio. Figure 5.38 represets a crossbar for coectig N odes to N odes. The coectio betwee each pair of odes is established by a crosspoit switch. The crosspoit switch ca be set o or off i respose to applicatio eeds. There are N 2 crosspoit switches for providig complete coectios betwee all the odes. The crossbar switch is a ideal etwork to use for small N. However, for large N, the implemetatio of the crosspoit switches makes this desig comple ad epesive ad thus less attractive to use. Figure 5.38 Crossbar switch. Sigle-stage etworks, also called recirculatig etworks, require routig algorithms to direct the flow of data several times through the etwork so that various coectios ad permutatios ca be costructed. Each time that the data traverse the etwork is called a pass. As a eample, Figure 5.39 represets a sigle-stage etwork based o the shuffle-echage coectio. Multistage etworks, such as the oe i Figure 5.36, are more comple from a hardware poit of view, but the routig of data is made simpler by virtue of permaet coectios betwee the stages of the etwork. Because there are more switches i a multistage etwork, the umber of possible permutatios o a sigle pass icreases; however, there is a higher ivestmet i hardware. There is also a possible reductio i the compleity of routig fuctios ad the time it takes to geerate the ecessary permutatios.

26 Figure 5.39 A sigle-stage etwork. Multistage etworks are further divided ito cocetrators ad coectors. Both of these techologies were established i the 950s by Bell Labs. A cocetrator itercoects a specific idle iput to a arbitrary idle output. Oe way to specify a cocetrator is by a triplet of itegers (I, O, C), where I>O C, ad where I is the umber of iputs, O is the umber of outputs, ad C is the capacity of the cocetrator. The capacity of a cocetrator is the maimum umber of coectios that ca be made simultaeously I through the etwork. Thus a cocetrator (I, O, C) is capable of itercoectig ay of the choices of K I I! iputs (K C) to some K of the outputs, where. For eample, Figure 5.40 represets K K!( I K)! a (6,4,4) cocetrator, called Masso's biomial cocetrator. I this etwork, the crosspoit switches 4 coectig the iputs to the outputs cosist of all the possible choices of two switches per iput lie. 2 There are si possible differet matchigs betwee si iput ad four output lies with two switches per iput lie. Ofte cocetrators are used for coectig several termials to a computer. Figure 5.40 A cocetrator with si iputs ad four outputs. A coector establishes a path from a specific iput to a specific output. I geeral, coector etworks ca be grouped ito three differet classes: oblockig, rearrageable, ad blockig etworks. I a oblockig etwork, it is always possible to coect a idle pair of termials (iput/output odes) without

27 disturbig coectios (calls) already i progress. This is called "oblockig i the strict sese" simply because such a etwork has o blockig states whatsoever. These type of etworks are said to be uiversal etworks sice they ca provide all possible permutatios. The rearrageable etworks are also uiversal etworks; however, i this type of etwork it may ot always be possible to coect a idle pair of termials without disturbig established coectios. I a rearrageable etwork, give ay set of coectios i progress ad ay pair of idle termials, the eistig coectios ca be reassiged ew routes (if ecessary) so as to make it possible to coect the idle pair at ay time. I cotrast, i a blockig etwork, depedig o what state the etwork may be i, it may ot be possible to coect a idle pair of termials i ay way. For each group of coectors, a class of dyamic etworks is show i the followig discussio. Noblockig etworks. Clos has proposed a class of etworks with iterestig properties [CLO 53]. Figure 5.4 shows oe eample of such a etwork. This particular etwork is called a three-stage Clos etwork. It cosists of a iput stage of m crossbar switches, a output stage of m crossbar switches, ad a middle stage of r r crossbar switches. This class of etworks is deoted by the triple N(m,,r), which determies the switches' dimesios. Figure 5.4 A oblockig etwork Clos has show that for m 2-, the etwork N(m,,r), is a oblockig etwork. For eample, the etwork N(3,2,2) i Figure 5.42 is a oblockig etwork. This etwork requires 2 crosspoit switches i every stage, or 36 switches i all. Note that a crossbar switch with the same umber of iputs ad outputs (i.e., 4) requires 6 switches. Thus i this case it is more ecoomical to desig a crossbar switch tha a Clos etwork. However, whe the umber of iputs, N, icreases, the umber of switches becomes much less tha N 2, as i the case of the crossbar. For eample, for N=36, oly 88 switches are ecessary i a Clos etwork, whereas i the case of a crossbar etwork 36 2, or 296, switches are required. Figure 5.42 A Clos etwork with =2, r=2, ad m=3. There should be at least 2- switches i the middle stage of the Clos etwork i order to become a oblockig etwork. To demostrate the ecessity of this coditio, let's cosider the followig eample. Figure 5.43 represets a sectio of a Clos etwork i which each iput (output) switch has three iputs

28 (outputs). Let's assume that we wat to coect iput C to output F. I this eample, four middle switches are required to permit iputs other tha C (i.e., A ad B) o a particular iput switch ad outputs other tha F (i.e., D ad E) o a particular output switch to have coectios to separate middle switches. I additio, oe more switch for the desired coectio betwee C ad F is required. Thus five middle switches are required (i.e., 2 * 3- switches). A similar argumet ca be give for a geeral etwork N(m,,r), i order to show that N is oblockig whe m=2-. Figure 5.43 A portio of a Clos etwork i which each iput/output switch has three termials. The total umber of switches for a three-stage Clos etwork N(2-,, ) ca be obtaied by aalyzig the umber of switches i each stage. Assumig that the etwork has N iput termials, where N= 2, the The iput stage cotais 2 (2-) switches. The middle stage cotais 2 (2-) switches. The output stage cotais 2 (2-) switches. Therefore, the total umber of switches, C(3), is C(3) = (2-)(3 2 )= I a similar way, the total umber of switches for a five-stage Clos etwork, show i Figure 5.44, is C(5) = (2-)( )+2 2 *(2-) = Figure 5.44 A five-stage Clos etwork; each of the middle-stage boes is a three-stage Clos etwork with N 2/3 iputs/outputs. Rearrageable etworks. Slepia ad Duguid showed that the etwork N(m,,r) is rearrageable if ad oly if m [BEN 62, DUG 59]. Later, Paull demostrated that whe m==r at most - eistig paths must be rearraged i order to coect a idle pair of termials [BEN 62, PAU 62]. Fially, Bees improved Paull's result by showig that a etwork N(,,r), where r 2, requires a maimum of r- paths to be rearraged [BEN 62]. Costructio of rearrageable etworks. The developmet of a rearrageable etwork depeds largely

29 o the desig of the switches ad the permutatio fuctios used to coect them. The followig method is a geeric approach to developig such etworks. To costruct a rearrageable etwork with a odd umber of stages, the followig structure ca be used. S S S, 2 s where S i represets the switches of i th stage, i represets the coectio betwee stage S i ad S i, ad s 3 represets umber of stages. This etwork should have the followig properties. Let i, for i, ( s ) / 2, deote the umber of iputs (outputs) for every switch i stage S i. This umber is chose such that ( s) / 2 i where i 2, ad N is the total umber of iputs to the etwork. The etwork should also satisfy the followig symmetric coditio: i s i for i=,, (s-)/2, S i S si for i=,, (s-)/2, where s i is iverse of the i coectio. i N I other words, the etire etwork will be symmetrical at the middle stage. To the left of the middle stage the coectios,, ( s) / 2 will coect stages S,, S ( s) / 2. To the right of the middle stage the iverse of these coectios will coect the stages S S ( s ) / 2,, s. To defie the coectio i (for i ( s ) / 2 ), take the first switch of S i ad coect each oe of its outputs to the iput of oe of the first i switches of S i+ ; go o to the secod switch of S i ad coect its i outputs to the iput of each of the et i switches of S i+. Whe all the switches of S i+ have oe lik o the iput side, start agai with the first switch. Proceed cyclically i this way utill all the outputs of S i are assiged. Figure 5.45 represets a rearrageable etwork, called a eight-iput Bees etwork. Note that = 2 =2. A alterative represetatio of Bees etwork is show i Figure This represetatio is obtaied by switchig the positio of switches 2 ad 3 i every stage ecept the middle oe. s Figure 5.45 A 8-to-8 rearrageable etwork.

30 Figure 5.46 A eight-iput Bees etwork. I geeral a Bees etwork ca be geerated recursively. Figure 5.47 represets the structure of a N=2 - iput Bees etwork. The middle stage cotais two sub-blocks; each sub-block is a N/2-iput Bees etwork. The costructio process ca be recursively applied to the sub-blocks util sub-blocks of size 2 iputs are reached. Sice the Bees etwork is a rearrageable etwork, it is possible to coect the iputs to the outputs accordig to ay of N! permutatios. Figure 5.47 Recursive structure of Bees etwork. Blockig etworks. Net, two well-kow multistage etworks, multistage cube ad omega, are discussed. These etworks are blockig etworks, ad they provide ecessary permutatios for may applicatios. Multistage Cube Network. The multistage cube etwork, also kow as the iverse idirect -cube etwork, provides a dyamic topology for a -cube etwork. It ca be used as a processor-to-memory or as a processor-to-processor itercoectio etwork. The multistage cube cosists of =log 2 N stages, where N is the umber of iputs (outputs). Each stage i the etwork cosists of N/2 switches. Each switch has two iputs, two outputs, ad four possible coectio states, as show i Figure Two cotrol lies ca be used to determie ay of the four states. Whe the switch is i upper broadcast (lower broadcast) state, the data o the upper (lower) iput termial are set to both output termials. Figure 5.48 The four possible states of the switch used i the multistage cube. As a eample, Figure 5.49 represets a multistage cube etwork whe N=8. The coectio patter betwee stages is such that at each stage the lik labels to a switch differ i oly bit. More precisely, at stage i the lik labels to a switch differ i the i th bit. The reaso that such a etwork is called multistage

31 cube is that the coectio patters betwee stages correspod to the -cube etwork. As show i Figure 5.50, for N=8, the patter of liks i stage 2,, ad 0 correspod, respectively, to vertical, diagoal, ad horizotal liks i the 3-cube. Figure 5.49 A eight-iput multistage cube etwork. Figure 5.50 Correspodece betwee the coectio patters of multistage cube ad 3-cube etworks. There are may simple ways for settig the states of the switches i a multistage cube etwork with N=2 iputs. Let's assume that a source S (with address s - s s 0 ) has to be coected to a certai destiatio D (with address d - d d 0 ). Startig at iput S, set the first switch [i the (-) th stage] that is coected to S to the straight state whe d - = s - ; otherwise, set the switch to the echage state. I the same way, bits d -2 ad s -2 determie the state of the switch located o the et stage. This process cotiues util a path is established betwee S ad D. I geeral, the state of the switch o the i th stage is straight whe d i = s i ; otherwise, the switch is set to echage. Figure 5.5 represets a path betwee source 2 (i.e., S = 00) ad destiatio 6 (i.e., D =0). I this figure, ote that the iputs of the switch o stage 2,, ad 0 are coected to the output liks d 2 s s 0, d 2 d s 0, ad d 2 d d 0, respectively.

32 Figure 5.5 Routig i a multistage cube. I the precedig method the differeces betwee the source ad destiatio addresses ca be stored as a tag, T, i the head of the packet. That is, T = S D = t -... t 0 determies the state of the switches o the path from source to destiatio. Oce the packet arrives at a switch i stage i, the switch eamies t i ad sets its state. If t i =0, the switch is set i the straight state; otherwise, it is set i the echage state. Aother way is to add destiatio D as a tag to the header. I this way, the iput of the switch o the i th stage is coected to the upper output whe d i = 0, otherwise, to the lower output. A multistage cube supports up to N oe-to-oe simultaeous coectios. However, there are some permutatios that caot be established by this kid of etwork. For eample, as show i Figure 5.5, a permutatio that requires source 3 ad 7 to be coected to destiatios ad 0, respectively, caot be established. I additio to oe-to-oe coectios, the multistage cube also supports oe-to-may coectios; that is, a iput device ca broadcast to all or a subset of the output devices. For eample, Figure 5.52 represets the state of some switches for broadcastig from iput 2 to outputs 4, 5, 6, ad 7. Figure 5.52 Broadcastig i a multistage cube. Omega Network. The omega etwork was origially proposed by Lawrie [LAW 75] as a itercoectio etwork betwee processors ad memories. The etwork allows coflict-free access to rows, colums, diagoals, ad square blocks of matrices [LAW 75]. This is importat for matri computatio. The omega etwork provides the ecessary permutatios (for certai applicatios) at a substatially lower cost tha a crossbar, sice the omega requires fewer switches. The omega etwork cosists of = log 2 N stages, where N is the umber of iputs (outputs). Each stage i the etwork cosists of a shuffle patter of liks followed by a colum of N/2 switches. As a eample, Figure 5.53 represets a omega etwork whe N=8. Similar to the multistage cube, each switch has two iputs, two outputs, ad four possible coectio states (see Figure 5.48). Each switch is cotrolled idividually. There is a efficiet routig algorithm for settig the states of the switches i the omega

33 etwork. Let's assume that a source S (with address s - s s 0 ) has to be coected to a certai destiatio D (with address d - d d 0 ). Startig at iput S, coect the iput of the first switch [i the (-) th stage] that is coected to S to the upper output of the switch whe d - = 0; otherwise, to the lower output. I the same way, bit d -2 determies the output of the switch located o the et stage. This process cotiues util a path is established betwee S ad D. I geeral, the iput of the switch o the i th stage is coected to the upper output whe d i = 0; otherwise, the switch is coected to the lower output. Figure 5.53 represets a path betwee source 2 (i.e., S = 00) ad destiatio 6 (i.e., D =0). The omega etwork is a blockig etwork; that is, some permutatios caot be established by the etwork. I Figure 5.53, for eample, a permutatio that requires sources 3 ad 7 to be coected to destiatios ad 0, respectively, caot be established. However, such permutatios ca be established i several passes through the etwork. I other words, sometimes packets may eed to go through several odes so that a particular permutatio ca be established. For eample, whe ode 3 is coected to ode, ode 7 ca be coected to ode 0 through ode 4. That is, ode 7 seds its packet to ode 4, ad the ode 4 seds the packet to ode 0. Therefore, we ca coect ode 3 to ode i oe pass ad ode 7 to ode 0 i two passes. I geeral, if we cosider a sigle-stage shuffle-echage etwork with N odes, the every arbitrary permutatio ca be realized by passig through this etwork at most 3(log 2 N)- times [WU 8]. I additio to oe-to-oe coectios, the omega etwork also supports broadcastig. Similar to the multistage cube etwork, the omega etwork ca be used to broadcast data from oe source to may destiatios by settig some of the switches to the upper broadcast or lower broadcast state. Figure 5.53 A eight-iput omega etwork. I geeral, the omega etwork is equivalet to a multistage cube etwork; that is, both provide the same set of permutatios. I fact, some argue that the omega etwork is othig more tha a alias for a multistage cube etwork. Figure 5.54 demostrates, for N=8, why this assertio may be true. By switchig the positio of switches 2 ad 3 i stage of the multistage cube etwork, the omega etwork ca be obtaied.

34 Figure 5.54 Mappig (a) a multistage cube etwork to (b) a omega etwork. Aother way to show equivalecy (uder certai assumptios) betwee the omega ad multistage cube etworks is through the represetatio of allowable permutatios for each of them. Ay permutatio i a etwork with N iputs, where N=2, ca be epressed as a collectio of switchig (Boolea) fuctios. For eample, cosider the followig permutatio for N=8: [(0,0),(,2),(2,4),(3,6),(4,),(5,3),(6,5),(7,7)]. Let X= 2 0 deote the biary represetatio of a source. Also, let F(X)=f 2 f f 0 deote the biary represetatio of the destiatio that X is coected to. The, the precedig permutatio ca be represeted as follows: X coected to F(X) 2 0 f 2 f f Each of the switchig fuctios f 0, f, ad f 2, therefore, ca be epressed as f f , , f

35 Thus, i geeral, every permutatio ca be represeted i terms of a set of switchig fuctios. I the followig, the switchig represetatio of omega ad multistage cube etworks is derived. Iitially, represetatios of basic fuctios, such as shuffle ad echage, are derived. These fuctios are the used to derive represetatios of omega ad multistage cube etworks. Shuffle ( ). The shuffle fuctio is defied as ( ) = This fuctio ca also be represeted as a set of switchig fuctios, such as 0 i i f i i Echage (E). The echage fuctio E is defied as. ) ( E This fuctio ca also be represeted as a set of switchig fuctios, such as 0 0 i i f i i Omega Network ( ). Recall that the omega etwork with stages is a sequece of shuffle-echage fuctios. That is, )). ())) ( ( ( ( E E E Thus, to determie to which destiatio a give source X= is coected, we must first apply fuctio, the E, et agai, ad so o. As show below, after applyig ad E times to the source X, the switchig fuctios ca be obtaied. First we apply :. ) ( 0 2 X Net, we apply E to. 0 2, )) ( ( 0 2 c X E where the bit c - represets the cotrol sigal to the switches of the (-) th stage, ad deotes the Boolea XOR fuctio. It is assumed that oe cotrol sigal c i goes to all the switches of the stage i, ad each switch ca have two states, straight (c i =0) ad echage (c i =). Note that the bit - is eclusive-or ed with c -, rather tha complemeted. This is because the bit c - determies whether a switch is i the straight state or the echage state, if a switch is i echage state the the echage fuctio should be applied. Now we apply ad the E to c 0 2, )))) ( ( ( ( c c X E E where the bit c -2 represets the cotrol sigal to the switches of the (-2) th stage. Fially, c c c X. 0 0 ) ( Thus c f i i i for 0 i Multistage cube (C). The multistage cube ca be represeted as )), ()))) ( ( ( ( ( 2 0 E E E C where i represets the coectio betwee the switches of stage i+ ad i, ad is the umber of stages. The fuctio i is defied as. ) ( i i i First we apply ad the E. )) ( ( 0 2 c X E where the bit c - represets the cotrol sigal to the switches of the (-) th. It is assumed that oe cotrol

36 sigal c i goes to all the switches of stage i, ad each switch ca have two states, straight (c i =0) ad echage (c i =). E( 2 ( E( ( X )))) c c2, E( 3 ( E( 2 ( E( ( X )))))) c 2 c c3. Fially, C(X ) c 2 c2 0 c0. Thus f i i ci for 0 i. Note that the omega etwork also has the same set of switchig fuctios; therefore, it is equivalet to the multistage cube. 5.3 INTERCONNECTION DESIGN DECISIONS A major problem i parallel computer desig is fidig a itercoectio etwork capable of providig fast ad efficiet commuicatio at a reasoable cost. There are at least five desig cosideratios whe selectig the architecture of a itercoectio etwork: operatio mode, switchig methodology, etwork topology, a cotrol strategy, ad the fuctioal characteristics of the switch. Operatio mode. Three primary operatig modes are available to the itercoectio etwork desiger: sychroous, asychroous, ad combied. Whe a sychroous stream of istructios or data is required by the etwork, a sychroous commuicatio system is required. I other words, sychroous commuicatio is eeded for establishig commuicatio paths sychroously for either data maipulatig fuctios or for a data istructio broadcast. Most SIMD machies operate i a lock-step fashio; that is, all active processig odes trasmit data at the same time. Thus sychroous commuicatio seems a appropriate choice for SIMD machies. Whe coectio requests for a itercoectio etwork are issued dyamically, a asychroous commuicatio system is eeded. Sice the timig of the routig requests is ot predictable, the system must be able to hadle such requests at ay time. Some systems are desiged to hadle both sychroous ad asychroous commuicatios. Such systems are able to do array processig by utilizig sychroous commuicatios, yet are also able to cotrol less predictable commuicatio requests by usig asychroous timig methods. Switchig methodology. The three mai types of switchig methodologies are circuit switchig, packet switchig, ad itegrated switchig. Circuit switchig establishes a complete path betwee source ad destiatio ad holds this path for the etire trasmissio. It is best suited for trasmittig large amouts of cotiuous data. I cotrast to circuit switchig, packet switchig has o dedicated physical coectio set up. Hece it is most useful for trasmittig small amouts of data. I packet switchig, data items are partitioed ito fied-size packets. Each packet has a header that cotais routig iformatio, ad moves from oe ode i the etwork to the et. The packet switchig icreases chael throughput by multipleig various packets through the same path. Most SIMD machies use circuit switchig, while packet switchig is most suited to MIMD machies. The third optio, itegrated switchig, is a combiatio of circuit ad packet switchig. This allows large amouts of data to be moved quickly over the physical path while allowig smaller packets of iformatio to be trasmitted via the etwork. Network topology. To desig or select a topology, several performace parameters should be cosidered. The most importat parameters are the followig.. VLSI implemetable. The topology of the etwork should be able to be mapped o two (or three) physical dimesios so that it ca produce a efficiet layout for packagig ad implemetatio i

37 VLSI systems. 2. Small diameter. the diameter of the etwork should grow slowly with the umber of odes. 3. Neighbor idepedecy. The umber of eighbors of ay ode should be idepedet of the size of the etwork. This allows the etwork to scale up to a very large size. 4. Easy to route. There should be a efficiet algorithm for routig messages from ay ode to ay other. The messages must fid a optimal path betwee the source ad destiatio odes ad make use of all of the available badwidth. 5. Uiform load. The traffic load o various parts of the etwork should be uiform. 6. Redudat Pathways. The etwork should be highly reliable ad highly available. Message pathways should be redudat to provide robustess i the evet of compoet failure. Cotrol strategy ad fuctioal characteristics of the switch. All dyamic etworks are composed of switch boes coected together through a series of liks. The fuctioal characteristics of a switch bo are its size, routig logic, the umber of possible states for the switch, fault detectio ad correctio, commuicatio protocols, ad the amout of buffer space available for storig packets whe there is cogestio. Most of the switches provide some of these capabilities, depedig o implemetatio requiremets relatig to efficiecy ad cost. I geeral, states of the switches of a etwork ca be set by a cetral cotroller or by each idividual switch. The former is a cetralized cotrol system, while the latter is a distributed oe. Cetralized cotrol ca be further broke dow ito idividual stage cotrol, idividual bo cotrol, ad partial stage cotrol. Idividual stage cotrol uses oe cotrol sigal to set all switch boes at the same stage. Idividual bo cotrol uses a separate cotrol sigal to set the state of each switch bo. This offers higher fleibility i settig up the coectig paths, but icreases the umber of cotrol sigals, which, i tur, sigificatly icreases cotrol circuit compleity. I partial stage cotrol, i+ cotrol sigals are used at stage i of the etwork. I a distributed cotrol etwork, the switches are usually more comple. I multistage itercoectio etworks, the switches have to deal with coflict resolutio, as well as with chages i routig due to faults or cogestio. Switches utilize protocols for hadshakig to esure that data may be correctly trasferred. Large buffers eable a switch to store data that caot be set forward due to cogestio. This allows icreased performace of the etwork by decreasig the umber of retrasmissios.