8 Interconnection. Networks and Clusters 8

Transcription

1 8 Itercoectio Networks ad Clusters 8 The edium is the essage because it is the medium that shapes ad cotrols the search ad form of huma associatios ad actios. arshall cluha Uderstadig edia (1964) The marvels of film, radio, ad televisio are marvels of oe-way commuicatio, which is ot commuicatio at all. ilto ayer O the Remote ossibility of Commuicatio (1967)

2 8.1 Itroductio A Simple Network Itercoectio Network edia Coectig ore Tha Two Computers Network Topology ractical Issues for Commercial Itercoectio Networks Examples of Itercoectio Networks Iteretworkig Crosscuttig Issues for Itercoectio Networks Clusters esigig a Cluster uttig It All Together: The Goggle Cluster of Cs Aother View: Iside a Cell hoe Fallacies ad itfalls Cocludig Remarks Historical erspective ad Refereces 654 Exercises Itroductio Thus far we have covered the compoets of a sigle computer, which has bee the traditioal focus of computer architecture. I this chapter we see how to coect computers together, formig a commuity of computers. Figure 8.1 shows the geeric compoets of this commuity: computer odes, hardware ad software iterfaces, liks to the itercoectio etwork, ad the itercoectio etwork. Itercoectio etworks are also called etworks or commuicatio subets, ad odes are sometimes called ed systems or hosts. The coectio of two or more itercoectio etworks is called iteretworkig, which relies o commuicatio stadards to covert iformatio from oe kid of etwork to aother. The Iteret is the most famous example of iteretworkig. There are two reasos that computer architects should devote attetio to etworkig. I additio to providig exteral coectivity, oore s Law shruk etworks so much that they coect the compoets withi a sigle computer. Usig a etwork to coect autoomous systems withi a computer has log bee foud i maiframes, but today this such desigs ca be foud i Cs too. Switches are replacig buses as the ormal commuicatio techique: betwee

3 564 Chapter 8 Itercoectio Networks ad Clusters Node Node Node Node SW iterface SW iterface SW iterface SW iterface HW iterface HW iterface HW iterface HW iterface Lik Lik Lik Lik Itercoectio etwork FIGURE 8.1 rawig of the geeric itercoectio etwork. computers, betwee I/O devices, betwee boards, betwee chips, ad eve betwee modules iside chips. As a result, computer architects must uderstad etworkig termiology, problems ad solutios i order to desig ad evaluate moder computers. The secod reaso architects should study etworkig is that today almost all computers are--or will be--etworked to other devices. Thus, uderstadig etworkig is critical; ay device without a etwork is somehow flawed. Just as a moder computer without a memory hierarchy broke hece a chapter just for it a moder computer without a etwork is broke too. Hece this chapter. This topic is vast, with portios of Figure 8.1 the subject of whole books ad college courses. Networkig is also a buzzword-rich eviromet, where may simply ideas are obscured behid acroyms ad uusual defiitios. To help you breakthrough the buzzword barrier, Figure 8.2 is a glossary of about 80 etworkig terms. The goal of this chapter is to provide computer architects a getle, qualitative itroductio to etworkig. It defies terms, helps you uderstad the architectural implicatios of itercoectio etwork techology, provides itroductory explaatios of the key ideas, ad give refereces to more detailed descriptios. ost of this chapter is o etworkig, but the fial quarter of this chapter focuses o clusters. A cluster is the coordiated use of itercoected computers i a machie room. I cotrast to the qualitative etwork itroductio, these sectios give a more quatitative descriptio of clusters, icludig may examples. It eds with a guided tour of the Google clusters.

4 8.1 Itroductio 565 Term adaptive routig AT atteuatio backpressure flow cotrol badwidth base statio bisectio badwidth bit error rate blade blockig bridge category 5 wire carrier sesig chael checksum circuit switchig cluster coaxial cable collisio collisio detectio efiitio Router picks best path based upo measure of delay o outgoig liks Asychroous Trasfer ode is a WAN desiged for real-time traffic such as digital voice Loss of sigal stregth as sigal passes through the medium over a log distace Whe the receiver caot accept aother message, separate wires betwee adjacet seders ad receivers tell the seder to stop immediately. It causes liks betwee two ed poits to freeze util the receiver makes room for the ext message. aximum rate the etwork ca propagate iformatio oce the message eters the it A etwork architecture that uses boxes coected via lad lies to commuicate to wireless hadsets Sum of the badwidth of lies that cross that imagiary dividig lie betwee two roughly equal parts of the etwork, each with half the odes BER, the error rate of a etwork, typically i errors per millio bits trasferred A removable computer compoet that fits vertically ito a box i a stadard VE rack Cotetio that prevets a message from makig progress alog a lik of a switch OSI layer 2 etworkig device that coects multiple LANs, which ca operate i parallel; i cotrast, a router coects etworks with icompatible addresses at OSI layer 3 Cat 5 twisted-pair, copper wire used for 10, 100, ad 1000 bits/sec LANs Listeig to the medium to be sure it is uused before tryig to sed a message I wireless etworks, it is a pair of frequecy bads that allow 2-way commuicatio A field of a message for a error correctio code A circuit is established from source to destiatio, reservig badwidth alog a path util the circuit is broke Coordiated use of itercoected computers i a machie room A sigle stiff copper wire is surrouded by isulatig material ad a shield; historically faster ad loger distace tha twisted pair copper wire Two odes (or more) o a shared medium try to sed at the same time Listeig to shared medium after sedig to see if a message collided with aother FIGURE 8.2 Networkig terms i this chapter ad their defiitios

5 566 Chapter 8 Itercoectio Networks ad Clusters Term collocatio site commuicatio subets credit-based flow cotrol cut-through routig destiatio-based routig determiistic routig ed systems ed-to-ed argumet Etheret fat tree FC-AL frequecy-divisio multiplexig full duplex header host hub Ifiibad efiitio A warehouse for remote hostig of servers with expasible etworkig, space, coolig, ad security Aother ame for itercoectio etwork To reduce overhead for flow cotrol, a seder is give a credit to sed up to N packets, ad oly checks for etwork delays whe the credit is spet The switch examies the header, decides where to sed the message, ad the start trasmittig it immediately without waitig for the rest of the message. Whe the head of the message blocks, the message stays strug out over the etwork. The message cotais a destiatio address, ad the switch picks a path to deliver the message, ofte by table lookup Router always picks the same path for the message Aother ame for itercoectio etwork ode as opposed to the itermediate switches Itermediate fuctios (error checkig, performace optimizatio, ad so o) may be icomplete as compared to performig the fuctio ed-to ed The most popular LAN, it has scaled from its origial 3 bits/secod rate usig shared media i 1975 to switched media at 1000 bits/secod i 2001; it shows o sigs of stoppig a etwork topology with extra liks at each level ehacig a simple tree, so badwidth betwee each level is ormally costat (see Figure 8.14 o page 595) Fibre Chael Arbitrated Loop; a SAN for storage devices ivide the badwidth of the trasmissio lie ito a fixed umber of frequecies, ad assig each frequecy to a coversatio. Two-way commuicatio o a etwork segmet The first part of a message that cotais o user iformatio, but cotets helps that etwork, such as providig the destiatio address Aother ame for itercoectio etwork ode A OSI layer 1 etworkig device that coects multiples LANs to act as oe A emergig stadard SAN for both storage ad systems i a machie room FIGURE 8.2 Networkig terms i this chapter ad their defiitios

6 8.1 Itroductio 567 Term iterferece iteretworkig I iscsi LAN message multimode fiber multipath fadig multistage switch efiitio I wireless etworks, reductio of sigal due to frequecy reuse; frequecy is reused to try to icrease the umber of simultaeous coversatios over a large area Coectio of two or more itercoectio etworks Iteret rotocol is a OSI layer 3 protocol, at the etwork layer SCSI over I etworks, it is a competitor to SANs usig I ad Etheret switches Local Area Network, for machies i a buildig or campus, such as Etheret The smallest piece of electroic mail set over a etwork A iexpesive optical fiber that reduces badwidth ad distace for cost I wireless etworks, iterferece betwee multiple versios of sigal that arrive at differet times, determied by time betwee fastest sigal ad slowest sigal relative to sigal badwidth a switch cotaiig may smaller switches that perform a portio of routig OSI layer Ope System Itercoect models the etwork as seve layers (see 8.25 o page 612 ) overhead packet switchig payload peer-to-peer protocol peer-to-peer wireless protocol rack uit receiver overhead router SAN I this chapter, etworkig overhead is seder overhead + receiver overhead + time of flight I cotrast to circuit switchig, iformatio is broke ito packets (usually fixed or maximum size), each with it s ow destiatio address, ad they are routed idepedetly The middle part of the message that cotais user iformatio Commuicatio betwee two odes occurs logically at the same level of the protocol Istead of commuicatig to base statios, peer-to-peer wireless etworks commuicate betwee hadsets The sequece of steps that etwork software follows to commuicate A R.U. is 1.7 iches, the height of a sigle slot i a stadard 19-ich VE rack; there are 44 R.U. i stadard 6-foot rack The time for the processor to pull the message from the itercoectio etwork OSI layer 3 etworkig device that coects multiples LANs with icompatible addresses Origially System Area Network but more recetly Storage Area Network, it coects computers ad/or storage devices i a machie room. FC-AL or Ifiibad are SANs. FIGURE 8.2 Networkig terms i this chapter ad their defiitios

7 568 Chapter 8 Itercoectio Networks ad Clusters Term seder overhead shadow fadig sigal-to-oise ratio simplex sigle-mode fiber source-based routig store-ad-forward TC throughput time of flight trailer trasmissio time trasport latecy twisted pairs virtual circuit WAN wavelegth divisio multiplexig widow wireless etwork wormhole routig efiitio The time for the processor to iject the message ito the etwork; the processor is busy for the etire time I wireless etworks, whe the received sigal is blocked by objects; buildigs outdoors or walls idoors SNR, the ratio of the stregth of the sigal carryig iformatio to the backgroud oise Oe-way commuicatio o a etwork segmet Sigle-wavelegth fiber is arrower ad more expesive tha multimode fiber but it offers greater badwidth ad distace The message specifies the path to the destiatio at each switch Each switch waits for the full message to arrive before it is set o to the ext switch Trasmissio Cotrol rotocol, it is a OSI layer 4 protocol (trasport layer) I etworkig, measured speed of the medium or etwork badwidth delivered to a applicatio; i.e., does ot give credit for headers ad trailers The time for the first bit of the message to arrive at the receiver The last part of a message that has o user iformatio but helps the etwork, such as error correctio code The time for the message to pass through the etwork (ot icludig time of flight) Time that the message speds i the itercoectio etwork (icludig time of flight) Two wires twisted together to reduce electrical iterferece A logical circuit is established betwee source ad destiatio for a message to follow Wide Area Network, a etwork across a cotiet, such as AT W seds differet streams simultaeously o the same fiber usig differet wavelegths of light ad the demultiplexes the differet wavelegths at the receiver I TC, the umber of TC datagrams that ca be set without waitig for approval A etwork that commuicates without physical coectios, such as radio The switch examies the header, decides where to sed the message, ad the starts trasmittig it immediately without waitig for the rest of the message. The tail cotiues whe the head blocks, potetially compressig the strug-out message ito a sigle switch FIGURE 8.2 Networkig terms i this chapter ad their defiitios

8 8.1 Itroductio 569 Let s start with the geeric types of itercoectios. epedig o the umber of odes ad their proximity, these itercoectios are give differet ames: Wide area etwork (WAN) Also called log haul etwork, the WAN coects computers distributed throughout the world. WANs iclude thousads of computers, ad the maximum distace is thousads of kilometers. AT is a curret example of a WAN. Local area etwork (LAN) This device coects hudreds of computers, ad the distace is up to a few kilometers. Ulike a WAN, a LAN coects computers distributed throughout a buildig or o a campus. The most popular ad edurig LAN is Etheret. Storage or System area etwork (SAN) This itercoectio etwork is for a machie room, so the maximum distace of a lik is typically less tha 100 meters, ad it ca coect hudreds of odes. Today SAN usually meas Storage area etwork as it coects computers to storage devices, such as disk arrays. Origially SAN meat a System area etwork to coect computers together, such as Cs i a cluster. A recet SAN tryig to etwork both storage ad system is Ifiibad. Figure 8.3 shows the rough relatioship of these systems i terms of umber autoomous systems coected, icludig a bus for compariso. Note the area of overlap betwee buses, SANs, ad LANs, which lead to product competitio. WAN/Iteret LAN Bus SAN Number of Autoomous Systems Coected FIGURE 8.3 Relatioship of four types of itercoects i terms of umber of autoomous systems coected: bus, system or storage area etwork, local area etwork, ad wide area etwork/iteret. Note that there are overlappig rages where buses, SANs, ad LANs compete. Some supercomputers have a switch-based custom etwork to itercoect up to thousads of computers; such itercoects are basically custom SANs.

9 570 Chapter 8 Itercoectio Networks ad Clusters These three types of itercoectio etworks have bee desiged ad sustaied by several differet cultures Iteret, telecommuicatios, workgroup/ eterprise, storage, ad high performace computig each usig its ow dialects ad its ow favorite approaches to the goal of itercoectig autoomous computers. This chapter gives a commo framework for evaluatig all itercoectio etworks, usig a sigle set of terms to describe the basic alteratives. Figure 8.22 i sectio 8.7 gives several other examples of each of these itercoectio etworks. As we shall see, some compoets are commo to all types ad some are quite differet. We begi the chapter i sectio 8.2 by explorig the desig ad performace of a simple etwork to itroduce the ideas. We the cosider the followig problems: which media to use as the itercoect (8.3), how to coect may computers together (8.4 ad 8.5), ad what are the practical issues for commercial etworks (8.6). We follow with examples illustratig the trade-offs for each type of etwork (8.7), explore iteretworkig (8.8), ad cross cuttig issues for etworks (8.9). With this getle itroductio to etworks i sectios 8.2 to 8.9, readers iterested i more depth should try the suggested readig i sectio Sectios 8.10 to 8.12 switch to clusters, ad give a more quatitative descriptio with desigs ad examples. Sectio 8.13 gives a view of etworks from the embedded perspective, usig a cell phoe ad wireless etworks as the example. We coclude i sectios 8.14 to 8.16 with the traditioal edig of the chapters. As we shall see, etworkig shares more characteristics with storage tha with processors ad memory. Like storage, the operatig system cotrols what features of the etwork are used. Agai like storage, performace icludes both latecy ad badwidth, ad queueig theory is a valuable tool. Like RAI, etworkig assumes failures occur, ad thus depedability i the presece of errors is the orm. 8.2 A Simple Network There is a old etwork sayig: Badwidth problems ca be cured with moey. Latecy problems are harder because the speed of light is fixed you ca t bribe God. Aoymous To explai the complexities ad cocepts of etworks, this sectio describes a simple etwork of two computers. We the describe the software steps for these two machies to commuicate. The remaider of the sectio gives a detailed ad the a simple performace model, icludig several examples to see the implicatios of key etwork parameters.

10 8.2 A Simple Network 571 Suppose we wat to coect two computers together. Figure 8.4 shows a simple model with a uidirectioal wire from machie A to machie B ad vice versa. At the ed of each wire is a first-i-first-out (FIFO) queue to hold the data. I this simple example, each machie wats to read a word from the other s memory. A message is the iformatio set betwee machies over a itercoectio etwork. achie A achie B FIGURE 8.4 A simple etwork coectig two machies. For oe machie to get data from the other, it must first sed a request cotaiig the address of the data it desires from the other ode. Whe a request arrives, the machie must sed a reply with the data. Hece, each message must have at least 1 bit i additio to the data to determie whether the message is a ew request or a reply to a earlier request. The etwork must distiguish betwee iformatio eeded to deliver the message, typically called the header or the trailer depedig o where it is relative to the data, ad the payload, which cotais the data. Figure 8.5 shows the format of messages i our simple etwork. This example shows a sigle-word payload, but messages i some itercoectio etworks ca iclude hudreds of words. Itercoectio etworks ivolve ormally software. Eve this simple example ivokes software to traslate requests ad replies ito messages with the appropriate headers. A applicatio program must usually cooperate with the operatig system to sed a message to aother machie, sice the etwork will be shared with all the processes ruig o the two machies, ad the operatig system caot allow messages for oe process to be received by aother. Thus, the messagig software must have some way to distiguish betwee processes; this distictio may be icluded i a expaded header. Although hardware support ca reduce the amout of work, most is doe by software. I additio to protectio, etwork software is ofte resposible for esurig reliable delivery of messages. The twi resposibilities are esurig that the message is either garbled or lost i trasit.

11 572 Chapter 8 Itercoectio Networks ad Clusters Header (1 bit) ayload (32 bits) 0 Address 1 ata 0= Request 1 = Reply FIGURE 8.5 essage format for our simple etwork. essages must have extra iformatio beyod the data. Addig a checksum field (or some other error detectio code) to the message format meets the first resposibility. This redudat iformatio is calculated whe the message is first set ad checked upo receipt. The receiver the seds a ackowledgmet if the message passes the test. Oe way to meet the secod resposibility is to have a timer record the time each message is set ad to presume the message is lost if the timer expires before a ackowledgmet arrives. The message is the re-set. The software steps to sed a message are as follows: 1. The applicatio copies data to be set ito a operatig system buffer. 2. The operatig system calculates the checksum, icludes it i the header or trailer of the message, ad the starts the timer. 3. The operatig system seds the data to the etwork iterface hardware ad tells the hardware to sed the message. essage receptio is i just the reverse order: 3. The system copies the data from the etwork iterface hardware ito the operatig system buffer. 2. The system calculates the checksum over the data. If the checksum matches the seder s checksum, the receiver seds a ackowledgmet back to the seder. If ot, it deletes the message, assumig that the seder will resed the message whe the associated timer expires. 1. If the data pass the test, the system copies the data to the user s address space ad sigals the applicatio to cotiue. The seder must still react to the ackowledgmet:

12 8.2 A Simple Network 573 Whe the seder gets the ackowledgmet, it releases the copy of the message from the system buffer. If the seder gets the time-out istead of a ackowledgmet, it reseds the data ad restarts the timer. Here we assume that the operatig system keeps the message i its buffer to support retrasmissio i case of failure. Figure 8.6 shows how the message format looks ow. Header (2 bits) ayload (32 bits) Trailer (4 bits) (Checksum) ata 00 = Request 01 = Reply 10 = Ackowledge request 11 = Ackowledge reply FIGURE 8.6 trailer. essage format for our simple etwork. Note that the checksum is i the The sequece of steps that software follows to commuicate is called a protocol ad geerally has the symmetric but reversed steps betwee sedig ad receivig. Note that this example protocol above is for sedig a sigle message. Whe a applicatio does ot require a respose before sedig the ext message, the seder ca overlap the time to sed with the trasmissio delays ad the time to receive. A protocol must hadle may more issues tha reliability. For example, if two machies are from differet maufacturers, they might order bytes differetly withi a word (see sectio 2.3 of Chapter 2). The software must reverse the order of bytes i each word as part of the delivery system. It must also guard agaist the possibility of duplicate messages if a delayed message were to become ustuck. It is ofte ecessary to deliver the messages to the applicatio i the order they are set, ad so sequece umbers may be added to the header to eable assembly. Fially, it must work whe the receiver s FIFO becomes full, suggestig feedback to cotrol the flow of messages from the seder (see sectio 8.4). Now that we have covered the steps i sedig ad receivig a message, we ca discuss performace. Figure 8.7 shows the may performace parameters of itercoectio etworks. This figure is critical to uderstadig etwork performace, so study it

13 574 Chapter 8 Itercoectio Networks ad Clusters Seder Seder overhead Trasmissio time (bytes/badwidth) Receiver Time of flight Trasmissio time (bytes/badwidth) Receiver overhead Trasport latecy Total latecy Time FIGURE 8.7 erformace parameters of itercoectio etworks. epedig o whether it is a SAN, LAN, or WAN, the relative legths of the time of flight ad trasmissio may be quite differet from those show here. (Based o a presetatio by Greg apadopolous of Su icrosystems.) well! Note that the parameters i Figure 8.7 apply to the itercoect i may levels of the system: iside a chip, betwee chips o a board, betwee computers i a cluster, ad so o. The uits chage, but the priciples remai the same, as does the badwidth that results. These terms are ofte used loosely, leadig to cofusio, so we defie them here precisely: Badwidth We use this most widely used term to refer to the maximum rate at which the etwork ca propagate iformatio oce the message eters the etwork. Ulike disks, badwidth icludes the headers ad trailers as well as the payload, ad the uits are traditioally bits/secod rather tha bytes/secod. The term badwidth is also used to mea the measured speed of the medium or etwork badwidth delivered to a applicatio. Throughput is sometimes used for this latter term. Time of flight The time for the first bit of the message to arrive at the receiver, icludig the delays due to repeaters or other hardware i the etwork. Time of flight ca be millisecods for a WAN or aosecods for a SAN. Trasmissio time The time for the message to pass through the etwork, ot icludig time of flight. Oe way to measure it is the differece i time betwee whe the first bit of the message arrives at the receiver ad whe the last bit of the message arrives at the receiver. Note that by defiitio trasmissio time is equal to the size of the message divided by the badwidth. This measure assumes there are o other messages to coted for the etwork.

14 8.2 A Simple Network 575 Trasport latecy The sum of time of flight ad trasmissio time. Trasport latecy is the time that the message speds i the itercoectio etwork. Stated alteratively, it is the time betwee whe the first bit of the message is ijected ito the etwork ad whe the last bit of the message arrives at the receiver. It does ot iclude the overhead of ijectig the message ito the etwork or pullig it out whe it arrives. Seder overhead The time for the processor to iject the message ito the etwork, icludig both hardware ad software compoets. Note that the processor is busy for the etire time, hece the use of the term overhead. Oce the processor is free, ay subsequet delays are cosidered part of the trasport latecy. For pedagogic reasos, we assume overhead is ot depedet o message size. (Typically, oly very large messages have larger overhead.) Receiver overhead The time for the processor to pull the message from the itercoectio etwork, icludig both hardware ad software compoets. I geeral, the receiver overhead is larger tha the seder overhead: for example, the receiver may pay the cost of a iterrupt. The total latecy of a message ca be expressed algebraically: essage size Total latecy = Seder overhead + Time of flight Receiver overhead Badwidth Let s look at how the time of flight ad overhead parameters chage i importace as we go from SAN to LAN to WAN. EXALE Assume a etwork with a badwidth of 1000 bits/secod has a sedig overhead of 80 microsecods ad a receivig overhead of 100 microsecods. Assume two machies. Oe wats to sed a byte message to the other (icludig the header), ad the message format allows bytes i a sigle message. Let s compare SAN, LAN, ad WAN by chagig the distace betwee the machies. Calculate the total latecy to sed the message from oe machie to aother i a SAN assumig they are 10 meters apart. Next, perform the same calculatio but assume the machies are ow 500 meters apart, as i a LAN. Fially, assume they are 1000 kilometers apart, as i a WAN. ANSWER The speed of light is 299,792.5 kilometers per secod i a vacuum, ad sigals propagate at about 63% to 66% of the speed of light i a coductor. Sice this is a estimate, i this chapter we ll roud the speed of light to 300,000 kilometers per secod, ad assume we ca achieve two-thirds of that i a coductor. Hece, we ca estimate time of flight. Let s plug the parameters for the short distace of a SAN ito the formula above:

15 576 Chapter 8 Itercoectio Networks ad Clusters essage size Total latecy = Seder overhead + Time of flight Receiver overhead Badwidth 0.01km bytes = 80µsecs , 000 km/sec µsecs bits/sec Covertig all terms ito microsecods (µsecs) leads to Total latecy = = = µsecs , 000 µsecs µsecs 100 µsecs µsecs µsecs + 80 µsecs µsec = µsecs 260µsecs Substitutig a example LAN distace ito the third equatio yields Total latecy = = = 0.5km bytes 80µsecs , 000 km/sec µsecs bits/sec 80 µsecs µsecs + 80 µsecs µsec = µsecs 262µsecs Substitutig the WAN distace ito the equatio yields 1000 km bytes Total latecy = 80µsecs , 000 km/sec µsecs bits/sec = 80 µsecs µsecs + 80 µsecs µsec = µsecs = 5260µsecs The icreased fractio of the latecy required by time of flight for log distaces, as well as the greater likelihood of errors over log distaces, are why wide area etworks use more sophisticated ad time-cosumig protocols. Complexity icreases from protocols used o a bus versus a LAN versus the Iteret as we go from te to hudreds to thousads of odes. Note that messages i LANs ad WANs go through switches which add to the latecy, which we eglected above. Geerally, switch latecy is small compared to overhead i LANs or time of flight i SANs. As metioed above, whe a applicatio does ot require a respose before sedig the ext message, the seder ca overlap the sedig overhead with the trasport latecy ad receiver overhead. Icreased latecy affects the structure of programs that try to hide this latecy, requirig quite differet solutios if the latecy is 1, 100, or 10,000 microsecods.

16 8.2 A Simple Network 577 Note that the example above shows that time of flight for SANs is so short relative to overhead that it ca be igored, yet i WANs, time of flight is so log that seder ad receiver overheads ca be igored. Thus, we ca simplify the performace equatio by combiig seder overhead, receiver overhead, ad time of flight ito a sigle term called Overhead: essage size Total latecy Overhead Badwidth We ca use this formula to calculate the effective badwidth delivered by the etwork as message size varies: essage size Effective badwidth = Total latecy Let s use this simpler equatio to explore the impact of overhead ad message size o effective badwidth. EXALE lot the effective badwidth versus message size for overheads of 25 ad 250 microsecods ad for etwork badwidths of 100, 1000, ad bits/secod. Vary message size from 16 bytes to 4 megabytes. For what message sizes is the effective badwidth virtually the same as the raw etwork badwidth? If overhead is 250 microsecods, for what message sizes is the effective badwidth always less tha 100 bits/secod? ANSWER Figure 8.8 plots effective badwidth versus message size usig the simplified equatio above. The otatio ox,bwy meas a overhead of X microsecods ad a etwork badwidth of Y bits/secod. To amortize the cost of high overhead, message sizes must be four megabytes for effective badwidth to be about the same as etwork badwidth. Assumig the high overhead, message sizes about 3K bytes or less will ot break the 100 bits/secod barrier o matter what the actual etwork badwidth. Thus, we must lower overhead as well as icrease etwork badwidth uless messages are very large. Hece, message size is importat i gettig full beefit of fast etworks. What is the atural size of messages? Figure 8.9 above shows the size of Network File System (NFS) messages for 239 machies at Berkeley collected over a period of oe week. Oe plot is cumulative i messages set, ad the other is cumulative i data bytes set. The maximum NFS message size is just over 8 KB, yet 95% of the messages are less tha 192 bytes log..figure 8.10 below shows the similar results for Iteret traffic, where the maximum trasfer uit was 1500 bytes.

17 578 Chapter 8 Itercoectio Networks ad Clusters 10, ,000.0 Effective badwidth (bits/sec) o25,bw10000 o25,bw1000 o25,bw100 o250,bw10000 o250,bw1000 o250,bw K 4K 16K 64K 256K 1 4 essage size (bytes) FIGURE 8.8 Badwidth delivered versus message size for overheads of 25 ad 250 microsecods ad for etwork badwidths of 100, 1000, ad bits/secod. Note that with 250 microsecods of overhead ad a etwork badwidth of 1000 bits/secod, oly the 4-B message size gets a effective badwidth of 1000 bits/secod. I fact, message sizes must be greater tha 256 B for the effective badwidth to exceed 10 bits/secod. The otatio ox,bwy meas a overhead of X microsecods ad a etwork badwidth of Y bits/secod. <<Artist: label lies, drop leged.>> Agai, 60% of the messages are less tha 192 bytes log, ad 1500-byte messages represeted 50% of the bytes trasferred. ay applicatios sed far more small messages tha large messages, sice requests ad ackowledgemets are more frequet tha data Summarizig this sectio, eve this simple etwork has brought up the issues of protectio, reliability, heterogeeity, software protocols, ad a more sophisticated performace model. The ext four sectios address other key questios: Which media are available to coect computers together? What issues arise if you wat to coect more tha two computers? What practical issues arise for commercial etworks?

18 8.2 A Simple Network % 90% 80% essages 70% 60% Cumulative percetage 50% 40% 30% 20% ata bytes 10% % essage size (bytes) FIGURE 8.9 Cumulative percetage of messages ad data trasferred as message size varies for NFS traffic. Each x-axis etry icludes all bytes up to the ext oe; e.g., 32 represets 32 bytes to 63 bytes. ore tha half the bytes are set i 8-KB messages, but 95% of the messages are less tha 192 bytes. Figure 8.50 (page 651) shows details of this measuremet. Collected at the Uiversity of Califoria at Berkeley. Cumulative percetage 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% essages ata bytes essage Size FIGURE 8.10 Cumulative percetage of messages ad data trasferred as message size varies for Iteret traffic. About 40% of the messages were 40 bytes log, ad 50% of the data trasfer was i messages 1500 bytes log. The maximum trasfer uit of most switches was 1500 bytes. Collect by Ver axto o CI Iteret traffic i 1998.

19 580 Chapter 8 Itercoectio Networks ad Clusters 8.3 Itercoectio Network edia Just as there is a memory hierarchy, there is a hierarchy of media to itercoect computers that varies i cost, performace, ad reliability. Network media have aother figure of merit, the maximum distace betwee odes. This sectio covers three popular examples, ad Figure 8.11 illustrates them. Category 5 Usheilded Twisted pair ("Cat5"): Coaxial cable: lastic coverig Braided outer coductor Isulator Copper core Fiber optics: Trasmitter is LE or Laser dieoge Claddig Total iteral reflectio Receiver is hotodiode Light source Silica Core Claddig Buffer FIGURE 8.11 Three etwork media. (From a presetatio by avid Culler of U.C. Berkeley.) The first medium is twisted pairs of copper wires. These are two isulated wires, each about 1 mm thick. They are twisted together to reduce electrical iterferece, sice two parallel lies form a atea but a twisted pair does ot. As they ca trasfer a few megabits per secod over several kilometers without amplificatio, twisted pair were the maistay of the telephoe system. Telephoe compaies budled together (ad sheathed) may pairs comig ito a buildig. Twisted pairs ca also offer tes of megabits per secod of badwidth over shorter distaces, makig them plausible for LANs.

20 8.3 Itercoectio Network edia 581 The desire to go at higher speeds with the less expesive copper led to improvemets i the quality of ushielded twisted-pair copper cablig systems. The origial telephoe-lie quality was called Level 1. Level 3 was good eough for 10 bits/secod Etheret. The desire for eve greater badwidth lead to the Level 5 or Category 5, which is sufficiet for 100 bits/secod Etheret. By limitig the legth to 100 meters, Cat5 wirig ca be used for 1000 bits/secod Etheret liks today. It uses the RJ-45 coector, which is similar to the coector foud o telephoe lies. Coaxial cable was deployed by cable televisio compaies to deliver a higher rate over a few kilometers. To offer high badwidth ad good oise immuity, isulatig material surrouds a sigle stiff copper wire, ad the cylidrical coductor surrouds the isulator, ofte wove as a braided mesh. A 50-ohm basebad coaxial cable delivers 10 megabits per secod over a kilometer. Coectig to this heavily isulated media is more challegig. The origial techique was a T juctio: the cable is cut i two ad a coector is iserted that recoects the cable ad adds a third wire to a computer. A less ivasive solutio is a vampire tap: a hole of precise depth ad width is first drilled ito the cable, termiatig i the copper core. A coector is the screwed i without havig to cut the cable. To keep up with the demads of badwidth ad distace, it became clear that the telephoe compay would eed to fid ew media. The solutio could be more expesive provided that it offered much higher badwidth ad that supplies were pletiful. The aswer was to replace copper with glass ad electros with photos. Fiber optics trasmits digital data as pulses of light. A fiber optic etwork has three compoets: 1. the trasmissio medium, a fiber optic cable; 2. the light source, a LE or laser diode; 3. the light detector, a photodiode. First, claddig surrouds the glass fiber core to cofie the light. A buffer the surrouds the claddig to protect the core ad claddig. Note that ulike twisted pairs or coax, fibers are oe-way, or simplex, media. A two-way, or full duplex, coectio betwee two odes requires two fibers. Sice light beds or refracts at iterfaces, it ca slowly spread as it travels dow the cable uless the diameter of the cable is limited to oe wavelegth of light; the it trasfers i a straight lie. Thus, fiber optic cables are of two forms: 1. ultimode fiber It uses iexpesive LEs as a light source. It is typically much larger tha the wavelegth of light: typically 62.5 micros i diameter vs. the 1.3-micro wavelegth of ifrared light. Sice it is wider it has more dispersio problems, where some wave frequecies have differet propagatio velocities. The LEs ad dispersio limit it to up to a few hudred meters at 1000 bits/secod or a few kilometers at 100 bits /secod. It is older ad less expesive tha sigle mode fiber.

21 582 Chapter 8 Itercoectio Networks ad Clusters 2. Sigle-mode fiber This sigle-wavelegth fiber (typically 8 to 9 micros i diameter) requires more expesive laser diodes for light sources ad curretly trasmits gigabits per secod for hudreds of kilometers, makig it the medium of choice for telephoe compaies. The loss of sigal stregth as it passes through a medium, called atteuatio, limits the legth of the fiber. Although sigle-mode fiber is a better trasmitter, it is much more difficult to attach coectors to sigle-mode; it is less reliable ad more expesive, ad the cable itself has restrictios o the degree it ca be bet. The cost, badwidth, ad distace of sigle-mode fiber is affected by the power of the light source, the sesitivity of the light detector, ad the atteuatio rate per kilometer of the fiber cable. Typically, glass fiber has better characteristics tha the less expesive plastic fiber, ad so is more widely used. Coectig fiber optics to a computer is more challegig tha coectig cable. The vampire tap solutio of cable fails because it loses light. There are two forms of T-boxes: 1. Taps are fused oto the optical fiber. Each tap is passive, so a failure cuts off just a sigle computer. 2. I a active repeater, light is coverted to electrical sigals, set to the computer, coverted back to light, ad the set dow the cable. If a active repeater fails, it blocks the etwork. These taps ad repeaters also reduce optical sigal stregth, reducig the useful distace of a sigle piece of fiber. I both cases, fiber optics has the additioal cost of optical-to-electrical ad electrical-to-optical coversio as part of the computer iterface. Hece, the etwork iterface cards for fiber optics are cosiderably more expesive tha for Cat5 copper wire. I 2001, most switches for fiber ivolve such a coversio to allow switchig, although expesive all optical switches are begiig to be available. To achieve eve more badwidth from a fiber, wavelegth divisio multiplexig (W) seds differet streams simultaeously o the same fiber usig differet wavelegths of light, ad the demultiplexes the differet wavelegths at the receiver. I 2001, W ca deliver a combied 40 Gbits/secod usig about 8 wavelegths, with plas to go to 80 wavelegths ad deliver 400 Gbits/secod. The product of the badwidth ad maximum distace forms a sigle figure of merit: gigabit-kilometers per secod. Accordig to esurvire [1992], sice 1975 optical fibers have icreased trasmissio capacity by tefold every four years by this measure. Let s compare media i a example.

22 8.4 Coectig ore Tha Two Computers 583 EXALE Suppose you have 25 magetic tapes, each cotaiig 40 GB. Assume that you have eough tape readers to keep ay etwork busy. How log will it take to trasmit the data over a distace of oe kilometer? Assume the choices are Category 5 twisted pair wires at 100 bits/secod, multimode fiber at 1000 bits/secod, ad sigle mode fiber at 2500 bits/ secod. How do they compare to deliverig the tapes by car? ANSWER The amout of data is 1000 GB. The time for each medium is give below: Twisted pair ultimode fiber Sigle-mode fiber b = = 81,920 secs = 22.8 hours 100 b/sec b = = 8192 secs = 2.3 hours 1000 b/sec b = = 3277 secs = 0.9 hours œ 2500 b/sec Car = Time to load car + Trasport time + Time to uload car 1 km = 300 secs secs = 300 secs secs secs kph = 720 secs = 0.3 hours A car filled with high-desity tapes is a high-badwidth medium! 8.4 Coectig ore Tha Two Computers Computer power icreases by the square of the umber of odes o the etwork. Robert etcalf ( etcalf s Law ) Thus far, we have discussed two computers commuicatig over private lies, but what makes itercoectio etworks iterestig is the ability to coect hudreds of computers together. Ad what makes them more iterestig also makes them more challegig to build. Shared versus Switched edia Certaily the simplest way to coect multiple computers is to have them share a sigle itercoectio medium, just as I/O devices share a sigle I/O bus. The most popular LAN, Etheret, origially was simply a bus shared by a hudred of computers. Give that the medium is shared, there must be a mechaism to coordiate ad arbitrate the use of the shared medium so that oly oe message is set at a time.

23 584 Chapter 8 Itercoectio Networks ad Clusters If the etwork is small, it may be possible to have a additioal cetral arbiter to give permissio to sed a message. (Of course, this leaves ope the questio of how the odes talk to the arbiter.) Cetralized arbitratio is impractical for etworks with a large umber of odes spread out over a kilometer, so we must distribute arbitratio. A first step towards arbitratio is lookig before you leap. A ode first checks the etwork to avoid tryig to sed a message while aother message is already o the etwork. If the itercoectio is idle, the ode tries to sed. Lookig first is ot a guaratee of success, of course, as some other ode may decide to sed at the same istat. Whe two odes sed at the same time, it is called a collisio. Let s assume that the etwork iterface ca detect ay resultig collisios by listeig to hear if the data were garbled by other data appearig o the lie. Listeig to avoid ad detect collisios is called carrier sesig ad collisio detectio. This is the secod step of arbitratio. The problem is ot solved. If every ode o the etwork waited exactly the same amout of time, listeed to be sure there was o traffic, ad the tried to sed agai, we could still have sychroized odes that would repeatedly bump heads. To avoid repeated head-o collisios, each ode whose message was garbled waits (or backs off ) a radom time before resedig. Note that radomizatio breaks the sychroizatio. Subsequet collisios result i expoetially icreasig time betwee attempts to retrasmit, so as ot to tax the etwork. Although this approach is ot guarateed to be fair some subsequet ode may trasmit while those that collided are waitig it does cotrol cogestio o the shared medium. If the etwork does ot have high demad from may odes, this simple approach works well. Uder high utilizatio, performace degrades sice the medium is shared. Aother approach to arbitratio is to pass a toke betwee the odes, with the toke givig the ode the right to use the etwork. If the shared media is coected i a rig, the the toke ca rotate through all the odes o the rig. Shared media have some of the same advatages ad disadvatages as buses: they are iexpesive, but they have limited badwidth. Ad like buses, they must have a arbitratio scheme to solve coflictig demads. The alterative to sharig the media is to have a dedicated lie to a switch that i tur provides a dedicated lie to all destiatios. Figure 8.12 shows the potetial badwidth improvemet of switches: Aggregate badwidth is may times that of a sigle shared medium. Switches allow commuicatio directly from source to destiatio, without itermediate odes to iterfere with these sigals. Such poit-to-poit commuicatio is faster tha a lie shared betwee may odes because there is o arbitratio ad the iterface is simpler electrically. Of course, it does pay the added latecy of goig through the switch, tradig off arbitratio overhead for switchig overhead. Give the obvious advatages, why were t switches always used? Earlier computers were much slower ad so could share media. I additio, applicatios

24 8.4 Coectig ore Tha Two Computers 585 Shared media (Etheret) Node Node Node Switched media (AT) Node Node Switch Node Node FIGURE 8.12 Shared medium versus switch. Etheret was origially a shared medium, ad but Etheret switches are ow available. All odes o the shared media must share the 100 b/sec itercoectio, but switches ca support multiple 100 b/sec trasfers simultaeously. Low cost Etheret switches are sometimes implemeted with a iteral bus with higher badwidth, but high-speed switches have a cross-bar itercoect. such as the World Wide Web rely o the etwork much more tha older applicatios. Fially, earlier switches would take several large boards, ad be as a large as a computer. I 2001, a sigle chip cotais a full 64-by-64 switch, or at least a large slice of it. oore s Law is makig switches more attractive, ad so techology treds favor switches today. Every ode of a shared lie will see every message, eve if it is just to check to see whether or ot the message is for that ode, so this style of commuicatio is sometimes called broadcast to cotrast it with poit-to-poit. The shared medium makes it easy to broadcast a message to every ode, ad eve to broadcast to subsets of odes, called multicastig. Switches allow multiple pairs of odes to commuicate simultaeously, givig these itercoectios much higher aggregate badwidth tha the speed of a shared lik to a ode. Switches also allow the itercoectio etwork to scale to a very large umber of odes. Switches are also called data switchig exchages, multistage itercoectio etworks, or eve iterface message processors (Is). epedig o the distace of the ode to the switch ad desired badwidth, the etwork medium is either copper wire or optical fiber.

25 586 Chapter 8 Itercoectio Networks ad Clusters EXALE Compare 16 odes coected three ways: a sigle 100 b/sec shared media; a switch coected via Cat5, each segmet ruig at 100 b/ sec; ad a switch coected via optical fibers, each ruig at 1000 b/ sec. The shared media is 500 meters log, ad the average legth of each segmet to a switch is 50 meters. Both switches ca support the full badwidth. Assume each switch adds 5 microsecods to the latecy. Calculate the aggregate badwidth ad trasport latecy. Assume the average message size is 125 bytes, ad igore the overhead of sedig or receivig a message ad cotetio for the etwork. ANSWER The aggregate badwidth of each example is the simplest calculatio: 100 b/sec for the shared media; , or 1600 b/sec for the switched twisted pairs; ad , or b/sec for the switched optical fibers. The trasport time is Trasport time = essage size Time of flight Badwidth For coax we just plug i the distace, badwidth, ad message size: 500/ Trasport time = shared , 000 µsecs µsecs = 2.5 µsecs + 10 µsecs = 12.5 µsecs For the switches, the distace is twice the average segmet, sice there is oe segmet from the seder to the switch ad oe from the switch to the receiver. We must also add the latecy for the switch. 50/ Trasport time = swtich µsecs 5 µsecs 3 300, µsecs = 0.5 µsecs + 5 µsecs + 10 µsecs = 15.5 µsecs 50/ Trasport time = fiber µsecs 5 µsecs 3 300, µsecs = 0.5 µsecs + 5 µsecs + 1 µsecs = 6.5 µsecs Although the badwidth of the switch is may times the shared media, the latecy for uloaded etworks is comparable.

26 8.4 Coectig ore Tha Two Computers 587 Switches allow commuicatio to harvest the same rapid advace from silico as have processors ad mai memory. Whereas the switches from telecommuicatios compaies were oce the size of maiframe computers, today we see sigle-chip switches. Just as sigle-chip processors led to processors replacig logic i a surprisig umber of places, sigle-chip switches are icreasigly replacig buses ad shared media etworks. Coectio-Orieted versus Coectioless Commuicatio Before computers arrived o the scee, the telecommuicatios idustry allowed commuicatio aroud the world. A operator set up a coectio betwee a caller ad a callee, ad oce the coectio is established, a coversatio ca cotiue for hours. To share trasmissio lies over log distaces, the telecommuicatios idustry used switches to multiplex several coversatios o the same lies. Sice audio trasmissios have relatively low badwidth, the solutio was to divide the badwidth of the trasmissio lie ito a fixed umber of frequecies, with each frequecy assiged to a coversatio. This techique is called frequecy-divisio multiplexig. Although a good match for voice, frequecy-divisio multiplexig is iefficiet for sedig data. The problem is that the frequecy chael is dedicated to the coversatio whether or ot there is aythig beig said. Hece, the log distace lies are busy based o the umber of coversatios, ad ot o the amout of iformatio beig set at a particular time. A alterative style of commuicatio is called coectioless, where each package is routed to the destiatio by lookig at its address. The postal system is a good example of coectioless commuicatio. Closely related to the idea of coectio versus coectioless commuicatio are the terms circuit switchig ad packet switchig. Circuit switchig is the traditioal way to offer a coectio-based service. A circuit is established from source to destiatio to carry the coversatio, reservig badwidth util the circuit is broke. The alterative to circuit-switched trasmissio is to divide the iformatio ito packets, or frames, with each packet icludig the destiatio of the packet plus a portio of the iformatio. Queuig theory i sectio 6.4 tells us that packets caot use all of the badwidth, but i geeral, this packetswitched approach allows more use of the badwidth of the medium ad is the traditioal way to support coectioless commuicatio. EXALE Let s compare a sigle 1000 bits/sec packet switched etwork with te 100 bits/sec packet-switched etworks. Assume that the mea size of a packet is 250 bytes, the arrival rate is 250,000 packets per secod, ad the iterarrival times are expoetially distributed. What is the mea respose time for each alterative? What is the ituitive reaso behid the differece?