Methodology of the direct measurement of the switching latency

Transcription

1 Tomas HEGR, Leos BOHC, Zbynek KOCUR, Miroslav VOZK, Per CHLUMSKY Czech Technical Universiy in Prague (), VSB Technical Universiy of Osrava () Mehodology of he direc measuremen of he iching laency bsrac. The aricle provides a measuremen mehodology based on he relaed RFCs. I brings a soluion of deermining he iching laency on he physical layer using common measuring devices. s a proof of concep were done a number of he experimenal measuremens, including analysis of he resuls. Swiching laency is an imporan performance parameer which paricipaes in he decision-making wheher o deploy he ich o low-laency environmens. This is especially imporan in indusrial neworks for real-ime sysems involving smar grids. Deermine he value of he iching laency is also an imporan sep in he evenual deploymen of he OpenFlow echnology on his field. Sreszczenie. Zaprezenowano meodę badania opóźnień w przełączaniu (iching laency) bazująca na RFC. Paramer en jes bardzo ważny w sieciach przemysłowych łącznie z sieciami ypu smar grid. (Meoda bezpośredniego pomiaru opóźnień przełączania w sieciach kompuerowych) Keywords: iching laency, measuremen mehodology, OpenFlow, indusrial neworks, low-laency, Smar Grids. Słowa kluczowe: opóźienie przełączania, sieci kompuerowe. Inroducion Massive global usage of Eherne as a ransmission echnology in local area neworks has led o is use in oher specialized secors and exacing environmens. These lowlaency environmens lay grea demands on he ransmission ime of criical messages inerchanged beween real-ime sysems. Criical fields of applicaion are Eherne neworks in subsaion auomaion (S). The sandard IEC 6850 deals wih hese ypes of indusrial neworks. I defines ha a nework as a whole has o ensure he ransmission delay less han 3 ms for GOOSE (Generic Objec Oriened Subsaion Even) ype messages. GOOSE messages conain informaion abou incidens, as described in he IEC [] or can carry sampled values from he IEC []. General requiremens for he ransmission delay of differen raffic ypes are defined in he IEEE Sd [3]. Grea demands placed on he ransmission delay in S neworks have requesed a ransmission of he GOOSE messages conen direcly on he second layer of ISO/OSI model wihou deceleraion arising from usage of he higher layers. This is he reason why i is possible o send and receive messages only wihin he bounds of local area nework. For solving he problem of ransmission delay of criical messages in he indusrial neworks here is ineresing he OpenFlow echnology. I is increasingly geing ino he producion environmen and allows cenrally managing daa flows in he nework from a conroller. I enables sysemaic compuing, implemenaion of opimal daa flow pahs and reserving ransmission resources for boh unicas and mulicas raffic. Swiching laency is an imporan parameer in he deploymen of he OpenFlow echnology [4]. The ransmission ime is no he only crucial requiremen for criical messages. We have o consider also is reliabiliy. Therefore, by is very naure here is used sore-and-forward soluion o he cu-hrough. Values of delay are ofen menioned by manufacurers in various forms and under differenly defined condiions which are no sufficien in highly challenging environmen. The iching laency can be measured by various mehods bu ofen only hrough very specialized and expensive measuring equipmen. Therefore i has been decided o creae a measuring mehodology direcly on a ransmission medium, in his case on he physical layer [5], [6]. The aim was o develop a measuremen mehodology ha would allow a delay deerminaion of he iching fabric for all common daa raes used in local neworks. amely i is he echnology 0Base-T, 00Base-TX and 000Base-T. The measuremen was designed under zero ich loads o avoid he influence of frame soring in queues and congesion. The resuls obained from experimenal measuremens have demonsraed an applicabiliy of his measuring mehodology and heir analysis brough some ineresing resuls. n accuracy of he measured laencies is in enhs of microseconds which is sufficien for comparison of he individual devices. Sandards and relaed works Requiremens for he ransmission delays are defined for example in he aforemenioned sandards [-3]. For purposes of he mehodology here have been used relaed RFCs published by he IETF RFCs which are coupled o his opic. The fundamenal descripion of he iching laency is based on RFC 4 [7]. In his RFC here is defined he laency of he sore-and-forward devices. Thus, processing frame ime is deermined as he saring ime inerval when he las bi of he inpu frame reaches he inpu por and as he ending ime inerval when he firs bi of he oupu frame is seen on he oupu por. This approach is ypically called LIFO. The second ype is cuhrough approach which uses he FIFO mehod. Cuhrough devices are ypically used in daa ceners and in oher compuing clusers (HPC). Furher documen ha covers he measuremen mehodology is universally known as RFC 544 [8]. Wide range of specialized measuring equipmen implemens his recommendaion. There are described, iner alia, ime inervals beween aking individual measuremens and also frame lenghs needed for measuremen. The number of reading repeiion is deermined o a leas 0. The RFC 899 of 000, which describes he area under examinaion, only specifies he laency a iching broadcas frames and i is no applicable o his mehodology [9]. Oher published works such as [0], which deals wih QoS and raffic shaping, use for laency measuremen synchronizaion via TP proocol. This soluion is no sufficienly accurae for measuremens a higher daa raes. For achieving beer device synchronizaion here can be used IEEE 588 bu a a cos of furher expenses on addiional hardware. Deploymen opions of such synchronizaion under laboraory condiions are described in he paper []. Swich archiecure and measuremen limis The archiecure of various ypes of iches can differ widely and i is a par of he know-how of each manufacurer. Generally, he ich can be viewed in wo perspecives. The firs is given by an arrangemen of physical componens in a real device. The second is he PRZEGLĄD ELEKTROTECHICZY, ISS , R. 89 R 7/03 59

2 iching logic iself. Thus, how a ich pus frames ino queues, how i schedules heir forwarding and how i implemens all his funcions ino a memory. From he hardware poin of view a general ich is composed of line cards, CPU and differen srucure memories (SRM, CM, TCM). The arrangemen is shown in Fig.. ll componens are conneced by inernal bus on he ich backplane. The backplane can become a boleneck if he archiecure is designed insufficienly. The line card conains a leas inerface for signal processing a he physical layer (PHY) and medium conrol access o he ransmission medium (MC). The archiecure of modular iches and large enerprise iches differs in boh backplane design and line card consrucion eiher. They are usually exended by addiional CPUs and memories. Fig.. The physical arrangemen of he componens of a general ich. The second view is from he poin of frame processing and memory usage. ll iches which were experimenally measured, and are widespread nowadays, use he shared memory. Due o he uilizaion of resources and he bes raio of delay o hroughpu oday i is very favoured he archiecure shown in Fig.. I is called CIOQ (Combined Inpu and Oupu Queuing wih Virual Oupu Queuing) []. In his archiecure he incoming frames are arranged ino a shared memory dedicaed o appropriae oupu por queues (Virual Oupu Queues). This soluion prevens head of he line blocking (HOL). fer processing he frame i is passed o he oupu queue of desinaion por. [3]. The minimum measurable iching laency of for he archiecure can be esimaed as in expression (): () lc sf iq oq where: oal iching laency, lc line card delay (PHY, MC), sf iching fabric delay, iq VOQ delay, oq oupu queue delay. Making usage of memories and heir arrangemen can vary considerably. Today s iches are no only designed o forward frames by MC addresses. Swiches provide more sophisicaed feaures in recen years such as ccess liss, QoS, L3 forwarding and even more. The general ich finds he oupu por by desinaion MC address is done via CM (Conen-ddressable Memory) which cells ake only binary saes. To allow advanced funcions Ternary CM have been inroduced. TCM provides he hird sae represening he do no care value. This sae allows using a mask for rouing or forwarding and allows he creaion of access rules wihou he need for soring hem in he memory for each individual address. The TCM has been opimized for a long ime and for specific applicaions. Tha is he reason why i is no possible o mach all OpenFlow uples by he hardware pah. The evaluaion of hese flow rules mus be realized by he sofware pah. Wheher he flow rule will be evaluaed in sofware or hardware depends on he used se of OpenFlow uples and implemenaion of a paricular ich. I is expeced ha he delay achieved by sofware processing ouside a TCM will be higher. Measuremen mehodology The fundamenal principle of he measuremen mehodology is o compare he ime difference beween he inpu and oupu ich por using he FIFO mehod direcly a he physical layer. ll measuremens mus be done using he 0Base-T Eherne on he measured inerfaces. This varian uses Mancheser encoding and ransmission channel which is no burdened by any oher broadcasing in he res. Due o hese characerisics here can be achieved unambiguous idenificaion of he passing frame. There is always signal broadcased on channel of varians 00Base- TX and 000Base-T o keep sender and receiver synchronized. Thus, i is no possible o deermine he ail and he head of he passing frame only by he oscilloscope. Fig. 3. The wiring of he measuring workplace wih acive differenial probes for he firs measuring mehodology of 0Base-T. Fig.. The CIOQ archiecure sors incoming frames ino virual oupu queues according o desinaion por and hen ino oupu queues. From all he menioned facs i is clear ha he overall processing ime of frame ransmission beween he inpu and oupu por is composed of a wide range of delays. Thereby he inegraed design of he iches in one box is no possible o measure he laency of he iching elemen direcly by means of CFrames, as suggesed in Tes raffic consiss of ICMP packes and is generaed by he sender saion using common ping applicaion. I is sufficien for measuring purposes and i allows laency measuremen wihou any addiional specialized sofware. On measured iches all heir unnecessary feaures generaing unwaned raffic mus be disabled. I would no be possible o idenify he es frame unambiguously. The unwaned raffic could fill up queues and cause disorion of he measured daa. I is primarily generaed by services like STP, RSTP, mulicas and he ime synchronizaion (IEEE 588 or TP). I is also appropriae o se saic MC address enries o reduce he RP. Wiring necessary for 60 PRZEGLĄD ELEKTROTECHICZY, ISS , R. 89 R 7/03

3 measuremen was designed in wo differen varians. The firs diagram is shown in Fig. 3. I is inended for measuring he iching laency of Eherne 0Base-T. Measuremen of he ime difference beween channels is performed on he oscilloscope which is conneced by acive differenial probes direcly o he ransmission medium. The probes have an inpu impedance of 00 Ω. These probes are commercially available, bu for experimenal measuremen were used prooypes creaed in anoher projec [4]. If i is possible, i is necessary o deacivae pors uomaic MDI/MDI-X feaure (pair apping). This can cause considerable difficulies by deerminaion of ransmiing and receiving pairs of symmeric lines a he ich inpu and oupu por. The measuremen is usually carried ou on he TD+ and TD- pair before and afer he ich in he direcion from sender o receiver of he es frame. Readings of he laencies are made according o RFC 544 in series of differen frame lenghs (64 B, 8 B, 56 B, 5 B, 04 B, 80 B, 58 B). The number of repeiions mus be a leas 0imes for each series of measuremen; hey should be carried ou in regular inervals of 60 sec. and in flow duraion a leas 0 sec. Higher number of repeiions reduces he saisical error of measuremen. n Oscilloscope ofen provides he auomaic delay readings. lernaively, i can be conneced o a PC and readings may be implemened programmaically. The digial oscilloscope mus have sufficien sampling frequency o mee he yquis-shannon sampling heorem (minimum sampling rae a 00 MS/s). The hreshold volage level is 500 mv and i is based on he resisance of he used probe. The selecion of he appropriae measuring pors is widely described by RFC 889. Experimenal measuremens were always carried ou only beween neighbouring and ouermos ich pors. For measuring he iching laency of oher Eherne ypes i was necessary o exend wiring diagram. By involving wo auxiliary devices, which use 0Base-T on inpu and oupu inerface here was achieved he possibiliy o measure 00Base-TX and 000Base-T Eherne. The diagram is in Fig. 4. This arrangemen is no longer direc wihin he meaning of measuring laency on he inpu and oupu por of he examined device bu sill i is a direc measuremen of he delay on he physical layer. firs i is necessary o measure he characerisic delay beween he auxiliary iches wihou insering he measured ich. The measuremen of he characerisics is he same procedure as described above for all frame lenghs and examined daa raes. I is hen possible o connec measured ich beween hose auxiliary ones and ake all measuremen series again. Fig. 4. The wiring of he measuring workplace for varians of 00Base-TX and 000Base-T. SWUX are auxiliary iches and SWMES is he examined ich. From he se of values obained by measuring individual series i is calculaed he mean value of he iching laency. For he firs ype of measuremen assuming a normal disribuion i is possible o use he expression (). I is necessary o cleanse resuls obained from he second measuremen mehodology of pre-measured characerisics of he auxiliary devices and ime required o ransfer frame hrough he newly formed segmen. I canno be included in he pre-measured characerisics and mus be compued. Since he measuremen iself is carried ou on he physical layer, i is used o compue he symbol rae ha is in boh cases 5 MBaud. Furhermore, i is necessary o compue he acual size of he frame given by applied coding raio. The compuaion of he mean value for given frame lengh can be performed using (3): () mesi i (3) i mes i aux l pr l r es s e where: number of measuremens in he series [-], mes value of laency from one measuremen [s], aux mean delay wih aux. iches [s], l pr preamble lengh (64 bis) [b], l es lengh of es frame [b], e r coding raio (4B/5B, 8b/0b) [-], r s symbol rae [Baud], mean value of he iching laency [s]. Type - sandard uncerainy of measured resuls for he firs measuremen mehodology can be evaluaed as sample experimenal sandard deviaion of he mean wrien down in (4). (4) u i ( ) ( ( ) ) mes i For he second measuremen mehodology may be used he same expression o deermine he sandard deviaion of he mean. However, he ype sandard uncerainy of measured resuls mus be evaluaed as he sum of he squares of he sandard deviaion s of measuremen wihou insered measured ich and wih i as in expression (5) [5]. (5) u ( ) s s aux The expanded sandard measuremen uncerainy hen can be deermined by a known expression (6). There u B ( ) corresponds o a sandard uncerainy of ype B primarily caused by he specific characerisics of he used oscilloscope as he sampling rae, resoluion and more. The signal propagaion delay on he wire can be negleced. For example if he lengh of he wire is m han he propagaion delay is up o 50 ns. To achieve a 95% probabiliy ha he searched value is in he given inerval, i is necessary o muliply he value of he expanded uncerainy u C ( ) by coverage facor k =. (6) u ( ) u ( ) u ( ) C nalysis of experimenal measuremens Experimenal measuremens were carried ou by he Tekronix DPO403 oscilloscope wih a sampling frequency.5 GS/s. chieved expanded uncerainy of measuremen of 00Base-TX having a coverage facor was for mos of he measured devices up o 0.5 µs. The expanded uncerainy for 0Base-T varian was in 0. µs due o more sraighforward mehodology of measuremen. For 000Base-T varian is he obained value in 0.5 µs of uncerainy oo large because is relaive value reaches 50%. The characerisic delays beween auxiliary iches (xrb0ls-i) had linear progression. Using he linear B r PRZEGLĄD ELEKTROTECHICZY, ISS , R. 89 R 7/03 6

4 regression were esimaed parameers shown in Tab.. From he able is obvious ha he coefficien of deerminaion R is very close o.00 so here is a high correlaion beween he oucomes are and heir prediced values. laency on frame lengh for all measured iches. s represenaive resuls here were seleced series of frame lenghs 64 B, 5 B and 58 B, hus he maximum Eherne frame size. Table. Linear regressed funcions Eherne ux. linear funcion R 0Base-T y=0.7993x+35, Base-TX y=0.080x+5, Base-T y=0.008x+5, The resuls of experimenal measuremens can be summarized in a performance division shown in Tab.. In he calculaion here are included worse resuls if here were carried ou more measuremens o several pors on a ich. Groups and B are he performance group (in erms of iching laency) ha are incompaible due o he large difference in he measured values of laency. The resuls were obained in more han six housand measuremens on foureen differen iches. Due o he lack of represenaives he resuls do no include measured iches which grealy exceeded he value of corresponding performance group. Table. Laency performance groups Eherne Group Mean [µs] Sandard dev. [µs] 64 B 5 B 58 B 64 B 5 B 58 B 000Base-T Base-TX B Base-T B Measuremens of aging 0Base-T brough he mos sraighforward resuls using he firs mehodology. Obained resuls of all measured iches are in Fig. 5. The values of ich laency are mainly in area of 0 µs. The only excepions are surprisingly Hirschmann indusrial iches ha show sharply higher values. In he firs case he Hirschmann MS0/MM3 has consanly higher laencies which could be caused by is modular archiecure and probably slow backplane. In he second case he Hirschmann RS FX/FX iching laencies more correspond o a sofware bridge mode hen o a L ich. Is progression is very similar o he RB0LS in a bridge mode which was measured o comparison. Such a soluion is for deploymen in indusrial neworks compleely inappropriae because he iching laency grows linearly wih he lengh of he frame. Fig. 6. Measured iching laencies for Eherne 00Base-TX. s menioned above, iches can be divided ino several groups according o he iching laency. In he firs group he iching laency is up o 3 µs (group ). In he second group i is over 5 µs (group B) and in he las one over 0 µs (poenial group C). In his group is he Hirschmann RS FX/FX indusrial ich again. Is iching laency sill increases linearly like in a bridge mode whereas he second Hirschmann ich has moved o he second hird of he char. Oher indusrial iches proved hemselves especially in he sabiliy of iching laency which was ofen nearly consan. Inexpensive home devices also showed low values when iching bu hey did no provide any oher services required in indusrial neworks. Some of he office and enerprise iches unexpecedly showed he opposie rend of decreasing laency wih increasing frame lengh. I is probably caused by inner iching opimizaion. The las resuls of 000Base-T were obained only for 6 iches. lmos all iches indicae laency values less han 5 µs. Only he RB0LS-I in a bridge and he HP 5406zl show higher laencies. The las one was measured wih he early revision firmware K which could cause hese higher values. Resuls are shown in Fig. 7. Fig. 7. Measured iching laencies for Eherne 000Base-T. Fig. 5. Measured iching laencies for Eherne 0Base-T. From he perspecive of he use frequency in indusrial neworks he 00Base-TX Eherne is paricularly imporan. The graph in Fig. 6 shows he dependence of iching In he view of poenial applicaion of he OpenFlow echnology is he sable value of iching laency a big advanage in he sense of downloaded flow rules execuion. If a conroller implemens he flow rules o he ich and hey are processed in hardware pah, no more delays will arise. For comparison where was carried ou anoher se of measuremens which should show a difference in he operaion of he general L ich and idenical OpenFlow soluion. The rules in he flow able were injeced 6 PRZEGLĄD ELEKTROTECHICZY, ISS , R. 89 R 7/03

5 proacively by he conroller before he measuremen in order o avoid is influence. Since iches from HP are no able o implemen hardware maching based on MC addresses due o he implemenaion of TCM, his evaluaion is done hrough he sofware pah [6]. From Tab. 3. i is obvious ha he iching laency of sofware processing of OpenFlow rules based on MC addresses is up o 00 µs. In comparing laency values wih and wihou OpenFlow here is an increase up o 7000%. Such laency in larger opologies may resul in failure of ime requiremens of criical raffic even in case of zero load. Table 3. The comparison of he OF and non-of modes Exp. Sd. Dev. of Mean [µs] he mean [µs] Swich Mode 64 B 5 B 58 B 64 B 5 B 58 B 000Base-T OF 000Base-T HP 00Base-TX zl OF 00Base-TX Base-T OF 0Base-T HP E Base-T OF 000Base-T Base-TX OF 00Base-TX Base-T OF 0Base-T Conclusions The described mehodology provides measuring soluion in he field of he deerminaion he minimum achievable iching laency using a digial oscilloscope and oher widely available devices. I is possible o verify he suiabiliy of he ich deploymen in indusrial environmens wihou he required informaion from he manufacurer. Resuls obained from experimenal measuremens have demonsraed ha he mehodology of measuremen is able o give very accurae resuls wih he expanded measuremen uncerainy of 0.5 µs for he mos common indusrial 00Base-TX Eherne neworks. Furhermore, he measuremens revealed possible problems caused by using iches wih a srong linear relaionship beween he frame lengh and he iching laency similar o a bridge mode. Wih he proposed mehodology i was possible o verify he suiabiliy of he OpenFlow deploymen in low-laency environmens. lhough he daa flows can be conrolled by using rules based on IP uples, i has no o be sufficien i.e. for L mulicas. I is used very much in S neworks for is sraighness and low overhead which leads o lower delays. Resuls also clearly show ha he applicaion of he echnology OpenFlow echnology is no recommended for such a purpose of evaluaing flow rules by he sofware pah. The possible enhancemen o his measuremen mehodology is in obaining he iching laency when forwarding differen load levels corresponding o he real operaion in indusrial neworks. REFERECES CKOWLEDGEMETS This work was suppored by Gran Projec SGS/86/OHK3/3T/3 and by he European Communiy s Sevenh Framework Programme (FP7/007-03) under gran agreemen no [] IEC. Communicaion neworks and sysems in subsaions: Specific Communicaion Service Mapping (SCSM) Mappings o MMS (ISO and ISO 9506-) and o ISO/IEC s ed. Geneva (003). [] IEC. Communicaion neworks and sysems in subsaions: Specific Communicaion Service Mapping (SCSM) Sampled values over ISO/IEC s ed. Geneva (003). [3] IEEE sandard communicaion delivery ime performance requiremens for elecric power subsaion auomaion. ew York (005). [4] S k o r pil V., Precechel R., Training a eural ework for a ew ode Elemen Design. Przeglad Elekroechniczny, Volume 89 (03), Issue B, [5] Fazio P., De Rango F., Velri F., Marano S., Performance evaluaion of he packe error rae of DS-SS physical layer in UWB neworks. Canadian Conference on Elecrical and Compuer Engineering, Oawa (006), [6] Komosny D., Balej J., Sahu H., Shukla R., Dolezel P., Cable lengh based geolocalisaion. Przeglad Elekroechniczny, Volume 88 (0), Issue 7, 6-3. [7] RFC 4, Benchmarking Terminology for ework Inerconnecion Devices. IETF (99). [8] RFC 544. Benchmarking Mehodology for ework Inerconnec Devices. IETF, (999). [9] RFC 889. Benchmarking Mehodology for L Swiching Devices. IETF, (000). [0] Loeser J., Haerig H., Low-laency hard real-ime communicaion over iched Eherne. In Proceedings Euromicro Workshop on Real-Time Sysems, Caania (004). 3-. [] Pravda M., Lafaa P.,Vodrazka J., Precision Clock Synchronizaion Proocol and Is Implemenaion ino Laboraory Eherne ework. In Proc.33rd Inernaional Conference on Telecommunicaion and Signal Processing, Baden near Vienna (00) [] Chao H., Liu B., High performance iches and rouers. Wiley-IEEE Press (007). [3] P o u rsepanj., Benchmarks Rae Swich-Fabric. In: Communicaion Sysem Design (003). [4] H a v lan M., cive differenial probes. In ccess Server, Prague (007). vailable: hp://access.feld.cvu.cz. [5] JCGM 00:008. Evaluaion of measuremen daa Guide o he expression of uncerainy in measuremen. BIPM: Join Commiee for Guides in Merology (008). vailable: hp:// _008_E.pdf [6] OpenFlow suppor wih HP Procurve 5400zl series. In: OpenFlow, 0. vailable: hp:// uhors: MSc. Tomas Hegr, Czech Technical Universiy in Prague, Faculy of Elecical Engineering, s. Technicka, 6000 Prague, Czech Republic, omas.hegr@fel.cvu.cz ; MSc. Leos Bohac, Ph.D., Czech Technical Universiy in Prague, Faculy of Elecical Engineering, s. Technicka, 6000 Prague, Czech Republic, leos.bohac@fel.cvu.cz; MSc. Zbynek Kocur, Czech Technical Universiy in Prague, Faculy of Elecical Engineering, s. Technicka, 6000 Prague, Czech Republic, E- mail: zbynek.kocur@fel.cvu.cz; ssoc. Prof. Miroslav Voznak, Ph.D., VSB Technical Universiy of Osrava, Faculy of Eelecrical Engineering and Compuer Science, s. 7. Lisopadu 5, Osrava, Czech Republic, voznak@ieee.org; MSc. Per Chlumsky, Czech Technical Universiy in Prague, Faculy of Elecical Engineering, s. Technicka, 6000 Prague, Czech Republic, per.chlumsky@fel.cvu.cz PRZEGLĄD ELEKTROTECHICZY, ISS , R. 89 R 7/03 63