Enabling a Converged World. 10-Gigabit Ethernet Switch Performance Testing

Transcription

1 Enabling a Converged World 10-Gigabit Ethernet Switch Performance Testing

2 White Paper 10-Gigabit Ethernet Switch Performance Testing sample test plans included

3 Copyright Ixia. All rights reserved. The information in this document is furnished for informational use only, is subject to change without notice, and should not be construed as a commitment by Ixia. Ixia assumes no responsibility or liability for any errors or inaccuracies that may appear in this document. Ixia and the Ixia logo are trademarks of Ixia. All other companies, product names, and logos are trademarks or registered trademarks of their respective holders. Ixia W. Agoura Road Calabasas, CA Phone: (818) Fax: (818) info@ixiacom.com Internet: 2 Copyright 2004, Ixia 10-Gigabit Ethernet Switch Performance Testing

4 Contents Abstract...5 Introduction Gigabit Ethernet LAN Gigabit Ethernet Metro Gigabit Ethernet WAN...6 What Is Inside a Multi-10GE Port Switch?...7 Packet buffer...7 Packet processing function...7 Classification tables...7 Context memory...7 Traffic management function...7 Switch fabric...7 What Happens to a Packet When It Enters a 10GE Port?...8 Storing the packet...8 Processing the packet...8 Metering and statistics recording Packet modification Traffic management Switch fabric What Are the Challenges in Processing Packets at 10G Rates? Stress Point 1: Ingress packet buffer Stress Point 2: Packet classification Stress Point 3: Traffic management Stress Point 4: Crossbar switch and backplane interconnect Stress Point 5: Multicast replication and queues Stress Point 6: Control plane Developing the Right Test Methodology Test methodology example Module-level testing System-level testing Test methodologies Anticipating the behavior of the 10GE switch Ixia s Test Solution IxExplorer IxScriptMate Appendix: 10GE Testing Examples Throughput Testing: RFC 2544 Throughput Test Testing for Latency: RFC 2544 Latency Tests Testing how 10GE ports handle prioritized streams based on QoS parameters GE OSPF Convergence Test Glossary Acknowledgements Gigabit Ethernet Switch Performance Testing Copyright 2004, Ixia 3

5 4 Copyright 2004, Ixia 10-Gigabit Ethernet Switch Performance Testing

6 10-Gigabit Ethernet Switch Performance Testing Abstract The first generation of 10-Gigabit Ethernet (10GE) switches, introduced in early 2002 after IEEE 802.3ae standardization, proved to be subpar. Some switches barely reached 50 percent of line rate throughput, even for large packet sizes, and exhibited poor latency, limited feature sets, and high per-port costs. The second generation of 10GE products has arrived, with substantial packet processing capabilities that enable additional services. Key functions in this technology include performing the necessary packet classification, header modification, policing of flows, and queuing/scheduling all at wire-speed rates. This white paper reviews the common building blocks of a multiple line card 10GE switch, identifies various stress points within the switch, and offers various test methodologies to test these stress points. Introduction Ethernet networking technology is now virtually ubiquitous; it has evolved from 10Base-T (IEEE 802.3) in 1983, Fast Ethernet (IEEE 802.3u) in 1995, 1-Gigabit Ethernet (802.3z) in 1998 to 10-Gigabit Ethernet (802.3ae) in The 10- Gigabit Ethernet standard provides a significant increase in bandwidth while maintaining compatibility with the installed base of standard interfaces, to protect existing investments in Ethernet technology. The IEEE 802.3ae 10-Gigabit Ethernet standard defines both LAN and WAN physical layers, the latter of which is compatible with existing SONET infrastructure. Ethernet not only dominates the LAN market, but is also taking hold in the Metro Area Network (MAN) market. It has extended into the WAN arena as both its distance and capacity has increased. Ethernet at 1-Gigabit and 10-Gigabit speeds has attracted increasing attention from carriers operating in the MAN and WAN market segments. After nearly two years of slow growth during 2001 and 2002, worldwide Layer 2 and Layer 3 Ethernet switch revenue began to accelerate in mid-2003, and is expected to grow from $11.8 billion in 2004 to $15 billion in 2006, according to Infonetics Research's quarterly market share and forecast service, L2 L3 Ethernet Switches. Worldwide port shipments are predicted to grow 75 percent from 161 million in 2003 to 279 million in The strongest growth in the Ethernet switch market has come from 10GE chassis ports and revenue, driven by 10GE deployment in service provider metro networks, and by the proliferation of 1GE to desktops/laptops as a standard interface. 10-Gigabit Ethernet has been developed to support a wide range of applications from the enterprise network, through the edge and metro network, and into the wide area network. 10-Gigabit Ethernet Switch Performance Testing Copyright 2004, Ixia 5

7 10-Gigabit Ethernet LAN The biggest market for 10-Gigabit Ethernet is likely to be in the corporate backbone. The cost of 1-Gigabit and 10-Gigabit Ethernet has dropped significantly, which will trigger the demand for 10GE services and products. In the enterprise network, 10-Gigabit Ethernet is used for: Switch-to-switch links for very highspeed connections within the same wiring closet or data center or between different buildings. Aggregation of multiple-fiber 1000BASE-X or copper 1000BASE-T segments into 10GE uplinks. Nearly all new desktops and laptops are now shipped with 10/100/1000BASE-T ports. System Area Networking (SAN) applications providing server interconnects for clusters of servers. 10-Gigabit Ethernet plays a key role in interconnecting the new generation of high-performance servers with multigigahertz 2/4/8-way processors required for moving large amount of data between server farms. 10-Gigabit Ethernet Metro At the Metro Area Network (MAN) level, service providers face tremendous pressure to expand capacity for broadband local access, high-speed WANs, storage, and campus /enterprise grids. Standardsbased 10-Gigabit Ethernet enables service providers to extend their 10GE networks into the metro edge using their dark fiber, especially when 10GE is combined with metro area optical fiber networks based on Wave Division Multiplexing (WDM). 10GE MANs can be deployed in either a star or ring topology. Unlike Resilient Packet Ring (RPR) networks, 10GE networks use the standard Ethernet MAC protocol. The latest 10GE metro switches can provide network reliability similar to that of networks based on SONET/SDH rings. 10-Gigabit Ethernet WAN 10GE WAN applications are similar to MAN applications, but leverage existing SONET networks rather than new infrastructure. SONET is the dominant transport protocol in the WAN backbone today, and most MAN public services are offered over SONET networks. The 10-Gigabit Ethernet interface (WAN PHY) is compatible with SONET-based Time Division Multiplexing (TDM) and with the payload rate of OC-192c/SDH VC-4-64c. The 10-Gigabit WAN interface facilitates the transport of native Ethernet packets across the WAN transport network, with no need for protocol conversion. This not only improves the performance of the network, but makes it simpler and less costly to manage. Ethernet s flexibility and universal availability will fuel the demand for 10- Gigabit Ethernet deployment in multiple network segments, from corporate networks, to data centers, and throughout service provider networks. This ongoing demand will require system vendors to deliver high-performing 10-Gigabit systems that meet service providers stringent requirements requirements that ensure high availability to corporate networks and performance that meets service level agreements (SLAs). 6 Copyright 2004, Ixia 10-Gigabit Ethernet Switch Performance Testing

8 What Is Inside a Multi-10GE Port Switch? Testing the new generation of 10GE switches requires a clear understanding of the various building blocks that make up a switch, and their interaction, to determine the various stress points within a switch and identify the weakest link in the chain. Figure 1 shows the major components of a 10GE switch, including the line card, the switch fabric card, and the controller card. The following subsections provide an overview of the 10GE switch/router components crucial to performance testing. Packet buffer The packet buffer is a temporary repository for arriving packets while they wait to be processed. Packet processing function The packet processor is an optimized ASIC or programmable device (Network Processing Unit, or NPU) for processing and forwarding packets in the data plane or fast path. It performs specific key tasks such as parsing the header, pattern matching or classification, table look-ups, packet modification, and packet forwarding, ideally at wire speed. Classification tables The classification table is a special memory used by the packet processor. It may contain the following databases: The routing table determines where to route incoming packets. Access Control Lists (ACLs) contain information to grant or deny permission to specific users or groups The flow classification table contains information about a particular user or group of users, protocols, and applications. Flow identification information is used to determine QoS treatment, packet policing, and perflow queuing and billing. The label table contains information about VLANs, stacked VLANs, MPLS labels, etc. Context memory Context memory contains instructions about whether to deny or forward packets, where to forward them, all the internal system headers needed to get a packet from an ingress to an egress port, and the external packet header (new MAC, IP, MPLS stacks, etc.). It also holds information relevant to metering the packet and other miscellaneous information about how to process a packet. Traffic management function The traffic management function is used to regulate the flow of traffic. It forwards traffic according to a user-defined set of rules pertaining to priority levels, latency and bandwidth guarantees, and congestion levels. It also provides the buffering required to work with any queuing mechanisms it uses to manage traffic flow across the switch fabric. It may also include the high-speed SERDES (serializer/deserializer) function used to connect to the switch fabric. Switch fabric The switch fabric provides data plane interconnection among all line cards in the system. The switch fabric typically employs a crossbar to move packets between its ingress and egress ports. An NxN crossbar switch fabric allows N line cards in a system to be interconnected in a chassis. This allows each slot to simultaneously send and receive traffic over the switch fabric. 10-Gigabit Ethernet Switch Performance Testing Copyright 2004, Ixia 7

9 What Happens to a Packet When It Enters a 10GE Port? Figure 1. 10GE switch building blocks. This section provides a high-level walkthrough of the life of a packet as it goes from a port on an ingress line card through the switch fabric card and exits on an egress line card. Figure 2 shows the logical operation of various line cards in a multi-10ge port switch, and identifies the stress points (numbered red circles). Storing the packet When a packet arrives on a 10GE port, it is stored in ingress buffer memory while waiting to be processed by the packet processor. When the packet processor is ready, the packet header is copied into the packet processor memory for processing. Processing the packet The packet processor (ASIC or NPU) examines the packet to determine whether a packet should be filtered or forwarded. It makes this determination by parsing the packet header and then performing packet and flow classification. The classification process maps information extracted from the packet header to information stored in local tables maintained by the control plane processor. The information in the classification tables typically includes forwarding tables, routing tables, and profiles or rules for a given packet or flow, such as policies for ACLs, Quality of Service (QoS), and Class of Service (CoS). The classification typically points to many fragments of information about a packet. This information is usually saved in context memory and may contain instructions about whether to deny or forward the packet, where to forward it, all additional header information to get a packet from ingress to egress port, the new header used when it leaves the egress port (new MAC, IP, MPLS stacks, etc.), information relevant to policing the packet (see "Metering and statistics recording" below), and other information about how to process the packet. 8 Copyright 2004, Ixia 10-Gigabit Ethernet Switch Performance Testing

10 Figure 2. Logical operation of line cards in a multi-10ge port switch with stress points. 10-Gigabit Ethernet Switch Performance Testing Copyright 2004, Ixia 9

11 Metering and statistics recording Packet metering checks the conformance of a flow to a subscribed traffic policy or contract. The metered information is then used to decide whether to discard or forward the packet. Bandwidth allocations may be established in contracts that specify the per-flow, steady state, and short term peak (burst) rates. Typically, a Token Bucket Algorithm is used at the ingress port to check the compliance of each flow to its contract. (A detailed discussion of the Token Bucket Algorithm is outside the scope of this document). The metered packets that violate the assigned contract can be marked for a range of treatments: unconditional drop, drop if no internal resources available, or pass on the packet dropping decision to the downstream device. The marking of packets can also be used by the egress line card to decide whether to delay packets within a packet stream in order to conform to some defined traffic profile, referred to as traffic shaping. Traffic shaping is used to smooth out packet flows and regulate the rate and volume of traffic admitted to the network. Packet modification Based on the outcome of the classification process, the next step is to modify the packet header and determine the packet s egress port. Context memory contains instructions about whether to deny or forward packets, where to forward them, and all the internal system headers needed to get a packet from an ingress to an egress port. On the egress line card, a new external packet header (new MAC, IP, MPLS stacks, etc.) will be inserted by the egress packet processing function. Traffic management The packet processor typically appends an internal forwarding header to each packet that it forwards to the traffic management device. This header contains information that tells the ingress/egress traffic managers where to forward the packet (which port and line card) and the characteristics of the flow. The traffic manager then passes through or discards flagged packets, depending on its policies for the ingress direction. Depending on the architecture, forwarded packets are organized into FIFO (First-In, First-Out) queues, input queues, or Virtual Output Queues (VOQs). Queued traffic is then scheduled for transmission to the switch fabric based on a scheduling algorithm for example, Weighted Round Robin (WRR), Weighted Fair Queuing (WFQ), etc. Depending on whether or not the switch fabric supports fixed or variable length packets, packet segmentation into fixed cells may occur before packets are transmitted onto the backplane. The ingress traffic manager typically interacts with the switch fabric and/or egress traffic manager to meet switch fabric scheduling requirements and avoid congestion. On the egress side, the traffic manager first performs a reassembly operation if segmentation was performed on the ingress side. Received packets are then placed into an output queue based on information from the header that was appended at ingress. If per-flow queuing is supported, the egress traffic manager maintains an additional output queue for each flow. Each output queue provides traffic shaping to maintain conformance to QoS policy. Switch fabric The switch fabric functions as a highperformance data plane interconnect between its ingress and egress ports. Most new crossbar switch architectures have embedded high speed SERDES (serializer/ deserializer) interconnecting line cards through centralized switch devices. A packet entering a switch is transferred to the switching engine. The switching engine 10 Copyright 2004, Ixia 10-Gigabit Ethernet Switch Performance Testing

12 then examines the packet header and decides where to forward the packet. When multiple ingress ports contend for an egress port, congestion becomes the primary issue at the egress switch port and, depending on the architecture, may cause Head-of-Line blocking (HOL). HOL blocking occurs with input queuing, when a cell (a segmented packet to be transmitted across the backplane) has to be held up, thereby blocking the progress of any cells behind it, even if they could otherwise be switched. Some switch architectures also provide multicast packet replication. To get around packet replication issues, most modern switch fabrics now allow for an incoming packet to be broadcast to any number of egress ports. Typically, on the ingress and egress line cards, separate queues and scheduling disciplines are maintained for unicast and multicast traffic. What Are the Challenges in Processing Packets at 10G Rates? This section examines the most common stress points within a switch, and discusses conditions that might push them toward a strained state. Refer to Figure 2. Stress Point 1: Ingress packet buffer The ingress packet buffer is a temporary repository for arriving packets waiting to be processed by the packet processor. Depending on the architecture and efficiency of the packet processor, data in the packet buffer could build up, resulting in intermittent (poor) latency, jitter, packet loss, or even service outage. In most architectures, when packet buffers begin to fill beyond a preset threshold, the packet processor initiates flow control to the upstream MAC device, requesting it to stop passing packets. The MAC device then transmits a special packet requesting remote ports to delay sending packets for a period of time. This special packet is called a pause frame. This helps prevent buffer overflow, but it does not solve the packet loss problem completely. If the packet flow continues and the flow control signal is not removed by the packet processor before the MAC device s buffer fills up, the MAC device will start dropping packets. Generally, the buffer in any part of the system can build up for two reasons: Local or downstream devices are exceeding their allocated processing budget; or There is contention for resources for instance, multiple ingress ports on a switch/router contending for an egress port. Buffer buildup can create a chain reaction, leading to unpredictable behavior in a switch or router. Another challenge in packet buffering for 10GE switches is dealing with back-to-back small packets. For example, at 10 Gbps speeds, arriving 64-byte packets must be deposited into buffer memory every 67 nanoseconds (ns), and departing packets must be retrieved from buffer memory every 67ns. Thus, to process a stream of back-to-back 64-byte packets, the packet buffer memory subsystem must support a write and a read every 67ns. Stress Point 2: Packet classification Packet classification is one of the most susceptible stress points. Classification maps information from the packet header to information stored in local tables maintained by a control plane processor (see "Stress Point 6: Control plane"). The packet processor parses various fields in the packet header to construct search keys. These keys are then used to address various tables. In most high-end architectures, Ternary Content Addressable Memory (TCAM) devices or equivalent technologies are used to map these keys to addresses, and are capable of holding millions of entries and 10-Gigabit Ethernet Switch Performance Testing Copyright 2004, Ixia 11

13 performing key searches in a matter of few internal clock cycles. However, complex applications or routing protocols may require multiple key searches to drive a look-up result. Furthermore, separate classification sequences may be required to determine what to do with a packet. For example, the packet processor may perform an ACL look-up first to decide whether to forward or deny a packet, and then do a route look-up to decide where to forward it. It might also perform a flow control look-up to provide enhanced services. The required degree of packet parsing and processing is one of the main criteria that identifies the class of a switch/router. A simple Layer 2-3 switch only inspects the L2 header (i.e., MAC header, VLAN), and the L3 header (i.e., IPv4, IPv6), and, in some cases, performs limited flow classification. Complex packets might require the classification of multiple L2-L3 headers for a given packet. Packets of a given protocol may be encapsulated within one or more tunnels of varying protocols, as shown in Figure 3. For example, a system supporting IPv6 over GRE requires two Layer 3 headers (IPv4 and IPv6) in addition to the Layer 2 MAC addresses. A more complex example is a Layer 2 VPN Martini Draft packet with frames arriving over Ethernet in a wide range of dispositions, including IPv4 routing, MPLS switching or Ethernet bridging. Flow classification. In addition to complex packet classification, flow classification might be required to provide enhanced services and policies. Flow classification provides a level of granularity that allows policies to be established based on the applications. Any number of combinations of Layer 3 and Layer 4 information could be employed to define the QoS or security policies that are then enforced. A flow is a collection of one or more packet streams. In some classes of switches/ routers, in addition to packet classification, flow classifiers perform stateful analysis of packets within the packet streams. Flow classifiers track the protocol state of each flow as the connection develops. This makes it possible to track control connections on well-known ports that spawn data connections on ephemeral ports. This is important, since many protocols establish connections and negotiate services on well-known Transmission Control Protocol (TCP) ports and then establish another ephemeral port to transfer the data for the network session. IPv6 over GRE packet Dest MAC Src MAC Type IPv4 header GRE IPv6 header TCP/UDP datagram Ethernet over MPLS packet Dest MAC Src MAC Type MPLS label EXP S TTL Dest MAC Src MAC Type IPv4 header TCP/UDP datagram Figure 3. Packets may be encapsulated in tunnels of varying protocols. 12 Copyright 2004, Ixia 10-Gigabit Ethernet Switch Performance Testing

14 Table 1. Maximum packet arrival rates over 10-Gigabit Ethernet. Wire rate LAN throughput for minimum-size packet Ethernet IPv4 Ethernet IPv6 Ethernet Over MPLS IPv4 Ethernet Over MPLS IPv million packets/sec million packets/sec million packets/sec million packets/sec Look-ups and performance. Classification table information is typically stored in a look-up table, usually held in a large TCAM or equivalent technology. When processing encapsulated packets or packets with multiple L2/L3 headers (i.e., IPv6 over IPv4, or MPLS stacks with Ethernet header, IP header, and TCP) the classification process might require multiple accesses to the look-up table for each packet. Table 1 shows the worst-case performance packet arrival rate for small packets. Depending on the packet arrival rate and number of required look-ups per packet, the packet processor or the classification device could become a resource bottleneck. For example, a back-to-back Ethernet packet with an IPv6 datagram requiring ACL and flow look-up might require 7 8 look-ups per packet. With 12 million packets arriving per second, this will require handling ~96 million look-ups/ sec. Stress Point 3: Traffic management The traffic management function provides advanced queuing, congestion management, and hierarchical scheduling of network traffic for large numbers of flows. It forwards traffic according to a user-defined set of rules pertaining to priority levels, latency and bandwidth guarantees, and varying congestion levels. It also provides the packet buffering required to work with any queuing mechanisms used to manage traffic flow across the switch fabric. It may also include the SERDES function. Communication from line card to switch fabric requires additional flow control information, creating overhead and increased bandwidth. This additional bandwidth is called speedup. See "Speedup." on page 14. The architecture of a switch/router can affect how the network behaves in times of heavy demand. An important concern is how packets are queued as they enter the switching fabric, that is, how traffic prioritization is handled and how different traffic flows are merged through the fabric. Some products forward all high-priority packets before any lower-priority packets; this is called strict priority queuing. Other products use mechanisms such as Weighted Fair Queuing (WFQ) to statistically multiplex packets into the fabric. On the ingress line card, WFQ allows packets from lower priority queues to be interleaved with higher priority traffic into the switch fabric. This prevents the higher priority traffic from completely blocking the lower priority traffic, since the queues are guaranteed access to the switch fabric for a predefined proportion of the time. Packet discarding is also part of traffic management. During times of congestion, the traffic manager may need to make discard decisions based on the availability of queue space, priority, or destination port, using a packet discard algorithm like 10-Gigabit Ethernet Switch Performance Testing Copyright 2004, Ixia 13

15 Random Early Detection (RED) or Weighted RED (WRED) for IP traffic. Some routers and switches are fundamentally limited by the small number of queues they have for QoS. Small numbers of queues are common for classbased queuing, which limits prioritization and fairness. Class-based queuing typically prioritizes traffic based on the Layer 2 header (virtual LAN and source or destination MAC address, for example), rather than on higher level information such as application or protocol type. Systems with large numbers of queues facilitate more granular prioritization and greater fairness. Queues established on a per-flow basis provide the possibility for each user session to get its own queue. However, a large number of CoS and QoS policies requires large amounts of memory and hierarchical schedulers capable of handling hundreds of thousands of flows. Stress Point 4: Crossbar switch and backplane interconnect Although most switch architectures for modern systems are non-blocking, three types of blocking can limit performance when multiple ingress ports are contending for an egress port: Head-of- Line (HOL) blocking, input blocking, and output blocking. HOL blocking can waste nearly half a crossbar switch's bandwidth if the cells waiting at each input are stored in a single First-In, First-Out (FIFO) queue. Modern switch architectures employ Virtual Output Queuing (VOQ). VOQ, in conjunction with a scheduling algorithm, eliminates most blocking issues. These scheduling algorithms require the traffic manager device and the switch fabric to exchange information, including requests for permission to transmit, granting of permissions to transmit, and other information. Speed-up. The additional bandwidth required to support VOQ and related scheduling algorithms is called speed-up. A 10GE line card that supports 15 Gbps to the switch fabric offers 50 percent speedup. Speed-up is a common way to reduce input and output blocking by running the crossbar switch faster than the external line rate. For example, if the crossbar switch runs twice as fast as the external line, the traffic manager can transfer two cells from each input port, and two cells to each output port during each cell time. The advantage of speed-up is obvious it offers more predictable delay and jitter across the switch ports by delivering more cells per cell time, and thus reducing the delay of each cell through the switch. In fact, sufficient speed-up can guarantee that every cell is immediately transferred to the output port, where its departure time can be precisely scheduled. Stress Point 5: Multicast replication and queues The biggest challenge in handling multicast packets is packet replication. Packet replication is generally accomplished in two stages. The first stage handles the branch replications from one ingress line card to multiple egress line cards. The second stage handles the leaf replications, and is typically accomplished on the egress line card. Depending on the switch architecture, the packet replication function could cause resource starvation in memory and CPU processing, as well as contention with unicast packets, as it accesses the data plane to forward the replicated packets. New generation switches take advantage of the natural multicast properties of a crossbar switch and perform cell replication within the fabric by closing multiple cross points simultaneously. This method relieves the ingress line card from performing packet replication. However, the second-stage replication on the egress line card can still cause resource starvation and congestion. 14 Copyright 2004, Ixia 10-Gigabit Ethernet Switch Performance Testing

16 Stress Point 6: Control plane The control plane processor handles the routing and switching control plane, as well as many system management operations, such as user configuration, background diagnostics, statistics and alarm collection and reporting, network management, etc. This document focuses only on how the control and data plane interact for the purpose of packet processing. The control plane processor runs the switch/router s operating system and is responsible for the operation of network routing protocols (OSPF, BGP, IS-IS, IGRP), network management (SNMP), console port, diagnostics, etc. In a distributed architecture, where each line card has its own control plane processor, a master control plane processor typically generates, synchronizes, and distributes routing tables and other information among line cards for local forwarding decisions. The control plane path interconnects the management processor(s) with various data plane blocks, to initialize, configure, perform diagnostics, and most importantly, to set up or update routing tables, Layer 2 tables, policies, and QoS/CoS tables. The control processor can read/write to any location in the forwarding table, context memory, and other memories to support route removal and addition, table flushing in route flaps, and policy for a given flow. The look-up and table management operation is asynchronous. The route table may be updated by the control plane processor while the packet processor is performing a look-up. During ordinary system operation and moderately stressed conditions, the control plane would not be called upon to modify more than a few thousand routes per second in response to a routing protocol update, while the system is at the same time forwarding data plane packets. However, during error conditions, or a topology alteration known as route flapping or route convergence, hundreds of thousands of routes may be modified each second while packets are still arriving on each interface at up to line rate. Therefore, determining how fast a switch/ router can update its routing table requires test beds that can create/modify hundreds of thousands of route updates per second while performing normal data plane operation, that is, forwarding packets at line rate. 10-Gigabit Ethernet Switch Performance Testing Copyright 2004, Ixia 15

17 Developing the Right Test Methodology Understanding the stress points in a switch allows the development of a test methodology that can focus on testing these areas. The test methodology should address multiple layers, and could include measuring for: Wire speed unicast data throughput and latency for Layer 2/3 traffic. The ability to filter packets at wirespeed based on MAC addresses, IP addresses, TCP or UDP ports, or a combination of these (N-tuple). The ability to perform prioritization based on QoS marking. The ability to police traffic based on user-defined rate limits. The ability to handle Head-of-Line blocking (HOL). Wire speed multicast performance. Test methodology example Following is an example of a test methodology for a multi-10ge port switch that supports the following features: 1. Wire-rate Layer 2/3 (for IPv4) switching with a minimum packet size of 64 bytes. 2. Switching capacity per slot that supports total port capacity on line card. 3. Filtering based on ACLs. 4. QoS handling based on 802.1p or IP Type of Service (TOS) bits. 5. Priority scheduling based on weighted round robin (WRR). 6. Traffic shaping. 7. Routing protocols, including multicast. The test methodology is broken out into module- and system-level testing. In the real world, local switching on the module occurs; this is the best-case scenario for switch performance, because there is no contention for the fabric. The worst-case scenario is when all traffic entering the switch must traverse the fabric, contending for backplane capacity and causing over-subscription. Module-level testing In this scenario, traffic will stay local to the line card. This means that switching will occur locally between ports on the same line card, and minimal traffic will traverse the backplane. System-level testing In this scenario, all of the ingress traffic is switched to ports on different line cards. This means that traffic will be contending for backplane capacity and will determine how the system handles oversubscription. It will be set up with partially meshed traffic patterns, with unique ingress ports mapped to unique egress ports, and multiple ingress ports mapped to a single egress port. Test methodologies 1. Layer 2 bidirectional throughput and latency test. This test determines the Device Under Test s (DUT s) maximum Layer 2 forwarding rate without traffic loss as well as average latency for different packet sizes. This test is performed full duplex with traffic transmitting in both directions. The DUT must perform packet parsing and Layer 2 address look-ups on the ingress port and then modify the header before forwarding the packet on the egress port. 2. Layer 2 throughput, QoS, and latency test. This test determines the DUT s maximum Layer 2 forwarding rate with packet loss and latency for different packet sizes. The DUT must perform a Layer 2 address lookup, check the 802.1p priority bit value on the ingress port, send it to the designated queue, and then modify the header before forwarding the packet on the egress port. 3. Layer 3 (IPv4) performance test with ACL and latency. This test determines the DUT s maximum IPv4 Layer 3 forwarding rate with packet loss and latency for different packet sizes. The DUT must perform 16 Copyright 2004, Ixia 10-Gigabit Ethernet Switch Performance Testing

18 packet parsing and route look-ups for both Layer 2 and Layer 3 packets on the ingress port and then modify the header before forwarding the packet on the egress port. The ACL test involves blocking or allowing traffic through, based on user-defined classifiers such as IP addresses or Layer 4 port numbers. Ixia routing emulation software is used to populate the DUT s routing table,. For example, OSPF emulation is used to generate OSPF LSAs to construct topological databases. 4. Layer 3 (IPv4) performance test with ACL, QoS, and latency. In addition to test 3 above, QoS values in each header will force the classification of the traffic based on IP Type of Service (TOS) field settings. On the ingress side, this QoS policy could also be used for assigning a packet to a specific queue, packet metering, and policing; on the egress side, it could be used for packet shaping. 5. Layer 3 (IPv6) performance test with ACL and latency. This test methodology is the same as the previous Layer 3 IPv4 with ACL performance test, except that it runs IPv6 traffic with a minimum-size packet of 84 bytes instead of 64 bytes. Due to the larger IPv6 header, the classification and table look-up functions will require more bandwidth and processing. 6. Layer 3 (IPv6) performance test with ACL, QoS, and latency. In addition to test 5 above, QoS values in each header force the classification of the traffic, based on the TOS field setting. On the ingress side, this QoS policy could also be used for assigning a packet to a specific queue, packet metering, and policing; on the egress side, it could be used for packet shaping. 7. Multicast test. This test uses a multicast protocol such as IGMP (IPv4) or MLD (IPv6) to set up multicast sessions between a multicast transmitter and groups of receivers. A multicast protocol emulation can be used to simulate one or more hosts while the DUTs function as IGMP/MLD routers. The simulation calls for groups of simulated hosts to respond to IGMP/MLD router-generated queries and to generate reports automatically at regular intervals. A number of IGMP groups are randomly shared across a group of hosts. 10-Gigabit Ethernet Switch Performance Testing Copyright 2004, Ixia 17

19 Anticipating the behavior of the 10GE switch Each row in the following table represents a test methodology and its expected impact on the different stress points mentioned earlier in this document, based on the switch s design criteria. As an example, in Table 2, row 2 (modulelevel, Full duplex Layer 2 performance and latency, with prioritization): all stress points show low or no stress, except for Stress Point 3, which points to the line card function that services the different priority queues. However, in row 6 (Layer 3 IPv6 performance, ACL and QoS), Stress Points 1 and 2 show that a high level of strain is to be expected. This is because we know that the switch is designed to handle wirespeed Layer 3 IPv4 packets, but not IPv6. This may mean that packet classification for IPv6 addresses may take more clock cycles, which may require more buffering on the ingress. Table 2. Module-level test methodologies and stress points. Test Methodology Stress Points Ingress Packet Buffer Packet Classification Traffic Management X-Bar Switch & Backplane Interconnect Multicast Replication & Queues Control Plane 1. Full duplex Layer 2 performance & latency test 2. Layer 2 QoS throughput & latency test low/no stress moderate stress 3. Layer 3 (IPv4) with ACL performance & latency test moderate stress low/no stress 4. Layer 3 (IPv4) with QoS & ACL performance & latency test 5. Layer 3 (IPv6) with ACL performance & latency test high stress 6. Layer 3 (IPv6) with QoS & ACL performance & latency test 7. Full duplex IGMP multicast test low/no stress moderate stress high stress 18 Copyright 2004, Ixia 10-Gigabit Ethernet Switch Performance Testing

20 With system-level testing, additional strain will occur with the traffic management and switching functions (Stress Points 3 and 4) because of a high level of traffic contending for backplane switching capacity. For example, row 4 (Mesh Layer 3 IPv6, with route flapping), shows additional strain not only in Stress Points 3 and 4, but also in Stress Point 6 (the control processor). Because of route flapping, it needs to modify the routing table as packets continue to arrive on each interface. Table 3. System-level test methodologies and stress points. Test Methodology Stress Points Ingress Packet Buffer Packet Classification Traffic Management X-Bar Switch & Backplane Interconnect Multicast Replication & Queues Control Plane Mesh L3 IPv4 with ACL performance & latency Mesh L3 IPv4 with QoS and ACL performance & latency low/no stress low/no stress moderate stress low/no stress Mesh L3 IPv6 with ACL performance & latency Mesh L3 IPv6 with QoS & ACL performance & route flapping high stress low/no stress moderate stress Mesh IGMP multicast low/no stress high stress 10-Gigabit Ethernet Switch Performance Testing Copyright 2004, Ixia 19

21 Ixia s Test Solution Ixia provides testing solutions that measure the performance and conformance of complex IP networks, devices, and applications. It was the first to offer support for a comprehensive 10GE wire rate test solution one that can generate, capture, characterize, and emulate network and application traffic, establishing definitive performance and conformance metrics for a 10GE network device or system under test. It is also the first to offer support for XAUI, XENPAK, XFP, XPAK, and X2 interfaces in the test industry. For more information on Ixia s 10GE products, refer to: index.php Ixia has developed two primary applications for Ethernet performance testing, IxExplorer and IxScriptMate, each with a distinct testing focus. IxExplorer IxExplorer provides a high level of flexibility and functionality in protocol emulation, traffic generation, and analysis. IxExplorer is the primary controlling application for Ixia s purpose-built hardware test platform, allowing detailed configuration of protocols and analysis of test results. Within IxExplorer, a comprehensive set of IP signaling and routing protocols is supported, including OSPF, IS-IS, RIP, BGP, LDP, and RSVP-TE, as well as multicast protocols. IxExplorer controls the protocol s operation on Ixia s test hardware architecture, which supports a CPU running Linux on each test port. This dedicated emulation environment allows hundreds of 10GE routers to be emulated on each network interface. Up to hundreds of thousands of LSPs can be signaled from each interface. Line rate traffic can be generated over the established connections. IxScriptMate IxScriptMate provides a framework for running automated test scenarios. Numerous test suites have been developed within the IxScriptMate environment for testing the session and circuit scalability, traffic throughput performance, latency, and failover recovery capabilities of switch/routers. IxScriptMate simplifies the configuration process by defining a configuration for the test and displaying the relevant parameters for user input. Tests then run automatically, and the results are presented to the user. This is especially useful where the test configuration can involve multiple protocols and complex traffic generation requirements. 20 Copyright 2004, Ixia 10-Gigabit Ethernet Switch Performance Testing

22 Table 4. Test methodologies and Ixia test solutions. Test Methodology 1. Full duplex Layer 2 performance & Latency 2. Layer 2 QoS throughput and latency test 3. Layer 3 (IPv4) with ACL performance and latency test 4. Layer 3 (IPv4) with QoS and ACL performance and latency test 5. Layer 3 (IPv6) with ACL performance and latency test 6. Layer 3 (IPv6) with QoS and ACL performance and latency test Ixia's Solution IxExplorer, IxScriptMate using RFC 2544 and RFC 2889 tests IxExplorer, IxScriptMate QoS Test IxExplorer, IxScriptMate RFC 2544, RFC 2889, ATSS tests IxExplorer, IxScriptMate QoS Test, BGP, OSPF tests IxExplorer, IxScriptMate RFC 2544, RFC 2889, ATSS tests IxExplorer, IxScriptMate RFC 2544, RFC 2889, ATSS tests 7. Full duplex IGMP multicast IxExplorer, IxScriptMate IP Multicast test Table 4 maps the specific Ixia solution available to test each of the seven test methodologies previously outlined for 10GE switches. 10-Gigabit Ethernet Switch Performance Testing Copyright 2004, Ixia 21

23 Appendix: 10GE Testing Examples The test plans presented in this section include several examples demonstrating how Ixia's solution addresses the challenges of 10GE switch testing. 1. Throughput Testing: RFC 2544 Throughput Test Objective. To characterize the 10GE switch data plane performance in forwarding traffic. IETF RFC 2544 defines a test methodology for this characterization, divided into the following three tests: The back-to-back test determines how the DUT responds to different quantities of frames with the minimum gap allowed by the protocol specification. The frame loss test determines how the DUT responds to streams with different loading. The throughput test finds the highest rate at which the DUT can forward frames. Setup. A minimum of two 10GE test ports is required for this test. Ixia's IxScriptmate application can be used to run RFC 2544 tests. Parameters. Select the test port pairs, frame size mode and frame sizes, and number of iterations. Figure 4. RFC 2544: Throughput Test setup. 22 Copyright 2004, Ixia 10-Gigabit Ethernet Switch Performance Testing

24 Methodology. This test should involve several test ports to match the DUT's port density. Ideally, the test should flood traffic to every input port of the DUT. A number of Ixia load modules will be connected to the DUT. Ixia's IxScriptMate is used to perform the RFC 2544 benchmark test. Results. The results of the test show the throughput rates, in frames per second, obtained for each frame size. The results also show the average throughput rates for all the trials. Figure 5. RFC 2544: Throughput Test results. 10-Gigabit Ethernet Switch Performance Testing Copyright 2004, Ixia 23

25 2. Testing for Latency: RFC 2544 Latency Tests Objective. To determine the latency of the DUT, and how much it varies with different frame sizes. In this test, frames are transmitted for a fixed duration. Once per second, the test tags a frame and transmits it halfway through the duration time. The test compares the tagged frame's timestamp when it was transmitted with the timestamp when it was received. The difference between the two timestamps is the latency. Setup. A minimum of two 10GE test ports will be used for this test, in conjunction with the IxScriptmate RFC2544 test. Latency will be measured in both directions. Parameters. Select the test port pairs, frame size mode and frame sizes, and number of iterations. Methodology 1. Packets will be sent to the DUT from each 10GE load module starting with the selected minimum frame size for an interval that lasts for selected test duration. 2. The average latency is then measured. 3. These two steps will continuously repeat, using selected frame sizes every interval until it reaches the maximum frame size selected. Results. The results of the test show the latency in nanoseconds (ns) obtained for each frame size and test port pair. The results also show the average latencies across the pairs. Figure 6. RFC 2544: Latency Test results. 24 Copyright 2004, Ixia 10-Gigabit Ethernet Switch Performance Testing

26 3. Testing how 10GE ports handle prioritized streams based on QoS parameters Objective. To determine the maximum rate at which the DUT can forward frames correctly according to their priority settings. The test sets up a configuration in which many ports, each with a different priority, sends traffic to one single port. The receive port may be overloaded in order to test the receipt of the highest priority frames. This test supports both MAC and IP layer frames; priority bits may be specified either in the IP precedence bits or in the 802.1p header. Latency may optionally be calculated per priority level. Test results are: the transmit and receive rates per priority, percent loss, and optionally, latency, for each priority per port. Setup. A minimum of two test ports will be used for this test, in conjunction with the IxScriptmate QoS Many-to-One test. In this example, one Ixia 10GE test port and two Ixia 1 GE test ports connected to the DUT all traffic directed to a single 10GE port on the DUT. Parameters. Select the three test port pairs, frame sizes, QoS parameters for each port, and number of iterations. 10 GE 1 GE 1 GE 10 GE Figure 7. Testing how 10GE ports handle prioritized streams based on QoS parameters setup. 10-Gigabit Ethernet Switch Performance Testing Copyright 2004, Ixia 25

27 Methodology 1. Packets will be sent from all ports. Multiple streams with varying QoS parameters will be sent to the one egress port. 2. Total frames received, latency, and frame forwarding rate are measured per priority. 3. Steps 1 and 2 continuously repeat, using selected frame sizes every interval until the test reaches the maximum frame size selected. Results. The results of the test show, on a per priority stream, the total frames received, the receive rate, and percentage loss for each frame size. Figure 8. Testing how 10GE ports handle prioritized streams based on QoS parameters results. 26 Copyright 2004, Ixia 10-Gigabit Ethernet Switch Performance Testing

28 4. 10GE OSPF Convergence Test Objective. Verify the ability of a router to switch between preferred and lesspreferred routes on its 10GE ports when the preferred routes are withdrawn and readvertised. This will test the control plane function on the DUT s 10GE line card. In this test, two 10GE ports simulate OSPF routers, Router 1 and Router 2. Both routers advertise the same route prefixes to the simulated network. However, the routes advertised by Router 1 will have smaller metrics (lower costs), which should cause the DUT to forward traffic over them instead of Router 2 s routes. After advertising the routes, the transmit port begins transmitting a stream of packets to an address in each of the advertised route prefixes. The DUT should forward all the packets over the route with the lower metric (Router 1). After a time interval has elapsed, the receive port simulating Router 1 withdraws its routes. The DUT should detect that the preferred route has gone down and switch traffic to Router 2 s routes. Router 1 then re-advertises its routes. The DUT should again detect a change re-route traffic to the receive port simulating Router 1. Setup. This test uses three test ports one to transmit and two to receive (see Figure 9 below). One Ixia test port (acting as the transmit port) and two Ixia 10GE (test ports 1 and 2 acting as receive ports) connect to the DUT. Receive ports will emulate OSPF routers. The traffic is unidirectional. The DUT must have three ports utilized with two enabled for OSPF. All three ports should be configured for IP and have unique subnets in which to communicate with the tester ports. Figure 9. 10GE OSPF Convergence Test setup. 10-Gigabit Ethernet Switch Performance Testing Copyright 2004, Ixia 27

29 Results. The test results provide an average convergence time for all routes. Figure 10 below displays example results for the automated OSPF convergence test in IxScriptMate. In addition to convergence time, this test also indicates the amount of lost packets caused by the convergence. Figure GE OSPF Convergence Test results. 28 Copyright 2004, Ixia 10-Gigabit Ethernet Switch Performance Testing