Cisco IOS Server Load Balancing and the Catalyst 6000 Family of Switches

WHITE PAPER Cisco IOS Server Load Balancing and the Catalyst 6000 Family of Switches The Catalyst 6000 family, which includes the Catalyst 6500 and the Catalyst 6000 series, delivers high-performance, multilayer switching solutions for server aggregation, high-end wiring closet, and backbone networks. The Catalyst 6000 family was designed to address the increased requirements for gigabit density, scalability, high availability, and multilayer switching in backbone and server-aggregation environments. Together, the Catalyst family delivers a wide range of intelligent solutions, enabling network managers to build networks that scale for mission-critical applications. Among the numerous requirements included in the ability to building scalable networks is the need to support server load balancing (SLB). SLB is important not only for scaling of Web services, but also for servers providing traditional IP services such as Domain Naming System (DNS) resolution and File Transfer Protocol (FTP) services. This white paper concentrates on the particular aspect of scalable network design using SLB and illustrates how the Catalyst 6000, with its embedded Cisco IOS SLB capabilities, provides network managers with an integrated solution to balance services across numerous servers at speeds of millions of packets per second (Mpps). The first platform in which IOS SLB will be deployed is the Catalyst 6000, specifically the Cisco IOS on the Catalyst 6000 Family of Switches supervisor software option. IOS SLB is a software solution which can provide complete SLB functionality, utilizing the advanced application-specific integrated circuits (ASICs) on the Catalyst 6000 to perform SLB at up to 15 Mpps. To explain how IOS SLB works in the Catalyst 6000 family, this paper provides an overview of the Catalyst 6000 family and its architecture, and then discusses traditional DNS methods of load-balancing and its shortcomings. It describes the Cisco approach to solving the load-balancing problem utilizing Cisco IOS and how, when deployed in the Catalyst 6000, this solution allows Cisco to provide a solution of SLB scaling to millions of packets per second. Switch Overview The Catalyst 6000 family consists of the Catalyst 6000 and the Catalyst 6500 series. Both series support six- and nine-slot versions, providing a wide range of configuration and price/ performance options. Both Catalyst 6000 and 6500 series switches also support a wide range of interface types and densities to include support for up to 384 10/100 Ethernet, 192 100BaseFX Fast Ethernet, and up to 130 Gigabit Ethernet ports. Network managers can also aggregate up to eight physical Fast Ethernet or Gigabit Ethernet links using either Fast EtherChannel or Gigabit EtherChannel technology, respectively, for logical connections up to 16 Gbps. The Catalyst 6000 family delivers an industry-leading Gigabit Ethernet backbone solution to meet the requirements of today s fast-growing backbones. (See Figure 1 and Table 1.) The architecture of the Catalyst 6500 series supports scalable switching bandwidth up to 256 Gbps and scalable multilayer switching up to 150 Mpps. For network managers not requiring the performance of the Catalyst 6500 series, the Catalyst 6000 Page 1 of 8

series provides a more cost-effective solution, with backplane bandwidth of 32 Gbps and multilayer switching scaling up to 15 Mpps. For investment protection, both switches support the same suite of Supervisor Engines, interface line cards, and power supplies, providing a wide range of price/performance options. Figure 1 Catalyst 6000 Family Table 1 Table 1: Catalyst Family Switches Catalyst 6000 Series Catalyst 6500 Series Slot Density 6 or 9 slots 6 or 9 slots Backplane Capacity 32 Gbps Scalable to 256 Gbps Multilayer Switching Scalable to 15 Mpps Scalable to 150 Mpps Redundant Supervisors Yes Yes Redundant Switch Fabrics No Yes Redundant Common Equipment Yes Yes Switch Architecture Both Catalyst 6000 and 6500 series switches employ a frame-based switching architecture using an input/output queuing model. Each slot is capable of supporting multiple transmit queues per port, all of which can support configurable thresholds and depths for maximum flexibility and support of mission-critical, unicast, and multicast/multimedia applications. Weighted random early detection (WRED) and weighted round robin (WRR) are also applied within each queuing complex to maximize utilization of available resources and minimize packet loss and delay, particularly for premium-class traffic for which users may be charged a premium. To ensure high system availability, the Catalyst 6000 family supports device-level fault tolerance, which includes redundant supervisor engines, power supplies, uplinks, and switch fabrics (Catalyst 6500 series only). All system elements including power supplies, fans, supervisors, line-card modules, and switch fabrics (Catalyst 6500 series only) are also hot-swappable such that elements can be added, removed, or replaced without service interruption of unrelated traffic flows. In dual supervisor configurations, Cisco Fast Switchover transfers switch control to the redundant supervisor within seconds for mission-critical applications that require maximum network availability. All system elements are field-replaceable units, maximizing serviceability and minimizing network downtime. Page 2 of 8

Load Balancing Load balancing has been traditionally assigned to a DNS server using a round robin algorithm to balance traffic. Without round robin, DNS maps host names to one IP address. With round robin, DNS randomly rotates through a list of several IP addresses, any one of which could be mapped to a DNS query for a Uniform Resource Locator (URL). As more server capacity is required, administrators can add IP addresses to the rotation while adding additional servers to the server farm. (See Figure 2.) The DNS round-robin method for load balancing has several limitations in a server-farm system made up of multiple servers: There is a level of system bias resulting from the rotation, which creates unequal and highly variable load distribution among individual servers. The result is that traffic is not being sent to the server that could most efficiently handle the load. The DNS round-robin method of scaling servers assumes mirrored data on each server. DNS round robin also presents an availability problem because this method has no knowledge of the status of the transmission control protocol (TCP) connection, server, or application. If a server crashes or is removed, DNS round robin continues to send client requests to that server and clients receive a server not available message. Cached information also presents a problem for DNS round-robin access to multiple servers. In an attempt to minimize redundancy and maximize efficiency, DNS uses a caching mechanism that allows it to remember data it has already retrieved. When a particular IP mapping has been removed from the rotation, several caches can continue to hold the information. Figure 2 Simple Load Balancing Using DNS DNS Server Internet 1 2 3 Web Servers Because of these basic limitations of DNS, newer techniques have been implemented to not only solve the load-balancing issues associated with round-robin DNS but also to provide more scalable and higher-availability solutions while providing mechanisms for server connection management and security. These solutions are commonly referred to as SLB products, and an example would be Cisco IOS SLB. By porting the software features of the Cisco Local Director server load balancing appliance into Cisco IOS, network managers now have the ability to deploy Cisco IOS and SLB on one hardware platform. The first platform in which IOS SLB will be deployed is the Catalyst 6000, specifically the Cisco IOS on the Catalyst 6000 Family of switches supervisor software option. Page 3 of 8

Server Load Balancing with IOS SLB IOS SLB intelligently load balances TCP/IP traffic across multiple servers. (See Figure 3.) It appears as one virtual server to the requesting clients. All traffic is directed toward a virtual IP address (virtual server) via DNS. Those requests are distributed over a series of real IP addresses on servers (real servers). The definition of a virtual IP address is an address that is in DNS and most likely has a domain name. A real IP address is physically located on a real server behind IOS SLB. IOS SLB provides the following benefits: High performance is achieved by distributing client requests across a cluster of servers. Administration of server applications is easier. Clients know only about virtual servers; no administration is required for real server changes. Security of the real server is provided because its address is never announced to the external network. Users are familiar only with the virtual IP address. Additionally, filtering of unwanted traffic can be based on both IP address and IP port numbers. Ease of maintenance with no downtime is achieved by allowing physical (real) servers to be transparently placed in or out of service. Figure 3 IOS SLB Presents a Virtual Address and Load Balances the Traffic Across Multiple Servers Web Servers 1 Internet IOS SLB 2 3 Server Load Balancing The Details This section discusses how server load balancing works, with the following sections outlining how IOS SLB works in conjunction with the Catalyst 6000 family of switches. IOS SLB allows users to represent a group of network servers (a server farm) as a single server instance, balance the traffic to the servers, and limit traffic to individual servers. The single server instance that represents a server farm is referred to as a virtual server. The servers that comprise the server farm are referred to as real servers. (See Figure 4.) Figure 4 Mapping Virtual to Real Addresses Virtual Address 192.168.1.200 Real Addresses 192.168.1.1-80 192.168.1.1-20 192.168.1.2-80 Page 4 of 8

In this environment, clients are configured to connect to the IP address of the virtual server. When a client initiates a connection to the virtual server, the SLB function chooses a real server for the connection based on a configured load-balancing algorithm. Representing server farms as virtual servers facilitates scalability and availability for the user. The addition of new servers and the removal/failure of existing servers can occur at any time without affecting the availability of the virtual server. IOS SLB can operate in one of two redirection modes: directed mode or dispatched mode. In directed mode, the virtual server can be assigned an IP address that is not known to any of the real servers. IOS SLB translates packets exchanged between a client and real server, translating the virtual server IP address to a real server address via network address translation (NAT). In dispatched mode, the virtual server address is known to the real servers and IOS SLB redirects packets to the real servers at the media access control (MAC) layer. Phase I of IOS SLB implements dispatch mode only for packet redirection. In this mode, the real servers must be Layer 2 adjacent to the device redirecting packets (the Catalyst 6000), or in other words, not beyond an additional router. However, as the Catalyst 6000 is able to multilayer switch many IP subnets directly, the servers need not be all on the same subnet. Directed mode will be supported in a following release. Server Algorithm There are two algorithm choices that may be used for load balancing or determining which server will receive a new connection request. (Weighted) Least connections The number of sessions assigned to a server is based on the number of current TCP connections. (Weighted) Round robin The round-robin predictor option directs the network connection to the next server and treats all servers as equals if unweighted, regardless of the number of connections or the response time. Connection Initiation and Decision Making It is important to understand how a hypertext transfer protocol (HTTP) connection or, for that matter, any TCP connection is established to understand how IOS SLB works on the Catalyst 6000 family of switches. A short discussion follows. (See Figure 5.) Figure 5 TCP/IP Handshake SYN client: this is my sequence number SYN/ACK server: I ACK your sequence number and this is mine ACK DATA client: I ACK your sequence number let s chat ACK I ACK your last message RST our conversation is over RST FIN/ACK OR our conversation is over FIN/ACK Page 5 of 8

It is important to understand how a HTTP connection or, for that matter, any TCP connection is established to understand LocalDirector and how Accelerated SLB works on the Catalyst 6000 Family of switches. A short discussion follows (see figure 5). For readers familiar with TCP this section can be skipped. For a connection to be established or initialized, the two TCP hosts synchronize on each other s initial sequence numbers. This process is done in an exchange of connection-establishing frames carrying a control bit called SYN (for synchronize) and the initial sequence numbers. The synchronization requires each side to send its own initial sequence number and to receive a confirmation in acknowledgment from the other side. Each side must also receive the other side s initial sequence number and send a confirming acknowledgment. Because Step 2 can be combined in a single message, this scenario is referred to as a three-way (or three-message) handshake. Now the two hosts are ready to communicate and the World Wide Web (WWW) pages can be transferred. After the last ACK, the connection is terminated by either a reset (RST) or a finish flag (FIN/ACK) being sent. IOS SLB Operation IOS SLB uses this handshake mechanism to identify the beginning and end of individual flows and, therefore, can make the appropriate decisions as the conversation progresses. The following discussion describes what happens in this sequence. The SYN is set in the first packet sent by the client to the IP address of the virtual server for which services are requested; in reality, this is the address of the IOS SLB in behalf of all the real servers, so the packet is sent to IOS SLB running on the multilayer switch feature card (MSFC) by the WAN router. IOS SLB maintains tables of all connections that are active and since this is a new connection (identified by source and destination IP address as well as port numbers), the software creates a connection entry for this flow. The next step is to decide which real server will service this connection request, based on configuration of the particular load-balancing algorithm. After this decision has been made, IOS SLB then forwards the packet to the appropriate server by changing the destination MAC address to that of the real server and sending the packet out to the server.the real server receives the packet, fetches the requested Web page, and sends the response back to IOS SLB, which then sends the packet back towards the original source workstation. If multiple pages are requested by this user s session, then this process continues until the user asks for no more data and requests the TCP connection to be closed, as indicated by the RST/FIN flags being sent in the last packet. At this point, the IOS SLB software resident on the Catalyst 6000 can flush its internal table of connection status for this particular session, and then calculate the server to receive the next new connection based on the deployed load-balancing algorithm. IOS SLB operating on the Catalyst 6000 moves the data forwarding into wire-speed application-specific integrated circuits (ASICs). IOS SLB How does IOS SLB work, both in the processor as well as with the Catalyst 6000 family hardware? As described earlier, the software component of making the decision of how to map a virtual server to a real server is still required, and to satisfy this requirement the Catalyst 6000 route processor is used. IOS SLB and Catalyst 6000 Hardware Operation IOS SLB takes advantage of both IOS SLB software services running on the Catalyst 6000 route processor and the wire-speed switching services of the Catalyst 6000. This scheme uses the handshake mechanism to identify the beginning and the end of individual flows and, therefore, can make the appropriate decisions as the conversation progresses. The following discussion describes the operation of IOS SLB software and Catalyst hardware. The SYN is set in the first packet sent by the client to the IP address of the virtual server for which services are requested; in reality, this is the address of the IOS SLB, in behalf of all the real servers, so the next hop router sends the packet to IOS SLB running on the Catalyst 6000. The Catalyst 6000 hardware sees this packet as a potential session for acceleration and caches the flow information. IOS SLB continues to maintain tables of all connections that are active and, since this is a new connection (identified by source and destination IP address as well as Layer 4 port numbers), it creates a connection entry for this flow. The next step is to decide which real server will service this based on the configuration of the particular load-balancing algorithm. Once this decision has been made, IOS SLB then forwards the packet to the appropriate server by changing the destination MAC address (phase I) to that of the real server and sending the packet out. At this point the Catalyst 6000 hardware ASICs see this packet as a valid SLB flow because it has been received and transmitted by the Page 6 of 8

IOS SLB, with a MAC address manipulation. All subsequent packets on the inbound direction will now be switched by the Catalyst 6000 without the intervention of the route processor, with the exception of any packets that have either the RST or FIN bits set. The real server receives the packet, fetches the requested Web page, and sends the response back to the Catalyst 6000 route processor, which then sends the packet to the next hop router. The Catalyst 6000 hardware sees this packet as a potential session in the reverse path for acceleration and caches the flow information. When IOS SLB sends the response packet, this action triggers the Catalyst 6000 hardware to see this flow as valid for IOS SLB, and all subsequent packets will be switched by the hardware without the intervention of the Catalyst 6000 route processor, with the exception of any packets that have the RST/ FIN bits set. In summary, two flows are established for each connection, one inbound and one outbound. If multiple pages are requested by this user s session, then this process continues until the user requires no further data and requests the TCP connection be closed, as indicated by the RST/ FIN flags being set in the last packet. As noted earlier, these bits indicate the end of a connection and these packets are sent to the Catalyst 6000 route processor directly. This scenario allows the IOS SLB to purge its connection cache for this particular flow and reevaluate its SLB algorithm for this particular server and subsequent flows. At this time the Catalyst 6000 also flushes its connection cache for both flows of this connection. have now provided switchovers in the subsecond range. In a later release IOS SLB functionality can be also backed up via HSRP so that if the primary Catalyst 6000 supervisor fails, a redundant supervisor will be able to take over its task within subseconds. Service Utilities Roadmap IOS SLB builds upon Accelerated Server Load Balancing (ASLB) where the functionality of the external LocalDirector has now been moved into the IOS code running on the Catalyst 6000. It leverages the hardware capabilities of the Catalyst 6000 to provide packet handling of millions of packets per second after making the intelligent load balancing decision via Cisco IOS SLB. Phase I of IOS SLB works in dispatch mode but later releases will include directed mode, as well as enhancements for redundancy and server health checks. Conclusion The Catalyst 6000 with its IOS SLB capabilities utilizing server cache technology provides network managers with the ability to balance services across multiple servers at speeds of millions of packets per second. As articulated in the service utilities roadmap, Cisco will continue to provide new features in IOS SLB as well as enhanced services at wire speed on the Catalyst platforms. Redundancy Ensures Uptime An important aspect of any network is that the design provides the redundancy that is required to ensure the uptime required by data centers. IOS SLB benefits from hot standby router protocol (HSRP) by allowing redundant routers to source and sink packets to and from the IOS SLB. In the unlikely event that a router fails, a standby router takes over the tasks of the failed router, with no effect on users session traffic. Routers communicate via hello packets to indicate their health and in the absence of three hellos from the primary router, the secondary router takes over communications of all sessions. Recent improvements in HSRP Page 7 of 8

Corporate Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA http://www.cisco.com Tel: 408 526-4000 800 553-NETS (6387) Fax: 408 526-4100 European Headquarters Cisco Systems Europe 11, Rue Camille Desmoulins 92782 Issy Les Moulineaux Cedex 9 France http://www-europe.cisco.com Tel: 33 1 58 04 60 00 Fax: 33 1 58 04 61 00 Americas Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA http://www.cisco.com Tel: 408 526-7660 Fax: 408 527-0883 Asia Headquarters Nihon Cisco Systems K.K. Fuji Building, 9th Floor 3-2-3 Marunouchi Chiyoda-ku, Tokyo 100 Japan http://www.cisco.com Tel: 81 3 5219 6250 Fax: 81 3 5219 6001 Cisco Systems has more than 200 offices in the following countries. Addresses, phone numbers, and fax numbers are listed on the Cisco Connection Online Web site at http://www.cisco.com/go/off ices. Argentina Australia Austria Belgium Brazil Canada Chile China Colombia Costa Rica Croatia Czech Republic Denmark Dubai, UAE Finland France Germany Greece Hong Kong Hungary India Indonesia Ireland Israel Italy Japan Korea Luxembourg Malaysia Mexico The Netherlands New Zealand Norway Peru Philippines Poland Portugal Puerto Rico Romania Russia Saudi Arabia Singapore Slovakia Slovenia South Africa Spain Sweden Switzerland Taiwan Thailand Turkey Ukraine United Kingdom United States Venezuela Copyright 2000, Cisco Systems, Inc. All rights reserved. Printed in the USA. Catalyst, Cisco, Cisco IOS, Cisco Systems, the Cisco Systems logo, EtherChannel, and IOS are registered trademarks of Cisco Systems, Inc. or its affiliates in the U.S. and certain other countries. All other trademarks mentioned in this document are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any of its resellers. (9912R) 12/99 BW5761