CSS HARDWARE LOAD BALANCING POLICY Version 2.5 Date: 04/11/2014 SECURITY WARNING The information contained herein is proprietary to the Commonwealth of Pennsylvania and must not be disclosed to un-authorized personnel. The recipient of this document, by its retention and use, agrees to protect the information contained herein. Readers are advised that this document may be subject to the terms of a non-disclosure agreement. DO NOT DISCLOSE ANY OF THIS INFORMATION WITHOUT OBTAINING PERMISSION FROM THE MANAGEMENT RESPONSIBLE FOR THIS DOCUMENT. EDC CSS HARDWARE LOAD BALANCING POLICY PAGE 1 OF 19
Version History Date Version Modified By / Approved By Section(s) Comment 03/14/2003 1.0 Fred Ost All Initial Draft 04/11/2005 2.0 Scott Winters All Modification 10/23/2007 2.1 Connie Reber All Updated to new template design. 07/30/2008 2.2 Connie Reber All Minor template and formatting updates. Changed title to reflect that document applies to Managed Services, MSL, and Co-Location. 11/06/2008 2.3 C. Reber Cover page Insert new OA logo onto cover page. 01/21/2010 2.0 S. Winters / C. Reber All Remove background architecture and update current. Insert web arch. Drawing. Remove co-location from cover page. Various verbiage updates 11/16/2011 2.4 C. Reber All Change ESF to EDC. 04/11/2014 2.5 C. Reber All Change Remedy references to general term incident Update cover page to OA standard EDC CSS HARDWARE LOAD BALANCING POLICY PAGE 2 OF 19
Table of Contents 1 CSS HARDWARE LOAD BALANCING INTRODUCTION... 4 1.1 OVERVIEW... 4 2 CSS HARDWARE LOAD BALANCING - INTERNET WEB SERVERS... 5 2.1 BACKGROUND... 5 2.2 FUNCTIONAL OVERVIEW... 5 2.3 SCHEMATIC DIAGRAM... 5 2.4 DEPLOYMENT OPTIONS... 6 2.4.1 Load Balancing... 6 2.4.2 SSL Acceleration... 7 2.4.3 Single Node Support... 8 2.5 EXTERNAL SECURITY ZONE DEPLOYMENT POLICY RECOMMENDATIONS... 9 2.6 AVAILABILITY... 9 3 CSS HARDWARE LOAD BALANCING - INTRANET WEB SERVERS... 10 3.1 BACKGROUND... 10 3.2 FUNCTIONAL OVERVIEW... 11 3.3 EDC INTERNAL SECURITY ZONE DEPLOYMENT POLICY RECOMMENDATIONS... 11 3.4 AVAILABILITY... 11 4 HOSTING SOLUTION ENGINE (HSE)... 12 4.1 BACKGROUND... 12 4.2 FUNCTIONAL OVERVIEW... 12 4.3 EDC HSE POLICY RECOMMENDATIONS... 14 4.4 AVAILABILITY... 14 5 GLOBAL LOAD BALANCING WITH GLOBAL SITE SELECTOR (GSS)... 15 5.1 BACKGROUND... 15 5.2 FUNCTIONAL OVERVIEW... 15 5.3 EDC GSS POLICY RECOMMENDATIONS... 16 5.4 AVAILABILITY... 16 6 SORRY SERVER SERVICES... 17 6.1 BACKGROUND... 17 6.2 FUNCTIONAL OVERVIEW... 17 6.3 EDC SORRY SERVER POLICY RECOMMENDATIONS... 17 6.4 AVAILABILITY... 17 7 CUSTOM NOTIFICATIONS AND KEEPALIVES... 18 7.1 BACKGROUND... 18 7.2 FUNCTIONAL OVERVIEW... 18 7.3 EDC CUSTOM NOTIFICATIONS POLICY RECOMMENDATIONS... 19 7.4 AVAILABILITY... 19 EDC CSS HARDWARE LOAD BALANCING POLICY PAGE 3 OF 19
1 CSS Hardware Load Balancing Introduction 1.1 OVERVIEW The Enterprise Data Center has implemented hardware load balancing support for Internet and Intranet web servers via Cisco Content Services Switches (CSS). The following information is described within this document: Chapter 1: Introduction Chapter 2: Internet Side Offerings including SSL Offload Section 3: Intranet Based Offerings Section 4: HSE Management Details Section 5: GSS Global Load Balancing Section 6: Sorry Server Services Section 7: Custom Notifications and Keepalives The primary audiences for this document are the EDC and commonwealth agencies who use these services. EDC CSS HARDWARE LOAD BALANCING POLICY PAGE 4 OF 19
2 CSS Hardware Load Balancing - Internet Web Servers 2.1 BACKGROUND The Enterprise Data Center has implemented hardware load balancing support for Internet and Intranet web servers via Cisco Content Services Switches (CSS), designated CSS-1 and CSS-2 in the diagrams in this section. 2.2 FUNCTIONAL OVERVIEW This design facilitates the use of external load balancing The CSS units are hardware load balancers, meaning they provide high availability services for web farm destinations for which they have been configured. In addition the CSS also provide Denial of Service attack protection. The Cisco CSS 11506 is configured or learns where specific content resides, either locally or remotely, and dynamically selects the best Web server or cache for specific content requests. Local server selection can be based on server load and application response time, as well as traditional least connection and round-robin algorithms. Any application that uses standard TCP or User Datagram Protocol (UDP) protocols can be load balanced. In addition, the CSS11506 supports SSL acceleration which can significantly reduce CPU loads on servers requiring secure connections. 2.3 SCHEMATIC DIAGRAM The framework against which the Internet External Security Zone front end load balancing will occur as depicted in Figure 1: Enterprise Data Center Agency firewall Commonwealth Agency MAN Busines Partne s Network r s S i Core Routers S i Internet ENCTCIZFW001 Web 1a Web 1b Si Si ENCTCIZFW002 Data Center (External) Security Zone CSS- 1 CSS-2 ASFPI01 ASFPI02 BLL BLL Server Node 1 ASFPI03 Server Node 2 Figure 1 Co-location Revised DMZ Network Framework EDC CSS HARDWARE LOAD BALANCING POLICY PAGE 5 OF 19
2.4 DEPLOYMENT OPTIONS For CTC based servers, EDC offers 3 different CSS service offerings: Load Balancing SSL Acceleration Support for a Single Node 2.4.1 Load Balancing The CSS is a hardware load balancer, capable of three main types of load balancing: HTTP load balancing, SSL load balancing, and load balancing of other services. CTC CSS Deployment - Load Balancing (option 1) CSS Dataflow: 1) Agency web server public IP address terminates on CSS via DNS client resolver. 2) CSS NAT's to Intranet routable RFC1918 addressing 3) Load balancing to destination physical IP's based on load, delay, other metrics 4) HTTP or HTTPS(server performs SSL) 5) Any TCP or UDP services can be load balanced CoPA MAN DMZ Internet 164.156.71.0/24 End User's Computer CSS 1 CSS 2 172.17.24x.y/24 (preferred) Intranet Switch Business Logic Layer EDC Figure 2 - Load Balancing Data Flow EDC CSS HARDWARE LOAD BALANCING POLICY PAGE 6 OF 19
HTTP Load Balancing 1. The traffic flow through the CSS to load balanced web servers (i.e., more than 1 server) is as follows: 2. Client requests connection to web server via DNS 3. DNS resolves and informs client. Client network connection to server destination attempt begins. 4. CSS receives request for destination IP address. The destination address is a Virtual IP Address (VIP). Based on pre configured load balancing metrics, CSS directs the connection request to the real IP address of a server in the group that is load balanced. SSL Load Balancing In the event that the destination IP addresses using HTTPS service needs to be end to end encrypted AND needs to be load balanced, the data flow is the same; however the packets are decrypted, load balancing decision made, and re-encrypted on the CSS in order to present end to end encrypted traffic to servers requiring it. In this event, since the public certificate has been previously exported to the CSS, new private certificates between the CSS and servers will be generated and used to re-encrypt traffic between the CSS and server(s). Load Balancing Other Services Though the CSS box is designed for HTTP or HTTPS traffic, any TCP or UDP services can be load balanced through it. It load balances the traffic the same way as demonstrated in the HTTP Load Balancing portion of this section. 2.4.2 SSL Acceleration SSL acceleration offers the ability to offload SSL processing from secure servers, reducing overhead and improving performance. The SSL accelerator card is installed in the CSS chassis and becomes the Certificate server for clients requesting secure connectivity to a given URL. In this configuration, the SSL server certificates are securely exported from the server(s) using symmetrical keys and kept on the CSS/SSL accelerator card*. Secure traffic requests from clients are routed to the CSS, after which the client performs key authentication with the CSS. After authentication, the traffic is load balanced (see option 1 above), and handed to the server. This can be to any port that is configured on the server to listen for web (typically port 80) or other services. Note that after decryption and authentication, traffic is converted to clear text between the CSS and server. This allows the servers to provide optimal application performance while maintaining encryption between the client and CSS. The diagrams below provide an illustration of how traffic is passed between secure and non-secure states, and how Denial of Service protection is provided: * CSS/SSL device, cert is stored in a part of memory, similar to nvram. EDC CSS HARDWARE LOAD BALANCING POLICY PAGE 7 OF 19
CTC CSS Deployment - SSL Acceleration (option 2) CSS SSL Acceleration Dataflow: 1) Agency web server public IP address terminates on CSS via DNS client resolver. 2) CSS NAT's to Intranet routable RFC1918 addressing **can optionally balance to public IP address. Less secure. 3) Client authenticates server SSL key via key negotiation***. 4) Traffic passed to server on designated port in clear text., thereby 'offloading' SSL processing from server. 5) Load balancing to destination physical IP's based on load, delay, other metrics. CoPA MAN DMZ Internet 164.156.71.0/24 ***Note: Assumes that secure server SSL Certificate exported to CSS and is available for client authentication End User's Computer CSS 1 CSS 2 Web Switches 172.17.24x.y/24 (preferred) Intranet Switch Business Logic Layer EDC Figure 2 - SSL Acceleration overview of the SSL acceleration process at EDC It should be noted that Agency security requirements may preclude use of SSL acceleration due to clear text data from CSS to server. In this event, SSL load balancing (option 1) should be used. 2.4.3 Single Node Support Just because the CSS is designed for load balancing doesn t mean that it can t accommodate or offer benefits for single node applications. By adding a single node application to the CSS the following benefits can be taken advantage of: Extra Security (only services specified in the CSS are even passed on to the actual server, much like a proxy). Custom Notifications (see the section of the same name) Sorry Server (see the section of the same name) EDC CSS HARDWARE LOAD BALANCING POLICY PAGE 8 OF 19
2.5 EXTERNAL SECURITY ZONE DEPLOYMENT POLICY RECOMMENDATIONS The hardware load balancing described in option 1 is to be deployed where server based load balancing facilities are not used. The SSL acceleration described in option 2 is to be deployed only where end to end encryption is not required. Traffic which requires end to end encryption can be load balanced via Option 1 above. 2.6 AVAILABILITY The DMZ load balancing option, SSL offloading and single node support, is all available for EDC customers at this time. EDC CSS HARDWARE LOAD BALANCING POLICY PAGE 9 OF 19
3 CSS Hardware Load Balancing - Intranet Web Servers 3.1 BACKGROUND Enterprise Data Center personnel are in the process of redesigning network services in the Enterprise Data Center (EDC) Internal Security Zone. This design will provide for hardware load balancing of Intranet web servers via Cisco Content Services Switches (CSS), designated CSS-3 and CSS-4 in the diagram below. Enterprise Data Center - CTC Agency firewall MAN Commonwealth Agency CTS1A CTS1B Business Partner Networks S i Core Routers S i ENCTCIZFW001 CSS-3 CSS-4 Database Outside ENCTCINT001 ENTCTINT002 Sync Secure Network ENCTCIZFW002 Database Inside ENCTCDBFW001 ENCTCDBFW002 EDC Mgmt Server ASF0 2 ASF0 1 Database Servers/ DC's EDC Database (Internal) Security Zone Figure 3 - EDC Intranet Load Balancing EDC CSS HARDWARE LOAD BALANCING POLICY PAGE 10 OF 19
3.2 FUNCTIONAL OVERVIEW This design facilitates the use of hardware load balancing The CSS11503 s (smaller versions of the CSS11506 s deployed in the web farm) will be deployed as hardware load balancers, meaning they provide high availability services for web farm destinations to which they are configured to actively track. In addition the CSS also provide Denial of Service attack protection. The Cisco CSS 11503 s is configured or learns where specific content resides, either locally or remotely, and dynamically selects the best Web server or cache for specific content requests. Local server selection is based on server load and application response time, as well as traditional least connection and round-robin algorithms. Any application that uses standard TCP or User Datagram Protocol (UDP) protocols can also be load balanced. All Intranet load balancing features described in Section 1 EXCEPT SSL acceleration will be offered. Please refer to Section 1a above for description of HTTP and HTTPS load balancing. Web servers that are in front of or behind the Database Firewalls (see Figure 7) can be load balanced. Web servers will have firewall functionality provided by either the PIX Firewall service modules or Checkpoint Database Firewalls. 3.3 EDC INTERNAL SECURITY ZONE DEPLOYMENT POLICY RECOMMENDATIONS The hardware load balancing described in Section 2 is to be deployed where server based load balancing facilities are not used. The SSL acceleration described in Section 2b will not be offered. 3.4 AVAILABILITY The DMZ load balancing option for EDC customers is available at this time. EDC CSS HARDWARE LOAD BALANCING POLICY PAGE 11 OF 19
4 Hosting Solution Engine (HSE) 4.1 BACKGROUND The Hosting Solution Engine (HSE) is a device that allows web management of CSS services. It allows agencies to directly control which load balanced servers are active and which are suspended in their CSS load balanced server groups. Everything is accomplished using a standard web browser and can be accessed from anywhere on the Commonwealth MAN, or remotely using Enterprise VPN services. 4.2 FUNCTIONAL OVERVIEW The HSE will allow agency users to make some changes to their assigned services on the CSS. Upon connecting to the HSE URL- https://hse.oaesf.state.pa.us they will be presented with a login screen. This screen will prompt for a login name and password that will be assigned by the EDC. In the near future the HSE will be integrated with AD for authentication. Upon logging in agency users will be allowed to suspend and activate load balanced members ONLY. No other configuration changes will be able to be done via the HSE (such as IP address changes, adding servers, etc.). These advanced changes must be completed by a Technical Operations Network Team member and can be requested via an incident ticket. Agency user access to the HSE can be completed via any web browser and uses this GUI interface. Services which represent the servers that are being load balanced are listed under the owner and content rule for the agency application. EDC CSS HARDWARE LOAD BALANCING POLICY PAGE 12 OF 19
The main parts of the interface that the agency user will deal with are objects representing the CSS, owners, content rules and services. In the figure above, the owner is represented by the owner object NAT_Test (owner). In practical application this will be the agency name. Under the owner is the content rule, in this case Test_Web_Servers (content rule). This content rule is designated for HTTP TCP port 80 and there is an IP address assigned to this content rule that is the VIP for the load balanced servers (this information can be seen in the right hand pane (see previous figure) by clicking on the content rule object). There is also a second content rule listed with the same VIP address (Test_Web_Servers_SSL) which is used for HTTPS TCP port 443. A content rule must be created for every service that is being run for a VIP address. Beneath those content rules are the services that represent the servers that are being load balanced (NAT_Test_1, NAT_Test_2, NAT_Test_1_SSL, NAT_Test_2_SSL). Each service represents an IP address and port on a load balanced server. For example NAT_Test_1 is the http tcp port 80 service for server 1 and NAT_Test_1_SSL is the https tcp port 443 service for the same server. The arrow in front of the service shows whether the service is up (GREEN arrow pointing up), failed (RED arrow pointing down) or suspended (YELLOW arrow pointing down). An agency contact with correct privileges can suspend or activate an agency service by checking the box directly in front of the service and then clicking the suspend button at the top of the screen. Reactivation of the service can be completed by again checking the box in front of the server and then clicking the activate button. More than one service can be activated or suspended at the same time by checking them all before clicking the Activate or Suspend button. A warning will come up in the right pane of the window and will ask for confirmation that the listed services are the ones that you want to suspend. When services have been suspended or activated as required, to save the changes to the CSS click on the CSS Object icon (listed as 172.17.234.50 (CSS/SSL) in this example). In the right pane a button to Save Device Configuration will appear. Click this button to make the service activation and suspensions permanent. When finished with CSS management click the Logout option in the upper right hand corner of the screen. EDC CSS HARDWARE LOAD BALANCING POLICY PAGE 13 OF 19
4.3 EDC HSE POLICY RECOMMENDATIONS If an agency is interested in using the HSE for management of their CSS load balanced servers they simply need to open an incident ticket or contact the call center (717) 506 1079 to request that access. Access will be granted on a user-by-user basis and each user will only be able to see objects representing resources that they have been allowed access to, as specified by the agency application owner(s). 4.4 AVAILABILITY The HSE is up and running now and will be made available for agency use in the near future. If you are interested in taking advantage of the HSE place contact the EDC Technical Operations Team via an incident ticket or by contacting the call center. EDC CSS HARDWARE LOAD BALANCING POLICY PAGE 14 OF 19
3 COMMONWEALTH OF PENNSYLVANIA 5 Global Load Balancing with Global Site Selector (GSS) 5.1 BACKGROUND The Global Site Selector (GSS) is available for global load balancing for critical applications that require the HIGHEST level of redundancy. The GSS offers the capability to load balance servers that are different physical sites. This offers the opportunity to put critical servers at multiple sites in case a major disaster occurred that would leave one site completely incapacitated. 5.2 FUNCTIONAL OVERVIEW The advantage of the GSS is that instead of running a Disaster Recovery (DR) site where servers are inactive and sit waiting for a disaster, the GSS allows active communication flows to be globally load balanced between the servers at two or more different sites. At the EDC there are currently two sites available for highly critical applications. The first is the CTC location of the EDC. The second is the EDC Interim Site located at the PennDot building on Cameron Street. This site is a DR site and is intended only as a standby facility in the case of a disaster and for highly critical applications to be located at for Global Load Balancing. The GSS allows Global Load Balancing between multiple sites by controlling DNS responses to clients. When Globally Load Balancing a URL, an NS record is setup in the domain s authoritative DNS server pointing the URL in question to the GSS unit. When a client queries for the IP address for the URL, the GSS checks with the CSS units at each site to see which is most capable to handle the request (based on many configurable load balancing settings). After determining the best site to send the end-user to, it returns the IP address for the server housing the application at that site. This way the end-user goes directly to the server in question. 3) ISP DNS server queries GSS for A record, GSS responds with IP address of most available server (using whatever algorithm has been selected) 1) Client request IP address for www.state.pa.us 4) Client directly contacts the server whose IP address was returned by GSS ISP DNS Server 1 End User Internet 4 4 3 2 PA Team (Internet/MAN) 2) ISP DNS server ask CoPA external DNS server for answer, DNS server replies with NS record, pointing client to GSS 2 Si Si Interim Site Data Center 3 Si Si CTC Data Center 4 state.pa.us External 4 state.pa.us External External DNS External DNS Cisc GS o S B Cisc GS o S A WWW.STATE.PA.US B WWW.STATE.PA.US A EDC CSS HARDWARE LOAD BALANCING POLICY PAGE 15 OF 19
5.3 EDC GSS POLICY RECOMMENDATIONS The EDC recommends highly critical applications be site load-balanced between the CTC and Interim Site. One server will be located at each are behind the local CSS at each location and then traffic will be diverted based on load to each physical location. For especially critical applications, a load-balanced pair of servers can be located at each site. 5.4 AVAILABILITY The EDC is currently implementing the GSS for the CTC and Interim site. It is targeted for June 2005. If you believe that you have a highly critical application that would benefit from Global Load Balancing via the GSS please contact the EDC Technical Operations Team. EDC CSS HARDWARE LOAD BALANCING POLICY PAGE 16 OF 19
6 Sorry Server Services 6.1 BACKGROUND As we have seen, the CSS units offer the ability to monitor agency servers with keepalives that offer the ability to failover to other available servers in the event of server failure. In conjunction with that capability, the CSS can offer a Sorry Server capability where if all servers for an agency application have failed, client requests can be directed to an EDC hosted server that will provide either a default sorry page or one customized for the application. This document describes the means by which an agency can take advantage of the EDC Sorry Service and details of how it can be used. 6.2 FUNCTIONAL OVERVIEW The EDC is hosting a Sorry Server in Managed Services that allows agencies to be directed to a default page in the event their CSS VIPed server(s) fail. The default sorry page is: We are sorry, the Commonwealth of Pennsylvania web site that you are trying to reach is either not available or is under going maintenance. Please try back later. Thank you for your Patience Any agency can take advantage of this feature with minimal changes to their existing CSS configuration by EDC personnel. In Phase 2 of the Sorry Server implementation the EDC will be exploring a means to allow agencies to add custom messages to Sorry Server pages. 6.3 EDC SORRY SERVER POLICY RECOMMENDATIONS The Sorry Server can be setup for any server located in the EDC that is taking advantage of the CSS load balancers. It is recommended that any interested parties test this service in Phase 1 of the deployment, so they will be prepared for Phase 2 implementation. 6.4 AVAILABILITY If any Commonwealth agency is interested in taking advantage of the EDC Sorry Service, or EDC Load Balancing services, please open a ticket with the EDC Call-Center at (717) 506 1079 option 1. Ask to have the ticket directed to the EDC Technical Operations Team. For additional questions on these services please feel free to contact that Technical Operations Team at: oa-esftot@state.pa.us. EDC CSS HARDWARE LOAD BALANCING POLICY PAGE 17 OF 19
7 Custom Notifications and Keepalives 7.1 BACKGROUND A final option that is available for agency applications that are using the CSS units is custom notifications. Custom notifications are E-mail notifications that are sent to necessary parties letting them know if the CSS sees their server as unavailable. 7.2 FUNCTIONAL OVERVIEW The CSS utilizes keepalives that constantly monitor the availability of configured servers. These keepalives can be as simple as a ping, tcp port probe, or an actual HTTP get or SSL hello message. When a server does not respond to a keepalive request 3 times the CSS will set the unit as unavailable and no longer send traffic to it. If it is part of a group of load balanced servers, it will divert all traffic to the other servers in its group. If it was the only active server in its group when it stopped responding the CSS can optionally redirect traffic to a Sorry Server. In any case, when the keepalive fails, the CSS will send an email notification to interested parties to inform them of the failure. EDC CSS HARDWARE LOAD BALANCING POLICY PAGE 18 OF 19
7.3 EDC CUSTOM NOTIFICATIONS POLICY RECOMMENDATIONS The preferred EDC keepalive is a non-persistent http head message sent to the listener used by the server fronted by the CSS. This is an HTTP HEAD request that asks the server to respond with a standard HTTP 200 code. However these require that the server is able to respond to a query to its IP address, otherwise a URI needs to be configured for the keepalive. Also, the server will be considered as down if it responds with any HTTP code other than a 200. For example, if the tested server responds with an HTTP redirect (300 code), the CSS would see it as down. If this type of keepalive can not be accommodated by the server, a TCP port ping can be used instead that verifies the server is listening on port 80, but does not assure that the web page is being displayed. Other advanced keepalives are available if this solution is not satisfactory, but these are in short supply and must be conserved for the most critical of applications. 7.4 AVAILABILITY The custom notifications feature of the CSS is available now. If you feel that you are receiving notifications that you should not, or are interested in receiving notifications for a given application, please contact the EDC Technical Operations team by opening an incident ticket or by contacting the call center at (717) 506 1079 option 1. EDC CSS HARDWARE LOAD BALANCING POLICY PAGE 19 OF 19