Disaster Recovery Design with Cisco Application Centric Infrastructure

Transcription

1 White Paper Disaster Recovery Design with Cisco Application Centric Infrastructure 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 1 of 46

2 Contents Overview... 3 Naming Conventions, IP Addresses, and VLANs... 4 Naming Conventions... 4 IP Addresses... 6 VLANs... 8 Design Requirements... 8 Tenant DMZ Tenant Server Farm Traffic Flow Internet Traffic Flows DCI Traffic Flows Internet Access through Disaster Recovery Disaster Recovery Topology and Service Flows Leaf and Spine Connectivity Server Connectivity with Leaf Switches Layer 4 Through 7 Device Connectivity to Leaf Switches Cisco ASR Router WAN Connectivity Cisco APIC Connectivity External Networking External Layer 2 Networks Service Architecture Design Firewall as the Gateway (NLB) Load Balancer in One-Armed Mode Traffic Flow Load Balancer as the Gateway Services Integration Service Device Packages Automated Service Insertion Cisco ASA Integration with Cisco ACI F5 Integration with Cisco ACI Virtual Machine Networking VMware vsphere Integration VMM Domain Configuration Management Network in Cisco ACI Out-of-Band Management Network Simple Network Management Protocol Syslog Network Time Protocol Conclusion For More Information Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 2 of 46

3 Overview One of the largest universities in the Europe, Middle East, Africa, and Russia (EMEAR) region is building a completely new (greenfield) disaster recovery site to meet service-level agreement (SLA) requirements for critical applications and services. Cisco Application Centric Infrastructure will be used to deliver these main services: Hosted applications for all faculty Research and development services for students Security, monitoring, and capacity planning Data center disaster recovery site capabilities Full benefits of F5 Layer 4 through 7 load-balancing capabilities Security enforcement for the application services using the Cisco Adaptive Security Appliances (ASA) firewall Figure 1 shows the building blocks of the network infrastructure. The existing primary IT services (ITS) data center is connected to the disaster recovery site through a data center interconnect (DCI). The Internet block at the disaster recovery site will be used for backup; in the event that the primary data center Internet goes down, the disaster recovery site Internet will be used. The ACI fabric will connect the servers, Layer 4 through 7 services, and the Internet and WAN blocks. Figure 1. Disaster Recovery Network Block The disaster recovery data center will house a number of devices, including blade servers, load balancers, firewalls, and virtualization appliances, all interconnected by networking equipment. These devices will require physical cabling with an increasing demand for higher performance and flexibility, requiring a reliable, scalable, and manageable cabling infrastructure. This document makes the following assumptions: On the basis of discussions with customer teams, initially the system will have two tenants. One VLAN and one IP network block per application tier is deployed. Data center servers generally are physical, with the use of some virtualized servers Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 3 of 46

4 The disaster recovery site hosts mission-critical applications: Microsoft SharePoint, Exchange, and Active Directory; Banner; Oracle E-Business Suite; web and portal content; file sharing, e-learning solutions; etc. The primary data center is the ITS site. The point-of-presence (POP) room at the Women s College of Science MDF (Main distribution frame) is the secondary site within the campus. The disaster recovery site is enabled for ACI. Dual 40-Gbps connectivity is used on leaf switches to connect to the spines. 1- and 10-Gbps connections are used on the servers to connect to the leaf switches. The disaster recovery site deploys three application tiers, in which all servers have gateways defined on the firewall. All front-end applications are network load balanced with SSL offload. All application and database servers are in same subnet. The F5 network load balancer is in single-arm mode, and for the Microsoft Exchange 2013 servers, the default gateway is on the F5 load balancer. All servers have static IP addresses and no need of Domain Host Configuration Protocol (DHCP). The disaster recovery site has a standalone Cisco ASA firewall deployed with two contexts and operates in routed mode. On the basis of the application flow, the F5 load balancer will be deployed using one-armed mode and routed mode. Cisco Catalyst 3750 Switches are used for out-of-band (OOB) network connectivity to all network devices. One Cisco Aggregation Services Router (ASR) 1001 Router will be connected to both leaf switches and the perimeter Internet router with two 1-Gbps fiber. Two Cisco ASR 1001 Routers are used at the disaster recovery site for DCI connectivity to the primary and secondary (co-location) sites. Role-based access control (RBAC) is used to define user administrative privileges and roles on the fabric. Services, including Simple Network Management Protocol (SNMP), Network Time Protocol (NTP), syslog, Domain Name System (DNS), FTP, Secure FTP (SFTP), and TACACS servers, use the out-of-band management network, and all these services are hosted in the ITS data center. VMware vcenter and Virtual Machine Manager (VMM) and service-graph integration for Cisco ASA firewall and F5 applications use the out-of-band management network. Naming Conventions, IP Addresses, and VLANs This section provides a summary of the naming conventions used by the customer network team and Cisco Advanced Services. Naming Conventions This section presents the naming conventions for customer disaster recovery site physical devices (Table 1), ACI logical constructs (Table 2), and access policies (Table 3) Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 4 of 46

5 Table 1. Naming Conventions for Physical Devices Building Floor Type and Manufacturer Device Type Sequence Number Host Name DRC 04 DR SP = Spine 01 DRC4DRSP01 DRC 04 DR SP = Spine 02 DRC4DRSP02 DRC 04 DR LF = Leaf 01 DRC4DRLF01 DRC 04 DR LF = Leaf 02 DRC4DRLF02 DRC 04 DR AP = Cisco Application Policy Infrastructure Controller (APIC) 01 DRC4DRAP01 DRC 04 DR AP = APIC 02 DRC4DRAP02 DRC 04 DR AP = APIC 03 DRC4DRAP03 DRC 04 DR RINT = Internet router 01 DRC4DRRINT01 DRC 04 DR RDCI = DCI router 01 DRC4DRRDCI01 DRC 04 DR F5 = F5 load balancer 01 DRC4DRF501 DRC 04 DR GTM = F5 Global Traffic Manager 01 DRC4DRGTM01 DRC 04 DR ASA = Cisco ASA firewall 01 DRC4DRASA01 Table 2. Naming Convention for Cisco ACI Logical Constructs (Tenant, Contexts, Bridge Domains, Endpoint Groups, etc.) Policy Construct Naming Format Naming Format Example Use Example Tenant <Name in CAPITAL letters identifying tenant function and transit-device details> INT_DMZ Tenant used for DMZ services Contexts <Name in CAPITAL letters with end-device type and traffic type and transit-device details> Transit_DMZ Context used for Internet transit traffic between Cisco ASA Instance <instance name> and Cisco ASR Bridge domain <Name in CAPITAL letters with end-device type and traffic type and transit-device details> FW_Internet_To_ASR_Int ernet Bridge domain used for VLAN traffic between the firewall and the Internet Application profile <Name in CAPITAL letters with end-device type and traffic type and transit-device details> Transit_ASR Application profile used to group endpoints; part of the traffic between the firewall and the Internet Endpoint group (EPG) <Name in CAPITAL letters with end-device type and traffic type and transit-device details> FW_Internet_To_ASR_Int ernet EPG used to connect to firewall interface <x/y> carrying Internet traffic 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 5 of 46

6 Table 3. Naming Conventions for Cisco ACI Access Policies Policy Construct Naming Format Naming Format Example Use Example VLAN pool <Maximum 10-letter PLATFORM NAME><Device or pair number if any> UCS-SHAREPOINT <Platform name> Physical domain <Maximum 10-letter PLATFORM NAME><Device or pair number if any> UCS-SHAREPOINT <Platform name> Attached Entity Profile <Maximum 10 letter PLATFORM NAME><Device or pair number if any> UCS-SHAREPOINT <Platform name> Link level <1/10 Gbps> 1G - Lightweight Access Control Protocol (LACP) <LACP-ACTIVE/PASSIVE> LACP-ACTIVE - Interface policy group <3- or 4-letter PLATFORM NAME>- <Interface type = vpc<number>/po<number>/ind<1/10 Gbps>-<Pair number>-<end device number> UCS-SHAREPOINT -vpc1-01 <Platform name> Interface profile <3- or 4-letter PLATFORM NAME>- <Interface type = vpc<number>/po<number>/ind<1/10 Gbps>-<Pair number>-<end device number> UCS-SHAREPOINT -vpc1-01 <Platform name> Switch profile <3- or 4-letter PLATFORM NAME>- <Interface type = vpc<number>/po<number>/ind<1/10 Gbps>-<Pair number>-<end device number> UCS-SHAREPOINT -vpc1-01 <Platform name> Port selector identity <3- or 4-letter PLATFORM NAME>- <Interface type = vpc<number>/po<number>/ind<1/10 Gbps>-<Pair number>-<end device number> UCS-SHAREPOINT -vpc1-01 <Platform name> IP Addresses ACI fabric requires a block of addresses to enable ACI infrastructure to initialize the fabric and assign IP addresses to different types of nodes. Here are some examples: Infrastructure loopback IP addresses (spine and leaf) Node-level communication with the APIC Tunnel endpoint (TEP) termination on the top-of-rack (ToR) switch Peering address for internal Border Gateway Protocol (ibgp), diagnostics, etc. Leaf loopback virtual PortChannel (vpc) TEP IP addresses Address of the logical virtual TEP (VTEP) shared between vpc peers 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 6 of 46

7 Leaf loopback fabric TEP IP addresses VTEP address used to communicate to downstream virtual switch (vswitch) VTEPs Identical across all leaf switches to allow downstream mobility of VTEP devices Spine loopback proxy anycast IP addresses Common anycast IP address shared by each fabric proxy redundancy group IPv4, IPv6, and Layer 2 MAC address proxies The fabric administrator specifies the infrastructure addressing scope. By default, the APIC assigns IP addresses from the /16 block (this setting is user configurable). For the disaster recovery project discussed here, new unique address spaces are used, as shown in Tables 4, 5, and 6. ACI provides both in-band and out-of-band capabilities to manage the ACI infrastructure. Table 4 shows the address spaces reserved for the ACI network. Table 4. IP Addresses for Cisco ACI Network Description TEP and infrastructure (infra) address In-band management (mgmt) address Out-of-band management address Router ID IP Range 10.xxx.11.0/23 10.xxx.13.0/24 10.xxx.14.0/24 10.xxx.0.128/29 Table 5 shows the address spaces used for fabric interconnectivity. Table 5. IP Addresses for Fabric Interconnectivity Description Point-to-point, fabric, and Cisco ASR connectivity Point-to-point, fabric, and Cisco ASA connectivity IP Range 10.xxx.1.128/27 10.xxx.1.160/27 Table 6 shows the address spaces used for management services. Table 6. IP Addresses for Management Services Description SNMP TACACS Syslog NTP Secure Copy (SCP) and FTP DNS IP Range yy.xxx.xxx.xxxx 10.xxx xxx yy.xxx.xxx.xxxx yy.xxx.xxx.xxz yy.xxx.xxx.xxxx 10.xxx xxx Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 7 of 46

8 VLANs Table 7 lists the VLANs allocated for disaster recovery in an ACI implementation. Table 7. VLANs in Fabric VLAN Description VLAN Range Server farm VLAN (static) Server farm VLAN (dynamic) Layer 2 in-band management Internet routers VLAN WAN/DCI routers VLAN Layer 2 connectivity from external network Testing VLAN Connectivity to the F5 load balancer Connectivity to the Cisco ASA Fabric infrastructure VLAN 4093 Design Requirements In Cisco ACI terminology, a tenant is the highest-level logical container in which objects and policies of a given group or organization can reside. A tenant itself isn t mapped directly to existing (legacy) network constructs such as virtual routing and forwarding (VRF) instances, virtual private networks (VPNs), interfaces, and security policies. Tenants provide the following high-level properties: Isolation: Tenants can be totally isolated from one another or share certain resources. RBAC: Users can be created that have visibility and specific privileges over only one or more specific tenants (this is called a domain). Inheritance: Objects inside a tenant inherit that tenant s top-level policies. In cases in which the departments within an organization do not all adhere to the same approach to network design, a single-tenant model can quickly become tedious to maintain at the design and operation levels. When a given department or service function is confined to its corresponding tenant in ACI, troubleshooting and fault isolation can be performed more efficiently. A tenant can choose to export (or import) objects to (or from) other tenants, thereby relaxing the strict isolation initially provided by the multitenant model. Implementation of more than one tenant provides a clean model for controlling inter-tenant communications when desirable. The use of multiple tenants lets you easily totally isolate one environment from another. Although it is true that inside a tenant, inter-epg communication is never allowed explicitly, the one environment = one tenant model lets you isolate or permit communication in a highly specific way. The ACI fabric design proposed in the context of a customer disaster recovery deployment addresses the needs of multiple internal network blocks. Those blocks, summarized in Table 8, are: Internet DMZ tenant at the disaster recovery perimeter Intranet server farm tenant 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 8 of 46

9 The Internet DMZ tenant provides services from the external world. The Cisco ASR Internet routers and WAN DCI routers are part of this tenant, as is the Internet context of the firewall. The server farm tenant addresses the internal applications. Each of these service functions has its specific applications and policies as well as traffic patterns. Mixing all of these within one overarching tenant results in operational complexity and little administrative flexibility. It also reduces the ease with which isolation and fault containment can be provided between departments. RBAC applies roles to domains. Domains can group from 1 to N tenants. With this model, you can easily create highly specific access control policies. Table 8. Tenant Description and Naming Description Internet DMZ tenant at disaster recovery perimeter Intranet server farm Tenant INT_DMZ SF Figure 2. High-Level View of Tenants and Required Services 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 9 of 46

10 Tenant DMZ The tenant DMZ provides Internet services and a pool of shared network and application services to the customer s users. The customer has an Internet link from two different service providers at the primary data center. The BGP routing protocol is used to provide dynamic failover between the two Internet service providers (ISPs). If the primary data center site is down, the Internet link at the disaster recovery site provides backup Internet services for campus and external users. The tenant DMZ provides the following services: Internet services for the primary data center in the event that the primary data center Internet service is down Internet services for all the services at the disaster recovery site Tenant Server Farm The tenant server farm provides mission-critical application services to the customer s users. The customer s secondary POP site within the campus functions as an alternative site for some core internal services that are required within the campus: for example, DHCP, DNS, IP telephony, and read-only domain controllers. The tenant server farm provides the following services: Microsoft SharePoint Microsoft Exchange 2013 Oracle ERP and E-Business Suite (Exadata) Banner systems (Oracle Exadata) E-learning (Blackboard) File sharing Web portal and content Domain controller Traffic Flow This section discusses the business- and customer-initiated traffic flows destined for the customer s disaster recovery data center. The traffic flows are of two main types: Internet traffic DCI traffic Internet Traffic Flows The Internet traffic flows provide access from the DMZ servers and load-balancing servers to the Internet, and also the access from the Internet to these servers. In the customer environment, the servers can access the Internet through a secure channel for software patching. Internet DMZ server access to Internet (outbound); flow (a) in Figure 3: All DMZ servers have a gateway on the Cisco ASA Internet firewall context, which sends traffic to the perimeter Cisco ASR router to access the Internet. Internet traffic reaching the Internet DMZ server (inbound); flow (b) in Figure 3: F5 GTM determines the application availability site (primary data center or disaster recovery site) based on DNS entries. All 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 10 of 46

11 incoming traffic travels through the perimeter firewall context (INT), and if incoming traffic is allowed, perimeter firewall traffic will go to servers in the DMZ. Server access to the Internet (outbound traffic); flow (c) in Figure 3: The server gateway is defined in the server farm context on the Cisco ASA firewall, which sends traffic to the Cisco ASA Internet context through the fabric (the fabric acts as Layer 2 transport). After the traffic reaches the perimeter firewall of the Cisco ASA Internet context, depending on the destination IP address, the traffic is forwarded to the Internet ASR router to reach the Internet. All front-end servers are load balanced with F5, which works in one-arm mode. Internet traffic reaching the servers (inbound traffic); flow (d) in Figure 3: GTM determines the application availability site (primary data center or disaster recovery site) based on DNS entries. All incoming traffic travels through the Cisco ASA perimeter firewall context (Internet context). If incoming traffic is allowed, after passing the perimeter, it is routed through the leaf (in this case, the fabric acts as Layer 2 transport) to the internal server farm firewall context. The server farm context on the Cisco ASA firewall s route to reach the virtual IP address or network load-balancing (NLB) server s IP address depends on the destination IP address. If the request is for NLB servers, the traffic will be forwarded to the F5 Local Traffic Manager (LTM), which will forward the traffic to a real server. Figure 3. One-Arm Load-Balancing Internet Traffic Flow 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 11 of 46

12 DCI Traffic Flows The DCI traffic flows provide access from the DMZ servers and load-balancing servers to the primary data center and co-location site and also access from the ITS data center applications to these servers. DMZ server access to ITS data center (outbound); flow (a) in Figure 4: All DMZ servers have a gateway on the Cisco ASA Internet DMZ firewall context, which sends traffic to the DCI Cisco ASR routers to access the primary and secondary sites. ITS traffic reaching the Internet DMZ server (inbound); flow (b) in Figure 4: All incoming traffic from the ITS data center travels through the perimeter firewall context (INT_DMZ), and if incoming traffic is allowed, after passing the perimeter firewall, the traffic will go to the servers in the DMZ. Server farm server access to ITS site (outbound traffic); flow (c) in Figure 4: The tenant server farm server s gateway is defined in the server farm SF context on the Cisco ASA firewall. On the basis of the destination IP address, the traffic will be forwarded to the DCI ASR routers to reach the primary or secondary site. All front-end servers are load-balanced with F5, which works in one-arm mode. ITS traffic reaching the disaster recovery server farm servers (inbound traffic); flow (d) in Figure 4: All incoming traffic from ITS travels directly to the server farm context on the Cisco ASA firewall. Its route to reach the virtual IP address or NLB server s IP address depends on the destination IP address. If the request is for load-balancing servers, the traffic is forwarded to the F5 LTM, and then the traffic is forwarded to a real server. Figure 4. DCI Traffic Flow for One-Arm Load Balancing 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 12 of 46

13 Internet Access through Disaster Recovery If the primary data center site is down in the primary data center, the Internet link at the disaster recovery site will be used for backup Internet service. Figure 5 shows the flow. Figure 5. Internet Access Through Disaster Recovery Disaster Recovery Topology and Service Flows Figure 6 shows the logical topology of both the ITS and disaster recovery data centers. Figure 6. Customer Data Center High-Level Topology Overview 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 13 of 46

14 Figure 6 is summarized here: Three main components of Cisco ACI Fixed-chassis spine (Cisco Nexus 9336PQ ACI Spine Switch) with 36 ports to connect with leaf switches Leaf (Cisco Nexus 9396PX Switch) with 48 x 1/10-Gbps fiber ports for host connectivity and 12 x 40- Gbps ports to connect with spines Cisco APIC appliances (Cisco UCS C220 M3 Rack Server) 40-Gbps BiDi optics between leaf and spine switches (OM4 cables) Two 1-Gbps connection between disaster recovery site and primary data center in ITS data center (through Cisco ASR routers) Two 100-Gbps connection used to connect each data center with Multiprotocol Label Switching (MPLS) cloud Hardware summary Two spine switches (Cisco Nexus 9336PQ) Two leaf switches (Cisco Nexus 9396PX) Two Cisco ASR 1000 Series routers used for WAN DCI connectivity One Cisco ASR 1000 Series router used for Internet connectivity One Cisco ASA firewall (Cisco ASA 5585-X) One F5 BIG-ADC10200V Two F5 GTM F5-BIG-DNS-2000S Leaf and Spine Connectivity Figure 7. Customer Disaster Recovery Physical Connectivity Overview 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 14 of 46

15 The main concepts of leaf and spine connectivity are summarized here: Endpoint devices (servers, firewalls, load balancers, etc.) and Cisco APIC connected to leaf nodes only Spine switches connected to leaf switches only Leaf switches rely on the spine s knowledge of all endpoints Depending on bandwidth requirements, all leaf nodes connect to both spines with one 40-Gbps uplink. A maximum of 12 uplinks can be used from the leaf switches to the spine switches. Server Connectivity with Leaf Switches The servers connect to leaf nodes only. Each leaf has a total 48 x 1/10-Gbps fiber ports available for endpoints, supporting individual port connectivity, active and standby mode connectivity, and vpc or Port Channel connectivity or both. Leaf switches support vpc. No vpc peer link is required. Peer communication occurs over the fabric. Path recovery also occurs over the fabric. Within the fabric, the vpc interfaces use an anycast VTEP that is active on both vpc peers. The servers can connect to leaf switches using individual interfaces, PortChannels, active and standby modes, and vpc. Figure 8. Server Connectivity Options with Leaf Switches 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 15 of 46

16 Layer 4 Through 7 Device Connectivity to Leaf Switches As mentioned earlier, the disaster recovery project uses a total of two leaf switches. The same leaf switches are used to connect Layer 4 through 7 devices: Cisco ASA firewall F5 load balancer The customer uses the F5 BIG-IP load balancer and Cisco ASA firewall services devices, enabled for Cisco ACI and configured and managed by Cisco APIC. The ACI fabric supports traditional service devices with features such as flooding bridge domain (Figure 9). Figure 9. F5 LTM and GTM and Cisco ASA Firewall Connectivity to Leaf Switches 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 16 of 46

17 Cisco ASR Router WAN Connectivity Figure 10 shows the connectivity between the Cisco ASR routers and the WAN. Figure 10. Cisco ASR and Leaf Switch Connectivity Cisco APIC Connectivity Three Cisco APIC devices are installed in the disaster recovery site using two 10-Gbps interfaces in active-standby mode and standard Linux interface bonding (Figure 11). This configuration is provided by default, and no additional configuration is required. The APIC cluster is a fully redundant multinode cluster, and upgrades and downgrades are performed through the cluster redundancy and not through In-Service Software Upgrade (ISSU) on individual nodes. (This solution uses a transactional redundancy model, and as in Hadoop and other distributed systems, it is designed to operate with node state changes. The addition of new nodes and the upgrading and removal of existing nodes are simple forms of cluster scale up and scale down.) 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 17 of 46

18 Figure 11. Cisco APIC Connectivity External Networking The ACI fabric uses the concepts of inside and outside networks. Anything designated as within, or inside, the fabric is subject to the forwarding rules and policy enforcement. However, external, or outside, entities almost always need to be connected to the fabric (WANs, existing networks, etc.), so a way is needed to designate these external entities as not being part of the fabric. To achieve this, the concept of external, or outside, networks is used. External networks can be at Layer 2 or Layer 3. In this scenario, the fabric is acting as Layer 2 between the firewall and Cisco ASR, so neither Layer 3 nor peering is required. External Layer 2 Networks External (outside) bridged and Layer 2 networks are required whenever the Layer 2 domain needs to be extended beyond the ACI fabric: for example, when a Layer 2 interconnect is required between data center sites. When a Layer 2 external network is configured, essentially a mapping is created between a fabric bridge domain and a VLAN external to the fabric. In addition, ports on one or more leaf switches are designated as border ports. These interfaces connect to the external device. The customer wants to extend a number of Layer 2 segments across data center sites. To support this requirement, the Cisco ASR 1000 Series routers use Cisco Overlay Transport Virtualization (OTV) technology to transport VLANs between the disaster recovery site and the primary data center Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 18 of 46

19 To support Layer 2 extension from the ACI fabric, a Layer 2 outside construct is used to extend bridge domains beyond the fabric. A Layer 2 outside construct works by extending a particular bridge domain using a specified VLAN. Figure 12 shows how the Layer 2 outside construct is used to extend bridge domains outside the fabric. Figure 12. Layer 2 Extension from the Cisco ACI Fabric In Figure 12, the fabric is Layer 2 connected to the DCI ASR 1001 Routers for Layer 2 extension between the ITS data center and the disaster recovery site. The Cisco ASR 1001 is connected to a leaf port, which is configured as a border leaf port during the creation of the Layer 2 outside construct. Note the following points about border ports: Border ports have Bridge Protocol Data Unit (BPDU) guard disabled (regular fabric edge ports have this feature enabled). Spanning-tree BPDUs are forwarded between border ports within the same bridge domain. When a bridge domain is extended using a Layer 2 outside construct, the bridge domain must operate in regular mode: in other words, Address Resolution Protocol (ARP) optimization should be disabled, and unknown unicast flooding should be enabled. Depending on the specific bridge domain being extended, these settings may already be configured (for example, bridge domain settings are different depending on where the default gateway resides). Layer 2 External constructs are configured from the APIC using the External Bridged Networks configuration option under Networking. Figure 13 shows how this item is configured in the APIC GUI Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 19 of 46

20 Figure 13. Layer 2 Outside Configuration Note that the Layer 2 outside construct must be mapped to a bridge domain. A VLAN must also be specified for the external encapsulation. Finally, the physical node and interface to which the external device (in this case, the Cisco ASR 1000 Series router) connects must be specified. Service Architecture Design The customer deploys most applications in a three-tiered architecture (web, application, and database tiers, or similar layers). Note the following points about three-tiered applications: The servers in a three-tiered architecture use one VLAN per application tier. The server farm and DMZ tenants use Cisco ASA firewalls for security enforcement. A standalone F5 load-balancer appliance is used for load balancing. Additional VRF instances would map to additional firewall contexts. Additional firewall contexts would use shared EPG and bridge domains for connection to the Cisco ASR and intercontext communication. There is one bridge domain per application tier, contained within a VRF instance, but a gateway is not configured on the fabric. The firewall is the default gateway for all hosts (except for the Microsoft Exchange application). Therefore, all east-west and north-south traffic passes through the firewall. The F5 load balancer is used in proxy Source Network Address Translation (SNAT) mode for most missioncritical services. Mainly the web and application tiers use the load-balancer service. For Microsoft Exchange application F5 BIG-IP LTM is deployed in Layer 3 inline (two-arm) mode, where the source IP address of the client is visible to server (direct server return). Firewalls and F5 BIG-IP run static routing with distribution switches. Distribution switches run dynamic routing protocol with WAN edge and provider edge routers. One load-balancing context is created per application group Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 20 of 46

21 For the solution discussed here, application-tier bridge domains are configured as follows: ARP flooding: Enabled. Within a bridge domain, servers must use ARP for the default gateway (the firewall). If the firewall is silent (that is, if it does not send out Generic ARP (GARP) or similar packets), the ACI fabric will not learn the MAC address information for the firewall. In this situation, ARP requests from the host to the firewall will fail. To avoid this scenario, regular ARP flooding is enabled in the application-tier bridge domains. In other words, ACI-optimized ARP handling is not enabled for these bridge domains. Unicast routing: Disabled. If unicast routing is enabled within a bridge domain, learning occurs for a host s IP and MAC addresses. In situations in which the fabric is not participating in routing (such as when a firewall acts as the gateway), IP addresses do not need to be learned. Therefore, this option is disabled for each of the application-tier bridge domains. Layer 2 unknown unicast: Flood. Hardware proxy handling of unknown unicast traffic is the default option. For topology 1, regular Layer 2 handling is enabled (flooding). Gateway and subnet addresses: Not configured. In this topology, all Layer 3 gateway services are running on the firewall, not the ACI fabric. Therefore, a subnet or gateway should not be configured in the application-tier bridge domains. For the customer s disaster recovery design, several topologies are supported for three-tiered applications. These topologies provide different options for integration of firewall and load-balancing services, as follows: Scenario 1 Firewall is the gateway for the servers. Load balancer service is not required. Scenario 2 Firewall is the gateway for the servers. Load balancer is in one-armed mode. Scenario 3 Load balancer acts as default gateway for hosts (load balancer operating in inline mode). Firewall is used in Layer 3 routed mode. Note that a combination of these topologies can be used for flexibility: for example, if one application group (web, application, and database) needs to use the firewall as the default gateway, and a second application group needs to use the load balancer as the gateway. The customer indicated that all application tiers will use the firewall as the default gateway except Microsoft Exchange, which uses the load balancer as the gateway. Firewall as the Gateway (NLB) In scenario 1, the firewall is used as the default gateway for all hosts and servers on a given VLAN. This means that all communication in and out of that VLAN is secured by the firewall Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 21 of 46

22 Figures 14 through 17 show how this topology works. Figure 14. NLB DMZ Servers: Firewall as Default Gateway (Internet Traffic) Figure 15. NLB DMZ Servers: Firewall as Default Gateway (DCI Traffic) 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 22 of 46

23 Figure 16. NLB Server Farm Servers: Firewall as Default Gateway (Internet Traffic) The following points relate to the topology in Figure 16: The Cisco ASA firewall contexts operate in routed mode. An EPG is created on the ACI fabric for each application tier. A bridge domain is created on the ACI fabric for each application tier. A separate bridge domain is created to handle dynamic routing between the Cisco ASA Internet context and the Cisco ASR 1001 Routers. A separate bridge domain is created to handle routing (static or Open Shortest Path First [OSPF]) between the Cisco ASA Internet context and the server farm context. The fabric does not actively participate in routing, but instead acts as the Layer 2 transport Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 23 of 46

24 Figure 17. NLB Server Farm Servers: Firewall as Default Gateway (DCI Traffic) The following points relate to the topology shown in Figure 17: The Cisco ASA firewall contexts operate in routed mode. An EPG is created on the ACI fabric for each application tier. A bridge domain is created on the ACI fabric for each application tier. A separate bridge domain is created to handle dynamic routing between the Cisco ASA Internet context and the Cisco ASR 1001 Routers. The fabric does not actively participate in routing, but instead acts as the Layer 2 transport. As mentioned previously, an EPG is created for each application tier: web, application, and database. By keeping the firewall interfaces in the same EPG as the hosts, you do not need to configure contracts between the host and the firewall (endpoints within the same EPG can communicate freely). For communication between the firewalls and the Cisco ASR 1001 Routers, a similar EPG arrangement is used. In this topology, the firewall and the Cisco ASR router are treated as regular hosts by the fabric; therefore, both the firewalls and the Cisco ASR 1001 Routers can reside in a dedicated INT_ASR EPG. On the basis of the current understanding of the applications to be deployed in the disaster recovery site, one context will be configured per tenant. In addition to the regular application tier bridge domains, a separate bridge domain is required for dynamic routing between the firewalls and the Cisco ASR 1001 Routers. In this topology, the dynamic-routing bridge domain should be configured with the same settings as the application-tier bridge domain Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 24 of 46

25 Load Balancer in One-Armed Mode There are many ways to insert the F5 load balancer into the traffic path in the ACI fabric. The actual traffic flow will depend on the service being load balanced and the configuration of the core components including the ACI, F5 load balancer, firewalls, and connecting infrastructure. According to customer application requirements, the gateway to all the traffic resides on the firewall, and F5 performs SNAT. The F5 BIG-IP LTM is connected to its gateway (firewall) using a single interface and VLAN only. It is not inline: that is, it does not sit in the traffic path, and only traffic that needs to be load balanced is directed to it. The traffic that the client initializes for the virtual server virtual IP address travels to the load balancer through service graph insertion. When the F5 load balancer operates in one-arm mode, it uses a pool of addresses, known as SNAT IP addresses. When traffic reaches the virtual IP address of the load balancer, the source IP address is replaced with an address from the SNAT pool. Return traffic from the real server is directed to the SNAT IP address; therefore, the load balancer sees all return traffic. In this topology, the firewall is used as the default gateway for all hosts and servers on a given VLAN. This means that all communication in and out of that VLAN is secured by the firewall. The load-sharing algorithm picks a physical server, and the load balancer forwards the traffic to the physical IP address of this server. In this topology, the F5 BIG-IP LTM load balancer operates in automated mode: that is, the device package is uploaded to the APIC, service graphs are used to redirect traffic to the load balancer, and configuration tasks are moved to the APIC. Figure 18 and 19 show how this topology works. Figure 18. Load Balancer in One-Arm Mode: Firewall as Gateway (Internet Traffic) 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 25 of 46

26 Figure 19. Load Balancer in One-Arm Mode: Firewall as Gateway (DCI Traffic) The following points relate to the topology in Figure 19: The firewalls operate in routed mode. The load balancer operates in one-arm mode. An EPG is created on the ACI fabric for each application tier. A bridge domain is created on the ACI fabric for each application tier. An EPG is created for the transit connection to F5 BIG-IP LTM. A bridge domain is created for the transit connection to F5 BIG-IP LTM. A separate bridge domain is created to handle dynamic routing between the firewalls and the Cisco ASR 1000 Series routers. Dynamic routing (OSPF) runs between the firewalls and Cisco ASR routers. This routing exchange occurs through the fabric (that is, the fabric does not actively participate in OSPF, but instead acts as the Layer 2 transport). No contract is necessary for host-to-firewall communication. The load-balancer configuration is provisioned using the service graph. A contract is created between the provider (server) and the consumer (client side) traffic, and the same contract binds the load-balancer graph function. By keeping the firewall interfaces in the same EPG as the hosts, contracts between the host and the firewall are not needed (endpoints within the same EPG can communicate freely). Because the load balancer is operating in one-arm mode, it is attached to the ACI fabric using a single arm. To accommodate this connection, a dedicated EPG is set up in which both the load-balancer interface and the firewall interface facing the load balancer reside. The load balancer and firewall can communicate freely without an ACI contract Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 26 of 46

27 For communication between the firewalls and the Cisco ASR 1000 Series routers, a similar EPG arrangement is used. In this topology, the firewall and the Cisco ASR router are treated as regular hosts by the fabric; therefore, both the firewalls and the Cisco ASR 1000 Series routers can reside in a dedicated INT_ASR EPG. A dedicated bridge domain is configured for each application tier. This design was chosen (instead of a single bridge domain) to prevent broadcast traffic from unnecessarily traversing application tiers. In addition to the regular application bridge domains, a separate bridge domain is required for dynamic routing between the firewalls and the Cisco ASR 1000 Series routers. In this topology, the dynamic-routing bridge domain should be configured with the same settings as the application-tier bridge domains. A dedicated bridge domain is also required for the transit connection between the firewall and the load balancer. Dynamic routing is required between the firewalls and the Cisco ASR 1000 Series routers to exchange routing information to and from external networks. Because the ACI fabric is not responsible for any Layer 3 services in this topology, dynamic routing is performed directly between the firewalls and the routers. Therefore, the ACI is operating purely as a Layer 2 transport. Although dynamic routing is taking place between the firewalls and the Cisco ASR 1000 Series routers, the fabric does not play an active role in this exchange; therefore the EPG and bridge domain configuration to achieve this behavior is identical to the configuration for the application tiers. Firewalls and Cisco ASR 1000 Series routers can all reside in the same EPG, eliminating the need to configure contracts to allow communication. Traffic Flow This section describes a traffic flow from an external client destined for a web server and the associated return flow. Figure 20 shows the traffic flow from the client to the server. Figure 20. Traffic Flow: Client to Server (DCI Traffic) Note: In the example in Figure 20, the database, application, and web servers are using the firewall as the default gateway. The database and application servers are not load balanced in this scenario Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 27 of 46

28 In Figure 20, the traffic enters the data center through the Cisco ASR 1000 Series router, where it is routed to the virtual IP address of the load balancer through the firewall. The firewall needs a route pointing to the virtual IP address through the load balancer interface (accessible through the bridge domain named LB_ONE-ARM). Upon receiving this traffic, the load balancer creates a new session with a source IP address from the IP address pool. This new flow is destined for the real server and is routed through the firewall. Finally, the firewall sends the traffic to the real server using the relevant application bridge domain and EPG (in this case, SRV1). The traffic flow is summarized here: A flow from outside is destined for the virtual IP address. The Internet ASR traffic is routed to the virtual IP address through the Cisco ASA DMZ context. The firewall DMZ context needs to route traffic to the virtual IP address. The firewall DMZ context has a route pointing to the virtual IP address through the firewall server farm context. The firewall server farm context has a route pointing to the virtual IP address of F5 BIG-IP LTM. Upon receiving the traffic, the load balancer creates a new session with a source IP address from the SNAT pool defined in the service graph parameters and changes the destination IP address on the packet to point to the back-end server. The new flow is destined now for the real servers based on the load-balancing method and is routed through the firewall as a gateway. The firewall server farm context routes the flow to the correct application bridge domain. Finally, the firewall sends the traffic to the real server using the relevant bridge domain and EPG (web EPG and web bridge domain in this case). The flow in Figure 21 is summarized here: Figure 21. Traffic Flow: Client to Server (DCI Traffic) 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 28 of 46

29 A flow from outside is destined for the virtual IP address. The DCI ASR route to the virtual IP address is through the Cisco ASA server farm context. The firewall server farm context has a route pointing to the virtual IP address of F5 BIG-IP LTM. Upon receiving the traffic, the load balancer creates a new session with a source IP address from the SNAT pool defined in the service graph parameters and changes the destination IP address on the packet to point to the back-end server. The new flow is destined now for the real servers based on the load-balancing method and is routed through the firewall as a gateway. The firewall server farm context routes the flow to the correct application bridge domain. Finally, the firewall sends the traffic to the real server using the relevant bridge domain and EPG (web EPG and web bridge domain in this case). Figures 22 and 23 show the return traffic flow. Figure 22. Traffic Flow: Server-to-Client Return Traffic (to Internet) 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 29 of 46

30 Figure 23. Traffic Flow: Server-to-Client Return Traffic (to DCI) For the return flow, the web server sends traffic to the IP address, which was derived from the address pool. This traffic is then routed through the firewall. The firewall routes the traffic to the load balancer s interface. Finally, the firewall sends the return traffic back to the original client through the firewall and Cisco ASR 1001 Router Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 30 of 46

31 The return flow is summarized here: The real server responds to the load-balancer address through the default gateway (firewall). The firewall routes the traffic to the load balancer address through the load-balancer interface. The load balancer sends traffic back to the client through the firewall. The firewall routes traffic back to the client through the Cisco ASR. Load Balancer as the Gateway In this topology (Figure 24), the load balancer is used as the default gateway for all hosts and servers on a given VLAN. This approach means that all communication in and out of that VLAN also must traverse the firewall. The major difference in this topology is that the load balancer operates in inline mode. Figure 24. Load Balancer as Default Gateway (Inline Mode Internet Traffic) In the example in Figure 24, the web server uses the load balancer as the default gateway. The application and database servers use the firewall as the gateway and are not load balanced in this scenario. The following main points relate to this topology (Figure 25): The load balancer and firewalls operate in inline routed mode. An EPG is created on the ACI fabric for each application tier. A bridge domain is created on the ACI fabric for each application tier. A separate bridge domain is created to handle traffic between the load balancers and the firewalls. A separate bridge domain is created to handle dynamic routing between the firewalls and the Cisco ASR 1000 Series routers. An EPG is deployed for the external side of the load balancer (in this case named LB_INLINE) Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 31 of 46

32 A contract is deployed between the server EPG (for example, Web) and the load balancer external EPG (for example, LB_INLINE). This contract refers to a service graph, which redirects the incoming traffic to the load balancer. The load-balancer configuration is provisioned using the service graph. A contract is created between the provider (server) and the consumer (client side) traffic, and the same contract binds the load-balancer graph function. Figure 25. Load Balancer as Default Gateway (Inline Mode DCI Traffic) Figures 26 and 27 show the return traffic for the inline mode deployment. The server sends the return traffic pointing the client IP address to F5 BIG-IP LTM. F5 BIG-IP has a route pointing to the Cisco ASA server farm context firewall and to the client IP address. Depending on the client s location, the traffic is forwarded to the DCI ASR or Internet ASR Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 32 of 46

33 Figure 26. Load Balancer as Default Gateway (Inline Internet Return Traffic) Figure 27. Load Balancer as Default Gateway (Inline DCI Return Traffic) 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 33 of 46

34 Services Integration Service Device Packages A device package is an archive of information containing the details required to manage an external service device (either from Cisco or a third party). A device package is presented as a compressed zip file containing the following: XML-based device specification Python script implementing the device script API The device specification contains XML code that specifies parameters such as version information, functions, and configuration options for the device. The device script handles the interface with the service device using its API (preferred) or the command-line interface (CLI). A device package can be uploaded to the APIC easily using the GUI. The L4-L7 Services tab contains a Packages submenu from which packages can be uploaded in zip file format as shown in Figure 28 Figure 28. Importing Device Packages Figure 29. Importing Device Packages from the Packages Menu 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 34 of 46