White Paper Disaster Recovery Design with Cisco Application Centric Infrastructure 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 1 of 46
Contents Overview... 3 Naming Conventions, IP Addresses, and VLANs... 4 Naming Conventions... 4 IP Addresses... 6 VLANs... 8 Design Requirements... 8 Tenant DMZ... 10 Tenant Server Farm... 10 Traffic Flow... 10 Internet Traffic Flows... 10 DCI Traffic Flows... 12 Internet Access through Disaster Recovery... 13 Disaster Recovery Topology and Service Flows... 13 Leaf and Spine Connectivity... 14 Server Connectivity with Leaf Switches... 15 Layer 4 Through 7 Device Connectivity to Leaf Switches... 16 Cisco ASR Router WAN Connectivity... 17 Cisco APIC Connectivity... 17 External Networking... 18 External Layer 2 Networks... 18 Service Architecture Design... 20 Firewall as the Gateway (NLB)... 21 Load Balancer in One-Armed Mode... 25 Traffic Flow... 27 Load Balancer as the Gateway... 31 Services Integration... 34 Service Device Packages... 34 Automated Service Insertion... 35 Cisco ASA Integration with Cisco ACI... 36 F5 Integration with Cisco ACI... 36 Virtual Machine Networking... 37 VMware vsphere Integration... 37 VMM Domain Configuration... 38 Management Network in Cisco ACI... 40 Out-of-Band Management Network... 41 Simple Network Management Protocol... 43 Syslog... 44 Network Time Protocol... 45 Conclusion... 45 For More Information... 45 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 2 of 46
Overview One of the largest universities in the Europe, Middle East, Africa, and Russia (EMEAR) region is building a completely new (greenfield) disaster recovery site to meet service-level agreement (SLA) requirements for critical applications and services. Cisco Application Centric Infrastructure will be used to deliver these main services: Hosted applications for all faculty Research and development services for students Security, monitoring, and capacity planning Data center disaster recovery site capabilities Full benefits of F5 Layer 4 through 7 load-balancing capabilities Security enforcement for the application services using the Cisco Adaptive Security Appliances (ASA) firewall Figure 1 shows the building blocks of the network infrastructure. The existing primary IT services (ITS) data center is connected to the disaster recovery site through a data center interconnect (DCI). The Internet block at the disaster recovery site will be used for backup; in the event that the primary data center Internet goes down, the disaster recovery site Internet will be used. The ACI fabric will connect the servers, Layer 4 through 7 services, and the Internet and WAN blocks. Figure 1. Disaster Recovery Network Block The disaster recovery data center will house a number of devices, including blade servers, load balancers, firewalls, and virtualization appliances, all interconnected by networking equipment. These devices will require physical cabling with an increasing demand for higher performance and flexibility, requiring a reliable, scalable, and manageable cabling infrastructure. This document makes the following assumptions: On the basis of discussions with customer teams, initially the system will have two tenants. One VLAN and one IP network block per application tier is deployed. Data center servers generally are physical, with the use of some virtualized servers. 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 3 of 46
The disaster recovery site hosts mission-critical applications: Microsoft SharePoint, Exchange, and Active Directory; Banner; Oracle E-Business Suite; web and portal content; file sharing, e-learning solutions; etc. The primary data center is the ITS site. The point-of-presence (POP) room at the Women s College of Science MDF (Main distribution frame) is the secondary site within the campus. The disaster recovery site is enabled for ACI. Dual 40-Gbps connectivity is used on leaf switches to connect to the spines. 1- and 10-Gbps connections are used on the servers to connect to the leaf switches. The disaster recovery site deploys three application tiers, in which all servers have gateways defined on the firewall. All front-end applications are network load balanced with SSL offload. All application and database servers are in same subnet. The F5 network load balancer is in single-arm mode, and for the Microsoft Exchange 2013 servers, the default gateway is on the F5 load balancer. All servers have static IP addresses and no need of Domain Host Configuration Protocol (DHCP). The disaster recovery site has a standalone Cisco ASA firewall deployed with two contexts and operates in routed mode. On the basis of the application flow, the F5 load balancer will be deployed using one-armed mode and routed mode. Cisco Catalyst 3750 Switches are used for out-of-band (OOB) network connectivity to all network devices. One Cisco Aggregation Services Router (ASR) 1001 Router will be connected to both leaf switches and the perimeter Internet router with two 1-Gbps fiber. Two Cisco ASR 1001 Routers are used at the disaster recovery site for DCI connectivity to the primary and secondary (co-location) sites. Role-based access control (RBAC) is used to define user administrative privileges and roles on the fabric. Services, including Simple Network Management Protocol (SNMP), Network Time Protocol (NTP), syslog, Domain Name System (DNS), FTP, Secure FTP (SFTP), and TACACS servers, use the out-of-band management network, and all these services are hosted in the ITS data center. VMware vcenter and Virtual Machine Manager (VMM) and service-graph integration for Cisco ASA firewall and F5 applications use the out-of-band management network. Naming Conventions, IP Addresses, and VLANs This section provides a summary of the naming conventions used by the customer network team and Cisco Advanced Services. Naming Conventions This section presents the naming conventions for customer disaster recovery site physical devices (Table 1), ACI logical constructs (Table 2), and access policies (Table 3). 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 4 of 46
Table 1. Naming Conventions for Physical Devices Building Floor Type and Manufacturer Device Type Sequence Number Host Name DRC 04 DR SP = Spine 01 DRC4DRSP01 DRC 04 DR SP = Spine 02 DRC4DRSP02 DRC 04 DR LF = Leaf 01 DRC4DRLF01 DRC 04 DR LF = Leaf 02 DRC4DRLF02 DRC 04 DR AP = Cisco Application Policy Infrastructure Controller (APIC) 01 DRC4DRAP01 DRC 04 DR AP = APIC 02 DRC4DRAP02 DRC 04 DR AP = APIC 03 DRC4DRAP03 DRC 04 DR RINT = Internet router 01 DRC4DRRINT01 DRC 04 DR RDCI = DCI router 01 DRC4DRRDCI01 DRC 04 DR F5 = F5 load balancer 01 DRC4DRF501 DRC 04 DR GTM = F5 Global Traffic Manager 01 DRC4DRGTM01 DRC 04 DR ASA = Cisco ASA firewall 01 DRC4DRASA01 Table 2. Naming Convention for Cisco ACI Logical Constructs (Tenant, Contexts, Bridge Domains, Endpoint Groups, etc.) Policy Construct Naming Format Naming Format Example Use Example Tenant <Name in CAPITAL letters identifying tenant function and transit-device details> INT_DMZ Tenant used for DMZ services Contexts <Name in CAPITAL letters with end-device type and traffic type and transit-device details> Transit_DMZ Context used for Internet transit traffic between Cisco ASA Instance <instance name> and Cisco ASR Bridge domain <Name in CAPITAL letters with end-device type and traffic type and transit-device details> FW_Internet_To_ASR_Int ernet Bridge domain used for VLAN traffic between the firewall and the Internet Application profile <Name in CAPITAL letters with end-device type and traffic type and transit-device details> Transit_ASR Application profile used to group endpoints; part of the traffic between the firewall and the Internet Endpoint group (EPG) <Name in CAPITAL letters with end-device type and traffic type and transit-device details> FW_Internet_To_ASR_Int ernet EPG used to connect to firewall interface <x/y> carrying Internet traffic 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 5 of 46
Table 3. Naming Conventions for Cisco ACI Access Policies Policy Construct Naming Format Naming Format Example Use Example VLAN pool <Maximum 10-letter PLATFORM NAME><Device or pair number if any> UCS-SHAREPOINT <Platform name> Physical domain <Maximum 10-letter PLATFORM NAME><Device or pair number if any> UCS-SHAREPOINT <Platform name> Attached Entity Profile <Maximum 10 letter PLATFORM NAME><Device or pair number if any> UCS-SHAREPOINT <Platform name> Link level <1/10 Gbps> 1G - Lightweight Access Control Protocol (LACP) <LACP-ACTIVE/PASSIVE> LACP-ACTIVE - Interface policy group <3- or 4-letter PLATFORM NAME>- <Interface type = vpc<number>/po<number>/ind<1/10 Gbps>-<Pair number>-<end device number> UCS-SHAREPOINT -vpc1-01 <Platform name> Interface profile <3- or 4-letter PLATFORM NAME>- <Interface type = vpc<number>/po<number>/ind<1/10 Gbps>-<Pair number>-<end device number> UCS-SHAREPOINT -vpc1-01 <Platform name> Switch profile <3- or 4-letter PLATFORM NAME>- <Interface type = vpc<number>/po<number>/ind<1/10 Gbps>-<Pair number>-<end device number> UCS-SHAREPOINT -vpc1-01 <Platform name> Port selector identity <3- or 4-letter PLATFORM NAME>- <Interface type = vpc<number>/po<number>/ind<1/10 Gbps>-<Pair number>-<end device number> UCS-SHAREPOINT -vpc1-01 <Platform name> IP Addresses ACI fabric requires a block of addresses to enable ACI infrastructure to initialize the fabric and assign IP addresses to different types of nodes. Here are some examples: Infrastructure loopback IP addresses (spine and leaf) Node-level communication with the APIC Tunnel endpoint (TEP) termination on the top-of-rack (ToR) switch Peering address for internal Border Gateway Protocol (ibgp), diagnostics, etc. Leaf loopback virtual PortChannel (vpc) TEP IP addresses Address of the logical virtual TEP (VTEP) shared between vpc peers 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 6 of 46
Leaf loopback fabric TEP IP addresses VTEP address used to communicate to downstream virtual switch (vswitch) VTEPs Identical across all leaf switches to allow downstream mobility of VTEP devices Spine loopback proxy anycast IP addresses Common anycast IP address shared by each fabric proxy redundancy group IPv4, IPv6, and Layer 2 MAC address proxies The fabric administrator specifies the infrastructure addressing scope. By default, the APIC assigns IP addresses from the 10.0.0.0/16 block (this setting is user configurable). For the disaster recovery project discussed here, new unique address spaces are used, as shown in Tables 4, 5, and 6. ACI provides both in-band and out-of-band capabilities to manage the ACI infrastructure. Table 4 shows the address spaces reserved for the ACI network. Table 4. IP Addresses for Cisco ACI Network Description TEP and infrastructure (infra) address In-band management (mgmt) address Out-of-band management address Router ID IP Range 10.xxx.11.0/23 10.xxx.13.0/24 10.xxx.14.0/24 10.xxx.0.128/29 Table 5 shows the address spaces used for fabric interconnectivity. Table 5. IP Addresses for Fabric Interconnectivity Description Point-to-point, fabric, and Cisco ASR connectivity Point-to-point, fabric, and Cisco ASA connectivity IP Range 10.xxx.1.128/27 10.xxx.1.160/27 Table 6 shows the address spaces used for management services. Table 6. IP Addresses for Management Services Description SNMP TACACS Syslog NTP Secure Copy (SCP) and FTP DNS IP Range yy.xxx.xxx.xxxx 10.xxx.32.21 10.51.200.135 10.xxx.32.22 yy.xxx.xxx.xxxx yy.xxx.xxx.xxz 10.0.0.1 10.0.0.2 yy.xxx.xxx.xxxx 10.xxx.35.35 10.xxx.35.36 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 7 of 46
VLANs Table 7 lists the VLANs allocated for disaster recovery in an ACI implementation. Table 7. VLANs in Fabric VLAN Description VLAN Range Server farm VLAN (static) 1-1500 Server farm VLAN (dynamic) 1501-3000 Layer 2 in-band management 3001-3010 Internet routers VLAN 3011-3020 WAN/DCI routers VLAN 3021-3030 Layer 2 connectivity from external network 3031-3050 Testing VLAN 3500-3600 Connectivity to the F5 load balancer 3601-3700 Connectivity to the Cisco ASA 3701-3800 Fabric infrastructure VLAN 4093 Design Requirements In Cisco ACI terminology, a tenant is the highest-level logical container in which objects and policies of a given group or organization can reside. A tenant itself isn t mapped directly to existing (legacy) network constructs such as virtual routing and forwarding (VRF) instances, virtual private networks (VPNs), interfaces, and security policies. Tenants provide the following high-level properties: Isolation: Tenants can be totally isolated from one another or share certain resources. RBAC: Users can be created that have visibility and specific privileges over only one or more specific tenants (this is called a domain). Inheritance: Objects inside a tenant inherit that tenant s top-level policies. In cases in which the departments within an organization do not all adhere to the same approach to network design, a single-tenant model can quickly become tedious to maintain at the design and operation levels. When a given department or service function is confined to its corresponding tenant in ACI, troubleshooting and fault isolation can be performed more efficiently. A tenant can choose to export (or import) objects to (or from) other tenants, thereby relaxing the strict isolation initially provided by the multitenant model. Implementation of more than one tenant provides a clean model for controlling inter-tenant communications when desirable. The use of multiple tenants lets you easily totally isolate one environment from another. Although it is true that inside a tenant, inter-epg communication is never allowed explicitly, the one environment = one tenant model lets you isolate or permit communication in a highly specific way. The ACI fabric design proposed in the context of a customer disaster recovery deployment addresses the needs of multiple internal network blocks. Those blocks, summarized in Table 8, are: Internet DMZ tenant at the disaster recovery perimeter Intranet server farm tenant 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 8 of 46
The Internet DMZ tenant provides services from the external world. The Cisco ASR Internet routers and WAN DCI routers are part of this tenant, as is the Internet context of the firewall. The server farm tenant addresses the internal applications. Each of these service functions has its specific applications and policies as well as traffic patterns. Mixing all of these within one overarching tenant results in operational complexity and little administrative flexibility. It also reduces the ease with which isolation and fault containment can be provided between departments. RBAC applies roles to domains. Domains can group from 1 to N tenants. With this model, you can easily create highly specific access control policies. Table 8. Tenant Description and Naming Description Internet DMZ tenant at disaster recovery perimeter Intranet server farm Tenant INT_DMZ SF Figure 2. High-Level View of Tenants and Required Services 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 9 of 46
Tenant DMZ The tenant DMZ provides Internet services and a pool of shared network and application services to the customer s users. The customer has an Internet link from two different service providers at the primary data center. The BGP routing protocol is used to provide dynamic failover between the two Internet service providers (ISPs). If the primary data center site is down, the Internet link at the disaster recovery site provides backup Internet services for campus and external users. The tenant DMZ provides the following services: Internet services for the primary data center in the event that the primary data center Internet service is down Internet services for all the services at the disaster recovery site Tenant Server Farm The tenant server farm provides mission-critical application services to the customer s users. The customer s secondary POP site within the campus functions as an alternative site for some core internal services that are required within the campus: for example, DHCP, DNS, IP telephony, and read-only domain controllers. The tenant server farm provides the following services: Microsoft SharePoint Microsoft Exchange 2013 Oracle ERP and E-Business Suite (Exadata) Banner systems (Oracle Exadata) E-learning (Blackboard) File sharing Web portal and content Domain controller Traffic Flow This section discusses the business- and customer-initiated traffic flows destined for the customer s disaster recovery data center. The traffic flows are of two main types: Internet traffic DCI traffic Internet Traffic Flows The Internet traffic flows provide access from the DMZ servers and load-balancing servers to the Internet, and also the access from the Internet to these servers. In the customer environment, the servers can access the Internet through a secure channel for software patching. Internet DMZ server access to Internet (outbound); flow (a) in Figure 3: All DMZ servers have a gateway on the Cisco ASA Internet firewall context, which sends traffic to the perimeter Cisco ASR router to access the Internet. Internet traffic reaching the Internet DMZ server (inbound); flow (b) in Figure 3: F5 GTM determines the application availability site (primary data center or disaster recovery site) based on DNS entries. All 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 10 of 46
incoming traffic travels through the perimeter firewall context (INT), and if incoming traffic is allowed, perimeter firewall traffic will go to servers in the DMZ. Server access to the Internet (outbound traffic); flow (c) in Figure 3: The server gateway is defined in the server farm context on the Cisco ASA firewall, which sends traffic to the Cisco ASA Internet context through the fabric (the fabric acts as Layer 2 transport). After the traffic reaches the perimeter firewall of the Cisco ASA Internet context, depending on the destination IP address, the traffic is forwarded to the Internet ASR router to reach the Internet. All front-end servers are load balanced with F5, which works in one-arm mode. Internet traffic reaching the servers (inbound traffic); flow (d) in Figure 3: GTM determines the application availability site (primary data center or disaster recovery site) based on DNS entries. All incoming traffic travels through the Cisco ASA perimeter firewall context (Internet context). If incoming traffic is allowed, after passing the perimeter, it is routed through the leaf (in this case, the fabric acts as Layer 2 transport) to the internal server farm firewall context. The server farm context on the Cisco ASA firewall s route to reach the virtual IP address or network load-balancing (NLB) server s IP address depends on the destination IP address. If the request is for NLB servers, the traffic will be forwarded to the F5 Local Traffic Manager (LTM), which will forward the traffic to a real server. Figure 3. One-Arm Load-Balancing Internet Traffic Flow 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 11 of 46
DCI Traffic Flows The DCI traffic flows provide access from the DMZ servers and load-balancing servers to the primary data center and co-location site and also access from the ITS data center applications to these servers. DMZ server access to ITS data center (outbound); flow (a) in Figure 4: All DMZ servers have a gateway on the Cisco ASA Internet DMZ firewall context, which sends traffic to the DCI Cisco ASR routers to access the primary and secondary sites. ITS traffic reaching the Internet DMZ server (inbound); flow (b) in Figure 4: All incoming traffic from the ITS data center travels through the perimeter firewall context (INT_DMZ), and if incoming traffic is allowed, after passing the perimeter firewall, the traffic will go to the servers in the DMZ. Server farm server access to ITS site (outbound traffic); flow (c) in Figure 4: The tenant server farm server s gateway is defined in the server farm SF context on the Cisco ASA firewall. On the basis of the destination IP address, the traffic will be forwarded to the DCI ASR routers to reach the primary or secondary site. All front-end servers are load-balanced with F5, which works in one-arm mode. ITS traffic reaching the disaster recovery server farm servers (inbound traffic); flow (d) in Figure 4: All incoming traffic from ITS travels directly to the server farm context on the Cisco ASA firewall. Its route to reach the virtual IP address or NLB server s IP address depends on the destination IP address. If the request is for load-balancing servers, the traffic is forwarded to the F5 LTM, and then the traffic is forwarded to a real server. Figure 4. DCI Traffic Flow for One-Arm Load Balancing 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 12 of 46
Internet Access through Disaster Recovery If the primary data center site is down in the primary data center, the Internet link at the disaster recovery site will be used for backup Internet service. Figure 5 shows the flow. Figure 5. Internet Access Through Disaster Recovery Disaster Recovery Topology and Service Flows Figure 6 shows the logical topology of both the ITS and disaster recovery data centers. Figure 6. Customer Data Center High-Level Topology Overview 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 13 of 46
Figure 6 is summarized here: Three main components of Cisco ACI Fixed-chassis spine (Cisco Nexus 9336PQ ACI Spine Switch) with 36 ports to connect with leaf switches Leaf (Cisco Nexus 9396PX Switch) with 48 x 1/10-Gbps fiber ports for host connectivity and 12 x 40- Gbps ports to connect with spines Cisco APIC appliances (Cisco UCS C220 M3 Rack Server) 40-Gbps BiDi optics between leaf and spine switches (OM4 cables) Two 1-Gbps connection between disaster recovery site and primary data center in ITS data center (through Cisco ASR routers) Two 100-Gbps connection used to connect each data center with Multiprotocol Label Switching (MPLS) cloud Hardware summary Two spine switches (Cisco Nexus 9336PQ) Two leaf switches (Cisco Nexus 9396PX) Two Cisco ASR 1000 Series routers used for WAN DCI connectivity One Cisco ASR 1000 Series router used for Internet connectivity One Cisco ASA firewall (Cisco ASA 5585-X) One F5 BIG-ADC10200V Two F5 GTM F5-BIG-DNS-2000S Leaf and Spine Connectivity Figure 7. Customer Disaster Recovery Physical Connectivity Overview 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 14 of 46
The main concepts of leaf and spine connectivity are summarized here: Endpoint devices (servers, firewalls, load balancers, etc.) and Cisco APIC connected to leaf nodes only Spine switches connected to leaf switches only Leaf switches rely on the spine s knowledge of all endpoints Depending on bandwidth requirements, all leaf nodes connect to both spines with one 40-Gbps uplink. A maximum of 12 uplinks can be used from the leaf switches to the spine switches. Server Connectivity with Leaf Switches The servers connect to leaf nodes only. Each leaf has a total 48 x 1/10-Gbps fiber ports available for endpoints, supporting individual port connectivity, active and standby mode connectivity, and vpc or Port Channel connectivity or both. Leaf switches support vpc. No vpc peer link is required. Peer communication occurs over the fabric. Path recovery also occurs over the fabric. Within the fabric, the vpc interfaces use an anycast VTEP that is active on both vpc peers. The servers can connect to leaf switches using individual interfaces, PortChannels, active and standby modes, and vpc. Figure 8. Server Connectivity Options with Leaf Switches 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 15 of 46
Layer 4 Through 7 Device Connectivity to Leaf Switches As mentioned earlier, the disaster recovery project uses a total of two leaf switches. The same leaf switches are used to connect Layer 4 through 7 devices: Cisco ASA firewall F5 load balancer The customer uses the F5 BIG-IP load balancer and Cisco ASA firewall services devices, enabled for Cisco ACI and configured and managed by Cisco APIC. The ACI fabric supports traditional service devices with features such as flooding bridge domain (Figure 9). Figure 9. F5 LTM and GTM and Cisco ASA Firewall Connectivity to Leaf Switches 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 16 of 46
Cisco ASR Router WAN Connectivity Figure 10 shows the connectivity between the Cisco ASR routers and the WAN. Figure 10. Cisco ASR and Leaf Switch Connectivity Cisco APIC Connectivity Three Cisco APIC devices are installed in the disaster recovery site using two 10-Gbps interfaces in active-standby mode and standard Linux interface bonding (Figure 11). This configuration is provided by default, and no additional configuration is required. The APIC cluster is a fully redundant multinode cluster, and upgrades and downgrades are performed through the cluster redundancy and not through In-Service Software Upgrade (ISSU) on individual nodes. (This solution uses a transactional redundancy model, and as in Hadoop and other distributed systems, it is designed to operate with node state changes. The addition of new nodes and the upgrading and removal of existing nodes are simple forms of cluster scale up and scale down.) 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 17 of 46
Figure 11. Cisco APIC Connectivity External Networking The ACI fabric uses the concepts of inside and outside networks. Anything designated as within, or inside, the fabric is subject to the forwarding rules and policy enforcement. However, external, or outside, entities almost always need to be connected to the fabric (WANs, existing networks, etc.), so a way is needed to designate these external entities as not being part of the fabric. To achieve this, the concept of external, or outside, networks is used. External networks can be at Layer 2 or Layer 3. In this scenario, the fabric is acting as Layer 2 between the firewall and Cisco ASR, so neither Layer 3 nor peering is required. External Layer 2 Networks External (outside) bridged and Layer 2 networks are required whenever the Layer 2 domain needs to be extended beyond the ACI fabric: for example, when a Layer 2 interconnect is required between data center sites. When a Layer 2 external network is configured, essentially a mapping is created between a fabric bridge domain and a VLAN external to the fabric. In addition, ports on one or more leaf switches are designated as border ports. These interfaces connect to the external device. The customer wants to extend a number of Layer 2 segments across data center sites. To support this requirement, the Cisco ASR 1000 Series routers use Cisco Overlay Transport Virtualization (OTV) technology to transport VLANs between the disaster recovery site and the primary data center. 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 18 of 46
To support Layer 2 extension from the ACI fabric, a Layer 2 outside construct is used to extend bridge domains beyond the fabric. A Layer 2 outside construct works by extending a particular bridge domain using a specified VLAN. Figure 12 shows how the Layer 2 outside construct is used to extend bridge domains outside the fabric. Figure 12. Layer 2 Extension from the Cisco ACI Fabric In Figure 12, the fabric is Layer 2 connected to the DCI ASR 1001 Routers for Layer 2 extension between the ITS data center and the disaster recovery site. The Cisco ASR 1001 is connected to a leaf port, which is configured as a border leaf port during the creation of the Layer 2 outside construct. Note the following points about border ports: Border ports have Bridge Protocol Data Unit (BPDU) guard disabled (regular fabric edge ports have this feature enabled). Spanning-tree BPDUs are forwarded between border ports within the same bridge domain. When a bridge domain is extended using a Layer 2 outside construct, the bridge domain must operate in regular mode: in other words, Address Resolution Protocol (ARP) optimization should be disabled, and unknown unicast flooding should be enabled. Depending on the specific bridge domain being extended, these settings may already be configured (for example, bridge domain settings are different depending on where the default gateway resides). Layer 2 External constructs are configured from the APIC using the External Bridged Networks configuration option under Networking. Figure 13 shows how this item is configured in the APIC GUI. 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 19 of 46
Figure 13. Layer 2 Outside Configuration Note that the Layer 2 outside construct must be mapped to a bridge domain. A VLAN must also be specified for the external encapsulation. Finally, the physical node and interface to which the external device (in this case, the Cisco ASR 1000 Series router) connects must be specified. Service Architecture Design The customer deploys most applications in a three-tiered architecture (web, application, and database tiers, or similar layers). Note the following points about three-tiered applications: The servers in a three-tiered architecture use one VLAN per application tier. The server farm and DMZ tenants use Cisco ASA firewalls for security enforcement. A standalone F5 load-balancer appliance is used for load balancing. Additional VRF instances would map to additional firewall contexts. Additional firewall contexts would use shared EPG and bridge domains for connection to the Cisco ASR and intercontext communication. There is one bridge domain per application tier, contained within a VRF instance, but a gateway is not configured on the fabric. The firewall is the default gateway for all hosts (except for the Microsoft Exchange application). Therefore, all east-west and north-south traffic passes through the firewall. The F5 load balancer is used in proxy Source Network Address Translation (SNAT) mode for most missioncritical services. Mainly the web and application tiers use the load-balancer service. For Microsoft Exchange application F5 BIG-IP LTM is deployed in Layer 3 inline (two-arm) mode, where the source IP address of the client is visible to server (direct server return). Firewalls and F5 BIG-IP run static routing with distribution switches. Distribution switches run dynamic routing protocol with WAN edge and provider edge routers. One load-balancing context is created per application group. 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 20 of 46
For the solution discussed here, application-tier bridge domains are configured as follows: ARP flooding: Enabled. Within a bridge domain, servers must use ARP for the default gateway (the firewall). If the firewall is silent (that is, if it does not send out Generic ARP (GARP) or similar packets), the ACI fabric will not learn the MAC address information for the firewall. In this situation, ARP requests from the host to the firewall will fail. To avoid this scenario, regular ARP flooding is enabled in the application-tier bridge domains. In other words, ACI-optimized ARP handling is not enabled for these bridge domains. Unicast routing: Disabled. If unicast routing is enabled within a bridge domain, learning occurs for a host s IP and MAC addresses. In situations in which the fabric is not participating in routing (such as when a firewall acts as the gateway), IP addresses do not need to be learned. Therefore, this option is disabled for each of the application-tier bridge domains. Layer 2 unknown unicast: Flood. Hardware proxy handling of unknown unicast traffic is the default option. For topology 1, regular Layer 2 handling is enabled (flooding). Gateway and subnet addresses: Not configured. In this topology, all Layer 3 gateway services are running on the firewall, not the ACI fabric. Therefore, a subnet or gateway should not be configured in the application-tier bridge domains. For the customer s disaster recovery design, several topologies are supported for three-tiered applications. These topologies provide different options for integration of firewall and load-balancing services, as follows: Scenario 1 Firewall is the gateway for the servers. Load balancer service is not required. Scenario 2 Firewall is the gateway for the servers. Load balancer is in one-armed mode. Scenario 3 Load balancer acts as default gateway for hosts (load balancer operating in inline mode). Firewall is used in Layer 3 routed mode. Note that a combination of these topologies can be used for flexibility: for example, if one application group (web, application, and database) needs to use the firewall as the default gateway, and a second application group needs to use the load balancer as the gateway. The customer indicated that all application tiers will use the firewall as the default gateway except Microsoft Exchange, which uses the load balancer as the gateway. Firewall as the Gateway (NLB) In scenario 1, the firewall is used as the default gateway for all hosts and servers on a given VLAN. This means that all communication in and out of that VLAN is secured by the firewall. 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 21 of 46
Figures 14 through 17 show how this topology works. Figure 14. NLB DMZ Servers: Firewall as Default Gateway (Internet Traffic) Figure 15. NLB DMZ Servers: Firewall as Default Gateway (DCI Traffic) 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 22 of 46
Figure 16. NLB Server Farm Servers: Firewall as Default Gateway (Internet Traffic) The following points relate to the topology in Figure 16: The Cisco ASA firewall contexts operate in routed mode. An EPG is created on the ACI fabric for each application tier. A bridge domain is created on the ACI fabric for each application tier. A separate bridge domain is created to handle dynamic routing between the Cisco ASA Internet context and the Cisco ASR 1001 Routers. A separate bridge domain is created to handle routing (static or Open Shortest Path First [OSPF]) between the Cisco ASA Internet context and the server farm context. The fabric does not actively participate in routing, but instead acts as the Layer 2 transport. 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 23 of 46
Figure 17. NLB Server Farm Servers: Firewall as Default Gateway (DCI Traffic) The following points relate to the topology shown in Figure 17: The Cisco ASA firewall contexts operate in routed mode. An EPG is created on the ACI fabric for each application tier. A bridge domain is created on the ACI fabric for each application tier. A separate bridge domain is created to handle dynamic routing between the Cisco ASA Internet context and the Cisco ASR 1001 Routers. The fabric does not actively participate in routing, but instead acts as the Layer 2 transport. As mentioned previously, an EPG is created for each application tier: web, application, and database. By keeping the firewall interfaces in the same EPG as the hosts, you do not need to configure contracts between the host and the firewall (endpoints within the same EPG can communicate freely). For communication between the firewalls and the Cisco ASR 1001 Routers, a similar EPG arrangement is used. In this topology, the firewall and the Cisco ASR router are treated as regular hosts by the fabric; therefore, both the firewalls and the Cisco ASR 1001 Routers can reside in a dedicated INT_ASR EPG. On the basis of the current understanding of the applications to be deployed in the disaster recovery site, one context will be configured per tenant. In addition to the regular application tier bridge domains, a separate bridge domain is required for dynamic routing between the firewalls and the Cisco ASR 1001 Routers. In this topology, the dynamic-routing bridge domain should be configured with the same settings as the application-tier bridge domain. 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 24 of 46
Load Balancer in One-Armed Mode There are many ways to insert the F5 load balancer into the traffic path in the ACI fabric. The actual traffic flow will depend on the service being load balanced and the configuration of the core components including the ACI, F5 load balancer, firewalls, and connecting infrastructure. According to customer application requirements, the gateway to all the traffic resides on the firewall, and F5 performs SNAT. The F5 BIG-IP LTM is connected to its gateway (firewall) using a single interface and VLAN only. It is not inline: that is, it does not sit in the traffic path, and only traffic that needs to be load balanced is directed to it. The traffic that the client initializes for the virtual server virtual IP address travels to the load balancer through service graph insertion. When the F5 load balancer operates in one-arm mode, it uses a pool of addresses, known as SNAT IP addresses. When traffic reaches the virtual IP address of the load balancer, the source IP address is replaced with an address from the SNAT pool. Return traffic from the real server is directed to the SNAT IP address; therefore, the load balancer sees all return traffic. In this topology, the firewall is used as the default gateway for all hosts and servers on a given VLAN. This means that all communication in and out of that VLAN is secured by the firewall. The load-sharing algorithm picks a physical server, and the load balancer forwards the traffic to the physical IP address of this server. In this topology, the F5 BIG-IP LTM load balancer operates in automated mode: that is, the device package is uploaded to the APIC, service graphs are used to redirect traffic to the load balancer, and configuration tasks are moved to the APIC. Figure 18 and 19 show how this topology works. Figure 18. Load Balancer in One-Arm Mode: Firewall as Gateway (Internet Traffic) 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 25 of 46
Figure 19. Load Balancer in One-Arm Mode: Firewall as Gateway (DCI Traffic) The following points relate to the topology in Figure 19: The firewalls operate in routed mode. The load balancer operates in one-arm mode. An EPG is created on the ACI fabric for each application tier. A bridge domain is created on the ACI fabric for each application tier. An EPG is created for the transit connection to F5 BIG-IP LTM. A bridge domain is created for the transit connection to F5 BIG-IP LTM. A separate bridge domain is created to handle dynamic routing between the firewalls and the Cisco ASR 1000 Series routers. Dynamic routing (OSPF) runs between the firewalls and Cisco ASR routers. This routing exchange occurs through the fabric (that is, the fabric does not actively participate in OSPF, but instead acts as the Layer 2 transport). No contract is necessary for host-to-firewall communication. The load-balancer configuration is provisioned using the service graph. A contract is created between the provider (server) and the consumer (client side) traffic, and the same contract binds the load-balancer graph function. By keeping the firewall interfaces in the same EPG as the hosts, contracts between the host and the firewall are not needed (endpoints within the same EPG can communicate freely). Because the load balancer is operating in one-arm mode, it is attached to the ACI fabric using a single arm. To accommodate this connection, a dedicated EPG is set up in which both the load-balancer interface and the firewall interface facing the load balancer reside. The load balancer and firewall can communicate freely without an ACI contract. 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 26 of 46
For communication between the firewalls and the Cisco ASR 1000 Series routers, a similar EPG arrangement is used. In this topology, the firewall and the Cisco ASR router are treated as regular hosts by the fabric; therefore, both the firewalls and the Cisco ASR 1000 Series routers can reside in a dedicated INT_ASR EPG. A dedicated bridge domain is configured for each application tier. This design was chosen (instead of a single bridge domain) to prevent broadcast traffic from unnecessarily traversing application tiers. In addition to the regular application bridge domains, a separate bridge domain is required for dynamic routing between the firewalls and the Cisco ASR 1000 Series routers. In this topology, the dynamic-routing bridge domain should be configured with the same settings as the application-tier bridge domains. A dedicated bridge domain is also required for the transit connection between the firewall and the load balancer. Dynamic routing is required between the firewalls and the Cisco ASR 1000 Series routers to exchange routing information to and from external networks. Because the ACI fabric is not responsible for any Layer 3 services in this topology, dynamic routing is performed directly between the firewalls and the routers. Therefore, the ACI is operating purely as a Layer 2 transport. Although dynamic routing is taking place between the firewalls and the Cisco ASR 1000 Series routers, the fabric does not play an active role in this exchange; therefore the EPG and bridge domain configuration to achieve this behavior is identical to the configuration for the application tiers. Firewalls and Cisco ASR 1000 Series routers can all reside in the same EPG, eliminating the need to configure contracts to allow communication. Traffic Flow This section describes a traffic flow from an external client destined for a web server and the associated return flow. Figure 20 shows the traffic flow from the client to the server. Figure 20. Traffic Flow: Client to Server (DCI Traffic) Note: In the example in Figure 20, the database, application, and web servers are using the firewall as the default gateway. The database and application servers are not load balanced in this scenario. 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 27 of 46
In Figure 20, the traffic enters the data center through the Cisco ASR 1000 Series router, where it is routed to the virtual IP address of the load balancer through the firewall. The firewall needs a route pointing to the virtual IP address through the load balancer interface (accessible through the bridge domain named LB_ONE-ARM). Upon receiving this traffic, the load balancer creates a new session with a source IP address from the IP address pool. This new flow is destined for the real server and is routed through the firewall. Finally, the firewall sends the traffic to the real server using the relevant application bridge domain and EPG (in this case, SRV1). The traffic flow is summarized here: A flow from outside is destined for the virtual IP address. The Internet ASR traffic is routed to the virtual IP address through the Cisco ASA DMZ context. The firewall DMZ context needs to route traffic to the virtual IP address. The firewall DMZ context has a route pointing to the virtual IP address through the firewall server farm context. The firewall server farm context has a route pointing to the virtual IP address of F5 BIG-IP LTM. Upon receiving the traffic, the load balancer creates a new session with a source IP address from the SNAT pool defined in the service graph parameters and changes the destination IP address on the packet to point to the back-end server. The new flow is destined now for the real servers based on the load-balancing method and is routed through the firewall as a gateway. The firewall server farm context routes the flow to the correct application bridge domain. Finally, the firewall sends the traffic to the real server using the relevant bridge domain and EPG (web EPG and web bridge domain in this case). The flow in Figure 21 is summarized here: Figure 21. Traffic Flow: Client to Server (DCI Traffic) 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 28 of 46
A flow from outside is destined for the virtual IP address. The DCI ASR route to the virtual IP address is through the Cisco ASA server farm context. The firewall server farm context has a route pointing to the virtual IP address of F5 BIG-IP LTM. Upon receiving the traffic, the load balancer creates a new session with a source IP address from the SNAT pool defined in the service graph parameters and changes the destination IP address on the packet to point to the back-end server. The new flow is destined now for the real servers based on the load-balancing method and is routed through the firewall as a gateway. The firewall server farm context routes the flow to the correct application bridge domain. Finally, the firewall sends the traffic to the real server using the relevant bridge domain and EPG (web EPG and web bridge domain in this case). Figures 22 and 23 show the return traffic flow. Figure 22. Traffic Flow: Server-to-Client Return Traffic (to Internet) 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 29 of 46
Figure 23. Traffic Flow: Server-to-Client Return Traffic (to DCI) For the return flow, the web server sends traffic to the IP address, which was derived from the address pool. This traffic is then routed through the firewall. The firewall routes the traffic to the load balancer s interface. Finally, the firewall sends the return traffic back to the original client through the firewall and Cisco ASR 1001 Router. 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 30 of 46
The return flow is summarized here: The real server responds to the load-balancer address through the default gateway (firewall). The firewall routes the traffic to the load balancer address through the load-balancer interface. The load balancer sends traffic back to the client through the firewall. The firewall routes traffic back to the client through the Cisco ASR. Load Balancer as the Gateway In this topology (Figure 24), the load balancer is used as the default gateway for all hosts and servers on a given VLAN. This approach means that all communication in and out of that VLAN also must traverse the firewall. The major difference in this topology is that the load balancer operates in inline mode. Figure 24. Load Balancer as Default Gateway (Inline Mode Internet Traffic) In the example in Figure 24, the web server uses the load balancer as the default gateway. The application and database servers use the firewall as the gateway and are not load balanced in this scenario. The following main points relate to this topology (Figure 25): The load balancer and firewalls operate in inline routed mode. An EPG is created on the ACI fabric for each application tier. A bridge domain is created on the ACI fabric for each application tier. A separate bridge domain is created to handle traffic between the load balancers and the firewalls. A separate bridge domain is created to handle dynamic routing between the firewalls and the Cisco ASR 1000 Series routers. An EPG is deployed for the external side of the load balancer (in this case named LB_INLINE). 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 31 of 46
A contract is deployed between the server EPG (for example, Web) and the load balancer external EPG (for example, LB_INLINE). This contract refers to a service graph, which redirects the incoming traffic to the load balancer. The load-balancer configuration is provisioned using the service graph. A contract is created between the provider (server) and the consumer (client side) traffic, and the same contract binds the load-balancer graph function. Figure 25. Load Balancer as Default Gateway (Inline Mode DCI Traffic) Figures 26 and 27 show the return traffic for the inline mode deployment. The server sends the return traffic pointing the client IP address to F5 BIG-IP LTM. F5 BIG-IP has a route pointing to the Cisco ASA server farm context firewall and to the client IP address. Depending on the client s location, the traffic is forwarded to the DCI ASR or Internet ASR. 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 32 of 46
Figure 26. Load Balancer as Default Gateway (Inline Internet Return Traffic) Figure 27. Load Balancer as Default Gateway (Inline DCI Return Traffic) 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 33 of 46
Services Integration Service Device Packages A device package is an archive of information containing the details required to manage an external service device (either from Cisco or a third party). A device package is presented as a compressed zip file containing the following: XML-based device specification Python script implementing the device script API The device specification contains XML code that specifies parameters such as version information, functions, and configuration options for the device. The device script handles the interface with the service device using its API (preferred) or the command-line interface (CLI). A device package can be uploaded to the APIC easily using the GUI. The L4-L7 Services tab contains a Packages submenu from which packages can be uploaded in zip file format as shown in Figure 28 Figure 28. Importing Device Packages Figure 29. Importing Device Packages from the Packages Menu 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 34 of 46
Automated Service Insertion ACI supports a significantly enhanced service insertion model compared to traditional VRF and VLAN stitchingbased approaches. By using the APIC as a centralized point of policy control, the traditional traffic steering challenges associated with services integration can be overcome. The first step in achieving automated service integration is importing the device package (described in the previous section). After a device package has been uploaded, the APIC is aware of the device features and functions, and configuration tasks can be moved from the device itself to the APIC. The second major part of the automated service insertion architecture is the service graph. A service graph is essentially a chain of service devices that traffic should pass through. A policy specifies that traffic should be redirected to a service graph, which may consist of firewalls, load balancers, or any other services (Figure 30). Figure 30. Logical Configuration After the service graph has been defined, it can be selected when defining a contract subject for an application policy. Figure 31 shows packet processing for the initial traffic requiring redirection to a service graph. Note that the initial traffic must be sent to the egress leaf node for redirection to the correct leaf switch on which the service resides; subsequent traffic will be redirected by the ingress leaf switch after learning occurs on that node. Figure 31. Services Redirection: Initial Packet Processing 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 35 of 46
Cisco ASA Integration with Cisco ACI The customer uses the Cisco ASA 5585-X firewall for Internet perimeter termination and for connection to the intranet server farm. The Cisco ASA firewall has the following logical and physical connectivity: Connectivity to the ACI fabric uses dual 10-Gbps fiber interfaces. One Cisco ASA 5586-X is connected to a pair of Cisco Nexus 9396PX leaf switches. A pair of 10 Gigabit Ethernet PortChannels connect to each of the Cisco Nexus 9396PX Switches. The Cisco ASA firewall peers with the Cisco ASR router using OSPF as the routing protocol. The Cisco ASA firewall operates in routed mode. The standalone deployment mode is used (no high availability). The Cisco ASA has two contexts defined: intranet server farm and Internet perimeter firewalls. Another 1-Gbps interface connects the Cisco ASA firewall connected to the Cisco Catalyst 3750 out-ofband switch for management connectivity. The Cisco ASA firewall and ACI integration will make up the out-of-band management connectivity. The Cisco APIC automates insertion of services (such as an Cisco ASA firewall) northbound between applications, also called EPGs. The APIC uses northbound APIs to configure the network and services. You use these APIs to create, delete, and modify a configuration using managed objects. When configuration is controlled through the APIC, you cannot change the configuration through the Cisco ASA CLI. This means that the CLI for any feature that you configure through the APIC are disabled on the Cisco ASA. However, you can use the CLI to configure management access to the Cisco ASA. Operational and status commands, such as troubleshooting commands and show commands, are also available through the CLI. When a service function is inserted in the service graph between applications, traffic from these applications is classified by the APIC and identified using a tag in the overlay network. Service functions use the tag to apply policies to the traffic. For Cisco ASA integration with the APIC, the service function forwards traffic using either routed firewall operation. F5 Integration with Cisco ACI With the Cisco ACI and F5 solution s fully programmable load-balancing services technology, customer can implement application-centric deployments with the Cisco ACI fabric, using contracts, filters, and service graphs to control traffic between application tiers. This model provides stateless load balancing for a three-tiered application in the data center with agility and full automation. Traffic can be redirected to F5 BIG-IP LTM load-balancing devices (both physical and virtual) using an appropriate device package integrated into Cisco APIC. The F5 load balancer has the following logical and physical connectivity considerations: One LTM (F5-Big-IP-10050) and one GTM (F5-Big-IP-2000) are used. Both F5 LTM and GTM support 1/10-Gbps connections (copper and fiber). However, the customer plans to use four 10-Gbps fiber connections for LTM and two 1-Gbps copper connections for GTM. This configuration may change in the final design. Both F5 LTM and GTM use Layer 2 LACP connectivity to both leaf switches. F5 LTM operates in mixed mode (one-arm anywhere mode) for all the servers; only for Microsoft Exchange 2013 does it operate in routed inline mode. The F5 LTM and Cisco ACI integration uses out-of-band management connectivity. 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 36 of 46
The Cisco APIC automates the insertion and provisioning of network services through the F5 BIG-IP platform: for example, SSL offload, server load balancing (SLB), and Microsoft SharePoint services. F5 BIG-IP LTM integrates with Cisco APIC through well-established and open APIs (Simple Object Access Protocol [SOAP] or Representation State Transfer [REST]). This integration automates network and service provisioning across the F5 services fabric, providing end-to-end telemetry and visibility of applications and tenants. Cisco APIC acts as a central point of configuration management and automation for Layer 4 through 7 services and tightly coordinates service delivery, serving as the controller for network automation. A service appliance (device) performs a service function defined in the service graph. One or more service appliances may be required to render the services required by a service graph. A single service device can perform one or more service functions. A maximum of two F5 BIG-IP LTM concrete device clusters in active-standby mode can be set up. Concrete devices have concrete interfaces connected to physical F5 BIG-IP interfaces, or to virtual network interface cards (vnics) if F5 BIG-IP VE is used. When a concrete device is added to a logical device cluster, the concrete interfaces are mapped to the logical interfaces (Figure 32). Figure 32. F5 Logical Device Cluster with Pair of Concrete Devices Virtual Machine Networking VMware vsphere Integration Cisco ACI can closely integrate with the server virtualization layer. In practice, this means that instantiating application policies through ACI results in the equivalent constructs at the virtualization layer (that is, port groups) being created automatically and mapped to the ACI policy. Integration with the server virtualization layer is defined through the creation of a policy construct known as a virtual machine management (VMM) domain. A VMM domain is defined as a container for one or more VMM systems (for example, VMware vcenter) with similar network policy requirements. Each VMM domain represents a live migration domain: that is, virtual machines can be live-migrated within the VMM domain, but not beyond it (Figure 33). 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 37 of 46
Figure 33. VMM Domains Each VMM domain is associated with a name space. The name space defines a range of identifiers (normally VLANs or VXLANs) that are used to identify traffic within the domain and map that traffic to EPGs. If VLANs are used to identify the traffic, then each VMM domain can support up to 4000 EPGs. VMM Domain Configuration To implement a VMM domain, the ACI administrator must establish connectivity with vcenter through the GUI or API. This is accomplished through the VM Networking tab in the APIC GUI and the Policies submenu (Figure 34). Figure 34. Creating a VMM Domain 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 38 of 46
Note the selection of the VMware Distributed Switch in this step. After the connection to vcenter has been established, a new virtual distributed switch (vswitch) will be automatically created in vcenter, as shown in Figure 35. Figure 35. Creating a Distributed Switch Note that physical uplinks (vmnics) are not automatically added to the new vswitch; the server administrator must perform this step. Assuming a successful connection, the APIC will now reflect information about the virtualized infrastructure, such as virtual machine information and vmnics, as shown in Figure 36. Figure 36. VMM Domain Inventory Display After the integration with vcenter is complete, the fabric or tenant administrator creates EPGs, contracts, and application profiles as usual. Upon the creation of an EPG, the corresponding port group is created at the virtualization level. The server administrator then connects virtual machines to these port groups (Figure 37). 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 39 of 46
Figure 37. Port Group Creation on VMware vcenter The vcenter connectivity can be established using two different modes: in-band and out-of-band mode. For the customer s disaster recovery design, out-of-band mode is used for vcenter and other Layer 4 through 7 integration. Note: In the customer s environment, services such as syslog, NTP, and TACACS as well as the VMM domain, F5, and Cisco ASA use the out-of-band network for management connectivity. Management Network in Cisco ACI Two tenants are created by default within Cisco ACI for management purposes: Infra: Used for TEP-to-TEP (leaf to leaf) traffic within the fabric and for bootstrap protocols within the fabric. Mgmt: Used for management connectivity between the APICs and switch nodes, as well as for connectivity to other management systems (authentication, authorization, and accounting [AAA] servers; vcenter; etc.). The infra tenant is preconfigured for settings related to the fabric infrastructure, including the context and bridge domain used for the fabric VXLAN overlay. Only EPGs configured under the infra tenant are allowed to associate with the overlay bridge domain, and these EPGs are implicitly allowed to communicate with each other with no contracts needed or allowed. The infra tenant can also be used to extend the fabric infrastructure to outside systems that support overlay protocols such as VXLAN and NVGRE. For example, the infra tenant s default EPG can be used to configure a fabric infrastructure VLAN on leaf ports that are connected to hypervisors or switches supporting VXLAN encapsulation. 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 40 of 46
It is highly recommended that users do not modify the infra tenant. This tenant is not used for general management functions. The mgmt tenant is used for general in-band and out-of-band management of the ACI solution. The mgmt tenant has a default context and private network named inb (for in-band). A single bridge domain is also created by default, also named inb. ACI switch nodes have: In-band access to infra and mgmt tenants On-board management port connected to the out-of-band management network Console ports connected to the terminal server APIC nodes have: Two 10 Gigabit Ethernet links to leaf nodes for control-plane and data-plane traffic Two management connections (connected to the out-of-band network) One Cisco Integrated Management Controller (IMC) port for appliance management One console port connected to the terminal server APIC can be configured to send a variety of system data to Call Home, SNMP, or syslog destinations. When an event triggers the sending of a report, information from selected queries is included in the report. It is also possible to configure a query based on a class name or a distinguished name, or to further qualify the query based on subtrees and object properties. As mentioned previously, the customer s disaster recovery site reserved the 10.xxx.14.0/24 block for the out-ofband network. Table 9 shows the distribution of the out-of-band block. Table 9. Out-of-Band and In-Band IP Block Assignment Details Type of Device Out-of-Band IP Block In-Band IP Block APIC 10.xxx.14.11-20 10.xxx.13.11-20 IMC 10.xxx.14.21-30 - LTM 10.xxx.14.31-46 - ASA 10.xxx.14.47-62 - VMM 10.xxx.14.63-78 - Free 10.xxx.14.63-79-100 - Leaf 10.xxx.14.101-150 10.xxx.13.101-150 Spine 10.xxx.14.201-204 10.xxx.13.201-204 Out-of-Band Management Network The APIC and spine and leaf switches have dedicated out-of-band management ports and provide connectivity to access these devices (Figure 38). 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 41 of 46
Figure 38. Out-of-Band Management Network The out-of-band management endpoint group consists of switches (leaf and spine switches) and APICs that are part of the associated out-of-band management zone. Each node in the group is assigned an IP address from the address pool associated with the corresponding out-of-band management zone. The allocated IP address is then configured on the out-of-band management port of the corresponding node (Figure 39). Hosts that are part of regular endpoint groups cannot communicate with the nodes in the out-of-band management endpoint group. Any host that is part of a special group known as the instance profile can communicate with the nodes in an out-of-band management endpoint group using special out-of-band contracts. Regular contracts cannot be used with this endpoint group. Figure 39. Create Node Management Addresses 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 42 of 46
Note: All the management services, including NTP, syslog, and TACACs, use the out-of-band management network Simple Network Management Protocol The management information base (MIB) can be retrieved from switch nodes (leaf and spine), and SNMP traps can be sent from switch nodes. SNMP support (MIB and trap) is available on fabric switches only. This section shows the steps for retrieving the MIB from switch nodes. After enabling the Admin state and assigning community policies, assign the default SNMP policy to the policy group (Figure 40). Figure 40. SNMP MIB Policy Figure 41. SNMP Policy Push the SNMP configuration to all switch nodes by configuring the fabric policies for SNMP (Figure 41). 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 43 of 46
The SNMP trap receiver configuration is required to send SNMP traps from the APIC switch nodes. Trap policy is also pod-level policy; after it is created on the APIC, it will be pushed to all switch nodes. Configure the destination group for external SNMP and trap servers. Destination servers can be on either an inband or out-of-band network, depending on the option selected from the Management EPG drop-down menu (Figure 42). Figure 42. SNMP Trap Destination Note: The ACI APIC does not support SNMP. However, SNMP polling and trapping can be configured on the APIC for each individual spine and leaf switch. Syslog Syslog messages can be sent through both in-band and out-of-band networks (Figure 43). Figure 43. Syslog Fabric Policy 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 44 of 46
Fabric monitoring policy for syslog (as shown in Figure 43) is pushed to all leaf and spine switches. The remote destination group is configured for external syslog servers. Destination servers can be on an in-band or out-of-band network, depending on the option selected from the Management EPG drop-down menu (Figure 44). Figure 44. Syslog Remote Destination Network Time Protocol NTP synchronizes time keeping among a set of distributed time servers and clients. This synchronization allows events to be correlated when system logs are created and other time-specific events occur. In the customer s disaster recovery setup, APIC and switch nodes (leaf and spine) are synchronized by external NTP servers. Conclusion Cisco Application Centric Infrastructure brings a lot of value to customer even when they are trying to build a greenfield disaster recovery data center. Customers were not able to bring up the network infrastructure quickly as well as integrate services and virtualization to bring their critical applications and meet their business deadlines. For More Information http://www.cisco.com/go/aci 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 45 of 46
Printed in USA C11-733786-00 02/15 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 46 of 46