JUNIPER DATA CENTER EDGE CONNECTIVITY SOLUTIONS Michael Pergament, Data Center Consultant EMEA (JNCIE 2 )
AGENDA Reasons to focus on Data Center Interconnect MX as Data Center Interconnect Connectivity options towards DC Interconnect Providing L2 services across multiple DC locations with VPLS EVPN Overview Network support for Seamless VM Mobility 2 Copyright 2012 Juniper Networks, Inc. www.juniper.net
REASONS TO FOCUS ON DC-INTERCONNECT REASON #1 Data-Center Consolidation and Distribution Scalability High Availability Compliance Multi- Tenancy 3 Copyright 2012 Juniper Networks, Inc. www.juniper.net
REASONS TO FOCUS ON DC-INTERCONNECT REASON #2 Geo-Clustering, Disaster Recovery Scalable L2- Stretch Traffic Engineering & Resiliency Low Latency & Jitter Fault Containment No STP 4 Copyright 2012 Juniper Networks, Inc. www.juniper.net
REASONS TO FOCUS ON DC-INTERCONNECT REASON #3 L2 Stretch and VM Mobility DC disaster recovery Storage replication Hybrid cloud services a strong SP trend Server maintenance No disruption to VMs Resource Optimization 5 Copyright 2012 Juniper Networks, Inc. www.juniper.net
JUNIPER S VISION: COMMON DATA CENTER MODEL Customer B - IT DC Junos Space SRX GbE/10GbE SERVERS MX EX QFX μf Pooled storage iscsi / NAS FC STORAGE Public Cloud Users SMB Customer A - IT DC Junos Space SRX GbE/10GbE SERVERS MX EX QFX μf Pooled storage iscsi / NAS FC STORAGE VPN Hybrid Cloud Hybrid Cloud VPN Junos Space Production Data Center A Production Data Center B Junos Space NAT FW LB IPSec SRX QFX MX Inter Data Center Connectivity MX QFX SRX NAT FW LB IPSec Pooled Storage (NAS) GbE/10GbE SERVERS GbE/10GbE SERVERS Pooled Storage (NAS) 6 Copyright 2012 Juniper Networks, Inc. www.juniper.net
DATA CENTER REFERENCE ARCHITECTURE SRX HA Junos Space Orchestration IP L3VPN E-VPN MX: Data Center Connectivity Best of the breed platforms Single JUNOS Optimized L2, L3, L4-7 services delivery JUNOS and JUNOS Space SDK for 3 rd party integration SRX: L4-7 Services Complex Fabric EX / QFX: Any port to any port L2/L3 connectivity Pooled Ethernet Storage iscsi / NAS Servers, Virtual Machines FC Storage 7 Copyright 2012 Juniper Networks, Inc. www.juniper.net
MX PROVIDING DC LAN & WAN CONNECTIVITY Proven platform Over 24,000 chassis shipped Over $3B revenues Over 2,500 customers Inline services, stateful services MX supporting extensive set of LAN features WAN / CORE MX providing W A N L A N market leading WAN features High scale, multitenancy, resiliency, deployment flexibility EDGE COLLAPSED CORE 8 Copyright 2012 Juniper Networks, Inc. www.juniper.net
MX L2 INSTANCE OVERVIEW Bridge-Domain L2 Flooding Domain Typically one BD per cloud tenant Assigned to tenant WAN instance BD level VLAN tag preserved over WAN IRB.0 Bridge Domain Automatic port level VLAN manipulation BD VLAN-ID used to identify tenant 4K VLAN / L2 learning domain per BD Extensive VLAN manipulations: swap, pop, push, pop-swap, swap-push, swapswap IFL 0: VLAN-ID 100 L3VPN.0 Bridge-Domain.0 VLAN-ID: 1001 IFL 1: VLAN 200 IFL 2: VLAN 300 VPLS.0 IFL 3: VLAN 400 WAN Instances Stitched per tenant Multi-tenancy Interface tags locally significant IRB per tenant for L3 connectivity 9 Copyright 2012 Juniper Networks, Inc. www.juniper.net
VIRTUAL-SWITCH OVERVIEW Virtual-Switch = L2 VRF Each L2 Domain independent of other Each virtual-switch Multiple bridge-domains Separate xstp instance Separate 4K VLAN-ID space Separate VPLS instance STP VPLS STP VPLS Virtual Switch 0 BD 0: VLAN 100-200 BD 1: VLAN 300 Virtual Switch 1 BD 0: VLAN 101-150 BD 1: VLAN OT:400, IT: 1001 Combines LAN and WAN switching in single place BD and Virtual-Switch combined High scale 8K Virtual-Switch support IFL 1 IFL 2 L2 Domain #1 STP #1 4K VLANs IFL 5 IFL 10 IFL 11 IFL 12 L2 Domain #2 STP #2 4K VLANs 10 Copyright 2012 Juniper Networks, Inc. www.juniper.net
MX HAS STRONG LAN FEATURES TAG Single & double tags Extensive manipulation capabilities (push, pop, swap, multiple operations) Local / global significance, label standardization LAG Aggregated Ethernet interface support LAG 16 member 64 member MC-LAG IRB Scale Integrated Routing and Bridging Single Interface IFL level resolution High scale MAC table: 1M MAC address support, 1M ARPs 128K IFL support (64bit RE, Trio chipset), high scale L2 filters User controlled MAC learning limits Mirror Layer-2 port mirroring Next-hop group capable: L2 and L3 nexthops Snoop IGMP and PIM Snooping Snooping with MC-LAG Further flooding optimization by Proxy-ARP, DHCP-Relay 11 Copyright 2012 Juniper Networks, Inc. www.juniper.net
DCI TODAY, VIRTUAL PRIVATE LAN SERVICES (VPLS) MX Series VPLS over MPLS (or) IP Remote Data Center Remote Data Center MX Series SRX NAT FW LB IPSec QFX SRX5800 EX4200 EX/MX GbE/10GbE SERVERS GbE/10GbE SERVERS 12 Copyright 2012 Juniper Networks, Inc. www.juniper.net
VPLS EMULATES AN ETHERNET SWITCH Common Characteristics: Forwarding of Ethernet Frames Forwarding of Unicast frames with an unknown MAC address Replication of broadcast and multicast frames Loop prevention Dynamic Learning of MAC address DC1 CE P P PE CE DC3 PE CE DC2 CE PE DC4 13 Copyright 2012 Juniper Networks, Inc. www.juniper.net
VPLS CHARACTERISTICS Virtual Private LAN Service (VPLS) provides VLAN Extension over a shared IP/MPLS network. Full Mesh VLAN Separation Provisioning Multicast, Broadcast and Flooding Availability Any-to-Any connectivity regardless of physical path Separate VPLS instances per VLAN. Allows network-wide segmentation with very large scale New site Auto Discovery, RSVP Automatic Mesh Point-to-Multipoint LSPs capabilities Underlying MPLS offers ECMP, Fast Reroute 14 Copyright 2012 Juniper Networks, Inc. www.juniper.net
CONNECTIVITY OPTIONS TOWARDS MX VPLS Multi-Homing Multi-Chassis LAG Standard LAG to MX VC MX Series MX Series VC MX Series NAT FW LB IPSec SRX QFX NAT FW LB IPSec SRX MC-LAG QFX NAT FW LB IPSec SRX LAG QFX 15 Copyright 2012 Juniper Networks, Inc. www.juniper.net
OPTION 1: VPLS MULTI-HOMING QFabric has one uplink to each MX (can be LAG) MX will allow traffic forwarding for particular VLAN only on one uplink Loop prevention implemented in BGP on MXs Traffic Load-Balancing Different VLANs can have different active uplinks VRRP Master for VLAN X VLAN X Active on this link NAT FW LB IPSec SRX QFX VRRP Master for VLAN Y MX Series VLAN X Active on this link 16 Copyright 2012 Juniper Networks, Inc. www.juniper.net
OPTION 2: MULTI-CHASSIS LAG A/P WITH VPLS MC-LAG Multi-Chassis Link Aggregation Group Allows a LAG interface to be established across multiple MX chassis One logical interface across 2x chassis ICCP MX Series Provides node level redundancy, multi-homing support, and loop-free Layer2 network without running Spanning Tree Protocol (STP) Uses Inter-Chassis Control Protocol (ICCP) to exchange control information between two MC-LAG nodes NAT FW LB IPSec SRX MC-LAG 1 QFX Client device device terminates physical links in a link aggregation group (LAG) Client device not aware of MC-LAG 17 Copyright 2012 Juniper Networks, Inc. www.juniper.net
OPTION 3: MX VIRTUAL CHASSIS, A/A LAG Benefits of a Virtual Chassis Performance and Scale Scaling Ports & Services beyond one chassis VC MX Series Easy to Manage Single image, single config One management IP address NAT FW LB IPSec SRX LAG QFX Single Control Plane Single protocol peering Single RT/FT 18 Copyright 2012 Juniper Networks, Inc. www.juniper.net
MPLS CONNECTIVITY: EVPN MAC update DC-1 DC-2 VM moves from DC-2 to DC-1 MPLS Cloud MAC update Load balancing DC-3 Ethernet-VPN a new standards based protocol to inter-connect L2 domains over MPLS Enhancing industry standard VPLS further Multi-vendor / open initiative non-proprietary MPLS investment protection - builds easily over VPLS, L2/L3VPN environments Enhancements delivered by EVPN: Active multi-homed Extended control plane (MAC address) scaling Faster convergence from edge failures using local repair Flooding AND Control Plane learning Increased granularity on MAC address reach-ability distribution increased support for host mobility policy based decisions 19 Copyright 2012 Juniper Networks, Inc. www.juniper.net
EVPN TERMINOLOGY MES : MPLS Edge Switch CE: Customer Edge Interface ES: Ethernet Segment ESI: Ethernet Segment Identifier (e.g. LAG Identifier) EFI: EVPN Forwarding Instance An E-VPN comprises CEs that are connected to MESs (PEs) that comprise the edge of the MPLS infrastructure. A CE may be a host, a router or a switch. 20 Copyright 2012 Juniper Networks, Inc. www.juniper.net
EVPN REFERENCE MODEL Host -A1 ESI 1, VLAN1 VPN A ESI 1, VLAN1 Host A5 EFI-A ESI 2, VLAN2 MES 1 VPN B Host-B1 EFI-B EFI-A RR VPN A Host-A4 ESI 3, VLAN1 MES 4 EFI-A Ethernet MES 2 Switch-B3 VPN B ESI 4, VLAN2 EFI-B ESI 5, VLAN1 Host-A3 VPN A MES 3 MESes are connected by an IP/MPLS infrastructure Transport may be provided by MPLS P2P or P2MP LSPs for multicast Transport may be also be provided by IP/GRE Tunnels 21 Copyright 2012 Juniper Networks, Inc. www.juniper.net
EVPN LOCAL MAC ADDRESS LEARNING A MES must support local data plane learning using vanilla Ethernet learning procedures When a CE generates a data plane packet such as an ARP request MESes may learn the MAC addresses of hosts in the control plane using extensions to protocols such as LLDP that run between the MES and the hosts MESes may learn the MAC addresses of hosts in the management plane 22 Copyright 2012 Juniper Networks, Inc. www.juniper.net
EVPN REMOTE MAC ADDRESS LEARNING EVPN introduces the ability for an MES to advertise locally learned MAC addresses in BGP to other MESes, using principles borrowed from IP VPNs EVPN requires an MES to learn the MAC addresses of CEs connected to other MESes in the control plane using BGP Remote MAC addresses are not learned in the data plane 23 Copyright 2012 Juniper Networks, Inc. www.juniper.net
ETHERNET AUTO-DISCOVERY (A-D) ROUTES ESI DCB EVPN DCB DCS DCS ESI RD ETHERNET TAG Label DCB RR DCB auto discovery through advertisement of Ethernet A-D routes Includes Ethernet Segment Identifier (ESI) to allow multi-homing of DCS to DCB Auto-discovery of Ethernet Tags (VLANs) on Ethernet Segments 24 Copyright 2012 Juniper Networks, Inc. www.juniper.net
KNOWN UNICAST FORWARDING - ACTIVE/ACTIVE LOAD BALANCING (DCS-DCB) DCB ESI DCS EVPN DCB DCS DCB Redundant connection between DCS and DCB appears as a LAG to the DCS (no STP required) The DCS connection to the DCB(s) is referred by the Ethernet Segment Identifier (ESI) 25 Copyright 2012 Juniper Networks, Inc. www.juniper.net
UNKNOWN BUM TRAFFIC- DF/BDF ELECTION (CE-PE) DF DCS EVPN BDF Redundant connection between DCS and DCB appears as a LAG to the DCS (no STP required) A Designated Forwarder (DF) is elected (can be per VLAN) using Ethernet A-D Route Other DCB becomes a backup designated forwarder (BDF) Highest IP address of DCB wins by default Support of Split Horizon To ensure that a multicast, broadcast or unknown unicast packet that is sent on one link by a DCS (that is dual homed) isn t sent back by the other link 26 Copyright 2012 Juniper Networks, Inc. www.juniper.net
CHALLENGES VM MOBILITY INTRODUCES Challenges L2 & L3 address no longer pinned to a site, interface Fast convergence of network paths as VM moves Ingress and Egress traffic convergence, optimization Learning and information distribution control L2 & L3 interaction for best user experience draft-raggarwa-data-center-mobility 27 Copyright 2012 Juniper Networks, Inc. www.juniper.net
VM DEFAULT GATEWAY SOLUTION: FIRST MECHANISM Each VLAN/subnet uses anycast IP and MAC addresses for its default gateway Each VLAN/subnet would have its own IP and MAC anycast addresses All the VMs on a given VLAN/subnet are (auto) configured with this IP (anycast) address The anycast default gateway IP and MAC address for a given VLAN/subnet must be configured on each MES that this VLAN/subnet could span This ensures that a particular MES can always perform IP forwarding on packets sent by a VM to the anycast default gateway MAC address. 28 Copyright 2012 Juniper Networks, Inc. www.juniper.net
VM DEFAULT GATEWAY SOLUTION: SECOND MECHANISM Eliminates the need to configure the anycast addresses for a given VLAN/subnet on each MES that is part of that VLAN/subnet Each MES that acts as the default gateway for a given VLAN/subnet propagates in the E- VPN control plane an E-VPN route that carries MES's IP and MAC address BGP Default Gateway Community is used to indicate that this E-VPN route is for the default gateway For a given VLAN/subnet the distribution scope of this route is the set of MESes that are spanned by that VLAN/subnet Each MES that receives such E-VPN route: Creates MAC forwarding state that enables it to apply IP forwarding to the packets destined to the MAC address carried in the route Replies to ARP requests that it receives from locally connected VMs destined to the default gateway IP address of the advertising MES 29 Copyright 2012 Juniper Networks, Inc. www.juniper.net
MACVPN L3VPN INTERACTION FOR INGRESS TRAFFIC STEERING Ingress VPN Traffic traffic to VM1 steers to new site VPN Site-3 L3VPN update DC Backbone DC Site-1 MACVPN update Site-2 VM moves to site-1 VM 1 30 Copyright 2012 Juniper Networks, Inc. www.juniper.net
Avoiding Triangular Routing for Inter-DC scenario: NHRP Solution Next Hop Resolution Protocol (NHRP) RFC2332 (1998) IETF Proposed Standard Implemented by multiple vendors (including Juniper and Cisco) Original application - eliminating (extra) IP hops when routing over ATM/FR Non- Broadcast Multiple Access (NBMA) media NHRP messages could be carried directly over IP (protocol 54), or over IP/GRE (protocol 54) NHC (NHRP Client): originates NHRP Request, receives NHRP Replies, receives NHRP Purge Request NHS (NHRP Server): receives NHRP Request, originates NHRP Replies, originates NHRP Purge Request 31 Copyright 2012 Juniper Networks, Inc. www.juniper.net
NHRP EXAMPLE: STEP BY STEP VM VM VM-A 10.1.1.1 ToR1 ToR4 ToR2 Data Center 1 DCBR1/NHS1 ToR5 Data Center 2 DCBR1/NHS1 advertise into IP routing 10.1.1/24 route (subnet of VM-A) DCBR2/NHS2 1. Client Site BR/NHC receives from host -A a packet destined to VM-A (10.1.1.1) 2. Client Site BR/NHC originates an NHRP Request (the Request carries 10.1.1.1). The NHRP Request is routed (relying on plain IP routing) towards DCBR1/NHS1 (as DCBR1/NHS1 advertises a route for 10.1.1/24) 3. Meantime, the packet is forwarded towards VM-A using plain IP routing, first to DCBR1 (as DCBR1 advertises a route for 10.1.1/24), and then (using the E-VPN procedures) from DCBR1 to DCBR2, to ToR4, and ultimately to VM-A 4. DCBR1/NHS1 (relying on the information provided by E-VPN) determines that VM-A is in Data Center 2 - DCBR2/NHS2 is the authoritative NHS for VM-A. So, DCBR1/NHS1 (using E-VPN procedures) forwards the NHRP Request to DCBR2/NHS2 5. When DCBR2/NHS2 receives the NHRP Request, it send back to Client Site BR/NHC an NHRP Reply (as DCBR2/NHS2 is the authoritative NHS for VM-A). The Reply carries IP address of DCBR2/NHS2 6. Once Client Site BR/NHC receives the NHRP Reply, Client Site BR/NHC installs in its FIB host route to VM-A, This route requires encapsulation (the destination address in the outer header is the address of the originator of the NHRP Reply DCBR2/NHS2) 7. From that moment traffic to VM-A from Client Site BR/NHC goes (directly) to DCBR2/NHS2 32 Copyright 2012 Juniper Networks, Inc. www.juniper.net 4 2 7 3 Host-A 192.9.20.1 1 Client site Client Site BR/NHC 5 FIB: 6 Cloud Dest: 10.1.1.1 Next-Hop: DCBR2/NHS2 Encap: GRE
NHRP EXAMPLE: VM MOVES STEP-BY-STEP VM-A 10.1.1.1 ToR1 ToR4 ToR2 Data Center 1 ToR5 Data Center 2 DCBR1/NHS1 advertise into IP routing 10.1.1/24 route (subnet X) DCBR1/NHS1 5 2 DCBR3/NHS3 DCBR2/NHS2 VM-A moves from from ToR4 to ToR6 Client Site BR/NHC Data Center 3 0. The initial state is the end state reached on previous slide 1. VM-A moved from ToR4 to ToR6 (from Data Center 2 to Data Center 3) 2. DCBR2/NHS2 (relying on the information provided by E-VPN) determines that VM-A moved to another DC. Therefore, DCBR2/NHS2 send an NHRP Purge to Client Site BR/NHC 3. When Client Site BR/NHC receives the Purge message, it deletes from its FIB the route to 10.1.1.1 4. Same as steps 2-7 on previous slide 0 6 1 5 4 7 Client site 9 ToR6 Host-A 192.9.20.1 FIB: Dest: 10.1.1.1 3 Next-Hop: DCBR2/NHS2 Encap: GRE 8 Dest: 10.1.1.1 Next-Hop: DCBR3/NHS3 Encap: GRE VM-A 10.1.1.1 33 Copyright 2012 Juniper Networks, Inc. www.juniper.net
SUGGESTED READING 1) EVPN - draft-ietf-l2vpn-evpn 2) Seamless VM Mobility - draft-raggarwa-data-center-mobility 34 Copyright 2012 Juniper Networks, Inc. www.juniper.net
35 Copyright 2012 Juniper Networks, Inc. www.juniper.net
36 Copyright 2012 Juniper Networks, Inc. www.juniper.net
37 Copyright 2012 Juniper Networks, Inc. www.juniper.net