Network Architecture Validated designs utilizing MikroTik in the Data Center

1-855-MIKROTIK Network Architecture Validated designs utilizing MikroTik in the Data Center P R E S E N T E D B Y: K E V I N M Y E R S, N E T W O R K A R C H I T E C T / M A N AG I N G PA R T NER I P A R C H I T E C H S M A N AG E D S E R V I C E S

Background Kevin Myers 16 + years in IT/Network Engineering Designed and implemented networks in Service Provider, Enterprise, Ecommerce and Government environments Areas of Design Focus: MikroTik integration with multi-vendor networks Design of BGP/MPLS/OSPF Service Provider Triple-Play networks Design of large enterprise Data Center networks Certifications MTCINE #1409INE006 Certified CCNP, CCNA, MCP, MTCRE, MTCTCE, MTCNA

IP ArchiTechs Managed Services Exhibitor at 2013 and 2014 MUM Please stop by our exhibitor booth and register to win an RC Helicopter! The first Carrier-Grade 24/7/365 MikroTik TAC (Technical Assistance Center) Three tiers of engineering support Monthly and per incident pricing available 1-855-MIKROTIK or support.iparchitechs.com AirMPLS - Private Nationwide 4G LTE MPLS backbone Partnership with Verizon Wireless - available anywhere in the Verizon service area Not Internet facing privately routed over our MPLS infrastructure Multiple Deployment options to carry public and private traffic including L2 adjacency Proactive Monitoring / Ticketing / Change Control / IPAM (IP Address Management) Carrier-Grade Network Engineering / Design in large (100,000+ nodes) environments

Introduction The MikroTik enabled Data Center Role within the Data Center Layer 3 Core Designs using CCRs with 10 Gbps interfaces Top of rack / End of Row L3 options for core connectivity External / Internal Firewall Internet reachability / protect critical internal networks with multiple layers (PCI) VPN Aggregation Multiple Vendors / Remote Mgmt Access MPLS P/PE router - Segregation of traffic within Data Center Role between Data Centers MPLS L2 VPN VLAN extension between Data Centers for VM mobility MPLS L3 VPN Segregate traffic as it routes between data centers VLAN Rewrites Used to deal with VLAN overlap between two or more sites Multiple Gateways for the same subnet at more than one site

Conventional Data Center

Multi-Million dollar DCs - Where does MikroTik fit in? MikroTik routers can be used in different areas of the Data Center and compete with mainstream vendors like Cisco, Juniper and HP within a specific set of design parameters. The goal of this presentation is to display the design elements required to build a Layer 3 infrastructure capable of up to 320 Gbps forwarding with off the shelf 10 Gig switches Why 320 Gbps? Relies on ECMP (server side) assuming 16 BGP paths 16 Paths is a conservative value for ECMP some go as high as 128 paths 16 CCRs with 20 Gbps LACP channels = 320 Gbps Design validation was tested with 2 CCRs which yielded 40 Gbps between servers

Multi-Million dollar DCs - Where does MikroTik fit in? Why? The business case for MikroTik in the Data Center CAPEX (Capital Expenditure) Savings Lower hardware replacement cost when a node fails Cisco Nexus Deployment for 320 Gbps $2,000,000 to $5,000,000 CAPEX MikroTik Deployment for 320 Gbps $50,000 to $100,000 CAPEX

The MikroTik enabled Data Center

Part 1 Desigining for High Availability 99.999% uptime Getting to five 9 s isn t easy can only have 5 minutes of unplanned downtime per year maintenance windows aren t included HA design elements Stackable switches enables multi-chassis LAG for CCRs and servers to provide survivability in the event of a failure of one of the switches LACP channeling/bonding at Layer 2 allows devices to aggregate speeds as well as prevent routing topology changes when a link fails Load Balancers Provide the ability to use multiple CCR chassis as a single firewall without breaking state. The LB has the ability to return traffic to the same source and track that relationship dynamically. BFD Bi-Directonal Forwarding Detection allows a network t o converge much more quickly than standard timers Multiple Internet BGP Peers When used along with BFD if the upstream carrier supports it, multiple tier 1 peerings provide a level of redundancy to ensure Internet traffic is uninterrupted

Part 2 Achieving 320 Gbps throughput Two mechanisms for achieving high throughput Method 1 - ECMP Equal Cost Multipath (ECMP) on the CCR RouterOS is capable of up to 128 gateways. Example below shows 16 gateways for one route Routes can be installed by either OSPF or Static. BGP can also be run on top of OSPF and utilize ECMP as well. ECMP Route with 16 Gateways

Part 2 Achieving 320 Gbps throughput ECMP Continued Using multiple gateway allows traffic egressing the router to balance along multiple paths but what about ingress? Server side ECMP is the key to scaling throughput when using independent routers. Support in multiple operating systems Microsoft and Linux both support ECMP in static routes along with OSPF and BGP

Achieving 320 Gbps throughput

Part 2 Achieving 320 Gbps throughput Utilizing BGP and OSPF at the server for dynamic ECMP Role of OSPF Converges quickly using adjusted standard timers (1 second hello 3 second dead) Converges even faster with Bidirectional Forwarding Detection (BFP) Provides Loopback reachability for BGP Is needed to implement ECMP dynamically through MikroTik routers until ECMP is added to BGP. Role of BGP Advertise data center subnets for servers, databases, web apps, etc, to the 16 router CCR core Can be utilized for traffic management

Achieving 320 Gbps throughput

Part 2 Achieving 320 Gbps throughput Method 2 Offset VRRP Gateways Each CCR is the VRRP master for one or more VLANS Requires setting priority for each VLAN/CCR Can be used in conjunction with ECMP when servers cannot be setup for ECMP CCR 1 Master for VLAN 100 Backup for VLAN 200 CCR2 Backup for VLAN 100 Master for VLAN 200

Part 2 Achieving 320 Gbps throughput LACP Channels for Routers and Servers LACP is an open standard for aggregating Layer 2 links 802.3ad Referred to as Channeling, Bonding, Teaming, Link Aggregation Can be trunked with multiple VLANs and multiple Layer 3 gateways Can be used with VRRP 20 Gigabit LACP channel on CCR-1036-8G-2S+

Part 2 Achieving 320 Gbps throughput LACP Channels for Routers and Servers Microsoft LACP example:

Part 2 Achieving 320 Gbps throughput LACP Channels for Routers and Servers Linux LACP example:

Part 2 Achieving 320 Gbps throughput Final Result 40 Gbps throughput Only 2 CCRs in the Core with 16 CCRs, the throughput will be roughly 320 Gbps

Part 3 Multiple Data Centers

Part 3 Multiple Data Centers Using MPLS in the Data Center CCRs can be used as MPLS edge routers to connect Data Centers. Used to segregate traffic within and between Data Centers L2VPN (VPLS ) Provides Layer 2 Connectivity and isolation L3VPN Provides Layer 3 connectivity and isolation VRF (Routing Marks) Used to separate customer routing tables so that more than one customer can use the same subnet without overlap

Part 3 Multiple Data Centers MPLS Customer Isolation at Layer 3

Part 3 Multiple Data Centers EoIP provides Layer 2 Connectivity and will allow MPLS to function across an encrypted internet link. Either EoIP or VPLS can be used for L2 connectivity.

Part 3 Multiple Data Centers VLAN Rewrites Problem: Data Center 1 uses Vlan 100 for web Servers on 10.1.1.0/24 Data Center 2 uses Vlan 100 for storage replication on 192.168.222.0/24 When extending the VLAN between Data Centers, one side must be rewritten CCRs can do this via bridging MikroTik routers with switch chips can use /switch to perform vlan rewrites

Part 3 Multiple Data Centers VLAN Rewrites change VLAN 100 traffic to VLAN 3100

Part 3 Multiple Data Centers VLAN Rewrites change VLAN 100 traffic to VLAN 3100 Create VLAN 100 and 3100 interface VLANs Create Bridge and add VLAN interface ports

Part 3 Multiple Data Centers Dual VRRP Gateways Problem when extending VLANs between Data Centers, If there is not a local gateway for hosts in that subnet, traffic must go all the way to the other Data Center via Layer 2 to hit the default gateway Solution: Dual VRRP gateways Data Center 1 VRRP GW 100.64.100.1/24 Data Center 2 VRRP GW 100.64.100.2/24 These are duplicate IPs How can this work? Because VRRP uses MAC addresses derived from the VRRP Group Number Hosts will always find the gateway in their own data center before going to the other Data Center

Part 3 Multiple Data Centers Dual VRRP Gateways Add input filter for VRRP on both edge routers IP Protocol 112 to prevent either gateway from becoming master for the other (bridges must be set to use IP Firewall) Add VRRP Gateway for 100.64.100.1 in both Data Centers

Part 3 Multiple Data Centers Dual VRRP Gateways

2014 Pittsburgh MUM RC Heli Giveaway!! 1-855-MIKROTIK 4 To Give Away!!! 17 RC Helicopters 24/7/365 MikroTik TAC Nationwide Private 4G LTE MPLS Proactive Network Monitoring Design / Engineering / Operations

Questions? The content of this presentation will be available at mum.iparchitechs.com Please come see us at the IP ArchiTechs booth in the Exhibitor Hall Email: kevin.myers@iparchitechs.com Office: (303) 590-9943 Web: Thank you for your time and enjoy the MUM!!