Managing the Management Switches. Freek Dijkstra, SURFsara 23 june 2016

Transcription

1 Managing the Management Switches Freek Dijkstra, SURFsara 23 june 2016

2 Outline 1. New datacentre, new management network 2. Bare Metal Switches 3. Why Dell? 4. Control plane & configuration management

3 New Datacenter Public tender in Won by Telecity (now sold to Equinix, sold to Digital Realty) Delivery in 8 days

4 Current management network Separate from production network, for security reasons. Mostly physically separated infrastructure Left-over switches from 1 to 10G upgrades in the past Unsupported switches: up to 10 years old (remember Nortel, Foundry, ) Hand-maintained (but few changes, and low SLA)

5 Desired management network Single CLI for all switches HUB A HUB B Automated configuration management 2Gbps LACP 1G-BASE-T 2Gbps LACP 1G-BASE-T Low(er) power switches Core (Fiber) mgmt switch 1 20Gbps Core (Fiber) mgmt switch 2 Replace hardware during datacentre migration Core (UTP) mgmt switch 3 20Gbps MLAG / VC 10GBASE-SR MM-fiber Core (UTP) mgmt switch 4 Tender for 78 switches February April Gbps LACP 10GBASE-SR MM-fiber 20Gbps LACP 10GBASE-SR MM-fiber Won by Dell EoR Console server NW-MGMT 1G-BASE-T ToR switch EoR switch 1G-BASE-T 1G-BASE-T 1G-BASE-T NW-MGMT EoR switch ToR switch EoR Console server ToR switch ToR switch CONSOLE RJ45 RS232 MGMT VLANS VLAN 800 SURFSARA-NW-MGMT SERIAL RS232 CONSOLE CONSOLE RJ45 RS232

6 Bare Metal ( White Label ) Switches All top of rack switches use the same network processors (ASIC) Decoupling of network operating system and hardware Similar to the rise of Linux and Windows with the IBM compatible PC Driven by cheap top-of-rack switches in big datacentres Bootloader: ONIE Hardware: Dell, Edge-core (Accton), Quanta, Penguin, Juniper, Software: Cumulus Linux, Big Switch, PicOS (Pica8) Freedom of Choice! Network Processors Broadcom Apollo2 Broadcom Firebolt3 Broadcom Helix4 Broadcom Tomahawk Broadcom Trident Broadcom Trident+ Broadcom Trident2 Broadcom Trident2+ Broadcom Triumph2 Mellanox Spectrum

7 Same Hardware

8 Why we choose Dell? Cheap - Significant price drop in last years - Not just this switch: also Z9100-ON and S6000-ON Good features for this switch - Redundant, swapable power supply and fans - Back-to-front airflow Dell groks Open Networking. - Choice of operating system: OS10, Cumulus Linux. - Yes, you can run JunOS on a Dell (or OS10 on a Juniper) Good support - Cumulus support through Dell (no pointing to each other of SW and HW vendor)

9 Previous Experience (Cumulus Linux on Dell) HPC Cloud (spring 2015) extension for Ceph storage environment GSP cloud (winter 2015) virtualisation of grid server park Management switches (summer 2016) LISA (summer 2016) move from 10G UTP to 40G fiber

10 Significant Change

11 Hardware and Software Integration Northbound interface API API Management plane Network Controller Control plane Data plane

12 Original Plan: Network Controller Considered different network controllers Not what we hoped for: No Northbound Authentication, Poor vendor abstraction NCS Network Controller OpenNaaS is advertised as a communitydriven Network-as-a-Service solution, which can integrate NaaS, SDN and NFV functionality. OpenDayLight is a open source SDN controller. It supports OpenFlow, NETCONF and other protocols as a southbound interface. Tail-f Network Control System (NCS) is a SDN platform which can support multiple network use-cases. Vendor / Community Open Source, supported by small, but active user community. Sponsored by EU FP7 projects Mantichore and GN3plus. Open source, supported by large vendorbacked community, including Brocade, Cisco, Dell, and HP. Proprietary. Acquired by Cisco in Will be marketed as "Cisco Network Service Orchestrator (NSO)" South-bound vendor support Only Juniper NETCONF support natively. Additional vendors require programming. Every vendor should be supported as long as it supports NETCONF, using YANG model abstraction. Many vendors supported, including Cisco, Juniper, and HP. Device implementation makes use of YANG model abstraction. North-bound GUI support Supported through opennaas-dc-gui. Native interface available, but not for NETCONF devices. Web GUI available. Granular authorization / delegation of permissions No extensive authorization currently available, but is under development. AAA module available, but highly under development. Granular authorization available. Ease of implementation Easy Difficult Medium

13 Final Implementation: Configuration Manager It runs Linux, so maintain like any other Linux cluster Generally more mature than network controllers Our choice: Ansible - Flexible, powerful - Support by most network vendors No integration with cloud and compute controllers yet.

14 First Experiences Datacentre-grade switches for a cheap price - Good airflow, dual power supply, swappable PSU and fans Ansible is great We love the freedom to chose what WE want Encountered a serious bug in the firmware (/sbin/shutdown bricked a S4048) - Who is responsible? - Support is important Courses Cumulus Linux (Remote, Cumulus), Ansible (Ton Kersten, AT-Computing) Need 3 uplinks - UTP for eth0, UTP for console, uplink - Need eth0: most ports are only enabled after installing the license Same license works for all Cumulus products - Makes automation (Ansible runbooks) easier

15 Freek Dijkstra