CERN Cloud Infrastructure Cloud Networking
Contents Physical datacenter topology Cloud Networking - Use cases - Current implementation (Nova network) - Migration to Neutron 7/16/2015 2
Physical network Topology restrictions - A machine has only network connectivity in a specific area of the CC Network DB Registration (LanDB) - DHCP (~2 minutes) - DNS (~10 minutes) 7/16/2015 Cloud Service 3
Cloud Network nova-network Provider Network VMx VMx VMx VMx VMx Project Router Project Router Private Network Private Network VMx VMx 7/16/2015 Cloud Service 4
Nova-network architecture Cloud Controller Public Network Compute Node(s) eth0 nova-api nova-network eth0 br0 nova-compute nova-scheduler nova-conductor VMx 7/16/2015 Cloud Service 5
Nova-network @ CERN (I) Hypervisor agents - KVM Linux Bridge - Hyper-V Virtual Switch 7/16/2015 Cloud Service 6
Nova-network @ CERN (II) LanDB + AD integration - SQL extra information + Code Patch - IP/MAC pairs registered beforehand - nova-api Rename LanDB device Update AD information Wait for DNS update - ec2 Choose one device and run Rename LanDB device 7/16/2015 Cloud Service 7
Migration to Neutron Nova-network will be deprecated Neutron as a replacement - Same end-user functionality - ML2 Plugin (Flat + Linux Bridge) - Integration with CERN services - On-demand IP registration 7/16/2015 Neutron @ CERN 8
Create instance with port 1. Machine created (nova) VMx VMx VMx VMx 2. Allocate a port (neutron) - Generate MAC address - Add port to DB - Lookup subnets on network - Allocate IP on subnet - Bind port on instance to host Subnet Subnet Subnet Provider Network 7/16/2015 Cloud Service 9
Create instance with port @ CERN 1. Create instance VMx VMx VMx VMx 2. Allocate a port on a network + host - Generate MAC address - Add port to DB - Lookup subnets on network host filter - Allocate IP on subnet (LanDB) - Bind port on instance to host IP Service IP Service IP Service Provider Network 7/16/2015 Cloud Service 10
Integration with LanDB Register port creation/deletion - Device + Interface registration on neutron Import network restrictions from LanDB - Subnet Clusters (aggregation of Subnets) - Host Restrictions 7/16/2015 Neutron @ CERN 11
Q & A 7/16/2015 Cloud Service 12
CERN Cloud Infrastructure Backup slides 7/16/2015 Neutron @ CERN 14
Why? Nova-network will be deprecated Neutron as a replacement - Same end-user functionality - ML2 Plugin (Flat + Linux Bridge) - Integration with CERN services - On-demand IP registration 7/16/2015 Neutron @ CERN 15
Create instance with port 1. Machine created (nova) VMx VMx VMx VMx 2. Allocate a port (neutron) - Generate MAC address - Add port to DB - Lookup subnets on network - Allocate IP on subnet - Bind port on instance to host Subnet Subnet Subnet Provider Network 7/16/2015 Cloud Service 16
Create instance with port @ CERN 1. Create instance VMx VMx VMx VMx 2. Allocate a port on a network + host - Generate MAC address - Add port to DB - Lookup subnets on network host filter - Allocate IP on subnet (LanDB) - Bind port on instance to host IP Service IP Service IP Service Provider Network 7/16/2015 Cloud Service 17
Integration with LanDB Register port creation/deletion - Device + Interface registration on neutron Import network restrictions from LanDB - Subnet Clusters (aggregation of Subnets) - Host Restrictions 7/16/2015 Neutron @ CERN 18
Improve LanDB integration Split device from interface registration Method to retrieve not available IPs - 1 Network, 1 Broadcast, 1 Gateway, 6 Reserved Avoid clustername on interface registration IPv4 IPv6 7/16/2015 Neutron @ CERN 19
Contents Overview of nova-network @ CERN Neutron basics and architecture Decisions Planning 7/16/2015 Cloud Service 20
Nova-network architecture Cloud Controller Public Network Compute Node(s) eth0 nova-api nova-network eth0 br0 nova-compute nova-scheduler nova-conductor VMx 7/16/2015 Cloud Service 21
Nova-network @ CERN (I) Hypervisor agents - KVM Linux Bridge - Hyper-V Virtual Switch 7/16/2015 Cloud Service 22
Nova-network @ CERN (II) LanDB + AD integration - SQL extra information + Code Patch - IP/MAC pairs registered beforehand - nova-api Rename LanDB device Update AD information Wait for DNS update - ec2 Choose one device and run Rename LanDB device 7/16/2015 Cloud Service 23
OpenStack Neutron Provides networking as a service (NaaS) Concepts - Networks - Subnets - Ports - Interfaces virtual network virtual port virtual interface Net1 10.0.0.0/24 VM1 VM1 7/16/2015 Cloud Service 24
Neutron Components API Server Queue Network node Agents - L2 - L3 - DHCP - Advanced 7/16/2015 Cloud Service 25
Neutron Architecture
Neutron Topology Provider Network VMx VMx VMx VMx VMx Project Router Project Router Private Network Private Network VMx VMx 7/16/2015 Cloud Service 27
Neutron Networks Private Networks - User created - Not shared - Routable/Isolated Provider Networks - Created by admins - Shared among tenants - Map to existing physical networks 7/16/2015 Cloud Service 28
Neutron Drivers (ML2) Core Plugin (ML2) Type Manager Mechanism Manager Type Driver Mechanism Driver flat GRE VLAN VXLAN Linux Bridge OvS Hyper-V Other 7/16/2015 Cloud Service 29
Decisions ML2 mechanism and type drivers - KVM Linux Bridge Open vswitch - HyperV Deployment architecture - Flat (provider networks) - Routed (private networks) - Mixed (both) 7/16/2015 Cloud Service 30
Create instance with port 1. Machine created (nova) VMx VMx VMx VMx 2. Allocate a port (neutron) - Generate MAC address - Add port to DB - Lookup subnets on network - Allocate IP on subnet - Bind port on instance to host Subnet Subnet Subnet Provider Network 7/16/2015 Cloud Service 31
Create instance with port @ CERN 1. Create instance VMx VMx VMx VMx 2. Allocate a port on a network + host - Generate MAC address skip?? - Add port to DB - Lookup subnets on network host filter - Allocate IP + MAC on subnet (LanDB) - Bind port on instance to host IP Service IP Service IP Service Provider Network 7/16/2015 Cloud Service 32
CERN Specifics AD integration - Where is the right point to do it? IP/MAC address handling - IP/MAC addresses registered beforehand - Scheduler awareness of host on creation Interference with APIs 7/16/2015 Cloud Service 33
Neutron Resources Core Networks Subnets Ports Extended Routers Floating IPs Firewalls Load Balancers 7/16/2015 Neutron @ CERN 34
Neutron Components Extensions API Core L3 FW Plugins Core Plugin L3 Plugin FW Plugin Service Plugins Agents L2 Agent (Hyper-V, LinuxBridge, ) L3 Agent 7/16/2015 Neutron @ CERN 35
Neutron Components @ CERN Extensions API Core CERN Extensions Plugins Core Plugin CERN Service Plugin Service Plugins Agents L2 Agent (LinuxBridge, Hyper-V) 7/16/2015 Neutron @ CERN 36
ML2 Plugin Core L2 Type Manager Mechanism Manager Extension Manager Type Drivers (Flat, GRE, VLAN, VXLAN) Mechanism Drivers (LinuxBridge, OVS, Hyper-V, Cisco, ) Extension Drivers 7/16/2015 Neutron @ CERN 37
ML2 Plugin @ CERN Core L2 Type Manager Mechanism Manager Extension Manager Flat Type Driver CERN Mechanism Drivers No Extension Drivers 7/16/2015 Neutron @ CERN 38
CERN Cloud Infrastructure Implementation 7/16/2015 Neutron @ CERN 39
Summary LanDB Custom Mechanism Drivers IP Service Clusters SubnetCluster API IP restrictions HostRestrictions API VM Metadata Security groups 7/16/2015 Neutron @ CERN 40
CERN LB Mechanism Driver Based on LinuxBridge driver. Handles LanDB registration / deletion. Queries Nova to resolve instance name and other LanDB metadata. If LanDB registration fails, port gets deleted, instance gets error state. If LanDB deletion fails, instance gets deleted anyway wasted IPs. Code is run by neutron-server. 7/16/2015 Neutron @ CERN 41
SubnetCluster API extension Defines new Cluster network resource Allows association of subnets with a clusters Neutron Subnet == CERN IP Service Provider Network Cluster X Cluster Y Subnet XX Subnet XY Subnet YX Subnet YY 7/16/2015 Neutron @ CERN 42
HostRestrictions API extension Exposes information about CERN network restrictions as part of the Neutron API. Hypervisor Allowed IP Service(s). Can choose an IP service based on different algorithms. 7/16/2015 Neutron @ CERN 43
Instance Metadata Problem: Neutron implementation depends on L3 or DHCP. Solution: Get rid of Neutron metadata. Add NAT rule to forward metadata requests on compute node. Chain PREROUTING (policy ACCEPT) DNAT tcp -- anywhere 169.254.169.254 tcp dpt:http to:128.142.134.168:8775 neutron-linuxbri-prerouting all -- anywhere anywhere Add FORWARD rule to allow packets from/to metadata server. Chain FORWARD (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere neutrondev.cern.ch neutron-filter-top all -- anywhere anywhere neutron-linuxbri-forward all -- anywhere anywhere Implemented as patch in IPTables Manager. Added metadata_host and metadata_port in neutron.conf 7/16/2015 Neutron @ CERN 44
Security Groups Supported with nova cells. IP Tables implementation. Default rules don t allow external ingress traffic. Cannot change default rules from API. Patch on default security group rules creation. 7/16/2015 Neutron @ CERN 45