Advanced Software Engineering Lecture 8: Data Center by Prof. Harold Liu
Agenda Introduction Design and Construction Management and Maintenance Hot Topics Real time traffic analysis SDN Wireless Data Center Networking
Core Elements of a Data Center Applications Databases Database Management System (DBMS) and the physical and logical storage of data Servers/Operating Systems Networks Storage Arrays 3
An Example Client Local Area Network Storage Area Network Storage Array Application User Interface Server Database Consider an order processing system consisting of: Application for order entry. Database Management System (DBMS) to store customer and product information. Server/Operating System on which the Application and Database programs are run. Networks that provide Connectivity between Clients and the Application/Database Server Connectivity between the Server and the Storage system. Storage Array
An Example..Closer Look Client Local Area Network Server Storage Area Network Storage Array Application User Interface A customer order is entered via the Application User Interface on a client.
An Example..Closer Look Client Local Area Network Server Storage Area Network Storage Array A customer order is entered via the Application User Interface on a client The client accesses the Server over a Local Area Network.
An Example..Closer Look Client Local Area Network Server Storage Area Network Storage Array O/S and DBMS A DBMS uses the operating system on the server to read and write this data to the physical location on a disk.
An Example..Closer Look Client Local Area Network Server Storage Area Network Storage Array A DBMS uses the operating system on the server to read and write this data to the physical location on disk. A Network provides the communication link between the server and the storage array, and transports the read/write commands and data between the server and the storage array.
An Example..Closer Look Client Local Area Network Server Storage Area Network Storage Array Database A DBMS uses the operating system on the server to read and write this data to the physical location on disk. A Network provides the communication link between the client and the server, and transports the read/write commands and data between the server and the storage array. A storage array receives the read/write commands and data from the server and performs the necessary operations to store the data on the physical disks.
Google Data Center Google 位 于 俄 勒 冈 州 的 Dalles 数 据 中 心
at Lenoir, North Carolina http://www.google.com/about/datacenters/inside/s treetview/
Gallery http://www.google.cn/about/datacenters/gallery/#/all
IBM Data Center
Equinix Data Center
Elements of a Data Center The Site Command Center Cable Management Network Infrastructure Terminal Servers Environmental Controls Power
Standards 1
Standard 2
Structural Aspects
Structural Aspects a raised floor ceiling Basement data center locations near water are not a good idea. Their must be a pathway for equipment to be moved in & out of the data center. Make sure the floor where the data center is to be located is rated for the estimated load.
Power and Environment Dual street power supply with UPS When necessary, a diesel generator as a second backup Environmental Control Facilities Temperature Moisture dust
HVAC (Heating, Ventilation, and Air Conditioning)
Hot Aisle and Cold Aisle Layout
Networking 3 layer architecture Core Layer Switch/Router Layer 2 Switch Internal NIC Ethernet connection Cisco Core Layer Router DLink 48ports Gbps Layer-2 Switch
Security Physical Access Levels of Access Monitoring
Past to Future a single computer occupied the space of an entire Data Center. That same space can be occupied by thousands of servers today.
Let us Make it Online HW SW Rack Deployment and Testing Maintenance
Servers Tower Low cost, customized Space cost, not for parallel computing Blade save space, high computing ability suitable for large size data center Rack Server small, easy to manage need good cooling
Rack Unit (RU) Unit=44.45mm (height) Width=19 inch or 23 inch 1U, 2U, 4U (half rack) 1U 4U
Server Rack and Cluster
Data Center Topology Core Internet Layer-3 router Data Center Aggregation Layer-2/3 switch Access Top of Rack Switch Layer-2 switch Servers
Top of Rack (ToR) Switch
Data Center Costs James Hamilton published basic 2008 breakdown Servers: 45% CPU, memory, disk Infrastructure: 25% UPS, cooling, power distribution Power draw: 15% Electrical utility costs Network: 15% Switches, links, transit
Power Cost Approximate distribution of peak power usage by hardware subsystem in one of Google s datacenters (circa 2007).
Data Center Challenges Traffic load balance Support for VM migration Achieving bisection bandwidth Power savings / Cooling Network management (provisioning) Security (dealing with multiple tenants)
Non Virtualized Data Centers Too many servers for too little work High costs and infrastructure needs Maintenance Networking Floor space Cooling Power Disaster Recovery
What is Virtualization? Run multiple OSes and user applications on the same hardware e.g., run both Windows and Linux on the same laptop How is it different from dual boot? Both OSes run simultaneously OSes are completely isolated from each other
Reduce costs by consolidating services onto the fewest number of physical machines
Dynamic Data Center Virtualization helps us break the one service per server model Consolidate many services into a fewer number of machines when workload is low, reducing costs Conversely, as demand for a particular service increases, we can shift more virtual machines to run that service We can build a data center with fewer total resources, since resources are used as needed instead of being dedicated to single services
VM Workload Multiplexing Separate VM sizing VM multiplexing s 3 s 2 s 1 We expect s 3 < s 1 + s 2. Benefit of multiplexing! Multiplex VMs workload on same physical server
So, it is just like Java VM, right?
Two Types of Hypervisors (or VMM) Hypervisor is a software layer that allows several VMs to run on a physical machine The physical OS and hardware are called the Host VM OS and applications are called the Guest Type 1 (bare metal) Type 2 (hosted) Process Process Process Process VM1 VM2 Guest Guest VM1 VM2 Process Hypervisor Host Hypervisor Hardware OS Hardware Host VMware ESX, Microsoft Hyper V, Xen VMware Workstation, Microsoft Virtual PC, Sun VirtualBox, QEMU, KVM
Bare metal or Hosted? Bare metal Has complete control over hardware Doesn t have to fight an OS Hosted Avoid code duplication: need not code a process scheduler, memory management system the OS already does that Can run native processes alongside VMs Familiar environment how much CPU and memory does a VM take? Use top! How big is the virtual disk? ls l Easy management stop a VM? Sure, just kill it! A combination Mostly hosted, but some parts are inside the OS kernel for performance reasons, e.g., KVM
VM on Multi core CPUs Each core can be configured for multiple VMs A Quad core CPU could be configured as a 32 node multi computer Limiting factor is often memory. Each guest OS has its own requirements (512 MB?) 51
Installing a Virtual machine Base OS is Windows 7 First install Sun VirtualBox as hypervisor Then, Guest OS will be Ubuntu 12.04.1 52
Installing Sun/Oracle VirtualBox 53
Installing VirtualBox 54
Installing VirtualBox 55
Installing VirtualBox 56
Installing Ubuntu VM 57
Installing Ubuntu VM 58
Installing Ubuntu VM 59
Installing Ubuntu VM 60
61
Installing Ubuntu VM 62
63
Installing Ubuntu VM 64
Data Center Topology Core Internet Layer-3 router Data Center Aggregation Layer-2/3 switch Access Top of Rack Switch Layer-2 switch Servers
Oversubscription
Motivation Flow based traffic monitoring Volume of processed data is reduced Popular flow statistics tools Cisco NetFlow sflow
What is sflow? Accurate sampling is simple enough to be performed in hardware, at wire speed. Good accuracy under heavy loads Detailed complete packet header and switching/routing information for L2 L7 traffic flows. Scalable capable of monitoring networks at 10Gbps, 100Gbps and beyond. Thousands of devices can be monitored by a single sflow Collector. Low Cost sflow Agent is very simple to implement and adds negligible cost to a switch or router. Timely an up to the minute view, for real time controls. QoS and DoS attack.
sflow Architecture
sflow in Operation Switch/Router sflow Datagram forwarding tables interface counters Switching ASIC sflow agent 1 in N sampling packet header src/dst i/f sampling parms forwarding user ID URL i/f counters eg 128B rate pool src 802.1p/Q dst 802.1p/Q next hop src/dst mask AS path communities localpref src/dst Radius TACACS sflow Collector & Analyzer
Statistical Model for Packet Sampling Estimating Traffic per Protocol Total number of frames = N Total number of samples = n Number of samples in class = c Number of frames in the class estimated by: N c = c n N Relative Sampling Error 100% % Error 75% 50% %error 196 1 c 25% 0% 1 10 100 1000 10000 Number of Samples in Class
Even so, Data Volume is Huge! Flow data in our campus network ( /16 prefix ) # of Routers 1 Day 1 Month 1 Year 1 1.2 GB 13 GB 156 GB 5 6 GB 65 GB 780 GB 10 12 GB 130 GB 1.5 TB 200 240 GB 2.6 TB 30 TB Short term period of flow data Massive flow data from anomaly traffic data of Internet worm and DDoS Cluster file system and cloud computing platform Google s programming model, MapReduce, big table Open source system, Hadoop 73
An Experiment: Travel Booking Website 2 BNT switches
Traditional Flow based Traffic Monitoring Run on a high performance central server Routers Flow Data Storage High Performance Server 75
Occupancy of Different Network Service Types Heavy Hitters Workload
Offline Spike Analysis
Google System Health
Software Defined Networking (SDN) What is SDN? Background An OS for networks What is OpenFlow? How it helps SDN The current status & the future of SDN Conclusions 79
Limitations of Current Networks Switches 80
We have lost our way Routing, management, mobility management, access control, VPNs, App App App Operating System Million of lines of source code 5400 RFCs Barrier to entry Specialized Packet Forwarding Hardware 500M gates 10Gbytes RAM Bloated Power Hungry Many complex functions baked into the infrastructure OSPF, BGP, multicast, differentiated services, Traffic Engineering, NAT, firewalls, MPLS, redundant layers, An industry with a mainframe mentality
Reality App App App Operating System Specialized Packet Forwarding Hardware App App Operating System App Specialized Packet Forwarding Hardware Lack of competition means glacial innovation Closed architecture means blurry, closed interfaces Vertically integrated, complex, closed, proprietary Not suitable for experimental ideas Not good for network owners & users Not good for researchers
Glacial process of innovation made worse by captive standards process Idea Standardize Deployment Wait 10 years Driven by vendors Consumers largely locked out Lowest common denominator features Glacial innovation
Old Ways to Configure a Network App App App Operating System Specialized Packet Forwarding Hardware App App App Operating System App App App Operating System Specialized Packet Forwarding Hardware App App App Operating System Specialized Packet Forwarding Hardware App App App Specialized Packet Forwarding Hardware Operating System Specialized Packet Forwarding Hardware 84
No control plane abstraction for the whole network! It s like old times when there was no OS 85 Wilkes with the EDSAC, 1949
Idea: An OS for Networks!!! App App App Closed Operating System Specialized Packet Forwarding Hardware App App App Operating System App App App Operating System Specialized Packet Forwarding Hardware App App App Operating System Specialized Packet Forwarding Hardware App App App Specialized Packet Forwarding Hardware Operating System Specialized Packet Forwarding Hardware
Idea: An OS for Networks Control Programs Network Operating System App App App Operating System Specialized Packet Forwarding Hardware App App App Operating System App App App Operating System Specialized Packet Forwarding Hardware App App App Operating System Specialized Packet Forwarding Hardware App App App Specialized Packet Forwarding Hardware Operating System Specialized Packet Forwarding Hardware 87
Idea: An OS for Networks Control Programs Network Operating System Simple Packet Forwarding Hardware Simple Packet Forwarding Hardware Simple Packet Forwarding Hardware Simple Packet Forwarding Hardware Simple Packet Forwarding Hardware 88
Idea: An OS for Networks NOX: Towards an Operating System for Networks Software Defined Networking (SDN) Control Programs Global Network View Network Operating System Control via forwarding interface Protocols Protocols 89
Trend App App App App App App Windows Windows Windows (OS) (OS) (OS) Linux Linux Linux Mac Mac Mac OS OS OS NOX Controller Controller 1 1 (Network OS) Controller Controller Network 22 OS Virtualization layer x86 (Computer) Virtualization or Slicing OpenFlow Computer Industry Network Industry
Outline What is SDN? Limitations of current networks The idea of Network OS What is OpenFlow? How it helps SDN The current status & the future of SDN Conclusions 92
OpenFlow OpenFlow: Enabling Innovation in Campus Networks Like hardware drivers interface between switches and Network OS 93
OpenFlow Control Path (Software) Data Path (Hardware) 94
OpenFlow OpenFlow Controller OpenFlow Protocol (SSL/TCP) Control Path OpenFlow Data Path (Hardware) 95
OpenFlow Basics Control Program A Control Program B Network OS OpenFlow Protocol SSL TCP IBM 10 gigabit ethernet OpenFlow switch G8264, which has 48 10 GbE SFP+ ports and 4 40 GbE QSFP+ ports
OpenFlow Switching Software Layer Hardware Layer MAC src MAC dst OpenFlow Client IP Src OpenFlow Table IP Dst TCP sport TCP dport Action * * * 5.6.7.8 * * port 1 Controller PC port 1 port 2 port 3 port 4 97 5.6.7.8 1.2.3.4
Flow Table Control Program A Control Program B Network OS Packet Forwarding If header = p, send to port 4 If header = q, overwrite header with r, add header s, and send to ports 5,6 If header =?, send to me Packet Forwarding Flow Table(s) Packet Forwarding
Flow Table Flow 1. Rule (exact & wildcard) Action Statistics Flow 2. Rule (exact & wildcard) Action Statistics Flow 3. Rule (exact & wildcard) Action Statistics Flow N. Rule (exact & wildcard) Default Action Statistics
Flow Entry Match fields Match against packets Action Modify the action set or pipeline processing Stats Update the matching packets Match Fields Action Stats In Port Src MAC Dst MAC Eth Type Vlan Id IP Tos IP Proto IP Src IP Dst TCP Src Port TCP Dst Port Layer 2 Layer 3 Layer 4 1. Forward packet to port(s) 2. Encapsulate and forward to controller 3. Drop packet 4. Send to normal processing pipeline 1. Packet 2. Byte counters
Examples Switching Switch Port MAC src MAC dst Eth type VLAN ID IP Src IP Dst IP Prot TCP sport TCP dport Action * * 00:1f:.. * * * * * * * port6 Flow Switching Switch Port MAC src MAC dst Eth type VLAN ID IP Src IP Dst IP Prot TCP sport TCP dport Action port3 00:20.. 00:1f.. 0800 vlan1 1.2.3.4 5.6.7.8 4 17264 80 port6 Firewall Switch Port MAC src MAC dst Eth type VLAN ID IP Src IP Dst IP Prot TCP sport TCP dport Action * * * * * * * * * 22 drop
Examples Routing Switch Port MAC src MAC dst Eth type VLAN ID IP Src IP Dst IP Prot TCP sport TCP dport Action * * * * * * 5.6.7.8 * * * port6 VLAN Switching Switch Port * MAC src MAC dst Eth type VLAN ID IP Src IP Dst IP Prot TCP sport * 00:1f.. * vlan1 * * * * * TCP dport Action port6, port7, port9
OpenFlow Usage OpenFlow Alice s Switch Rule Alice s code Controller PC Decision? OpenFlow Protocol Alice s OpenFlow Rule Switch Alice s OpenFlow Rule Switch 10 3
OpenFlow Usage» Alice s code: Simple learning switch Per Flow switching Network access control/firewall Static VLANs Her own new routing protocol: unicast, multicast, multipath Home network manager Packet processor (in controller) IPvAlice Alice s code Controller PC 10 4
OpenFlow Standard way to control flow tables in commercial switches and routers Just need to update firmware Essential to the implementation of SDN 105
Centralized/Distributed Control Onix: A Distributed Control Platform for Large scale Production Networks, UNIX OSDI 2010. Centralized Control Distributed Control Controller Controller OpenFlow Switch OpenFlow Switch Controller OpenFlow Switch OpenFlow Switch OpenFlow Switch OpenFlow Switch
Virtualizing OpenFlow Network operators Delegate control of subsets of network hardware and/or traffic to other network operators or users Multiple controllers can talk to the same set of switches Imagine a hypervisor for network equipments Allow experiments to be run on the network in isolation of each other and production traffic
Switch Based Virtualization Research VLAN 2 Research VLAN 1 Production VLANs Flow Table Flow Table Normal L2/L3 Processing Controller Controller 108
FlowVisor A network hypervisor developed by Stanford A software proxy between the forwarding and control planes of network devices
FlowVisor based Virtualization Topology discovery is per slice Aaron s Controller Heidi s Controller Craig s Controller OpenFlow Protocol OpenFlow Switch OpenFlow FlowVisor & Policy Control OpenFlow Protocol OpenFlow Switch OpenFlow Switch 110
Outline What is SDN? Limitations of current networks The idea of Network OS What is OpenFlow? How it helps SDN The current status & the future of SDN Conclusions 11 1
OpenFlow Building Blocks oftrace oflops openseer Monitoring/ debugging tools Stanford Provided ENVI (GUI) LAVI n Casting Expedient Applications NOX Beacon Trema Maestro ONIX Controller FlowVisor Console FlowVisor Slicing Software Commercial Switches HP, NEC, Pronto, Juniper.. and many more Stanford Provided Software NetFPGA Ref. Switch OpenWRT PCEngine WiFi AP Broadcom Ref. Switch Open vswitch OpenFlow Switches 112
Current status of SDN Hardware support Juniper MX series NEC IP8800 WiMax (NEC) HP Procurve 5400 Netgear 7324 PC Engines Pronto 3240/3290 Ciena Coredirector More coming soon... 113
Current status of SDN Industry support Google built hardware and software based on the OpenFlow protocol VMware purchased Nicira for $1.26 billion in 2012 IBM, HP, NEC, Cisco and Juniper also are offering SDNs that may incorporate OpenFlow, but also have other elements that are specific to that vendor and their gear.
Future Focuses of SDN New policies for security Programmable WLANs The placement of controllers (amount; location; centralized/distributed) Debugger for SDN 11 5
Conclusions What is SDN? A system layered abstraction Programmable, flexible, and extensible What is OpenFlow? Interface between switches and controllers Enabling SDN Future SDN Enabling innovation 11 6
Wireless Data Center Networking 60 GHz spectrum 7 GHz (57 64 GHz) waveband data rate over 1 Gb/s 10 meters Line of Sight (LoS)