Advanced Software Engineering. Lecture 8: Data Center by Prof. Harold Liu

Transcription

1 Advanced Software Engineering Lecture 8: Data Center by Prof. Harold Liu

2 Agenda Introduction Design and Construction Management and Maintenance Hot Topics Real time traffic analysis SDN Wireless Data Center Networking

3 Core Elements of a Data Center Applications Databases Database Management System (DBMS) and the physical and logical storage of data Servers/Operating Systems Networks Storage Arrays 3

4 An Example Client Local Area Network Storage Area Network Storage Array Application User Interface Server Database Consider an order processing system consisting of: Application for order entry. Database Management System (DBMS) to store customer and product information. Server/Operating System on which the Application and Database programs are run. Networks that provide Connectivity between Clients and the Application/Database Server Connectivity between the Server and the Storage system. Storage Array

5 An Example..Closer Look Client Local Area Network Server Storage Area Network Storage Array Application User Interface A customer order is entered via the Application User Interface on a client.

6 An Example..Closer Look Client Local Area Network Server Storage Area Network Storage Array A customer order is entered via the Application User Interface on a client The client accesses the Server over a Local Area Network.

7 An Example..Closer Look Client Local Area Network Server Storage Area Network Storage Array O/S and DBMS A DBMS uses the operating system on the server to read and write this data to the physical location on a disk.

8 An Example..Closer Look Client Local Area Network Server Storage Area Network Storage Array A DBMS uses the operating system on the server to read and write this data to the physical location on disk. A Network provides the communication link between the server and the storage array, and transports the read/write commands and data between the server and the storage array.

9 An Example..Closer Look Client Local Area Network Server Storage Area Network Storage Array Database A DBMS uses the operating system on the server to read and write this data to the physical location on disk. A Network provides the communication link between the client and the server, and transports the read/write commands and data between the server and the storage array. A storage array receives the read/write commands and data from the server and performs the necessary operations to store the data on the physical disks.

10 Google Data Center Google 位 于 俄 勒 冈 州 的 Dalles 数 据 中 心

11 at Lenoir, North Carolina treetview/

12 Gallery

13 IBM Data Center

14 Equinix Data Center

15

16

17 Elements of a Data Center The Site Command Center Cable Management Network Infrastructure Terminal Servers Environmental Controls Power

18 Standards 1

19 Standard 2

20 Structural Aspects

21 Structural Aspects a raised floor ceiling Basement data center locations near water are not a good idea. Their must be a pathway for equipment to be moved in & out of the data center. Make sure the floor where the data center is to be located is rated for the estimated load.

22 Power and Environment Dual street power supply with UPS When necessary, a diesel generator as a second backup Environmental Control Facilities Temperature Moisture dust

23 HVAC (Heating, Ventilation, and Air Conditioning)

24 Hot Aisle and Cold Aisle Layout

25 Networking 3 layer architecture Core Layer Switch/Router Layer 2 Switch Internal NIC Ethernet connection Cisco Core Layer Router DLink 48ports Gbps Layer-2 Switch

26 Security Physical Access Levels of Access Monitoring

27 Past to Future a single computer occupied the space of an entire Data Center. That same space can be occupied by thousands of servers today.

28 Let us Make it Online HW SW Rack Deployment and Testing Maintenance

29 Servers Tower Low cost, customized Space cost, not for parallel computing Blade save space, high computing ability suitable for large size data center Rack Server small, easy to manage need good cooling

30 Rack Unit (RU) Unit=44.45mm (height) Width=19 inch or 23 inch 1U, 2U, 4U (half rack) 1U 4U

31 Server Rack and Cluster

32 Data Center Topology Core Internet Layer-3 router Data Center Aggregation Layer-2/3 switch Access Top of Rack Switch Layer-2 switch Servers

33 Top of Rack (ToR) Switch

34 Data Center Costs James Hamilton published basic 2008 breakdown Servers: 45% CPU, memory, disk Infrastructure: 25% UPS, cooling, power distribution Power draw: 15% Electrical utility costs Network: 15% Switches, links, transit

35 Power Cost Approximate distribution of peak power usage by hardware subsystem in one of Google s datacenters (circa 2007).

36 Data Center Challenges Traffic load balance Support for VM migration Achieving bisection bandwidth Power savings / Cooling Network management (provisioning) Security (dealing with multiple tenants)

37 Non Virtualized Data Centers Too many servers for too little work High costs and infrastructure needs Maintenance Networking Floor space Cooling Power Disaster Recovery

38

39

40 What is Virtualization? Run multiple OSes and user applications on the same hardware e.g., run both Windows and Linux on the same laptop How is it different from dual boot? Both OSes run simultaneously OSes are completely isolated from each other

41

42

43

44 Reduce costs by consolidating services onto the fewest number of physical machines

45 Dynamic Data Center Virtualization helps us break the one service per server model Consolidate many services into a fewer number of machines when workload is low, reducing costs Conversely, as demand for a particular service increases, we can shift more virtual machines to run that service We can build a data center with fewer total resources, since resources are used as needed instead of being dedicated to single services

46 VM Workload Multiplexing Separate VM sizing VM multiplexing s 3 s 2 s 1 We expect s 3 < s 1 + s 2. Benefit of multiplexing! Multiplex VMs workload on same physical server

47 So, it is just like Java VM, right?

48

49 Two Types of Hypervisors (or VMM) Hypervisor is a software layer that allows several VMs to run on a physical machine The physical OS and hardware are called the Host VM OS and applications are called the Guest Type 1 (bare metal) Type 2 (hosted) Process Process Process Process VM1 VM2 Guest Guest VM1 VM2 Process Hypervisor Host Hypervisor Hardware OS Hardware Host VMware ESX, Microsoft Hyper V, Xen VMware Workstation, Microsoft Virtual PC, Sun VirtualBox, QEMU, KVM

50 Bare metal or Hosted? Bare metal Has complete control over hardware Doesn t have to fight an OS Hosted Avoid code duplication: need not code a process scheduler, memory management system the OS already does that Can run native processes alongside VMs Familiar environment how much CPU and memory does a VM take? Use top! How big is the virtual disk? ls l Easy management stop a VM? Sure, just kill it! A combination Mostly hosted, but some parts are inside the OS kernel for performance reasons, e.g., KVM

51 VM on Multi core CPUs Each core can be configured for multiple VMs A Quad core CPU could be configured as a 32 node multi computer Limiting factor is often memory. Each guest OS has its own requirements (512 MB?) 51

52 Installing a Virtual machine Base OS is Windows 7 First install Sun VirtualBox as hypervisor Then, Guest OS will be Ubuntu

53 Installing Sun/Oracle VirtualBox 53

54 Installing VirtualBox 54

55 Installing VirtualBox 55

56 Installing VirtualBox 56

57 Installing Ubuntu VM 57

58 Installing Ubuntu VM 58

59 Installing Ubuntu VM 59

60 Installing Ubuntu VM 60

61 61

62 Installing Ubuntu VM 62

63 63

64 Installing Ubuntu VM 64

65 Data Center Topology Core Internet Layer-3 router Data Center Aggregation Layer-2/3 switch Access Top of Rack Switch Layer-2 switch Servers

66 Oversubscription

67

68 Motivation Flow based traffic monitoring Volume of processed data is reduced Popular flow statistics tools Cisco NetFlow sflow

69 What is sflow? Accurate sampling is simple enough to be performed in hardware, at wire speed. Good accuracy under heavy loads Detailed complete packet header and switching/routing information for L2 L7 traffic flows. Scalable capable of monitoring networks at 10Gbps, 100Gbps and beyond. Thousands of devices can be monitored by a single sflow Collector. Low Cost sflow Agent is very simple to implement and adds negligible cost to a switch or router. Timely an up to the minute view, for real time controls. QoS and DoS attack.

70 sflow Architecture

71 sflow in Operation Switch/Router sflow Datagram forwarding tables interface counters Switching ASIC sflow agent 1 in N sampling packet header src/dst i/f sampling parms forwarding user ID URL i/f counters eg 128B rate pool src 802.1p/Q dst 802.1p/Q next hop src/dst mask AS path communities localpref src/dst Radius TACACS sflow Collector & Analyzer

72 Statistical Model for Packet Sampling Estimating Traffic per Protocol Total number of frames = N Total number of samples = n Number of samples in class = c Number of frames in the class estimated by: N c = c n N Relative Sampling Error 100% % Error 75% 50% %error c 25% 0% Number of Samples in Class

73 Even so, Data Volume is Huge! Flow data in our campus network ( /16 prefix ) # of Routers 1 Day 1 Month 1 Year GB 13 GB 156 GB 5 6 GB 65 GB 780 GB GB 130 GB 1.5 TB GB 2.6 TB 30 TB Short term period of flow data Massive flow data from anomaly traffic data of Internet worm and DDoS Cluster file system and cloud computing platform Google s programming model, MapReduce, big table Open source system, Hadoop 73

74 An Experiment: Travel Booking Website 2 BNT switches

75 Traditional Flow based Traffic Monitoring Run on a high performance central server Routers Flow Data Storage High Performance Server 75

76 Occupancy of Different Network Service Types Heavy Hitters Workload

77 Offline Spike Analysis

78 Google System Health

79 Software Defined Networking (SDN) What is SDN? Background An OS for networks What is OpenFlow? How it helps SDN The current status & the future of SDN Conclusions 79

80 Limitations of Current Networks Switches 80

81 We have lost our way Routing, management, mobility management, access control, VPNs, App App App Operating System Million of lines of source code 5400 RFCs Barrier to entry Specialized Packet Forwarding Hardware 500M gates 10Gbytes RAM Bloated Power Hungry Many complex functions baked into the infrastructure OSPF, BGP, multicast, differentiated services, Traffic Engineering, NAT, firewalls, MPLS, redundant layers, An industry with a mainframe mentality

82 Reality App App App Operating System Specialized Packet Forwarding Hardware App App Operating System App Specialized Packet Forwarding Hardware Lack of competition means glacial innovation Closed architecture means blurry, closed interfaces Vertically integrated, complex, closed, proprietary Not suitable for experimental ideas Not good for network owners & users Not good for researchers

83 Glacial process of innovation made worse by captive standards process Idea Standardize Deployment Wait 10 years Driven by vendors Consumers largely locked out Lowest common denominator features Glacial innovation

84 Old Ways to Configure a Network App App App Operating System Specialized Packet Forwarding Hardware App App App Operating System App App App Operating System Specialized Packet Forwarding Hardware App App App Operating System Specialized Packet Forwarding Hardware App App App Specialized Packet Forwarding Hardware Operating System Specialized Packet Forwarding Hardware 84

85 No control plane abstraction for the whole network! It s like old times when there was no OS 85 Wilkes with the EDSAC, 1949

86 Idea: An OS for Networks!!! App App App Closed Operating System Specialized Packet Forwarding Hardware App App App Operating System App App App Operating System Specialized Packet Forwarding Hardware App App App Operating System Specialized Packet Forwarding Hardware App App App Specialized Packet Forwarding Hardware Operating System Specialized Packet Forwarding Hardware

87 Idea: An OS for Networks Control Programs Network Operating System App App App Operating System Specialized Packet Forwarding Hardware App App App Operating System App App App Operating System Specialized Packet Forwarding Hardware App App App Operating System Specialized Packet Forwarding Hardware App App App Specialized Packet Forwarding Hardware Operating System Specialized Packet Forwarding Hardware 87

88 Idea: An OS for Networks Control Programs Network Operating System Simple Packet Forwarding Hardware Simple Packet Forwarding Hardware Simple Packet Forwarding Hardware Simple Packet Forwarding Hardware Simple Packet Forwarding Hardware 88

89 Idea: An OS for Networks NOX: Towards an Operating System for Networks Software Defined Networking (SDN) Control Programs Global Network View Network Operating System Control via forwarding interface Protocols Protocols 89

90 Trend App App App App App App Windows Windows Windows (OS) (OS) (OS) Linux Linux Linux Mac Mac Mac OS OS OS NOX Controller Controller 1 1 (Network OS) Controller Controller Network 22 OS Virtualization layer x86 (Computer) Virtualization or Slicing OpenFlow Computer Industry Network Industry

91

92 Outline What is SDN? Limitations of current networks The idea of Network OS What is OpenFlow? How it helps SDN The current status & the future of SDN Conclusions 92

93 OpenFlow OpenFlow: Enabling Innovation in Campus Networks Like hardware drivers interface between switches and Network OS 93

94 OpenFlow Control Path (Software) Data Path (Hardware) 94

95 OpenFlow OpenFlow Controller OpenFlow Protocol (SSL/TCP) Control Path OpenFlow Data Path (Hardware) 95

96 OpenFlow Basics Control Program A Control Program B Network OS OpenFlow Protocol SSL TCP IBM 10 gigabit ethernet OpenFlow switch G8264, which has GbE SFP+ ports and 4 40 GbE QSFP+ ports

97 OpenFlow Switching Software Layer Hardware Layer MAC src MAC dst OpenFlow Client IP Src OpenFlow Table IP Dst TCP sport TCP dport Action * * * * * port 1 Controller PC port 1 port 2 port 3 port

98 Flow Table Control Program A Control Program B Network OS Packet Forwarding If header = p, send to port 4 If header = q, overwrite header with r, add header s, and send to ports 5,6 If header =?, send to me Packet Forwarding Flow Table(s) Packet Forwarding

99 Flow Table Flow 1. Rule (exact & wildcard) Action Statistics Flow 2. Rule (exact & wildcard) Action Statistics Flow 3. Rule (exact & wildcard) Action Statistics Flow N. Rule (exact & wildcard) Default Action Statistics

100 Flow Entry Match fields Match against packets Action Modify the action set or pipeline processing Stats Update the matching packets Match Fields Action Stats In Port Src MAC Dst MAC Eth Type Vlan Id IP Tos IP Proto IP Src IP Dst TCP Src Port TCP Dst Port Layer 2 Layer 3 Layer 4 1. Forward packet to port(s) 2. Encapsulate and forward to controller 3. Drop packet 4. Send to normal processing pipeline 1. Packet 2. Byte counters

101 Examples Switching Switch Port MAC src MAC dst Eth type VLAN ID IP Src IP Dst IP Prot TCP sport TCP dport Action * * 00:1f:.. * * * * * * * port6 Flow Switching Switch Port MAC src MAC dst Eth type VLAN ID IP Src IP Dst IP Prot TCP sport TCP dport Action port3 00: :1f vlan port6 Firewall Switch Port MAC src MAC dst Eth type VLAN ID IP Src IP Dst IP Prot TCP sport TCP dport Action * * * * * * * * * 22 drop

102 Examples Routing Switch Port MAC src MAC dst Eth type VLAN ID IP Src IP Dst IP Prot TCP sport TCP dport Action * * * * * * * * * port6 VLAN Switching Switch Port * MAC src MAC dst Eth type VLAN ID IP Src IP Dst IP Prot TCP sport * 00:1f.. * vlan1 * * * * * TCP dport Action port6, port7, port9

103 OpenFlow Usage OpenFlow Alice s Switch Rule Alice s code Controller PC Decision? OpenFlow Protocol Alice s OpenFlow Rule Switch Alice s OpenFlow Rule Switch 10 3

104 OpenFlow Usage» Alice s code: Simple learning switch Per Flow switching Network access control/firewall Static VLANs Her own new routing protocol: unicast, multicast, multipath Home network manager Packet processor (in controller) IPvAlice Alice s code Controller PC 10 4

105 OpenFlow Standard way to control flow tables in commercial switches and routers Just need to update firmware Essential to the implementation of SDN 105

106 Centralized/Distributed Control Onix: A Distributed Control Platform for Large scale Production Networks, UNIX OSDI Centralized Control Distributed Control Controller Controller OpenFlow Switch OpenFlow Switch Controller OpenFlow Switch OpenFlow Switch OpenFlow Switch OpenFlow Switch

107 Virtualizing OpenFlow Network operators Delegate control of subsets of network hardware and/or traffic to other network operators or users Multiple controllers can talk to the same set of switches Imagine a hypervisor for network equipments Allow experiments to be run on the network in isolation of each other and production traffic

108 Switch Based Virtualization Research VLAN 2 Research VLAN 1 Production VLANs Flow Table Flow Table Normal L2/L3 Processing Controller Controller 108

109 FlowVisor A network hypervisor developed by Stanford A software proxy between the forwarding and control planes of network devices

110 FlowVisor based Virtualization Topology discovery is per slice Aaron s Controller Heidi s Controller Craig s Controller OpenFlow Protocol OpenFlow Switch OpenFlow FlowVisor & Policy Control OpenFlow Protocol OpenFlow Switch OpenFlow Switch 110

111 Outline What is SDN? Limitations of current networks The idea of Network OS What is OpenFlow? How it helps SDN The current status & the future of SDN Conclusions 11 1

112 OpenFlow Building Blocks oftrace oflops openseer Monitoring/ debugging tools Stanford Provided ENVI (GUI) LAVI n Casting Expedient Applications NOX Beacon Trema Maestro ONIX Controller FlowVisor Console FlowVisor Slicing Software Commercial Switches HP, NEC, Pronto, Juniper.. and many more Stanford Provided Software NetFPGA Ref. Switch OpenWRT PCEngine WiFi AP Broadcom Ref. Switch Open vswitch OpenFlow Switches 112

113 Current status of SDN Hardware support Juniper MX series NEC IP8800 WiMax (NEC) HP Procurve 5400 Netgear 7324 PC Engines Pronto 3240/3290 Ciena Coredirector More coming soon

114 Current status of SDN Industry support Google built hardware and software based on the OpenFlow protocol VMware purchased Nicira for $1.26 billion in 2012 IBM, HP, NEC, Cisco and Juniper also are offering SDNs that may incorporate OpenFlow, but also have other elements that are specific to that vendor and their gear.

115 Future Focuses of SDN New policies for security Programmable WLANs The placement of controllers (amount; location; centralized/distributed) Debugger for SDN 11 5

116 Conclusions What is SDN? A system layered abstraction Programmable, flexible, and extensible What is OpenFlow? Interface between switches and controllers Enabling SDN Future SDN Enabling innovation 11 6

117 Wireless Data Center Networking 60 GHz spectrum 7 GHz (57 64 GHz) waveband data rate over 1 Gb/s 10 meters Line of Sight (LoS)