Ant Rowstron. Joint work with Paolo Costa, Austin Donnelly and Greg O Shea Microsoft Research Cambridge. Hussam Abu-Libdeh, Simon Schubert Interns

Transcription

1 Ant Rowstron Joint work with Paolo Costa, Austin Donnelly and Greg O Shea Microsoft Research Cambridge Hussam Abu-Libdeh, Simon Schubert Interns

2 Thinking of large-scale data centers Microsoft, Google, Yahoo, Amazon, Facebook Data centers run: External facing applications Hotmail, Search, Facebook feeds, etc. Internal services (building blocks for the apps) Microsoft: Autopilot, Dryad Google: GFS, BigTable, MapReduce Yahoo: Hadoop, HDFS Amazon: Dynamo, Astrolabe Facebook: Cassandra, Scribe CamCube: Rethinking the Data Center 2/24

3 Distributed systems abstractions vs. reality Key-based Explicit structure Symmetry of role Oversubscription Fate-sharing TCP incast Failures CamCube: Rethinking the Data Center 3/24

4 Tweak things (in isolation) More bandwidth (fat-trees, clos networks) Centralized flow scheduling TCP-tweaking Flat layer 2 addressing Fundamental issue: Technology developed for a different environment Services have little or no control over the network CamCube: Rethinking the Data Center 4/24

5 Design a data center from the perspective of a distributed systems builder Distributed Systems CamCube HPC Networking CamCube: Rethinking the Data Center 5/24

6 Simple composition Service Architecture TCP/IP Link and key-based Software Stack Network Topology CamCube: Rethinking the Data Center 6/24

7 Simple composition Service Architecture TCP/IP Link and key-based Software Stack Network Topology CamCube: Rethinking the Data Center 7/24

8 Nodes are commodity x86 servers with local storage Container-based model 1,500-2,500 servers Two types of networks: Direct-connect network for intra-server traffic Six 1 Gbps Ethernet ports / server Switched network for inter-camcube traffic Not all servers need to be connected Multiple independent networks can be used CamCube: Rethinking the Data Center 8/24

9 Nodes are commodity x86 servers with local storage Container-based model 1,500-2,500 servers Two types of networks: Direct-connect network for intra-server traffic Six 1 Gbps Ethernet ports / server Switched network for inter-camcube traffic Not all servers need to be connected Multiple independent networks can be used CamCube: Rethinking the Data Center 9/24

10 Traditional data centers are provisioned for peak inter- and intra traffic. Nodes are commodity x86 servers with local storage Container-based model 1,500-2,500 servers Two types of networks: Direct-connect network for intra-server traffic Six 1 Gbps Ethernet ports / server Switched network for inter-camcube traffic Not all servers need to be connected Multiple independent networks can be used CamCube: Rethinking the Data Center 10/24

11 The CamCube API Nodes have (x,y,z)coordinate Defines key-s Simple one hop API to send/receive packets to/from 1-hop neighbours Key or address-based Coordinates re-mapped in case of failures Enable KBR-like API Merge physical and virtual topologies y (1,2,1) z (1,2,2) (1,2,2) (1,2,1) x CamCube: Rethinking the Data Center 11/24

12 (0,0,0) (1,0,0) (2,0,0) CamCube: Rethinking the Data Center 12/24

13 Fetch mail Antr:1234 (1,1,0) (0,0,0) (1,0,0) (2,0,0) Packets move from service to service and server to server Services can intercept and process packets along the way Keys are re-mapped in case of failures Server resources (RAM, Disks, CPUs) are accessible to all services CamCube: Rethinking the Data Center 13/24

14 (1,1,0) (0,0,0) (1,0,0) (2,0,0) Packets move from service to service and server to server Services can intercept and process packets along the way Keys are re-mapped in case of failures Server resources (RAM, Disks, CPUs) are accessible to all services CamCube: Rethinking the Data Center 14/24

15 (1,1,0) (0,0,0) (1,0,0) (2,0,0) Packets move from service to service and server to server Services can intercept and process packets along the way Keys are re-mapped in case of failures Server resources (RAM, Disks, CPUs) are accessible to all services CamCube: Rethinking the Data Center 15/24

16 (1,1,0) (0,0,0) (1,0,0) (2,0,0) Packets move from service to service and server to server Services can intercept and process packets along the way Keys are re-mapped in case of failures Server resources (RAM, Disks, CPUs) are accessible to all services CamCube: Rethinking the Data Center 16/24

17 (1,1,0) (0,0,0) (1,0,0) (2,0,0) Packets move from service to service and server to server Services can intercept and process packets along the way Keys are re-mapped in case of failures Server resources (RAM, Disks, CPUs) are accessible to all services CamCube: Rethinking the Data Center 17/24

18 Exploit properties of a data center Response data size usually known in advance E.g. Windows TransmitFile All or nothing value to response data Simple API Send(destination, service, byte [] data) CamCube: Rethinking the Data Center 18/24

19 Must be multi-path: Resilience to failure Exploit bandwidth available CamCube: Rethinking the Data Center 19/24

20 Can exploit per-hop functionality: (0,0,0) (1,0,0) (2,0,0) CamCube: Rethinking the Data Center 20/24

21 Can exploit per-link functionality: (0,0,0) (1,0,0) (2,0,0) Ethernet Flow Control Low packet loss rates CamCube: Rethinking the Data Center 21/24

22 CamCube provides link-state based routing protocol Uses shortest paths only CamCube: Rethinking the Data Center 22/24

23 Service provides reliability (e2e retransmission) Hot potato routing at source Inject traffic into network quickly Hop-by-hop reliability Logarithmic ack vectors e2e Large buffers per server (hop) How large (1GB+?) Store packets for long periods? (seconds, mins, hours?) What to include QoS (?) Explicit per-link congestion notification Think Ethernet PAUSE frames Want to include admission control Graceful degradation at high utilization rates CamCube: Rethinking the Data Center 23/24

24 27-server (3x3x3) prototype Quad-core Intel Xeon Ghz 4GB RAM Windows Server 2008 R2 6 Intel PRO/1000 PT 1 Gbps ports Version 2 of the runtime 12Gbps aggregate throughput for <25% CPU usage CamCube: Rethinking the Data Center 24/24

25 Distributed Systems High Performance Computing MPI hides the underlying topology CamCube makes topology explicit Services can intercept packet on-path Failure resilience Multiple independent services Distributed systems HPC Key-space naturally mapped on the physical topology No need to reverse-engineer the network Networking No switches or routers (symmetry of role) Cross-layer approach Not using TCP/IP CamCube Networking CamCube: Rethinking the Data Center 25/24

26 Building large-scale services is hard Mismatch between physical topology and abstractions used The network is a black box CamCube Designed from the perspective of distributed systems developers Explicit topology Supports key-based abstraction Benefits Simplified development More efficient applications and better fault-tolerance CamCube: Rethinking the Data Center 26/24