Open Cirrus: Enabling System Software Research in Computer Science

Transcription

1 OpenCirrus: Enabling System Software Research in Computer Science through Clusters of Bare Metal Michael Kozuch Intel Labs, Pittsburgh November 2011

2 2 Motivation

3 Connecting the Cloud to Open Research Cloud Computing Clearly important whatever it is! Industry developing rapidly Research still plays a role in disruptive advances Open Communities enable rapid innovation Exchange of ideas: Knowledge grows Constructive Darwinism: Best tools survive/evolve Cloud researchers need access to experimental tools at reasonable scale 3

4 Good News Many Options Open Cirrus Type of research Systems and services Approach Participants Federation of heterogeneous data centers CESGA, CMRI, CMU, CNIS, ETRI, HP, GaTech, GSTZ, Intel, IDA, KIT, MIMOS, RAS, UIUC, Yahoo! Google/ IBM cluster Data- intensive applications Cluster supported by Google and IBM Google, IBM, MIT, Stanford Univ., Univ. of Washington Distribution 15 sites Centralized one data center in Atlanta Teragrid PlanetLab Emulab Open Cloud Amazon EC2 PRObE Scientific applications Multisite heterogeneous clusters for supercomputing Many universities and organizations Systems and services Nodes hosted by research institution Many universities and organizations 11 partners in US More than 700 nodes worldwide Systems Single- site cluster with flexible control Univ. of Utah More than 300 machines Interloperability across clouds using open APIs Multisite heterogeneous clusters Commercial use Systems Raw access to virtual machines Reuse of LANL s retiring clusters Four centers Amazon CMU, LANL, NSF 480 cores distributed in four locations Several unified data centers Each test bed has strengths and weaknesses, need to match the resource to the research Thousands of older but still useful nodes at one site 4 Open Cirrus: A Global Cloud Computing Testbed, IEEE Computer, April 2010

5 Agenda Motivation Overview of Open Cirrus * Organization of Open Cirrus Open Cirrus Software Services Example Open Cirrus Site Closing Thoughts 5

6 6 Overview of Open Cirrus

7 Open Cirrus * Cloud Computing Testbed Collaboration between industry and academia, sharing hardware infrastructure software infrastructure research applications and data sets UIUC* KIT* ISPRAS* CMU* GaTech* CESGA* MIMOS* IDA* ETRI* China Telecom* Chinese Acad. Sci.* China Mobile* 7 Sponsored by HP, Intel, and Yahoo! (with additional support from NSF) 15 sites currently, target of around 20 in the next two years

8 Open Cirrus * Objectives Foster systems research around cloud computing Vendor-neutral open-source stacks and APIs for the cloud Expose research community to enterprise level requirements Provide realistic traces of cloud workloads How Open Cirrus differ from typical cloud services Support for systems research and applications research Federation of heterogeneous datacenters Collection of interesting data sets 8

9 9 Organization of Open Cirrus

10 Open Cirrus * Management Central Management Office, oversees Open Cirrus Currently run by HP Labs with Dejan Milojicic as Director Each site Runs its own research and technical teams Contributes individual technologies Operates some of the global services Contributes a member to the Steering Committee Steering Committee Provides long-term guidance to the effort Independently-managed sites providing a cooperative research testbed 10

11 User Access to Open Cirrus * User access is organized around Research Projects Led by Principal Investigator (PI) Project PIs apply to each site separately Identifying additional team members Contact information for applications to each site are available on the Open Cirrus Web site ( Each Open Cirrus site decides which users and projects get access to its site. 11

12 Open Cirrus * Research Projects Example research areas of interest Datacenter federation Datacenter management Web services Data-intensive systems Projects typically not of interest Traditional HPC app development Production apps looking for free cycles Closed-source system development 12

13 Open Cirrus Summits 13 Past summits, 2009-now HP, Palo Alto, 6/2009; Yahoo, Sunnyvale, 1/2010; ETRI, Seoul, Korea, 6/2010; Intel/CMU, Pittsburgh, 10/2010 Summits in 2011 [Fifth] Moscow, June 1-3 & [Sixth] Atlanta, October Formal events, IEEE Technically Co-sponsored >40 papers submitted, 17-member program committee ~25 accepted papers, ~40 posters Each was 2 intense days of interactive exchange Future summits 7th Summit, Beijing, China, May 2012 Workshop on Cloud Computing Testbeds in September 2012 at ICAC (San Jose, CA)

14 2011 Summit Papers Hadoop in the cloud A Study of Skew in MapReduce Applications, Univ. Washington (Best student award) Volume Management of Hadoop DataNode, ChinaMobile Linux PageCache Optimization for Hadoop, Yahoo, India Clusterken: A Reliable Object-Based Messaging Framework to Support Data Center Processing, HP Experiences and apps Experience Using Open Cirrus for a Security-Aware Cloud-Based Business Process Modeling and Execution Environment, UFPE Brazil, and HP Supporting Test-Driven Development of Web Service Choreographies, USP Brazil, HP High-Performance Algorithm for the Coupled Conductive-Radiative Heat Transfer Problems, Far Eastern Federal University, Russia Distributed Cloud Storage Services with FleCS Containers, Georgia Tech, CERCS Lessons from Adopting Cloud-like Architectures in Real-life Financial Applications, Singapore Management University

15 2011 Summit Papers, Cont. Measurements High-Performance Testing, ISP RAS, Russia Virtual Machine based Disk I/O Bandwidth Controller, ETRI, Korea Software performance estimation in a virtualized environment based on atomic tests, Lomonosov Moscow State University, Russia Adding a silver lining to the cloud Controlling Traffic Ensembles in Open Cirrus, HP Practical Cloud Federations, HP and Dortmund University Cloud Sustainability Dashboard, HP Cloud technology and management An Open-source Cloud Management Platform Comparison, CESGA, Spain CAS Data Cloud: Integrated services of data, middleware and infrastructure, Chinese Academy of Sciences

16 2011 Summit Papers, Cont. Systems in the cloud Distributed, Robust Auto-Scaling Policies for Power Management in Compute Intensive Server Farms, Anshul Gandhi, Mor Harchol-Balter, Ram Raghunathan and Michael Kozuch CMU, Intel Labs An Open Job Scheduling Service for Large-Scale Data Processing, Zhenghua Xue, Jianhui Li, Yuanchun Zhou, Yang Zhang and Geng Sheng CNIC Using Active NVRAM for Cloud I/O, Sudarsun Kannan, Dejan Milojicic, Ada Gavrilovska, Karsten Schwan, Hasan Abbasi and Vanish Talwar GaTech, HP Labs, ORNL Applications in the cloud The Application and Practice of Parallel Cloud Computing in ISP, Zhilan Huang, Guoliang Yang and Shengyong Ding (China Telecom) Globally Distributed BookPrep, Dejan Milojicic, Shariff Dudekula, Susanth Puthanveedu and Prakash Reddy HP Labs US, India Running Interactive Perception Applications on OpenCirrus, Qian Zhu, Nezih Yigitbasi and Padmanabhan Pillai Accenture, TUDelft, Intel Labs Evaluation of HPC Applications on Cloud, Abhishek Gupta and Dejan Milojicic HP Labs, UIUC 16

17 17 Open Cirrus Software Services

18 Cloud Software Stack Key Learnings Enable use of application frameworks (Hadoop, Maui-Torque) Enable general IaaS use Provide Big Data storage service Enable physical resources allocation Application Frameworks IaaS Why Physical? 1. Virtualization overhead 2. Access to phys resource 3. Security issues Storage Service Resource Allocator Node Node Node Node Node Node

19 Key Software Services Zoni: Used to allocate mini-clusters to researchers. E.g. bare-metal access for power management projects HDFS: Distributed file system from Hadoop. Used to provide high bandwidth access to Big Data using OTS server systems. E.g. bare-metal access for power management projects Tashi: A VM instantiation service. Particularly useful to users with custom software stacks. E.g. MPI jobs Maui/Torque: Job submission service. Used primarily by users with straightforward applications. E.g. microarchitecture simulation runs Hadoop: Service for users who want to experiment with MapReduce. E.g. evaluation of new HDFS management strategies

20 Zoni Functionality Provides each project with a mini-datacenter Allocation Isolation Assignment of physical resources to users Allow multiple mini-clusters to co-exist without interference Provisioning Booting of specified OS Management Debugging OOB power management OOB console access Isolation of experiments Domain 0 PXE/DNS/DHCP Server Pool 0 Domain 1 DNS/PXE/DHCP Server Pool 0 Server Pool 1 Gateway

21 Access Model At a minimum, sites must expose a ssh gateway Sites may also provide additional external connections Some provision for web services is highly recommended Sites may also be divided into resource pools by service Some services may require a front-end machine (e.g. hadoop) Site Firewall/NAT exported web services port 80/443 Open Cirrus * cluster access port 22 ssh gw Front-end machines

22 22 Example Open Cirrus Site: Intel Labs Pittsburgh

23 45 Mb/s T3 to Internet 20 nodes: 1 Xeon (single-core) [Irwindale] 6GB DRAM 366GB disk 10 nodes: 2 Xeon 5160 (dual-core) [Woodcrest] 4GB RAM 2 75GB Disk 10 nodes: 2 Xeon E5345 (quad-core) [Clovertown] 8GB DRAM 2 150GB Disk Key: rxry=row X rack Y TOR Switch (sw0-r2r1) Blade Rack 1 Gb/s 40 nodes (x4x4 p2p) rxrycz=row X rack Y chassis Z (r2r1c1-4) 1 Gb/s (x4) 1 Gb/s (x4) TOR Switch (sw0-r2r2) Blade Rack 40 nodes 1 Gb/s Xeon E5345 (quad-core) [Clovertown] 8GB DRAM 2 150GB Disk (r2r2c1-4) TOR Switch (sw1-r1r2) 1 Gb/s (x4) (x4x4 p2p) 1 Gb/s x1 (x4) TOR (r1r1) 1 Gb/s Switch (sw0-r1r1) (x2x5 p2p) 1 Gb/s 1U Rack 15 nodes (x15 p2p) 2 Xeon E5420 (quad-core) [Harpertown] 8GB DRAM 2 1TB Disk 1 Gb/s (x4) TOR 1 Gb/s (x8) Switch (sw0-r1r2) 1U Rack 15 nodes Xeon E5420 (quad-core) [Harpertown] 8GB DRAM 2 1TB Disk 3U Rack 5 storage nodes TB Disk 1 Gb/s (r1r5) (x15 p2p) x1 (r1r2) TOR 1 Gb/s Switch (sw2-r1r2) 12 nodes Xeon X5650 (six-core) TOR Switch (sw0-m1) (x4) 10 Gb/s 1 Gb/s [WestmereEP] 48GB DRAM 6 0.5TB Disk 1 Gb/s (x4) (x15 p2p) 1 Gb/s (x8 p2p) 10G Switch (sw0-r3r3) TOR Switch (sw0-r?r?) 1 Gb/s 2U Rack 15 nodes (x15 p2p) Xeon E5440 (quad-core) [Harpertown] 8GB DRAM 6 1TB Disk x3 (r1r3,r1r4,r2r3) Mobile Rack 8 (1u) nodes Xeon E5440 (quad-core) [Harpertown] 16GB DRAM 2 1TB Disk 10 Gb/s TOR Switch (sw0-r3r?) 1 Gb/s 2U Rack 15 nodes (x15 p2p) Xeon E5520 (quad-core) [Nehalem-EP] 16GB DRAM 6 1TB Disk x2 (r3r2,r3r3)

24 24 Intel BigData Cluster Dashboard

25 Example Project 1: AutoScale Idea: Save data center power by turning machines off when load decreases Investigate the trade-off between power and response time Open Cirrus resources used: 2.5 racks 25

26 Our Data Center Unknown Power Aware Load Balancer Load Balancer DB key-value workload mix of CPU & I/O Mor Harchol-Balter, CMU 28 Application servers 7 Memcached Experimental platform deployed on Open Cirrus cluster at Intel Labs Pittsburgh 26

27 Metrics T 95 T avg P avg 95%-tile response time mean power consumption 400 ms 95% Resp. time (ms) Single Server arrival rate (req/s) AlwaysOn ON/OFF N = (max req. rate)/60 Leave N servers on always. N(t) = (req. rate(t))/60 Mor Harchol-Balter, CMU 60 req/s

28 AlwaysOn ON/OFF Num. servers --- load o n busy+idle x n busy+idle+setup Num. servers --- load o n busy+idle x n busy+idle+setup Time (min) Time (min) T 95 =291ms, T avg =137ms P avg =2,323W T 95 =11,003ms, T avg =3,537ms P avg =1,281W I m late, I m late! Mor Harchol-Balter, CMU

29 Towards Better Dynamic Provisioning: AutoScale Problems with Existing Policies: PROBLEM 1: SHUTTING OFF Quick to shut servers off to save power Then suffer huge setup lag. Fixes: Wait until server goes idle. Then wait some more Reduce #servers. by packing servers to SLA point.* PROBLEM 2: TURNING ON Existing policies turn on based on arrival rate. *Converges to optimal # spare servers, w/o knowing arrival rate Mor Harchol-Balter, CMU [Gandhi, Gupta, Harchol-Balter, Kozuch 2010]. Far more accurate & robust to use #jobs in system.

30 ON/OFF AutoScale Num. servers --- load o n busy+idle x n busy+idle+setup Num. servers --- load o n busy+idle x n busy+idle+setup Time (min) Time (min) T 95 =11,003ms, T avg =3,537ms P avg =1,281W T 95 =474ms, T avg =210ms P avg =1,387W Mor Harchol-Balter, CMU I m late, I m late! T 95 =291ms, T avg =137ms P avg =2,323W

31 Results of Implementation AlwaysOn ON/OFF AutoScale T 95 P avg T 95 P avg T 95 P avg 291 ms 2323 W 11,003 ms 1281 W 474 ms 1387 W 271 ms 2205 W 3,802 ms 759 W 466 ms 1016 W 289 ms 2363 W 4,227 ms 1,391 W 480 ms 1729 W 377 ms 2263 W > 1 min 849 W 556 ms 1412 W Mor Harchol-Balter, CMU

32 Example Project 2: Rabbit Idea: Develop a file system with the good properties of HDFS that can scale with load Open Cirrus resources used: 2 racks 32 Robust and Flexible Power-Proportional Storage, SOCC, 2010

33 Power Proportionality and Big Data 2000 Number of blocks stored on node i The Hadoop Filesystem (10K blocks) Possible power savings: ~66% ~0% Optimal: ~95% Node number i i=100

34 Rabbit Filesystem A reliable, power-proportional filesystem for Big Data workloads Simple Strategy: Maintain a primary replica 34

35 Example Project 3: CloudConnect Idea: Add inexpensive optical interconnect to the data center to improve bandwidth Optical links are circuit-switched. Open Cirrus resources used: 1 Rack 35 c-through: Part-time Optics in Data Centers, SIGCOMM, 2010

36 Current solutions for increasing data center network bandwidth FatTree BCube 1. Hard to construct 2. Hard to expand 36

37 Hybrid packet/circuit switched network architecture Electrical packet-switched network for low latency delivery Optical circuit-switched network for high capacity transfer Optical paths are provisioned rack-to-rack A simple and cost-effective choice Aggregate traffic on per-rack basis to better utilize optical circuits

38 Testbed setup 16 servers with 1Gbps NICs Emulate a hybrid network on 48-port Ethernet switch 100Mbps links Ethernet switch Optical circuit emulation Optical paths are available only when hosts are notified During reconfiguration, no host can use optical paths 10 ms reconfiguration delay 4Gbps links Emulated optical circuit switch 38

39 MapReduce sort 10GB random data Completion time (s) Electrical network 128 KB 50 MB 153s 100 MB 300 MB 500 MB 135s Full bisection bandwidth c-through c-through varying socket buffer size limit (reconfiguration interval: 1 sec) 39

40 Yahoo Gridmix benchmark 3 runs of 100 mixed jobs such as web query, web scan and sorting 200GB of uncompressed data, 50 GB of compressed data 40

41 41 Closing Thoughts

42 Where we are now Open Cirrus Beginning our second three-year effort Open Cirrus continues to grow Good research, good community events What could be better (particularly at Intel)? Scheduling idle VMs, idle nodes, etc Federation needs good use model Tracing could be incredibly valuable for research Better tools for creating/uploading images Unlimited resources 42

43 Summary Open Communities can shape the development of Cloud Computing Open Cirrus* is a multi-partner test bed for research in Cloud Computing The Open Cirrus software stack provides a good starting point for open-source cloud computing software development Hadoop, Tashi, Zoni are Apache Software Foundation* efforts 43