Open Cirrus : A Global Testbed for Cloud Computing Research

Transcription

1 Open Cirrus : A Global Testbed for Cloud Computing Research David O Hallaron Director, Intel Labs Pittsburgh Carnegie Mellon University

2 Open Cirrus Testbed Sponsored by HP, Intel, and Yahoo! (w/additional support from NSF). 9 sites worldwide, target of around 20 in the next two years. Each site cores. Shared hardware infrastructure (~15K cores), s, research, apps. 2 Dave O Hallaron DIC Workshop, 2009

3 Open Cirrus Context Goals 1. Foster new systems and s research around cloud computing 2. Catalyze open-source stack and APIs for the cloud Motivation Enable more tier-2 and tier-3 public and private cloud providers How are we different? Support for systems research and applications research Access to bare metal, integrated virtual-physical migration Federation of heterogeneous datacenters Global signon, monitoring, storage s. 3 Dave O Hallaron DIC Workshop, 2009

4 Intel BigData Cluster Open Cirrus site hosted by Intel Labs Pittsburgh Operational since Jan nodes, 1440 cores, 1416 GB DRAM, 500 TB disk Supporting 50 users, 20 projects from CMU, Pitt, Intel, GaTech Cluster management, location and power aware scheduling, physical virtual migration (Tashi), cache savvy algorithms (Hi- Spade), realtime streaming frameworks (SLIPstream), optical datacenter interconnects (CloudConnect), log-based architectures (LBA) Machine translation, speech recognition, programmable matter simulation, ground model generation, online education, realtime brain activity decoding, realtime gesture and object recognition, federated perception, automated food recognition. Idea for a research project on Open Cirrus? Send short abstract to Mike Kozuch, Intel Labs Pittsburgh, michael.a.kozuch@intel.com 4 Dave O Hallaron DIC Workshop, 2009

5 Open Cirrus Stack Management and control subsystem Compute + network + storage resources Power + cooling Physical Resource set (PRS) 5 Dave O Hallaron DIC Workshop, 2009 Credit: John Wilkes (HP)

6 Open Cirrus Stack PRS clients, each with their own physical data center Research Tashi NFS storage HDFS storage PRS 6 Dave O Hallaron DIC Workshop, 2009

7 Open Cirrus Stack Virtual clusters (e.g., Tashi) Virtual cluster Virtual cluster Research Tashi NFS storage HDFS storage PRS 7 Dave O Hallaron DIC Workshop, 2009

8 Open Cirrus Stack BigData App Hadoop 1. Application running 2. On Hadoop 3. On Tashi virtual cluster 4. On a PRS 5. On real hardware Virtual cluster Virtual cluster Research Tashi NFS storage HDFS storage PRS 8 Dave O Hallaron DIC Workshop, 2009

9 Open Cirrus Stack BigData app Hadoop Experiment/ save/restore Virtual cluster Virtual cluster Research Tashi NFS storage HDFS storage PRS 9 Dave O Hallaron DIC Workshop, 2009

10 Open Cirrus Stack Platform s BigData App Hadoop Experiment/ save/restore Virtual cluster Virtual cluster Research Tashi NFS storage HDFS storage PRS 10 Dave O Hallaron DIC Workshop, 2009

11 Open Cirrus Stack Platform s User s BigData App Hadoop Experiment/ save/restore Virtual cluster Virtual cluster Research Tashi NFS storage HDFS storage PRS 11 Dave O Hallaron DIC Workshop, 2009

12 Open Cirrus Stack Platform s User s BigData App Hadoop Experiment/ save/restore Virtual cluster Virtual cluster Research Tashi NFS storage HDFS storage PRS 12 Dave O Hallaron DIC Workshop, 2009

13 System Organization Compute nodes are divided into dynamically-allocated, vlanisolated PRS subdomains Open research Tashi development Apps running in a VM mgmt infrastructure (e.g., Tashi) Apps switch back and forth between virtual and phyiscal. Production storage Proprietary research Open workload monitoring and trace collection 13 Dave O Hallaron DIC Workshop, 2009

14 Open Cirrus Stack - PRS PRS goals Provide mini-datacenters to researchers Isolate experiments from each other Stable base for other research PRS approach Allocate sets of physical co-located nodes, isolated inside VLANs. PRS code from HP Labs being merged into Apache Tashi project. Credit: Kevin Lai (HP), Richard Gass, Michael Ryan, Michael Kozuch, and David O Hallaron (Intel) 14 Dave O Hallaron DIC Workshop, 2009

15 Open Cirrus Stack - Tashi An open source Apache Software Foundation project sponsored by Intel, CMU, and HP. Research infrastructure for cloud computing on Big Data Implements AWS interface Daily production use on Intel cluster for 6 months Manages pool of 80 physical nodes ~20 projects/40 users from CMU, Pitt, Intel Research focus: Location-aware co-scheduling of VMs, storage, and power. Integrated physical/virtual migration (using PRS) Credit: Mike Kozuch, Michael Ryan, Richard Gass, Dave O Hallaron (Intel), Greg Ganger, Mor Harchol-Balter, Julio Lopez, Jim Cipar, Elie Kravat, Anshul Ghandi, Michael Stroucken (CMU) 15 Dave O Hallaron DIC Workshop, 2009

16 Tashi High-Level Design Most decisions happen in the scheduler; manages compute/storage/power in concert Scheduler Cluster Manager Node Virtualization Service Node Storage Service Node Services are instantiated through virtual machines Node Node Data location and power information is exposed to scheduler and s The storage aggregates the capacity of the commodity nodes to house Big Data repositories. Node CM maintains databases and routes messages; decision logic is limited 16 Dave O Hallaron DIC Workshop, 2009 Cluster nodes are assumed to be commodity machines

17 Throughput/disk (MB/s) 3.6X 3.5X 11X 9.2X Location Matters (calculated) 300 Calculated (40 racks * 30 nodes * 2 disks) Disk-1G SSD-1G Disk-10G SSD-10G Random Placement Location-Aware Placement 17 Dave O Hallaron DIC Workshop, 2009

18 Throughput/disk (MB/s) 2.9X 4.7X Location Matters (measured) Measured (2 racks * 14 nodes * 6 disks) ssh xinetd Random Placement Location-aware Placement 18 Dave O Hallaron DIC Workshop, 2009

19 Open Cirrus Stack Hadoop An open-source Apache Software Foundation project sponsored by Yahoo! Provides a parallel programming model (MapReduce), a distributed file system, and a parallel database (HDFS) Dave O Hallaron DIC Workshop, 2009

20 Typical Web Service Data center db db Application Application server Application server Application server server HTTP server Query Result External client Characteristics: Small queries and results Little client computation Moderate server computation Moderate data accessed per query Examples: Web sites serving dynamic content 20 Dave O Hallaron DIC Workshop, 2009

21 Big Data Service Data-intensive computing system (e.g. Hadoop) External data sources Parallel data server Parallel compute server Parallel query server Query Result External client d 1 d 2 d 3 Parallel file system (e.g., GFS, HDFS) Source dataset Derived datasets Characteristics: Small queries and results Massive data and computation performed on server 21 Dave O Hallaron DIC Workshop, 2009 Examples: Search Photo scene completion Log processing Science analytics

22 Streaming Data Service External data sources Parallel data server Parallel compute server Parallel query server Continuous query stream Continuous query results External client and sensors d 1 d 2 d 3 Source dataset Derived datasets Characteristics: Application lives on client Client uses cloud as an accelerator Data transferred with query Variable, latency sensitive HPC on server Often combines with Big Data 22 Dave O Hallaron DIC Workshop, 2009 Examples: Perceptual computing on high data-rate sensors: real time brain activity detection, object recognition, gesture recognition

23 Streaming Data Service Gestris Interactive Gesture Recognition Two-player Gestris (gesture-tetris) implementation 2 video sources Uses a simplified volumetric event detection algorithm 10 cores, 3GHz each: -1 camera input, scaling -1 game + display -8 for volumetric matching (4 for each video stream) Achieves full 15fps rate Arm gesture selects action Credit: Lily Mummert, Babu Pillai, Rahul Sukthankar (Intel), Martial Hebert, Pyre Matikainen (CMU) 23 Dave O Hallaron DIC Workshop, 2009

24 Streaming Data Meets Big Data Real-time Brain Activity Decoding Magnetoencephalography (MEG) measures the magnetic fields associated with brain activity. Temporal and spatial resolution offers unprecedented insights into brain dynamics. MEG ECoG Credit: Dean Pomerleau (Intel), Tom Mitchell, Gus Sudre and Mark Palatucci (CMU), Wei Wang, Doug Weber and Anto Bagic (UPitt) 24 Dave O Hallaron DIC Workshop, 2009

25 Localizing Sources of Magnetic Activity Goal: determine spatiotemporal pattern of brain activity most likely to have caused measured magnetic field Magnetic Field Measurements Estimated Brain Activity Ill-posed problem that applies to both MEG and EEG. Very computationally expensive Important for better mapping to fmri results, further neuroscience understanding of brain processes and (maybe) improve decoding. 25 Dave O Hallaron DIC Workshop, 2009

26 Big Data Background Processing Source localization pipeline MRI data MEG or EEG field data Reconstruct brain (~ 40 hr/ subject) Create co-registered boundary model (~ 1 hr / subject) Pre-processing & Filtering (~ 1 hr / session) Brain Structural Information Model of electro-magnetic field from sources to sensors (~ 5 min / session) Brain activity estimates (movies, time series) (~ 15 min / session) Electromagnetic Field Measurements 26 Dave O Hallaron DIC Workshop, 2009

27 Hand Foot Celery Streaming/Big Data Service Real-Time MEG/EEG Decoding Real-Time Decoding Of Brain Activity Decoded Results Brain Activity Decoding Hand Celery Hand Foot Airplane Off-line Decoder Training (once) Brain activity estimates Cloud cluster Source Localization Preprocess & filter Stimulus MEG/EEG Electro-magnetic Imaging field Data 27 Dave O Hallaron DIC Workshop, 2009 Off-line Source Modeling (once)

28 Summary and Lessons Using the cloud as an accelerator for interactive streaming/big data apps is an important usage model. Location-aware and power-aware workload scheduling still open problems. Need integrated physical/virtual allocations to combat cluster squatting. Storage models are still a problem. GFS-style storage systems not mature, impact of SSDs unknown We need open source architecture and reference implementations. Access model Local and global s Application frameworks Need to investigate new application frameworks Map-reduce/Hadoop not always appropriate 28 Dave O Hallaron DIC Workshop, 2009