An Overview of the Open Cloud Consortium

Transcription

1 An Overview of the Open Cloud Consortium Robert Grossman Open Cloud Consortium OMG Cloud Computing Interoperability Workshop July 13, 2009

2 This talk represents my personal opinions and does not necessarily represent a consensus position of the Open Cloud Consortium or positions of any of its members. 2

3 Part 1. Introduction 3

4 Two Types of Clouds On-demand resources & services over a network at the scale of a data center On-demand computing instances (IaaS) IaaS: Amazon EC2, S3, etc.; Eucalyptus supports many Web 2.0 applications/users On-demand cloud services for large data cloud applications (Large Data PaaS) GFS/MapReduce/Bigtable, Hadoop, Sector, Manage and compute with large data (say 100+ TB) 4

5 Delivering On-Demand Computing Instances App App App App App App App App Table-based Data (BigTable) Services Relational Data Services (Pig, etc.) App App Cloud Compute Services (MapReduce, Variants & Generalizations) App App Cloud Storage Services For both architectures, you are interested in filling a data center. 5

6 Activities Currently Focused Around Five Use Cases 1. Moving an existing cloud application from Cloud 1 to Cloud 2 without changing the application. 2. Providing surge capacity for an application on Cloud 1 using any of the Clouds 2, 3, (without changing the application). 1. Migrate / port 2. Surge / burst Cloud 1 Cloud 2

7 Large Data Cloud Use Cases 3. Moving a large data cloud application from one large data cloud storage service to another. 4. Moving a large data cloud application from one large data cloud compute service to another. App 1 App 2 Large Data Cloud Compute Services Large Data Cloud Storage Services

8 Inter-Cloud Use Case 5. Inter-cloud communication between two HIPAA compliant clouds. HIPAA Cloud 1 HIPAA Cloud 2

9 Part 2. The Open Cloud Consortium (OCC) 9

10 OCC is a member driven organization that: 1. Supports the development of standards. 2. Supports the development of benchmarks. 3. Supports reference implementations for cloud computing, preferably open source. 4. Sponsors workshops and other events related to cloud computing. 10

11 OCC Members Current Members Aerospace, Cisco, MIT Lincoln Labs, Northwestern University, Open Data Group, Sector Project, University of Chicago, University of Illinois at Chicago and Yahoo. Current Contributing Members & Partners CalIT2, Johns Hopkins University, National Lambda Rail

12 OCC Structure OCC is a consortium operated by a not-for-profit. OCC is member-driven. OCC is oriented to companies and organizations, not individuals. Governance structure is available from web site. 12

13 OCC Working Groups Members set up working groups. 1. Large Data Clouds Working Group 2. Open Cloud Testbed Working Group. 3. Standard Cloud Performance Measurement (SCPM) 1. Working collaboratively with CCIF 4. Information Sharing and Security Working Group.

14 Part 3. Large Data Clouds Benchmarks and Interoperability Working Group * *OCC is not affiliated with the Hadoop Project. 14

15 Cloud Benchmarks To help inform purchase decisions. To help inform the design of cloud applications. To help inform cloud architecture decisions.

16 Benchmarks for Large Data Clouds Until recently, the only benchmark used was Terasort (sorting 10 billion 100 byte records) Replaced by Gray Sort and Minute Sort Gray Sort tries to maximize TB / min sorted on 100 TB or more of data. Hadoop holds the current Gray Sort and Minute Sort records. Problem: sort is just one of the types of work load for analytic applications

17 MalStone MalGen generates synthetic data with realistic distributions. MalStone A & B stylized computations that can be used as benchmarks for architectures, software and systems for large data clouds. Open source and available at code.google.com/p/malgen 17

18 MalStone B sites entities d k-2 d k-1 d k time 18

19 MalStone B Benchmark MalStone B Hadoop v min Hadoop Streaming v min Sector v min # Nodes 20 nodes # Records 10 Billion Size of Dataset 1 TB 19

20 ThriftStore Cloud App 1 Cloud App 2 App n ThriftStore Hadoop DFS KFS Sector DFS Each cloud application can access any of the large data cloud storage services without changing any of its APIs. Based upon the open source Thrift project.

21 Part 4. Standard Cloud Performance Measurement (SCPM) Working Group 21

22 Establish Benchmarks Appropriate for Four Use Cases 1. Moving an application between two clouds. 2. Obtaining burst instances from multiple cloud service providers for a private/public hybrid application. 3. Moving a large data cloud application to another large data cloud storage service. 4. Moving a large data cloud application to another large data cloud compute service.

23 Hierarchy of Benchmarks Benchmarks for basic operations Kernel Benchmarks Basic functions and routines Stripped down versions of programs Full applications Goal is to develop small suite of these benchmarks for four use cases.

24 Please Get Involved The Standard Cloud Performance Measurement (SCPM) Working Group is just starting, please get involved.

25 Part 5. OCC Sponsored Events 25

26 OCC/CCIF Jointly Sponsored Events International Summit on Governmental Cloud Computing Ottawa's Westin Hotel on October 6, 2009 Cloud Computing and Its Applications 2009 Chicago on October 20-21, 2009

27 We Welcome New Members Companies and organizations are welcome to join the Open Cloud Consortium (OCC) Join one of our working groups Large Data Clouds Working Group Standard Cloud Performance Measurement (SCPM) Working Group Information Sharing & Security Working Group