VISION Cloud: Data-intensive Storage Services on Clouds. OGF 35 June 17-19, 2012 Delft, Netherlands. Dimosthenis Kyriazis

Transcription

1 OGF 35 June 17-19, 2012 Delft, Netherlands VISION Cloud: Data-intensive Storage Services on Clouds Dimosthenis Kyriazis National Technical University of Athens

2 Data Deluge: The Emerging Zettabyte Age CAGR Entities (67x) (# files/containers) Information (44x) Storage (30x) Source: IDC 2010 Digital Universe Study IT Professionals (1.4x) 2020 Source: 2

3 Growth of Unstructured Data is a Key Driver for Storage Clouds Some statistics Unstructured data accounts for more than 90% of the digital universe* Medical images may be up to 30% of the world s storage** According to the IDC, by 2020, 15% of all digital info will be in a cloud and another 20% will be touched by the cloud Clouds hold a large portion of this storage Amazon S3 holds more than 900 billion objects (1Q12)*** YouTube streams 4B videos daily**** Source: IDC's Digital Universe Study, sponsored by EMC, June 2011 *The Exploding Digital Universe, 2008, and Digital Universe Study, April 2010, June 2011 ** *** **** 3

4 Adopting cloud storage solutions? 86% [Above the Clouds: A Berkley View of Cloud Computing] [L. Willcocks, W. Venters, E. Whitley - Accenture] [Randy Bias VP Technology Strategy of GoGrid, ServePath] [CeBIT 2011] [M. Malek - Google] 4

5 VISION Cloud: Virtualized Storage Services Foundation for the Future Internet Goal Architect and implement an infrastructure for the reliable and effective delivery of data-intensive storage services, facilitating the convergence of ICT, media and telecommunications Facts A 3-year project, started Oct M (total budget all partners) 5

6 Enabling data-intensive storage services on clouds (1/2) Data Mobility and Federation A layer enabling unified access to data across storage clouds and federation of data objects maintained by users across different administrative domains Raise the Abstraction Level of Storage A new data model enabling the encapsulation of storage into objects with user-defined and system-defined attributes / metadata used for handling both an object and its content Content-Centric Storage Access Access to data objects through information about their content and its relationships (build content networks) rather than details of underlying storage containers 6

7 Enabling data-intensive storage services on clouds (2/2) Computational Storage Computational agents (storlets) released into the cloud and activated by events on data. The storlets define not only the computation but also triggering conditions whereby storlets are activated Advanced Capabilities for Cloud-based Storage Support multi-tenancy, guarantee secure and authorized access to the data and services, check compliance with standards and regulations, optimize service provision with regard to cost, monitor, analyze and manage SLA considering content-related terms 7

8 Emphasizing on Data Mobility Goal: Single view of storage across multiple providers Barriers are both technological and business-related VISION Cloud is focusing on techniques for data deduplication data transfer technologies Proposed architecture A layer that enables unified access to data across storage clouds, and federates sets of data objects maintained by users across different administrative domains Built in network optimizations to overcome limitations by employing new mechanisms, extending lazy copying techniques, network deduplication and network overlays Allows leveraging of the information captured by metadata for executing data transfers 8

9 The VISION Cloud Architecture Data Access Management / Control Access and Interface Layer Data Access Layer (DAL) Management Interface Layer (MIL) Operating Layer Data Operating Layer (DOL) Management Operating Layer (MOL) DATA SERVICE Content networks/objects, Computation on storage, Mobility, availability, reliability, security MANAGEMENT SERVICE Monitoring, Metering, Billing, Security management, Tenant/User management, SLA

10 Unified Management Model

11 Use Cases Multi-tenant cloud-native business applications Healthcare data access based on content and its relationships Mobile users accessing content Create, repurpose & distribute content through multiple channels

12 Data: storing, retrieving, associating, analyzing, distributing 5 Million TB of data on the internet, Google has indexed only the 200TB (0.004%) 12

13 Session Overview - Agenda VISION Cloud - Towards Interoperable Storage Clouds A media use case for storage clouds (Mirko Lorenz - Deutsche Welle) Use of CDMI & challenges (Alex McDonald, SNIA Cloud Storage Initiative) Highlighting challenges on Federation & Interoperability for data storage cloud (Massimo Villari - University of Messina) Logging management for storage cloud compliance (Andrea Manieri - Engineering) Discussion / roundtable on VISION Cloud interoperability issues and how OGF can address them 13

14 Thank You Visit our Website: Dimosthenis Kyriazis National Technical University of Athens

15 Backup

16 Content Centric Interface Basic Object Interface Object Relations Interface Metrics Object Interface Content Centric Interface Metadata Upload Interface Operational Statistics Interface e.g. put object e.g. define relations e.g. search for e.g. import of large e.g. how many coordinates metdata sets downloads VISION Cloud 16

17 Storlet Life Cycle and States

18 Physical Model 100s of Data Centers (DC) Each DC 10s of storage clusters Each storage cluster 100s of servers with direct attached disks Data Center Data Center Storage Cluster 1 Storage Cluster 2 Storage Cluster 3 Data Center Data Center

19 Data Access and Operating Layers Application requests Communicate with peer in other clusters Data Access Layer Request Handler Computational Storage Service Global View Content Centric Service Secure Access Service Global Catalog Data Operating Layer Secure Access Control Component User Services Identity/Policy Services Resource Map Replicator Object Service Storlet Execution Communicate with peer in other clusters Catalog Storage Service Access Enforcer Logically global components Components that have an independent instance on each cluster node Logically global components

20 Secure Access Management Interface and Operating Layers Customer Developer Service API Management Interface Layer Tenants, subtenants and user accounts management SLA management Models development Global View Compliance Accounting and Billing Containers management Resource Map Global Catalog User Services Monitor KB Management Operating Layer Identity/Policy Services Analysis SLA Management DM Placement and Allocation

21 Logical View of Storlet Execution Environment

22 We deduplicate content within a tenant while ensuring there are replicas at each desired location Tenant j s logical view