Storage in Cloud Environments Cloud Storage Storage Cloud

Transcription

1 Dietmar Noll IBM Storage Software Development 11 March 2010 Storage in Cloud Environments Cloud Storage Storage Cloud

2 Objectives Understand the role of storage in cloud environments Understand the storage-related requirements / goals in a cloud, both from the consumer and from the provider perspective Know the different solution approaches available today to implement storage in cloud environments Understand the functionality needed for managing storage in the cloud 2 Storage in Cloud Environments Dietmar Noll [email protected]

3 Introduction Requirements Implementation Options Storage Management Aspects 3 Storage in Cloud Environments Dietmar Noll [email protected]

4 Introduction 4 Storage in Cloud Environments Dietmar Noll [email protected]

5 Cloud Goals First and foremost, clients want a lower cost option They also want to only pay for what they use. And they want to get it fast, when they need it. Finally, they want it to be really easy to manage. Monitor & Manage Services & Resources Datacenter Infrastructure Access Services Service Consumers Cloud Administrator IT Cloud Service Catalog, Component Library Component Vendors/ Software Publishers Publish & Update Components, Service Templates 5 Storage in Cloud Environments Dietmar Noll [email protected]

6 Why is storage so relevant in the cloud environment? Data Growth is Exponential 128 GB/ person The World s total data per person. Digital Information Created, Captured, Replicated WW 2006: 180 exabytes 2007: 280 exabytes : 1800 exabytes (1800 billion gigabytes) Variety of Information Information Technology holds the promise of bringing a variety of new types of information to the people who need it Volume of Data Data is growing exponentially 24 GB/ person Expected compound annual growth rate is almost 60% Velocity of Change Acquisitions Mergers 0.8 GB/ person Sources: IDC, Worldwide Disk Storage Systems Forecast Update, Doc # IDC Whitepaper: The Diverse and Exploding Digitall Universe, March 2008 Consolidations ILM, Data Retention initiatives 6 Storage in Cloud Environments Dietmar Noll [email protected]

7 Why is storage so relevant in the cloud environment? The Information Tidal Wave Continues... Hard Drives are CHEAP! $96.93 $58.27 $34.90 External disk shipments & price (History & Forecast) $ $ $11.11$7.71 $5.22 $3.48 $2.30 $1.51 $ Source: IDC 2008 DC, PB $/GB Yes, it s a good thing that the price of disk drives keeps coming down because we re going to need a lot more of them to hold all that data. Buy more disk? More floor space? More power requirements? More people? More skills? More technologies? What s our strategy? Leverage an integrated management approach, providing smart support to better utilize the storage environment 7 Storage in Cloud Environments Dietmar Noll [email protected]

8 Globally, storage requirement is 80% file-based unstructured data, and growing Worldwide Storage Capacity Shipped by Segment, Explosion of data, transactions, and digitally-aware devices strains IT infrastructure and operations. Storage capacity is doubling every 18 months. Majority of this data is unstructured filebased, such as user files, medical images, web and rich media content, growing at 63% Source: IDC, State of File-Based Storage Use in Organizations: Results from IDC's 2009 Trends in File-Based Storage Survey: Dec 2009: Doc # Block storage, while still well suited for existing OLTP/database workloads, is not where majority of strategic analytics-based applications and strategic storage initiatives are being deployed 8 Storage in Cloud Environments Dietmar Noll [email protected]

9 Primary Workloads for Storage Clouds Collaboration Data & General File Storage Digital Media Web Content Store High Performance Analytics Energy & Geo- Sciences CAE General purpose file storage environments where clients are challenged with the manageability of current NAS systems High performance, simplified management for widely varying use cases in digital media environments. Hyper-scalable storage for large Web 2.0 stores and for other vendors looking to build their own Cloud/SaaS applications Business applications such as financial services interested in cloud deployments with single namespace Energy exploration and geo-sciences require huge addressable namespaces and very high performance. Auto / Aero / Electronics design processes experiencing rapid file-centric storage growth as simulation expands. Retail Banking & Financial Markets Chemical & Petroleum Healthcare 9 Storage in Cloud Environments Dietmar Noll [email protected]

10 Cloud Computing Reference Architecture Overview Cloud Storage Focus Cloud Service Consumer Partner Clouds Consumer In-house IT Cloud Service Provider Cloud Services IT capability provided to Cloud Service Consumer (Virtualized) Infrastructure Server, Storage, Network, Facilities Infrastructure for hosting Cloud Services and Common Cloud Management Platform Common Cloud Management Platform BSS Business Support Services Business-level functionality for management of Cloud Services OSS Operational Support Services Operational-level functionality for management of Cloud Services Security & Resiliency Cloud Service Developer Service Development Tools File Storage Systems Block Storage Systems Backup Systems Archiving Systems SAN Connectivity Storage Provisioning Storage Monitoring and Event Management Capacity and Performance Management Storage Virtualization Management Data Lifecycle Management 10 Storage in Cloud Environments Dietmar Noll

11 Requirements 11 Storage in Cloud Environments Dietmar Noll

12 Storage-related Requirements Consumer Perspective Pay as you go Reduce Costs Reduce CAPEX and OPEX Increase Efficiency through virtualization and optimization Flexible sourcing Elasticity Manage Risk Security Resiliency Compliance Simplicity Improve Service Higher Quality Higher Availability Higher Flexibility 12 Storage in Cloud Environments Dietmar Noll

13 Storage-related Requirements Consumer Perspective Reduce Costs Reduce CAPEX and OPEX Increase Efficiency through virtualization and optimization Flexible sourcing Manage Risk Security Resiliency Compliance Improve Service Higher Quality Higher Availability Higher Flexibility Reduce Information Management Costs Reduce/avoid investments in storage HW and SW Leverage latest virtualization technology and efficient processes Avoid vendor dependencies Manage Storage-related Risks Ensure physical and logical security Leverage backup and disaster recovery skills and technologies Ensure information retention periods and accessibility Improve Storage Services Provide service based on application requirements Leverage storage availability (HA) skills and technologies Quickly react on requests without dependency on own HW/SW. 13 Storage in Cloud Environments Dietmar Noll

14 Storage-related Requirements Provider Perspective Multi-Tenancy Provide appropriate management separation for the storage environment while still allowing integrated management of all storage aspects Security Prevent (logically and physically) unauthorized access or even modification to data stored in the environment High Availability Ensure the data can be accessed according to the needs at any time Utilization Optimally leverage the assets available in the infrastructure to avoid unnecessary investments Monitoring Achieve situation awareness about the health of the environment and react on events in the environment 14 Storage in Cloud Environments Dietmar Noll

15 Storage-related Requirements Provider Perspective - continued Metering Capture information about the environments capabilities (storage capacity, performance, etc.) and usage and keep historical records of it Reporting Allow visualization and analysis of the data collected about the storage environment to be used for regular reports, problem determination, etc. Planing Combine the information about the current and historic environment as well as other sources to plan future changes to the environment to optimize ROI Automation Achieve a high level of automation of configuration tasks to minimize OPEX. 15 Storage in Cloud Environments Dietmar Noll [email protected]

16 Defining Requirements for Storage Services Service Level Categories Service Level Objectives Accessability Initial Access Time Data Sharing Requires Access Transparency May Out Of Space Duration Availability Availability Period Planned Downtime Max. Unplanned Downtime Aggregate Max Unplanned Downtime Per Instance Recovery Point Objective (RPO) Recovery Time Objective (RTO) Consistency Number of Copies Number of Versions Retain Deleted Performance Avg. I/O Rate Avg. Data Throughput Retention / Compliance Immutability Disposal Durability Retention Period Security Accountability Integrity Authenticity Confidentiality Physical Security 16 Storage in Cloud Environments Dietmar Noll [email protected]

17 Defining Requirements for Storage Services Service classes / SLA templates to allow simple classification Service Class Platinum Service Class Platinum Service Class Gold Service Class Gold Service Class Silver Service Class Silver Accessability Accessability Availability Availability Performance Performance Consistency Consistency Retention / Compliance Retention / Compliance Security Security Service Class Bronze Service Class Bronze 17 Storage in Cloud Environments Dietmar Noll [email protected]

18 Implementation Options 18 Storage in Cloud Environments Dietmar Noll

19 Storage Approaches Block Storage File Storage DAS SAN NAS Block Storage File Storage DAS SAN NAS ATA SATA SAS SCSI FC Network FCP FCoE iscsi LAN NFS SMB/CIFS FTP SCP 19 Storage in Cloud Environments Dietmar Noll

20 Traditional Storage Overview Type/Technology Benefits Constraints Application Media CD/DVD/Diskettes cheap, convenient, portable, compatible, long life,. low capacity, moderate speed,.. Archiving, distribution, migration, local sharing,. HD - Internal/External most common form of storage, high speed limited capacity, local environment storage in single computer Tape low cost, portability, unlimited capacity Slow/uneasy recovery of individual files/groups of files data archiving, low-budget businesses, offsite storage DAS - Direct-Attached Storage simplicity, low initial cost, ease of management individual server, admin, and data transfer in network env. Data sharing, backup, archiving, app sharing Disk Library high speed, capacity, availability not quickly accessible as DAS "write once, read rarely Data RAID - Redundant Array of Independent Disks high speed, capacity, availability, reliability security & fault tolerance recovery could be difficult high cost for system optimization swap files Internet Service Providers redundant storage SAN - Storage Area Network large block data storage Reliability, availability, fault tolerance, scalability high cost lack of standardization management complexity large databases applications need bandwidth mission-critical applications NAS Network-Attached Storage multi-client fast file access easy file sharing, replication, redundancy, consolidation Less convenient than SAN for moving large blocks of data data backup data archiving redundant storage Fibre Channel for data transmission in SAN gigabit speed large data xfer flexible distance btw devices high cost management complexity see SAN iscsi for IP-based data transmission in SAN IP protocols longer distance than Fibre Slower than Fibre management complexity see SAN 20 Storage in Cloud Environments Dietmar Noll [email protected]

21 Storage in the Cloud Implementation Approaches and Management OSS Integration between different management domains to coordinate operations Infrastructure Computer Management System Mgmt. Path Applications Operating System Computer Virtualized Computer Management Path Data Path Different interfaces used for application level / infrastructure level data access Storage Management System Block Storage Mgmt. Path Additional management path needed to give applications/operating systems control over certain aspects of storage management 21 Storage in Cloud Environments Dietmar Noll [email protected] File Storage Data Services

22 Data Management in Cloud Storage Cloud Storage Storage Management System Primary copy of data managed, which the application is using during its normal operation. Mirror Copies Backup Copies Meta Data & Policies Supplemental data about the data managed as well as policies related to the management of this data Online Storage Multiple copies managed by the cloud to fulfill SLAs Archive Copies 22 Storage in Cloud Environments Dietmar Noll

23 Storage Technologies in Cloud Storage Storage Virtualization Source: Storage Networking Industry Association, SNIA 23 Storage in Cloud Environments Dietmar Noll

24 Storage Technologies in Cloud Storage Block Storage Virtualization Large number of direct connections make management complex and inefficient Virtualized environment reduces dependencies and increases performance and availability Virtualizer needs to have appropriate scalability and redundancy characteristics Distinct storage systems mean separation of resources, which leads to imbalanced utilization Changes to the storage systems have direct impact on the storage consumers, potentially causing downtime Virtualized Block Storage Virtualization hides complexity of the environment to ease management from a consumer perspective, increase utilization and decrease dependency 24 Storage in Cloud Environments Dietmar Noll [email protected]

25 Block Storage Virtualization IBM SAN Volume Controller How does it work? Logical Volumes Virtual Disks MDisks Groups Unmanaged Managed Disks Backend Volumes 25 Storage in Cloud Environments Dietmar Noll

26 Block Storage Virtualization IBM SAN Volume Controller Dynamic Infrastructure Support Dynamically scale performance For performance sensitive applications, dynamically add more performance to your existing capacity by adding controller pairs or mix with additional capacity Dynamically scale capacity For high capacity applications such as archive, dynamically add capacity by adding disk enclosures or mix with additional performance Scale capacity in tiers Implement a tiered storage infrastructure with common management interfaces and software functions 26 Storage in Cloud Environments Dietmar Noll [email protected]

27 Storage Technologies in Cloud Storage File Storage Virtualization Equivalent approach to achieve the same goals as seen for block storage: increased utilization, reduced dependencies, improved availability and performance Additional complexity due to high number of entities, identity management and high dynamics Virtualized File Storage Same problems as seen for block storage: high dependency/complexity, reduced utilization/availability 27 Storage in Cloud Environments Dietmar Noll

28 File Storage Virtualization IBM SoNAS How does it work? /home/appl/data/web/important_big_spreadsheet.xls /home /appl IBM Scale Out NAS Policy Engine /data Storage nodes Interface nodes /web Global Namespace /home/appl/data/web/big_architecture_drawing.ppt /home/appl/data/web/unstructured_big_video.mpg Interface Interface nodes nodes >.. Storage.. Storage... nodes nodes > scale out scale out Note: all three files, in same directory, but each allocated to different physical storage pool Data striped across all disks in storage pool. High performance, auto-tuning, autoload balancing Logical Physical Tier 1: SAS drives Tier 2: 1TB SATA drives Tier 3: 2TB SATA drives 28 Storage in Cloud Environments Dietmar Noll

29 File Storage Virtualization IBM SoNAS Dynamic Infrastructure Support Dynamically scale performance For performance sensitive applications, dynamically add more performance to your existing capacity by adding controller pairs or mix with additional capacity Dynamically scale capacity For high capacity applications such as archive, dynamically add capacity by adding disk enclosures or mix with additional performance Scale capacity in tiers Implement a tiered storage infrastructure with common management interfaces and software functions 29 Storage in Cloud Environments Dietmar Noll [email protected]

30 Storage Cloud Offering Examples Name Google Docs Web providers Web Digital Image sites Facebook Amazon S3 Nirvanix SDN VMWare & Dell IBM Smart Business Storage Cloud Description Allows users to upload documents, spreadsheets and presentations to Google's data servers. Users can edit files using a Google application. Users can also publish documents so that other people can read them or even make edits. like Gmail, Hotmail and Yahoo! Mail store messages on their own servers. Users can access their from computers and other devices connected to the Internet. Sites like Flickr and Picasa host millions of digital photographs. Their users create online photo albums by uploading pictures directly to the services' servers. Social networking sites like Facebook and MySpace allow members to post pictures and other content. All of that content is stored on the respective site's servers. A storage service that let low to high end subscribers store, retrieve, and share data objects that are application agnostic. Web Service API access. Customer examples include saving personal data to Web startups, offering online services, back up databases or store archival data. Managed cloud storage for the enterprise. Stores, delivers, and processes storage requests in system selected location of network of storage nodes. CloudNAS application and Web Service API access. Customer examples include off-site data protection, Tier-N storage, distributed content and collaboration, and embedded storage Integrated compute and storage infrastructures for cloud services. Virtualze Storage within Cloud Computing Infrastructures using VMware vsphere 4.0 and Dell EqualLogic iscsi SANs Scalable NAS solution provided a client and its partner network. Customers store, retrieve, and share information in a private environment that drives efficiency, standardization and best practices while retaining greater customization and control. Application Cloud Storage Cloud Storage Service Cloud Storage Infrastructure 30 Storage in Cloud Environments Dietmar Noll [email protected]

31 Storage Management Aspects 31 Storage in Cloud Environments Dietmar Noll

32 Integration of Storage Management into Service Management Visibility of. all elements and services in the storage infrastructure 32 Storage Infrastructure Topology Visualization Storage Infrastructure Reporting Storage Infrastructure Health / Event Monitoring Control of. changes to the storage infrastructure IBM CONFIDENTIAL Consistent execution of provisioning operations Raise alerts based on events and conditions in the infrastructure Continuous configuration checking Automation of. common actions in the storage infrastructure to reduce administration costs Smart selection of target entity of control provisioning operations Continuous optimization of storage infrastructure Problem Determination (Root Cause Suspect Identification) 32 Storage in Cloud Environments Dietmar Noll [email protected]

33 Storage Management in the Cloud Functional Areas Discovery & Categorization Identify entities in storage environment and collect information about them Identify capabilities based on data collected Change Management Monitoring & Problem Determination Receive events from storage infrastructure and evaluate for relevance / importance Constantly check monitored entities for abnormal conditions and SLA violations Support identification of problem root causes and corrective actions Reporting & Metering Provide access to information about storage infrastructure entities and items Correlate data to allow metering at various levels, like per application, per user, etc. Keep historical data to allow usage based charging Provisioning & Optimization Support autonomic provisioning of storage resources Provide intelligence to de-couple service request from storage infrastructure, e.g. TPC planners Continuously optimize storage infrastructure for efficient SLA-compliant resource usage 33 Storage in Cloud Environments Dietmar Noll [email protected]

34 Storage Management in the Cloud Direct Functional Coverage Cloud Service Consumer Cloud Service Provider Cloud Service Developer Business-Process-as-a-Service Consumer End user Cloud Services User Interface API Software-as-a-Service Platform-as-as-Service Infrastructure-as-a-Service Developer Partner Clouds Consumer Business Manager Customer In-house IT Virtualized Infrastructure Server, Storage, Network, Facilities Common Cloud Management Platform Service Delivery Portal API BSS Business Support System Offering Mgmt Order Mgmt Accounting & Billing Contract Mgmt OSS Operational Support Service Templates System Service Request Management Provisioning Monitoring & Event Management Customer Mgmt Entitlements Invoicing SLA Reporting Metering, Analytics & Reporting Service Delivery Catalog Service Automation Management Configuration Mgmt Incident, Problem & Change Management IT Asset & License Management Virtualization Mgmt Consumer Administrator Components with red Service Provider Portal frame are directly Service Business Manager Service Transition Manager related to functional coverage for storage Security & Resiliency management 34 Storage in Cloud Environments Dietmar Noll Pricing & Rating Subscriber Mgmt Peering & Settlement Service Offering Catalog Image Lifecycle Management IT Service Level Management Capacity & Performance Management Service Development Portal Service Operations Manager Service Security Manager Service Development Tools Service Definition Tools For the other components, the storage management contributes to some extent, but there is no direct coverage. Image Creation Tools

35 Storage Management in the Cloud Levels of Context Infrastructure Performance Data & Event Information Asset & Configuration Data Infrastructure Entity Service Instance Best Practices, Compliance Rules & Policies Storage management in the cloud needs to relate information about the current infrastructure with the targets for the infrastructure This has to happen at various levels of context to realize visibility, control and automation concurrently This knowledge of the environment and the requirements/goals is applied to support the service management during provisioning as well as reporting/monitoring/metering. 35 Storage in Cloud Environments Dietmar Noll

36 Storage in Cloud Environments Summary Almost any non-trivial cloud-based workload also needs appropriate storage available, although in general the cloud consumer should not consciously notice the complexity in storage management. The storage management has to be closely integrated into the overall service management to achieve cloud goals from a provider perspective. Most of the cloud-based workload will be file storage or application storage/storage service, but block storage is also needed for certain workloads and storage management needs to expose the same level of functionality for all approaches. Technology for implementing the I/O path is available today, while the integrated and consolidated storage management still needs to evolve. 36 Storage in Cloud Environments Dietmar Noll [email protected]

37 Thank you! For more information, please visit: ibm.com/cloud Or contact me at: 37 Storage in Cloud Environments Dietmar Noll