Cloud Storage: Key to Future of IT Infrastructure Anil Vasudeva President & Chief Analyst imex@imexresearch.com 408-268-0800
Agenda (1) The Need for Cloud Storage Data Centers Infrastructure IT Roadmap to Clouds Cloud Computing Ecosystem & Architectural Stack Storage Usage in Data Centers & in Cloud Mega-Centers Applications mapped by Key Workload & Storage Metrics (2) Cloud Infrastructure Technology Enablers Virtualization: Key to Cloud Storage Infrastructure SSDs: A New SCM Filling Price Performance Gaps Networks: Fast March to 100Gb/sec Hadoop: Meeting Big Data/BI/Data Base Paradigm (3) Drivers for Cloud Storage Adoption Storage Efficiencies and Costs Reduction Tier 4 Storage for Data Protection Real-Time Online Analytics Technology Innovations HW, SW 2005-2015 (3) Cloud Segments: Challenges & Solutions Adopting Public/Private/Hybrid Clouds: Pros & Cons (4) Cloud Storage Requirements Adopting Public/Private/Hybrid Clouds: Pros & Cons Best Practices: Getting Ready for the Cloud (5) Key Take-Aways
Data Centers & Cloud Infrastructure Public CloudCenter Enterprise VZ Data Center On-Premise Cloud Vertical Clouds IaaS, PaaS SaaS Servers VPN Switches: Layer 4-7, Layer 2, 10GbE, FC Stg Supplier/Partners ISP ISP Internet ISP FC/ IPSANs Remote/Branch Office ISP ISP Core Optical ISP Edge ISP Caching, Proxy, FW, SSL, IDS, DNS, LB, Web Servers Database Servers, Middleware, Data Mgmt Cellular Wireless Web 2.0 Social Ntwks. Facebook, Twitter, YouTube Cable/DSL Home Networks Tier-1 Edge Apps Application Servers HA, File/Print, ERP, SCM, CRM Servers Tier-2 Apps Directory Security Policy Middleware Platform Tier-3 Data Base Servers Management 4 Source:: IMEX Research - Cloud Infrastructure Report 2009-12
IT Industry Journey - Roadmap IT Industry Roadmap IMEX Research Cloudization Integration/Consolidation On-Premises > Private Clouds > Public Clouds DC to Cloud-Aware Infrast. & Apps. Cascade migration to SPs/Public Clouds. Integrate Physical Infrast./Blades to meet CAPSIMS IMEX Cost, Availability, Performance, Scalability, Inter-operability, Manageability & Security Standardization Automation Automatically Maintains Application SLAs (Self-Configuration, Self-Healing IMEX, Self-Acctg. Charges etc) Virtualization Pools Resources. Provisions, Optimizes, Monitors Shuffles Resources to optimize Delivery of various Business Services Standard IT Infrastructure- Volume Economics HW/Syst SW (Servers, Storage, Networking Devices, System Software (OS, MW & Data Mgmt SW) Analytics BI Predictive Analytics - Unstructured Data From Dashboards Visualization to Prediction Engines using Big Data.
Cloud Computing Ecosystem Service Providers Cloud Services Providers Public Mutitenancy,OnDemand Private - On Premises, Enterprise Hybrid Interoperable P2P Examples Public - BT, Telstra, T-Systems France Telecom Private Hybrid IBM/Cloudburst, SaaS Software-as-a-Service - Servers, Network, Storage - Management, Reporting Examples email - Yahoo!,Google Collaboration - Facebook,Twitter Bus.Apps - SalesForce, GoogleApps, Intuit PaaS Platform Tools & Services - Deploy developed platforms ready for Application SW on Cloud Aware Infrastructure Examples Amazon EC2 Force.com Navitaire IaaS Infrastructure HW & Services - Servers, Network, Storage - Management, Reporting Examples Amazon S3 Nirvanix
EJB Ruby Python.Net PHP Management SLA SLA SLA SLA SLA Virtualized Cloud Infrastructure Private Cloud Enterprise Cloud Computing Hybrid Cloud Public Cloud Service Providers SaaS Applications SaaS App App App App App Platform Tools & Services PaaS.... Operating Systems IaaS Virtualization Resources (Servers, Storage, Networks) Application s SLA dictates the Resources Required to meet specific requirements of Availability, Performance, Cost, Security, Manageability etc.
% of I/O Accesses SSD 100% 95% Data Storage Usage In Corporate Data Centers I/O Access Frequency vs. Percent of Corporate Data 75% 65% Cache Logs Journals Temp Tables Hot Tables Disk Arrays Tables Indices Hot Data Tape Libraries Back Up Data Archived Data Offsite DataVault 1% 2% 10% 50% 100% % of Corporate Data Source:: IMEX Research - Cloud Infrastructure Report 2009-12 8
Applications Best Suited for Cloud Temporary Storage Test & Dev Data, Excess Storage from Data Centers Internet Web Content Web 2.0 user-generated content, Digital Catalogs, Public Files, iphone App Content Data Archiving & Records Retention Regulatory compliance, Corporate governance, Audits & searches, Legal, Intellectual Property, Risk Management Data Back Up & Restore Offsite data protection, Historical Versioning, Online Access, Automated Retrieval Intranet File Storage Employee Portals, Digital Libraries, Document Collaboration, Training Media, File Shares, Historical Versioning, Online Access, Automated Retrieval Source: IMEX Research SSD Industry Report 2010-122 9
% of I/O Accesses NextGen Data Storage Usage in Cloud I/O Access Frequency vs. Percent of Corporate Data 95% 75% 65% SSD Logs Journals Temp Tables Hot Tables FCoE/ SAS Arrays Tables Indices Hot Data Cloud Storage SATA Back Up Data Archived Data Offsite DataVault 1% 2% 10% 50% 100% % of Corporate Data Source:: IMEX Research - Cloud Infrastructure Report 2009-12 10
Market Segments by Applications IOPS* (*Latency-1) 1000 K 100 K 10K 1K Transaction Processing (RAID - 1, 5, 6) OLTP ecommerce Data Warehousing Business Intelligence OLAP Scientific Computing (RAID - 0, 3) HPC 100 TP HPC Imaging Audio Video Web 2.0 10 1 5 10 50 *IOPS for a required response time ( ms) *=(#Channels*Latency-1) Source:: IMEX Research - Cloud Infrastructure Report 2009-12 100 MB/sec 500
Cloud Storage Requirements Data Protection Back Up/Archive/DR RAID 0,1,5,6,10 Virtual Tape Replication Storage Efficiency Auto Tiered Storage Storage Virtualization Thin Provisioning Deduplication MAID DRAM Flash SSD Performance Disk Capacity Disk Tape Auto Tiering System using SSDs Data Class (Tiers 0,1,2,3) Storage Media Type (Flash/Disk/Tape) Policy Engines (Workload Mgmt) Transparent Migration (Data Placement) File Virtualization (Uninterrupted App.Opns.in Migration) Source: IMEX Research SSD Industry Report 2011
Workloads Consolidation using VZ A single server 1.5x larger than standard 2-way server will handle consolidated load of 6 servers. VZ manages the workloads + important apps get the compute resources they need automatically w/o operator intervention. Physical consolidation of 15-20:1 is easily possible Reasonable goal for VZ x86 servers 40-50% utilization on large systems (>4way), rising as dual/quad core processors becomes available Savings result in Real Estate, Power & Cooling, High Availability, Hardware, Management Source: Dan Olds & IMEX Research 2009
For TCO Analysis, EM: imex@ imexresearch.com (408) 268-0800 Cost over 3 years TCO Savings with Virtualization $16,000 995 Pre-Virtualization (VZ) Servers 78 VZ Servers $12,000 $8,000 Provisioning Downtime Disaster Recovery DC Real Estate Power & Cooling Network SAN Hardware $4,000 $- w/o VZ w VZ VZ SW & Support
Cloud Storage Requirements Data Protection Back Up/Archive/DR RAID 0,1,5,6,10 Virtual Tape Replication Storage Efficiency Auto Tiered Storage Storage Virtualization Thin Provisioning Deduplication MAID DRAM Flash SSD Performance Disk Capacity Disk Tape Auto Tiering System using SSDs Source: IMEX Research SSD Industry Report 2011 Data Class (Tiers 0,1,2,3) Storage Media Type (Flash/Disk/Tape) Policy Engines (Workload Mgmt) Transparent Migration (Data Placement) File Virtualization (Uninterrupted App.Opns.in Migration)
SSD Technology Filling Price/Perf Gaps Price $/GB CPU SDRAM NAND NOR SCM DRAM PCIe SSD DRAM getting Faster (to feed faster CPUs) & Larger (to feed Multi-cores & Multi-VMs from Virtualization) Tape HDD SATA SSD HDD becoming Cheaper, not faster SSD segmenting into PCIe SSD Cache - as backend to DRAM & SATA SSD - as front end to HDD Source: IMEX Research SSD Industry Report 2010-12 Performance I/O Access Latency 16
IOPS $/IOP Cost $K SSD in SAN Storage - Perf & TCO SAN Storage SAN Performance & TCO Improvements using SSD 300 250 200 150 100 50 0 Improvements using SSD $/IOPS Improvement 800% FC-HDD Only IOPS Improvement 475% SSD/SATA-HDD 10 9 8 7 6 5 4 3 2 1 0 250 200 150 100 50 0 SAN TCO using HDD vs. Hybrid Storage 145 0 75 14.2 HDD Only HDD-FC HDD- SATA SSD RackSpace Pwr/Cool 0 36 64 28 5.2 HDD/SSD Performance (IOPS) $/IOP Power & Cooling RackSpace SSDs HDD SATA HDD FC Source: IMEX Research SSD Industry Report 2010-12 17
% of Drives Failing (AFR %) SSD Challenges & Solutions: Goals & Best Practices 20% 15% 10% 5% 0% 0 1 2 3 4 Years of Use 5 JEDEC SSD Std. <=3% Concerned about SSD Adoption in your Enterprise? Be aware of Tools & Best Practices And you should be OK!! Best Practices By leveraging Error Avoidance Algorithms, and Best Practices of Verification Testing, to keep total functional failure rate <=3% (with defects and wear-outs issues combined) In practice, endurance ratings are likely to be significantly higher than typical use, so data errors and failures will be even less. Capacity Over-provisioning will provide large increases in random performance and endurance. Select SSD based on confirmed EVT Ratings Use MLC within requirements of Endurance Limits Using Best-of-Breed Controllers to achieve <=3% AFR and JEDEC Endurance Verification Testing should allow Enterprise Capabile SSDs Source: Intel IDF 10 & IMEX Research SSD Industry Report 2011 IMEX 2010-12
I/O Forensics for Auto Storage-Tiering Storage-Tiered Virtualization Storage-Tiering at LBA/Sub-LUN Level Physical Storage Logical Volume SSDs Arrays Hot Data HDDs Arrays Cold Data LBA Monitoring and Tiered Placement Every workload has unique I/O access signature Historical performance data for a LUN can identify performance skews & hot data regions by LBAs Automatic Migration (Policy Based) Source: IBM & IMEX Research SSD Industry Report 2011 IMEX 2010-12 19
AutoSmart Storage-Tiering SW: Enhancing Database Throughput DB Throughput Optimization Every workload has unique I/O access signature and historical behavior identify hot database objects and smartly placed in the right tier. Scalable Throughput Improvement - 300% Substantial IO Bound Transaction Response time Improvement - 45%-75% Productivity (Response Time) Improvement Using automated reallocation of hot spot data to SSDs (typically 5-10% of total data), significant performance improvements is achieved : Response time reduction of around 70+% or IOPS increase of 200% for any I/O intensive loads Verticals benefitting from Online Transactions : Airlines Reservations, Investment Banking Wall St. Stock Transactions Financial Institutions Hedge Funds etc. plus Low Latency seeking HPC Clustered Systems etc. Source: IBM & IMEX Research SSD Industry Report 2011 IMEX 2010-12 20
Big Data Driving New Storage Data Stored by Large US Enterprises Big Data Storage Potential Data Stored by Large US Enterprise Stored Data by Industry (in US 2009 PB) Discrete Manafacturing Government Communications and Media Process manufacturing Banking Health Care Providers Securities & Investment Srvcs Professional Services Retail Education Insurance Transportation Wholesale Utilities 4% 4% 3% 3% 3% Big Data Cloud Storage Potential Data Stored by Large US Enterprises 5% 6% 6% 6% Resource Industries 2% 1,792 Consumer and Recreational 2% 1,312 Services Construction 1% 967 9% 10% 10% 12% 14% 150 Stored Data TB/Firm (>1K Employees US) 231 319 278 370 536 825 801 697 870 831 1,507 1,931 3,866
Harnessing Big Data for Business Insights Data Sources Big Data Infrastructure Business Insights Information is at the center of New Wave of opportunity Majority of data growth is being driven by unstructured data and billions of large objects 80% of world s data is unstructured driven by rise in Mobility devices, collaboration machine generated data.
Type of Data Organizations Analyze Types of Data Organizations Analyze Customer/member data Transactional data from applications Application Logs Other Types of Event Data Network Monitoring/Network Traffic Online Retail Transactions Other Log Files Call Data Records Web Logs Text data from social media and online Search logs Trade/quote data Intelligence/defense data Multimedia (audio/video/images) Weather Smartmeter data Other (please specify) 9% 8% 3% 3% 6% 3% 5% 15% 11% 18% 15% 18% 11% 23% 21% 21% 33% 33% 34% 26% 28% 32% 37% 36% 36% 41% 44% 46% 51% 59% 64% Hadoop 68% 68% 69% Non Hadoop
Data: IMEX Research & Panasas NextGen Drivers: Big Data Cloud Smart Mobile Devices Commercial Visualization Bio/Health Analytics Predictive Bus. Intelligence Entertainment- On Demand 24 Instant On Boot Ups Rugged, Low Power 1GB/s, ms Rendering (Texture & Polygons) Very Read Intensive, Small Block I/O Data Warehousing Random IO, High OLTPM Most Accessed Videos Very Read Intensive 10 GB/s, ms 10 GB/s, ms 4 GB/s, ms
Big Data Architectural Goals Meet Requirements of V3 Meet Enterprise Criterion Big Data Platform Analyze Data in Native Format
Rise of Big Data Analytics
Demand for Real Time Analytics
Technology Innovations - HW
$ / TPMc TPMc / Processor Technology Innovations: DB SW Tech Innovation 1985 1990 1995 2000 2005 2010 2015 OLTP Transactions DB SW Rows Locking Optimizer Parallel Query Clustering XML Grid OpenSource / Hadoop OLAP- Analytics DB SW Indexing Partitioning Columnar Materialized View Bit Mapped Index In-Memory Query Binding Hardware 32 bit SMP NUMA 64 bit Multi-core/ Blades Flash MPP Big Data Multi-core Columnar In-Memory MPP Visualization OLTP Database Innovation Progress Big Data: Analytics DB Technology Impact 100000 10000 100,000 100000 10000 1000 100 10 1 0.1 $/TPMc TPMc/Processor 1985 1990 1995 2000 2005 2010 2015 1000 100 10 1 0.1 0.01 10,000 1,000 100 10 1.10.01 0 $/GB DW Size TB 1985 1990 1995 2000 2005 2010 2015 10000 1000 100 10 1 0.1 0.01 0.001 0.0001
Big Data Product Segments Big Data - Product Metrics Data Set Size Data Structure Access/Use Parallel Processing Memory DB Technique Data Cataloging SW PB TB GB Transaction Machine Unstructured Other Transaction Search Analytics Appliance Cluster < 1K Cluster > 1K In-Memory Flash Columnar Zero Sharing No SQL Text Image Audio Video
Big Data Ecosystem
Big Data Stack BI Framework - Interoperable with Enterprise Data Warehousing EDW Big Data (Connectors) BI APPLICATIONS (Query, Analytics, Reporting, Statistics) Big Data Orchestration Framework HBase Avro Flume ZooKeeper Big Data Storage Framework (HDFS) Big Data Access Framework Pig Hive Sqoop Network Big Data Processing Framework (MapReduce) Backup & Recovery Security Management Hypervisor/VMs Operating System Servers
Cloud Types - Benefits & Risks Private Cloud (On Premises) Public Cloud (Remote) Hybrid Cloud (Co-Oper) Characteristics IT acts as provider of services Available to Internal Users only Bidirectional Scalability Reservationless Auto Allocation of Resources Pay/Use Metering Mechanism Application & SW Agnostic Owned and delivered by 3rd Party Application on User Side & App Services and Resources on Cloud Srvcs Provider side Cloud Provider sells Srvcs to anyone, Data stored in Shared Facility serving multitenants - impact on Perf., BW, & Availability of Apps served. Agreed upon Cooperation between OnPremises user owned by User and Cloud Services Provider (CSP) on Cloud Side Onsite Gateway Appliance with abstraction layer between usr location & public cloud site of CSP Benefits Instant Provisioning Agility Full Physical Control & Security + User s access to Infrastructure Customizable Srvcs & Solns to meet User s specific Needs Predictable & Lower Costs solutions for large/midrange sites Lower Costs using Shared Offsite Stg Instant Provisioning Infinite Scalability Efficient Data Sharing by user particularly if info shared by users dispersed in multilocations Storage Costs CAPEX >> OPEX Improves Performance. by serving Hot I/Os On-Premise Deduped Data sent to Cloud > Lowers reqd. BW & Monthly Networking Costs Only Encrypted Data Sent > ensures security for MC Apps Risks Demands extra capacity be available for unforeseen demand in provisioning and management Building Private Cloud Infrast. difficult, requires new expertise New Cos.now moving as Private Cloud Infras. Builders vs CSPs Requires Monitoring, DR Solns, Change Mgmt, Testing, SLA/ Charge Back mechanisms etc. SLA managed as outsourced contract Full understanding of Performance, Availability, Costs required per App before migration Shared Infrastructure awareness for sensitive data despite encryption Infrastructures Failures/Hacked SensitiveData Exposure visible in Press Cloud Provider can abruptly exit Biz Lower HA than achievable in Pvt Cloud
Cloud Storage Requirements Expect more from your storage Hyper-efficient Storage Storage that analyzes, adapts, and improves performance Cloud ready storage, with features you can use today Technology & Best Practices enable storage efficiency Real-time Compression up to 80% less space Auto Tiering 3x more performance with just 2% solid-state Storage Virtualization up to 30%more utilization Thin Provisioning up to 35%more utilization Data Protection/HA: Fileset level snapshots and writeable file clones Capability for remote mirror File replication and file level snapshots for business continuity and DR Manageability: Unified SAN & NAS (block and file) storage system with a tightly integrated management console Support for NFS/CIFS/FTP/HTTPS/SCP file protocols Storage administration with a single user interface Policy-based management of files with user-defined policies!
Storage Costs Reduction Cloud Storage Issues & Solns. Virtualization (VZ) requires Shared Storage for - VMotion - Storage VMotion - HA/DRS - Fault Tolerance Additional Capacity Consumed for - VZ snapshots, - VM Kernel etc Following techniques Reduce Storage Costs Traditional Data Growth Thin Replication ~ 95% Snapshots ~ 75% Virtual Clones ~80% Thin Provisioning ~30% CAPACITY SAVINGS ~ xx % DeDuplication ~ 25-95% Auto-Tiering 65-95% RAID*DP ~ 40% vs R10 Capacity Requirements
Cloud Storage Requirements Efficient Storage Back Up Automated DR Replication Thin Provisioning & Others Simplifying the increase capacity utilization Automatic avoidance of disk hotspots Extent-based volume management Efficient cache management at disk system (controller & disk arrays) Optimized performance through automated data placement Dynamic QoS management
Operational Flexibility Private Cloud - Requirements VZ App Silos Costs Public Cloud Storage Private Cloud Storage Self- Service Access Resources on demand to speed deployment and delivery Scale Resource Up/down to optimize their usage, Release when not needed Service Catalog Choose pre-defined IT Services/user/dept. Define SLA to efficiently meet services Private Cloud Storage Automation Automated Provisioning - Moving Data & Processes Seamlessly Allocation, Self Tuning of Resources to meet Workload Requirements Service Analytics Monitor & Analyze Usage for Charge back Interactively auto-tune performance Availability with SLA requirements
Service Catalog by User/Workload Class Storage Availability Operational Recovery Disaster Recovery Performance Security Service Catalog by Workloads Metric Initial Capacity TB Storage Protocol (NFS/SCSI ) Downtime/Year (Max Unplanned) Mts Copy Retention Time Hrs/Yrs RPO Recovery Point RTO Recovery Time Recovery Consistency Recovery Assurance % Min # Recovery Points RPO Recovery Point RTO Recovery Time Recovery Consistency Recovery Assurance % Min # Recovery Points I/O Response Time ms (Max/Avg) IOPs (Min/Assured) Bandwidth MB/sec (Min/Assured) Encryption DLP Copy Retention Time Hrs/Yrs Copy Destruction Method -Quick/Secure Erase Authencity Guarantee
Cloud Storage - Requirements Enable Policy Based Efficient File Management Locally - Identify files for backup or replication - Move to right tier of storage hierarchy including cloud - Delete expired or unwanted files Globally - Localize data > Improve file access performance and reduce network costs. - Allow multiple sites to collaborate on information exchange - Virtualize to a single/global namespace > Multiple users can view same files from universal locations Locally Globally Collaboration in Cloud - Local or long distance users sharing a dedicated home file System but with individual home directory. - Each site/system has own file set at their local cluster. -Data Center has all Home dirs and manages BUs/hsm - Remote sites periodically prefetch (via policy) or pull on demand - Data Center maintains BUs & R-kive Copy of data. - Each location maintains logs/records for that Geography Data Center
Usage Dept/User IT Infrastructure in the Cloud SLA Business Priorities Cost of IT Ops/Charge Back Methods Response Time/Availability/Throughput, QoS Transactions/Sessions/Events/Analysis/Reporting Business Services Managed & Charged Cloud IT Services Provider Usage Profiles Users/Services/Workloads Applications (OLTP/BI/HPC/Data Streaming) Execution: Rules Driven, Adaptive Provisioning Services Abstraction, Adaptive Provisioning Assets Location Cloud Services Hosting Platform OS/Processors/#/Speed/Type Pooled Infrastructure Resources by Application Metrics Pooled Capacity Provisioning: Processing, Bandwidth, Storage, Repository For copy of case study on how a major financial institution implemented Cloud Infrastructure email imex@imexresearch.com
Key Takeaways Cloud Storage is increasing being adopted IT-as-a-Service is being delivered through a virtualized, Shared IT infrastructure, automation and Service Efficiencies using Private and Hybrid Cloud Infrastructure Savings through Storage Efficiency using automation that is being integrated Due diligence in careful planning, a three year roadmap, selection and testing of vendors/products required before starting migration from existing DC operations IT Infrastructure to meet needs of your Business Applications/SLA Let your business needs determine your Cloud Strategy. Identify and define specific SLA metrics per user/dept/division. Strategize for Long Term but take incremental approach to implementing Clouds Identify and screen promising Cloud Service Providers their Enterprise Class, Security and Compliance adherence, Change Controls, Referenced High Availability & Performance Results. Be aware of technology and best practices available Engage the expertise of consultants and system integrators to validate your strategy and implementation plans 41