Addressing Storage Management Challenges using Open Source SDS Controller

Transcription

1 Addressing Storage Management Challenges using Open Source SDS Controller Anjaneya Reddy Chagam, Intel Chief SDS Architect, Data Center Group Shayne Huddleston, Oregon State University Infrastructure Architect, Information Services

2 Agenda o Data Explosion & Storage Pain Points o Software Defined Storage (SDS) framework o Controller Prototype and CoprHD (TBD) o Customer Perspective (Oregon State University) o Summary 2

3 Agenda o Data Explosion & Storage Pain Points o Software Defined Storage (SDS) framework o Controller Prototype and CoprHD (TBD) o Customer Perspective (Oregon State University) o Summary 3

4 The Problem for IT From 2013 to 2020, the digital universe will grow by a factor of 10, from 4.4 ZB to 44 ZB. It more than doubles every two years. COST CHALLENGES CONTINUE TO GROW 125 IT PROS WILL SHOULDER A GREATER STORAGE BURDEN Storage Capacity In TB 62% CAGR IT Budget 2% CAGR GB PER IT PRO 28 MILLION IT PROS WORLDWIDE ,231 GB PER IT PRO 36 MILLION IT PROS WORLDWIDE Data needs are growing at a rate unsustainable with today s infrastructure and labor costs. Source: IDC The Digital Universe of Opportunities: Rich Data and the Increasing Value of the Internet of Things - April

5 The Problem with Storage Infrastructure Storage Silos (Traditional): Storage dedicated to application Isolated resources Policies unique to vertical solutions APPLICATION SERVERS A A A A A A A A A Challenges: Cost: Managing massive data growth Managing volume and complexity of data Maintenance, operations and support Choice: Lock-in Lack of interoperability and flexibility Difficult to access shared data value and insight Cloud Deployment Need for rapid service delivery Infrastructure scalability ISOLATED STORAGE SILOS Traditional storage management is too complex and inefficient for new data demands 5

6 The Industry s Response 1 INTELLIGENT 2 SCALE-OUT 3 4 STORAGE NON-VOLATILE MEMORY SOFTWARE DEFINED 6

7 Agenda o Data Explosion & Storage Pain Points o Software Defined Storage (SDS) framework o Controller Prototype and CoprHD (TBD) o Customer Perspective (Oregon State University) o Summary 7

8 Intel s Approach to Software Defined Infrastructure (SDI) Application A Services Delivery Application B Application C Application D SERVICE ASSURANCE Policies and intelligent monitoring trigger dynamic provisioning and service assurance as applications are automatically deployed and maintained Orchestration Software Orchestration Software Infrastructure Attributes Power Performance Security Thermals Utilization Location PROVISIONING MANAGEMENT Orchestration provisions, manages and optimally allocates resources based on the unique requirements of an application Resource Pool Storage Network Compute POOLED RESOURCES Network, Storage and Compute elements are abstracted into resource pools 8

9 Software-Defined Storage (SDS): A Key Component of SDI Dynamic, policy-driven storage management SERVICES DELIVERY App #1 App #2 App #3 App #4 Orchestration Software Software-Defined Storage Controller STORAGE SYSTEMS Infrastructure Attributes POWER PERFORMANCE THERMALS ULTILIZATION LOCATION LATENCY DURABILITY Orchestrate application access to diverse storage systems through SLAs, increasing flexibility and handling data complexity Provision resources dynamically (pay-as-yougrow), increasing efficiency Aggregate diverse equipment, leverage legacy, increase flexibility, and drive down costs Abstract software from hardware, providing flexibility and scalability SDS framework delivers a scalable, costeffective solution to meet the needs of tomorrow s datacenter. 9

10 SDS Architecture DATA SERVICES Storage function in data plane to optimize storage Does not persist data For example: Data Deduplication, Erasure Code, Tiering APPLICATIONS ORCHESTRATOR Northbound API DATA SERVICES SDS CONTROLLER DATA SERVICES Southbound API SDS CONTROLLER Has visibility into and control of ALL storage resources Coordinates between apps, orchestrator, and storage systems Allocates storage resources to meet SLAs Storage System Storage System [SAN] [Capacity] Storage System [Performance] Storage System [NAS] SAN Node Node Node Node Node Node NAS

11 SDS Controller Workflow SDS Controller Servers with OS, SW Pre Configured Appliances Provision (Distributed Storage System) Assign resource Provision resource Note: Servers already have OS. Focus on 3 rd party s/w integration with framework Legend Not in OpenStack Exists in OpenStack CUDL = Create, Update, Delete, List Discover (Storage Capabilities) Discover storage system, tiers Discover QoS Discover capabilities Compose (Virtual Pools) CUDL virtual pools Attach/Detach tiers to virtual pools Admin UI for pool composition Object support Data Type Life Cycle Mgmt CUDL volumes & shares Attach & Detach volumes & shares Attach object store Clone, Snapshot, Backup, Extend, Shrink Get volume stats QoS monitor, alert & enforce Policies, data life-cycle mgmt. SLO based resource requests, scheduling Data services SDN integration Need end to end automation to address storage management pain points Maintenance (Storage Systems, Pools) Deassign resource Unprovision resource Retire storage system & virtual pools Support for maintenance mode Metrics aggregation 11

12 Agenda o Data Explosion & Storage Pain Points o Software Defined Storage (SDS) framework o Controller Prototype and CoprHD (TBD) o Customer Perspective (Oregon State University) o Summary 12

13 SDS Controller Prototype SDS Controller Provision (Distributed Storage System) Discover (Storage Capabilities) Compose (Virtual Pools) Data Type Life Cycle Mgmt Maintenance (Storage Systems, Pools) Expand Cluster (Ceph VSM, SwiftStack Controller) Discover storage system, tiers, capabilities - driver framework (Swift, Ceph) Prototype (Intel Demo Booth) Note: Servers already have OS. Focus on 3 rd party s/w integration. CUDL virtual pools Attach/Detach tiers to virtual pools Admin UI for pool composition Object support CUDL volumes & shares Attach & Detach volumes & shares Attach object store Clone, Snapshot, Backup, Extend, Shrink Get volume stats QoS monitor, alert & enforce Policies, data life-cycle mgmt. SLO based resource requests, scheduling Data services SDN integration Deassign resource Unprovision resource Retire storage system & virtual pools Support for maintenance mode Metrics aggregation Legend Not in OpenStack Exists in OpenStack CUDL = Create, Update, Delete, List 13

14 SDS Controller OpenStack Architecture Heat Horizon CLI Ceilometer Provision Pool mgmt Policy mgmt Data Life cycle mgmt. Data Service mgmt. QoS Monitoring, Maint SDS Controller App (e.g. Nova) Cinder attach/ detatch volume Volume, Share, Container, Image mgmt App (e.g. VM) Manila attach/ detatch share App (e.g. Nova) Glance Boot from image App (e.g. Webapp) PUT/GET/DELETE account, container, object Object Storage Cinder+ Drivers Manila+ Drivers Glance+ Drivers Storage Servers Storage System Storage System Storage System E.g., Object Storage Swift, Ceph, Atmos, Amplidata Work with the community and ecosystem to enable an Open and Interoperable SDS controller 14

15 15 CoprHD Slide (being explored)

16 Agenda o Data Explosion & Storage Pain Points o Software Defined Storage (SDS) framework o Controller Prototype and CoprHD (TBD) o Customer Perspective (Oregon State University) o Summary 16

17 Oregon State University DC Infrastructure Central IT provider for OSU (Infrastructure as a Service) Mission is to enable and support Students, Faculty & Science 3 central data centers (distributed) Infrastructure - OpenStack and VMWare Orchestration - Heavily virtualized - High availability metro cluster architecture - Puppet configuration management - Active/active replicated block storage - SAN, NAS storage islands - Distributed storage (piloting) 17

18 Oregon State University IT Workloads Multitude of different workloads that we must support - Virtual Machines - Virtual Desktops (VDI) - Research workloads Genome sequencers generate 20TB/hour sustained high bandwidth write - OLTP - Oracle, MSSQL, mysql - Business analytics (Hadoop, etc.) - Long term digital archive 18

19 Oregon State University IT Challenges Storage demand is growing faster - Growing at over 40%/year Storage is not optimized for diverse workloads Traditional storage management is costly and hard to scale Lot of touch points to administer, provision, and consume storage resources Allocation of storage resources to user workloads is 1 to 1 IT budget is relatively flat Majority of storage cost is hardware and operations 19

20 Oregon State University Focus Areas & Plans Storage growth explosion - Shift to cost optimized, pay as you grow storage architecture (OpenStack, Swift, Ceph) Storage optimized for diverse workloads - SLA and policy driven storage resource allocation - Storage allocation based on workloads, performance, cost and resiliency requirements Storage management - All storage operations managed by single controller in the DC Administrator should not have to go beyond the controller except in extraordinary circumstances - Recognition and classification of performance problems in central dashboard - Fully automated provisioning and simplified maintenance New capacity automatically added to appropriate service and controlled by policies OSU is collaborating with Intel to address storage challenges via open source SDS controller initiatives 20

21 Agenda o Data Explosion & Storage Pain Points o Software Defined Storage (SDS) framework o Controller Prototype and CoprHD (TBD) o Customer Perspective (Oregon State University) o Summary 21

22 Summary o o o Software Defined Storage is needed to address tomorrow s challenges o o Data needs are growing at a rate unsustainable with today s infrastructure and labor costs Traditional storage silos drives management complexity and inefficient SDS is a framework aimed at serving the needs of emerging storage requirements. o Industry wide focus is needed to create an open and interoperable controller Intel is working with community and ecosystem to enable an Open and Interoperable SDS controller

23 Call to Action o Engage with the community to address storage challenges o Provide feedback on storage requirements o Help us drive for an open source, interoperable, standards-based Software Defined Storage controller

24 Backup Slides 24

25 SDS Controller Architecture USERS SERVICE/TENANT PORTAL APPLICATIONS SLAs ORCHESTRATION SLA SLO COMPUTE CONTROLLER SDN CONTROLLER OPTIONAL PLUGINS: e.g., Analytics, Replication, Backup, etc. SDS CONTROLLER PROVISIONING DISCOVERY Northbound API Northbound Interface (REST) VMWare Microsoft OpenStack POOLING DATA TYPE Mgmt Life Cycle Mgmt Monitoring/Maint Southbound Interface (REST) VASA Cinder Manila Swift SMI-S S3 Southbound API QoS SLO Cloud DATA PLANE DATA SERVICES Storage Systems (SAN, NAS, Scale-Out, Hyper- Converged) UNPROVISIONED STORAGE SYSTEM OBJECT SLOs STORAGE SYSTEM FILE/BLOCK SLOs Storage Hardware, Media 25

26 SDS Functional Partitioning SDS Controller Block File Object Storage Management (Control Plane) Provision (Storage System) Discover (Storage Capabilities) Compose (Virtual Pools) Monitoring & Maintenance (Storage Systems, Pools) Data Life Cycle Mgmt Scheduling (Storage resources) Data Services Framework Policy, SLO, QoS Mgmt SDN integration Volume Operations (Control Plane) QoS Monitoring, Alerts CUDL volumes Clone, Snapshot, Backup, Extend, Shrink, Migrate volumes Attach volume Detach volume Get volume stats Share Operations (Control Plane) QoS Monitoring, Alerts CUDL shares Clone, Snapshot, Backup, Extend, Shrink, Migrate shares Attach share Detach share Get share stats Object Operations (Control Plane) QoS Monitoring, Alerts Attach object store, tier Legend Not in OpenStack Exists in Cinder or Manila CUDL = Create, Update, Delete, List 26

27 Provision Expand Capacity (Ingest Nodes) Horizon SDS Controller UI 1 Get Available Servers Virtual Storage Manager (VSM) Note: Prototype uses VSM for getting list of servers end goal Ironic 2 3 Add Server, Role VSM Ceph Ceph (Block) Refresh new storage capacity Swift Swift Stack Controller Swift (Object) Ceph Driver Cinder- Volume Stats Swift Driver Cinder-API Stats Get Stats from cache Cinder- Scheduler 27

28 Discovery Discover Compose Consume Database Horizon SDS Controller UI SDS API Discover Horizon SDS Controller UI Cinder API Volume volume_type to contain volume_backend_name for volumes share_backend_name for shares object_backend_name for objects cinder.conf to contain [swift-x] object_driver=<> object_backend_name=<> swift_proxies=<> etc. etc. Ceph (Block) Swift (Object) scheduler_default_filters=datatypefilt er,availabilityzonefilter,capacityfilter,c apabilitiesfilter (NOTE: not in scope) NOTE: Discovery is driver based framework. For e.g., drivers can be Swift, CDMI, S3, Proprietary 28

29 Disclaimer Intel technologies features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration. No computer system can be absolutely secure. Check with your system manufacturer or retailer or learn more at [intel.com]. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. Include as footnote in appropriate slide where performance data is shown: o Configurations: [describe config + what test used + who did testing] o For more information go to Intel, the Intel logo, {List the Intel trademarks in your document} are trademarks of Intel Corporation in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others Intel Corporation. 29

30 Software defined infrastructure (sdi) Application A Services Delivery Application B Application C Application D SERVICE ASSURANCE Policies and intelligent monitoring trigger dynamic provisioning and service assurance as applications are automatically deployed and maintained Orchestration Software Orchestration Software Infrastructure Attributes Power Performance Security Thermals Utilization Location Resource Pool Storage Network Compute PROVISIONING MANAGEMENT Orchestration provisions, manages and optimally allocates resources based on the unique requirements of an application POOLED RESOURCES Network, Storage and Compute elements are abstracted into resource pools

31 Intel s Approach to Software Defined Infrastructure ONLINE SALES CRM ORCHESTRATION LAYER ERP BROADEST ENABLED ECOSYSTEM Integrated and optimized for more seamless Data Center operations Service Provisioning EXPOSED AND INTEGRATED TELEMETRY Deeper insight & optimal provisioning management SOFTWARE DEFINED SERVERS 10/40GbE SOFTWARE DEFINED STORAGE SOFTWARE DEFINED NETWORK Power Performance Security Thermals Utilization Location Latency Intel True Scale NVMe * Other names and brands may be claimed as the property of others. PLATFORM AND ARCHITECTURAL LEADERSHIP Standards-based compute, network and storage resource pools. Network workloads and Storage workloads may run as SW appliances on IA servers (virtual network function, SDN or SDS controllers, storage optimization workloads) 31