Object-based Storage in Big Data and Analytics. Ashish Nadkarni Research Director Storage IDC

Transcription

1

2 Object-based Storage in Big Data and Analytics Ashish Nadkarni Research Director Storage IDC

3 IDC s definition of Big Data and Analytics (BDA) Mix of data, talent, technology, processes, and services that allow for effective management of potentially large volumes of multistructured and/or high-velocity data Enables a range of business intelligence and analytic applications to support tactical, operational, and strategic decision-making processes across the organization, delivering greater business value Applications Analytics & Discovery Tools Data Organization & Management Tools Infrastructure

4 Size of the BDA Market 2013* Big Data Software: $2.5 B Hardware: $4.6 B Services: $3.8 B Total WW: $10.9 B CAGR: 32% 2013* Business Analytics Software: $38 B Hardware: $19 B Services: $47 B Total WW: $104 B CAGR: 10%

5 What does Big Data comprise of? Hadoop Ecosystem NoSQL DBMSs Analytic Applications Rich Media Analytics Advanced & Predictive Analytics Value Added Content Providers Content Analytics Risk Management Decision Management Internet of Things Visual Discovery Skills and Org Structure Data Warehousing Big Data & Analytics Infrastructure Source: Google Trends, August 22, 2013

6 Types of BD/A applications deployed BD/A landscape is still dominated by commercial suppliers Opensource/community based platforms are playing an increasingly dominant role Q. What type of data analytics applications do you run on your data analytics infrastructure?

7 Big Data Infrastructure Workflow An IDC View Data IO Profiles Supporting Infrastructure Analytics Software Value derivation loop and/or and/or Data collected is in one or more formats and/or from one ore more locations Data sampled is more than 100TB at rest Object Storage! Data captured is via ultra-highspeed streaming Deployed on dynamically adaptable infrastructure Analyzed via batch, parallel and/or distributed processing framework Business value is continuously extracted and applied and/or and and Data generated is growing at >60% per year Value Volume Velocity Variety On-premise or cloudbased On-demand Resilient Optimized Intelligence Self-tuned Enduring Time-sensitive High-Impact Business-Tangibility Data Infrastructure Applications Business Impact

8 Dynamic Infrastructure for Big Data Variable Computing Compustorage IT Efficiency Capacity on demand Mobile Analytics Agile Business Apps integration SAP/Hana

9 What is object storage? Utilize APIs and REST-based object access methods for storing, retrieving and looking up data Use a flat Account-Container-Object approach for data organization and placement Use a referable and programmable metadata repository that is tied to a global and geo-dispersed namespace Meta data includes info about create, access, modified and other dates, permissions, security, application or file type, or other attributes Primary access is via an API where an object can be of arbitrary size (up to the maximum of the object system) along with variable sized meta data (depends on the object system/service implementation) Many object-platforms make use of NoSQL databases for data storage

10 The trifecta of object-storage platform suppliers Suppliers Open-source or community based Software ISVs (Deliver appliances) Traditional Hardware suppliers OpenStack, Basho Riak CS, and Ceph are examples of open-source or community based platforms Software ISVs include Scality, Cleversafe and Amplidata Traditional suppliers include EMC, NetApp and DDN

11 APPLICATION Most object storage platforms are built using shared nothing architectures APPLICATION APPLICATION CIFS NFS HTTP Block CIFS NFS HTTP Block Global Namespace Local persistent mory CPU/Me storage CPU/Me mory CPU/Me mory CPU/Me mory CPU/Me mory Local persistent mory CPU/Me storage CPU/Me mory CPU/Me mory CPU/Me mory CPU/Me mory Local persistent mory CPU/Me storage CPU/Me mory CPU/Me mory CPU/Me mory CPU/Me mory Autonomous node Site 1 Site 2 Site n

12 Object platforms with unified data access Some of them also leverage a unified access model for flexible data ingestion capabilities Ceph is an example of shared nothing architectures and unified (file/block/access) access LIBRADOS App (Direct) App (REST) RADOSGW (object access) Ceph Object Store (RADOS) (Distributed object-based storage) Local File system Local File system CPU + Memory CPU + Memory Local File system CPU + Memory 12 Host or VM RBD (block access) FS Client CEPHFS (file access) Individual Ceph Nodes

13 Common terms used in conjunction with objectplatforms Shared nothing architectures Data sharding Geo-dispersed namespaces Multi-master or master-less configurations Erasure coding (distributed vs. local)

14 Why object-storage in Big Data? BD/A analytics environments deal with large unstructured data sets - analyzed using applications with distributed workloads (e.g. Map/Reduce) Many of these workloads run on commodity hardware nodes that are clustered together It is expensive to use traditional storage platforms for distributed analytics With most object storage systems/services you can specify anywhere from a few KB of user defined meta data or GB This bodes well for workloads that need to analyze data of all types together

15 The growing influence of object-storage in Big Data IDC recently conducted a survey on storage patterns in Big Data More than 95% respondents said that they had either deployed or had plans to deploy object storage for their BD/A infrastructure Of this group, 55% plan to deploy it exclusively as the storage platform for BD/A infrastructure Q. Does your organization deploy or plan to deploy file- and object-based storage in your data analytics infrastructure?

16 Where data lives before it is processed 50% respondents said that they plan to store preprocessed data in an onpremises or cloud-based object-platform Geo-distributed and latency-resilient access mechanism of object storage platforms make it suitable for distributed analytics applications Q. On what storage medium is data stored before it is processed?

17 Where data lives after it is processed More than 47% respondents said that they plan to store postprocessed data in an onpremises or cloud-based object-platform The ability to use off-theshelf or commodity components to build objectplatforms makes them economical platforms for post-processed data Q. On what storage medium is data stored after it is processed?

18 The use of object-platforms as an archive tier Object platforms are excellent for archival of data economical and compliant They can quickly adapt for in-place analytics Several object-platforms are getting certified in regulatory and compliance environments Q. What kinds of data retention policies are in place in your data analytics infrastructure for the resulting data set after the raw data is processed/analyzed?

19 Growing popularity of object-platforms Platforms used in place of HDFS in Hadoop In a recent study conducted by IDC, respondents were asked Which independent file systems or database platforms do they use or plan to use with your Hadoop Infrastructure as a replacement for or to augment HDFS (Hadoop data store)? Amongst distributed file systems, object platforms made it to the list Ceph Openstack Swift Cleversafe dsnet Scality RING Amazon S3

20 Object platforms and adjacent non-storage workloads (Compustorage) Q. What were your reasons for selecting an alternative data store to augment or replace HDFS? Hadoop Map/Reduce is a common workload that can be run directly on the objectplatforms Many object-platform suppliers provide HDFS-connectors or replacements Preferred because of native deficiencies of HDFS (resiliency and performance in particular)

21 Essential Guidance The platform has to be open The platform is no good unless it enables solutions The platform itself cannot be a bottleneck (i.e. it is all about scale) It's all about geo-dispersed analytics Consider the workloads Don't forget the combination structured and unstructured data Don't forget the objects

22 Thank You! Ashish IDC Visit us at IDC.com and follow us on 22