Big Data Storage: Convergence and Efficiency

Transcription

1 I D C T E C H N O L O G Y S P O T L I G H T Big Data Storage: Convergence and Efficiency May 2014 Frank Cai, William Zhang, Craig Stires Sponsored by Huawei IDC Opinion The emergence of the big data age changes the way businesses operate and how people live. Today, data assets are increasingly becoming the core competency of companies because big data analysis enables companies to capture market dynamics with higher sensitivity and thus detect more sales and marketing opportunities. The explosively growing amounts of data, however, exceed the capacity that the companies' exsisting IT infrastructure can handle or cope with. For enterprises that have decided to get involved with big data, the very first thing to consider is the infrastructure's capability in carrying volumes of data and its time efficiency in data computing. Based on such needs, it is crucial that the infrastructure has high scalability and high performance in carrying big data applications. According to IDC's China market data, 60% of investments in big data were used on infrastructure in After all, big data application can hardly be realized without highly efficient infrastructure. The big data system is toppling the traditional IT architecture of enterprises, in which the storage systems are a core component. The ability to quickly analyze large volumes of varied data types fast is exactly where the value of big data lies. An IDC survey conducted on a global scale shows that volume, variety, and fast processing speed are the three indispensible factors that bring value to big data. The high efficiency of big data is made possible by the high-performance processing platform and flexible scalability, which are powerful capabilities sought by enterprise customers. In addition, for big data application, enterprises also want to evaluate the accessibility and manageability of the big data infrastructure. In the face of complicated data types and tremendous amounts of data, finding the means to obtaining maximum value is a big challenge IT professionals are confronted with. In a traditional IT infrastructure, data processing is section-specific and department-specific, that is, the acquisition, storage, backup and filing of data, as well as data mining and analysis, are conducted by different subsystems. Improving the performance and capacity of a single subsystem can only solve part of the problem. For enterprise users, the big data processing platform can optimally converge all the capabilities of these subsystems with unified view and management, so that data can flow freely and support application needs from storage and filing to analysis anytime. Furthermore, this allows the deployment model to evolves from the traditional construction that centers around application and subsystems to a new model that centers around data. The deployment of an all-in-one architecture and the construction of a big data processing platform that can carry out data life-cycle management will raise the efficiency of enterprise data management, provide stronger support to business, and free up IT administrators from tedious work, in order to devote themselves to innovative work with greater value. IDC 1559

2 The selection of an appropriate storage system is crucial to the construction of big data system. IDC suggests that IT departments focus on the following factors: High efficiency of the storage system. The nature of big data determines that data analysis needs to be highly real-time, and that the big data storage system has to be high-performance. A survey of IDC reveals that 68.6% of users regard high performance as the single most important factor when choosing big data storage system. High scalability of the storage system. The important characteristics of the big data storage system are it is massive in amount and it keeps increasing fast. The big data storage system needs to be highly scalable to meet the linear growth in demand for capacity and performance by the big data system. Diversified data type supported by the storage system platform. A prominent feature of the big data storage system is the variety of data types it carries, including structured data, nonstructured data, half-structured data, and object data. The big data storage system needs to simultaneously support and administer a variety of data types. Manageability of storage system platform and adaptibility to the application programming interface (API) standard. The big data infrastructure is extraordinarily complicated. So it is important for the big data storage system to provide unified and simplifed administration. In addition, for the sake of a centralized automatic administration of storage and computing, the big data storage system needs to be adaptable to the API standard. In This Technology Spotlight In this Technology Spotlight, we will discuss the all-in-one infrastructure of Huawei OceanStor 9000 big data storage system to get an in-depth understanding of its design in terms of system architecture, data storage, and business operation protection. We will also analyze how to fully meet the customer demand in storage system life-cycle application in order to maximize the value for customers. Market Overview According to IDC statistics, by 2020 the amount of data in the entire world will be eight times more than in 2012, and in China this will be 24 times more. The emergence of the big data age is toppling the traditional IT architecture, especially challenging the enterprise storage architecture. The big data storage system not only stores and protects data, but it also needs to provide an all-inone architecture that is capable of life-cycle management and data analysis optimization, including features of data storage, filing, analysis, and management. The purpose is to offer powerful support for customers to build a complete big data ecosystem and downsize the complexity for customers to manage data. IDC's China market data reveals that 67.2% investments in big data were used on infrastructure in 2013 (see Figure 1). Among all the categories, the storage system investment is the highest with a five-year compound average growth rate (CAGR) of more than 60%. 2

3 Figure 1 China Big Data Revenue Breakdown, 2013 Software (14.9%) Service (17.9%) Infrastructure (67.2%) Total: US$238 Million Source: IDC, 2014 Huawei OceanStore 9000 Storage Solution Distributed File System WushanFS Distributed File System WushanFS, a distributed file system Huawei developed independently, is the core system that supports the OceanStor The system employs a hardware architecture of fully symmetric distributed clusters, and in networking adopts the full connection and full redundancy strategy. Deployed on X86 servers, the WushanFS system could leverage latest features of hardware development to cut short development cycle. Logically, WushanFS distributed file system aggregates all disks within the servers to form a resource pool and provide a unified namespace, so that different levels of data redundancy protections can be provided cross nodes and cross stacks. The WushanFS distributed system adopts decentralized technologies including cluster metadata service (MDS) and internode data redundancy. File data and management data (metadata) of the system are stored in each node to avoid competition for resources. Breaking this bottleneck, the system sees the volume of storage and the computing performance in linear growth with the increase of nodes. WushanFS distributed file system has a global cache, the volume of which also grows linearly as the number of nodes increases. With the global cache, data hit ratio rises greatly while the hard disk I/O visit frequency drops and the overall system performance is lifted. When users access the file data, the WushanFS file system first locates the metadata server related to the original data and acquires data distribution information from the metadata server, the exact places and nodes the original data reside in. Reading and writing of the data is done through visiting these nodes. 3

4 Figure 2 WushanFS Distributed System Architecture Switch Switch Application Management Server 1B/10G Switch 1G Switch 1B/10G Switch Storage A B C D N Source: Huawei, 2014 High Performance High performance is a primary characteristic of the big data storage system, and also the cornerstone for the storage, analysis, and filing of big data. The SPECsfs 2008 evaluation on OceanStor 9000 shows that, with 100 nodes and under the Network File System (NFS) protocol, the system could reach the performance of 5,030,264 operations (OPS), which tops the current standards of the industry. The scale-out architecture of OceanStor 9000 ensures the linear expansion of the system. Every new node added to the system will see an increase of storage volume and performance with the business being unaffected. This evaluation assesses the performance of OceanStor 9000 at 10 nodes, 20 nodes, and 40 nodes, demonstrating the scale-out and performance growth at different number of nodes. 4

5 Figure 3 Huawei OceanStor 9000 SPEC Evaluation Throughput(ops/sec) Response(msec) Source: Huawei, 2014 Seamless Scale-Out OceanStor 9000 allows dynamic on-demand node expansion from the 3 up to 288 nodes, without interupting the business during system expansion. OceanStor 9000 could use different types of nodes to support different applications, and support a maximum namespace of up to 40 petabyte (PB). IT administrators do not have to manage multiple namespaces, thus reducing the complexity of system administration. By eliminating the namespaces, the data silos caused by the namespaces are also broken down. OceanStor 9000 is a flexible storage system that is easy to extend in accordance to users' requests. This reduces the total ownership cost for users with unchanged high reliability and performance. Lean Data Management InfoProtector High Reliability Data Protection The data protection technologies of OceanStor 9000 include distribution model and node redundancy. Data entering the system is divided into a number of data fragments designated as N, and the system then computes out another number of redundant data fragments designated as M. All these data fragments will be saved in different nodes that amount to the number of N+M. This way, OceanStor 9000 storage system is able to restore the data not just when disks are faulty, but also when the whole node is faulty. The system can be restored to service as long as the number of faulty nodes is less than M. By reconstructing data, the system can recover the lost data and restore the overall system data stability. OceanStor 9000 has another protection mode called N+M:B, which allows a maximum of M disks or B nodes to fail at the same time without problem. This mode is effective in small systems where the nodes number is smaller than N+M. OceanStor 9000 offers multiple redundancy configurations in the nodes of N+M or N+M:B, and users can configure in accordance with business requirement at the management interface. All folder or files fall into the scope of this configuration. Moreover, users can set different redundancy 5

6 ratio for different files within one folder. This facilitates users in establishing most appropriate degrees of reliability by flexibly designating data redundancy according to their practical requirements. Info Tier Automatic Storage Tiering The Info Tier automatic storage tiering of OceanStor 9000 allows classification of physical nodes into different nodes pool within one file system. s with same characteristics, physical types, and access performance fall into one node pool. File pool refers to the collection of files that has certain features, for instance, files with the size of more than 100M. The file pool policy defines the rules of file pools in node pool, for instance, files bigger than 100M is stored in the other node pool. File pool and node pool correlate through file pool policy. Automatic storage tiering allows users to define the value of data in workflow through setting the file pool policy. Files of high value or importance will be stored in devices of high usability and performance, which in turn are more expensive, while files of low value or importance in devices of low cost, and relatively low performance and usability. Dynamic storage tiering is triggered by file pool policy, and migrates data automatically across nodes. The whole process is transparent to business access and will not affect normal operation. InfoEqualizer Load Balancing Technology OceanStor 9000 InfoEqualizer client connected load balancing is a domain name system (DNS)- based technology. At the stage of domain name translation, OceanStor 9000 load balancing service through an algorithm assigns one node among the cluster for user's access, while the business data interaction in the wake will be completed between the client and this node. OceanStor 9000 load balancing service is designed as a cluster system. At the initial stage of the cluster, a node will be selected out as the main node through Paxos algorithm. There is only one main node throughout the load balancing process. Each of the nodes in the cluster collects the loading status information and reports regularly to the main node, including information regarding the number of CPU cores, CPU clock speed, RAM size, network adaptor status, CPU utilization, RAM utilization, network throughput, and NAS client connection status. The main node conducts load balancing based on the information collected. OceanStor 9000 supports unified service access domain name, and domain name inquiry service is integrated into the load balancing service. When users launch a request for domain name inquiry, load balancing service will perform computations based on the load balancing policy configured, and revert an appropriate node back to client for the access into the OceanStor 9000 system. Architecture of All-in-One Life-Cycle Management of Data One of the biggest challenges for big data is the life-cycle management of the mass data. End users often face the challenge of how to administrate data of online store, near store, and offline store with high efficiency and low cost. Huawei raised the concept of innovative storage architecture that combines storage, analysis, and filing together; and practices automated full lifecycle management on information. By applying high hardware performance to the concept of allin-one of functions in product design and integrating the storage, analysis, and filing of big data, Huawei OceanStor 9000 effectively improves the data life-cycle management efficiency. Huawei OceanStor 9000 is capable of providing full life-cycle management on one platform, and deploying a variety of business applications within one architecture, thus realizing a holistic vision on data flow. The full life-cycle management of online store, near store, and off store data simplifies the management and reduces operation cost. 6

7 In addition, full life-cycle management on the same platform will spare the system cost and data loss risks during data migration, making it more efficient to use the function of dynamic tiering storage. OceanStor 9000 provides abundant interfaces including NFS, CIFS, POSIX, HDFS, MR, and SQL. Through the management software, OceanStor 9000 is able to apply unified devices management, NFS/CIFS/POSIX/HDFS file write in, MR/SQL inquiry and life-cycle management of the data. The application logic is therefore streamlined and business development sped up. Figure 4 OceanStor 9000 Full Life-Cycle Data Management Media Assets Internet HPC Behavior Forecast, Instant Analysis Unstructured Data, Object Data, Structured Data Big data sharing BI Big data analysis NFSCIFSHDFSObject SQL MR/Hbase Distributed Database Enterprise Hadoop Distributed File System Archiving Source: Huawei, 2014 Analysis-Oriented Optimization Data analysis systems usually fall into two categories. One is structured data analysis that develops from relational database; the other is unstructured data analysis based on Hadoop. Both of these two types have downsides. For structured data analysis, most data analysis products require data to be processed and sorted before entering the database. For unstructured data analysis, Hadoop is developer dependent and cannot be delivered directly, thus secondary development is needed in considerations of reliability, usability, and functionality. Huawei OceanStor 9000 big data analysis subsystem is based on distributed storage, and incorporates Hadoop's innovative analyzing components. It has standard SQL interface for access and analysis of structured data. It also supports standard interface of NFS/CIFS/POSIX for access of structured and semi-structured data. In addition, the analysis module is able to stream access while concurrently conducting analysis on the structured and semi-structured data received from the standard interfaces. With the evolvement of digitalization and the growth of unstructured data, enterprises will have more ticket, image, audio, video, and document files than ever before. Management of these data is often no more than to store them in network-attached storage (NAS). As the volume of data keeps swelling, the 7

8 legacy system is unable to fulfil the requirements on performance and extensibility, and a system that is easy to extend, fast to retrieve, and able to integrate with business software, is desperately needed. Huawei's big data platform provides a nice capability on speed and extension, and allows users to define the metadata by themselves. To achieve a swifter pace, the system also supports metadata retrieval to both system metadata and the metadata of the business files loaded into the system. Opportunities and Challenges of OceanStor 9000 IDC believes that big data is in the nascent stage, which makes the investment on the deployment of infrastructure and data management as the first step. IDC sees huge market space for big data storage and forecasts that the average compound growth rate of the big data storage will stand as high as 40.2% from 2012 to Huawei has gained an advantage in the market as a pioneer in big data storage by introducing the OceanStor IDC, however, has also noticed that Huawei will come across some challenges in promoting its big data solution. As the eternal goal of big data is to dig out and analyze the valuable data from the large volume and fast-expanding data warehouses, the storage and management of data is only the first step. Huawei needs to provide its clients an end-to-end solution, which runs from the infrastructure up to the processing and analysis platform; and is loaded with industry modelling, data mining, and business analysis software to truly realize the value of big data. To maximize the value of the data, business model should be built and insights be gained for each industry. Therefore, Huawei needs to partner with ISVs that have knowledge in vertical industries to construct a complete ecosystem. Conclusion At this stage, a successful infrastructure is the first step towards advancing business operation and enhancing competitiveness for organizations today. With the onset of the big data era, building an infrastructure of low cost, high efficiency, and multi-function is an important step to address the challenges of the future. In this technology spotlight, IDC summarizes for IT administrators the features of big data storage on full life-cycle management from the aspect of performance, usability, and open protocol. In addition to this, IT managers when evaluating storage solutions should also pay attention to the following reminders: Consider the reliability of the storage systems within the framework of cost Apply lean management when reliability of the storage system is ensured Think about the security along with the efficiency of the storage system ABOUT THIS PUBLICATION This publication was produced by IDC Go-to-Market Services. The opinion, analysis, and research results presented herein are drawn from more detailed research and analysis independently conducted and published by IDC, unless specific vendor sponsorship is noted. IDC Go-to-Market Services makes IDC content available in a wide range of formats for distribution by various companies. A license to distribute IDC content does not imply endorsement of or opinion about the licensee. COPYRIGHT AND RESTRICTIONS Any IDC information or reference to IDC that is to be used in advertising, press releases, or promotional materials requires prior written approval from IDC. For permission requests contact the GMS information line at or gms@idc. com. Translation and/or localization of this document requires an additional license from IDC. For more information on IDC visit For more information on IDC GMS visit IDC China Office: Room 1206, Tower D, Global Trade Center No.36 North 3rd Ring Road, Beijing P F