Enterprise-grade Hadoop: The Building Blocks

Transcription

1 Enterprise-grade Hadoop: The Building Blocks An Ovum white paper for MapR Publication Date: 24 Sep 2014 Author name

2 Summary Catalyst Hadoop was initially developed for trusted environments that did not require bulletproof security, running highly specialized analytics, and accessible only by a select cadre of elite programmers at big Internet companies. But as mainstream enterprises implement Hadoop, there is the expectation that the platform will deliver the capabilities that are expected of enterprise data platforms. Progress in filling Hadoop s gaps regarding enterprise-grade functionality has been significant; for instance, in the 2.x version, the open source community has made important advancements in opening the platform to multiple workloads and addressing some single point of failure issues. On the horizon, there are numerous open source projects aimed at addressing gaps in role-based access control, data security, and other areas. However, as an emerging platform, commercial distributions are differentiating; organizations should closely scrutinize their platform supplier s strategy for delivering an enterprisegrade platform. Ovum view Ovum believes that for Hadoop (and other Big Data technologies) to gain enterprise adoption, it must become a "first class citizen" in the enterprise. To do so, Hadoop must be capable of behaving like an enterprise platform. If Hadoop is to become a platform on which enterprises store system of record data, and rely on for competitive analytics insights and operational applications or decisions, it must deliver: Predictable performance -- Deliver consistent performance/availability/reliability to meet business SLAs. Security Provides the same degree of granular security addressing access rights, privacy, and protection unauthorized actions as enterprise databases. Data protection Ensure the same degree (e.g., backup and recovery, privacy, access, obfuscation, and auditing of usage) that is offered by enterprise databases. Data governance and stewardship -- Support the policy-driven lifecycle management of data, like any enterprise database. As a platform, Hadoop is making excellent progress toward meeting these requirements. MapR has been proactive in charting a dual-pronged strategy, supporting critical mass open source projects targeting security and manageability, while extending unique capabilities that address high availability and fault tolerance, data protection, multi-tenancy/resource-management, and performance consistency. Hadoop must become a first class citizen in the enterprise Hadoop originated with Internet companies that had -- at the time -- unique computing and analytic problems. As such, these companies jointly developed the platform in the open source community based on work performed at Google, Yahoo, Facebook, LinkedIn and other organizations. Early 2014 Ovum. All rights reserved. Unauthorized reproduction prohibited. Page 2

3 adopters typically implemented Hadoop as a special, often standalone platform, and run by special teams. The rules of engagement are changing as enterprises adopt Hadoop. The standalone, SWAT team model of Hadoop deployment will not be sustainable in the broader enterprise. Instead, IT departments deploying Hadoop on behalf of multiple business groups must integrate it into the data processing fabric, covered by the same policies regarding access, data protection and retention, resource management, and security. With Hadoop increasingly being utilized as an operational platform for both decision support and data-driven applications, SLA compliance regarding performance and uptime become extremely relevant. Hadoop must become a first class citizen in the enterprise. It must become part of the enterprise data fabric because it will operate alongside existing OLTP, data warehousing, and other decision support data stores. The technology must be accessible to the people and skills that already form the IT organization. Hadoop must map to existing data center infrastructures, policies, and practices that are already applied to data platforms. To become a first class citizen in the enterprise, Hadoop must become enterprise-ready for maintaining security, delivering consistent performance and high availability, providing the management capabilities to ensure sound operation, protecting data, and ensuring data governance. The Five Building Blocks for Enterprise Hadoop 1. Security Hadoop was not originally designed for security. Like the Internet, Hadoop was originally designed for use in trusted environments where there was no perceived need for enterprise-grade security features. Because Hadoop clusters were originally deployed behind firewalls with few having the knowledge to access them, the main security concern wasn t hacking; instead, it was facilitating the pooling of disjointed clusters. That resulted in support of Kerberos for token-based authentication to remote clusters. The original core Hadoop platform also made provision for controlling data access through a coarse-grained approach involving simple user permissions at the file system and directory levels. Third-party tools could enforce protection at the perimeter through firewalls and packages that guard against malware and intrusions, but collectively, all of these measures were just the first step for any data platform that is supposed to be enterprise-grade. An enterprise-grade solution includes full authentication, authorization, and accounting ( AAA ) measures that ensure that that users are who they claim, they can only access data for which they are authorized, and all actions are logged to support administrative audits. At minimum, this requires a unified, single-sign-on capability for all Hadoop modules that can be linked to existing enterprise directories (e.g., LDAP, Active Directory), with authorization for role-based access to specific categories of data. Fault-tolerant authentication must be supported from the client to all nodes of the cluster. 2. Performance and High Availability Enterprise data stores are expected to perform and be available on a consistent, predictable basis. That encompasses high performance, fast response times, and high uptime, and requires features such as automated, dynamic load balancing, rolling updates, and immediate failover. Hadoop s 2014 Ovum. All rights reserved. Unauthorized reproduction prohibited. Page 3

4 MapReduce heritage until recently limited the platform to batch mode operations. As such, expectations for performance and conformance to service level agreement (SLA) commitments were modest. However, as Hadoop accommodates new forms of workloads such as interactive SQL query, real-time stream processing, database operations, and search, it will be utilized increasingly as an operational platform; this is where SLA compliance; uptime and availability; operational efficiency; and managed recovery become especially critical. This places high importance on eliminating single points of failure (SPOF), especially with the NameNode, which the Hadoop community addressed with v 2.0, and that MapR has supported in a more distributed way from the outset. This also places high importance on having predictable performance, both for interactive query and reporting workloads. Furthermore, the purpose of running real-time streaming for operational applications or decision support will be undermined if processing is unexpectedly interrupted. 3. Manageability Enterprise-grade data platforms are expected to be reliable. This requires powerful management capabilities that provide full visibility of operations and granular management of resources and recovery. Although not a high priority for early adopters who utilized Hadoop primarily as a batch analytics platform, manageability grows essential as enterprises deploy it in an operational role and across multiple business units. They entails providing complete visibility, down to node level, of all running services and resource consumption across all clusters. Making operations transparent is an important first step towards meeting the service levels that users expect. With visibility comes the ability to manage resources granularly, isolating resources down to service and node level, along with the ability to dynamically shift loads according to demand and priority. It also requires the ability to maintain availability through capabilities such as: Enabling organizations to take advantage of the latest versions of Hadoop through rolling updates that avoid taking entire clusters offline, and allowing the core Hadoop platform to be upgraded separately from the Apache ecosystem components; Building and running standard workflows for system stops, restarts, and reconfiguration; Granular multi-tenancy across groups or applications which allows for resource isolation and scheduling, resource quotas for data storage and processing, security and, and reporting/auditing of system utilization for ongoing optimization; and Policy-based disaster recovery that prioritizes mirroring of operations by importance or criticality. Manageability becomes essential as Hadoop evolves to a multi-purpose platform, capable of running multiple types of workloads side by side across different portions of the cluster a capability that is partially opened up by Hadoop 2.x s YARN resource management framework. This is a critical development for Hadoop to evolve into becoming a truly shared resource that can be utilized by multiple business units concurrently. Additionally, as Hadoop usage grows to support multiple business units or applications as a shared service from a single cluster, the need for a true multitenant environment become more pronounced Ovum. All rights reserved. Unauthorized reproduction prohibited. Page 4

5 4. Data protection Enterprise Hadoop platforms are expected to support either within the platform, or through enabling technology that third parties leverage provisions to protect privacy and authorize access to specific categories of data. Although not a major issue in Hadoop s early days, when it was utilized as an batch processing platform, as new operational workloads emerge, provisions must be made to ensure that the data always remains sacrosanct. Data must be fully recoverable; features such as disaster recovery, snapshots, and rollbacks will become critical. Because no system can be made 100% available, provisions must be made for snapshotting capabilities that pinpoint the status of all operations on Hadoop files or HBase and Hive tables at specific points in time. Additionally, within the cluster, the platform should have the capability to intelligently replicate data across multiple nodes deployed across different racks, and for providing selective mirroring onto remote clusters that pinpoints the most sensitive data. And there should be straightforward automation to replicate data to remote clusters to protect against geographical disasters. Additionally, the platform must provide enabling technology to support complete or selective obscuring of data as dictated by enterprise policy or external regulation. That encompasses data encryption (for data at rest or on the wire), data masking, and data redaction. This includes protection of data in movement (on the wire) and at rest. These capabilities may not necessarily be provided by the Hadoop platform provider; but Hadoop must enable selective application of such measures. 5. Data governance While data retained in Hadoop will be more varied than data stored in traditional enterprise transaction databases and data warehouses, it is not immune from enterprise policies regarding data retention and lifecycle management, including full data lineage tracking (e.g., sources of data, how the data was transformed or consumed, and by whom) to generate audit trails. The challenge is compounded by the extended breadth and depth of data sources that will be stored in Hadoop compared to a traditional enterprise data warehouse. With its more economical storage, it becomes more practical to store longer timeframes of data in Hadoop; for instance, instead of storing 1 3 years of customer data, an enterprise could keep 7 10 years of data live. Likewise, because Hadoop (unlike traditional data warehouses) can accommodate variably structured data, the sources of data may become more diverse. That makes data governance a broader challenge for Hadoop, although with most organizations just beginning their implementations, that may not be at the top of their task list. But, with more and more varied data stored in Hadoop, it will become extremely critical to enforce strict role-based access controls by data set, while carefully tracking lineage to document data utilization. With more varied sources of data, it will become important to assign confidence levels regarding the validity and cleanliness of data. With extended periods of data stored, data retention policies will have to be enforced where applicable. Ovum does not necessarily expect Hadoop platform providers to specialize in delivering complete portfolios of data governance tools, but the underlying platform must be capable of enforcing those controls at the necessary level of granularity Ovum. All rights reserved. Unauthorized reproduction prohibited. Page 5

6 MapR s strategy for supporting enterprise-grade Hadoop Supporting and accelerating Apache Hadoop Hadoop is not a single monolithic platform, but a collection of open source projects that commercial distribution providers test, integrate, package and support. The MapR Data Platform is built on a distributed computing architecture that was designed from the outset for high availability, reliability, and manageability. MapR s strategy is to support all major Apache Hadoop open source projects, and support industrydefined APIs for areas where MapR has written differentiated implementations, like HDFS and HBase. MapR supports a broad variety of Apache Hadoop and other open source software projects and in many cases, offers choices of competing open source projects in areas such as interactive SQL query. At the core of the MapR Data Platform is the MapR File System (MapR-FS). It provides an NFS and POSIX-compliant file system that replaces HDFS and supports consistent snapshots, mirroring, and random read/writes and provides a NAS-like interface for enterprise data sources. With the MapR Enterprise Database Edition of the distribution, MapR provides MapR-DB, an integrated in-hadoop NoSQL database that provides a high-performance alternative to HBase, while maintaining HBase API compatibility. MapR also supports native Apache HBase. Enterprise capabilities enabled by the MapR platform The Hadoop community has made important advancements with the 2.x generation with regard to the ability to run multiple workloads, take snapshots, and provide a degree of redundancy with the NameNode, which was formerly a single point of failure. MapR follows a dual strategy for supporting a variety of Apache projects that are addressing gaps in the Hadoop platform for manageability, security, and data protection, while leveraging its unique architecture to provide some capabilities that are beyond the capabilities of HDFS. NFS compatibility and POSIX compliance This provide several advantages. Industry-standard NFS compatibility enables existing enterprise applications to access Hadoop data as if they were accessing a NAS; it also allows existing enterprise data sources to be ingested into Hadoop without need for special connectors or utilities such as Flume or Sqoop. POSIX compliance allows the random read/write capabilities that are taken for granted with enterprise databases. Platform Resiliency MapR approaches this requirement in several ways. Its architecture distributes NameNode metadata across all worker nodes in the cluster. While Hadoop 2.x has added a secondary backup NameNode, MapR s architecture offers a different approach that distributes metadata in shards across the cluster, co-residing with the data with which it is associated. This capability is available out-of-the-box, requiring no additional configuration or hardware, and allows instant recovery, with files and tables available rapidly after node failures or cluster restarts. And, if nodes fail while MapReduce batch jobs are running, MapR lets the jobs run to completion, rather than requiring a complete restart from scratch. Snapshots are also supported in Hadoop 2.x, but they are inconsistent if any writes are being 2014 Ovum. All rights reserved. Unauthorized reproduction prohibited. Page 6

7 performed on files; for MapR, snapshots are consistent as you would expect in an enterprise storage system, capturing the status of all files at a specific point in time, even if they are open. Mirroring is configurable; users can designate the volumes and directories to mirror, rather than covering the entire cluster by default. Security This is a work in progress for the Hadoop platform in general. The community has responded with a variety of projects, while established third party tooling addresses some of the gaps. This is an area where MapR is adopting a dual strategy, addressing key customer requirements in the core platform, and implementing security projects from Apache for additional layers of protection. Authentication has been built directly on top of the core platform, with MapR supporting both Kerberos as well as simplified username/password based authentication to integrate into environments that haven t adopted Kerberos. Authorization is available at both a file-level, with permissions bits, and on a column and column-family level for tables stored in MapR-DB, controlled by Boolean access control expressions (ACEs). Wire-level encryption is also supported for all types of communication between nodes in the cluster. To achieve more granular project-specific authorization for projects like Hive and Impala, MapR will also support Sentry. Data protection This is also a work in progress for all Hadoop players. MapR currently supports wire level encryption for data in motion, but currently relies on third parties for encryption, key management, and masking for data at rest. For data replication, it ensures that at least one replica is deployed on a separate rack. MapR s snapshotting capability can pinpoint file status at specific points of time, even while files are open. And for mirroring, MapR provides granularity to choose data by specific volumes or directories, providing a more selective alternative to covering the entire cluster. Manageability and Multi-tenancy Most commercial Hadoop distributions provide node level monitoring and management. In addition to this, the MapR Control System (MCS) provides detailed management of jobs and data, the latter controlled through a unique concept called a volume. A volume is a logical partition of the file system that can be associated with a rich data management policy. Using MCS, users can modify several attributes of a volume, including quota, snapshot and mirror policy and schedule, administrative permissions, and data placement policy. Combined with YARN or other resource managers such as Mesos, this provide fine-grained resource and workload management for multi-tenant environments. MapR takes a few steps further with data placement control and job placement control. These capabilities ensure that the multiple user groups in a MapR cluster can run jobs in segregated portions of the cluster to minimize resource contention. Combined with integrated security, organizations can deploy a MapR cluster with multiple distinct user groups and data sets, a configuration that requires separate clusters on other Hadoop distributions. Additionally, the MapR Control System provides flexibility, supporting access for a GUI, command level interface, or through third party tools via RESTful APIs Ovum. All rights reserved. Unauthorized reproduction prohibited. Page 7

8 Appendix Author Tony Baer, Principal Analyst, Ovum IT Information Management Ovum Consulting We hope that this analysis will help you make informed and imaginative business decisions. If you have further requirements, Ovum s consulting team may be able to help you. For more information about Ovum s consulting capabilities, please contact us directly at consulting@ovum.com. Copyright notice and disclaimer The contents of this product are protected by international copyright laws, database rights and other intellectual property rights. The owner of these rights is Informa Telecoms and Media Limited, our affiliates or other third party licensors. All product and company names and logos contained within or appearing on this product are the trademarks, service marks or trading names of their respective owners, including Informa Telecoms and Media Limited. This product may not be copied, reproduced, distributed or transmitted in any form or by any means without the prior permission of Informa Telecoms and Media Limited. Whilst reasonable efforts have been made to ensure that the information and content of this product was correct as at the date of first publication, neither Informa Telecoms and Media Limited nor any person engaged or employed by Informa Telecoms and Media Limited accepts any liability for any errors, omissions or other inaccuracies. Readers should independently verify any facts and figures as no liability can be accepted in this regard - readers assume full responsibility and risk accordingly for their use of such information and content. Any views and/or opinions expressed in this product by individual authors or contributors are their personal views and/or opinions and do not necessarily reflect the views and/or opinions of Informa Telecoms and Media Limited Ovum. All rights reserved. Unauthorized reproduction prohibited. Page 8

9 CONTACT US INTERNATIONAL OFFICES Beijing Dubai Hong Kong Hyderabad Johannesburg London Melbourne New York San Francisco Sao Paulo Tokyo 2014 Ovum. All rights reserved. Unauthorized reproduction prohibited. Page 9