How To Run Apa Hadoop 1.0 On Vsphere Tmt On A Hyperconverged Network On A Virtualized Cluster On A Vspplace Tmter (Vmware) Vspheon Tm (

Transcription

1 Apache Hadoop 1.0 High Availability Solution on VMware vsphere TM Reference Architecture TECHNICAL WHITE PAPER v 1.0 June 2012

2 Table of Contents Executive Summary... 3 Introduction... 3 Terminology... 5 High Level Use Cases... 5 Reference Architecture... 6 Design Principles for High Availability... 6 Key Components of Apache Hadoop 1.0 High Availability Solution using VMware vsphere TM.. 6 New components... 8 Failover Test Scenarios... 8 Conclusion... 9 Appendix... 9 Test Configuration... 9 VMware vmotion TM Technology VMware Fault Tolerance VMware Dynamic Resource Scheduling (DRS) References About VMware About Hortonworks... 13

3 Executive Summary There is a need in the Apache Hadoop ecosystem for production quality of High Availability (HA) solutions in Hadoop 1.0 environments. Hadoop 1.0 is adopted in large scale in many corporations as a key data management platform running in production and needs to be up and available as continuously as possible. This paper describes a Reference Architecture for running key Apache Hadoop 1.0 services on VMware vsphere TM platform, providing a HA solution to these services. All testing was done using a pre-release build of the Hortonworks Data Platform, powered by Apache Hadoop and is a joint work of Hortonworks and VMware. Hadoop enhancements required for this reference architecture have been contributed to the Apache Foundation. Introduction An Apache Hadoop cluster is designed to be very fault tolerant and resilient to failures. Such a cluster is composed of many servers (machines or VMs) each of which runs one or more processes that cooperate to provide the Hadoop services (HDFS, MapReduce, etc.). The majority of servers in a Hadoop cluster are worker nodes. Hadoop has built in redundancy and failure recovery algorithms to detect and repair a Hadoop cluster, without interruption, when a worker fails. A Hadoop cluster does depend on a small number of key master nodes being available for the cluster to function. These masters provide key services necessary for Hadoop to function. The NameNode and JobTracker are the most critical of these masters. Without the NameNode, data stored in Hadoop (HDFS) cannot be accessed or modified. Without the JobTracker, end user programs (MapReduce Jobs) cannot be executed. Currently the stable version of Hadoop (Hadoop 1.0) does not provide an HA solution for its masters and work to improve this situation in the Hadoop 2.0 development branch is incomplete and only addresses NameNode availability. This document outlines a solution to NameNode availability in stable Hadoop 1.0 line that also can be applied to the other Hadoop masters, providing a complete HA solution for Hadoop using VMware HA. Solutions to ensure higher levels of availability have traditionally been very costly, hard to implement, and difficult to manage. VMware vsphere TM makes it simpler and less expensive to provide higher levels of availability for important applications. The revolutionary VMware vmotion TM capabilities in vsphere TM make it possible to perform planned hardware maintenance with zero application downtime. VMware HA, a feature of vsphere TM, specifically reduces unplanned downtime by leveraging multiple VMware ESXi hosts configured as a cluster, to provide rapid recovery from outages as well as cost-effective high availability for applications running in virtual machines. VMware HA protects application availability in the following ways: It protects against hardware failure and network disruptions by restarting virtual machines on active hosts within the cluster. It protects against operating system (OS) failures by continuously monitoring a virtual machine and restarting it as required. It provides a mechanism to react to application failures. It provides the infrastructure to protect all workloads within the cluster. After VMware HA is configured, no actions are required to protect new virtual machines. They are automatically protected. One can combine VMware HA with VMware Distributed Resource Scheduler (VMware DRS) to protect against failures and to provide load balancing across the hosts within a cluster.

4 Figure 1: VMware HA solution VMware HA on Hadoop 1.0 offers a mature and tested alternative to the evolving HA facilities planned in Hadoop 2.0. VMware HA can also be applied to the full stack of Hadoop service, while current work in Hadoop 2.0 only covers NameNode HA. We believe VMware HA will continue to offer improved manageability with future version of Hadoop. Simply enable HA on the appropriate VMs, and the VMware cluster will automatically restart those services in the event of a host or software failure. One of the most important reasons to consider VMware HA is that it is a generic service that can cover all components of Hadoop ecosystem. So rather than treating the NameNode as a special case, one can apply the same technology to all key Hadoop services. These include: HDFS (NameNode, Check Point Server) MapReduce (JobTracker) Hive / HCatalog metadata server & Database Ambari Oozie Others.

5 Terminology High Availability (HA) - an application s ability to restore itself to full operating capabilities when faced with intermittent or permanent failure because of any dependency from network through operating system to application, itself. NameNode (NN) - the master service or interface called by all Hadoop filesystem operations. The NameNode manages files which exist in a Hadoop system as well as where the data for those files reside. JobTracker (JT) - the master service or interface called by all MapReduce clients attempting to execute code and work inside a Hadoop cluster. Worker - A VM or physical machine running the Hadoop TaskTracker to execute user code and the DataNode to store user data. Hadoop is designed to transparently handle failure of these nodes. Master A Hadoop service that is critical to Hadoop s functionality, such as the NameNode or JobTracker. For a Hadoop cluster to be HA, these services must be made HA. Cold Standby is an HA strategy where a replacement service is started only after a service configured for HA is determined to have failed. Hot Standby - is an HA strategy where a service configured for HA has a second version kept in lockstep with the active server. Clients can failover to this hot standby automatically w/o external hardware or software intervention. High Level Use Cases High availability capabilities are typically meant for use by operations and DevOps teams. To that end, the use cases listed below are focused on production scenarios. Planned Downtime: A Hadoop cluster is often shutdown in order to perform operating system or hardware maintenance servers, switches etc. If this maintenance impacts a Hadoop master s hardware or OS, that service is typically taken out of service for minutes to hours. From a user experience perspective, it is desired that users should be able to continue to use the cluster without interruption caused by master downtime. In a vsphere TM virtualized cluster, VMware vmotion TM technology and HA solution can address this requirement, either through live migration of the VM or by explicitly triggering a failover from the vcenter TM console with no or minimal downtime. Unplanned Downtime or Unresponsive Service: The failover of the NameNode service can occur due to hardware or OS failure, a failure of the NameNode daemon, or because the NameNode daemon becomes unresponsive for a few minutes. While this is not common, the failure can occur at unexpected times and may have an impact on meeting the SLAs of some critical applications. VMware HA solution automatically detects any hardware or VM failure to take appropriate action. In addition, the HA solution also provides a mechanism to detect failures of the NameNode service running in a VM. Detection of failures automatically triggers HA failover without any human intervention. In essence, the difference between planned and unplanned modes of failover is that planned failover is initiated by a human and unplanned failover is initiated by the system itself. Planned upgrades cause the biggest unavailability of the HDFS service because HDFS NameNode failures are not very common. Since most outages are planned, operators are in control of when those outages occur and can schedule in low utilization times. Hence a cold standby and failover HA solution should suffice for Namenode and can be enabled by VMware HA feature.

6 To support hot standby HA, VMware offers a feature called Fault Tolerance (FT). See Appendix for more details. Reference Architecture This section describes the Reference Architecture for providing a HA solution to key Apache Hadoop 1.0 services such as the NameNode. Design Principles for High Availability The key to architecting a highly available computing environment is to eliminate single points of failure. With the potential of occurring anywhere in the environment, failures can affect both hardware and software. Building redundancy at vulnerable points helps reduce or eliminate downtime caused by hardware failures. These include redundancies at the following layers: Server components such as network adaptors and host bus adaptors (HBAs) Servers, including blades and blade chassis Networking components Storage arrays and storage networking VMware vcenter TM Server is the management focal point for any VMware environment. Although VMware High Availability will continue to protect your environment without vcenter TM Server, the ability to manage the environment is severely impacted. It is highly recommended that you protect your vcenter TM Server instance as best as possible. This can be accomplished in the following ways: Use of VMware vcenter TM Server Heartbeat A specially designed HA solution for vcenter TM. Use of VMware High Availability Useful in environments where the vcenter TM Server instance is virtualized, such as when using the vsphere TM vcenter TM Server Appliance. Refer to VMware High Availability Deployment Best Practices [1] for further guidance on storage, networking, and cluster HA design considerations. Key Components of Apache Hadoop 1.0 High Availability Solution using VMware vsphere TM An Apache Hadoop cluster consists of two classes of hosts - masters and workers. Hadoop masters include the HDFS NameNode and MapReduce JobTracker. These nodes are single points of failure in Hadoop 1.0. If the hardware, OS, or master software for these hosts should fail, the Hadoop cluster becomes unavailable. In our reference architecture, this is resolved by deploying these hosts in a virtualized VMware HA cluster. The remaining hosts in a Hadoop cluster are workers. Apache Hadoop 1.0 has built in redundancy and failure recovery algorithms to detect and repair a Hadoop cluster, without interruption, when a worker fails. While these nodes do not need HA support, when designing a Hadoop cluster, there are many advantages to a uniform design where vsphere TM is used to manage all hosts. Virtualizing the entire Hadoop cluster is an option that brings additional operational and performance benefits such as automation, datacenter consolidation, rapid provisioning, higher resource utilization, a uniform system management, and monitoring framework for all nodes, leading to reduced TCO. [2]

7 Figure 2: VMware HA solution Ref. Arch. for Apache Hadoop 1.0 Reference Architecture described here consists of at least 2 ESX hosts that are configured to run in a single HA cluster, hosting the NameNode and JobTracker VM s. If an active Namenode or JobTracker fails, any other host in the HA cluster can become the active NameNode or JobTracker. There is no explicit host in the cluster configured as a hot or cold standby NameNode. A SAN is used to manage VM images and the server state of the NameNode and JobTracker. VMware vcenter TM is used to configure and manage the cluster. As Hadoop manages worker nodes failure transparently, the worker nodes are not configured for HA. They may be virtualized in an ESX cluster as described here [2] or deployed on physical hardware. Table 1: Reference Architecture Components COMPONENT NAME Name Node Job Tracker Workers (TaskTracker, and Data Nodes) CONFIGURATION One VM running the NameNode and monitoring tool on host1 with Shared Storage (SAN) One VM running the JobTracker on host2 with Shared Storage (SAN) One VM or physical machine running both components with direct attached storage (DAS) for HDFS and MapReduce.

8 New components Application health monitoring components VMware HA solution enables the HA agent on a host to heartbeat with a virtual machine through VM Tools or to an agent running within the virtual machine that is monitoring the application health. After the loss of a defined number of VMware Tools heartbeats with the virtual machine, HA will reset the virtual machine. Hortonworks has contributed an open source monitoring agent to the Apache Ambari project. [3] Client Patch To complete the HA solution, the failure of the HDFS NameNode not only needs to be detected and repaired, HDFS clients also need to behave gracefully, while the HA system detects and repairs the fault in the HDFS master. To support this, the standard HDFS client s existing retry logic was extended to support a configurable window during which it regularly attempts to reconnect to the NameNode. Assuming NameNode service is restored within that window, clients seamlessly continue their work, allowing HA to be transparent to HDFS. [4] Failover Test Scenarios Name Node host or VM Failure When the physical host running NameNode is failed, or the host OS is crashed, VMware HA solution automatically restarts the NameNode VM on one of the available hosts in the cluster. When the VM/Guest OS running the NameNode fails, VMware HA solution automatically restarts the NameNode VM on the same physical host in the cluster. In both cases, the failure is transparent to the user or any other services in the system, and failover involved no human intervention, except for when triggered manually. This capability is provided by VMware HA solution out-of-the box and applies to planned as well as unplanned downtime. vmotion TM was tested as well, a capability that can be used for manually moving the live NameNode VM from one host to another without any downtime. Name Node application Failure The new Monitoring Tool, introduced earlier in the document is used to enable and demonstrate the HA capability of the NameNode application running in a VM. When the NN process fails to start or crashes for reasons such as JVM crash, failure is reported by the monitoring tool to vsphere TM which in turn takes appropriate action to restart the NameNode VM.

9 Conclusion VMware vsphere TM 5.0 and VMware HA helps eliminate single points of failure in the deployment of business-critical applications in virtual machines. VMware HA solution works well with Apache Hadoop master services like NameNode to greatly simplify application provisioning while also providing an easy-touse, cost-effective high-availability, and failover solution. The results presented here represent a small part of ongoing work surrounding HA solution for virtualized Hadoop. Future documents will deal with results of applying FT and HA solution to other Hadoop master services, as well as performance and sizing analysis. Appendix Test Configuration Hardware Servers: o Dell R510 o 2 x Intel Xeon Processor X GHz 6 Core o 6 x 8GB 1333Mhz Dual Ranked LV RDIMMs o 2 x 500GB 7.2K RPM SATA 3.5 HD (ESX OS in Raid 1) o 8 x 1TB 7.2K RPM SATA 3.5 HD (JBOD) o 1 x PERC H700 Integrated RAID Controller 512MB Cache o 1 x QLogic (QLE2560) 8Gbps PCI-E FC HBA o o 1 x Broadcom 5716 dual-port gigabit Ethernet without TOE enabled 1 x Broadcom NetXtreme II Dual Port SFP+ Direct Attach 10GbE NIC PCIe x8 with TOE and iscsi Switches: o Extreme Summit x port switch o Arista 7124SX 24-port 10GbE switch o Brocade DS-300 FC switch Storage: o EMC VNX5700 Linux Distribution : CentOS bit Java : Sun Java 1.6.0_22 Native OS CPU, memory and disk 1VM per host, same as hardware Hypervisor vsphere TM ESXi 5.0, Build No vcenter TM 5.0.0, Build No Virtual Machines VMware Tools : installed Virtual network adapter : E1000

10 Virtual SCSI controller : LSILogicParallel Disks : VMDK 1 VM per host : 2 Master Nodes: 12G memory, 4 vcpus, shared storage 4 Worker Nodes : 12G memory, 4 vcpus, 8 local disks + shared storage (used for OS image) Hadoop Distribution : Hortonworks Data Platform Active NameNode : Host1 Active JobTracker : Host2 Workers : All hosts/vms from Host3 Host6 java.opts = -Xmx2048m Xmn512m Cluster topology o Native, 1 VM per host: all hosts in the default rack Figure 3: Test Cluster Layout

11 VMware vmotion TM Technology vmotion TM eliminates the need to schedule application downtime due to planned server maintenance through live migration of virtual machines across servers with no disruption to users or loss of service. With vmotion TM, resources can be dynamically reallocated across physical hosts. Figure 4: VMware vmotion TM Technology VMware Fault Tolerance Fault Tolerance (FT) means zero down time, and should not include any human intervention. In VMware virtualized environment, VMware Fault Tolerance (FT) can be designed to do so with minimum impact to the SLA. Using the VMware s vlockstep technology, FT on the ESXi host platform provides continuous availability by protecting a virtual machine (the Primary virtual machine) with a shadow copy (secondary virtual machine). The Secondary virtual machine can take over execution at any point without service interruption or loss of data because it is in virtual lockstep with the primary virtual machine. Figure 5: VMware Fault Tolerance (FT) Solution

12 VMware Distributed Resource Scheduling (DRS) In deployments with shared storage, Dynamic Resource Scheduling (DRS) dynamically load balances server resources to deliver the right resource to the right application based on business priority; allowing applications to shrink and grow as required. DRS performs an initial placement of the virtual machine on a host. As the cluster conditions change (for example, load and available resources), DRS uses vmotion TM to migrate virtual machines to other hosts as required. This capability can be used by the master nodes when more than one host is available in the cluster to dynamically balance the load across physical hosts. Figure 6: VMware DRS Solution References 1. VMware High Availability (VMware HA) Deployment Best Practices: 2. Virtualizing Hadoop: 3. Application health Monitoring Tool: HDFS Client Patch: About VMware VMware (NYSE:VMW), the global leader in virtualization and cloud infrastructure, delivers customer-proven solutions that accelerate IT by reducing complexity and enabling more flexible, agile service delivery. VMware enables enterprises to adopt a cloud model that addresses their unique business challenges. VMware s approach accelerates the transition to cloud computing while preserving existing investments and improving security and control. With more than 300,000 customers and 25,000 partners, VMware solutions help organizations of all sizes lower costs, increase business agility and ensure freedom of choice.

13 About Hortonworks Hortonworks was formed by the key architects and core Hadoop committers from the Yahoo! Hadoop software engineering team that led the effort to design and build every major release of Apache Hadoop from 0.1 to the most current stable release, contributing more than 80% of the code along the way. The team continues to be the leading contributor to core Hadoop (MapReduce and HDFS) as well as many other important Hadoop projects. We are also the major driving force behind the next generation of Apache Hadoop.