Hadoop, Easier than Ever

Transcription

1 Hadoop, Easier than Ever Etu TM Appliance. The Only Choice for Hadoop Platforms Enjoy the beauty of Hadoop cluster deployment and simple management

2 Why is an Etu TM Appliance Cluster indispensable to a Hadooper? For the sake of concision... high-performance, security, reliability, and easy management. For a developer and system administrator, dealing with Hadoop is pretty well known as a challenge or more simply put a pain in the neck. Think about this. As a Hadoop-based application developer or a Hadoop system administrator, what is a reasonable amount of time for you to get a 10-node Hadoop cluster up and running? Would it be three days, one day, three hours, or maybe one hour? Etu TM Appliance guarantees as short as 10+ minutes. Imagine a product designed for running Hadoop jobs that enables you to process Big Data on the same cluster scale with a five-fold leap in performance every day compared to current systems. Would it not make more sense to have your system capable of satisfying your users by replacing the existing application system with one that offers instantaneous access to it? Etu TM Appliance is characterized as a high-performance product offering a computing capacity that is three to twelve times more powerful than those self-installed ones. Are you still trying to figure out a solution for Hadoop Name Node High Availability? In reality, the job has been done with a broader built-in HA that covers the whole cluster system: computing, storage, host, network services, security mechanisms, network interfaces, all beyond a Hadooper s imagination of HA. Etu TM Appliance sets a new benchmark for the reliability of a Hadoop system. In regards to storage and computing of Big Data, how does one get the full benefit of an emphasized multi-tenancy in cloud computing? Imagine achieving the goal of authenticating and authorizing different users and applications to access segmented data. Etu TM Appliance incorporates a highly integrated Kerberos and LDAP as an ingenious information artisan to guarantee multi-tenant security. As a business grows a Hadoop cluster should ideally be able to scale-out accordingly. Imagine you have a 50-node cluster that needs a system adjustment overhaul, parameter optimization, and software maintenance. How many hosts do you have to connect to and how many command lines are needed to accomplish this task? Etu TM Appliance boosts first call resolution with one GUI and one URL. 2 Etu TM Appliance

3 Key Features 1. Highly Integrated Appliance A Hadoop-based Big Data processing platform that combines distributed storage with parallel computing, which integrates software and hardware. Etu TM Appliance is a design of simplicity and optimization that is an agglomeration of high-performance, effective management, security, reliability, scalability, and openness. 2. High-Performance Operating System Etu TM OS is specifically tailored for Hadoop Big Data clusters. It is the key for optimization for operational effectiveness as well as simplified deployment/management. 3. One-Click Rapid Deployment Starting from bare metal to up and running with an installed comprehensive software stack and optimized configuration, an Etu TM Appliance cluster with more than 100 s can be automatically deployed and configured in 10+ minutes. 4. Full System High Availability Using a full system rather than just the Hadoop Name Node High Availability comprehensively prevents the risk for a single point failure. 6. Ad Hoc Network Design Etu TM Appliance is designed to combine the multi-interface bandwidth and adapter fault tolerance allowing considerable high-performance network transmission and high availability. 7. Human-Based Centralized Etu TM Appliance offers a web-based graphical user interface that designates central management and settings across nodes in cluster. 8. Low-Risk Adoption The minimal Etu TM Appliance cluster contains 1 Master Node and 2 s, which offers the flexibility in purchasing less than one full Hadoop rack or even half. 9. Linear Scalability Depending on the unit(s) of s, an Etu TM Appliance cluster can be scaled-out up to tens of hundreds folds. 10. Open and Value-Added Platform The open Big Data processing platform enables a variety of applications to be deployed on it. 5. Multi-Tenant Security Etu TM Appliance is built in with a multi-tenant identity and access control mechanism providing a level of high data security comparable to a public cloud. 3 Etu TM Appliance

4 The differences between an Etu TM Appliance and a general Hadoop platform Etu TM Appliance General Hadoop Platform Operating System Etu TM OS is specialized for running Hadoop jobs. General Linux distributions. Computing Performance Offers an increase in computing performance 3 to 12 times higher than a general Hadoop platform through a full system optimization. General system with a non calibrated computing performance. Data Acquisition An Etu TM Log Collector carries the capability of receiving UDP packets higher than 60,000 EPS per node, along with an accuracy rate of more than 99.99%. General data collection performance without optimization. Cluster Deployment The ability to initiate one click deployment, meaning it has the ability to get deployment and setup finished for more than 100 s from bare metal to up and running within 10+ minutes while giving attention to a HA system architecture. Before the installation of a Hadoop Ecosystem software stack it requires the manual installation of an operating system and cluster services for each node. After this then it can be configured to run. Distinctive Network Design Combines the multi-interface bandwidth that allows for HA. General network lacks in design for performance and fault tolerance. High Availability Architecture The built-in full system HA is highly integrated covering from computing, storage, host, network services, security mechanisms, network interfaces, and etc. to avoid a single point of failure in all aspects. Generally, a Hadoop Name Node HA mechanism is provided and users are required to carry out many detailed steps, which might lead to a high failure rate. Multi-tenant Security The built-in Kerberos and LDAP fully meet the requirements of an enterprise-level multi-tenant platform authentication and authorization. Users are required to implement Kerberos and LDAP integration, which ensues a lengthy and complicated process. Other than that, threshold of a system integration is very high. Cluster A web-based graphical user interface that designates central management and settings across nodes in cluster. A root access is needed to enter each node to manage the host system. 4 Etu TM Appliance

5 How Etu TM Appliance Works Etu has taken into consideration an enterprise s investment augmentation Minimum Package: 1+2 model and acknowledge the concerns of efficiency and effectiveness. Therefore, an Etu Appliance allows your enterprise to immediately benefit from the effects whether the implementation ranges from small-scale to large-scale. Again remember, because of the flexibility that Etu TM Appliance allows, transitioning into Switch large-scale is efficiently done when it comes time for future expansions. An Master Node Etu TM Appliance cluster simply deploys three nodes (one Master Node with two Scale-out up to thousands of nodes s) to scale out and allows the option to grow by simply adding Figure 1. Etu TM Appliance minimum package and additional s to consistently scale-out architecture (Note: Switch is not included in Etu TM offering) handle the increasing workload created by Hadoop Big Data. During the scale-out, the operation requires no downtime. The integration of an Etu TM Appliance is painless and virtually seamless with an advantage of being fluidly flexible to meet the expansion needs for all enterprises. Begin by investing in a minimum package and expand accordingly to your business growth needs. A minimum package for an Etu TM Appliance consists of a cluster of three nodes (1 Master Node + 2 Worker Nodes). As your Hadoop Big Data processing workloads grow we simply increase s through scale-out expansions without interrupting running services. Benefits Enterprises that implement a Hadoop Big Data platform with an Etu TM Appliance can easily gain the following benefits: 1. Staff can do their Hadoop-related jobs respectively and make progress all together, which improves professional performance. When it comes to storage and computing for multi-structured Big Data, neither developers nor system administrators are free from taxing Hadoop Ecosystem problems. These challenges range from software stacks that are unfavorable to installation, to difficulty in managing system and applications, to not knowing the right way to reach optimization, and etc. Etu TM Appliance saves time for developers and system administrators, which is literally improving the performance of your professionals through efficiency. 2. More confidence in Mission Critical Big Data computing. Etu TM Appliance recognizes the Full System HA, enabling enterprises to feel more at ease and gain a peace of mind in knowing that the relatively more crucial Big Data computing tasks can be shifted to a Hadoop platform that will continuously extract value from multi-structured Big Data. 3. A Big Data PaaS is ready for a full run. Etu TM Appliance s centralized Web-based Console functions as a PaaS user portal for enterprises and internet service providers that deserve the best application platform solution to erect both public and private clouds. Besides simplifying Hadoop cluster management, companies can achieve enterprise-level multi-tenant authentication, authorization, and meet major requirements in self-service, security, scalability, and stability. 4. Ability to access and compute accumulated data repeatedly, which can offer unlimited business value. This platform is characterized of having a high level of openness, allowing it to have multiple applications run on it. As time goes by, the data that has been saved in an Etu TM Appliance cluster will increase the value of your business. 5. Performance doubles, TCO drops To focus on performance optimization we start from the distinctive operating system and then continue forward throughout the whole system. Etu TM Appliance compliments Hadoop and takes advantage of the features such as parallel computing architecture and linear scalability. Compared to general Hadoop platforms, the adoption of an Etu TM Appliance cluster reduces the number of nodes undertaking computing tasks, which significantly cuts down on an enterprise s CAPEX and management costs. 5 Etu TM Appliance

6 Etu TM Appliance Functionalities Etu TM Module Apache Hadoop Ecosystem Module Etu TM Console Application Table File Data Source Cluster Data Source Data Processing Module Etu TM Clusterware Sqoop Pig HiveQL Mahout SNMP FTP MapReduce Deployment Syslog Account Etu TM DataFlow Data Store Module Security Hive Meta Store HBase Configuration HDFS High Availability Etu TM OS Figure 2. Etu TM Appliance Software Architecture Software Module List Etu TM OS Etu TM Clusterware Auto-deployment Account Security Auto-configuration Etu TM Console Application Cluster Table File Data Source High Availability SNMP Big Data Store Etu TM Data Source FTP Syslog HDFS Hive Meta Store HBase Etu TM DataFlow Sqoop Big Data Processing MapReduce Pig HiveQL Mahout Data Formats Semi-structured data: such as log, XML, CSV, Meta Data, and other text files with pre-defined fields Unstructured data: such as full text, web page, content, and various binary files* * Specific processing module might be required to respectively process file formats 6 Etu TM Appliance

7 Etu TM Appliance 2 Edition Comparison Chart Cluster One-click Deployment Hadoop Ecosystem Deployment & Readiness Checks Operating System Deployment 1 Bare Metal Deployment for Built-in NTP/DNS/DHCP Services System Level Cluster Configuration Service Multiple NIC Bonding for Bandwidth Aggregation 2 Multiple NIC Bonding for Failover 2 High Availability 3 Built-in Storage HA with Distributed File System Full System Failover Isolation Mechanism 4 Hadoop Name Node HA Hadoop Ecosystem HA 5 Hive Meta Store HA DNS/DHCP Service HA Deployment/Configuration Services HA LDAP/Kerberos Services HA Configuration Role Instance Configuration and HDFS / MapReduce / HBase / Hive Configuration Etu TM Clusterware Configuration Operating System Kernel and System Configuration Automated Time Sync Configuration Automated DNS/DHCP/Host Configuration Automated SSH Key and Host CA Automated Network Configuration Automated Monitoring Configuration Cluster Monitoring Proactive Health Checks Status and Healthy Summary Performance Monitoring Host Monitoring (CPU/RAM/HDD/NIC) JVM Metrics Monitoring SNMP Integration Security Kerberos Principals and Keytabs Deployment Built-in Kerberos Built-in LDAP Automated Principal Creation Automated Keytabs Creation Automated User Environment Configuration Console Cluster Cluster Status Node Service User License Cluster Monitoring High Availability S S: Standard Edition E: Enterprise Edition E S E Application Application Status Application Deployment Application Execution Application Scheduling Data Source DataFlow FTP Service Syslog Service HDFS HDFS Cluster Status HDFS File Browser HBase HBase Cluster Status HBase Table 1. Etu TM OS is applied as the operating system 2. Extra NIC is required for implementing the Multiple NIC Bonding for Network Failover or Multiple NIC Bonding for Bandwidth Aggregation. 3. An extra Master Node is required to enable the Etu TM HA option. It is available in the Standard Edition as well as the Enterprise Edition. 4. To acquire the highest-level of Full System Failover Isolation Mechanism it may require user to purchase additional network interface cards for all the nodes and switches. 5. Hadoop Ecosystem HA consists of HDFS / MapReduce (Job Tracker) / HBase / Hive Thrift. Etu TM Appliance Specifications By Role Model Specifications Role CPU Memory Hard Drive Network Interface Card Power User Data Etu 1000M Series Master Node 64-Bit 12 Core 48GB K RPM SAS 300GB x 2 (RAID 1) 1 Gbit Ethernet Dual Port x 1 or 2 Dual Power N/A Etu 1000W Series 64-Bit 12 Core 48GB K RPM SATA 2TB x 4 or 3TB x 4 1 Gbit Ethernet Dual Port x 1 or 2 Single Power 4~40TB or 6~60TB (Depends on compression ratio) Etu 2000W Series 64-Bit 12 Core 48GB K RPM SATA 2TB x 8 or 3TB x 8 1 Gbit Ethernet Dual Port x 1 or 2 Single Power 8~80TB or 12~120TB (Depends on compression ratio) 1. Etu TM Appliance Specification Sheet is provided by request. 2. One Etu TM Appliance cluster at least includes 1 Master Node and 2 s. 3. One Etu TM Appliance cluster with HA at least includes 2 Master Nodes and 2 s. 4. User Data can be stored with more than two replicas if necessarily and store them on different s. 7 Etu TM Appliance

8 About Etu Etu is a Big Data pioneer in providing Big Data End-to-End solutions for Enterprises in Asia. Etu is dedicated to develop Hadoop-based Big Data platform technology in particular processing analytics. Cooperating with ISV/SI partners who have strong capabilities in application development and integration from various vertical markets, Etu is committed to help customers to discover, unlock, and connect business values hidden in semi-structured and unstructured data. Our team members have large-scale Big Data processing experience in online services and several of them have earned high level Cloudera Certified Developer/Administrator for Apache Hadoop certifications. Contact information For more information about Etu Appliance, please visit Etu at our official website: Trademark Disclaimer Etu, Etu Appliance and Etu Recommender are trademarks of SYSTEX GROUP. All other brands and trademarks referenced herein are acknowledged to be trademarks or registered trademarks of their respective holders. EA2_EN_2 Eco-friendly Printing with Soy Ink 〡 FSC Certified Paper