Lenovo Big Data Reference Architecture for Cloudera Distribution for Hadoop

Transcription

1 Lenv Big Data Reference Architecture fr Cludera Distributin fr Hadp Last update: 19 August 2015 Cnfiguratin Reference Number: BDACLDRXX53 Industry-leading Lenv prducts prvide a scalable, flexible, integrated platfrm CDH is the mst cmplete, tested, and ppular distributin f Apache Hadp Uses pwerful, versatile new Lenv System x3650 M5 server Leading security and reliability Innvative energy efficiency and perfrmance He DaShuang (Lenv) Ajay Dhlakia (Lenv) Dwai Lahiri (Cludera)

2 Table f Cntents 1 Intrductin Business prblem and business value Business prblem Business value Requirements Functinal requirements Nn-functinal requirements Architectural verview Cmpnent mdel Operatinal mdel Hardware descriptin Lenv System x3650 M5 Server Lenv System x3550 M5 Server Lenv RackSwitch G Lenv RackSwitch G Cluster ndes Data ndes Management ndes Systems management Netwrking Data netwrk Hardware management netwrk Multi-rack netwrk Predefined cluster cnfiguratins Deplyment cnsideratins Increasing cluster perfrmance Designing fr lwer cst ii Lenv Big Data Reference Architecture fr Cludera Distributin fr Hadp

3 7.3 Designing fr high ingest rates Estimating disk space Scaling cnsideratins High availability cnsideratins Netwrking cnsideratins Hardware availability cnsideratins Strage availability Sftware availability cnsideratins Migratin cnsideratins Appendix: Bill f Materials Management nde Data nde Management/Administratin netwrk switch Data netwrk switch Rack Cables Acknwledgements Resurces iii Lenv Big Data Reference Architecture fr Cludera Distributin fr Hadp

4 1 Intrductin This dcument describes the reference architecture fr the Big Data Slutin frm Cludera. It prvides a predefined and ptimized hardware infrastructure fr the Cludera Distributin fr Hadp (CDH versin 5.4, a distributin f Hadp with value added capabilities frm Cludera. This reference architecture prvides the planning, design cnsideratins, and best practices fr implementing CDH with Lenv prducts. The Lenv and Cludera teams wrked tgether n this dcument, and the reference architecture that is described herein was validated by Lenv and Cludera. Cludera brings the pwer f Hadp t the enterprise. Apache Hadp is the pen surce sftware framewrk that is used t reliably manage large vlumes f structured and unstructured data. Cludera enhances this technlgy t withstand the demands f yur enterprise, adding administrative, wrkflw, prvisining, and security features. The result is that yu get a mre develper and user-friendly slutin fr cmplex, large-scale analytics. The predefined cnfiguratin prvides a baseline cnfiguratin fr a big data slutin, which can be mdified based n the specific custmer requirements, such as lwer cst, imprved perfrmance, and increased reliability. The intended audience f this dcument is IT prfessinals, technical architects, sales engineers, and cnsultants t assist in planning, designing, and implementing the big data slutin with Lenv hardware. It is assumed that yu are familiar with Apache Hadp cmpnents and capabilities. Fr mre infrmatin abut Hadp, see Acknwledgements This reference architecture dcument has benefited very much frm the detailed and careful review cmments prvided by Mike Perks f Lenv. Resurces n page Lenv Big Data Reference Architecture fr Cludera Distributin fr Hadp

5 2 Business prblem and business value This sectin describes the business prblem that is assciated with big data envirnments and the value that is ffered by the Cludera slutin that uses Lenv hardware. 2.1 Business prblem By 2009, the wrld generated 800 billin GB f data, a level that is expected t increase t 40 trillin GB by In all, 90% f the data in the wrld tday was created in the last tw years alne. This data cmes frm everywhere, including sensrs that are used t gather climate infrmatin, psts t scial media sites, digital pictures and vides, purchase transactin recrds, and cell phne glbal psitining system (GPS) signals. This data is big data. Big data spans the fllwing dimensins: Vlume: Big data cmes in ne size: large. Enterprises are awash with data, easily amassing terabytes and even petabytes f infrmatin. Velcity: Often time-sensitive, big data must be used as it is streaming int the enterprise t maximize its value t the business. Variety: Big data extends beynd structured data, including unstructured data f all varieties, such as text, audi, vide, click streams, and lg files. Big data is mre than a challenge; it is an pprtunity t find insight int new and emerging types f data t make yur business mre agile. Big data als is an pprtunity t answer questins that, in the past, were beynd reach. Until nw, there was n effective way t harvest this pprtunity. Tday, Cludera uses the latest big data technlgies, such as the massive map-reduce scale-ut capabilities f Hadp, t pen the dr t a wrld f pssibilities. 2.2 Business value Apache Hadp is the pen surce sftware framewrk that is used t reliably manage large vlumes f structured and unstructured data. Cludera enhances this technlgy t withstand the demands f yur enterprise, adding administrative, wrkflw, prvisining, and security features. The result is that yu get a mre develper and user-friendly slutin fr cmplex, large-scale analytics. Hw can businesses prcess tremendus amunts f raw data in an efficient and timely manner t gain actinable insights? Cludera allws rganizatins t run large-scale, distributed analytics jbs n clusters f cst-effective server hardware. This infrastructure can be used t tackle large data sets by breaking up the data int chunks and crdinating data prcessing acrss a massively parallel envirnment. After the raw data is stred acrss the ndes f a distributed cluster, queries and analysis f the data can be handled efficiently, with dynamic interpretatin f the data frmatted at read time. The bttm line: Businesses can finally get their arms arund massive amunts f untapped data and mine that data fr valuable insights in a mre efficient, ptimized, and scalable way. Cludera that is deplyed n Lenv System x servers with Lenv netwrking cmpnents prvides superir perfrmance, reliability, and scalability. The reference architecture supprts entry thrugh high-end cnfiguratins and the ability t easily scale as the use f big data grws. A chice f infrastructure cmpnents prvides flexibility in meeting varying big data analytics requirements. 2 Lenv Big Data Reference Architecture fr Cludera Distributin fr Hadp

6 3 Requirements The functinal and nn-functinal requirements fr this reference architecture are desribed in this sectin. 3.1 Functinal requirements A big data slutin supprts the fllwing key functinal requirements: Varius applicatin types, including batch and real-time analytics Industry-standard interfaces s that applicatins can wrk with Cludera Real-time streaming and prcessing f data Varius data types and databases Varius client interfaces Large vlumes f data 3.2 Nn-functinal requirements Custmers require their big data slutin t be easy, dependable, and fast. The fllwing nn-functinal requirements are key: Easy: Ease f develpment Easy management at scale Advanced jb management Multi-tenancy Dependable: Data prtectin with snapsht and mirrring Autmated self-healing Insight int sftware/hardware health and issues High availability (HA) and business cntinuity Fast: Superir perfrmance Scalability Secure: Strng authenticatin and authrizatin Kerbers supprt Data cnfidentiality and integrity 3 Lenv Big Data Reference Architecture fr Cludera Distributin fr Hadp

7 4 Architectural verview Figure 1 shws the main features f the Cludera reference architecture that uses Lenv hardware. Users can lg int the Cludera client frm utside the firewall by using Secure Shell (SSH) n prt 22 t access the Cludera slutin in the crprate netwrk. Cludera prvides several interfaces that allw administratrs and users t perfrm administratin and data functins, depending n their rles and access level. Hadp applicatin prgramming interfaces (APIs) can be used t access data. Cludera APIs can be used fr cluster management and mnitring. Cludera data services, management services, and ther services run n the ndes in cluster. Strage is a cmpnent f each data nde in the cluster. Data can be incrprated int the Cludera Hadp strage thrugh the Hadp APIs r netwrk file system(nfs), depending n the needs f the custmer. A database is required t stre the data fr Cludera manager, hive meta stre, and s n. Cludera prvides an embedded database fr test r prf f cncept (POC) envirnments and an external database is required fr prductin envirnment. Figure 1. Cludera architecture verview 4 Lenv Big Data Reference Architecture fr Cludera Distributin fr Hadp

8 5 Cmpnent mdel The CDH slutin prvides features and capabilities that meet the functinal and nn-functinal requirements f custmers. It supprts missin-critical and real-time big data analytics acrss different industries, such as financial services, retail, media, healthcare, manufacturing, telecmmunicatins, and gvernment rganizatins and leading Frtune 100 and Web 2.0 cmpanies. CDH is the wrld s mst cmplete, tested, and ppular distributin f Apache Hadp and related prjects. All f the packaging and integratin wrk is dne fr yu, and the entire slutin is thrughly tested and fully dcumented. By taking the guesswrk ut f building ut yur Hadp deplyment, CDH gives yu a streamlined path t success in slving real business prblems with big data. CDH is 100% Apache-licensed pen surce and is the nly Hadp slutin t ffer unified batch prcessing, real-time stream prcessing, interactive SQL, and interactive search and discvery, and rle-based access cntrls. The Cludera platfrm fr big data can be used fr varius use cases frm batch applicatins that use MapReduce r Spark with data surce, such as click streams, t real-time applicatins that use sensr data. Figure 2 shws the CDH key capabilities that meet the functinal requirements f custmers. Figure 2. CDH key capabilities 5 Lenv Big Data Reference Architecture fr Cludera Distributin fr Hadp

9 The CDH slutin cntains the fllwing cmpnents: Analytic SQL: Cludera Impala Cludera Impala is the industry s leading massively parallel prcessing (MPP) SQL query engine that runs natively in Apache Hadp. The Apache-licensed, pen surce Impala prject cmbines mdern, scalable parallel database technlgy with the pwer f Hadp, enabling users t directly query data stred in HDFS and Apache HBase withut requiring data mvement r transfrmatin. Impala is designed frm the grund up as part f the Hadp system and shares the same flexible file and data frmats, metadata, security, and resurce management framewrks that are used by MapReduce, Apache Hive, Apache Pig, and ther cmpnents f the Hadp stack. Search Engine: Cludera Search Cludera Search is Apache Slr that is integrated with CDH, including Apache Lucene, Apache SlrClud, Apache Flume, Apache Tika, and Apache Hadp MapReduce and HDFS. Cludera Search als includes valuable integratins that make searching mre scalable, easy t use, and ptimized fr near-real-time and batch-riented indexing. These integratins include Cludera Mrphlines, which is a custmizable transfrmatin chain that simplifies lading any type f data int Cludera Search. Machine Learning: Cludera ML Cludera ML is an Apache licensed cllectin f Java libraries and cmmand line tls t aid data scientists in perfrming cmmn data preparatin and mdel evaluatin tasks. Cludera ML is intended t be an educatinal resurce and reference implementatin fr new data scientists that want t understand the mst effective techniques fr building rbust and scalable machine learning mdels n tp f Hadp. Stream Prcessing: Apache Spark Apache Spark is an pen surce, parallel data prcessing framewrk that cmplements Apache Hadp t make it easy t develp fast, unified big data applicatins that cmbine batch, streaming, and interactive analytics n all yur data. In cllabratin with Databricks (the cmpany that is leading the develpment f Spark) Cludera ffers cmmercial supprt fr Spark with Cludera Enterprise. Spark is times faster than MapReduce, which delivers faster time t insight n mre data and results in better business decisins and user utcmes. Cludera Manager Cludera Manager is the industry s first and mst sphisticated management applicatin fr Apache Hadp and the enterprise data hub. Cludera Manager sets the standard fr enterprise deplyment by delivering granular visibility int and cntrl ver every part f the data hub, which empwers peratrs t imprve perfrmance, enhance quality f service, increase cmpliance, and reduce administrative csts. Cludera Manager makes administratin f yur enterprise data hub simple and straightfrward, at any scale. With Cludera Manager, yu can easily deply and centrally perate the cmplete big data stack. 6 Lenv Big Data Reference Architecture fr Cludera Distributin fr Hadp

10 The applicatin autmates the installatin prcess, which reduces deplyment time frm weeks t minutes; gives yu a cluster-wide, real-time view f ndes and services running; prvides a single, central cnsle t enact cnfiguratin changes acrss yur cluster; and incrprates a full range f reprting and diagnstic tls t help yu ptimize perfrmance and utilizatin. Cludera Navigatr Cludera Navigatr is the first fully integrated data management applicatin fr Apache Hadp-based systems. It prvides all f the capabilities that are required fr administratrs, data managers, and analysts t secure, manage, and explre the large amunts f diverse data that land in Hadp. Cludera Navigatr prvides the fllwing features: Data Audit and Access Cntrl: Verify apprpriate user r grup permissins and reprt n data access fr files (HDFS), recrds (HBase), and metadata (Hive). Discvery and Explratin: Determine what data is available and hw it is structured s that yu can use it effectively. Lineage: Trace the prgressin f data sets back t their riginal surces t verify reliability f results. Lifecycle Management: Ensure the crrect placement and retentin f data that is based n value r plicies. Cludera Manager Metrics Cludera Manager mnitrs a number f perfrmance metrics fr services and rle instances that are running n yur clusters. These metrics are mnitred against cnfigurable threshlds and can be used t indicate whether a hst is functining as expected. Yu can view these metrics in the Cludera Manager Admin Cnsle, which displays metrics abut yur jbs (such as the number f currently running jbs and their CPU r memry usage), Hadp services (such as the average HDFS I/O latency and number f cncurrent jbs), yur clusters (such as average CPU lad acrss all yur hsts) and s n. Cludera Manager Backup And Disaster Recvery Cludera Manager prvides an integrated, easy-t-use management slutin fr enabling data prtectin in the Hadp platfrm. Cludera Manager prvides rich functinality that is aimed twards replicating data that is stred in HDFS and accessed thrugh Hive acrss data centers fr disaster recvery scenaris. When critical data is stred n HDFS, Cludera Manager prvides the necessary capabilities t ensure that the data is available at all times, even in the face f the cmplete shutdwn f a data center. Cludera Manager als prvides the ability t schedule, save, and (if needed) restre snapshts f HDFS directries and HBase tables. Cludera Manager API The Cludera Manager API prvides cnfiguratin and service life cycle management, service health infrmatin and metrics, and allws yu t cnfigure Cludera Manager. The API is served n the same hst and prt as the Cludera Manager Admin Cnsle, and des nt require an extra prcess r extra cnfiguratin. The API supprts HTTP Basic Authenticatin, accepting the same users and credentials as the Cludera Manager Admin Cnsle. Table 1 n page 8 lists a Cludera Prduct Cmparisn with different features. 7 Lenv Big Data Reference Architecture fr Cludera Distributin fr Hadp

11 Table 1. Cludera Prduct Cmparisn Features Licensing Term Cludera Express Unlimited/ Free Cludera Enterprise Basic Editin Flex Editin Data Hub Editin Annual Subscriptin 100% Open Surce Data Platfrm, pwered by Apache Hadp (CDH) Cre: Hadp, Flume, HBase, HCatalg, Hive, Hue, Impala, Kafka, Mahut, Ozie, Pig, Cludera Search, Sentry, Spark, Sqp, Whirr, ZKeeper System Management (Cludera Manager) Deplyment and Cnfiguratin Service Management Service and Hst Mnitring Security Management Diagnstics (Lg Search, Events, and mre) Extensibility and APIs Rlling Updates/Restarts AD Kerbers Integratin SNMP Supprt LDAP Integratin Cnfiguratin Histry and Rllbacks Operatinal Reprts Scheduled Diagnstics Backup & Disaster Recvery Supprted Cmpnents Cre Hadp 1 Cludera Directr Online NSQL DBMS (Apache HBase r Accumul) Interactive SQL (Impala) Interactive Analytics (Apache Spark) Chse One per Cluster 1. Supprt fr Cre Hadp includes: Hadp, Flume, HCatalg, Hive, Hue, Kafka, Mahut, Ozie, Pig, Sentry, Sqp, Whirr, and ZKeeper 8 Lenv Big Data Reference Architecture fr Cludera Distributin fr Hadp

12 Features Search (Cludera Search) Data Gvernance, Audit, Lineage, and Discvery (Cludera Navigatr) Agile Deplyment Optins Cludera Express Cludera Enterprise Basic Editin Flex Editin Data Hub Editin Cludera Directr Supprt Features Dedicated Glbal Supprt Team Practive Technical Guidance Predictive Issue Analysis Cmprehensive Knwledge Base Prductin slutin Guides Prduct Radmap Input 8x5 2 r 24x7 Expert Supprt Premium Supprt 3 Optin Open Surce Legal Assurance / Indemnificatin Fr mre infrmatin, see this website: cludera.cm/cntent/cludera/en/prducts-and-services/prduct-cmparisn.html The Cludera slutin is perating system independent. Cludera supprts many Linux perating systems, including Red Hat Linux and SUSE Linux. Fr mre infrmatin abut the versins f supprted perating systems, see this website: cludera.cm/cntent/cludera/en/dcumentatin/cre/latest/tpics/cm_ig_cm_requirements.html. 2. Custmer's lcal time 3. 24x7 with 15 minute guaranteed respnse time fr critical issues 9 Lenv Big Data Reference Architecture fr Cludera Distributin fr Hadp

13 6 Operatinal mdel This sectin describes the peratinal mdel fr the Cludera reference architecture. T shw the peratinal mdel fr different sized custmer envirnments, fur different mdels are prvided fr supprting different amunts f data. Thrughut the dcument, these mdels are referred t as starter rack, half rack, full rack, and multi-rack cnfiguratin sizes. The multi-rack is three times larger than the full rack. A Cludera deplyment cnsists f cluster ndes, netwrking, pwer, and racks. The predefined cnfiguratins can be implemented as is r mdified based n specific custmer requirements, such as lwer cst, imprved perfrmance, and increased reliability. Key wrklad requirements, such as the data grwth rate, sizes f datasets, and data ingest patterns help in determining the prper cnfiguratin fr a specific deplyment. A best practice when a Cludera cluster infrastructure is designed is t cnduct the prf f cncept testing by using representative data and wrklads t ensure that the prpsed design wrks. 6.1 Hardware descriptin This reference architecture uses Lenv System x3650 M5 and x3550 M5 servers and Lenv RackSwitch G8052 and G8272 tp f rack switches Lenv System x3650 M5 Server The Lenv System x3650 M5 server (as shwn in Figure 3) is an enterprise class 2U tw-scket versatile server that incrprates utstanding reliability, availability, and serviceability (RAS), security, and high-efficiency fr business-critical applicatins and clud deplyments. It ffers a flexible, scalable design and simple upgrade path t inch hard disk drives (HDDs) r slid-state drives (SSDs), r inch HDDs with dubled data transfer rate thrugh 12 Gbps serial-attached SCSI (SAS) internal strage cnnectivity and up t 1.5 TB f TruDDR4 Memry. On-bard it prvides fur standard embedded Gigabit Ethernet prts and tw ptinal embedded 10 Gigabit Ethernet prts withut ccupying PCIe slts. Figure 3. Lenv System x3650 M5 Cmbined with the Intel Xen prcessr E v3 prduct family, the Lenv x3650 M5 server ffers an even higher density f wrklads and perfrmance that lwers the ttal cst f wnership (TCO) per virtual machine. Its pay-as-yu-grw flexible design and great expansin capabilities slidify dependability fr any kind f virtualized wrklad with minimal dwntime. The x3650 M5 server prvides internal strage density f up t 87.6 TB in a 2U frm factr with its impressive array f wrklad-ptimized strage cnfiguratins. It als ffers easy management and saves flr space and pwer cnsumptin fr mst demanding strage virtualizatin use cases by cnslidating strage and server int ne system. 10 Lenv Big Data Reference Architecture fr Cludera Distributin fr Hadp

14 The reference architecture recmmends the strage-rich System x3650 M5 mdel fr the fllwing reasns: Strage capacity: The ndes are strage-rich. Each f the inch drives has raw capacity up t 6 TB and each f tw 2.5-inch drives has raw capacity f 1.8 TB fr a ttal f 87.6 TB per nde and ver 1 petabyte per rack. Perfrmance: This hardware supprts the latest Intel Xen prcessrs and TruDDR4 Memry. Flexibility: Server hardware uses embedded strage, which results in simple scalability (by adding ndes). Mre PCIe slts: Up t 8 PCIe slts are available if rear disks are nt used, and up t 2 PCIe slts if bth Rear 3.5-inch HDD Kit and Rear 2.5-inch HDD Kit are used. They can be used fr netwrk adapter redundancy and increased netwrk thrughput. Better pwer efficiency: Innvative pwer and thermal management prvides energy savings. Reliability: Lenv is first in the industry in reliability and has exceptinal uptime with reduced csts. Fr mre infrmatin, see the fllwing Lenv System x3650 M5 website : lenvpress.cm/tips Lenv System x3550 M5 Server The Lenv System x3550 M5 server (as shwn in Figure 4) is a cst- and density-balanced 1U tw-scket rack server. The x3550m5 features a new, innvative, energy-smart design with up t tw Intel Xen prcessrs f the high-perfrmance E v3 prduct family prcessrs a large capacity f faster, energy-efficient TruDDR4 Memry, up t twelve 12Gb/s SAS drives, and up t three PCI Express (PCIe) 3.0 I/O expansin slts in an impressive selectin f sizes and types. The server s imprved feature set and exceptinal perfrmance is ideal fr scalable clud envirnments. Figure 4: Lenv System x3550 M5 Fr mre infrmatin, see the fllwing Lenv System x3550 M5 website: lenvpress.cm/tips Lenv RackSwitch G8052 The Lenv System Netwrking RackSwitch G8052 (as shwn in Figure 5) is an Ethernet switch that is designed fr the data center and prvides a virtualized, cler, and simpler netwrk slutin. The Lenv RackSwitch G8052 ffers up t 48 1 GbE prts and up t 4 10 GbE prts in a 1U ftprint. The G8052 switch is always available fr business-sensitive traffic by using redundant pwer supplies, fans, and numerus high-availability features. Figure 5. Lenv RackSwitch G Lenv Big Data Reference Architecture fr Cludera Distributin fr Hadp

15 Lenv RackSwitch G8052 has the fllwing characteristics: A ttal f 48 1 GbE RJ45 prts Fur standard 10 GbE SFP+ prts Lw 130W pwer rating and variable speed fans t reduce pwer cnsumptin Fr mre infrmatin, see this website: lenvpress.cm/tips Lenv RackSwitch G8272 Designed with tp perfrmance in mind, Lenv RackSwitch G8272 is ideal fr tday s big data, clud, and ptimized wrklads. The G8272 switch ffers up t Gb SFP+ prts in a 1U frm factr and is expandable with fur 40 Gb QSFP+ prts. It is an enterprise-class and full-featured data center switch that delivers line-rate, high-bandwidth switching, filtering, and traffic queuing withut delaying data. Large data center grade buffers keep traffic mving. Redundant pwer and fans and numerus HA features equip the switches fr business-sensitive traffic. The G8272 switch (as shwn in Figure 6) is ideal fr latency-sensitive applicatins, such as client virtualizatin. It supprts Lenv Virtual Fabric t help clients reduce the number f I/O adapters t a single dual-prt 10 Gb adapter, which helps reduce cst and cmplexity. The G8272 switch supprts the newest prtcls, including Data Center Bridging/Cnverged Enhanced Ethernet (DCB/CEE) fr supprt f FCE and iscsi and NAS. Figure 6: Lenv RackSwitch G8272 The enterprise-level Lenv RackSwitch G8272 has the fllwing characteristics: 48 x SFP+ 10GbE prts plus 6 x QSFP+ 40GbE prts Supprt up t 72 x 10Gb cnnectins using break-ut cables 1.44 Tbps nn-blcking thrughput with lw latency (~ 600 ns) Up t 72 1Gb/10Gb SFP+ prts OpenFlw enabled allws fr easily created user-cntrlled virtual netwrks Fr mre infrmatin, see this website: lenvpress.cm/tips Cluster ndes The Cludera reference architecture is implemented n a set f ndes that make up a cluster. A Cludera cluster cnsists f tw types f ndes, data ndes and management ndes. Data ndes use System x3650 M5 servers with lcally attached strage and management ndes use System x3550 M5 servers. Data ndes run data (wrker) services fr string and prcessing data. Management ndes run the fllwing types f services: Management (cntrl) services fr crdinating and managing the cluster Miscellaneus services (ptinal) fr file and web serving 12 Lenv Big Data Reference Architecture fr Cludera Distributin fr Hadp

16 6.2.1 Data ndes Table 2 lists the recmmended system cmpnents fr data ndes. Table 2. Data nde cnfiguratin Cmpnent System Prcessr Memry - base Data nde cnfiguratin System x3650 M5 2 x Intel Xen prcessr E v3 2.3GHz 10-cre 128 GB: 8 x 16GB 2133MHz RDIMM (minimum) Disk (OS) 900 GB drives: 2 x 900GB SAS 2.5 inch 1.2 TB drives: 2 x 1.2TB SAS 2.5 inch 1.8 TB drives: 2 x 1.8TB SAS 2.5 inch Disk (data) 3 TB drives: 14 x 3TB NL SATA 3.5 inch (42 TB Ttal) 4 TB drives: 14 x 4TB NL SATA 3.5 inch (56 TB Ttal) HDD cntrller Hardware strage prtectin Hardware management netwrk adapter Data netwrk adapter 12 Gb JBOD cntrller Nne (JBOD). By default, Cludera maintains a ttal f three cpies f data stred within the cluster. The cpies are distributed acrss data servers and racks fr fault recvery. Integrated 1GBaseT Adapter Mellanx CnnectX-3 10 GbE Adapter The Intel Xen prcessr E v3 is recmmended t prvide sufficient perfrmance. A minimum f 128 GB f memry is recmmended fr mst MapReduce wrklads with 256 GB r mre recmmended fr HBase, Spark, and memry-intensive MapReduce wrklads. Tw sets f disks are used: ne set f disks is fr the perating system and the ther set f disks is fr data. Fr the perating system disks, RAID 1 mirrring shuld be used. Each data nde in the reference architecture has an internal directly attached strage. External strage is nt used in this reference architecture. Available data space assumes the use f Hadp replicatin with three cpies f the data, and 25% capacity reserved fr efficient file system peratin and t allw time t increase capacity, if needed. In situatins where higher strage capacity is required, the main design apprach is t increase the amunt f data disk space per nde. Using 4 TB drives instead f 3 TB drives increases the ttal per nde data disk capacity frm 42 TB t 56 TB, a 33% increase. 13 Lenv Big Data Reference Architecture fr Cludera Distributin fr Hadp

17 When increasing data disk capacity, there is a balance between perfrmance and thrughput. Fr sme wrklads, increasing the amunt f user data that is stred per nde can decrease disk parallelism and negatively affect perfrmance. Increasing drives sizing als affects rebuilding and reppulating the replicas if there is a disk r nde failure. Higher density disks r ndes results in higher rebuild times. Drives that are larger than 4 TB are nt recmmended based n the balance f capacity and perfrmance. In this case, higher capacity can be achieved by increasing the number f ndes in the cluster. Fr higher IO thrughut, the data nde can be cnfigured with inch SAS drives, which have less strage capacity but much higher IO thrughut. Fr the HDD cntrller, just a bunch f disks (JBOD) is the best chice fr a Cludera cluster. It prvides excellent perfrmance and, when cmbined with the Hadp default f 3x data replicatin, als prvides significant prtectin against data lss. The use f RAID with data disks is discuraged because it reduces perfrmance and the amunt data that can be stred. Data ndes can be custmized accrding t client needs. A minimum f three data ndes are required as Hadp has three cpies f data by default. Three data ndes shuld be used fr test r POC envirnments nly. A minimum f five data ndes are required fr prductin envirnment if there are data nde failures Management ndes The Management Nde is the nucleus f the Hadp Distributed File System (HDFS) and supprts several ther key functins that are needed n a Cludera cluster. The Management Nde hsts the fllwing functins: YARN ResurceManager: Manages and arbitrates resurces amng all the applicatins in the system. Hadp NameNde: Cntrls the HDFS file system. The NameNde maintains the HDFS metadata, manages the directry tree f all files in the file system, and tracks the lcatin f the file data within the cluster. The NameNde des nt stre the data f these files. Zkeeper: Prvides a distributed cnfiguratin service, a synchrnizatin service, and a name registry fr distributed systems. Jurnal Nde: Cllects, maintains, and synchrnize updates frm NameNde. HA ResurceManager: Standby ResurceManager that can be used t prvide autmated failver. HA NameNde: Standby NameNde that can be used t prvide autmated failver. Other hadp cmpnent management services; fr example, HBase master, HiveServer2, and Spark Histry Server. Cludera Manager: Optinal management platfrm fr CDH. Table 3 lists the recmmended cmpnents fr a management nde. Management ndes can be custmized accrding t client needs. Table 3. Management nde cnfiguratin Cmpnent System Prcessr Management nde cnfiguratin System x3550 M5 2 x Intel Xen prcessr E v3 2.3 GHz 10-cre 14 Lenv Big Data Reference Architecture fr Cludera Distributin fr Hadp

18 Memry - base 128 GB 8 x 16 GB 2133 MHz RDIMM (minimum) Disk (OS / lcal strage) 300 GB drives: 10 x 300GB SAS 2.5 inch r 600 GB drives: 10 x 600GB SAS 2.5 inch HDD cntrller Hardware management netwrk adapter Data netwrk adapter ServeRAID M5210 SAS/SATA Cntrller Integrated 1GBaseT Adapter Mellanx CnnectX-3 10 GbE Adapter The Intel Xen prcessr E v3 is recmmended t prvide sufficient perfrmance. A minimum f 128 GB f memry is recmmended. A chice f 300 GB and 600GB drives is suggested fr the perating system and lcal strage. The management nde uses 10 HDDs fr the fllwing strage pls: Tw drives are cnfigured with RAID 1 fr perating system Tw drives are cnfigured with RAID 1 fr NameNde metastre Fur drives are cnfigured with RAID 10 fr database One drives is cnfigured with RAID 0 fr Zkeeper One drive cnfigure with RAID 0 fr Qurum Jurnal Nde stre This design separates the data stres fr different services and prvides best perfrmance. SSD and PCIe flash strage can be used t prvide imprved I/O perfrmance fr the database. Because the management nde is respnsible fr many memry-intensive tasks, multiple management ndes might be needed t split functins. Fr mst implementatins, the size f the Cludera cluster is a gd indicatr f hw many management ndes are needed.table 4 prvides a high-level guideline fr a cluster that prvides HA NameNde and ResurceManager failver when cnfigured with multiple management ndes. Fr a medium size cluster with data ndes, cnsider the use f a better cnfiguratin fr the management ndes, such as mre memry and CPU cre. Table 4. Number f management ndes Number f Data Ndes Number f Management Ndes Breakut f functin < Cludera Manager, Jurnal Nde, Zkeeper ResurceManager, HA Hadp NameNde, Jurnal Nde, Zkeeper HA ResurceManager, Hadp NameNde, Jurnal Nde, Zkeeper > Cludera Manager, Jurnal Nde, Zkeeper ResurceManager, HA Hadp NameNde, Jurnal 15 Lenv Big Data Reference Architecture fr Cludera Distributin fr Hadp

19 Nde, Zkeeper HA ResurceManager, Hadp NameNde, Jurnal Nde, Zkeeper Jurnal Nde, Zkeeper, ther rles Jurnal Nde, Zkeeper, ther rles Nte: If there are multiple racks, management ndes shuld be separated acrss racks. If yu plan t scale up the cluster significantly, it might be best t separate ut each f these functins frm the beginning, even if the starting cnfiguratin is smaller and requires fewer management ndes. The number f Management Ndes can be custmized based n specific needs. 6.3 Systems management Systems management includes cluster system management and hardware management. Cluster systems management uses the Cludera Manager and is adapted frm the standard Hadp distributin, which places the management services n separate servers than the data servers. Because the Management Nde hsts imprtant and high-memry functins, it is imprtant that it is a pwerful and fast server. The recmmended management ndes can be custmized accrding t client needs. Hardware management uses the Lenv XClarity Administratr, which is a centralized resurce management slutin that reduces cmplexity, speeds up respnse, and enhances the availability f Lenv server systems and slutins. The Lenv XClarity Administratr prvides agent-free hardware management fr Lenv s System x rack servers and Flex System cmpute ndes and cmpnents, including the Chassis Management Mdule (CMM) and Flex System I/O mdules. Figure 7 shws the Lenv XClarity administratr interface in which Flex System cmpnents and rack servers are managed and are seen n the dashbard. Lenv XClarity Administratr is a virtual appliance that is quickly imprted int a virtualized envirnment server cnfiguratin. 16 Lenv Big Data Reference Architecture fr Cludera Distributin fr Hadp

20 Figure 7: XClarity Administratr interface Als, xcat prvides a scalable distributed cmputing management and prvisining tl that prvides a unified interface fr hardware cntrl, discvery, and perating system deplyment. It can be used t facilitate r autmate the management f cluster ndes. Fr mre infrmatin abut xcat, see Acknwledgements This reference architecture dcument has benefited very much frm the detailed and careful review cmments prvided by Mike Perks f Lenv. Resurces n page Netwrking Regarding netwrking, the reference architecture specifies tw netwrks: a data netwrk and an administrative r management netwrk. Figure 8 shws the netwrking cnfiguratin fr Cludera. 17 Lenv Big Data Reference Architecture fr Cludera Distributin fr Hadp

21 Figure 8: Cludera netwrk cnfiguratin Data netwrk The data netwrk is a private cluster data intercnnect amng ndes that is used fr data access, mving data acrss ndes within a cluster, and imprting data int the Cludera cluster. The Cludera cluster typically cnnects t the custmer s crprate data netwrk. Tw tp f rack switches are required fr the data netwrk that is used by Cludera. Tw 1 GbE r 10 GbE switches can be used (a 1 Gb Ethernet switch is sufficient fr sme wrklads). The recmmended 1 GbE switch is the Lenv RackSwitch G8052. The 10 Gb Ethernet switch can prvide extra I/O bandwidth fr better perfrmance. The recmmended 10 GbE switch is the Lenv System Netwrking RackSwitch G8272. The tw Mellanx 10 GbE prts f each nde are link aggregated t the recmmended G8272 rack switch fr better perfrmance and imprved HA. The data netwrk is cnfigured t use a virtual lcal area netwrk (VLAN) Hardware management netwrk The hardware management netwrk is a 1 GbE netwrk that is used fr in-band perating system administratin and ut-f-band hardware management. In-band administrative services, such as SSH r Virtual Netwrk Cmputing (VNC) that is running n the hst perating system enables cluster ndes t be administered. Thrugh the integrated management mdules II (IMM2) within the System x3650 M5 server, ut-f-band management enables the hardware-level management f cluster ndes, such as nde deplyment r basic input/utput system (BIOS) cnfiguratin. Hadp has n dependency n the IMM2. Based n custmer requirements, the administratin links and management links can be segregated nt separate VLANs r subnets. The administrative r management 18 Lenv Big Data Reference Architecture fr Cludera Distributin fr Hadp

22 netwrk is typically cnnected directly t the custmer s administrative netwrk. When the in-band administrative services n the hst perating system are used, Cludera is cnfigured t use the data netwrk nly. The reference architecture requires ne 1 Gb Ethernet tp-f-rack switch fr the hardware management netwrk. Administratrs als can access all f the ndes in the cluster thrugh the custmer admin netwrk, as shwn in Figure 8 n page 18. This rack switch fr the hardware management netwrk is cnnected t each f the ndes in the cluster by using tw physical links (ne fr in-band perating system administratin and ne link fr ut-f-band IMM2 hardware management). On the ndes, the administratin link cnnects t prt 1 n the integrated 1 GBaseT adapter and the management link cnnects t the dedicated IMM2 prt Multi-rack netwrk The data netwrk in the predefined reference architecture cnfiguratin cnsists f a single netwrk tplgy. Apprpriate ther Lenv RackSwitch G8316 cre switches per cluster is required fr crss racks cmmunicatin. In this case, the secnd Mellanx 10 GbE prt can be cnnected t the secnd Lenv RackSwitch G8272. The ver-subscriptin rati fr G8272 is 1:2. Figure 9 n page 20shws hw the netwrk is cnfigured when the Cludera cluster is installed acrss mre than ne rack. The data netwrk is cnnected acrss racks by tw aggregated 40 GbE uplinks frm each rack s G8272 switch t a cre G8316 switch. 19 Lenv Big Data Reference Architecture fr Cludera Distributin fr Hadp

23 Figure 9: Cludera crss rack netwrk cnfiguratin 4 A 40GbE switch is recmmended fr intercnnecting the data netwrk acrss multiple racks. Lenv System Netwrking RackSwitch G8316 is the recmmended switch. A best practice is t have redundant cre switches fr each rack t avid a single pint f failure. Within each rack, the G8052 switch can ptinally be cnfigured t have tw uplinks t the G8272 switch t allw prpagatin f the administrative r management VLAN acrss cluster racks thrugh the G8316 cre switch. Fr large clusters, the Lenv System Netwrking RackSwitch G8332 is recmmended because it prvides a better cst value per 40 Gb prt than the G8316. Many ther crss rack netwrk cnfiguratins are pssible and might be required t meet the needs f specific deplyments r t address clusters larger than three racks. If the slutin is initially implemented as a multi-rack slutin, r if the system grws by adding racks, the ndes that prvide management services are distributed acrss racks t maximize fault tlerance. 6.5 Predefined cluster cnfiguratins The intent f the predefined cnfiguratins is t ease initial sizing fr custmers and t shw example starting pints fr fur different-sized wrklads. The starter rack cnfiguratin cnsists f three ndes and a pair f rack switches. The half rack cnfiguratin cnsists f nine ndes and a pair f rack switches. The full rack cnfiguratin (a rack fully ppulated) cnsists f 18 ndes and a pair f rack switches. The multi-rack cntains a ttal f 57 ndes; in each rack there are 19 ndes, a management nde and a pair f switches. Table 5 lists fur predefined cnfiguratins fr the Cludera reference architecture. The table als lists the amunt f space fr data and the number f ndes that each predefined cnfiguratin prvides. Strage space is described in tw ways: the ttal amunt f raw strage space when 3 TB r 4 TB drives (raw strage) are used and the amunt f space fr the data that the custmer has (available data space). Available data space assumes the use f Hadp replicatin with three cpies f the data and 25% capacity that is reserved fr intermediate data (scratch strage). The estimates that are listed in Table 5 d nt include extra space that is freed up by using cmpressin because cmpressin rates can vary widely based n file cntents. Table 5. Pre-defined cnfiguratins Starter rack Half rack Full rack Multi-rack Raw strage (3TB) 126 TB 378 TB 756 TB 2,394 TB Available data space (3TB) 31.5 TB 94.5 TB 189 TB TB Raw strage (4 TB) 168 TB 504 TB 1,008 TB 3,192 TB Available data space (4 TB) 42 TB 126 TB 252 TB 798 TB Number f Data Ndes Number f Management Ndes T simplify the diagram, nly ne G8272 is drawn in the diagram, and in recmmended cnfiguratin, tw G8272 are cnfigured fr HA. 20 Lenv Big Data Reference Architecture fr Cludera Distributin fr Hadp

24 Starter rack Half rack Full rack Multi-rack Number f Racks Number f 10 GbE cables Number f 1 GbE cables Figure 10 n page 21shws an verview f the architecture in three different ne-rack sized clusters withut netwrk redundancy: a starter rack, a half rack, and a full rack. Figure 11 n page 22shws a multi-rack-sized cluster withut netwrk redundancy. Figure 10: Starter rack, half rack, and full rack Cludera predefined cnfiguratins 21 Lenv Big Data Reference Architecture fr Cludera Distributin fr Hadp

25 Figure 11: Multi-rack Cludera predefined cnfiguratins 22 Lenv Big Data Reference Architecture fr Cludera Distributin fr Hadp

26 7 Deplyment cnsideratins This sectin describes ther cnsideratins fr deplying the Cludera slutin. 7.1 Increasing cluster perfrmance There are tw appraches that can be used t increase cluster perfrmance: increasing nde memry and the use f a high-perfrmance jb scheduler and MapReduce framewrk. Often, imprving perfrmance cmes at increased cst and yu must cnsider the cst-t-benefit trade-ffs f designing fr higher perfrmance. In the Cludera predefined cnfiguratin, nde memry can be increased t 256 GB by using GB RDIMMs. An even larger memry cnfiguratin can prvide greater perfrmance, depending n the wrklad. 7.2 Designing fr lwer cst There are tw key mdificatins that can be made t lwer the cst f a Cludera reference architecture slutin. When lwer-cst ptins are cnsidered, it is imprtant t ensure that custmers understand the ptential lwer perfrmance implicatins f a lwer-cst design. A lwer-cst versin f the Cludera reference architecture can be achieved by using lwer-cst nde prcessrs and lwer-cst cluster data netwrk infrastructure. The nde prcessrs can be substituted with the Intel Xen E v3 2.4 GHz 8-cre prcessr. This prcessr requires 1866 MHz RDIMMs, which can als lwer the per-nde cst f the slutin. The use f a lwer-cst netwrk infrastructure can significantly lwer the cst f the slutin, but can als have a substantial negative effect n intra-cluster data thrughput and cluster ingest rates. T use a lwer-cst netwrk infrastructure, use the fllwing substitutins t the predefined cnfiguratin: Within each nde, substitute the Mellanx 10GbE dual SFP+ netwrk adapter with the extra prts n the integrated 1GBaseT adapters within the System x3650 M5 server. Within each rack, substitute the Lenv RackSwitch G8272 tp-f-rack switch with the Lenv RackSwitch G8052. Within each cluster, substitute the Lenv RackSwitch G8316 cre switch with the Lenv RackSwitch G8272. Althugh the netwrk wiring schema is the same as the schema that is described in Netwrking, the media types and link speeds within the data netwrk changed. The data netwrk within a rack that cnnects the cluster ndes t the lwer-cst ptin G8052 tp-f rack-switch is nw based n tw aggregated 1GBaseT links per nde. The physical intercnnect between the admin/management netwrks and the data netwrks within each rack is nw based n tw aggregated 1GBaseT links between the administrative r management netwrk G8052 switch and the lwer cst data netwrk G8052 switch. Within a cluster, the racks are intercnnected thrugh tw aggregated 10 GbE links between the substitute G8052 data netwrk switch in each rack and a lwer cst G8272 cre switch. 7.3 Designing fr high ingest rates Designing fr high ingest rates is difficult. It is imprtant t have a full characterizatin f the ingest patterns and vlumes. The fllwing questins prvide guidance t key factrs that affect the rates: On what days and at what times are the surce systems available r nt available fr ingest? 23 Lenv Big Data Reference Architecture fr Cludera Distributin fr Hadp

27 When a surce system is available fr ingest, what is the duratin fr which the system remains available? D ther factrs affect the day, time, and duratin ingest cnstraints? When ingests ccur, what is the average and maximum size f ingest that must be cmpleted? What factrs affect ingest size? What is the frmat f the surce data (structured, semi-structured, r unstructured)? Are there any data transfrmatin r cleansing requirements that must be achieved during ingest? T increase the data ingest rates, cnsider the fllwing pints: Ingest data with MapReduce jb, which helps t distribute the I/O lad t different ndes crss the cluster. Ingest during cluster lad is nt high, if pssible. Cmpressing data is a gd ptin in many cases, which reduces the I/O lad t disk and netwrk. Filter and reduce data in earlier stage saves mre csts. 7.4 Estimating disk space When yu are estimating disk space within a Cludera Hadp cluster, cnsider the fllwing pints: Fr imprved fault tlerance and perfrmance, the Cludera Hadp file system replicates data blcks acrss multiple cluster data ndes. By default, the file system maintains three replicas. Cmpressin rati is an imprtant cnsideratin in estimating disk space and can vary greatly based n file cntents. If the custmer s data cmpressin rati is unavailable, assume a cmpressin rati f 2.5:1. T ensure efficient file system peratin and t allw time t add mre strage capacity t the cluster if necessary, reserve 25% f the ttal capacity f the cluster. Assuming the default three replicas maintained by the Cludera Hadp file system, the raw data disk space, and the required number f ndes can be estimated by using the fllwing equatins: Ttal raw data disk space = (User data, uncmpressed) * (4 / cmpressin rati) Ttal required data ndes = (Ttal raw data disk space) / (Raw data disk per nde) Yu shuld als cnsider future grwth requirements when estimating disk space. Based n these sizing principals, Table 6 n page 25shws an example fr a cluster that must stre 500 TB f uncmpressed user data. The example shws that the Cludera cluster needs 800 TB f raw disk t supprt 500 TB f uncmpressed data. The 800 TB is fr data strage and des nt include perating system disk space. A ttal f 15 ndes are required t supprt a deplyment f this size. 24 Lenv Big Data Reference Architecture fr Cludera Distributin fr Hadp

28 Table 6. Example f strage sizing with 4TB drives Descriptin Size f uncmpressed user data Value 500 TB Cmpressin rati 2.5:1 Size f cmpressed data 200 TB Strage multiplicatin factr 4 Raw data disk space needed fr Cludera cluster Strage needed fr Cludera Hadp 3x replicatin Reserved strage fr headrm Raw data disk per nde (with 4 TB drives) 800 TB 600 TB 200 TB 56 TB Minimum number f ndes required (800/56) Scaling cnsideratins The Hadp architecture is linearly scalable but it is imprtant t nte that sme wrklads might nt scale cmpletely linearly. When the capacity f the infrastructure is reached, the cluster can be scaled ut by adding ndes. Typically, identically cnfigured ndes are best t maintain the same rati f strage and cmpute capabilities. A Cludera cluster is scalable by adding System x3650 M5 data ndes, management ndes, and netwrk switches. As the capacity f racks is reached, new racks can be added t the cluster. When a Cludera reference architecture implementatin is designed, future scale ut shuld be a key cnsideratin in the initial design. There are tw key aspects t cnsider: netwrking and management. These aspects are critical t cluster peratin and becme mre cmplex as the cluster infrastructure grws. The crss rack netwrking cnfiguratin that is shwn in Figure 9 n page 20 prvides rbust netwrk intercnnectin f racks within the cluster. As racks are added, the predefined netwrking tplgy remains balanced and symmetrical. If there are plans t scale the cluster beynd ne rack, a best practice is t initially design the cluster with multiple racks (even if the initial number f ndes fit within ne rack). Starting with multiple racks can enfrce prper netwrk tplgy and prevent future re-cnfiguratin and hardware changes. As racks are added ver time, multiple G8316 switches might be required fr greater scalability and balanced perfrmance. Als, as the number f ndes within the cluster increases, s d many f the tasks f managing the cluster, such as updating nde firmware r perating systems. Building a cluster management framewrk as part f the initial design and practively cnsidering the challenges f managing a large cluster pays ff significantly in the lng run. 25 Lenv Big Data Reference Architecture fr Cludera Distributin fr Hadp

29 Practive planning fr future scale ut and the develpment f cluster management framewrk as a part f initial cluster design prvides a fundatin fr future grwth that can minimize hardware recnfiguratins and cluster management issues as the cluster grws. 7.6 High availability cnsideratins When a Cludera cluster n System x is implemented, cnsider availability requirements as part f the final hardware and sftware cnfiguratin. Typically, Hadp is cnsidered a highly reliable slutin. Hadp and Cludera best practices prvide significant prtectin against data lss. Generally, failures can be managed withut causing an utage. There is redundancy that can be added t make a cluster even mre reliable. Sme cnsideratin must be given t hardware and sftware redundancy Netwrking cnsideratins Optinally, a secnd redundant switch can be added t ensure HA f the hardware management netwrk. The hardware management netwrk des nt affect the availability f the Cludera Hadp file system functinality, but it might affect the management f the cluster; therefre, availability requirements must be cnsidered. T supprt HA in the netwrk, link aggregatin is used between the 10 Gb prts f a server netwrk adapter and the tp-f-rack switch. Virtual Aggregatin Grups (vlag) can be used between redundant switches. If 1 Gbps data netwrk links are used, it is recmmended that mre than ne link is used per nde t increase thrughput Hardware availability cnsideratins The redundancy f each individual data nde is nt necessary with Cludera. Hadp 3x replicatin prvides built-in redundancy and makes lss f data unlikely. If Hadp best practices are used, an utage frm a data nde lss is extremely unlikely as the wrklad can be dynamically re-allcated. The lss f a data nde cannt stp wrklad prcessing; wrklad is autmatically re-allcated t anther data nte. Multiple management ndes are recmmended s that if there is a failure, functin can be mved t an peratinal management nde. Having multiple management ndes des nt reslve the issue f the NameNde being a single pint f failure. Fr mre infrmatin, see Sftware availability cnsideratins. Within racks, switches and ndes must have redundant pwer feeds with each pwer feed cnnected frm a separate PDU Strage availability If default 3x replicatin is nt sufficient fr availability requirements, RAID is anther ptin fr insuring against data lss. HDFS disks can be set up with RAID, but perfrmance is negatively affected. In mst cases, HDFS 3x replicatin prvides mre than sufficient prtectin. Cludera als prvides manual r scheduled snapshts f vlumes t prtect against human errr and prgramming defects. Snapshts are useful fr rllback t a knwn data set. 26 Lenv Big Data Reference Architecture fr Cludera Distributin fr Hadp