Pr oven Tools t o Simplif y Hadoop Envir onment s. [ DELL Kor ea Next Gener at ion Comput e Solut ions 최주열부장 ]

Size: px

Start display at page:

Download "Pr oven Tools t o Simplif y Hadoop Envir onment s. [ DELL Kor ea Next Gener at ion Comput e Solut ions 최주열부장 ]"

Mervyn Hoover
7 years ago
Views:

1 Pr oven Tools t o Simplif y Hadoop Envir onment s [ DELL Kor ea Next Gener at ion Comput e Solut ions 최주열부장 ]

2 Hadoop Oper at ional Models Tradit ional Dat acent er Assigned Ser ver s Rigid Policies Tiered Sof t ware Cloud Inf rast ruct ure Elast ic Resour ces Ser vices (APIs) Dist ribut ed Sof t ware

3 Oper at ional Challenges Deployment Complex because of scale (60 nodes t o 1000 nodes) Cumber some because of high-t ouch pr ocesses Conf igurat ion & Tuning Error-prone conf iguration management St at e management Monit or ing and Management Complex t r oubleshoot ing and diagnost ics No proactive notif ication of problems Perf ormance Opt imizat ion Limitations of traditional tools 3

management St at e management Monit or ing and Management Complex t r oubleshoot ing and

4 CloudOps Fr amewor k

5 Thr ee aspect s of r evolut ionar y clouds Two Sides of Cloud Ecosystem +API Cloud = Operat ions Ops Black Box HW SW OPS CloudOps APIs Ops Cloud O/S Physical 5

6 Images vs. Layer s: Over view Images: Single Unit Layer s: St acked Pieces Configuration Integrations + Applicati ons + Utilities + Operating Sy stem Configuratio on Integrations Application Foo Application Bar Utilities Operating System

7 Images vs. Layer s: Lif ecycle Images: Replacement Layers: Upgrade Config I+A+U+O /S Config I+A+U+O /S Config I+A+U+O /S Config Foo I Bar v1 U OS Config Foo I Bar v2 U OS Config I+A+U+O /S Bar v2

8 Modular Design: Bar clamps APIs, User Access, & Ecosyst em Part ners Nagios Ganglia Dashboar d Dell Cro owbar Ops Mana agement Cloud Inf rast ruct ure & Dell IP Ext ensions Core Component s & Oper at ing Syst ems Hadoop Cr owbar Deployer DNS NTP Logging Pr ovisioner BIOS IPMI Physical Resour ces Net wor k RAID

Dell IP Ext ensions Core Component s & Oper at ing Syst ems Hadoop Cr owbar

9 Cr owbar = Inst all St at e Machine Traditional Dat acenter

10 Cloud = Ops We have capable hardware & sof t ware, t he real quest ion is how are we going t o operat e it as a service? HW SW OPS Cloud Ops This is CloudOps Sof tware mindset to inf rastructure Sof tware is constantly changing Fluid resources instead of servers Manual t ouch is unaccept able

HW SW OPS Cloud Ops This is CloudOps Sof tware mindset to inf rastructure

11 Second Act

12 Plat f or m Select ion Dell PowerEdge C2100 Designed wit h big dat a in mind Compact 2U f orm f act or 2-socket 6-core Int el Xeon 5620 processor High perf ormance memory syst em Expansive disk st orage Recommended Conf igur at ion Int el Xeon Processor 5600 series 4-6 1TB or 2TB 7200 RPM SATA SSD 12-24GB DDR3 R-ECC RAM 1-2 dual-por t 1GigE Linux kernel or lat er Sun Java 6u14 or lat er Hadoop version 0.20.x or lat er Int el Whit epaper: Opt imizing Hadoop Deployment s (ht t p://sof t ware.int el.com/f ile/31124)

series 4-6 1TB or 2TB 7200 RPM SATA SSD 12-24GB DDR3 R-ECC RAM 1-2 dual-por t 1GigE Linux kernel 2.6.30 or lat er Sun Java 6u14 or lat er Hadoop version 0.

13 So what seems t o be t he pr oblem? Dat af low and high level abst ract ion make it dif f icult t o underst and runt ime behaviors Large dist ribut ed syst em makes it dif f icult t o correlat e concurrent perf ormancerelat ed act ivit ies

t o underst and runt ime behaviors Large dist ribut ed syst

14 HiTune: Hadoop Per f or mance Analyzer Collects met rics f rom each node Aggregat es dat a using Chukwa Analyzes r esult s using Hadoop Generates report s f or visualizat ion Syst em met ric (CPU, Disk I/O, Net work IO, Memory) Syst em met ric (CPU, Disk I/O, Net work IO, Memory) Hadoop met rics (NameNode, Dat anode, JobTr acker, TaskTr acker, JV M met r ics) Dat af low based st at ist ics (Job, MapTasks, Reduce Tasks, Thr eaddump f or M/R) Summary view of a single j ob Summary view by comparing mult iple j obs

I/O, Net work IO, Memory) Hadoop met rics (NameNode, Dat anode, JobTr acker, TaskTr acker, JV M met r ics) Dat af low based st

15 HiTune Ar chit ect ur e Tr acker Light weight agent running on each node in t he Hadoop clust er Sysst at, Hadoop logs and met rics, Java inst rument at ion Aggr egat ion engine Merges t he result s of all t he t rackers in a dist ribut ed f ashion Analysis engine Generat es report s based on dat a f low model

ion Aggr egat ion engine Merges t he result s of all t he t rackers in a dist

16 Case St udy Partitioned Input D A T A Map Tasks map spill map spill map Spill map spill shuffle copier merge sort reduce copier copier shuffle shuffle merge merge Reduce Tasks sort reduce sort reduce Aggregated Output Terasort wit h zlib Streaming dataflow Sequential dataflow Large gap bet ween end of map and end of shuf f le No CPU, I/O, or net work bandwidt h bot t lenecks Adding copiers does not change shuf f le f et chers busy percent = 100 Terasort wit h LZO Copier t hreads idle 80% wait ing f or memory merge t hreads Memory merge t hreads busy most ly due t o compression Changing compression codec t o LZO closes t he gap Improves j ob running t ime by 2.3x

CPU, I/O, or net work bandwidt h bot t lenecks Adding copiers does not change shuf f le f et chers busy percent = 100 Terasort wit h LZO Copier t hreads idle 80% wait

17 Have at it Pull Cr owbar ht t ps://git hub.com/dellcloudedge/cr owbar Pull HiTune ht t ps://git hub.com/hitune/hitune

Dell Reference Configuration for Hortonworks Data Platform

Dell Reference Configuration for Hortonworks Data Platform A Quick Reference Configuration Guide Armando Acosta Hadoop Product Manager Dell Revolutionary Cloud and Big Data Group Kris Applegate Solution