Hadoop Hardware : Size does matter. @joep and @eecraft Hadoop Summit 2013 v2.3
About us Joep Rottinghuis Software Engineer @ Twitter Engineering Manager Hadoop/HBase team @ Twitter Follow me @joep Jay Shenoy Hardware Engineer @ Twitter Engineering Manager HW @ Twitter Follow me @eecraft HW & Hadoop teams @ Twitter, Many others 2
Agenda Scale of Hadoop Clusters Single versus multiple clusters Twitter Hadoop Architecture Hardware investigations Results 3
Scale Scaling limits JobTracker 10 s thousands of jobs per day; 10 s Ks concurrent slots Namenode 250-300 M objects in single namespace Namenode @~100 GB heap -> full GC pauses Shipping job jars to 1,000 s of nodes JobHistory server at a few 100 s K job history/conf files # Nodes 4
When / why to split clusters? In principle preference for single cluster Common logs, shared free space, reduced admin burden, more rack diversity Varying SLA s Workload diversity Storage intensive Processing (CPU / Disk IO) intensive Network intensive Data access Hot, Warm, Cold 5
Cluster Architecture 6
Hardware investigations 7
Service criteria for hardware Hadoop does not need live HDD swap Twitter DC : No SLA on data nodes Rack SLA : Only 1 rack down at any time in a cluster 8
Baseline Hadoop Server (~ early 2012) E56xx PCH NIC GbE Characteristics: HBA Standard 2U server 20 servers / rack E56xx Expander E5645 CPU Works for the general cluster, but... Need more density for storage Potential IO bottlenecks Dual 6-core 72GB memory 12 x 2TB HDD 2 x 1 GbE 9
Hadoop Server: Possible evolution E5-26xx or E5-24xx NIC GbE 10GbE? Characteristics: + CPU performance E5-26xx or E5-24xx HBA Expander 16 x 2T? 16 x 3T? 24 x 3T?? 20 servers / rack Candidate for DW Can deploy into the general DW cluster, but... Too much CPU for storage intensive apps Server failure domain too large if we scale up disks 10
Rethinking hardware evolution Debunking myths Bigger is always better One size fits all Back to Hadoop Hardware Roots: Scale horizontally, not vertically Twitter Hadoop Server - THS 11
THS for backups E3-12xx NIC GbE Characteristics: + IO Performance Few fast cores SAS HBA PCH E3-1230 V2 CPU 16 GB memory 12 x 3 TB HDD Storage focus: Cost efficient (single socket, 3T drives) SSD boot 2 x 1 GbE Less memory needed 12
THS variant for Hadoop-Proc and HBase E3-12xx NIC 10GbE Characteristics: + IO Performance Few fast cores SAS HBA PCH E3-1230 V2 CPU 32 GB memory 12 x 1 TB HDD Processing / throughput focus: Cost efficient (single socket, 1T drives) SSD boot 1 x 10 GbE More disk and network IO per socket 13
THS for cold cluster E3-12xx NIC GbE Characteristics: Disk Efficiency Some compute SAS HBA PCH E3-1230 V2 CPU 32 GB memory 12 x 3 TB HDD Combination of previous 2 use cases: Space & power efficient 2 x 1 GbE Storage dense and some processing capabilities 14
Rack-level view 1G TOR 1G TOR 1G TOR 10G TOR 1G TOR 1G TOR Baseline Twitter Hadoop Server Backups Proc Cold Power ~ 8 kw ~ 8 kw ~ 8 kw ~ 8 kw CPU sockets; DRAM 40; 1440 GB 40; 640 GB 40; 1280 GB 40; 1280 GB Spindles; TB raw 240; 480 TB 480; 1,440 TB 480; 480 TB 480; 1,440 TB Uplink; Internal BW 20 ; 40 Gbps 20 ; 80 Gbps 40 ; 400 Gbps 20 ; 80 Gbps 15
Processing performance comparison Benchmark Baseline Server THS (-Cold) TestDFSIO (write replication = 1) 360 MB/s / node 780 MB/s / node TeraGen (30TB replication = 3) 1:36 hrs 1:35 hrs TeraSort (30 TB, replication = 3) 6:11 hrs 4:22 hrs 2 Parallel TeraSort (30 TB each, replication = 3) 10:36 hrs 6:21 hrs Application #1 4:37 min 3:09 min Application set #2 13:3 hrs 10:57 hrs Performance benchmark set up: Each clusters 102 nodes of respective type Efficient server = 3 racks, Baseline 5+ racks Dated stack: CentOS 5.5, Sun 1.6 JRE, Hadoop 2.0.3 16
Results 17
LZO performance comparison 16 18
Recap At a certain scale it makes sense to split into multiple clusters For us: RT, PROC, DW, COLD, BACKUPS, TST, EXP For large enough clusters, depending on use-case, it may be worth to choose different HW configurations 19
Conclusion our Twitter Hadoop Server not only saves many $$$, it is also faster! 20
#ThankYou @joep and @eecraft Come talk to us at booth 26