Hadoop Size does Hadoop Summit 2013

Transcription

1 Hadoop Hardware : Size does Hadoop Summit 2013 v2.3

2 About us Joep Rottinghuis Software Twitter Engineering Manager Hadoop/HBase Twitter Follow Jay Shenoy Hardware Twitter Engineering Manager Twitter Follow HW & Hadoop Twitter, Many others 2

3 Agenda Scale of Hadoop Clusters Single versus multiple clusters Twitter Hadoop Architecture Hardware investigations Results 3

4 Scale Scaling limits JobTracker 10 s thousands of jobs per day; 10 s Ks concurrent slots Namenode M objects in single namespace GB heap -> full GC pauses Shipping job jars to 1,000 s of nodes JobHistory server at a few 100 s K job history/conf files # Nodes 4

5 When / why to split clusters? In principle preference for single cluster Common logs, shared free space, reduced admin burden, more rack diversity Varying SLA s Workload diversity Storage intensive Processing (CPU / Disk IO) intensive Network intensive Data access Hot, Warm, Cold 5

6 Cluster Architecture 6

7 Hardware investigations 7

8 Service criteria for hardware Hadoop does not need live HDD swap Twitter DC : No SLA on data nodes Rack SLA : Only 1 rack down at any time in a cluster 8

9 Baseline Hadoop Server (~ early 2012) E56xx PCH NIC GbE Characteristics: HBA Standard 2U server 20 servers / rack E56xx Expander E5645 CPU Works for the general cluster, but... Need more density for storage Potential IO bottlenecks Dual 6-core 72GB memory 12 x 2TB HDD 2 x 1 GbE 9

10 Hadoop Server: Possible evolution E5-26xx or E5-24xx NIC GbE 10GbE? Characteristics: + CPU performance E5-26xx or E5-24xx HBA Expander 16 x 2T? 16 x 3T? 24 x 3T?? 20 servers / rack Candidate for DW Can deploy into the general DW cluster, but... Too much CPU for storage intensive apps Server failure domain too large if we scale up disks 10

11 Rethinking hardware evolution Debunking myths Bigger is always better One size fits all Back to Hadoop Hardware Roots: Scale horizontally, not vertically Twitter Hadoop Server - THS 11

12 THS for backups E3-12xx NIC GbE Characteristics: + IO Performance Few fast cores SAS HBA PCH E V2 CPU 16 GB memory 12 x 3 TB HDD Storage focus: Cost efficient (single socket, 3T drives) SSD boot 2 x 1 GbE Less memory needed 12

13 THS variant for Hadoop-Proc and HBase E3-12xx NIC 10GbE Characteristics: + IO Performance Few fast cores SAS HBA PCH E V2 CPU 32 GB memory 12 x 1 TB HDD Processing / throughput focus: Cost efficient (single socket, 1T drives) SSD boot 1 x 10 GbE More disk and network IO per socket 13

14 THS for cold cluster E3-12xx NIC GbE Characteristics: Disk Efficiency Some compute SAS HBA PCH E V2 CPU 32 GB memory 12 x 3 TB HDD Combination of previous 2 use cases: Space & power efficient 2 x 1 GbE Storage dense and some processing capabilities 14

15 Rack-level view 1G TOR 1G TOR 1G TOR 10G TOR 1G TOR 1G TOR Baseline Twitter Hadoop Server Backups Proc Cold Power ~ 8 kw ~ 8 kw ~ 8 kw ~ 8 kw CPU sockets; DRAM 40; 1440 GB 40; 640 GB 40; 1280 GB 40; 1280 GB Spindles; TB raw 240; 480 TB 480; 1,440 TB 480; 480 TB 480; 1,440 TB Uplink; Internal BW 20 ; 40 Gbps 20 ; 80 Gbps 40 ; 400 Gbps 20 ; 80 Gbps 15

16 Processing performance comparison Benchmark Baseline Server THS (-Cold) TestDFSIO (write replication = 1) 360 MB/s / node 780 MB/s / node TeraGen (30TB replication = 3) 1:36 hrs 1:35 hrs TeraSort (30 TB, replication = 3) 6:11 hrs 4:22 hrs 2 Parallel TeraSort (30 TB each, replication = 3) 10:36 hrs 6:21 hrs Application #1 4:37 min 3:09 min Application set #2 13:3 hrs 10:57 hrs Performance benchmark set up: Each clusters 102 nodes of respective type Efficient server = 3 racks, Baseline 5+ racks Dated stack: CentOS 5.5, Sun 1.6 JRE, Hadoop

17 Results 17

18 LZO performance comparison 16 18

19 Recap At a certain scale it makes sense to split into multiple clusters For us: RT, PROC, DW, COLD, BACKUPS, TST, EXP For large enough clusters, depending on use-case, it may be worth to choose different HW configurations 19

20 Conclusion our Twitter Hadoop Server not only saves many $$$, it is also faster! 20

21 Come talk to us at booth 26