PERFORMANCE MODELS FOR APACHE ACCUMULO:

Transcription

1 Securely explore your data PERFORMANCE MODELS FOR APACHE ACCUMULO: THE HEAVY TAIL OF A SHAREDNOTHING ARCHITECTURE Chris McCubbin Director of Data Science Sqrrl Data, Inc.

2 I M NOT ADAM FUCHS But perhaps I m still an interesting guy MS in CS from UMBC in Network Security and Quantum Computing 8 years at JHU/APL working on UxV Swarms 4 years at JHU/APL and TexelTek creating Big Data Applications for the NSA Co-founder and Director of Data Science at Sqrrl 2

3 SO, YOUR DISTRIBUTED APPLICATION IS SLOW Today s distributed applications run on tens or hundreds of library components Many versions so internet advice could be ineffective, or worse, flat out wrong Hundreds of settings Some, shall we say, could be better documented Shared-nothing architectures are usually sharedlittle architectures with tricky interactions Profiling is hard and time-consuming What do we do? 3

4 TODAY S TALK 1. Quick intro to performance optimization 2. Tricks and techniques for targeted distributed application modeling performance improvement 3. A deep dive into improving bulk load application performance 4

5 The Apache Accumulo sorted, distributed key/value store is a secure, robust, scalable, high performance data storage and retrieval system. Many applications in real-time storage and analysis of big data : Spatio-temporal indexing in non-relational distributed databases - Fox et al 2013 IEEE International Congress on Big Data Big Data Dimensional Analysis - Gadepally et al IEEE HPEC 2014 Leading its peers in performance and scalability: Achieving 100,000,000 database inserts per second using Accumulo and D4M - Kepner et al IEEE HPEC 2014 An NSA Big Graph experiment (Technical Report NSA-RD v1) Benchmarking Apache Accumulo BigData Distributed Table Store Using Its Continuous Test Suite - Sen et al 2013 IEEE International Congress on Big Data For more papers and presentations, see 5

6 SCALING UP: DIVIDE & CONQUER Well- Known Loca9on (zookeeper) Metadata Tablet 1 - to Encyclopedia:Ocelot Root Tablet - to Metadata Tablet 2 Encyclopedia:Ocelot to Collections of KV pairs form Tables Tables are partitioned into Tablets Metadata tablets hold info about other tablets, forming a 3-level hierarchy A Tablet is a unit of work for a Tablet Server Table: Adam s Table Table: Encyclopedia Table: Foo Data Tablet - : thing Data Tablet thing : Data Tablet - : Ocelot Data Tablet Ocelot : Yak Data Tablet Yak : Data Tablet - to 6

7 PERFORMANCE ANALYSIS CYCLE Start: Create Model Simulate & Experiment Outputs: Better Code + Models Refine Model Analyze Modify Code 7

8 MAKING A MODEL Determine points of low-impact metrics Add some if needed Create parallel state machine models with components driven by these metrics Estimate running times and bottlenecks from a-priori information and/or apply measured statistics Focus testing on validation of the initial model and the (estimated) pain points Apply Amdahl s Law Rinse, repeat 8

9 BULK INGEST OVERVIEW Accumulo supports two mechanisms to bring data in: streaming ingest and bulk ingest. Bulk Ingest Goal: maximize throughput without constraining latency. create a set of Accumulo Rfiles, then register those files with Accumulo. RFiles are groups of sorted key-value pairs with some indexing information MapReduce has a built-in key sorting phase: a good fit to produce RFiles 9

10 BULK INGEST MODEL Map Reduce Register Time 10

11 BULK INGEST MODEL Hypothetical Resource Usage Map Reduce Register 100% CPU 20% Disk 0% Network 46 seconds 40% CPU 100% Disk 20% Network 168 seconds 10% CPU 20% Disk 40% Network 17 seconds Time 11

12 INSIGHT Spare disk here, spare CPU there can we even out resource consumption? Why did reduce take 168 seconds? It should be more like 40 seconds. No clear bottleneck during registration is there a synchronization or serialization problem? Map Reduce Register 100% CPU 20% Disk 0% Network 46 seconds 40% CPU 100% Disk 20% Network 168 seconds 10% CPU 20% Disk 40% Network 17 seconds Time 12

13 LOOKING DEEPER: REFINED BULK INGEST MODEL Map Thread Map Setup Map Sort Spill Merge Serve Parallel Latch Reduce Thread Shuffle Sort Reduce Output Time 13

14 BULK INGEST MODEL PREDICTIONS We can constrain parts of the model by physical throughput limitations Disk -> memory (100Mbps avg 7200rpm seq. read rate) Input reader Memory -> Disk (100Mbps) Spill, OutputWriter Disk -> Disk (50Mbps) Merge Network (Gigabit = 125Mbps) Shuffle And/or algorithmic limitations Sort, (Our) Map, (Our) Reduce, SerDe 14

15 PERFORMANCE GOAL MODEL Performance goals obtained through: Simulation of individual components Prediction of available resources at runtime 15

16 INSTRUMENTATION application version SYSTEM DATA application sha 8d17baf8 node num 1 input type arcsight yarn.nodemanager.resource.memory-mb map num containers 20 input block size 32 yarn.scheduler.minimum-allocation-mb 2048 red num containers 20 input block count 20 yarn.scheduler.maximum-allocation-mb cores physical 12 input total yarn.app.mapreduce.am.resource.mb 2048 cores logical 24 output map yarn.app.mapreduce.am.command-opts -Xmx1536m disk num 8 output map:combine input records mapreduce.map.memory.mb 2048 disk bandwidth 100 output map:combine records out mapreduce.map.java.opts -Xmx1638m replication 1 output map:spill mapreduce.reduce.memory.mb 2048 monitoring TRUE output final mapreduce.reduce.java.opts -Xmx1638m output map:combine mapreduce.task.io.sort.mb 100 TIME mapreduce.map.sort.spill.percent 0.8 map:setup avg 8 RATIOS mapreduce.task.io.sort.factor 10 map:map avg 12 input explosion factor mapreduce.reduce.shuffle.parallelcopies 5 map:sort avg 12 compression intermediate mapreduce.job.reduce.slowstart.completedmaps 1 map:spill avg 12 load combiner output mapreduce.map.output.compress FALSE map:spill count 7 total ratio mapred.map.output.compression.codec n/a map:merge avg 46 description baseline map total 290 CONSTANTS red:shuffle avg 6 avg schema entry size (bytes) 59 red:merge avg 38 red:reduce avg 68 effective MB/sec red:total avg 112 red:reducer count 20 job:total

17 PERFORMANCE MEASUREMENT Baseline (naive implementation) Map Thread Map Setup Map Sort Spill Merge Serve Reduce Thread Shuffle Sort Reduce Output 17

18 PATH TO IMPROVEMENT 1. Profiling revealed much time spent serializing/ deserializing Key 2. With proper configuration, MapReduce supports comparison of keys in serialized form 3. Rewriting Key s serialization lead to an order-preserving encoding, easy to compare in serialized form 4. Configure MapReduce to use native code to compare Keys 5. Tweak map input size and spill memory for as few spills as possible 18

19 PERFORMANCE MEASUREMENT Optimized sorting Improvements: Time for map-side merge went down Sort performance drastically improved in both map and reduce phases 300% faster 19

20 PERFORMANCE MEASUREMENT Optimized sorting Map Thread Map Setup Map Sort Spill Merge Serve Reduce Thread Shuffle Sort Reduce Output Insights: Map is slower than expected Output is disk bound maybe we can move more processing to Reduce Reverse Amdahl s law Intermediate data inflation ratio (output/input for map) is very high 20

21 PATH TO IMPROVEMENT 1. Profiling revealed much time spent copying data 2. Evaluation of data passed from map to reduce revealed inefficiencies: Constant timestamp cost 8 bytes per key Repeated column names could be encoded/ compressed Some Key/Value pairs didn t need to be created until reduce 21

22 PERFORMANCE MEASUREMENT Optimized map code Improvement: Big speedup in map function Twice as fast Reduced intermediate inflation sped up all steps between map and reduce 22

23 DO TRY THIS AT HOME Hints for Accumulo Application Optimization With these steps, we achieved 6X speedup: Perform comparisons on serialized objects With Map/Reduce, calculate how many merge steps are needed Avoid premature data inflation Leverage compression to shift bottlenecks Always consider how fast your code should run 23

24 SOME CURRENT ACCUMULO PERFORMANCE PROJECTS Optimize metadata operations Batch to improve throughput (ACCUMULO-2175, ACCUMULO-2889) Remove from critical path where possible Optimize write-ahead log performance Maximize throughput Reduce flushes Parallelize WALs (ACCUMULO-1083) Avoid downtime by pre-allocating 24

25 Securely explore your data SQRRL IS HIRING! QUESTIONS? Chris McCubbin Director of Data Science Sqrrl Data, Inc.