Can t We All Just Get Along? Spark and Resource Management on Hadoop

Transcription

1 Can t We All Just Get Along? Spark and Resource Management on Hadoop

2 Introduc=ons So>ware engineer at Cloudera MapReduce, YARN, Resource management Hadoop commider

3 Introduc=on Spark as a first class data processing framework alongside MR and Impala Resource management What we have already What we need for the future

4 Bringing Computa=on to the Data Users want to ETL a dataset with Pig and MapReduce Fit a model to it with Spark Have BI tools query it with Impala Same set of machines that hold data must also host these frameworks

5 Cluster Resource Management Hadoop brings generalized computa=on to big data More processing frameworks MapReduce, Impala, Spark Some workloads are more important than others A cluster has finite resources Limited CPU, memory, disk and network bandwidth How do we make sure each workload gets the resources it deserves?

6 How We See It Impala MapReduce Spark HDFS

7 How They Want to See It Engineering - 50% Finance - 30% Marketing - 20% Spark MR Spark Spark MR MR Impala Impala Impala HDFS

8 Central Resource Management Impala MapReduce Spark YARN HDFS

9 YARN Resource manager and scheduler for Hadoop Container is a process scheduled on the cluster with a resource alloca=on (amount MB, # cores) Each container belongs to an Applica=on

10 YARN Applica=on Masters Each YARN app has an Applica=on Master (AM) process running on the cluster AM responsible for reques=ng containers from YARN AM crea=on latency is much higher than resource acquisi=on

11 YARN JobHistory Server ResourceManager Client NodeManager NodeManager Container Map Task Container Application Master Container Reduce Task

12 YARN Queues Cluster resources allocated to queues Each applica=on belongs to a queue Queues may contain subqueues Root Mem Capacity: 12 GB CPU Capacity: 24 cores Marketing Fair Share Mem: 4 GB Fair Share CPU: 8 cores R&D Fair Share Mem: 4 GB Fair Share CPU: 8 cores Sales Fair Share Mem: 4 GB Fair Share CPU: 8 cores Jim s Team Fair Share Mem: 2 GB Fair Share CPU: 4 cores Bob s Team Fair Share Mem: 2 GB Fair Share CPU: 4 cores

13 YARN app models Applica=on master (AM) per job Most simple for batch Used by MapReduce Applica=on master per session Runs mul=ple jobs on behalf of the same user Recently added in Tez AM as permanent service Always on, waits around for jobs to come in Used for Impala

14 Spark Usage Modes Mode Long Lived/Multiple Jobs Multiple Users Batch No No Interactive Yes No Server Yes Yes

15 Spark on YARN Developed at Yahoo Applica=on Master per SparkContext Container per Spark executor Currently useful for Spark Batch jobs Requests all resources up front

16 Enhancing Spark on YARN Long- lived sessions Mul=ple Jobs Mul=ple Users

17 Long- Lived Goals Hang on to few resources when we re not running work Use lots of the cluster (over fair share) when it s not being used by others Give back resources gracefully when preempted Get resources quickly when we need them

18 Mesos Fine- Grained Mode Allocate sta=c chunks of memory at Spark app start =me Schedule CPU dynamically when running tasks

19 Long- Lived Approach A YARN applica=on master per Spark applica=on (SparkContext) Which is to say an applica=on master per session One executor per applica=on per node One YARN container per executor Executors can acquire and give back resources

20 Long- Lived: YARN work YARN long lived YARN YARN not built with apps that would s=ck around indefinitely Miscellaneous work like renewable container tokens YARN resizable containers

21 Long- Lived: Spark Work YARN fine- grained mode Changes to support adjus=ng resources in Spark AM Memory?

22 The Memory Problem We want to be able to have memory alloca=ons preempted and keep running RDDs stored in JVM memory JVMs don t give back memory

23 The Memory Solu=ons Rewrite Spark in C++ Off- heap cache Hold RDDs in executor processes in off- heap byte buffers These can be freed and returned to the OS Tachyon Executor processes don t hold RDDs Store data in Tachyon Punts off- heap problem to Tachyon Has other advantages, like not losing data when executor crashes

24 Mul=ple User Challenges A single Spark applica=on wants to run work on behalf of mul=ple par=es Applica=ons are typically billed to a single queue We d want to bill jobs to different queues Rajat from Marketing Cluster Spark App Sylvia from Finance

25 Mul=ple Users with Spark Fair Scheduler Full- features Fair Scheduler within a Spark Applica=on Two level scheduling Difficult to share dynamically between Spark and other frameworks

26 Mul=ple Users with Impala Impala has same exact problem Solu=on: Llama (Low Latency Applica=on MAster) Adapter between YARN and Impala Runs mul=ple AMs in a single process Submits resource requests on behalf of relevant AM Jobs billed to the YARN queues they belong in Cluster Spark App AM for Marketing Queue AM for Finance Queue Rajat from Marketing Sylvia from Finance

27 Spark Other Hadoop processing frameworks