Managing large clusters resources

Transcription

1 Managing large clusters resources ID2210 Gautier Berthou (SICS)

2 Big Processing with No Locality Job( /crawler/bot/jd.io/1 ) submi t Workflow Manager Compute Grid Node Job This doesn t scale. Bandwidth is the bottleneck

3 MapReduce Locality Job( /crawler/bot/jd.io/1 ) submi t Job Tracker Task Tracker Task Tracker Task Tracker Task Tracker Task Tracker Task Tracker Job Job Job Job Job Job DN DN DN DN DN DN R R = esultfile(s) R R

4 First step Single Processing Framework Batch Apps Hadoop 1.x MapReduce (resource mgmt, job scheduler, data processing) HDFS (distributed storage) 4

5 MapReduce Locality Job( /crawler/bot/jd.io/1 ) submi t Job Tracker Task Tracker Task Tracker Task Tracker Task Tracker Task Tracker Task Tracker Job Job Job Job Job Job DN DN DN DN DN DN R R = esultfile(s) R R

6 The Job Tracker Job Distribute the map and reduce tasks on the s of the cluster Ensure fairness of the cluster resource attributions Track the progress of these tasks Authenticate job tenants and make sure that each job is isolated from the others Etc. 6

7 Limitations of MapReduce [Zaharia 11] MapReduce is based on an acyclic data flow from stable storage to stable storage. - Slow writes data to HDFS at every stage in the pipeline Acyclic data flow is inefficient for applications that repeatedly reuse a working set of data: - Iterative algorithms (machine learning, graphs) - Interactive data mining tools (R, Excel, Python) Map Reduce Input Map Output Map Reduce

8 Limitations Only one programming model. The map reduce framework is not using the cluster at its maximum. The job tracker is a bottle neck. The job tracker is a single point of failure. 8

9 Goals for a new Scheduler Being able to run different frameworks Scale Provide advance scheduling policies Run efficiently with different kind of workloads 9

10 Second step Multiple Processing Frameworks Batch, Interactive, Streaming Hadoop 2.x MapReduce (data processing) Others (spark, mpi, giraph, etc) YARN (resource mgmt, job scheduler) HDFS (distributed storage) 10

11 Examples of scheduling Policies Capacity scheduler: - Applications have different levels of priorities. Fair scheduler: - Applications have different levels of priorities. - Used resources can be preempted. Reservation-based scheduler (1) : - Applications can indicate how long they will run and when they have to be finished. (1) Reservation-based Scheduling: If you re late don t blame us!, C. Curino & al., Microsoft tech-report 11

12 Scenario 3 kinds of jobs: - Emergency jobs: need to be run as soon as possible. - Production jobs: have a deadline, a known running time and are very exigent on the s they can be scheduled on. - Best effort jobs: interactive jobs that have lower priority, but on which users expect low latency. 12

13 Capacity scheduler Best effort emergency production Best effort Best effort Best effort Best effort Best effort production Now Production work deadline 13

14 Fair scheduler Best effort emergency Best effort Best production effort Best effort Best effort Now Production work deadline 14

15 Reservation-based scheduler Best effort emergency Best effort production production Best effort Best effort Now Production work deadline 15

16 Scheduler Architectures Omega: flexible, scalable schedulers for large compute clusters, Malte Schwarzkopf & al., EuroSys 13 16

17 The monolithic Scheduler Yarn: - Apache Hadoop YARN: Yet Another Resource Negotiator, V. K. Vavilapalli & al., SoCC 13. Borg: - Large-scale cluster management at Google with Borg, A. Verma & al., EuroSys

18 Architecture 1/3 18

19 Architecture 2/3 Resources Manager 19

20 Architecture 3/3 zookeeper Standby Master Resources Manager Standby Resources Resources Manager Manager Master Resources Manager 20

21 Pros and Cons Pros: - Fine knowledge of the state of the cluster state -> optimal use of the cluster resources. - Easy to implement new scheduling policies. Cons: - Bottle neck. - The failure of the master scheduler has a big impact on the cluster usage. 21

22 Two level Scheduler Mesos: A Platform for Fine-Grained Resource Sharing in the Center, B. Hindman & al., NSDI 11 22

23 Architecture 1/2 23

24 Architecture 2/2 MapReduce Scheduler Partial State Spark Scheduler Flink Scheduler Mesos Master 24

25 Pros and Cons Pros: - Scale out by adding schedulers. - Concurrent scheduling of tasks. Cons: - Suboptimal use of the cluster. Especially when there exist long running tasks. 25

26 Shared State Scheduler Omega: flexible, scalable schedulers for large compute clusters, M. Schwarzkopf & al. EuroSys 13 26

27 Architecture 1/2 27

28 Architecture 2/2 MapReduce Scheduler Global state` Spark Scheduler Flink Scheduler State Manager 28

29 Architecture 2/2 MapReduce Scheduler Global state Spark Scheduler Conflict Flink Scheduler State Manager 29

30 Pros and Cons Pros - Scalable. - Good use of the cluster resources. Cons - Unpredictable interaction between the different schedulers policies. 30

31 Comparison 31

32 Performance comparison 1/ Simple monolithic Advence monolithic Two-level Shared state 32

33 Performance Comparison 2/ What the previous evaluation does not show about the Two-level scheduling: 33

34 Performance Comparison 3/ Trying to handle more batch jobs in Omega by running several batch schedulers in parallel. How to write an good accepted paper. paper. 34

35 Sum up Two-Level and Shared state Schedulers scale better. Shared state Schedulers use the cluster resources more optimally than Two-level Schedulers. Monolithic Scheduler are a potential Bottleneck. But as Monolithic schedulers are easier to design, allow finer allocation of resources and more advance scheduling policies, they are the ones used in practice. 35

36 Making Yarn more scalable HOPS YARN: a one and a half level scheduler 36

37 Hadoop Yarn HA Implementation zookeeper Standby Master Resources Manager Standby Resources Resources Manager Manager Master Resources Manager 37

38 Hops Yarn HA Implementation3/3 NDB Standby Master Resources Manager Standby Resources Resources Manager Manager Master Resources Manager 38

39 MySQL Cluster (NDB) Shared Nothing DB SQL API NDB API Distributed, In-memory 2-Phase Commit - Replicate DB, not the Log! Real-time - Low TransactionInactive timeouts Commodity Hardware Scales out - Millions of transactions/sec - TB-sized datasets (48 s) Split-Brain solved with Arbitrator Pattern SQL and Native Blocking/Non- Blocking APIs 30+ million update transactions/second on a 30- cluster 39

40 Standby is boring Master Resources Manager Standby Resources Manager Standby Resources Manager 40

41 Dificulties 1/2 41

42 Difficulties 2/2 Pulling from the database when the state is needed is inefficient. Having an independent thread that regularly pull from the database is difficult to tune and cause lock problems. 42

43 Solution Luckily NDB has an event API. 43

44 With streaming Master Resources Manager Standby Resources Manager Standby Resources Manager 44

45 Conclusion There exists three architectures for large cluster resource scheduling: - Monolithic - Two-levels - Shared State Each of these architectures has pros and cons. The monolithic architectur is the one presently used because it is easyer to use and develop. At KTH and SICS we are exploring the possibilities for a new architecture ensuring more scalability while keeping the advantages of the monolithic architecture.

46 References Reservation-based Scheduling: If you re late don t blame us!, C. Curino & al., Microsoft tech-report Omega: flexible, scalable schedulers for large compute clusters, Malte Schwarzkopf & al., EuroSys 13 Apache Hadoop YARN: Yet Another Resource Negotiator, V. K. Vavilapalli & al., SoCC 13. Large-scale cluster management at Google with Borg, A. Verma & al., EuroSys 15. Mesos: A Platform for Fine-Grained Resource Sharing in the Center, B. Hindman & al., NSDI 11