Strategies for scheduling Hadoop Jobs. Pere Urbon-Bayes

Transcription

1 Strategies for scheduling Hadoop Jobs Pere Urbon-Bayes

2 $ whoami Software Architect with > 10 years of experience. Interested in data centric applications and graph analytics. I run the GraphDevRoom at FOSDEM. Movie and Series Geek, Runner, doing everything I can to enjoy this life. I believe we're getting payed for solving problems, not for having strong religious fights between technologies.

3 Scheduler.register(talk, Time.now) Houston, we've had a problem!. An state of the art. How to approach an internal solution.

4 Scheduler.register(talk, Time.now) Houston, we've had a problem!. An state of the art. How to approach an internal solution.

5 The situation We build large solar fields, for example in Templin, the north of Berlin, DE hectares photovoltaic modules MW 205 millions Euro Inside knowledge is very much appreciated.

6

7 A platform for this? The scheduler

8 The scheduler A nice scheduler should: Run hadoop jobs, but also PIG or shell scripts, etc. Consider executions idempotent. Have a nice interaction way, for example a REST api. Support workflows but also cron alike jobs. Track the job execution. Provide live statistics for monitoring.

9 Scheduler.register(talk, Time.now) Houston, we've had a problem!. An state of the art. How to approach an internal solution.

10 State of the art

11 State of the art Database Oozie Server HDFS REST api Oozie Client MapReduce JobTracker TaskTracker

12 State of the art MapReduce JobTracker TaskTracker Database Azkaban Server Azkaban Executor Client

13 State of the art Luigi scheduler Luigi Worker Pool HDFS Deployment throw codebase copy Client MapReduce JobTracker TaskTracker

14 State of the art Lang oozie Complexity (LOC) Framework Logs Community Docs java High > 105k Pig, hive, sqoop, mapreduce decentrali Good - ASF zed excellent azkaban java Moderate > 26k Pig, hive, mapreduce Centraliz ed. UI accessibl e Few users good Hive, postgress, scalding, python, streaming decentrali Few users zed good luigi phython Simple > 5.9k

15 State of the art Configuration oozie Command line, property files, xml azkaban Bundled inside the workflow zip, sytems defaults luigi Command line, python init file Replay Customization Testing Oozie job - difficult Mini oozie rerun Partial plugin? rerun UI Authorization Kerberos, simple, custom Xml based, custom Remove subclass output idempotent Linux, OS based Python unit test

16 State of the art So why not jumping into Oozie, it have: Great community and features A complete integration with the common Hadoop tooling Has a good documentation! BUT

17 State of the art But, it also have: An XML based configuration Scheduling based on map tasks A Hard setup An obscure object model And in this project case, we don't like plain Java that much.

18

19 Scheduler.register(talk, Time.now) Houston, we've had a problem!. An state of the art. How to approach an internal solution.

20 The wish list Easy to interact with, ex. REST api. Manage workflows and simple cron alike jobs. Have a centralized logging infrastructure To run not just Hadoop jobs, but also support scripting. Easy configuration and deployment model. Simplified extension model.

21 The wish list Be able to reply jobs. This means jobs data to be idempotent and status management. A report and monitoring module. Track the job execution. This include failures, timing, memory consumption, etc. But, for sure they are not all necessary at once

22

23 Our friends of convenience

24 A possible architecture? Registry Scheduler Server Cron Manager Scheduler Worker Database Workflow Manager REST api GraphDB File System MapReduce JobTracker TaskTracker

25 The workflow graph Task 2 S Task 1 Fork Task 3 Join Task 5 E Task 4 defined as Direct Acyclic Graph (DAG)

26

27

28 Hadoop (First approach) Submit and run tasks one after another.

29 Hadoop (second approach) A non blocking alternative

30 JobClient.runJob

31 JobClient.submitJob

32 JobClient.setJobEndNotificationURI

33 Remote Job

34 RemoteJob core-site.xml mapred-site.xml

35

36 Scheduler.add_job

37 Scheduler.run_job

38 Scheduler::Internals.run_java

39 Scheduler.register(talk, Time.now) Houston, we've had a problem!. An state of the art. How to approach an internal solution.

40 Strategies for scheduling Hadoop jobs Thanks for your attention! Questions? Pere Urbon-Bayes