Programming and Debugging Large- Scale Data Processing Workflows. Christopher Olston and many others Yahoo! Research

Transcription

1 Programming and Debugging Large- Scale Data Processing Workflows Christopher Olston and many others Yahoo! Research

2 Context Elaborate processing of large data sets e.g.: web search pre- processing cross- dataset linkage web informa=on extrac=on inges?on storage & processing serving

3 Context storage & processing workflow manager e.g. Nova dataflow programming framework e.g. Pig distributed sor=ng & hashing e.g. Map- Reduce Overview scalable file system e.g. GFS Detail: Inspector Gadget, RubySky Debugging aides: Before: example data generator During: instrumenta=on framework ABer: provenance metadata manager

4 Pig: A High- Level Dataflow Language and Run=me for Hadoop Web browsing sessions with happy endings. Visits = load /data/visits as (user, url, time);! Visits = foreach Visits generate user, Canonicalize(url), time;!! Pages = load /data/pages as (url, pagerank);!! VP = join Visits by url, Pages by url;! UserVisits = group VP by user;! Sessions = foreach UserVisits generate flatten(findsessions(*));! HappyEndings = filter Sessions by BestIsLast(*);!! store HappyEndings into '/data/happy_endings';!

5 vs. map- reduce: less code! "The [Hofmann PLSA E/M] algorithm was implemented in pig in lines of pig-latin statements. Took a lot less compared to what it took in implementing the algorithm in Map-Reduce Java. Exactly that's the reason I wanted to try it out in Pig. It took 3-4 days for me to write it, starting from learning pig. " " -- Prasenjit Mukherjee, Mahout project" /20 the lines of code Minutes /16 the development =me Hadoop Pig Hadoop Pig performs on par with raw Hadoop

6 vs. SQL: step- by- step style; lower- level control "I much prefer writing in Pig [Latin] versus SQL. The step-by-step method of" creating a program in Pig [Latin] is much cleaner and simpler to use than the single block method of SQL. It is easier to keep track of what your variables are, and where you are in the process of analyzing your data. " " -- Jasmine Novak, Engineer, Yahoo!" "PIG seems to give the necessary parallel programming construct (FOREACH, FLATTEN, COGROUP.. etc) and also give sufficient control back to the programmer (which purely declarative approach like [SQL on top of Map- Reduce] doesnʼt). " " -- Ricky Ho, Adobe Software"

7 Conceptually: A Graph of Data Transforma=ons Find users who tend to visit good pages. Load Visits(user, url, =me) Load Pages(url, pagerank) Transform to (user, Canonicalize(url), =me) Join url = url Group by user Transform to (user, Average(pagerank) as avgpr) Filter avgpr > 0.5

8 Load Visits(user, url, =me) Load Pages(url, pagerank) Illustrated! Transform to (user, Canonicalize(url), =me) (Amy, cnn.com, 8am) (Amy, hhp:// 9am) (Fred, 11am) (Amy, 8am) (Amy, 9am) (Fred, 11am) Join url = url Group by user (Amy, 8am, 0.9) (Amy, 9am, 0.4) (Fred, 11am, 0.4) ( 0.9) ( 0.4) (Amy, { (Amy, 8am, 0.9), (Amy, 9am, 0.4) }) (Fred, { (Fred, 11am, 0.4) }) Transform to (user, Average(pagerank) as avgpr) ILLUSTRATE lets me check the output of my lengthy (Amy, 0.65) batch jobs and their (Fred, 0.4) custom functions without having to do a lengthy run of a long pipeline. [This feature] enables me to be productive. Filter " " avgpr > Russell Jurney, LinkedIn" (Amy, 0.65)

9 (Naïve Algorithm) Load Visits(user, url, =me) Transform to (user, Canonicalize(url), =me) (Amy, cnn.com, 8am) (Amy, hhp:// 9am) (Fred, 11am) (Amy, 8am) (Amy, 9am) (Fred, 11am) Join url = url Group by user Transform to (user, Average(pagerank) as avgpr) Load Pages(url, pagerank) ( 0.9) ( 0.4) Filter avgpr > 0.5

10 Pig Project Status Produc=zed at Yahoo (~12- person team) 1000s of jobs/day 70% of Hadoop jobs Open- source (the Apache Pig Project) Offered on Amazon Elas=c Map- Reduce Used by LinkedIn, Twiher, Yahoo,...

11 Next: NOVA storage & processing workflow manager e.g. Nova dataflow programming framework e.g. Pig distributed sor=ng & hashing e.g. Map- Reduce scalable file system e.g. GFS Debugging aides: Before: example data generator During: instrumenta=on framework ABer: provenance metadata manager

12 Why a Workflow Manager? Con=nuous data processing (simulated on top of Pig/Hadoop sta=c processing layer) Independent scheduling of workflow modules

13 Example RSS feed NEW Workflow template detec=on ALL news ar=cles ALL news site templates ALL NEW template tagging NEW NEW shingling NEW shingle hashes seen ALL NEW NEW de- duping NEW unique ar=cles

14 Data Passes Through Many Sub- Systems Nova datum X inges=on Pig Map- Reduce low- latency processor serving datum Y GFS provenance of X? metadata queries

15 Ibis Project metadata Ibis metadata queries integrated metadata answers data processing sub- systems metadata manager users Benefits: Provide uniform view to users Factor out metadata management code Decouple metadata life=me from data/subsystem life=me Challenges: Overhead of shipping metadata Disparate data/processing granulari=es

16 What s Hard About Mul=- Granularity Provenance? Inference: Given rela=onships expressed at one granularity, answer queries about other granulari=es (the seman;cs are tricky here!) Efficiency: Implement inference without resor=ng to materializing everything in terms of finest granularity (e.g. cells)

17 Next: INSPECTOR GADGET storage & processing workflow manager e.g. Nova dataflow programming framework e.g. Pig distributed sor=ng & hashing e.g. Map- Reduce scalable file system e.g. GFS Debugging aides: Before: example data generator During: instrumenta=on framework ABer: provenance metadata manager

18 Mo=vated by User Interviews Interviewed 10 Yahoo dataflow programmers (mostly Pig users; some users of other dataflow environments) Asked them how they (wish they could) debug

19 Summary of User Interviews # of requests feature 7 crash culprit determina=on 5 row- level integrity alerts 4 table- level integrity alerts 4 data samples 3 data summaries 3 memory use monitoring 3 backward tracing (provenance) 2 forward tracing 2 golden data/logic tes=ng 2 step- through debugging 2 latency alerts 1 latency profiling 1 overhead profiling 1 trial runs

20 Our Approach Goal: a programming framework for adding these behaviors, and others, to Pig Precept: avoid modifying Pig or tampering with data flowing through Pig Approach: perform Pig script rewri=ng insert special UDFs that look like no- ops to Pig

21 Pig w/ Inspector Gadget load filter load join IG coordinator group count store

22 Example: Crash Culprit Determina;on Phases 1 to n- 1: record counts Phase n: records load filter load join IG coordinator group Phases 1 to n- 1: maintain count lower bounds Phase n: maintain last- seen records count store

23 Example: Forward Tracing load load filter IG coordinator traced records join group tracing instruc=ons report traced records to user count store

24 Flow end user result dataflow program + app. parameters applica=on IG driver library launch instrumented dataflow run(s) raw result(s) load load IG coordinator filter join dataflow engine run;me store

25 Agent & Coordinator APIs Agent Class init(args) tags = observerecord(record, tags) receivemessage(source, message) finish() Agent Messaging sendtocoordinator(message) sendtoagent(agentid, message) senddownstream(message) sendupstream(message) Coordinator Class init(args) receivemessage(source, message) output = finish() Coordinator Messaging sendtoagent(agentid, message)

26 Applica=ons Developed Using IG # of requests feature lines of code (Java) 7 crash culprit determina=on row- level integrity alerts 89 4 table- level integrity alerts 99 4 data samples 97 3 data summaries memory use monitoring N/A 3 backward tracing (provenance) forward tracing golden data/logic tes=ng step- through debugging N/A 2 latency alerts latency profiling overhead profiling trial runs 93

27 RubySky: Unified Scrip=ng Framework for Processing and Debugging Scenario: ad- hoc data analysis e.g. we just did a crawl; is it too biased toward Yahoo content? In this scenario, some=mes using new data; always running new code à Bugs are the norm à Debugging should be a first class ci=zen!

28 RubySky Example :Crawl_data <= load(context, "/crawl_data", schema {string :url;...})!! # quick-and-dirty determination of "domain" from "url" (e.g. -> yahoo.com)! :With_domains <= Crawl_data.foreach(as {string :url; string :domain}) do r! url = r[:url]! prefix = if (url[0, prefix.length] == prefix)! domain = url[prefix.length, url.index('/', prefix.length) - prefix.length]! else! throw Don t know how to parse this url: + url! end! Tuple.new([url, domain])! end!! :Yahoo_only <= With_domains.filter do r! r[:domain].match(/^yahoo/)! end!! :Grouped <= Yahoo_only.group!! :Grouped.each { r puts NUMBER OF YAHOO URLs: + r[:yahoo_only].size }!

29 RubySky Example (cont.) :Crawl_data <= load(context, "/crawl_data", schema {string :url;...})!! # quick-and-dirty determination of "domain" from "url" (e.g. -> yahoo.com)! :With_domains <= Crawl_data.foreach(as {string :url; string :domain}) do r! url = r[:url]! prefix = if (url[0, prefix.length] == prefix)! domain = url[prefix.length, url.index('/', prefix.length) - prefix.length]! else! domain = punt url! end! Tuple.new([url, domain])! end!! # SPY: examine a few extracted domains, as sanity check! :Domain_sample <= With_domains.foreach(as {string :domain}) do r! $count += 1! ($count < SAMPLE_SIZE)? [Tuple.new([r[:domain]])] : []! end! show Domain_sample!! # SPY: alert if any extracted domain is empty or null! :Domain_missing <= With_domains.foreach(as {string :alert}) do r! (r[:domain] == "" or r[:domain] == nil)? [Tuple.new(["ALERT! Missing domain from: " + r[:url]])] : []! end! show Domain_missing!! :Yahoo_only <= With_domains.filter do r! r[:domain].match(/^yahoo/)! end!! :Grouped <= Yahoo_only.group!! :Grouped.each { r puts NUMBER OF YAHOO URLs: + r[:yahoo_only].size }!

30 RubySky Execu=on launch load client punt request response (code and/or data) domain samples empty domain alerts extract domain spy agent filter group answer (# yahoo URLs) count

31 Summary storage & processing workflow manager e.g. Nova dataflow programming framework e.g. Pig distributed sor=ng & hashing e.g. Map- Reduce scalable file system e.g. GFS Nova Pig Debugging aides: Before: example data generator During: instrumenta=on framework ABer: provenance metadata manager Dataflow Illustrator Inspector Gadget, RubySky Ibis

32 Related Work Pig: DryadLINQ, Hive, Jaql, Scope, rela;onal query languages Nova: BigTable, CBP, Oozie, Percolator, scien;fic workflow, incremental view maintenance Dataflow illustrator: [Mannila/Raiha, PODS 86], reverse query processing, constraint databases, hardware verifica;on & model checking Inspector gadget, RubySky: XTrace, taint tracking, aspect- oriented programming Ibis: Kepler COMAD, ZOOM user views, provenance management for databases & scien;fic workflows

33 Collaborators Shubham Chopra Tyson Condie Anish Das Sarma Alan Gates Pradeep Kamath Ravi Kumar Shravan Narayanamurthy Olga Natkovich Benjamin Reed Santhosh Srinivasan Utkarsh Srivastava Andrew Tomkins