How To Write A Program On A Microsoft.Com (For Microsoft) With A Microsail (For Linux) And A Microosoft.Com Microsails (For Windows) (For Macintosh) ( For Microsils

Transcription

1 Programming and Debugging Large- Scale Data Processing Workflows Christopher Olston Google Research (work done at Yahoo! Research, with many colleagues)

2 What I m Working on at Google We ve got fantasjc cloud building blocks: BigTable, MapReduce, Pregel, and on and on (and so do you: EC2, Hadoop, Redis, ZooKeeper, ) To build your app: 1. Think hard, and choose a few building blocks (BB s) 2. SJck your app logic into a blender, and pour it into the various BB abstracjons (keys/values, MR funcjons, callbacks, ) 3. Tune it: stupid map- reduce tricks ; set bazillions of flags 4. Hope that Your BB choices were right, and stay right for a while Nobody ever has to understand your app by reading the code You never a_empt big changes to your app logic or algorithms

3 What I m Working on at Google We ve got fantasjc cloud building blocks: BigTable, MapReduce, Pregel, and on and on (and so do you: EC2, Hadoop, Redis, ZooKeeper, ) To build your app: 1. Think hard, and choose a few building blocks (BB s) 2. SJck What your app if we logic could into separate a blender, the and app pour logic it into the various BB abstracjons from the assemblage (keys/values, of MR building funcjons, blocks? callbacks, ) 3. Tune it: stupid map- reduce tricks ; set bazillions of flags 4. Hope that Your BB choices were right, and stay right for a while Nobody ever has to understand your app by reading the code You never a_empt big changes to your app logic or algorithms

4 Research at Google High- risk/high- reward research happening across the company Successful research projects oben become successful products (e.g. speech recognijon) No pressure to publish incremental papers Interdisciplinary In my case: DB + PL + AI We re hiring J

5 Now, back to your regularly scheduled program

6 Context Elaborate processing of large data sets e.g.: web search pre- processing cross- dataset linkage web informajon extracjon inges)on storage & processing serving

7 Context storage & processing workflow manager e.g. Nova dataflow programming framework e.g. Pig distributed sorjng & hashing e.g. Map- Reduce Overview scalable file system e.g. GFS Detail: Inspector Gadget, RubySky Debugging aides: Before: example data generator During: instrumentajon framework Aber: provenance metadata manager

8 Pig: A High- Level Dataflow Language & RunJme for Hadoop Web browsing sessions with happy endings. Visits = load /data/visits as (user, url, time);! Visits = foreach Visits generate user, Canonicalize(url), time;!! Pages = load /data/pages as (url, pagerank);!! VP = join Visits by url, Pages by url;! UserVisits = group VP by user;! Sessions = foreach UserVisits generate flatten(findsessions(*));! HappyEndings = filter Sessions by BestIsLast(*);!! store HappyEndings into '/data/happy_endings';!

9 vs. map- reduce: less code! "The [Hofmann PLSA E/M] algorithm was implemented in pig in lines of pig-latin statements. Took a lot less compared to what it took in implementing the algorithm in Map-Reduce Java. Exactly that's the reason I wanted to try it out in Pig. It took 3-4 days for me to write it, starting from learning pig. " " -- Prasenjit Mukherjee, Mahout project" /20 the lines of code Minutes /16 the development Jme Hadoop Pig Hadoop Pig performs on par with raw Hadoop

10 vs. SQL: step- by- step style; lower- level control "I much prefer writing in Pig [Latin] versus SQL. The step-by-step method of" creating a program in Pig [Latin] is much cleaner and simpler to use than the single block method of SQL. It is easier to keep track of what your variables are, and where you are in the process of analyzing your data. " " -- Jasmine Novak, Engineer, Yahoo!" "PIG seems to give the necessary parallel programming construct (FOREACH, FLATTEN, COGROUP.. etc) and also give sufficient control back to the programmer (which purely declarative approach like [SQL on Map-Reduce] doesnʼt). " " -- Ricky Ho, Adobe Software"

11 Conceptually: A Graph of Data TransformaJons Find users who tend to visit good pages. Load Visits(user, url, Jme) Load Pages(url, pagerank) Transform to (user, Canonicalize(url), Jme) Join url = url Group by user Transform to (user, Average(pagerank) as avgpr) Filter avgpr > 0.5

12 Load Visits(user, url, Jme) Load Pages(url, pagerank) Illustrated! Transform to (user, Canonicalize(url), Jme) (Amy, cnn.com, 8am) (Amy, h_p:// 9am) (Fred, 11am) (Amy, 8am) (Amy, 9am) (Fred, 11am) Join url = url Group by user (Amy, 8am, 0.9) (Amy, 9am, 0.4) (Fred, 11am, 0.4) ( 0.9) ( 0.4) (Amy, { (Amy, 8am, 0.9), (Amy, 9am, 0.4) }) (Fred, { (Fred, 11am, 0.4) }) Transform to (user, Average(pagerank) as avgpr) (Amy, 0.65) (Fred, 0.4) Filter avgpr > 0.5 (Amy, 0.65)

13 Load Visits(user, url, Jme) Load Pages(url, pagerank) Illustrated! Transform to (user, Canonicalize(url), Jme) (Amy, cnn.com, 8am) (Amy, h_p:// 9am) (Fred, 11am) (Amy, 8am) (Amy, 9am) (Fred, 11am) Join url = url Group by user (Amy, 8am, 0.9) (Amy, 9am, 0.4) (Fred, 11am, 0.4) ( 0.9) ( 0.4) (Amy, { (Amy, 8am, 0.9), (Amy, 9am, 0.4) }) (Fred, { (Fred, 11am, 0.4) }) Transform to (user, Average(pagerank) as avgpr) ILLUSTRATE lets me check the output of my lengthy (Amy, 0.65) batch jobs and their (Fred, 0.4) custom functions without having to do a lengthy run of a long pipeline. [This feature] enables me to be productive. Filter " " avgpr > Russell Jurney, LinkedIn" (Amy, 0.65)

14 (Naïve Algorithm) Load Visits(user, url, Jme) Transform to (user, Canonicalize(url), Jme) (Amy, cnn.com, 8am) (Amy, h_p:// 9am) (Fred, 11am) (Amy, 8am) (Amy, 9am) (Fred, 11am) Join url = url Group by user Transform to (user, Average(pagerank) as avgpr) Load Pages(url, pagerank) ( 0.9) ( 0.4) Filter avgpr > 0.5

15 Original UI Prototype (2008)

16 InteracJve Graphics Programming UI by Bret Victor, Apple Designer (2012) Creators need an immediate connection to what they create If you make a change you need to see the effect of that immediately. " " -- Bret Victor, Apple"

17 Pig Today Open- source (the Apache Pig Project) Dev./support/training by Cloudera, Hortonworks Offered on Amazon ElasJc Map- Reduce Used by LinkedIn, Nexlix, Salesforce, Twi_er, Yahoo... At Yahoo, as of early 2011: 1000s of jobs/day 75%+ of Hadoop jobs Has an interacjve example- generator command, but no side- by- side UI L

18 Next: NOVA storage & processing workflow manager e.g. Nova dataflow programming framework e.g. Pig distributed sorjng & hashing e.g. Map- Reduce scalable file system e.g. GFS Debugging aides: Before: example data generator During: instrumentajon framework Aber: provenance metadata manager

19 Why a Workflow Manager? ConJnuous data processing (simulated on top of Pig/Hadoop stajc processing layer) Independent scheduling of workflow modules

20 Example RSS feed NEW Workflow template detecjon ALL news arjcles ALL news site templates ALL NEW template tagging NEW NEW shingling NEW shingle hashes seen ALL NEW NEW de- duping NEW unique arjcles

21 Data Passes Through Many Sub- Systems Nova datum X ingesjon Pig Map- Reduce low- latency processor serving datum Y GFS provenance of X? metadata queries

22 Ibis Project metadata Ibis metadata queries integrated metadata answers data processing sub- systems metadata manager users Benefits: Provide uniform view to users Factor out metadata management code Decouple metadata lifejme from data/subsystem lifejme Challenges: Overhead of shipping metadata Disparate data/processing granularijes

23 Example Data and Process GranulariJes Workflow Table MR program Pig script Web page Row Cell Column group Column AlternaJve Pig job MR job MR job phase MR task Pig logical operajon Pig physical operajon Task a_empt data granulari7es process granulari7es

24 What s Hard About MulJ- Granularity Provenance? Inference: Given relajonships expressed at one granularity, answer queries about other granularijes (the seman;cs are tricky here!) Efficiency: Implement inference without resorjng to materializing everything in terms of finest granularity (e.g. cells)

25 Inferring TransiJve Links map phase reduce phase IMDB web page Yahoo! Movies web page IMDB extracted table )tle year lead actor Avatar 2009 Worthington IncepJon 2010 DiCaprio Yahoo extracted table )tle year lead actor Avatar 2009 Saldana IncepJon 2010 DiCaprio map output 1 map output 2 combined table )tle year lead actor Avatar 2009 A1: Worthington A2: Saldana IncepJon 2010 DiCaprio Q: Which web page said Saldana?

26 Next: INSPECTOR GADGET storage & processing workflow manager e.g. Nova dataflow programming framework e.g. Pig distributed sorjng & hashing e.g. Map- Reduce scalable file system e.g. GFS Debugging aides: Before: example data generator During: instrumentajon framework Aber: provenance metadata manager

27 MoJvated by User Interviews Interviewed 10 Yahoo dataflow programmers (mostly Pig users; some users of other dataflow environments) Asked them how they (wish they could) debug

28 Summary of User Interviews # of requests feature 7 crash culprit determinajon 5 row- level integrity alerts 4 table- level integrity alerts 4 data samples 3 data summaries 3 memory use monitoring 3 backward tracing (provenance) 2 forward tracing 2 golden data/logic tesjng 2 step- through debugging 2 latency alerts 1 latency profiling 1 overhead profiling 1 trial runs

29 Our Approach Goal: a programming framework for adding these behaviors, and others, to Pig Precept: avoid modifying Pig or tampering with data flowing through Pig Approach: perform Pig script rewrijng insert special UDFs that look like no- ops to Pig

30 Pig w/ Inspector Gadget load filter load join IG coordinator group count store

31 Example: Integrity Alerts load filter load alert! join IG coordinator propagate alert to user group count store

32 Example: Crash Culprit Determina;on Phases 1 to n- 1: record counts Phase n: records load filter load join IG coordinator group Phases 1 to n- 1: maintain count lower bounds Phase n: maintain last- seen records count store

33 Example: Forward Tracing load load filter IG coordinator traced records join group tracing instrucjons report traced records to user count store

34 Flow end user result dataflow program + app. parameters applicajon IG driver library launch instrumented dataflow run(s) raw result(s) load load IG coordinator filter join dataflow engine run;me store

35 Agent & Coordinator APIs Agent Class init(args) tags = observerecord(record, tags) receivemessage(source, message) finish() Agent Messaging sendtocoordinator(message) sendtoagent(agentid, message) senddownstream(message) sendupstream(message) Coordinator Class init(args) receivemessage(source, message) output = finish() Coordinator Messaging sendtoagent(agentid, message)

36 ApplicaJons Developed For IG # of requests feature lines of code (Java) 7 crash culprit determinajon row- level integrity alerts 89 4 table- level integrity alerts 99 4 data samples 97 3 data summaries memory use monitoring N/A 3 backward tracing (provenance) forward tracing golden data/logic tesjng step- through debugging N/A 2 latency alerts latency profiling overhead profiling trial runs 93

37 RubySky: ScripJng Framework with Built- in Debugging Support Scenario: ad- hoc data analysis e.g. we just did a crawl; is it too biased toward Yahoo content? In this scenario, somejmes using new data; always running new code à Bugs are the norm à Debugging should be a first class cijzen!

38 RubySky Example :Crawl_data <= load(context, "/crawl_data", schema {string :url;...})!! # quick-and-dirty determination of "domain" from "url"! # (e.g. => yahoo.com)! :With_domains <= Crawl_data.foreach() do r! url = r[:url]! prefix = if (url[0, prefix.len] == prefix)! domain = url[prefix.len, url.index('/', prefix.len) - prefix.len]! else! throw Don t know how to parse this url: + url! end! Tuple.new([url, domain])! end!! :Yahoo_only <= With_domains.filter do r! r[:domain] =~ /^yahoo/! end!! :Grouped <= Yahoo_only.group!! :Grouped.each { r puts NUM. YAHOO URLs: + r[:yahoo_only].size }!

39 RubySky Example: w/debug stmts...! :With_domains <= Crawl_data.foreach() do r!...! else! domain = PUNT url! end! Tuple.new([url, domain])! end!! # examine a few extracted domains, as sanity check! :Domain_sample <= With_domains.foreach() do r! $count += 1! ($count < SAMPLE_SIZE)? [Tuple.new([r[:domain]])] : []! end! SHOW Domain_sample!! # alert if any extracted domain is empty or null! :Domain_missing <= With_domains.foreach() do r! (r[:domain] == "" or r[:domain] == nil)?! [Tuple.new(["ALERT! Missing domain from: " + r[:url]])] : []! end! SHOW Domain_missing!!...!

40 RubySky ExecuJon launch load client punt request response (code and/or data) domain samples empty domain alerts extract domain spy agent filter group answer (# yahoo URLs) count

41 Summary storage & processing workflow manager e.g. Nova dataflow programming framework e.g. Pig distributed sorjng & hashing e.g. Map- Reduce scalable file system e.g. GFS Nova Pig Debugging aides: Before: example data generator During: instrumentajon framework Aber: provenance metadata manager Dataflow Illustrator Inspector Gadget, RubySky Ibis

42 Related Work Pig: DryadLINQ, Hive, Jaql, Scope, rela;onal query languages Nova: BigTable, CBP, Oozie, Percolator, scien;fic workflow, incremental view maintenance Dataflow illustrator: [Mannila/Raiha, PODS 86], reverse query processing, constraint databases, hardware verifica;on & model checking Inspector gadget, RubySky: XTrace, taint tracking, aspect- oriented programming Ibis: Kepler COMAD, ZOOM user views, provenance management for databases & scien;fic workflows

43 Collaborators Shubham Chopra Tyson Condie Anish Das Sarma Alan Gates Pradeep Kamath Ravi Kumar Shravan Narayanamurthy Olga Natkovich Benjamin Reed Santhosh Srinivasan Utkarsh Srivastava Andrew Tomkins