Workflow Management System for Stratosphere
|
|
- Dorthy Fleming
- 8 years ago
- Views:
Transcription
1 Workflow Management System for Stratosphere 1 THESIS PRESENTATION BY SURYAMITA HARINDRARI SEPTEMBER 5 TH, 2014 THESIS ADVISOR: ASTERIOS KATSIFODIMOS, PHD THESIS SUPERVISOR: PROF. DR. VOLKER MARKL DATABASE & INFORMATION MANAGEMENT (DIMA) TECHNISCHE UNIVERSITÄT BERLIN
2 Background Agenda Workflow & Workflow Management System Control Flow vs Data Flow Related Work Motivation Approach Stage 1: Translating AST to Control Flow Graph Abstract Syntax Tree (AST) Control Flow Graph Stage 2: Adding Data Flow to the Control Flow Graph Data Flow Analysis Stage 3: Generate Code for Underlying System Evaluation: Productivity & Generality Conclusion Future Work 2
3 Workflows & Workflow Management System 3 Big Data Analytics à Complex applications to process large datasets on distributed resources Workflow: Automate procedures that otherwise needed to be carried out manually [Deelman et al, 2009] Sequence of steps or computation [Crobak, 2012] Workflow Management System (WMS): Defines, manages and executes workflows Order of execution is driven by a computer representation of the workflow logic [Hollingsworth et al, 1993]
4 Simple Workflow vs Complex Workflow 4 Promoter Identification Workflow [Ludäscher et al, 2005] ETL Process Workflow [Crobak, 2012]
5 Taxonomy of a Workflow Workflow Taxonomy [Yu et al, 2005] 5
6 Data Flow Data Flow vs Control Flow Related Work on Data Flow Systems: Hadoop MR, Stratosphere, Pig, Hive, Jet Limitations: Does not support control structures Low level optimized code à reduce productivity High overhead in learning new language i.e. Pig Latin Control Flow Related Work on Workflow Systems: Oozie, Luigi, Azkaban, Kepler, Spark Limitations: Markup languages à cumbersome Graphical representation à limited Tasks & Data dependencies defined manually 6
7 Problem Motivation Stratosphere à does not support control flow outside UDFs Existing workflow systems à dependencies specified manually Solution WMS that automatically detects the control flow and data dependencies between tasks from pure program code Intuitive way for the programmer to define the workflow Goals Design and develop a WMS that works on top of Stratosphere Define a workflow domain specific language (DSL) to make defining workflows easier 7
8 Workflow Design: Our Taxonomy The Design of Our Workflow System 8
9 Approach 9 Translate the program code into target code: Translate user program to Intermediate Representation (IR) Control Flow Graph (CFG) Add data flow to the CFG Generate code for underlying system WMS execute the jobs
10 Stage 1 Part 1: Translate User Program to AST 10 Compiler constructs a sequence of Intermediate Representations (IR) which can have a variety of forms Abstract Syntax Trees (AST) à data structure that represents program constructs. Each node in AST represents operator Children of a node in AST represent the operands of the operator
11 Grammar Definition & AST Representation Grammar Definition supported by our DSL 11
12 Our Tool: Scala AST Reuse the Scala AST given freely by the Scala compiler Scala Macros Compile time metaprogramming Expand trees at compile time enabling programmers to hack and manipulate AST within compilation scope Scala AST Classes [Stocker, 2010] Block List of statements and return value of expression ValDef Immutable and mutable variable or statements Assign non-initial assignments to variables If consists of cond, thenp, and elsep sub-tree LabelDef represents iteration statement 12
13 Generating AST from User Program Sample program in our workflow DSL val e1 = DataSource(..") val e2 = DataSource(..") var e3: DataSet[(String, Int, Int)] = null var i = 0 while(i < 0) { if (e1.map(x => x._2) > 50) e3 = e1.map { x => (x._1, x._ , x._3)} else e3 = e2.map { x => (x._1, x._ , x._3)} i = i + 1 } val e4 = e3.write( ) e4 13
14 Stage 1 Part 2: Generate Control Flow Graph from AST 14 Control Flow Graph Directed graph in which the nodes represent basic blocks and the edges represent control flow paths [Allen, 1970] Basic Blocks à sequences of instructions or statements that are always executed together Edges represent possible flow of control from the end of one basic block to the beginning of another
15 CFG for Various Statements 15
16 Generated CFG from AST 16
17 Generated CFG from AST Algorithm (1 of 2) 17
18 Create CFG from AST Algorithm (2 of 2) 18
19 Stage 2: Generate CF-Enriched Data Flow 19 Data Flow Analysis [Lam et al, 2006] Transmission of information through program variables missing in CFG Derive the information about the flow of data along with program execution paths Traverse the CFG to detect data dependencies Add another type of edges which presents information on the data dependencies between the blocks
20 Generate Def-Use Pair Compute the set of variables defined def B and the set of variables used in each block of the CFG use B Association between the block and variable of the program: def(b,v) holds, for a variable v and a vertex B, if B defines v use(b,v) holds, for a variable v and a vertex B, if B uses the value of v Generate the Def-Use pair information for each of the block in G(V,E) Add an edge from block B1 to block B2 that depicts the data flow of variable v given that def (B1,v) reaches use (B2,v) def (B1,v) reaches use (B2,v) when there is a definition clear path from B1 to B2 20
21 CFG with Def-Use Pair 21 val e1 = DataSource(..") val e2 = DataSource(..") var e3: DataSet[(String, Int, Int)] = null var i = 0 while(i < 0) { if (e1.map(x => x._2) > 50) e3 = e1.map { x => (x._1, x._ , x._3)} else e3 = e2.map { x => (x._1, x._ , x._3)} i = i + 1 } val e4 = e3.write( ) e4
22 Output: G(V,E,DFE) Adding Data Flow to the CFG 22
23 Control-Flow-Enriched Data Flow
24 Stage 3: Generate Code for Underlying System 24 Assumptions Code generated will run only for systems with a specified set of primitives that are currently supported by Stratosphere Transform each block in G(V,E,DFE) to a Stratosphere job Output: Stratosphere jobs to be executed in the WMS with order according to the dependencies defined in the IR
25 Code Generation Algorithm (1 of 2) Each incoming DFE to a block à Stratosphere job of that block requires the input of the data or variable contained in the DFE 25 Each outgoing DFE from a block à Stratosphere job of that block need to output the variable contained in the DFE WMS automatically selects which job to be run
26 Code Generation Algorithm (2 of 2) 26 J à sequence of Stratosphere job j(i,o) I à data source set of all input variables to the job O à data sink set of all output variables from the job
27 Use Case: Ingestion Process Evaluation: Productivity 27
28 Oozie vs Workflow DSL Implementation (1 of 2) Oozie Implementation Specify two XML definitions, for the main process and the subprocess. Each XML definition contains the action nodes and decision nodes based on the overall workflow The input and output directory of each subprocess is also defined manually in the XML definition. 28 A part of Oozie Implementation of SubDirectory Subprocess [Source: ]
29 Oozie vs Workflow DSL Implementation (2 of 2) 29 Workflow DSL Implementation Specify one workflow definition for both the main process and sub- process Intuitive à Ex: the fork node in the main process can be replaced by a general while style iteration Body of the iteration is the subprocess itself à the conditionals branching based on the directory information var temp = new Directories() var dirlist = temp.get var i: Int = 0 while (i < temp.getsize) { var dir = new DirInfo(dirList(i)) var dirage = dir.getage var dirsize = dir.getsize } if( if(dirage < 1) dirsize > 23 else dirsize > 0) { if(dirage > 6 dirsize > 23) { var ingest = ingestfile(dir.getname) var archive = archivefile(dir.getname) } else { var reminder = sendreminder(dir.getname) } } i = i+1
30 Evaluation: Generality High-level declarative interface which adheres only for Stratosphere at the moment 30 Deeply embedded in Scala - same syntax and semantics with some restrictions Possible to compile a program written in our DSL to other underlying platforms i.e. Spark can understand the general-style if statement and while statement supported by our DSL
31 Logistic Regression in Spark & Workflow DSL 31 Spark val data = spark.hdfstextfile(...).map(readpoint).cache() var w = Vector.random(D) for (i <- 1 to ITERATIONS) { var gradient = spark.accumulator(vector.zeros(d)) data.foreach(p => { val scale = (1/(1+exp(-p.y*(w dot p.x))) - 1) * p.y gradient += scale * p.x }) w -= gradient.value } Our workflow DSL val data = spark.hdfstextfile(...).map(readpoint).cache() var w = Vector.random(D) while(i < ITERATIONS) { w -= data.map(p => { val scale = (1/(1+exp(-p.y*(w dot p.x))) - 1) * p.y scale * p.x }).reduce(_+_) i = i + 1 } println("final w: " + w) println("final w: " + w) Source:
32 Conclusion Define a workflow DSL to enable the programmer to implement their algorithm Deeply embedded in Scala à avoids overhead for the programmer 32 Generate a control-flow-enriched data flow and target code from user program via static analysis of the program code Static analysis of Scala code detects the control flow and data dependencies Increase productivity compared to the implementation in other existing WMS (Oozie) Extensibility to be run on top of other frameworks
33 Future Work Extend grammar of our DSL i.e. For-comprehension 33 Extend our DSL to other frameworks Possible to generate the code or job scripts of the workflow for any execution framework Run program written in our DSL on multiple platforms
34 References 34 [Deelman et al, 2009] Ewa Deelman, Dennis Gannon, Matthew Shields, and Ian Taylor. Workflows and e- science: An overview of workflow system features and capabilities. Future Generation Computer Systems, 25(5): , [Hollingsworth et al, 1993] David Hollingsworth and UK Hampshire. Workflow management coalition the workflow reference model. Workflow Management Coalition, 68, [Ludäscher et al, 2005] Ludäscher Bertram, Ilkay Altintas, Chard Berkley, Dan Higgins, Efrat Jaeger, Matthew Jones, Edward A. Lee, Jing Tao, and Yang Zhao. Scientific Workflow Management and the Kepler System. Concurrency and Computation: Practice and Experience 18 no. 10, , [Yu et al, 2005] Jia Yu and Rajkumar Buyya. A taxonomy of workflow management systems for grid computing. Journal of Grid Computing, 3(3-4): , [Stocker, 2010] Mirko Stocker. Scala Refactoring. PhD thesis, HSR Hochschule für Technik Rapperswil, [Lam et al, 2006] Monica Lam, Ravi Sethi, JD Ullman, and Alfred Aho. Compilers: Principles, Techniques, and Tools. Addison-Wesley, [Kelly, 2011] Peter M Kelly. Applying functional programming theory to the design of work- flow engines
35 References 35 [Ackermann et al, 2012] Stefan Ackermann, Vojin Jovanovic, Tiark Rompf, and Martin Odersky. Jet: An embedded dsl for high performance big data processing. In International Workshop on End-to-end Management of Big Data (BigData 2012), number EPFL-CONF , [Alexandrov et al, 2014] Alexander Alexandrov, Rico Bergmann, Stephan Ewen, Johann-Christoph Frey- tag, Fabian Hueske, Arvid Heise, Odej Kao, Marcus Leich, Ulf Leser, Volker Markl, et al. The stratosphere platform for big data analytics. The VLDB Journal, pages 1 26, [Allen, 1970] Frances E Allen. Control flow analysis. In ACM Sigplan Notices, volume 5, pages ACM, [Ewen et al, 2012] Stephan Ewen, Kostas Tzoumas, Moritz Kaufmann, and Volker Markl. Spinning fast iterative data flows. Proceedings of the VLDB Endowment, 5(11): , [Burmako, 2013] Eugene Burmako. Scala macros: Let our powers combine!: On how rich syn- tax and static types work with metaprogramming. In Proceedings of the 4th Workshop on Scala, page [Islam et al, 2012] Mohammad Islam, Angelo K Huang, Mohamed Battisha, Michelle Chiang, San- thosh Srinivasan, Craig Peters, Andreas Neumann, and Alejandro Abdelnur. Oozie: towards a scalable workflow management system for hadoop. In Pro- ceedings of the 1st ACM SIGMOD Workshop on Scalable Workflow Execution Engines and Technologies, page 4. ACM, [Crobak, 2012]
Massive scale analytics with Stratosphere using R
Massive scale analytics with Stratosphere using R Jose Luis Lopez Pino jllopezpino@gmail.com Database Systems and Information Management Technische Universität Berlin Supervised by Volker Markl Advised
More informationSpark in Action. Fast Big Data Analytics using Scala. Matei Zaharia. www.spark- project.org. University of California, Berkeley UC BERKELEY
Spark in Action Fast Big Data Analytics using Scala Matei Zaharia University of California, Berkeley www.spark- project.org UC BERKELEY My Background Grad student in the AMP Lab at UC Berkeley» 50- person
More informationSpark. Fast, Interactive, Language- Integrated Cluster Computing
Spark Fast, Interactive, Language- Integrated Cluster Computing Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael Franklin, Scott Shenker, Ion Stoica UC
More informationSURVEY ON SCIENTIFIC DATA MANAGEMENT USING HADOOP MAPREDUCE IN THE KEPLER SCIENTIFIC WORKFLOW SYSTEM
SURVEY ON SCIENTIFIC DATA MANAGEMENT USING HADOOP MAPREDUCE IN THE KEPLER SCIENTIFIC WORKFLOW SYSTEM 1 KONG XIANGSHENG 1 Department of Computer & Information, Xinxiang University, Xinxiang, China E-mail:
More informationEarly Cloud Experiences with the Kepler Scientific Workflow System
Available online at www.sciencedirect.com Procedia Computer Science 9 (2012 ) 1630 1634 International Conference on Computational Science, ICCS 2012 Early Cloud Experiences with the Kepler Scientific Workflow
More informationCA4003 - Compiler Construction
CA4003 - Compiler Construction David Sinclair Overview This module will cover the compilation process, reading and parsing a structured language, storing it in an appropriate data structure, analysing
More informationSpark and Shark. High- Speed In- Memory Analytics over Hadoop and Hive Data
Spark and Shark High- Speed In- Memory Analytics over Hadoop and Hive Data Matei Zaharia, in collaboration with Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Cliff Engle, Michael Franklin, Haoyuan Li,
More informationThe Stratosphere Big Data Analytics Platform
The Stratosphere Big Data Analytics Platform Amir H. Payberah Swedish Institute of Computer Science amir@sics.se June 4, 2014 Amir H. Payberah (SICS) Stratosphere June 4, 2014 1 / 44 Big Data small data
More informationComparison of Distributed Data-Parallelization Patterns for Big Data Analysis: A Bioinformatics Case Study
Comparison of Distributed Data-Parallelization Patterns for Big Data Analysis: A Bioinformatics Case Study Jianwu Wang, Daniel Crawl, Ilkay Altintas San Diego Supercomputer Center University of California,
More informationBig Data looks Tiny from the Stratosphere
Volker Markl http://www.user.tu-berlin.de/marklv volker.markl@tu-berlin.de Big Data looks Tiny from the Stratosphere Data and analyses are becoming increasingly complex! Size Freshness Format/Media Type
More informationComparison of Distributed Data- Parallelization Patterns for Big Data Analysis: A Bioinformatics Case Study!
Comparison of Distributed Data- Parallelization Patterns for Big Data Analysis: A Bioinformatics Case Study! Jianwu Wang, Daniel Crawl, Ilkay Altintas! Kostas Tzoumas, Volker Markl! San Diego Supercomputer
More informationBig Data Research in Berlin BBDC and Apache Flink
Big Data Research in Berlin BBDC and Apache Flink Tilmann Rabl rabl@tu-berlin.de dima.tu-berlin.de bbdc.berlin 1 2013 Berlin Big Data Center All Rights Reserved DIMA 2015 Agenda About Data Management,
More informationhttp://glennengstrand.info/analytics/fp
Functional Programming and Big Data by Glenn Engstrand (September 2014) http://glennengstrand.info/analytics/fp What is Functional Programming? It is a style of programming that emphasizes immutable state,
More informationReport from the first workshop on Scalable Workflow Enactment Engines and Technology (SWEET 12)
Report from the first workshop on Scalable Workflow Enactment Engines and Technology (SWEET 12) Jan Hidders TUDelft, The Netherlands a.j.h.hidders@tudelft.nl Paolo Missier Newcastle University, UK paolo.missier@ncl.ac.uk
More informationSURVEY ON THE ALGORITHMS FOR WORKFLOW PLANNING AND EXECUTION
SURVEY ON THE ALGORITHMS FOR WORKFLOW PLANNING AND EXECUTION Kirandeep Kaur Khushdeep Kaur Research Scholar Assistant Professor, Department Of Cse, Bhai Maha Singh College Of Engineering, Bhai Maha Singh
More informationHIGH PERFORMANCE BIG DATA ANALYTICS
HIGH PERFORMANCE BIG DATA ANALYTICS Kunle Olukotun Electrical Engineering and Computer Science Stanford University June 2, 2014 Explosion of Data Sources Sensors DoD is swimming in sensors and drowning
More informationSpark: Making Big Data Interactive & Real-Time
Spark: Making Big Data Interactive & Real-Time Matei Zaharia UC Berkeley / MIT www.spark-project.org What is Spark? Fast and expressive cluster computing system compatible with Apache Hadoop Improves efficiency
More informationSpark: Cluster Computing with Working Sets
Spark: Cluster Computing with Working Sets Outline Why? Mesos Resilient Distributed Dataset Spark & Scala Examples Uses Why? MapReduce deficiencies: Standard Dataflows are Acyclic Prevents Iterative Jobs
More informationA Multi-layered Domain-specific Language for Stencil Computations
A Multi-layered Domain-specific Language for Stencil Computations Christian Schmitt, Frank Hannig, Jürgen Teich Hardware/Software Co-Design, University of Erlangen-Nuremberg Workshop ExaStencils 2014,
More informationBig Data Analytics. Chances and Challenges. Volker Markl
Volker Markl Professor and Chair Database Systems and Information Management (DIMA), Technische Universität Berlin www.dima.tu-berlin.de Big Data Analytics Chances and Challenges Volker Markl DIMA BDOD
More informationSystems Engineering II. Pramod Bhatotia TU Dresden pramod.bhatotia@tu- dresden.de
Systems Engineering II Pramod Bhatotia TU Dresden pramod.bhatotia@tu- dresden.de About me! Since May 2015 2015 2012 Research Group Leader cfaed, TU Dresden PhD Student MPI- SWS Research Intern Microsoft
More informationLanguage-Driven, Technology-Enhanced Instructional Systems Design
Language-Driven, Technology-Enhanced Instructional s Design Iván Martínez-Ortiz, José-Luis Sierra, Baltasar Fernández-Manjón Fac. Informática. Universidad Complutense de Madrid C/ Prof. José García Santesmases
More informationAnalysis Pipelines for Benchmarking Big Data Systems
Analysis Pipelines for Benchmarking Big Data Systems Thomas Bodner thomas.o.bodner@campus.tu-berlin.de Ref. Code: Berlin_EN_2182, Suggested starting date: May 15, 2013 Today, practically everyone ranging
More informationPerformance Optimization Techniques and Tools for Data-Intensive Computation Platforms
Performance Optimization Techniques and Tools for Data-Intensive Computation Platforms Licentiate Seminar Presentation June 11, KTH Kista, Sweden Vasiliki Kalavri kalavri@kth.se Outline 1. Introduction
More informationn Introduction n Art of programming language design n Programming language spectrum n Why study programming languages? n Overview of compilation
Lecture Outline Programming Languages CSCI-4430 & CSCI-6430, Spring 2016 www.cs.rpi.edu/~milanova/csci4430/ Ana Milanova Lally Hall 314, 518 276-6887 milanova@cs.rpi.edu Office hours: Wednesdays Noon-2pm
More informationAdvanced compiler construction. General course information. Teacher & assistant. Course goals. Evaluation. Grading scheme. Michel Schinz 2007 03 16
Advanced compiler construction Michel Schinz 2007 03 16 General course information Teacher & assistant Course goals Teacher: Michel Schinz Michel.Schinz@epfl.ch Assistant: Iulian Dragos INR 321, 368 64
More informationBuilding a real-time, self-service data analytics ecosystem Greg Arnold, Sr. Director Engineering
Building a real-time, self-service data analytics ecosystem Greg Arnold, Sr. Director Engineering Self Service at scale 6 5 4 3 2 1 ? Relational? MPP? Hadoop? Linkedin data 350M Members 25B 3.5M 4.8B 2M
More informationA Brief Introduction to Apache Tez
A Brief Introduction to Apache Tez Introduction It is a fact that data is basically the new currency of the modern business world. Companies that effectively maximize the value of their data (extract value
More informationOntology construction on a cloud computing platform
Ontology construction on a cloud computing platform Exposé for a Bachelor's thesis in Computer science - Knowledge management in bioinformatics Tobias Heintz 1 Motivation 1.1 Introduction PhenomicDB is
More informationA Framework for Distributed Data-Parallel Execution in the Kepler Scientific Workflow System
Available online at www.sciencedirect.com Procedia Computer Science 9 (2012 ) 1620 1629 International Conference on Computational Science, ICCS 2012 A Framework for Distributed Data-Parallel Execution
More informationData processing goes big
Test report: Integration Big Data Edition Data processing goes big Dr. Götz Güttich Integration is a powerful set of tools to access, transform, move and synchronize data. With more than 450 connectors,
More informationUsing Eclipse CDT/PTP for Static Analysis
PTP User-Developer Workshop Sept 18-20, 2012 Using Eclipse CDT/PTP for Static Analysis Beth R. Tibbitts IBM STG tibbitts@us.ibm.com "This material is based upon work supported by the Defense Advanced Research
More informationLDIF - Linked Data Integration Framework
LDIF - Linked Data Integration Framework Andreas Schultz 1, Andrea Matteini 2, Robert Isele 1, Christian Bizer 1, and Christian Becker 2 1. Web-based Systems Group, Freie Universität Berlin, Germany a.schultz@fu-berlin.de,
More informationMap Reduce Workflows
2012 coreservlets.com and Dima May Map Reduce Workflows Originals of slides and source code for examples: http://www.coreservlets.com/hadoop-tutorial/ Also see the customized Hadoop training courses (onsite
More informationProgramming Hadoop 5-day, instructor-led BD-106. MapReduce Overview. Hadoop Overview
Programming Hadoop 5-day, instructor-led BD-106 MapReduce Overview The Client Server Processing Pattern Distributed Computing Challenges MapReduce Defined Google's MapReduce The Map Phase of MapReduce
More informationDeveloping Scalable Smart Grid Infrastructure to Enable Secure Transmission System Control
Developing Scalable Smart Grid Infrastructure to Enable Secure Transmission System Control EP/K006487/1 UK PI: Prof Gareth Taylor (BU) China PI: Prof Yong-Hua Song (THU) Consortium UK Members: Brunel University
More informationDetection of DOM-based Cross-Site Scripting by Analyzing Dynamically Extracted Scripts
Detection of DOM-based Cross-Site Scripting by Analyzing Dynamically Extracted Scripts Suman Saha 1, Shizhen Jin 2,3 and Kyung-Goo Doh 3 1 LIP6-Regal, France Suman.Saha@lip6.fr 2 GTOne, Seoul, Korea jinshzh@gmail.com
More informationA Scala DSL for Rete-based Runtime Verification
A Scala DSL for Rete-based Runtime Verification Klaus Havelund Jet Propulsion Laboratory California Institute of Technology, California, USA Abstract. Runtime verification (RV) consists in part of checking
More informationE6893 Big Data Analytics Lecture 2: Big Data Analytics Platforms
E6893 Big Data Analytics Lecture 2: Big Data Analytics Platforms Ching-Yung Lin, Ph.D. Adjunct Professor, Dept. of Electrical Engineering and Computer Science Mgr., Dept. of Network Science and Big Data
More informationApache Flink Next-gen data analysis. Kostas Tzoumas ktzoumas@apache.org @kostas_tzoumas
Apache Flink Next-gen data analysis Kostas Tzoumas ktzoumas@apache.org @kostas_tzoumas What is Flink Project undergoing incubation in the Apache Software Foundation Originating from the Stratosphere research
More information1/20/2016 INTRODUCTION
INTRODUCTION 1 Programming languages have common concepts that are seen in all languages This course will discuss and illustrate these common concepts: Syntax Names Types Semantics Memory Management We
More informationHadoop Ecosystem Overview. CMSC 491 Hadoop-Based Distributed Computing Spring 2015 Adam Shook
Hadoop Ecosystem Overview CMSC 491 Hadoop-Based Distributed Computing Spring 2015 Adam Shook Agenda Introduce Hadoop projects to prepare you for your group work Intimate detail will be provided in future
More informationIntroduction. Compiler Design CSE 504. Overview. Programming problems are easier to solve in high-level languages
Introduction Compiler esign CSE 504 1 Overview 2 3 Phases of Translation ast modifled: Mon Jan 28 2013 at 17:19:57 EST Version: 1.5 23:45:54 2013/01/28 Compiled at 11:48 on 2015/01/28 Compiler esign Introduction
More informationLanguage Processing Systems
Language Processing Systems Evaluation Active sheets 10 % Exercise reports 30 % Midterm Exam 20 % Final Exam 40 % Contact Send e-mail to hamada@u-aizu.ac.jp Course materials at www.u-aizu.ac.jp/~hamada/education.html
More informationDistributed Aggregation in Cloud Databases. By: Aparna Tiwari tiwaria@umail.iu.edu
Distributed Aggregation in Cloud Databases By: Aparna Tiwari tiwaria@umail.iu.edu ABSTRACT Data intensive applications rely heavily on aggregation functions for extraction of data according to user requirements.
More informationChapter 1. Dr. Chris Irwin Davis Email: cid021000@utdallas.edu Phone: (972) 883-3574 Office: ECSS 4.705. CS-4337 Organization of Programming Languages
Chapter 1 CS-4337 Organization of Programming Languages Dr. Chris Irwin Davis Email: cid021000@utdallas.edu Phone: (972) 883-3574 Office: ECSS 4.705 Chapter 1 Topics Reasons for Studying Concepts of Programming
More informationApache MRQL (incubating): Advanced Query Processing for Complex, Large-Scale Data Analysis
Apache MRQL (incubating): Advanced Query Processing for Complex, Large-Scale Data Analysis Leonidas Fegaras University of Texas at Arlington http://mrql.incubator.apache.org/ 04/12/2015 Outline Who am
More informationSCALABLE GRAPH ANALYTICS WITH GRADOOP AND BIIIG
SCALABLE GRAPH ANALYTICS WITH GRADOOP AND BIIIG MARTIN JUNGHANNS, ANDRE PETERMANN, ERHARD RAHM www.scads.de RESEARCH ON GRAPH ANALYTICS Graph Analytics on Hadoop (Gradoop) Distributed graph data management
More informationDatabase Application Developer Tools Using Static Analysis and Dynamic Profiling
Database Application Developer Tools Using Static Analysis and Dynamic Profiling Surajit Chaudhuri, Vivek Narasayya, Manoj Syamala Microsoft Research {surajitc,viveknar,manojsy}@microsoft.com Abstract
More informationAn Industrial Perspective on the Hadoop Ecosystem. Eldar Khalilov Pavel Valov
An Industrial Perspective on the Hadoop Ecosystem Eldar Khalilov Pavel Valov agenda 03.12.2015 2 agenda Introduction 03.12.2015 2 agenda Introduction Research goals 03.12.2015 2 agenda Introduction Research
More informationThe Stratosphere platform for big data analytics
The VLDB Journal DOI 10.1007/s00778-014-0357-y REGULAR PAPER The Stratosphere platform for big data analytics Alexander Alexandrov Rico Bergmann Stephan Ewen Johann-Christoph Freytag Fabian Hueske Arvid
More informationAUTOMATED TEST GENERATION FOR SOFTWARE COMPONENTS
TKK Reports in Information and Computer Science Espoo 2009 TKK-ICS-R26 AUTOMATED TEST GENERATION FOR SOFTWARE COMPONENTS Kari Kähkönen ABTEKNILLINEN KORKEAKOULU TEKNISKA HÖGSKOLAN HELSINKI UNIVERSITY OF
More informationSpark and Shark: High-speed In-memory Analytics over Hadoop Data
Spark and Shark: High-speed In-memory Analytics over Hadoop Data May 14, 2013 @ Oracle Reynold Xin, AMPLab, UC Berkeley The Big Data Problem Data is growing faster than computation speeds Accelerating
More informationData Governance in the Hadoop Data Lake. Michael Lang May 2015
Data Governance in the Hadoop Data Lake Michael Lang May 2015 Introduction Product Manager for Teradata Loom Joined Teradata as part of acquisition of Revelytix, original developer of Loom VP of Sales
More informationSpark Application Carousel. Spark Summit East 2015
Spark Application Carousel Spark Summit East 2015 About Today s Talk About Me: Vida Ha - Solutions Engineer at Databricks. Goal: For beginning/early intermediate Spark Developers. Motivate you to start
More informationSupporting Software Development Process Using Evolution Analysis : a Brief Survey
Supporting Software Development Process Using Evolution Analysis : a Brief Survey Samaneh Bayat Department of Computing Science, University of Alberta, Edmonton, Canada samaneh@ualberta.ca Abstract During
More informationApache Mahout's new DSL for Distributed Machine Learning. Sebastian Schelter GOTO Berlin 11/06/2014
Apache Mahout's new DSL for Distributed Machine Learning Sebastian Schelter GOO Berlin /6/24 Overview Apache Mahout: Past & Future A DSL for Machine Learning Example Under the covers Distributed computation
More informationClassification of Natural Language Interfaces to Databases based on the Architectures
Volume 1, No. 11, ISSN 2278-1080 The International Journal of Computer Science & Applications (TIJCSA) RESEARCH PAPER Available Online at http://www.journalofcomputerscience.com/ Classification of Natural
More informationStratoSphere Above the Clouds
Stratosphere Parallel Analytics in the Cloud beyond Map/Reduce 14th International Workshop on High Performance Transaction Systems (HPTS) Poster Sessions, Mon Oct 24 2011 Thomas Bodner T.U. Berlin StratoSphere
More informationOptimizations. Optimization Safety. Optimization Safety. Control Flow Graphs. Code transformations to improve program
Optimizations Code transformations to improve program Mainly: improve execution time Also: reduce program size Control low Graphs Can be done at high level or low level E.g., constant folding Optimizations
More informationWorkshop on Hadoop with Big Data
Workshop on Hadoop with Big Data Hadoop? Apache Hadoop is an open source framework for distributed storage and processing of large sets of data on commodity hardware. Hadoop enables businesses to quickly
More informationHYBRID WORKFLOW POLICY MANAGEMENT FOR HEART DISEASE IDENTIFICATION DONG-HYUN KIM *1, WOO-RAM JUNG 1, CHAN-HYUN YOUN 1
HYBRID WORKFLOW POLICY MANAGEMENT FOR HEART DISEASE IDENTIFICATION DONG-HYUN KIM *1, WOO-RAM JUNG 1, CHAN-HYUN YOUN 1 1 Department of Information and Communications Engineering, Korea Advanced Institute
More informationExpanding the CASEsim Framework to Facilitate Load Balancing of Social Network Simulations
Expanding the CASEsim Framework to Facilitate Load Balancing of Social Network Simulations Amara Keller, Martin Kelly, Aaron Todd 4 June 2010 Abstract This research has two components, both involving the
More informationWHITE PAPER. Peter Drucker. intentsoft.com 2014, Intentional Software Corporation
We know now that the source of wealth is something specifically human: knowledge. If we apply knowledge to tasks we already know how to do, we call it productivity. If we apply knowledge to tasks that
More informationBig Data Open Source Stack vs. Traditional Stack for BI and Analytics
Big Data Open Source Stack vs. Traditional Stack for BI and Analytics Part I By Sam Poozhikala, Vice President Customer Solutions at StratApps Inc. 4/4/2014 You may contact Sam Poozhikala at spoozhikala@stratapps.com.
More informationSan Diego Supercomputer Center, UCSD. Institute for Digital Research and Education, UCLA
Facilitate Parallel Computation Using Kepler Workflow System on Virtual Resources Jianwu Wang 1, Prakashan Korambath 2, Ilkay Altintas 1 1 San Diego Supercomputer Center, UCSD 2 Institute for Digital Research
More informationComponent visualization methods for large legacy software in C/C++
Annales Mathematicae et Informaticae 44 (2015) pp. 23 33 http://ami.ektf.hu Component visualization methods for large legacy software in C/C++ Máté Cserép a, Dániel Krupp b a Eötvös Loránd University mcserep@caesar.elte.hu
More informationPHP FRAMEWORK FOR DATABASE MANAGEMENT BASED ON MVC PATTERN
PHP FRAMEWORK FOR DATABASE MANAGEMENT BASED ON MVC PATTERN Chanchai Supaartagorn Department of Mathematics Statistics and Computer, Faculty of Science, Ubon Ratchathani University, Thailand scchansu@ubu.ac.th
More informationThe Big Data Ecosystem at LinkedIn Roshan Sumbaly, Jay Kreps, and Sam Shah LinkedIn
The Big Data Ecosystem at LinkedIn Roshan Sumbaly, Jay Kreps, and Sam Shah LinkedIn Presented by :- Ishank Kumar Aakash Patel Vishnu Dev Yadav CONTENT Abstract Introduction Related work The Ecosystem Ingress
More informationHadoop Ecosystem B Y R A H I M A.
Hadoop Ecosystem B Y R A H I M A. History of Hadoop Hadoop was created by Doug Cutting, the creator of Apache Lucene, the widely used text search library. Hadoop has its origins in Apache Nutch, an open
More informationSecuring PHP Based Web Application Using Vulnerability Injection
International Journal of Information and Computation Technology. ISSN 0974-2239 Volume 3, Number 5 (2013), pp. 391-398 International Research Publications House http://www. irphouse.com /ijict.htm Securing
More informationHADOOP ADMINISTATION AND DEVELOPMENT TRAINING CURRICULUM
HADOOP ADMINISTATION AND DEVELOPMENT TRAINING CURRICULUM 1. Introduction 1.1 Big Data Introduction What is Big Data Data Analytics Bigdata Challenges Technologies supported by big data 1.2 Hadoop Introduction
More informationApache Flink. Fast and Reliable Large-Scale Data Processing
Apache Flink Fast and Reliable Large-Scale Data Processing Fabian Hueske @fhueske 1 What is Apache Flink? Distributed Data Flow Processing System Focused on large-scale data analytics Real-time stream
More informationBig Data for Investment Research Management
IDT Partners www.idtpartners.com Big Data for Investment Research Management Discover how IDT Partners helps Financial Services, Market Research, and Investment Management firms turn big data into actionable
More informationLog Mining Based on Hadoop s Map and Reduce Technique
Log Mining Based on Hadoop s Map and Reduce Technique ABSTRACT: Anuja Pandit Department of Computer Science, anujapandit25@gmail.com Amruta Deshpande Department of Computer Science, amrutadeshpande1991@gmail.com
More informationWhat is Analytic Infrastructure and Why Should You Care?
What is Analytic Infrastructure and Why Should You Care? Robert L Grossman University of Illinois at Chicago and Open Data Group grossman@uic.edu ABSTRACT We define analytic infrastructure to be the services,
More informationDice. David Watkins Emily Chen Khaled Atef Phillip Schiffrin. djw2146 ec2805 kaa2168 pjs2186. Manager System Architect Testing Language Guru
Dice David Watkins Emily Chen Khaled Atef Phillip Schiffrin djw2146 ec2805 kaa2168 pjs2186 Manager System Architect Testing Language Guru September 30 th, 2015 1 DESCRIPTION Dice is a distributed systems
More informationKepler + Hadoop : A General Architecture Facilitating Data-Intensive Applications in Scientific Workflow Systems
Kepler + Hadoop : A General Architecture Facilitating Data-Intensive Applications in Scientific Workflow Systems Jianwu Wang, Daniel Crawl, Ilkay Altintas San Diego Supercomputer Center, University of
More informationScoping (Readings 7.1,7.4,7.6) Parameter passing methods (7.5) Building symbol tables (7.6)
Semantic Analysis Scoping (Readings 7.1,7.4,7.6) Static Dynamic Parameter passing methods (7.5) Building symbol tables (7.6) How to use them to find multiply-declared and undeclared variables Type checking
More informationCase Study : 3 different hadoop cluster deployments
Case Study : 3 different hadoop cluster deployments Lee moon soo moon@nflabs.com HDFS as a Storage Last 4 years, our HDFS clusters, stored Customer 1500 TB+ data safely served 375,000 TB+ data to customer
More informationTechnical paper review. Program visualization and explanation for novice C programmers by Matthew Heinsen Egan and Chris McDonald.
Technical paper review Program visualization and explanation for novice C programmers by Matthew Heinsen Egan and Chris McDonald Garvit Pahal Indian Institute of Technology, Kanpur October 28, 2014 Garvit
More informationMoving From Hadoop to Spark
+ Moving From Hadoop to Spark Sujee Maniyam Founder / Principal @ www.elephantscale.com sujee@elephantscale.com Bay Area ACM meetup (2015-02-23) + HI, Featured in Hadoop Weekly #109 + About Me : Sujee
More informationSemester Review. CSC 301, Fall 2015
Semester Review CSC 301, Fall 2015 Programming Language Classes There are many different programming language classes, but four classes or paradigms stand out:! Imperative Languages! assignment and iteration!
More informationVoice Driven Animation System
Voice Driven Animation System Zhijin Wang Department of Computer Science University of British Columbia Abstract The goal of this term project is to develop a voice driven animation system that could take
More informationHow To Optimize Data Processing In A Distributed System
Performance Optimization Techniques and Tools for Data-Intensive Computation Platforms An Overview of Performance Limitations in Big Data Systems and Proposed Optimizations VASILIKI KALAVRI Licentiate
More informationBig Data for the JVM developer. Costin Leau, Elasticsearch @costinl
Big Data for the JVM developer Costin Leau, Elasticsearch @costinl Agenda Data Trends Data Pipelines JVM and Big Data Tool Eco-system Data Landscape Data Trends http://www.emc.com/leadership/programs/digital-universe.htm
More informationCS555: Distributed Systems [Fall 2015] Dept. Of Computer Science, Colorado State University
CS 555: DISTRIBUTED SYSTEMS [SPARK] Shrideep Pallickara Computer Science Colorado State University Frequently asked questions from the previous class survey Streaming Significance of minimum delays? Interleaving
More informationSpark and the Big Data Library
Spark and the Big Data Library Reza Zadeh Thanks to Matei Zaharia Problem Data growing faster than processing speeds Only solution is to parallelize on large clusters» Wide use in both enterprises and
More informationBig Data Analytics Platform @ Nokia
Big Data Analytics Platform @ Nokia 1 Selecting the Right Tool for the Right Workload Yekesa Kosuru Nokia Location & Commerce Strata + Hadoop World NY - Oct 25, 2012 Agenda Big Data Analytics Platform
More informationAN AI PLANNING APPROACH FOR GENERATING BIG DATA WORKFLOWS
AN AI PLANNING APPROACH FOR GENERATING BIG DATA WORKFLOWS Wesley Deneke 1, Wing-Ning Li 2, and Craig Thompson 2 1 Computer Science and Industrial Technology Department, Southeastern Louisiana University,
More informationBig Data Analytics with Spark and Oscar BAO. Tamas Jambor, Lead Data Scientist at Massive Analytic
Big Data Analytics with Spark and Oscar BAO Tamas Jambor, Lead Data Scientist at Massive Analytic About me Building a scalable Machine Learning platform at MA Worked in Big Data and Data Science in the
More informationConjugating data mood and tenses: Simple past, infinite present, fast continuous, simpler imperative, conditional future perfect
Matteo Migliavacca (mm53@kent) School of Computing Conjugating data mood and tenses: Simple past, infinite present, fast continuous, simpler imperative, conditional future perfect Simple past - Traditional
More informationA visual DSL toolkit in Lua
A visual DSL toolkit in Lua Past, present and future Alexander Gladysh Lua Workshop 2013 Toulouse 1 / 44 Outline Introduction The Problem Classic third-party alternatives Past generations
More informationShark Installation Guide Week 3 Report. Ankush Arora
Shark Installation Guide Week 3 Report Ankush Arora Last Updated: May 31,2014 CONTENTS Contents 1 Introduction 1 1.1 Shark..................................... 1 1.2 Apache Spark.................................
More informationBigData. An Overview of Several Approaches. David Mera 16/12/2013. Masaryk University Brno, Czech Republic
BigData An Overview of Several Approaches David Mera Masaryk University Brno, Czech Republic 16/12/2013 Table of Contents 1 Introduction 2 Terminology 3 Approaches focused on batch data processing MapReduce-Hadoop
More informationDeploying Hadoop with Manager
Deploying Hadoop with Manager SUSE Big Data Made Easier Peter Linnell / Sales Engineer plinnell@suse.com Alejandro Bonilla / Sales Engineer abonilla@suse.com 2 Hadoop Core Components 3 Typical Hadoop Distribution
More informationSemantic Workflows and the Wings Workflow System
To Appear in AAAI Fall Symposium on Proactive Assistant Agents, Arlington, VA, November 2010. Assisting Scientists with Complex Data Analysis Tasks through Semantic Workflows Yolanda Gil, Varun Ratnakar,
More informationThe Internet of Things and Big Data: Intro
The Internet of Things and Big Data: Intro John Berns, Solutions Architect, APAC - MapR Technologies April 22 nd, 2014 1 What This Is; What This Is Not It s not specific to IoT It s not about any specific
More informationData Science in the Wild
Data Science in the Wild Lecture 4 59 Apache Spark 60 1 What is Spark? Not a modified version of Hadoop Separate, fast, MapReduce-like engine In-memory data storage for very fast iterative queries General
More informationGSiB: PSE Infrastructure for Dynamic Service-oriented Grid Applications
GSiB: PSE Infrastructure for Dynamic Service-oriented Grid Applications Yan Huang Department of Computer Science Cardiff University PO Box 916 Cardiff CF24 3XF United Kingdom Yan.Huang@cs.cardiff.ac.uk
More information