Conquering Big Data with Apache Spark
|
|
|
- Rolf Blankenship
- 7 years ago
- Views:
Transcription
1 Conquering Big Data with Apache Spark Ion Stoica November 1 st, 2015 UC BERKELEY
2 The Berkeley AMPLab January faculty > 50 students 3 software engineer team Organized for collaboration achines lgorithms eople AMPCamp (since 2012) 3 day retreats (twice a year) 400+ campers (100s companies)
3 The Berkeley AMPLab Governmental and industrial funding: Goal: Next generation of open source data analytics stack for industry & academia: Berkeley Data Analytics Stack (BDAS)
4 Generic Big Data Stack Processing Layer Resource Management Layer Storage Layer
5 Hadoop Stack Hive Pig HadoopMR Storm Impala Processing Layer Giraph Processing Yarn Resource Management Layer Res. Mgmnt Storage HDFS Layer Storage
6 BDAS Stack Spark Streaming BlinkDB SparkSQL Sample Clean SparkR Processing Layer Spark Core GraphX Mesos Resource Management Hadoop Layer Yarn Succinct Tachyon BDAS Stack Storage Layer 3 rd party MLBase MLlib Velox Velox HDFS, S3, Ceph, Processing Res. Mgmnt Storage
7 Today s Talk Spark Streaming BlinkDB SparkSQL Sample Clean SparkR Spark Core GraphX Mesos Resource Management Hadoop Layer Yarn Succinct Tachyon BDAS Stack Storage Layer 3 rd party MLBase MLlib Velox Velox HDFS, S3, Ceph, Today s Talk Processing Res. Mgmnt Storage
8 Overview 1. Introduction 2. RDDs 3. Generality of RDDs (e.g. streaming) 4. DataFrames 5. Project Tungsten
9 Overview 1. Introduction 2. RDDs 3. Generality of RDDs (e.g. streaming) 4. DataFrames 5. Project Tungsten
10 A Short History Started at UC Berkeley in 2009 Open Source: 2010 Apache Project: 2013 Today: most popular big data project
11 What Is Spark? Parallel execution engine for big data processing Easy to use: 2-5x less code than Hadoop MR High level API s in Python, Java, and Scala Fast: up to 100x faster than Hadoop MR Can exploit in-memory when available Low overhead scheduling, optimized engine General: support multiple computation models
12 Analogy First cellular phones Specialized devices Unified device (smartphone)
13 Analogy Better Phone First cellular phones Better Games Better GPS Specialized devices Unified device (smartphone)
14 Analogy Batch processing Specialized systems Unified system
15 Analogy Real-time analytics Instant fraud detection Better Apps Batch processing Specialized systems Unified system
16 General Unifies batch, interactive comp. SparkSQL Spark Core
17 General Unifies batch, interactive, streaming comp. SparkSQL Spark Streaming Spark Core
18 General Unifies batch, interactive, streaming comp. Easy to build sophisticated applications Support iterative, graph-parallel algorithms Powerful APIs in Scala, Python, Java, R Spark SparkSQL MLlib GraphX SparkR Streaming Spark Core
19 Easy to Write Code WordCount in 3 lines of Spark WordCount in 50+ lines of Java MR
20 Fast: Time to sort 100TB 2013 Record: Hadoop 2100 machines 72 minutes 2014 Record: Spark 207 machines 23 minutes Also sorted 1PB in 4 hours Source: Daytona GraySort benchmark, sortbenchmark.org
21 Community Growth June 2014 June 2015 total contributors contributors/month lines of code 175, ,000
22 Meetup Groups: January 2015 source: meetup.com
23 Meetup Groups: October 2015 source: meetup.com
24 Community Growth Summit Attendees Meetup Members Developers Contributing K K
25 Large-Scale Usage Largest cluster: 8000 nodes Largest single job: 1 petabyte Top streaming intake: 1 TB/hour 2014 on-disk sort record
26 Spark Ecosystem Distributions Applications
27 Overview 1. Introduction 2. RDDs 3. Generality of RDDs (e.g. streaming) 4. DataFrames 5. Project Tungsten
28 RDD: Resilient Distributed Datasets Collections of objects distr. across a cluster Stored in RAM or on Disk Automatically rebuilt on failure Operations Transformations Actions Execution model: similar to SIMD
29 Operations on RDDs Transformations f(rdd) => RDD Lazy (not computed immediately) E.g., map, filter, groupby Actions: Triggers computation E.g. count, collect, saveastextfile
30 Example: Log Mining Load error messages from a log into memory, then interactively search for various patterns
31 Example: Log Mining Load error messages from a log into memory, then interactively search for various patterns Driver
32 Example: Log Mining Load error messages from a log into memory, then interactively search for various patterns lines = spark.textfile( hdfs://... ) Driver
33 Example: Log Mining Load error messages from a log into memory, then interactively search for various patterns Base RDD lines = spark.textfile( hdfs://... ) Driver
34 Example: Log Mining Load error messages from a log into memory, then interactively search for various patterns lines = spark.textfile( hdfs://... ) errors = lines.filter(lambda s: s.startswith( ERROR )) Driver
35 Example: Log Mining Load error messages from a log into memory, then interactively search for various patterns Transformed RDD lines = spark.textfile( hdfs://... ) errors = lines.filter(lambda s: s.startswith( ERROR )) Driver
36 Example: Log Mining Load error messages from a log into memory, then interactively search for various patterns lines = spark.textfile( hdfs://... ) errors = lines.filter(lambda s: s.startswith( ERROR )) messages = errors.map(lambda s: s.split( \t )[2]) messages.cache() Driver messages.filter(lambda s: mysql in s).count()
37 Example: Log Mining Load error messages from a log into memory, then interactively search for various patterns lines = spark.textfile( hdfs://... ) errors = lines.filter(lambda s: s.startswith( ERROR )) messages = errors.map(lambda s: s.split( \t )[2]) messages.cache() messages.filter(lambda s: mysql in s).count() Driver Ac5on
38 Example: Log Mining Load error messages from a log into memory, then interactively search for various patterns lines = spark.textfile( hdfs://... ) errors = lines.filter(lambda s: s.startswith( ERROR )) messages = errors.map(lambda s: s.split( \t )[2]) messages.cache() Driver Block 1 messages.filter(lambda s: mysql in s).count() Block 2 Block 3
39 Example: Log Mining Load error messages from a log into memory, then interactively search for various patterns lines = spark.textfile( hdfs://... ) errors = lines.filter(lambda s: s.startswith( ERROR )) messages = errors.map(lambda s: s.split( \t )[2]) messages.cache() messages.filter(lambda s: mysql in s).count() Driver tasks tasks Block 1 tasks Block 2 Block 3
40 Example: Log Mining Load error messages from a log into memory, then interactively search for various patterns lines = spark.textfile( hdfs://... ) errors = lines.filter(lambda s: s.startswith( ERROR )) messages = errors.map(lambda s: s.split( \t )[2]) messages.cache() messages.filter(lambda s: mysql in s).count() Block 3 Driver Read HDFS Block Block 1 Read HDFS Block Block 2 Read HDFS Block
41 Example: Log Mining Load error messages from a log into memory, then interactively search for various patterns lines = spark.textfile( hdfs://... ) errors = lines.filter(lambda s: s.startswith( ERROR )) messages = errors.map(lambda s: s.split( \t )[2]) messages.cache() messages.filter(lambda s: mysql in s).count() Block 3 Driver Cache 3 Process & Cache Data Cache 1 Block 1 Process & Cache Data Cache 2 Block 2 Process & Cache Data
42 Example: Log Mining Load error messages from a log into memory, then interactively search for various patterns lines = spark.textfile( hdfs://... ) errors = lines.filter(lambda s: s.startswith( ERROR )) messages = errors.map(lambda s: s.split( \t )[2]) messages.cache() messages.filter(lambda s: mysql in s).count() Block 3 Driver results Cache 3 results results Cache 1 Block 1 Cache 2 Block 2
43 Example: Log Mining Load error messages from a log into memory, then interactively search for various patterns lines = spark.textfile( hdfs://... ) errors = lines.filter(lambda s: s.startswith( ERROR )) messages = errors.map(lambda s: s.split( \t )[2]) messages.cache() Driver Cache 1 Block 1 messages.filter(lambda s: mysql in s).count() messages.filter(lambda s: php in s).count() Block 3 Cache 3 Cache 2 Block 2
44 Example: Log Mining Load error messages from a log into memory, then interactively search for various patterns lines = spark.textfile( hdfs://... ) errors = lines.filter(lambda s: s.startswith( ERROR )) messages = errors.map(lambda s: s.split( \t )[2]) messages.cache() messages.filter(lambda s: mysql in s).count() messages.filter(lambda s: php in s).count() Driver Block 3 tasks Cache 3 tasks tasks Cache 1 Block 1 Cache 2 Block 2
45 Example: Log Mining Load error messages from a log into memory, then interactively search for various patterns lines = spark.textfile( hdfs://... ) errors = lines.filter(lambda s: s.startswith( ERROR )) messages = errors.map(lambda s: s.split( \t )[2]) messages.cache() messages.filter(lambda s: mysql in s).count() messages.filter(lambda s: php in s).count() Block 3 Driver Cache 3 Process from Cache Cache 1 Block 1 Process from Cache Cache 2 Block 2 Process from Cache
46 Example: Log Mining Load error messages from a log into memory, then interactively search for various patterns lines = spark.textfile( hdfs://... ) errors = lines.filter(lambda s: s.startswith( ERROR )) messages = errors.map(lambda s: s.split( \t )[2]) messages.cache() messages.filter(lambda s: mysql in s).count() messages.filter(lambda s: php in s).count() Block 3 Driver results Cache 3 results Cache 1 results Block 1 Cache 2 Block 2
47 Example: Log Mining Load error messages from a log into memory, then interactively search for various patterns lines = spark.textfile( hdfs://... ) errors = lines.filter(lambda s: s.startswith( ERROR )) messages = errors.map(lambda s: s.split( \t )[2]) messages.cache() Driver Cache 1 Block 1 messages.filter(lambda s: mysql in s).count() messages.filter(lambda s: php in s).count() Cache your data è Faster Results Full-text search of Wikipedia 60GB on 20 EC2 machines 0.5 sec from mem vs. 20s for on-disk Block 3 Cache 3 Cache 2 Block 2
48 Language Support Python lines = sc.textfile(...) lines.filter(lambda s: ERROR in s).count() Scala val lines = sc.textfile(...) lines.filter(x => x.contains( ERROR )).count() Java JavaRDD<String> lines = sc.textfile(...); lines.filter(new Function<String, Boolean>() { Boolean call(string s) { return s.contains( error ); } }).count(); Standalone Programs Python, Scala, & Java Interactive Shells Python & Scala Performance Java & Scala are faster due to static typing but Python is often fine
49 Expressive API map reduce
50 Expressive API map filter groupby sort union join leftouterjoin rightouterjoin reduce count fold reducebykey groupbykey cogroup cross zip sample take first partitionby mapwith pipe save...
51 Fault Recovery: Design Alternatives Replication: Slow: need to write data over network Memory inefficient Backup on persistent storage Persistent storage still (much) slower than memory Still need to go over network to protect against machine failures Spark choice: Lineage: track sequence of operations to efficiently reconstruct lost RRD partitions
52 Fault Recovery Example Two-partition RDD A={A 1, A 2 } stored on disk 1) filter and cache à RDD B 2) joinà RDD C 3) aggregate à RDD D RDD A A 1 filter B 1 join C 1 agg. D A 2 filter B 2 join C 2
53 Fault Recovery Example C 1 lost due to node failure before reduce finishes RDD A A 1 filter B 1 join C 1 agg. D A 2 filter B 2 join C 2
54 Fault Recovery Example C 1 lost due to node failure before reduce finishes Reconstruct C 1, eventually, on different node RDD A A 1 filter B 1 join C 1 agg. D A 2 filter B 2 join C 2
55 Fault Recovery Results Iteratrion time (s) Failure happens Iteration
56 Overview 1. Introduction 2. RDDs 3. Generality of RDDs (e.g. streaming) 4. DataFrames 5. Project Tungsten
57 Spark Streaming: Motivation Process large data streams at second-scale latencies Site statistics, intrusion detection, online ML To build and scale these apps users want: Integration: with offline analytical stack Fault-tolerance: both for crashes and stragglers
58 Traditional Streaming Systems Event-driven record-at-a-times Each node has mutable state For each record, update state & send new records State is lost if node dies input records input records mutable state node 1 node 3 node 2 Making stateful stream processing be fault-tolerant is challenging
59 Spark Streaming Data streams are chopped into batches A batch is an RDD holding a few 100s ms worth of data Each batch is processed in Spark data streams receivers batches
60 How does it work? Data streams are chopped into batches A batch is an RDD holding a few 100s ms worth of data Each batch is processed in Spark Results pushed out in batches Streaming data streams receivers batches results
61 Streaming Word Count val lines = context.sockettextstream( localhost, 9999) val words = lines.flatmap(_.split(" ")) val wordcounts = words.map(x => (x, 1)).reduceByKey(_ + _) create DStream from data over socket split lines into words count the words wordcounts.print() ssc.start() print some counts on screen start processing the stream
62 Word Count object NetworkWordCount { def main(args: Array[String]) { val sparkconf = new SparkConf().setAppName("NetworkWordCount") val context = new StreamingContext(sparkConf, Seconds(1)) val lines = context.sockettextstream( localhost, 9999) val words = lines.flatmap(_.split(" ")) val wordcounts = words.map(x => (x, 1)).reduceByKey(_ + _) wordcounts.print() ssc.start() ssc.awaittermination() } }
63 public class WordCountTopology { public static class SplitSentence extends ShellBolt implements IRichBolt { Word Count Spark Streaming object NetworkWordCount { def main(args: Array[String]) { val sparkconf = new SparkConf().setAppName("NetworkWordCount") val context = new StreamingContext(sparkConf, Seconds(1)) val lines = context.sockettextstream( localhost, 9999) val words = lines.flatmap(_.split(" ")) val wordcounts = words.map(x => (x, 1)).reduceByKey(_ + _) wordcounts.print() ssc.start() ssc.awaittermination() } } Storm public SplitSentence() { super("python", "splitsentence.py"); public void declareoutputfields(outputfieldsdeclarer declarer) { declarer.declare(new Fields("word")); public Map<String, Object> getcomponentconfiguration() { return null; } } public static class WordCount extends BaseBasicBolt { Map<String, Integer> counts = new HashMap<String, public void execute(tuple tuple, BasicOutputCollector collector) { String word = tuple.getstring(0); Integer count = counts.get(word); if (count == null) count = 0; count++; counts.put(word, count); collector.emit(new Values(word, count)); public void declareoutputfields(outputfieldsdeclarer declarer) { declarer.declare(new Fields("word", "count")); } } public static void main(string[] args) throws Exception { TopologyBuilder builder = new TopologyBuilder(); builder.setspout("spout", new RandomSentenceSpout(), 5); builder.setbolt("split", new SplitSentence(), 8).shuffleGrouping("spout"); builder.setbolt("count", new WordCount(), 12).fieldsGrouping("split", new Fields("word")); Config conf = new Config(); conf.setdebug(true); if (args!= null && args.length > 0) { conf.setnums(3); StormSubmitter.submitTopologyWithProgressBar(args[0], conf, builder.createtopology()); } else { conf.setmaxtaskparallelism(3); LocalCluster cluster = new LocalCluster(); cluster.submittopology("word-count", conf, builder.createtopology()); Thread.sleep(10000); cluster.shutdown(); } } }
64 Machine Learning Pipelines tokenizer = Tokenizer(inputCol="text", outputcol="words ) hashingtf = HashingTF(inputCol="words", outputcol="features ) lr = LogisticRegression(maxIter=10, regparam=0.01) pipeline = Pipeline(stages=[tokenizer, hashingtf, lr]) df = sqlctx.load("/path/to/data") model = pipeline.fit(df) lr df0 tokenizer df1 hashingtf df2 lr.model df3 Pipeline Model
65 Powerful Stack Agile Development Hadoop Storm Impala (SQL) MapReduce (Streaming) non-test, non-example source lines Giraph (Graph) Spark
66 Powerful Stack Agile Development Hadoop Storm Impala (SQL) MapReduce (Streaming) Giraph (Graph) Spark Streaming non-test, non-example source lines
67 Powerful Stack Agile Development Hadoop Storm Impala (SQL) MapReduce (Streaming) Giraph (Graph) Spark SparkSQL Streaming non-test, non-example source lines
68 Powerful Stack Agile Development Hadoop Storm Impala (SQL) MapReduce (Streaming) Giraph (Graph) Spark GraphX SparkSQL Streaming non-test, non-example source lines
69 Powerful Stack Agile Development Hadoop Storm Impala (SQL) MapReduce (Streaming) Giraph (Graph) Spark Your App? GraphX SparkSQL Streaming non-test, non-example source lines
70 Benefits for Users High performance data sharing Data sharing is the bottleneck in many environments RDD s provide in-place sharing through memory Applications can compose models Run a SQL query and then PageRank the results ETL your data and then run graph/ml on it Benefit from investment in shared functionality E.g. re-usable components (shell) and performance optimizations
71 Overview 1. Introduction 2. RDDs 3. Generality of RDDs (e.g. streaming) 4. DataFrames 5. Project Tungsten
72 Beyond Hadoop Users Spark early adopters Users Understands MapReduce & functional APIs Data Engineers Data Scientists Statisticians R users PyData
73
74 DataFrames in Spark Distributed collection of data grouped into named columns (i.e. RDD with schema) Domain-specific functions designed for common tasks Metadata Sampling Project, filter, aggregation, join, UDFs Available in Python, Scala, Java, and R
75 Spark DataFrame Similar APIs as single-node tools (Pandas, dplyr), i.e. easy to learn > head(filter(df, df$waiting < 50)) # an example in R ## eruptions waiting ## ## ##
76 Spark RDD Execution Java/Scala frontend opaque closures (user-defined functions) Python frontend JVM backend Python backend
77 Spark DataFrame Execution DataFrame frontend Logical Plan Intermediate representation for computation Catalyst optimizer Physical execution
78 Spark DataFrame Execution Python DF Java/Scala DF R DF Simple wrappers to create logical plan Logical Plan Intermediate representation for computation Catalyst optimizer Physical execution
79 Benefit of Logical Plan: Simpler Frontend Python : ~2000 line of code (built over a weekend) R : ~1000 line of code i.e. much easier to add new language bindings (Julia, Clojure, )
80 Performance RDD Python Java/Scala Runtime for an example aggregation workload
81 Benefit of Logical Plan: Performance Parity Across Languages SQL DataFrame R Python Java/Scala RDD Python Java/Scala Runtime for an example aggregation workload (secs)
82 Overview 1. Introduction 2. RDDs 3. Generality of RDDs (e.g. streaming) 4. DataFrames 5. Project Tungsten
83 Hardware Trends Storage Network CPU
84 Hardware Trends 2010 Storage Network 50+MB/s (HDD) 1Gbps CPU ~3GHz
85 Hardware Trends Storage 50+MB/s (HDD) 500+MB/s (SSD) Network 1Gbps 10Gbps CPU ~3GHz ~3GHz
86 Hardware Trends Storage 50+MB/s (HDD) 500+MB/s (SSD) 10X Network 1Gbps 10Gbps 10X CPU ~3GHz ~3GHz L
87 Project Tungsten Substantially speed up execution by optimizing CPU efficiency, via: (1) Runtime code generation (2) Exploiting cache locality (3) Off-heap memory management
88 From DataFrame to Tungsten Python DF Java/Scala DF R DF Logical Plan Initial phase in Spark 1.5 Tungsten Execution More work coming in 2016
89 Project Tungsten: Fully Managed Memory Spark s core API uses raw Java objects for aggregations and joins GC overhead Memory overhead: 4-8x more memory than serialized format Computation overhead: little memory locality DataFrame s use custom binary format and off-heap managed memory GC free No memory overhead Cache locality
90 Example: Hash Table Data Structure Keep data closure to CPU cache
91 Example: Aggregation Operation
92 Unified API, One Engine, Automatically Optimized language frontend SQL Python Java/Scala R DataFrame Logical Plan Tungsten backend JVM LLVM SIMD 3D XPoint
93 Refactoring Spark Core SQL Python SparkR Streaming Advanced Analytics DataFrame (& Dataset) Tungsten Execution
94 Summary General engine with libraries for many data analysis tasks Access to diverse data sources Streaming SQL ML Graph Simple, unified API Major focus going forward: Easy of use (DataFrames) Performance (Tungsten)
Unified Big Data Processing with Apache Spark. Matei Zaharia @matei_zaharia
Unified Big Data Processing with Apache Spark Matei Zaharia @matei_zaharia What is Apache Spark? Fast & general engine for big data processing Generalizes MapReduce model to support more types of processing
Spark: Making Big Data Interactive & Real-Time
Spark: Making Big Data Interactive & Real-Time Matei Zaharia UC Berkeley / MIT www.spark-project.org What is Spark? Fast and expressive cluster computing system compatible with Apache Hadoop Improves efficiency
Fast and Expressive Big Data Analytics with Python. Matei Zaharia UC BERKELEY
Fast and Expressive Big Data Analytics with Python Matei Zaharia UC Berkeley / MIT UC BERKELEY spark-project.org What is Spark? Fast and expressive cluster computing system interoperable with Apache Hadoop
Conquering Big Data with BDAS (Berkeley Data Analytics)
UC BERKELEY Conquering Big Data with BDAS (Berkeley Data Analytics) Ion Stoica UC Berkeley / Databricks / Conviva Extracting Value from Big Data Insights, diagnosis, e.g.,» Why is user engagement dropping?»
Spark ΕΡΓΑΣΤΗΡΙΟ 10. Prepared by George Nikolaides 4/19/2015 1
Spark ΕΡΓΑΣΤΗΡΙΟ 10 Prepared by George Nikolaides 4/19/2015 1 Introduction to Apache Spark Another cluster computing framework Developed in the AMPLab at UC Berkeley Started in 2009 Open-sourced in 2010
Apache Spark 11/10/15. Context. Reminder. Context. What is Spark? A GrowingStack
Apache Spark Document Analysis Course (Fall 2015 - Scott Sanner) Zahra Iman Some slides from (Matei Zaharia, UC Berkeley / MIT& Harold Liu) Reminder SparkConf JavaSpark RDD: Resilient Distributed Datasets
Introduc8on to Apache Spark
Introduc8on to Apache Spark Jordan Volz, Systems Engineer @ Cloudera 1 Analyzing Data on Large Data Sets Python, R, etc. are popular tools among data scien8sts/analysts, sta8s8cians, etc. Why are these
Introduction to Big Data! with Apache Spark" UC#BERKELEY#
Introduction to Big Data! with Apache Spark" UC#BERKELEY# This Lecture" The Big Data Problem" Hardware for Big Data" Distributing Work" Handling Failures and Slow Machines" Map Reduce and Complex Jobs"
Data Science in the Wild
Data Science in the Wild Lecture 4 59 Apache Spark 60 1 What is Spark? Not a modified version of Hadoop Separate, fast, MapReduce-like engine In-memory data storage for very fast iterative queries General
Architectures for massive data management
Architectures for massive data management Apache Spark Albert Bifet albert.bifet@telecom-paristech.fr October 20, 2015 Spark Motivation Apache Spark Figure: IBM and Apache Spark What is Apache Spark Apache
Beyond Hadoop with Apache Spark and BDAS
Beyond Hadoop with Apache Spark and BDAS Khanderao Kand Principal Technologist, Guavus 12 April GITPRO World 2014 Palo Alto, CA Credit: Some stajsjcs and content came from presentajons from publicly shared
Unified Big Data Analytics Pipeline. 连 城 lian@databricks.com
Unified Big Data Analytics Pipeline 连 城 lian@databricks.com What is A fast and general engine for large-scale data processing An open source implementation of Resilient Distributed Datasets (RDD) Has an
Spark in Action. Fast Big Data Analytics using Scala. Matei Zaharia. www.spark- project.org. University of California, Berkeley UC BERKELEY
Spark in Action Fast Big Data Analytics using Scala Matei Zaharia University of California, Berkeley www.spark- project.org UC BERKELEY My Background Grad student in the AMP Lab at UC Berkeley» 50- person
Using Apache Spark Pat McDonough - Databricks
Using Apache Spark Pat McDonough - Databricks Apache Spark spark.incubator.apache.org github.com/apache/incubator-spark user@spark.incubator.apache.org The Spark Community +You! INTRODUCTION TO APACHE
Moving From Hadoop to Spark
+ Moving From Hadoop to Spark Sujee Maniyam Founder / Principal @ www.elephantscale.com sujee@elephantscale.com Bay Area ACM meetup (2015-02-23) + HI, Featured in Hadoop Weekly #109 + About Me : Sujee
Spark. Fast, Interactive, Language- Integrated Cluster Computing
Spark Fast, Interactive, Language- Integrated Cluster Computing Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael Franklin, Scott Shenker, Ion Stoica UC
Scaling Out With Apache Spark. DTL Meeting 17-04-2015 Slides based on https://www.sics.se/~amir/files/download/dic/spark.pdf
Scaling Out With Apache Spark DTL Meeting 17-04-2015 Slides based on https://www.sics.se/~amir/files/download/dic/spark.pdf Your hosts Mathijs Kattenberg Technical consultant Jeroen Schot Technical consultant
Spark and the Big Data Library
Spark and the Big Data Library Reza Zadeh Thanks to Matei Zaharia Problem Data growing faster than processing speeds Only solution is to parallelize on large clusters» Wide use in both enterprises and
The Flink Big Data Analytics Platform. Marton Balassi, Gyula Fora" {mbalassi, gyfora}@apache.org
The Flink Big Data Analytics Platform Marton Balassi, Gyula Fora" {mbalassi, gyfora}@apache.org What is Apache Flink? Open Source Started in 2009 by the Berlin-based database research groups In the Apache
Spark and Shark. High- Speed In- Memory Analytics over Hadoop and Hive Data
Spark and Shark High- Speed In- Memory Analytics over Hadoop and Hive Data Matei Zaharia, in collaboration with Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Cliff Engle, Michael Franklin, Haoyuan Li,
Next-Gen Big Data Analytics using the Spark stack
Next-Gen Big Data Analytics using the Spark stack Jason Dai Chief Architect of Big Data Technologies Software and Services Group, Intel Agenda Overview Apache Spark stack Next-gen big data analytics Our
CS 294: Big Data System Research: Trends and Challenges
CS 294: Big Data System Research: Trends and Challenges Fall 2015 (MW 9:30-11:00, 310 Soda Hall) Ion Stoica and Ali Ghodsi (http://www.cs.berkeley.edu/~istoica/classes/cs294/15/) 1 Big Data First papers:»
Big Data Processing. Patrick Wendell Databricks
Big Data Processing Patrick Wendell Databricks About me Committer and PMC member of Apache Spark Former PhD student at Berkeley Left Berkeley to help found Databricks Now managing open source work at Databricks
Apache Spark : Fast and Easy Data Processing Sujee Maniyam Elephant Scale LLC sujee@elephantscale.com http://elephantscale.com
Apache Spark : Fast and Easy Data Processing Sujee Maniyam Elephant Scale LLC sujee@elephantscale.com http://elephantscale.com Spark Fast & Expressive Cluster computing engine Compatible with Hadoop Came
Ali Ghodsi Head of PM and Engineering Databricks
Making Big Data Simple Ali Ghodsi Head of PM and Engineering Databricks Big Data is Hard: A Big Data Project Tasks Tasks Build a Hadoop cluster Challenges Clusters hard to setup and manage Build a data
Programming Hadoop 5-day, instructor-led BD-106. MapReduce Overview. Hadoop Overview
Programming Hadoop 5-day, instructor-led BD-106 MapReduce Overview The Client Server Processing Pattern Distributed Computing Challenges MapReduce Defined Google's MapReduce The Map Phase of MapReduce
Hadoop2, Spark Big Data, real time, machine learning & use cases. Cédric Carbone Twitter : @carbone
Hadoop2, Spark Big Data, real time, machine learning & use cases Cédric Carbone Twitter : @carbone Agenda Map Reduce Hadoop v1 limits Hadoop v2 and YARN Apache Spark Streaming : Spark vs Storm Machine
Apache Flink Next-gen data analysis. Kostas Tzoumas ktzoumas@apache.org @kostas_tzoumas
Apache Flink Next-gen data analysis Kostas Tzoumas ktzoumas@apache.org @kostas_tzoumas What is Flink Project undergoing incubation in the Apache Software Foundation Originating from the Stratosphere research
What s next for the Berkeley Data Analytics Stack?
What s next for the Berkeley Data Analytics Stack? Michael Franklin June 30th 2014 Spark Summit San Francisco UC BERKELEY AMPLab: Collaborative Big Data Research 60+ Students, Postdocs, Faculty and Staff
HiBench Introduction. Carson Wang (carson.wang@intel.com) Software & Services Group
HiBench Introduction Carson Wang (carson.wang@intel.com) Agenda Background Workloads Configurations Benchmark Report Tuning Guide Background WHY Why we need big data benchmarking systems? WHAT What is
Apache Spark and Distributed Programming
Apache Spark and Distributed Programming Concurrent Programming Keijo Heljanko Department of Computer Science University School of Science November 25th, 2015 Slides by Keijo Heljanko Apache Spark Apache
CS555: Distributed Systems [Fall 2015] Dept. Of Computer Science, Colorado State University
![CS555: Distributed Systems [Fall 2015] Dept. Of Computer Science, Colorado State University](/thumbs/26/8135278.jpg "CS555: Distributed Systems [Fall 2015] Dept. Of Computer Science, Colorado State University")
CS 555: DISTRIBUTED SYSTEMS [SPARK] Shrideep Pallickara Computer Science Colorado State University Frequently asked questions from the previous class survey Streaming Significance of minimum delays? Interleaving
Architectures for massive data management
Architectures for massive data management Apache Kafka, Samza, Storm Albert Bifet albert.bifet@telecom-paristech.fr October 20, 2015 Stream Engine Motivation Digital Universe EMC Digital Universe with
High-Speed In-Memory Analytics over Hadoop and Hive Data
High-Speed In-Memory Analytics over Hadoop and Hive Data Big Data 2015 Apache Spark Not a modified version of Hadoop Separate, fast, MapReduce-like engine In-memory data storage for very fast iterative
How To Create A Data Visualization With Apache Spark And Zeppelin 2.5.3.5
Big Data Visualization using Apache Spark and Zeppelin Prajod Vettiyattil, Software Architect, Wipro Agenda Big Data and Ecosystem tools Apache Spark Apache Zeppelin Data Visualization Combining Spark
In-memory data pipeline and warehouse at scale using Spark, Spark SQL, Tachyon and Parquet
In-memory data pipeline and warehouse at scale using Spark, Spark SQL, Tachyon and Parquet Ema Iancuta iorhian@gmail.com Radu Chilom radu.chilom@gmail.com Buzzwords Berlin - 2015 Big data analytics / machine
Big Data Analytics with Spark and Oscar BAO. Tamas Jambor, Lead Data Scientist at Massive Analytic
Big Data Analytics with Spark and Oscar BAO Tamas Jambor, Lead Data Scientist at Massive Analytic About me Building a scalable Machine Learning platform at MA Worked in Big Data and Data Science in the
Big Data Research in the AMPLab: BDAS and Beyond
Big Data Research in the AMPLab: BDAS and Beyond Michael Franklin UC Berkeley 1 st Spark Summit December 2, 2013 UC BERKELEY AMPLab: Collaborative Big Data Research Launched: January 2011, 6 year planned
StratioDeep. An integration layer between Cassandra and Spark. Álvaro Agea Herradón Antonio Alcocer Falcón
StratioDeep An integration layer between Cassandra and Spark Álvaro Agea Herradón Antonio Alcocer Falcón StratioDeep An integration layer between Cassandra and Spark Álvaro Agea Herradón Antonio Alcocer
Streaming items through a cluster with Spark Streaming
Streaming items through a cluster with Spark Streaming Tathagata TD Das @tathadas CME 323: Distributed Algorithms and Optimization Stanford, May 6, 2015 Who am I? > Project Management Committee (PMC) member
Hadoop Ecosystem B Y R A H I M A.
Hadoop Ecosystem B Y R A H I M A. History of Hadoop Hadoop was created by Doug Cutting, the creator of Apache Lucene, the widely used text search library. Hadoop has its origins in Apache Nutch, an open
An Open Source Memory-Centric Distributed Storage System
An Open Source Memory-Centric Distributed Storage System Haoyuan Li, Tachyon Nexus haoyuan@tachyonnexus.com September 30, 2015 @ Strata and Hadoop World NYC 2015 Outline Open Source Introduction to Tachyon
Cloud Computing and Big Data Processing
Cloud Computing and Big Data Processing Shivaram Venkataraman UC Berkeley, AMP Lab With slides from Matei Zaharia UC BERKELEY Cloud Computing, Big Data Hardware Open MPI Software Google 1997 Data, Data,
Big Data Approaches. Making Sense of Big Data. Ian Crosland. Jan 2016
Big Data Approaches Making Sense of Big Data Ian Crosland Jan 2016 Accelerate Big Data ROI Even firms that are investing in Big Data are still struggling to get the most from it. Make Big Data Accessible
Tachyon: memory-speed data sharing
Tachyon: memory-speed data sharing Ali Ghodsi, Haoyuan (HY) Li, Matei Zaharia, Scott Shenker, Ion Stoica UC Berkeley Memory trumps everything else RAM throughput increasing exponentially Disk throughput
CSE-E5430 Scalable Cloud Computing Lecture 11
CSE-E5430 Scalable Cloud Computing Lecture 11 Keijo Heljanko Department of Computer Science School of Science Aalto University keijo.heljanko@aalto.fi 30.11-2015 1/24 Distributed Coordination Systems Consensus
Real Time Data Processing using Spark Streaming
Real Time Data Processing using Spark Streaming Hari Shreedharan, Software Engineer @ Cloudera Committer/PMC Member, Apache Flume Committer, Apache Sqoop Contributor, Apache Spark Author, Using Flume (O
Developing Scalable Smart Grid Infrastructure to Enable Secure Transmission System Control
Developing Scalable Smart Grid Infrastructure to Enable Secure Transmission System Control EP/K006487/1 UK PI: Prof Gareth Taylor (BU) China PI: Prof Yong-Hua Song (THU) Consortium UK Members: Brunel University
Spark: Cluster Computing with Working Sets
Spark: Cluster Computing with Working Sets Outline Why? Mesos Resilient Distributed Dataset Spark & Scala Examples Uses Why? MapReduce deficiencies: Standard Dataflows are Acyclic Prevents Iterative Jobs
This is a brief tutorial that explains the basics of Spark SQL programming.
About the Tutorial Apache Spark is a lightning-fast cluster computing designed for fast computation. It was built on top of Hadoop MapReduce and it extends the MapReduce model to efficiently use more types
WHITE PAPER. Building Big Data Analytical Applications at Scale Using Existing ETL Skillsets INTELLIGENT BUSINESS STRATEGIES
INTELLIGENT BUSINESS STRATEGIES WHITE PAPER Building Big Data Analytical Applications at Scale Using Existing ETL Skillsets By Mike Ferguson Intelligent Business Strategies June 2015 Prepared for: Table
Apache Flink. Fast and Reliable Large-Scale Data Processing
Apache Flink Fast and Reliable Large-Scale Data Processing Fabian Hueske @fhueske 1 What is Apache Flink? Distributed Data Flow Processing System Focused on large-scale data analytics Real-time stream
Berkeley Data Analytics Stack:! Experience and Lesson Learned
UC BERKELEY Berkeley Data Analytics Stack:! Experience and Lesson Learned Ion Stoica UC Berkeley, Databricks, Conviva Research Philosophy Follow real problems Focus on novel usage scenarios Build real
How Companies are! Using Spark
How Companies are! Using Spark And where the Edge in Big Data will be Matei Zaharia History Decreasing storage costs have led to an explosion of big data Commodity cluster software, like Hadoop, has made
Making Sense at Scale with Algorithms, Machines & People!
UC BERKELEY Making Sense at Scale with Algorithms, Machines & People PI: Michael Franklin University of California, Berkeley Expeditions in Computing PI Meeting May 15, 2013 2 The Berkeley AMPLab Sources
Functional Query Optimization with" SQL
" Functional Query Optimization with" SQL Michael Armbrust @michaelarmbrust spark.apache.org What is Apache Spark? Fast and general cluster computing system interoperable with Hadoop Improves efficiency
Making Sense at Scale with the Berkeley Data Analytics Stack
Making Sense at Scale with the Berkeley Data Analytics Stack Michael Franklin February 3, 2015 WSDM 2015 Shanghai UC BERKELEY Agenda A Little Bit on Big Data AMPLab Overview BDAS Philosophy: Unification
Big Data Analytics Hadoop and Spark
Big Data Analytics Hadoop and Spark Shelly Garion, Ph.D. IBM Research Haifa 1 What is Big Data? 2 What is Big Data? Big data usually includes data sets with sizes beyond the ability of commonly used software
Big Data Analytics with Cassandra, Spark & MLLib
Big Data Analytics with Cassandra, Spark & MLLib Matthias Niehoff AGENDA Spark Basics In A Cluster Cassandra Spark Connector Use Cases Spark Streaming Spark SQL Spark MLLib Live Demo CQL QUERYING LANGUAGE
The Internet of Things and Big Data: Intro
The Internet of Things and Big Data: Intro John Berns, Solutions Architect, APAC - MapR Technologies April 22 nd, 2014 1 What This Is; What This Is Not It s not specific to IoT It s not about any specific
SPARK USE CASE IN TELCO. Apache Spark Night 9-2-2014! Chance Coble!
SPARK USE CASE IN TELCO Apache Spark Night 9-2-2014! Chance Coble! Use Case Profile Telecommunications company Shared business problems/pain Scalable analytics infrastructure is a problem Pushing infrastructure
The Berkeley Data Analytics Stack: Present and Future
The Berkeley Data Analytics Stack: Present and Future Michael Franklin 27 March 2014 Technion Big Day on Big Data UC BERKELEY BDAS in the Big Data Context 2 Sources Driving Big Data It s All Happening
Unlocking the True Value of Hadoop with Open Data Science
Unlocking the True Value of Hadoop with Open Data Science Kristopher Overholt Solution Architect Big Data Tech 2016 MinneAnalytics June 7, 2016 Overview Overview of Open Data Science Python and the Big
Distributed DataFrame on Spark: Simplifying Big Data For The Rest Of Us
DATA INTELLIGENCE FOR ALL Distributed DataFrame on Spark: Simplifying Big Data For The Rest Of Us Christopher Nguyen, PhD Co-Founder & CEO Agenda 1. Challenges & Motivation 2. DDF Overview 3. DDF Design
Hadoop & Spark Using Amazon EMR
Hadoop & Spark Using Amazon EMR Michael Hanisch, AWS Solutions Architecture 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Agenda Why did we build Amazon EMR? What is Amazon EMR?
Spark and Shark: High-speed In-memory Analytics over Hadoop Data
Spark and Shark: High-speed In-memory Analytics over Hadoop Data May 14, 2013 @ Oracle Reynold Xin, AMPLab, UC Berkeley The Big Data Problem Data is growing faster than computation speeds Accelerating
Machine- Learning Summer School - 2015
Machine- Learning Summer School - 2015 Big Data Programming David Franke Vast.com hbp://www.cs.utexas.edu/~dfranke/ Goals for Today Issues to address when you have big data Understand two popular big data
Big Data Analytics. Lucas Rego Drumond
Big Data Analytics Big Data Analytics Lucas Rego Drumond Information Systems and Machine Learning Lab (ISMLL) Institute of Computer Science University of Hildesheim, Germany Apache Spark Apache Spark 1
The Berkeley AMPLab - Collaborative Big Data Research
The Berkeley AMPLab - Collaborative Big Data Research UC BERKELEY Anthony D. Joseph LASER Summer School September 2013 About Me Education: MIT SB, MS, PhD Joined Univ. of California, Berkeley in 1998 Current
Big Data and Scripting Systems beyond Hadoop
Big Data and Scripting Systems beyond Hadoop 1, 2, ZooKeeper distributed coordination service many problems are shared among distributed systems ZooKeeper provides an implementation that solves these avoid
Hadoop MapReduce and Spark. Giorgio Pedrazzi, CINECA-SCAI School of Data Analytics and Visualisation Milan, 10/06/2015
Hadoop MapReduce and Spark Giorgio Pedrazzi, CINECA-SCAI School of Data Analytics and Visualisation Milan, 10/06/2015 Outline Hadoop Hadoop Import data on Hadoop Spark Spark features Scala MLlib MLlib
Introduction to Spark
Introduction to Spark Shannon Quinn (with thanks to Paco Nathan and Databricks) Quick Demo Quick Demo API Hooks Scala / Java All Java libraries *.jar http://www.scala- lang.org Python Anaconda: https://
Significantly Speed up real world big data Applications using Apache Spark
Significantly Speed up real world big data Applications using Apache Spark Mingfei Shi(mingfei.shi@intel.com) Grace Huang ( jie.huang@intel.com) Intel/SSG/Big Data Technology 1 Agenda Who are we? Case
Brave New World: Hadoop vs. Spark
Brave New World: Hadoop vs. Spark Dr. Kurt Stockinger Associate Professor of Computer Science Director of Studies in Data Science Zurich University of Applied Sciences Datalab Seminar, Zurich, Oct. 7,
Tachyon: Reliable File Sharing at Memory- Speed Across Cluster Frameworks
Tachyon: Reliable File Sharing at Memory- Speed Across Cluster Frameworks Haoyuan Li UC Berkeley Outline Motivation System Design Evaluation Results Release Status Future Directions Outline Motivation
Mesos: A Platform for Fine- Grained Resource Sharing in Data Centers (II)
UC BERKELEY Mesos: A Platform for Fine- Grained Resource Sharing in Data Centers (II) Anthony D. Joseph LASER Summer School September 2013 My Talks at LASER 2013 1. AMP Lab introduction 2. The Datacenter
Systems Engineering II. Pramod Bhatotia TU Dresden pramod.bhatotia@tu- dresden.de
Systems Engineering II Pramod Bhatotia TU Dresden pramod.bhatotia@tu- dresden.de About me! Since May 2015 2015 2012 Research Group Leader cfaed, TU Dresden PhD Student MPI- SWS Research Intern Microsoft
Architectures for Big Data Analytics A database perspective
Architectures for Big Data Analytics A database perspective Fernando Velez Director of Product Management Enterprise Information Management, SAP June 2013 Outline Big Data Analytics Requirements Spectrum
Big Data Frameworks: Scala and Spark Tutorial
Big Data Frameworks: Scala and Spark Tutorial 13.03.2015 Eemil Lagerspetz, Ella Peltonen Professor Sasu Tarkoma These slides: http://is.gd/bigdatascala www.cs.helsinki.fi Functional Programming Functional
Open source Google-style large scale data analysis with Hadoop
Open source Google-style large scale data analysis with Hadoop Ioannis Konstantinou Email: ikons@cslab.ece.ntua.gr Web: http://www.cslab.ntua.gr/~ikons Computing Systems Laboratory School of Electrical
BIG DATA APPLICATIONS
BIG DATA ANALYTICS USING HADOOP AND SPARK ON HATHI Boyu Zhang Research Computing ITaP BIG DATA APPLICATIONS Big data has become one of the most important aspects in scientific computing and business analytics
Can t We All Just Get Along? Spark and Resource Management on Hadoop
Can t We All Just Get Along? Spark and Resource Management on Hadoop Introduc=ons So>ware engineer at Cloudera MapReduce, YARN, Resource management Hadoop commider Introduc=on Spark as a first class data
The Hadoop Eco System Shanghai Data Science Meetup
The Hadoop Eco System Shanghai Data Science Meetup Karthik Rajasethupathy, Christian Kuka 03.11.2015 @Agora Space Overview What is this talk about? Giving an overview of the Hadoop Ecosystem and related
Analytics on Spark & Shark @Yahoo
Analytics on Spark & Shark @Yahoo PRESENTED BY Tim Tully December 3, 2013 Overview Legacy / Current Hadoop Architecture Reflection / Pain Points Why the movement towards Spark / Shark New Hybrid Environment
Data-Intensive Programming. Timo Aaltonen Department of Pervasive Computing
Data-Intensive Programming Timo Aaltonen Department of Pervasive Computing Data-Intensive Programming Lecturer: Timo Aaltonen University Lecturer timo.aaltonen@tut.fi Assistants: Henri Terho and Antti
Hadoop IST 734 SS CHUNG
Hadoop IST 734 SS CHUNG Introduction What is Big Data?? Bulk Amount Unstructured Lots of Applications which need to handle huge amount of data (in terms of 500+ TB per day) If a regular machine need to
Big Data at Spotify. Anders Arpteg, Ph D Analytics Machine Learning, Spotify
Big Data at Spotify Anders Arpteg, Ph D Analytics Machine Learning, Spotify Quickly about me Quickly about Spotify What is all the data used for? Quickly about Spark Hadoop MR vs Spark Need for (distributed)
Dell In-Memory Appliance for Cloudera Enterprise
Dell In-Memory Appliance for Cloudera Enterprise Hadoop Overview, Customer Evolution and Dell In-Memory Product Details Author: Armando Acosta Hadoop Product Manager/Subject Matter Expert Armando_Acosta@Dell.com/
CS54100: Database Systems
CS54100: Database Systems Cloud Databases: The Next Post- Relational World 18 April 2012 Prof. Chris Clifton Beyond RDBMS The Relational Model is too limiting! Simple data model doesn t capture semantics
Large scale processing using Hadoop. Ján Vaňo
Large scale processing using Hadoop Ján Vaňo What is Hadoop? Software platform that lets one easily write and run applications that process vast amounts of data Includes: MapReduce offline computing engine
Tachyon: A Reliable Memory Centric Storage for Big Data Analytics
Tachyon: A Reliable Memory Centric Storage for Big Data Analytics a Haoyuan (HY) Li, Ali Ghodsi, Matei Zaharia, Scott Shenker, Ion Stoica June 30 th, 2014 Spark Summit @ San Francisco UC Berkeley Outline
Real-Time Analytical Processing (RTAP) Using the Spark Stack. Jason Dai jason.dai@intel.com Intel Software and Services Group
Real-Time Analytical Processing (RTAP) Using the Spark Stack Jason Dai jason.dai@intel.com Intel Software and Services Group Project Overview Research & open source projects initiated by AMPLab in UC Berkeley
Managing large clusters resources
Managing large clusters resources ID2210 Gautier Berthou (SICS) Big Processing with No Locality Job( /crawler/bot/jd.io/1 ) submi t Workflow Manager Compute Grid Node Job This doesn t scale. Bandwidth
Making big data simple with Databricks
Making big data simple with Databricks We are Databricks, the company behind Spark Founded by the creators of Apache Spark in 2013 Data 75% Share of Spark code contributed by Databricks in 2014 Value Created
DRIVING INNOVATION THROUGH DATA ACCELERATING BIG DATA APPLICATION DEVELOPMENT WITH CASCADING
DRIVING INNOVATION THROUGH DATA ACCELERATING BIG DATA APPLICATION DEVELOPMENT WITH CASCADING Supreet Oberoi VP Field Engineering, Concurrent Inc GET TO KNOW CONCURRENT Leader in Application Infrastructure
Shark Installation Guide Week 3 Report. Ankush Arora
Shark Installation Guide Week 3 Report Ankush Arora Last Updated: May 31,2014 CONTENTS Contents 1 Introduction 1 1.1 Shark..................................... 1 1.2 Apache Spark.................................
1. The orange button 2. Audio Type 3. Close apps 4. Enlarge my screen 5. Headphones 6. Questions Pane. SparkR 2
SparkR 1. The orange button 2. Audio Type 3. Close apps 4. Enlarge my screen 5. Headphones 6. Questions Pane SparkR 2 Lecture slides and/or video will be made available within one week Live Demonstration
In Memory Accelerator for MongoDB
In Memory Accelerator for MongoDB Yakov Zhdanov, Director R&D GridGain Systems GridGain: In Memory Computing Leader 5 years in production 100s of customers & users Starts every 10 secs worldwide Over 15,000,000
Big Data With Hadoop
With Saurabh Singh singh.903@osu.edu The Ohio State University February 11, 2016 Overview 1 2 3 Requirements Ecosystem Resilient Distributed Datasets (RDDs) Example Code vs Mapreduce 4 5 Source: [Tutorials
Productionizing a 24/7 Spark Streaming Service on YARN
Productionizing a 24/7 Spark Streaming Service on YARN Issac Buenrostro, Arup Malakar Spark Summit 2014 July 1, 2014 About Ooyala Cross-device video analytics and monetization products and services Founded