Why Spark on Hadoop Matters

Transcription

1 Why Spark on Hadoop Matters MC Srivas, CTO and Founder, MapR Technologies Apache Spark Summit - July 1,

2 MapR Overview Top Ranked Exponential Growth 500+ Customers Cloud Leaders 3X bookings Q1 13 Q % software licenses 80% of accounts expand 3X < 1% lifetime churn > $1B in incremental revenue generated by 1 customer 2

3 Rapidly Evolving Landscape Management Batch Tez* Spark Cascading Pig MR v1 & v2 ML, Graph GraphX MLLib Mahout APACHE HADOOP AND OSS ECOSYSTEM SQL NoSQL & Streaming Data Security Search Integrtn. & Access Drill* Shark Impala Hive YARN EXECUTION ENGINES Accumulo* Solr HBase Storm* Spark Streaming Hue HttpFS Flume Sqoop MapR Data Platform Knox* Workflow & Data Gov. Falcon* Provision Savannah* Juju Whirr Sentry* Oozie ZooKeeper DATA GOVERNANCE AND OPERATIONS * 2014 TIMELINE 3

4 The Complete Spark Stack on Hadoop Management Batch Tez* Spark Cascading Pig MR v1 & v2 ML, Graph GraphX MLLib Mahout APACHE HADOOP AND OSS ECOSYSTEM SQL NoSQL & Streaming Data Security Search Integrtn. & Access Drill* Shark Impala Hive YARN EXECUTION ENGINES Accumulo* Solr HBase Storm* Spark Streaming Hue HttpFS Flume Sqoop MapR Data Platform Knox* Workflow & Data Gov. Falcon* Provision Savannah* Juju Whirr Sentry* Oozie ZooKeeper DATA GOVERNANCE AND OPERATIONS * 2014 TIMELINE 4

5 A Winning Combination 5

6 Spark Advantages: Easier APIs Python, Scala, Java EASE OF DEVELOPMENT IN-MEMORY PERFORMANCE RDDs DAGs Unify Processing Shark, ML, Streaming, GraphX COMBINE WORKFLOWS 6

7 Hadoop Advantages: UNLIMITED SCALE Multiple data sources Multiple applications Multiple users Reliability Multi-tenancy Security ENTERPRISE PLATFORM WIDE RANGE OF APPLICATIONS Files Databases Semi-structured 7

8 The Combination of Spark on Hadoop UNLIMITED SCALE IN-MEMORY PERFORMANCE WIDE RANGE OF APPLICATIONS EASE OF DEVELOPMENT ENTERPRISE PLATFORM COMBINE WORKFLOWS Operational Applications Augmented by In-Memory Performance 8

9 Case Studies MapR MapR Technologies Technologies 9

10 Industry Leading Ad-Targeting Platform High performance analytics over MapR M7 NoSQL Load from M7 table into RDD to augment scoring in real-time Results fed back to M7 for other applications 10

11 Leading Pharma Company: NextGen Genomics Existing process takes several weeks to align chemical compounds with genes ADAM on Spark allows realignment in a few hours Geneticists can minimize engineering dependency 11

12 Cisco: Security Intelligence Operations Sensor data lands in M7 Spark Streaming on M7 for first check on known threats Data next processed on GraphX and Mahout Results queried using SQL via Shark and Impala 12

13 Insurance Giant: Addressing Health Care Regulations Patient information in M7 combined with clinical records to compute readmittance probability Process uses Spark with transactional data in M7 Insurance options decided in real-time on online portals 13

14 In Summary MapR MapR Technologies Technologies 14

15 Spark on Hadoop gains traction for Real-time applications 15

16 Pick the Right Tool for the Job 16

17 MapR is Unbiased Open Source (a la Linux) Open source distribution is about providing choice Linux includes MySQL, PostgreSQL and SQLite Linux includes Apache httpd, nginx and Lighttpd MapR Distribution for Hadoop Distribution C Distribution H Spark Spark (all of it) and Shark Spark only No Interactive SQL Shark, Impala, Drill, Hive/Tez One option (Impala) One option (Hive/Tez) Versions Hive 0.10, 0.11, 0.12, 0.13 Pig 0.11, 012 HBase 0.94, 0.98 One version One version 17

18 Thank you Engage with maprtech mapr-technologies MapR maprtech 18