Hortonworks: We Do Hadoop.

Transcription

1 Hortonworks: We Do Hadoop. Our mission is to enable your Modern Data Architecture by delivering One Enterprise Hadoop November 2013 Page 1

2 Recent Announcements October 23 Hortonworks Data Platform 2.0 GA Culmination of years of work, Hortonworks delivers YARN from the community to the enterprise to further cements Hadoop s role in the data architectures of tomorrow YARN Stinger Phase 2 Platform and Operational Services October 15 Real Time Stream Processing with Storm Announcing Hortonworks investment roadmap for deeply integrating Apache Storm with Hadoop for analyzing sensor and machine data Page 2

3 HDP 2.0: Investment Themes FLEXIBLE Delivering YARN from the community to the enterprise to extend Hadoop into a multi-use platform Hadoop beyond batch COMPLETE Delivery of Stinger phase 2 Provides management of YARN and Hadoop 2.0 with Ambari As always, we deliver a tested, stable distribution across all the most recent Apache release INTEGRATED Certified by partners and customers Page 3

4 HDP: Reliable, Consistent & Current HDP demonstrates most recent community innovation OCT 2013 May 2013 FEB 2013 SEPT 2012 JUNE 2012 HDP 2.0 HDP 1.3 HDP 1.2 HDP 1.1 HDP Hadoop Pig HCatalog Hive HBase Sqoop Flume Oozie Zookeeper Mahout HMC1.1 HMC1 Ambari Hortonworks Data Platform

5 Hadoop Beyond Batch A shift from the old to the new Single Use System Batch Apps HADOOP 1.0 MapReduce (cluster resource management & data processing) HDFS (redundant, reliable storage) MapReduce (batch) Multi Use Data Platform Batch, Interactive, Online, Streaming, HADOOP 2.0 Tez (interac6ve) YARN (opera6ng system: cluster resource management) HDFS2 (redundant, reliable storage) Others (varied) Page 5

6 Hadoop: a FLEXIBLE Multi-use Data Platform Apache YARN: the Hadoop 2.0 Operating System Apache YARN Enables data processing models beyond MapReduce (batch), such as interactive, online, streaming and beyond. Interact with all data in multiple ways simultaneously Data Processing Engines Run Na?vely IN Hadoop BATCH MapReduce INTERACTIVE Tez ONLINE HBase STREAMING Storm GRAPH Giraph REEF LASR, HPA OTHERS YARN (opera6ng system: cluster resource management) HDFS2 (redundant, reliable storage) Page 6

7 YARN: Efficiency with Shared Services 2 YARN allows you to double processing in Hadoop x on the same hardware while providing more predictable performance & quality of service Standard SQL Processing Hive Batch MapReduce Interac?ve Tez Online Data Processing HBase, Accumulo Real Time Stream Processing Storm Efficient Cluster Resource Management & Shared Services (Apache YARN) Redundant, Reliable Storage (HDFS2) others Shared Services YARN provides a stable, common set of shared resources across multiple, coordinated workloads - Manage & Monitor - Multi-tenancy - Security - High Availability - Disaster Recovery Page 7

8 Batch AND Interactive SQL-IN-Hadoop Stinger Initiative A broad, community-based effort to drive the next generation of HIVE Goals: Speed Improve Hive query performance by 100X to allow for interactive query times (seconds) Scale The only SQL interface to Hadoop designed for queries that scale from TB to PB SQL Support broadest range of SQL semantics for analytic applications running against Hadoop all IN Hadoop S?nger Project (announced February 2013) S?nger Phase 1: Base Op6miza6ons SQL Types SQL Analy6c Func6ons ORCFile Modern File Format S?nger Phase 2: SQL Types SQL Analy6c Func6ons Advanced Op6miza6ons Performance Boosts via YARN S?nger Phase 3 Hive on Apache Tez Query Service (always on) Buffer Cache Cost Based Op6mizer (Op6q) 70% complete in 6 months Delivered May 2013 HIVE 0.11 (HDP 1.3) Delivered September 2013 HIVE 0.12 (HDP 2.0) Coming Soon Page 8

9 SPEED: Increasing Hive Performance Human Acceptable Query Times across ALL use cases Simple and advanced queries across petabytes in seconds Integrates seamlessly with existing tools Currently a 60x improvement in just six months Performance Improvements included in Hive 12 Vectorization Base & advanced query optimization Startup time improvement Join optimizations < 10s Page 9

10 SCALE: Interactive Query at Petabyte Scale Sustained Query Times Apache Hive 0.12 provides sustained acceptable query times even at petabyte scale Smaller Footprint Better encoding with ORCFile in Apache Hive 12 reduces resource requirements for your cluster File Size Comparison Across Encoding Methods Dataset: TPC DS Scale 500 Dataset 585 GB (Original Size) 505 GB (14% Smaller) Parquet 221 GB (62% Smaller) Hive GB (78% Smaller) Larger Block Sizes Columnar format arranges columns adjacent within the file for compression & fast access Encoded with Text Encoded with RCFile Encoded with Parquet Encoded with ORCFile Page 10

11 SQL: Enhancing SQL Semantics Hive SQL Datatypes INT TINYINT/SMALLINT/BIGINT BOOLEAN FLOAT DOUBLE STRING TIMESTAMP BINARY DECIMAL ARRAY, MAP, STRUCT, UNION DATE VARCHAR CHAR Hive SQL Seman?cs SELECT, INSERT GROUP BY, ORDER BY, SORT BY JOIN on explicit join key Inner, outer, cross and semi joins Sub queries in FROM clause ROLLUP and CUBE UNION Windowing Func6ons (OVER, RANK, etc) Custom Java UDFs Standard Aggrega6on (SUM, AVG, etc.) Advanced UDFs (ngram, Xpath, URL) Sub queries for IN/NOT IN, HAVING INTERSECT / EXCEPT Expanded JOIN Syntax SQL Compliance Hive 12 provides a wide array of SQL datatypes and semantics so your existing tools integrate more seamlessly with Hadoop Available Hive 0.12 (HDP 2.0) Roadmap Page 11

12 HDFS2 Highlights HDFS Federation/NameSpaces (further scales # of files & nodes) Automated failover with a hot standby and full stack resiliency for the NameNode master service Standard NFS read/write access to HDFS Point in time recovery with Snapshots in HDFS Wire Encryption for HDFS Data Transfer Protocol MapReduce (batch) HADOOP 2.0 Tez (interac6ve) YARN (opera6ng system: cluster resource management) HDFS2 (redundant, reliable storage) Others (varied) Page 12

13 HDP 2.0 Certified Partners Page 13

14 Announcements October 23 Hortonworks Data Platform 2.0 GA Culmination of years of work, Hortonworks delivers YARN from the community to the enterprise to further cements Hadoop s role in the data architectures of tomorrow YARN Stinger Phase 2 Platform and Operational Services October 15 Real Time Stream Processing with Storm Announcing Hortonworks investment roadmap for deeply integrating Apache Storm with Hadoop for analyzing sensor and machine data Page 14

15 Hadoop 2.0: GA in Apache Community YARN based architecture of Hadoop 2.0 enables new processing approaches Single Use System Batch Apps HADOOP 1.0 Multi Use Data Platform Batch, Interactive, Online, Streaming, HADOOP 2.0 MapReduce (cluster resource management & data processing) HDFS (redundant, reliable storage) MapReduce (batch) Tez (interac6ve) YARN (cluster resource management) HDFS2 (redundant, reliable storage) Others (varied) Page 15

16 Stream Processing in Hadoop Stream processing has emerged as a key use case Driven by new types of data Sensor/Machine Server logs Clickstream Storm with Hadoop enables new business opportunities Low-latency dashboards Quality, Security, Safety, Operations Alerts Improved operations Real-time data integration HADOOP 2.0 Multi Use Data Platform Batch, Interactive, Online, Streaming, MapReduce (batch) Tez (interac6ve) YARN (cluster resource management) HDFS2 (redundant, reliable storage) Apache STORM (streaming) Page 16

17 Apache Storm Leading for Stream Processing Developed by Twitter, adopted by large enterprises In use and proven at Yahoo! and others Apache Project with active community of developers Key Capabilities of Storm Ingest millions of events / second Perform arithmetic and aggregations on the data as it arrives Alert on boundary conditions Persist to Hive, HBase and HDFS Integration with queuing Machine Server log Events STORM Processing & Events Dashboards Analytics HDFS Page 17

18 Hortonworks Storm Investment Plans Bringing innovation from the community to the Enterprise Hortonworks Investment in Storm Goals: Unlock new uses of data Real-time event processing for sensor networks and business activity monitoring Phase 1: Streaming IN Hadoop Storm on YARN Installa6on with Ambari Ganglia & Nagios monitoring Phase 2: Enterprise connec?vity Q Ease of use Connected with Hadoop and the enterprise. Integrated developer and operations tools Scale Ingesting millions of events per second. Fast query on petabytes of data Data ingest Spouts Bolts for no6fica6on and data persistence: HDFS, HBase AD/LDAP plugin for authen6ca6on High Availability management w/ambari Phase 3: Visual stream development and management Declara6ve wiring Hive update support Advanced scheduler OpenStack Savanna support all IN Hadoop Page 18

19 Hortonworks Approach to Enterprise Hadoop Community Driven Enterprise Apache Hadoop Identify and introduce enterprise requirements into the public domain Work with the community to advance and incubate open source projects Apply Enterprise Rigor to provide the most stable and reliable distribution Page 19

20 One Hadoop: Interoperable & Familiar APPLICATIONS BusinessObjects BI DEV & DATA TOOLS DATA SYSTEM RDBMS EDW MPP HANA OPERATIONAL TOOLS INFRASTRUCTURE SOURCES Exis?ng Sources (CRM, ERP, Clickstream, Logs) Emerging Sources (Sensor, Sen?ment, Geo, Unstructured) Page 20

21 Hortonworks: The Value of Open for You Connect With the Hadoop Community We employ a large number of Apache project committers & innovators so that you are represented in the open source community Visit Try sandbox Follow twitter.com/hortonworks Avoid Vendor Lock Hortonworks Data Platform remain as close to the open source trunk as possible and is developed 100% in the open so you are never locked in The partners you rely on, rely on Hortonworks We work with partners to deeply integrate Hadoop with data center technologies so you can leverage existing skills and investments Certified for the Enterprise We engineer, test and certify the Hortonworks Data Platform at scale to ensure reliability and stability you require for enterprise use Support from the experts We provide the highest quality of support for deploying at scale. You are supported by hundreds of years of Hadoop experience Page 21