Modernizing Your Data Warehouse for Hadoop

Transcription

1 Modernizing Your Data Warehouse for Hadoop Big data. Small data. All data. Audie Wright, DW & Big Data Specialist O , C

2 Unlock Insights on Any Data Taking an End-toEnd Approach to BI and Analytics Modernizing Your Data Warehouse for Hadoop

3 The traditional data warehouse data warehousing has reached the most significant tipping point since its inception. The biggest, possibly most elaborate data management system in IT is changing. Gartner, The State of Data Warehousing in 2012

4 The traditional data warehouse 2 Real time data 1 Increasing 1 data Increasing data 3 New data sources volumes volumes and types 4 Cloud-born data

5 The modern data warehouse

6 Microsoft s modern data warehouse SQL Server 2014 Analytics Platform System Microsoft Azure HDInsight Data Platform

7

8

9 Scale out relational data to petabytes From terabytes to multi-petabytes Scale out technologies in Analytics Platform System APS / HDInsight APS / HDInsight APS / HDInsight APS / HDInsight APS / HDInsight APS / HDInsight APS 0TB 6PB

10 Scale Out non-relational data Scale out big data Scale out non-relational data in HDInsight (for Microsoft Azure or APS)

11

12

13

14

15 In-memory performance In-memory Columnstore for next-generation performance Columnstore index representation

16 Concurrency and mixed workloads Great performance for mixed workloads Query Results

17 Near real-time insights Real-time with complex event processing Event Sources Event Targets

18

19

20

21

22 What is big data? Petabytes Data complexity: variety and velocity

23 What is Hadoop? Distributed, scalable system on commodity HW Operational services Data services AMBARI OOZIE FALCON FLUME SQOOP HBASE PIG HIVE & HCATALOG Core Services LOAD & EXTRACT NFS WebHDFS MAP REDUCE YARN HDFS Hadoop Cluster Hadoop clusters provide scale-out storage and distributed data processing on commodity hardware compute & storage compute & storage

24 IT infrastructure optimization Legal discovery Social network analysis Traffic flow optimization Web app optimization Churn analysis Natural resource exploration Weather forecasting Healthcare outcomes Fraud detection Life sciences research Advertising analysis Equipment monitoring Smart meter monitoring

25 Hadoop offerings on-premise and cloud Real-time with complex event processing Microsoft Azure

26

27 Integrate relational data and Hadoop Integrated query with PolyBase in SQL APS Select Result set Microsoft Azure HDInsight Analytics Platform System PolyBase Hortonworks (Windows, Linux), Cloudera Microsoft HDInsight

28

29

30 Microsoft s modern data warehouse SQL Server 2014 Analytics Platform System Microsoft Azure HDInsight Data Platform

31 Freedom of deployment options and hybrid solutions

32 Appliance vs. Reference Architecture Buying an a appliance Reference Architecture Order SKU from a list of configuration options Factory Order builds hardware & tests from a BOM Hardware vendor installs & connects Customer builds & configures Microsoft validates function & performance Installs software, drivers, firmware, Hands over the etc. keys to the customer Microsoft is the single point of contact for Customer support manages multiple support channels

33 ! Sign up for a free architectural design session for APS with your Microsoft rep! Visit Analytics Platform System at Try HDInsight at Try SQL Server for data warehousing in Microsoft Azure VMs at Try Hortonworks Data Platform for Windows at products/hdp-windows/! Try SQL Server 2014 at sql-server-2014.aspx

34

35 Growth Topology PDW Region Only Scale Unit Base Unit Base UnitExtension

36 Growth Topologies Hadoop Region Extend Min

37 About Analytics Platform System SQL Server Parallel Data Warehouse PolyBase Microsoft HDInsight

38 About Hortonworks Data Platform For Windows

39 About Microsoft Azure HDInsight Microsoft Azure

40 Microsoft Contributions to Hadoop 6,000+ Engineering hours Hive (Improve performance 40x with Stinger) Contributed FileSystem implementation for Microsoft Azure Storage HDFS permissions model mapped to Windows HDP ,000+ Code line contributions Windows, a first class OS for Hadoop REEF for creation and execution of machine learning jobs 9

41 Hortonworks and Microsoft Engineering alignment Corporate alignment Field Alignment

42 ü Data sources Non-Relational Data

43 Microsoft Azure ü

44 44

45 PDW Customers

46 HDInsight Customers

47

48

49

50

51

52

53

54

55

56

57 .