BITKOM& NIK - Big Data Wo liegen die Chancen für den Mittelstand?

Transcription

1 BITKOM& NIK - Big Data Wo liegen die Chancen für den Mittelstand?

2 The Big Data Buzz

3 big data is a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. (Wikipedia)

4 Apache Hadoop the leading big data analytics technology won the Media Guardian Innovator of the Year Award

5 Big Data Analytics

6 Is there a comprehensive strategy for Big Data in your company? Please estimate the yearly growth of data for reporting and analysis No 63% % 66% 19% 7% Planned 23% % 54% 35% 8% % 48% 36% 13% Yes 14% 0% 20% 40% 60% 80% 0% 20% 40% 60% 80% 100% Negative growth / No growth 1-25% growth 25%-50% growth Over 50% growth Survey with 274 participants from DACH, France, Nordics, Netherlands,

7 What problems have you encountered when using Big data? In which areas does your company use Big data analysis? Controlling 24% Inadequate technical know-how 46% Marketing 19% Inadequate analytical know-how 44% Sales 18% Lack of compelling business case 36% IT 18% Technical problems 34% Production 17% Cost 33% Research and development 14% Data privacy issues 25% Supply Chain 7% Can not make Big data usable for end-users 15% 0% 20% 40% 0% 20% 40% 60%

8 The database journey continuous: Big Data

9 Scale-Up (SMP) Scale-out (MPP) Up to 256 Cores in Windows today Parallel Data Warehouse

10 but is this already Big Data?

11

12

13 The large hadron collider produces 15 PB/year*

14 But what if my customer doesn t own a large hadron collider

15 Large scale plants Vehicle fleets Smart Grids Green Energy Stock Exchanges Host Protocols Computer Centers Web Farms Twitter Facebook Google Analytics

16 Source: The Importance of 'Big Data': A Definition, Mark Beyer, Douglas Laney, G "Big data" is high-volume, high-velocity and high-variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making.

17

18 200 Mio Feeds 100 PB 20 PB

19 Social analytics different sources data structures sophisticated algorithms

20

21 Up to 75 control units in one car Up to possible special equipments About 15 GB data on board (incl. navigational data) Up to stores for onboard diagnosis More than car diagnoses happening each day

22 How to deal with the 3 Vs?

23

24 Hadoop

25

26

27 The Hadoop Ecosystem (simplified) Quelle: Tom White s Hadoop: The Definitive Guide

28 Scalable machine learning library that leverages the Hadoop infrastructure Key use cases: Recommendation mining Examine user behavior, build recommendation model Clustering Grouping data into related topics Classification Learn from classified documents to assign categories to unlabeled data Algorithmns: K-means Clustering, Naïve Bayes, Decision Tree, Neural network, Hierarchical Clustering, Positive Matrix Factorization and more

29 Zookeeper Ambari HCatalog Oozie HBase/Cassandra/Couch/ MongoDB Hive Mahout R Cascad-ing Pig Flume Sqoop HBase (Column DB) Avro Hadoop = MapReduce + HDFS Hortonworks HStreaming Karmasphere Splunk Cloudera Hadapt MapR Datameer

30

31 Use of Big Data

32

33

34 180 PB raw data in > computers (polystructured)* Biggest Hadoop cluster: nodes (2x4 CPUs, 4x1 TB disks, 16 GB RAM) Ad Impressions: Cube with 207 Measures 24 Dimensions 247 Attributes Desktop Clients (Excel & Tableau): < 6s ad hoc query time

35 Query engine for SQL & Hadoop Cost base optimizer. Decides on: Rendering operators in Map/Reduce-Jobs or Moving HDFS data into RDBMS storage HDFS-Bridge for parallelized Data Transport Regular T-SQL Results PDW V2 & HDFS Data Nodes

36 maturing Not every problem questions simple

37 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation.