Big Data Zurich, November 23. September 2011

Transcription

1 Institute of Technology Management Big Data Projektskizze «Competence Center Automotive Intelligence» Zurich, November 11th 23. September 2011 Felix Wortmann Assistant Professor Technology Management, Robert Winter, Felix for Wortmann Scientific Director Bosch IoT Lab Institut für Wirtschaftsinformatik

2 Goals Goals for today Understand big data starting point : Companies have 1 invested for more than 20 years into analytics. 2 Understand big data value proposition: Where does big data go beyond the capabilities of today s systems? Goals Understand key technologies and their development: 3 There are converging technology streams. Slide 2

3 Big Data Big Data Where are Companies Today? Big Data Hype or Disruptive Innovation? Big Data A New Understanding of Volume, Velocity and Variety Big Data Converging Technology Streams Conclusion and Discussion Slide 3

4 BI as a Basis for Transparency and Decision Making Business intelligence (BI) is a broad category of technologies, applications, and processes for gathering, storing, accessing, and analyzing data. Quelle: Wortmann, 2012; Wixom & Watson, Slide 4

5 Example This dashboard summarizes the daily status of a hospitality company's key performance indicators across 14 locations. (Source: SAP.com) Slide 5

6 The BI approach: Hub and spoke architecture Presentation (Frontend) Reporting, Analyse, Planung (Data Marts) Data Mart Data Mart Data modelling Metadata management (Core) Data Warehouse Metadata Data population Data consolidation Staging Area Slide 6

7 Big Data Big Data Where are Companies Today? Big Data Hype or Disruptive Innovation? Big Data A New Understanding of Volume, Velocity and Variety Big Data Converging Technology Streams Conclusion and Discussion Slide 7

8 The rise of Big Data Big Data emerged three years ago challenging existing analytical technologies, platforms, applications and processes. Business Intelligence Big Data Source: google trends, Slide 8

9 Business Intelligence is there to stay The fundamental BI/DWH architecture is stable for more than 20 years. BI/DWH has matured and evolved over the last two decades. Early 1990ies: BI/DWH architecture Value Proposition Integration of internal data sources High quality information store Proven basis for decision making The Challenges Quelle: Devlin, Murphy, Complexity and costs Speed and performance Integration of unstructured data Real Time Analysis Slide 9

10 Structured Data Retail banking: 250 terabyte DWH Slide 10

11 Unstructured Data Google: 1 Petabyte of Data per hour Source: Economist, Slide

12 Service Use Case: Rapid Social Response Slide 12

13 Sales Use Case: Social Lead Generation Beispiel SDN Slide 13

14 KLM Semi-structured geo-spatially tagged Data mobile transactions per day US 600 mobile billion networks: 600 billion geo-spatially transactions per day Source: ParStream, Slide 14 Source: IBM, 2011.

15 Big Data: Three years ago Big data is a category of technologies, applications, and processes for gathering, storing, accessing, and analyzing data that goes beyond the capabilities of todays BI systems: Volume: From gigabyte and terabyte to petabyte and 1 exabyte. Velocity: From hours and minutes to seconds and 2 milliseconds. Three Converging Domains A. Business intelligence B. Content intelligence C. Real time intelligence Variety: From structured data to semi- and unstructured 3 data. Source: TDWI, 2011; Wixom and Watson, Slide 15

16 Big Data: Where are we today? Big data is not mainstream yet. Innovators like google have pioneered the fundamental technologies. Vendors are pushing hard to bring these technologies to the Fortune Diffusions of innovations Source: Moore, Slide 16

17 Conclusion Big Data The end of traditional BI? Business Intelligence: Integration of internal, structured 1 data. There to stay. Big data: Big data goes beyond the capabilities of today s 2 BI systems, e.g. external, unstructured information. Conclusion Adoption: Enterprises are just starting to explore the 3 opportunities of Big Data. Slide 17

18 Big Data Big Data Where are Companies Today? Big Data Hype or Disruptive Innovation? Big Data A New Understanding of Volume, Velocity and Variety Big Data Converging Technology Streams Conclusion and Discussion Slide 18

19 Big Data Today People realize the fundamental changes Slide 19

20 Big Data Today A strategic initiative Slide 20

21 Q What are the potentials of Big Data for a retailer or retail bank? Question Slide 21

22 Variety: New information sources and customers Variety is more than capturing social media or web data. Variety means to systematically broadening and exploiting the information base with a clear business case in mind. External Information customer Postfinance Spend analysis for customers Business Intelligence Amazon Product reviews Cosibon Quality and return management Internal Internal External Source of information Slide 22

23 Variety: Example PostFinance Data Analysis for Customers. Slide 23

24 Variety: Example Amazon Leveraging customer generated data for own and customer purposes. Slide 24

25 Variety: Example Cosibon Exploiting new data sources Going Beyond Today s A Store Management B Going Beyond Today s Product Management Slide 25

26 Volume: Deeper understanding Often, it is not about more raw data but about moving from the average (customer segment analysis) to the individual FROM: THE AVERAGE TO: THE INDIVIDUAL Slide 26

27 Volume: Deeper understanding Example Telco: From sophisticated yet oversimplified analysis to more realistic and effective models. FROM: THE ISOLATED OBJECT TO: THE HUMAN BEEING Source: Ryals, Knox, Slide 27

28 Velocity: Where big data is really needed The sweat spot: Sub-second response time & continuous data load Response Time Source: ParStream, Import Latency Slide 28

29 Velocity is more than machine response time Slide 29

30 Big Data A New Understanding of the 3 Vs Conclusion Volume: Enterprises leverage the full potential of existing 1 data assets first. Velocity: Velocity has two dimensions, i.e. response time 2 and import latency. Reporting starts to go real time. Conclusion Variety: Variety means to systematically broadening and 3 exploiting the information base. Source: TDWI, 2011; Wixom and Watson, Slide 30

31 Big Data Big Data Where are Companies Today? Big Data Hype or Disruptive Innovation? Big Data A New Understanding of Volume, Velocity and Variety Big Data Converging Technology Streams Conclusion and Discussion Slide 31

32 Big Data Technology A young and fragmented market Slide 32

33 Big Data Technology Domains Three major big data technology domains do exist. Technology Domains A. Business Intelligence B. NoSQL & Content Intellig. C. Real Time Intelligence > > > > Domain of relational databases Innovation on the basis of main memory approaches Innovation on the basis of Hadoop capabilities Relational databases are there to stay > > > > Domain of Hadoop software stack HDFS and HBase for data storage MapReduce, Pig and Hive for data processing Lucene and Mahout for language processing > > > > Current discussion dominated by Kafka, Storm Kafka as a distributed, persistent message broker Storm as a real time computation system Both proven technologies but early Apache projects Structured Data Unstructured /structured Data Data Streams Slide 33

34 A. Business Intelligence Next generation business intelligence tools build upon three pillars. Leverage latest hardware Multi core & main memory Scale out capability Run on multiple nodes Intelligent databases Run calculations in DB Slide 34

35 Q What does «run calculations in DB» mean for the classical three tier architecture? Discussion Slide 35

36 A. Business Intelligence Impact of «Big Data» on Data Warehouse Landscape Established Architecture New Architecure repor)ng and analy)cal applica)ons repor)ng and analy)cal applica)ons Mart Mart Mart Mart ETL layer in-memory database data warehouse data warehouse ETL layer ETL layer source systems source systems Slide 36

37 B. Content Intelligence Hadoop is an open-source software framework and an Apache top-level project for storage and large scale processing of data-sets on clusters of commodity hardware. FUNDAMENTAL COMPONENTS Mahout Lucene MapReduce Pig Hive HDFS HBase Slide 37

38 B. Content Intelligence Short intro Map-reduce Source: Slide 38

39 B. Content Intelligence Short intro Map-reduce Map: How should the input data be processed? Result: key-value pairs elgar => 1 fleisch => 1 migros => 1 fleisch => 1 buendner => 1 fleisch => 1 {elgar => 1} {fleisch => 1, fleisch => 1, fleisch => 1} {migros => 1} {buendner => 1} Slide 39

40 B. Content Intelligence Short intro Map-reduce Reduce: How should the data be reduced? Result: key-value pairs {elgar => 1} {fleisch => 1, fleisch => 1, fleisch => 1} {migros => 1} {buendner => 1} {elgar => 1} {fleisch => 3} {migros => 1} {buendner => 1} Slide 40

41 C. Real Time Intelligence Storm is a real time computation system. It complements map-reduce. Source: Slide 41

42 Big Picture: Technology is Converging With Hadoop 2.0 Hadoop is not limited to batch processing any more, but can also be used e.g. for streaming and interactive applications. Source: hortonworks, Slide 42

43 Big Picture: Example ebay Different platforms for different user groups and use cases Slide 43

44 Big Picture: Example ebay There is no silver bullet to address all use cases. Each technology has its strength and weaknesses. Source: ebay, Slide 44

45 Big Data Converging Technology Streams Conclusion Big Data Market: The big data market is young and 1 fragmented. Data operating system: Hadoop is emerging as a data 2 operating system. Conclusion Beyond either or: There is no silver bullet to address all 3 use cases. Slide 45

46 Big Data Big Data Where are Companies Today? Big Data Hype or Disruptive Innovation? Big Data A New Understanding of Volume, Velocity and Variety Big Data Converging Technology Streams Conclusion and Discussion Slide 46

47 Conclusion Big Data The end of traditional BI? Hype or Disruptive Innovation: Big Data is disruptive. 1 However, it will not disruptive every use case. Use Cases: Big data goes beyond the capabilities of 2 today s BI systems: volume, velocity, variety Conclusion Key developments: There are three major Big Data 3 technology streams which have to converge. Slide 47

48 Institute of Technology Management Big Data Projektskizze «Competence Center Automotive Intelligence» Zurich, November 11th 23. September 2011 Robert Winter, Felix Wortmann Felix Wortmann Institut für Wirtschaftsinformatik Assistant Professor for Technology Management,