Microsoft Big Data 解決方案與案例分享

Transcription

1 DBI 312 Microsoft Big Data 解決方案與案例分享 Rich Ho Technical Architect 微軟技術中心

2 Agenda What is Big Data? Microsoft Big Data Strategy Key Benefits of Microsoft Big Data Demo Case Study

3 What is Big Data?

4 The world of Data is changing 10x increase every five years Data explosion 85% from new data types Volume Velocity Variety Hadoop Easy Accessibility of External Data Cheap, Distributed Storage & Processing Cloud By 2015, organizations that build a modern information management system will outperform their peers financially by 20 percent. Gartner, Mark Beyer, Information Management in the 21st Century

5 Three V in Big Data 10x increase every connected VOLUME VELOCITY years devices per adult Relational Data VARIETY 85% from new data types Big Data

6 We are moving to Big Data Terabytes Data Volume Petabytes+ Structured Data Variety Unstructured Batch Data Velocity Streaming Garbage? Traditional Data Big Data Any Data, Any Size, Anywhere

7 With Big Data, says: 以前我們將尿布和啤酒 放在一起而創造新業績 現在我們將比老爸更早 一步知道他女兒懷孕了!

8 Traditional data flow

9 New exploratory data flow

10 What is Big Data? Increasing Data Volumes Increasing Data & Analysis Complexing Emerging Technologies

11 Microsoft Big Data Strategy

12 Microsoft Big Data Stacks SELF-SERVICE INSIGHTS MOBILE REAL-TIME PREDICTIVE COLLABORATIVE DATA ENRICHMENT DISCOVER AND RECOMMEND TRANSFORM AND CLEAN SHARE AND GOVERN DATA MANAGEMENT RELATIONAL NON-RELATIONAL MULTIDIMENSIONAL STREAMING MARKETPLACE External Data and Services OPERATIONAL

13 Enterprise Big Data Blueprint Big Data Sources (Raw, Unstructured) Data & Compute Intensive Application Business Insights Data Marts Sensors Load Summarize & Load Devices Fast Reporting Distributed environments for Non-structured data Bots Enterprise Data Warehouse Historical Data (Beyond Active Window) Interactive Reports Integrate/Enrich Multi-dimensional Analysis Crawlers Performance Scorecards Enterprise ETL with SSIS, DQS, MDS ERP CRM LOB Source Systems APPS Online Transaction

14 Hadoop on Windows Hadoop on Azure

15 Microsoft Big Data Platform Big Data Sources (Raw, Unstructured) Data & Compute Intensive Application Business Insights Data Marts Sensors Load Summarize & Load Devices Fast Reporting Hadoop on Windows Azure Bots Hadoop on Windows Server Enterprise Data Warehouse Historical Data (Beyond Active Window) Interactive Reports Integrate/Enrich Multi-dimensional Analysis Crawlers Performance Scorecards Enterprise ETL with SSIS, DQS, MDS ERP CRM LOB Source Systems APPS Online Transaction

16 Microsoft Big Data Platform SQL Server PDW Big Data Sources (Raw, Unstructured) SQL Server FT Data & Compute Intensive Application Business Insights Data Marts Sensors Load Summarize & Load Devices Fast Reporting Hadoop on Windows Azure Bots Hadoop on Windows Server Enterprise Data Warehouse Historical Data (Beyond Active Window) Interactive Reports Integrate/Enrich Multi-dimensional Analysis Crawlers Performance Scorecards Enterprise ETL with SSIS, DQS, MDS SQL Server EE ERP CRM LOB Source Systems APPS Online Transaction

17 Microsoft Big Data on Public Cloud Hadoop on Windows Azure Big Data Sources (Raw, Unstructured) SQL Azure Data & Compute Intensive Application Business Insights Data Marts Sensors Load Summarize & Load Devices Fast Reporting Hadoop on Windows Server Bots Enterprise Data Warehouse Historical Data (Beyond Active Window) Interactive Reports Integrate/Enrich Multi-dimensional Analysis Crawlers Performance Scorecards Enterprise ETL with SSIS, DQS, MDS ERP CRM LOB Source Systems APPS Online Transaction

18 Hadoop on Azure (CTP)

19 Request a New Cluster (4 nodes, 2 TB)

20 Allocating NN & DN

21 Starting NN & DN

22 NN & DN ready! (after 9 mins)

23 Hadoop on Azure ready! (after 13 min s)

24 Access Hadoop on Azure through RDP

25 Key Benefits of Microsoft Big Data

26 Microsoft Big Data Benefits Breakthrough Insights Broader Access of Hadoop Enterprise Ready Analyze Big Data with familiar tools (Excel, PowerPivot, Power View) New Hadoop-based distribution on Windows / Azure System Center Integration JavaScript based simple programming Hive ODBC driver Hive add-in for Excel Active Directory Integration

27 Breakthrough Insights Benefits Breakthrough Insights Key Features Hive ODBC Driver integrates Hadoop to SQL Server Analysis Ser vices, PowerPivot, and Power View

28 Broader Access of Hadoop Benefits Broader Access of Hadoop Key Features Hive add-in for Excel Hive ODBC driver

29 Enterprise Ready (SCOM Integration) Benefits Enterprise Ready Key Features System Center to monitor performances of Head nodes and Data nodes

30 1. Hadoop on Azure - JavaScript Console - Job Execution 2. Excel Hive Connectivity 3. Windows Marketplace & Hadoop on Azure Integration

31 Case Study: Mining Big Mobility Data

32 Microsoft Big Data Case Study: T-Drive

33 Microsoft Big Data Case Study: T-Drive

34 Video Microsoft Research Asia Big Data Case Study

35 Thank You

36 Resources Connect. Share. Discusss Microsoft Certification & Training Resources Resources for IT Professionals Resources for Developers / /

37

38 請協助完成 本課程問卷 並在離開 教室時交給工作人員 填妥大會背包中的大會問卷 可於活動 第三天兌換問卷禮哦 感謝您的合作

39