AGENDA. What is BIG DATA? What is Hadoop? Why Microsoft? The Microsoft BIG DATA story. Our BIG DATA Roadmap. Hadoop PDW

Transcription

1

2 AGENDA What is BIG DATA? What is Hadoop? Why Microsoft? The Microsoft BIG DATA story Hadoop PDW Our BIG DATA Roadmap

3 BIG DATA? Volume 59% growth in annual WW information 1.2M Zetabytes (10 21 bytes) this year growing 44X in the next decade

4 BIG DATA? Volume Velocity In 2000 the Sloan Digital Sky Survey collected more data in its 1 st week than was collected in the entire history of Astronomy By 2016 the New Large Synoptic Survey Telescope in Chile will acquire 140 terabytes in 5 days - more than Sloan acquired in 10 years The Large Hadron Collider at CERN generates 40 terabytes of data every second

5 BIG DATA? Volume Velocity Variety Structured & Unstructured Data Sensors, Devices, Bots, Crawlers, Twitter, Facebook, Social Trends,

6 AGENDA What is BIG DATA? What is Hadoop? Why Microsoft? The Microsoft BIG DATA story Hadoop PDW Our BIG DATA Roadmap

7 HADOOP OVERVIEW The Apache Hadoop project develops open-source software for reliable, scalable, distributed computing Current most popular distributed storage and computing system in Online world Hadoop known users: AOL ebay Facebook Google Twitter Yahoo Baidu Hulu Alibaba/Taobao

8 THINKING AT SCALE Facebook Hadoop Data Warehouse Capacity (May 2010) 21 PB of storage in a single HDFS cluster 2000 machines 12 TB per machine (a few machines have 24 TB each) 12 TB of compressed data added per day 800 TB of compressed data scanned per day 25,000 map-reduce jobs per day 65 millions files in HDFS

9 TRADITIONAL E-COMMERCE DATA FLOW

10 NEW EXPLORATORY E-COMMERCE DATA FLOW

11 SO HOW DOES IT WORK? FIRST, STORE THE DATA

12 SO HOW DOES IT WORK? SECOND, TAKE THE PROCESSING TO THE DATA // Map Reduce function in JavaScript var map = function (key, value, context) { var words = value.split(/[^a-za-z]/); for (var i = 0; i < words.length; i++) { if (words[i]!== "") {context.write(words[i].to LowerCase(), 1);} }}; var reduce = function (key, values, context) { var sum = 0; while (values.hasnext()) { sum += parseint(values.next()); } context.write(key, sum); };

13 HADOOP ARCHITECTURE Task tracker Task tracker Map Reduce Layer HDFS Layer Job tracker Name node Data node Data node Reference:

14 TRADITIONAL RDBMS VS MAP REDUCE Traditional RDBMS MapReduce Data Size Gigabytes (Terabytes) Petabytes (Exabytes) Access Interactive and Batch Batch Updates Read / Write many times Write once, Read many times Structure Static Schema Dynamic Schema Integrity High (ACID) Low Scaling Nonlinear Linear Reference: Tom White s Hadoop: The Definitive Guide

15 THE POTENTIAL: SOLVING SPECIFIC INDUSTRY PROBLEMS ecommerce: mining web logs: collaborative filtering, user experience optimisation Manufacturing: detecting trends and anomalies in sensor data: predicting and understanding faults Capital Markets: joining market and external data: correlation detection for investment strategy identification, risk calculations Retail Banking: historical transaction mining: fraud detection, customer segmentation Health: research in tissue density, anomalies, pattern analysis etc.

16 AGENDA What is BIG DATA? What is Hadoop? Why Microsoft? The Microsoft BIG DATA story Hadoop PDW Our BIG DATA Roadmap

17 WHY MICROSOFT? 500M Active Live IDs 59 markets and 36 languages 355M Active Accounts Over 4B WW Queries Each Month 100 PB Over 459M Unique Users 14B Ads Per Month Over 303M Users 76 markets and 48 languages Over 6M Songs In The Catalog 25M Users 2-4 billion s per day 200+ CLOUD SERVICES

18 WHAT IS AZURE?

19 AGENDA What is BIG DATA? What is Hadoop? Why Microsoft? The Microsoft BIG DATA story Hadoop PDW Our BIG DATA Roadmap

20 MICROSOFT BIG DATA SOLUTION PowerView Excel with PowerPivot Predictive Analytics Familiar End User Tools Embedded BI SSAS BI Platform S S R S Hadoop On Windows Azure Hadoop On Windows Server Connectors Microsoft PDW Sensors Devices Bots Crawlers Unstructured and Structured Data ERP CRM LOB APPs

21 MICROSOFT HADOOP STRATEGY Submit changes back to Apache Foundation Just works on Windows Azure and Server Integration with Visual Studio Generic based Framework for.net Hadoop MapReduce Job Submission work-for-composing-af656ef7 Performance, Scale, High Availability Management, Ease of use Security, Data Governance Integration with AD and SC. Integration with SQL Server

22 APACHE HADOOP AS A SERVICE ON AZURE

23 Parallel Data Warehouse (PDW) Complete Business Analytics Platform Massive Scale & Performance Simplicity & Value 23

24 A Complete Business Analytic Platform Integration with SQL Server BI Complementary tools Non-Microsoft BI & ETL Tools SQL Server Integration Services, Analysis Services PowerPivot & Power View Hadoop StreamInsight Master Data Services Informatica Business Objects Microstrategy and SAS 24

25 Massive Scale and Performance Complete Business Analytics Platform Massive Scale & Performance Simplicity & Value Benefits 25

26 Parallel Data Warehouse is a True MPP Appliance With a Shared nothing architecture Control Rack Data Racks SQL SQL Client Drivers SQL SQL SQL SQL Support/Patching SQL ETL Load Interface SQL SQL SQL Corporate Backup Solution SQL CONTROL RACK DATA RACK 26

27 Microsoft offers Optimized Data Warehouse Solutions all built on SQL Server to meet specific customer needs Software Reference Architectures Appliances Fast Track for Dell Parallel Data Warehouse HP Enterprise Data Warehouse Dell Quickstart Data Warehouse HP Business Data Warehouse 27

28 AGENDA What is BIG DATA? What is Hadoop? Why Microsoft? The Microsoft BIG DATA story Hadoop PDW Our BIG DATA Roadmap

29 MICROSOFT BIG DATA ROADMAP Microsoft is extending its leadership in business intelligence and data warehousing to provide insights to all users by activating new types of data of any size. Microsoft is announcing an end-to-end roadmap for Big Data that embraces Apache Hadoop TM by distributing enterprise class Hadoop based solutions on both Windows Server and Windows Azure. To accelerate the delivery of Microsoft s Hadoop based solution for Windows Server and service for Windows Azure, Microsoft has announced a partnership with Hortonworks. Microsoft is committed to broadening accessibility and usage of hadoop to end users, developers and IT professionals in organizations of all sizes. On the horizon: Support for YARN (Hadoop/MapReduce 2.0) Real time queries on Hadoop data!

30