BIG DATA AND MICROSOFT. Susie Adams CTO Microsoft Federal

Transcription

1 BIG DATA AND MICROSOFT Susie Adams CTO Microsoft Federal

2 THE WORLD OF DATA IS CHANGING Cloud

3 What s making this possible? Electrical efficiency of computers doubles every year and ½. Laptops and mobile phones own their existence to thistrend trend. We now have wireless no battery sensors. So where are we today? If a MacBook Air operated at the efficiency of 1991 it would last 2.5 sec. This efficiency will drive innovation and revolutionize how we collect, analyze and make better decisions off of data.

4 Four Megatrends Will Dominate The Next Decade Gartner By 2020, mobile, cloud, social and big data technologies will drive at least 80% of the industry s growth CIO's increasingly see technologies such as analytics/business intelligence, mobility, cloud and social in combination rather than isolation to address business priorities. IDC In 2012 the ICT industry's shift to its third In 2012, the ICT industry s shift to its third major platform of growth built on mobile, cloud, social, and Big Data technologies will accelerate.

5 Who Uses Hadoop? source:

6 New Big Data Applications Data Mining Fraud detection Predictive Analytics Spam detection Machine Learning High energy physics Pattern Recognition Log analysis Threat Analysis Patient medical record analysis Critical infrastructure security and Recommendation engines operational analysis Price aggregation Ad campaign analytics Satellite image processing Graph creation Search engines Page ranking Semantic text analysis Gaming player behavior analysis Transcoding video on demand Genomic analysis Wb Web crawling

7 Gartner Big Data/BI: Research Highlights g Unstructured data will grow some 80% over the course of the next five years. Analytics/business siness intelligence was the topranked technology for 2012 as CIOs are combining analytics with other technologies to create new capabilities. 80% Organizations are finding ways to turn the explosion in size, volume and complexity of data into insight and value. This is occurring across structured and unstructured content from internal and external sources. IDG Perhaps the biggest challenge facing those pursuing big data is getting a platform that can store and access all current and future information and make it available online for analysis cost-effectively. Big Data will join mobile and cloud as the next "must Deloitte 2.7 have" competency as the volume of digital content grows to 2.7ZB (1ZB = 1 billion terabytes) in 2012, up ZETABYTES 48% from 2011, rocketing toward 8ZB by IN 2012 IDC Two-thirds of all mobile apps developed in 2012 will be integrated with analytics offerings. McKinsey three barriers [to more effective use of data and analytics in decision making] : a cultural preference for experience over data; a lack of skills in synthesizing and translating the analytics and data for decision makers; and concerns that the data quality is poor. We predict big data and analytics will be the CIO issue of the year in InformationWeek big data analytics isn't an out-of-the-box software solution." Companies will need to hire or contract people with chops in math and statistics, in addition to buying the latest software tools. data analytics is moving from a wonkish craft to a mainstream discipline.

8 The Three V s of Big Data Volume Storage challenge Massive Analysis challenge Variety Tabular, Hierarchical, Telemetry, Metering, etc. Velocity 56% YoY Data Growth Currently producing 10 Zettabytes (Billion Trillion) per year and growing g Biggest growth is in unstructured data 80% of new data is big data

9 The Value Is Not The Data The value is Deriving new insights Recognizing relationships Making increasingly accurate predictions

10 Big data requires a holistic approach Thatleverages TRADITIONALANDNEWCAPABILITIES AND NEW TRADITIONAL Relational Database Management System NEW Petabyte-Scale Services

11 Multiple Components: Big Data Challenges. Hadoop Common Hadoop Distributed File System (HDFS ) Hadoop MapReduce Sub Projects: Avro Cassandra Chukwa HBase Hive Mahout Pig ZooKeeper Hadoop (Yahoo) MapReduce (Google) HDFS Big Table (Google) Mahout Pegasus Pig Hive Cassandra Sqoop Linux GFS Apache NoSQL Zookeeper Lack of Skillsets. Hbase Chukwa Avro

12 Make Big Data accessible to everyone Focus on your Mission not your Infrastructure! Immersive Insight, Wherever you are Analyze Big Data with familiar tools Immersive insights from any data JavaScript based simple programming Connecting with the World s Data Share your data with the world via Azure Marketplace Enrich with social media data via Social Analytics Advanced analytics with Hadoop Any Data, Any Size Anywhere Simplicity it and manageability of Windows to Hadoop Extended data warehousing with Hadoop Scale & elasticity of cloud

13 We deliver insights to everyone by enabling big data analysis with familiar enduser tools Benef fits Interaction and analysis of unstructured data in Hadoop from Microsoft Excel Key Fea atures Hive add-in for Excel

14 unlocking new insights from all data with Microsoft BI tools Benef fits Key Fea atures Familiar BI tools with structured and unstructured data Hive ODBC Driver integrates Hadoop to SQL Server Analysis Services, PowerPivot, and Power View

15 Microsoft uniquely Connects Hadoop to the world via Windows Azure Marketplace Benefits Sharing of data and insights through Windows Azure Marketplace Mashing up of internal and public data sets via Data Explorer Ke ey Feature es Integration with Windows Azure Marketplace Integration with third-party data and services

16 Enriches analysis with social media data via social analytics Benefits Stronger customer relationships Models augmented with publicly available data from social media sites Integration of social information with business applications Ke ey Feature es Integration with social media sites Microsoft Codename "Social Analytics"

17 Microsoft brings the simplicity and manageability of windows to hadoop Benefits Simplified management of Hadoop on Windows Enterprise-class security Easy setup on-premises and in the cloud Key Features Integration ti with Microsoft Integration ti with Windows System Center Server Active Directory Smart packaging of Hadoop on premises Fast deployment of Hadoop on Azure

18 While extending your Enterprise Data Warehouse with hadoop Benefits Integration with enterprise BI solutions Integration with Microsoft Enterprise Data Warehouses Deeper insights from structured and unstructured data Key Feature es Microsoft SQL Server connector for Apache Hadoop with SQOOP (SQL to Hadoop) SQL Server Parallel Data Warehouse connector for Apache Hadoop with SQOOP

19 A holistic BIG DATA Solution from Microsoft spanningrelational and non relational Worlds SELF-SERVICE OPERATIONAL PREDICTIVE DISCOVER AND RECOMMEND INSIGHTS DATA ENRICHMENT TRANSFORM AND CLEAN MOBILE REAL-TIME COLLABORATIVE SHARE AND GOVERN External Data a and Services MARKETPLA ACE DATA MANAGEMENT RELATIONAL NON-RELATIONAL MULTIDIMENSIONAL STREAMING

20

21

22 Hadoop on Windows & Azure: Roadmap INSIGHTS Excel Integration Preview 2 Hive Add-in for Excel PowerPivot Add-in for Excel Power View for SharePoint DATA ENRICHMENT Hadoop Connectors Azure Data Market Hive ODBC Driver Preview 2 Azure Labs Data Explorer Social Analytics Data Hub (Private Data Market) DATA MANAGEMENT Hadoop on Azure Private CTP Hadoop on Azure Preview 2 Hadoop on Server Private TAP More capacity Hadoop Core & Common Disaster Recovery for HDFS JavaScript Framework Support for Mahout Hadoop on Azure GA Portal Integration & Billing Azure SDK integration Hadoop on Server GA JavaScript, PIG, Hive, Hbase Active Directory Integration Systems Center Integration CY H

23 This Innovation Will enable us to control industrial processes with more precision. ii Assess the results of our actions quickly and effectively Rapidly reinvent our institutions and business models dl to reflect new realities. So that t we can test tour assumptions with real dt data in real time, and modify those assumptions as reality dictates.

24 CREATING NEW BUSINESS OPPORTUNITIES Increases ad revenue by processing 3.5 billion events per day Massive Volumes Measures and ranks online user influence by processing 3 billion signals per day Cloud Connectivity Uses sentiment analysis and web analytics for its internal cloud Real-Time Insight Processes 464 billion rows per quarter, with average query time under 10 secs. Connects across 13 social networks via the cloud for data and API access Improves operational decision making for IT managers and users 1. Klout Case Study: SQL Server 2012 Enterprise/Klout/Data Services Firm Uses Microsoft BI and Hadoop to Boost Insight into Big Data/

25 Microsoft EDW Azure on-prem connectors txt ERP CRM LOB Apps