Big Data One size doesn t fit all. Dr. Jean-Laurent Philippe, PhD Directeur Avant-Vente, Intel EMEA ICAR 2013

Transcription

1 Big Data One size doesn t fit all Dr. Jean-Laurent Philippe, PhD Directeur Avant-Vente, Intel EMEA ICAR 2013

2 Legal Disclaimer INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. A "Mission Critical Application" is any application in which failure of the Intel Product could result, directly or indirectly, in personal injury or death. SHOULD YOU PURCHASE OR USE INTEL'S PRODUCTS FOR ANY SUCH MISSION CRITICAL APPLICATION, YOU SHALL INDEMNIFY AND HOLD INTEL AND ITS SUBSIDIARIES, SUBCONTRACTORS AND AFFILIATES, AND THE DIRECTORS, OFFICERS, AND EMPLOYEES OF EACH, HARMLESS AGAINST ALL CLAIMS COSTS, DAMAGES, AND EXPENSES AND REASONABLE ATTORNEYS' FEES ARISING OUT OF, DIRECTLY OR INDIRECTLY, ANY CLAIM OF PRODUCT LIABILITY, PERSONAL INJURY, OR DEATH ARISING IN ANY WAY OUT OF SUCH MISSION CRITICAL APPLICATION, WHETHER OR NOT INTEL OR ITS SUBCONTRACTOR WAS NEGLIGENT IN THE DESIGN, MANUFACTURE, OR WARNING OF THE INTEL PRODUCT OR ANY OF ITS PARTS. Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined". Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information. The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order. Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling , or go to: Copyright 2012 Intel Corporation.

3 Agenda Big Data as Insights that provides ROI Why work with Intel on Big Data 3

4 Evolution to Big Data Processing Date Paradigm Processing Style/ Scale Out Form Factor 90s Reporting / Data Mining High Cost / Isolated use Batch sales reports Sequential SQL queries RDMS Scale Multi-core 2000s Model-based discovery High Cost / Dept Use Batch-ie correlated buying pattern No SQL. parallel analysis Shared disk/memory Scale Node Node Node No SQL RDMS Proprietary MPP/ DW Appliance Today Unbounded Map Reduce Query Low Cost / Enterprise Use Arrival of vast amounts of unstructured data Real-time- ie recommend engine storage node Built-in data replication/reliability Shared nothing, in memory Unlimited Linear Scale Distributed node addition Open Source SW loosely coupled to commodity HW Node Node Node

5 Virtuous Cycle of Data-Driven Innovation 40 Zettabytes of data will be generated WW in Clients Cloud Richer user experiences Intelligent Systems Richer data to analyze 2.8 Zettabytes of data will be generated WW in (1) IDC Digital Universe 2020, (2) IDC Richer data from devices 5

6 Hadoop - One Tool in Building a Data Refinery Actionable Insights (Existing Biz Infra, Web Apps, Ops) Discover, Archive, Transact (EDW, NOSQL, DataMining) Refine, Explore, Enrich (MapReduce, HDFS) Big Data Sources & Capture (transactions, observations, interactions)

7 Variety - Right Data Methods For Right Data Structure Unstructured Multi-format Data Emerging Technologies Analytical Paradigms MapReduce /Hive Structured Data Relational Database EXALYTICS

8 TCO Comparisons Storage & Analytics Apache Hadoop on Xeon Nodes Typical Cost/node Storage (TBs) Initial Cost Annual Cost 3 year hardware & software cost 188 $12,500 2,600 $2,350,000 $564,000 $4,042,000 MPP Typical Cost/node Storage (TBs) Initial Cost Annual Cost 3 year hardware & software cost $50,000 2,600 $130,000,000 $26,000,000 $208,000,000

9 Characteristics of Big Data Categorize and Handle: Unstructured & Structured Data Marketing Automation CMS Data Velocity Real Time Near Real Time Need for Faster: Time to Insights SMS Audio Periodic Social Tech Automated Demand Generation Data Variety Video Reports Web Data Base Photo Table Batch KB MB GB TB PB Data Volume Scale to Comprehend: Massive Existing & Incoming Data 9

10 Forces Driving Big Data Advancement Intel TrueScale Infiniband HPC Enabling exascale computing on massive data sets Cloud Helping enterprises build open interoperable clouds Open Source Contributing code and fostering ecosystem * Other names and brands may be claimed as the property of others. 10

11 Enterprise and Big Data How is it used in different sectors? Big data can generate significant financial value across sectors US health care $300 billion value per year ~0.7 percent annual productivity growth Europe public sector administration 250 billion value per year ~0.5 percent annual productivity growth Global personal location data $100 billion+ revenue for service providers Up to $700 billion value to end users US retail 60+% increase in net margin possible percent annual productivity growth Manufacturing Up to 50 percent decrease in product development, assembly costs Up to 7 percent reduction in working capital FMCG: Problem solving ideas, product decisions, predict market reaction, enhance brand relevance Financial: To detect credit card fraud, greater accuracy and granularity in risk assessment Cable and TV: Customize TV ads to individual household Mobile Ads: Ads based on locations Retail: Insight and understanding of customer likes, dislikes, influences and behaviors General business: To discover unknown and untapped behaviors and attitudes Healthcare: Evidence based medicine Utilities: Understanding of individualized use patterns and better manage demand Online businesses: Better understanding of customer preferences and social interactions Crime prevention/intelligence: Better analysis of patterns Solving today s problems faster SOURCE: McKinsey Global Institute analysis SOURCE: McKinsey 11

12 The Potential of New Information Products The Experts The Field Only business model Tech has left. Forbes, 2011 Data are becoming the new raw material of business: an economic input almost on a par with capital and labor. The Economist, 2010 Information will be the oil of the 21st century. Gartner, 2010 Retail: increase margins 60% Manufacturing: 50% decrease in production costs Cellular: $150B to Providers Public Sector: $250B growth. McKinsey 2010 The 4 Vs Genomics Provider Billing Smart City Telco Billions of correlations for new patient discovery services Real time access to subscriber billing records Predictive traffic forecasting & license fraud detection New customer segmentation for realtime campaigns Unstructured datasets whose Volume, Variety, Velocity and Values is beyond typical database software tools to capture, store, manage and analyze. * Source: McKinsey Global Institute Analysis

13 The Four Pillars and Associated Challenges of Big Data Volume Variety Velocity Value Massive scale and growth of unstructured data 80%~90% of total data Growing 10x~50x faster than structured (relational) data 10x~100x of traditional data warehousing Heterogeneity and variable nature of Big Data Many different forms (text, document, image, video...) No schema or weak schema Inconsistent syntax and semantics Real-time rather than batch-style analysis Data streamed in, tortured, and discarded Making impact on the spot rather than after-the-fact Predictive analytics for future trends and patterns Deep, complex analysis (machine learning, statistic modeling, graph algorithms ) versus Traditional business intelligence (querying, reporting )

14 Intel Leverages the Power of Big Data Chip Design Validation: Cut Product Time to Market by 25% Faster analysis process for validating results Streamlined debug process through analysis of large volumes of historical test data Reseller Channel Management: Increased sales by $5M per Qtr. Decreased cost by $6M per Qtr. Smarter reseller engagement prioritization by leveraging advanced customer profile algorithms Cost efficient detection of non-complaint claims MALWARE Malware Detection: Proof of Concept (POC) Collecting and analyzing large amounts of server security data at the system, network, and application levels lead to discovery of new malware threats before they arise. MILLION new malware samples per quarter 1 CYBER ATTACKS 1 McAfee Threats Report: Second Quarter 2012, McAfee, (PDF) 2 Koebler, Jason, U.S. Nukes Face Up to 10 Million Cyber Attacks Daily, U.S. News & World Report (2012), MILLION U.S. cyber attacks per day 2 14

15 Reimagining the Possibilities with Big Data Analytics Move to Value & Vision Enhance understanding, drive innovation, and accelerate personalized medical cures Create new business models and transform organizational processes Enhance public safety and transportation, increase energy efficiency and reduce carbon footprint 15

16 Big Data is more than just an IT decision Customers face many decision points assessing a big data opportunity LOB may take the lead without engaging IT CMOs clearly driving discussions CMOs recently reported that the percent of their companies marketing budgets devoted to big data will increase from 6% to 10% over the next three years. Forbes, March 2013 Role of the data scientist Data is useless without asking the right questions Usage Models will be key drivers

17 Big Data Adoption and Deployment Phases Investigate Discover Plan Implement Understand business model Organization alignment Market & technology trends Define problem statement Identify business use cases Gather requirements Develop success metrics Identify high value, high visibility use cases Define scope & ROI for proof of concept (POC) Identify Big Data reference architecture Pilot the POC Promote and extend the POC result for other projects Extend and enhance more advanced analytic capabilities What insights would best benefit your business? What results are you really trying to get? What do you want to do with your data? What kind of data & correlations are you interested in mining? How much return are you expecting on your investment? What is your timeline for getting results? Are there other industries or uses you re using for a model? 17

18 Virtuous Cycle of Data Inside and Outside the Box Transform / Analyze Compute Data Move Networking Persist Storage 18

19 Analytics: Distributed and Disruptive Analytics at the Edge The Innovations in Open Source The Shift to In-Memory Taking us to near real-time insight * Other names and brands may be claimed as the property of others. 19

20 With a Variety of Delivery Mechanisms Workload Specific Appliances Best of Breed Building Blocks As-A-Service Aster Big Analytics Appliance Petabyte Cloud Open Source Packaged Apps PureData* & PureApplication* Systems Exadata*, Exalytics*, Big Data Appliance Supercomputing Analytic Services * Other names and brands may be claimed as the property of others. 20

21 Choice High Availability Secure Energy Efficient Responsive Big Data A Foundation For Delivering Big Value Big Data Building Blocks Compute Network Storage Software & Technologies Intel Xeon Product Family E3- E5-E7 Intel Atom Intel Ethernet Controllers Intel Ethernet Adapters Intelligent Storage 1 Scale-out Storage 1 Scale-up Storage 1 Intel Distribution for Apache Hadoop Intel Data Center Manager Intel Xeon Phi TM Intel Ethernet Switch Silicon Intel True Scale Fabric Intel SSD 710 series, DC S3700 (SATA) Intel SSD 910 series (PCIe) Intel Node Manager Intel Expressway Service Gateway Intel Cache Acceleration Software Intel s Lustre Intel VT and Intel TXT Intel AES-NI Intel s Foundational Technologies Offer Advanced Solutions for Big data Analytics Xeon-based storage systems are available in a wide range of configuration options from the industry s leading storage vendors

22 Intel Optimized Big Data Products Compute Intel Xeon processor E5-and E7 based servers up to 80% performance boost with hardwareenhanced security Storage Intelligent scale-out storage built with Intel Xeon processor E5-based storage NETWORK STORAGE COMPUTE Technical Compute TCP Intel Xeon Phi Integrated Systems Embedded Analysis Solutions From Intel ISG Fast Fabric & Caching Investing in New Fabric Approaches Non-Volatile Memory That Provide Capacity Caching for Data Velocity Network Intelligent scale-out Networking From 10GBe 40GBe Scaling Workloads Interconnect Efficiency Visibly Mobile Benchmark Proven Optimization Intel Software EcoSystem Lustre In-Memory In Stream Data Analysis End to End Security Performance Client Rich Modeling Support Client Server Application Management

23 Big Data Compute Platform Optimizations Intel Xeon E5 Family Intel Xeon E7 Family RAM QPI 1 Xeon E QPI 2 CORE 1 CORE 2 QPI 3 CORE 3 CORE 4 Integrated PCI Express* 3.0 Up to 40 lanes per socket Up to 4 channels DDR MHz memory Up to 8 cores Up to 20 MB cache QPI 4 4 QPI 1.0 Lanes for robust scalability CORE 5 CORE 6 CORE 7 CORE 8 CORE 9 CORE 10 CACHE Up to 8 channels DDR MHz memory Up to 10 cores Up to 30 MB cache SCALE-OUT with Hadoop and analytic/dw engines SCALE-UP in-memory analytic engines and databases: Oracle*, SAS*, SAP Hana* Proof point: E5 Analytics 25X Improvement Hadoop on E5 Proof point: SAP HANA 23

24 Intel Ethernet Reduces Time to Process Large Data Sets Trends and Challenges Big data is hitting the enterprise with unprecedented volume, velocity, variety, complexity, and OPPORTUNITY 1GbE Network Connections VM VM VM Hypervisor VM VM VM Hypervisor Intel Ethernet Solution 10 Ports 1GbE 2 Ports 10GbE Up to 20x performance boost over legacy infrastructure with optimizations on Intel Xeon processors, Intel SSD storage, and 10Gb Intel Ethernet networking 10 Gigabit Ethernet allows quicker import and export of large data sets for processing Moving the Data with 10GbE Up to 80% Reduction in Cables & Switch ports Up to 15% Reduction in Infrastructure Costs Up to 2x Improved Bandwidth per Server

25 Intel CAS with Intel SSD Solution Added as cache layer accelerates Big Data workloads 50X IOPS 3X TPC-C 20X TPC-H Performance near equal to replacing all hard drives with SSDs at significantly lower cost throughput performance

26 Data Methods for the Right Data Structure Unstructured Data Emerging Technologies Analytical Paradigms MapReduce /Hive Structured Data Relational Database E X A L Y T I C S * Other names and brands may be claimed as the property of others. 26

27 Intel Distribution for Apache Hadoop* & Tools HiTune (URL) MapReduce File-based Encryption in HDFS Up to 20x faster decryption with AES-NI* Role-based access control for Hadoop services Instrument Aggregation Engine Report Engine Up to 8.5X faster Hive queries using HBase co-processor Optimized for SSD with Cache Acceleration Software Adaptive replication in HDFS and HBase Integrated text search with Lucene HiTune Controller HiBench (URL) 1 2 Simplified deployment & comprehensive monitoring Deployment of HBase across multiple datacenters Automated configuration with Intel Active Tuner Detailed profiling of Hadoop jobs Simplified design of HBase schemas (+ in 2.4) REST APIs for deployment and management (+ in 2.4) Micro Benchmarks Sort WordCount TeraSort 3 Machine Learning Bayesian Classification K-Means Clustering HiBench Web Search Nutch Indexing Page Rank Result = many Hadoop optimization tips (IDF2012 presentation Big Data Analytics on a Performance-optimized Hadoop Infrastructure ) 4 HDFS Enhanced DFSIO 27

28 Unleash the power of platform TeraSort for 1TB sort: >4 hour process time Nearly 50x increase in your ability to discover insights Upgrade processor Intel Xeon 5600 HDD 1GbE ~50% reduction Upgrade to SSD ~80% reduction Upgrade to 10GbE ~50% reduction Hadoop processing time: <10 minutes with complete Intel-based solution Intel distribution ~40% reduction 28 Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. Source: Intel Internal testing For more information go to : intel.com/performance ` *Other brands and names are the property of their respective owners Whitepaper

29 Intel Intelligence at the Edge Intel Intelligent Systems Framework: Simplifying the Internet of Things Wind River Intelligent Device Platform Driving Secure Interoperability Unlocking Edge Data Filtering Data Connectivity Manageability Security Billions of devices that need to share data with each other and the cloud Edge systems need to react to streaming data in real time Data volume outpacing network and storage efficiency Pre-integrated smart and connected capabilities enable rich network options to save development time and costs Validated and flexible firmware providing an extensive network of connectivity choices, including broad modem support and PAN, LAN, and WAN network access Platform customization significantly reduces time to product while increasing productive life of M2M devices Intuitive web-based tool reduces configuration and support costs and allows for anytime provisioning and management of devices Dynamic post dynamic Services framework (OSGi) enables modularized, hardware agnostic deployment of new apps Security features designed for M2M development that protect critical data throughout the device lifecycle Customizable SRM to ensure the integrity of the end devices via secure boot, provide encrypted communication between device and management console in the cloud, and offer device resource management to limit system exposure of untrusted applications 29

30 Big Data A Foundation For Delivering Big Value Intel Data Research Worry-Free Data Protections to allow you to control how your data is used Future Data Experiences Data Economy, Vibrant Data, Data Visualization Analytical Applications Model-Based workloads, Everyday Analytics Machine Learning Algorithms Relationship Discovery, Real-time learning GraphLab, GraphBuilder Distributed Data Computing Graph, parallel, statistical computing partitioned across cloud, client and edge Data Ecosystem Infrastructure Platforms, architectures, DBMSs, smart networks, and the Internet of Things Dozens of Academic Industry & Gov Research Collaborations 30 Disaggregation / Silicon Photonics

31 Lustre for Big Data Technical Computing Evaluating Hadoop + Lustre configurations for technical computing and other high-performance, parallel opportunities Intel Manager for Lustre Simple provisioning, monitoring and management Browser, CLI and REST interfaces, fully scriptable Lowers complexity and costs, highly extensible Priced per number of attached storage devices Lustre File System Proven sustained performance at massive scale Flexible, scale-up and scale-out solutions Fee-paid development of new features and enhancements Intel Support Services for Lustre Unrivaled Lustre support expertise Professional services offerings ensures success Priced per storage server 31

32 Summary: Intel in Big Data The pervasiveness of Intel Architecture democratizes the implementation and performance of Big Data everywhere Optimized ISV software stacks and services Accelerate analytics: CPU, storage, and network Foster the growth of market partners Solution research and academia engagement Distribute analytics to the edge 32

33

34 Legal Disclaimers All products, computer systems, dates, and figures specified are preliminary based on current expectations, and are subject to change without notice. Intel processor numbers are not a measure of performance. Processor numbers differentiate features within each processor family, not across different processor families. Go to: Intel, processors, chipsets, and desktop boards may contain design defects or errors known as errata, which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel Virtualization Technology requires a computer system with an enabled Intel processor, BIOS, virtual machine monitor (VMM). Functionality, performance or other benefits will vary depending on hardware and software configurations. Software applications may not be compatible with all operating systems. Consult your PC manufacturer. For more information, visit No computer system can provide absolute security under all conditions. Intel Trusted Execution Technology (Intel TXT) requires a computer system with Intel Virtualization Technology, an Intel TXT-enabled processor, chipset, BIOS, Authenticated Code Modules and an Intel TXT-compatible measured launched environment (MLE). Intel TXT also requires the system to contain a TPM v1.s. For more information, visit Requires a system with Intel Turbo Boost Technology capability. Consult your PC manufacturer. Performance varies depending on hardware, software and system configuration. For more information, visit Intel AES-NI requires a computer system with an AES-NI enabled processor, as well as non-intel software to execute the instructions in the correct sequence. AES-NI is available on select Intel processors. For availability, consult your reseller or system manufacturer. For more information, see Intel product is manufactured on a lead-free process. Lead is below 1000 PPM per EU RoHS directive (2002/95/EC, Annex A). No exemptions required Halogen-free: Applies only to halogenated flame retardants and PVC in components. Halogens are below 900ppm bromine and 900ppm chlorine. Intel, Intel Xeon, Intel Core microarchitecture, the Intel Xeon logo and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. Copyright 2011, Intel Corporation. All rights reserved. 34

35 Legal Disclaimers: Performance Performance tests and ratings are measured using specific computer systems and/or components and reflect the approximate performance of Intel products as measured by those tests. Any difference in system hardware or software design or configuration may affect actual performance. Buyers should consult other sources of information to evaluate the performance of systems or components they are considering purchasing. For more information on performance tests and on the performance of Intel products, Go to: Intel does not control or audit the design or implementation of third party benchmarks or Web sites referenced in this document. Intel encourages all of its customers to visit the referenced Web sites or others where similar performance benchmarks are reported and confirm whether the referenced benchmarks are accurate and reflect performance of systems available for purchase. Relative performance is calculated by assigning a baseline value of 1.0 to one benchmark result, and then dividing the actual benchmark result for the baseline platform into each of the specific benchmark results of each of the other platforms, and assigning them a relative performance number that correlates with the performance improvements reported. SPEC, SPECint, SPECfp, SPECrate. SPECpower, SPECjAppServer, SPECjEnterprise, SPECjbb, SPECompM, SPECompL, and SPEC MPI are trademarks of the Standard Performance Evaluation Corporation. See for more information. TPC Benchmark is a trademark of the Transaction Processing Council. See for more information. SAP and SAP NetWeaver are the registered trademarks of SAP AG in Germany and in several other countries. See for more information. INFORMATION IN THIS DOCUMENT IS PROVIDED AS IS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO THIS INFORMATION INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. Performance tests and ratings are measured using specific computer systems and/or components and reflect the approximate performance of Intel products as measured by those tests. Any difference in system hardware or software design or configuration may affect actual performance. Buyers should consult other sources of information to evaluate the performance of systems or components they are considering purchasing. For more information on performance tests and on the performance of Intel products, reference 35

36 Optimization Notice Intel's compilers may or may not optimize to the same degree for non-intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Notice revision #