Big Data. Value, use cases and architectures. Petar Torre Lead Architect Service Provider Group. Dubrovnik, Croatia, South East Europe May, 2013

Transcription

1 Dubrovnik, Croatia, South East Europe May, 2013 Big Data Value, use cases and architectures Petar Torre Lead Architect Service Provider Group Cisco and/or its affiliates. All rights reserved. Cisco Connect 1

3 Legal Disclaimer Intel may make changes to specifications and product descriptions at any time, without notice. Performance tests and ratings are measured using specific computer systems and/or components and reflect the approximate performance of Intel products as measured by those tests. Any difference in system hardware or software design or configuration may affect actual performance. Buyers should consult other sources of information to evaluate the performance of systems or components they are considering purchasing. For more information on performance tests and on the performance of Intel products, visit Intel Performance Benchmark Limitations Intel does not control or audit the design or implementation of third party benchmarks or Web sites referenced in this document. Intel encourages all of its customers to visit the referenced Web sites or others where similar performance benchmarks are reported and confirm whether the referenced benchmarks are accurate and reflect performance of systems available for purchase. Intel processor numbers are not a measure of performance. Processor numbers differentiate features within each processor family, not across different processor families. See for details. Intel, processors, chipsets, and desktop boards may contain design defects or errors known as errata, which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel Virtualization Technology requires a computer system with a processor, chipset, BIOS, virtual machine monitor (VMM) and applications enabled for virtualization technology. Functionality, performance or other virtualization technology benefits will vary depending on hardware and software configurations. Virtualization technologyenabled BIOS and VMM applications are currently in development. 64-bit computing on Intel architecture requires a computer system with a processor, chipset, BIOS, operating system, device drivers and applications enabled for Intel 64 architecture. Performance will vary depending on your hardware and software configurations. Consult with your system vendor for more information. Intel, Intel Xeon, Intel Core microarchitecture, and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. 3 Copyright 2013 Intel Corporation. All rights reserved

5 Possibilities with Big Data Analytics Move to Value & Vision Enhance understanding, drive innovation, and accelerate personalized medical cures Create new business models and transform organizational processes Enhance public safety and transportation, increase energy efficiency and reduce carbon footprint 5 Copyright 2013 Intel Corporation. All rights reserved

6 Data Refinery Evolution Traditional Data Analysis Big Data Analysis Structured Transaction Relational Database Data Warehouse Analyze Unstructured Streamed Data Node Node Organize Analyze Batch Cluster SQL Devices MapReduce Hive Volume? Gigabytes to Terabytes Petabytes and Beyond Velocity? Batch Real-Time Data Analytics Variety? Centralized, Data Moves to Analytics Distributed, Analytics Moves to Data Value? Reactive, Query, Reporting, Proprietary Predictive Machine Learning, Optimized Algorithms, on Standard HW 6 Copyright 2013 Intel Corporation. All rights reserved

7 Data Refinery Evolution Traditional Data Analysis Big Data Analysis Structured Transaction Relational Database Batch Data Warehouse Analyze Unstructured Streamed Data Node Node Cluster Organize Analyze SQL Devices MapReduce Hive Gigabytes to Terabytes Volume Traditional Processing $50K Petabytes and Beyond Velocity Batch Data Processing Costs Per TeraByte Real-Time Data Analytics Variety Value Centralized, Data Moves to Analytics Big Data with Hadoop < $1K Reactive, Query, Reporting, Proprietary Distributed, Analytics Moves to Data Predictive Machine Learning, Optimized Algorithms, on Standard HW 7 Copyright 2013 Intel Corporation. All rights reserved

9 Hadoop case study: China Mobile Group, Guangdong province Challenge: Deliver real time access to Call Data Records (CDR) for billing self service Solution: Chose Hadoop + Xeon over RDMS to remove data access bottlenecks, increase storage, and scale system Benefits: Lower TCO, 30x performance increase, stable operation, analytics on subscriber usage for targeted promotions Data Characteristics: 30TB billing data/month Real-time retrieval of 30 days CDRs 300k records/second, 800k insert speed/sec 15 analytics queries 133 server nodes $57k/TB for traditional MPP versus <$1k/TB for Hadoop big data Analytics 9 Copyright 2013 Intel Corporation. All rights reserved

10 Big Data use cases Chip Design Validation: Cut Product Time to Market by 25% Faster analysis process for validating results Streamlined debug process through analysis of large volumes of historical test data Reseller Channel Management: Increased sales by $5M per Qtr. Decreased cost by $6M per Qtr. Smarter reseller engagement prioritization by leveraging advanced customer profile algorithms Cost efficient detection of non-complaint claims Malware Detection: Proof of Concept (POC) Collecting and analyzing large amounts of server security data at the system, network, and application levels lead to discovery of new malware threats before they arise. 1 McAfee Threats Report: Second Quarter 2012, McAfee, (PDF) 2 Koebler, Jason, U.S. Nukes Face Up to 10 Million Cyber Attacks Daily, U.S. News & World Report (2012), www/us/en/itmanagement/intel-it-bestpractices/mining-big-data-in- the-enterprise-for-betterbusiness-intelligence.html MALWARE MILLION new malware samples per quarter 1 CYBER ATTACKS MILLION U.S. cyber attacks per day 2 10 Copyright 2013 Intel Corporation. All rights reserved

12 Big Data Adoption and Deployment Phases Investigate Discover Plan Implement Understand business model Organization alignment Market & technology trends Define problem statement Identify business use cases Gather requirements Develop success metrics Identify high value, high visibility use cases Define scope & ROI for proof of concept (POC) Identify Big Data reference architecture Pilot the POC Promote and extend the POC result for other projects Extend and enhance more advanced analytic capabilities What insights would best benefit your business? What results are you really trying to get? What do you want to do with your data? What kind of data & correlations are you interested in mining? How much return are you expecting on your investment? What is your timeline for getting results? Are there other industries or uses you re using for a model? 12 Copyright 2013 Intel Corporation. All rights reserved

13 Hadoop project best practices Take advantage of internal and external resources Start small to reduce rework and redesign Address training requirements Engage in proactive customer interaction Use Agile Methodology Invest in automation and standardization 13 Copyright 2013 Intel Corporation. All rights reserved

15 Hadoop intro Hadoop is: A flexible, extensible open source framework Key advantages: Open Source Scale horizontally (no linear cost escalation) Faster project delivery (no schema and ETL overheads) Source: Never discard any data (store interactions, not just transactions) SW and HW costs 15 Copyright 2013 Intel Corporation. All rights reserved

16 Typical Hadoop Architecture STRUCTURED DATA PLATFORMS ANALYTICS Consume/Review Legacy Data Mining Mobile Analysis Logs CONSUME UNSTRUCTURED Social & Web Sensor/ Machine Data Create Map Node Node Hadoop REDUCE Node Hadoop Infrastructure IMPORT Enterprise Data Warehouse No-SQL In Memory DB SQL RDBMS IMPORT IMPORT Visualize Spreadsheets APPS Streaming Data Analysis Docs & Audio Files 16 Copyright 2013 Intel Corporation. All rights reserved

17 Intel Optimized Hadoop Architecture DATA PLATFORMS STRUCTURED ANALYTICS Consume/Review Legacy Data Mining HiBench Node Logs UNSTRUCTURED Social & Web Sensor/ Machine Data Create Map CONSUME Node Node 10G IMPORT NICs HiTune Enterprise Data Warehouse Hadoop Intel Distr. for Apache* Hadoop Intel X520/ 540 Intel Intel SSDs X520/ 910 & S3700 In Memory DB 540 Series IMPORT Intel Manager Cache Acceleration Software CAS Copyright 2013 Intel Corporation. All rights reserved Visualize Spreadsheets APPS SQL CAS AES-NI (TXT) 17 10G NICs IMPORT No-SQL REDUCE Luster File System Docs & Audio Files Mobile Analysis RDBMS Streaming Data Analysis

18 Intel Portfolio Delivers Balanced Performance Hadoop TeraSort 1 TB Sort Processing Time: >4 hours Improved 1 TB sort from 4 h to 7 min Upgrade to Intel Xeon Processor E processor ~50% reduction Processing time: ~7 minutes Intel Xeon Processor HDD Upgrade to Intel SSD 520 Series ~80% reduction Upgrade to Intel 10GbE Adapters ~50% reduction Intel Distribution for Apache Hadoop* software ~40% reduction 1GbE Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. Configurations: Intel internal measurements as of 26 February 2013 using Intel Distribution for Apache Hadoop * Software 2.1.1, Xeon E5-5600, Xeon E5-260 with 3 HDD and 4 SSD drives. Pllease reference slide speaker notes for configuration details. For more information go to 18 Copyright 2013 Intel Corporation. All rights reserved *Other brands and names are the property of their respective owners.

19 Intel Distribution for Apache Hadoop* & Tools HiTune ( MapReduce File-based Encryption in HDFS Up to 20x faster decryption with AES-NI* Role-based access control for Hadoop services Instrument Aggregation Engine Report Engine Up to 8.5X faster Hive queries using HBase co-processor Optimized for SSD with Cache Acceleration Software Adaptive replication in HDFS and HBase Integrated text search with Lucene HiTune Controller HiBench ( 1 2 Simplified deployment & comprehensive monitoring Deployment of HBase across multiple datacenters Automated configuration with Intel Active Tuner Detailed profiling of Hadoop jobs Simplified design of HBase schemas (+ in 2.4) REST APIs for deployment and management (+ in 2.4) Micro Benchmarks Sort WordCount TeraSort 3 Machine Learning Bayesian Classification K-Means Clustering HiBench Web Search Nutch Indexing Page Rank Result = many Hadoop optimization tips (IDF2012 presentation Big Data Analytics on a Performance-optimized Hadoop Infrastructure ) 4 HDFS Enhanced DFSIO 19 Copyright 2013 Intel Corporation. All rights reserved

24 Summary: Intel in Big Data The pervasiveness of Intel Architecture democratizes the implementation and performance of Big Data everywhere Optimized ISV software stacks and services Accelerate analytics: CPU, storage, and network Foster the growth of market partners Solution research and academia engagement Distribute analytics to the edge 24 Copyright 2013 Intel Corporation. All rights reserved

25