Einsatz von Big Data Lösungen. Dipl.Ing.Wolfgang Nimführ, Information Agenda Executive Conultant, Big Data Tiger Team IBM Softare Group Europe

Transcription

1 Einsatz von Big Data Lösungen Dipl.Ing.Wolfgang Nimführ, Information Agenda Executive Conultant, Big Data Tiger Team IBM Softare Group Europe

2 Welcome to the Instrumented Interconnected World! 2

3 Challenge Study a Large Volume and Variety of Data to Find New Insights Multi-channel customer sentiment and experience a analysis Detect life-threatening conditions at hospitals in time to intervene Predict weather patterns to plan optimal wind turbine usage, and optimize capital expenditure on asset placement Make risk decisions and frauds detection based on real-time transactional data Identify criminals and threats from disparate video, audio, and data feeds 3

4 Leveraging Big Data Analytics How do you address the challenges presented by empowered market participants generating mountains of data? Can you capture data generated by these interactions? Can you do it in realtime? Source: 1 Barrera, Clod and Wojtowecz. Cloud Leads Five Storage Trends for CIO. Januar y 27, nternetworldstats.com/stats.htm. 3 ess/3584-more+than+seven+trillion+sms+messages+will+be+sent+in+2011 Can you turn that data into insights to predict customer / competitive / market behavior? 4

5 How does Big Data Analytics impact business? Deploying these competencies extensively correlates to long-term financial performance Listen and Anticipate consistently deployed across the enterprise correlate to higher compound annual growth rates (5-year CAGR, ) Source: Outperforming in a Data Rich, H yp er Connect ed W orld, an IBM C enter for Applied Insights research report. Cop yright IBM

6 Leveraging Big Data Analytics can improve Customer Experience Client Mgr Teller e-commerce Call Center Information Management Capabilities Natural Language External Data Internal Data Web Logs Twitter feeds Facebook chats YouTube Video Blogs/Posting Appraisal data Credit bureau data Big Data Analytics Hub Relationship / risk data Product profitability data correspondents Company website logs Event triggers Customer Profitability analysis Complaint Data Voice to Text Data Transactional data Policy & Procedure data 6

7 Maximum Benefit Requires Combining Deep and Reactive Analytics Exa Deep Analytics Hypotheses Deep Predictions Real time Optimization 100,000 updates/sec, 5 ms/decision Round-trip automation 10 PB f or Deep Analytics Peta History Predictive Analytics 100,000 records/sec, 6B/day 10 ms/decision 6 PB f or Deep Analytics Data Scale Tera Giga Integration Integration Feedback Reality Smart Traffic 250K GPS probes/sec 630K segments/sec 2 ms/decision, 4K vehicles 7 Mega Kilo Traditional Data Warehouse and Business Intelligence Integration Observations yr mo wk day hr min sec ms µs Occasional Frequent Real-time Decision Frequency Fast Reactive Analytics Actions DeepQA 100s GB for Deep Analytics 3 sec/decision 1 PB training corpus

8 Requirements for a Big Data Solution Platform Analyze a Variety of Information Novel analytics on a broad set of mixed information that could not be analyzed before Multiple relational & non-relational data types and schemas Analyze Information in Motion Streaming data analysis Large volume data bursts & ad-hoc analysis Analyze Extreme Volumes of Information Cost-efficiently process and analyze petabytes of information Manage & analyze high volumes of structured, relational data Discover & Experiment Ad-hoc analytics, data discovery & experimentation 8 Manage & Plan Enforce data structure, integrity and control to ensure consistency for repeatable queries

9 Traditional Approach vs Big Data Approach Traditional Approach Structured & Repeatable Analysis Business Users Determine what question to ask Big Data Approach Iterative & Exploratory Analysis IT Delivers a platform to enable creative discovery IT Structures the data to answer that question Business Explores what questions could be asked Monthly sales reports Profitability analysis Customer surveys Brand sentiment Product strategy Ma ximum asset utilization 9

10 Big Data there are use cases across all industries Financial Services Fraud detection Risk management 360 View of the Customer Utilities Weather impact analysis on power generation Transmission monitoring Smart grid management Transportation Weather and traffic impact on logistics and fuel consumption Health & Life Sciences Epidemic early warning system ICU monitoring Remote healthcare monitoring IT Transition log analysis for multiple transactional systems Cybersecurity Retail 360 View of the Customer Click-stream analysis Real-time promotions Telecommunications CDR processing Churn prediction Geomapping / marketing Network monitoring Law Enforcement Real-time multimodal surveillance Situational awareness Cyber security detection 10

11 Monetizing Relationships, Not Just Transactions Calling Network Merged Network Amy Bearn Telco company 32, Married, mother of 3, Accountant Telco Score: 91 CPG Score: 76 Fashion Score: 88 How v aluable is Amy to my mobile phone network? How likely is she to switch carriers? How many other customers will f ollow Telco Retail 11 Social Network Public Database How v aluable is Amy to my retail sales? Who does she influence? What do they spend?

12 Big Data 360 Multi-Channel Customer Sentiment Analysis Business Processes Drive consistent behavior across all applications and processes Events and Alerts Big Data Platform web traffic and social media Insight Insight generates customer churn alert Master Data Management Website Logs Social Media Internet Scale Analytics Information Integration Data Warehouse Campaigns Campaign Management Call Detail Reports (CDRs) Streaming Analytics Call behavior and experience insight Customer Sentiment BI Customer Insight BI 12

13 Big Data 360 Lead Generation Personal Personal Attributes Attributes Identifiers: name, address, age, gender, Identifiers: name, address, age, gender, occupation occupation Interests: sports, pets, cuisine Interests: sports, pets, cuisine Life Cycle Status: marital, parental Life Cycle Status: marital, parental Life Life Events Events Life-changing events: relocation, having a Life-changing events: relocation, having a baby, getting married, getting divorced, buying baby, getting married, getting divorced, buying a house a house Relationships Relationships Personal relationships: family, friends and Personal relationships: family, friends and roommates roommates Business relationships: co-workers and Business relationships: co-workers and work/interest network work/interest network Monetizable intent to buy products Social Media based 360-degree Consumer Profiles Life Events Timely Timely Insights Insights Intent to buy various products Intent to buy various products Current Location Current Location Sentiment on products, services, campaigns Sentiment on products, services, campaigns Incidents damaging reputation Incidents damaging reputation Customer satisfaction/attrition Customer satisfaction/attrition Products Products Interests Interests Personal preferences of products Personal preferences of products Product Purchase history Product Purchase history Suggestions on products & services Suggestions on products & services I need a new digital camera for my food pictures, any I need a new digital camera for my food pictures, any recommendations around 300? recommendations around 300? What should I buy?? A mini laptop with Windows 7 OR a Apple What should I buy?? A mini laptop with Windows 7 OR a Apple MacBook!??! MacBook!??! Location announcements I'm 13 I'm at at Starbucks Starbucks Parque Parque Tezontle Tezontle College: Off to Stanford for my MBA! Bbye chicago! College: Off to Stanford for my MBA! Bbye chicago! Looks like we'll be moving to New Orleans sooner than I thought. Looks like we'll be moving to New Orleans sooner than I thought. Intent to buy a house I'm thinking about buying a home in Buckingham Estates per a I'm thinking about buying a home in Buckingham Estates per a recommendation. Anyone have advice on that area? #atx #austinrealestate recommendation. Anyone have advice on that area? #atx 2012 #austinrealestate #austin IBM Corporation #austin

14 Big Data 360 Lead Generation Real-time Real-time product product intents intents enriched enriched with with consumer consumer attributes attributes Micro-segmentation Micro-segmentation of of product product intents intents by by occupation occupation Entries Entries contain contain promotional promotional messages, messages, wishful wishful thinking, thinking, questions, questions, etc etc Integration Integration across across Social Social Media Media sites sites Real-time Real-time tracking tracking by by micro-segmentation micro-segmentation For For many many of of the the attributes attributes we we need need to to extract, extract, cleanse, cleanse, normalize normalize and and categorize categorize Micro-segmentation Micro-segmentation of of consumers consumers by by hobbies hobbies 14

15 Institutional Risk Application Comprehensive view of publicly traded companies and related people based on regulatory filings Extract Integrate 15

16 IBM Big Data Platform for Ingest, Data and Analytics Analytics reveal Business Value New analytic applications drive the requirements for a big data platform Integrate and manage the full variety, velocity and volume of data Apply advanced analytics to information in its native form Visualize all available data for adhoc analysis Development environment for building new analytic applications Workload optimization and scheduling Security and Governance Infrastructure enables Business Value BI / Reporting Visualization & Discovery Hadoop System Analytic Applications Exploration / Visualization Functional App IBM Big Data Platform Application Development Accelerators Stream Computing Industry App Predictive Analytics Systems Management Data Warehouse Information Integration & Governance Content Analytics 16

17 Big Data Challenges and Solutions Big Data Challenges IBM Big Data Solutions SQL Data NoSQL Data Streaming High volume of structured data Valuable Information Compute intensive analytics Low latency response on queries Business Intelligence and Analytics Understanding the customer through segmentation and analysis Very high volumes (TBs to PBs) unstructured data Exploration and discovery Text, Entity and Social Media Analytics Real time processing Detect failure patterns High volume, low latency processing Scoring and decision analytics IBM Netezza Analytic appliance for high speed, advanced analytics on large structured data sets IBM BigInsights Hadoop-based processing for analytics on variety and volumes of data IBM Streams Low latency analytics for streaming data 17

18 High Level Architecture View *) Sensors Regulations Streaming Structured or Unstructured Unstructured Real Time Scoring and Response IBM Streams Analytics and Reporting Streaming Smart Grid Analytics Distribution Grid Monitoring Root Cause Failure Analysis Demand Response Effectiveness Social Unstructured Data Asset Landscape Generation Trading Marketing Transmission Supplier Maintenance Exploration/Discovery Queryable Archive Distribution Orders Employee IBM BigInsights Smart Meters Customer GIS ETL Improv ed Analytics Structured Improv ed Analytics Structured InfoSphere Warehouse Analytics and Reporting Analytics and Reporting Web/social Sentiment analysis Call Centre analysis Log analysis Outage Information Micro customer segmentation Offering Management Foundational Meter Data Management Customer Portals Smart Meter Analytics Demand Forecasting Generation Scheduling Customer Segmentation Campaign Management Outage Management Estimate Load Shedding Time of Use Tariffs Maintenance Scheduling 18 *) Example for Industry Energy & Utility

19 InfoSphere BigInsights Analytical platform for persistent Big Data Based on open source & IBM technologies Distinguishing characteristics Built-in analytics... enhances business knowledge Enterprise software integration... complements and extends existing capabilities Production-ready platform with tooling for analysts, developers, and administrators... speeds time-to-value and simplifies development/maintenance IBM advantage Combination of software, hardware, services and advanced research BI / Exploration / Functional Industry Predictiv e Content Reporting Visualization App App Analytics Analytics Visualization & Discovery Hadoop System Analytic Applications IBM Big Data Platform Application Development Accelerators Stream Computing Systems Management Data Warehouse Information Integration & Governance 19

20 InfoSphere BigInsights Embrace and Extend Hadoop Analytics BigSheets Text Analytics ML Analytics *) Interface Application Zookeeper IBM LZO Compression Pig Hive Jaql MapReduce AdaptiveMR FLEX BigIndex Oozie Lucene Av ro Management Console (browser based) Developing Tooling (Eclipse Plug-Ins) Storage HBase Rest API (for Applications) HDFS GPFS-SNC *) Data Sources/ Connectors Streams Data Stage Netezza DB2 BoardReader CSV / XML / JSON R SPSS IBM Open Source Flume JDBC Web Crawler *) future release 20

21 BigSheets A visual tool for data manipulation and prototyping Ad-hoc analytics for LOB user Analyze a variety of data - unstructured and structured Spreadsheet metaphor for exploring/ visualizing data Browser-based 21

22 Text Analytics Turns disparate words into measurable insights Physically assemble data, standardize formats, address auto-identify language, process punctuation and non-grammatical characters, standardize spelling. Part-of-speech identification, standard and customized extraction dictionaries, proper noun identification, concept categorization, synonyms, exclusions, multi-terms, regular expressions, fuzzymatching Identify positive or negative sentiment, NLP-based analytics, define variables, macros and rules. Iterative classification using automated and manual techniques. Concept derivation & inclusion, semantic networks and cooccurrence rules Reporting/Monitoring social commentary, combination w /structured data, clustering, associated concepts, correlated concepts, autoclassification of documents, sites, posts. 22 Pre-configured text annotators ready for distributed processing on Big Data Support for native languages including double-byte

23 Text Analytics Highly accurate analysis of textual content How it works Parses text and detects meaning with annotators Understands the context in which the text is analyzed Hundreds of pre-built annotators for names, addresses, phone numbers, along others Accuracy Highly accurate in deriving meaning from complex text Performance AQL language optimized for MapReduce Unstructured text (document, , etc) Football World Cup 2010, one team distinguished themselves well, losing to the eventual champions 1-0 in the Final. Early in the second half, Netherlands striker, Arjen Robben, had a breakaway, but the keeper for Spain, Iker Casillas made the save. Winger Andres Iniesta scored for Spain for the win. Classification and Insight 23

24 ML Analytics Statistical and Predictive Analysis Framework for machine learning (ML) implementations on Big Data Large, sparse data sets, e.g. 5B non-zero values Runs on large BigInsights clusters with 1000s of nodes Productivity Build and enhance predictive models directly on Big Data High-level language Declarative Machine Learning Language (DML) E.g lines of Java code boils down to 15 lines of DML code Parallel SPSS data mining algorithms implementable in DML Optimization Compile algorithms into optimized parallel code For different clusters and different data characteristics E.g. 1 hr. execution (hand-coded) down to 10 mins E xecution Time (sec) # non zeros (million) 24 Java Map-Reduce Sy stemml Single node R

25 Workload Optimization Optimized performance for big data analytic workloads Adaptive MapReduce Algorithm to optimize execution time of multiple small jobs Performance gains of 30% reduce overhead of task startup Hadoop System Scheduler Identifies small and large jobs from prior experience Sequences work to reduce overhead Task Map (break task into small parts) Adaptive Map (optimization order small units of work) Reduce (many results to a single result set) 25

26 Public wind data is available on 284km x 284 km grids (2.5o LAT/LONG) More data means more accurate and richer models (adding hundreds of variables) - Vestas wind library at 2.5 PB: to grow to over 6 PB in the near-term - Granularity 27km x 27km grids: driving to 9x9, 3x3 to 10m x 10m simulations Reduced turbine placement identification from weeks to hours Perspective: The Vestas Wind library

27 InfoSphere Streams Analytical platform for in-motion Big Data Analytic Applications Built to analyze data in motion Multiple concurrent input streams Massive scalability BI / Exploration / Functional Industry Predictiv e Content Reporting Visualization App App Analytics Analytics Visualization & Discovery IBM Big Data Platform Application Development Systems Management Process and analyze a variety of data Accelerators Structured, unstructured content, video, audio Hadoop System Stream Computing Data Warehouse Advanced analytic operators Information Integration & Governance 27

28 Stream Computing Analyze Data in Motion Traditional Computing Stream Computing Historical fact finding Find and analyze information stored on disk Batch paradigm, pull model Query-driven: submits queries to static data Current fact finding Analyze data in motion before it is stored Low latency paradigm, push model Data driven bring the data to the query 28

29 Streams approach illustrated tuple 29

30 Massively Scalable Stream Analytics Linear Scalability Clustered deployments unlimited scalability Automated Deployment Automatically optimize operator deployment across clusters Performance Optimization JVM Sharing minimize memory use Fuse operators on same cluster Telco client 25 Million messages per second Analytics on Streaming Data Analytic accelerators for a variety of data types Streaming Data Sources Optimized for real-time performance Deployments Source Adapters Analytic Operators Streams Runtime Sync Adapters Streams Studio IDE Automated and Optimized Deployment Visualization 30

31 University of Ontario Institute of Technology Use case Neonatal infant monitoring Predict infection in ICU 24 hours in advance Solutions 120 children monitored :120K msg/sec, billion msg/day Trials expanding to include hospitals in US and China Event Preprocesser Analysis Framework Sensor Network Stream-based Distributed Interoperable Health care Infrastructure Solutions (Applications) 31

32 Cisco turns to IBM big data for intelligent infrastructure management Optimize building energy consumption with centralized monitoring Automate preventive and corrective maintenance Capabilities Utilized: Streaming Analytics Hadoop System Business Intelligence 32 Applications: Log Analytics Energy Bill Forecasting Energy consumption optimization Detection of anomalous usage Presence-aware energy mgt. Policy enforcement

33 Without a Big Data Platform You Code IBM Big Data Platform Over 100 sample applications and toolkits with industry focused toolkits with 300+ functions and operators Event Handling Custom SQL and Scripts Multithreading Check Pointing Application Management HA Accelerators and Tool kits Streams provides development, deployment, runtime, and infrastructure services Connectors Performance Optimization Debug Security TerraEchos developers can deliver applications 45% faster due to the agility of Streams Processing Language Alex Philip, CEO and President, TerraEchos 33

34 Big Data Platform Directions 1.Mature enterprise capabilities Scalability and manageability Robust file system and information lifecycle management Deployment options: software, appliances, cloud Deep integration with enterprise systems and applications 2.Ecosystem support Ease of use for all types of users: Developers, Business Users, Partners and Data Scientists Enhanced development environment Self-service application development and visualization tools 3.Accelerators to drive faster time to value Extensive analytic techniques for different uses Industry-specific models and use cases 4.Enhanced focus on the 4th V:Veracity Managing data, process and model uncertainty 34

35 Questions & Discussions ibm.com/smarteranalytics ibm.com/bigdata 35

36 Thank You! Think Big BIG DATA Dipl.Ing. Wolfgang Nimführ IBM Österreich Obere Donaustrasse 95 Information Agenda Executive Consultant Tel Big Data Tiger Team IBM Software Group Europe 36