How To Understand The Power Of Big Data

Transcription

1 Conquering Big Data with Information Architecture Carlos Augusto, Sales Consulting Manager Mark Johnson, Director of Engineered Systems and Big Data Programs

2 If you had a magic wand Where would you like to be today? Brazil? Disney World? Switzerland? Great Planet Earth! Couch

3 Now consider this

4 Further Analyzing it It is hard to believe that other like Earth planets don t exist. We don t know what we don t know.

5 Organizations have a Data Milky Way They need to find the next Earth Copyright 2012, Oracle and/or its affiliates. All rights reserved.

6 executives who would grade themselves C or lower in preparedness 93 believe 89 % executives who say drawing intelligence their organization from data is top is priority losing revenue as a result of not being able to fully leverage information 67 % 6 Copyright 2012, Oracle and/or its affiliates. All rights reserved. Source: Oracle Research Study - From Overload to Impact: An Industry Scorecard on Big Data Business Challenges, July 2012

7 Big Data Definitions Big Data: Techniques and Technologies that Enable Enterprises to Effectively and Economically Analyze All of their Data Forrester Research

8 Big Data is ALL Data Unstructured, Semi-Structure and Structured What is the main difference in this data? There is always structure. But its not formally defined or anticipated. Social Media, RSS feeds, Videos, DOCs, PDFs, Graphics Volume, Velocity, Variety, Value These Characteristics Challenge your Existing Architecture and your Thought Processes Semi-Structured. Does not conform to DB tables, but still contains tags or semantic elements. s, log files, machine generated content

9 What Makes it Big Data? VOLUME Very large quantities of data VELOCITY Extremely fast streams of data VARIETY Wide range of datatype characteristics

10 Public Sector Big Data Use Cases Big Data can add significant value FRAUD PREVENTION Detect abnormal behavior patterns MAINTENANCE & UTILITIES Reduce maintenance costs CONSTITUENT SENTIMENT Detect trends through social media activities THREAT IDENTIFICATION Detect suspicious activities HEALTH CARE Improve treatments and lower costs OPEN GOVERNMENT Increase transparency to constituents REGULATORY COMPLIANCE Identify discrepancies in records across systems JUSTICE & PUBLIC SAFETY Enhance Intelligence & Policing Platform REVENUE MANAGEMENT Increase Collections

11 National Cancer Institute, Oracle & SAIC Identifying Cancer & Genome Relationships Solution Cross-referenced the relationships between 20,000 genes in a population of 900M people with 22 million medical publication abstracts Achieved over 100x performance improvement with 367TB of data Our mouths were open in amazement Dr. Robert Stephens Director of Bioinformatics, National Cancer Institute * Winner Best Government Big Data Project for 2012 CTO Vision

12 The solution to Big Data isn t a single product DATA LIFECYCLE NEW REQUIREMENTS DECIDE ANALYZE ACQUIRE ORGANIZE New Tools for Acquiring and Organizing Information Easy Integration of New Infrastructure In-memory Processing Advanced Analytics

13 Big Data - Tools Social Data Hadoop (Map/Reduce)

14 Big Data Technologies Basic Principles and Minimum Tech Foundation Dominant Technologies Apache Hadoop Software Framework for data intensive distributed applications Three major components/modules: Hadoop Distributed File System (HDFS) Parallel Scanning, Batch Oriented YARN A framework for job scheduling and cluster resource management Map-Reduce A programming model based on YARN for processing the large data sets in parallel

15 Big Data Technologies Some additional Hadoop components Component Apache Hive Apache Pig Apache Mahout Apache Flume Apache Cassandra Capability Provides simple SQL-like interface to hadoop cluster High level data flow language used during hadoop data analysis. Pig scripts are automatically converted into Map-Reduce jobs Data mining library for the Hadoop framework. Delivers a core set of algorithms designed for clustering, classification and batch-based filtering Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of streaming event data (log, for example). Multi-master database with no single points of failure

16 Big Data requires Big Architecture... Leveraging what you own and Focus on Business Value DECIDE ANALYZE ACQUIRE ORGANIZE Exploiting Synergies via Information Architecture Values and Principles

17 Information Architecture Capability Model Infrastructure Security Governance Enterprise Data Model Sharing & Delivery Diverse Data Realms BI & Data Warehouse Master Data Mgmt Integration Content Management Data Realms Master Transaction Reference Analytical Metadata Unstructured Big Data An Architect s Approach to Enterprise Initiatives Simplify Unify Standardize Focus on Business Value

18 Structured Traditional Information Architecture Approach Management, Security, Governance Reference & Master Data Transaction Data Enterprise Integration Data Warehouse Analytic Capabilities Key Architecture Principles Ensure Quality, Timeliness, Security Viewing data as an asset through value, cost and risk lens Key EA Responsibility Establish Balanced Governance to - Ensure Business Alignment Create Center of Excellence for - Architecture Review - Standards and Training

19 Unstructured Big Data Information Architecture Approach Machine Generated Text, Image, Audio, Video Distributed File System Key Value Data Store Map Reduce Sandboxes Data Warehouse Analytic Capabilities Key Architecture Principles Ensure Quality, Timeliness, Security Volume Drive to zero latency Variety Dynamic schema Velocity Continuous increase Value Explore and correlate Key EA Responsibility Establish Balanced Governance to - Ensure Business Alignment Create Center of Excellence for - Architecture Review - Standards and Training

20 Unified Information Architecture Enable Organizations to Make Decision Using All Data Data Sources Knowledge Discovery Engine Big Data Ecosystem User Tools Information Discovery High Volume Distributed File System Data Warehouse BI Tools and Dashboards Custom/Advanced Analytics NoSQL DB Machine Learning Algorithms Real-Time Recommendations

21 Structured Unstructured Semistructured Integrated Big Data Architecture Capabilities Management Security, Governance Acquire Organize Analyze Decide Master & Ref Data Transaction Data DBMS (OLTP) ETL/ELT CDC Real-Time ODS DW/DM Reporting & Dashboards Alerting EPM BI Applications Machine Generated Social Media HDFS NoSQL Message- Based Hadoop (MapReduce) Streaming (CEP Engine) In-Database Analytics Text Analytics and Search Advanced Analytics Visual Discovery Text, Image Video, Audio Servers Storage Network OS

22 Making Key Architecture Decisions Balancing Cost, Value and Risk Challenge: low cost, high volume data exploration Challenge: real-time analytics for low-density data Challenge: correlation with structured data

23 Adding Unstructured Sources Healthcare EMR Nurse & Physician Notes, HIE-CCD Documents, etc.... Keep experiencing frequent headaches and dizzy spells. This has been ongoing since.. The endotracheal tube and left sub-clavian vein catheter was in satisfactory position... Asthma symptoms, which include coughing, wheezing, and chest tightness, are common in an asthma attack.. My husband is having trouble with a chest cold and his lungs hurt, what can he do to feel better?.. PCP Office Notes EMR Note Fields 3 rd Party Web Sites Social Media

24 Endeca Information Discovery Rapid, intuitive exploration and analysis of data from any combination of structured and unstructured sources Unstructured Analytics Physician Notes Mr. Edwards is a 36 yo auto mechanic from Chicago, IL with a history of mild asthma and seasonal allergies. He complains that he became ill 2 weeks ago with a summer cold wile in Orlando, and rather than getting well he got worse several days ago. He has facial pain, pain in the upper teeth on the feels somewhat better with Arfin and Tylenol. He denies associated symptoms of shortness of breath, but does have a mild non-productive cough. He had a recent normal dental evaluation summer cold wile in Orlando, and rather than getting well he got worse several days ago. He has facial pain, pain in the upper teeth on the right, blood tinged nasal discharge, subjective fevers and mild sore throat Unique Features Robust Content Acquisition and Text Enrichment Contextual, Natural Language Search Dynamic Data Modeling Surfaces Relationships Intuitive Navigation & Analytics In-Memory Performance Benefits Unprecedented Information Visibility Uncovers Data Relationships Not Visible With Standard Tools Self-Service Data Discovery for Clinicians

25 Location ID Diagnoses EE Name Endeca Has Unique Capabilities to Crawl the Data and Index ALL Data Relationships Labor Data Primary EMR Medications Unit / Dept. Manager Surgeries Payer ID Modifier Codes Nurses Notes CHF Dr. Lingle Operations Data Diagnostic Codes Region Item Cost Revenue Cycle Data Comorbidities

26 Information Discovery Interface Overview Search Guided Navigation Analytics

27 Relational Analysis Power in Action Unique, User-Friendly, Natural Language Search Capability Helps user pick desired filter attributes based on natural language entries

28 Relational Analytics Basic Readmission / Outcome Discovery A Rapid selection of patient / diagnostic groups to analyze C Instant update of key metrics on selected groups B Configurable Standard filters used most frequently D Configurable graphic analyses of multiple, selectable parameters

29 Unexpected Relationships Which Data Attributes Have Highest Incidence/Correlation? No matter what data source they came from or what you thought to look for! A Tag Cloud gives visual relational indicators B Note this example where two specific procedures & a specific physician had high relationships to all Iatrogenic Pneumothoraces

30 Dynamic Visual Analysis Interactive refreshing of data upon every click All metrics & graphs update instantly as you progressively select additional parameters to explore

31 Information Discovery and Hadoop Complementary Big Data Capabilities Oracle Endeca Information Discovery Massive scalability Batch execution Technical experts develop analyses Deep Large-Scale Processing Data Variety - structured and unstructured data No fixed pre-defined model required Populates Informs In-memory analytics Interactive query response times Targets business users Fast Self-service Information Discovery

32 Data Exploration Conceptual View HDFS SQL Tools Business Opportunity Sessionized weblogs Abandoned shopping carts Technology Options Mount HDFS w/dbms Load data into DBMS DBMS Architecture Decisions No data movement Use existing SQL skills

33 External Table Data Exploration Logical View Oracle Big Data Appliance Key Components FUSE Oracle SQL Developer (any SQL tools) OR Oracle Big Data Appliance or other HDFS Solution Oracle Database External Table OR OBIEE (Big Data Analysis) FUSE (Filesystem in Userspace) SQL Developer Oracle BI Enterprise Edition

34 Correlation & Analytics Conceptual Expand Data Warehouse with Granular Data Store Business Intelligence Online Data Scalable Factory Flexible Cost Effective Σ Data Warehouse Σ Marts Archiving

35 Correlation & Analytics Logical HDFS & NoSQL, MapReduce Oracle BD Connectors Oracle Advanced Analytics Oracle BI Foundation and Endeca Key Components Oracle Big Data Appliance Infini Band InfiniBand Oracle Data Integrator Oracle Exadata Oracle Data Integrator Oracle Advanced Analytics Oracle Exalytics Oracle Big Data Appliance Oracle Exadata Oracle Exalytics Oracle BI and Endeca

36 Summary of Oracle Big Data Platform Big Data Appliance CEP Exadata Endeca Information Discovery Exalytics Oracle Business Intelligence ACQUIRE ORGANIZE ANALYZE DECIDE

37 In Summary Big Data Best Practices Align Big Data with Specific Business Goals Think Big and Start Small Go for a Quick Win Easy Skills Shortage with Standards, Governance, Tools and Appliances Optimize Knowledge Transfer with a Center of Excellence Top Payoff is Aligning Unstructured with Structured Data Plan your Sandboxes for Performance Align with Cloud Operating Model Take Advantage of Big Data Technologies and Go Find the Next Earth!!!

38

39 Questions?