A holistic approach to Big Data

Transcription

1 A holistic approach to Big Data Raul F. Chong Senior Big Data and Cloud Program Manager Big Data University Community Leader 2013 BigDataUniversity.com

2 Agenda The state of Big Data adoption Big Data A holistic approach The 5 high value Big Data use cases Technical details of key Big Data components The future of Big Data and Cloud Demos Resources

4 Big Data Adoption Phases

5 What is your Big Data source? What type of data/records are you planning to analyze using big data technologies?

6 What is your Big Data source? What type of data/records are you planning to analyze using big data technologies? Multiple responses accepted

7 What do you want to do with the Big Data collected? What kind of analytics do you want to perform on this big data?

8 What do you want to do with the Big Data collected? What kind of analytics do you want to perform on this big data? Multiple responses accepted

9 Use of Big Data globally and in the financial sector Multiple responses accepted

11 KTH Swedish Royal Institute of Technology Reducing Traffic Congestion Deployed real-time Smarter Traffic system to predict and improve traffic flow. Analyzes streaming real-time data gathered from cameras at entry/exit to city, GPS data from taxis and trucks, and weather information. Predicts best time and method to travel such as when to leave to catch a flight at the airport Results Enables ability to analyze and predict traffic faster and more accurately than ever before Provides new insight into mechanisms that affect a complex traffic system Smarter, more efficient, and more environmentally friendly traffic 11

12 University of Southern California Innovation Lab Monitors Political Debates Benefits Real-time display of public sentiment as candidates respond to questions Debate winner prediction based on public opinion instead of solely political analysts

13 Big Data A holistic approach Big Data is Not Only Hadoop! Examples where Hadoop is not entirely applicable: Cyber security, Stock market, Traffic control, Sensor information, monitoring trends in Social Media What if your company has many silos of information, difficult to move to HDFS? What about governance? Can we trust the source of this data?

14 Big data holistic approach: A platform Solutions Analytics and Decision Management Big Data Platform Big Data Infrastructure

15 Big data holistic approach: A platform The IBM Big Data Platform Solutions Analytics and Decision Management Big Data Platform Data Warehouse Delivers deep insight with advanced indatabase analytics & operational analytics Big Data Infrastructure

16 Big data holistic approach: A platform Solutions Analytics and Decision Management Big Data Platform Stream Computing Data Warehouse Analyze streaming data and large data bursts for real-time insights Big Data Infrastructure

17 Big data holistic approach: A platform The IBM Big Data Platform Solutions Analytics and Decision Management Big Data Platform Cost-effectively analyze Petabytes of unstructured and structured data Hadoop System Stream Computing Data Warehouse Big Data Infrastructure

18 Big data holistic approach: A platform Solutions Analytics and Decision Management Big Data Platform Hadoop System Stream Computing Data Warehouse Govern data quality and manage the information lifecycle Information Integration & Governance Big Data Infrastructure 18

19 Big data holistic approach: A platform Solutions Analytics and Decision Management Big Data Platform Speed time to value with analytic and application accelerators Accelerators Hadoop System Stream Computing Data Warehouse Information Integration & Governance Big Data Infrastructure

20 Big data holistic approach: A platform The IBM Big Data Platform Solutions Analytics and Decision Management Discover, understand, search, and navigate federated sources of big data Visualization & Discovery Big Data Platform Application Development Systems Management Accelerators Hadoop System Stream Computing Data Warehouse Information Integration & Governance Big Data Infrastructure

21 Big data holistic approach: A platform Process any type of data Structured, unstructured, inmotion, at-rest, in-place Built-for-purpose engines Designed to handle different requirements Manage and govern data in the ecosystem Enterprise data integration Grow and evolve on current infrastructure The whole is greater than the sum of parts Integrated components Out of the box, standards-based services Start small (value is additive) Solutions Analytics and Decision Management Visualization & Discovery Hadoop System Big Data Platform Application Development Accelerators Stream Computing Information Integration & Governance Big Data Infrastructure Systems Management Data Warehouse 21

22 An example of the big data platform in practice Ingestion and Real-time Analytic Zone Streams Analytics and Reporting Zone Warehousing Zone BI & Reporting Connectors Hadoop Enterprise Warehouse Predictive Analytics MapReduce Hive/HBase Col Stores Data Marts Visualization & Discovery Documents in variety of formats ETL, MDM, Data Governance Landing and Analytics Sandbox Zone Metadata and Governance Zone

24 The 5 High Value Big Data Use Cases Big Data Exploration Find, visualize, understand all big data to improve business knowledge Enhanced 360 o View of the Customer Achieve a true unified view, incorporating internal and external sources Security/Intelligence Extension Lower risk, detect fraud and monitor cyber security in real-time Operations Analysis Analyze a variety of machine data for improved business results Data Warehouse Augmentation Integrate big data and data warehouse capabilities to increase operational efficiency

25 Big Data Exploration: Illustrated Find, visualize and understand all big data to improve business knowledge Greater efficiencies in business processes New insights from combining and analyzing data types in new ways Develop new business models with resulting increased market presence and revenue Streams Connector Framework Hadoop UI / User App Builder Integration & Governance Data Explorer Warehouse CM, RM, DM RDBMS Feeds Web 2.0 Web CRM, ERP File Systems

26 Big Data Exploration: Example in Practice Airplane Manufacturer Blinded for confidentiality Exploring 4 TB to drive point business solutions (supplier portal, call center, etc.) Single-point of data fusion for all employees to use Reduced costs & improved operational performance for the business Is Big Data Exploration Right for You? How do you separate the noise from useful content? How do you perform data exploration on large and complex data? How do you find insights in new or unstructured data types (e.g. social media and )? How do you enable employees to navigate and explore enterprise and external content? Can you present this in a single user interface? How do you identify areas of data risk before they become a problem? What is the starting point for your big data initiatives? Big Data Platform Component Starting Point: Data Explorer

27 Enhanced 360º View of the Customer: Illustrated SOURCE SYSTEMS CRM Name: J Robertson Address: 35 West 15 th Address: Pittsburgh, PA ERP Name: Janet Robertson Address: 35 West 15 th St. Address: Pittsburgh, PA Legacy Name: Address: Jan Robertson 36 West 15 th St. Address: Pittsburgh, PA Master Data Management 360 View of Party Identity First: Last: Address: City: State/Zip: Gender: Age: DOB: Janet Robertson 35 West 15 th St Pittsburgh PA / F 48 1/4/64 Hadoop Streams Warehouse Unified View of Party s Information

28 Enhanced 360º View of the Customer: Insight from user s photos Pins / Re-pins Likes / Dislikes Tweets Favorites Photo Albums and Pinboards Photo Semantic Analysis User Segmentation Style Kitchen Gallery Consumer Dream Home Wedding Computer Advertisements Promotions Campaigns Planning Retailers, Marketers and Planners Preferred Styles Designs Products Interests 28

29 Enhanced 360º Customer View: Customer Example Leading Medical Equipment Supplier Blinded for confidentiality Increase revenue and decrease cost in the call center Increase customer & employee satisfaction Leverage new data types in customer analysis Is the Enhanced 360º Customer View Right for You? How do you identify and deliver all data as it relates to a customer, product, competitor to those to need it? How do you gather insights about your customers from social data, surveys, support s, etc.? How do you combine your structured and unstructured data to run analytics? How are you driving consistency across your information assets when representing your customer, clients, partners etc.? How do you deliver a complete view of the customer enhance to your line of business users to ensure better business outcomes? Big Data Platform Component Starting Point: Data Explorer, Hadoop

30 Security/Intelligence Extension: Illustrated New Considerations Traditional Security Operations and Technology Logs Events Alerts Collection, Storage and Processing Big Data Analytics Configuration information System audit trails Network flows and anomalies External threat intelligence feeds Web page text and social activity Identity context Video/audio surveillance Business process data Customer transactions Collection and integration Size and speed Enrichment and correlation Analytics and Workflow Visualization Unstructured analysis Learning and prediction Customization Sharing and export

31 Reconstructing Events Integrating Multimedia from Diverse Sources Security Cameras Mobile Cameras Overhead Social Media 100K security cameras (static cameras, slowly changing topology) 10M mobile photos/day (limited knowledge about locations) 50M social media photos/video (uncertain geo-temporal context) Moving vehicles (patrol cars), overhead drones, broadcast, retail, 311, etc. Correlate multimedia content across a wide diversity of sources and dynamic topology of cameras Exploit partial overlaps in field of view, reidentification of objects/people and contextual information Obtain real-time operational picture across diverse content 31

32 Security/Intelligence Extension: Customer Example Captured and analyzed 42TB of daily traffic in real-time for tracking persons of interest to take suitable action and reduce risk. Would the Security / Intelligence Extension benefit you? What are your plans to enrich your security or intel system with unused or underleveraged data sources (video, audio, smart devices, network, Telco, social media)? How will you address the need sub second detection, identification, resolution of physical or cyber threats? How do you intend to follow activities of criminals, terrorists, or persons in a blacklist? How do you plan to enhance your surveillance system with real-time data from video, acoustic, thermal or other security sensors? Do you want to correlate lots of technical or human intel data and sources looking for associations or patterns (big data forensics)? How are you going to deal with unstructured data ( , social, etc.) in your Security Information & Event Management (SIEM) solution to improve cyber threat detection & remediation? Big Data Platform Component Starting Point: Streams, Hadoop

33 Operations Analysis: Illustrated Indexing, Search Raw Logs and Machine Data Only store what is needed Machine Data Accelerator Statistical Modeling Root Cause Analysis Real-time Analysis Federated Navigation & Discovery

34 Operations analysis is a Business Imperative Cost of System Down Time 49 percent of Fortune 500 companies experience > 80 hours of system down time/year 1 Cost of down time varies between $90,000/hr to $6.48 million/hr 80 hours * $6.48M = approx $500M per year System downtown costs North American businesses $26.5 billion a year in lost revenue

35 Operations Analysis: Customer Example Intelligent Infrastructure Management: log analytics, energy bill forecasting, energy consumption optimization, anomalous energy usage detection, presence-aware energy management Optimized building energy consumption with centralized monitoring; Automated preventive and corrective maintenance Utilized InfoSphere Streams, InfoSphere BigInsights, IBM Cognos Would Operations Analysis benefit you? Do you deal with large volumes of machine data? How do you access and search that data? How do you perform root cause analysis? How do you perform complex real-time analysis to correlate across different data sets? How do you monitor and visualize streaming data in real time and generate alerts? Big Data Platform Component Starting Point: Hadoop, Streams

36 Data Warehouse Augmentation: Needs Integrate big data and data warehouse capabilities to increase operational efficiency Need to leverage variety of data Structured, unstructured, and streaming data sources required for deep analysis Low latency requirements (hours not weeks or months) Required query access to data Extend warehouse infrastructure Optimized storage, maintenance and licensing costs by migrating rarely used data to Hadoop Reduced storage costs through smart processing of streaming data Improved warehouse performance by determining what data to feed into it

37 Data Warehouse Augmentation: Illustrated Hadoop Filter and summarize big data for the warehouse

38 Data Warehouse Augmentation: Illustrated Hadoop Hadoop as a query-ready archive for a data warehouse

39 Data Warehouse Augmentation: Customer Example Improved analysis performance by over 40 times, reduced wait time from hours to seconds, and increased campaign effectiveness by 20+%. Could Data Warehouse Augmentation benefit you? Are you drowning in very large data sets (TBs to PBs) that are difficult and costly to store? Are you able to utilize and store new data types? Are you facing rising maintenance/licensing costs? Do you use your warehouse environment as a repository for all data? Do you have a lot of cold, or low-touch, data driving up costs or slowing performance? Do you want to perform analysis of data in-motion to determine what should be stored in the warehouse? Do you want to perform data exploration on all data? Are you using your data for new types of analytics? Big Data Platform Component Starting Point: Hadoop, Streams

41 Sentiment Analysis using IBM Text Analytics (Basic example) 2013 BigDataUniversity.com

42 Sentiments for movie Ra.One :-( 2013 BigDataUniversity.com

43 Sentiments for movie Swades :-) 2013 BigDataUniversity.com

44 Architecture Diagram Annotated Document Stream AQL Text Analytics Optimizer Compiled Operator Graph (.aog) Text Analytics Runtime Rule language with familiar SQL-like syntax Specify annotator semantics declaratively Choose an efficient execution plan that implements the semantics Highly scalable, embeddable Java runtime Input Document Stream 2013 BigDataUniversity.com

45 How Streams Works continuous ingestion Continuous ingestion Continuous analysis

46 How Streams Works Continuous ingestion Continuous analysis Filter / Sample Infrastructure provides services for Scheduling analytics across hardware hosts, Establishing streaming connectivity Transform Annotate Correlate Classify Achieve scale: By partitioning applications into software components By distributing across stream-connected hardware hosts Where appropriate: Elements can be fused together for lower communication latency

47 Scalable Stream Processing Streams programming model: construct a graph Mathematical concept not a line -, bar -, or pie chart! OP Also called a network OP Familiar: for example, a tree structure is a graph Consisting of operators and the streams that connect them The vertices (or nodes) and edges of the mathematical graph A directed graph: the edges have a direction (arrows) Streams runtime model: distributed processes Single or multiple operators form a Processing Element (PE) Compiler and runtime services make it easy to deploy PEs On one machine Across multiple hosts in a cluster when scaled-up processing is required All links and data transport are handled by runtime services Automatically With manual placement directives where required OP OP stream OP OP OP

48 From Essential Elements to Running Jobs Streams application graph: A directed, possibly cyclic, graph A collection of operators Src Connected by streams Each complete application is a potentially deployable job Src OP stream OP OP Sink Sink Jobs are deployed to a Streams runtime environment, known as a Streams Instance (or simply, an instance) An instance can include a single processing node (hardware) Or multiple processing nodes node node h/w node node node node node node Streams instance

49 Streams Runtime Illustrated Meters Company Filter Usage Model Temp Action Optimizing scheduler assigns jobs to hosts, and continually manages resource allocation Usage Contract Text Extract Season Adjust Daily Adjust Commodity hardware laptop, blades or high performance clusters x86 host x86 host x86 host x86 host

50 Streams Runtime Illustrated Optimizing scheduler assigns PEs to hosts, and continually manages resource allocation Commodity hardware laptop, blades or high performance clusters Dynamically add hosts and jobs New jobs work with existing jobs Meters Meters Company Filter Usage Model Temp Action Usage Contract Text Extract Season Adjust Daily Adjust Text Extract Degree History Compare History Store History x86 host x86 host x86 host x86 host x86 host

51 Streams Runtime Includes High Availability PEs on busy hosts can be moved manually by the Streams administrator A PE failing on one host can be moved automatically to another; communications are automatically rerouted Meters Meters Company Filter Usage Model Temp Action Usage Contract Text Extract Season Adjust Daily Adjust Text Extract Degree History Compare History Store History x86 host x86 host x86 host x86 host x86 host

52 Social Data Analytics Accelerator Architecture Social Media Online flow: Data-in-motion analysis Stream Computing and Analytics Real time analytics. Pre-defined views and charts Data Ingest and Prep Extract Buzz, Intent, Sentiment Entity Analytics: Profile Resolution Dashboard Social Media Data BigInsights System and Analytics Extract Buzz, Intent, Sentiment And Consumer Profiles Entity Analytics and Integration Comprehensive Social Media Customer Profiles Pre-defined Workbooks and Dashboards Offline flow: Data-at-rest analysis Data Explorer Index using Push API Ad hoc access Optional: Indexed Search

53 Social Data Analytics Accelerator

54 Machine Data Analytics Accelerator Preventing outages Data Administrator Data Scientist End User Import Logs Extract Transform Analyze Visualize Business requirement Improve ability to understand, correct and anticipate outages Solution Overview Provide faceted search across log records from multiple systems to find events Link and correlate events across systems Discover interesting patterns Solution Detail BigInsights applications for Import, Extract, Transform, Analyze, Visualize

56 The Future of Big Data and Cloud SQL for Hadoop support improvements towards full ANSI support Hive Impala (Cloudera) Big SQL (IBM) Stinger (Hortonworks) Drill (MapR) HAWQ (Pivotal) SQL-H (Teradata) Improvements in Multimedia Analytics Growth in usage and adoption of R programming language Cloud Bare metal support helping with Hadoop workloads Private network Full support with APIs

59 Big Data University (bigdatauniversity.com) BigInsights on the Cloud - Making Learning Hadoop Easy Flexible on-line delivery allows and Fun place pace Free courses, free study materials. Cloud-based sandbox for exercises zero setup with Robust Course Management System and Content Distribution infrastructure 108,000 registered students. Free IBM Hadoop, BigInsights Publications

60 Big Data University (bigdatauniversity.com) BigInsights on the Cloud - Making Learning Hadoop Easy and Quick Fun Start Editions available (Free, nonproduction, no time bomb): IBM InfoSphere BigInsights (IBM s Hadoop Distribution) ibm.co/quickstart IBM InfoSphere Streams ibm.co/streamsqs

61 My contact information My contact information Contact Info: Facebook: facebook.com/raul.f.chong LinkedIN: linkedin.com/pub/raul-f-chong/8/aa2/b63 61

62 Thank You! 2013 BigDataUniversity.com