A New Era Of Analytic

Transcription

1 Penang egovernment Seminar 2014 A New Era Of Analytic Megat Anuar Idris Head, Project Delivery, Business Analytics & Big Data

2 Agenda Overview of Big Data Case Studies on Big Data Big Data Technology Readiness & Expansion Big Data Initiatives

3 Common understanding Big Data Is High Performance Computing (HPC) Big Data Is About Massive Data Volume Big Data Means Hadoop (Big Data Framework) Big Data Need A Very Large Data Warehouse Big Data Means Collection of Unstructured Data Big Data Is for Social Media & Sentiment Analysis

4 Big Data Is, According to A new generation of technologies and architectures designed to economically extract value from very large volumes of a wide variety of data by enabling high-velocity capture, discovery, and/or analysis International Data Corporation (IDC) is the premier global provider of market intelligence, advisory services, and events for the information technology, telecommunications and consumer technology markets

5 Big Data Aggregation g Not just Very Very Large Data Volume

6

7 Industry Evolution..

8 Big Data Analytics Growing Focus in Malaysia

9 How BIG is Big Data?

10 Illustration 1

11 Illustration 2 1 GB = 1,000 MB 1 TB = 1,000 GB 1PB= 1,000 TB 1 EB = 1,000 PB 1 ZB = 1,000 EB 1 YB = 1,000 ZB 1 BB = 1,000 YB

12 Illustration 3

13 Illustration 4

14 Illustration 5

15 Where Is This Big Data Coming From? 12+ TBs of ftweet tdata every day 30 billion RFID tags today (1.3B in 2005) 4.6 billion camera phones world wide of >100 PBs data 500+ TBs of data every day 100s of millions of GPS enabled devices sold annually 2+ billion people 80 million smart on the meters in 2009 Web 200 million, 2014

16 4V Characteristics of Big Data. Cost efficiently processing the growing Volume 2010 Responding to the increasing Velocity 30 50x 35 Billion ZB 2020 RFID sensors and counting Collectively Analyzing the broadening Variety 80% of the worlds data is unstructured Establishing the Veracity of big data sources 1 in 3 business leaders don t trust the information they use to make decisions

17 With Big Data, We ve Moved into a New Era of Analytics 2.5+ quintillionbytes 5+ million of data created daily. trade events per second. Volume Velocity 100 s of different types of data. Variety Veracity 1in3 Only decision makers trust their information.

18 The number of organizations who see analytics as a competitive advantage is growing. g 63% 2010 BUSINESS 2011 IMPERATIVE 2012 business initiative IQ

19 Analytic With Data-In-Motion & Data At Rest Opportu unity Star rts Here st Nowcas Adaptive Analytics Model ast Foreca F

20 Illustration 6 Real-time Analytics World's largest community-based crowdsourced traffic and navigation information service application. 1.5 million users Malaysia (2013). 50 million global. Users are contributing tons of real-time traffic info. It s a world of connected, goal-driven intelligent agents that work collaboratively to solve problems Dynamic Case Management. Solution recommendation by the online analytic System is a combination of human and machine intelligence. New Business Process Management.

21 Agenda Overview of Big Data Case Studies on Big Data Big Data Technology Readiness & Expansion Big Data Initiatives

22 The 5 Key Big Data Use Cases Big Data Exploration Find, visualize, understand all big data to improve decision making Enhanced 360 o View of the Public Extend existing public views (MDM, CRM, etc) by incorporating additional internal and external e information sources Security/Intelligence Extension Lower risk, detect fraud and monitor cyber security in real-time Operations Analysis Analyze a variety of machine data for improved business results Data Warehouse Augmentation Integrate big data and data warehouse capabilities to increase operational efficiency

23 Big Data Exploration: Value & Diagram Find, Visualize & Understand all big data to improve business knowledge Greater efficiencies in business processes New insights from combining and analyzing data types in new ways Develop new business models with resulting increased market presence and revenue Application/ Users

24 The 5 Key Big Data Use Cases Big Data Exploration Find, visualize, understand all big data to improve decision making Enhanced 360 o View of the Public Extend existing public views (MDM, CRM, etc) by incorporating additional internal and external information sources Security/Intelligence Extension Lower risk, detect fraud and monitor cyber security in real-time Operations Analysis Analyze a variety of machine data for improved business results Data Warehouse Augmentation Integrate big data and data warehouse capabilities to increase operational efficiency

25 Enhanced 360º View of the Public: Needs Extend existing public views (MDM, CRM, etc) by incorporating additional internal and external information sources Need a deeper understanding of local/ public sentiment from both internal and external sources Desire to increase public trust and satisfaction by understanding what meaningful actions are needed Challenged getting the right information to the right people to provide public what they need to solve immediate problems

26 Enhanced 360º View of the Public

27 Enhanced 360º View of the Public Opinion Exposure based on Location

28 The 5 Key Big Data Use Cases Big Data Exploration Find, visualize, understand all big data to improve decision making Enhanced 360 o View of the Public Extend existing public views (MDM, CRM, etc) by incorporating additional internal and external e information sources Security/Intelligence Extension Lower risk, detect fraud and monitor cyber security in real-time Operations Analysis Analyze a variety of machine data for improved business results Data Warehouse Augmentation Integrate big data and data warehouse capabilities to increase operational efficiency

29 Security/Intelligence Extension: Needs Security/Intelligence Extension enhances traditional security solutions by analyzing all types and sources of under-leveraged data Enhanced Intelligence & Surveillance Insight Analyze data-in-motion & at rest to: Find associations Uncover patterns and facts Maintain currency of information Real-time Cyber Attack Prediction & Mitigation Crime prediction & protection Reduce Customer Churn Analyze network traffic to: Discover new threats early Detect known complex threats Take action in real-time Analyze Telco & social data to: Gather criminal evidence Prevent criminal activities Proactively apprehend criminals Customer Retention

30 Supplier Auditing for Performance Monitoring System Detailed Scoring Triplewise visualization shows how visually displaying the results of algorithmic data mining can show patterns of poor performance delivery Troubled suppliers?b 30

31 Supplier Auditing for Performance Monitoring System Detailed Scoring Triplewise visualization shows how visually displaying the results of algorithmic data mining can show patterns of poor performance delivery Troubled suppliers?b 31

32 The 5 Key Big Data Use Cases Big Data Exploration Find, visualize, understand all big data to improve decision making Enhanced 360 o View of the Public Extend existing public views (MDM, CRM, etc) by incorporating additional internal and external e information sources Security/Intelligence Extension Lower risk, detect fraud and monitor cyber security in real-time Operations Analysis Analyze a variety of machine data for improved business results Data Warehouse Augmentation Integrate big data and data warehouse capabilities to increase operational efficiency

33 Operations Analysis: Needs Analyze a variety of machine data for improved business results Business Challenges: Complexity and rapid growth of machine data Difficult to capture small fraction of machine for better decision In-ability to analyze machine data and combine it with enterprise data for a full view analysis Benefits: Gain real-time visibility into operations, customer experience, transactions and behavior Proactively plan to increase operational efficiency Identify and investigate anomalies Monitor end-to-end infrastructure to proactively avoid service degradation or outages

34 Operations Analysis: Multi-dimensional views Ra aw Logs and Mach hine Data a Only store what is needed Machine Data Accelerator Indexing, Search Statistical Modeling Root Cause Analysis Real-time Analysis Federated d Navigation & Discovery

35 The 5 Key Big Data Use Cases Big Data Exploration Find, visualize, understand all big data to improve decision making Enhanced 360 o View of the Public Extend existing public views (MDM, CRM, etc) by incorporating additional internal and external e information sources Security/Intelligence Extension Lower risk, detect fraud and monitor cyber security in real-time Operations Analysis Analyze a variety of machine data for improved business results Data Warehouse Augmentation Integrate big data and data warehouse capabilities to increase operational efficiency

36 Data Warehouse Augmentation: Needs Integrate big data and data warehouse capabilities to increase operational efficiency Need to leverage variety of data Extend warehouse infrastructure Structured, unstructured, and streaming Optimized storage, maintenance and data sources required for deep analysis licensing costs by migrating rarely used data Low latency requirements to Hadoop (hours not weeks or months) Reduced d storage costs through h smart Required query access to data processing of streaming data Improved warehouse performance by determining what data to feed into it

37 Data Warehouse Augmentation: Needs

38 Agenda Overview of Big Data Case Studies on Big Data Big Data Technology Readiness & Expansion Big Data Initiatives

39 Big Data Platform and Products/Tools

40 Big Data Strategy: Move the ANALYTICS Closer to the Data Analytic Applications New analytic applications drive the BI / Exploration / Functional Industry Predictive Content Reporting Visualization App App Analytics Analytics requirements for a big data platform Visualization & Discovery Hadoop Hadoop System Big Data Platform Application Development Accelerators Stream Computing Systems Management Data Warehouse Information Integration & Governance Integrate and manage the full variety, velocity and volume of data Apply advanced analytics to information in its native form Visualize all available data for ad-hoc analysis (even in motion!) Development environment for building new analytic applications Workload optimization and scheduling Security and Governance Grow and evolve on your current IT infrastructure re

41 Big Data Platform : Typical Infrastructure (Hadoop) Analytics Server & Tool Data Warehouse Hadoop Cluster Management/ Analysts

42 Big Data Platform : Typical Infrastructure (DW) Analytics Server & Tool Storage Area Network (SAN) Data Warehouse Server (Single/Cluster) DW Software Licenses Operating System Licenses Management/ Analysts

43 Big Data Platform : Appliance Analytics Server & Tool Big Data Appliance Management/ Analysts

44 Big Data Platform : Appliance

45 Big Data Platform : Appliance

46 Record loading performance : 315,532 per sec 18,931,920 per min 1,135,915,200 per hour

47 Big Data Strategy: Move the ANALYTICS Closer to the Data

48 Agenda Overview of Big Data Case Studies on Big Data Big Data Technology Readiness & Expansion Big Data Initiatives

49 Big Data Strategy : A Must for Better Analytics

50 The 5 Key Big Data Components to Get Started Big Data Exploration Find, visualize, understand all big data to improve decision making Enhanced 360 o View of the Public Extend existing public views (MDM, CRM, etc) by incorporating additional internal and external e information sources Security/Intelligence Extension Lower risk, detect fraud and monitor cyber security in real-time Operations Analysis Analyze a variety of machine data for improved business results Data Warehouse Augmentation Integrate big data and data warehouse capabilities to increase operational efficiency

51 Studies show that organizations competing on analytics outperform their peers substantially outperform IBM IBV/MIT Sloan Management Review Study 2011 Copyright Massachusetts Institute of Technology x Revenue x 20 Stock Price 5 Growth Appreciation 2.0x EBITDA Growth

52 Big Data Analytics : New Job Data Scientist

53 Penang egovernment Seminar 2014 Thank you