Penang egovernment Seminar 2014 A New Era Of Analytic Megat Anuar Idris Head, Project Delivery, Business Analytics & Big Data
Agenda Overview of Big Data Case Studies on Big Data Big Data Technology Readiness & Expansion Big Data Initiatives
Common understanding Big Data Is High Performance Computing (HPC) Big Data Is About Massive Data Volume Big Data Means Hadoop (Big Data Framework) Big Data Need A Very Large Data Warehouse Big Data Means Collection of Unstructured Data Big Data Is for Social Media & Sentiment Analysis
Big Data Is, According to A new generation of technologies and architectures designed to economically extract value from very large volumes of a wide variety of data by enabling high-velocity capture, discovery, and/or analysis International Data Corporation (IDC) is the premier global provider of market intelligence, advisory services, and events for the information technology, telecommunications and consumer technology markets
Big Data Aggregation g Not just Very Very Large Data Volume
Industry Evolution..
Big Data Analytics Growing Focus in Malaysia www.bigdataanalytics.my/ www.facebook.com/dataanalytics.my
How BIG is Big Data?
Illustration 1
Illustration 2 1 GB = 1,000 MB 1 TB = 1,000 GB 1PB= 1,000 TB 1 EB = 1,000 PB 1 ZB = 1,000 EB 1 YB = 1,000 ZB 1 BB = 1,000 YB
Illustration 3
Illustration 4
Illustration 5
Where Is This Big Data Coming From? 12+ TBs of ftweet tdata every day 30 billion RFID tags today (1.3B in 2005) 4.6 billion camera phones world wide of >100 PBs data 500+ TBs of data every day 100s of millions of GPS enabled devices sold annually 2+ billion people 80 million smart on the meters in 2009 Web 200 million, 2014
4V Characteristics of Big Data. Cost efficiently processing the growing Volume 2010 Responding to the increasing Velocity 30 50x 35 Billion ZB 2020 RFID sensors and counting Collectively Analyzing the broadening Variety 80% of the worlds data is unstructured Establishing the Veracity of big data sources 1 in 3 business leaders don t trust the information they use to make decisions
With Big Data, We ve Moved into a New Era of Analytics 2.5+ quintillionbytes 5+ million of data created daily. trade events per second. Volume Velocity 100 s of different types of data. Variety Veracity 1in3 Only decision makers trust their information.
The number of organizations who see analytics as a competitive advantage is growing. g 63% 2010 BUSINESS 2011 IMPERATIVE 2012 business initiative IQ
Analytic With Data-In-Motion & Data At Rest 01011001100011101001001001001 0110100101010011100101001111001000100100010010001000100101 11000100101001001011001001010 01100100101001001010100010010 01100100101001001010100010010 11000100101001001011001001010 01100100101001001010100010010 01100100101001001010100010010 01100100101001001010100010010 01100100101001001010100010010 11000100101001001011001001010 Opportu unity Star rts Here st Nowcas Adaptive Analytics Model ast Foreca F 01100100101001001010100010010 01100100101001001010100010010 01100100101001001010100010010 01100100101001001010100010010 01100100101001001010100010010 11000100101001001011001001010 01100100101001001010100010010 01100100101001001010100010010 01100100101001001010100010010 11000100101001001011001001010 1
Illustration 6 Real-time Analytics World's largest community-based crowdsourced traffic and navigation information service application. 1.5 million users Malaysia (2013). 50 million global. Users are contributing tons of real-time traffic info. It s a world of connected, goal-driven intelligent agents that work collaboratively to solve problems Dynamic Case Management. Solution recommendation by the online analytic System is a combination of human and machine intelligence. New Business Process Management.
Agenda Overview of Big Data Case Studies on Big Data Big Data Technology Readiness & Expansion Big Data Initiatives
The 5 Key Big Data Use Cases Big Data Exploration Find, visualize, understand all big data to improve decision making Enhanced 360 o View of the Public Extend existing public views (MDM, CRM, etc) by incorporating additional internal and external e information sources Security/Intelligence Extension Lower risk, detect fraud and monitor cyber security in real-time Operations Analysis Analyze a variety of machine data for improved business results Data Warehouse Augmentation Integrate big data and data warehouse capabilities to increase operational efficiency
Big Data Exploration: Value & Diagram Find, Visualize & Understand all big data to improve business knowledge Greater efficiencies in business processes New insights from combining and analyzing data types in new ways Develop new business models with resulting increased market presence and revenue Application/ Users
The 5 Key Big Data Use Cases Big Data Exploration Find, visualize, understand all big data to improve decision making Enhanced 360 o View of the Public Extend existing public views (MDM, CRM, etc) by incorporating additional internal and external information sources Security/Intelligence Extension Lower risk, detect fraud and monitor cyber security in real-time Operations Analysis Analyze a variety of machine data for improved business results Data Warehouse Augmentation Integrate big data and data warehouse capabilities to increase operational efficiency
Enhanced 360º View of the Public: Needs Extend existing public views (MDM, CRM, etc) by incorporating additional internal and external information sources Need a deeper understanding of local/ public sentiment from both internal and external sources Desire to increase public trust and satisfaction by understanding what meaningful actions are needed Challenged getting the right information to the right people to provide public what they need to solve immediate problems
Enhanced 360º View of the Public
Enhanced 360º View of the Public Opinion Exposure based on Location
The 5 Key Big Data Use Cases Big Data Exploration Find, visualize, understand all big data to improve decision making Enhanced 360 o View of the Public Extend existing public views (MDM, CRM, etc) by incorporating additional internal and external e information sources Security/Intelligence Extension Lower risk, detect fraud and monitor cyber security in real-time Operations Analysis Analyze a variety of machine data for improved business results Data Warehouse Augmentation Integrate big data and data warehouse capabilities to increase operational efficiency
Security/Intelligence Extension: Needs Security/Intelligence Extension enhances traditional security solutions by analyzing all types and sources of under-leveraged data Enhanced Intelligence & Surveillance Insight Analyze data-in-motion & at rest to: Find associations Uncover patterns and facts Maintain currency of information Real-time Cyber Attack Prediction & Mitigation Crime prediction & protection Reduce Customer Churn Analyze network traffic to: Discover new threats early Detect known complex threats Take action in real-time Analyze Telco & social data to: Gather criminal evidence Prevent criminal activities Proactively apprehend criminals Customer Retention
Supplier Auditing for Performance Monitoring System Detailed Scoring Triplewise visualization shows how visually displaying the results of algorithmic data mining can show patterns of poor performance delivery Troubled suppliers?b 30
Supplier Auditing for Performance Monitoring System Detailed Scoring Triplewise visualization shows how visually displaying the results of algorithmic data mining can show patterns of poor performance delivery Troubled suppliers?b 31
The 5 Key Big Data Use Cases Big Data Exploration Find, visualize, understand all big data to improve decision making Enhanced 360 o View of the Public Extend existing public views (MDM, CRM, etc) by incorporating additional internal and external e information sources Security/Intelligence Extension Lower risk, detect fraud and monitor cyber security in real-time Operations Analysis Analyze a variety of machine data for improved business results Data Warehouse Augmentation Integrate big data and data warehouse capabilities to increase operational efficiency
Operations Analysis: Needs Analyze a variety of machine data for improved business results Business Challenges: Complexity and rapid growth of machine data Difficult to capture small fraction of machine for better decision In-ability to analyze machine data and combine it with enterprise data for a full view analysis Benefits: Gain real-time visibility into operations, customer experience, transactions and behavior Proactively plan to increase operational efficiency Identify and investigate anomalies Monitor end-to-end infrastructure to proactively avoid service degradation or outages
Operations Analysis: Multi-dimensional views Ra aw Logs and Mach hine Data a Only store what is needed Machine Data Accelerator Indexing, Search Statistical Modeling Root Cause Analysis Real-time Analysis Federated d Navigation & Discovery
The 5 Key Big Data Use Cases Big Data Exploration Find, visualize, understand all big data to improve decision making Enhanced 360 o View of the Public Extend existing public views (MDM, CRM, etc) by incorporating additional internal and external e information sources Security/Intelligence Extension Lower risk, detect fraud and monitor cyber security in real-time Operations Analysis Analyze a variety of machine data for improved business results Data Warehouse Augmentation Integrate big data and data warehouse capabilities to increase operational efficiency
Data Warehouse Augmentation: Needs Integrate big data and data warehouse capabilities to increase operational efficiency Need to leverage variety of data Extend warehouse infrastructure Structured, unstructured, and streaming Optimized storage, maintenance and data sources required for deep analysis licensing costs by migrating rarely used data Low latency requirements to Hadoop (hours not weeks or months) Reduced d storage costs through h smart Required query access to data processing of streaming data Improved warehouse performance by determining what data to feed into it
Data Warehouse Augmentation: Needs
Agenda Overview of Big Data Case Studies on Big Data Big Data Technology Readiness & Expansion Big Data Initiatives
Big Data Platform and Products/Tools
Big Data Strategy: Move the ANALYTICS Closer to the Data Analytic Applications New analytic applications drive the BI / Exploration / Functional Industry Predictive Content Reporting Visualization App App Analytics Analytics requirements for a big data platform Visualization & Discovery Hadoop Hadoop System Big Data Platform Application Development Accelerators Stream Computing Systems Management Data Warehouse Information Integration & Governance Integrate and manage the full variety, velocity and volume of data Apply advanced analytics to information in its native form Visualize all available data for ad-hoc analysis (even in motion!) Development environment for building new analytic applications Workload optimization and scheduling Security and Governance Grow and evolve on your current IT infrastructure re
Big Data Platform : Typical Infrastructure (Hadoop) Analytics Server & Tool Data Warehouse Hadoop Cluster Management/ Analysts
Big Data Platform : Typical Infrastructure (DW) Analytics Server & Tool Storage Area Network (SAN) Data Warehouse Server (Single/Cluster) DW Software Licenses Operating System Licenses Management/ Analysts
Big Data Platform : Appliance Analytics Server & Tool Big Data Appliance Management/ Analysts
Big Data Platform : Appliance
Big Data Platform : Appliance
Record loading performance : 315,532 per sec 18,931,920 per min 1,135,915,200 per hour
Big Data Strategy: Move the ANALYTICS Closer to the Data
Agenda Overview of Big Data Case Studies on Big Data Big Data Technology Readiness & Expansion Big Data Initiatives
Big Data Strategy : A Must for Better Analytics
The 5 Key Big Data Components to Get Started Big Data Exploration Find, visualize, understand all big data to improve decision making Enhanced 360 o View of the Public Extend existing public views (MDM, CRM, etc) by incorporating additional internal and external e information sources Security/Intelligence Extension Lower risk, detect fraud and monitor cyber security in real-time Operations Analysis Analyze a variety of machine data for improved business results Data Warehouse Augmentation Integrate big data and data warehouse capabilities to increase operational efficiency
Studies show that organizations competing on analytics outperform their peers substantially outperform IBM IBV/MIT Sloan Management Review Study 2011 Copyright Massachusetts Institute of Technology 2011 16 1.6x Revenue 25 2.5x 20 Stock Price 5 Growth Appreciation 2.0x EBITDA Growth
Big Data Analytics : New Job Data Scientist https://datajobs.com/big-data-salary
Penang egovernment Seminar 2014 Thank you