HP Big Data & Analytics for CSPs and Customers Fouad Bendris / Big Data Lead Enterprise Group PreSales & Strategic Pursuits - EMEA
Agenda Big Data Trends - Challenges & Opportunities Haven Platform : The Big Data foundation HP SPS - Haven for CSPs Use Cases Predictive & Prescriptive Analytics 2
Big Data Trends! Challenges & Opportunities
IoT - Internet of things The new Data tsunami! Big Data Big Volume! We have gone beyond the decimal system In the near future, Brontobyte will be the measurement to describe the type of sensor data that will be generated from the IoT (Internet of Things) 4 This is our digital universe today = 250 trillion of DVDs Exabyte 1 EB of data is created on the internet each day = 250 million DVDs worth of information. The proposed Square Kilometer Array telescope will generated an EB of data per day 10 24 10 27 Brontobyte This will be our digital universe tomorrow Zettabyte Yottabyte 10 21 1.3 ZB of network traffic Terabyte 500TB of new data per day are ingested in Facebook databases Megabyte 10 18 10 12 10 15 Petabyte 10 6 10 9 Gigabyte by 2016 The CERN Large Hadron Collider generates 1PB per second
Expansion of the Digital Universe Big Volume = Big opportunity? 2005 2010 0.1ZB 1.2ZB 2012 2.8ZB 2015 8.5ZB Growing by a Factor of 33 in 10 years! More than doubling every 2 years 70 % created by individuals Mostly Unstructured Data 2020 40ZB IDC projects that the Digital Universe will reach 40 ZB by 2020, an amount that exceeds previous forecasts by 5 ZBs. *Digital Universe = Digital information created or captured in a year 5
Accelerating Innovation & Change AOL took 9 years to get to 1 million users Draw Something took 9 days... Facebook took 9 months 6
The Age of Unicorns The billion-dollar tech startup was supposed to be the stuff of myth Now they seem to be everywhere! 7
The changing Big Data landscape Insight from 100% of Data ~100% Machine Data 90% of Information Human Information Annual Growth ~10% Business Data 8 10% of Information
Analytics by numbers: external challenges Required new forms of processing to enable decision making Traditional Enterprise Data 75% of currently deployed data warehouses will not scale sufficiently to meet demand by 2016 1 Big Data 56% Aren t sure how to get value from Big Data 2 Dark Data CRM ERP Data Warehouse Web Social Log Files Machine Data Semi-structured Unstructured Note: Dark data is a type of unstructured, untagged and untapped data that is found in data repositories and has not been analyzed or processed 9
Business drivers Adopting Big Data is not a matter of If, but When! Solutions to problems not solvable by traditional approach When existing traditional Data processing solutions can t scale anymore When it is too expensive to extend existing Data processing solutions 10 Opportunities To handle new types of data unstructured and semi-structured To value data that are not used dark data To reduce costs for data storage and processing To speed up processes become more competitive, more profitable To react in real time interact in real time with customers, partners, suppliers, employees, To analyse and forsee the customer behavior (predictive analysis) To develop new product and service offerings based on data
How Telco s can leverage Big Data
How Telcos can use big data New revenues stream Process optimization Cloud CEM Advertising CDN & IPTV Network Analytics Actionable Experience OTT Co-petition B2B M2M Personalize d Mobile Experience B2C Churn Mgt Ecosystem Indoor Positioning Services Mobility E- Commerce Revenue Intelligence Mobility Next Best Offer 12 In all these initiatives - Big Data & Analytics play a significant role!
Low Complexity High Complexity CSPs Big Data : Market Evolution and Economical Impact Customer Experience Management Challenging Today Strategic Next 3 years 13 User Data Consolidatio n Subscription Management & control Advanced Churn and Loyalty Management Archive Optimization Customer experience Assurance Customer Self-care Subscriber Network usage analytics Automatied promotions Revenue Assurance & Protection Hanging Fruits Subscriber preference & Information brokering Subscriber digital identity Intelligent IP routing Customer Care optimization (call reduction) Mobile Advertising Content Recommendation Next best Action engine Automatic problem diagnostic Adoption Complexity Customer Care optimization (Automatic resolution) Next Best Offer (Marketing automation) Intelligent Firewalls Analytics as a Services M2M Analytics WiFi Indoor analytics Actionable Networks (PCRF 2.0) Store of the future B2B Value Added Services Tactical Automatic cyber attack prevention Augmented reality Customer Experience Management Area Actionable Networks ( NFV) Analytics will drive network optimization Time to adopt Use case evolution path Analytics will support new revenue generation Analytics will improve security and revenue protection Analytics will drive Marketing personalization
The new Analytics Paradigm 1.0 Descriptive Analytics what did happen and why it did happen? KPIs, KQIs, KBIs, 2.0 Predictive Analytics what is likely to happen now? Net Promoter Scores, Churn Scores, 3.0 Prescriptive Analytics What should I do now? Promote, Recommend, Upsell, 14
From Batch To Actionable Information Actionable Real time Streaming and Pattern Detection Predictive Prescriptive Data Value Required Market Focus Interactive Parameterized reports Drilldown visualization Exploration Non-interactive Data preparation Incremental batch processing Dashboards, scorecards Batch Current Market Focus: Data Warehousing and Analytics Operational batch processing Enterprise reports Data mining Sub-second Response Time Hours 15 Transaction Support Data Integrity Real-time Performance Operational Query Optimization Workload Management
Introduction to Haven
HP Haven Big Data platform HP applications Customer applications Developer applications Gain insight from 100% of your data Analyze machine, business, human data Connect to any existing data source system Scale 50-1000x faster than legacy systems Develop modern data-driven applications & web services Haven Defined programming interfaces Analytics, context and categorization Premise & OnDemand Scalable data stores & data connectors Social media Video Audio Email Texts Mobile Transactional Documents IT/OT Search data engine Images Records Compliance archives 17
Haven Big Data Platform Haven Hadoop/ HDFS Autonomy IDOL Vertica Enterprise Security n Apps Catalog massive volumes of distributed data Process and index all information Analyze at extreme scale in real-time Collect & unify machine data Powering HP Software + your apps 18 Transactional Social media Video Audio Email Texts Mobile data Documents IT/OT Search engine Images
HP Haven Hadoop 19 HP Reference Architectures and Services for Apache Hadoop Helps design and implement a high-performance, integrated Big Data platform Hadoop Map Reduce Powerful distributed Data processing framework (Java, batch oriented) Hadoop Distributed FileSystem (HDFS) PetaByte scale clustered storage solution with inherent High Availability and Performance features HP Reference Architectures for Hadoop Comprehensive, pre-tested, reference architectures (Cloudera, MapR, Hortonworks) HP Enterprise Design Service for Hadoop Detailed specification design for Hadoop distributions as the guideline for a planned implementation HP Implementation Service for Hadoop Provides risk reduction based on skill sets, project acceleration as a result of applied experience. HP consultants install, configure, deploy the client s architecture, and test it Strategy, design, implementation and infrastructure support
HP Haven Hadoop 5 Reasons Why 1 Open Uses Commodity Hardware - no expensive or proprietary hardware required 2 Flexible Supports and processes all data Structured and Unstructured 3 Open Source Developed by hundreds, commercially supported by many 4 ROI Scales horizontally using commodity infrastructure only buy what you need 5 Safe Bet Supported and backed by the largest IT companies in the World Who Uses Hadoop? Large-scale web companies : Yahoo, Facebook, Twitter, Linked-in, ebay, Amazon, Banks, Telco's, Energy providers, Retails, Healthcare, Governments, and HP! HP provides tuned, ProLiant based, reference architectures for Hadoop
HP Haven Autonomy IDOL HP Autonomy IDOL platform High-performance human information processing 400+ connectors Seamlessly access virtually any enterprise content repository, including file systems, email, or knowledge bases 1,000+ file types Process virtually any file type such as text (email, tweet, document), audio, video, and even people profiles & behavior Over 500 functions Leverage the power of functions like sentiment, categorization, and clustering to deliver intelligence and insight. 15 years and over $280M in R&D, >170 Patents Distributable architecture Achieve big data scalability and high performance with distributable ingest and query architecture All data types, all content repositories unmatched understanding 21
HP Haven - Vertica HP Vertica Analytics platform High-performance data analytics platform purpose-built for big data Blazing fast analytics Gain insight into your data in near-real time by running queries 50x-1,000x faster than legacy products Massive scalability Infinitely scale your solution by adding an unlimited number of industry-standard servers Open architecture Protect and embrace your investment in hardware and software with built-in support for Hadoop, R, and a range of ETL and BI tools Optimized data storage Store 10x-30x more data per server than row databases with patented columnar compression Speed, scalability, and openness at lower TCO 22
HP Haven ESP ArcSight HP ArcSight Universal log management platform High-performance universal log management to consolidate machine data across IT 315+ connectors Collect, normalize, and categorize machine data such as logs, events, and flows from any device, any time, anywhere from any vendor Search over 1,000,000 events per second The unified machine data through filtering and parsing is enriched with rich metadata, which allows you to search machine data through simple text-based keywords without the need of domain expertise Store years worth of data The unified data is stored through high compression ratio in any of your existing storage formats, eliminating the need for expensive databases and DBAs Analytics & intelligence Built-in content packs, algorithms, rules, and the unified machine data help you deploy IT security, IT operations, IT GRC, and log analytics Collect, store, and analyze any machine data across IT 23
n Apps, the n in Haven http://dev.hp.com/haven HP Operations Analytics HP Service Anywhere HP Digital Marketing Hub Analytical security Commercial insurance risk assessment Ecommerce predictive analytics 24 Healthcare analytics Insider threat analysis Service analytics for manufacturing
COLLECTION MANAGEMENT CONSUMPTION Analytics platform High level functional Architecture BUSINESS ANALYSTS MANAGEMENT DATA SCIENTISTS APPLICATIONS! Structured Analysis Multi format Analysis Unstructured Analysis Adhoc Reports OLAP Canned Reports Alerts KPI Dashboards Real time Scoring Interaction Optimization Analytic DW Unstructured Human Data Business Data System Data Relational DW Unstructured Human Interaction Data Custom Semi-structured/System Data Enterprise Application Data Chat Call Center Web & Mobile Social Media Audio Video email News Media Business Owned 3rd Party Logs Config Status Sensor Data Finance Sales & Marketing Services Supply Chain 25
HP & Partner Services ALM BSM ESM Haven Framework Data Flows Apps HP Ops Analytics HP Promote Community & Partner Visual Reporting & Visualization (Tableau, MicroStrategy, SaS, SAP BO, Etc.) Process Integrated Analytics ( SAS, Revolution Analytics, INFR, etc.) Big Data Storage Hadoop Unstructured Storage In Database Analytics (SQL, R & KXEN) Vertica Interactive Analytic Accelerator Source Data Autonomy Unstructured Categorization Arcsight Logger Sensor & System Generated Vertica Structured Data Develop, Deploy, Secure & Manage 26 Transactional Social media Video Audio Email Texts Mobile data Documents IT/OT Search engine Images
Example: Analyzing Twitter Feeds with Haven Extracted tweets from Hadoop Structured tweets in Vertica Sentiments with Autonomy Tableau dashboard HP Anywhere mobile platform 27
HP SPS (Smart Profile Server) Haven for CSPs
HP Smart Profile Server: Big Data & Analytics Platform Applications (specific analytic use cases) Targeted products and marketing offers IT and Network Experience Optimization New business Models enablement Proactive Data Strategies Software ( BigData architecture) VERTICA - Structured Analytic Database IDOL - Unstructured Analytic Engine HADOOP - Distributed Big Data file systems HAVEn + eium Real Time data collection SPS Analytics run-time engine and designer DRAGON Blue DPI network probes DEL Data Exposure Layer Data Sources Market data Subscriber data Network data Application data 29 Transactional Social media Video Audio Email Texts Mobile data Documents IT/OT Search engine Images
HP & Partner Services HP Smart Profile Server: Haven for Telco s Apps Visual Process HP Ops Analytics HP Promote Community & Partners Reporting & Visualization (Tableau, MicroStrategy, SaS, SAP BO, Etc.) Integrated Analytics ( SAS, Revolution Analytics, etc) SPS Value Packs Campain Mgr 3 rd party Businesses RT Reports/Notifications/APIs RT Value Packs Big Data Storage Hadoop Unstructured Storage SPS / Vertica Interactive Analytics SQL, R Industry Data Models SPS CEP Real-time Analytics Source Data Autonomy Unstructured Categorization Arcsight Logger Security Logs SPS / Vertica Structured Data CEP Streaming Data 30 Transactional Social media Video Audio Email Texts Mobile data Documents IT/OT Search engine Images
Complex Event Processing Requirements for CSPs A massively scalable, real time Complex Event Processing Ability to process millions of events/streams per seconds onfew nodes of commodity hardware Ability to Combine of rules based pattern detection and online based calculations in a box (fraud management, customer experience management, social networks, campaigns feedbacks, clickstreams) A Simplified usage across the event processing lifecycle (development, configuration, deployment, Monitoring) Ability to develop and to deploy rules, kpi logic in minutes Ability to Combine pre-built operators (filters, aggregator, correlators ) without specific skills Ability to Update pattern detection logic based at runtime Transformation of streams into insights with low latency Ability to Collect, detect and notify in milliseconds to update business processes in real-time (Next best Offer, Marketing Campaign optimization) 31
Complex Event Processing Terminology Streaming Data : Data coming as a continuous flow from one or multiple sources, synchronous or asynchronous. Streams/events: are tuplets of data regrouping multiple fields such as, session events (start, duration etc ) and measurements (volume, latency measurements) and context/content information such as location, MSISDN, tweets.. Real-time: SPS CEP is designed to process streaming data in real-time e-g between a few milliseconds to a second processing time Event Processing Graphs: SPS CEP Engine executes Event Processing Graphs (EPG) a-k-a topologies. An EPG is a combination of collectors, event processing agents (EPA) and notifiers. A EPG is materially a Storm Topology embedding, some HP IP, a streaming SQL language runtime. They are deployed and run with SPS CEP real-time engine. Their lifecycle (start, stop, ) is done through SPS operation console Event Processing Agents (EPA): are implementing the online calculation/pattern detection logic SPS CEP Designers: are used to build Specify and build automatically EPGs as part of Value Packs SPS Analysis: a set of customizable CEP EPG to allow rapid specification of business rules by business analysts. The SPS BRE relies on SPS CEP technology. 32
Use Cases with SPS Predictive & Prescriptive Analytics
HP SPS Value Packs - Specific Analytics Packages for CSPs Advertising Next Best Offer Analyzing Subscriber s traffic and application usage to discovery the best tariff plan and best offer to propose for them Ad Experience Personalization Enabling customers to partner with Ad networks, search engines and commerce sites Churn Management Analyzing All Subscriber's inputs (data, traffic, call center requests, application usage, social information, trouble tickets', ) to identify churn probability and propose mitigate actions (new plans, discounts, free services, etc.) Massive Network Traffic Storage Helping Customers to optimize data archives size Content Delivery Network Analytics Real Time analysis of IPTV and Content Delivery Network helps recognize how media contents are consumed by subscriber. Actionable Experience Management This solution helps Customerss Operators to immediately resolve issues of customer experience. Subscriber Network Usage Analytics Real-time analysis of Network usage helps recognize situations of service cannibalization between different services or by OTT services. Customer Experience Management Analyze e2e networks and services to proactively determine subscriber impact and manage subscriber relationship. 34 Revenue Assurance Helping CSPs can reduce or eliminate subscription fraud and proactively recognize Churn. Mobile Experience Personalization Customers can better connect and engage with their users using a mobile app installed on their device.
What is NBO? Next-best offer makes use of the predictive analytics and complex event processing technologies to identify the products or services that customers are most likely to be interested in for their next purchase NBO Trigger based marketing Target a customer with his specific need Collaborative Filtering Product preference Computer Based on trouble ticket system and KPI Engine Content Based Recommender Search customers to inform about the offer Find and suggest the best offers to him SPS Recommendation Engine Weighted Average Computer CURRENT FUTURE 35
Next best offer Relevant, Contextual and Personalized promotions Tier One Russia: CPR Increase by 5%* Call Center Network Events RT Analytics and Recommendation Engines (HP Smart Profile Server) Internet Email Usage Data Audio Audio Texts CRM Marketing Data Campaign and promotions engine (HP Campaign Manager) Offers Catalogue Self-Care Mobile interaction (HP SPS MEP) Optional module SMS Smart App Shop 3 rd party Business Self-Care Customer profile management Customer Social Media CDR/XDR Network Documents $ Images Transactional data Customer traffic & service usage Real 36 value to customer + Revenue uptake + Relationship interactions + Churn Mitigation
Where NBO Is Used? Harvest customer leads from sales, marketing & other sources Insights into customer acquisitions Target offers to boost customer lifetime value Insights to tailor product creation Improved customer experience Achieving Better ROI Campaign efficiency Incentive management Minimize the net present value of cost for acquiring, retaining, and growing relationship with customer 37
Subscriber network usage analytics Enabling CSPs to identify customer behavior inside mobile Internet traffic 38
Personalized Advertising Benefits for subscriber and the CSP Get relevant promotions or advertising Get discounts or special offers for CSPs services Enable new revenue streams (3rd party advertising) Improve customer loyalty 84% of Mobile subscribers are Willing to sign up for a plan in which they accept advertising in exchange for free features. Subscriber CSP ~$20B annual revenue by 2015 91% CAGR. 39 Bell Canada
Customer experience assurance Empower growth through improved customer loyalty and preference Big Data Analytics 40
Why HP? hp.com/go/bigdata hp.com/go/haven hp.com/go/telcobigdata 41 Make Data Matter!
Thank You