HP Software Big Data overview

Transcription

1 HP Software Big Data overview 23 rd April 2015 David Kemp, Big Data Consultant

2 Big Data from the Internet of Things We have gone beyond the decimal system Today, data scientists use Yottabytes to describe how much government data the NSA or FBI have on people altogether. In the near future, a Geopbyte will be the measurement to describe the type of data generated from the IOT. Brontobyte This will be our digital universe tomorrow Yottabyte This is our digital universe today Exabyte 1 EB of data is created on the internet each day Terabyte 500TB of new data per day are ingested in Facebook databases Megabyte Geopbyte This will take us beyond our decimal system Zettabyte 1.3 ZB of network traffic by Petabyte The CERN Large Hadron Collider generates 1PB per second 10 9 Gigabyte 2

3 Enterprise data growth Burroughs IBM NEC Costs of managing data Unisys Hitachi Product configurator Mainframe Kilobytes Fijitsu Bull Claim processing Payroll Sales tracking & marketing Commissions Bills of material SCM ERP Costing Database Manufacturing projects CRM Quality control HCM SAP Engineering HP MRM Inventory EMC Cost management Cash management Time and Expense Accounts receivable Billing Activity management Training Time & attendance Data warehousing Service Order entry Joyent Client/server Megabytes Rostering HCM Fixed assets PLM Intacct DCC Saba CCC IntraLinks Adobe Microsoft Cornerstone ondemand Softscape NetSuite Plex Systems ebay Quickbooks Kenexa OpSource Hosting.com Tata Communications Datapipe PPM NetDocuments Microsoft Google Alterian Corel Yahoo Saba Softscape Volusion Ariba OpenText Workscape ADP VirtualEdge Quadrem Xerox SugarCRM FinancialForce.com Avid NetReach Zoho Serif Yahoo! CyberShift Sage Sonar6 Hyland Qvidian Music kaggle SuperCam SLI Systems Elemica SCM Kinaxis Xactly CyberShift SmugMug ihandy The Internet Gigabytes NetSuite Exact Online PaperHost Sonar6 Facebook Fring Rackspace SolidFire Snapfish Dragon Diction GoGrid Cookie Doodle Ah! Fasion Girl buzzd Renren Atlassian Paint.NET DocuSign Hootsuite Education Flickr UPS Mobile salesforce.com Zynga Pandora Amazon Bromium Scanner Pro Foursquare nebula ischedule BrainPOP Khan Academy MobileFrame.com myhomework Toggl Xing MailChimp AppFog Amazon Web Services LimeLight News LinkedIn Workday Navigation SuccessFactors Associatedcontent MobilieIron Twitter PingMe cloudability CloudSigma HP eprint Fed Ex Mobile RightScale YouTube Business Amazon Utilities Scribd. SmugMug Zillabyte Reference Games Productivity TripIt Atlassian Twitter Baidu Atlassian Tumblr. New Relic Urban Parse Yandex Finance Entertainment Jive Software Qzone Travel Lifestyle box.net Sport ScaleXtreme Workbrain Yandex Mozy Zynga Mixi Heroku CYworld BeyondCore Taleo Splunk Pinterest Mobile, social, Big Data & the cloud Zettabytes Photo & Video Yammer Answers.com Viber Social networking dotcloud PingMe Every 60 seconds 1,820 TB of data created The volume, velocity and breadth of channels often overwhelms Information Management strategies leading to dark data Storage costs are visible, soft costs such as opportunity & risk costs are less so, but no less real TCO for unstructured data varies between $4/GB to $100/GB annually, but $25GB is a good rule of thumb* 3 *Source: ESG White Paper The Cost of Managing Unstructured Data, May 2014

4 Acceleration Driven by Technology Advancements Big Data End Point Virt & Cloud Always-On Data doubles every months New, unstructured data types to manage e.g. Hadoop Storage growth is outpacing IT budget Data is everywhere, on a sea of devices 63% of enterprises have no formal endpoint protection in place Windows XP EOL will force the refresh of millions of enterprise PCs 11% of organizations changed their backup & recovery solution in 2013, triple the rate of the previous year Physical, virtual, cloud & hybrid environments Emerging workload specific, hybrid solutions, cloud storage Escalating regulatory and higher volumes Bar has been raised from data protection to delivering business resiliency Data accessibility from Anywhere, Anytime 4

5 Internal challenges for all corporates for Information Governance & Management Corporate Governance monitoring and enforcement Social media monitoring - internal & external where permitted Ability to freeze data across a complex IT legacy architecture Cross-media visibility and comprehension De-duplication, clustering and synthesis of mass data Necessity to respect national and international data privacy standards Fast and effective response to the Business 5

6 External increasing regulatory powers: EMEA Rule / Regulation A/K/A Summary Bribery Act 2011 BA UK Worldwide sanctions by UK Serious Fraud Office including fines and jail for condoning senior management Spanish Criminal Code anti- Bribery Law 13 March 2014 SBL Worldwide sanctions by Spanish courts now with jurisdiction outside Spain for allegations against a Spanish citizen / habitual Spanish resident, employee of Spanish firm EU Competition Law ECL Anti-trust raid, investigation, sanctions with National Police support for suspected anticompetitive behavior by National entities with sanctions up to 10% of annual sales. Market Abuse Directive updated April 2014 effective June 2014 European Markets Infrastructure Regulation Data Protection Regulation approved 11 March 2014 by EU Parliament FCA Dealing Room Audio Records Management November MAD EMIR DPR DRAR Aligns MAD with Markets in Financial Instruments Directive includes market manipulation of spot commodity contracts, market abuse in Algorithmic & High Frequency Trading and Insider Information planning breach Governance of OTC derivatives for timely risk mitigation, reconciliation and reporting etc updated May 2014 Update of the 1995 Regulation requiring higher standards of data privacy, the right to be forgotten as well as new corporate enforcement in the light of reasonably justified compliance breach expected effective 2015 Mandatory capture of all mobile device traffic effecting trades on UK dealing floors and retention for 3 years so Constructive Knowledge of content 6 Basel III introduction Basel III Capital adequacy, stress testing and market liquidity risk priorities including additional capital buffers and increase in minimum leverage rate

7 7

8 What is legacy data and dark data? Redundant, obsolete, trivial, and the unknown Legacy data resides in: Legacy applications and repositories Unmanaged SharePoint sites, file shares and mail systems Legacy data can contain or be: Redundant Duplicates and unauthorized copies Obsolete No longer in use or out of date Determined through creation, last modified or accessed date and retention policy Trivial File type with no content value Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

9 What is dark data? What lies hidden in your enterprise data the unknown Beyond legacy data Dark data tends to be: Human readable Unstructured Unindexed Unmanaged Inactive Orphaned Dark data resides in: File servers SharePoint servers Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

10 The risks of ignoring legacy and dark data Legacy & dark data sitting outside the information governance strategy exposes the organization to risk: Spiralling costs: DEFENSIBILITY Expanding information footprint and storage costs Litigation and ediscovery costs ( smoking gun or inability to deliver) Security breaches and reputational damage: SECURITY Sensitive information unprotected (personally identifiable information, privacy regulations) Data leakage and misuse Poor business execution and performance: OPERATIONAL EFFICIENCY Incorrect context Decisions based on outdated information Duplicate effort spent re-creating information Inability to exploit information for advancing the business and increasing revenue: REVENUE Inability to correlate Client Relationship Management Data Inability to create a Client 360 degree view Inability to be pro-active as to client needs Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

11 Accuracy and insight Engaging 100% of data to gain competitive advantage Traditional enterprise data Big Data Dark data CRM ERP Data warehouse Web Social Log files Machine data Semi-structured Unstructured Data volumes 11

12 It takes a Big Data platform to cash in your data assets Siloed data challenge Minimal data in the average organization is tagged and analyzed Information silos - everywhere Tools for finding and understanding information, tied to origin and format Queries take too long and are too rigid, difficult to uncover opportunities, patterns or unexpected threats Big Data platform answer Ad hoc discovery find what s in the data without pre-structuring Ubiquitous but secure data access Real time data collection and analysis, any format, any data source An extensible platform to harness100% of your data, on-premise, in the cloud 12

13 HP Haven Big Data platform HP applications Customer applications Developer applications Gain insight from 100% of your data Connect to all of your machine, business, & human data sources Analyze at volume and velocity of data Develop modern data-driven applications Haven Defined programming interfaces Analytics, context and categorization Scalable data stores Data connectors In the Cloud Social media Video Audio Texts Mobile Transactional Documents IT/OT Search data engine Images Records Compliance archives On-premise 13

14 HP IDOL: Understand & Act on 100% of your information Seize your competitive advantage Move faster. Make better decisions. Anticipate change. Unlock opportunities. Streamline processes. Identify breaking trends. Identify fraud. Effectively diagnose illnesses. Impact your business. Pivot. Move faster. Make better decisions. Anticipate change. Unlock opportunities. Streamline processes. Identify breaking trends. Identify fraud. Effectively diagnose illnesses. Impact your business. Pivot. Move faster. Make better decisions. Anticipate change. Unlock opportunities. Streamline processes. Identify breaking trends. Identify fraud. Effectively diagnose illnesses. Impact your business. Pivot. Move faster. Make better decisions. Anticipate change. Unlock opportunities. Move Find 2D/3D clustering, Acoustic signature, Active matching, Agents, Alerting, Auto language detection, Auto query guidance, Boolean & legacy, Operations, Breaking news clustering, Categorization, Collaboration, Community, Concept highlighting, Concept-query, Summarization, Conceptual retrieval, Context summarization, Cross-modal suggest, Dynamic n-dimensional, Taxonomy generation, Dynamic +500 XML, powerful Explicit HP IDOL functions profiling, Face recognition, Field modulation, Frame analysis, Fuzzy matching, Hot clustering, Hyperlinking, Image analysis, Image association, Implicit profiling, Keyword search, Mail object identification, Melody classification, Melody identification, Metadata recognition, Natural language retrieval, Object identification, Object recognition, Ontology generation, Parametric refinement, Phrase spotting, Proper name identification, Query by example, Real-time aggregation, Routing, Scene detection, Script alignment, Sentiment analysis, Sound matching, Speaker, Proper name identification, Fuzzy matching, Phrase spotting, Frame analysis, Enterprise HP Archiving HP Data Protection HP Marketing Optimization and more Enrich Analyze Act HP IDOL Act on 100% of your information External and Cloud News Forums Blogs and more Language independent Language independent Customer Social media Video Audio Mobile Documents Texts communications 14 Transactional data IOT Images Search engine

15 HP Vertica HP Vertica Analytics platform High-performance data analytics platform purpose-built for big data Blazing fast analytics Gain insight into your data in near-real time by running queries 50x-1,000x faster than legacy products Massive scalability Infinitely scale your solution by adding an unlimited number of industry-standard servers Open architecture Protect and embrace your investment in hardware and software with built-in support for Hadoop, R, and a range of ETL and BI tools Optimized data storage Store 10x-30x more data per server than row databases with patented columnar compression Speed, scalability, and openness at lower TCO 15

16 HP Haven - complementary technologies Hadoop IDOL Vertica Big Data scalability? Optimized for? Unstructured Unstructured Structured Interactive Query? Accessible by Business users? Built-in Analytical Functions? Data collection from all repositories incl. web, social? Limited Access to any data in any unstructured document format? Limited 3 rd party libraries file formats Audio, Video, Image file processing? 16

17 The right tool for the right job Inquire Investigate Interact Improve Search your data Analyze your data Personalize your data Enhance your data Automatic hyperlinking Conceptual search Keyword search Fieldtext search Phrase search Phonetic search Field modulation Fuzzy matching Automatic Query Guidance Highlighting Parametric refinement Summarization Real-time predictive query Metadata extraction Automatic tagging Faceted navigation Implicit profiling Explicit profiling Community and expertise network Agents Intent-based ranking Alerting Social feedback Eduction Automatic clustering Clustering 2D/3D Autoclassification Auto language detection Sentiment analysis Automatic taxonomy generation 17

18 HP Software -Big Data Data Portfolio Store Explore Govern Protect Serve Hadoop SQL on Hadoop Archiving Backup & recovery Sr. executives HP s Archive/RM/ ECM repositories Unstructured data repositories Structured data repositories Cloud repositories Offsite/removable Information Analytics Enterprise search ediscovery/eca Collaboration Knowledge management Curated workspace File analysis Structured data archiving Records management Policy management Application retirement Disaster recovery Rights management Legal holds Surveillance/ supervision Platform security Legal & compliance IT Security & Risk Finance Line of business End users 18

19 Wider Information Governance Story Analyze Classify Take action Secure Protect Dispose Know what information you have Unstructured Data Repositories HP ControlPoint HP Records Manager Know where the information is located Know your information is secure Structured Data Repositories HP Structured Data Manager Additional Active Repositories Know the information is managed appropriately 19

20 HP ControlPoint Shining a Light on the Unstructured World HP ControlPoint Measure >>>>>> Manage Examples Classify Data Tag Data Execute Policy Domain-appropriate classification; segment content by value Perpetuate & persist categories across the estate to enrich data Design and implementation of agreed policies. No Value Redundant, Obsolete, Trivial. Business Contract, SOP, Test Procedure, HR. Class IP Sensitive, Export Control, PII. Security Open, Confidential, Secret RM Business Record, Non-Business Record Business Product Name, Employee ID Move Delete Secure To DMS, Archive, Cloud, Tier 2 Storage Now, In 6 Months, After Review & Approval.. Leave, Put on Hold, Enforce Retention If you can t measure it, you can t manage it Peter Drucker : Management Consultant, Educator & Author 20

21 Volume and growth of data Candidates to file, delete, etc. Type of files 21

22 Consolidated Architecture for Structured and Unstructured Content IM Voice Compliance Policy Authority Front Office Compliance Record Files Unstructured Scan/Fax Connector Framework Manage Review Report Understand Meaning Classification Govern Records Management Structured Archive Application Database Archiving 22

23 This is a rolling (up to 3 year) roadmap and is subject to change without notice Use case # 1: Smart / Safe City Improving public safety by detecting high-risk activities and investigating threats Deployment Environment - Ingest data from 2,000+ CCTV cameras in Auckland View network of road and environmental sensors Social media trending, broadcast monitoring, and real time web news Phase 1 scene analysis and license plate recognition Future Phase - Integrate Vertica to uncover breaking trends and facilitate incident responses IDOL eduction sends interesting data to Vertica for statistical analysis and slice/dice Combine Vertica s pattern-matching and graph-analysis at scale with IDOL s ability to model concepts and enrich data 23

24 This is a rolling (up to 3 year) roadmap and is subject to change without notice Use case # 2: Catch Insider Traders Financial Services Information Surveillance & Digital Forensics Solution Multiple data sources: HP Digital Safe data Transactional trading data Financial news feeds Social media , voic recordings, instant messaging Phase 1 complex policies such as highlighting suspect trades where no communication can be found between related Bank A and Bank B contacts Future Phase - Integrate Vertica for trend and anomaly detection IDOL eduction sends interesting data to Vertica for statistical analysis and slice/dice Combine Vertica s pattern-matching and graph-analysis at scale with IDOL s ability to model concepts and enrich data 24

25 This is a rolling (up to 3 year) roadmap and is subject to change without notice Use case # 3: Smart Retail / Voice of Customer Prevent churn, analyze surveys, react to product/warranty issues Multiple data sources: Phase 1: 25 Enterprise documents, , ticketing systems, CRM cases, videos Customer social media, blogs, forums, UGC, surveys Public Websites, News Sentiment detection, clustering Eduction people, places, credit card #s Link expansion, Gender detection Curation, tagging, alerts Future Phase - Integrate Vertica for demographic profiling IDOL eduction sends interesting data to Vertica for statistical analysis and slice/dice Combine Vertica s pattern-matching and graph-analysis at scale with IDOL s ability to model concepts and enrich data

26 Use case #4: Derivatives Records Management The Problem: Dodd Frank Act Compliance (Regulatory Record Keeping) Manage millions of pieces of content for long term retention The Solution: HP Records Management, ControlPoint and Teleform Why We Won: Efficient and cost-effective solution for capturing and retaining millions of content for lengthy retention Strong Professional Services capability and delivery Functional completeness ControlPoint s ability to identify content and HP RM s ability to ingest large volumes of classified content We are facing increased regulatory pressure to produce records in almost real-time. HP provided the best and most complete solution. 26

27 HP is powering the world s leading firms for risk management and regulatory compliance 10 of the top 10 Global Banks 77 of the top 100 Global Law Firms 19 of the top 25 FT Financial Institutions 27

28 Check out the websites

29 Q&A Contact Us David Kemp EMEA Business Development Manager +44 (0) Liam Scahill Ireland HP Solutions Sales Executive Copyright 2012 Autonomy Inc., an HP Company. All rights reserved. Other trademarks are registered trademarks and the properties of their respective owners.