A new IT era for a third generation platform demand. Pivotal Field Engineering and Customer Success

Transcription

1

2 A new IT era for a third generation platform demand Pivotal Field Engineering and Customer Success

3 Every Business is Becoming a Digital Business Software is Eating the World Data Is Fueling Software $18B valuation Transportation $20B valuation Entertainment $26B valuation Automotive 3

4 In the beginning shipped on physical media hard to change after release runs on other peoples computers have to worry about bugs process doesn t run very long no worries about uptime The shift to services the internet changes everything runs on your computers can change your computers still have to worry about bugs process runs a very long uptime is everything 4

5 Getting to mobile/multidevice Enterprise mobile apps must be consumer-grade Simple, clean, intuitive UI High performance Built with modern technology stack In 2015, 79% of surveyed organizations plan to increase spending on mobile development by 36%, and many will explore new technology and sourcing options 5

6 90% Of All Enterprise Apps Will Be Mobile By % of enterprise apps are mobile % of enterprise apps will be desktop and mobile Source: Gartner, "The Transformation of Mobile Middleware", Van Baker, April 14,

7 Moving to the Third Platform 7

8 IT underpins the new digital Era Smart Mobile Devices Pervasive Telemetry In Products/Things Huge Data Sets Massive On-Demand, Affordable Compute Capability New Levels Of Agility And Affordability 8

9 The Data Driven Enterprise Journey STORE ANALYZE DEVELOP INNOVATE Structured Predictive Analytics Advanced Analytic Pipelines Agile Dev Expertise Unstructured Machine Learning Realtime Analytical Applications DevOps High Volume High Velocity Advance Data Science Realtime Analytics Global Scale Data-Driven Applications Enterprise, Consumer, IoT, and Mobile Hybrid Cloud Continuous Delivery Closed Loop Applications BIG DATA PREDICTIVE ANALYTICS ENTERPRISE PAAS AGILE DEVELOPMENT 9

10 Open Source is the New Standard Address the core needs across community Differentiated features/value-add OSS also enables customer-specific backlogs/features 10

11 JOURNEY TO A DATA-DRIVEN ENTERPRISE Perform advanced analytics Discover insights Deploy analytic apps and automate at scale Modernize data infrastructure 11

12 Move from Inflexible Infrastructure to Agile Data Architecture Based on scaledup, proprietary systems Siloed and scattered data sets Elastic, Scale-out storage and processing Handle any data type or source Storage limited with rigid data models Cloud friendly and open-source based Copyright 2015 Pivotal. All rights reserved. 12

13 Move from Historical BI to Data Science Long running queries Massive parallel processing of queries Stale dashboards Fresh data Summarized data sets Machine learning and advanced analytics tools Copyright 2015 Pivotal. All rights reserved. 13

14 Move from Scale-Limited Apps to Analytic-Powered Apps at Scale Service failure or data loss at scale Long innovation cycles Resilient, scale-out messaging and processing Agile development with CF-based data services Poor experience at scale Low-latency, inmemory computing Copyright 2015 Pivotal. All rights reserved. 14

15 LARGE ENTERPRISE BIG DATA TROUBLE But 80% of CEOs thinking data mining and analysis are strategically important (1) 0% of CIOs think their IT infrastructure is fully prepared for big data (3) 44% of new applications failed to meet performance expectations (5) 4% of companies use analytics effectively (2) 30% of companies have deployed advanced analytics, 11% big data analysis (4) 2X 90% of companies allocate at least 2X more cloud capacity than needed to ensure performance (6) (1) 2015 PWC CEO Survey; (2)2013 Baine and Company - The Value of Big Data; (3) 2014 IT Infrastructure Conversation - IBM; (4) Ernest and Young Enterprise IT Trends and Investments; (5) 2014 Riverbed Tecnologies - The Transformers; (6) 2014 ElasticHosts CIO Study 15

16 Issues with EDWs and How to Address Them Greenplum DB EDW Issues Best-in-Class ADW High Cost Custom built and priced for analytics Missing drill-down detail Scale out storage Not geared for big data Cost-based query optimization Rigid data models=high ETL Costs Flexible structured data models Vendor locked-in systems of record Offload analytics to ADW Limited deployment Appliance/Commodity/IaaS deployments Limited advanced analytics support Integrated advanced analytics support 16

17 Added value of data analytics: maturity model Value of Analytics ($) What happened? Descriptive Analytics Why did it happen? Diagnostic Analytics What will happen? Predictive Analytics How can we make it happen? Prescriptive Analytics Complexity 17

18 Apps/ Users Front end layer for analytics Pivotal approach: move the analysis closer to the data Preparing data area, to support analysis with performance landing temp area Collection area huge spaces Operational reports Defining a unique point of view, not in silos Star schemas, data warehousing processes Denormalization, data mining preparation Hadoop Data Exploration Sandbox Data Ingestion: Batch, stream, message queues, web services DATA SCIENTISTS Source Data Structured Traditional scale up relational databases Apps/ packaged applications Semi-structured Location data, xml, etc Sensors, Internet of things, web logs Social Data Unstructured Images, video, etc. Customer Facing new Agile Apps 18

19 How Data Science and Data Engineering can help you? Pivotal Data Scientists are technical professionals with strong programming skills, anchored in vertical/ horizontal domains or in specialized academic research, able to identify real-world problems requiring predictive analytics, formulate these mathematically, and solve them by applying machine learning and statistical algorithms, on Big Data, in Pivotal and third-party technologies. Data Science + Pivotal Data Engineers are Big Data experts and industry veterans with a passion to leverage these skills to drive business value for Pivotal customers. They possess expert knowledge and skills with the Pivotal data products and excel at architecting enterprise scale solutions to the most demanding data and analytic challenges. Data Engineering Copyright 2015 Pivotal. All rights reserved. 19

20 Technical Observations SQL is today and will remain the most valuable workload on Hadoop While Hadoop continues to mature, focused MPP SQL will remain important Scale out in-memory processing will have significant enterprise adoption and impact into the future Streaming and Machine Learning will continue to gain value Open Source is becoming critical to enterprise investment decisions 20

21 MPP Shared Nothing Architecture Flexible framework for processing large datasets Master Host and Standby Master Host Master coordinates work with Segment Hosts Master Host SQL Standby Master Greenplum DB Segment Host with one or more Segment Instances Interconnect Segment Instances process queries in parallel Segment Hosts have their own CPU, disk and memory (shared nothing) node1 Segment Host Segment Instance Segment Instance Segment Instance Segment Instance node2 Segment Host Segment Segment Instance Host Segment Segment Instance Instance Segment Segment Instance Instance Segment Segment Instance Instance Segment Instance node3 Segment Host Segment Instance Segment Instance Segment Instance Segment Instance noden Segment Host Segment Instance Segment Instance Segment Instance Segment Instance High speed interconnect for continuous pipelining of data processing 21

22 Functions Oct 2014 Predictive Modeling Library Generalized Linear Models Linear Regression Logistic Regression Multinomial Logistic Regression Cox Proportional Hazards Regression Elastic Net Regularization Robust Variance (Huber-White), Clustered Variance, Marginal Effects Matrix Factorization Singular Value Decomposition (SVD) Low Rank Linear Systems Sparse and Dense Solvers Linear Algebra Other Machine Learning Algorithms Principal Component Analysis (PCA) Association Rules (Apriori) Topic Modeling (Parallel LDA) Decision Trees Random Forest Support Vector Machines Conditional Random Field (CRF) Clustering (K-means) Cross Validation Naïve Bayes Support Vector Machines (SVM) Time Series ARIMA Descriptive Statistics Sketch-Based Estimators CountMin (Cormode-Muth.) FM (Flajolet-Martin) MFV (Most Frequent Values) Correlation Summary Inferential Statistics Hypothesis Tests Support Modules Array Operations Sparse Vectors Random Sampling Probability Functions Data Preparation PMML Export Conjugate Gradient 22

23 Performing a linear regression on 10 million rows in seconds Why you want to move your analysis close to your data Hellerstein, Joseph M., et al. "The MADlib analytics library: or MAD skills, the SQL. 23

24 Hadoop Ecosystem Challenges Pivotal HD Current Challenges Mature Hadoop Ecosystem Fragmented Hadoop distributions Common ODP core-based distributions Recertifying Hadoop-based tools Minimize recertification using ODP Transportability of Hadoop-based apps Apps transportable over distributions Proprietary Hadoop file systems Common HDFS file system Proprietary management interfaces Hadoop-native management interfaces Tracking to fast moving project versions Integrated set of production-quality versions 24

25 Business Data Lakes: A Data Exploration Sandbox New architecture for data driven enterprise Match technology need to data need Allow enterprise to store everything Accelerate time to value from data Store Everything Obsessively collect data Keep it forever Put the data in one place Analyze Anything Cleanse, organize, and manage your data lake Make the right tools available Use the resources wisely to compute, analyze, and understand data Build what you need Use insights to iteratively improve your product 25

26 SQL on Hadoop Offerings: Explore your data HAWQ Typical SQL on Hadoop Mature SQL on Hadoop Complex joins not supported Complex joins at performance Limited advanced analytics support Advanced analytics at scale within SQL Interactive query latency issues Fast interactive queries on large data Ad-hoc query performance issues Strong ad-hoc query support in optimizer SQL analytic query coverage issues Full analytic SQL compliance Concurrent query throughput issues High query throughput for mixed workloads 26

27 MPP on Hadoop HAWQ Feature Benefit Rich and compliant SQL dialect Powerful and portable SQL apps Leverage large SQL-based ecosystems TPC-DS compliance Enable a wide range of use cases Avoid surprises in production Flexible/efficient joins at linear scale Deep analytics + machine learning Data federation capabilities Off-load EDW workloads at a much lower cost Predictive/advanced learning use cases at scale Build use cases with diverse/external data assets without data movement High availability and fault tolerance Native Hadoop file format support Off-load business critical workloads from EDW Reduce ETL and data movement = lower costs 27

28 Traditional Data Stores Can t Handle Cloud Scale Low-Latency Apps GemFire Traditional RDBMS Distributed In-Memory DB s, NoSql, IMDG Disk based, high latency Memory based, low latency Limited scale up and out via expensive hardware and planning Elastically scale up or down on demand without down time Designed for monolithic applications Built for cloud scale applications Support complex SQL for transactions AND analytics Built for general business applications and reporting requirements Built to to optimize application-specific data access patterns Built to ensure low latency data access at any scale Configurable data consistency 28

29 NoSql In memory Data Grid High Level Architecture Java Client C# Client C++ Client Clients can embed cache with disk overflow Locators Locators provide both discovery and load balancing services. Machines can be added dynamically to expand capacity Data partitioning and replication is handled transparently to clients. Redundant storage assures continuous availability (memory or disk) M.. M1 M2 M3 Updates are sent to subscribers as objects change Disk-Stores for data persistence and backup Data Data Data Synchronous read through, write through, or asynchronous write behind to other data sources 29

30 Putting it All Together DATA FEEDS TRANSACTIONAL APPS ANALYTIC APPS Data Stream Pipeline Distributed Computing Real-Time Data Expert Systems & Machine Learning Advanced Analytics Data Lake HDFS 30

31 IoT Pipeline React Apps Edge Ingest Process (Tx, RTA) Analyze (Big ML) Store Platform 31

32 Agile Data: Rapid Development of Industry Use Cases Retail CRM Customer Scoring Store Siting and Layout Fraud Detection / Prevention Supply Chain Optimization Advertising & Public Relations Demand Signaling Ad Targeting Sentiment Analysis Customer Acquisition Financial Services Algorithmic Trading Risk Analysis Fraud Detection Portfolio Analysis Media & Telecommunications Network Optimization Customer Scoring Churn Prevention Fraud Prevention Manufacturing Product Research Engineering Analytics Process & Quality Analysis Distribution Optimization Energy Smart Grid Exploration Government Market Governance Counter-Terrorism Econometrics Health Informatics Healthcare & Life Sciences Pharmaco-Genomics Bio-Informatics Pharmaceutical Research Clinical Outcomes Research 32

33 Pivotal offers different data analysis choices From traditional scenarios to big data innovation Hawq making analytics on top of Hadoop Data Mining Data exploration using statistical models Olap performances OLAP queries on distributed HAWQ tables Data Exploration Explorative OLAP queries directly on offloaded data 33

34 Italian success story: activities & goals Project Management Reporting migration Dramatic improvement of data load Data movement SQL conversion Data Quality General improvement of front-end performance Very few post delivery issues No business disruption during the change 34

35 Italian success story: ETL improvements ETL chains time comparison (min) Summary of % gain Staging 91% Level 1 70% Level 2 82% Weekly jobs 88% Monthly jobs 89% 35

36 Pivotal for Automotive Innovative ingestion scenarios to big data analysis Connected cars Front end representation of data Querying and associating hot data and geography Data Ingestion Sensor data from IoT Message queuing, IMDG Mantaining operational data 36

37 Telco and IoT 37

38 Pivotal and social data analysis Innovative ingestion scenarios to big data analysis Social networks, Spring XD, Hadoop Front end representation of data Querying and associating hot data and geography Adding info to flowing data Data Ingestion Creating flows from Twitter Managing flows with Spring XD 38

39 The Power of PaaS (On Premise & Off Premise) Business Value, Agility & Cost Savings Traditional IT IaaS PaaS You Manage Applications Data Runtime Middleware O/S Virtualization Servers Storage You Manage Applications Data Runtime Middleware O/S Virtualization Servers Storage You Manage IaaS Applications Data Runtime Middleware O/S Virtualization Servers Storage Pivotal CF + Choice of IaaS Networking Networking Networking Copyright 2013 Pivotal. All rights reserved. 39

40 CLOUD FOUNDRY - THE HARDWARE ANALOGY... Traffic Management & Load Balancing... Mobile Services App Server Message Bus Etc... Etc... Etc... $B s Data Services Auto-Scaling & Availability Linux Pivotal CF Dev Ops: App & Container Management Physical Servers IBM, HP, Dell,... EHC Modern Infrastructure... 40

41 Software is Eating the World: You need PaaS Ø Developer Agility: End-to-end platform where cloud services enable them to build and update applications easily Ø Operational Agility: Built-in operational benefits to de-risk new projects: HA, logging, APM, auditability, ID integration, etc.. Ø Choice of Infrastructure: Optimize efficiency, cost, geographic distribution, capacity planning and regulatory compliance 41

42 Continuous Delivery/Deployment Microservices, not anymore monolithic apps 42

43 What Agile Means What Agile Means Agile software development is a way to build software in the face of changing requirements. It drives the evolution of a product through an iterative development cycle based on ongoing end-user feedback. Traditional/Waterfall (every 6-12 months) Requirements Design Iterative (daily/weekly) Prioritize Code Feedback Automated Testing Rather than a big-bang approach, releasing every few months, Agile proposes that software is built and deployed much more frequently - perhaps even every few hours - in order to get immediate feedback Implementation Verification Maintenance Release Evolve Design as a product takes shape. 43

44 Powering Digital Transformation Pivotal enables enterprises to provide modern software-driven experiences for their customers and workforces. 44

45 Q&A 45

46