Predictive Analytics: Seeing the Whole Picture

Transcription

1 Webinar will begin shortly Predictive Analytics: Seeing the Whole Picture Presented by Caserta Concepts, Zementis, FICO June 18, 2015 Copyright 2015 Zementis, Inc. All rights reserved. 2

2 Predictive Analytics: Seeing the Whole Picture Moderator Presenters Mark Rabkin Director Business Development Zementis Copyright 2015 Zementis, Inc. All rights reserved. Joe Caserta Founder / CEO Caserta Concepts Michael Zeller CEO Zementis Shalini Raghavan Senior Director Product Management FICO 4

3 Next Up Copyright 2015 Zementis, Inc. All rights reserved. 5

4 Topic: Predictive Analytics Presented by:

5 About&Caserta&Concepts& Technology&innova6on&company&with&exper6se&in&data&analysis:& Big&Data&Solu6ons& Data&Warehousing& Business&Intelligence& Solve&highly&complex&business&data&challenges& AwardAwinning&solu6ons& Business&Transforma6on& Maximize&Data&Value& Industry&Recognized&Workforce& Core&focus&in&the&following&industries:& ecommerce&/&retail&/&marke6ng& Financial&Services&/&Insurance& Healthcare&/&Ad&Tech&/&Higher&Ed& Services& Strategy,&Roadmap,&Implementa6on& Data&Science&&&Analy6cs& Data$Science$&$Analy.cs$ Data&on&the&Cloud& Data&Interac6on&&&Visualiza6on&

6 Why do we need predictive analytics today?

7 The&Progression&of&Analy6cs& Business Value What happened? Descrip6ve& Analy6cs& Why did it happen? Diagnos6c& Analy6cs& What will happen? Predic6ve& Analy6cs& How can we make It happen? Prescrip6ve& Analy6cs& Reports! Correlations! Predictions! Recommendations Data Analytics Sophistication Source: Gartner

8

9

10 Today s&data&environment& Enrollments& & & Claims& Finance& Others & ETL& Tradi6onal& EDW& ETL& Traditional BI AdAHoc/Canned& Repor6ng& Big Data Analytics Big Data Lake NoSQL& Databases& & Data&Science& Spark$ MapReduce$ Pig/Hive$ AdAHoc&Query& N1& N2& N3& N4& Hadoop Distributed File System (HDFS) N5& Horizontally Scalable Environment - Optimized for Analytics Canned&Repor6ng&

11 The&Big&Data&Pyramid&! &Data&has&different&governance&demands&at&each&6er.&! &Only&top&6er&of&the&pyramid&is&fully&governed.& Purpose& User$adIhoc$queries$and$repor.ng$ Agile$business$insight$through$dataI munging,$machine$learning,$blending$ with$external$data,$development$of$ toibe$bdw$facts$ Big& Data& Warehouse& Data&Science&Workspace& Governance& Fully$Data$Governed$($trusted)& Metadata&!&Catalog$ $$$$ILM&!&who&has&access,&how&long&do&we& manage&it & $$$$$$$$$Data$Quality$and$Monitoring&!&Monitoring&of& &&&&&&&&&&&&&&completeness&of&data& Data$is$ready$to$be$turned$into$ informa.on:$organized,$well$ defined,$complete.$$$$ Raw$machine$data$ collec.on,$collect$ everything$ Data$Lake& &Integra6on&Layer& Landing&Area& &Source&Data&in& Full&Fidelity & Metadata&!&Catalog& $$$ILM&!&who&has&access,&how&long&do&we& &&&&&&&& manage&it & $$$$$$$$$$$$Data$Quality$and$Monitoring&!& &&&&&&&&&&&&&&&&Monitoring&of&completeness&of&data& Metadata&!&Catalog& $$$ILM&!&who&has&access,& &&&&&&&how&long&do&we&&& &&&&&&&&&& manage&it &

12 Notable&Predic6ve&Analy6c&Tools& Open Source Tools: scikit-learn KNIME OpenNN Orange R Weka GNU Octave Apache Mahout Most Popular: SAS SPSS MATLAB R Commercial Tools: Alpine Data Labs BIRT Analytics Angoss KnowledgeSTUDIO IBM SPSS Statistics and IBM SPSS Modeler KXEN Modeler Mathematica MATLAB Minitab Oracle Data Mining (ODM) Pervasive Predixion Software RapidMiner RCASE

13 Are&there&Standards?& CRISP-DM: Cross Industry Standard Process for Data Mining 1. Business Understanding Solve a single business problem 2. Data Understanding Discovery Data Munging Cleansing Requirements 3. Data Preparation ETL 4. Modeling Evaluate various models Iterative experimentation 5. Evaluation Does the model achieve business objectives? 6. Deployment PMML; application integration; data platform; Excel

14 1.&Business&Understanding& In this initial phase of the project we will need to speak to humans. It would be premature to jump in to the data, or begin selection of the appropriate model(s) or algorithm Understand the project objective Review the business requirements The output of this phase will be conversion of business requirements into a preliminary technical design (decision model) and plan. Since this is an iterative process, this phase will be revisited throughout the entire process.

15 2.&Data&Understanding& Data Discovery! understand where the data you need comes from Data Profiling! interrogate the data at the entity level, understand key entities and fields that are relevant to the analysis. Cleansing Requirements! understand data quality, data density, skew, etc Data Munging! collocate, blend and analyze data for early insights! Valuable information can be achieved from simple group-by, aggregate queries, and even more with advanced SQL and Python Significant iteration between Business Understanding and Data Understanding phases.

16 3.&Data&Prepara6on& ETL (Extract Transform Load) 90+% of Data Science time goes into Data Preparation! Select required entities/fields Address Data Quality issues: missing or incomplete values, whitespace, bad data-points Join/Enrich disparate datasets Transform/Aggregate data for intended use: Sample Aggregate Pivot

17 Data&Quality&and&Monitoring& &Build&an&automated&robust&data& quality&subsystem:& &Metadata&and&error&event&facts& &Orchestra6on& &Based&on&Data&Warehouse&ETL& Toolkit& Each&error&instance&of&each&data& quality&check&is&captured& Implemented&as&subAsystem& acer&inges6on& Each&fact&stores&unique& iden6fier&of&the&defec6ve& source&row&

18 4.&Modeling& Requires Algebra & Statistics Expertise Evaluate various models/algorithms Classification Clustering Regression Many others.. Tune parameters Iterative experimentation Different models may require different data preparation techniques (ie. Sparse Vector Format) Additionally we may discover the need for additional data points, or uncover additional data quality issues!

19 What&to&use&When?& #predictiveanalytics

20 5.&Evalua6on& What problem are we trying to solve again? Our final solution needs to be evaluated against original Business Understanding Did we meet our objectives? Did we address all issues?

21 6.&Deployment& Engineering Time! It s time for the work products of data science to graduate from new insights to real applications. Processes must be hardened, repeatable, and generally perform well too! Full Data Governance applied PMML (Predictive Model Markup Langauge): XML based interchange format

22 Closing&Thoughts &&! &Analy6cs&requires&the&convergence& of:&&! &Data&governance&&! &Advanced&data&engineering&&! &Math&and&sta6s6cs&! &Business&smarts&! &Analy6cs&must&be&guided&by&best& prac6ces&and&standards&a&crispadm&! &Tools&and&techniques&must& ul6mately&be&plaform&agnos6c& (portable)&! &Work&with&experts&that&have&done&it& before!&

23 Agile&DW&&&ETL&Training&in&NYC,&2015& Two unique workshops taught by international data warehousing veterans. Workshops: Sept (2 days), Agile Data Warehousing with Lawrence Corr Sept (2 days), ETL Architecture and Design with Joe Caserta SAVE $300 BY REGISTERING BEFORE JUNE 30TH! Thanks! We look forward to seeing you there.

24 Thank&You& Joe Caserta President, Caserta Concepts (914) Next&up.&&Michael&Zeller,&CEO,&Zemen6s&

25 Zementis: Company Snapshot Zementis provides predictive analytics solutions for big data Software for Operational Predictive Analytics Company: Global customer base Deployments across multiple industries Partnerships with software and analytics industry leaders Industry analyst coverage Offices in USA, Asia Products: Vendor-neutral architecture for - Data mining tools - Analytics and data warehouse platforms Multi-platform support: Hadoop, MPP and SQL PMML industry standard centric Flexible deployment cloud or on-site Capabilities: Separates model development from deployment Supports wide range of predictive modeling techniques Rapidly deploys and executes predictive models Scores data in batch and real-time modes Accelerates business insight Copyright 2014 Zementis, Inc. All rights reserved. Confidential 6

26 Big Data - Big Confusion? Hadoop Ecosystem Hive Spark Storm Batch Streaming Real-time Cloud On-premise Copyright 2015 Zementis, Inc. All rights reserved.

27 Path to Business Value Predictive analytics helps organizations unlock the value of their big data Big Data Predictive Analytics Business Insights Decisions & Actions Business Value Applications Databases Cloud Log Files RSS Feeds Other Sources Predictive Models Machine Learning Techniques Data Mining Tools More relevant More accurate More comprehensive More nuanced Faster Lower risk Greater positive impact Accelerated time-tomarket More precise targeting Real-time responsiveness Enhanced operational agility Competitive advantage Higher revenue growth rates Greater profitability Copyright 2015 Zementis, Inc. All rights reserved. 8

28 Traditional Deployment Cycle but model deployment challenges can often erode much of the value that predictive analytics can deliver Develop Operationalize Utilize Business Decisions Data Scientist IT Engineer Business Professional Predictive model deployment becomes a rework cycle Extensive manual coding Cross-checking Fixing coding errors Delayed insight Less accurate decisions Missed opportunities Loss of value Copyright 2015 Zementis, Inc. All rights reserved. 9

29 Deployment with Zementis & PMML Enter Zementis, whose solutions accelerate time-to-insight for predictive analytics Economic Value Time-to-insight Within 2 days * ~ 6 months Accelerated deployment timeline Reduced model deployment cycle time Reduced model deployment expense Increased model throughput Enhanced accuracy Minimal rework, if any Model Deployment Cycle Time Without Zementis With Zementis * And sometimes even within a few hours! Rapid insight = Rapid time-to-value from predictive analytics Copyright 2015 Zementis, Inc. All rights reserved. 10

30 What is PMML? Predictive Model Markup Language (PMML) industry standard reduces the complexity of operationalizing models Mature standard developed by the DMG (Data Mining Group) to avoid proprietary issues and incompatibilities and to deploy models XML-based language used to define statistical and data mining models and to share these between compliant applications Supported by most leading data mining tools, commercial and open-source Data handling and transformations (pre-and post-processing) are a core component of the PMML standard Allows for the clear separation of tasks: Model development vs. model deployment Eliminates the need for custom code and proprietary model deployment solutions Copyright 2015 Zementis, Inc. All rights reserved. 11

31 Predictive Analytics Workflow PMML in action, covering a complete workflow from raw data input to decision output PMML File Raw Inputs Model Signature Input Validation Data Pre- Processing Predictive Model Data Post- Processing Prediction Data and operational types Outliers, Missing Values, Invalid Values Normalize, Discretize, Bin, Map, etc. Derived Model Inputs Model Outputs Scaling, Business Decisions, Thresholds, etc. Copyright 2014 Zementis, Inc. All rights reserved. Confidential 12

32 Execute Anywhere Vendor-neutral architecture enables compatibility with industry-leading analytics and data warehouse platforms Data Mining Tools Compatible with most leading commercial and open-source data mining tools Tools include: - IBM SPSS - KNIME - Python - R - SAS Predictive Modeling Techniques Supports multiple broad categories of predictive modeling techniques Examples: - Clustering Models - Decision Trees - Neural Network Models - Naïve Bayes Classifiers - Random Forest Models Copyright 2015 Zementis, Inc. All rights reserved. ADAPA UPPI Analytics and Data Warehouse Platforms Cloud On-site In-database Hadoop Amazon Web Services FICO Analytic Cloud Microsoft Azure Cloud SAG Apama IBM WebSphere RedHat JBoss / Tomcat Oracle WebLogic SAP HANA IBM PureData / Netezza Pivotal / EMC Greenplum SAP Sybase IQ Teradata + Teradata Aster Hive / Storm / Spark Datameer Cloudera Hortonworks IBM 13

33 Broad Applicability ADAPA and UPPI accelerate predictive model insights for multiple industries and business use cases Fraud & Risk Scoring Internet of Things (IoT) Marketing & Sales Financial institutions Scoring bureaus Online transaction processing Advanced decision management Predictive maintenance Quality control Sensor & device data processing Manufacturing Healthcare Up- /cross-sell and nextbest-offer Marketing campaign optimization Real-time recommendations Copyright 2015 Zementis, Inc. All rights reserved. 14

34 Next Up Copyright 2015 Zementis, Inc. All rights reserved. 15

35 Data Driven Decisions Going from Raw and Disparate to Decision-Ready Shalini Raghavan Senior Director, FICO 2015 Fair Isaac Corporation. Confidential. This presentation is provided for the recipient only and cannot be reproduced or shared without Fair Isaac Corporation s express consent.

36 FICO is a leading predictive analytic software company that helps businesses grow by making data-driven decisions Organizations in 90+ countries turn to FICO to: Improve customer engagement Automate decisions for greater speed and scalability Optimize every decision for higher ROI 2015 Fair Isaac Corporation. Confidential. 2

37 FICO Analytics: Fast, Smart and Everywhere 9,000 per second card transactions analyzed for fraud milliseconds to perform 15,000 fraud risk calculations 2.5 billion payment cards protected from fraud 10 billion+ FICO Scores purchased per year 4,000 predictive models for one retail client 65% credit cards worldwide managed 2015 Fair Isaac Corporation. Confidential. 3

38 Analytic Vendor Landscape Summarize Past and Current Behavior Predict Future Behavior and Adapt Tools Solutions Scores Tools Solutions Decision Value Business Intelligence Descriptive Analytics Predictive Analytics Decision Optimization Understand the trends in the business 2015 Fair Isaac Corporation. Confidential. 4 Make different offers to groups of customers Analytic Capability Target each decision to a customer s future behavior Automatically take the optimal action on each individual

39 FICO Drives Benefits Across Industries Banking Insurance Retail/ Consumer Goods Health Care/ Pharma Government More than Half of the Top 100 Banks Globally Two Thirds of the Top US P&C Insurers One Third of the Top US Retailers 8 of the Top 10 Pharma Companies 100+ Government Agencies Fair Isaac Corporation. Confidential. 5

40 Challenges These are the challenges we are addressing Fair Isaac Corporation. Confidential. 6

41 Current Solutions A Patchwork Quilt of Technologies Batch ETL Reporting Streaming Business Process Automation Big Data Automation CEP Dataflow Automation Social Data Connectors Enterprise Data Connectors 2015 Fair Isaac Corporation. Confidential. 7

42 Activating Data for Competitive Advantage Raw Decision-Ready Product Usage Offer Propensity Payment Rate Transaction Frequency Customer Attrition Collection 2015 Fair Isaac Corporation. Confidential. 8

43 Customer Imperatives Leverage all data Big, Small, Fast, Slow, Different Integrate and analyze data without lengthy programming cycles Continuously turn data into decision ready information Integrate with existing architectures and data stores Hide complexity of technology Lower your Total Cost of Ownership 2015 Fair Isaac Corporation. Confidential. 9

44 FICO Data Management Integration Platform Data Sources Decision Consumers Scheduled Batch (Pull) Data DMIP Capabilities 3 rd Party Applications Continuous Stream (Push)! Un-Structured Semi-Structured Structured Identity Resolution Event Processing Normalization Unified Data Acquisition Visualization Analytic Processing Validation Distributed Processing Decision Ready Data Analytic & Decisioning Solutions 2015 Fair Isaac Corporation. Confidential. 10

45 FICO Data Management Integration Platform Data Sources Decision Consumers Distributed Processing Scheduled Batch (Pull) Job Job Step 1 n Data 3 rd Party Applications Continuous Stream (Push) Topic Analytic & Decisioning Solutions! Distributed Messaging 2015 Fair Isaac Corporation. Confidential. 11

46 Real-Time Visualization Live Visualizations Reduced latency for operational use cases Immediate insight Fully configurable by business user Fair Isaac Corporation. Confidential. 12

47 Processing Online Activity with Augmentation BATCH POLL REAL-TIME Simply change a single inbound job step to switch from batch to real-time processing. No need to write code Product Data Customer Data ftp http In Stream In CSV Unpack CSV Norm Field Valid NVP Norm Regex Norm RDBMS Fetch RDBMS Fetch Predict Topic Out Jobs can be composed and re-used Job steps are configurable through a web UI without need to write code / scripts or use developer tooling Autonomous job steps enable faster processing All jobs once built are spun up for execution without intervention from a very skilled development user Fair Isaac Corporation. Confidential. 13

48 Sense and Respond Marketing GOAL Customer-focused real-time marketing in response to customer actions CONSIDERATIONS Many channels for customer interaction, including 3rd party sources Reduce long latencies from initial customer engagement by driving personalized, relevant actions in real-time. Improve customer growth and retention 2015 Fair Isaac Corporation. Confidential. 14

49 Sense and Respond Marketing Requirements Requires access to real-time customer data Access to other relevant sources of data, e.g. customer data Requires infrastructure to hold and manage the identity resolution data Benefits Increases the responsiveness of marketing actions Integrates information to provide a more comprehensive view of the customer Gain early insight to customer behavior through real-time visualization 2015 Fair Isaac Corporation. Confidential. 15

50 Operational Intelligence GOAL Reduce impact of outages, risk, losses and inefficiency CONSIDERATIONS Many devices and infrastructure components with vulnerabilities and potential for failures Current fault and vulnerability detection is contained in siloes Costs are escalating due to faults and vulnerabilities! 2015 Fair Isaac Corporation. Confidential. 16

51 Operational Intelligence Requirements Requires access to real-time devices/components in the infrastructure Requires infrastructure to hold and manage the linkage networks Benefits Accelerates the detection of faults and vulnerabilities Extracts information from unstructured data to provide deeper meaning Integrates events to catch correlations for more comprehensive coverage Gain early insight to anomalies through real-time visualization 2015 Fair Isaac Corporation. Confidential. 17

52 Thank You Shalini Raghavan Senior Director, FICO 2015 Fair Isaac Corporation. Confidential. This presentation is provided for the recipient only and cannot be reproduced or shared without Fair Isaac Corporation s express consent.

53 Thank You! Questions? Copyright 2015 Zementis, Inc. All rights reserved. 16