Predictive Analytics & Predictive Modeling December 2 3, Catherine Snyder Supervisor US Dealer Audit, Audit Services General Motors Company

Transcription

1 Predictive Analytics & Predictive Modeling December 2 3, 2014 Catherine Snyder Supervisor US Dealer Audit, Audit Services General Motors Company

2 AGENDA Overview of General Motors Company Predictive Analytics Definition and References Dealer Risk Management & Optimization (DRMO) Model Key Takeaways DRMO Global Program Objectives Operating Environment Process Overview Risk Indicators and What Could Go Wrongs VIN / Warranty Claim Selection Process DRMO Model Development Keys to Success 2

3 THE NEW GENERAL MOTORS Top Global Automaker Sales of 9.7 million vehicles #1 in U.S.; #2 in China Fortune 7 company Sales in 120 Countries Production in 30 Countries 396 facilities touching 6 continents 219,000 Employees 21,000 Dealerships 3

4 GM TOP GLOBAL AUTOMAKER Sales in 120 Countries Production Locations in 30 Countries Percentage of deliveries by region 4

5 GM BRANDS Baojun Wuling Jiefang

6 6

7 7

8 Predictive Analytics 8

9 DEFINITION AND REFERENCES Business Analytics Defined Thomas H. Davenport, professor at Babson College Defines descriptive, predictive, and prescriptive analytics and when to use each. How Managers Should Use Data

10 PREDICTIVE ANALYTICS Encompasses various statistical techniques from modeling, machine learning, and data mining. Analyze current and historical facts, capture relationships between explanatory and predicted variables from past occurrences, to make predictions about the future, unknown, events, trends and behavior patterns. Accuracy and usability of results depend greatly on the rigor of data analysis and the quality of assumptions. 10

11 PREDICTIVE MODELS Capture relationships among many factors to enable assessment of risk or potential associated with a particular set of conditions, guiding decision for candidate transactions. Model the relationship between specific performance of a unit in a sample and one or more known attributes of the unit. Objective is to assess the likelihood that a similar unit in a different sample will exhibit the specific performance. Examples: Real-time Credit Card Transaction Predictive Model Credit Scoring Model Customer Relationship Management (CRM) Model Clinical Decision Support Systems Customer Retention/Silent Attrition Model Fraud Risk Assessment Model 11

12 PREDICTIVE ANALYTICAL TOOLS Alpine Data Labs BIRT Analytics Angoss KnowledgeSTUDIO IBM SPSS Statistics and IBM SPSS Modeler KXEN Modeler Mathematica MATLAB Minitab Oracle Data Mining (ODM) Pervasive Predixion Software Revolution Analytics SAP SAS and SAS Enterprise Miner STATA STATISTICA TIBCO FICO 12

13 13

14 14

15 Dealer Risk Management & Optimization (DRMO) Model 15

16 KEY TAKEAWAYS Use audit results to drive Investment in prevention activities Sell Win-Win Model examines 100% of the population, removes individual bias, and provides consistent results Combine datasets to improve risk assessment Integrate automated controls when you can No model is perfect Depending on the audit objectives: Predictive analytic is not always the answer Traditional audit approaches could be your best friends You may only need very few samples Be alert to opportunities 16

17 DRMO GLOBAL PROGRAM OBJECTIVES Implement global, common process to improve audit objectivity, effectiveness and efficiency Engage global subject matter experts Perform advanced analytics on 100 % of the claims to aggregate dollars-at-risk per dealer Allocate resources to highest risk transactions/dealers Refresh model based on audit results and changes Expedite identification of global issues and root cause analyses to facilitate global corrective actions and process improvement Consolidate risk profiles and audit results to provide insights for business and operational improvements Improve visibility of risk on a timely basis Provide interactive reporting and analytics Develop scorecard for monitoring effectiveness of risk management /optimization program 17

18 OPERATING ENVIRONMENT IN UNITED STATES 4,300 dealers 2.8 million vehicles delivered in 2013 Incentive/Warranty programs by brand/model Statutes of Limitation vary by states: 6, 9, 12, 18 and 24 months Fraud provisions and impact on audit rights vary by states 18

19 COMMON INCENTIVE PROGRAM PARAMETERS Who When Where What How 19

20 PROCESS OVERVIEW What Could Go Wrongs (WCGWs) Risk Indicators Model Evaluation Aggregate by Dealer Claim Data Claim- Level Risk Score Weighted At-Risk Amount (WARA) % of High Risk Claims Audit Plan Pull List Actual Audit Results 20

21 RISK INDICATORS / WHAT COULD GO WRONG What Could Go Wrong (WCGW) Identify common deviation reasons for claims Risk Indicators (RI) Evaluate claim attributes used to identify one or more of the WCGWs Perform calculations on each claim Compare against historical exceptions to identify the strength of correlation of each indicator, both individually and in various combinations (not all indicators or combinations of indicators are equal in their predictive power) Use in the scoring formula for a given claim if the indicators show strong historical correlation to exceptions Example Risk Indicator Description What Could Go Wrongs RI-IC-21 RI-WTY-5 Claim Date for a Program Across all Models: Box plot which shows high-level anomalies in claim dates across an entire incentive program. Abnormal claim volume: Identification of dealers claiming an abnormal percentage of warranty claims related to a particular part when compared to the full population of dealers. WCGW-IC-2 WCGW-IC-3 WCGW-IC-5 WCGW-IC-7 WCGW-IC-11 WCGW-WTY-1 WCGW-WTY-2 An incentive claim may be submitted and paid for an expired program or a program that has not yet begun. An incentive claim may be submitted and paid for an inapplicable model. An incentive claim may be submitted for an applicable program date where actual delivery was outside of the program's eligible period. An incentive code may be added to a claim submission after the original submission date in order to take advantage of a program or pocket an incentive and not pass it on to the customer. Vehicles may be reported as sold, but a sale has not actually occurred. Warranty claims are processed for repairs that did not occur. Warranty claims are processed for non-covered parts. WCGW-WTY-14 Warranty claims are processed for non-covered services. 21

22 EXAMPLES: WHAT COULD GO WRONGS Sales Incentive 40 WCGWs Incentives were claimed on deliveries that did not qualify Incentives were claimed for customers that did not qualify Vehicles not sold may be reported as sold Incentives may be claimed for deliveries outside the eligible period Dealer may misreport sales to manipulate vehicle allocation Warranty 30 WCGWs Multiple claims were submitted for the same repair using different problem codes or operation codes Warranty claims were processed for non-covered services Warranty claims were processed for repairs that did not occur Warranty claims were processed for vehicles in dealer possession for work that is not required Dealers submitted more hours for a service than actually incurred 22

23 EXAMPLES: INCENTIVE RISK INDICATORS % of Incentive Claims to Market Share Comparison by Geographic Region Analyze claims monthly to determine which dealers claim more incentives than their predicted relative geographic market share of deliveries Dealer Penetration % Compared to National Average Penetration % Monthly comparison of total delivery % per incentive code at a dealership to the national average % on a brand-by-brand basis High/Low outliers are flagged Abnormal Customer Names Customer names similar to the dealer name Deliveries made to the same customer Abnormal Amount of Incentives Claimed on Vehicle Deliveries Vehicles with an abnormal amount of incentives claimed when compared to average number of incentives claimed/vehicle for specific incentive category Volume of Deliveries with Claims at Month-End Claims for deliveries where the dealer s delivery volume in the last part of the month appeared abnormal compared to other dealers 23

24 EXAMPLES: WARRANTY RISK INDICATORS Frequency of problem code use Identification of dealers using specific problem codes at a higher percentage of total claims than the overall dealer population. Flags claims related to those dealer/problem code combinations as high risk Abnormal claim volume Identification of dealers claiming an abnormal percentage of warranty claims related to a particular part when compared to the full population of dealers Abnormal labor hours by operation and problem code Box plot which shows the total hours (regular labor and other labor hours) for each operation and problem code combination to identify claims that are outliers when compared to the population Labor only claims abnormal for problem/operation code combination Identification of specific labor-only claims where it is abnormal for that particular claim type to be labor-only based on the total population of claims with a specific problem/operation code combination Warranty claims for vehicles in dealer possession Identification of claims for vehicles without a warranty start date and where vehicle mileage is less than

25 VIN/WARRANTY CLAIM SELECTION PROCESS New claims loaded into the database Risk indicator values calculated for all claims Risk score calculated for each claim based on the risk indicator values and incentive category or warranty type scoring function (model) Different incentive categories / warranty types may have unique risk score functions (models) Average risk score for each VIN / Warranty Claim calculated VINs / Warranty Claims with higher risk scores predict a higher exception rate based on historical patterns and behaviors VINs / Warranty Claims with highest risk scores flagged as selected as these pose the highest likelihood of exception Model ranks dealers for audit based on having the highest aggregate Weighted At-Risk Amount (WARA) for highest risk VINs (Incentives) or having the highest % of high risk warranty claims (Warranty) NOTE: WARA = claim risk score (probability of failure) * claim amount 25

26 AUDIT PLANNING PROCESS Model ranks dealers for audit based on number of high risk Sales Incentive/Warranty claims 26

27 AUDIT PULL LISTS The Claim Risk Score and Top 3 Primary Risk Indicators are provided in the Audit Pull List 27

28 28

29 KEYS TO SUCCESS Understand the program objectives, rules, risks and processes Leverage diversified knowledge, experience and skills on development and feedback Apply agile model development approach to respond to changes and new information Make the model user-friendly Use visualization to facilitate communication Analyze, learn, discuss, improve, and validate 29

30 30