CART/MARS Risk Assessment in Automobile Loans and Leases John Trimble Wells Fargo
Appeal of Data Mining Quality More accurate prediction Better analysis from same skill set Quantity Improved efficiency Same modeling job takes less time Increasing total accomplishment in given period
Topic of This Presentation Two Case Studies Where Efficiency Really Mattered Quick replacement of legacy scorecard (model) Eliminated yearly licensing fee Where Efficiency Frees Time for Higher Quality Pursuit in Accounting for loan prepayments Leading to better credit risk assessment
About Wells Fargo Auto Finance
Auto Loan Originations Automated Approval Car Dealer Legacy Model Wells Fargo Lender #2 Lender #3
Wells Fargo Portfolio Approximately $13 billion in loans and leases (retail and wholesale) 840,000 Customers Approve new loans with credit scoring models to large degree Model assesses risk of approx. 1.8 million individuals per year (800,000 applications)
First Case Study Using MARS to Replace a Legacy Model
Replacing A Legacy Model Eliminate Annual Licensing Fee Legacy Model Performing well Historical inputs and performance available New Model Only legacy input variables available New data platform under construction
Legacy Model Inputs and Model Inputs Performance Credit Bureau Variables in Month t Application Variables in Month t Model Performance Measure Delinquency 60 dpd or worse In 18 subsequent months (i.e. t+1,,t+18)
Limited # Input Variables LEGACY MODEL' S VARI ABLES No. S o u r c e La b e l 1 Application Vehicle Age 2 C. Bu r e a u CBR Te n u r e 3 C. Bureau Del i nquenci es 4 Application % Down Payment 5 C. Bureau Credit Utilization 6 Application Gross Mont hl y I ncome 7 C. Bur eau # I nqui r i es 8 C. Bur eau Maj or Der og Rat i ngs 9 C. Bur eau Revol vi ng Bur den 10 C. Bureau Recent Del i nquenci es 11 C. Bureau # Account s
Variables Selected by MARS MARS TRANS FORMATI ONS OF LEGACY MODEL' S VARI ABLES # Del i nquenci es NO Delinquencies Long CBR Tenur e Shor t CBR Tenur e Hi gh Revol vi ng Bur den Low Revolving Burden Recent Del i nquency NO Recent Del i nquency Hi gh # Recent Inqui ri es Low # Recent Inquiries Do wn Pa y me n t ( %) NO Do wn Pa y me n t Vehi cl e Age
MARS Generated An Excellent Replacement Legacy and New Model Lorenz Curves 100 90 80 70 Legacy New Random % of Bads 60 50 40 30 20 10 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 % of Obs
Benefits and Costs of the Replacement Benefit: Eliminated substantial annual licensing fee Costs: Data preparation (1 week) Model development (1 day) Model implementation (2 weeks)
2nd Case Study Using CART and MARS to Understand Who Prepays Auto Loans
When Loans Prepay Expected Return < Realized Return Approx. 2-3% of loans voluntarily pay off each month, 33% prepay over 18 mos Prepays Impose additional administrative burden Expose the business to reinvestment risk If Can Predict Prepayment Tendencies Can price in prepayment risks Develop other mitigation strategies
Prepayment Behavior Underlying Causes Likely Complex Natural Causes Auto accidents Theft Voluntary Causes Advantageous refinancing opportunities Vehicle replacement Interest rate levels Manufacturer incentives Retention Sometimes is lack of attractive alternatives
Predicting Prepayments Available Attributes Nearly 70 Credit Bureau and Application Attributes Available Examples include: Credit Inquiries Loan Maturity Recent Delinquencies Credit Utilization # Accounts Debt Burden Delinquency History
Search for Salient Attributes Search for Common Attributes of Prepayers Regardless of when prepay CART Trees Developed for Each Month of Performance Window Variable Importance Scores (VIS) Averaged for each attribute Attributes Ranked by Average VIS
VIS Attributes of Prepayers Top 10 by Average VIS 59.57 Loan Maturity Attribute 54.05 Utilization of Credit Union Accounts 51.48 Number of Credit Union Accounts 41.48 Extent of Credit Seeking Activity 34.80 Credit Utilization 34.14 Number of Accounts 33.83 Extent of Credit Union Utilization 33.71 Volume of Recent Credit Seeking Activity 33.15 Average Number of Months Trades Open
Prepays CART Tree 47 Nodes
Prepays CART Tree 1 Obs Per Applicant Gini Splitting Rule Standard CART settings Top 7 Node Splits (47 Nodes Total) Node 1 Loan Maturity (<=37 Mos) (Left Node Terminal) Node 2 # Credit Union Accounts in Good Standing (<= 1) Node 3 Credit Utilization. (<= 21.6%) Node 4 Extent of Credit Seeking Activity (<= 44%) Node 5 # New Accounts. (<= 3) Node 6 Extent of Credit Seeking Activity (<= 51%) Node 7 Credit Utilization. (<= 28.3%)
Salient Revelations from CART Loan maturity seems to be a key attribute Shorter term loans prepay at greater rate But term also enters in more complex ways Access to credit union financing is important, but relationship not simple Active credit union customers prepay at 1.6 times rate as non-credit-union customers Credit utilization and credit worthiness are also important
Logistic Regression Lorenz CART Nodes Learn and Test Sample Lorenz Curves % of Attrites 100 90 80 70 60 50 40 30 20 10 Learn Test Random 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 % of Population
CART Logistic Model Lift is relatively modest Prepayment motivated by many underlying factors Natural causes not measured Model still offers useful predictions Could use in present form to adjust pricing To compensate for prepayment risk Application data not in current study potentially may add to predictive accuracy
Logistic Regression Lorenz Top 5 CART Nodes+ MARS Vars Learn and Test Sample Lorenz Curves % of Attrites 100 90 80 70 60 50 40 30 20 10 Learn Test Random 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 % of Population
CART vs. MARS MARS: CART Dummies Plus Vars Model: CART Dummy variables Let MARS pick other variables MARS model adds modest lift over strictly CART model MARS Logistic Model Provides: Smoothes lift Continuous probability predictions
Logistic Regression Lorenz Top 5 CART Nodes Interacted with MARS Vars Learn and Test Sample Lorenz Curves 100 % of Attrites 90 80 70 60 50 40 30 20 10 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 % of Population Learn Test Random
MARS vs. MARS CART Interacted with MARS Vars Model: CART dummies interacted with variables picked by MARS Each segment modeled separately then combined into single Logistic model Commonly used approach to model building Believed to improve prediction But adds imperceptible lift over previous MARS model with CART dummies and MARS variables Does add new statistically significant variables
What Have We Learned? About Who Prepays
Prepayer Profiles Greatest Propensity to Prepay Preference for shorter maturity (< 42 mos) Shorter the maturity more likely to prepay Access to lower-cost financing Credit Union member (1+ accts open) Home equity line (tax-advantaged rate) Low credit utilization (< 58%) Increases with decreases in credit utilization Longevity with smaller # accounts (>2yrs) Positive balances on only small # such accts More accounts opened recently (>62%)
Prepayer Profiles Least Propensity to Prepay Preference for longer maturity (> 42 mos) Longer the maturity less likely to prepay Lower # accts opened recently (< 62%) Increases with increases in % opened recently High credit utilization (> 58%) Larger # accts with balances (8+) Balances in good standing May be Credit Union member, but access to lowcost financing stymied by debt burden Never used credit union much in past (0-2 times)
What About The Timing of Prepayment?
Linear Regression Lorenz MARS Timing-of of-prepayment Predictors Learn and Test Sample Lorenz 100 90 % of Attrites Each Month 80 70 60 50 40 30 20 10 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 % of Population Learn Test Random
Timing-of of-prepayment Lorenz Lift appears to be quite small But is continuous target variable Target is Month of Prepayment Prediction in each month good on average But variance quite large Small R 2 (1.2%) Still get useful indicators of timing of prepayment
Prepayer Profiles Greatest Propensity to Prepay Early Prefer longer maturity (> 24 mos) Propensity declines with maturity Very low credit utilization (<= 15%) Increases with decreases in credit utilization High % accts recently opened (>=167%) Increases with # such accounts Not many recent accounts used (< 24%) Increases with decreases in % such accounts High # Credit Union accounts (2+) Increases with # of such accounts Good credit rating FICO Score > 780
Prepayer Profiles Least Propensity to Prepay Early Prefer shorter maturity (<= 24 mos) Propensity declines with maturity Carries balances on several accts (1+) Balances in good standing Lower % accts opened recently (< 167%) Propensity to prepay decreases with % of accts recently opened