Plastic Card Fraud Detection using Peer Group analysis

Transcription

1 Plastic Card Fraud Detection using Peer Group analysis David Weston, Niall Adams, David Hand, Christopher Whitrow, Piotr Juszczak 29 August, /08/07 1 / 54

2 EPSRC Think Crime Peer Group - Peer Group Work Crime Prevention & Detection Funding 12 projects Also feasibilty studies and more Think Crime Project Develop Fraud Detection Tools Real Data 29/08/07 2 / 54

3 ThinkCrime Team Peer Group - Peer Group Work Members of the team are David Hand Niall Adams Christopher Whitrow Piotr Juszczak David Weston Gordon Blunt Collaborating banks Abbey National, Alliance and Leicester, Capital One, Lloyds TSB 29/08/07 3 / 54

4 Overview Peer Group - Peer Group Work Peer Group Applied to Time-Aligned Multivariate Continuous Data Applied to Credit Card Transaction Data Work 29/08/07 4 / 54

5 Peer Group - Approaches to Fraud Detection Anomaly Detection Peer Group Anomaly Detection to Peer Groups I Anomaly Detection to Peer Groups II Anomaly Detection to Peer Groups III Peer Group - Peer Group Work 29/08/07 5 / 54

6 Approaches to Fraud Detection Broadly 2 approaches to statistical fraud detection Supervised or Anomaly Detection Peer Group - Approaches to Fraud Detection Anomaly Detection Peer Group Anomaly Detection to Peer Groups I Anomaly Detection to Peer Groups II Anomaly Detection to Peer Groups III Peer Group Work 29/08/07 6 / 54

7 Approaches to Fraud Detection Peer Group - Approaches to Fraud Detection Anomaly Detection Peer Group Anomaly Detection to Peer Groups I Anomaly Detection to Peer Groups II Anomaly Detection to Peer Groups III Broadly 2 approaches to statistical fraud detection Supervised or Anomaly Detection Supervised Historical Instances of Fraud Less likely to falsely flag a transaction as fraudulent Approach Chris is taking Peer Group Work 29/08/07 6 / 54

8 Anomaly Detection Peer Group - Approaches to Fraud Detection Anomaly Detection Peer Group Anomaly Detection to Peer Groups I Anomaly Detection to Peer Groups II Anomaly Detection to Peer Groups III Does not use historical Instances of Fraud Build a profile of usual behaviour Significant deviations considered frauds More likely to falsely flag a transaction as fraudulent Potential to adapt to changing fraud patterns Approach Piotr is taking Peer Group Work 29/08/07 7 / 54

9 Peer Group Peer Group - Approaches to Fraud Detection Anomaly Detection Peer Group Anomaly Detection to Peer Groups I Anomaly Detection to Peer Groups II Anomaly Detection to Peer Groups III Similar to anomaly detection methods Do not need to build a model of usual behaviour for account holder Determine a peer group Find other accounts that you expect will behave similarly to the account holder Find accounts that have behaved similarly in the past Monitor account holder s behaviour with respect to peer group Anomalous behaviour, should account holder deviate strongly from peer group Peer Group Work 29/08/07 8 / 54

10 Anomaly Detection to Peer Groups I Peer Group - Approaches to Fraud Detection Anomaly Detection Peer Group Anomaly Detection to Peer Groups I Anomaly Detection to Peer Groups II Anomaly Detection to Peer Groups III The weekly amount spent on a credit card for a particular account Week 1 to Week n y 1,...,y n 1,y n Target Account Wish to determine if the amount spent in week n is anomalous Anomaly Detection based on account profile y 1 y 2 y n 1 y n Peer Group Work 29/08/07 9 / 54

11 Anomaly Detection to Peer Groups II Peer Group - Approaches to Fraud Detection Anomaly Detection Peer Group Anomaly Detection to Peer Groups I Anomaly Detection to Peer Groups II Anomaly Detection to Peer Groups III Population Normalised Anomaly Detection x m,1 x m,2 x m,n 1 x m,n. x 2,1 x 2,2 x 2,n 1 x 2,n x 1,1 x 1,2 x 1,n 1 x 1,n y 1 y 2 y n 1 y n Peer Group Work 29/08/07 10 / 54

12 Anomaly Detection to Peer Groups III Peer Group - Approaches to Fraud Detection Anomaly Detection Peer Group Anomaly Detection to Peer Groups I Anomaly Detection to Peer Groups II Anomaly Detection to Peer Groups III Peer Group Sort accounts in order of decreasing similarity, π(i) x π(m),1 x π(m),2 x π(m),n 1 x π(m),n. x π(k),1 x π(k),2 x π(k),n 1 x π(k),n. x π(2),1 x π(2),2 x π(2),n 1 x π(2),n x π(1),1 x π(1),2 x π(1),n 1 x π(1),n y 1 y 2 y n 1 y n Peer Group size k. Work 29/08/07 11 / 54

13 Peer Groups Example /08/07 12 / 54

18 Peer Group - Peer Group Detecting Anomalies Detecting Anomalies Robustifying Peer Groups Robustifying Peer Groups Peer Group Peer Group Quality Whitening the Population Work 29/08/07 17 / 54

19 Detecting Anomalies Peer Group - Peer Group Detecting Anomalies Detecting Anomalies Robustifying Peer Groups Robustifying Peer Groups Peer Group Quality Whitening the Population Assuming we already have a peer group set of accounts for our target account. y n is multivariate (column vector) and continuous Mahalanobis distance of the target from the mean of its peer group µ is mean of x π(1),n,...,x π(k),n C is covariance matrix of x π(1),n,...,x π(k),n Mahalanobis distance of a target from its peer group (y n µ) T C 1 (y n µ) Work 29/08/07 18 / 54

20 Detecting Anomalies If the distance is above an externally selected threshold, then we flag the target as fraudulent. Peer Group - Peer Group Detecting Anomalies Detecting Anomalies Robustifying Peer Groups Robustifying Peer Groups Peer Group Quality Whitening the Population Peer Group Target Work /08/07 19 / 54

21 Robustifying Peer Groups Peer Group - Peer Group contaminated by fraudulent transactions Outlier Masking Outlier Swamping Peer Group Detecting Anomalies Detecting Anomalies Robustifying Peer Groups Robustifying Peer Groups Peer Group Quality Whitening the Population Peer Group Target Work /08/07 20 / 54

22 Robustifying Peer Groups Peer Group - Robustify the covariance matrix for the Mahalanobis Distance evaluation Use Heuristic Peer Group Detecting Anomalies Detecting Anomalies Robustifying Peer Groups Robustifying Peer Groups Peer Group Quality Whitening the Population Work 29/08/07 21 / 54

23 Robustifying Peer Groups Peer Group - Peer Group Detecting Anomalies Detecting Anomalies Robustifying Peer Groups Robustifying Peer Groups Peer Group Quality Whitening the Population Robustify the covariance matrix for the Mahalanobis Distance evaluation Use Heuristic An account that has deviated strongly from its peer group at time t should not contribute to any peer group at time t Work 29/08/07 21 / 54

24 Robustifying Peer Groups Peer Group - Peer Group Detecting Anomalies Detecting Anomalies Robustifying Peer Groups Robustifying Peer Groups Peer Group Quality Whitening the Population Robustify the covariance matrix for the Mahalanobis Distance evaluation Use Heuristic An account that has deviated strongly from its peer group at time t should not contribute to any peer group at time t For each peer group select 75% closest to their own peer groups Work 29/08/07 21 / 54

25 Peer Group Quality Peer Group - Peer Group Detecting Anomalies Detecting Anomalies Robustifying Peer Groups Robustifying Peer Groups Peer Group Quality Whitening the Population Work It is not necessarily the case that peer group analysis can be successfully deployed on all accounts. q t = 1 k k (y t x π(i),t ) T (y t x π(i),t ) (1) i=1 where T is the transpose. This is a simple measure of how close the members of the peer group are to the target. A good quality peer group is one that closely follows the target over time. Q s,e = 1 t e t s t e t=t s q t. (2) 29/08/07 22 / 54

26 Whitening the Population Peer Group - Peer Group Detecting Anomalies Detecting Anomalies Robustifying Peer Groups Robustifying Peer Groups Peer Group Quality Whitening the Population Whitening the population to make the scatter of a peer group (of size 2) commensurate across time The smaller the value of Q s,e the better the peer group tracks the target over time. Peer Group Members Population Target t=1 t=2 t=3 Work 29/08/07 23 / 54

27 Building Peer Groups Peer Group - Peer Group Detecting Anomalies Detecting Anomalies Robustifying Peer Groups Robustifying Peer Groups Peer Group Quality Whitening the Population Possible to know apriori the peer group membership Employee fraud detection, people with the same job description can be naturally grouped together. IBM FAMS. Health care fraud. Geography, speciality Infer peer group membership from the time series itself Measuring similarity of time series Work 29/08/07 24 / 54

28 Peer Group - Peer Group Time Alignment & Feature Extraction Time Alignment & Feature Extraction Outlier Detection from Peer Groups Active and Inactive Accounts Work 29/08/07 25 / 54

29 Time Alignment & Feature Extraction Peer Group - Peer Group Time Alignment & Feature Extraction Time Alignment & Feature Extraction Outlier Detection from Peer Groups Active and Inactive Accounts Accounts transactions are asynchronous data streams Synchronise account time series by extracting features from the data streams at regular time intervals M(s, e, A) summarise transactions of account A occurring from day s to day e inclusive Mean amount spent Number of transactions Entropy of Merchant Category Groups 16 Groups +1 for ATMs Returns 1 point in 3 dimensional space Work 29/08/07 26 / 54

30 Time Alignment & Feature Extraction 100 Account A Amount Withdrawn Amount Withdrawn Day M(7,10,A) Account B Day M(7,10,B) 29/08/07 27 / 54

31 Outlier Detection from Peer Groups Peer Group - Peer Group Time Alignment & Feature Extraction Time Alignment & Feature Extraction Outlier Detection from Peer Groups Active and Inactive Accounts Once a day at midnight Summary statistic for day t, behaviour of the past d days M(t d + 1,t,A) Smaller d, the more sensitive to new transactions Mahalanobis distance in 3 dimensional space Work 29/08/07 28 / 54

32 Active and Inactive Accounts Peer Group - Peer Group Time Alignment & Feature Extraction Time Alignment & Feature Extraction Outlier Detection from Peer Groups Active and Inactive Accounts Account inactive on day t if it has not performed any transactions on that day Do not test for outlierness for inactive accounts Unusually long periods of inactivity will not be considered fraudulent Work 29/08/07 29 / 54

33 Active and Inactive Accounts Peer Group - Peer Group Time Alignment & Feature Extraction Time Alignment & Feature Extraction Outlier Detection from Peer Groups Active and Inactive Accounts Account inactive on day t if it has not performed any transactions on that day Do not test for outlierness for inactive accounts Unusually long periods of inactivity will not be considered fraudulent Account not active over entire summary statistic window Active peer group members. Closest k accounts that are active on at least one day of the summary statistic window Work 29/08/07 29 / 54

34 Building Peer Groups Peer Group - Peer Group Time Alignment & Feature Extraction Time Alignment & Feature Extraction Outlier Detection from Peer Groups Active and Inactive Accounts Subdivide training data into n non-overlapping windows M(1, L n,a),...,m((n 1)L n + 1,L,A) Point in 3n dimensional space Complication, potential for bias Standardise each window by whitening Work 29/08/07 30 / 54

35 Building Peer Groups 100 Account A Amount Withdrawn M(1,3 1 M(6 2 3,A) M(3 1 3,6 2 3,A) 3,10,A) 100 Account B Amount Withdrawn M(1,3 1 3,B) M(3 1 3,6 2 3,B) M(6 2 3,10,B) 29/08/07 31 / 54

36 Building Peer Groups Peer Group - Peer Group Time Alignment & Feature Extraction Time Alignment & Feature Extraction Outlier Detection from Peer Groups Active and Inactive Accounts Find k nearest neighbours Large number of accounts Accounts that have high volume of transactions unlikely to be tracked by accounts with low volume First sort by number of transactions in training data Work 29/08/07 32 / 54

37 Peer Group - Peer Group Performance Criteria Performance Metric Performance Curve Average Performance Curve Work 29/08/07 33 / 54

38 Performance Criteria Peer Group - Peer Group Performance Criteria Performance Metric Performance Curve Average Performance Curve Reduce total amount lost to fraud Reduce number of fraudulent transactions Reduce the time between fraud starting and fraud detection Reduce the number of account holders affected by flagging legitimate transactions as fraud Number of possible performance metrics Work 29/08/07 34 / 54

39 Performance Metric Peer Group - Peer Group Performance Criteria Performance Metric Performance Curve Average Performance Curve Work If an account has been flagged as containing fraudulent transactions. The card issuer would need to investigate this account. minimise the amount of fraud given the number of investigations the card company can make Performance Curve x-axis number of fraudulent accounts missed as a proportion of the number of fraudulent accounts y-axis number of fraud flags raised as a proportion of the number of accounts Different to ROC curve. The smaller the area under the curve the better the performance. Random classification is represented by a diagonal line from the top left to the bottom right. 29/08/07 35 / 54

40 Performance Curve 1 Number of Fraud Flags Raised per Day as a Proportion of the Population Proportion of Frauds not found The lower the curve the better the performance. Twice Area under Curve [0,1], smaller the area the better the performance 29/08/07 36 / 54

41 Average Performance Curve Produce one curve for each day Take the average of the curves. For a given proportion of fraud flags raised 1 Number of Fraud Flags Raised per Day as a Proportion of the Population Proportion of Frauds not found 29/08/07 37 / 54

42 Peer Group - Peer Group Experiments Varying Length of Summary Statistic Window Varying Length of Summary Statistic Window Varying Length of Summary Statistic Window Varying Length of Summary Statistic Window Varying Length of Summary Statistic Window Global Outlier Detector Peer Groups 29/08/07 38 / 54 Performance

43 Experiments Peer Group - Peer Group Data 4 months of data Accounts with > 80 transactions and fraud free for first 3 months. About 4000 accounts 6% defrauded in final month Performed Peer Group once a day for the remaining month Experiments Varying Length of Parameters Summary Statistic Window Varying Length of Peer Group building 8 segments Summary Statistic Window Summary Statistic window size 7 days Varying Length of Summary Statistic Active Peer Group Size 100 Window Varying Length of Robustifying Peer Groups not used Summary Statistic Window Varying Length of Summary Statistic Window Global Outlier Detector Peer Groups 29/08/07 39 / 54 Performance

44 Varying Length of Summary Statistic Window day Number of Fraud Flags Raised per Day as a Proportion of the Population Proportion of Frauds not Found 29/08/07 40 / 54

45 Varying Length of Summary Statistic Window day 3 days Number of Fraud Flags Raised per Day as a Proportion of the Population Proportion of Frauds not Found 29/08/07 41 / 54

46 Varying Length of Summary Statistic Window Number of Fraud Flags Raised per Day as a Proportion of the Population day 3 days 5 days Proportion of Frauds not Found 29/08/07 42 / 54

47 Varying Length of Summary Statistic Window Number of Fraud Flags Raised per Day as a Proportion of the Population day 3 days 5 days 7 days Proportion of Frauds not Found 29/08/07 43 / 54

48 Varying Length of Summary Statistic Window Number of Fraud Flags Raised per Day as a Proportion of the Population day 3 days 5 days 7 days 14 days Proportion of Frauds not Found 29/08/07 44 / 54

49 Global Outlier Detector Peer Group - Peer Group Is peer group analysis doing nothing more than finding outliers to the population? Special case, use largest possible peer group All accounts apart from target account Subtract Performance Curve for Peer Group from Global. Values less than zero imply Peer Group method is performing better. Experiments Varying Length of Summary Statistic Window Varying Length of Summary Statistic Window Varying Length of Summary Statistic Window Varying Length of Summary Statistic Window Varying Length of Summary Statistic Window Global Outlier Detector Peer Groups 29/08/07 45 / 54 Performance

50 Peer Groups Performance Non Robust Number of Fraud Flags Raised per Day as a Proportion of the Population Proportion of Frauds Not Found 29/08/07 46 / 54

51 Peer Groups Performance Number of Fraud Flags Raised per Day as a Proportion of the Population Non Robust Non Robust without Fraud Contamination Proportion of Frauds Not Found 29/08/07 47 / 54

52 Peer Groups Performance Number of Fraud Flags Raised per Day as a Proportion of the Population Non Robust Non Robust without Fraud Contamination Robust Proportion of Frauds Not Found 29/08/07 48 / 54

53 Peer Groups Performance Number of Fraud Flags Raised per Day as a Proportion of the Population Non Robust Non Robust without Fraud Contamination Robust Global Proportion of Frauds Not Found 29/08/07 49 / 54

54 Peer Groups Versus Global Outlier Detector Performance of the peer group analysis compared with global population outlier detector. Performance Difference Robustified Peer Group Peer Group Number of Fraud Flags Raised per Day as a Proportion of the Population 29/08/07 50 / 54

55 Peer Groups Versus Global Outlier Detector Performance of the robustified peer group analysis compared with global population outlier detector on screened data Performance Difference Number of Fraud Flags Raised per Day as a Proportion of the Population 29/08/07 51 / 54

56 Peer Group - Peer Group Work Conclusions 1 Day Symposium, 23rd November 2007 Work 29/08/07 52 / 54

57 Conclusions Peer Group - Peer Group We have demonstrated there exist credit card transaction accounts that evolve sufficiently closely to enable fraudulent behaviour to be detected. Finding frauds that are not global outliers to the population. Current work Combining Methods Work Conclusions 1 Day Symposium, 23rd November /08/07 53 / 54

58 1 Day Symposium, 23rd November 2007 Statistical and machine learning approaches to detecting fraud and predicting consumer behaviour Competing Risks in Retail Finance, Crowder MJ Event History for Debt Collection Portfolios, Zhou F, Hand DJ, Heard NA A dynamic scorecard for monitoring baseline performance with application to tracking a mortgage portfolio, Whittaker J, Whitehead C, Somers M Estimating the iceberg: how much fraud is there in the UK? Blunt G, Hand DJ Evaluating Fraud Detection Systems, Hand DJ Transaction Aggregation: A Winning Strategy vs. Fraud? Whitrow C, Weston D, Juszczak P, Hand DJ, Adams N Detecting Plastic Card Fraud using Peer Group, Weston D, Whitrow C, Juszczak P, Hand DJ, Adams N Behavioural finance as a multi-instance learning problem,juszczak P, Hand DJ 29/08/07 54 / 54