AML Mdel Validatin Beynd the Guidance By: Salvatre Cangialsi February, 2014 Intrductin The Office f the Cmptrller f the Currency and the Federal Reserve have bth issued guidance n Mdel Risk Management. The supervisry guidance applies bradly t all quantitative mdels used thrughut a banking rganizatin in the peratin f their business. Such mdels are widely used within AML cmpliance grups. They are primarily implemented thrugh the AML sftware slutins; but are als implemented via spreadsheets and ther tls used by the cmpliance rganizatin. A key aspect f Mdel Risk Management is a rbust validatin prcess. The validatin prcess emplyed by AML cmpliance grups has received increasing attentin frm regulatrs. Sme banks have been unprepared fr the level f regulatry review and rigr expected ver Mdel Validatins. This article will prvide real wrld insights and guidance n the current issues related t the validatin f mdels used fr AML cmpliance. Overview f the Guidance as Applied t AML SR Letter 11-7 f April 4, 2011 (http://www.federalreserve.gv/bankinfreg/srletters/sr1107a1.pdf) prvides revised and cmprehensive guidance n Mdel Risk Management. The guidance addresses amng ther tpics Mdel Validatin. As applied t AML, the guidance is meant t assure that: The prper mdels are chsen, The mdels perate crrectly, and The implementatin and use f the mdels are apprpriate fr the risk f the bank. The primary mdels used by mst banks include: Custmer nbarding and retentin, Custmer and AML risk rating, Suspicius activity detectin scenaris, Scring, and OFAC/Sanctins vilatin detectin. These mdels must be independently validated by all banks. The validatin must be independent f the develpers and users f the mdels. The independent validatin may be perfrmed by an internal grup such as audit. It may als be perfrmed by third party cnsultants having adequate expertise. Hwever, the bank remains respnsible fr verseeing the results f the wrk perfrmed by third parties.
A key element f the guidance is hw the risk, business activity, and the cmplexity f the mdels shuld be cnsidered in the manner in which a validatin is perfrmed. Recent Trends In general, AML cmpliance examinatins are increasing in scpe and cmplexity. Findings are becming mre difficult t address while the risk f a regulatry enfrcement actin has increased. It has als been rumred that there will be a higher prbability f criminal prsecutins related t egregius AML deficiencies. Given this backdrp, which is nt new t the Chief Cmpliance Officer (CCO), it is prudent t track examinatin trends and adapt peratins as apprpriate t the bank. The fcus f this sectin is n the recent trends applicable t Mdel Validatins which are as fllws: Increased Examinatin Fcus. Nearly all f the banks we have spken with have fund that a review f the Mdel Validatin prcess was part f their verall AML examinatins. This has been nging and appears t als nw apply t a wider range f small financial institutins. Additinally, the depth f review f the Mdel Validatin prcess has increased. Evlving Requirements. The manner in which the Mdel Validatin prcess is reviewed seems t vary substantially acrss individual regulatrs. T sme extent, this is related t the risk and activities f the bank. Hwever, much f the variance des nt appear t be explained by risk alne. One can assume that, given the newness f this increased fcus, best practices acrss regulatrs have nt fully matured. Cnsequently, it may be difficult fr a bank t anticipate the level f review and their expected perfrmance in this area. Expected Rigr and Quantitative Prcess. What has been a cnsistent trend in recent examinatins is the expected rigr and demand fr quantitative supprt fr judgments reached in a Mdel Validatin. This is nt surprising given that a quantitative apprach is an essential aspect f verall Mdel Risk Management. Nevertheless, the mathematic, statistical, ecnmic, and analytic skills needed fr the expected level f rigr are nt always available r anticipated by a bank prir t an examinatin. The CRAD. The Cmpliance Risk Analysis Divisin f the OCC is cmprised f highly skilled prfessinals in the area f statistical analysis and ecnmetric mdels. The grup is primarily staffed with Ph. D.s in ecnmics and statistics. They prvide brad supprt fr the OCC's supervisry and regulatry initiatives. A number f the banks we have spken with have had the CRAD play a rle in their AML examinatins. The grup has reviewed and prvided substantive challenges t the suspicius activity mdels used by thse banks. With respect t thse challenges, the bank is expected t justify, in a quantitative manner, detectins rules chsen and the threshlds applied t rules. A Questin f Cst vs Scpe. The OCC has prvided guidance n the use f cnsultants as part f an enfrcement actin. See http://www.cc.treas.gv/news-issuances/bulletins/2013/bulletin- 2013-33.html. Althugh this des nt directly apply t Mdel Validatins, it clearly highlights the OCC's interest ver a bank's due diligence ver third party cnsultants and the cntracts entered int with them. We call attentin t this as a bank pinted ut the OCC's cncern with the cst f
an assessment prject. In essence the OCC felt that the cst prvided was t lw t cver the full scpe f wrk that wuld be needed t perfrm a prper Mdel Validatin. Althugh there was misunderstanding by the OCC as t what the actual prject's gals were, it suggests that banks shuld take care in the definitin f the scpe f wrk and the due diligence applied twards third party cnsulting cntracts. Cnfusin with System Assessments. Many banks have perfrmed AML system assessments. Again, this wrk is ften carried ut by independent cnsultants. It is cmmn that the definitin f a system assessment varies frm ne cnsultant t anther. Generally, the assessment will cver a review f rules that partially r fully meets the criteria fr a Mdel Validatin. With increased regulatry fcus n Mdel Validatins, the clarity f the definitin applied t a system assessment must be assured. Challenges Encuntered by Banks With an increasing fcus n a larger range f banks, Mdel Validatins need t be cnsidered a pririty by all CCOs. In develping a prcess fr sund validatins, ne must understand the challenges t that utcme. These challenges can be bradly categrized as: Validatin Apprach. Althugh the OCC guidelines apply t all Mdel Validatins, its implementatin can vary greatly frm bank t bank. The variability is based n several factrs including AML specific requirements, risk prfile, and the views f the regulatr. Withut the benefit f several refinements t the prcess, a bank may face strng criticism that can lead t regulatry actins. System Limitatins. Mdel Validatins are inherently system fcused. Mst AML systems are acquired frm a sftware vendr. The vendr has a prprietary interest in maintaining their intellectual prperty rights and will ften prvide mdels as a "black bx". Withut adequate disclsure frm the sftware vendr, the crrectness f mdel design and implementatin cannt be directly assessed. Validatin is limited t a testing and analysis methdlgy which is less cmprehensive. Required Skill Set. This is the mst difficult challenge fr many banks. A prper Mdel Validatin requires a range f skill sets that may nt be available r are nt easily accessible t the bank. Essentially a team apprach is needed fr the validatin. The team shuld be cmpsed f members with the fllwing skills: AML cmpliance dmain expertise AML system expertise Ecnmetric, statistical, and mathematical Data analytics Audit Needed Tls. Accessing and analyzing data, perfrming tests, and interpreting results are greatly enhanced with the availability f autmated tls. These tls include: AML typlgy mdels
Data analytics platfrms Data analytics platfrms such as Tableau, Sptfire, and many thers can be acquired frm cmmercial vendrs. Hwever, they require a learning curve r existing in-huse expertise. AML typlgy tls are mre prblematic. They are highly cmplex, are nt cmmercially available, and very few practitiners in the AML Mdel Validatin space have the expertise t develp these systems. Reliance n Cnsultants. With the range f skills required fr a validatin, banks may emply an independent cnsulting firm t perfrm part r mst f the required wrk. Where the bank is lacking familiarity f the verall requirements fr an AML Mdel Validatin, the selectin prcess can be difficult. Further adding t the difficulty is the large number f small cnsulting firms that will engage cntractrs fr the wrk perfrmed. The use f cntractrs by the cnsulting firm can lead t incnsistent prcesses and a lack f cntinuity ver subsequent engagements. Cmmunicating with Regulatrs. There is much latitude in the implementatin f a Mdel Validatin prcess. When the bank takes an apprach that is nt fully understd by the regulatr r where the regulatr suggest an alternate apprach, it is required that the bank adequately explain all aspects f the validatin apprach. The explanatin may need t be made t the primary regulatr as well as t specialized teams such as the CRAD. Each f these grups may require a different level f detail and explanatin. Having the apprpriate staff t prvide these explanatins is ften a challenge. Budgeting. Senir management must fully understand the imprtance, scpe f wrk, and time cmmitment needed fr a successful Mdel Validatin. Whether the wrk is dne with in-huse staff r by third party cnsultants, adequate resurces must be made available. Given this need, the CCO alng with audit must develp a justificatin apprach that will be successful. Recmmendatins Perfrming a Mdel Validatin fr a financial institutin is clearly nt a simple undertaking. The range f skill sets needed and the challenges inherent in the prcess call fr a well-rganized apprach. In this sectin, we prvide a number f categrized recmmendatins that shuld be cnsidered. Gvernance 1. It is highly recmmended that the CCO assure the invlvement f senir management. 2. Final reprts shuld be delivered t the bard f directrs r similar versight structure. 3. The firm r internal grup perfrming the Mdel Validatin must be clearly independent f the creatin and use f the mdels. Regulatry Cnsideratins 1. The bank's regulatrs shuld be cnsulted peridically cncerning plans fr the Mdel Validatin.
2. The CCO shuld develp a netwrk f peers and cnsultants that he can reach ut t peridically t discuss current regulatry expectatins and findings. 3. Cmprehensive dcumentatin shuld be develped t supprt the Mdel Validatin. This dcumentatin at a minimum shuld include: a. The statement f wrk r ther dcument describing the scpe f the engagement and any specific limitatins. It is essential that the scpe f wrk be clearly detailed. It is als imprtant that the scpe f wrk indicate that it will be perfrmed in accrdance with OCC guidelines fr Mdel Validatins. b. Bis f the peple participating in the validatin. c. Detailed and cmprehensive prject plan. d. Written reprt with separate versin cntrl dcumenting the reasn that changes were made. e. Wrk papers. Qualificatins 1. The qualificatins f the firm and the specific team assigned t a Mdel Validatin must be reviewed. It is recmmended that the engagement shuld be perfrmed by a team. The team shuld be made up f individuals with the fllwing skills: Planning a. Subject matter expertise in AML and Sanctins Cmpliance with strng experience guiding Mdel Validatins. b. Quantitative analysis. This individual shuld have demnstrable skills evidenced by an apprpriate mathematics r statistics degree and a prven recrd f experience. c. Technical business analyst. A persn able t access and evaluate data frm multiple system surces and with a gd understanding f the business needs f the AML Cmpliance Grup. d. Strng prject management experience. It is nt necessary that these skills be represented by separate peple. Fr example, in a smaller engagement the subject matter expert may als prvide verall prject management. Further, it is cmmn that the quantitative analyst will have the database and ther technical skills t wrk with the varius systems. 1. Prir t starting an engagement, it is essential that the cntract, statement f wrk, r internal prject descriptin clearly describe that a Mdel Validatin is t be perfrmed in accrdance with
OCC and Federal Reserve guidelines. Fr a number f reasns, a bank may decide that the prject will nt fully cnfrm t thse guidelines. Perhaps sme f the wrk will be dne by bank staff and ther wrk by a third party cnsulting firm. It is essential that all wrk that will be perfrmed, as well as wrk that is excluded, be clearly described at the pint f apprving the prject and dcumented in the final reprt t the CCO. 2. Dcumentatin shuld als identify verall respnsibility fr the prject and specific respnsibilities fr key phases, particularly when wrk is dne by third party firms. 3. When parts f an verall Mdel Validatin are dne by separate grups, the verall prject plan shuld dcument hw the separate reprts will be reviewed and relied upn t issue an pinin. 4. The prject plan itself shuld be prepared prir t the cmmencement f wrk. It shuld detail all majr tasks, respnsibilities, and timelines fr cmpletin. Peridic review f prgress against the plan shuld be undertaken and dcumented. 5. The validatin shuld be guided by a cmprehensive framewrk. The framewrk is ne that shuld be applicable in general t any independent assessment and als cntain the detailed requirements fr a Mdel Validatin. In particular the framewrk shuld address: Review a. Key stakehlders and their invlvement b. Prject planning and reprting c. Data cnfidentiality and security prtcls d. Required tls and ther supprts e. Infrmatin gathering prcess f. Methds f analysis g. List f all dcumentatin and artifacts needed h. Test perid, plans, cases, and results i. Reprt preparatin, review, and apprval j. Wrk paper management k. Methdlgy fr fllwing up n findings 1. Develp r update the catalg f all mdels used by the AML cmpliance grup. The catalg shuld cntain: a. Name f the mdel b. Purpse c. Descriptin f its peratin d. Data requirements e. Cntrl parameters f. Expected results g. Scheduling h. Date validated prir t use
i. Date implemented j. Date f last validatin k. Other pertinent ntes 2. Gather and evaluate the change cntrl prcess used fr mdel develpment and implementatin. 3. Obtain the AML Risk Assessment, business requirements dcument and ther infrmatin that describes the ratinale fr the mdels used by the AML grup. 4. Dcument and perfrm an assessment f verall gvernance related t Mdel Risk Management, plicies, and cntrls. 5. Befre analyzing mdels, it is essential that the mdel develpment prcess be reviewed t assure that it meets the OCC guidelines. Fr mdels prvided by third parties, the bank must receive sufficient infrmatin t judge cnfrmance. Sme f the infrmatin that shuld be reviewed includes: a. BIOs f key individuals invlved with mdel develpment b. Dcumentatin f all mdels describing their purpse, expected results, and cntrl parameters r ther mechanisms that influence prcessing results. 6. Obtain and review all prir Mdel Validatin and User Acceptance Test reprts. 7. Fr each mdel in the catalg, review and assess the sundness f its implementatin. Testing 1. All tests perfrmed f mdels shuld be cnducted with a data repsitry develped specifically fr the purpse f Mdel Validatin. The data repsitry shuld be cnstructed t supprt all tests and the range f utcmes expected. 2. A statistically valid methdlgy fr data sampling must be determined and dcumented. Nte that data sampling may be needed fr varius test gals and therefre the sampling methdlgy shuld be cnsistent with the data, vlumes, and test gals. 3. Perfrm a cmprehensive data quality review. The data quality review shuld address: a. Accuracy f data elements used by the mdels. Fr example, des the database cntain the latest values fr each transactin? Or, might data be added t the database prir t a mdificatin in the riginating system. b. Cmpleteness f the data. Are all relevant fields representing a transactin present? Often the details frm the riginating message are separate frm the recrding f the banking transactin. In this case are the tw surces cmbined apprpriately? Als cnsider that data elements may be ptinal in the riginating system. If s, is this addressed in the AML
system? Cnsider further, data that may nt exist in the riginating message. Fr example, jurisdictinal data such as cuntries which are essential t mnitring may be mitted. c. Cnsistency. Are the same data values represented in a cnsistent manner? An example f this issue is in the use f incnsistent abbreviatins. Anther frm f incnsistency wuld be the rder in which names are recrded. 4. Tests shuld be cnducted that verify that all mdels perfrm in accrdance with their intended functinality. This can be accmplished in several ways and are generally termed back testing. One methd is t use a tl that simulates hw each mdel shuld perfrm. The simulatin is then applied t histrical data t assure that the same results that were riginally btained against histrical data are again received. Any variatin shuld be explained as acceptable r nt. 5. In additin t back testing, a series f "Abve the Line" (ATL) and "Belw the Line " (BTL) tests shuld be perfrmed. These tests are designed t evaluate each mdel's behavir against changes in their threshlds r ther cntrl mechanisms. A key utcme f the tests is an assessment f the threshld changes n false psitives and false negatives. One way f visualizing the impact f the changes is with a Dispsitin Curve. This graph will shw the increasing r decreasing rate f prductive alerts fr the changes made t threshlds. Analysis The analysis prcess shuld be dcumented and cmprehensive. The specific analytical methds will vary based n the factrs discussed abve. But ften they shuld include: 1. Review and explanatin f differential alerts frm Abve and Belw the Line Testing 2. Assessment f the change in the number if prductive alerts as threshlds are mdified. 3. Assessment f prductivity levels against AML risk assessment and acceptable risk 4. Dispsitin curves that graphically present the relatinship between parameter changes and prductive alerts 5. Assessment f false psitives and false negatives 6. Analysis f time spent n varius alert types 7. Review f quality cntrls