Comply and Compete: Model Management Best Practices

Transcription

1 Comply and Compete: Model Management Best Practices Improve model validation and tracking practices to prepare for a regulatory audit and boost decision performance Number 55 Regulators worldwide have become increasingly concerned about the soundness of decision making and capital adequacy within financial services. As a result, they are taking an even closer look at how financial institutions manage risk and use scoring models. Regulations demand rigorous documentation across the model lifecycle, with particular vigilance in monitoring ongoing model performance and use in production. While the Fed/OCC and Basel regulators have recently issued guidelines on what they expect, they are often just that guidelines. The onus falls on financial institutions to develop a rigorous model management framework to satisfy audit requirements. While increased scrutiny from regulators carries an added burden, financial institutions can take the opportunity to make improvements that translate into increased analytic performance and risk reduction. This paper highlights best practices in model management, focusing on nine critical areas on the radar of regulators: 1. Review credit risk policies regularly. This paper outlines best practices that not only help you prepare for a regulatory audit, but also evaluate and refine model performance to boost portfolio profitability. 2. Prepare a suitable data sample. 3. Ensure segmentation transparency. 4. Choose the right model type. 5. Validate model effectiveness. 6. Track performance. 7. Monitor overrides. 8. Defend decision strategies. 9. Document thoroughly. Make every decision count TM

2 »The» Compliance Challenge Financial institutions are now using predictive models on an increasingly broader scale, to measure capital reserve requirements and manage complex decisions across the credit account lifecycle. The greater complexity and number of models in use makes it even more difficult to meet the latest regulatory guidelines, particularly around model tracking and validation. Globally, the Basel Framework attaches great importance to model validation. Under Basel, financial institutions calculating capital requirements must fully understand how models used in internal ratings-based and risk-weighted asset calculations are developed; conduct regular, ongoing validation of such models; and prove they are responding to findings from their ongoing analyses. Basel also sets strict requirements for governance and documentation. Best Practices for Compliance 1. Review credit risk policies regularly Key Takeaway Conduct policy reviews every six months since these have a direct impact on your bottom line. Basel is intended as a set of guidelines, allowing individual countries to implement the regulations to fit their needs. While the core principles remain the same regardless of country, there are nuances in interpretation by regulatory authorities and even individual regulators assigned to a particular institution. 2. Prepare a suitable data sample 3. Ensure segmentation transparency 4. Choose the right model type 5. Validate model effectiveness Be able to demonstrate that your sampling techniques are complete, responsible and relevant. The best automated tools help ensure transparency for regulators, while enabling performance improvements. Select a model type that s appropriate for the decision type and available data, as well as one that ensures transparency and palatability. It is essential to revalidate models on an ongoing basis minimum once a year, but more often in a dynamic economy. 6. Track performance Employ standard reporting and analysis that provide critical insight into the health of models that drive your critical decisions. 7. Monitor overrides Monitor overrides carefully, and be able to prove that they are based on clear and consistent guidelines. 8. Defend decision strategies 9. Document thoroughly Since decision strategies have become increasingly complex, interactive strategy exploration functionality is critical for tracking strategies, strategy changes and results. Automate production and review of standard reports, and document any findings. Similarly, in the US, the Board of Governors of the Federal Reserve System (Fed) and the Office of the Comptroller of the Currency (OCC) recently released updated guidelines for financial institutions using predictive models across their business. Model validation still dominates the new guidance, but it is more clearly defined as within the broader scope of model management and governance. As with Basel, the Fed/OCC guidelines can be highly subjective and open to interpretation. While implementing new regulations is proving an enormous undertaking, the usefulness of required tracking, monitoring and documenting is not limited to compliance. Improving processes also enables institutions to evaluate and refine model performance and risk management practices in ways that control losses and boost portfolio profitability. In this paper, we outline best practices for preparing for a regulatory audit with this dual goal of compliance and competitive advantage in mind. April page 2

3 »» 1. Review Credit Risk Policies Regularly Regulatory compliance must be central to credit risk policies, which reflect an organization s broader objectives in terms of risk appetite. Because policies have a direct impact on the bottom line, reviewing policies every six months is considered good business practice. In the US, the OCC/Fed guidance requires a review of policies at least annually. A thorough review should ask: Does each policy serve a purpose? Policies have a tendency to become part of corporate culture. Regular policy reviews ensure you are not retaining a policy when it is no longer useful in the current business environment or overlooking newly emerging requirements. Are your policies defensible? Examine your policy requirements closely to determine if they are truly indicative of risk and empirically defensible. Are your policies consistent? Policies should be consistent throughout an account lifecycle and across channels, taking into consideration that you may need to interact differently with clients in person, over the internet or through a call center. Are your policies redundant/do gaps exist? Review policy performance to eliminate redundancies, reduce contradictions and identify gaps or overlaps. Gaps can result in excessive risk being introduced into your portfolio. In contrast, redundant risk mitigation can limit your opportunity to book good quality accounts. In other words, both extremes can impact your bottom line. Be sure to document the reasons behind each policy you observe. You may also consider hiring an independent third party to review your policies within the context of industry benchmarks, and challenge policies accepted as the norm within your organization. Top Model Management Challenges Manage growing model portfolios. Financial institutions are using predictive models on an increasingly broader scale. The challenge is finding an efficient way to manage and track models in production which can number in the thousands for large lenders and ensure these models are still performing well. Respond to regulatory requests. By implementing consistent, repeatable processes at every stage of the model lifecycle, lenders can respond to regulator requests promptly with sufficient documentation and a complete audit trail. Retain transparency. Models should be strong predictors, but they should also be easy to understand, defend and explain. This is important, for example, when explaining to a regulator why a particular characteristic or segmentation was used. Determine what to prioritize. By putting in place a model inventory and reporting schedule, financial institutions can more easily monitor high-impact models and identify ones most in need of review and redevelopment. Keep up with documentation. Solutions that standardize documentation and automate workflow enable financial institutions to demonstrate compliance and respond more rapidly to management and regulatory requests. page 3

4 »» 2. Prepare a Suitable Data Sample Since improper sampling can result in poorly suited models, regulators require you demonstrate that your sampling and model validation techniques are complete, responsible and relevant. This holds true for both the initial validation after you develop the model, as well as your ongoing model validations. Basel guidelines require that you use relevant internal or external data when testing and validating models. For your initial validation, the sample you use to validate a model should be independent of the development sample. This can inform whether a model is over-fit to training data, and provides a more realistic benchmark for how it would perform in production. For ongoing validation of models, we recommend that you: Avoid sampling when possible. It is best to use all records from a given time period to validate the model. When seasonality is an issue, choose a scoring window outside of that time period. If sampling, ensure a representative and adequate sample. Insufficient sample size can lead to poor conclusions in the model validation. When possible, select a random sample that adequately represents all subpopulations of interest. Keep in mind the economic, market and product situation during the timeframe in which you pull your sample, since this may impact the accuracy of your results. Enhancements to model oversight and validation processes are re-investments in a bank s intellectual property. This not only aligns with current regulatory guidance, but truly gives a bank solid support for treating this asset as capital today and will provide tomorrow s differentiator in competitive advantage. Pasquale Lapomarda, Retail Credit Risk Analytics Manager, TD Bank Be aware of data bias. It is highly likely that your data will be biased, for example, by the decision strategies you apply to new applicants. For accounts scoring well above your cutoff, you should expect a reliable odds-to-score relationship. However, near the cutoff score, this pattern may reasonably weaken or even reverse due to the influence of well-chosen overrides. Furthermore, because business outcomes are not known for rejected applicants, you won t see as strong of a separation between goods and bads compared with the development sample. Be prepared to defend these influences to regulators. Regulators will also inquire about your data hygiene processes. You should understand and document the accuracy of data sources, inputs, outputs, transformations and calculations for both model development and validation. Be prepared to demonstrate how you treat outliers and missing values. FICO recommends validating the reliability and quality of data sources yearly. 3. Ensure Segmentation Transparency Regulators require that you clearly document how you segmented the subpopulations within your model and how you determined the unique actions you took against each subpopulation. You also need to demonstrate that your segmentation supports your business objectives. page 4

5 Figure 1: Automate the discovery of predictive segments Regulators will ask whether you defined your subpopulation empirically or by a domain expert, and how your segmentation fits in with your decision strategies. In some countries, you may also need to demonstrate that your segmentation does not discriminate based on age, race or gender. Basel mandates that if you segment by product, you must do it under the umbrella of Residential Mortgage (RM), Qualified Revolving Retail Exposures (QRRE) and Other Retail (OR). The key to successful segmentation is in identifying the right variables to split a population into actionable segments. Automated tools and techniques now make this process significantly faster and easier. The best solutions deliver optimized segmentation schemes that substantially improve a model s precision, while maintaining a transparent and interpretable scoring solution for use with regulators. New tools, such as the Segmented Ensembles module in FICO Model Central Solution, make the process of finding optimal segmentation schemes much quicker and easier. Segmented Ensembles automates the process of searching the countless combinations of segmentation variables, split points and sequencing to find the best segmented model system. Remember that in order to validate and track a segmented modeling solution effectively, you will want to evaluate the entire system, as well as the individual segment models. 4. Choose the Right Model Type Financial institutions should select a model type appropriate for data type and decision area, and one that will provide robust predictions. For both business and regulatory purposes, also consider the following at model design time when choosing a model type: Transparency. Your model type should be easy to understand and explain, for both regulators and customers. Look for interpretable features that allow you to identify and explain what is driving a score result. A risk model should include reason codes, which many regulators require you give to customers when an adverse action is taken based on the score. Palatability. Regulators will ask about model outcomes, so it is important that model scores and reason codes have a high degree of face validity. Palatability is about intuition page 5

6 Figure 2: Ensuring transparency in model type Characteristic Points Length of Credit History in Months Less than or more 75 Number of Credit Accounts with Balance > Scorecards are commonly used model types in risk modeling because they are highly transparent and interpretable. This sample section of a scorecard shows how points are assigned for different values within a category of information. It s easy to see how each factor relates to an individual s score. and common sense, not complex mathematics. Does your model behave intuitively from a business context and is it directionally correct? For instance, as length of good credit history increases, does the risk score improve? Ease of engineering. During development, you may need to engineer or fine-tune a model to ensure it will address your identified business goal. This may require you to substitute or remove predictive characteristics that may be contentious in order to address regulatory requirements or customer concerns. You may wish to alter variable binnings or apply pattern constraints to smooth noisy data and improve the model s robustness. 5. Validate Model Effectiveness Once you have developed a model, you need to validate that it works according to your business objectives. You also need to revalidate on an ongoing basis once a year at a minimum, but more often in a dynamic economy. Validation evaluates your model s behavior over a range of input values and identifies any segments where the model has degraded. Overall, you should: Strive for clarity, consistency. Regulators want to see that you validate on a consistent basis, and that your process is repeatable. Regulators also want to know what threshold metrics you ve put in place and actions you are taking (such as more frequent reassessment, recalibration or rebuilding) when a model falls below an identified threshold. Create a supervisory review. In the US, the OCC/Fed requires your validation processes be reviewed by parties independent of those developing the model and designing and implementing the validation process. Globally, Basel puts an equally strong emphasis on governance. An independent reviewer should have the authority to challenge the recommendations of model developers. In addition, the reviewer should be given the authority to sign off on the final determination, in order to ensure that input is considered carefully rather than summarily overruled. Never validate in a standalone environment. Models and the scores they produce rarely operate in a vacuum; rather, they are intimately tied to business rules and decision strategies. Test a model s interactions with these other elements and simulate the impact of an updated model with respect to your customer portfolio. Include standard performance measures. Your validation checklist should include standard measures (K-S, divergence, ROC area, Gini coefficient, etc.), along with metrics that ensure the model rank-orders by score range. Measure both the model performance, as well as the score and attribute stability. You should evaluate this at least quarterly, and ideally monthly, in order to quickly identify changes. page 6

7 Regulators want you to demonstrate that you fully understand the philosophy around your validation process. You will need to document information such as the boundaries of your model s effectiveness and responses to a variety of market conditions. Figure 3: Use a variety of measures to assess model performance 100% DIVERGENCE 100% KOLMOGOROV-SMIRNOV STATISTIC (K-S) 100% RECEIVER OPERATING CHARACTERISTIC (ROC) % OF POPULATION Bads How far apart? How much overlap? Goods CUMULATIVE % OF ACCOUNTS Bads K-S Goods CUMULATIVE % OF BADS ROC Area Random 0% Low SCORE High 0% L ow SCORE High 0% 0% CUMULATIVE % OF TOTAL POPULATION 100% Divergence, K-S and ROC area are three useful measures of a score s predictive power. Divergence measures the separation of the score distributions between outcome classes (e.g., good vs. bad accounts). K-S quantifies the maximum separation between the score distributions. ROC measures how well the score classifies across the entire population. 6. Track Performance Over time, many factors can impact model performance. These include shifts in population makeup or behavior, economic changes, and changes to credit and collection policies. Regulators expect you to monitor models on a continual basis so you can recalibrate and rebuild them in a timely manner and modify your strategies accordingly. Figure 4: Using dashboards for ongoing model health checks Technology tools such as FICO Model Central Solution can be used to get an overall view of the health of the models across the organization. This should include functionality such as automated alerts indicating shifts in model performance or the distribution of scores and attributes. page 7

8 Tracking outcomes is vital to understanding how well business strategies are performing. This requires capturing what was known at the time of a decision, what actions were taken and what the resulting outcomes were. Automating the analysis provides faster feedback about predictions and assumptions, and makes it easier to identify and adjust for emerging trends and market fluctuations. Alerts can be used to identify when performance has shifted out of the target range. For a more detailed overview of several key model tracking reports, review the FICO white paper: Effective Tracking and Reporting Is Key to Precise Risk Management. Besides aiding in compliance, regularly producing these reports allows you to quickly detect deteriorating model effectiveness and emerging portfolio delinquency changes. This lets you modify approval and collection strategies more quickly, and avoid future losses. 7. Monitor Overrides Anytime you override a score, regulators will require that you document and monitor that decision carefully. Your overrides should be based on clear and consistent guidelines. Regulators will ask questions such as: What is your cutoff for an override? What authority level do you require for override approval? How many overrides are you doing every month? What is your policy for authorizing an override? Figure 5: Monitor overrides with an Override Tracking Report Override Code Each override reason should be assigned an identifying code for tracking in order to evaluate an underwriter s decisions. Use codes that allow for efficient and effective analysis. Strive to eliminate vague codes such as general or miscellaneous. Reasons for high-side overrides (accounts that score above the cutoff, but are declined) should be examined carefully to make sure you are not turning away potentially good customers and that no disparate impact is evident. Disparate impact is a fairness test that looks at whether certain minority groups are impacted by the decisions made differently than the majority, even if the scoring models themselves do not evaluate race or other characteristics that define the minority. Reasons for Decline Total Low Side High Side 1 Previous Derogatory (internal) Previous Derogatory (external) Debt Ratio High (>45%) Debt Ratio Low (<10%) Deposit Accounts w/ >$10K Bank Customer >10 years Miscellaneous You should also analyze reasons for low-side overrides (accounts that scored below the cutoff, but are approved). If the percentage is significant or unexpected, you should re-evaluate your override policy and follow up to see that it is being applied correctly. If an underwriter is constantly overriding a score, find out why. Does the underwriter have the proper understanding of how scores work? Or is the model deteriorating at an extent to which the underwriter feels it is no longer accurate? Total Overrides The Override Tracking Report enables you to pinpoint the reason for overrides. Strive to eliminate vague codes such as general or miscellaneous. The report should be produced quarterly at a minimum, and more often in a volatile economy. When tracking the business performance of overrides, a general rule of thumb is: An account booked as a result of a low-side override should perform no worse than one at the score cutoff. page 8

9 8. Defend Decision Strategies No matter how complex your decision strategies, regulators will expect you to explain and defend them with empirical results. Regulators will want to know how you develop, track and implement your strategies. You must also show the results of your strategies, including the realized losses, gains and exposures arising from your decisions. Most importantly, regulators will want to know how you balance the need to increase profits with the need to contain risk. Carefully document all your strategy decisions, as well as changes to those strategies. You should document what your subpopulations are, what actions you ve taken and where cutoff scores are applied. You must also demonstrate that your segments are homogeneous and actionable. Regulators might also ask you to pull performance and score information on an isolated subpopulation, which they define. They may ask you to pull a sample of your declines to make sure you are not engaging in regulatory violations, such as disparate impact or redlining (discrimination based on where a person lives). Since decision strategies have become increasingly complex, with hundreds or even thousands of nodes, an automated solution is essential to track such strategies, strategy changes and business results. You also want a solution that can simulate various what-if scenarios so you can understand the projected results of a decision, fine-tune decision strategies, balance risk and profits, and optimize business performance. Figure 6: Improved visualization tools aid in transparency of decision strategies Even in a simple strategy, multiple paths lead to the same action. Notice that both Letter and Phone Queue interrupt each other in other words, they aren t always adjacent on the decision tree graph. This makes it difficult to understand the pathways leading to a single action. This problem would be exacerbated in a real-world decision tree, which is times this size. Here, we see which paths lead to Phone Queue and which lead to Letter. Every action is presented only once on the graph, making the strategy more digestible. A strong strategy visualization tool also allows you to focus on a single action and see all the pathways that lead to that action. Decision trees are visually complex, sometimes having hundreds or even thousands of nodes. Advanced visualization solutions allow strategies to be viewed in ways that make it easier to explain and defend them, both within an organization and to regulators. In this graphic, FICO s Decision Graph offers multiple simplified views of the same collections treatment strategy. page 9

10 9. Document Thoroughly Regulators worldwide place tremendous importance on documentation and oversight. When a regulator asks you for proof of when you last ran a model validation, who reviewed the results and what action was taken, you need the right tools in place to quickly retrieve the supporting evidence. Improving Model Management FICO Model Central Solution provides a complete environment for managing predictive models in a reliable, automated and integrated way. The solution: Presents a management dashboard of overall model health, alerting personnel to performance degradation for action before business decisions are impacted. Creates a standardized process for easy management and monitoring of models, freeing resources to focus on other pressing business tasks and issues. Coordinates model validation, tracking tasks and management reporting, storing complete and annotated audit trails to satisfy compliance requirements. Deploys new models quickly and efficiently up to 50% faster speeding time to value and improving return on investment. Integrates models from various programming languages into one environment, further saving time and IT resources. Improves business outcomes dramatically with the integration of simulation and testing capabilities, and optimization of strategy decisions. Learn more about FICO Model Central Solution. Decision Execution Scoring Services Decision Simulation Decision Optimization ADVANCED DECISIONING PROFESSIONAL DEVELOPMENT Deployment & Verification Development & Calibration Model Data Mart Tracking Alerts FOUNDATION Monitoring Ongoing Validation Management Reporting With that in mind, you should keep an inventory of every model within your operating environment, cataloguing its purpose, usage and restrictions on use. List the types and sources of inputs. Your documentation should be detailed enough so that anyone unfamiliar with the model can understand how it operates, its limitations and your key assumptions. You also should be able to retrieve documentation for any vendorsupplied models, and demonstrate that you understand it. Your inventory should indicate if a model is functioning properly. It should include a description and dates of any updates, and a list of policy exceptions. It should also include names of individuals responsible for validation, a list of validation plans, findings of validations performed and any actions taken as a result. You should also have a complete audit trail of who modified a model and for what purpose, with a traceable path to the outcome of each modification. All annotations should be digitally captured and attributed to an individual, and the sequence of any changes should be apparent. By putting in place a technology that automatically ensures documentation and validation processes are managed correctly and consistently, financial institutions can ensure that highly trained analysts can focus on ad hoc regulator queries and new model developments, rather than being consumed with producing standard validation and tracking reports. And by centrally documenting a model s design and limitations, you reduce the risk of misapplying a model. FICO Model Central Solution is available in three tiers: Foundation for validation, monitoring, management reporting, alerting and administration; Professional Development, which includes Foundation services plus full model development and deployment capabilities; and Advanced Decisioning, which includes professional services and capabilities for testing and rapid learning adaptation. page 10

11 »Conclusion» Financial institutions today must operate in a highly regulated world. Although new regulations come with their share of overhead, once a financial institution has the people, processes and technology in place for proper model management and validation, it can go from merely complying with new regulations to proactively improving model performance. Stronger models, in turn, drive better decision making to improve business results. The Insights white paper series provides briefings on best practices, research findings and product innovations from FICO. To subscribe, go to For more information North America Latin America & Caribbean Europe, Middle East & Africa Asia Pacific (0) info@fico.com LAC_info@fico.com emeainfo@fico.com infoasia@fico.com FICO, Model Central and Make every decision count are trademarks or registered trademarks of Fair Isaac Corporation in the United States and in other countries. Other product and company names herein may be trademarks of their respective owners Fair Isaac Corporation. All rights reserved. 2811WP 04/14 PDF