EMEA Banking Model Governance Framework

Transcription

1 EMEA Banking Model Governance Framework WHITE PAPER Sponsored by: FICO Michael Versace July 2012 Global Headquarters: 5 Speen Street Framingham, MA USA P F IDC FINANCIAL INSIGHTS OPINION As business analytics and complex modeling take center stage as differentiating capabilities during this time of financial recovery, financial institutions will find new challenges to governance programs that have traditionally focused solely on data and software development activities. Just as with the predecessor technology of spreadsheets, today's more sophisticated modeling engines are a priceless blessing to business analysts, investment officers, liquidity managers, and others thanks to the huge flexibility they provide and the ability of users to shape their own solutions even for complex computing needs. Financial institutions rely heavily on quantitative analysis and models in most aspects of financial decision making. IDC Financial Insights believes that the increased dependency on complex analytic models creates significant business, operational, and technology risks and highly opaque opportunities for rogue operations, and that real, quantitative impacts may result from modeling errors due to the lack of effective model governance programs. Developing and maintaining strong governance, policies, and controls over analytics and model risk is fundamental to the effectiveness of business strategies and financial decision making. A weak governance function reduces the value of modeling investments and quality of overall enterprise risk management programs. Model governance capabilities are necessary to support today's expanding analytical capabilities in order to: Promote the reliability and usefulness of modeling methods Encourage the effective use of well documented models and methods for critical business decisions, with the appropriate recognition of the capabilities and limitations of models and analytics used Establish accountabilities for the selection and use of modeling assumptions and the recognition of risks and uncertainties in the models July 2012July 2012, IDC Financial Insights #IDCWP24U

2 Create safeguards and methods to ensure data quality standards that ultimately limit the materiality of impacts on modeling results and business decision making IN THIS WHITE PAPER This paper defines the topic of model governance and examines the issues surrounding the effectiveness of related programs and practices across the financial sector. To complete this research, IDC Financial Insights developed a set of detailed questions and conducted workshop-like meetings, as well as qualitative interviews, with experts in analytics, modeling, and governance program development across the EMEA region. Meetings with over 40 practitioners were held in Stockholm, Frankfurt, Istanbul, and Madrid in June 2012 to form the evidential basis for this paper. SITUATION OVERVIEW Analytic excellence has quickly become a core requirement for success in today's financial market. Today's analytic discipline involves a wide range of calculations of varying degrees and complexities performed electronically or otherwise across the financial spectrum for everything from credit pricing to capital planning, and many risk-related operations in between. In describing today's environment, analytic experts interviewed for this research agreed that analytics and modeling functions are found in many planning, product, and operational corners of the enterprise. For example, experts spoke of analytics for: Retail credit capacity planning, and pricing individual retail products Relationship pricing to determine the best mix of product and pricing strategies for individual customers Predicting consumer behavior, and using advanced analytics to optimize target market campaigns and advertising Modeling to determine the impact of new vertical market strategies when integrated with existing lending and investment portfolios Estimating the impact of portfolio and counter-party risks to a bank's capital reserves Investigating financial fraud and predicting trends and areas of fraud reoccurrence Page 2 #IDCWP24U 2012 IDC Financial Insights

3 TABLE 1 Framework of Analytic Forms and Degrees of Insight Analytic Form Insights Generated Example Optimization What are the best options? What is the optimal retail product and pricing strategy for consumers with newly established residential mortgages in the southwest? Prediction What might happen next? What might we expect for repayment trends given current credit capacities of our customers given regional and national employment and housing trends? Exploration or forecast Will trends continue? How might non-performing loans trend given the forecast changes in the housing market and household incomes? Statistics Proof for what happened For the last 12 months, what were the low- and high-side lending overrides, including volume, reasons, frequency, and overall percentage of overall products scored? Alerts What needs to happen now? Our risk adjusted performance metric for consumer profitability is within 5% of a 6 month low. How should this affect our pricing strategies? Ad hoc query Who, what, where, and when Show me all the customer accounts with at least one portfolio with investments outside of the target asset mix for this period. Reports What happened Balance sheet as of end of quarter Source: IDC Financial Insights, 2012 As shown in Table 1, analytic operations come in many forms, according to those interviewed for this paper, and there are varying degrees of knowledge and insight that come of each operational form. For instance, some referred to standard reporting as an analytical operation, while noting that standard reporting provides a look in the rear-view mirror (tell me what happened) versus looking ahead. Statistics are another form of analytics, traditionally used to provide proof or further details into an activity or action of the past, helping determine the cause and effect of a specific management decision or market event. At the other end of the spectrum, predictive analytics provide insight into what might happen next or sometime in the future. For example, experts spoke about the use of predictive analytic models to understand the future credit capacity of consumers based on earnings 2012 IDC Financial Insights #IDCWP24U Page 3

4 history and longevity of employment and home ownership. Optimization, the highest form of analytic operations described, tends to be reserved for those operations that are option-based, where the user is looking for the best path to take for a specific risk scenario. For example, in cash forecasting and working capital planning, analytic modeling is used to optimize the use of assets (e.g., cash and securities) to finance an acquisition or provide sufficient liquidity reserves and meet regulatory constraints. Also optimization (often used in conjunction with predictive models that consider all aspects of a decision or "decision models") is increasingly being used by portfolio managers to drive scenario analysis focused on improving credit underwriting lending decisions or operational decisions in collections and recoveries. END-USER INSIGHTS Organizational Approach As expected, most firms we met spoke of increasing the use of business analytics and modeling techniques as the toolset to help their firms compete more effectively, navigate business through these challenging economic periods, and satisfy regulatory requirements. The following are examples of the experiences and projects shared during our workshop meetings: A leading relationship bank in the Nordic region continues significant investments in talent and infrastructure to establish risk based performance through decision analytics in its trade financing, SME lending, retail and private investing business lines. Analytic models, which have increased in both numbers and importance at a large European P&C insurer since the introduction of new capital requirements under Solvency II, have sparked interest by IT in grid and high performance architectures to meet the data and reporting requirements. Similar evaluations are taking place at a growing Turkish retail bank to help support its expanding credit card and risk services business. A large German cooperative bank reported a leveling of its investments in analytics and infrastructure after three years of a major ramp-up in its capital markets division. In all cases, these experts report an increased role for senior management in supporting the growth of analytics and models in credit and capital functions over the past 24 months. Governance approaches varied, however, from institution to institution, and few could speak with authority about who actually owned the responsibility for model governance at their institution. In some organizations, the governance or control function existed inside the specific business function, such as credit risk. In other organizations, a Page 4 #IDCWP24U 2012 IDC Financial Insights

5 separate risk control function had some form of governance and oversight responsibility for analytics and modeling across all lines of business. The control and oversight functions were almost always part of the enterprise risk group, holding specific responsibilities for: Understanding the inventory of key modeling and analytic functions across the enterprise and in specific lines of business Knowing strategies for which key modeling and analytic operational functions were being performed Ensuring proper management tests and sign-offs were obtained prior to significant changes in modeling or analytic operations, assumptions, or data Monitoring model variances and responding to alerts Periodically providing independent tests of models and analytic operations to look for errors or anomalies Model creation, testing, and validation processes are largely the responsibility of the individual business units that own the strategy. In credit risk, for example, the strategy for the lending or card business might be to grow cardholder balances and interest income by 10% over a defined forecast horizon. The card or lending line of business would own this strategy, and the new model, testing, and validation processes that establish the financial viability of executing against this strategy. IT, on the other hand, may own the process for controlling the movement of the new model(s) from a user environment to production. Once in production, IT also has responsibility for ensuring the integrity of the model, providing end-user access controls and security, and meeting model service and performance levels defined by the business unit. These responsibilities were fairly well defined and clear cut. What was less clear in this environment was who owns the data used to test and validate new models or models being changed. In many cases, standard ETL and FTP processes would be used to extract data from production business warehouses. The extracted data is then loaded to end-user computing databases where the data can be more easily manipulated. In this situation, data governance issues are created where custody and ownership responsibilities are split between business owners and IT. Drivers/Imperatives Risk-based performance management and regulatory reporting were cited as the most important drivers behind the increased use of enterprise analytics and modeling. Specifically, with regulatory constraints on capital set to increase as a result of regulations such as the BaFin Solvency Ordinance, which define how credit, market, and operational risks are to be determined, and in the coming years as 2012 IDC Financial Insights #IDCWP24U Page 5

6 Basel III is phased in, capital optimization and operational efficiency has become even more fundamental to the operational performance of financial institutions in Europe. And with the continued compression on margins across most business lines as reported by these European banking experts, it is clear that optimization of capital will continue to be central to driving profitability. For European insurers, the data modeling requirements and increased reporting for Solvency II across all balance sheet accounts, assets, and liabilities represents the most significant driver behind investments in analytic systems and resources. One insurer reported a sixfold increase in data modeling, simulation execution, and reporting requirements. At the same time, firms have seen a dramatic increase in regulatory scrutiny over governance programs as they are applied to analytic and modeling functions. There has been a tremendous increase in regulatory request for financial, risk, and performance data to validate regulatory reports, for example, and a similar increase in the presence of regulators on site and hands-on examination and testing of the effectiveness of governance controls. For all key modeling and analytic functions, regulators are keenly interested in the materiality of input and output assumptions, data quality, validation methods and checks, documentation, and other change management matters related to the models they examine. These factors have a significant influence on the management decisions these models support and are as important as the complexity of the calculation of the model itself. Industry Challenges Model risk is an outgrowth of the increased importance and use of scenario-based or predictive modeling and analytics. Model risk equates to the potential for negative results from decisions based on incorrect or incorrectly applied models and analytic operations. Model risk can lead to productivity loss, reputation damage, or financial consequences. At the highest level, model risk can come from a number of sources, including: Models may have fundamental errors that produce inaccurate results and ultimately poor business decisions. These errors can occur at any point, from design through implementation, and can come from a number of sources: Formula errors Poor sample design Incomplete validation and test design/execution A model may be used incorrectly or outside the environment for which it was designed. For example, a bank may apply an existing Page 6 #IDCWP24U 2012 IDC Financial Insights

7 predictive model for a new retail product or service without giving due consideration to current market conditions or customer behavior changes. Lack of data governance. A lack of certainty behind the source and official nature of data creates, in the eyes of end users, the most significant risk to modeling and analytic operations. And as data used in model validation moves further from its original application source and into end-user computing environments, the integrity of the data decreases as data governance capability degrades. From an operational and capacity point of view, end users voiced these additional concerns over their ability to establish effective model governance programs: The sheer growth in modeling and impact on inventory, maintainability, and accountability Data management efficiency and quality and establishing an aspect of "certainty" to inputs and outputs of all modeling efforts Change management and traditional independent reviews and testing prior to live use Ongoing validation and alerting when models fall out of established boundaries The split between IT and business responsibilities for data when data used in modeling is spread across multiple data marts and LOB servers and storage platforms Integration of model governance techniques with common GRC platforms (what's the opportunity for industry standards?) Need for more automation and audit trails to cover the complete life cycle and history of analytic models 2012 IDC Financial Insights #IDCWP24U Page 7

8 FIGURE 1 Model Governance Maturity Model Limited Governance Characteristics 1. Ad hoc governance: not recognized, little process rigor and repeatability 2. Inventories/activities not managed: complexity and delays, unknown data quality, no validation 3. Little separation of business and control functions; audit failures, after the fact 4. Solutions are a collection of data sets and content systems: little integration, automation, accountability 5. Overreliance on IT, resource constraint 6. Change delays, long development cycles, data inconsistencies Desired Outcomes/Value High Low Model Governance Must Be Considered as Part of an Enterprise Risk Maturity Model Lagging Filling Gaps Laggards Governance not formal or institutional, costly, errorprone, not defensible or repeatable, no audit trails or ongoing validation Operational/Manual Investment Horizon Optimized Leaders Official and accountable governance process, common repositories for data, models, assumptions, ongoing validation, repeatable with audit trails Maturing Characteristics 1. Persistent model governance culture: model risk performance and accountability; fact-based culture 2. Full model governance & IDE life cycle, from concept, to development, to validation and operation with audit trails, quality data 3. Centralized policy with federated accountability and operations shared by leadership; particularly trading and banking P&Ls 4. Data and variables repositories, and model registries with clear separation from production 5. Clear separation, with audit/examination interface for review and testing Source: IDC Financial Insights, 2012 The framework, goals, and best practices for effective model governance fit within the discipline of enterprise risk management. Figure 1 describes each end of a maturity model for model governance in this context, showing limited or lagging characteristics as well as characteristics of leading enterprises. Limited characteristics include ad hoc policy, limited inventory and change management automation, long development cycles and data inconsistency from model to model, execution to execution. In contrast, leading enterprises demonstrate a well-balanced model governance strategy that leverages organizational and technical infrastructures with capabilities and business disciplines that display: A risk culture across the enterprise, a complete accounting capability, and performance values for modeling across the credit, market, and operational risk disciplines High quality data that can defend measurable and predictable business outcomes An analytic orientation and platform with advanced, flexible, scalable, and auditable capabilities that demonstrate value wherever analytics are deployed CRM, product pricing, balance Page 8 #IDCWP24U 2012 IDC Financial Insights

9 sheet modeling, liquidity forecasting, regulatory reporting, capital usage, portfolio investment strategies, and fraud management Timeliness of activities, transparency and collaboration among model risk stakeholders, including internal and external parties A reliable, scalable, and assured risk infrastructure to maintain model risk and data quality service levels Model Governance Framework Model governance is the processes, human capital, and technologies necessary to ensure that a high degree of reliability and integrity can be placed on an analytic model's information relevancy, the transparency of a model's assumptions, the completeness and comprehensiveness of a model's accounting, and the communication of any uncertainty inherent in the model, its input data, or output analysis. A framework for model governance helps ensure the: Integrity of data and data sources used Accounting of models in use and their purpose Sufficient testing and validation of models and model changes Clarity of documentation and defensibility of audit trails Transparency of defined assumptions The goals for establishing a framework for a model governance program are to: Promote the reliability and usefulness of analytic and modeling methods Ensure the effective use of well-developed, tested, and documented analytic methods and models with the appropriate recognition of the capabilities and limitations Establish accountabilities and responsibilities for the selection and use of analytic and modeling assumptions and the recognition of risks and uncertainties, including data uncertainties Satisfy regulatory scrutiny and compliance requirements particularly where specific guidance has been offered, most notably through the FSA and the Office of the Comptroller in the U.S. Create safeguards and methods for ensuring data quality and data certainty 2012 IDC Financial Insights #IDCWP24U Page 9

10 Best Practices IDC Financial Insights recommends that firms expanding modeling and analytic capabilities establish a set of model governance best practices to achieve the above noted goals. Best practices must include: Inventories and classifications for all models across financial business units, in product groups, and from syndicated vendor solutions, including definitions of use, metadata references, and decisioning strategy supported The development, adoption, and use of standard production and deployment processes, including analysis of organizational responsibilities and technical standards for rigor, approach, and degree of effectiveness, efficiency, and defensibility Use of standard model validation frameworks and independent audit policies and practices that allow models to be continually validated. Model validation best practices must encompass at least five essential addressable areas to determine model performance: Underlying assumptions Data quality, protection, and decay Testing, out-of-bound conditions, and guidelines for model decay Output validation Audit trails Data risk management. The ultimate objective of any governance process is to build trusted relationships between various stakeholders involved in the creation, use, management, and accountability of data. The proxies for this need for trust may be represented in corporate policy, statutes, or regulations. A critical best practice for effective model governance therefore is the process of understanding the unique risks associated with data used in modeling operations and the probabilities of such risks impacting model outputs. Once all inherent risks are understood, specific methodologies should be developed to manage and control these risks in an integrated, balanced, and comprehensive manner. Optimized model governance frameworks. To this end, start by looking at the way in which model governance/control frameworks are feeding into decision management and ultimately performance optimization (the value chain). Based on this value assessment, governance and controls should be built into the process, not simply bolted on. Page 10 #IDCWP24U 2012 IDC Financial Insights

11 CONCLUSIONS With the growth and increased importance of modeling and analytics capabilities in the financial industry, information and IT governance programs can no longer be limited to traditional application software environments. Analytics extend governance responsibilities beyond data and IT functions and to the end-user modeling activities that form the core of tomorrow's modern financial enterprise. As modeling and analytic operations rise in importance as a differentiating business capability and compliance requirement, financial firms must reorient governance programs and explicitly focus on frameworks, goals, and best practices for model governance. DEFINITIONS Model. The term model in this document refers to a quantitative method, system, or approach that applies statistical, economic, financial, or mathematical techniques and assumptions to process data into quantitative-based decision making. Model risk. Model risk is the potential for adverse consequences from decisions based on incorrect or misused model inputs and outputs. Model governance. Model governance is the processes, human capital, and technologies necessary to ensure that a high degree of reliability and integrity can be placed on an analytic model's information relevancy, the transparency of a model's assumptions, the completeness and comprehensiveness of a model's accounting, and the communication of any uncertainty inherent in the model, its input data, or output analysis. ABOUT IDC FINANCIAL INSIGHTS IDC Financial Insights is one of the world's leading providers of independent research, custom consulting, and advisory services focusing on the business, technology, and operational issues within the financial services community. We are the preferred research partner for over 250 of the world's largest financial institutions and technology companies. For the past 5+ years we have been helping our clients understand and manage the challenges they face from a rapidly changing business, operational, and technological environment. We employ the industry's most talented minds, which gives our clients the insights that they rely on and advice they trust. Furthermore, we are the only research firm that has a significant physical presence worldwide, with analyst communities based in the Americas, Asia/Pacific, and Europe IDC Financial Insights #IDCWP24U Page 11

12 Copyright Notice Copyright 2012 IDC Financial Insights. Reproduction without written permission is completely forbidden. External Publication of IDC Financial Insights Information and Data: Any IDC Financial Insights information that is to be used in advertising, press releases, or promotional materials requires prior written approval from the appropriate IDC Financial Insights Vice President. A draft of the proposed document should accompany any such request. IDC Financial Insights reserves the right to deny approval of external usage for any reason. Page 12 #IDCWP24U 2012 IDC Financial Insights