Model Risk Management Nav Vaidhyanathan Director, Head of Model Risk Management Wintrust Financial Corporation Global Association of Risk Professionals August 2014
These materials reflect the views and opinions of the speaker and are not intended to reflect the views, policies or practices of Wintrust Financial or its affiliated companies. They are for informational purposes only. The views expressed in the following material are the author s and do not necessarily represent the views of the Global Association of Risk Professionals (GARP), its Membership or its Management. 2
Agenda Model Risk Management is a broad, evolving, and an expanding topic. This presentation will cover only select components at a high level. Inherent risk (model tier) and residual risk of model A framework for model monitoring Model risk aggregation framework Model validation for stress test (e.g., DFAST) models 3 2014 Global Association of Risk Professionals. All rights reserved.
Model Risk Axioms* The following can potentially be considered as self-evident truths for Model Risk. Given everything else equal, All models have limitations Model Risk is related to model limitations Model Risk increases with more models (this might fall in the grey area) E.g., two Tier 1 models potentially pose higher risk than 1 Tier 1 model (everything else being the same, i.e., similar limitations, similar validation status, similar materiality, similar findings, etc.) Model Risk for Tier 1 model > Model Risk for Tier 2 model Model Risk for validated model Model Risk for unvalidated model (for the same model) More findings more Model Risk (same category and level of findings) Certain types of findings will have higher Model Risk than others Models with deteriorating performance pose higher Model Risk Model Risk is exhibited by the variance between model output and observed actual (this is just one of the ways that Model Risk exhibits itself) * may not be axioms in the purest sense of the term 4 2014 Global Association of Risk Professionals. All rights reserved.
Residual Risk Rating Inherent Risk and Residual Risk of Model Residual Risk Rating could be driven by: Model validation findings: Category Level # of Open Findings Model performance / monitoring High Moderate Low Low (Tier 3) Moderate High (Tier 1) Inherent Risk Rating (Model Tier) Model owners can reduce residual risk by improving models, improving controls, and/or remediating findings Inherent Risk Rating could be driven by: Materiality Type of model Used for regulatory / financial reporting or critical business decisions Feeds into another Tier 1 model? Model owners cannot (usually) reduce inherent risk 5 2014 Global Association of Risk Professionals. All rights reserved.
Residual Risk Rating Framework Residual Risk Rating (High, Moderate, Low) is based on the Residual Risk Score range E.g., RRS > 200 High, RRS 100 Low If the model is deemed unfit for use, the RRR is High The Residual Risk Score Range to Residual Risk Ratings mapping needs to be monitored periodically The ranges can be determined by typical observations from model validation and model monitoring exercises If a model is unvalidated, the residual risk rating is HIGH 6 2014 Global Association of Risk Professionals. All rights reserved.
Model Monitoring Model monitoring should evaluate whether changes (including anticipated) in products, exposures, activities, clients, or market conditions necessitate adjustment, redevelopment, or replacement of the model Ongoing activities include review of input data and assumptions, outcome analysis, model stability checks, review of overlays and expert judgments, benchmarking, etc. Key considerations Model performance monitoring Pros and cons of model performance measures Model type Model use Type of measure Periodicity of monitoring Thresholds for model recalibration or rebuild From performance measure to model performance rating 7 2014 Global Association of Risk Professionals. All rights reserved.
A Bottom Up Framework For Model Risk Aggregation Impact of Failure (e.g., on Capital) Tier 1 Tier 2 Tier 3 Validated Yes No High Residual Risk Moderate Residual Risk Low Residual Risk As part of an RCSA process, the impact of the model failure is estimated by model owners The end nodes in the decision tree can be used for aggregating model risk For DFAST projection models, for Severely Adverse Scenario, the impact on capital can be assessed using backtesting during Great Recession period for all models (adjust the variance by severity of the scenario compared to the Great Recession) By looking at the distribution, a conservative starting point for capital can be chosen for each model Tier 8 2014 Global Association of Risk Professionals. All rights reserved.
Some Considerations in Validation of Stress Test Models Are estimates conditioned on macroeconomic scenarios? Do the models produce outcomes that show separation and order between the baseline, adverse, and severely adverse scenarios? Are the scenario and assumptions consistent across all models? Are exposures modeled using appropriate levels of segmentation? Qualitative elements of the model Relevance of third party vendor models to company specific characteristics, especially when using proxy portfolio Sensitivity analysis around assumptions Use of weak models vs. expert judgment For models not validated, compensating controls and conservativeness Documentation Capital buffer for model risk 9 2014 Global Association of Risk Professionals. All rights reserved.
C r e a t i n g a c u l t u r e o f r i s k a w a r e n e s s Global Association of Risk Professionals 111 Town Square Place 14th Floor Jersey City, New Jersey 07310 U.S.A. + 1 201.719.7210 2nd Floor Bengal Wing 9A Devonshire Square London, EC2M 4YN U.K. + 44 (0) 20 7397 9630 www.garp.org About GARP The Global Association of Risk Professionals (GARP) is a not-for-profit global membership organization dedicated to preparing professionals and organizations to make better informed risk decisions. Membership represents over 150,000 risk management practitioners and researchers from banks, investment management firms, government agencies, academic institutions, and corporations from more than 195 countries and territories. GARP administers the Financial Risk Manager (FRM ) and the Energy Risk Professional (ERP ) Exams; certifications recognized by risk professionals worldwide. GARP also helps advance the role of risk management via comprehensive professional education and training for professionals of all levels. www.garp.org. 10 2014 Global Association of Risk Professionals. All rights reserved.
Model Risk Management for Non Banks Michelle McCarthy Managing Director, Risk Management Nuveen Investments Global Association of Risk Professionals August 2014
Disclaimers These materials reflect the views and opinions of the speaker and are not intended to reflect the views, policies or practices of Nuveen Investments or its affiliated companies. They are not intended to provide investment advice and are for informational purposes only. The views expressed in the following material are the author s and do not necessarily represent the views of the Global Association of Risk Professionals (GARP), its Membership or its Management. 2
Contents Differences between models and their uses at banks vs. the buy side What should be in scope for model risk management activity in the asset management business? Potential tiering of asset management models, and effect on model risk management activities 3 2014 Global Association of Risk Professionals. All rights reserved.
Most literature on model risk management has concerned banks OCC 2000-16 OCC 2011-12, FRB SR 11-7 The case of AXA Rosenberg brought the conversation to the buy side, without much guidance on how bank practices can and should differ from asset managers practices 4 2014 Global Association of Risk Professionals. All rights reserved.
Different uses of models in asset managers vs. banks Asset managers are not computing regulatory capital Bank-owned asset managers may need to provide operational risk capital calculations, but this is not in service of the asset management business We rarely use valuation models as the sole source of valuation; we frequently use external sources Many asset management models are an input to a decision, but not the sole driver of that decision 5 2014 Global Association of Risk Professionals. All rights reserved.
Potential scope of model management activity for asset managers Includes models that are used for investing or hedging Drive investing activity to some extent Value key assets Does not include calculators; model must provide estimates and involve uncertainty to require model risk management Good change controls always an important operational control, but calibrations and validations should be reserved for models that estimate uncertainty Would not include compliance systems Would not include price-to-yield calculations Would not include simple screens for fundamental stock characteristics Would not include pure index replication models without meaningful uncertainty Does not include analyses that help develop an investment thesis only the tools used to execute the thesis, or to value assets once purchased 6 6 2014 Global Association of Risk Professionals. All rights reserved.
Tiering for materiality A potential tiering of asset management models: Tier 1 Tier 2 Tier 3 Tier 4 Models that: have high potential direct impact on company results are broadly applied across the entire company are understood and maintained by a relatively narrow range of experts at the company Result in instant electronic trading with little or no ability to intervene, and/or affecting a large portion of the portfolio are tightly coupled to trading but permit human intervention, and/or affect a small portion of the business result in trading with little other input than direction from a model (quant strategies) provide hedge ratios for using derivatives, or to balance proportions of fixed income instruments, in portfolios value smaller, less significant balance sheet items than those captured in Tier 1 apply a screen to a universe of stocks, using some measure of uncertainty, such as a measurement of the dispersion of some fundamental characteristic (low priceto-earnings ratios, for example) compare characteristics of securities to assist the portfolio manager in finding securities that he or she deems desirable may support a certain amount of decision making, but with many other inputs to these decisions outside the model. are outside the company s areas of key business risk are understood by a broad range of business people Such as: Statistical arbitrage model High frequency trading model A model that is rigorously adhered to, is key to a broad set of strategies and is disclosed as a critical feature of the investment process Significant valuation models where third party sources are unavailable A strategy that strongly adheres to rebalancing if Barra or Yieldbook measures exceed a threshold An economic indicator model that is the basis of an investment strategy High frequency trading models Models used for delta hedging or asset allocation rebalancing Minor valuation models where third party sources are unavailable Using Barra or Yieldbook to see where a portfolio is overweight, in order to select trades that will balance the portfolio better Economic indicator screens to provide general investment signals Index replication models with tilts Risk oversight models Valuation models where third party valuations are available Measures of portfolios that are not used to drive action Index replication models that approach 100% replication 7 2014 Global Association of Risk Professionals. All rights reserved.
Slotting typical asset management models into tiers Tier I Tier II Tier III Tier IV Investment Models Fundamental Business Models x x Stock Selection Screen x x Quantitative Stock Selection x x Portfolio Construction x x Index Replication with Tilts x Index Replication x Statistical Arbitrage x x High Frequency Trading x x Leverage x Hedging x Economic Models Growth x x Inflation x x Industry x x Credit x x Liquidity x x x Risk Models Capital Loss x Liquidity x x Ex Ante Tracking Error x x Ex Post Tracking Error x Duration, Beta, Factor Exposure Limits x x Valuation Models Third Party Pricing Services x x Pricing Model - Third Party Developed x x x Pricing Model - In-House Developed x x x 8 2014 Global Association of Risk Professionals. All rights reserved.
What steps may make sense for model management of the different tiers? Inventory For material models (Tiers 1 to 2) Initial independent validation Where independence may or may not require the different reporting lines set forth for banks: effective challenge Documentation (model theory, assumptions, limitations, code, user instructions) Change validation Periodic formal, documented calibration Periodic re-validation given results of calibration Model management for Tier 3 models could be lighter in terms of frequency, intensity, formality of independence The challenge of third party vendor models 9 2014 Global Association of Risk Professionals. All rights reserved.
C r e a t i n g a c u l t u r e o f r i s k a w a r e n e s s Global Association of Risk Professionals 111 Town Square Place 14th Floor Jersey City, New Jersey 07310 U.S.A. + 1 201.719.7210 2nd Floor Bengal Wing 9A Devonshire Square London, EC2M 4YN U.K. + 44 (0) 20 7397 9630 www.garp.org About GARP The Global Association of Risk Professionals (GARP) is a not-for-profit global membership organization dedicated to preparing professionals and organizations to make better informed risk decisions. Membership represents over 150,000 risk management practitioners and researchers from banks, investment management firms, government agencies, academic institutions, and corporations from more than 195 countries and territories. GARP administers the Financial Risk Manager (FRM ) and the Energy Risk Professional (ERP ) Exams; certifications recognized by risk professionals worldwide. GARP also helps advance the role of risk management via comprehensive professional education and training for professionals of all levels. www.garp.org. 10 2014 Global Association of Risk Professionals. All rights reserved.