Delegate Pack Solvency II: event on technical issues for life insurers intending to use an internal model

Transcription

1 Delegate Pack Solvency II: event on technical issues for life insurers intending to use an internal model Prudential Regulation Authority 20 Moorgate, London EC2R 6DA

2 Contents Plenary 1. Solvency II: technical issues for life insurers intending to use an internal model Sid Malik and Johann Meeke Workshops 2. Invested credit risk modelling Jemima Ayton and Daniel Curtis 3. Dependency and aggregation Vanessa Leung and Dan Georgescu 4. Longevity risk modelling Jemima Ayton and Rajeev Shah 5. Market risk Min Wei Chan, Niall Kavanagh and Manuel Sales 6. Proxy modelling validation Vanessa Leung and Dan Georgescu Additional information 7. QR code and website details

3 Solvency II: technical issues for life insurers intending to use an internal model Sid Malik Johann Meeke Date: 2 April

4 Aims for the afternoon a. Assist firms in improving the quality (not quantity) of submissions b. Discuss methods by which work done and judgements made can be effectively justified and communicated c. Be an open and transparent regulator Continued focus on fairness and consistency Experiences from reviews shared in response to requests Opportunity for firms to ask questions and discuss technical issues The PRA wants to help life insurers improve the quality of their internal model submissions Technical issues for life insurers plenary session 3 Overview of the afternoon Time Topic Location 12:30-13:15 Plenary session Auditorium 13:15-14:45 Workshop session one Market risk Longevity risk Dependencies Meet Kieran Barnes outside the front left door of the Auditorium Meet Laurienne Sherriff outside the front right hand door of the Auditorium Remain in the Auditorium 14: Coffee Outside Auditorium :35 Workshop session two Market risk Proxy models Credit Risk Meet Kieran Barnes outside the front left door of the Auditorium Meet Laurienne Sherriff outside the front right hand door of the Auditorium Remain in the Auditorium 16:35-17:30 Refreshments Outside Auditorium 17:30 Close Technical issues for life insurers plenary session 4

5 Agenda: plenary session 1. Framework 2. The PRA context 3. My team at the PRA 4. Focus on achieving fairness and consistency 5. Observations from review work to date 6. Key messages 7. Workshop logistics Technical issues for life insurers plenary session 5 1. Framework The material provided today is offered on the following basis: We have prepared these slides on the basis of the information available at this time. This workshop is being held in order to facilitate readiness, but it should be acknowledged that this discussion is not taking place against the background of final adopted texts and firms should ensure they familiarise themselves with all obligations as they are published. The delegated acts have not yet been published and there are still Implementing Technical Standards (ITS) and Regulated Technical Standards (RTS) and guidelines to come (see the EIOPA website for a timeline). Ultimately, the adopted Level 2 regulations (delegated act and EIOPA ITS and RTS) will set out definitively what is required to comply with the Solvency II regime. Unless expressly permitted to do so by the Solvency II regime, the PRA will not be placing interpretations or other requirements on firms. Therefore: providing firms are complying with the Level 2 regulations they will have complied with their obligations under EU law; and as Level 2 regulations are directly applicable, firms do not need to seek out a PRA view or interpretation of the provisions in order to comply. The comments made in these slides and during the workshops do not represent a statement of PRA policy. None of the comments made in these slides or during the workshops should be taken to mean that the PRA considers that Level 2 should apply in advance of its formal coming into effect. There may be discussion about actions firms can take in readiness for the Solvency II regime, but this does not create any legal obligation to comply earlier than the implementation date of 1 January Any emphasis on a particular provision in the Solvency II regime in these materials or at the workshops should not be taken to imply that it is a priority or an area of particular supervisory concern for the PRA, or vice versa. The PRA will have a duty to supervise the entire Solvency II regime consistently with the obligations in Level 1 and 2 texts. Where approaches for compliance with Level 2 are discussed, they merely represent a possible method of complying with Level 2 obligations and should not be interpreted as representing the only way which the PRA will accept. Approaches are put forward for consideration rather than to be prescriptive. Presenters may mention certain good practices based on reviews to date for a sample of firms. More good practices or even better practices may emerge as other firms models are reviewed. In addition, what is considered as a good practice for a particular type of firm may not necessarily be considered such for all other firms. Technical issues for life insurers plenary session 6

6 2. The PRA context The PRA has two statutory objectives: To promote the safety and soundness of the firms it regulates, focusing on the adverse effects that they can have on the stability of the UK financial system. A specific objective to insurance firms, to contribute to ensuring that policyholders are appropriately protected. And a secondary objective: Facilitate effective competition The PRA has a risk assessment framework which: focuses on issues of the most importance; pays particular attention to reducing the impact of firm failure, and; is an assessment backed by proactive intervention framework which provides a ladder of intervention Technical issues for life insurers plenary session 7 3. My team at the PRA (1/2) Life Insurance and Pensions Risk Department within the PRA Made up of 40 plus risk specialists Line staff with firm knowledge supplemented by specialist teams Our support to Supervision covers both Groups and Retail Life firms Collaborative Other technical specialists Supervisory colleagues Bank and FCA colleagues Technical Issues for Life Insurers Plenary Session 8

7 3. My team at the PRA (2/2) Our role is to: ensure that firms follow principles and practices which do not threaten the PRA s statutory objectives ensure compliance with Solvency II use our knowledge and expertise to provide technical support to internal stakeholders lead on capital assessments and support other firm initiated activities The PRA s actuarial experts treat firms fairly and consistently Technical issues for life insurers plenary session 9 4. Focus on achieving fairness and consistency (1/2) Specialisms Dedicated teams with expertise and deep knowledge of key risks Developed PRA approach on methodology and calibrations and trained staff Lead and support on firm internal model reviews Multi-disciplinary review teams Supervision and risk specialists supported by Prudential Legal Function and Insurance Policy Focused on correct interpretation and application of Solvency II Knowledge sharing Quality assurance and peer review Peer review two-fold: within specialist reviews for each firm as a whole Audit trail Management challenge Technical issues for life insurers plenary session 10

8 4. Focus on achieving fairness and consistency (2/2) Developed tool to identify key risk drivers in firms complex stochastic models and informs individual capital guidance Role of benchmarks Independent panel process A discipline for bringing together the views of the PRA on a firm in one place Senior level challenge Enables consistency between firms Technical issues for life insurers plenary session Observations from review work to date Documentation Quality not quantity Complete, clear and easy to follow Evidence to justify the conclusions Expert judgement Justify clearly Qualify the experts Gold-plating Not allowed Taken very seriously at PRA Technical issues for life insurers plenary session 12

9 6. Key messages a. Enhance the quality (not quantity) of submissions b. Work done and judgements made need to be justified and communicated c. Open and transparent regulator Experiences from reviews shared in response to requests Opportunity for firms to ask questions and discuss technical issues Continued focus on fairness and consistency Feel free to ask questions and make comments and observations both during the workshops and throughout the day. Technical issues for life insurers plenary session Workshop logistics Time Topic Location 13:15-14:45 Workshop session one Market risk Longevity risk Dependencies Meet Kieran Barnes outside the front left door of the Auditorium Meet Laurienne Sherriff outside the front right hand door of the Auditorium Remain in the Auditorium 14: Coffee Outside Auditorium :35 Workshop session two Market risk Proxy models Credit Risk Meet Kieran Barnes outside the front left door of the Auditorium Meet Laurienne Sherriff outside the front right hand door of the Auditorium Remain in the Auditorium 16:35-17:30 Refreshments Outside Auditorium 17:30 Close Technical issues for life insurers plenary session 14

10 Solvency II: technical issues for life insurers intending to use an internal model Sid Malik Johann Meeke Enjoy the workshops 2 April

11 Invested credit risk modelling Daniel Curtis Jemima Ayton 2 April Objectives Discuss key elements of the modelling process for invested credit risk. Share examples of what the PRA has seen to date in internal model reviews. Discuss and share views. No matching adjustment discussion, the focus is purely asset-side models. Feel free to ask questions and make comments and observations during the workshop Invested credit risk modelling workshop 18

12 Agenda 1. Key modelling choices 2. More detailed model mechanics 3. Credit-specific discussion of over-arching topics 4. Conclusions 5. Questions and comments Invested credit risk modelling workshop Key modelling choices a. Reconciliation of short-term and long-term views of invested credit risk b. Should the credit model include the asset model? c. Risks included in the credit model d. Role of risk identification in optimal model design Invested credit risk modelling workshop 20

13 1a. Reconciliation of short-term and long-term views of invested credit risk All the models the PRA has seen to date are reduced-form spread models designed to capture the change in asset value over one year There are challenges in reconciling these models with a long-term view of default risk which is the usual perspective adopted by annuity writers ORSA can act as a bridge between these two views of the world Firms should consider how they can reconcile their short and longer term views of invested credit risk Invested credit risk modelling workshop 21 1b. Should the credit model include the asset model? A spread model in isolation cannot provide an updated asset value Firms typically use asset terms and conditions models or proxy models, to which spread is an input To date the PRA has seen insufficient attention placed on either the documentation or validation of the asset models, or the links between the asset and spread models Discussion: Should the credit model include the asset model? Invested credit risk modelling workshop 22

14 1c. Risks included in the credit model Solvency II Directive (Article 101) requires firms to take into account all material and quantifiable risks in their SCR calculation To date the PRA has seen limited evidence of firms undertaking risk identification exercises and so cannot be sure that firms are meeting the Article 101 requirements Examples of invested credit sub-risks include (but are not necessarily limited to): Spread Transition Default Callability Subordination/ranking Explicit risk modelling may not be appropriate in all cases Credit models should include all material and quantifiable risks Invested credit risk modelling workshop 23 1d. Role of risk identification in model design Lack of good risk identification processes poses challenges in terms of model design The PRA has seen limited evidence of assets being split into homogenous risk groups and then modelled using a model appropriate to the risk characteristics Some examples of what we have seen in this area: Standard spread models used for all assets, including non-vanilla bonds Standard spread model with uplifts for more complex assets Development of non-spread models for material holdings of non-standard asset classes Stresses applied to the valuation models Firms should consider how good risk identification work can aid model design Invested credit risk modelling workshop 24

15 2. More detailed model mechanics a. Calibration hot topic: Data Invested credit risk modelling workshop 25 2a. Calibration hot topic: Data (1/2) Firms should consider how they can obtain a complete, accurate and appropriate set of data to model credit risk and how data can be supplemented by expert judgement What data set should be used when calibrating a credit model? Areas where firms have been considering this issue include: Modelling UK exposures using US data? The extent to which data should be split by duration? The extent to which data should be split by sector? Length of the dataset to use i.e. credibility vs. appropriateness Should regime changes be reflected in the data? Invested credit risk modelling workshop 26

16 2a. Calibration hot topic: Data (2/2) There are usually differences between a firm s asset holdings and the calibration data used Duration Type Geography Sector It is unlikely that any data set will be entirely free from basis risk Firms should consider the extent to which the data they have chosen are adequately reflective of their own portfolio Invested credit risk modelling workshop Credit-specific discussion of over-arching topics a. Use test b. Validation Invested credit risk modelling workshop 28

17 3a. Use test (1/3) Model design There are many ways in which invested credit risk can be modelled including: Reduced form vs. structural Simple vs. complex Asset by asset vs. portfolio level Point-in-time vs. through-the-cycle There is no right answer to model design but the model chosen should be justifiably appropriate for the risk profile of the firm Invested credit risk modelling workshop 29 3a. Use test (2/3) Model granularity Model granularity (or lack thereof) limits the ability of the model to be useful in a firm s risk management and system of governance Overall model granularity is a function of asset and spread model granularity Detailed asset model + spread modelled purely at rating model = relatively non-granular risk modelling Spread modelled using rating, duration and sector + single-factor proxy model = relatively non-granular risk modelling The PRA has seen limited evidence for, and demonstration that, the level of granularity used is appropriate for uses other than calculating the SCR Invested credit risk modelling workshop 30

18 3a. Use test (3/3) Discussion: How far should firms push the invested credit risk component of their internal model to meet a range of uses vs. using stand-alone specialised credit models? Potential uses: Calculate required economic capital Allocate required economic capital Set risk limits Monitor risk limits Set investment mandates Monitor investment mandates Test new asset strategies? Invested credit risk modelling workshop 31 3b Validation Validation has tended to focus on model input parameters rather than outputs The PRA have seen the following tools used: Scenario analysis Back-testing Comparison with risk appetite Sensitivities Firms senior management are likely to be significantly interested in how the model outputs compare to recent extreme events Discussion: What are the views of the audience on what good validation would look like for the invested credit risk component of an internal model? Invested credit risk modelling workshop 32

19 4. Conclusions Firms key modelling decisions may include: How to reconcile short and long-term views of credit risk The role of the asset model in the overall credit model How to ensure all material and quantifiable risks are included How to use risk identification to its best advantage A key hot topic in calibration is data. There are generic as well as portfolio-specific considerations. Use test and validation are important Invested credit risk modelling workshop Final comments/questions Invested credit risk modelling workshop 34

20 Dependency and aggregation Vanessa Leung Dan Georgescu 2 April

21 Objectives Talk over key issues in dependency and aggregation modelling Share examples of what the PRA has seen in the reviews conducted to date Discuss and share views Feel free to ask questions and make comments and observations during the workshop Dependency and aggregation workshop 37 Agenda 1. Overview of dependency/aggregation methodologies commonly adopted by firms 2. Specific considerations a) Correlation matrix aggregation approach b) Gaussian copula dependency structure 3. Calibration techniques a) Data analysis b) Expert judgment 4. Conclusions 5. Final questions and comments Dependency and aggregation workshop 38

22 1. Overview of dependency/aggregation methodologies commonly adopted by firms 1. The use of a variance/covariance approach (also known as correlation matrix approach) a) Single stage b) Multi-stage (e.g SII standard formula) c) Risk geographies 2. The use of fully stochastic models a) Copula + marginal distributions approach b) Structural economic scenario generators The PRA does not prefer a particular modelling approach for dependency but asks firms to consider the appropriateness and limitations of their chosen approach. 39 Agenda 1. Overview of dependency/aggregation methodologies commonly adopted by firms 2. Specific considerations a) Correlation matrix aggregation approach b) Gaussian copula dependency structure 3. Calibration techniques a) Data analysis b) Expert judgment 4. Conclusions 5. Final questions and comments Dependency and aggregation workshop 40

23 2a. Specific considerations - Correlation matrix approach 1. The implicit assumptions of this technique can be summarised as : A: the relationship between risk and capital are linear and additive, B: all of the individual risks are normally distributed (or elliptically distributed); and C: the relationship between risks is assumed to be the same in moderate and extreme scenarios. 2. Some firms use non-linearity adjustments to address the above simplifications Discussion: How effective are non-linearity adjustments in addressing the deficiencies in this aggregation technique? Are there remaining material limitations and how are these addressed? Dependency and aggregation workshop 41 2a. Specific considerations - Correlation matrix approach 1. How do you calculate the PDF at other percentiles a) Explicit considerations for the appropriateness of the assumptions and calculation for each percentile to the same rigour as the 99.5 th percentile? b) Consistency between the assumptions made in the calculation for other points and that made for the 99.5 th percentile? 2. Implications given low richness of the probability distribution forecast a) Risk management b) Capital allocation c) Model validation Firms should consider whether the chosen aggregation methodology allows them to generate a PDF that is rich and accurate enough to capture all relevant characteristic of the risk profile and support decision making Dependency and aggregation workshop 42

24 2b. Specific considerations - Gaussian copula dependency structure Firms may want to consider the appropriateness of the Gaussian copula for the dependency of their own risks: a) Do you allow for tail dependency? b) Do you allow for dependency between risks which varies according to the direction of the risks? c) Where the same assumptions are used for both Group and solo, do you consider the appropriateness of the assumptions for each entity separately? d) How do you ensure that PSD (or other mechanical) adjustment does not materially change key assumptions against your views from initial analysis? Discussion: How well does a Gaussian copula reflect the underlying dependency modelled across all risks? Are key features in the dependency between specific risks captured by this copula? Dependency and aggregation workshop 43 Agenda 1. Overview of dependency/aggregation methodologies commonly adopted by firms 2. Specific considerations a) Correlation matrix aggregation approach b) Gaussian copula dependency structure 3. Calibration techniques a) Data analysis b) Expert judgment 4. Conclusions 5. Final questions and comments Dependency and aggregation workshop 44

25 3a. Calibration techniques - Data analysis Actual data suggests ve correlation suggests +ve correlation Change in interest rate (%) Scatter plot % -20% -10% 0% 10% 20% 30% Oct 87-2 Change in equity Dependency and aggregation workshop 45 3a. Calibration techniques - Data analysis Scatterplots of the data at the start of data analysis Effect of correlations varying through time and according to market conditions Rationale behind the choice of calibration window Forward-looking calibration Tail effects Copulas are approximations to the empirical joint distribution of the data Expert judgement overlay Goodness-of-fit tests Sensitivity testing Communication to senior managers Dependency and aggregation workshop 46

26 3b. Calibration techniques Expert judgement Doc. Criteria to identify Assess Validation Challenge Governance Evidence, rationale Dependency and aggregation workshop 47 Agenda 1. Overview of dependency/aggregation methodologies commonly adopted by firms 2. Specific considerations a) Correlation matrix aggregation approach b) Gaussian copula dependency structure 3. Calibration techniques a) Data analysis b) Expert judgment 4. Conclusions 5. Final questions and comments Dependency and aggregation workshop 48

27 4. Conclusions Correlation matrix approach - Consider the assumptions - Document limitations and deficiencies - Assess effectiveness of adjustments (e.g. for non-linearity) - Consider whole PDF Copula approach - Document methodology for copula selection for material risk pairs - Assess effectiveness, ongoing appropriateness - Triggers for recalibration Calibration techniques - Where there is data: analyse it, use recognised statistical techniques to calibrate, assess fit, communicate appropriation to audience - Where there is little data and expert judgment is used, it should be appropriately documented so that it is visible to the users of the expert judgment Dependency and aggregation workshop Final questions and comments Dependency and aggregation workshop 50

28 Longevity risk modelling Jemima Ayton Rajeev Shah 2 April

29 Objectives Talk over key elements of the longevity risk modelling process Share examples from the internal model reviews conducted to date Discuss and share views Feel free to ask questions and make comments and observations during the workshop Longevity risk modelling workshop 53 Agenda 1. Key modelling choices 2. More detailed model mechanics 3. Longevity-specific discussion of over-arching topics 4. Conclusions 5. Questions and comments Longevity risk modelling workshop 54

30 1. Key Modelling Choices a. Risk definition and risk identification b. Granularity of modelling c. Consistency with technical provisions Longevity risk modelling workshop 55 1a. Key modelling choices: risk definition and risk identification The definition of longevity risk used and identification of relevant sub-risks is important as it is likely to drive the modelling approach taken. The PRA has seen two broad definitions to date: i. Longevity risk defined in terms of actual outcomes ii. Longevity risk defined in terms of changes in assumptions These definitions tend to map to one of two different modelling approaches: i. The run-off method for actual outcomes ii. The direct one-year or expectations-based method for changes in assumptions The PRA does not prefer a particular modelling approach for longevity risk but asks firms to ensure all relevant sub-risks are captured in their chosen approach. Longevity risk modelling workshop 56

31 1b. Key modelling choices: granularity of modelling Firms generally break longevity risk into sub-components for modelling purposes. Most firms reviewed to date have split risks in two ways: i. Base mortality risk - The risk that current mortality experience differs from the base table used for the best estimate assumption. This can then split into various sub-risks such as random volatility, parameter uncertainty, anti-selection, relevance and identification. ii. Trend risk - The risk that future improvements in annuitant mortality differ from the assumed improvements used to set technical provisions. This can also split down further. The PRA does not mandate any particular split of longevity risk into component parts, but notes that firms should think about how they will demonstrate compliance with Article 101(3) for longevity risk. Longevity risk modelling workshop 57 1c. Key modelling choices: consistency with technical provisions (1/2) The market for longevity risk is not deep and liquid. Firms cannot therefore refer to a market price for longevity risk when constructing their ICA/SII balance sheets. Instead an assumption about current and future longevity risk experience should be made to determine best estimate liabilities. Firms should consider what consistency with the methods used to calculate technical provisions means for longevity risk Longevity risk modelling workshop 58

32 1c. Key modelling choices: consistency with technical provisions (2/2) Discussion: When calculating the SCR how closely should the process for determining the longevity risk distribution relate to the best estimate assumption setting process? 1. Should the longevity risk model be applied at the same level of granularity at which the technical provisions are set? Why? 2. Should different modelling approaches be used for setting technical provisions and for determining the SCR/ICA? 3. Should different modelling approaches be used for setting technical provisions and for determining the SCR/ICA? 4. What difference should one expect between the actual best estimate and the best estimate implied by the capital model and should it matter? Longevity risk modelling workshop More detailed model mechanics a. Longevity risk data b. Longevity risk models Longevity risk modelling workshop 60

33 2a. More detailed model mechanics: Longevity risk data A balance often has to be struck between using expert judgement and using data. Internal data vs. external data: Firms generally do not have sufficient internal data particularly for modelling future improvements in longevity. External data are more plentiful but may pose challenges regarding completeness, accuracy and appropriateness. Common appropriateness issues in respect of data include use of males data for modelling females etc. Firms should ask what role data can, and should, play in their longevity risk modelling Longevity risk modelling workshop 61 2b. More detailed model mechanics: Longevity risk models There are a range of longevity risk modelling approaches available. Considerations that we have seen being made when choosing a model or models include: Is the model picking up all of the features of the data? Is model error being allowed for? How up-to-date is the approach? Is the model too complex? Is it well understood? What are its limitations? How risk responsive is the model? Firms should consider whether all appropriate considerations are being taken into account when they select their longevity risk models Longevity risk modelling workshop 62

34 3. Longevity-specific discussion of over-arching topics a. Expert judgement b. Validation c. Use of the longevity model Longevity risk modelling workshop 63 3a. Longevity-specific discussion of over-arching topics: Expert judgement Discussion: What makes a good longevity risk expert judgement? Who should the experts be? What does the expert need to know? Are there enough skills outside the expert group to challenge and validate expert judgements? Longevity risk modelling workshop 64

35 3b. Longevity-specific discussion of over-arching topics: Validation Discussion: What amount and type of validation is required for longevity risk modelling? Who carries out the validation? What needs to be validated for longevity risk? Data Risk identification and definition Modelling methodology Material assumptions (implicit and explicit) Calibrations Other? Longevity risk modelling workshop 65 3c. Longevity-specific discussion of over-arching topics: Use of the longevity model Discussion: What does use mean for the longevity risk component of an internal model? How should use of the model relate to the materiality of the risk? How should the longevity risk model link to the firm s management of longevity risk? How does the quality of the modelling impact use? How do the insights obtained depend on the modelling approach used? Longevity risk modelling workshop 66

36 4. Conclusions The PRA does not mandate a particular modelling approach or split of risks for longevity. To date longevity reviews have shown positive progress from firms towards meeting the Solvency II tests and standards. Areas that require further consideration include: Achieving consistency with the approach used to set technical provisions Use of data Range and type of models used The role of expert judgement The validation that should be undertaken for longevity risk What use means in a longevity risk context Longevity risk modelling workshop Final comments/questions Longevity risk modelling workshop 68

37 Solvency II industry conference Market risk workshop Min Wei Chan Niall Kavanagh Manuel Sales 2 April

38 Objectives Highlight some key points of interest in our internal model market risk methodology reviews Share the PRA s observations and some examples from our reviews of firms methodologies to date Discuss and share views Feel free to ask questions and make comments and observations during the workshop Market risk workshop 71 Agenda 1. Key modelling choices 2. Methodology 3. Validation 4. Summary 5. Questions and discussion Market risk workshop 72

39 1. Key modelling choices Risk identification Basis risk Granularity of modelling Market risk workshop Key modelling choices Risk identification (e.g. analysis of own exposures) What characteristics of the firm and exposures are being taken into account? What are the asset and liability profiles? Proportionate to the materiality of the risk Basis risk To what extent is there a risk that actual investment exposure differs from exposure assumed in the model, especially in stressed conditions? Granularity Is it sufficient for the model to be useful e.g. use-test, risk management? How are private equities, hedge funds etc. treated? Market risk workshop 74

40 1. Key modelling choices Risk identification (e.g. analysis of own exposures) What characteristics of the firm and exposures are being taken into account? What are the asset and liability profiles? Proportionate to the materiality of the risk Basis risk To what extent is there a risk that actual investment exposure differs from exposure assumed in the model, especially in stressed conditions? Granularity Is it sufficient for the model to be useful e.g. use-test, risk management? How are private equities, hedge funds etc. treated? Market risk workshop Key modelling choices Risk identification (e.g. analysis of own exposures) What characteristics of the firm and exposures are being taken into account? What are the asset and liability profiles? Proportionate to the materiality of the risk Basis risk To what extent is there a risk that actual investment exposure differs from exposure assumed in the model, especially in stressed conditions? Granularity Is it sufficient for the model to be useful e.g. use-test, risk management? How are private equities, hedge funds etc. treated? Market risk workshop 76

41 1. Key modelling choices Risk identification (e.g. analysis of own exposures) What characteristics of the firm and exposures are being taken into account? What are the asset and liability profiles? Proportionate to the materiality of the risk Basis risk To what extent is there a risk that actual investment exposure differs from exposure assumed in the model, especially in stressed conditions? Granularity Is it sufficient for the model to be useful e.g. use-test, risk management? How are private equities, hedge funds etc. treated? Analysis of exposures and basis risk is important because it demonstrates a firms understanding of its risks and informs their modelling approach. The model should be proportionate but granular enough to be useful. Market risk workshop Methodology a. Choice of model b. Choice of data c. Model fitting Market risk workshop 78

42 2a. Choice of model What is being modelled and why? Equities: absolute returns, log returns, returns in excess of risk free rate, etc. Interest rates: approaches to modelling the yield curve; absolute or multiplicative changes? What are the implications of these choices? Justification for choice of model How well does it capture the underlying characteristics of the risk / data? Justification for a conditional or unconditional modelling approach How are the model assumptions appropriate? Governance process to ensure the model remains adequately calibrated? Market risk workshop 79 2a. Choice of model What is being modelled and why? Equities: absolute returns, log returns, returns in excess of risk free rate, etc. Interest rates: approaches to modelling the yield curve; absolute or multiplicative changes? What are the implications of these choices? Justification for choice of model How well does it capture the underlying characteristics of the risk? Justification for a conditional or unconditional modelling approach How are the model assumptions appropriate? Governance process to ensure the model remains adequately calibrated? Market risk workshop 80

43 2a. Choice of model What is being modelled and why? Equities: absolute returns, log returns, returns in excess of risk free rate, etc. Interest rates: approaches to modelling the yield curve; absolute or multiplicative changes? What are the implications of these choices? Justification for choice of model How well does it capture the underlying characteristics of the risk? Justification for a conditional or unconditional modelling approach How are the model assumptions appropriate? Governance process to ensure the model remains adequately calibrated? Market risk workshop 81 2a. Choice of model What is being modelled and why? Equities: absolute returns, log returns, returns in excess of risk free rate, etc. Interest rates: approaches to modelling the yield curve; absolute or multiplicative changes? What are the implications of these choices? Justification for choice of model How well does it capture the underlying characteristics of the risk? Justification for a conditional or unconditional modelling approach How are the model assumptions appropriate? Governance process to ensure the model remains adequately calibrated? The PRA is interested in the justification for the choice of model, why it was chosen over others and the process to ensure it remains appropriate. The PRA does not prefer a particular type of model. Market risk workshop 82

44 2b. Choice of data Market risk workshop 83 2b. Choice of data Choice of data How closely does it reflect own exposures? Alternatives? Length of data period Equities relevance and impact of older data? Interest rates consideration of regime changes? Justification for adopting a shorter or longer data period? Data enrichment Non-overlapping or overlapping? What are the limitations and/or biases of each approach? Market risk workshop 84

45 2b. Choice of data Choice of data How closely does it reflect own exposures? Alternatives? Length of data period Equities relevance and impact of older data? Interest rates consideration of regime changes? Justification for adopting a shorter or longer data period? Data enrichment Non-overlapping or overlapping? What are the limitations and/or biases of each approach? Market risk workshop 85 2b. Choice of data Choice of data How closely does it reflect own exposures? Alternatives? Length of data period Equities relevance and impact of older data? Interest rates consideration of regime changes? Justification for adopting a shorter or longer data period? Data enrichment Non-overlapping or overlapping? What are the limitations and/or biases? Market risk workshop 86

46 2b. Choice of data Choice of data How closely does it reflect own exposures? Alternatives? Length of data period Equities relevance and impact of older data? Interest rates consideration of regime changes? Justification for adopting a shorter or longer data period? Data enrichment Non-overlapping or overlapping? What are the limitations and/or biases? The PRA is interested in how firms strike a balance between the range of available data to inform their calibration. Market risk workshop 87 2c. Model fitting Choice of fitting method Is the consideration of appropriate method focussed on material risks? What are the advantages, disadvantages and limitations? How sensitive is the calibration to alternative choices? How good a fit to the data can be achieved (across the pdf)? The PRA is interested in the justification for the choice of model fitting approach over others especially as the choice can have an impact on the final calibration. The PRA does not prefer a particular model fitting approach. Market risk workshop 88

47 3. Validation a. Validating overall approach b. Sensitivity testing c. Goodness of fit Market risk workshop 89 3a. Validating overall approach Key considerations Risk identification Data (e.g. choice of dataset, period, overlapping / non-overlapping etc.) Modelling methodology Material assumptions and expert judgements Calibrations A good approach to independent validation covers each key step of the model calibration process, including the initial risk identification/exposure analysis stage and calibration of the final output. Market risk workshop 90

48 3b. Sensitivity testing Sensitivity testing of alternative choices As part of calibration process: How is this being quantified? How does this inform the final calibration? As part of validation process: How is it used to inform independent validation? How are key sensitivities made known to senior management? Sensitivity testing can form an important and informative part of both the model calibration and validation process. Market risk workshop 91 3c. Goodness of fit Does the model achieve a good overall fit to the underlying data? Broader qualitative considerations Does the model fit well at the key percentiles of interest? Why is it acceptable for the model to fit less well at certain points? What are the issues and limitations of the goodness of fit tests used? What happens if the goodness of fit tests fail? How does the model compare to past events? A good approach to assessing goodness of fit takes account of broader qualitative considerations, i.e. it is more than just a statistical pass or fail. The PRA does not have a preference for particular goodness of fit tests. Market risk workshop 92

49 4. Summary A number of alternative approaches are likely to meet Solvency II requirements The PRA has seen good progress by firms Areas for further consideration by firms may include: Risk identification understanding of exposure and how this informs approach Basis risk how material this is, and whether specific allowance should be made Model choice why was it was chosen, and how will it remain appropriate Data choice how a balance is achieved between all relevant data Model fitting justification for choice, especially if choice affects calibration Validation including evidence of independent validation of risk ID and calibration Sensitivity testing how this informs the calibration and validation process Goodness of fit consider this to be more than just a pass or fail test Market risk workshop Questions and discussion Market risk workshop 94

50 Proxy modelling Validation Dan Georgescu Vanessa Leung 2 April

51 Objectives Talk over the key features of validation of proxy models Share examples of observations the PRA has made in the reviews conducted to date Feel free to ask questions and make comments and observations during the workshop Proxy modelling workshop 97 Agenda 1. What is a proxy model and the role of validation? 2. Out-of-sample testing 3. Evaluating the results from out-of-sample tests 4. Other considerations 5. Summary 6. Questions and comments Proxy modelling workshop 98

52 Agenda 1. What is a proxy model and the role of validation? 2. Out-of-sample testing 3. Evaluating the results from out-of-sample tests 4. Other considerations 5. Summary 6. Questions and comments Proxy modelling workshop What is a proxy model and the role of validation? What is a proxy model? A model which approximates the results of the technical provision models For example, replicating portfolios or replicating formulae Most firms have developed some form of proxy model to produce a probability distribution forecast of changes in basic own funds and calculate capital requirements. Proxy modelling workshop 100

53 1. What is a proxy model and the role of validation? Relevant articles from the SII Directive A121(1): The internal model, and in particular the calculation of the probability distribution forecast underlying it, shall comply with the criteria set out in paragraphs 2 to 9. So, the whole PDF must comply with SQS not just the biting scenario Proxy modelling workshop What is a proxy model and the role of validation? Relevant articles from the SII Directive A121(2): The methods used to calculate the probability distribution forecast... shall be consistent with the methods used to calculate technical provisions. Proxy modelling workshop 102

54 1. What is a proxy model and the role of validation? A124: The model validation process shall include an effective statistical process for validating the internal model which enables the undertakings to demonstrate to their supervisory authorities that the resulting capital requirements are appropriate. The statistical methods applied shall test the appropriateness of the probability distribution forecast compared not only to loss experience but also to all material new data and information relating thereto. The model validation process shall include an analysis of the stability of the internal model and in particular the testing of the sensitivity of the results of the internal model to changes in key underlying assumptions. Proxy modelling workshop 103 Agenda 1. What is a proxy model and the role of validation? 2. Out-of-sample testing 3. Evaluating the results from out-of-sample tests 4. Other considerations 5. Summary 6. Questions and comments Proxy modelling workshop 104

55 2. Out-of-sample testing An example of a statistical process Out-of-sample (OOS) testing Scenarios (not used in fitting) are evaluated using both the TP model and the proxy model. OOS are a good way to demonstrate that there is an effective statistical process for validating the proxy model Proxy modelling workshop Out-of-sample testing Scatterplot of OOS testing Light (change in BoF m) Heavy (change in BoF m) # Light Heavy H minus L 1 (796) (764) 32 2 (927) (1008) (81) 3 (981) (1055) (74) 4 (960) (1167) (207) 5 (832) (484) (1018) (1153) (135) 7 (981) (942) 39 8 (1054) (1109) (55) 9 (985) (991) (6) 10 (1147) (1108) (1072) (1041) (927) (797) (1010) (1044) (34) 14 (962) (1030) (68) 15 (869) (801) (1041) (902) (972) (901) (911) (1025) (114) (1060) (1058) 2 20 (913) (801) 112 Proxy modelling workshop 106

56 2. Out-of-sample testing The PRA has observed that some firms perform a substantial amount of out-of-sample testing with carefully chosen scenarios to rigorously test the fit of their proxy models. Discussion: Factors which should drive level of OOS testing Proxy modelling workshop Out-of-sample testing Choosing scenarios Combined scenarios, from the neighbourhood of the SCR Risk driver scenarios, from the risk driver distribution At different levels of aggregation Covering the majority of the pdf Proxy modelling workshop 108

57 Agenda 1. What is a proxy model and the role of validation? 2. Out-of-sample testing 3. Evaluating the results from out-of-sample tests 4. Other considerations 5. Summary 6. Questions and comments Proxy modelling workshop Evaluating the results from out-of-sample tests 3a. Defining success 1. In order to assess whether the proxy model is fit for purpose, there is a need to define what a pass or fail means on the validation carried out. 2. Considerations on setting the success/acceptance criteria a) Linking this to the materiality/confidence interval for the SCR b) Group/solo entity c) Different types of businesses d) Errors measured with respect to the Assets/Liabilities alone or the net asset value/loss Discussion: How do you define your acceptance criteria? Proxy modelling workshop 110

58 3. Evaluating the results from out-of-sample tests 3b. Investigating errors 1. How should the errors be investigated? a) Individually? b) averaging fitting errors? 2. Investigating the outliers a) What are outliers? b) Do they point to more systematic issue? Example of a SCR calculation with 10 scenarios only: A. Heavy model produces: 3, 1, 6, 4, 10, 9, 16, 18, 24, 26 B. Proxy model produces: 1, 3, 4, 6, 8, 11, 14, 20, 22, 28 Average difference is zero, the 90 th percentile would be understated Firms should consider how the errors are investigated and whether further insights can be drawn from the results Proxy modelling workshop 111 Agenda 1. What is a proxy model and the role of validation? 2. Out-of-sample testing 3. Evaluating the results from out-of-sample tests 4. Other considerations 5. Summary 6. Questions and comments Proxy modelling workshop 112

59 4. Other considerations Communicating the suitability of the proxy models to the users of the model How well do you articulate the percentiles and situations where the fit of the model is good and where it is poor/uncertain? Do the validation reports clearly identify the percentiles/situations tested? What are the limitations of the model, and are they well understood by users and reviewers of the model? Firms should consider how the appropriateness and limitations of the proxy model can be communicated to users of the model Proxy modelling workshop Other considerations Failure resolution and frequency of recalibration 1. If the validation indicates that the proxy models are not fit to be used for the reporting cycle, then what is the process to be followed? a) What is plan B? b) What further developments would need to be carried out? c) Should not ignore the bad news 2. How often should the proxy models be recalibrated? a) Linked to frequency of calculation of SCR? b) What other circumstances could require a recalibration of the proxy models? Firms should plan for the eventuality that the proxy model is not fit for purpose Proxy modelling workshop 114

60 5. Summary The PRA does not prescribe how to validate proxy models The discussions today have highlighted areas firms may consider when validating their proxy models: The use of out-of-sample testing Coverage of the out-of-sample tests scenarios Interpreting the results from the validation tests Communication and failure resolution Proxy modelling workshop Final questions and comments Proxy modelling workshop 116

61 QR code and website details