Session 190 PD, Model Risk Management and Controls Moderator: Chad R. Runchey, FSA, MAAA

Transcription

1 Session 190 PD, Model Risk Management and Controls Moderator: Chad R. Runchey, FSA, MAAA Presenters: Michael N. Failor, ASA, MAAA Michael A. McDonald, FSA, FCIA Chad R. Runchey, FSA, MAAA

2 SOA 2014 Annual Meeting Session 190 PD, Model Risk Management and Controls System Access and Change Control Michael Failor Modeling Actuary SCOR Global Life Americas

3 Agenda Systems vs. Models Reasons for System Access and Change Control Change Control Components Leading Practices Systems Development Life Cycle Spreadsheet Specific Issues 2

4 System Access and Change Control Systems vs. Models Reasons for System Access and Change Control Change Control Components Leading Practices Systems Development Life Cycle Spreadsheet Specific Issues 3

5 Systems vs. Models Characteristics of Actuarial Systems Systems often contain a myriad of options and settings in which to build and run actuarial models Systems provide the underlying code (actuarial formulas) needed to run projection models Open systems provide users the ability to modify system code / formulas Closed systems prevent user-level changes to underlying system code System coding errors can affect every model built on the system (similar to DNA coding errors) 4

6 Systems vs. Models (cont.) Characteristics of Actuarial Models Often run on projection systems Defined by selecting options, settings and inputs accommodated by the underlying projection system (similar to genes in DNA) Models may be independent of each other Models are always dependent on underlying system calculations 5

8 Reasons for System Access and Change Control Modeling Demands ( givens ) Models will continue to evolve Models must be quickly modified to remain viable Model flexibility cannot be at the expense of controls Model integrity must be sustained 7

9 Reasons for System Access and Change Control (cont.) Modeling Demands Increased Systems Development More code development in more applications Vender provided open code actuarial projection systems Use of Excel (VBA) SQL interfaces C++ SAS Some major drivers Stochastic modeling Solvency II VM-20 System consolidations Product creativity Modeling Has Evolved. Are System Controls Keeping Pace? 8

10 Reasons for System Access and Change Control (cont.) Actuarial Modeling Controls: A Survey of Actuarial Modeling Controls in the Context of a Model-Based Valuation Framework (December 2012) Sponsored by the SOA Research and Analysis by Deloitte SCOR Participated in the Project Oversight Group Compared Current State of Controls Against Those Expected To Be in Place for Model-Based Valuation (MBV) Approaches Includes Analysis of Controls Surrounding System Coding Changes 9

11 Reasons for System Access and Change Control (cont.) On a scale of 1 to 5 (1 being the best), system change control received a 4 the worst score of all the categories. Note that system coding changes fall under this category. Modeling Governance Theme Score Current State Synopsis Governance Standards 3 While many companies employ a variety of model governance policies, few companies have a holistic, formal and documented model governance structure. General Modeling Process 3 Many companies have multiple models and modeling platforms and few companies incorporate a model steward role in the modeling processes. System Access and Change Control 4 Model changes are not generally governed by a formal change process. Model Assumption Management 3 Assumptions are regularly reviewed and updated, but with few controls in place to ensure assumptions are approved and input appropriately. Model Input Management 2 Many companies use automated feeds from admin systems for model inputs of liabilities. Other model inputs are often less automated. Model Output Management 2 Model output used for financial reporting purposes is generally well controlled, while model output for analysis and other purposes is generally less controlled. Source: Actuarial Modeling Controls -- A Survey of Actuarial Modeling Controls in the Context of a Model-Based Valuation Framework. Society of Actuaries, December

12 Reasons for System Access and Change Control (cont.) Majority of the survey respondents had no formal change control process Under system access and change control, the SOA sponsored report recommends two key next steps to move from the current state to leading practices: Implement a formal change management process for governing model code changes and model updates Determine the calendar for internal model releases to ensure consistency of the model of record across the organization Source: Actuarial Modeling Controls -- A Survey of Actuarial Modeling Controls in the Context of a Model-Based Valuation Framework. Society of Actuaries, December

14 Change Control Components Code Changes Are Often Performed by Actuarial Staff IT involvement beneficial Adoption and modification of existing IT control processes Code Comparison Tools May be provided in modeling system software Third party code comparison tools work with VBA (Excel and MS-Access) Code Changes Should Be Peer Reviewed Rigor varies from company to company SOA 2013 Valuation Actuary Symposium, Session 14 (Reviewing and Validating Actuarial Models), Systems Peer Review presentation 13

15 Change Control Components (cont.) Model Steward Must have top-down support Roles vary from organization to organization Secures production models Archives models When changes are made Periodic scheduled archival Applies version controls Assures that procedures have been properly followed Assigns access levels 14

16 Change Control Components (cont.) Clearly Defined System Access Controls Levels Run model Read only Change model inputs Change code Locations Production grids Testing servers Models and Systems Actuarial projection systems Specific models Spreadsheets Data 15

17 Change Control Components (cont.) Test Beds (or Test Packs) Need to be kept up to date Test beds have their limitations Management / Departmental Approvals Valuation Pricing Risk Management Others Communication and Documentation Start to finish High-level descriptions down to comments in the code Archived and linked to: Production models Test results Rollout schedules 16

18 Change Control Components (cont.) Different Modeling Environments Centralized models Model of record is maintained Simultaneous changes by different coders Requires a formal code aggregation process Integration testing Non-Centralized models No single model of record Pricing models vs. valuation models Potentially divergent modeling results May require code duplication efforts 17

20 Leading Practices* Typically Have Four Different Steps in the Control Process 1. Establish procedure to identify and prioritize model changes (change request process) 2. Evaluate coding changes in a test environment and analyze impact on financial results 3. Perform additional testing on the model code changes (depending on nature of model change) Regression testing Sensitivity testing Peer reviews 4. Document and seek formal approval Document changes and rationale, results and discussion, any sensitivity tests Appropriate parties review documentation and either approve or deny release If approved, schedule system release for production use (try to coordinate release date well in advance of quarter close) *Source: Actuarial Modeling Controls -- A Survey of Actuarial Modeling Controls in the Context of a Model-Based Valuation Framework. Society of Actuaries, December

21 Leading Practices* (cont.) Modeling Teams Responsible for Managing Change Requests Determine change request procedures Maintain and publish logs of all change requests Prioritize change requests IT Involvement in the Code Change Process For open code systems and centralized models Increase control and consistency around code *Source: Actuarial Modeling Controls -- A Survey of Actuarial Modeling Controls in the Context of a Model-Based Valuation Framework. Society of Actuaries, December

23 Systems Development Life Cycle What Is a Systems Development Life Cycle (SDLC)? Clearly defined process for planning, creating, testing, and deploying systems and systems modifications. SDLC Benefits Risk reduction Well defined roles and responsibilities Improved communication and documentation Auditability Are Actuaries Employing SDLC Methods? UK Actuaries: Institute and Faculty of Actuaries (2009 Best Practices) 22

24 Systems Development Life Cycle (cont.) Change Request Log Vital for any SDLC process Formalized documentation of requested system changes Reasons for the change? Is it a system correction or a system enhancement? Date when change needs to be in production? Consequences if change is not made? Other relevant details (i.e., change request ID, date, etc.) Each change request is assigned a priority level based on the above Published and archived 23

25 Systems Development Life Cycle V Methodology* Business Requirements User Acceptance Test Functional and System Design System Test Regression Test Technical Specification Unit Test BUILD *Source: Report From the Actuarial Processes and Controls Best Practice Working Party Life Insurance. Institute and Faculty of Actuaries (IFoA), May 31,

26 Systems Development Life Cycle V Methodology* (cont.) Business Requirements (BR) Produced by the actuaries requesting the change Details the required changes Includes calculation methodologies (i.e., reserve requirements) Functional and System Design Describes model design and structure as a solution to the BR Can be a source for an actuarial programmer to go directly to code Technical Specification Documentation at a more detailed coding level Required only for non-actuarial programmers (i.e., pure IT coders) 25

27 Systems Development Life Cycle V Methodology* (cont.) Unit Testing Ties back to technical specification Low-level testing of code segments in isolation System Testing Ties back to functional and system design Testing on a single policy or model point Scenario testing End-to-end testing Stress testing User Acceptance Testing (UAT) Ties back to business requirements Run in production mode Checks for reasonableness of results Regression Testing Performed periodically throughout the build and test cycle Assures that changes do not inadvertently impact other calculations 26

29 Spreadsheet Specific Issues At the Far End of the Flexibility vs. Control Paradigm Reduce the Number of Spreadsheets Periodic Archival Inventory and Document Workbook Components VBA / SQL code How to run VBA / macros and use spreadsheet Input data sources and links Description of outputs Microsoft add-ins Random number Others 28

30 Spreadsheet Specific Issues (cont.) Apply Workbook Access Controls Need to lock down models Limit directory access Protect Workbooks Apply Change Controls Workbook versioning VBA Code management methods apply Third party code comparison tools Evaluate code whenever workbook is modified 29

31 Spreadsheet Specific Issues (cont.) Additional Testing Concerns Workbook testing should be Excel version specific VBA testing may be affected by Excel session settings Common Risks Difficult to analyze large complex spreadsheets Key person risk Different levels of user / tester expertise 30

32 Thank You Michael Failor Modeling Actuary SCOR Global Life Americas 31

33 Model Risk Management and Controls Model Risk Management Chad Runchey, FSA MAAA Ernst & Young LLP

34 Model risk management Background Drivers of model risk Model risk management framework Common challenges 2

35 What is model risk? possible adverse consequences (including financial loss) of decisions based on models that are incorrect or misused. Federal Reserve Supervisory Letter 11-7 Can have many causes: Design errors Inappropriate or incorrect use Model calculation errors Poor data Misunderstood or poorly communicated assumptions or results Increases with greater model complexity, input/assumption uncertainty, breadth of use and materiality Model risk is not a new risk 3

36 What is driving the increased focus? More model-driven financial reporting and analysis Regulatory environment Solvency II internal capital models Federal Reserve comprehensive scope NAIC limited requirements Management/Board pressure Leading practices being set by other insurers CRO Council article on validation principles 4

37 Model risk management framework 1 st Line of Defense (Model Owners) Model Lifecycle Business Purpose Model Development Model Implementation Model Operation Performance Monitoring Model Change Management Model Governance 2 nd Line of Defense (Model Governance & Validation) Enterprise Governance Model Review Components Conceptual Soundness Data & IT Infrastructure Process & Controls Independent Testing. Roles and responsibilities 1 st line: Owns model lifecycle and related activities 2 nd line: Establishes model risk management framework and standards, provides review and challenge, and can perform independent model testing 3 rd line: Performs independent testing and verification of 1 st and 2 nd line activities 3 rd Line of Defense (Internal Audit) 5

38 Model risk policy A policy formalizes the framework and clearly states the roles and responsibilities of each group within the organization. Other key components of a policy include: Model definition Risk-rating approach Documentation standards Model validation requirements Compliance dates Ongoing assessment 6

39 First line of defense: Model owners Business purpose Model development and implementation Ensure design, theory and logic underlying model is supportable and documented Ensure that model components work as intended, are appropriate for intended business purpose, and are conceptually sound and mathematically and statistically correct Compare with alternative theories and approaches Demonstrate data is suitable for model and consistent with theory behind approach Test model s accuracy, robustness and stability, assessing potential limitations and documenting results Model operation Performance monitoring Assess performance over time under various conditions and as model applications change Gather insights from observation of outcomes and incorporate into feedback loop Model change management Implement model changes in a manner consistent with model change management framework Provide appropriate documentation 7

40 Second line of defense: Independent risk management Model Governance Assess model access rules Assess change management protocols Conceptual Soundness Ensure model methodology is theoretically correct and fit for purpose Ensure applicability of assumptions and other inputs Data & IT Infrastructure Assess applicability, accuracy, completeness and availability of all input data, including data mapping and transformation procedures Assess the sufficiency of the IT infrastructure Process and Controls Assess the efficiency of model processes and potential performance issues Assess the design and effectiveness of process controls related to the use and change management of models Independent Testing Assign risk ratings to all models and determine level of testing warranted Perform independent model testing and analyze results 8

41 Common challenges Governance and policy Model definition Model validation Management expectations Independence Sustainability Model documentation 9

42 Contact information Chad Runchey (212)

43 SOA 2014 Annual Meeting Session 190 PD, Model Risk Management and Controls MODEL VALIDATION AND CONTROLS Michael McDonald Consulting Actuary Actuarial Design

44 Overview Model Validation Model Controls Current Trends in Valuation Transformations Useful Links

46 Model Validation Why is it needed? Reserve and capital valuations are moving to model-based approaches Complexity of models has increased likelihood of material modeling errors Increased scrutiny from regulators and senior management Basis for Actuarial analysis of inforce exposure Sensitivity analysis Business Planning Asset-liability management

47 Model Validation Model Review Identify Errors Modeling methodology Calculations

48 Model Validation Model Review Identify model setup deficiencies vs. best practices Consistency Transparency Organization Intuitive object/variable names Hard coded values are stored in central location Well documented Model user-manual Documentation provides link between model calculation and source assumptions Conceptual description of calculations

49 Model Validation Model Vetting Validate Model results Vetting tools should be independent of model Build simplified calculator Compare results of simplified scenario for reasonability Build robust calculator for interim calculations Projected mortality calculation Projected NAAR calculation Dynamic lapse assumption

50 Model Validation Sensitivity Analysis Review against pricing model and/or historical sensitivities SENSITIVITY RECONCILIATION Pricing Sensitivity Differences Inforce Distribution Inforce Aging Actuarial Assumptions Economic Assumptions Unknown Differences TOTAL Valuation Sensitivity (x + y) x a b c d e y z

51 Model Validation Sensitivity Analysis Review against pricing model and/or historical sensitivities SENSITIVITY RECONCILIATION Pricing Sensitivity Differences Inforce Distribution Inforce Aging Actuarial Assumptions Economic Assumptions Unknown Differences TOTAL Valuation Sensitivity (x + y) x a b c d e y z

52 Model Validation Case Testing Roll forward inforce compare reserves vs baseline projected Inforce Aging Process time 0 Extract aged for 5 years Projections run for 5 years Reserve (0) Reserve(5) Projection Model

53 Model Validation Case Testing Stress-test dynamic assumptions Single policy unique attributes

55 Model Controls Documentation Model User manual/documentation manual Internal model documentation Issue log Version Controls Linear development Standardized model naming Model comparison tools Track object timestamps

56 Model Controls Model Development Validation Change summary Testing plan with expected impact Change management checklist Peer review Impact quantification Sign-off from valuation actuary

57 Model Controls Inclusion control User access Important for both production and development environment Consolidate calculations into one model Eliminate spreadsheet models! Too flexible Hard to document Slower calculations Different individual style

59 Valuation Transformation Legacy Design Extracts and assumptions reside in business function area Model output consolidated in back-end spreadsheets Next Generation Design All model assumptions reside in external databases (pricing and valuation) Models reference tables populated with admin data through data warehouse Model output is loaded directly into back-end data stores

60 Legacy Illustration PRICING TEAM ADMIN FINANCE Extract Extract Extract Extract PRICING Model Hypothetical Policy Data VALUATION Model Policy Data EXPERIENCE MONITORING Policy Data Queries Assumptions Product features Actuarial Feedback Assumptions Product features Actuarial Feedback Analysis CONSOLIDATION TOOLS Spreadsheets Desktop Apps FINANCIAL STATEMENTS

61 PRICING TEAM Next Generation Illustration ADMIN FINANCE ASSUMPTION DATABASE Product Features Actuarial Assumptions Scenario s Policy Info DATA WAREHOUSE Claims/Pre miums/exp enses Pricing Distribution Historic Versioning PRICING Model VALUATION Model FINANCIAL REPORTING SYSTEM Consolidates, organizes, aggregates financial results Query Tools for Inforce Analysis EXPERIENCE MONITORING Queries Analysis FINANCIAL STATEMENTS

62 Next Generation Illustration Product Features ASSUMPTION DATABASE Actuarial Assumptions Scenarios Policy Info DATA WAREHOUSE Claims/Prem iums/expens es Pricing Distribution FINANCIAL REPORTING SYSTEM Consolidates, organizes, aggregates financial results Query Tools for Inforce Analysis FINANCIAL STATEMENTS

63 Valuation Transformation Robust automated controls and processes Actuarial staff can focus time on analyzing results Non modeling staff can review model assumptions More granular model output available for analysis (ie. seriatim projections) Financial statement info from single source Increased transparency

65 Useful Links SOA Survey Model Control Modeling-Control.aspx EU Study Model Processes and Controls &ved=0cdsqfjaf&url=http%3a%2f%2fwww.actuaries.org.uk%2fsystem%2f files%2fdocuments%2fpdf%2factuarialprocessescontrolswplife200905rpt.pdf&ei =qppzu_c7grjlsqs29idiag&usg=afqjcnhhvm8nlnxsrg5hyjzwod0xtus Sjw&sig2=mWsOJGTI1aq6un5CWkE43w

66 QUESTIONS? Michael McDonald Consulting Actuary Actuarial Design