Methods to Measure Operational Risk in the Superannuation Industry

Transcription

1 Methods to Measure Operatonal Rsk n the Superannuaton Industry Amandha Ganegoda Abstract: Australan Government Treasury (2001a) recognzes operatonal rsk (OpRsk) as the most sgnfcant rsk for superannuaton funds. Gven the sze of the super ndustry and the role t plays n the communty t s of utmost mportance that super fund managers take relevant actons to measure and mange ther OpRsks. However measurng OpRsk s a dffcult task for reasons such dffculty n dentfyng rsk, nature of certan OpRsk (e.g. rogue traders), scarcty of data, budget and tme constrants etc. The focus of ths artcle s to explore ways n whch OpRsk can be measured n practce whle overcomng these dffcultes. I dscuss smple top-down approaches such as scalars, benchmarks, Black swan approach, Regresson and trend analyss, fnancal statement models and more advanced bottom-up approaches such as Expected loss models, LDA, Secnaro-based models and Process approaches. Artcle provdes bref explanatons on how to mplement each of these methods and then dscuss the advantages and dsadvantages of each method and sutablty of them to overcome the aforementoned dffcultes of measurng OpRsk. 1

2 1. Introducton Superannuaton ndustry s the second largest sector n the fnancal servces ndustry n Australa n terms of assets under management (Reserve Bank of Australa, 2007). APRA (2007a) reports, as of September 2007 super funds held A$1.2 trllon n assets. To most households ther superannuaton savngs represent the most mportant asset after home ownershp (The Australan Government Treasury, 2001a). The magntude of the ndustry t self and the role t plays n the communty makes the superannuaton ndustry one of the most mportant sectors n the fnancal servces ndustry. Though most super funds do not provde explct guarantees on retrement benefts for ther members, fact that many Australans depend on the system as ther man source of retrement ncome makes t necessary that super funds are regulated properly to mnmze the probablty of falure. Australan Government Treasury (2001a) recognzes operatonal rsk as the most sgnfcant rsk for a super fund gven that market rsk and nvestment rsk s borne by the fund members. However, surprsngly there are only lmted regulatory requrements for super funds to manage ther operatonal rsk compared to other ndustres lke bankng and nsurance. For example, current regulatory captal requrement for super funds requre approved trustees to hold only A$5 mll assets regardless of the fund sze or operatonal process. And for those funds that operate wthout an approved trustee, there s no captal requrement. There have been recent government dscussons to reform the prudental captal requrement for super funds n order to algn the captal requrement wth fund sze and the operatonal process (Treasury, 2001b). Thus t s hghly lkely that n near future superannuaton ndustry may need to quantfy ther operatonal rsk n order set up prudental captal. Ths paper ams to ntroduce a consstent framework to measure OpRsk for the superannuaton ndustry. In the precedng secton we ntroduce a basc framework to measure OpRks by categorzng OpRsk nto three categores: known-known rsk, known-unknown rsk and unknown-unknown rsk. In secton 3, 4 and 5 we dscuss dfferent methodologes avalable to measure each rsk category by lookng at technques used by dfferent ndustres such as Bankng, Insurance, Nuclear, Chemcal and Avaton to measure ther OpRsk. Here we dscuss 2

3 pros and cons of each methodology and possblty of extendng them to measure OpRsk n the superannuaton funds. Secton 6 concludes. 2. A Framework to Measure Operatonal Rsk The framework we propose to measure OpRsk s nspred by a statement gven by the Unted States former secretary of defense, Donald Rumsfeld. Thus we called the method Rumsfeld Approach. We broadly categorze all OpRsk nto three categores Known-Known Rsk that we know exst and know how to model Known-Unknown rsk that we know exst but do not know how to model (or hard to model) (e.g. legal rsk) Unknown-Unknown rsk that we are unaware of The motvaton behnd the Rumsfeld approach s to provde a consstent framework that would take nto account all three categores of OpRsk to determne the rsk captal. The subsequent sectons dscuss the sutable methods avalable to measure the OpRsk for each category. 3. Modelng Known-Known OpRsk OpRsk that we know exst and can be modeled are categorzed as known-known OpRsk. There are many approaches that have been used by the fnancal ndustry as well n other ndustres to model such rsk. Most of these approaches can be broadly subdvded nto two categores: Top-down approaches whch attempt to model aggregate operatonal losses wthout gvng attenton to underlyng operatonal process and bottom-up approaches whch attempt to model losses at each rsk cell by mappng the operatonal process to rsk cells and then aggregate them across. 3.1 Top-Down Models Scalars Scalars are smplstc top-down methods to measure operatonal rsk. Approach assumes operatonal rsk to be a pre-determned percentage of a busness scalar such as gross ncome, operatonal costs, assets, funds under management etc (Lawrence, 2000). Two well known 3

4 examples of ths approach are the basc ndcator approach and the standardzed approach specfed for banks under Basel II. 1) Basc ndcator approach The basc ndcator approach s the smplest method recommended by the Basel II commttee to measure operatonal rsk for banks. Banks usng ths approach need to hold operatonal rsk captal equal to the average postve annual gross ncome over the prevous three years scaled by a fxed percentage set by the regulator. Formula to calculate the OpRsk captal charge under basc ndcator approach s gven n (3.1), where GI + s the postve annual gross ncome for the th year and α s the fxed scalar, set by the regulator. Currently α s taken as 15 per cent. 3 + α GI Captal Charge = = 1 (3.1) 3 2) Standardzed approach Standardze approach for banks under Basel II requre them to map ther gross ncome nto eght busness lnes; corporate fnance, tradng and sales, retal bankng, commercal bankng, payment and settlement, agency servces, asset management, and retal brokerage. Captal charge s then calculated by scalng the annual gross ncome for each busness lne by a fxed percentage specfed by the regulator and aggregatng them across. Formula for calculatng the captal charge s gven n (3.2) 3 3 max ( β GI, j),0 j= 1 = 1 Captal Charge = (3.2) 3 where, GI, j s the gross ncome for the th year on j th busness lne. β s the scalar, set by the regulator for each of the busness area. Currently values for β range from 12 per cent to 18 per cent dependng on the rsk nherent n the busness lne. 4

5 Both basc ndcator approach and the standardzed used by banks can be easly mplemented n a super fund by ether takng gross ncome or funds under management as the busness scalar for equatons (3.1) and (3.2). Man advantage of scalars s that t s a low cost method to measure OpRsk. However the man dsadvantage s that the captal charge s not drectly lnked to the loss data, operatonal process or the control process. Thus the accuracy of the model s questonable. Furthermore model does not provde any nformaton on the types of operatonal rsk events, ther rsk profles and how to control them. In addton these models do not attempt to quantfy the low frequency-hgh severty (LF/HS) events whch mpose the hghest threat to the solvency of the frm Regresson and Trend Analyss The models based on regresson and trend analyss attempt to dentfy the key rsk ndcators (KRI) (e.g. audt ratngs, employee turnover, transacton volume etc.) that drves the operatonal rsk and then use these KRI to montor and measure OpRsk. The objectve s to obtan a functon of the form (3.3) and estmate constants α and β by regressng aganst hstorc losses. Operatonal loss = α + β f( KRI ) (3.3) There are several benefts n use of KRIs as an nput to measure OpRsk. Snce KRI are drectly lnked to the operatonal process, model gves the lne mangers behavoral ncentve to keep the KRIs at a desred level n order to manage the OpRsk. Furthermore, changes n the control envronment are usually reflected much quckly n the KRIs makng the output of the model forward-lookng compared to other types of models that solely depend on hstorc data. Therefore these types of models can be useful to montor OpRsk and provde early warnng sgnals to the management. A man drawback of the models based on regresson and trend analyss s the dffculty n fndng a functon of the form (3.3) that would clearly explan the relatonshp between operatonal losses and KRIs. Furthermore, these types of models are not effcent n measurng LF/HS rsk as regresson technques requre large amount of data. Therefore they are more approprate to use at a busness unt level to montor/manage hgh frequency operatonal rsk or 5

6 use as a technque to allocate rsk captal among busness unts that has been determned by other means (Ceske et al., 2000) Fnancal Statement Models Fnancal statement models assume operatonal rsk has an nfluence on the fnancal data such as stock returns, earnngs, expenses, proftablty etc. Frst step n the modelng process s to dentfy a target varable whch s hghly nfluenced by the operatonal rsk. Then the target varable s modeled usng external rsk factors whch drve the target varable. Operatonal rsk s measured as the varance n the target varable that s unexplaned by the external rsk factors. An example of ths approach s CAPM-based models (Hoffman, 2002, Ceske et al., 2000). CAPM-based models assume OpRsk s the dfferental between rsk measured by CAPM and the rsk measured separately by the credt and market rsk models. Another example of ths approach s the Mult-factor stock return model (Saunders et al., 2004). Ths model specfes the stock return can be modeled usng a smlar formula as n (3.4) R = α + β f( I ) + ε (3.4) where, R s the stock return, f ( I ) are functons of the external rsk factors I such as ASX200, CPI etc, α, β are constants and ε s the resdual term. Operatonal rsk s measured as the varance of the resdual term ( σ ). More examples of fnancal statement models can be found n Saunders et al. (2004) and Ceske (2000). 2 ε The man advantage of fnancal statement models s that they are easy to mplement and nputs are readly avalable. They look at the frm-wde vew, thus sutable for determnng the total OpRsk captal charge for a frm. However due to ther frm-wde vew they cannot be employed to determne the OpRsk captal allocaton among busness unts. Another short comng of ths approach s that not all OpRsk are senstve for external rsk factors. It s possble that some mportant OpRsk such as fraud can be overlooked n such nstances. Another drawback of the method s that they are neffcent n measurng LF/HS rsk events 6

7 snce model do not perform well when contnuty of the fnancal data has been dsrupted by a large scale event such as catastrophc loss, mergers and acqustons etc. (Saunders et al., 2004). 3.2 Bottom-Up Models Expected Loss Models Expected loss models attempt to project future expected operatonal losses by usng nsttuton s nternal loss data as a key nput to measure operatonal rsk. Thus n contrast to scalars and benchmarks, captal charge s drectly lnked to the nsttuton s loss dstrbuton. These models assume unexpected losses (the tal of the loss dstrbuton) can be extrapolated usng expected losses (the mean of the loss dstrbuton). A well known example of ths approach s the nternal measurement approach (IMA), one of the methods prescrbed under Basel II advanced measurement approach (AMA) 1. Key steps n IMA can be dentfed as follows 1) Identfy the key rsk events n a frm and categorze them on a matrx by busness lne and event type. 2) Expected loss (EL) for each rsk cell n the matrx s calculated usng frm s nternal data. 3) Extrapolate the unexpected losses (UL) usng EL. To acheve ths IMA assumes there s a lnear relatonshp between UL and EL and the rato between UL and EL s taken as gamma ( γ ). Gamma s estmated by developng an ndustry wde operatonal loss dstrbuton and takng the rato of ndustry EL to a hgh percentle of the ndustry loss dstrbuton, say 99%. (see fg 3.1). The product of EL for each rsk cell and gamma wll gve the maxmum amount of loss per holdng perod. However the gamma for each frm wll be dfferent due to reasons such as operatonal process, busness & control envronment, etc. For example frms wth weak OpRsk control process wll tend to have a long tal operatonal loss dstrbuton. Thus ther gamma would be greater than 1 Methods recommended under Basel II advanced measurement approach ncludes: Internal Measurement Approach (IMA), Scenaro Based Advanced Measurement Approach (sbama), Rsk Drvers and Controls Approach (RDCA) and Loss Dstrbuton Approach (LDA) 7

8 the ndustry gamma. Converse wll be true for frms wth good OpRsk management. In order to account for the dfference n the shape between frm s loss dstrbuton and ndustry s loss dstrbutons, the product of EL and gamma s multpled by a rsk profle ndex (RPI) (see fg 3.1). Formula to calculate the OpRsk charge for IMA s gven n (3.5), where (,j) combnaton represent each rsk cell. ( γ, j, j EL, j) Captal Charge = RPI (3.5) all j all Fgure 3.1: Estmaton of Gamma and RPI Source: Mor & Harada (2001) 8

9 It should be noted that the IMA does not take nto account the rsk dversfcaton between rsk cells. IMA smply aggregate the captal charges for each rsk cell to obtan the total captal charge for the frm. It s possble to develop an expected loss model whch takes nto account the rsk dversfcaton. However accountng for correlatons between the rsk cells s qute dffcult, especally when there sn t enough data. The man advantage of the expected loss model approach s that t drectly lnks the frm s loss experence nto OpRsk captal calculatons oppose to the top-down models dscussed earler. Furthermore method s not as data ntensve as compared to other models that use hstorcal loss data such as LDA (dscussed below). In absence of suffcent hstorc data for partcular rsk cell t s possble to make subjectve estmates of the EL usng expert opnon. However, the man drawback of ths method s that they do not attempt to assess the tal of the loss dstrbuton drectly. Assumpton of a stable relatonshp between unexpected losses and the expected losses need to be tested more thoroughly before t s beng used. In addton model s backward-lookng as t solely depends on the hstorc data to model OpRsk. Hstory does not necessarly have to reflect the future; especally when there have been consderable changes n the busness and control processes. However ths ssue can be easly overcome by adjustng the estmated EL usng expert opnon to reflect the changes n the busness and control envronment. Another crtcsm of the method s that t does not provde much nformaton regardng how to manage OpRsk snce captal charge estmaton procedure s not drectly lnked to the busness and control process Loss Dstrbuton Approach (Actuaral Approach) Loss dstrbuton approach (LDA) s a borrowed technque from actuaral ndustry whch has beng used to model nsurance losses for many years. Smlar to IMA, LDA categorze the rsk events on a matrx by busness lne and event type. But rather than compute the expected losses, LDA estmates two separates dstrbutons for frequency of losses and severty of losses for each rsk cell usng nternal data. Then usng the model gven n (3.6), where L s the loss for a gven rsk cell, N s the number of losses (frequency) and X k s the severty of losses, 9

10 frequency and severty dstrbutons are combned usng convoluton technque such as Monte- Carlo smulaton to obtan a total loss dstrbuton over a holdng perod L N = X k = 1 Then VaR for each rsk cell s calculated and they are aggregated across to obtan the captal charge. Captal Charge VaR ( α) k = j j (3.6) In theory LDA s able to provde superor results than the expected loss models descrbed earler snce t makes full use of the nternal data to drectly measure the unexpected losses. However many research ncludng Moscadell (2004), (Evans et al., 2007) have demonstrated that due to hghly skewed nature of operatonal loss data, conventonal frequency and severty models used n LDA are unable provde adequate results n descrbng the loss data; especally n the extreme percentles. A further short comng of the LDA s that t does not take nto account rsk dversfcaton when calculatng the captal charge. One can argue that most OpRsk categores are not correlated thus addng up captal charges for each rsk cell to obtan the total captal charge s hghly conservatve. Another ssue wth the LDA s the ntense need for data. In practce t s dffcult to fnd adequate nternal loss data for each busness lne and event type combnaton. A further crtcsm of LDA s that t s backward-lookng as method reles on hstorc data as the prmary nput. Thus LDA s not sutable to be used at nstances when sgnfcant changes have occurred n the busness or control envronment. In addton LDA s not drectly lnked to the busness process or the control envronment thus behavoral ncentve provded by LDA s farly lmted Improvements to LDA Many mprovements for the LDA approach have been proposed by varous authors to address the aforementoned ssues. One method that has been put forward to model the tal of the dstrbuton s to use the extreme value theory (EVT). Moscadell (2004) demonstrates usng peak-over-threshold method (POT) to model the tal of the loss dstrbuton provdes sgnfcantly better ft to the operatonal loss data n the extreme percentles. Smlar results have been reported by Evans et al (2007). Further dscusson of the use of EVT model for 10

11 extreme OpRsk losses can be found n Embrechts et al (2004), Chavez-Demouln et al (2006) and Chavez-Demouln & Embrechts (2004). Though EVT provdes a method to model extreme losses, scarcty of data makes t dffcult to estmate the parameters of the model wth suffcent confdence level. Basel II recommends combnng ndustry data and expert opnon n order to supplement scarce nternal data. Wüthrch et al (2007) propose an elegant soluton to combne dfferent data sources by usng Bayesan nference. They estmate the pror dstrbuton for frequency and severty by usng ndustry data and then pror dstrbutons are weghted by the actual observatons and expert opnon to obtan the posteror dstrbutons. These posteror dstrbutons are then convoluted to obtan the aggregate loss dstrbuton Process Approach Models based on the process approach focus on the actual operatonal process of the frm to dentfy and measure rsk, thereby provdng behavoral ncentve for better rsk management. There are many dfferent types of models that can be classfed under process approach. Some of them nclude: () Rsk Drvers and Control Approach (RDCA) RDCA approach whch s also known as scorecards s one of the alternatve methods specfed by Basel II under AMA regme (Basel, 2001). Ths approach heavly reles on control self assessment (CSA) technques to dentfy the prncpal drvers and controls of OpRsk. Followng s the outlne of the basc steps of the RDCA approach. 1) Collectng raw data: Managers are gven questonnares whch consst of seres of weghted, rsk-based questons. These questons are desgn to collect nformaton on organzaton s unque rsk drvers, controls and managers expert opnon. Answers gven by the managers are then analyzed to dentfy the statstcs that are most relevant n explanng the rsk exposure of the busness process. 11

12 2) Buldng KRIs: Second step s to buld KRIs usng the raw data that would properly reflect the OpRsk n the organzaton or the busness unt. Buldng meanngful KRIs s a challenge as we would lke to keep the number of KRIs that we have to montor as lttle as possble n order to mnmze the complexty of the model. Ths requres takng nto account the correlaton between statstcs when desgnng the KRIs. Also we would want our KRIs to be forward lookng so they could serve as an early warnng system. Furthermore, KRIs should be easy to measure and montor. More on how to desgn KRI can be found n Scandzzo (2005). 3) Desgnng scorecards: scorecard s a frm s self-assessment of rsk event. Scorecard would typcally contan the defnton of the rsk event, lkelhood of event, mpact f event occurs, types of controls, etc. A score s gven to each scorecard card by takng approprate weghts of the KRIs. Weghts are usually decded by regresson analyss of hstorc data. In the absence of past data, one may use expert opnon and valdate the weghts once the data become avalable. Once the scorecards are created, they can be used as a montorng tool for OpRsk and evaluaton tool for new control measure. 4) Determnng captal charge: scorecards can be used to determne the captal charge by smulatng rsks and controls. Blunden (2003) descrbes three possbltes for such smulatons; ) smulate the controls frst and f controls fals, then smulate the rsk; ) smulate the rsk frst and, f a rsk happens, then smulate the controls; ) smulate both rsk and controls together. Blunden (2003) states that whch method to choose wll depend on the data avalablty, effcency of smulaton, capablty of the technology etc. The RDCA approach has the advantage of nvolvng the lne managers n the modelng process. Method s more transparent to managers as rsk exposures are measured usng statstcs they are famlar wth whereby provdng behavoral ncentves at the front lne. It s possble to compare the performance of managers by comparng the scores of the busness unts they are responsble for. Management practce of the unt wth the best scores can be adopted by other managers promptng best practce wthn the organzaton. 12

13 Another advantage of the RDCA approach s the model output s more forward-lookng compared to methods prmarly rely on hstorc data. An undesred level of score can serve as a red flag that sends escalaton warnng to the senor management. Thus RDCA approach not only provdes means to quantfy OpRsk but also to montor and manage OpRsk. Some OpRsk experts beleve f NAB had an OpRsk management system whch uses KRIs they could have avoded the A$360 mllon operatonal loss experenced n the early 2004 (Aduse- Poku, 2005). However t should be noted that badly desgned scorecards or wrong expert opnon can provde terrbly wrong predctons. Mark Lawrence (2000) at ANZ Bank states Relyng completely on scorecards could be compared to drvng wthout lookng n the rear-vew mrror. Thus t s necessary to valdate the model usng hstorc data. Anders & Brnk (2004) propose four technques to valdate the model outputs. They are a) Organzatonal nformaton: Qualty of nputs s ensured by evaluatng expert opnon by a thrd person or nternal audt. Independent oversght wll ensure consstency across buldng questonnares, workshops procedure etc. b) Fnancal nformaton: A comparson of the sum of estmated OpRsk cost for each scenaro wth the past operatonal costs recorded n the fnancal accounts can valdate the model output. c) Key rsk ndcators: assesses whether past development of KRI are consstent wth the scenaro assessments. d) Psychometrc analyss: use psychometrc tools to analyze the qualty of data collecton methods. A major drawback of the RDCA approach s that t cannot be used effcently n quantfyng LF/HS OpRsk. As dscussed earler, key nputs of the RDCA s obtaned from managers through questonnares, workshops etc. Usually managers have lttle or no experence n LF/HS events and therefore are unable to provde good enough estmates of such events. Thus t s necessary take nto account LF/HS events by other means such as scenaro analyss. 13

14 () Delta-EVT Delta-EVT s a method developed by Kng (2001) to quantfy OpRsk. Method measures operatonal rsk as the uncertanty n earnngs due to two types of Operatonal losses. Frst are the hgh probablty-low severty losses that can be mapped to causal factors. The second are the rare extreme losses that cannot be mapped to causal factors (e.g. control breakdowns). Kng (2001) proposes to model the low severty losses usng Delta method and extreme losses usng EVT. The Delta method s a technque whch s based on the law of error propagaton; a theory well known to anyone who has taken a Physcs or Chemstry lab course n ther undergraduate years. The delta method attempts to quantfy the aggregate uncertanty n profts propagated from uncertantes n each OpRsk factors. For example consder the smple case where lablty of a superannuaton fund wth m nvestment optons calculated usng the formula (3.7) m m ( ) ( ) Lablty (L) = L = total number of unts TU unt prce UP (3.7) = 1 = 1 Accordng to formula (3.7) an operatonal loss due to msprcng of lablty can happen due to two reasons: a unt prcng error or an accountng error n the total number of unts. Thus takng unt prcng errors and accountng errors as the rsk factors, we can wrte the formula for volatlty of lablty due to OpRsk n th nvestment usng the delta method as followng L L δl = δup + δtu δ prefx represent the error term UP TU ( ) ( ) ( ) ( TU ) ( δup ) ( UP ) ( δtu ) = + ( TU ) σ ( UP ) UP σtu = + Then assumng unt prcng and accountng methods used to value each nvestment methods are smlar, n other words assumng perfect correlaton among lablty valuaton errors for each nvestment opton, total error n valuaton due to OpRsk s 14

15 ( δl) 2 m = δl = 1 m = + = 1 2 ( TU ) σ ( UP ) UP σtu 2 The Delta method only measures the operatonal rsk that can be related to causal factors. Most of the catastrophc losses and control breakdowns are not related to causal factors. Thus Kng (2001) suggest the use of EVT to quantfy such rsk due to rare events. He proposes to obtan the maxmum operatng loss due to causal factors usng Delta method and set t as a threshold to flter the large losses. Then use EVT to model the excess losses. Delta-EVT method has many advantages. Delta method employed to quantfy hgh frequency events s based on the classc error propagaton law whch s an ISO standard for measurng uncertanty (ISO, 1993). Delta method s farly forward-lookng as key nputs uncertantes of the rsk factors are very much senstve for the operatonal exposure and the changes n the control envronment. For nstance, n the above example any changes n the valuaton models used for the purpose of unt prcng s reflected straghtaway fromσ 2 UP. Furthermore, even when hstorc data s not avalable Delta method can be employed usng expert judgment to estmate standard devatons. Coupled wth EVT to measure hgh percentle losses Delta-EVT method provdes and elegant soluton to quantfy OpRsk. However there are few drawbacks n the methodology as well. Frstly not all operatonal losses can be explaned usng determnstc functons (e.g. fraud). Furthermore, delta method assumes uncertantes assocated wth rsk factors have normal dstrbutons and they are small enough so that the propagated errors can be estmated usng frst order approxmaton. Another ssue s the dffculty n dentfyng all the relevant rsk factors and modelng ther nteractons. Thus t s necessary to verfy the loss dstrbutons wth the past experence whenever possble. 15

16 () Causal Models Bayesan Belef Networks (BBN) Bayesan belef networks are a type of causal model whch employ Bayesan probablty theory to model cause and effect n a system. In contrast to Detla-EVT method descrbed earler where OpRsk events are modeled usng lst of determnstc relatonshps, BBNs allow us to model events where casual events exst but, determnstc relatonshps cannot be obtaned (Mast et al., 1999). Applcatons of BBNs can be found n Nuclear ndustry (Santoso et al., 1999), medcal dagnoss (Nkovsk, 2000), data mnng (Heckerman, 1997), ntellgent trouble shootng systems (Nkovsk, 2000, Heckerman & Breese, 1996), and Avaton falure dagnoss (Mast et al., 1999). A BBN s a set of varables called nodes whch are connected by arrows (a.k.a. drected edges, or arcs) representng the dependences among the nodes such that there are no drected cycles. In other words BBNs are Drected Acyclc Graphs (DAGs). Frst step n desgnng a BBN s to dentfy the varables that mpact the operatonal losses and connect them accordng to dependences. For example consder the smple BBN network gven n fgure 2 whch models the losses due to erroneous beneft payments n a superannuaton fund. Model assumes total losses due to erroneous beneft payments depend on the number of erroneous payments and the exposure whch s measured by the medan beneft sze. Furthermore, number of erroneous beneft payments s depended on the number of benefts pad (volume) and the level of tranng of the staff who handles the beneft payments. Once the BBN has been setup next step would be to estmate the condtonal probablty dstrbutons for each state of the nodes such that each node A n the BBN wth parents P1, P2,..., P n has a probablty table Pr( A P1, P2,..., P n) attached to them. Usually BBN s parameterzed before operatonal data s observed thus probabltes for each node s estmated by subjectve opnon. As the data become avalable t s possble to update the network n order to reflect the new experence. There are many algorthms to update a BBN gven new nformaton s avalable for a set of nodes. As cted n Aduse-Poku (2005) some of them nclude; poly-tree algorthm (Pearl, 1988), the clque-tree (Laurtzen & Spegelhalter, 1988) and juncton-tree algorthms (Cowell et al., 1999). 16

17 Once the BBN has been bult t can be used to measure, montor and manage OpRsk. For nstance n the example gven n fgure 2 we fnd the expected loss from erroneous beneft payments s( 50k k k k ) = An advantage of usng a BBN to model OpRsk s that t allows management to dynamcally observe the changes to the loss dstrbuton wth respect to changes n the busness and control envronment. In the gven example f the management decded to gve all the staff a level 2 tranng then the expected loss wll reduce down to Ablty to smulate the mpact of manageral decsons allows management to carry out cost-beneft analyss to asses ther strateges. Ths type of analyss can also be used for stress testng. However the man dsadvantage of BBNs s the complexty nvolved n desgnng such a model. Many frms would fnd there sn t smply enough tme and money avalable to buld a BBN n an enterprse-wde bass. Thus t s more feasble to use BBNs to quantfy and mange operatonal rsk n partcular busness unts wth large OpRsk. Fgure 2: Smple BBN for Losses Due To Erroneous Beneft Payments 17

18 (v) Other Models usng Process Approach There are many other models based on the prncpals of process approach. Some of them nclude Relablty models, Connectvty models, System dynamcs, and Neural networks. Relablty models have been n use for many years n engneerng to measure and mtgate OpRsk n power plants, nuclear reactors etc. They model the tme between OpRsk events rather than ther frequency (Saunders et al., 2004). Thus these models may become useful n measurng partcular operatonal rsk (e.g. IT falure) at busness unt level oppose to organzatonal level. System dynamcs approach s another casual model that has been proposed to measure OpRsk. A dscusson contrbuted by Jerry Mccols and Samr Shah of the use of system dynamc approach at Tllnghast/ Tower Perrn can be found n Hoffman (2002). Another promsng method n the development stage s the neural networks. Perera (2000) clams that the neural networks developed at NASA to analyze relablty of mcroelectromechancal systems (MEMS) can be adapted to measure operatonal rsk for fnancal nsttutons. Due to complexty of these models t s hghly unlkely they wll become popular n the near future. However they wll at least be useful to model partcular complex operatonal rsk at busness unts. 3.3 So whch method to choose? As dscussed above there many methods avalable to model known-known OpRsk. Whch method to choose wll depend on the factors such as problem n hand, data avalablty, tme and budget constrants, etc. Top-down methods such as scalars, fnancal statement models and trend analyss s sutable for small super funds that do not have the capablty to carry out sophstcated modelng. On the other hand large frms may fnd bottom-up methods such as LDA, casual modelng more attractve as they wll allow the frm to gan better understandng of frm s OpRsk. Among the bottom-up methods avalable, causal models provdes superor results n terms of accuracy, forward lookng captal estmates, behavoral mpact and early warnng sgnals. However tme and cost nvolve n settng up such a model can be qute sgnfcant. Attractve alternatve for casual modelng s the RDCA (scorecards). Use of KRI n the RDCA approach makes RDCA some what forward lookng. If properly set up, RDCA can provde valuable nformaton to OpRsk managers on how to manage rsk and even early 18

19 warnng sgnals. Accordng to Dr. Mark Lawrence, the chef rsk offcer at ANZ bank of Australa, after consderng many dfferent methods to model OpRsk n the bank, ANZ fnally settle down to usng RDCA rather than causal models due to budget and tme constrants(lawrence, 2000). Fgure 3 llustrate how ANZ ranked the dfferent technques n terms of tme/cost, accuracy and behavoral mpact. Though casual models are too costly to mplement at organzatonal level, one should not rule them out completely. Causal models could be qute useful to measure partcular hgh rsk OpRsk at busness unt level. For example, gven most of the recent large scale losses such as Sumtomo, Barng, Alled Irsh, NAB etc. were due to fraudulent tradng actvtes, tme and cost nvolve n buldng a causal model may not be unreasonable to measure and montor such OpRsk. Fgure 3: ANZ Rankng of Dfferent OpRsk Measurement Methods n Terms of Tme/Cost, Accuracy and Behavoral Impact Source: Lawrence (2000) 19

20 4. Assessng Known-Unknown OpRsk In order to model rsk that we know exst but cannot model (e.g. legal rsk) we propose a methodology based on the solvency II loss-gven-default approach whch we call Black Swan approach. Under Solvency II European nsurers need to stress test the survvablty of the frm under large catastrophc ndustry losses. Ths method s called the loss-gven-default approach. The method smply looks at whether nsurer has enough captal to cover ther exposures under a gven set of catastrophc ndustry losses (e.g. European wndstorm causng $4bll ndustry loss). Method makes no attempt to quantfy the frequency of losses. The Black Swan approach outlned below s a slghtly modfed verson of the loss-gven-default approach such that t wll be sutable to measure OpRsk. The basc steps of the Black Swan approach are as follows 1) Identfy the types OpRsk classes that we cannot model usng conventonal models. 2) Obtan ndustry loss data for each rsk class and make correctons for nherent bas. 3) Use the adjusted data to fnd the maxmum operatonal loss experenced by smlar organzatons n the ndustry under each OpRsk class. 4) Survvablty of the frm s stress tested aganst the maxmum loss. If the frm cannot survve, then necessary captal or rsk management practce s put forward. 4.1 Correctng for Inherent Bas Most mportant step n the Black Swan approach s to correct for nherent bas n the data snce data that has not been corrected for bas can yeld perverse captal estmates. Accordng to APRA (2007b) there are manly three types of bas whch external data s affected from ) Reportng bas occurs when dfferent threshold has been used by nsttutons to report losses ) Control bas occurs when data s collected from nsttutons wth dfferent control systems ) Scale bas occurs when data s collected from nsttutons wth dfferent szes 20

21 There are many technques suggested by varous authors on how to correct nherent bas. De Fontnouvelle et al. (2003) proposed usng a stochastc truncaton model to correct for reportng bas where they treat each nsttuton s loss reportng threshold as an unobserved random varable. Dahen & Donn (2007) and Na et al. (2005) provdes an elegant method to remove scale bas by assumng operatonal losses can be broken nto two components: a component that s common for all frms n the ndustry and an dosyncratc component specfc to each frm. They assume dosyncratc component depends on factors such as frm sze, locaton, busness lnes etc. They estmate the nfluence of each factor usng regresson analyss. Correctng for control bas s much harder snce t s dffcult to obtan nformaton about the control mechansm of the nsttuton whch data s comng from due to dsclosure constrants n the external databases. Thus the modeler needs to select data usng hs own opnon on how relevant the data s to hs company gven the dfference n standard rsk management practce n the ndustry and hs own company. A detal dscusson on recent advances n correctng nherent bas for external data sources can be found n APRA (2007b). Man advantage of the Black Swan approach s that t s smple to mplement and the nputs are readly avalable. Unlke most of the methods dscussed earler, black swan approach focuses on the LF/HS events, rsk that posses greatest threat to the solvency of the company. One major drawback of the method s that t does not take nto account the probablty of the losses. Thus method may lead to conservatve captal estmates. 5. Rsk Margn for Unknown-Unknown OpRsk Unknown-unknown OpRsk s the OpRsk that the frm s exposed to but s unaware of. The only sensble way to account for these types of rsk s to add a rsk margn above the captal charge calculated under known-known and known-unknown OpRsk. Benchmarkng s one of the easest ways to determne the approprate rsk margn. The method smply looks at the OpRsk captal held by smlar frms n the ndustry to manage ther unknown-unknown rsk and benchmark aganst t. Ths s a proxy measure rather than a real quantfcaton. Therefore t s necessary to use expert judgment when benchmarkng aganst 21

22 each other so that the nature of the frm s busness envronment and control process s taken nto account when decdng the rsk margn. 6. Dscusson In ths paper we have ntroduced a consstent framework to measure operatonal rsk such that all three categores of rsk: known-known, known-unknown and unknown-unknown rsks are taken nto account when determnng the captal charge. We look nto dfferent technques avalable for super funds to measure ther OpRsk by drawng technques used by varous ndustres to measure ther OpRsk. We report that there are many methods avalable to measure known-known rsks. Choce of the method wll depend on the factors such as problem n hand, data avalablty and tme & budget constrants. In summery most of the top-down models avalable to measure known-known rsks are easy to mplement, low cost and sutable when n need of quck answers. However the accuracy of these models are questonable. On the other hand bottom-up approaches are able to provde more accurate captal estmates as they make full use of the nternal loss data and n some cases even external data and expert opnon. Causal models such as Bayesan networks seems to provde best results n terms of accuracy, but however, due to tme and cost nvolve n settng up a causal model most frms mght fnd these models less feasble to mplement. Therefore they mght be useful to measure OpRsk n hgh rsk busness unts rather than n a frm-wde level. On the other hand RDCA seems to provde a good trade-off between performance and tme/cost. If properly set up RDCA can provde farly forward-lookng estmates of OpRsk captal charge, behavoral ncentves and early warnng sgns to managers. In order to asses the rsk due to known-unknown OpRsk we propose to stress test usng the black swan approach, whch s a modfed verson of the loss-gven-default approach. Unknown-unknown rsk s taken nto account by addng a rsk margn above the captal estmates obtan under known-known rsk and known-unknown rsk. 22

23 References: Aduse-Poku, K. (2005) "Operatonal Rsk management - Implementng a Bayesan Network for Foregn Exchange and Money Market Settlement". - A thess submtted n fulfllment of the requrements for the degree of Doctor of Phlosophy (PhD): Faculty of Economcs and Busness Admnstraton, Unversty of Göttngen Germany. Anders, U. and Brnk, G. J. V. D. (2004) "Implementng a Basel II Scenaro-Based AMA for Operatonal Rsk". In The Basel handbook : a gude for fnancal practtoners. ed. Mchael K. Ong. London: Rsk. Australan Prudental Regulatory Authorty (2007a) "Quarterly Superannuaton Performance for September 2007 ". Australan Prudental Regulaton Authorty (2007b) "A Revew of Correcton Technques for Inherent Bases n External Operatonal Rsk Loss Data". Workng Paper Seres Basel Commttee on Bankng Supervson (2001) "Workng Paper on the Regulatory Treatment of Operatonal Rsk". Blunden, T. (2003) "Scorecard Approaches". In Operatonal rsk, regulaton, analyss and management. ed. C. Alexander. London: Pearson Educaton. Ceske, R., Ernández, J. V. H. and Sánchez, L. M. (2000) "Quantfyng Event Rsk: The Next Convergence". The Journal of Rsk Fnance. Chavez-Demouln, V. and Embrechts, P. (2004) "Advanced Extremal Models for Operatonal Rsk". Avalable: (Accessed: Aug 27, 2007 Chavez-Demouln, V., Embrechts, P. and Neslehova, J. (2006) "Quanttatve models for operatonal rsk: Extremes, dependence and aggregaton". Journal of Bankng & Fnance. 30 (10): Cowell, R. G., Dawd, P. A., Laurtzen, S. L. and Spegelhalter, D. J. (1999) "Probablstc networks and expert systems". New York: Sprnger. Dahen, H. and Donne, G. (2007) "Scalng Models for the Severty and Frequency of External Operatonal Loss Data". In Workng Paper Montréal, Canada Centre nterunverstare sur le rsque, les poltques économques et l emplo (Inter-Unversty Center on RIsk, Economc Polces and Employment). De Fontnouvelle, P., De Jesus-Rueff, V., Jordan, J. S. and Rosengren, E. S. (2003) "Usng Loss Data to Quantfy Operatonal Rsk". In Workng Paper Seres. Department of Supervson and Regulaton, Federal Reserve Bank of Boston. Embrechts, P., Kaufmann, R. and Samorodntsky, G. (Eds.) (2004) "Run theory revsted: stochastc models for operatonal rsk". Frankfurt: European Central Bank. Evans, J., Womersley, R. and Wong, D. (2007) "Operatonal Rsks n Banks: An Analyss of Emprcal Data From An Australan Bank". In Bennal Conventon Chrstchurch, New Zealand: Insttute of Actuares of Australa. Heckerman, D. (1997) "Bayesan Networks for Data Mnng". Data Mnng and Knowledge Dscovery. 1 (1): Heckerman, D. and Breese, J. S. (1996) "Causal ndependence for probablty assessment and nference usng Bayesan networks". Systems, Man and Cybernetcs, Part A, IEEE Transactons on. 26 (6): Hoffman, D. (2002) "Managng operatonal rsk : 20 frmwde best practce strateges". New York: Wley. 23

24 Internatonal Organzaton for Standardzaton. (1993) "Gude to the expresson of uncertanty n measurement". [Geneva]: Internatonal Organzaton for Standardzaton. Kng, J. L. (2001) "Operatonal Rsk : Measurement and Modellng". New York: Wley. Laurtzen, S. L. and Spegelhalter, D. J. (1988) "Local computatons wth probabltes on graphcal structures and ther applcaton to expert systems". Journal of the Royal Statstcal Socety. 50 (2): Lawrence, M. "Markng the Cards at ANZ".artcle orgnally appeared n the November 2000 ssue of Rsk magazne: Operatonal Rsk Specal Report. Mast, T. A., Reed, A. T., Yurkovch, S., Ashby, M. and Adbhatla, S. (1999) "Bayesan Belef Networks for Fault Identfcaton n Arcraft Gas Turbne Engnes". IEEE Internatonal Conference on Control Applcaton, Kohala Coast, HI, USA, 08/22/ /27/ : Mor, T. and Harada, E. (2001) "Internal Measurement Approach to Operatonal Rsk Captal Charge (Dscusson Paper)". In. Bank of Japan. Moscadell, M. (2004) "The modellng of operatonal rsk: experence wth the analyss of the data collected by the Basel Commttee". In Tem d dscussone (Economc workng papers) Bank of Italy, Economc Research Department. Na, H. S., Mranda, L. C., Berg, J. v. d. and Lepoldt, M. (2005) "Data Scalng for Operatonal Rsk Modellng". In ERIM Report Seres - ERS LIS. Erasmus Research Insttute of Management (ERIM). Nkovsk, D. (2000) "Constructng Bayesan networks for medcal dagnoss from ncomplete and partally correct statstcs". Transactons on Knowledge and Data Engneerng. 12 (4): Pearl, J. (1988) "Probablstc reasonng n ntellgent systems : networks of plausble nference". San Mateo, Calf.: Morgan Kaufmann Publshers. Perera, J. "Neural Networks Do All The Work".artcle orgnally appeared n the November 2000 ssue of Operatonal Rsk supplement to Rsk magazne. Reserve Bank of Australa (2007) "The Man Types of Fnancal Insttutons n Australa". Santoso, N. I., Darken, C., Povh, G. and Erdmann, J. (1999) "Nuclear plant fault dagnoss usng probablstc reasonng". Power Engneerng Socety Summer Meetng, IEEE. 2: vol.712. Saunders, A., Boudoukh, J. and Allen, L. (2004) "Understandng market, credt, and operatonal rsk : the value at rsk approach". Malden, Mass.: Blackwell. Scandzzo, S. (2005) "Rsk Mappng and Key Rsk Indcators n Operatonal Rsk Management". Economc Notes. 34 (2): The Australan Government Treasury (2001a) "Optons for Improvng the Safety of Superannuaton - Background Paper ". The Australan Government Treasury (2001b) "Optons for Improvng the Safety of Superannuaton - Issues Paper". Wüthrch, M., Lambrgger, D. D. and Shevchenko, P. V. (2007) "The Quantfcaton of Operatonal Rsk Usng Internal Data, Relevant External Data and Expert Opnon". Avalable: (Accessed: Aug 23,