Irving Fisher Committee on Central Bank Statistics. IFC Working Papers. No 11. Optimizing Checking of Statistical Reports. by Peter Askjær Drejer

Transcription

1 Irving Fisher Commiee on Cenral Ban Saisics IFC Woring Papers No 11 Opimizing Checing of Saisical Repors by Peer Asjær Drejer July 2013

2 IFC Woring Papers are wrien by he saff of member insiuions of he Irving Fisher Commiee on Cenral Ban Saisics, and from ime o ime by, or in cooperaion wih, economiss and saisicians from oher insiuions. The views expressed in hem are hose of heir auhors and no necessarily he views of he IFC, is member insiuions or he Ban for Inernaional Selemens. This publicaion is available on he BIS websie ( Ban for Inernaional Selemens All righs reserved. Brief excerps may be reproduced or ranslaed provided he source is saed. ISBN (online)

3 Opimizing Checing of Saisical Repors Concepualized Experiences from a Cenral Ban Peer Asjær Drejer 1 Absrac The aim of his paper is o convey experiences and progress of Danmars Naionalban during he recen years in opimizing he process of checing saisical repors. While he empirical bacground is derived from wor wihin he ineres rae saisics, he paper generalizes and caegorizes our experiences in order o mae hem applicable o oher saisical areas and oher collecion sysems. The paper considers five differen areas for opimizaion: "Checing aggregaes", "Oulier idenificaion", "Worflow", "IT ools" and "Encouraging checs by he reporing eniy". 1 Money and Baning Saisics, Naional Ban of Denmar, Havnegade 5, 1093 Copenhagen, Denmar, ( pad@naionalbanen.d) IFC Woring Paper No 11 - Opimizing Checing of Saisical Repors 1

4 Conens 1. Inroducion A Framewor for Defining a Chec Checing Aggregaes Oulier Idenificaion The IT Applicaion for Human Evaluaion Encouraging Checs by he Reporing Eniy References IFC Woring Paper No 11 - Opimizing Checing of Saisical Repors

5 1. Inroducion 2 Recen years have seen significan developmens in boh echnologies and saisical requiremens. While innovaions wihin IT faciliae compilers' as of checing incoming daa, an increased demand for granular daa and requiremens for more imely daa poses new challenges. In his ligh, i is worhwhile o reassess exising procedures. This paper aims a conveying some experiences and progress of Danmars Naionalban during he recen years in opimizing he checing process. While he empirical bacground is derived from wor wihin he ineres rae saisics, we have ried o generalize and caegorize our experiences in order o mae hem generally applicable o oher areas of saisics and oher sysems. The purpose of he daa checing process is o ensure ha daa repored are wihou subsanial errors and ha he compiler learns abou he sory behind imporan changes in repored figures. In general, he daa cleansing process can be divided ino wo seps: Firs a daa validaion process followed by he process of plausibiliy esing. 3 Daa validaion rules are buil ino he reporing sysem and he daa validaion process will chec wheher he daa pass or fail he validaion rules. This process does no require analyical involvemen of compilers. For his reason, he main focus of opimizaions in his paper will be wihin he plausibiliy esing process. The core of he plausibiliy esing process consiss of idenifying possible errors. Thus, a naural way of opimizing is o address he abiliy o idenify ouliers. A he same ime, however, i is a fac ha plausibiliy esing aes up significan resources a he compiling insiuion and poenially also for he saisical reporer. Hence, he qualiy of he checing process should be measured no jus on how well errors are correced, bu also on he size of he burden i pus on he compilers and reporers. 2 3 The opinions expressed in his paper are hose of he auhor and do no necessarily represen hose of Danmars Naionalban. I han Seen Ejersov, Alexander Khalileev, Tue Mollerup Mahiasen, Miel Kragelund, Andreas Kuchler, Klaus Theill Jensen and Bria Gaarde for discussions and valuable commens and suggesions. I also han he anonymous reviewers for valuable commens. The division ino a daa validaion and a plausibiliy esing follows he erminology of IMF (2008), p Noice ha his paper does no apply he subsequen division of he plausibiliy esing ino hree phases. IFC Woring Paper No 11 - Opimizing Checing of Saisical Repors 3

6 Char 1 Opimizaion areas in relaion o he checing process Area 5: Encouraging checs by reporers Reporer Collecion Colleced daa Valid daa Verified daa Communicaion Failing checs Validaion checs Furher producion Daa validaion Area 3: Worflow Area 4: IT ools Area 2: Oulier idenificaion Communicaion Queries Human evaluaion Plausibiliy esing Ouliers Auomaic oulier idenificaion Area 1: Checing aggregaes Square boxes represen daa. Boxes wih round edges represen processes and boxes wih doed line and round edges represen processes wih human involvemen. This paper describes five areas for opimizaion: Checing aggregaes, Oulier idenificaion, Worflow, IT ools and Encouraging checs by he reporing eniy. The areas and heir relaion o a sylised checing process are depiced in Char 1. Each area will be described in a separae secion. This paper sars ou wih a secion describing a sandardized framewor for defining a chec. While such a framewor a firs may seem superfluous, i will be useful for undersanding he subsequen secions. In auumn 2011 a worshop was held, focussing on hese five areas, wih he paricipaion of cenral bans and oher compilers of baning saisics from around he world. As a follow-up, a survey was conduced on curren pracices and opimizaion effors (see Drejer (2012)). All 30 insiuions paricipaing in he worshop answered he survey quesionnaire. Resuls from his survey will be referred o in his paper. 2 A Framewor for Defining a Chec The daa checing process can be seen as made up of a number of checs ha need o be performed. This secion oulines a framewor for describing any chec performed on saisical repors. When chec definiions become complex, heir implemenaion and ongoing managemen will be grealy faciliaed by such a framewor. In general, i is our experience ha he framewor leads o a clearer discussion of opimizaion issues and faciliaes he dialogue beween daa analyss and echnical saff. The framewor will describe 5 seps ha are needed in order o define a single chec. While hese seps will include noaion for checs on aggregaes (secion 3), hey will sill be 4 IFC Woring Paper No 11 - Opimizing Checing of Saisical Repors

7 applicable o simpler checs wihou any aggregaion. The framewor definiion will be illusraed using an example of checing loan daa repored by a ban (see Char 2). The same example will be coninued under secion 3. Sep 1: Chec Iem A chec will be done o verify a repored figure. This figure may be direcly repored by he reporing eniy or i may be indirecly repored, aing he form of an aggregae wihin a single repor or beween muliple repors. This figure will be ermed he chec iem: Chec iem = c If he chec is on an aggregaed figure, hen a subcomponen srucure should be defined (see secion 3). Subcompone n = s i S, where c where si denoes each subcomponen and S is he oal amoun of subcomponens. In he example depiced in Char 2, he chec iem is an aggregae being he oal loans repored by he ban. The subcomponens of he chec iem are loans by secor. Anoher example of a chec iem could be on he direcly repored figure for loans o Households wihou defining a subcomponen srucure. i s i Sep 2: Benchmar Daa The nex sep is o define which daa should be used o verify he chec iem. For each chec we will define a vecor of benchmar daa: Benchmar daa = b The benchmar daa can come ino use in all he subsequen seps. I could be used for ransforming he chec iem (see nex sep), i could be used in he oulier idenificaion funcion (sep 4) or i can serve a more indirec purpose as bacground informaion for he human evaluaion (sep 5). In our example, we use he simple benchmar of las period's observaions of he chec iem and subcomponens. IFC Woring Paper No 11 - Opimizing Checing of Saisical Repors 5

8 Char 2 Example of sep 1-3 of a chec definiion Benchmar daa Secor -1 Subcomponens Iem growh Growh conribuion MFIs % -2% Public secor % 2% Households % 0% Non-financial corp % 4% Invesmen funds % 2% Oher financial ins % 15% Oher % 0% Toal loans % Subcomponen conribuions Chec Iem Chec Iem Transformaion Sep 3: Chec Iem Transformaion The nex sep is o define he basis for he oulier idenificaion. This will be ermed he chec iem ransformaion and can be seen as he funcion ha ransforms he chec iem ino a esable measure: Chec iem ransformaion = = f ( c, b) If he chec iem is an aggregae wih subcomponens, we will define subcomponen conribuions o describe how much each subcomponen conribues o he chec iem ransformaion: Subcomponen conribuion of i = sc( i) = SC( c, where SC(*) is he funcion ha defines subcomponen conribuions subjec o he consrain f ( c, b ) = sc i. i Growh rae calculaions and seasonal adjusmens are boh examples of chec iem ransformaions. In our example, we use las period's observaions (our benchmar) o creae a chec iem ransformaion in he form of a growh rae (see Char 2). Accordingly he subcomponen conribuions are calculaed as he growh rae conribuions of each subcomponen. s i, b) Sep 4: Auomaic Oulier Idenificaion The nex sep is o esablish boundaries for deermining which values of he chec iem ransformaion should be reurned as auomaically deeced ouliers. This sep requires a definiion of he mehodology behind he oulier idenificaion. We will denominae he oulier 6 IFC Woring Paper No 11 - Opimizing Checing of Saisical Repors

9 funcion by O (*,b). The inpu o he oulier funcion can boh be he chec iem ransformaion and subcomponen conribuions. In he example of Char 2, we can search for ouliers among chec iem ransformaions O( f (, b ), b). The funcion could reurn he value 1 (oulier) if he oal loan growh, f ( c, b), exceeds +/-15%, and oherwise 0 (no oulier). Anoher possibiliy is o search for ouliers among subcomponens O ( sc i, b). Sep 5: Human Evaluaion The las sep will be he human evaluaion of auomaically idenified ouliers. For each chec, we need o define which daa he analys performing his as should be presened wih. This includes he definiion of error ex messages and graphs and able definiions. Finally i should be defined how he oulier should be presened o he reporer in case he analys decides o mae a query abou he iem. Coninuing our example of checing oal loans, he presenaion depends on wheher in sep 4 we idenified he oulier a iem or subcomponen level. Assuming he laer we could define ha he analys should be presened wih a graph of he hisory of he oulier subcomponen and is conribuion o he oal growh. In addiion here would probably be ex explaining he oulier's origin. IFC Woring Paper No 11 - Opimizing Checing of Saisical Repors 7

10 3. Checing Aggregaes The concep of checing daa a aggregaed levels is now a well-nown phenomenon ha is being increasingly employed in saisical producion. Van den Dool e al. (2008) described a macro-micro approach o compiling saisics ha has been adoped by De Nederlandsche Ban. Among compilers of baning saisics, 87% used some ind of aggregaed checing in 2011 (Drejer (2012)). The example in Char 2 illusraed how a basic chec on an aggregae wors. If our oulier search was based on he monh-o-monh growh of each individual secor iem (he direcly repored daa) we would ge a relaively large number of suspicious iems and we would need o assess he validiy of each oulier. Using aggregaed checing echniques help us focus on imporan developmens ha affec he aggregae. In his insance, here was a relaively large aggregae increase of 20% which can easily be raced bac o he responsible subcomponen (loans o "Oher financial insiuions"). In general, checing of aggregaes can be applied o almos any ind of saisics. However, compared o more radiional checing approaches, i implies some design challenges. This secion is focused on how o define an aggregaed chec. Relaed o his, secion 5 will consider he implicaions o he worflow. 3.1 Type of Aggregaion Aggregaed checing can be performed boh wihin an individual repor and by aggregaing across individual repors. By checing aggregaed figures wihin a repor, our gain is ha we do no have o chec he mos disaggregaed figures repored. This ype of chec is illusraed in Example 1 of Char 3 and will be referred o as a chec wihin aggregaion. On he oher hand, by checing aggregaes across individual repors, our gain is ha we do no have o chec each individual repor, bu only hose conribuing o he aggregae. This ype of chec is illusraed in Example 2 of Char 3 and will be referred o as a chec wih reporer aggregaion. I is possible o mae checs ha have boh wihin and reporer aggregaion. This is illusraed in Example 3 of Char 3. Noice ha here can be inermediae seps of reporer aggregaion, where no all reporers bu only subgroups of reporers are aggregaed. Also noice ha a wihin aggregaion can be boh an explicily repored aggregaed figure and a calculaed aggregaion. 8 IFC Woring Paper No 11 - Opimizing Checing of Saisical Repors

11 Char 3 Differen ypes of aggregaion Example 1 Wihin aggregaion Example 2 Reporer aggregaion Example 3 Combinaion Repor R1 Repor R1 Repor R2 R1 + R2 Repor R1 Repor R2 R1 + R2 (Aggregaed) Chec iem Subcomponens 3.2 Defining Subcomponens and Subcomponen Conribuions In mos insances of aggregaed checing, we wan o be able o brea down he aggregae in order o idenify which iem or which reporer affeced he aggregae. In order o do his efficienly, i is imporan o define which subcomponens mae up he aggregaed figure and define how he conribuion from each of hese subcomponens should be calculaed. When we define subcomponens o a chec iem, in effec wha we are doing is defining o wha deail we subsequenly wan answers. As oulined in he framewor, his measure of how much a subcomponen affeced he aggregae chec iem ransformaion will be ermed he subcomponen conribuion. Noice ha hese subcomponen srucures and conribuions should be viewed concepually and need no necessarily be performed compuaionally for each chec. Depending on he level of checing, one may decide o mae he acual calculaion of subcomponen conribuions condiional o developmens in he chec iem (see 3.3.2) Subcomponens As oulined in he framewor, he subcomponens mae up he chec iem. The se of subcomponens, S, could be defined across muliple dimensions including he reporer dimension. As an example, we firs consider a chec iem wih pure reporer aggregaion, say oal loans o he household secor graned by bans. Here, our chec iem and subcomponens can be defined as: Chec iem = c = i s i [E1] IFC Woring Paper No 11 - Opimizing Checing of Saisical Repors 9

12 where s i is he oal loans o households of ban i I, where I is he oal populaion of bans. We may wan o be able o dig furher ino each ban's oal household loans. This could be done by defining subcomponens ha include loan ypes and mauriy dimensions: s, Chec iem = c s i, l, m i l m = where i, l m is he household loans of loan ype l L, where L is he complee se of loan ypes, in mauriy band caegory m M, where M consiues all mauriy band caegories, of ban i Subcomponen Conribuions Once he subcomponens have been defined, heir conribuions o he chec iem ransformaion should be calculaed. As oulined in sep 3 of he general framewor, his calculaion is done by defining a funcion ha defines each subcomponen's share of he chec iem ransformaion. Coninuing he example above where we were checing he amoun of oal loans we migh have a chec iem ransformaion ha gives he monhly growh rae of he aggregae. If we have he same subcomponen srucure as in E1, he subcomponen conribuions would be defined as each reporer's growh rae conribuion: s, i sc( i) = I [E2] s i= 0 1, i Using Subcomponen Conribuions as Analyical Tools Besides is use in he checing process, he breadown of aggregaed level figures is ofen a useful analyical ool o explain wha drives changes in he overall saisics. An example of his is he ECB's publishing of a breadown of euro area ineres level developmens ino weigh and ineres rae effecs. 3.3 Approaches o Checing Aggregaes Four Ways of Checing Aggregaes When we perform he oulier idenificaion for checs involving aggregaes, we can search boh a he aggregaed chec iem level and a he subcomponen level, corresponding o using eiher he chec iem ransformaions or he subcomponen conribuions as inpu for he oulier idenificaion. By combining he differen ypes of aggregaion and he wo possible levels of oulier idenificaion, we can idenify four caegories of checing based on aggregaes (see Char 4). Noice ha Char 4 only depics pure aggregaion forms. As shown in Char 3, here can also be combinaions of reporer and wihin aggregaion. The erm macro checing seems mainly o have been used for Mehod 1 for cases wih reporer aggregaion and oulier idenificaion a he chec iem level. In many cases, i will be useful o consider he whole palee of possibiliies. 10 IFC Woring Paper No 11 - Opimizing Checing of Saisical Repors

13 Char 4 Four mehods of checing aggregaes Oulier idenificaion a chec iem level Oulier idenificaion a subcomponen level Reporer aggregaion Mehod 1 Mehod 3 Wihin aggregaion Mehod 2 Mehod Choice of Tes Level In he example used above of checing aggregaed household loans across insiuions, we had a chec iem ransformaion being he monhly change and subcomponen conribuions defined as monhly growh rae conribuions (as in E2). In he search for ouliers, we eiher consider aggregaed growh rae, O 1( ), or he growh rae conribuions, O 2 ( sci ). In he firs case, he oulier is found a he aggregaed level and he subcomponen conribuions are used as supplemenary informaion for he subsequen evaluaion of he oulier. In he second case, he chec will direcly idenify ouliers on he subcomponen conribuions. Boh mehods have advanages and disadvanages. An advanage of focusing on he oulier idenificaion on he aggregaed level is ha i helps o prioriize which suspicious subcomponen should be addressed firs. This is in paricular an advanage when considering reporer aggregaes. According o Drejer (2012), 77% of compilers using checs on aggregaes perform he oulier search on he aggregaed chec iem level. In many cases, however, i is worhwhile o consider doing he search a a subcomponen level. I is clear ha he searching of ouliers among subcomponen conribuions poenially leads o a more direc and faser idenificaion which is an advanage in he human evaluaion process. In addiion, his mehod has some imporan advanages when considering he overall worflow (see secion 5 for a discussion of his issue). An imporan difference beween he wo approaches is wheher similar-sized movemens in opposie direcion wihin subcomponens will be idenified as ouliers. When focusing a he aggregae such movemens will cancel ou and no oulier be idenified. On he oher hand, when focusing on he subcomponens, such movemens will be deeced. Wheher or no deecing hese movemens is preferable will depend on he naure of he repored daa. In some insances, par of he raionale behind doing checs a he aggregaed level is ha offseing movemens should indeed no be considered. Consider for example a case where he chec iem is he profi of a firm and we define he subcomponens as he main iems of he profi/loss accoun. Assume ha since las repor, his firm has expanded is operaions and is reporing correcly. If we focus our chec on subcomponen conribuions we will probably ge boh urnover and cos as ouliers. On he oher hand, if we had focused on he IFC Woring Paper No 11 - Opimizing Checing of Saisical Repors 11

14 chec iem (profi), we would have had a more normal developmen and probably no oulier, which in his case probably would be preferable. I is worh noing ha he wo approaches can coexis. We may wan o search for ouliers a subcomponen level because i gives us a direc idenificaion; however, in ligh of he above example, we may wan o mae his search condiional on he aggregaed level chec. This could be implemened using a logic relaion lie (if O ( ) is rue hen O ( sc ) ). 1 2 i Using Differen Levels of Reporer Aggregaion There can be muliple uses of checing a reporer aggregaion level. If he chec iem is based on he inpu of he enire reporing populaion, he purpose of he chec is o address implicaions for he final saisics. Consrucion of chec iems a a lower level, however, can serve o ensure a higher qualiy a he reporer level. Ofen a large par of he reporing populaion will be small reporers. The conribuions of oher reporers o he oal aggregae will probably be negligible and even drasic ouliers a he individual level will no affec he oal. This would be deeced if we run checs on each reporer, bu hen we would have o chec each one. By consrucing subpopulaions of reporers ha normally would be assumed o have conribuions of similar size, he micro level errors could o a much higher degree be deeced in he macro checs. Hence, checs on aggregaions of subpopulaions can be a useful ool ha can help more efficienly o ensure daa qualiy a he reporer level. Anoher applicaion of sublevel reporer aggregaions is when he saisics are divided ino differen sraa and we wan o ensure qualiy a he sraa level Daa Availabiliy One major issue of performing checs on reporer aggregaes is he limiaion ha all repors, ha will conribue o he final aggregae, needs o be available before he checing can sar. As his will no always necessarily be he case, i is preferable o have a procedure by which a preliminary "sand in" repor can be generaed. 3.4 Examples of Checs on Aggregaes Bans' Reporing of Loans and Choice of Subcomponen Srucure To illusrae various definiions of subcomponen srucures, we expand he example from he secion oulining he framewor (see Char 2) and consider a slighly more elaborae subcomponen srucure (see Char 5). In his example, our chec iem is sill oal loans and he new and more deailed subcomponen srucure now includes a breadown of counerpary residence counry. As benchmar daa, we sill use las period's repored loans, and he chec iem ransformaion will be he change in he chec iem. The subcomponen conribuions will again be he growh conribuions. The example shows ha he expanded subcomponen srucure allows us o pin down he abnormal developmen a a more deailed level. While he more deailed subcomponen srucure may generally seem preferable o he simpler one used in Char 2, he preferable level of subcomponens need no always be he lowes. Specifically, when repors are very granulaed, oo much deail in he subcomponen srucure may poenially lead o a huge amoun of calculaions and exend he wor of he analys owards oo much deail. Also, if he evaluaion of ouliers is done in a sandardized IT sysem, i may be preferable ha checs have he same number or a maximum number of subcomponens. 12 IFC Woring Paper No 11 - Opimizing Checing of Saisical Repors

15 Char 5 Examples of expanded subcomponen srucures for wihin aggregaion Secor Counry -1 Subcomponens Iem growh Growh conribuion MFIs Domesic % 0% Foreign % -2% Public secor Domesic % 3% Foreign % -1% Households Domesic % 0% Non-financial corp. Domesic % 0% Foreign % 3% Invesmen funds Domesic % 2% Foreign % 0% Oher financial ins. Domesic % 15% Foreign % 0% Oher Domesic % 0% Toal loans % Subcomponen conribuions Chec Iem Chec Iem Transformaion The example may also illusrae aspecs of he choice of approach owards search for ouliers in aggregaed checing. When he oulier idenificaion considers he overall chec iem ransformaion (Mehod 2 in Char 4), he considered measure is 20%. If his is found o be an oulier, we proceed o evaluae each subcomponen conribuion. Alernaively, he oulier idenificaion could occur direcly a he subcomponen conribuions (Mehod 4 in Char 4), where he conribuion of 15% would probably have been singled ou as an oulier. I is apparen ha he laer mehod in his case leads o a more direc idenificaion of he oulier Ineres Rae Saisics Implemenaion Since 2004, we have based our checs of he ineres rae saisics on reporer aggregaes and have found ha i significanly increases boh he efficiency of he producion and he qualiy of he saisics. For he ineres rae saisics, he chec iem will generally be an aggregaed weighed ineres rae and in he checing process we wan o evaluae how i changed, hus he chec iem ransformaion is he change in he aggregae ineres rae. In defining subcomponens and heir conribuions, we will firs consider he case for pure reporer aggregaion. Since he chec iem is a weighed sum, finding subcomponen conribuions is less sraighforward han when he chec iem ransformaion is a simple sum, as in he above example. A way of decomposing a change in a weighed sum is by using he exended Marshall-Edgeworh decomposiion (see Huerga and Selacova (2008)): IFC Woring Paper No 11 - Opimizing Checing of Saisical Repors 13

16 14 IFC Woring Paper No 11 - Opimizing Checing of Saisical Repors = = = = K K i w i w I I I 0 1, 1, 0,, 1 ( ) ( ) = = = K K I i I i w w w i 0 1 1,, 0 1,, 2 2 ) ( where I denoes he aggregaed ineres rae in period, w is he weigh for subcomponen in period and i is he ineres rae of subcomponen in period. There is a oal of K subcomponens. This formula decomposes he change in he aggregae ineres rae ino a sum of individual ineres rae effecs (firs erm on he righ hand side) and a sum of individual weigh effecs (second erm). The inuiion behind he decomposiion is as follows: he ineres rae change of an individual reporer affecs he aggregae change proporional o he average weighing over he wo periods. A change in he weigh affecs he aggregae ineres in a slighly more suble way; here he effec depends on how far away he ineres rae of he individual reporer was from he aggregaed ineres on average. An imporan feaure of his decomposiion is ha i isolaes he conribuion of each individual reporer o he aggregae whereby we can define our subcomponen conribuion: ( ) ( ) = 2 2 ) ( ) ( 1 1,, 1,, I i I i w w w i sc where denoes he individual reporer. The subcomponen conribuions can be evaluaed eiher as a oal or by he decomposed ineres rae effec and weigh effec. The above formula is based on reporer aggregaion of K reporers. I is sraighforward o subsiue individual insiuions wih wihin aggregaion over J iems. In his case, we could for example explain he developmen in he overall ineres rae due o ineres and weigh changes in subcomponens (subsiuing J for K and j w for w and j i for i ).

17 4. Oulier Idenificaion The process of idenifying ouliers will generally consis of wo phases: 1) a compuer searches for liely errors (ouliers) given pre-specified algorihms, and 2) he auomaically idenified ouliers are invesigaed by an analys who decides wheher he ouliers were acually liely errors ha should lead o furher quesions o he reporer. In secion 4.1 we consider he differen sources of informaion ha can be used as foundaion for evaluaing ouliers. This benchmar informaion can boh be used in he ransformaion of he chec iem, as par of he oulier deerminaion or as supplemenary informaion in he human evaluaion. Hereafer, secion 4.2 considers he issue of auomaic oulier deerminaion. The human evaluaion of ouliers is covered by boh secion 5 and 6. A his poin, i is worh noing ha auomaically idenified ouliers will in general have o undergo human evaluaion and ha his wor will be roughly proporional o he number of ouliers. For his reason, while we of course wan o minimize he amoun of undiscovered errors (false negaives), we also should pay aenion o minimizing he amoun of iems idenified as ouliers ha are no errors (false posiives). I should also be noed ha sophisicaed mehods, boh wihin benchmar daa rerieval and oulier deecion, can be complex o implemen and may require subsanial resources o mainain. In addiion, if he process leading o an oulier classificaion is complex, i may no easily be undersood by he analys. Hence, one should mae a mehodological cos/benefi analysis before developing sysems. 4.1 Available Benchmar Informaion Various forms of informaion can be exploied for benchmaring daa. Compilers radiionally have a endency o rely enirely on he hisory of he chec iem iself. However, here are several oher possibiliies ha will ofen be able o provide valuable benchmars. Below is a descripion of areas ha could provide reasonable benchmars (see Char 6). The exen o which differen ypes of benchmar daa is used by compilers are given in Char 7. IFC Woring Paper No 11 - Opimizing Checing of Saisical Repors 15

18 Char 6 Poenial sources for benchmar daa Reporer A Reporer B Reporer C Oher exernal informaion Oher sources for same daa Hisory [T,-1] B C Period A Checed repor D E F Oher daa in he same repor (A) When checing one par of a repor, oher pars of i may ofen be useful as benchmars. An example: If a repor conains a firm's profi-loss accouns, we can assume a posiive correlaion beween repored urnover and repored expenses. When using his ind of benchmar daa, aggregaed chec mehodologies will ofen be preferable Repor Hisory (B) The hisory of a repor can be used o esablish some saisical feaures of he underlying daa generaing process. The mos frequenly used benchmar is probably las period's observaion of he chec iem ha is being used o consruc he chec iem ransformaion of an absolue movemen or percenage movemen. In addiion, he hisory of hese changes could be used as a basis for esablishing boundaries for wha would be an oulier General Reporer Hisory (C) If here are similariies beween he ways daa are generaed across reporers, hen oher reporers' hisorical repors for a specific iem could be used o esablish general feaures of an iem applicable o all or subgroups. In addiion, his ind of daa will ofen have a panel srucure ha may be exploied for saisical inference procedures Conemporaneously Incoming Repors (D) If he reporer's business can be assumed o be correlaed, hen conemporaneously incoming repors could conain valuable informaion abou he normal developmen for a given iem for he curren reporing period. As such figures are preliminary and sill poenially conain errors, i is imporan ha any benchmaring deduced from hese will no be sensiive o ouliers (e.g. by using medians, rimmed means, ec.) Exernal Sources (E) Oher sources of informaion may be correlaed wih he informaion repored. For example, colleced daa on bans may be relaed o financial mare daa. 16 IFC Woring Paper No 11 - Opimizing Checing of Saisical Repors

19 4.1.6 Sources for he Same Daa (F) If some ind of parallel reporing exiss for he figures repored, his can be used as a checup. An example of his is financial repors as benchmars for colleced balance shee daa. Char 7 Exen o which sources for benchmar daa is used Longer hisory of checed iem (B) Las period's observaion of checed iem (B) Conemporaneously incoming repors (D) General reporer hisory for he checed iem (C) Exernal sources for he same daa (F) Oher exernal informaion (E) No Low High Source: Drejer (2012) 4.2 Auomaic Oulier Deerminaion The as of idenifying ouliers can be broen down ino wo seps: 1) we need a benchmar value, and 2) we need o se some confidence inervals around his in order o deermine wheher deviaions from he benchmar are unusual. Using he noaion from sep 4 of he framewor descripion in secion 2, his amouns o defining he funcion O(*, b ). This can be done eiher by rying o model he underlying saisical disribuion or by applying more simple rules. Among compilers of baning saisics, simple rules were he predominan way of idenifying ouliers, while oher saisical modelling was generally no used or only used o a very low exen (Drejer (2012)) Heurisic Rules Benchmars and oulier bands can be se wihou he use of saisical measures. If, for example, he benchmar daa is las period's observaion and he chec iem ransformaion is he percenage movemen, we can define a fixed percenage hreshold ha defines an oulier. While his approach riss having many false posiives, i may be preferable for is simpliciy and provide analyss confidence in nowing ha everyhing ha changed by more han x% compared o las period has been checed Saisical Model-based Deerminaion When using saisical mehods, we ry o model he underlying probabiliy funcion behind he chec iem ransformaion in order o deermine he expeced mean and esablish lielihood bands for his. Based on his saisical modelling, we can deermine ouliers in erms of lielihood of occurrence. There are numerous ways of saisically modelling disribuions assuming various models for he daa generaing process. Some main IFC Woring Paper No 11 - Opimizing Checing of Saisical Repors 17

20 assumpions are he definiion of univariae or mulivariae relaionship and wheher here is a ime series model aspec. For an applicaion wih ime series mehodology o oulier idenificaion (see Caporello and Maravall (2003)) Machine Learning The human evaluaion basically consiss of presening he analys wih an oulier ogeher wih a fixed se of addiional informaion and he analys maing a yes/no decision. If he oucome of his process is sored, he compiling insiuion will gradually build up a se of daa conaining he decisions of analyss. This ind of daa se can be used o rain algorihms o perform he yes/no decision, see for example Kosianis e al (2006). Such algorihms may aid or even replace he human evaluaion. 4.3 Examples Benchmar Daa For he ineres rae saisics we have carried ou empirical ess of numerous alernaives in order o find he opimal benchmar (see Barmann and Drejer (2011)). On his basis, we found ha he opimal source for rerieving benchmar daa generally is conemporaneously incoming repors Defining Ouliers by Empirical Disribuions In he ineres rae saisics, we perform a chec on each ineres rae, where he chec iem is he change compared o las period, normalised by a benchmar change. The benchmar change is he median of conemporaneous repors for he same iems. In order o find ouliers, we wan o describe he saisical feaures of he chec iem ransformaion. Since he chec iem ransformaion is benchmared, we do no expec deviaions o vary over ime and, in addiion, we expec he disribuion of daa o follow he same disribuion across reporers. On his bacground, we pool hisorical chec iem ransformaions over ime and reporers in order o derive empirical disribuion. 4 In his process, we ae he inermediae sep of pooling ogeher iems wih similar disribuional feaures in order o derive a smooher disribuions. Char 8 shows he lower and upper ails of hree of hese disribuions described by seleced perceniles for he chec iem ransformaion. Since we have he complee disribuional feaures of he chec iem ransformaion, we can se any level of confidence hreshold. For each iem, he analys decides he imporance and ses he confidence level accordingly. For example, ineres rae changes on he overall ineres rae for loans o households is of high prioriy, and accordingly, we wan o examine approximaely 15% of all observaion, which corresponds o seing he lower and upper limis o he 7h and 93rd perceniles of he empirical disribuion. Since his iem belongs o he lowes variance group, is disribuion is given by he blue line in Char 8. This means ha an oulier is idenified if he deviaion of he ineres rae change in relaion o he median ineres rae change is below -0,14 pp. or above 0,14 pp. The selecion is pre-programmed which means ha he analys will no have o worry abou which level o chec. 4 The chec iem ransformaions are aen from he firs received repors, sill including errors. 18 IFC Woring Paper No 11 - Opimizing Checing of Saisical Repors

21 Char 8 Empirical disribuions for he ineres rae change (chec iem ransformaion) (Iners change - median ineres rae change) 0,8 0,6 0,4 0,2 0-0,2-0,4-0,6-0,8 P3 P5 P7 P10 P15 [ ] P85 P90 P93 P95 P97 Perceniles Group wih lowes variance Group wih 3rd lowes variance Group wih 5h lowes variance IFC Woring Paper No 11 - Opimizing Checing of Saisical Repors 19

22 5. Worflow The opimal soluion on how he worflow should be organized wihin a saisical producion process, across differen saisics and across collecing insiuions, depends on facors ha are specific o each saisics or compiling insiuion. Exising IT sysems, qualiy sandards, he division of receiving and checing daa beween deparmens, ec., will all imply limis for how he worflow can be arranged. However, some consideraions will apply generally. Mapping exising worflows in process diagrams is he foundaion of finding ways o opimize he overall worflow. The processes mapped can be anyhing from he overall flow of daa beween deparmens and down o he specific ass ha an analys performs when checing incoming repors. Ofen i maes sense o sar a he very op level, as exemplified in Char 1, and from here furher expand individual processes. 5.1 Basic Worflows of Daa Checing Plausibiliy checing can involve differen inds of checs/ass, which have various inds of sandardizaion and worflow. Since he producion of saisics spans a limied number of days, here is a paricular need for sandardizaion and efficiency in he plausibiliy esing ha aes place as par of he producion. One way of dealing wih his problem is o focus he producion checs on changes, while he more elaborae checs, ensuring ha he levels of he saisics are plausible, are done on an ad hoc basis beween producions. Below are described 4 differen inds of worflows ha can ae place as par of he plausibiliy esing, each of which can be he subjec of opimizaion Sandardized Plausibiliy Checing of Repors (Producion) These are sandardized checs performed during every producion. Each chec follows a definiion as given in he framewor secion. In order o mae he evaluaion of ouliers efficien he checing process should also be sandardized Follow-up Evaluaion (Producion) Following he iniial checing of repors, reporers will ofen send revisions of heir repors. These revisions have o be evaluaed in erms of how well he problems have been addressed: Which iems have changed since he las version? Did new errors occur? How were he quesions addressed in he daa? And how did he revision affec he aggregaed picure? In order o answer hese quesions many differen inds of daa are needed and if he worflow surrounding his process is no very well-organized and suppored by specific IT ools, his par of he process can be he mos ime consuming Early Assessmen of Compiled Aggregaes (Producion) This is a chec of reporer aggregaions ha is supposed o give an early indicaion of major qualiy issues in he final saisics. The assessmen can be made on he basis of ime series of aggregaes, graphs or ables wih drill-down feaures. Noice ha his differs from he chec on reporer aggregaes in ha his is no sandardized, wih defined oulier idenificaion. Even hough his is an assessmen wihou any explici oulier idenificaion, he ools involved may adop some of he mehodology of aggregaed checs and subcomponen and heir conribuions o help pinpoin he responsible reporer/iem Ad Hoc Checs (Producion and Non-Producion) Some ypes of chec are no suiable o be implemened as sandardized checs eiher because daa is no ye available, because he chec is of a complex analyical naure, or ha heir relevance is confined o cerain ime periods. Examples of his are longer-erm developmens in iem composiions and comparisons wih oher daa sources for he same 20 IFC Woring Paper No 11 - Opimizing Checing of Saisical Repors

23 daa (e.g. financial repors). These checs are essenial for ensuring levels and composiion of he saisics. The worflow surrounding hese checs is generally unrelaed o he producion and can be performed a any ime. 5.2 Coordinaing Checing Beween Differen Compilaions The daa conained in a repor will ofen be used for muliple purposes. As an example, he repor of a ban will boh spill ino he Loans and Deposis Saisics, bu also ino he Financial Naional Accouns and he Balance of Paymens Saisics. The analys in he compiling insiuion who receives a specific repor may be analyically involved in only pars of he final use of ha repor and his analyical checing may hus be focused on his par. In his case, oher analyss will subsequenly have o validae he repor seen from heir analyical perspecive. This can lead o a siuaion where plausibiliy checs are done more han once, by differen analyss and a differen poins in ime, adding much complexiy o he checing process for boh compilers and reporers. If all users of daa define heir daa checing needs for each ype of reporing and all checs are implemened by he insiuional eniy receiving daa, such worflow complexiies are o a large exen circumvened. This common definiion is in urn faciliaed by having a universal language for defining checs, as oulined in he Framewor secion. 5.3 Worflow Implicaions of Checs wih Reporer Aggregaion As oulined in secion 1, an imporan way o improve he efficiency of he saisical producion is by using chec iems wih reporer aggregaion. This, however, poenially implies a new way of arranging he worflow Choosing Level of Checs In a seup where all checs relae only o a single repor, he worflow has he advanage of giving a simple worflow, where an analys checs he repor and gives feedbac independenly of oher checing. This case is shown in panel A of Char 9. In secion 3, wo differen ways of performing aggregaed checs were described, wih oulier idenificaion a eiher he chec iem level or he subcomponen level. If we perform aggregaed checs wih oulier idenificaion a he chec iem level, hen our checing process will no be based on he individual repors, bu insead around he chec iem (see panel B of Char 9). Since here is a 1-o-many relaion beween a chec and repors he problem of coordinaing he feedbac o reporers arises, and if he checs are performed by differen persons, i creaes inerdependencies beween he worflow of each person. If we perform aggregaed checs on subcomponen conribuions, he worflow can again be confined o a single repor because he ouliers idenified have a 1-o-1 relaion o repors (see panel C of Char 9). This illusraes some imporan worflow implicaions of doing checs based on reporer aggregaion wih oulier idenificaion a he chec level (Mehod 1 of Char 4) vs. using oulier idenificaion a he subcomponen level (Mehod 3 of Char 4) Combining Checs on Reporer Aggregaes wih Individual Checs In cases where compilers also use daa in disaggregaed form, hey canno rely enirely on checs done on reporer aggregaes, bu also need o evaluae each individual repor. For IFC Woring Paper No 11 - Opimizing Checing of Saisical Repors 21

24 such compilers, i has been common pracise o sar by performing checs on he individual repors and, once hese checs are passed, proceed o he checs on reporer aggregaes. According o Drejer (2012), 81% of compilers ha use checs on reporer aggregaes uilize his worflow. This, however, poenially has wo disadvanages. Firsly, when performing wo separae rounds of checing, here is a poenial for wo rounds of feedbac and wo differen analyss being involved in checing a repor which creaes a need for coordinaion. A way of overcoming his problem is o perform checs on reporer aggregaes a he subcomponen level (Mehod 3 of Char 4). As described in 5.2.1, he checing on reporer aggregaes in his case becomes aribuable o he individual repors 22 IFC Woring Paper No 11 - Opimizing Checing of Saisical Repors

25 Char 9 Differen ways of organising he worflow A. Checing wihou reporer aggregaion Repor 1 Auo oulier idenificaion 1 Ouliers 1 Human evaluaion 1 Feedbac 1 Repor 2 Auo oulier idenificaion 2 Ouliers 2 Human evaluaion 2 Feedbac 2 Repor 3 Auo oulier idenificaion 3 Ouliers 3 Human evaluaion 3 Feedbac 3 B. Checing wih reporer aggregaion, oulier idenificaion based on chec iem Feedbac 1 Repor 1 Auo oulier idenificaion 1 Ouliers A(1,2,3) Human evaluaion A Repor 2 Feedbac 2 Repor 3 Auo oulier idenificaion 2 Ouliers B(1,2,3) Human evaluaion B Feedbac 3 C. Checing wih reporer aggregaion, oulier idenificaion based on subcomponen conribuions Repor 1 Repor 2 Repor 3 Repor 1 Auo oulier idenificaion 1 Ouliers 1 Human evaluaion 1 Feedbac 1 Repor 2 Auo oulier idenificaion 2 Ouliers 2 Human evaluaion 2 Feedbac 2 Repor 3 Auo oulier idenificaion 3 Ouliers 3 Human evaluaion 3 Feedbac 3 Square boxes represen daa. Boxes wih round edges represen processes and boxes wih doed line and round edges represen processes wih human involvemen. and hence he checing can be merged wih he individual checs. Thus, here is a worflow wih one analys doing one chec and performing one feedbac. Secondly, in cases where checs on reporer aggregaes are done focusing on he aggregaed chec iem (Mehod 1 of Char 4) one of he main advanages is ha i helps o prioriize beween which repors o invesigae firs. By saring wih individual checs in a more arbirary fashion, compilers canno exploi his advanage. IFC Woring Paper No 11 - Opimizing Checing of Saisical Repors 23

26 A Danmars Naionalban we have he need o chec daa a individual reporer level for he ineres rae saisics. As we wan an inegraed approach o checing daa, we have chosen o merge individual checing wih checing of reporer aggregaes focusing on he subcomponen level. However, in order o exploi he prioriizing advanages of checing reporer aggregaes a he aggregae level, we have supplemened he checing process wih a ool ha enables us o chec a he aggregaed level (Mehod 3 of Char 4) and drill down o reporer level. This ool is used in he very beginning of he producion process o ge a firs overview of overall developmens and significan conribuors o help prioriize he checing and hroughou he producion o eep rac of he curren sae. 6. The IT Applicaion for Human Evaluaion The human evaluaion is aided by IT applicaions. Since hey are ools used for repeiive evaluaion oulier afer oulier, repor afer repor, producion afer producion, hey need o be designed for pracical implemenaion. In general, if assessing an oulier involves many manual procedures, i significanly slows down he process, ires he analys and increases he lielihood ha procedures will be sipped. An imporan general aspec in he design is o confine and specialize he ool o he specific as i needs o do. Too much flexibiliy in he ools will ofen slow down he process because of addiional daa processing ime and he lac of guidance of wha o do. 6.1 Experiences from he Ineres Rae Saisics The curren IT applicaion used for checing ineres rae saisics a Danmars Naionalban has coninuously been developed in close collaboraion wih daa analyss. 5 Over he years differen inds of feaures were added, some of which have laer been abandoned as hey urned ou no o be durable, however, oher survived and urned ou o add grea efficiency gains. In he following, we will ry o sum up some of he conceps ha were really durable. The descripion covers wo main ools and some remars on echnical aspecs Tool for Evaluaing Auomaically Idenified Ouliers For he process of checing he (poenially large) amoun of auomaically idenified ouliers, we have found a need for ool wih he following feaures: Providing Supplemenary Informaion Alongside he oulier he analys should receive informaion ha can suppor he decision maing. The beer informaion available, he easier and more qualified decisions can be made by he analys. Graphical Presenaion Daa relaed o he oulier, should be presened graphically. An inspecion of 2-3 graphs can give he same overall informaion as considering 7 daa series, bu in a much faser and inuiive way. Also imporanly, when looing hrough numerous ouliers, reading/considering 5 The ool is echnically based on a combinaion of SAS and Excel, which enables a large degree of flexibiliy for changing he seup and adding new feaures. 24 IFC Woring Paper No 11 - Opimizing Checing of Saisical Repors

27 figures is iresome. In oal, a graphic presenaion offers a subsanial gain o efficiency and should be considered a mus in any checing ool. Inegraing Hisorical Decisions and Explanaions A good amoun of ouliers will be due o facors ha are more or less sysemaic in naure and will mae hem loo erroneous even hough hey are no. Boh informaion on previous decisions abou he oulier and explanaions o previous ouliers may grealy help o explain he curren oulier. Hence presening decision hisory and previous dialogue for ouliers along wih he curren oulier will ofen grealy help he checing process. Inegraing Communicaion wih he Reporer Adminisraive ass such as generaing a join error repor or handling conac informaion are unrelaed o he conen of he saisical repors and hus disrac he analys from his core wor. Therefore adminisraive ass relaing o he communicaion wih reporers should be minimized. Ideally he feedbac sysem is inegraed in he checing sysem so ha a repor will be auomaically generaed Tool for Assessing Revisions When reporers send revised repors, i is imporan o be able o direcly compare he failing checs of he curren version wih hose for earlier versions for he same period. If he ool does no incorporae such comparisons, he analys will have o do his manually which disurbs he worflow Technical Aspecs of he IT Applicaions Some general feaures can mae IT applicaions beer ools: No Waiing Time Apar from slowing down he checing process by is duraion, waiing ime relaed o IT ools can also seriously harm concenraion and produciviy for he analys. Therefore, he aim should be no waiing ime, which is generally bes achieved by having generaed all daa used in he checing applicaion before he checing sars. Ergonomics Looing hrough a lis of ouliers is a sandardized as ha requires submiing he same commands repeiively o he compuer. I is our experience ha his is done faser and more ergonomically friendly by he use of eyboard raher han mouse. While i migh seem lie a minor poin, i can have major impac on he speed and comfor of he human evaluaion process. One Tool Preferably, all seps of he human evaluaion process in a producion seing should be consolidaed ino one applicaion, so ha he analys will no have o swich beween applicaions. While such a goal is mos easily achieved when he applicaion is developed inhouse, here is also scope for approaching his goal when purchasing sofware. Keeping he goal in mind hroughou he design phase and when purchasing sofware will help achieve i. IFC Woring Paper No 11 - Opimizing Checing of Saisical Repors 25