An Error Detecting and Tagging Framework for Reducing Data Entry Errors in Electronic Medical Records (EMR) System

Transcription

1 201 IEEE Internatonal Conference on Bonforatcs and Boedcne An Error Detectng and Taggng Fraework for Reducng Data Entry Errors n Electronc Medcal Records (EMR Syste Yuan Lng, Yuan An College of Coputng and Inforatcs, Drexel Unversty Phladelpha, USA {yl68, ya45}@drexel.edu Abstract we develop an error detectng and taggng fraework for reducng data entry errors n Electronc Medcal Records (EMR systes. We propose a taxonoy of data errors w ree levels: Incorrect Forat and Mssng error, Out of Range error, and Inconsstent error. We a to address e challengng proble of detectng erroneous nput values at look statstcally noral but are abnoral n edcal sense. Detectng such an error needs to take patent edcal hstory and populaton data nto consderaton. In partcular, we propose a probablstc eod based on e assupton at e nput value for a feld depends on e hstorcal records of s feld, and s affected by oer felds rough dependency relatonshps. We evaluate our eods usng e data collected fro an EMR Syste. The results show at e eod s prosng for autoatc data entry error detecton. Keywords Data Entry Errors; Electronc Medcal Records (EMR; Error Detectng; Error Taggng I. INTRODUCTION Currently, large-scale data analyss s conducted n alost every area. Advanced analytcal process and data nng technques have been used to dscover valuable nforaton and knowledge fro large-scale data. However, ost data nng technques assue at data has already been cleaned [1]. Ths s not always true n real stuatons. Data preparaton process and data qualty ssues are crtcal to relable data nng results, because data w errors wll coprose e credblty of e data nng results. The data lfecycle of an applcaton typcally conssts of e followng steps: data collecton, transforaton, storage, audtng, cleanng, and analyzng to akng decsons [2]. Data errors can be reduced n each step of e data lfecycle. Snce data collecton s e frst step of e lfecycle, reducng data entry errors n e data collecton stage s e earlest opportunty to enable data qualty. Error and outler detecton have been studed n varous applcaton doans, such as credt card fraud detecton[], network ntruson detecton[4], and data cleanng n data warehouse[5]. In general, errors can be classfed nto ree types: pont errors, contextual errors, and collectve errors [6]. In e healcare doan, data errors n electronc edcal databases have been studed recently [7, 8]. Generally, Mengwen Lu, Xaohua Hu College of Coputng and Inforatcs, Drexel Unversty Phladelpha, USA {l94, xh29}@drexel.edu Electronc Medcal Records (EMR data errors have been classfed as ncoplete, naccurate, or nconsstent [9]. More specfcally, based on EMR st notes, e preous study [10] classfes e data entry errors nto Inconsstent Inforaton, Mscellaneous Errors, Mssng Sectons, Incoplete Inforaton, and Incorrect Inforaton [10]. For our case, we conducted a coprehensve lterature reew. As a result, we suarzed dfferent levels of data errors n a ore detaled anner. Three levels of data entry errors are proposed as shown n Table I. Exaple errors are dscussed n e followng paragraphs. TABLE I. Level 1: Incorrect Forat and Mssng Level 2: Out of Range Level : Inconsstent LEVELS OF DATA ENTRY ERRORS L1-1: Incorrect Forat Data L1-2: Mssng Data L2-1: Out of noral range L2-2: Out of populaton trends L2-: Out of personal trend L-1: Personal Inconsstent L-2: Populaton Inconsstent L1-1 errors can be easly controlled w pror knowledge about e data forat. Constrants can be set for each feld. For exaple, only nubers are allowed to be entered nto blood pressure feld. L2-1 error also can be easly controlled as long as ere are knowledge about e noral range of each feld. For exaple, heght values can be constraned n e range of 0 to eters. Detectng L1-2 errors, however, s a dffcult research task, but we can learn to detect ssng data accordng to clncal gudelnes and caregng practces. Once e ssng values are detected, we can set constrants to s partcular feld as we dd for L1-1 and L2-1. For L2-2 errors, usng statstcs for detectng erroneous values for a sngle attrbute s well-studed and robust results can be obtaned. For exaple, L2-2 errors can be calculated usng a unvarate outler detecton technque called Hapel X84 [11]. In s paper, we do not focus on solng e L1-1 to L2-2 error probles. The challenge les n e detecton of values at look statstcally noral but are abnoral n edcal sense lke L2-, L-1, and L-2 errors. For exaple, e weght value of a patent ncreases to 20lbs fro 180lbs n ree ons. The weght value of 20lbs s n e noral populaton dstrbuton /1/$ IEEE 249

2 range, but for s partcular patent, t s suspcous f he/she has a healy lng style and does not have ajor edcal probles. It s categorzed as a L2- error. The followng llustrates oer exaples. A record showng a ale patent has e status of pregnancy would be consdered as a L-1 error. An obese and dabetc patent w BMI (Body ass ndex larger an 5 s lkely to have a cardovascular dsease anfested as hypertenson and dyslpdea. If e patent s blood pressure values are sgnfcantly low, but stll n e noral range, e blood pressure value would be nconsstent w e BMI value fro e populaton perspectve. Ths sgnals a L-2 type of error. In s paper, we propose an error detectng and taggng echans n real te to address ese challenges based on hstorcal data and dependency relatonshps. The entre fraework conssts of ree an coponents: 1 Fnd a slar patent group. Error detecton s based on hstorcal datasets. For a sngle patent, e hstorcal datasets should be coposed fro patents who have equvalent edcal stuatons, clncal condtons, and basc deographc nforaton. Because t s dffcult to fnd a group of patents w exactly e sae edcal condtons, we detect errors based on a dataset fro patents w slar condtons. 2 Learn a probablstc odel. Based on e dataset fro a group of slar patents, we learn paraeters for our proposed odel. The probablstc odel s used to caculate e nput value probablty for each feld. Return a tag. Error taggng s based on results fro e probablstc odel. If an nput value probablty for a feld s relatvely hgh, en e value has a hgher chance to be a correct value and a Noral tag wll be returned. Whereas f e probablty s relatvely low, e tag s Suspcous. We use a reshold to decde whch tag should be returned. Our experental results show at our error detectng and taggng eod s prosng. The predctve accuracy of e probablstc odel s between 70% and 85% usng our experental data (explaned n e Evaluaton secton. We also copare our eod w exstng eods. The results ndcate at our eod has a better perforance. In suary, e contrbutons of our paper are: 1 We suarze dfferent levels of data entry errors based on a coprehensve lterature reew. Our eod focuses on detectng and taggng errors at cannot be detected by sple statstcal eods. 2 We desgn a probablstc odel based on two assuptons: frst, e current patent s nput value for a feld depends on hs/her hstorcal records n s feld; and second, e nput value for a feld depends on values fro oer felds.we cobne e nfluences fro ese two aspects n our odel. We develop an error detectng and taggng fraework w fve stages: selectng a classfcaton odel, fndng slar patent group, buldng a probablstc odel, settng a reshold, and returnng tags. 4 We test our eod on dataset fro an EMR syste. Our experental results show at e predctve accuracy of our probablstc odel s better an Bayesan Network. Our odel also generates better results an e oer eod assung e nput value for a feld s based solely on hstorcal records. The rest of e paper s organzed as follows: Secton II descrbes related work. Secton III presents our error detectng and taggng fraework. Secton IV ntroduces our probablstc odel and error detectng eod. Secton V presents our evaluaton process and experental results. Secton VI presents our conclusons and future research drectons. II. RELATED WORK Statstcal eods have been used to analyze and detect data errors at data cleanng stage [12-14]. However, we need soe eods to reduce date entry error n e data collecton stage[15]. There are dfferent ways to detect errors and control data qualty, ncludng classfcaton based neural networks, Bayesan networks[16], support vector achne, rule-based eods, unsupersed clusterng, and nstance-based eods. The Usher syste s developed to detect errors on for entry felds by usng Bayesan network and a graphcal odel w explct error odelng[17]. The syste ncludes a probablstc error odel based on a Bayesan network for estatng contextualzed error lkelhood for each feld on a for. The eod uses Bayesan network to learn dependency relatonshps aong felds, but t does not take hstorcal records for a partcular feld nto consderaton. In s paper, we detect errors based on dependency relatonshps aong felds as well as hstorcal data for a partcular feld. In addton, we classfy patents based on soe basc nforaton to cluster e nto slar groups before we apply error detecton. As our eod s probablstc n nature, selectng e rght dataset for estatng e probabltes s crtcal. For exaple, t s unreasonable to predct e nput value for a person whose age s 50 usng a dataset for people whose ages are around 10. Usng taggng echans has been shown to decrease certan types of errors [18]. Data entry nterfaces can be desgned to help prong data qualty. We propose a syste at classfes e nput values of each feld as Suspcous or Noral, and prodes s nforaton as feedback to e user. If e Suspcous tag occurs, en t rends e user to check e nput value. Consequently, t would help reducng data entry errors. III. ERROR DETECTING AND TAGGING FRAMEWORK A. Error Detectng and Taggng Fraework Our error detectng and taggng fraework s presented n Fg. 1. A data entry nterface contans ultple felds. After a user nputs a value for a feld, a tag ( Noral or Suspcous wll be returned. 250

3 probablstc odel and e reshold settng, we return a Noral or Suspcous tag to e current user. IV. DATA ENTRY ERROR DETECTING METHOD Fg. 1. Error Detectng and Taggng Fraework Fg. 2. Hstrocal Records for Error Detectng Task A. Data Entry Error Detectng Meod Gven a data entry nterface I, let F { Fv,, Fj,, Fk } be a set of nput felds on e nterface I. For exaple, n Fg. 4, e feld for enterng e Weght value can be represented as F Weght. Here we use a sple exaple to llustrate e hstorcal data forat and e error detectng task. Fg. 2 shows a set of records for dfferent patents. There are fve felds for each record: Date (Observaton Date, BP_S (Systolc Blood Pressure value, BP_D (Dastolc Blood Pressure value, BMI, and Weght. Our error detectng task s to tell wheer e patent w ID has a noral or suspcous nput value for BP_S, BP_D, BMI and Weght at e observaton date: 1/1/2010. The feedback s generated based on e forer 8 records fro e patent w ID, and also e oer 11 records fro patents w ID 1 and ID 2, assung ese ree patents have slar condtons. Based on e data, we wll learn dependency relatonshps aong BP_S, BP_D, BMI and Weght. These relatonshps are captured n a probablstc graphcal odel. B. Error Detectng Process Based on our error detectng and taggng fraework, we propose e error detectng process as shown n Fg.. A patent s odeled as a set of data felds descrbng e patent s deographc nforaton and clncal condtons. A for s used to collect a set of specfc types of nforaton durng an encounter, for exaple, blood pressure, weght, etc. For our eod, we dde e entre patent nforaton nto two categores: ndependent felds whch are consdered as fxed durng e error detecton process and dependent feld whch wll be collected and flagged as noral or suspcous. Fg.. Error Detectng Process As shown n Fg., our error detectng process conssts of e followng fve steps. Frst, patent records are categorzed nto groups based on nforaton about e basc ndependent felds fro e current patent and oer hstorcal records usng a classfcaton odel. Second, we fnd a slar patent group for e current patent. Thrd, we buld a probablstc odel based on clncal felds and hstorcal records fro a group of slar patents. Four, we set a reshold for e probablty of nput value. Fnally, based on results fro e Fg. 4. A Data Entry For Fg. 5. An exaple of relatonshps aong felds For a sngle feld F v, we can use a sequence of values v { v1,, v 1, } to represent all hstorcal records for a patent, where v s e current value entered by a user for e feld F v, { v 1,, v 1} s e hstorcal record sequence for e feld F and v v s e value entered for e feld 1 F v at e ( 1 te. We assue at a patent s current nput value for a feld depends on hstorcal nput records of s feld. Abnoral changes of nput values coparng w hstorcal records need to get attenton. In probablty eory, such a sequentally dependent process can be descrbed by a Markov odel. Accordng to 1 order Markov assupton, e current value for e feld F v at te depends only on e value for e sae feld at( 1 te. Therefore, for a gng nput value v, we can use e value v to predct e probablty of e value 1 for e feld F v at e current te. The probablty can be represented as q ( 1. Slarly, we can have a probablty accordng to 2 order Markov assupton as q ( 1, 2. But for a new patent, ere are no hstorcal records for value of feld F v. So we can assue at e current value v for e feld F v at te does not depend on preous hstorcal data fro e patent. Therefore e predcted probablty can be represented as q v. ( Our goal s to predct e probablty of an nput value v for e feld F v, whch can be represented as q M ( v. Usng 1 order Markov assupton, 2 order Markov assupton, and a pror probablty, we can calculate q v as n Equaton (1: M ( q M ( v all _ condtons α q ( v 1, v 2, all _ condtons + α 2 q ( v v 1, all _ condtons + α q ( v all _ condtons 1 (1 251

4 all_condtons eans at e probablty s calculated based on a dataset only fro patents at have equal deographc and clncal condtons. In e followng equatons, we wll ot all_condtons n each ter assung at ey have been taken nto consderaton. The ree paraeters α, 1 α, and 2 α are used to balance e ree probabltes [24], andα 1 + α2 + α 1. Gven a set of records, we can estate each part of probablty accordng to Equaton (2, Equaton (, and Equaton (4, respectvely. q ( v v q ( v v q ( v, v all 2, 1, v, v 1, v For Equaton (2 and Equaton (, N ( v 2, 1 s e occurrence nuber of nput value v for feld 2 F at v e ( 2 te followed by e nput value v for feld 1 F v at e ( 1 te for a patent. Slarly, 2, 1, s e occurrence nuber of nput value n a sequence of { 2, 1, } for e ( 2, ( 1, and te for a patent. For Equaton (4, all s e total nuber of values for feld F v for hstorcal records fro all slar patents. Slar patents eans a group of patents who are slar to e current patent. The slarty s easured n ters of er deographc and clncal condtons such as age, gender, dsease condtons, edcatons, treatents, socal acttes, self-anageent, etc. N ( v s e occurrence nuber of v for feld F v fro all hstorcal records based on all slar patents. The Equaton (1 only uses e hstorcal records fro a sngle feld. However, ere are dependency relatonshps aong dfferent felds on a data entry nterface I. The relatonshps aong dfferent felds can be obtaned based on pror knowledge fro experts or learned fro hstorcal data [17]. The relatonshps can be learned fro hstorcal records w a Bayesan network usng e Max-Mn Hll-Clbng (MMHC eod [19]. We pleented e eod usng e R packages: bnlearn [20] and gran [21]. An exaple of learned relatonshps aong felds s showed n Fg. 5. A Condtonal Probablty Table (CPT wll also be learned for e Bayesan Network. For e exaple shown n Fg. 5, e value v of F depends v on e value F w and F u, so we can represent e probablty as q B( w, u. By addng e probablty of q B( w, u to e Equaton (1, we get e followng Equaton (5: q v β q ( v + γ q ( v w, u (5 ( M B where e two paraeters β andγ, β + γ 1. We already get e value of q M ( v by Equaton (1. We can get e value of q v w, u fro CPT of e learned Bayesan B ( (2 ( (4 Network as Fg. 5. By pluggng n e above values, we copute a fnal probablty q( v all _ condtons. B. Error Taggng Meod We assue at for each feld F v e set of possble values s fnte and dscrete. If e values for a feld are contnuous, we apply approaprate eods for dscretzng e. Let ' '' V { v, v, } be e set of possble values for a feld F v. The current nput value v at e te s a eber of e set. For each possble nput values n e set V, we can calculate probablty based on e Equaton (5. We get a tag result for current nput value v for e feld F v as Equaton (6. Noral tag( Suspcous q( θ q( v < θ For e reshold value θ n e Equaton (6, ere are eods at can be appled to set e reshold. For exaple, 80/20 rule, Lnear Utlty Functon [22] and Maxu Lkelhood Estaton[2]. In s paper, we wll use 80/20 rule to dynacally decde e reshold value. In partcular, probablty locates before 20% hghest values eans e correspondng value s Noral, oerwse t s Suspcous. V. EVALUATION The preous sectons descrbed e eoretcal dervatons of e odel and coputatonal steps. In s secton, we present an experent usng a set of lted data for evaluatng e perforance of e odel. We aed to gan soe frsand practcal confdence rough a plot study. The ltaton of e data s two-fold. Frst, we only obtaned e data for a sall nuber of people, about Second, we consder a patent odel w a sall nuber of felds ncludng age, sex, BP, weght, and BMI. The experent s by no eans orough and coplete because e real odel for descrbng a patent s far ore coplcated an just sevaral felds. The purpose of e experent s to deonstrate e effectveness of e probablstc odel for error detecton takng hstorcal data and nter-feld dependency relatonshps nto consderaton. We obtaned e data fro a heal clnc afflated w Drexel Unversty n Phladepha, PA, USA. All e data are de-dentfed for HIPAA (Heal Insurance Portablty and Accountablty Act coplance. We only obtaned records w sx felds wout any addtonal clncal nforaton. A. Data Sets We selected sx felds fro databases: Age, Sex, Dastolc Blood Pressure (BP_D value, Systolc Blood Pressure (BP_S value, Weght, and Body Mass Index (BMI. we extracted 1448 patents w records fro all ese sx felds fro e EMR syste. Because our dataset s lted and patents nforaton s nsuffcent, we cannot guarantee all patents have equal clncal condtons and treatents n s sall scale prelnary study. Only Age and Sex are used as basc felds to conduct classfcaton process to fnd a slar patents group for current patent. Based on e two basc attrbutes: age and sex. Patents are dded nto 14 categores and e nubers of (6 252

5 patents for each category are shown n Table II. The average nuber of hstorcal records for each patent s 8.4. We exane our data entry error detectng eod for each group. TABLE II. 16 KINDS OF SIMILAR PATIENTS Patents/Records Group Age Sex Nuber 1 M 0/15 [20, 0 2 F 6/567 M 68/28 [0, 40 4 F 12/97 5 M 122/64 [40, 50 6 F 188/ M 191/151 [50, 60 8 F 270/ M 10/850 [60, F 181/ M 20/195 [70, F 55/658 1 M 4/6 [80, F 21/172 a. M stands for Male, and F stands for Feale BP_D, BP_S, Weght, and BMI are transfored to dscrete value and used as clncal felds to buld a Bayesan Network to fnd e nter-feld dependency relatonshp aong ese clncal felds. In our experent, we bult a unfed Bayesan network usng all records. The Max-Mn Hll-Clbng (MMHC eod s used to generate e Bayesan Network structure as shown n Fg. 6. Fg. 6. Learned Bayesan Network Fg. 7. Predctve Accuracy and Nuber of Records of 14 Groups The varable BP_D depends on e varable BP_S. The jont probablty between BP_D and BP_S can be calculated fro e CPT (Condtonal Probablty Table of Bayesan Network. B. Experental Results Based on our dataset, we used our eod to predct e value of varable BP_D. The predctve power of our eod s easured n nddual -fold cross valdaton experents for each group. The average predctve accuracy and nuber of records for each group s shown n Fg. 7. The accuracy s calculated as correctly predcted nstance nuber dded by total nstance nuber. The paraeters set here are α α α 1/, β / 4, and γ 1/ Fg. 8. Coparatve Perforance of Usher w Our Meod Fg. 9. Accuracy Coparson for Dfferent Paraeters Set The average predctve accuracy for all groups s 71%. The predctve power s stable for all ese groups, whch ranges fro 70% to 85%, except for group1, group1, and group14. Ther nubers of records are lower an 200, so er predctve accuracy are relatvely lower. We pleent Usher s error odel [17] on our dataset. Usher nduced e probablty of akng an error usng an error odel. The coparatve results of Usher and our eod s shown n Fg. 8. Usher s average accuracy s 6.2%, whle our eod s 71%. Except for group 1 and group 1 n whch e nubers of records are too sall, our eod has better perforance an Usher. Snce e nuber of records and accuracy of group 8 s relatve hgh. We test dfferent sets of paraeters on e dataset of group 8. The dfferent paraeter sets and correspondng accuracy results are as shown as Table III and Fg. 9. There are 9 dfferent sets of paraeters. TABLE III. ACCURACY RESULTS FOR DIFFERENT PARAMTERS SET Set α1 α α 2 β γ Accuracy / 1/ 1/ / 1/ 1/ / 1/ 1/ As shown n Table III, n Set 1 and Set 2, β 1 andγ 0, whch eans at predcton only depends on past hstorcal values of e current sngle feld. In set, β 0 andγ 1, whch eans at predcton only depends on oer felds, but does not depend on hstorcal values of current sngle feld. That s, e result of set s e result of only usng Bayesan network. The accuracy results fro set 1, 2 and are relatvely low. In suary, e accuracy results obtaned by e eods purely dependng on past hstorcal values of e current sngle feld or on dependency relatonshps aong ultple felds learned fro Bayesan network are not as good as e results obtaned by cobnng ese two aspects. In Fg. 9, sets 6, 7, 8, and 9 have relatvely hgher accuracy. In ese four sets, paraeters β and α have hgher weght. That eans predcton dependng on all forer 2 stages of hstorcal values of e current sngle feld and dependency relatonshps aong ultple felds proves e accuracy. The set 9 has hghest accuracy. But n practce, t s not recoended, because for soe sall datasets, α ght equal to 0. The recoended paraeter sets are Set 6, 7, and 8. We also conduct an experent to copare e perforance of our odel to at of e Hapel X84[11]. Based on our eod, e Noral ranges for a patent s BP_D value can dynacally narrow down to [70, 90] as shown n Fg. 10. a, whle e Hapel X84 gave Noral to a broader range [45., 114.7] as shown n Fg. 10. b for all populaton. The narrower range ndcates at our eod s ore senstve to error detecton. 25

6 a Our eod b Hapel X84 Fg. 10. Our eod copared w Hapel X84 VI. DISCUSSION AND FUTURE WORK Ths paper proposes a fve-step error detectng and taggng fraework for reducng data entry errors at pose sgnfcant challenges to exstng statstc eods. Our probablstc odel for error detectng s based on e assupton at nput values for a feld can be predcated usng hstorcal data and nter-feld dependency relatonshps aong felds. We conducted plot experents to evaluate our eod. Results show at our probablstc odel has better accuracy an a Bayesan network, t s also better an e eod based on a Markov assupton at e nput value for a feld s based only on hstorcal records of s sngle feld. We also copare our eod w Usher s error odel. The result shows our eod has a better perforance. Moreover, our eod has ore precse results an e Hapel X84 eod. Our future research work wll focus on followng aspects: 1 Currently, our taggng eod can only return two knds of tags as Noral and Suspcous. We would lke to extend our odel to detect ultple errors as shown n Table I. 2 Our error detectng eod only focuses on dscrete nput value. We plan to prove our eod to deal w contnuous nput values. We plan to conduct a ore orough experental evaluaton by collectng ore patent data w ore attrbutes n e future. ACKNOWLEDGMENT Ths work s supported n part by a Drexel Jupstart grant on Heal Inforatcs and e NSF grant IIP for e center for sual and decson nforatcs (CVDI. REFERENCES [1] Dasu, T. and T. Johnson, Exploratory data nng and data cleanng. Vol : Wley-Interscence. [2] Batn, C. and M. Scannapeca, Data qualty. 2006: Sprnger. [] Chan, P.K., et al., Dstrbuted data nng n credt card fraud detecton. Intellgent Systes and er Applcatons, IEEE, (6: p [4] Lazarec, A., et al., A coparatve study of anoaly detecton schees n network ntruson detecton. Proc. SIAM, 200. [5] Rah, E. and H.H. Do, Data cleanng: Probles and current approaches. IEEE Data Eng. Bull., (4: p. -1. [6] Chandola, V., A. Banerjee, and V. Kuar, Anoaly Detecton: A Survey. Ac Coputng Surveys, (. [7] Arts, D.G., N.F. De Kezer, and G.-J. Scheffer, Defnng and prong data qualty n edcal regstres: a lterature reew, case study, and generc fraework. Journal of e Aercan Medcal Inforatcs Assocaton, (6: p [8] Beretta, L., et al., Iprong e qualty of data entry n a low-budget head njury database. Acta Neurochrurgca, (9: p [9] Phllps, W. and Y. Gong, Developng a noenclature for er errors, n Huan-Coputer Interacton. Interactng n Varous Applcaton Doans. 2009, Sprnger. p [10] Khare, R., An, Y., Wolf, S., Nyrjesy, P., Lu, L., & Chou, E, Understandng e EMR error control practces aong gynecologc physcans. Conference 201 Proceedngs, p [11] Hellersten, J.M., Quanttatve data cleanng for large databases. Unted Natons Econoc Cosson for Europe (UNECE, [12] Nah, M.L., C.F. Peper, and M.M. Cunnngha, Quantfyng Data Qualty for Clncal Trals Usng Electronc Data Capture. Plos One, (8. [1] Goldberg, S.I., A. Neerko, and A. Turchn, Analyss of data errors n clncal research databases. AMIA Annu Syp Proc, 2008: p [14] Jacobs, B., Electronc edcal record, error detecton, and error reducton: a pedatrc crtcal care perspectve. Pedatr Crt Care Med, (2 Suppl: p. S [15] Goldberg, S.I., et al., A Weghty Proble: Identfcaton, Characterstcs and Rsk Factors for Errors n EMR Data. AMIA Annu Syp Proc, : p [16] Wong, W., et al. Bayesan network anoaly pattern detecton for dsease outbreaks. n MACHINE LEARNING-INTERNATIONAL WORKSHOP THEN CONFERENCE [17] Chen, K., et al., Usher: Iprong data qualty w dynac fors. Knowledge and Data Engneerng, IEEE Transactons on, (8: p [18] Chen, K., J.M. Hellersten, and T.S. Parkh. Desgnng adaptve feedback for prong data entry accuracy. n Proceedngs of e 2nd annual ACM syposu on User nterface software and technology ACM. [19] Tsaardnos, I., L.E. Brown, and C.F. Alfers, The ax-n hllclbng Bayesan network structure learnng algor. Machne learnng, (1: p [20] Scutar, M., Learnng Bayesan networks w e bnlearn R package. arxv preprnt arxv: , [21] Højsgaard, S., Graphcal Independence Networks w e gran package for R, 2012, Journal. [22] Arapatzs, A., et al., KUN on e TREC-9 Flterng Track: Increentalty, decay, and reshold optzaton for adaptve flterng systes [2] Zhang, Y. and J. Callan. Maxu lkelhood estaton for flterng resholds. n Proceedngs of e 24 annual nternatonal ACM SIGIR conference on Research and developent n nforaton retreval ACM. [24] Mchael Collns. Language Modellng, llns/