Generation of Synthetic Data Sets for Evaluating the Accuracy of Knowledge Discovery Systems

Transcription

1 Geeratio of Sythetic Data Sets for Evaluatig the Accuracy of Kowledge Discovery Systes Daiel R. Jeske Uiversity of Califoria Deartet of Statistics Riverside, CA Behrokh Saadi Lucet Techologies Perforace Aalysis Deartet Holdel, NJ Pegyue J. Li La e, Sea Cox, Rui Xiao Ted ouglove, Mih Ly Doug Holt, Rya Rich Uiversity of Califoria, Riverside CA ABSTRACT Iforatio Discovery ad Aalysis Systes (IDAS) are desiged to correlate ultile sources of data ad use data iig techiques to idetify otetial evets that could occur i the future. Alicatio doais for IDAS are uerous ad iclude the eergig area of hoelad security. Develoig test cases for a IDAS requires backgroud data sets ito which hyothetical future scearios ca be overlaid. The IDAS ca the be easured i ters of false ositive ad false egative error rates. Obtaiig the test data sets ca be a obstacle due to both rivacy issues ad also the tie ad cost associated with collectig ultile istaces of a diverse set of data sources. I this aer, we give a overview of the desig ad architecture of a IDAS Data Set Geerator (IDSG) that will eable a fast ad corehesive test of a IDAS. The IDSG will geerate data usig sytactical rule-based algoriths ad also seatic grahs that rereset iterdeedecies betwee attributes. The IDSG tool will feature a default set of attribute geeratio caabilities as well as a wizard that allows users to grow the scoe of data the tool ca geerate. e describe the data geeratio algoriths for soe default attributes such as ae, geder, age, address, hoe uber, driver s licese uber, social security uber, credit card uber, icoe level, occuatio, ad credit card trasactios. The credit card trasactio exale illustrates the use of seatic grahs to cature attribute deedecies, ad also illustrate our aroach for aroxiatig high diesioal ultivariate distributios. Categories ad Subect Descritors D..5 [Testig ad Debuggig]: Testig Tools. D..4 [Software/Progra erificatio]: Statistical Methods, alidatio. I..4 [Kowledge Reresetatios Foraliss ad Methods]: Seatic Grahs. Geeral Ters Algoriths, Desig, erificatio Keywords Iforatio Discovery, Data Miig, Data Geeratio.. INTRODUCTION. Backgroud Moder data collectio ethods i cobiatio with the raid icreases i iforatio techology have ade ossible the asseblig of extesive data sets o idividuals. Iforatio Discovery ad Aalysis Systes (IDAS) for a iortat tool for turig large quatities of data that have bee collected ito iforatio that ca be used. IDAS s are desiged to correlate ultile sources of data ad use data iig techiques to fid relatioshis withi disarate data sets that could be used to redict evets that occur i the future. A IDAS extracts iforatio fro data by fidig atters, threads, ad relatioshis. To do this, IDASs use a hybrid of statistical ad artificial itelligece ethodologies such as atter recogitio, classificatio, categorizatio, ad learig, ad eloy data iig tools based o eural etworks, Bayesia etworks, classificatios schees, ad regressio odels, aog others, to cull out ad aggregate iforatio across differet data sources. IDAS s have bee a aor asset to busiess alicatios such as fraud revetio [,7] ad are i use i the edical field i a wide variety of alicatios icludig hel i diagosis [5] ad aalysis of edical videos [3].A recet survey by the US Geeral Accoutig Office foud that 5 federal agecies are coductig or la to coduct 99 searate data iig efforts, with 3 of these curretly oeratioal [4]. It is believed that IDAS s could be equally effective for itelligece alicatios such as rovidig leadig idicators of terrorist acts.. Motivatig Proble A critical techical issue with IDASs is their ability to rovide accurate iferece. Give the diversity of techiques used to develo a IDAS, it is desirable to have a baselie aroach for testig their ability to ake accurate iferece as well as their ability to deal with large iut data sets with varyig degrees of accuracy. A iortat art of testig of IDAS s is the geeratio of sythetic data sets for use i test cases. Develoig test cases for a IDAS requires backgroud data sets ito which hyothetical future scearios ca be overlaid. The IDAS ca the be easured i ters of false ositive ad false egative error rates. Obtaiig the test data sets ca be a obstacle due to both rivacy issues ad also the tie ad cost

2 associated with collectig ultile istaces of a diverse set of data sources. I this aer, we give a overview of the desig ad architecture of a IDAS Data Set Geerator (IDSG) that will eable a fast ad corehesive test of a IDAS. This IDSG is curretly uder develoet for the Deartet of Hoelad Security (DHS) by the Uiversity of Califoria, Riverside ad Lucet Techologies. The secific goal is to desig ad develo a rototye syste to exlore the feasibility of sythesizig data sets for testig the effectiveess of IDASs desiged to ucover otetial threat scearios. Such a data sythesis caability is a attractive alterative for testig a IDAS whe realistic data sets are ot readily available due to rivacy or restrictive access issues, or the challege ad costs of aassig diverse data fro ultile sources is iosig. Further, a sythetic data caability allows various features ad caabilities of a IDAS to be exercised i ultile test cases, thus erittig a eaigful figure of erit of its efficacy to be established. There are several difficulties i the geeratio of IDAS test data i geeral, ad for the DHS roect i articular. Geeral difficulties iclude: Multivariate data IDASs use large ubers of data fields with ukow correlatios betwee fields. Multile data tyes Data for these systes ca be cotiuous, oial, ordial, textual, or iage data. Lack of available high diesio data Readily available data, the tye eeded to quickly develo test cases, tyically are reseted i two or at ost three diesios. DHS secific difficulties iclude: Privacy issues The Deartet of Defese, Office of the Isector Geeral issued a reort fidig that the Defese Advaced Proects Agecy (DARPA) had ot adequately cosidered rivacy cocers associated with the Total Iforatio Awareess Progra terroris data iig roect [3]. Security issues The eed to aitai security for the DHS recludes the direct iteractio with the cliet to idetify relevat data fields. Lack of traiig data Because of security eeds, traiig data for evaluatig the effectiveess of the data sythesis tools are ot available. The roble of develoig a data set geerator that ca create realistic data for all ossible IDAS data iut tyes for all ossible IDASs is foridable, ad really is ot our goal. Istead, our aroach is to sythesize data sets of sufficiet quality to eable IDAS evaluatio. Our visio is a exerietal latfor that geerate data sets cotaiig user-desiged threat sceario iforatio as well as backgroud data whose quality ca be icreased as eeded by iosig relatioshis, correlatios, ad other statistical coditios o the data attributes withi or across data sets..3 Obective Exales of geeratio of sythetic data through various eas for testig uroses are available i ay areas of research. For exale, sythetic data sets were geerated for testig a Robust/Resistat crystallograhic refieet rocedure usig two sets of Gaussia rado errors, a Gaussia error distributio cotaiated with 0% draw fro a high variace Gaussia distributio, ad a log-tailed rado error distributio [8]. Sythetic data has bee geerated for use i groud truth testig of differet rotei sot detectio software ackages through the use of a Gaussia ixture odel obtaied fro traiig data [9]. Sythetic hadwritig data have bee geerated usig rado erturbatios of actual hadwritig for use i testig cursive had-writig recogitio systes [0]. Alteratively, the US Cesus Bureau is exlorig the use of sythetic data as a eas of rotectig the cofidetiality of the ublic while aitaiig statistical quality []. I ultivariate cases, articularly where there are colex covariace structures requirig a uderstadig of the iterdeedecies betwee variables, the data sythesis rocess becoes ore colex. Cetral to the aroach i this aer is the develoet of a obect-orieted iforatio odel to rereset data obects that readily erit iosig additioal data structure ad coditios o data attributes. Our sythetic data geeratio desig is based o the use of seatic grahs. For this roect, a seatic grah is defied as a grahical deictio of the iterrelatioshis betwee data fields of iterest. Seatic grahs have bee used i various fields for suarizig colex relatioshis betwee ultile factors. For exale, docuet suaries have bee roduced usig seatic sub-grahs to rereset the various characteristics of the docuet but i a vastly reduced size [6]. Colex relatioshis betwee airs of ous i seteces have bee suarized usig autoatic grah algoriths [] as well. A key asect of our obective to sythesize evaluative IDAS iut data is to rovide a scalable desig that ca be adated to icrease data quality, add data tyes, ad address coutatioal costraits dictated by our users. The rest of this aer is orgaized as follows. Sectio describes the geeral architecture used i our data geeratio aroach. Sectio 3 rovides a few illustrative alicatios of our roosed desig, with a exale draw fro geeratig credit card trasactio data. Sectio 4 suarizes the ricial fidigs todate of this roect.. ARCHITECTURE. User Sceario To better deostrate the fuctioality of IDSG, we reset a tyical user sceario i Figure. As show i the figure, the user iitially eters iforatio o the tye of data set, tables, table relatioshis, the attributes withi each table, ad the attribute relatioshi.

3 Figure. A Tyical User Sceario At this oit, the user eeds to defie the relatioshis aogst the attributes. This will be doe by a SeaticGrahBuilder. As a exale, for a credit card trasactio data set, the user ay require two tables, CardHolder ad CardTrasactio, with -toay relatioshi betwee the CardHolder ad the CardTrasactio Tables. The table CardHolder has attributes: Nae, Address, TelehoeNuber, Date-of-Birth, Geder, Icoe, CardNuber, CustoerSiceDate. CardTrasactio table icludes attributes CardNuber, TrasactioDate, Trasactio Aout, PurchasedIte, ad BusiessNae. For selectio of the attributes, the user will be rovided with a set of re-defied attributes to choose fro. However, if desired, a IDSG izard will allow the user to defie ew attributes. Oce the above two sets of iforatio are rovided, IDSG ay build the DataSetSchea. For the schea, the user eeds to iut the ultilicity relatioshi betwee the tables. I this exale, the user eeds to secify how ay trasactio records er CardHolder/card to geerate. For this, IDSG will rovide a list of robability distributios to select fro. O the scree a artially costructed seatic grahs will aear (Figure ). The grah is artially costructed sice soe of the attributes or relatioshis ay ot have bee redefied ad thus eed to be secified by the user. For exale, the relatioshi betwee Age ad Icoe ay already exist, but the user ay decide to add a additioal relatioshi betwee Geder ad Icoe. This will add a additioal lik to the seatic grah betwee the Geder ad Icoe attributes ad oe a widow for secificatio of the tye of relatioshi. Alteratively, the user ay deterie to aitai the structure of the grah (odes ad edges) ad oly odify araeters. For exale, the user ay decide to chage the araeters for Geder distributio fro to 40-60, if the uber of Male-Feale credit card holders are a ratio. Figure. A Partially Costructed Seatic Grah Oce the schea ad the seatic grah are rereseted, the user rovides the iforatio o other araeters. For exale, the uber of records i each table, the duratio of tie for the evets, the geograhic area (for addresses), the atioality (for aes), ad the tye of ites urchased (eg, airlie, hotels, ad car retals oly). The above araeters further costrai the tye of data geerated. Oce these araeters are iut, the IDSG data geerator odule ca geerate the data sets ad store the i soe secific forat. Iitially, this forat is a coa searated file.. Seatic Grahs ad Related Tables A key ste i our ethodology is the use of a seatic grah to rereset deedecy relatioshis aog differet data sets. hile seatic grahs tyically are used to suarize reexistig data, i our ethodology the seatic grah serves as a guide for the creatio of the sythetic data. The seatic grah is also used to illustrate the data geeratio sequece to be used i the sythesizig rocess. The first hase of the data geeratio rocess is the costructio of the seatic grah. A key assutio i our ethodology is that seatic grahs ca adequately cature the relatioshis withi the data. The IDSG tool uses a seatic grah that describes these data relatioshis. The basic eleets of our seatic grahs are data attributes ad directed deedecies. Ovals i the seatic grah rereset various data attributes, ad arrows idicate the deedecy relatioshi betwee two data attributes. The uber of arrows coig ito a data attribute dictates the diesio of a ier table that holds iforatio about coditioal robability distributios for the data attribute. The coditioal distributio of the data attribute is theoretically secified oce the values of all of the uer level data attributes are kow. Data attributes without icoig arrows such as illustrated i Figure 3a will be assiged a distributio fro either the syste or the user. Sice this tye of data obect does ot have icoig arrows, they are ideedet of other data attributes. The geeratio of this tye of data attribute is doe with sile rado salig.

4 .3 Aroxiatig High-Diesioal Data Deedecies Figure 3. a) No icoig deedecies, b) sigle icoig deedecy, ad c) two icoig deedecies To geerate a data obect with a sigle icoig arrow, such as Figure 3b we will assue its uer level data attribute A is already geerated ad takes the value a. The we ca geerate a realizatio of data attribute B accordig to the coditioal robability distributio of B, give A = a, which is secified i the ier table associated with data attribute B. To geerate a data attribute with ultile icoig arrows, such as Figure 3c we eed to the coditioal distributios of C, give all the ossible cobiatios of values for A ad B. The set of coditioal distributios for C, give A ad B, is show i Table. Table would, i theory, be the ier table associated with data attribute C. The table etry is k robability vector associated with the k levels of C, give A = a ad B = b. Ufortuately, high diesioal distributio tables are difficult to obtai i ractice without doig extesive custoized survey work. The tye of iforatio that is ore easily foud is the set of (weighted) row ad colu averages show i Table as { i} i = ad { }, which corresod to the coditioal = distributios of C, give A ad give B, resectively. I Sectio.3, we describe a aroxiatio to of the for ˆ = αi + ( α ), for 0 α. Clearly ˆ is also a robability vector, ad the fact it deeds oly o the ore easily obtaied row ad colu averages of Table ake the estiator useable. Give A = ai ad B = b, a value of for C is the geerated fro the distributio ˆ. I the ext sectio, we elaborate o what value to use for the ixig araeter α. i As described i the revious sectio, we roose to aroxiate with ˆ = αi + ( α ). These aroxiatios are a iortat art of our IDSG sice obtaiig tables such as Table is difficult i ractice, where as obtaiig is relatively easier. The = iforatio such as { i} i = ad { } otivatio for aroxiatios of the for ˆ ( ) = αi + α icreases whe higher diesioal seatic grahs are utilized. For exale, if a variable D has three iuts, say A, B, ad C, the a 3-diesioal table with etries k would be eeded. Here, k would corresod to the robability distributio for D, give A = ai, B = b ad C = c k. Such a table would be extreely difficult to obtai. However, is is quite coceivable that row, colu ad laar averages { i} i =, { } X r k = ad = { k } would be available ad the ˆ k = αki + βk + ( αk βk ) Xk could be used to aroxiate k. For the 3-diesioal case, there are two ixig araeters 0 αk, βk. I the ext sectio, we describe our ethodology for deteriig the values of ixig araeters, describig the aroach for the case two diesios where a sigle araeter 0 α is required. Extesios to the higher diesioal cases are straightforward. Table. Coditioal Distributios for C, give A ad B A \ B b b b a a Row Average a Colu Average

5 .3. Algorith for Mixig Paraeter The weighted row ad colu averages show i Table are fuctios of weights derived fro the oit distributio for A ad B, which is show i Table. Here, = Pr ( A= a, B= b ). i Table. Joit Distributio of A ad B A \ B b b b a a Margial for A i i a Margial for B i i i It follows that the weighted row ad colu averages show i Table ca be exressed as i i i i i i i i = i + i + + i, i ii ii ii = + + +, I geeral alicatios, the robability vectors ad = (,,, ) will ot be kow. e odel the vectors as ideedet idetically distributed k-diesioal uifor Dirichlet rado vectors, ad as a -diesioal uifor Dirichlet rado vector that is ideedet of all the. I soe sese, our odel corresods to a Bayesia oiforative rior, though our use of the odel is ot withi the tyical Bayesia aradig. Rather, we use this aroach to defie a structural characterizatio of the ad values that we ca otiize agaist. hile we caot exect the Dirichlet assutios to hold across all alicatios, it ca be argued that it is the ost reasoable thig to do i the absece of ay other iforatio. For a give realizatio of ({ }, ) we ca fid the otial value of α by iiizig the total error su of squares Q = αi ( α ). It is easy to show that the iiizig α, over the iterval (0,) is the value ( ) ( i) αˆ = Mi, Max 0,. ( i) ( i) The quatity ˆ α is a rado variable, varyig through the Dirichlet distributios used for the ad. The distributio of ˆ α deeds o, ad k, but does ot deed o either i or. Our choice for the ixig araeter is the ea value of ˆ α, which we obtai uerically by the followig algorith: ) Siulate ideedet realizatios of fro a k- diesioal uifor Dirichlet distributio ) Siulate a oe realizatio of fro a -diesioal uifor Dirichlet distributio 3) Coute ˆ α 4) Reeat stes ()-(3) a large uber (e.g., 0,000) ties 5) Coute the average of the geerated values of ˆ α Table 3 shows the ea values for ˆ α for selected choices of, ad k. Siilar tables ca easily be built for higher diesioal robles. For exale, i the case where three iuts A, B ad C drive the value of a 4 th variable D, the table would have colus for,, r ad k, where r reresets the uber of levels for C ad k is the uber of levels of D. The algorith described above is easily exteded to coute the ea values of the two ixig araeters ˆk α ad ˆk β. Table 3. Mixig Paraeter as a Fuctio of (,,k) k Mea of ˆ α Illustratio e ow illustrate the rocedure described i the revious sectio with a sile exale ivolvig exediture atters as a fuctio of Age ad Geder. I this exale, Age ad Geder corresod to the variables A ad B, resectively. The variable C, exediture tye, is a two-level categorical variable that idetifies a exediture as either ecessary (N) or uecessary (U). Figure 4 is a aroriate seatic grah for this exale, ad we have =3, =, ad k=. Our obective is to siulate a value for C, give we have already siulated values for A ad B. I order to ileet our ethodology, we eed the two tables show i Table 4 ad Table 5. Table 4 gives the roortio of N ad U exeditures for each geder, ad Table 5 gives the

6 roortio of N ad U exeditures for each of the three age grous. Table 4. Probability of N ad U Exeditures For Geder Male Feale = Pr ( C = U) = Pr ( C = U) Table 5. Probability of N ad U Exeditures For Age Grou Age Grou Age Grou Age Grou 3 = Pr ( C = U) = Pr ( C = U) = Pr ( C = U) Suose we wated to siulate exediture tye for a Male ( A = a ) i Age Grou ( B = b ). Referrig to Table 3, we fid the ixig araeter for ( k,, ) = (,3,) is e would thus estiate usig ˆ Pr( C = N Male, Age-Grou ) ( 0.44) = ˆ Pr( C = U Male, Age-Grou ) = ( 0.44) A siulated value of C, give A = a ad B = b, is the obtaied by radoly accordig to the robabilities give by ˆ ad ˆ. versios of IDSG ay iclude database forats as outut. This decisio will be ade at soe oit i the future whe we have ore kowledge of user s IDAS ad ore develoet tie. 3. ILLUSTRATIE APPLICATIONS 3. Nae Geeratio I order to geerate aes, we first eed to fid a aes database that we ca sale fro. e obtaied our sets of aes fro olie source such as the Cesus Bureau. The aes are the stored i the aster database as ale first ae, feale first ae, ad last ae i searate files. Siilar files are created for each atioality. he the user rus the geerator, it will go through the database ad radoly ick out a last ae, the cobie it with either a first ae fro the ale list or the feale list to geerate a full ae with geder. The user of the tool ca also secify which atioality of origi they wat the aes, which will geerate a ore realistic dataset. The geerator will go through the database ad cobie last ae files fro secific atioality with first ae files to geerate the desire atioality. This rocess ca be illustrated by usig a seatic grah (Figure 6). Natioality Geder Last Nae First ae Figure 4. Nae Geeratio Scheatic. 3. Credit Card Nuber Geeratio To geerate realistic credit card ubers, we use the seatic grah show i Figure 7. Card Tye.4 Outut Forats The IDSG ust be caable of oututtig geerated sythetic datasets i ortable forats. These data sets ay be stored i volues of CDs or DDs. The datasets will be dyaically geerated based o user secified araeters with otetially very large size which eas that ortability, silicity, ad flexibility are the three ai requireets. Storage of outut data i a database, i flat files/cs, or XML are soe of the ai choices. Storage of the outut i a database has eough flexibility to allow the user to reserve all data relatioshis ad also to otiize the dataset size. But a Alicatio Prograig Iterface (API) will be required for the IDAS, which icreases the colexity ad tie of syste develoet. e have chose coa searated values (CS) as the iitial choice of IDSG for the outut forat sice it is ortable, sile ad flexible. However, this requires the outut of all data i flat forat to reserve the data relatioshis. This will icrease the dataset size sigificatly. This advatage ca be offset by chea storage edia ad requirig oly oe-tie iortig to a IDAS. Future Card Issuer MII Credit card Nuber Figure 5. Credit card uber seatic grah The first digit o a credit card is the Maor Idustry Idetifier (MII) which reresets the source fro where the credit card was issued. There are te issuer categories, but for our urose we will oly use four of the for radoly geeratig a credit card uber. For exale, a credit card uber startig with 6 is assiged for erchadisig ad bakig uroses, such as i the case of the Discovery card. Credit card ubers startig with 4

7 ad 5 are used for bakig ad fiacig uroses, as i the case of isa ad MasterCard. The last issuer category we used is 3 to rereset travel ad etertaiet used, for istace the Aerica Exress card. Table 9 is a overview of the rules for uberig credit card. The first six ubers icludig the MII reresets the issuer idetifier. The rest of the digits o the credit card rereset the cardholder s accout uber. Issuer Table 6. Credit Card Paraeters Idetifier Discovery 60xx 6 Mastercard 5xxxx-55xxx 6 Aerica Exress 34xxx, 37xxx 5 isa 4xxxxx 3, 6 Legth (Nubers) exediture category for a 30-year old erso with a icoe of $50,000. A uifor rado uber geerator ca be used to radoly ick oe of the categories accordig to the robabilities ˆ. i 8 Table 7. Detailed Joit Distributio of Exediture by Age. The credit card uber geeratio rocess begis by radoly geeratig the card tye (Master, isa, Aerica Exress, or Discovery) deedig o the robability distributio for the tye of cards i circulatio. After a card tye is icked, a aroriate MII uber is assiged to the card, which icludes a radoly saled uber, varyig i legth fro to 5 digits log, deedig o the issuer tye. The fial ste i the rocess is to geerate a rado strig of ubers whose legth deeds o the card tye is geerated. e use the Luh algorith to check whether or ot the credit card uber we radoly geerated is a valid uber or ot. To use this algorith, we first double the value of alteratig digits of the card uber, startig with the first digit. Next, we subtract 9 fro those ubers that are greater tha 9. Fially, we su the trasfored ubers together ad the divide by 0. If there are o reaiders, the card uber is cosidered valid. Thus, this is a valid visa credit card uber sice there are o reaiders. 3.3 Credit Card Trasactio Geeratio I this sectio, we illustrate the geeratio of oe articular asect of a credit card urchase trasactio. Our exale akes use of Figure 4 which shows how exediture category (variable C) deeds o age grou (variable A) ad icoe level (variable B). Exediture category has 7 levels: Food, Housig, Aarel, Trasortatio, Health Care, Etertaiet, ad Other. Table 7 shows the robability of each exediture category as a fuctio 7 of Age Grou. The colus of Table 7 corresod to the { i } i= vectors i Sectio. Table 8 shows the robability of each exediture category as a fuctio of Icoe Level. The colus 9 of Table 8 corresod to the { } = vectors i Sectio. Tables 7ad 8 were obtaied fro the U.S. Cesus Bureau. To siulate a exediture category for a 30-year old erso with a Icoe Level of $50,000, we use the algorith described i Sectio.3 takig ( k,, ) = (6,9,7) to fid that the ixig araeter is The estiated robability distributio for exediture category is thus ˆ 8 = ( 0.47) 8. The eleets of i rereset the estiated robabilities of each Table 8. Joit Distributio of Exediture by Icoe. 4. SUMMAR e have described a desig ad architecture for a test data geeratio tool that ca facilitate buildig test cases for IDAS tools. Our tool eables a IDAS develoer to overcoe tie ad cost issues associated with gatherig real data to build test cases by akig sythetic data available. Sythetic test data of sufficiet quality is a viable ecooic alterative to gatherig real data sets, which i ay cases ight ot eve be ossible. The develoet iterval for our tool exteds through the caledar year 005. e aticiate our data geeratio latfor will lay a iortat role i coarig the effectiveess of differet IDAS. By overlayig secific sceario o our backgroud data sets, a IDAS evaluator ca easure coetig IDAS i ters of false egative ad false ositive error rate. 5. REFERENCES [] Abowd, J.M. ad Lae, J.I. Sythetic Data ad Cofidetiality Protectio. U.S. Cesus Bureau, LEHD Progra Techical Paer No. TP-003-0, (003) [] Cha, P.K., Fa,., Prodroidis, A.L., ad Stolfo, S.J. Distributed Data Miig i Credit Card Fraud Detectio. IEEE Itelliget Systes 4(6), (999). [3] Deartet of Defese, Office of the Isector Geeral. Iforatio Techology Maageet: Terroris Iforatio Awareess Progra. Reort No. D (004). [4] Geeral Accoutig Office, Data Miig: Federal Efforts Cover a ide Rage of Uses. GAO (004).

8 [5] Kusiak, A., Kerstie, K.H., Ker, J.A., McLaughli, K.A., ad Tseg, T.L. Data Miig: Medical ad Egieerig Case Studies. Proceedigs of the Idustrial Egieerig Research 000 Coferece, Clevelad, Ohio, May -3, (000), -7. [6] Leskovec, J. Grobelik, M., ad Millic-Fraylig, N. Learig Sub-structures of Docuet Seatic Grahs for Docuet Suarizatio. LikKDD 004, August 004, Seattle A, USA. (004). [7] Orerod, T., Morley, N., Ball, L., Lagley, C., ad Seser, C. Usig Ethograhy To Desig a Mass Detectio Tool (MDT) For The Early Discovery of Isurace Fraud. CHI 003, Aril 5-0, 003, Ft. Lauderdale, Florida, USA. ACM /03/0004 (003) [8] Price, E., ad Nicholso,.L. A Test of a Robust/Resistat Refieet Procedure o Sythetic Data. Acta Cryst., A39, (983), [9] Rogers, M. Graha, J., ad Toge, R.P. Usig Statistical Iage Models for Obective Evaluatio of Sot Detectio i Two-Diesioal Gels. Proteoics, Jue, 3(6) (003), [0] arga, T. ad Buke, H. Geeratio of Sythetic Traiig Data for a HMM-based Hadwritig Recogitio Syste. Proceedigs of the Seveth Iteratioal Coferece o Docuet Aalysis ad Recogitio (ICDAR 003) /03 IEEE Couter Society (003). [] iddows, D. ad Dorow, B. A Grah Model for Usuervised Lexical Acquisitio. 9 th Iteratioal Coferece o Coutatioal Liguistics (COLING 9). Taiei, August (00) [] u,.t., Stefaova, L., Mitra, A.K., ad Krishaurti, T.N.. Multi-Model Sythetic Suereseble Predictio Syste. Acta Cryst., A39, (983), [3] Zhu, X., Aref,.G., Fa, J., Catli, A.C., ad Elagarid, A.K. Medical ideo Miig for Efficiet Database Idexig, Maageet, ad Access. IEEE It. Cof. O Data Egieerig (ICDE 03), Bagalore, Idia, March 5-March 8, (003), -.