Demonstrating the ROI of Data Discovery

Transcription

1 WHITE PAPER: DATA DISCOVERY AND PROFILING Demostratig the ROI of Data Discovery OCTOBER 2009 Malcolm Chisholm Ph.D. idepedet cosultat

2 Table of Cotets Executive Summary SECTION 1 2 The Growig Data Crisis What are Data Discovery ad Profilig? Source Data Aalysis The Mirage of Legacy Data Quality SECTION 2 6 Ehacig Data Models The Role of Techology SECTION 3 8 Preparig a Busiess Case SECTION 4: CONCLUSIONS 8 SECTION 5: REFERENCES 9 SECTION 6: ABOUT THE AUTHOR 9 Copyright 2009 CA. All rights reserved. All trademarks, trade ames, service marks ad logos refereced herei belog to their respective compaies. This documet is for your iformatioal purposes oly. To the extet permitted by applicable law, CA provides this documet As Is without warraty of ay kid, icludig, without limitatio, ay implied warraties of merchatability or fitess for a particular purpose, or oifrigemet. I o evet will CA be liable for ay loss or damage, direct or idirect, from the use of this documet, icludig, without limitatio, lost profits, busiess iterruptio, goodwill or lost data, eve if CA is expressly advised of such damages.

3 Executive Summary Challege As data cotiues to grow i importace to eterprises worldwide, a glarig problem with its quality ad accessibility has come to light. Orgaizatios are strugglig to idetify all viable data sources ad leverage it to aswer such fudametal questios as what is the umber of curret customers? As seior maagemet, busiess users ad IT grow icreasigly frustrated at the iability to fully leverage corporate data, may are lookig for a better solutio. Opportuity Data discovery ad profilig is a relatively ew data maagemet techique that uses tools to idetify all data sources throughout the orgaizatio ad uearth the meaig of the iformatio itself. As with ay ew paradigm, today s budget holders ad decisio makers must first be educated o data discovery ad profilig ad the show the value it ca brig to the orgaizatio through a clear retur o ivestmet (ROI) aalysis. Beefits The growig eed to access ad leverage data that is hidig withi may disparate systems ad applicatios ad across the highly complex, disjoited IT ifrastructures commo i today s eterprises, has made data discovery ad profilig a ecessity. Data discovery ad profilig tools help complemet may data maagemet techologies i use today ad help improve efficiecies ad effectiveess, while mitigatig risk. WHITE PAPER: DATA DISCOVERY AND PROFILING 1

4 SECTION 1 The Growig Data Crisis Durig the past forty years, eterprises have bee gradually implemetig complex ifrastructures for iformatio processig. Origially, the focus was o automatig busiess processes. Today, however, it has shifted to data. While data was oce viewed as merely the by-product of automatio, it is ow recogized as a valuable resource i its ow right. It ca be used to do thigs that were uthikable a few decades ago, such as measure the efficiecy of busiess processes, yield precious marketig isight ad show the results of busiess decisios. At least that is the promise. I reality, the ad-hoc, orgaic growth of eterprise IT architectures ad lack of meaigful data maagemet have created productio data ladscapes of almost uimagiable scale that are almost opaque. I fact, the followig geeral questios caot always be reliably aswered: What data does the eterprise maage? What does it mea? Where it is located? Worse, may orgaizatios are strugglig to ascertai key busiess data, such as the umber of curret customers. Ad, customer details are ievitably distributed over may databases, but few if ay kow exactly which oes. Iformatio may eve be hidde i trasactio records, sice compaies put more emphasis o havig a ew customer complete a trasactio tha properly capturig all of his or her details i a separate customer record. Eve where a specific customer database ca be idetified, the data cotet is ofte plagued with such problems as duplicate records. The full cosequeces of this crisis are ot yet completely uderstood, but they are becomig clearer. Busiess users are icreasigly frustrated that the iformatio they eed caot be provided. Ad IT departmets are strugglig to itegrate eterprise iformatio assets i order to aswer busiess eeds but fid that such projects cosume a very high level of resources, delivery is delayed ad outcomes are ofte plagued by data quality issues. Oe prove way to mitigate these problems is through the use of data discovery ad profilig. However, it is a relatively ew data maagemet techique. As a result, seior maagemet may be ufamiliar with it ad uable to readily uderstad its value. This paper provides the iformatio eeded to clearly demostrate the retur o ivestmet (ROI) for discovery ad profilig. What are Data Discovery ad Profilig? The first steps to explaiig the ROI of data discovery ad profilig to seior maagemet ivolve describig what it is, why it is eeded ow ad why it was ot eeded i the past. Because discovery ad profilig are ew ad may ot be ituitive, it s importat to esure that this message is always cosistet. This is particularly true for seior executives outside of IT, who ievitably focus o busiess value. For them, discovery ad profilig ca be explaied as follows: Data Discovery. I most orgaizatios, the size ad complexity of the productio data is so large that o oe kows where critical busiess iformatio is located. Yet, the busiess demads this iformatio be provided i cosolidated ad aggregated forms. With thousads of database tables, ad hudreds of thousads of database colums i a average mid-sized eterprise, it is virtually impossible to do so maually. Huma aalysts simply caot look at data o this scale to fid customers, products, sales ad so o, to figure out how to brig the data ito itegrated eviromets where it ca be accessed to satisfy busiess iformatio eeds. WHITE PAPER: DATA DISCOVERY AND PROFILING 2

5 However, a data discovery tool ca overcome this problem of scale because it ca sca large eviromets ad idetify data i a fractio of the time required by a team of huma aalysts. This tool offers a much greater chace of fidig the best sources of critical busiess data. Data Profilig. Eve if huma aalysts were to fid a particular source of critical busiess data i the ladscape, they still have to uderstad it before they ca use it for busiess itelligece purposes. This is remarkably difficult because most data is locked i vedor-supplied packages with proprietary database desigs or home-grow legacy applicatios whose databases are equally mysterious. Uless the meaig of the data ad its relatioships i these eviromets are properly uderstood, it is impossible to itegrate it to produce the kid of actioable reportig the busiess demads. Data profilig tools look at the cotet of the data itself to fid relatioships ad patters makig it much easier to uearth the meaig behid the iformatio. Profilig tools also work o a far greater scale tha a team of huma aalysts tryig to browse the data to maually ifer relatioships. Additioally, data profilig tools ca quickly spot icosistecies i the source data. Before data profilig, these icosistecies ofte appeared uoticed i data warehouses ad marts. The data problems were evetually foud whe busiess users discovered aomalies i their reports. All too ofte, such episodes reduced trust i the reportig eviromets, but left busiess users without a reliable alterative. Source Data Aalysis Defiig discovery ad profilig is just the first step. Seior maagemet will wat to kow how these techiques ad the tools that implemet them ca either save or make the eterprise moey. This is the core of ay demostratio of ROI, ad oe area i which it ca easily be doe is source data aalysis (SDA). SDA was rarely eeded i the early days of IT, because applicatios were beig developed to automate existig busiess processes, rather tha reprocess data that had already bee populated i databases. For this reaso, it ever appeared i such programmig-cetric methodologies of IT as waterfall or agile. At the time, aalysis i these activities cosisted of lookig at documets, or talkig to users, to gather requiremets ad uderstad processes. But SDA is completely differet. It aims at providig a sufficiet uderstadig of the cotet of existig data sources that are to be itegrated i order to satisfy busiess requiremets. What s more, it simply caot be doe by a few huma aalysts i the same way that traditioal aalysis ca. ESTIMATING SOURCE DATA ANALYSIS It is fairly easy to determie the time required for SDA. Table 1 provides estimates of the size of small, medium ad large eterprise database eviromets. The estimates i the table are take from the author s experiece i workig o projects ivolvig limited maual SDA. They ca be substituted by estimates for a give eterprise, but because of the ormal large scale of data ladscapes, the coclusios will be more or less the same. Table 1 shows that, except for the very smallest eterprises, the work required to perform maual SDA o ay itegratio project is very sigificat. For example, a itegratio project that requires 2,000 colums to be aalyzed would require 5,000 work hours. At a cost of $100 per aalyst work hour this would total $500,000. It would require a team of three aalysts workig for about 1 year to complete the task. WHITE PAPER: DATA DISCOVERY AND PROFILING 3

6 Table 1: Volume Estimates for Maual Data Discovery ad Profilig Volume Metric Small Medium Large Number of Productio Databases ,000 Average Tables per Database Average Colums per Database Average Rows per table 10,000 20, ,000 Total Tables 200 5,000 75,000 Total Colums 1,600 75,000 2,250,000 Total Rows 2,000, ,000,000 7,500,000,000 Average Aalyst Profilig Effort (Work Hours to Maually Discover ad Profile a Sigle Colum) Work-hours to Maually Discover ad Profile All Colums 1, ,500 9,000,000 Cost per aalyst work-hour $100 $100 $100 Total cost for Maual Discovery ad Profilig (potetial savigs for automatio) $160,000 $18,750,000 $900,000,000 Average Number of Colums per Itegratio Project 150 2,000 5,000 Work-hours to Maually Discover ad Profile All Colums 150 5,000 20,000 Cost per aalyst work-hour $100 $100 $100 Total cost for Maual Discovery ad Profilig for Itegratio (potetial savigs for automatio) $15,000 $500,000 $2,000,000 WHITE PAPER: DATA DISCOVERY AND PROFILING 4

7 Clearly, this is uacceptable ad does ot happe o actual projects. Ufortuately, because SDA is so ew, it is rarely budgeted for explicitly ad facts about costs are difficult to glea. For example, durig a itegratio project for a major fiacial compay that had spet 30% of its budget o maual SDA, this level of effort was uforesee ad caused a major project overru. It oly occurred after serious aomalies were foud i the data ad the SDA effort was stopped after six moths because it was deemed to be holdig up the project (which was ultimately usuccessful). Uderscorig the lack of plaig ad budgetig for SDA, a seior maager i a leadig discovery tool compay asserted that i his compay approximately 30% of all resources o data itegratio projects are devoted to SDA. He too felt that this was ot early eough to do the job well if it had to be doe maually. If the task of SDA could be automated i ay way, the ivestmet i maual labor could be reduced. Data discovery ad profilig tools provide this capability. If the above estimate is correct for a average data itegratio project i a eterprise with a medium-sized data ladscape, the oe project would easily pay for the acquisitio of such a tool. It is also worth otig that today most aalysis is actually performed o a ifrastructure that IT has already implemeted. I fact, programmers sped a lot of their time reverse-egieerig existig code ad databases estimated to be more tha 50% accordig to a 1996 report by CASE Associates 1. Maagemet is likely aware that aalysts must sped time tryig to uderstad existig applicatios ad databases which should be take ito cosideratio durig ay discussio of ROI for data profilig. The Mirage of Legacy Data Quality Oe issue that sometimes makes it difficult to propose data profilig to CIOs ad other seior maagemet is a feelig that data quality may be better tha it actually is. All CIOs are aware of the status of their productio applicatios ad typically receive daily reports o aythig that impacts the service levels of these applicatios. Such reports usually iclude few istaces of data quality problems that cause service outages. As a result, data profilig is assumed to be uecessary; the fact that legacy applicatios are ruig smoothly is iterpreted as cofirmatio that there must be relatively few data quality problems. This is the mirage of legacy data quality. It is caused by the fact that, over the years, program logic ad productio data cotet have become boud together i legacy applicatios. The program code filters, trasforms, ehaces ad relates applicatio data to fit every output that the applicatio produces. The data is ot used as is, but oly as redered by the applicatio. Therefore, just because the legacy applicatio works does ot mea that data quality problems are abset, oly that they are beig hadled. Whe data is moved out of this eviromet, data quality problems are o loger addressed, ad they become apparet. Of course, may CIOs realize this ad are oly too aware of the eed to profile legacy data before it is used i differet situatios. However, the lack of visibility ito legacy data problems makes it impossible to gauge just how bad the problem really is, ad may lead to the coclusio that profilig is ot really ecessary. This ca result i a headlog rush ito data itegratio projects i which legacy data is pulled ito data warehouses or MDM hubs or used i messagig i ear real time amog operatioal systems. It is much later that the problems iheret i legacy data are properly uderstood. But by that time, a uacceptable level of resources has bee expeded. Whe the mirage of legacy data quality is a problem, the data aalyst must fid a way to covicigly demostrate a eed for data profilig before ay itegratio project becomes too advaced. This is a difficult task, because i such situatios the aalyst is ulikely to have a eterprise-class data profilig tool available ad maual profilig has prove to be too time cosumig. However, the aalyst does ot eed to do a full-scale profilig exercise. A sample of data is all that is eeded to idetify data quality issues. Next, a scalig factor ca be used to estimate the full scope of data quality problems ad provide a icotestable case for full profilig of the legacy data. Today, there are low-cost data profilig tools available, so the aalyst eed ot ecessarily rely o a large, eterprise-class solutio. Ad if the low-cost tools are ot available, a aalyst with reasoable kowledge of the data may still be able to costruct a sample set usig SQL queries. WHITE PAPER: DATA DISCOVERY AND PROFILING 5

8 Table 2 shows how results from such a samplig approach might be preseted to seior maagemet. I order to estimate the gross umber of quality problems likely to occur, the umber of errors foud is multiplied by the ratio of the data objects sampled to the total. TABLe 2: PRESENTATION OF RESULTS FROM A SAMPLING APPROACH TO ESTIMATING DATA QUALITY PROBLEMS Volumes Number of Tables Number of Colums Data Quality Test Results Overloaded Colums Code Colums without Paret Code Tables Colums with Orphaed Child Values Sampled Number Foud i Sample Total Estimated to Occur David Sharo, Meetig the Challege of Software Maiteace, IEEE Software, vol. 13, o. 1, pp , Jauary, SECTION 2 Ehacig Data Models A importat part of data profilig is buildig data models that accurately represet the uderlyig databases. However, i eterprises where data modelig has ot bee performed i the past, this may ot be see as helpful, makig it difficult to get seior maagemet buy-i. Yet, eve i eterprises that have doe data modelig, there may still be a eed to covice maagemet of the eed for profilig. Databases are very expesive pieces of ifrastructure. Ideally, logical data models would represet them perfectly ad physical data models would specify them exactly. However, they rarely do. Models may be etirely missig, the logical may ot match the physical or the physical may ot match the database. Profilig ca help tremedously i these areas by producig completely reliable data models that are reverse-egieered from productio databases. Still, this may ot resoate with maagemet i some eterprises. They may cotiue to questio the eed for data modelig, sice it was ot eeded before. However i the past, over a period of decades, each silo was built idividually ad was self-cotaied. Thus, it was possible, albeit poor practice, for idividual applicatio teams to uderstad the represetatio ad specificatio of these databases as they were beig built without retaiig or passig o this kowledge. WHITE PAPER: DATA DISCOVERY AND PROFILING 6

9 Such kowledge is usually lost if ot cotaied i data models. What s more, through the years, applicatios get ehaced, modified, fixed ad patched a little bit at a time, resultig i eve more drift away from the origial desig. While all this is happeig, persoel come ad go, ad kowledge that is maitaied tribally is imperfectly trasmitted if at all. The result is that applicatios ievitably evolve ito black boxes ad obody is sure what is goig o iside. Eve a small chage is risky ad ivolves a disproportioate amout of reverse-egieerig. Chages to database structures are particularly feared. This argumet is uderscored by the fact that the older a applicatio, the less it is uderstood. I fact, cotrary to most experiece, the best uderstood applicatios are always the ewest. Agai, it would ot matter if black box databases were t eeded for aythig beyod themselves, or if there were just oe or two of them. But data itegratio casts a wide et, ad data is ultimately eeded from all trasactio applicatios. Furthermore, the cumulative set of databases that has bee built up over decades eeds to be cosidered. This is completely differet from sayig that a data modelig tool was ot eeded whe teams built or implemeted oe or two databases per year. Tryig to uderstad the structure of a black box database, either to ehace it or to use it as a source of data, meas developig a data model of the database i questio. The least efficiet ad least effective way of doig this is to rely o huma aalysts to somehow uderstad, remember ad commuicate the structure of the database. No eterprise will have the aalyst resources to do this for more tha oe or two databases at a time. If data models are eeded to uderstad databases, ad there is a scarcity of aalyst resources, the there is o alterative but to use data profilig tools to automatically geerate sharable data models of the databases. The Role of Techology I helpig seior maagemet uderstad the ROI for data maagemet, it may also be ecessary to overcome the perceptio that existig techology ivestmets ca solve all data-related problems. The fact is that techology has created a great deal of the data mess i which most eterprises fid themselves. For istace, there are may classes of tools that ca quickly ad efficietly move data aroud ad they work very well ad have bee used extesively. As a result, however, itegratio eviromets have bee plagued by data quality problems that wet urecogized due to the lack of data profilig. It ca be difficult to make the case that aother piece of techology a profilig tool is required to make the ivestmet i other tools pay off. The vedors of data movemet tools, data warehouse products, MDM hubs, BI tools ad the like will ot ecessarily raise the eed for a data profilig tool. I fact, they may eve dowplay it i order to make a sale. What s more, they may claim that their tools have some features that assure data quality, such as guarateed message delivery i a middleware product. The role of such claims eeds to be carefully uderstood whe makig a case for the ROI of data profilig. Seior maagemet may ot metio the claims they have heard from vedors of other products, so it is importat to place data profilig tools i a product matrix that shows how they will complemet tools curretly i use. More to the poit, without data profilig, the ivestmet i these other tools may well be wasted. If possible, the egagemet with seior maagemet should attempt to flush out ay argumets or assumptios about how existig tools ca meet data profilig use cases. WHITE PAPER: DATA DISCOVERY AND PROFILING 7

10 SECTION 3 Preparig a Busiess Case Gatherig the iformatio about ROI for data profilig is oe thig, but successfully presetig it as part of a well thought out busiess case is aother. The followig three mai areas should be covered i ay busiess case that attempts to demostrate ROI: Efficiecy. Efficiecy is the reductio of existig costs, or the act of doig more with the curret set of resources. This part of the busiess case is the easiest to preset i quatitative terms, ad maagemet expects to see quatificatio i at least oe major area for ROI. Developig the kids of matrices show i Table 1 ca be helpful here. Effectiveess. Effectiveess is harder to quatify tha efficiecy. Basically, it meas eablig the eterprise to meet its stated goals. Today, IT is see as a major problem i adjustig to ew busiess eviromets, because the architecture that IT has developed over the decades is brittle ad iflexible. I this part of the pla, it is ecessary to show how data profilig ca cotribute to agility. Risk Mitigatio. This part of the busiess case is frequetly overlooked. Some quatificatio may be possible if iformatio is available about losses caused by data-related problems that would have bee picked up by profilig. A more direct, but possibly cotroversial, approach is to look at IT projects that have failed to meet expectatios because of lack of data profilig. Obviously, this should be hadled tactfully. Presetatio is importat because data is full of abstract cocepts that are difficult to grasp. The busiess case should be as easy for the iteded audiece to read as possible. There is o eed to dwell o techical details. Seior maagemet has may demads o its time ad attetio, so presetig the case ad ROI as clearly as possible will be much appreciated. SECTION 4 Coclusios While the eed for Data Discovery ad Profilig tools has become a ecessity, the ability to successfully prove its value to seior maagemet remais a challege. The ROI techiques preseted i this paper ca be used to help executives uderstad the eed for this ew paradigm, appreciate its place withi the existig arseal of IT tools ad look forward to reapig the beefits of it. For more iformatio about how CA ERwi Data Profiler ca help your busiess icrease the quality of your data by performig cross system aalysis ad profilig of both database ad legacy systems, visit ca.com/modelig or for local iformatio, go to ca.com/cotact/rmdm. WHITE PAPER: DATA DISCOVERY AND PROFILING 8

11 SECTION 5 Refereces 1. David Sharo, Meetig the Challege of Software Maiteace, IEEE Software, vol. 13, o. 1, pp , Jauary, Malcolm Chisholm Ph.D. Idepedet Cosultat SECTION 6 About the Author Malcolm Chisholm Ph.D. is a idepedet cosultat with over 25 years of experiece i IT, ad has worked i fiace, maufacturig, govermet, defese, telecommuicatios, pharmaceuticals ad isurace. He is author of the books Maagig Referece Data i Eterprise Databases (Morga Kaufma, 2000) ad How to Build a Busiess Rules Egie (Morga Kaufma, 2003). Malcolm is a thought leader i the fields of master data maagemet ad data goverace, ad writes ad speaks ofte o these topics. He maitais the websites com, ad ad ca be cotacted at DataGoverace@gmail.com. WHITE PAPER: DATA DISCOVERY AND PROFILING 9

12