White Paper. Agile data management with X88

Transcription

1 White Paper Agile data management with X88 A White Paper by Bloor Research Author : Philip Howard Publish date : June 2011

2 This paper is a call for some more forward thinking from data management practitioners and vendors and a move towards a more agile stance for the development of data management rules. One of the few companies that has recognised this deficiency and has responded to it is X88. Philip Howard

3 Introduction This paper does not concern itself with the benefits of using data management (data quality, data integration, data governance, data enrichment, data transformation, test data management and so on) tools and processes. That is assumed. It is also assumed that you understand that these are not one-off exercises but ongoing requirements. Further, we are going to make the additional assumption that you realise the benefits of automating as much of your data management processes as possible. This is on the basis that the more that is automated then the less manual intervention will be required and this will be both faster and more cost-effective. Of course, such processes cannot be entirely automated but one would like to do as much as one can. The way that you automate data management processes depends on the type of management under consideration and the type of tool in use. Here we need to make a distinction between test data management on the one hand and data quality and data integration processes on the other. If we consider agile development in general, the reason why this has not been as successful as it might have been is precisely because development may have been agile but the data to support it has not. During application development, data rules and data models often change but when you have such changes you need to re-generate relevant test data. In practice, using traditional techniques, a typical lifecycle for such re-generation is measured in weeks when you would really like it to be in hours. The issue of agile data management with respect to test data is, precisely, the data itself, and the automation of the re-generation of that data. project. If you catch an error early on it is much easier and quicker to rectify it at that stage than after you have completed development when there may be other complications caused by that error. In particular, the practice of testing early and often includes user acceptance testing, which prevents the sort of specification mismatch that can occur when what the user meant wasn t actually reflected in the specification and/or wasn t correctly interpreted by the developer, or worse, the user was mistaken about what was needed. In other words, we are arguing that the development of data management rules should be treated in exactly the same way as any other development project. Further, it is our view that an agile development methodology is particularly well suited to both data quality and data integration environments. This is because users should be heavily involved in the project in any case, either because they are sponsoring the project directly or because it is part of a larger data governance initiative, data migration or data warehousing project. This paper will therefore discuss the issues raised by data management rules and their development, which are actually common to rules-based environments in general, and consider potential solutions. We will also discuss X88 s Pandora product, with particular reference to how that product supports data prototyping in an agile environment. In so far as data quality and data integration tools are concerned these are either rulesbased or semantic. The former dominate the market and it is with these that this paper is concerned and, since it is the rules themselves that provide whatever degree of automation is offered, it is on the creation and subsequent maintenance of these rules that this paper is focused. In our view, not enough attention has been paid to rule definition and management by (most) vendors in this space. In particular, we will argue that an agile approach to rules development should be adopted, for precisely the same reasons that it makes sense to use an agile approach to development in general. That is, that it is more productive to test early and often rather than at the end of a development A Bloor White Paper Bloor Research

4 The issues with rules What does a data management rule do? It takes data from one or more sources, manipulates it and then puts the result/data somewhere. The key is in the manipulation: you may improve values through data quality processes, you may derive values to load into an OLAP cube, you may enrich the data with geo-spatial coordinates, you may apply complex business logic, you may look for correlations between different pieces of data, you may in fact, do almost anything. Clearly, different types of data management tools have widely different inbuilt capabilities. Data integration tools, for example, typically come with a variety of pre-built functions so that you can add, concatenate and apply statistical functions to incoming data. However, these are not rules but merely functions that can be used to help in the development of business rules. Data quality tools, on the other hand, are typically delivered with various out-of-thebox rules that can be applied generically and which do not need to be developed. For example, you might have a rule that a credit card number has to be of the format xxxx xxxx xxxx xxxx where each x is a number between 0 and 9. Similarly, you might have pre-built rules about the validity of postal codes. There is no problem with rules such as these: credit card formats are unlikely to change and, as long as you keep up-to-date with PAF (postal address files) then post code validation should be automatic. However, when it comes to rules that are specific to your business then issues can become much more complex. In some cases you may be able to customise rules that are provided out-of-the-box by the vendor but in many others you may have to build appropriate rules from scratch. As a general principle we can say that you will always have to develop appropriate rules for data integration tasks and you will almost always have to do so (at least for any non-trivial tasks) when using data quality tools. As an illustration of this consider the following reallife example from the healthcare sector. The organisation involved is owned by nearly 200 non-profit hospital and healthcare systems servicing more than 2,000 hospitals and over 53,000 other healthcare sites across the United States. Its primary mission is to drive down healthcare costs while improving the quality of patient care. What the company wants to do is improve its supply chain management. In order to do this it needs to cleanse and de-duplicate its supplier master records (not simple: 3M, for example, has more than 200 valid variations of its name, including subsidiaries), cleanse and de-duplicate its product records (very complex, because the same product can be supplied by different distributors with different product codes and different descriptions) and extract relevant cleansed data from source systems, transform it and then load that data into the data warehouse for analysis. At the same time we can assume that the company in question has recognised that simply fixing its data quality issues as a one-off project is not enough and has implemented a data governance project across the enterprise. This involves, amongst other things, monitoring both data quality and compliance with relevant regulations. The results are reported within a dashboard that displays appropriate KPIs (key performance indicators). Where do these KPIs come from? They might, as an example, represent weighted aggregations of data quality metrics. In other words, you gather raw data quality scores and then apply relevant business rules to these figures in order to calculate these KPIs. Or, as another example from within the healthcare sector, you might need to monitor that sensitive patient data is being appropriately masked or redacted so that unauthorised persons cannot view that data. Here again, you would need to define relevant business rules, monitor actual practice and report the results in a KPI. In other words, there are a whole series of sets of business rules that need to be developed and tested that range from traditional data quality rules through transformation rules to the calculation of KPIs. Moreover, these data management rules, whether for data quality or data integration or for monitoring data governance or compliance, can be highly complex. Moreover, it is only a business analyst or domain expert that is likely to understand what rules are appropriate to the business but, and here is the rub, you will likely need a developer to actually build the rule. This leads us onto the issue of exactly how you develop data quality, monitoring, transformation and other data management rules. In practice, development follows a traditional paradigm: a specification from the business, followed by development using the tool in question, followed by acceptance testing. Bear in mind that we are not just talking about one-off Bloor Research A Bloor White Paper

5 The issues with rules rules but perhaps thousands. Moreover, there may be relationships within the data that need to be maintained: if you have accounts with both Federal Express and FedEx, you may want to match and then merge these but you also need to ensure that any related orders (both current and historic) still retain their relationships with the merged entity. The problem with this approach to development is twofold. Firstly, there are the commonplace issues that bedevil all specifications: that business and IT do not fully understand each other, typically speaking different (technical) languages; the business changes its mind; both sides can make errors and mistakes; and there is often ambiguity in the specification. Secondly, if there is a specification mismatch you won t find out about it until the very end of the process when you perform acceptance testing, which is the most expensive time to find out. To put this another way: this is not an agile approach. A further problem is that when you have thousands or tens of thousands of data quality rules, or any sort of rules for that matter, then maintenance of those rules can become an issue: you can end up spending so much resource maintaining existing rules that you do not have time, or very little time, to develop new rules, and this may mean that, for example, you cannot extend your data management processes to new business domains as fast as the organisation would like. It is therefore important that the environment enables rule maintenance in as easy to use a manner as possible, as well as providing an optimal environment for the development of new rules. Next, there is the question of where the development takes place. You develop new rules within whatever tool you are using but what do you test this against? The live data? That would be dangerous and, besides, it will put an additional strain on the operational system, which will be counter-productive. Alternatively, you may take a copy of the operational system or a subset of it, as you would do in conventional testing environments. Unfortunately, this has other consequences: it means that you will need extra licenses for the development and test systems, as well as extra hardware to run these on. Finally, there is the question of sensitive data: who is allowed to see personal information (for example) within the data? If developers and/or testers are not permitted to see this then you will need to mask the test data in some way in order to comply with data privacy regulations. Of course, the very transformations that you build into data quality rules can be used to mask data but there remains the question of who tests the tester, so to speak. In any case, it is not as simple as replacing 16 digits in a credit card number with an x we will discuss this further in due course. To summarise: we do not believe that most companies (and vendors) think of building data management rules as analogous to developing other types of application. We think they should: data rules and transformations suffer from exactly the same sort of issues as more generalised development and the same sort of answers apply. Specifically, there are three major considerations: 1. An environment that supports the testing, development and validation of the rules and processes under consideration and, in particular, which supports early-stage user acceptance testing. Needless to say, this should be collaborative, and as easy to use and productive as possible. 2. The creation, generation or abstraction of a dataset that can be used as a test bed for the rules or processes that are of interest, and which can be re-generated or recreated, as required, in an agile manner. By preference, the generation or re-generation of this dataset should have as little impact on operational systems as possible. 3. Relevant output. If you are developing transformation rules for data integration, as an example, then you would like the output to either generate all the code or directly integrate with the data integration tool in question so that all relevant transformation processes can be generated automatically by that tool or, if you want a vendor-neutral approach, then you would like to generate a complete, documented specification that can be used to create the relevant workflows required to support the data integration process. In the following sections we will discuss each of these requirements. A Bloor White Paper Bloor Research

6 Agile development Aside from the facilities provided by data management vendors themselves which, as we have discussed, tend to be locked into a traditional and non-agile approach the principal alternative approach is known as data prototyping. This is complementary with, rather than competitive to, the various data quality, integration and governance solutions that are available in the market, though it could also be profitably used where no such solutions are in place. Data prototyping While we might argue with the following definition, according to Wikipedia a data prototype is a form of functional or working prototype. It states that the justification for its creation is usually a data migration, data integration or application implementation project and the raw materials used as input are an instance of all the relevant data which exists at the start of the project. Wikipedia goes on to say that the objectives of data prototyping are to produce: A set of data cleansing and transformation rules which have been seen to produce data that is all fit for purpose. A dataset that is the result of those rules being applied to an instance of the relevant raw (source) data. To achieve this, a data architect uses a graphical interface to interactively develop and execute transformation and cleansing rules using raw data. The resultant data is then evaluated and the rules refined. Beyond the obvious visual checking of the data on-screen by the data architect, the usual evaluation and validation approaches are to use data profiling software and then to insert the resultant data into a test version of the target application and trial its use. In other words, this offers an agile approach to the development of data quality rules and transformations: build a rule, test it, refine it, test it again and so on, and get users involved in acceptance testing whenever it is appropriate. Perhaps the most important aspect of this is that this is built on top of data profiling capabilities. This is fundamental for data integration, as you must understand the relationships (Federal Express and its orders) that exist within the data in order to be able to satisfactorily build a prototype and incrementally test it. However, there is a secondary issue: most data profiling tools profile the live data in-situ. This is not practical in a development environment because you are continually degrading the performance of the operational system by re-profiling it. This is why data quality providers do not typically offer prototyping: because there would be too big a performance overhead on live systems. So this approach will only be practical if the profiling software takes a copy of the relevant data and metadata when it reads it for the first time. Such an approach will mean that all subsequent analysis has no impact on operational systems and, as well, such analyses should perform better in their own right. The other point, of course, is that you do not actually need live data to test against: just representative data. We will discuss this further in due course. In the same way that agile development allows a programmer to validate their work as they develop, prototyping allows the incremental validation of the specification of rules during their development by the subject matter experts. This is achieved by interactively studying and even profiling the data, which results from the application of each (part of a) rule. In the case of Data Integration projects the final validation step is to load the data from the data prototype into the target application to ensure that it actually works: after all the analyst could have made a mistake. This validation of a result by subject matter experts, rather than an English specification or pseudo-code, is fundamentally different to the current iterative approach and can bring the enormous time and effort savings alluded to in the introduction to this paper Bloor Research A Bloor White Paper

7 Agile data In order to test application software, and we can think of data management rules as a form of application software, you need to have data to test against. The traditional ways of achieving this are to copy the operational database (often one copy for development and another for testing) or to subset the database. The problem with the former, and to a lesser extent the latter, is that it is expensive in terms of license costs and additional hardware support. Furthermore, the actual process of taking a copy of the database can cause a problem. In a data migration, for example, the schema for the new database may change during development, which effectively means that you need a new copy of the data to test against. This can seriously slow down the agility of the environment if, for example, it takes two weeks (which is fairly typical) to get a new copy of the database. Another problem with taking a database copy is that the current data in the database will not exhibit all the potential errors that you might want to process through your rules: many outliers (which are most interesting from a testing perspective) will only be present occasionally. This problem is exacerbated, of course, if you subset the data because that process can eliminate even more outliers. However, by using all the data, the rule development and validation can at least take account of all the actual variations in the data rather than trying to second-guess what might be in there. Finally, if data within your dataset is sensitive then it may need to be masked. However, this is not as simple as it may appear. For example, you could simply hide a credit card number by replacing each digit with an x, which will be fine if you are only concerned with data protection. However, if you want to test a payment application then you will need to work with real (pseudo-) numbers in order to test your applications. Similarly, simple shuffling techniques (for example, replacing zip code with 54321) will not work if your application requires a valid zip code. So, you will need to mask in such a way that the data remains valid. has a consulting physician who practices in a particular hospital and uses a designated operating theatre. If you scramble the data so that a patient with flu ends up having open heart surgery then your software may break down simply because your masking routines have not ensured that important relationships remain intact. So, discovery of these relationships may be essential and falls within the domain of data profiling and, by inference, data prototyping. There is another way, which is to generate synthetic data. This requires an understanding of the data model underlying the database(s) in question, which will typically be based on the database schema(s) complemented by relationships discovered through use, again, of data profiling. The two main advantages of using this approach are that the data does not need to be masked, and synthetic data can be re-generated very quickly (typically in a matter of hours rather than weeks) when required. There is also a third potential benefit in that, if you have a complete understanding of the data model and all its associated business rules, then you should be able to generate a dataset that includes all potential outliers. However, this is dependent on the thoroughness of your data model, the completeness of your data profiling, and the accuracy of your understanding of the processes involved. This can never by guaranteed to be 100% so using synthetic data, together with data prototyping, may well be a successful marriage because, with data prototyping, you are effectively going beyond the analysis by trialling the transformed data within the new application to see if it actually works. Further, it may not simply be a question of identifying what data needs to be masked and then hiding it. This is because you need to ensure that data relationships remain intact during the masking process as otherwise testing may break down. This will, of course, be dependent on the application but in complex environments it can be critical. For example, a patient has a disease, which has a treatment, which A Bloor White Paper Bloor Research

8 Agile output We have little to add to the brief description provided above. If we assume that you are using data prototyping then you would either like it to generate code, or be integrated with relevant data quality and data integration platforms, or generate a specification. In the last case specification really does not do justice to what has been produced: what you have is a fully tested (including acceptance testing) and complete document describing precisely what needs to be done. Given that classical specification-based approaches frequently lead to misunderstandings and failures, calling this sort of output a specification does it a disservice. In addition, as is sometimes the case in industry, the act of building a prototype may be all that is required to answer a question, prove a point or evaluate feasibility. The actual data produced by the data prototype may itself be the deliverable, especially if the requirement is a one-off, As a result the output of an Agile approach to data rule development could simply be that further phases of the project are not necessary Bloor Research A Bloor White Paper

9 X88 Pandora Pandora, from X88 Software, is a data management product which exists in three versions: Data Discovery and Profiling Edition Data Integration Prototyping Edition Data Quality Management Edition. In fact this is somewhat misleading as the three editions are not different products, as their names might suggest, but supersets of one another with the Data Quality Management Edition being the broadest in terms of functionality. In other words, both the Data Integration Prototyping and Data Quality Management editions include prototyping capabilities and both depend on the data discovery and profiling capabilities of the entry-level product. Data discovery and profiling is at the heart of the product. We do not intend to discuss this in detail. However, where Pandora differs from most profiling products in the market is that it copies the data from the system or systems being analysed into its own repository. This has the advantage that the profiling, relationship discovery and dependency and key analysis you perform have no impact on source systems. Furthermore, you can potentially do a lot more with the information you retrieve when working with your own source of data rather than relying on operational systems. In particular, it is this that enables data prototyping. Specifically, the two more advanced editions of Pandora include a graphical rule builder and automatic mapping report generation as well as a business glossary (see Figure 1) and support for the definition of user defined functions. The glossary is important because it supports collaboration between business analysts and developers while user-defined functions allow you to define your own functions if the 300+ functions included within Pandora are not sufficient for you. However, the main point is that instead of defining rules (which may be data quality, data integration or transformation rules) in a spreadsheet or in a document, which is the typical approach adopted in most organisations, you can design these within the graphical rule builder and then test them, within the same environment, to see if they work in the way that you expect. Figure 1: Pandora Business Glossary options The graphical rule-builder uses a point-andclick interface that does not require any programming or scripting, and does not require knowledge of SQL. Indeed, the repository is not relational and does not have the limits that SQL can impose on certain types of functions. Major capabilities include: Joins you can join data based on the relationships discovered during profiling. Any type of multi-system join is available, all of which are performed within the Pandora repository and without reference back to source systems. One specific feature worth noting is the dynamic table union function. This allows you to create the union of multiple source tables even when these are dissimilar. This is particularly useful in supporting MDM implementations, for example. Sorts the product supports single and multi-column sorting as well as reverse sorts. In particular, you can sort on a sub-value within a datatype. Filters data can be filtered via fuzzy matching, value contains options, pattern matching, range and datatype tests. Grouping Pandora supports multi-column group by operations on both native and calculated or derived values. This is often combined with aggregation functions to provide record count and value summations. Other pre-built functions include support for aggregations (see Figure 2) and look-ups from reference tables, amongst others. A Bloor White Paper Bloor Research

10 X88 Pandora Figure 2: Aggregation options in Pandora Figure 3: Business-friendly mapping diagram In practice, once you have defined your rules, tested them and confirmed that they do what you want (which will, no doubt, be an iterative process), then you are ready to generate your mappings. This is done automatically for you by the product, which takes into account all the rules and functions you have deployed and automatically generates the optimal mapping to meet those requirements. Note that this is illustrated (see Figure 3) using business terminology in order to support business/it collaboration and the sort of agile approach we have advocated in this paper. To do this, Pandora uses metadata (stored in its repository) matching. It uses exact field-name matches in the first instance, then a series of fuzzy matching rules to ascertain the most likely target fields for available source fields. Where there are multiple opportunities for mapping, it provides the user with a list of potential candidates from which to choose. The auto-mapper validates all of the mappings to ensure that the source and target are compatible in terms of datatype, length, nullability and syntax rules. Inconsistencies are highlighted visually. The key point here is that you can now just load the resulting data generated from your mappings into the target application and see how it runs for acceptance testing purposes. Once you are happy with the results you can use Pandora to generate, again automatically, a mapping specification document that, together with the associated data, marks a clear handover from Design to Build, and an objective sign-off for the Build when that is completed. Best practice will be to provide the Build team with the Pandora-generated specification document (an example of which is illustrated in Figure 4) and the same source data used to develop it. If the process developed by the Build produces Bloor Research A Bloor White Paper

11 X88 Pandora output data that is identical to the Pandora data prototype, then the Build can be signed off. Otherwise it does not conform to the specification. This makes managing outsourced and similar developments much simpler than is typically the case. Note that, since the data prototype uses all the source data, most test cases will be catered for by default. Finally, bear in mind that the mapping specification generated and passed on to the Build team is much more than a traditional mapping document or specification. This is because you know that it will work thanks to the testing and design work that you have already done. The only thing that we could wish for is that you could import the mappings directly into a relevant data integration tool and could then use them to generate the relevant workflows within that tool. However, even without this capability, we believe Pandora to be extremely useful and it is in use at a number of X88 customers in conjunction with high-end data integration products such as Informatica and Ab Initio. Figure 4: Pandora mapping report Finally, it is worth noting that X88 Software is working on a Data Integration Edition of Pandora to provide an end-to-end solution for departmental projects or for organisations that do not have data integration products in place. A Bloor White Paper Bloor Research

12 Conclusion The environments in which data management rules are developed and tested in most organisations today are old-fashioned, time consuming and error prone. This is primarily because data management is not treated as a form of application development, where things have moved on from the sort of approach that is classically presented within data quality and data integration environments. This paper is therefore a call for some more forward thinking from data management practitioners and vendors and a move towards a more agile stance for the development of data management rules. One of the few companies that has recognised this deficiency and has responded to it is X88. The company s Pandora tool supports data prototyping for data integration, data quality and data management environments generally. Properly used this approach can eliminate the specification mismatch that is the cause of so many overrunning, delayed or cancelled data management projects. Further Information Further information about this subject is available from Bloor Research A Bloor White Paper

13 Bloor Research overview Bloor Research is one of Europe s leading IT research, analysis and consultancy organisations. We explain how to bring greater Agility to corporate IT systems through the effective governance, management and leverage of Information. We have built a reputation for telling the right story with independent, intelligent, well-articulated communications content and publications on all aspects of the ICT industry. We believe the objective of telling the right story is to: Describe the technology in context to its business value and the other systems and processes it interacts with. Understand how new and innovative technologies fit in with existing ICT investments. Look at the whole market and explain all the solutions available and how they can be more effectively evaluated. Filter noise and make it easier to find the additional information or news that supports both investment and implementation. Ensure all our content is available through the most appropriate channel. Founded in 1989, we have spent over two decades distributing research and analysis to IT user and vendor organisations throughout the world via online subscriptions, tailored research services, events and consultancy projects. We are committed to turning our knowledge into business value for you. About the author Philip Howard Research Director - Data Philip started in the computer industry way back in 1973 and has variously worked as a systems analyst, programmer and salesperson, as well as in marketing and product management, for a variety of companies including GEC Marconi, GPT, Philips Data Systems, Raytheon and NCR. After a quarter of a century of not being his own boss Philip set up what is now P3ST (Wordsmiths) Ltd in 1992 and his first client was Bloor Research (then ButlerBloor), with Philip working for the company as an associate analyst. His relationship with Bloor Research has continued since that time and he is now Research Director. His practice area encompasses anything to do with data and content and he has five further analysts working with him in this area. While maintaining an overview of the whole space Philip himself specialises in databases, data management, data integration, data quality, data federation, master data management, data governance and data warehousing. He also has an interest in event stream/complex event processing. In addition to the numerous reports Philip has written on behalf of Bloor Research, Philip also contributes regularly to and and was previously the editor of both Application Development News and Operating System News on behalf of Cambridge Market Intelligence (CMI). He has also contributed to various magazines and published a number of reports published by companies such as CMI and The Financial Times. Away from work, Philip s primary leisure activities are canal boats, skiing, playing Bridge (at which he is a Life Master) and walking the dog.

14 Copyright & disclaimer This document is copyright 2011 Bloor Research. No part of this publication may be reproduced by any method whatsoever without the prior consent of Bloor Research. Due to the nature of this material, numerous hardware and software products have been mentioned by name. In the majority, if not all, of the cases, these product names are claimed as trademarks by the companies that manufacture the products. It is not Bloor Research s intent to claim these names or trademarks as our own. Likewise, company logos, graphics or screen shots have been reproduced with the consent of the owner and are subject to that owner s copyright. Whilst every care has been taken in the preparation of this document to ensure that the information is correct, the publishers cannot accept responsibility for any errors or omissions.

15 2nd Floor, St John Street LONDON, EC1V 4PY, United Kingdom Tel: +44 (0) Fax: +44 (0) Web: