The Cost of Duplicate Data in Enterprise Content Management

Transcription

1 The Cost of Duplicate Data in Enterprise Content Management Sheryl Arnold Partner/CTO

2 Introduction Duplicate records in databases and product information are a serious issue for data and content management systems. We have all experienced one aspect of this problem when we receive several mass-market mailings all addressed differently to our home: Mike Smith is the same as Michael Smith is the same as Mike R Smith problem. Significant effort has been made to identify and remove duplicates to prevent the obvious cost waste for these mail/address databases. In the area of product information and enterprise resource planning (ERP) data management, the issue is similarly expensive. Beside the costs of maintaining duplicate data, the issue is magnified as this data becomes available across the enterprise in a variety of applications. For example, duplicate product records in an eprocurement system can cause a variety of problems. These range in importance from a minor nuisance when a catalog user pauses to try to understand the differences in products that appear identical in the catalog but might be different. On the other end of the spectrum is the issue of product obsolescence and misidentification when duplicate records remain in a system. It becomes critical as part of a data and content management system to provide a method to identify and remove duplicates and to prevent the introduction of duplicates into these systems when possible. This process is often called deduping. This paper addresses the issue of duplicates and provides a strategy for the prevention, identification, and removal of duplicate data records. Why Duplicates Matter in eprocurement Systems Duplicate records are troublesome in most applications and eprocurement systems are no different. For example, if the fields used for a product key are identical, a critical operation on the product data record may be performed in error. These affected operations include product updates, price and availability changes, obsolescence of products, catalog search results, general maintenance of the product catalog, and equipment maintenance. It is also interesting to note that the average system contains up to 65% duplicates. Even though many product content management systems provide functionality for the loading and maintenance of product information, duplicate identification and removal requires additional business processes. An effective solution must be sophisticated enough to provide automated software technology as well as learned intelligence capture and exception handling. Failure to do both of these will result in a loss of productivity, efficiency, and accuracy when manual steps are repeated over time. How Duplicates Affect the Bottom Line The day-to-day operations involved in keeping manufacturing or construction equipment and fleet operations at optimum levels can be tedious at best. It s always a fair assumption that supporting operations such as Procurement, Maintenance, Sourcing, and Asset Management should be able to access the right product at the right price from the right supplier. Depending on the duplicate volume and complexity levels that have been allowed to sweep across the enterprise system the challenge in controlling costs can seem insurmountable. In this paper there are multiple examples of duplicates at different complexity levels. In each example there is a cost risk based on different pricing for the same part. If the data for these products is from varied sources (systems, business units, or regions) then costs will climb exponentially based on usage and duplicate volume. 1 P a g e

3 Understanding Duplicates Duplicate identification can be an expensive and involved process. It is important to understand the scope and issues related to duplication identification and removal before a strategy can be devised and implemented. The examples used in this paper are simple by design, but the concepts also apply to complex, highly attributed, and direct material product information. An important aspect of understanding duplicates is in understanding different classifications, or levels of duplicates. Generally, three levels of duplicates exist in any given system, and each level requires a different process and effort for identification and removal. A description of each of these levels is shown on the following page. Level-1 Duplicates: Exact Records The simplest and most easily understood duplicates occur when two or more records are an exact match. These records are known as a Level-1 duplicates. In Table 1 below, notice that the first and last records are exact. These records may be in a single source file or one record may be in a source file to load while one record already exists in the destination database. In some cases, a complete record may be spread across multiple source files. These files all contain a unique key to the data but have different components that are used to make up a single product description. This situation applies to all levels of duplicate identification. Table 1: Example of Exact Records Mfr Name Description Mfr Part # UOM Supplier Price Fleetguard Filter Fuel Spin On FF5324 EA ABC Corporation Fleetguard Filter Fuel Spin On FF5321 EA ABC Corporation Fleetguard Filter Fuel Spin On FF5324 EA ABC Corporation Level-2 Duplicates: Near-Exact and Similar Records Level-2 duplicates are records that vary by some value but are, in fact, the same record. This condition can be caused by a number of factors ranging from typographical errors and incomplete data to differing descriptions by different suppliers for the same record. Notice that the six items in the table below vary by some slight deviation. This variation is all that is required for a computer to believe that these are six unique records. 2 P a g e

4 Table 2: Example of Near-Exact Records Mfr Name Description Mfr Part # UOM Supplier Price Fleetguard Filter Fuel Spin On #FF5324 EA ABC Fleetguard Filter Fuel Spin On FF5324 EA ABC Corp Fleetguard Filter Fuel Spin On FF5324 Each ABCCorp Fleetguard Filter Fuel Spin On FIL5324 EA ABC Fleetguard Filter Fuel Spin On FF 5324 EA ABCCorp Fleetguard Filter Fuel Spin On FF5324 EA ABC Level-3 Duplicates: Different Records Representing Identical Records The third and most complex level of duplicates occurs when the records are the same but their descriptions and even part numbers are different. Level-3 duplicates require sophisticated matching algorithms and sometimes human intervention in order to identify and resolve the duplicate. Table 3: Example of Complex Records Mfr Name Description Mfr Part # UOM Supplier Price Fleet Gard Filter, Fuel 5324 EA ABCCO FleetGuard Fuel Filter FG EA ABC Corp Fleetguard Filter, Spin On FG FF5324 EA ABCCorp Level-4 Duplicates: Identical products with different manufacturers The fourth level of duplicates represents the identification of duplicate products that are being provided by multiple manufacturers or suppliers. This type of duplicate is typically referred to as an alternate or cross reference to the original part and provides the exact same fit, form and function as the original. Table 4: Example of Identical Products Mfr Name Description Mfr Part # UOM Supplier Price Fleetguard Filter Fuel Spin On FF5324 EA ABC Corp Baldwin Fuel Filter BF7634 EA AAA Parts Caterpillar Filter 1R0759 EA ABC Corp Donaldson Fuel spin-on primary P EA XYZ Inc P a g e

5 Duplicate Identification and Removal One primary reason for classifying the duplicates into four levels is that the process to identify and remove them is addressed differently for each one. Level-1 duplicates are the easiest to identify. Conversely, Level-3 identification of duplicates is costly in time, resources, and money while Level-4 duplicates produce results that can drive business initiatives such as manufacturer consolidation, deep discount and contract negotiation purchasing strategies. There may be slight variances for each of the levels outlined below depending on the type of data and the type of system (i.e., ERP, CRM, legacy 1, legacy 2, eprocurement, etc.) the data is from. Level-1 Duplicate Records At first glance, Level-1 duplicate identification and removal looks quite simple. However, it is more difficult than it seems to find exact matches. This is directly related to how a computer analyzes data. The precision that computer software provides for identifying matches does not help in the case where # 123 is not the same as #123. This issue compounds significantly when there are long text descriptions associated with a record. One technique to mitigate this problem is to define keys for a duplicate where long text fields are ignored. A Level-1 duplicate check should be performed on any new data entering the system after an initial duplicate removal project has been performed. It is desirable to have a strict key definition (i.e. as many fields of the record as possible). It is also critical to have these potential duplicates routed into an exception-based workflow for action. Level-2 Duplicate Records While there is usually gain from a Level-1 duplicate project, the majority of duplicate records will be identified through a Level-2 process. This will require data cleansing and normalization as part of the process and is often performed in conjunction with those activities. Since typical keys selected for duplicate identification for eprocurement product content are manufacturer and manufacturer part number, these are a good starting point for cleansing and normalization. Unit of measure (UOM) is also a common attribute to add to the key. Typical record cleansing and normalization activities are shown as follows: Part numbers need to be normalized (notice spaces, dashes, and special characters) Manufacturer names need to be normalized by using a reference database All text requires case normalization (sentence or title casing), spell checking, and abbreviation expansion Subtleties for Level-2 Duplicate Identification Additionally, there are often subtleties associated with Level-2 duplicate identification. Several of these are as follows: Text fields are notoriously difficult (i.e., Table 2) Too simple key definition can lead to many false duplicates Part numbers often have many representations with dashes, spaces Trailing spaces may need trimming and cannot be visually identified Multiple suppliers of the same part might be desired but flagged as a duplicate 4 P a g e

6 Level-3 Duplicate Records The most difficult and, therefore, most costly of duplicate record identification is when the records do not share many, if any, identical fields even after a cleansing and normalization exercise. These records are typically a small subset of the overall number of duplicate records, but can become a majority of the cost. The effort and resources to identify these duplicates vary by commodity type but typically require subject matter experts (SMEs) or the involvement of the manufacturer or supplier in order to determine the accuracy of the product description. Subtleties in Level-3 Duplicate Identification The subtleties manifest themselves in how the SMEs perform the exercise. In particular, replacements, similar records, and newer versions often cause errors. It is important in a Level-3 exercise to have at least two independent SMEs validate the duplicate. In some cases, it may be necessary to contact the manufacturer or supplier. If a decision is made to remove or not remove, then that decision needs to be documented, archived, and possibly internal procurement processes need to be revised to eliminate the potential danger of adding duplicates in the future. Level-4 Duplicate Records Level-4 duplicates that are managed by corporate business practices may not need to be removed from the procurement system since they provide buying power and potential supplier consolidation. In most cases where corporate business practices have been unable to identify these types of duplicates there is a high risk of unused inventory and unrealized cost savings. The effort required to identify these duplicates vary by commodity type and utilize all levels of duplicate identification. Subtleties in Level-4 Duplicate Identification The subtleties manifest themselves in the effectiveness of the duplicate identification processes, how they are exercised and the involvement of SMEs. Duplicate Identification and Removal Work Flow Once an analysis is completed and a strategy is determined, a workflow can be established for the management of all four levels of duplicate identification and removal. It is critical at this stage to ensure the capture of all intelligence about the data records with any nuances, the processes, and lessons learned in this initial project. It is likely that these patterns will repeat themselves over time when new data records are encountered or additional processing is performed on the existing data. A sample workflow is shown in the Figure below. It does not include details such as how the software compares records technically or what specific action is performed at each step. These will vary depending on the result of the analysis and the software used in the workflow. 5 P a g e

7 Figure 1: Duplicate Identification and Removal Workflow Legacy Database De-duped Database Level-1 De-dupe Remove Exact Records Level-2 De-dupe Remove Same- Keyed Records Validate Normalize Cleanse Classify Enrich Data Normalization and Cleansing The validation step is typically used to ensure that required fields exist and values are within specified ranges. The normalize step ensures common values are used for fields where possible (e.g., units of measure) and standards are applied. The cleanse step corrects errors in the record. The classify step is used to place similar records in a taxonomy (i.e., typically a category of like products with their associated attributes). The enrich step is often used to enhance missing, incomplete, and inaccurate descriptions of the product.. After records receive this processing, they run though a Level-2 duplicate identification and removal process. Ideally, the product content management software would be able to save these steps with a context for each action required (e.g., by supplier). Level-3 duplicate identification is performed after the classification step with software that provides a search capability by category. It is typically performed by having SME s examine records after the processing shown in the Figure above. 6 P a g e

8 Conclusion It is important to understand the cost of duplicates and scope of the problem before embarking on a duplicate identification and removal strategy. This cost will determine which strategy to employ and the level of duplicate identification required. It is recommended that every system implement Level-1 duplicate record identification and removal. More importantly, implement a system to prevent Level 1 duplicates from entering the system with future record additions or imports. Most analyses will determine that Level-2 duplicate identification and removal is also required. Level-3 is a system-by-system analysis while Level-4 may require corporate initiatives for improved manufacturer and supplier management. In all cases, ensure that any knowledge gained in the process is captured and used to prevent the addition of duplicates upon additional record imports. It is important at this step to determine if the software and product content systems in your enterprise or under evaluation for your enterprise are equipped to handle duplicate identification and removal processes. Costs associated with the project will be directly proportional to the automated capabilities of these systems. Once the identification and removal project is successful, it is critical to establish on-going procedures, processes and technology automation to prevent duplicate entries. If the system you use has the ability to capture a combination of processes and learned knowledge (e.g., EMI-SMS Data Quality Solutions), then cost savings and efficiencies can be realized. Additionally, this knowledge can be leveraged in other projects within the enterprise that require or desire duplicate record removal. Author - Sheryl Arnold, The Cost of Duplicate Data in Enterprise Content Management, WPDQ P a g e