Data De-duplication from the data sets using Similarity functions

Transcription

1 Data De-dplication from the data sets sing Similarity fnctions M.Chitrarpa 1, V.Mniraj Naid 2 1 M.Tech Stdent Department of CSE Adisankara College of Engineering & Technology Gdr, Nellore district, India 2 Assoc. Prof. Department of CSE Adisankara College of Engineering & Technology Gdr, Nellore district, India Abstract: The fndamental isse of dplicate detection is that inexact dplicates in a database may refer to the same real world object de to errors and missing data. Dplicate elimination is hard becase it is cased by different types of errors like typographical errors, missing vales, abbreviations and different representations of the same logical vale. The recent poplarity in merging data from mltiple data sorces has introdced the problem of dplicate records in databases. For Instance, one can find two records that have different syntactic representation of data, bt yet describe the same real world entity. Themselves weakened by the presence of those dplicates in databases. Sch dplicates often negatively inflence the reslts when analyzing data. In this paper we are sing Levenshtein Edit-Distance similarity fnction for strings. Keywords: Dplicate detection, Similarity fnction: Edit-distance, K-means clstering, Syntactic and Entity. 1. Introdction Data qality problems arise with the constantly increasing qantity of data stored in real-world databases that are assred by the vital data cleaning process. The fndamental element of data cleaning is sally termed as dplicate record identification that is the process of identifying the record pairs signifying the same entity (dplicate records). In this paper, we have developed a domain independent approach to detect dplicate followed by data sorces as they are independent. Normally, organizations become conscios of practical precise disparities or inconsistencies while integrating data from diverse sorces to implement a data warehose. Sch problems belong to the category called data heterogeneity. Erroneos dplication of data occrs when information from diverse data sorces that store overlapping information is integrated bt, errors like spelling mistakes, convicting cstoms across records presented in large databases. In the data sorces, omitted fields etc., normally exist in training phase, the record level similarity is specified by the featre vector as inpt for training. The experimentation is performed on the realworld datasets and the performance is evalated with the evalation metrics. Data warehoses that are archives of data gathered from nmeros data sorces constitte the fondation of the majority of existing decision the data accepted at the data warehose from external sorces. Incoming data tples from external sorces need validation and refinement for providing high data qality. An `error-free' procedre in the data warehose is recommended by data qality. Data cleaning techniqes are essential to improve the qality of data. Data cleaning also called data cleansing or spport applications and CRM (Cstomer scrbbing, enhance the qality of data by Relationship Management). Preciseness of identifying and eradicating errors and decision spport analysis on data warehoses is critical becase important bsiness decisions are inflenced by sch analysis. Independent and inconsistencies from the data [5]. It aims at enhancing the overall data compatibility by concentrating on eradication of changes in data practically incompatible standards may be contents and minimizing data repetition. Record 36

2 dplicates, omitted vales, record and field resemblances and dplicate eradications are detected by crrent data cleaning techniqes [8], [4]. Detection of other or several records that signify one distinct real world entity or object is performed by the dplicate record detection process [2, 31]. The problem of dplicate detection is actally to find ot for all objects represented in the database whether the same real-world object is represented by two or more distinct database entries. Record linkage, object identification, record matching etc., are alternative names for Dplicate detection". It is a greatly researched topic and has high importance in fields sch as master data management, data warehosing and ETL (Extraction, Transformation and Loading), cstomer relationship management, and data integration [2]. The two innate problems that mst be addressed by dplicate detection are qick detection of all dplicates in large data sets (efficiency) and proper determination of dplicates and non-dplicates (effectiveness) [9]. Clstering is the categorization of objects into diverse grops, or more exactly, the division of a data set into sbsets (clsters), so that the data in each sbset (ideally) reveal a few common traits C freqently proximity based on certain defined distance measre. The process of splitting database into a set of mtally exclsive sbsets (blocks) sch that matches do not occr across blocks is commonly termed as blocking. Hence, the efficiency of dplicate detection is increased by blocking which sbstantially improves the speed of the comparison process [12]. For example, classifying a set of people records based on the zip codes in the address fields, avoids comparing records that have different zip codes [11]. Fingerprint-based and fll text-based are two types of common dplicate detection methods [10]. Textal similarity, typically qantified sing a similarity fnction [20] sch as, edit distance or cosine similarity is tilized by most of the crrent approaches to determine if two representations are dplicates. In this paper, we employ an efficient detects the dplicates by comparing a pair of records sing attribte similarities. Here, is trained to discriminate between pairs of records corresponding to dplicates and nondplicates sing the training vectors generated by these attribte similarities. In or proposed approach, we employ two phases that is training phase and dplicate detection phase. In the training phase, the dataset contains the labeled dplicates and nondplicates.first we consider a pair of dplicate or non-dplicate records and then, compte the similarity sing three similarity measres to obtain the similarity distance between two individal records. After that, we combine all the similarity vales of all records to generate a featre vector. The above two steps are repeated for all pair of records in the training dataset to obtain a set of featre vectors. After that, we introdce the training system, according to the featre vectors. In the dplicate detection phase, an efficient K- means clstering is sed to partition the dataset into small partitions based on some common featres. It redces the time taken for record comparison to increase the efficiency of the dplicate detection process. Hence, the similarity comptations and other processes are performed on the particlar clster in which the record falls. Ths, we identify the dplicate and nondplicate records for the inpt dataset to improve the qality of or database. The experimentation is carried ot on the real dataset containing dplicate records and the reslts demonstrate that or approach improves the accracy in dplicate detection. 2 Dplicate Record Detection Using Similarity Metrics Dplicate detection depending on the particlar domain is complicated for two reasons. First one is dplicate representations are not similar as they differ slightly in their vales and Second one is theoretical reqired comparison of all pairs of records is impractical for hge qantities of data. 2.1 Proposed Work: Let s consider a database D, that contains records composed of k different fields. Given database, D = {R1; R2; _... ; Rn} where each record Ri incldes k fields sch as name, address, street and city. The fields are often called as attribtes in data cleaning. First, the proposed work takes the training dataset and directly comptes the similarity vales for each record sing edit distance. Bt there exists m different similarity metrics for example edit distance[15, 16], cosine similarity, ID5 and many more. 2.2 Similarity fnctions 37

3 Similarity fnctions are formlas that have been invented to measre the distance between two data points. In this paper the similarity fnctions are sed to compte the similarity between two vales of an attribte. The similarity is defined as a vale indicating how close the two elements are depending on how big or small the vale is. A distinction needs to be drawn here between the terms distance fnctions and similarity fnctions. Given two strings s and t, a distance fnction generates a comparative vale v where a bigger v represents less similarity and a smaller v represents a greater similarity between the two strings. In contrast, a similarity fnction rns on two strings will generate a bigger vale only if the two strings are compared to be very similar. In this paper, the two terms will be sed withot distinction, nless clearly specified. In this dplicate detection phase, we have sed clstering method, K-means [16] to avoid the similarity comptation between all pairs of records. Then, the similarity comptation is performed on the particlar clster in which the record falls. It is sefl for the fast identification of dplicate records. Dplicate detection process 1. Similarity Comptation for records sing Levenshtein distance fnction 2. Combining similarity vales 3. Clstering of inpt dataset 4. Dplicate detection and elimination of dplicate records Step 1.Similarity Comptation for records sing Levenshtein distance fnction In this paper we have sed Levenshtein distance to measre the amont of difference between two seqences (i.e. an edit distance). T he Levenshtein distance between two strings is the minimm nmber of edits that are reqired to transform one string into the other, with the help of edit operations like insertion, deletion or sbstittion of a single character. This edit distance is normalized by dividing the maximm of the length of two name strings. The cost of compting this distance is proportional to the prodct of the two string lengths. In or example, we compare the strings 'Divanov' and "Diane". D I V A N O V D I A N E Table 1. Levenshtein distance between two strings The distance between these two strings is 3 (i.e. a minimm of three edit operations is reqired to transform the first string into the second). Let s consider a sample dataset in Table 1. contains asset of records of stdent details of Name, Address, Street and City. Name Address Street City Mrs.Snith a D/oJohn 51Bharat Nagar d-5 Mrs.Sneet ha D/oJohn Bharat Nagar d-5 Haritha D/oRam HyderNaga r d Raj S/osom shivajiroad Delhi-01 Harita D/oRam HyderNaga r d Raj S/swam BhagyaNag Delhi-01 i ar Table 2 sample dataset We calclate the Levenshtein distance fnctions for each pair of record in the dataset. Here, the similarity comptation process is explained for two records R1and R2 present in the table shown above. Let s consider only the Name" attribte of these two records. The distance between these two strings is 2 (i.e. a minimm of two edit operations is reqired to transform the first string into the second).in this way we find distance between all pairs of records in the dataset. 38

4 Table 2. Sample dataset International Jornal of Advanced Engineering and Global Technology 0 M r s. S n i t h a M r s S n e e t h a Step2. Combining similarity vales all the mltiple fields We combine these similarity vales obtained from different similarity measres to compte the distance between any two records. Here, we can represent similarity between any pair of records by a featre vector in which each component has the similarity vale between two records of anyone of the similarity measre. When considering a dataset that contains records composed of n different fields and a set of m distance metrics, we can represent similarity between any pair of records by an m-length vector. Each component of the vector represents the compted similarity vale between two records that is calclated sing one of the m distance metrics. Step3. Clstering The clstering of data records is carried ot sing the k-means clstering that is widely accepted clstering method among the data mining commnity. The basic steps for clstering are given as: (1) Initialize k-centroids, one for each clster. (2) Compte the similarity of each k-centroid with the data records presented in the dataset. (3) Assign data records to clster Ci whose similarity measre is high. (4) Update the k-centroids. (5) Repeat Step 2 to step 4, ntil there is no movement of the data records between the clsters. We apply K-means clstering to the data set to separate the fields contains same average distance. Step 4. Detection and elimination of Dplicate records In this dplicate detection phase, for each data records, the comparison is done only the data records presented within the clster to redce the nmber of record comparisons. Pairs of records that fall nder each clster will be candidates for a fll similarity comptation and then perform the record comparison by tilizing previos steps in the same way as sed. Here, the similarity metrics are sed to calclate distances for each field of each pair of potential dplicate or non-dplicate records. Inpt and obtaining a binary matrix which distingishes the dplicate and Non-dplicate records based on the threshold vale. We are considering the threshold vale is 0.9(i.e. 90%).If two records similarity is 90% then the second record is consider as dplicate record and less than threshold vale remaining all records are nondplicate records. Reslts Analysis: The reslts of the proposed approach for dplicate record detection are explained in this section. Experimental reslts consider the table 2 as follows: R1 &R2 : Distance vector= [ ] Similarity measres mapping into 0 to 1 is [ ] 0.91 R1&R3 : Distance vector= [ ] The complete similarity matrix, Similarity measres mapping into 0 to 1 is [ ] 0.5 R6 R1 0 R1 R2 R3 R4 R5 R R R R

5 R We are considering the threshold vale is 0.9(i.e 90%).If two records similarity is 90% then the second record is consider as dplicate record and less than threshold vale remaining all records are non-dplicate records. Here R1, R2 90%, R2 is removed and R3 & R5 are 96% similar, R5 is removed. After dplicate records removing the data set consists of R1, R3, R4 and R6. Conclsion: We followed a simple approach to detect dplicate records occr in large databases. The approach makes se of similarity fnctions in detecting the dplicate records. The similarity was compted with the help of Levenshtein distance, similarity to accrately identifying the dplicate records. Finally, the experimentation was carried ot sing the real-world datasets and the performance of the proposed approach is evalated based on the Symmetric metrics. The experimental evalation ensred that the proposed approach detects dplicates efficiently. References [1] Srajit Chadhri, Kris Ganjam, Venkatesh Ganti and Rajeev Motwani, Robst and Efficient Fzzy Match for Online Data Cleaning", In Proceedings of the ACM SIGMOD International Conference on Management of Data, pp , New York, USA, [2] Ahmed K. Elmagarmid, Panagiotis G. Ipeirotis and Vassilios S. Verykios, Dplicate Record Detection: A Srvey", IEEE Transactions on Knowledge and Data Engineering, Vol. 19, No. 1, pp. 1 { 16, Janary [3] Srajit Chadhri, Anish Das Sarma, Venkatesh Ganti and Raghav Kashik, Leveraging Aggregate Constraints for Dedplication", In Proceedings of the ACM SIGMOD International Conference on Management of Data, pp. 437 { 448, New York, USA, 2007}. [4] J. Jebamalar Tamilselvi and Dr. V. Saravanan, A Uni_ed Framework and Seqential Data Cleaning Approach for a Data Warehose", International Jornal of Compter Science and NetworkSecrity, Vol. 8, No. 5, May [5] Erhard Rahm and Hong Hai Do, \Data Cleaning: Problems and Crrent Approaches", IEEE Data Engineering Blletin, Vol. 23, No. 4, December [6] Matsakis, Nicholas E and Leslie Pack Kaelbling, Active dplicate detection with Bayesian non-parametric models", Massachsetts Institte of Technology, Thesis (Ph. D.), MIT libraries, [7]RIDDLE data repository from [8] Arthr D. Chapman, Principles and Methods of Data Cleaning Primary Species and Species- Occrrence Data", Version 1.0, Report for the Global Biodiversity Information Facility, Copenhagen, [9] Uwe Draisbach and Felix Namann, A Comparison and Generalization of Blocking and Win-dowing Algorithms for Dplicate Detection", In Proceedings of the 7th International Workshop on Qality in Databases at VLDB, Lyon, France, [10] Hi Yang and Jamie Callan, Near-Dplicate Detection by Instance-Level Constrained Clstering", In Proceedings of the Twenty-Ninth Annal International ACM SIGIR Conference on Re-search and Development in Information Retrieval, pp. 421 { 428, Seattle, WA, USA, 2006}. [11] Steven Eijong Whang, David Menestrina, Georgia Kotrika, Martin Theobald, and Hector Garcia-Molina, \Entity Resoltion with Iterative Blocking", In Proceedings of the 35th SIGMOD international conference on Management of data, Providence, pp. 219 { 232, Rhode Island, USA, Jne 29 { Jly 2, 2009}. [12] J. Jebamalar Tamilselvi and V. Saravanan, Token- Based Method of Blocking Records for Large Data Warehose", Advances in Information Mining, Vol. 2, No. 2, pp. 5 { 10, [13] Microsoft Research `Data Cleaning' from lt.aspx. [14] Israr Ahmed, Abdl Aziz, \Dynamic Approach for Data Scrbbing Process", International Jornal on Compter Science and Engineering (IJCSE), Vol. 2, No. 2, pp. 416 { 423, 2010}. [15] Jaro, M.: Advances in record-linkage methodology as applied to matching the 1985 censs of tampa, orida. Jornal of the American Statistical Association pp (1989). [16]. Levenshtein, V.: Binary codes capable of correcting deletions, insertions, and reversals. In: Soviet physics doklady. vol. 10, pp. 707{710 (1966)}. [17] MacQeen, J.B., \Some Methods for classi_cation and Analysis of Mltivariate Observations", in Proceedings of 5th Berkeley Symposim on Mathematical Statistics and Probability. pp. 281 { 297,1967}. 40