Enhancing Quality of Data using Data Mining Method

Transcription

1 JOURNAL OF COMPUTING, VOLUME 2, ISSUE 9, SEPTEMBER 2, ISSN Enhancing Quality of Data using Data Mining Method Fatemeh Ghorbanpour A., Mir M. Pedram, Kambiz Badie, Mohammad Alishahi Abstract Data is asset for companies and organizations. Because data and the information obtained from data analysis play an important role in decision making. The quality of data affects the quality of decisions and the incorrect data causes incorrect decision making. Recently, a great deal of researches has focused on enhancing data quality. It is infeasible or very difficult to improve quality of data through manual inspection. Because data quality is one of the complicated and non-structured concepts and data cleansing process can not be done without the help of professional domain experts, and detection of errors require a thorough knowledge in the related domain of the data. Therefore (semi-)automatic data cleansing methods is employed to find data errors and defects and solve them. Data mining methods are appropriate for enhancing different dimensions of data quality, since they are aimed at finding abnormal patterns in large volumes of dataset. In this paper, a new approach is presented to detect the errors inside the dataset using fuzzy association rules. Fuzzy association rules are used to build a model that is intended to capture the structure of the regarded data. Finally, Experimental results of the proposed approach show the effectiveness of the proposed method to find errors in datasets. Index Terms data quality, data mining, fuzzy association rules INTRODUCTION IDESPREAD use of data and decisions based on Wdata analysis focuses on data quality in today s business success. Nevertheless, a study by the Mta group reveals that 4% of projects based on data analysis will be failed. As one of the main reasons, they identified insufficient data quality leading to wrong decisions []. Therefore, traditional methods of cleaning data can be used rarely. It is normally infeasible to guarantee data quality by manual inspection, especially when data are collected over long periods of time and through multiple generations of databases. Therefore, (semi-)automatic data cleaning methods have to be used []. Since the early 9s, knowledge discovery in databases (KDD) has been introduced as a well established field of research, and over the years new methods together with scalable algorithms have been developed to analyze effciently very large datasets. Unfortunately most of the orientation is towards the particular and theoretical problems. The application of data mining methods to improve data quality is a relatively new and promising approach from research and usage viewpoint and can present new domains of their application outside the domain of pure data analysis [2]. Data quality studies have been accomplished using data mining methods. For example, Brodley and Friedl F. Ghorbanpour A. is with the Department of Computer Engineering, Islamic Azad University, Science and Research Branch, Tehran,Iran M.M. Pedram is with the Department of Computer Engineering, Faculty of Engineering, Tarbiat Moallem University, Karaj/Tehran,Iran K. Badie is with the Information Technology Research Center,Telecom Research center, Tehran,Iran M. Alishahi is with the Department of Industrial Engineering, Sharif University of Technology, Tehran,Iran used data mining in preprocessing data in the process of knowledge discovery in database. By filtering the probably incorrect training data, they can significantly reduce the misclassifications, while their goal is to improve final classification accuracies, not detection of errors in training data [3]. In addition, Hipp and et al proposed a developed algorithm of data mining to extract the structure of the data. Deviation from this structure can then be hypothesized to be incorrect [4]. Grüning showed that classifiers can be used in detection of conflicts in datasets and gave practical recommendations for data correction. This approach uses support vector machines as a classification algorithm [5]. Marcus and Maletic used different methods of data mining which include statistical methods, clustering, pattern-based methods and ordinal association rules for atomatic error detection in real dataset. Their Experiments showed tthaeeexperiments showed that ordinal association rule is more efficient than the other methods, but it is appropriate only for the datasets whose lots of their attribute type is decimal or date [6]. In this paper fuzzy association rules are used to enhance the data quality. The structure of this paper is as follows: Section 2 presents an introduction of data quality and reasons of using data mining for improving data quality. In Section 3, the proposed method is explained, and in Section 4, the experimental results of proposed method have been given and analyzed. 2 DATA QUALITY AND DATA MINING Definition of data quality depends on the considered purpose, so there is no unique definition which can be stated formally. In literature, "appropriate for use" or "to meet end user needs" are used. According to the general

2 JOURNAL OF COMPUTING, VOLUME 2, ISSUE 9, SEPTEMBER 2, ISSN definition of quality management, we define quality of data to meet customer needs. Contrary to popular belief, the quality is not necessarily error at the level of zero [7]. Data will have high quality if they are appropriate for applications, decision making and planning. Researches have defined different dimensions for data quality. Each dimension shows a particular view of quality. The more interesting dimensions are as follow: Consistency: the rate of violation from the defined significant rules on the dataset. Availability: the rate of data availability, being easy and the speed of data retrieval. Accuracy: the distance between value of v and value of v ' that represents the entity which v tries to portray it. Completeness: the database includes all of the related entities. Value added: the rate of being useful. Data mining is an automatic process to extract the patterns as an implicit knowledge in the database. Data mining utilizes multiple scientific fields simultaneously, such as the technology of data base, artificial intelligence, machine learning, neural networks, statistics, pattern recognizer, knowledge based systems and information retrieval. Data mining methods are appropriate for improving different dimensions of data quality, since they are aimed at finding abnormal patterns in large volumes of dataset. In 2, Hipp introduced data quality mining as a new approach from research and application viewpoint. The goal of data quality mining is to employ data mining methods in order to detect, quantify, explain and correct data quality deficiencies in huge databases [4]. In this paper, the accuracy dimension of data quality has been considered using data mining method. 3 THE PROPOSED IDEA As Among data mining methods, association rules is a very active research topic that has been used widely in lots of cases such as basket analysis, decision maker support systems, theft detection, etc. Data quality is also one of the topics where using association rules can be useful, because In comparison with other methods of data mining, association rules are understandable and can clearly discover and describe the dependency between data. Also association rules are independent of each other. Therefore, recision of simpler rules has no effect on the other rules. One of the defects of using association rules for data cleaning is that association rules can not show properly the associations between quantitative values. As an example in Table I, job and degree are categorical attributes and is numerical attribute. If we want to obtain association rules in these data with support>, via Apriori algorithm, only we can find some BS degrees, whereas there is an association between and job or degree and. For example, some one who have a high degree level have more than the people who have a mean degree level. Apriori algorithm is not capable of discovering such associations between quantitative values. As much of data in real world contain quantitative values and there may be errors in this type of values, the proposed method in [4] cannot be an applicable to detect error. In this paper, the presented method in [4] will be extended and association rules between quantitative values will be discovered with the help of fuzzy sets concept. 3. Fuzzy Association Rule Fuzzy association rules are the rules which are extracted from a fuzzy dataset. A fuzzy rule in the form of A B where A, B are fuzzy sets and A B, is called fuzzy association rule. Using of fuzzy concepts has some advantages. First, the discovered association rules are more understandable. These concepts make a transition between numerical values of data and categorical concepts. Second, these concepts help discover the rules between quantitative values. For example, the following fuzzy sets are defined for the data shown in Table I: High job level = {manager, deputy} Mean job level = {designer} High degree grade = {PHD, MS} Mean degree grade = {BS, 2th} High = {4-5} Mean = {-2} TABLE INSTANCE DATABASE Trans_id Job Degree Income desiger BS 2 desiger BS 5 3 desiger 2th 2 4 manaer PHD 4 5 manage MS 45 6 deputy PHD 5 The following fuzzy association rules may be stated by the mentioned fuzzy sets: High job level High degree grade Mean job level Mean degree grade High job level High Mean job level Mean Now assume the following transaction: Transaction_id 7: job = deputy, degree = PHD, = Considering the dataset shown by Table I, an error can be detected in the above transaction, as the person who is deputy cannot have of, while his is expected to be much higher. This transaction violates "High job level High " fuzzy rule. It should be noted that the error would not be detected by the rules extracted by Apriori algorithm. 3.2 proposed method The proposed method is consisted of steps that the description of each step is as follows:

3 JOURNAL OF COMPUTING, VOLUME 2, ISSUE 9, SEPTEMBER 2, ISSN a) Data preprocessing: in this step, data is converted to standard format and lost values are managed. For example, the values of date attribute in some may be in the form of YYYY/MM/DD and in some other in the form of YYYY-MM-DD. To obtain a correct model of the associations, implementation of this step is necessary. b) Mapping data to fuzzy values: fuzzy sets and fuzzy membership functions for quantitative data are defined in accordance with the knowledge of experienced experts. Then, data values are mapped to fuzzy values by fuzzy membership functions. Table II shows fuzzy mapping for and job attributes of Table I. Fuzzytrans_id TABLE 2 MAPPING TABLE TO FUZZY VALUE High Mean High job level Mean job level.3. c) Fuzzy association rules extraction: Fuzzy association rules are extracted, via the presented algorithm in [9], in which all attributes are weighted equally. In this step, the rules with high confidence level are considered. Thus, rule set R is limited to R r R confidence ( r) where is the minimum confidence. Table III shows some extracted rules from Table II. TABLE 3 FUZZY ASSOCIATION RULES EXTRACTED FROM TABLE 2 Fuzzy association rule Confidence High job level High.44 Mean job level Mean.48 d) Consistency check of with discovered rules: the consistency of with discovered rules is checked in this step. Each of may violate some rules and may be consistent with other rules. Also, it may not fire some of the rules. Let R be the set of fuzzy association rules and Let D be the database of fuzzy. Let r X Y be a Fuzzy association rule and be membership function of X. X the mapping that determines whether a fuzzy transaction T D violates a rule r R, is defined as: In (), Y shows the degree that fuzzy transaction T violates r, the more Y, the more inconsistency of transaction T with rule r. As an example, the consistency check for transaction 7 is shown in Table IV. TABLE 4 CONSISTENCY CHECK FOR TRANSACTION6 WITH RULES IN TABLE 3 Fuzzytrans_id 7 Violate: D R [,] Y if X rule_ satisfy Y rule_ satisfy else rule_satisfy parameter is the threshold for fuzzy rule satisfaction. Noncorrelated rule Mean job level Mean Consistent rule High job level High degree grade Inconsistent rule High job level High e) Scoring and ranking the : A score is assigned to each transaction by summing the confidence values of the rules it violates. The score of each transaction is computed as following: score ( R T r R ) : D R confidence ( r). violate ( T, r) The tuning parameter R allows assessing the confidences depending on their value [4]. For example the score of transaction 7 is (.44). The that have scores higher than score_treshold, the minimal threshold for scores, will be presented as a list sorted according to the rules that are violated or hold. Based on this information together with her background knowledge the user will decide upon the trustworthiness of single or groups of similar and finally upon the quality of the whole dataset. Used algorithm with according to upper proposed method is shown in figure. Preprocess records // for enhancing data quality Map dataset to fuzzy value Generate fuzzy association rules with determined support and confidence For all unmarked transaction in the dataset For all generated fuzzy association rules If transaction doesn t satisfy the rule Then mark transaction as a possible error and save confidence of rule End for Set score of each transaction with summing saved confidence End for Sort marked transaction by the score in descending order Output marked with consequent of the violated rules Fig.. Proposed algorithm

4 JOURNAL OF COMPUTING, VOLUME 2, ISSUE 9, SEPTEMBER 2, ISSN EVALUATION RESULTS In this section, proposed method is evaluated and the results are compared with presented method in [4]. All the evaluations have been implemented on a system equipped with Core 2 Duo 2.26 GHz CPU, 3GB RAM and Win 7. Usually for evaluating the performance of information retrieval and classification algorithms, precision and recall measures are used. These measures are defined by (3) and (4) based on the confusion matrix entries, shown in Table V. In this paper, in addition to these two measures, the runtime of the two methods has been also compared. Real Classification Correct Incorrect Precision = TP TP FP Recall = TP TP FN TABLE 5 CONFUSION MATRIX Transactions classification by proposed mthod Correct TP FP Incorrect FN TN In order to evaluate the proposed method, two datasets has been used. One of the datasets is Adult dataset from UCI and another dataset is real dataset that is Personnel data of IT company. Adult includes two numerical attributes called and hpw and two categorical attributes called job and degree. Also, personnel data includes two numerical attributes called and valuation score and two categorical attributes called job and degree. instances of the of this datasets have been selected and manipulated to make imitation errors. Then, proposed method and the presented one in [4] are applied to the dataset. In the tests related to precision and recall, min_support, min_confidence, score_treshold and min_rule_satisfy are considered to be.2,.7,.3 and.4, respectively. The tuning parameter is equal 7 in two methods. Because based on the experimental result of Hipp this value is well when it is important for us that don t violate the rules with high confidence value [4]. Figure 2 compares the precision of the proposed method with the method proposed in [4], in terms of number of created errors in Adult and Personal datasets. The figure shows that the proposed method outperforms the method proposed in [4], as lots of the association between quantitative data in datasets are not discovered by Apriori algorithm and play no role in ranking. Some attributes values are in a range which (3) (4) causes rules support would not be satisfied (above example). Precision curves have wavy shapes, because any change in data causes a change in the support of rules which can affect their detections. Precision Number Of Error In Dataset Adult Dataset-Fuzzy Association Rule Adult Dataset-Apriorit Real Dataset-Fuzzy Association Rule Real Dataset-Apriori Fig. 2. The precision measure for the proposed method and the method described in [4] for two datasets Figure 3 also compares the recall measure of Adult and Personal datasets for two methods. It is clear that the proposed method delivers better results. Recall Number Of Error In Dataset Adult Dataset-Fuzzy Association Rule Adult Dataset-Apriorit Real Dataset-Fuzzy Association Rule Real Dataset-Apriori Fig. 3. The recall measure for the proposed method and the method described in [4] for two datasets Figures 4, 5 compare the precision and recall of the proposed method with the method proposed in [4], in terms of varying min_support and min_confidence for Adult dataset. The figures show that increasing the min_support or min_confidence causes the decreasing the precision and recall. Because increasing the min_support or min_confidence causes some association rules are not discovered and play no rule in error detection. Therefore some errors are not detected.

5 JOURNAL OF COMPUTING, VOLUME 2, ISSUE 9, SEPTEMBER 2, ISSN Precision-Recall Varying min_support Precision-Fuzzy Association Rule Precision-Apriori Recall-Fuzzy Association Rule Recall-Apriori Fig. 4 The precision and recall measure for the proposed method and the method described in [4] for Adult Figure 6 shows the runtime of the proposed method and the method proposed in [4]. Precision-Recall Varying min_confidence Precision-Fuzzy Association Rule Precision-Apriori Recall-Fuzzy Association Rule Recall-Apriori Fig. 5 The precision and recall measure for the proposed method and the method described in [4] for Adult RunTime Number Of Transactions Fuzzy Association Rule Apriori Fig. 6 The runtime of the proposed method and the mothod presented in [4] for Personal dataset According to figure 6, the runtime of the proposed method is better. The main reason is due to the difference in runtime of association rules discovery. Fuzzy association rules algorithm is much faster than Apriori algorithm, because Apriori algorithm must check all of the numerical and categorical data in the dataset to find frequent itemsets. Also, it should find the support of each of itemsets which is time consuming due to variety of numeric values. While fuzzy association rules algorithm only works with the fuzzy sets whose numbers are more limited and are defined by user. 5 CONCLUSION Automated data quality improvement is a necessity for finding incorrect data in large databases. In this paper, a method based on data mining approaches is presented to improve data quality. This paper s proposed approach for data quality improvement uses fuzzy association rules, by which hidden rules in datasets will be discovered. Then, the consistency of with these rules is checked and a score is assigned to each transaction. The high scores are assigned to the which are suspicious to have defects. User s knowledge on suspicious and background knowledge about data will be used in the process of determining the accuracy of and ultimately total quality of dataset. Evaluation results show the effectiveness of the proposed method. REFERENCES [] D. Luebbers, U. Grimmer, M. Jarke, Systematic Development of Data Mining-Based Data Quality Tools, Proc. of the 29-th International Conference on Very Large Data Bases, Berlin, Germany, pp , 23. [2] J. Hipp, M. Müller, J. Hohendorff, F. Naumann, Rule-Based Measurement of Data Quality in Nominal Data, Proc. of the 2th International Conference on Information Quality (ICIQ), Cambridge, USA, 27. [3] C. Brodley, M. Friedl, Identifying Mislabeled Training Data, Journal of Artificial Intelligence Research. Vol., pp. 3-67, 999. [4] J. Hipp, U. Güntzer, U. Grimmer, Data Quality Mining Making a Virtue of Necessity, Proc. of the 6-th ACM SIGMOD Workshop on Research Issues in Data Mining and Knowledge Discovery, Santa Barbara, California, pp , 2. [5] F. Grüning, Data Quality Mining: Employing Classifiers for Assuring consistent Datasets, Proc. of the 2-th Information Technologies in Environmental Engineering, Springer-Verlag Berlin Heidelberg, pp , 27. [6] J. Maletic, A. Marcus, Data cleansing: Beyond integrity analysis, Proc. of the Conference on Information Quality, MIT, Boston, pp. 2-29, 2. [7] J. Geiger, Data Quality Management: The Most Critical Initiative You Can Implement, Intelligent Solutions, Inc., Boulder, Paper 98-29, 24. [8] J. Maletic, A. Marcus, K. Lin, Ordinal Association Rules for Error Identification in DataSets, Proc. of -th Intl. conf. Information and Knowledge Management(CIKM), Atlanta, GA, pp , 2 [9] D. Olson, Y. Li, Mining Fuzzy Wighted Association Rules, Proc. of the 4th IEEE Intl. Conference on System Sciences, Hawaii, 27. []

6 JOURNAL OF COMPUTING, VOLUME 2, ISSUE 9, SEPTEMBER 2, ISSN Fatemeh Ghorbanpour A. received B.Sc. and M.Sc. degrees in Computer Engineering- Software from Iran Universities. Her current research interests include information retrieval and data mining. She is a member of the IEEE. Dr. M. Mohsen Pedram received his B.Sc., M.Sc. and Ph.D. in electronic engineering from the Iran universities. His major research interests are Machine Learning, Image Processing, Artificial intelligence, Data mining, and Pattern Recognition. He has published many papers in various fields. At present, he is teaching in Department of Computer Engineering in Iran. Dr. Kambiz Badie received his B.Sc., M.Sc. and Ph.D. in electronic engineering from the Tokyo Institute of Technology, Japan, majoring in Pattern Recognition & Artificial Intelligence. His major research interests are Machine Learning, Cognitive Modeling, and Systematic Knowledge Processing in general, and Analogical Knowledge Processing Experience-Based Modeling, and Interpretative Modeling in particular with emphasis on idea and technique generation. He has published many papers in various fields. At present, he is the Director of IT Faculty at Iran Telecom Research Center Mohammad Alishahi received B.Sc. degree in Computer Engineering- Software and M.Sc. degree in Industrial Engineering from Iran Universities. His current research interests include data mining and Management information systems.