Information Quality Assessment in Context of Business Intelligence System

Transcription

1 Information Quality Assessment in Context of Business Intelligence System Samuel Otero Schmidt Universidade de São Paulo, Brasil Maria Aparecida Gouvêa Universidade de São Paulo, Brasil Abstract This study identified the relevant Information Quality (IQ) dimensions in Business Intelligence (BI) system from user s perspective. A case study was conducted, through a sample of 170 individuals, in a big Brazilian financial company. Results highlighted twenty IQ dimensions that can be used by companies in evaluation of their IQ. Keywords: Information Quality, Business Intelligence System, Information Technology. Introduction The BI system is important to help companies to be competitive, providing more accurate information for the decision making. Organizations require these systems to improve performance and increase profits (Lonnqvist and Pirttimaki 2006). The studies conducted by Luftman et al. (2010) and Luftman et al. (2012) include an annual survey conducted with over 600 companies across the world and more than 150 companies in the United States, on the priorities of IT investments and the concerns of executives. Investment in BI was determined as the number one priority in 2010 and 2011 at the global level. In the United States, this investment has been kept in the first place since These same authors state that IT executives believe their companies are rich in terms of the amount of data, but still lack quality information. According to Lonnqvist and Pirttimaki (2006) and Popovic et al. (2009), one of the main direct benefits of BI systems is to improve the quality of information for users, providing more accurate data, easily accessible and more consistent. These aspects permeate the concept of Information Quality (IQ), which according to Kahn and Strong (1998) is the characteristic of the information regarding to being in line with the specifications and meet or exceed the users expectations. Thus, IQ can be used as a way to subjectively evaluate the success of the BI system through the users perception (Popovic and Jaklic 2009). The IQ is one of the most used concepts in studies about information systems (Lee et al. 2002, Delone and Mclean 1992). The authors Rai et al. (2002) and Delone and Mclean (1992) point out that this concept may individually impact 1

2 on the activities of users, which makes them more effective, creating more value for the organization. There are several authors, who have analyzed IQ and BI before, but there is no consensus on which dimensions are important to the context of the BI system (Hou 2012, Isik et al 2011, Nelson et al 2005). Therefore this will be the research problem: Which IQ dimensions are relevant in the context of the BI system from the users perspective? Objectives and Hypothesis The objectives of the study are: (I) Identify which and how many dimensions are actually categories that are relevant to the user of a BI system; and (II) Identify the dimensions that most influence the overall IQ of the BI system. The hypothesis of the study and the corresponding references that inspired its formulation are shown in Table 1. Table 1 Hypothesis Hypothesis Description References H1 H2 There are relevant categories of IQ dimensions that determine the user s perception of the BI system. There are dimensions that have a greater relative influence on the overall IQ than others. (Kandari et al 2011, Alkhattabi et al 2010) (Alkhattabi et al 2010) Background - Information Quality (IQ) and Business Intelligence (BI) According to Kahn and Strong (1998), IQ is the information characteristic and is also the indicator to measure if the users expectations were met. Bailey and Person (1983) identified the complexity of measuring IQ from the users viewpoint; so in order to enable this measurement, they developed a tool to evaluate the satisfaction of IT systems users through the use of IQ dimensions, for example: precision, reliability, currency, security. Other important aspects are which tools are used to evaluate how the IQ dimensions are influenced by Information Systems (IS) contexts inside the companies (Ong and Lai 2007, Ge and Helfert 2007, Wu and Wang 2006) and the fact that an evaluation tool dedicated to BI context was not identified on the literature. On the basis of the literature, particularly on studies regarding IQ and BI systems context, 30 dimensions were selected to be part of this research and can be considered to have relevance for this context (Table 2). 2

3 IQ References / Dimensions Ariyachandra and Watson (2006) Bharati and Chaudhury (2004) Table 2 IQ dimensions and references Isik, et al (2013) Doll and Torkzadeh (1988); Hou (2012); Chen et al (2000) Nelson et al (2005) Popovič et al (2009) Eppler (2006) Wang and Strong (1996) Comprehensiveness Accessibility Currency Clarity Completeness Conciseness Believability Consistency Content Convenience Correctness Believability Understandability Ease of use Format Interactivity Interpretability Free-of-error Objectivity Relevancy Accuracy Appropriate amount of data Lee et al (2002) Pipino et al (2002); Wixom and Watson (2001) Jarke and Vassiliou (1997) Boove et al (2003) Delone and McLean (1992) Delone (2003) Yeoh and Koronios (2010) Foshay et al (2007) Stvilia et al (2007) Total Traceability

4 Reputation Security Timeliness Value-added Total Research Method This research is exploratory, quantitative and the methodology employed in only based on a single case study with a survey. This strategy investigates contemporary phenomena in their real context, where the boundaries between a phenomenon and the context itself are not clearly defined (Yin 2010). The case study was conducted at a large Financial Company (Company A) in Brazil. Company A has more than 30,000 employees of which over 2,000 work in the area of IT. The main activities of this company are accounting, transactions, issuing credit cards, internet banking and insurance. There are offices and agencies spread throughout all of the states in Brazil and other countries. The revenue is more than $15 billion dollars per year. The annual investments on BI systems projects and maintenance are more than $500 million dollars. There are over 3,000 users of BI systems. The survey was conducted through a web questionnaire. The survey was conducted in the second half of It is worth to note that the with the invitation to participate in the survey was sent to approximately 1,500 people and obtained 175 participants. The analysis of the data was based on the recommendations of Hair et al. (2010). There was no occurrence of missings in the metric variables. Three outliers were identified and removed from the sample. We also conducted an analysis of consistency of observations, considering the scores given by the respondents for all independent variables (30 dimensions of IQ) and the score given to the dependent variable (Overall score of IQ). As result of this analysis, two cases were removed. The univariate normality was assessed using the Kolmogorov-Smirnov test. Out of the 31 variables, 23 accept the hypothesis of normality. Normality is not a strong assumption for the use of multivariate techniques. Therefore, no corrective actions will be taken in relation to the few variables that do not have a normal distribution. Data Analysis Exploratory Factorial Analysis To distinguish the term IQ dimension (for the variables used in this study) of the term data dimension, which is used to express the results of the groups of variables of factor analysis, the term dimension will be used only in the context of IQ, while the term data dimension will be replaced by the term factor. After twelve rounds, the model stabilized with very good KMO and MSA, above 0.9; satisfactory commonalities; 66% of total explained variance; four factors with 20 dimensions; and well-defined factor loadings, which are shown in Table 3. Table 4 shows the total explained variance for each factor: Factor 1 explains 49%; Factor 2 explains 6.9%; Factor 3 has 5.7% and the last factor explains 4.9% of the total variance. 4

5 Table 3 Rotated Component Matrix Variables Factors V2b.768 V3b.668 V4b.679 V5b.691 V6b.665 V7b.558 V8b.552 V11b.801 V14b.698 V15b.672 V16b.503 V17b.635 V19b.627 V20b.723 V21b.802 V22b.802 V23b.647 V24b.653 V25b.729 V28b.666 Table 4 Total Explained Variance Stabilized Model of Factor Analysis Factor Initial Eigenvalues Total Variance % Accumulated %

6 After the validated model, it was possible to interpret and label the factors. According to Hair et al. (2010), when performing the interpretation of the factors, it is necessary to use a theoretical framework and pay attention to the variables with higher factor loadings. The IQ dimensions associated with each factor with their respective names are shown in Table 5. Table 5 Denomination of the Factors Factor Category Variable IQ Dimension Factor Loading 1 Content Quality 2 Intrinsic Quality 3 Information Value 4 Contextual Quality V19b Content1.627 V20b Content2.723 V21b Content3.802 V22b Content4.802 V23b Comprehensiveness.647 V24b Clarity.653 V28b Convenience.666 V4b Consistency.679 V6b Accessibility.665 V7b Understandability.558 V11b Believability.801 V15b Reputation.672 V17b Free-of-error.635 V5b Relevancy.691 V14b Objectivity.698 V25b Value-added.729 V2b Timeliness.768 V3b Completeness.668 V8B Currency.552 V16b Appropriate amount of data.503 It can be seen that Factor 1 has seven IQ dimensions, and four of them (Content1, Content2, Content3 and Content4) represent a single concept (dimension Content) according to Doll and Torkzadeh (1988). These variables have been widely used in the literature on IQ and in the context of BI (Bharati and Chaudhury 2004, Hou 2012, Chen et al. 2000) which confirms the importance of these variables for this factor and for this study. There are also three other dimensions (Comprehensiveness, Clarity and Convenience) associated with this factor. The dimensions Comprehensiveness and Clarity have conceptual basis for this association in Eppler (2006) and Popovič et al. (2009) as they categorize as Content Quality. The dimension Convenience is the one that has the least theoretical background, as these authors categorize it as 6

7 Media Quality. After this comparison with the literature, we decided to name Factor 1 as Content Quality due to the conceptual affinity. Factor 2 has six dimensions (Consistency, Accessibility, Understandability, Believability, Reputation and Free-of-Error) and we followed the same process of comparison with the literature, where it was possible to identify that most of these dimensions predominated in the category Intrinsic, which is included in the studies of Wang and Strong (1996), Jarke and Vassiliou (1997) and Ballou et al. (1998). The only two dimensions that do not fall in this category are: Accessibility and Understandability. For Wang and Strong (1996), the first has a specific category called IQ Accessibility. For these authors, this dimension shares this category with the dimension Security (indicated as variable V12b in this study); however, the dimension Security was not included in the model, which explains why the dimension Accessibility was associated with the dimensions of the category Intrinsic. The dimension Understandability, on the other hand, according to the same authors, is part of the category Representational, which supports its categorization in Factor 2. Even with lower factor loading (0.558), it is believed that the user has the perception that this dimension is associated with the Intrinsic aspect of the data. Based on these studies, we named Factor 2 as Intrinsic Quality because there is a large part of dimensions associated with the category Intrinsic and a relevant dimension that refers to Accessibility. Factor 3 has a predominance of the relationship between the variables Relevancy, Objectivity and Value-Added. Compared to other studies, we found no clear predominance of one specific category to classify them. We noticed that this group is peculiar to this research, showing that the benefits generated by the use of information in the daily tasks are particularly relevant to users of BI systems. Therefore, this factor was named Information Value. The last factor has four dimensions (Timeliness, Completeness, Currency and Amount of Data). Three factors, except Currency, are categorized as Contextual (Wang and Strong 1996). Currency has a low theoretical support to be in this fator; however, in the user s perception it is highly associated with Timeliness and Completeness. This may occur due to the fact that BI systems have a large amount of data history with high levels of detail, also known as high granularity. It is the function of the component DW to keep this history available to the user. Therefore, there is this peculiarity in the context of the BI system, which strengthens the association between these four variables. Therefore, we decided to name Factor 4 as Contextual Quality. Therefore, as a result of the Exploratory Factor Analysis, we obtained four relevant factors to assess the IQ perception in the BI context. The factor scores obtained were used as independent variables in the multiple regression analysis to identify how much they can forecast the general IQ of the BI system. Multiple Regression To identify the relative importance of all dimensions in relation to their respective factors, multiple regression can be used, with the Enter method, to forecast the factors. This analysis of relative importance was conducted in the study of Alkhattabi et al. (2010) involving the IQ dimensions in the context of e-learning systems. After finding the partial squared correlation coefficients, they must be summed to identify the relative importance using equation 1. 7

8 (1) where is the partial correlation for the variable in the corresponding factor. Based on the equation of relative importance, we calculated the relative importance for the four factors in relation to the overall IQ (Table 6). After the calculation, it was possible to consolidate the relative importance of each dimension within the four factors and the importance of each factor in the overall IQ (Table 7). Table 6 Relative importance of factors in relation to the overall IQ Factors Squared partial correlation Relative Importance overall IQ Content Quality % Intrinsic Quality % Information Value % Contextual Quality % Factor Content Quality (1) Intrinsic Quality (2) Contextual Quality (4) Table 7 Relative importance of IQ dimensions and factors Importance (Factor -> Overall IQ) 47.90% 32.10% 13.10% Variables Importance (Dimensions -> Factors) V21b 42.7% V22b 30.6% V24b 19.9% V20b 3.0% V23b 1.4% V28b 1.3% V19b 1.1% V11b 23.7% V4b 17.0% V15b 16.6% V6b 16.3% V17b 14.9% V7b 11.5% V2b 37.0% V3b 28.0% V8b 19.1% V16b 15.9% 8

9 Information value (3) 6.90% V14b 35.5% V5b 32.6% V25b 31.9% Conclusions The exploratory factor analysis identified that at least 20 of the 30 dimensions are important in the context of the BI system. We identified four factors named as IQ categories: Content Quality ; Intrinsic Quality, Information Value and Contextual Quality. Therefore, H1 was confirmed. H2 was confirmed, as the hierarchy of dimensions was obtained through the relative importance identified through the results of the Multiple Regression, with the Enter method. It was found that the categories Content Quality and Intrinsic Quality have a greater importance in determining the overall IQ of the BI system from the user s perspective than the other two categories. This result corroborates the study of Alkhattabi et al (2010), which highlights the importance of identifying categories for the IQ dimensions, facilitating their understanding and prioritizing the user s perspective in a given context. In order to meet the objective I, we identified 20 out of the 30 IQ dimensions, raised in the literature review that form four categories, Content Quality, Intrinsic Quality, Information Value and Contextual Quality, which determine the IQ on the user s perspective in the context of the BI system. Objective II was met and its results confirm that the IQ categories may vary depending on the context, as stated by Kandari et al (2011) and Alkhattabi et al (2010). Overall, the categories and their relative importance identified in this study may be useful for IT analysts and managers, who develop or are responsible for providing information in BI systems, as the IQ dimensions and categories detected may help understand the user s perception, identifying problems and prioritizing actions to improve the IQ of these systems. There are some limitations such as: (a) convenience sample rather than probabilistic, which reduces the generalizability of the results; (b) the difficulty to obtain a stratified sample of the users of BI systems by department in the company selected; (c) the study was applied to only one company in the financial sector, being necessary to replicate it in other sectors to allow the generalization of the results. For future studies, we suggest: (a) using the model with the four categories of IQ dimensions proposed to validate if indeed they are relevant in other samples; (b) conduct case studies in other companies to compare the results. References Alkhattabi, M., D. Neagu, A. Cullen Information quality framework for e-learning systems. Knowledge Management & E-Learning: An International Journal 2(4): Ariyachandra, T., H. J Watson Which data warehouse architecture is most successful? Business Intelligence Journal, 11(1): 4 6. Bailey, J. E., S. W. Pearson Development of a tool for measuring and analyzing computer user satisfaction. Management Science 29(5): Bharati, P., A. Chaudhury An empirical investigation of decision-making satisfaction in web-based decision support systems. Decision support systems 37(2):

10 Bovee, M., R. Srivastava, B. Mak A conceptual framework and belief function approach to assessing overall information quality. International journal of intelligent systems 18(1): Chen, L. D., K. S. Soliman, E. Mao, Frolick M. N Measuring user satisfaction with data warehouses: an exploratory study. Information & Management 37(3): DeLone, W.H., E. R McLean Information Systems Success: The Quest for the Dependent Variable. Information Systems Research. 3(1): Delone, W. H The DeLone and McLean model of information systems success: a ten-year update. Journal of Management Information Systems 19(4): Doll, W., G. Torkzadeh The measurement of end user computing satisfaction. MIS Quarterly 12(2): Eppler, M. J Managing information quality: increasing the value of information in knowledge-intensive products and processes. Springer. Foshay, N., A. Mukherjee, A. Taylor Does data warehouse end-user metadata add value?. Communications of the ACM 50(11): Ge, M., M. Helfert A review of information quality research Develop a research agenda. In Proceedings of the 12th International Conference on Information Quality. Cambridge, MA: MIT. Hair, J., W. Black, B. Babin, R. Anderson Multivariate data analysis (7th ed.). Englewood Cliffs: Prentice Hall. Hou, C. K Examining the effect of user satisfaction on system usage and individual performance with business intelligence systems: An empirical study of Taiwan s electronics industry. International Journal of Information Management 32(6): Isik, O., M. C. Jones, A. Sidorova Business intelligence success: The roles of BI capabilities and decision environments. Information & Management 50(1): Jarke, M., Y. Vassiliou Foundations of Data Warehouse Quality A Review of the DWQ Project. In Proceedings of the 2nd International Conference on Information Quality Kahn, B. K., D. M. Strong Product and service performance model for information quality: An update. In Proceedings of the 1998 Conference on Information Quality. Cambridge, MA: MIT Kandari, J., E. C. Jones, F. F. H. Nah, R. R. Bishu Information quality on the World Wide Web: development of a framework. International Journal of Information Quality 2(4): Lee, Y. W., D. M. Strong, B. K. Kahn, R. Y. Wang AIMQ: a methodology for information quality assessment. Information & Management 40(2): Lonnqvist, A., V. Pirttimaki The measurement of business intelligence. Information Systems Management 23(1): Luftman, J., T. Ben-zvi Key issues for IT executives 2010: judicious IT investments continue postrecession. MIS Quarterly Executive 9(4): Luftman, J., H. S. Zadeh, B. Derksen, M. Santana, E. H. Rigoni, Z. D. Huang Key information technology and management issues : an international study. Journal of Information Technology 27(3): Nelson, R., P. Todd, B. Wixom Antecedents of information and system quality: an empirical examination within the context of data warehousing. Journal of Management Information Systems 21(4): Ong, C. S., J. Y. Lai Measuring user satisfaction with knowledge management systems: Scale development, purification, and initial test. Computers in Human Behavior 23(3): Pipino, L., Y. W. Lee, R. Y. Wang Data Quality Assessment. Communications of the ACM 45(4): Popovic, A., P. Coelho, J. Jaklic The impact of business intelligence system maturity on information quality. Information Research 14( 4). Stvilia, B., T. Gasser, M. B. Twidale, L. Smith A framework for information quality assessment. Journal of the American Society for Information Science and Technology 58(12): Wu, J. H., Y. M. Wang Measuring KMS success: A respecification of the DeLone and McLean s model. Information & Management 43(6): Wang, R. Y., D. M. Strong Beyond accuracy: What data quality means to data consumers. Journal of management information systems 12(4): Wixom, B., H. Watson An Empirical Investigation of the Factors Affecting Data Warehousing Success. MIS Quarterly 25(1): Yeoh, W., A. Koronios Critical success factors for business intelligence systems. Journal of Computer Information Systems 50(3): Yin, R. K Case Study Research: Design and Methods (4th ed.). Sage: London. 10