Online Supplement: A Mathematical Framework for Data Quality Management in Enterprise Systems

Transcription

1 Online Supplement: A Mathematical Framework for Data Quality Management in Enterprise Systems Xue Bai Department of Operations and Information Management, School of Business, University of Connecticut, Storrs, Connecticut xue.bai@business.uconn.edu 1 Detailed Literature Review Our work follows a large body of data quality research in the accounting literature and the information systems literature. The literature in accounting has mainly studied the reliability of internal control systems and error characteristics in accounting data (Yu and Neter, 1973; Cushing, 1974; Hamlen, 1980; Stratton, 1981; Ham et al., 1985; Lea et al., 1992). Yu and Neter (1973) developed a Markov model to evaluate the reliability of internal control systems. The authors first model the financial information system as a stochastic process. They then define the error states, the operational probability matrix, the transformation probability matrix, and the control policy matrix. The outcome error probabilities are computed by multiplying these matrices. Their approach presents a first attempt to the quantitative and probabilistic estimation of the reliability of internal control systems. However, the work presents no specifications with regard to the control allocations, nor have the authors provided control solutions to error reduction in the system. Our research adopts the same basic concepts of data errors and control in this work, but further extends it by explicitly considering the control allocation as a decision variable. Cushing (1974) developed a mathematical model for measuring the reliability of an internal control system. He developed a measure of reliability using the probability that the system is error-free. From that he estimates both the cost of executing error correction procedures and the risk of undetected errors in the system. Cushing s reliability model, however, considers the control allocations as fixed, and it does not provide solutions toward 1

2 meeting a threshold reliability measure under cost constraints. Our research extends Cushing s model by adding task-level attributes about errors and control procedures, and provides error reduction strategies. Hamlen (1980) proposed a mixed-integer programming model for the design of an internal control system. Hamlen s model minimizes the cost of controls subject to a given percentage of quality improvement desired in the system s output. The system is modeled as a set of controls that can correct a set of error types in various ledgers. However, solving the problem is NP-hard. We extend Hamlen s approach by considering realistic control features that may be selectively applied to individual error sources, and quantifies the system-wise impact of control application on data errors of in the audit targets that are linked directly or indirectly to the error sources. Our model, though more sophisticated than Hamlen s, is easier to solve. Stratton (1981) proposed a reliability model to evaluate an internal control system. The author first simulates a raw material purchasing process with weaknesses, which results in data errors in ending dollar balances. The reliability model is then used to estimate the parameters associated with the reliability function and the reliability importance scores. The model takes the control allocation as fixed. Controls and process execution are modeled as binary variables with one being correctly performed; and zero otherwise. Error probabilities in the ending balances are then calculated based on the error status of the process. Similar to Yu and Neter (1973) and Cushing (1974), no optimization formulation was developed to offer better control allocation solutions under cost or risk constraints. Ham et al. (1985) studied the error characteristics in accounting systems in several industries. The study examined the empirical distributions of four types of error rates and their implications to accounting data. The types of error rates studied include the ones that are defined in terms of the frequency of erroneous transactions, and those defined in terms of the magnitude of erroneous monetary value. We adopt two error metrics from their study, the error incidence rate and the proportion of net monetary error, as measures of error magnitude in our model. Lea et al. (1992) presented an audit risk assessment model in accounting systems. The authors model how the risks of error in various transaction streams are related to the risk of error at the account balance level to which they contribute. The authors showed that the level of tolerable error at the transaction stream level cannot be assumed to be the same as that for the account balance level. Our model follows their motivation to decompose an account balance into its constituent transaction streams, and to include in our model the 2

3 volume of transactions in various transaction streams as well as the network structure of these streams among transaction sources, error sources, and audit targets. Research on data quality in the information systems literature is extensive and diverse. Notable publications include but are not limited to work on data integrity enhancement in databases (Ballou and Tayi, 1989), data quality analysis in relational databases (Wang et al., 1995), assessment of information quality in information manufacturing systems (Ballou et al., 1998), data quality management at the organizational level (Wang, 1998), functional forms of data quality (Pipino et al., 2002), process-oriented ontologies for data reliability assessment in accounting information systems (Krishnan et al., 2005), and multi-dimensional data quality assessment methodologies (Even and Shankaranarayanan, 2009). Existing work has so far either focused on the control strategies in the data repositories or the identification of important characteristics that define data quality. Most studies have viewed the underlying business processes as a black box. There has been little research done at the task level in business processes, where data are actually produced and transformed. Our work fills the gap in literature by tackling the data quality issues at the task level in business processes. Ballou and Tayi (1989) presented methodology for allocating resources to maintain data integrity in multiple data sets. Their methodology takes into account data quality attributes such as error rates, costs, and effectiveness of data maintenance procedures. An integer programming model is used to identify the most effective distribution of the data maintenance resources available for identifying and correcting data errors. Using simulated examples and a heuristic procedure, the authors illustrated how the model parameters can be accurately obtained to facilitate the IP optimization. Wang et al. (1995) proposed an attribute-based data model with data quality indicators. This model is developed based upon query algebra and data integrity rules, and aims to facilitate cell-level tagging with quality measures of data in the relational database. The authors showed that using these indicators, the user could assess the quality of data for the intended application. Along with the data model, an ER-based data quality requirement analysis methodology was presented to specify the types of quality indicator to be modeled. Wang (1998) developed a set of concepts, principles, and procedures for defining, measuring, analyzing, and improving information products. The purpose of the approach is to facilitate the implementation of an organization s overall data quality policy. Ballou et al. (1998) proposed a model to measure the quality of information products in information manufacturing systems. Their model considers key system parameters such as 3

4 timeliness, data quality, and the value and cost of information products to the customer. In particular, the output data quality of the system is defined as a function of both input data quality and processing effectiveness. A mathematical formulation is developed for identifying the optimal configurations of the key system parameters under study. Pipino et al. (2002) introduced a set of functional forms to measure the quality of data in organizational databases. The measures proposed include simple ratios, min or max operators, and weighted averages. Based on the functional forms, the authors developed metrics for various data quality dimensions. Using these metrics, the authors presented an approach that combines subjective and objective assessments of data quality, and demonstrated its effectiveness in practice. Even and Shankaranarayanan (2009) proposed a dual-assessment methodology for data quality in customer databases. This methodology considers two types of data quality concepts, impartial data quality and contextual data quality. The authors demonstrated the methodology in the context of Customer Relationship Management (CRM), using large data samples from real-world datasets. Computational steps involved in the methodology are provided, and implications for applying the methodology in other business contexts and data environments are discussed in the study. Krishnan et al. (2005) developed a process-oriented ontology for accounting information systems and demonstrated how this ontology can be used to support audit and control planning using a set-covering decision model. Their ontology specifies an accounting information system in terms of Information Transformation Processes (ITPs) and information flows. Although the concept of ITPs differs from that of atomic tasks in the standard BPMN notation (OMG, 2006), the focus on ITPs is well suited to the accounting information systems settings. In our study, we adopt the ontological structure from Krishnan et al. (2005) in modeling information flows, error introduction, and error aggregation in a general business process setting. The key elements of our process model hence include transaction sources, tasks (or error sources), controls, and audit targets, which are similar to the concepts of economic events, ITPs, control procedures, and general ledger accounts, respectively, in Krishnan et al. (2005). This more general terminology is necessary and suitable to our problem context because it enables the study of data quality issues in broader enterprise settings than accounting information systems. Furthermore, we extend the ontology in Krishnan et al. (2005) to include quantitative attributes about the flow of transactions, probabilities of errors at the tasks, and the effectiveness and cost of the controls designed to detect and 4

5 correct errors. This extension enables our objective of quantitative data quality assessment, and optimization of the placement of controls. In summary, despite the large and diverse literature on data quality in the information systems field, there has been little quantitative work that offers practical and effective solutions for managing data quality in an enterprise system at the task level in business processes. Since the data in an enterprise system are actually transformed, and data errors introduced, through tasks in a business process, we argue that a task-level approach to data quality control would be more effective and practical than the traditional approaches. Our work contributes to the literature by developing task-level control solutions to data quality management. 2 Description of Control Procedures (Table 1). 5

6 Table 1: Description of the available control procedures in the revenue realization process. Units U1 U2 U3 U4 U5 U6 U7 U8 U9 U10 U11 U12 Description Identify channel transfer transactions to be reviewed and approved by accounting and brand financial management at the time of the request. For those transfers not approved, any revenue recorded will be reversed. Provide information on specific equipment that is uninstalled >30 days prior to review date with a value greater than an established clip level. This information will be reviewed by accounting and by brand financial management, and a decision will be made to reverse any recorded revenue, if appropriate. Validate each transaction to ensure it meets all published firm order guidelines prior to claiming revenue. Review for persuasive evidence of an arrangement, fixed or determinable fees, and reasonable expectation of collectability. This may contain but is not limited to contracts, offer letters, e-pricer reports, purchase orders, and customer correspondence. Assess customer credit based on published transaction guidelines. For cash transactions valued > 250K & > 100K for a third party, the backlog contract support review must have an approved Company Global Financing credit evaluation. For transactions under the cash and third party clip levels, paytrend analysis must be done. Review accuracy of transaction pricing against supporting pricing documents, and review for the existence of unusual contract terms or future products at no charge or substantial discounts, trade-in price/term exceptions, or transfer of credit from one contract to another. Review transactions for appropriate supplements and contracts, and reference the correct base agreements. Transactions should be reviewed for unusual contract terms or future contingencies, including but not limited to exceptions to standard provisions for risk of loss, title, or installation; acceptance criteria; or performance guarantees. Ensure appropriate contract management of complex or special offerings. Transactions are reconciled based on established reconciliation schedule. Revenue adjustments are processed accordingly. Validate each transaction to assure it meets all published firm order guidelines prior to claiming revenue. Review for persuasive evidence of an arrangement, fixed or determinable fees, and reasonable expectation of collectibility. On a monthly basis, the process owner selects 30 orders for which the Company Global Financing transaction audit number is not electronically retrieved through SAP and submits list to Company Global Financing to verify that the transaction audit number was valid and appropriate for that order; the results of the review are reported to the business process owner by Company Global Financing. Review accuracy of transaction pricing against supporting pricing documents and review for the existence of unusual contract terms or future contingencies. Identify all shipments with 500 units or more to a single location for the shipment month. The shipment address for these transactions are certified to ensure that they are valid end user locations and that any third party locations do not involve any company financial consideration for storage costs. Identify shipments in the testing month that have ship to override activity. Address changes are reviewed to assure they are valid end users. 6

7 3 Model Implementation Steps The implementation of our proposed methodology comprises the following steps. 1. Create a process model for an existing business process of interest (for example, the modeling diagram in Figure 1 in the paper). Various modeling tools are commercially available for this purpose. 2. Identify the transaction sources, error sources, and audit targets using the process modeling framework developed in Section 2 of the paper For transaction sources, obtain or estimate the volume of transactions (V s ) over a given time period (e.g., per day, month, quarter, etc.) and estimate the average book values (z s ). This may be a simple average book value or a probability distribution based on historical transaction data For error sources, obtain the probability of errors prior to the application of any controls (p iɛ ). This may be obtained from the logs of controls that already exist. For a new business process or for error sources that do not have logs of past control activities, an estimation must be done based on comparable error sources with available data. The taint (ρ i ) of the error sources must also be obtained from historical logs or otherwise estimated For audit targets, specify the error classes (ɛ) that need to be asserted, and the threshold risk due to undetected errors ( ˆR) the control system needs to achieve. 3. Estimate error metrics (α incidence (ɛ, j)) and (α monetary (ɛ, j)), and the risk due to errors in the audit targets (R). For a model with probability distributions, a Monte Carlo simulation can be performed to estimate error metrics and the expected loss in terms of probability distributions. The process analyst may develop multiple scenarios in order to test different expectations of future process changes, including changes in transaction volumes, business process topology, and policies. 4. Apply the control framework developed in Section 2.4 to associate error sources with a set of control units. These control units are existing and available ones. For each control, estimate the probability of detecting and correcting errors (q ki ). This data is available if the control units are periodically subject to internal or external auditing, 7

8 where they are evaluated with test data with errors that are known a priori. The cost of control (ξ k ) can be estimated from the time and resources a control unit requires to search for and then fix errors. 5. Construct the three-dimensional array (Θ) using the mapping schema described in Section 2.4.2, and the matrix of costs of control (Ω) described in Section The process analyst may run multiple scenarios with different control selections as well as the scenarios developed in step 3 above. 6. Apply the optimization model to find the optimal control strategy for a data-quality related managerial problem of interest. As a special case developed in Section 3 of the paper, a two-stage multiple-choice knapsack model is applied to find the optimal control strategy (X) that achieves a threshold risk ( ˆR) at minimum cost of control. 7. When the optimal control solution is intractable, heuristics may be applied. 8

9 References Ballou, D., R. Wang, H. Pazer, G.K. Tayi Modeling information manufacturing systems to determine information product quality. Management Science 44(4) Ballou, D.P., G.K. Tayi Methodology for allocating resources for data quality enhancement. Communications of the ACM 32(3) Cushing, B. E A mathematical approach to the analysis and design of internal control systems. The Accounting Review 49(1) Even, A., G. Shankaranarayanan Dual assessment of data quality in customer databases. Journal of Data and Information Quality 1(3). Ham, J., D. Losell, W. Smieliauskas An empirical study of error characteristics in accounting populations. The Accounting Review 60(3) Hamlen, S. S A chance-constrained mix integer programming model for internal control systems. The Accounting Review 55(4) Krishnan, R., J. Peters, R. Padman, D. Kaplan On data reliability assessment in accounting information systems. Information Systems Research 16(3) Lea, R. B., S. J. Adams, R. F. Boykin Modeling of the audit risk assessment process at the assertion level within an account balance. Auditing: A Journal of Practice & Theory 11(Supplement) OMG Business process modeling notation specification. URL Pipino, L. L., Y W. Lee, R. Y. Wang Data quality assessment. Communications of the ACM 45(4) Stratton, W. O Accounting systems: The reliability approach to internal control evaluation. Decision Sciences 12(1) Wang, R.Y A product perspective on total data quality management. Communications of the ACM 41(2) Wang, R.Y., M.P. Reddy,, H.B. Kon Toward quality data: An attribute-based approach. Decision Support Systems 13(3-4)

10 Yu, S., J. Neter A stochastic model of the internal control system. Journal of Accounting Research 11(2)