Evaluation of Business Intelligence Systems

Transcription

1 Evaluation of Business Intelligence Systems Francesco Di Tria, Ezio Lefons, and Filippo Tangorra Dipartimento di Informatica, Università degli Studi di Bari Aldo Moro via Orabona 4, Bari Italy francescoditria, lefons, Abstract Business Intelligence is an activity based on a set of processes and software tools. Its aim is to support the decisional making phase, by extracting information from synthetical data. As the success of such an activity depends on the effectiveness of several business processes and the correct integration of independent software tools, nowadays standardization is strongly needed, in order to define a methodology to obtain high-quality information, really useful for the improvement of the business processes of an Information System. In this context, our study focuses on a framework that encapsulates current emerging criteria to evaluate every facet of Business Intelligence systems. In our case study, we tested the criteria to evaluate Business Intelligence platforms, by developing a real OLAP application in an Academic Information System. 1. Introduction Keywords: Data Warehouse, Standard Criteria, Metrics Evaluation. Business Intelligence (BI) is a high-level process, widely supported by Information Technology, to manage data supporting enterprise users in business decisions making. A crucial activity related to BI is to collect historical data coming from heterogeneous operational databases into a single integrated data warehouse. The data warehouse is then utilized to support multidimensional analyses, in order to produce information and knowledge that can be used by decisional makers for the improvement of the business processes and the achievement of business goals. The collection and integration of historical data require (a) the design and the implementation of a database optimized for multidimensional analyses, and (b) a feeding plan that regularly updates the database. In turn, multidimensional analyses include the activities of decision support systems, query reporting, On-Line Analytical Processing (OLAP), statistical analysis, forecasting, and Data Mining. All these tasks of the BI process are performed by adequate software tools, which represent specific components of the so-called BI systems. Not too long time ago, BI was not so popular, as the costs to develop and to maintain a BI system were too expensive. The only companies interested to BI were banks, financial services, and chains of supermarkets, for which the customer satisfaction is the key of success. Today, because of the lowering of the developing cost, BI has entered in new working contexts. Therefore, most major database management system vendors as well as data analysis system vendors are now marketing, at acceptable costs, tools supporting BI, that are tightly integrated and easy-to-use [1]. Several open source products are also developed, that have become serious alternative to traditional proprietary licensed software, providing a wide variety of tools for data warehousing and full BI suites. This great increase of BI tools has often involved a mix of many different implementations of BI solutions also inside the same company. Therefore, BI appears now to have a mature technology for which the appropriate standardization is essential to save costs while reducing overlapping tools, and to increase the control over the information from data integration to the data warehouse, and to end-user reports. Current standardization purposes regard both aspects of the BI activity: the methodological processes and the very high number of different tools and systems supporting single tasks or the entire process. As concerns processes, standardization is devoted to consolidate the sequence of events in performing a BI project, fixing in each event the tasks to be performed. On the other side, as concerns systems, standardization is devoted to define a (minimal) set of products covering all phases of BI processes with as less overlap as possible. In fact, existing BI platforms do not support adequately all process phases and further tools are unavailable to perform the entire lifecycle of the BI applications. Therefore, one goal of BI standardization actually is to reduce as much as possible the number of the International Journal of Information Processing and Management(IJIPM) Volume4, Number3, May 2013 doi: /ijipm.vol4.issue

2 tools for accomplishing the BI process. In this context, a set of evaluation criteria can be used to compare the BI tools and platforms in order to choose the group of products satisfying the organization requirements of standardization. In this paper, first we propose a framework composed of emerging current criteria to evaluate each component of a BI system. For each criterion, we detected opportune metrics for the quantitative assessment. Then, we chose those criteria useful to evaluate commercial BI platforms covering several tasks of the whole process of the development of BI applications. To test this framework, we have made an effective comparison of three BI platforms, by developing a real BI application able to extract information from a data warehouse designed and used in the Academic context [2]. Thus, the original and main contribution of the paper consists of a non-experience-based evaluation framework of BI platforms that can serve as a landmark for the strategic choices of software tools to integrate in an Information System. The paper is organized as follows. Section 2 outlines the BI process and the reference architecture, and describes the involved technologies. Section 3 introduces criteria and measurement techniques used for evaluating the tools supporting the several BI process tasks. Section 4 contains the case study: in particular, it (a) shortly illustrates the BI platforms we chose for comparison, (b) presents the Academic Data Warehouse on which we developed BI applications using the selected platforms, and (c) explains the functional complexity measurement process and values assigned to each evaluated software tool. Then, in Section 5, we discuss the results of the carried out experimental comparison. At last, we conclude with some our future perspectives. 2. Business Intelligence Architecture BI term refers to a broad category of models, methodologies, and technologies for exploring data from which to obtain information for making business decisions. Therefore, the BI application is a complex object that takes several aspects into account such as the process to obtain business results, the technology to support each process type, and finally, the information obtained from the process. The BI process involves several technologies to be effective including data modeling, reporting and information visualization, ad-hoc querying, data warehousing, OLAP, data mining, and statistical/ analytical methods and tools, for example. Thus, the BI process is commonly performed by different tools. However, the technology has become mature offering BI platforms and software suites that substitute mere tools and cover a wide range of activities in the BI process. This is also a sign of the need of BI technology for standardization. The core of a Business Intelligence System is represented by the Data Warehouse (DW), or the central database designed according to a multidimensional model [3]. This is a conceptual model based on the metaphor of data cube, where a fact of interest typically an occurring event is measured by numeric values and a single value is pointed by coordinates of analysis. For this reason, DW is a database optimized to support the analytical processing rather than the transactional one. Moreover, DW can be subdivided into autonomous databases, the so-called data marts, where each data mart covers a specific departmental area of the Information System (see, Figure 1). The multidimensional model of a DW is described by the metadata, stored in a separate repository. The actual standard for metadata definition is the Common Warehouse Metamodel [4], developed by the Object Management Group (OMG). This research group has defined the XML Metadata Interchange as the XML-based physical layer to best allow the interoperability and the portability of metadata among the single components of a BI system [5]. A DW is populated by a periodically executed feeding process. This task, the so-called ETL (from Extraction, Transformation and Loading), consists of feeding the central repository with data coming from several and heterogeneous sources. Finally, Decisional Systems are software tools that access the DW, in order to develop BI applications. The architecture shown in Figure 1 is widely accepted by business companies and it is known as two-layer architecture (Sources level and DW level). However, there exist two variations to this architecture: in the one, the DW is virtual and it is represented by a middleware (one-layer architecture); in the other, the ETL process does not feed directly the DW but a global and reconciled database that, in turn, feeds the Data Warehouse (three-layer architecture). 223

3 Data Mining Figure 1. Business Intelligence architecture. In the general BI architecture, it is possible to distinguish two main areas: (1) the back-end area, constituted of the Sources level, the Refresh level, and the DW level managed by the OLAP Server; and (2) the front-end area, constituted of the Analytical level. Clearly, a BI system is the result of the integration of different technologies. In the back-end, the main challenges are related to the data management issues. In fact, the data at the Sources level can be stored according to different formats. The most utilized are XML files, flat files (csv files, for example), Excel documents, and relational databases. Each of these objects represents a data source for the BI system that must be able to uniformly access these data. In particular, transferring data among components is the main issue of such systems, which must integrate an efficient network infrastructure and must support several standard communication protocols, such as TCP/IP and ODBC. The ETL task physically moves data from sources into the DW, according to an established feeding plan, that periodically updates the DW with integrated and cleaned data. In fact, the decision making process must be supported by reliable analyses, and it is strongly determined by the quality of available data. In addition, dirty data and missing values can easily lead to a misrepresentation of the results of an analytical activity. Thus, a hard challenge of BI is to be able to obtain and to store data, according to high levels of quality and reliability, via an ETL process. This process, essentially based on data integration, includes an important sub-process, whose aim is to clean data. For these reasons, this activity must be able to guarantee a high-level quality of data [6, 7], as these data will be used to provide information and knowledge for decision making. In fact, the main aim of a BI system consists of performing analytical queries and developing data mining applications, in order to produce information and knowledge that will be used for improving the business processes of the Information System. The execution of ETL must be repeated at regular intervals of time and the right interval must be addressed depending on the refresh necessity. Moreover, in the DW implementation step, it is necessary to choose the right logical model supported by the DBMS. Two logical models exist: ROLAP, based on relational technology, and MOLAP, based on multidimensional vectors. Of course, each of these logical models has its own advantages. As an 224

4 example, relational technology ensures high scalability and easy maintenance, while multidimensional vectors guarantee rapid time to access data. In order to combine the advantages of both technologies, hybrid logical models, such as HOLAP, are increasingly implemented by vendors. In the front-end, BI platforms are Decision Support Systems supporting Machine Learning, based on Data Mining algorithms, and statistical analysis of data, based on OLAP and Approximate Query Processing [8]. Graphical modeling for correct data visualization is also important to deploy the results of the elaboration to end users. To this end, dashboards and scorecards are the most important graphical tools allowing decision makers to access information produced in analytical processes. 3. Framework to Evaluate Business Intelligence Systems On the basis of the architecture and the process model described in the previous Section, we identify the following main components of a BI system: (a) Data Warehouse; (b) ETL tool; (c) OLAP server; and (d) BI platform. For each of these, we are interested in defining the main targets that can be object of evaluation. While in the past BI was managed by ad-hoc solutions, over the years some de-facto standard methodologies are emerged, thanks to the experience of researchers and specialized companies. Nowadays, along with the methodologies, it is important to address the main criteria with which to standardize the evaluation of the effectiveness of a BI system Data Warehouse There are two main methodologies to design a DW: requirement-driven and data-driven. The requirement-driven approach starts from the requirements analysis, that is, the phase when end-users needs are investigated, in order to understand what kind of information they are interested in. The DW design, based on the user requirements, can be expressed according to the i* for data warehousing framework [9], that leads to a conceptual model represented using UML [10, 11]. The evaluation of UML schemata is based on ad-hoc metrics [12]. A similar methodology adopts the Tropos framework [13] to express business goals and to derive facts schemata by mapping multidimensional concepts on data sources [14]. On the contrary, the data-driven approach is essentially based on the schemata of the source databases and the conceptual model is obtained through a process of remodeling the global schema coming from the integration of the logical schemata of source databases [15]. Since both approaches present several benefits, the current trend is towards hybrid design methodologies [16, 17], where the most part of the steps to be performed are largely done automatically [18, 19]. However, in despite of the chosen design approach, the DW must be optimized to allow analytical processing of data and must support multidimensional views of data. In fact, while in transactional databases the objective to achieve is normalization, in data warehouse the objective is multidimensionality. Thus, we are interested in the criteria that must be satisfied in order to verify whether the DW is a multidimensional database. In [20, 21], Kimball defines 20 criteria, useful to compare systems, in order to establish what makes a system multidimensional. These criteria are divided into 3 groups: (a) architecture, (b) administration, and (c) expression. In architecture criteria, Kimball defines the key factors that must be respected, in order to produce consistent and integrated information to the user. The administration criteria include the capability to preserve historical information and, as business needs change, the possibility to add new information, sharing it with supply chain partners, in order to provide a wide and contextualized view of the information system. The expression criteria ensure that repetitive tasks, such as querying, navigating, analyzing, and reporting, are easy and intuitive. An expressive DW is one that allows users to exploit the power of a multidimensional system and to obtain relevant information, in order to improve business decision making. Kimball also suggests a metrics for the evaluation. For each criteria, he suggests to assign the value 0 (bad) or 1 (good). At the end of assignments, it is necessary to add up the 0s and 1s. The total is a value between zero, representing a system that does not support the multidimensional model, and 20, representing a system completely designed according to Kimball s criteria. We point out that this 225

5 metrics is not objective while it is not clear how to assign the weights. In fact, each assignment depends on subjective judgements. In order to evaluate any relational schemata, further metrics have been proposed to check the quality of star/snow-flake schemata [22]. The metrics measure the complexity of a table, by counting the number of its attributes and foreign keys, a star, by counting the number of its dimension tables, for example, a schema, by counting the number of its dimension and fact tables, and so on. These metrics assume that a good schema must report low values in counting. However, we highlight that a schema must be evaluated in reference to the information requirements, that is, the information it allows to extract from. In fact, a complex schema may support a larger number of queries than a non-complex schema (which might not allow the decision makers to execute all the queries they are interested in) ETL After the design and the implementation steps, a DW must be populated with data, via an ETL process. This process is planned and executed using ETL systems, which are software tools able to process and elaborate large quantity of data, during the migration from a set of sources into the data warehouse. Therefore, the main aim of ETL is the achievement of data integration, based on a plan to feed the DW with data coming from operational databases and to update it regularly. As source databases may contain dirty data, data integration includes data cleaning as its own sub-activity. Then, the success of this process is determined by the achievement of high data quality. Low levels of data quality cannot allow the decision makers to obtain the useful information and knowledge necessary for the improvement of the Information System s processes. In [23], the authors propose a model for the evaluation of the data quality of a DW. This model is based on metadata able to provide diagnostic information about possible sources of errors. Moreover, they formalize a cost/benefit model to identify low quality data and to compute the cost related to the improvement of data quality. As concerns the BI component, a methodology to support companies in the choice of the best ETL software tool that accords to their own needs, is presented in [24]. It includes (a) a set of standard selection criteria, grouped into eight categories, (b) a set of figures of merit, that represent measures of quality (cost, ease of use, speed, ), and (c) a metrics evaluation to convert qualitative measures into quantitative ones. The result is an evaluation matrix, where the rows correspond to the criteria and the columns to the figures of merit. In every cell, the matrix reports the score obtained by the software tool for those specific criterion and figure of merit OLAP Server The DW is managed by a DBMS supporting OLAP. A DBSM supporting OLAP is the so-called OLAP server and represents the server-side of a BI architecture. Such DBMSs usually integrate MOLAP and/or ROLAP technologies [25]. For OLAP servers, two targets of evaluation are established: Features. These criteria define all the functionalities that must be implemented in an OLAP server. A complete checklist to evaluate OLAP servers is proposed in [26], and it includes criteria referring to the kind of database architecture that is supported, the ability to execute OLAP functions, and back-end and front-end integration. Each criterion must have an associated weight; the metrics evaluation is performed assigning a score to each criterion, according to its weight. System performance. This target of evaluation includes the criteria to evaluate the performance of the system, e.g., the query response time. The OLAP Council defined a set of criteria in order to simulate a realistic OLAP workload [27]. The goal of this analytical process benchmark is to measure the OLAP server s overall performance rather than the performance of specific tasks. The operations performed on the database have been chosen 226

6 among those really executed in common business analytical queries. On the other hand, TPC-H [28] is an ad-hoc decision support benchmark, which includes the so-called TPC-H Composite Query-per-Hour Performance Metrics that considers several aspects of the capability of the system to process queries. A proposal of standardization to extend the current TPC-H to systems performing Approximate Query Processing is presented in [29] BI Platform The BI platform is the client-side component of a BI architecture. At the back-end level, it interacts with the OLAP server, by establishing a connection and executing analytical queries produced by a code generator. The code generator is a fundamental component that translates user s analytical queries into SQL clauses via metadata, representing the multidimensional model of the DW. At the front-end level, it allows users to develop OLAP and data mining applications. In general, OLAP is an activity that consists of performing analytical processing, by formulating statistical queries over large quantity of structured data, in order to obtain synthetic information hidden in numeric values. In the last years, new multidimensional-query-languages have been developed. An example is MDX, the language proposed by Microsoft to express analytical queries in a better way than traditional SQL [30]. The results of OLAP are always visualized with the support of reports and graphical charts. Data Mining is the process of extracting knowledge from large volumes of data, by using procedures able to automatically detect semantic relationships among data. Differently than traditional statistical analysis, BI is interested in data mining because it is often possible to discover hidden knowledge by pattern matching and machine learning methods. Moreover, it is possible to predict future trends, useful in market analyses for example. In BI, this new knowledge, obtained with automatic procedures, is used to support the decision making process in order to improve the business processes, such as the selling increase. Now, great efforts are being attempted to standardize all data mining aspects: the process model, the supporting technology, the tasks with their formal inputs and outputs, and the architecture [31]. It is also important to obtain an objective evaluation of data mining algorithms. Multi-criteria based metrics that can be used in order to evaluate data mining algorithms can be found in [32]. The metrics allows users to obtain an objective evaluation of data mining algorithms, thanks to a set of criteria that take into account not only positive facets but also the negative ones. This approach, that considers both advantages and disadvantages related to a data mining algorithm, is called Data Envelopment Analysis, according to which the final value is calculated by the ratio of the positive properties to the negative properties. For BI platforms, one can establish two targets of evaluation: Features. This target defines the features that must be provided to final users by a BI platform. Recently, the research has pointed out the attention on the definition of formalized standard criteria for evaluating and comparing the technical and functional characteristics that these software tools must own [33]. The criteria are summarized in a Capability Matrix, but no metrics have been yet defined for them. Strategy. This target attempts to examine the market strategy suggested by vendors and adopted by BI platforms. The adoption of a BI platform impacts with the management of the business processes of an information system. In [34], a set of criteria to evaluate the business strategy of vendors is presented. These criteria are grouped in: (a) Ability to Execute, and (b) Completeness of Vision. The Ability to Execute provides seven criteria to evaluate how much the vendor has been able to impose and to make winning its personal point of view of business market. The Completeness of Vision provides eight criteria to evaluate how much the vendor has been able to introduce methodologies in order to exploit new trends in business market. On the base of these criteria, vendors are classified in Leaders, Challengers, Visionaries, and Niche Players. Each criterion is expressed via a question that must be answered with numeric values. However, no method has been proposed to assign a weight to each question in an objective manner. 227

7 Evaluation Criteria for BI Platforms In order to make a comparison, we utilize the criteria defined by the Capability Matrix. We assume that a capability is an evaluation criterion and, for each capability grouped in the following three main areas, we establish a set of tasks on which to perform the comparison of the investigated software tools. Information delivery o Reporting. This capability comprises the task of creating and formatting interactive reports, by performing on-line analytical queries on both relational and multidimensional data sources, while hiding the complexity of the warehouse s logical schema. The ability of scheduling and sharing reports among end users is also considered. o Dashboards. This capability is logically linked with the previous one and it concerns the ability to build, to publish and to update a set of meaningful and interactive charts to a web-based application. o Ad hoc queries. This capability allows users to create their own queries. Here, users need to know the data warehouse s logical schema and SQL programming language. o Microsoft Office integration. Many users utilize Microsoft Excel to create their own reports. The MS Office integration capability comprises the tasks that a user has to do to create a report using Excel as an OLAP client and the BI platform as a middleware. Integration o BI infrastructure. This capability includes all the tasks regarding the implementation of rules for the security administration. o Metadata management. The process of metadata creation is the first and the most important task to carry out the integration of the BI platform with the OLAP Server. o Development environment. A BI platform must be equipped with a set of reusable components to be integrated in a BI application. o Workflow and collaboration. This capability includes all the tasks that allow users to share information, to communicate each other in a public way, or to implement business rules to generate information by trigger-driven events. Analysis o OLAP. This capability comprises all the tasks that allow users to execute traditional OLAP queries (e.g., drilling) and to define their own functions. o Visualization. In some cases, users need to visualize a report containing multi-dimensional data so as to get an optimal view even on a two-dimensional screen; as an example, this effect can be reached by defining the graphics details of the tool. o Predictive modeling and data mining. This capability comprises the tasks that allow users to manage a predictive modeling environment. o Scorecarding. This capability regards the tasks needed to design strategy maps that align key performance metrics with the achievement of strategic objectives Metrics Evaluation As we noted previously, it does not suffice to define a set of criteria. We also need a method to obtain a quantitative evaluation that must be as much as possible objective. In order to obtain such kind of assessment, we need to define a metrics evaluation. While criteria represent analysis focus points, metrics evaluation is the way to obtain a measurement of such points. Software measurement is a field of the Software Engineering and it consists of a quantitative evaluation of a tool. A facet of software measurement is the functional complexity. Presently, there are two standards for functional size measurement: IFPUG FP and COSMIC FFP: 228

8 IFPUG FP. Function Point (FP) analysis, proposed by IFPUG [35], has been the most utilized metrics for the functional size measurement of software in the last years. Its main feature is to be platform-independent, not only from hardware technology but also from the programming language used for the development. Moreover, the function point analysis is carried out from the user s point of view, not the developer s one. The Function Point analysis measures the features which an application is composed of, by listing all the real elements that are enumerable by the end user. The key-factor is that the Function Point metrics provides a normalization technique to compare systems of different vendors. In fact, this metrics measures the application on the basis of two evaluation areas: the first is based on the Unadjusted Function Point value, which reflects the features provided to the user by the application; the second provides the Adjustment Factors value, which emphasizes the complexity of the general features provided to the user. The final value of this metrics depends on both the values. The first step consists of determining the type of functional counting, in reference to the state of the development of the application. The second step establishes the counting context; this context is determined by the scope of the counting and it identifies the tasks that must be evaluated. COSMIC FFP. Full Function Point (FFP) analysis, proposed by COSMIC [36], is a new standard proposal for the software size measurement. This group has a partnership with ISBSG [37], with which launched a joint study to compare software tools. In particular, a guideline has been defined to evaluate the functional size of the BI platform by the COSMIC FFP [38] Evaluation Targets In the experiment illustrated in this paper, we utilized the Function Point metrics. We applied metrics at the Application Counting level, that measures installed applications. This counting is a baseline metrics and estimates the features actually provided to the user, referring to the tasks presented at the beginning of this Section. In any case, it is possible to convert metrics evaluation executed by IFPUG FP into that obtainable by COSMIC FFP [39]. Table 1 summarizes the criteria and metrics we propose as standard framework to evaluate BI systems with reference to the architecture components depicted in Figure 1. Table 1. Evaluation framework per BI component. BI Component Target Criterion Metrics Data warehouse ETL tool OLAP server BI platform Multidimensional design Kimball s criteria Data warehouse quality Data model Metrics for DW Quality Data quality Degradation rate Lost / Improvement Costs System performance Engineering trade study Features Checklist to evaluate OLAP servers System performance TPC-H Features BI Platform Matrix Cosmic FFP Strategy Magic Quadrant for BI Platforms 4. Case Study A Business Intelligence platform is a software tool that allows business users to develop Business Intelligence applications. These applications, usually made up by reports and charts, are inspected in order to find relevant and hidden information. Once extracted, the information is used for decision making, with the aim to improve the process of the Information System. 229

9 4.1. Business Intelligence Platforms Here, we briefly describe the three Business Intelligence platforms that we selected for the evaluation, namely Business Object [40], Pentaho [41], and Oracle Business Intelligence Enterprise Edition (BIEE) [42]. Of course, these platforms are only a part of wider BI suites. The general architecture of our system is shown in Figure 2. The data warehouse is managed by Oracle OLAP Server a sub-component of Oracle DBMS. There are several ways to connect to Oracle DBMS. The most used is based on the Open Database Connectivity (ODBC) protocol that guarantees a flexible and easy interaction. Via ODBC connection, a Business Intelligence platform is able to perform the analytical processing. Figure 2. Case study BI system. The characteristics of the three selected platforms are as follows. Pentaho is a suite of open source software tools of Business Intelligence. Currently, it includes a tool for each topic of Business Intelligence: Weka for data mining, Kettle for data integration and ETL, and Pentaho BI for analyses and reporting. The technology adopted is JSP (Java Servlet Page) and the Application Server is based on Java JEE. Oracle BIEE is a Business Intelligence platform that gives business users the ability to access information stored in a data warehouse, providing a business view that hides the complexity of the underlying data structures. The Administration Tool is the software used to manage metadata through different levels of abstraction: the Physical layer deals with the connection to the data source and the import of tables/views; the Business Model and Mapping layer is focused on the definition of metadata using the objects of the physical layer; while the Presentation layer aims at organizing metadata to be correctly represented in the Presentation Catalog the component that deploys information to end-users. BI Answers is the tool for analyses and reporting, and it uses an abstract view of the data source. The results can be shared among final users through dashboards. Business Object is the BI platform of SAP and adopts a service-oriented architecture. This architecture is very scalable, extensible, and fault-tolerant. The main tools provided are Designer and Desktop Intelligence. The former is the tool used by designers and developers to create a conceptual model of the underlying database (the so-called universe). The universe is devoted at supporting non-expert users during the querying phase. In fact, the universe is based on objects, which represent dimensions and indicators, for example. In this way, it is not necessary for users (a) to know the logical schema of the database, and (b) to use a system query language (e.g., SQL). The second tool is used by final and non-expert users to perform analyses using the universe. The results are visualized through reports and charts. Figure 3 summarizes the features of the chosen BI suites synthetically. We stress that both Pentaho and Business Object do not include a DBMS but they rely on (any) external database servers. However, 230

10 they are equipped with OLAP servers (or OLAP engines) that interact with DBMSs and perform data aggregation in a memory cache, reproducing multidimensional views of data and supporting analytical processing. Business Intelligence Platform DBMS ETL OLAP Server Reporting OLAP Data Mining Approximate Query Processing Oracle BIEE Pentaho, Business Object Figure 3. BI suites features The Data Mart In order to have a real case study to apply FP metrics and the evaluation criteria introduced in Sub- Section 3.4.2, we have chosen a Data Mart of the Academic Data Warehouse as the repository for developing a Business Intelligence application [43], using the three Business Intelligence platforms object of the present evaluation. In particular, the Didactic Data Mart contains data about the students enrolled to the Faculties of the University of Bari. This Data Mart has been designed through the integration of the logical schemas of two transactional databases: (a) ESSE3 (Secretary and Services for Students), that is the current database that supports all the didactic curricula, and the administrative processes and services to the students in accordance with the didactic autonomy of the University; and (b) NOGE (NOt ManaGEd), that is a secondary database that stores residual historical data about students enrolled before the ESSE3 introduction. The Didactics Data Mart s conceptual model can be thought of as a set of data cubes, whose main dimensions are time, student, and course of study, which are base dimensions for they represent the minimum information to express who, where and when aggregation levels for business analysis. According to these coordinates, it is possible to find data; a single datum is stored in a cell of the cube and it represents the value of a measure; a measure is the quantitative description of a fact; and, in the business context, a fact is a meaningful event to be analyzed. In general, the Didactics Data Mart has got 4 cubes: tax, examination, university degree, and enrolment. The tax cube is represented by a fact table, whose scope is to control the payment of the taxes by the student; it has only the student, time and degree course dimensions and the amount field as measure. Figure 4 shows the conceptual schema of the tax cube, represented according to the data warehouse Dimensional Fact Model. faculty_ description course_ description faculty course tax amount day month year month_ description student birthday name sex surname Figure 4. Tax cube. The complete logical schema of the Didactics Data Mart is shown in Figure

11 4.3. Experimental Data Here, we report the comparison carried out on the three Business Intelligence platforms: namely Business Object, Pentaho, and Oracle BIEE. The comparison consists of the measurement of the software functional complexity based on the Cosmic method. In detail, we first show a case of measurement of the drilling task. To execute this task, we started from the report in Table 2. As Degree Course is the dimension of the Enrolment fact table (cf., Figure 5) and it has a one-to-many relationship with Faculty table, this relationship allowed us to create a new report, where the grouping fields were Degree Course and Year. Figure 5. Didactics data mart. Table 2. Percentage of university students per Faculty and Year ( ). Faculty Educational Sciences Law Medicine and Surgery Economics Math, Phys, and Natural Sci Arts and Philosophy Pharmacy Foreign Languages and Lit Political Sciences Law (Taranto city) Veterinary Medicine Agricultural Sciences Economics (Taranto city) We created the report using the three platforms and the results are those illustrated in Table 3 for Business Object, Table 4 for Pentaho, and Table 5 for Oracle BIEE. Each task is called Functional User Requirement (FUR) and it is composed of several basic operations, or Functional Processes (FPs). For each FP, we have to count the involved data movements, which are the functional steps where data are moved as a single and unique group. Data movements are classified in the following types: 232

12 Entry, that moves data from the user to the process that requires it, Exit, that moves data from the process to the user that requires it, Read, that moves data from the secondary memory to the process that requires it, Write, that moves data from the process to the secondary memory. The functional complexity of a task is given by the count of the data movements per FP. In our experiment, Pentaho seems to have the highest functional complexity for the drilling task, while Oracle BIEE reports the minimum value of complexity. Table 3. Data movements for drilling task in Business Object. Functional Process Entry Exit Read Write Subtotal Login Connection to the universe Analysis selection Level navigation Total 18 Table 4. Data movements for drilling task in Pentaho. Functional Process Entry Exit Read Write Subtotal Login Creation of data source Execution of OLAP query Total 22 Table 5. Data movements for drilling task in Oracle BIEE. Functional Process Entry Exit Read Write Subtotal Login Execution of OLAP query Total 11 To detail the case of measurement, we explain how we counted the data movements in the Level navigation FP in Business Object. The counting details are given in Table 6. Table 6. Detailed data movements for Level navigation functional process in Business Object. Data movement Trigger event Sub-process description Data group CFP type The user executes the command for level navigation The system retrieves the information about available levels The system displays the available levels Type of analysis Entry 1 Data about the hierarchical structure Data about the hierarchical structure Read 1 Exit 1 The user selects the level Selected level Entry 1 The system performs the drill Selected level Read 1 down operation The system displays the report and feedback messages Report Exit 1 Total 6 233

13 First, each FP, which includes one or more sub-processes, is triggered by a user event that is the main process. Then, each process uses and/or produces a group of data according to the previously introduced types. Therefore, the FP includes the main process, which is a data entry about the kind of analysis to be performed depending on the user s choice. Then, several sub-processes start. The system retrieves the hierarchical structure of the dimension by performing a data reading from the secondary memory. Next, it presents these data to the user, who is allowed to choose a level to which drilling down. At last, the system performs the operation by reading the new data and showing the final report, along with feedback messages to the user. By counting the number of data movements performed, we obtain the value 6 for the Level navigation during the drilling task in Business Object (cf., Table 3). The complete benchmark of our experimentation is reported in Table 7. Table 7. Functional Complexity of BI platforms. Benchmark Area Capability Task Business Object Score Pentaho Oracle BIEE Information Delivery reporting creating reports dashboards creating charts ad-hoc queries defining ad-hoc queries Subtotal Integration BI infrastructure security and privileges metadata management metadata creation Subtotal Analysis OLAP drilling Total Discussion To measure the functional complexity, each task is considered to be a function with I/O parameters. Therefore, the cost of the task is evaluated in terms of the number of data groups that it moves among the process, the user, and/or the secondary memory. Of course, the measurement methods that evaluate the functional complexity are not able to capture the difficulty that the user may encounter in the task execution. Moreover, it is important to apply the measurement to systems with the same coverage, that is, those systems supporting the same set of tasks. A system including many tasks will obtain a higher complexity value than ones supporting few tasks. Summing up, our experimental results show that Pentaho has the highest functional complexity at all. This means that it performs a higher number of data movements rather than the other two platforms. 5. Concluding Remarks In this paper, we have presented a framework to evaluate a complete and efficient system supporting the decisional making process. Since we need to choose the right tools that best fit the business requirements and in order to decrease the confusion determined by the overlapping of the features, this framework focuses on the main components of a BI system. In particular, we included in the framework the emerging standard criteria that can help business companies to evaluate every single component of the system. Moreover, since the evaluation must be performed objectively, we have associated metrics to each criterion. As case study, we have evaluated three popular Business Intelligence platforms: Business Object, Pentaho, and Oracle BIEE. The evaluation has been carried out using a software measurement method consisting in the analysis of the functional complexity. The experimental showed that Pentaho presents the highest functional complexity, due to the several repetitive tasks to be manually executed. However, in our opinion, the standard metrics evaluation based on linearly counting the different types of data movements needs to be adjusted in order to assign proper weights to each kind of 234

14 operation. In fact, we recognized some limitations of this measurement method, which we illustrate with the following example. Suppose we have two platforms reporting the value, say, 10 of functional complexity for the same task. We are then allowed to say they have the same functional complexity and, therefore, the same performance. However, if the measure 10 derives from Read:3, Entry:4, Exit:2, Write:1 as to the first platform, and Read:3, Entry:2, Exit:3, Write:2 as to the second one, then we could state that the value 10 (= ) for the second platform hides a more complex task than that for the first one (10= ), for the Write operation is more expensive than Exit/Entry operations. Therefore, we deem that it should be advisable to couple data movement types with suitable weights, so modifying the evaluation method in Ref. 36. (Our preliminary tests indicate suitable weights 0.15 for Entry, 0.2 for Exit, 0.3 for Read, and 0.35 for Write.) Further future work is to extend the benchmark in breadth and depth. In particular, given a report as the one shown in Table 7, its extension in breadth makes it is possible to add columns relative to different Business Intelligence platforms; while extending in depth makes it possible to insert rows relative to capabilities not considered here, including more tasks than one for each capability. 6. References [1] C. J. White, The IBM Business Intelligence Software Solution, disc99/disc/ibm/bisolution.pdf, [2] C. dell Aquila, F. Di Tria, E. Lefons, and F. Tangorra, An Academic Data Warehouse, In Proceedings of the International Conference on Applied Informatics and Communications, Athens, WSEAS Press, pp , [3] R. Kimball, M. Ross, and R. Merz, The Data Warehouse Toolkit: The Complete Guide to Dimensional Modeling, 2nd ed., Wiley, New York, [4] Object Management Group, Common Warehouse Metamodel (CWM) Specification, vers. 1.1, vol. 1, [5] Object Management Group, XML Metadata Interchange (XMI) Specification, vers. 2.0, [6] B. Piprani and D. Ernst, Model for Data Quality Assessment, In Proceedings of the OTM Confederated International Workshops and Posters on the Move to Meaningful Internet Systems, Monterrey, pp , [7] B. Stvilia, L. Gasser, M. B. Twidale, and L. C. Smith, A Framework for Information Quality Assessment, Journal of the American Society for Information Science and Technology, vol. 58, no. 12, pp , [8] S. Chaudhuri, U. Dayal, and V. Ganti, Database Technology for Decision Support Systems, Computer, vol. 34, no. 12, pp , [9] J. Mazón, J. Trujillo, M. Serrano, and M. Piattini, Designing Data Warehouses: From Business Requirement Analysis to Multidimensional Modeling, In Proceedings of the 1 st International Workshop on Requirements Engineering for Business Need and IT Alignment, Paris, pp , [10] S. Mora and J. Trujillo, Physical Modeling of Data Warehouses Using UML, In Proceedings of the 7 th ACM International Workshop on Data warehousing and OLAP, Washington, DC, pp , [11] S. Lujan-Mora, J. Trujillo, and I. Y. Song, A UML Profile for Multidimensional Modelling in Data Warehouses, Data & Knowledge Engineering, vol. 59, no. 3, pp , [12] M. Serrano, J. Trujillo, C. Calero, and M. Piattini, Metrics for Data Warehouse Conceptual Models Understandability, Information and Software Technology, vol. 49, no. 8, pp , [13] P. Giorgini, M. Kolp, J. Mylopoulos, and M. Pistore, The Tropos Methodology: An Overview, Methodologies and Software Engineering for Agent Systems, Kluwer Academic Press, [14] P. Giorgini, S. Rizzi, and M. Garretti, GRAnD: A Goal-oriented Approach to Requirement Analysis in Data Warehouses, Decision Support Systems, vol. 45, no. 1, pp. 4-21, [15] M. Golfarelli, D. Maio, and S. Rizzi, The Dimensional Fact Model: a Conceptual Model for Data Warehouses, International Journal of Cooperative Information Systems, vol. 7, pp ,

15 [16] F. Di Tria, E. Lefons, and F. Tangorra, Hybrid Methodology for Data Warehouse Conceptual Design by UML Schemas, Information and Software Technology, vol. 54, no. 4, pp , [17] F. Di Tria, E. Lefons, and F. Tangorra, GrHyMM: A Graph-Oriented Hybrid Multidimensional Model, In Proceedings of Advances in Conceptual Modeling. Recent Developments and New Directions, Brussels, Belgium, Lecture Notes in Computer Science 6999, Springer Verlag, pp , [18] C. dell Aquila, F. Di Tria, E. Lefons, and F. Tangorra, Logic Programming for Data Warehouse Conceptual Schema Validation, In Proceedings of the 12 th International Conference on Data Warehousing and Knowledge Discovery, Bilbao, Spain, Lecture Notes in Computer Science 6263, Springer Verlag, pp. 1-12, [19] C. dell Aquila, F. Di Tria, E. Lefons, and F. Tangorra, Dimensional Fact Model Extension via Predicate Calculus, In Proceedings of the the 24 th International Symposium on Computer and Information Sciences, North Cyprus, IEEE Press, pp , [20] R. Kimball, Rating Your Dimensional Data Warehouse, Intelligent Enterprise Magazine, vol. 3, no. 7, [21] R. Kimball, Is Your Dimensional Data Warehouse Expressive?, Intelligent Enterprise Magazine, vol. 3, no. 8, [22] M. Serrano, C. Calero, and M. Piattini, Metrics for Data Warehouse Quality, Effective databases for text & document management, IGI Publishing Hershey, PA, USA, [23] Y. C. Chu, S. S. Yang, and C. C. Yang, Enhancing Data Quality Through Attribute-Based Metadata And Cost Evaluation In Data Warehouse Environments, Journal of the Chinese Institute of Engineers, vol. 24, no. 4, pp , [24] S. Henry, S. Hoon, M. Hwang, D. Lee, and M. D. DeVore, Engineering Trade Study: Extract, Transform, Load Tools for Data Migration, In Proceedings of the 2005 IEEE Systems and Information Engineering Design Symposium, pp. 1-8, [25] S. Chaudhuri and U. Dayal, An Overview of Data Warehousing and OLAP Technology, ACM Sigmod Record, vol. 26, no. 1, pp , [26] D. Bulos, How to Evaluate OLAP Servers A Comprehensive Checklist for Evaluating the Latest OLAP Server Products, DBMS, vol. 8, pp , [27] OLAP Council, APB-1 OLAP Benchmark Release II, OLAP Council, 1998, [28] TPC BENCHMARK H (Decision Support), Standard Specification, Revision , (16 June 2011). [29] F. Di Tria, E. Lefons, and F. Tangorra, Metrics for Approximate Query Engine Evaluation, In Proceedings of the ACM Symposium on Applied Computing, SAC 2012, Riva, Trento, Italy, March 26-30, pp , [30] B. C. Smith and C. Ryan Clay, MicroSoft SQL Server 2008 MDX Step by Step, MicroSoft Press, [31] C. Clifton and B. Thuraisingham, Emerging Standards for Data Mining, Computer Standards & Interfaces, vol. 23, no. 3, pp , [32] G. Nakhaeizadeh and A. Schnabl, Development of Multi-Criteria Metrics for Evaluation of Data Mining Algorithms, In Proceedings of the Fourth International Conference on Knowledge Discovery in Databases & Data Mining, pp , [33] K. Schlegel and B. Sood, Business Intelligence Platform Capability Matrix, [34] R. L. Sallam, J. Richardson, J. Hagerty, and B. Hostmann, Magic Quadrant for Business Intelligence Platforms, [35] International Function Point Users Group, Function Point Counting Practices Manual, Release 4.1.1, [36] The COSMIC Functional Size Measurement Method version Measurement Manual, [37] ISBSG, International Software Benchmarking Standards Group, [38] Guideline For Sizing Business Application Software version 1.1, [39] J. J. Cuadrado-Gallego, F. Machado-Piriz, and J. Aroba-Páez, On the Conversion between IFPUG and COSMIC Software Functional Size Units: A Theoretical and Empirical Study, The Journal of Systems and Software, vol. 81, no. 5, pp , [40] SAP, [41] Pentaho, 236

16 [42] Oracle, [43] M.C. Lee, "The Combination of Knowledge Management and Data mining with Knowledge Warehouse", International Journal of Advancements in Computing Technology, vol. 1, no. 1, pp ,