A Systematic Evaluation of Business Intelligence Tools: An Investigation into Potential Tangible/Intangible Value

1 Paper 04 A Systematic Evaluation of Business Intelligence Tools: An Investigation into Potential Tangible/Intangible Value 2 Contents: 1. Introduction 4 2. BI Tools and Evaluation overview...5 2.1 BI overview..5 2.2 Evaluation overview...5 3. BISE proposed framework..6-8 3.1 Tangible value of BISE framework.7 3.2 Intangible Value of BISE framework..8 4. Research Methodology..9 5. Conclusion 9-10 6. References and Bibliography 10-12 3 Abstract Business Intelligence (BI) tools are becoming increasingly important due to ir ability to produce accurate and timely knowledge. This allows organisations to have a better understanding of internal and external environments which facilitate ir decision-making process and support m achieving competitive advantage. However, re is an urgent need of a systematic evaluation framework for BI tools. This study propose an initial Business Intelligence Systematic Evaluation (BISE) framework. The framework will help organisations identify tangible and intangible value of BI tools. The framework was developed through use of Information Systems (IS) evaluation concepts and models and BI tools main functionalities. Keywords: Business Intelligence (BI), Information Systems (IS), Tangible/Intangible value, BISE framework. 4 1. Introduction

BI tools are a fundamental part of organisations strive to achieve competitive edge. They provide businesses with ability to produce accurate and timely knowledge using highly sophisticated visualisation techniques (i.e. dashboards) and advanced analytics (i.e. data mining) that will help improve business performance and decision making process (Cody et al. 2002). BI tools are becoming increasingly important in a wide variety of industries where marketing reports of BI tools value are found extensively. In fact, a report conducted by IDC1 in 2006, investigated BI Vendor Shares worldwide. It shows an 11.5% growth rate that reached 6.25 billion in worldwide revenue. Moreover, anor research predicted a greater adoption of BI tools and an increasing interest in Predictive Analytics (Vesset, McDonough June 2007). In contrast, academic research on BI tools is very limited. In this study, we propose an initial Business Intelligence Systematic Evaluation (BISE) framework developed using IS evaluation frameworks and BI tools functions. BISE framework will facilitate identification of tangible and intangible value of BI tools. It will also help improve tool performance and hence decision making. Moreover, it will ensure a better integration of BI tools functions with business operations. This study is also intended to provide a better understanding of BI tools evaluation process. Research questions for BI evaluation include; how prevalent is BI evaluation? Which employees are usually involved in BI evaluation? What evaluation criteria are relevant for BI? And what are benefits and barriers to BI evaluation? The reminder of paper is organised as follows. In section 2, we define BI tools and evaluation. In section 3, we introduce an initial BISE framework and discuss potential tangible/non-tangible value of framework. Section 4 introduces research methodology that will be used to carry out this study. Finally, a brief conclusion is presented. 1 IDC is premier global provider of market intelligence, advisory services, and events for information technology, telecommunications, and consumer technology markets. 5 2. BI Tools and Evaluation overview

This section provides a general explanation of BI tools and evaluation. 2.1 BI overview The term BI originated from Gartner Group in 1989 by Howard Dresner and it was in that period (early 90 s) that BI emerged within industrial world. Academic interest came later in mid-90 s and evolved greatly ever since (Ou, Peng 2006); (Golfarelli, Rizzi & Cella 2004). (Howson 2008) describes BI as; Business Intelligence allows people at all levels of an organisation to access, interact with, and analyse data to manage business, improve performance, discover opportunities, and operate efficiently. There are two main analytical methodologies used in BI tools market. First, Query, Reporting, and Analysis (QRA) consist of analysis tools i.e. dashboards that support ad hoc data access and report building. Second, Advanced Analytics employs data mining, statistical software and knowledge discovery that includes different techniques i.e. classification and clustering in data mining to obtain valuable information and knowledge from huge raw of data. These techniques can be used in applications such as market basket analysis and loan applications (Vesset, McDonough July 2006). Business Performance Management (BPM) is a combination of BI and Performance Management tools that help companies improve ir performance. 2.2 Evaluation overview There are two types of Information Technology (IT) evaluation. First, Prior- Operational Use evaluation is sometimes referred to as strategic, preimplementation or formative evaluation. This type of evaluation is primarily performed to support IT investment justification and predict estimated costs and benefits, return on investment and management. Second, Operational Use evaluation is sometimes referred to postimplementation or summative evaluation. It is undertaken to measure actual impact of new system. In or words, it is aimed to establish positive or negative impact of system, a better understanding of system performance, and what it has accomplished in terms of its stated objectives. The worth of an IT application is inherently related to assessment of systems features (Al- Yaseen et al. 2006); (Beynon-Davies, Owens & Williams 2004). This study take into

consideration only second type of evaluation namely; Operational Use Evaluation. 6 3. BISE proposed framework There are no current frameworks for BI tools evaluations in academia. This study proposes BISE framework designed specifically to evaluate BI tools. The main benefit of BISE will be facilitating identification of tangible and intangible value of BI tools. As mentioned in section 2.2, identifying value of a system signifies assessing its main features. That is to say, framework is constructed through correlation of evaluation criteria. In order to select appropriate criteria for evaluation of BI, re are two important aspects that need to be investigated in detail. First, re has to be a clear understanding of main functions and capabilities of BI tools. Second, re has to be an extended review of previous IS evaluation frameworks. The identification of BI tools main functions was achieved through review of three major BI vendors namely; BusinessObjects, SQL Server 2005 and Oracle. The review was conducted using a wide variety of resources such as white papers, technical overviews, product description reports and so on. Table 1 is a summary of resulting output of review. Reporting Analysis Or capabilities Ad Hoc Query Enterprise Reporting Report building and publishing Report management Scorecards Dashboards BI Search BI Spreadsheets EIS OLAP Visualisation techniques (i.e. sophisticated gauges) Root-cause analysis What-if analysis

Predictive Analytics Unified Graphical Interface Embedded BI Web-Based BI Data Integration (ETL) Alert Notification Mobile BI User rights and security management Table 1: BI main functions and or capabilities After an extensive review of evaluation frameworks in IS field, two models have been selected to be used for developing an initial BISE framework. The two main models used to design proposed framework are; DeLone and McLean IS Success model and Casual diagram of perceptual evaluation of IS. However, it important to 7 emphasize proposed framework is just an initiative of potential BISE framework. In fact, a finalised model can not be developed at this stage as empirical data collection (surveys, case studies etc) are polar for design of an effective model. Figure1: BISE framework 3.1 Tangible value of BISE framework Organisations commonly buy and implement an IT system without considering any kind of evaluation techniques for that system. The introduction of a new system for organisational use should comprise identification of impacts that system will have on business processes of organisation. That is to say, evaluation is as important as procurement and implementation of an IT system (Scholtz, Steves 2004). Moreover, (Smithson, Hirschheim 1998) described evaluation as a necessary evil. This is a strong definition that reflects important benefits of evaluation. A systematic evaluation provides a basic managerial feedback function as well as evolving organisational learning process. Also, it gives benchmarks for BI success

in terms of investment and operations (Lycett 2000). Furrmore, in context of function, evaluation is essential for problem diagnosis, planning and reduction of uncertainty. Finally, an efficient assessment of system help improve ir use, thus increase system potential and also can help in design of future ones (Mende, Brecht & \{O}sterle 1994). Indeed, improved BI tools will lead to better knowledge and hence improve decision making. 8 3.2 Intangible Value of BISE framework BI tools are successful from technological perspective, but in practice problems rise in operational use of BI tools. In fact, a survey conducted in 2005 by TDWI2 established that an average of 18 percent of potential BI users actively utilise BI tools (Fryman 2006). Moreover, high level of iteration between data analysts and business users causes time needed for overall cycle of collecting, analysing, and acting on enterprise data to be longer. In addition, lack of clear business goals and metrics can have a negative effect on businesses due to unrealistic expectation of advanced analytics capabilities (Kohavi, Rothleder & Simoudis 2002). In last decade, many companies have failed to create a convincing business value out of ir BI tool, thus failing to achieve Return On Investment (ROI). This is due to first, BI tools are IT driven which leads companies to consider mostly technical issues surrounding integration of system. And second, many companies paid moer attention to ROI than strategic alignments with business processes (Williams 2004). There are several technical issues surrounding BI tools, but non-technical problems mentioned above are under-researched areas. BI tools offer companies with attractive features which if integrated effectively with business processes can generate business value. Moreover, drawing threads of previous argument, most companies that adopt BI tool have non-technical issues with it. Therefore, problem lies in organisations failure to have a formal procedure to integrate BI with business operations and strategy. Based on non-technical issues faced by BI, we believe that BISE framework will

be beneficial in tackling se issues. The framework can identify main functions where users are less comfortable with. It can also provide a better understanding of current tasks and business processes carried out using BI functions, and hence give useful indicators on BI current alignment with business strategy. Moreover, it will allow a better understanding of users difficulties using specific functions and provide a starting platform to find long-term approach for broadening use of BI and make users more independent from data analysts and experts. Finally, developing a framework will reduce barriers to perform evaluation. 2 TDWI is premier provider of in-depth, high-quality education and research in business intelligence and data warehousing industry. 9 4. Research Methodology This research developed an initial framework through refinement of concepts in field of IS and refore can be defined as constructive research (Cornford, Smithson 2006). The study will be performed in three stages. The first stage is concerned with an in-depth review of BI tools and IS evaluation practices. The second stage will be design of a questionnaire. The selection of a questionnaire as a method is due to fact this study objective needs different types of data and its research questions require high rate of respondents in order to be generalised. (Galliers 1992) describes surveys as Obtaining snap shots of practices, situations or views at a particular point in time (via questionnaires or interviews) from which inferences are made (using quantitative analytical techniques) regarding relationships that exist in past, present and future. After conducting survey, final stage is concerned with design of a case study to produce more empirical data to support refinement and development of BISE framework. 5. Conclusion Although BI tools are being covered increasingly by academics, research publications are still limited. This is reflected by number of conference and journal papers about BI in literature. In fact, current interest of BI tools in industry does

not reflect level of academic research dedicated to it. Furrmore, an indepth investigation of BI existing solutions and latest technological trends, justified urgent need to research this area more extensively. The main challenges of BI tools are related to users, data, time, strategy which can be tackled through use of an effective evaluation method. BISE framework can tackle main issues surrounding BI tools. Evaluation should be a dynamic and ongoing process. BISE framework helps organisations identify tangible/non-tangible value of ir BI tools. It can also assist companies to gain a better understanding of impact of BI on organisational performance and thus enhance utilisation of tool. This study also intends to provide an extensive review into technological and organisational aspects surrounding improvement of knowledge and understanding of BI tools evaluation. The main objective is to generate empirical data through use of a systematic review, survey and case studies in order to enhance initial BISE framework. An effective questionnaire with high responses will provide a clear understanding and view of current BI tools evaluation practices. Finally, an effective BISE framework can 10 demonstrate accountability, gain knowledge and enhance development of evaluated BI tool. It will also make evaluation of BI tools a formal procedure. 6. References and Bibliography Al-Yaseen, H., Eldabi, T., Lees, D. & Paul, R. 2006, "Operational Use evaluation of IT investments: An investigation into potential benefits", European Journal of Operational Research, vol. 173, no. 3, pp. 1000-1011. Beynon-Davies, P., Owens, I. & Williams, M.D. 2004, "Information systems evaluation and information systems development process", Journal of Enterprise Information Management, vol. 17, no. 4. Butler Group 2006, Business intelligence: a strategic approach to extending and standardising use of BI, Butler Group, Hull. Cody, W.F., Kreulen, J.T., Krishna, V. & Spangler, W.S. 2002, "The integration of business intelligence and knowledge management", IBM Systems Journal, vol. 41, no. 4, pp. 697-713. Cornford, T. & Smithson, S. 2006, Project Research in Information Systems, Palgrave Macmillan, Basingstoke, Hampshire.

Dekkers, J., Versendaal, J. & Batenburg, R. 2007, "Organising for Business Intelligence: A framework for aligning use and development of information", 20th Bled econference, emergence: Merging and Emerging Technologies, Processes, an Institutions. Delone, W.H. & McLean, E.R. 2003, "The DeLone and McLean Model of Information Systems Success: A Ten-Year Update", Journal of Management Information Systems, vol. 19, no. 4, pp. 9-30. Folinas, D. 2007, "A conceptual framework for business intelligence based on activities monitoring systems", International Journal of Intelligent Enterprise, vol. 1, pp. 65-80(16). Fryman, H. 2006, "Four Strategies to Broaden BI Adoption", Business Intelligence Journal, vol. 11, no. 3, pp. 7. 11 Galliers, R. 1992, Information Systems Research, Blackwell Scientific Publications, Oxford. Golfarelli, M., Rizzi, S. & Cella, I. 2004, "Beyond data warehousing: what's next in business intelligence?", DOLAP '04: Proceedings of 7th ACM international workshop on Data warehousing and OLAPACM, New York, NY, USA, pp. 1. Howson, C. 2008, Successful Business Intelligence: Secrets to Making BI a Killer App, McGraw-Hill, London. Irani, Z. 2002, "Information systems evaluation: navigating through problem domain", Information & Management, vol. 40, no. 1, pp. 11-24. Irani, Z., Gunasekaran, A. & Love, P.E.D. 2006, "Quantitative and qualitative approaches to information systems evaluation", European Journal of Operational Research, vol. 173, no. 3, pp. 951-956. Kohavi, R., Rothleder, N.J. & Simoudis, E. 2002, "Emerging trends in business analytics", Communications of ACM, vol. 45, no. 8, pp. 45-48. Kumar, K. 1990, "Post implementation evaluation of computer-based information systems: current practices", Communications of ACM, vol. 33, no. 2, pp. 203-212. Leclercq, A. 2007, "The perceptual evaluation of information systems using construct of user satisfaction: case study of a large french group", SIGMIS Database, vol. 38, no. 2, pp. 27-60. Lycett, M. 2000, Component-based information systems: toward a framework for evaluation. Mende, M.W., Brecht, L. & \{O}sterle, H. 1994, "Evaluating existing information

systems from a business process perspective", SIGCPR '94: Proceedings of 1994 computer personnel research conference on Reinventing IS : managing information technology in changing organizationsacm, New York, NY, USA, pp. 289. Ortiz, S.,Jr. 2002, "Is business intelligence a smart move?", Computer, vol. 35, no. 7, pp. 11-14. 12 Ou, L. & Peng, H. 2006, "Knowledge and Process Based Decision Support in Business Intelligence System", Computer and Computational Sciences, 2006. IMSCCS '06. First International Multi-Symposiums on, pp. 780. Rouibah, K. & Ould-ali, S. 2002, "PUZZLE: a concept and prototype for linking business intelligence to business strategy", The Journal of Strategic Information Systems, vol. 11, no. 2, pp. 133-152. Schiff, M. & Michaels, J. 2007, "MAKING A STRATEGIC BUSINESS INTELLIGENCE CHOICE-A CHECKLIST",. Scholtz, J. & Steves, M.P. 2004, "A framework for real-world software system evaluations", CSCW '04: Proceedings of 2004 ACM conference on Computer supported cooperative workacm, New York, NY, USA, pp. 600. Smithson, S. & Hirschheim, R. 1998, "Analysing information systems evaluation: anor look at an old problem", Eur.J.Inf.Syst., vol. 7, no. 3, pp. 158-174. Vesset, D. & McDonough, B. June 2007, "Worldwide Business Intelligence Tools 2006 Vendor Shares", vol. 1. Vesset, D. & McDonough, B. July 2006, "Worldwide Business Intelligence Tools 2005 Vendor Shares", vol. 1. Weiss, P. 2005, Employee Ability Shapes Business Intelligence, InformationWeek Research. Wilhide, K. & Vesset, D. January 2006, "Evaluation Criteria for Selecting a Successful Business Performance Management Solution",. Willen, C. 2002, Airborne opportunities, Intelligent Enterprise. Williams, S. 2004, "Assessing BI Readiness: A Key to BI ROI", Business Intelligence Journal, vol. 9, no. 3, pp. 15. Yusof, M.M. 2006, Towards a Framework for Health Information Systems Evaluation.