Key Metrics for Determining ROI for Business Intelligence Implementations. Elliot King, Research Analyst August 2007

Key Metrics for Determining ROI for Business Intelligence Implementations Elliot King, Research Analyst August 2007 Sponsored by Produced by

2 TABLE OF CONTENTS Introduction and Key Findings............................................................................................................................3 Respondent Pool..............................................................................................................................................4 Conducting ROI Studies in IT.............................................................................................................................4 Figure 1: How frequently does your company conduct ROI analysis for significant IT projects?.......................................... 4 Figure 2: For which kinds of IT projects does your company conduct ROI analysis?......................................................... 5 Figure 3: In general, when it comes to investments in IT, who is primarily responsible for conducting an ROI analysis?...........5 BI and ROI.......................................................................................................................................................6 Figure 4: When is the last time you implemented a new BI application (including an upgrade of a current product)?...............6 Figure 5: How many people regularly access your business intelligence applications?.................................................... 6 Figure 6: Did you have to perform a cost-justification prior to implementing your BI solution?........................................... 7 Figure 7: Has your company conducted an ROI analysis for your investments in business intelligence?................................7 Figure 8: Have you ever calculated the total cost of ownership for your BI implementation?............................................. 7 Key Cost Metrics for Calculating BI ROI..............................................................................................................8 Figure 9: Rank how important it is to measure each of the following direct costs for BI applications..................................8 Figure 10: Rank how important it is to measure each of the following indirect costs in calculating ROI for BI applications........ 9 Figure 11: Rank how important it is to measure each of the following related costs in calculating ROI for BI applications........ 9 Key Benefit Metrics for Calculating BI ROI.........................................................................................................10 Figure 12: Rank how important it is to quantify each benefit for staff in calculating ROI for BI applications...........................10 Figure 13: Rank how important it is to quantify each benefit for senior managers in calculating ROI for BI applications..........10 Figure 14: Rank how important it is to quantify each benefit for IT staff in calculating ROI for BI applications......................11 Figure 15: Rank how important it is to quantify each general benefit in calculating ROI for BI applications.........................11 Figure 16: Would making a BI implementation accessible to a larger community of users have a material impact on the ROI calculation?........................................................................................ 12 Figure 17: Would improving the ease-of-use of your BI solution have a material impact on the ROI calculation?...................12 Challenges in Calculating ROI for BI Implementations.........................................................................................13 Figure 18: What are the most significant challenges to developing a meaningful ROI analysis?....................................... 13 Figure 19: Do you agree or disagree with the idea that the intangible benefits of BI are more important than the tangible, measurable benefits?........................................................................................... 13 Conclusion..................................................................................................................................................... 14 Detailed Demographic Breakdown of Respondents.............................................................................................14 Figure 20: How many employees are in your organization?..................................................................................... 14 Figure 21: Approximately what is your organization's annual revenue?...................................................................... 15 Figure 22: What is your primary industry?.......................................................................................................... 15 Figure 23: What is your primary job title?..........................................................................................................16 Data collection and analysis performed with Survey Methods

3 Introduction Measuring the return on investment for technology acquisitions is seen as a key management process. Nonetheless, for a variety of reasons, many companies do not conduct formal return on investment studies. ROI studies can be complex. New IT investments generally have direct, indirect, and related expenses, so calculating the appropriate investment can be tricky. Calculating benefits and return is also a challenge. Once again, IT projects generally have both tangible and intangible benefits. And many of the benefits may not be easily assigned a monetary value. Measuring the ROI for business intelligence projects is even more challenging than for other kinds of IT investments. The overall objective of business intelligence is to improve company performance by putting the right information into the right hands at the right time. Many BI implementations are intended to allow business analysts to identify significant trends in enterprise operations and to provide senior management with better information with which better decisions can be made in a timely fashion. Over the past several years, business intelligence implementations have also been intended to provide key metrics to front-line personnel, including the sales force, call center operators and even companies customers. Of course, many different factors have an impact on overall company performance, so precisely measuring the influence of a business intelligence implementation can be difficult, if not impossible. In the same way, it is hard to assign a measurable advantage to providing improved access to better information for a company s customers. To understand the role of ROI analysis in IT generally and the key metrics for measuring ROI for business intelligence specifically, Unisphere Research conducted a Web-based survey on behalf of Noetix. Email solicitations were sent to readers of Unisphere Media publications as well as to a list of IT professionals supplied by Noetix. In total, 415 people participated in the survey. Of that number, approximately 80 percent of the respondents indicated that at least one business intelligence/business analytics tool was in use in their enterprises. Among the study s key findings are: Most companies in the study conduct ROI studies for at least some of their IT projects. A team of IT and line-of-business staff are generally responsible for conducting ROI studies. A majority of respondents in the study have not conducted ROI studies for their BI investments. The key cost metrics for an ROI study for BI is the direct cost of the software licenses and the cost of needed hardware. The key benefit metric for BI implementation for senior management and line-of-business staff is improved access to data. The key benefit metric for IT staff is the time it takes to develop a report. The key metric to judge the impact on decision-making is the quality of information delivered. A majority of respondents felt that the intangible benefits of BI were more important than the tangible benefits. The difficulty in accurately measuring both costs and benefits are the chief hurdles in conducting ROI studies for BI. Data collection and analysis performed with Survey Methods

4 Respondent Pool Respondents came from a wide range of companies of different sizes participating in many different industries and held a variety a job titles. Approximately 64 percent of the respondents indicated they have primarily IT-related responsibilities and close to 23 percent have a combination of IT and line-of-business-related responsibilities. The respondent pool, however, was weighted towards large companies. Approximately 54 percent of the respondents said they were at companies with more than 1,000 employees (nearly 32 percent are at companies with more than 5,000 employees). Judging size by revenue, the spread was more even. About 27 percent of those surveyed said they work at companies with more than $1 billion in revenue while 21 percent work at enterprises with less than $50 million in annual revenue. And while technology companies represented the largest single market sector with around 26 percent of the survey participants, manufacturing, financial services, government/non-profit, healthcare, and retail were also well represented. In terms of job titles, database administrators and systems administrators made up the largest classification, generating about 26 of the respondents. Directors and managers of IT/IS were the next largest group, accounting for more than 18 percent of the respondents. A complete breakdown of the respondent pool can be found in Figures 20 through 23. Conducting ROI Studies in IT Survey respondents routinely conduct ROI analyses of their IT investments. As Figure 1 shows, nearly 90 percent of respondents analyze ROI at least from time to time for what are considered significant IT projects. However, only a little more than a quarter of those surveyed (26.9%), assess ROI for all IT projects. Figure 1: How frequently does your company conduct ROI analysis for significant IT projects? Never 10.91% From time to time 28.43% For every project 26.9% For selected projects 33.76%

5 Not surprisingly, the term significant is interpreted differently in different organizations. In almost one-third of the enterprises (29.5%), ROI studies are conducted for all budgeted IT projects. In about a quarter of the enterprises (23.9%), corporate management determines the project for which return on investment will be calculated. And in around 10 percent of the organizations, ROI is only used in special situations. (See Figure 2.) Figure 2: For which kinds of IT projects does your company conduct ROI analysis? All budgeted IT projects 29.5% All projects with an initial cost of more than $50,000 only 9.73% All projects with an initial cost of more than $100,000 only 11.21% All projects with an initial cost of more than $500,000 only 3.24% All projects with an initial cost of more than $1 million 0.88% Selected projects as determined by corporate management 23.89% Selected projects as determined by IT management 12.39% Special situations 9.14% Charts do not add up to 100 percent due to rounding. 0 20 40 60 80 100 As Figure 3 shows, ROI studies are conducted either by a team consisting of both IT staff and line-of-business staff (30%) or by IT managers (28.9%). Figure 3: In general, when it comes to investments in IT, who is primarily responsible for conducting an ROI analysis? A team from IT and the business side 30.09% Other 4.13% The office of the Chief Financial Officer 7.67% The office of the Chief Information Officer 15.93% Line-of-business application owner 5.31% IT application owner 7.96% IT manager 28.91%

6 BI and ROI The use of BI and analytics was widespread among the respondents. Around 80 percent had at least one business intelligence or analytics application in operation and another six percent planned to implement a BI solution within a year. As Figure 4 shows, around half of the companies had put a new BI application online or upgraded an existing implementation within the last year. As measured by the number of users, the reach of the BI application varied greatly among the enterprises. On the one hand, more than 17 percent of the end-users said more than 500 users accessed their BI applications. On the other hand, almost 14 percent said less than 10 people in their organization used BI. (See Figure 5.) Figure 4: When is the last time you implemented a new BI application (including an upgrade of a current product)? Within the last 5 years 12.38% More than 5 years ago 6.67% Within the last year 50.16% Within the last 2 years 30.79% Figure 5: How many people regularly access your business intelligence applications? 500 or more 17.46% 0-9 13.97% 100-499 20% 10-49 29.21% 50-99 19.37% Charts do not add up to 100 percent due to rounding.

7 As Figure 6 shows, most enterprises do justify costs before implementing a BI application. However, fewer than half of the companies surveyed formally calculate their return on investment for the BI implementation (Figure 7). And even fewer companies have ever calculated the total cost of ownership of their BI applications (Figure 8). Figure 6: Did you have to perform a cost-justification prior to implementing your BI solution? No 31.55% Yes 68.45% Figure 7: Has your company conducted an ROI analysis for your investments in business intelligence? No 54.76% Yes 45.24% Figure 8: Have you ever calculated the total cost of ownership for your BI implementation? Yes 23% No 77%

8 Key Cost Metrics for Calculating BI ROI Calculating the return on investment for IT projects is a formidable activity and in some ways calculating the ROI for business intelligence is more challenging than for other kinds of IT applications. The ROI equation has two parts determining the real costs associated with a project and then being able to measure the real benefits. To accurately assess ROI for business intelligence, both direct and indirect costs must be measured. But assigning costs to a specific project often calls for judgment. For example, if more storage is not purchased specifically for a business intelligence project but will be required in the future to manage the data associated with BI, how should those costs be treated in terms of calculating ROI? Figure 9 ranks the most important direct costs to measure when determining ROI for BI. Not surprisingly, the initial cost of software and new servers required as well as the consulting services needed to implement the project are the most important direct costs to take into consideration. Figure 9: Rank how important it is to measure each of the following direct costs for BI applications. (Scale of 1 to 5, with 5 being very important) Initial cost of software licenses 4.14 Initial cost of new servers needed 4.03 Initial cost of consulting services for implementation 3.94 Initial cost of staff hours to implement the project 3.80 Cost of subsequent upgrades and ongoing maintenance 3.79 Initial costs of changes needed in the network 3.66 Initial cost of additional storage 2.57 0 1 2 3 4 5

9 But the calculation of direct costs is only one step in determining the cost side of the ROI equation. BI implementations also generate indirect costs and other, related costs that must be measured. As could be anticipated, in general, respondents found that the calculation of indirect and related costs was less important than calculating direct costs. Nevertheless, the most important indirect cost to take into consideration is the cost of upgrading related software (Figure 10), while the most important related cost is time for training end-users, both business analysts and other end-users (Figure 11). Figure 10: Rank how important it is to measure each of the following indirect costs in calculating ROI for BI applications. (Scale of 1 to 5, with 5 being very important) Cost of upgrading related software 3.88 Impact on use of existing servers 3.69 Impact on existing network traffic 3.67 Impact on existing storage 3.62 Increase in support calls to IT 3.55 Increase in maintenance of hardware 3.54 0 1 2 3 4 5 Figure 11: Rank how important it is to measure each of the following related costs in calculating ROI for BI applications. (Scale of 1 to 5, with 5 being very important) Time for training other end-users 3.73 Time for training business analysts 3.61 Time for training IT staff 3.51 0 1 2 3 4 5

10 Key Benefit Metrics for Calculating BI ROI Calculating the benefits from an investment can be even more challenging than calculating the costs. Some of the anticipated benefits include the potential to increase productivity among several different communities, including IT staff, senior management and line-of-business personnel. Moreover, business intelligence is intended to improve the quality of the decisions taken. The question is, what are the best measures to determine the potential benefits produced by the use of business intelligence? As Figures 12 and 13 show, the most important metric for improving the productivity of both line-of-business personnel and senior management regarding the use of business intelligence is the time it takes for each community to improve the access to data for making decisions. For line-of-business personnel, the second most important criterion is the number of reports end-users can create on their own. For senior management, the second most important criterion is time senior managers must spend on analysis. Figure 12: Rank how important it is to quantify each of the following benefits for staff in calculating ROI for BI applications. (Scale of 1 to 5, with 5 being very important) Access to data for decision-making 4.06 Number of reports end users can create on their own 3.89 Time spent on analysis 3.78 Time needed to create reports 3.76 Number of end-users who can receive reports 3.65 Adoption of BI tools by other end-users 3.65 0 1 2 3 4 5 Figure 13: Rank how important it is to quantify each of the following for senior managers in calculating ROI for BI applications? (Scale of 1 to 5, with 5 being very important) Access to data for decision-making 3.97 Time spent on analysis 3.80 Increase use of BI by senior managers 3.67 Number of senior managers who can receive reports 3.65 Adoption of BI tools by additional senior managers 3.60 Time needed to create reports 3.49 Number of reports senior managers receive 3.38 0 1 2 3 4 5

11 The productivity measures for IT staff differ from those for senior management and line-of-business personnel. The most important metric for IT staff is the time it takes to develop a report followed by the staff time required to respond to users requests. (See Figure 14.) Figure 14: Rank how important it is to quantify each of the following benefits for IT staff in calculating ROI for BI applications. (Scale of 1 to 5, with 5 being very important) Time it takes IT to develop a report 3.90 Staff time required to respond to user requests 3.83 Number of requests to IT to create reports 3.74 Total number of reports generated by IT 3.60 Time it takes to distribute reports 3.40 0 1 2 3 4 5 While developing metrics to gauge increased productivity from all three communities of users involved IT staff, senior management and line-of-business personnel is important for analyzing ROI, the promise of business intelligence is the potential to improve the quality of decisions made. The most important criterion for measuring the quality of decisions made, according to survey respondents, is the quality of the information available to decision-makers, followed by the quality of information available to frontline personnel. The least important variable to measure that is associated with the quality of decision-making is the quantity of information available to front-line or line-of-business staff (see Figure 15). Figure 15: Rank how important it is to quantify each of the following general benefits in calculating ROI for BI applications? (Scale of 1 to 5, with 5 being very important) Quality of information available to senior management 4.26 Quality of information available to front-line personnel 4.08 Bottom-line impact of specific decision 3.94 Speed at which decisions are made 3.94 Quantity of information available to senior management 3.75 Quantity of information available to front-line personnel 3.68 0 1 2 3 4 5

12 As Figures 16 and 17 show, other factors could also have a material impact on calculating ROI, including making BI more available to a wider community of users and vendors' improvement of the ease of use of BI tools. Figure 16: Would making a BI implementation accessible to a larger community of users have a material impact on the ROI calculation? No 22% Yes 78% Figure 17: Would improving the ease-of-use of your BI solution have a material impact on the ROI calculation? No 10% Yes 90%

13 Challenges in Calculating ROI for BI Implementations Calculating the ROI for BI implementations is a complicated task. As Figure 18 shows, the greatest hurdle companies face in calculating ROI is that both real costs and tangible benefits are hard to quantify. Figure 18: What are the most significant challenges to developing a meaningful ROI analysis? (Multiple responses allowed) Real costs are hard to estimate 59.94% Specific benefits are hard to quantify 57.49% Impact on overall corporate performance is hard to determine 47.09% Process is too complicated 35.47% Process takes too long 30.89% No single person is responsible for calculating ROI 27.52% There is no time to conduct post hoc ROI calculations 24.77% Calculating ROI has no real value 3.67% 0 20 40 60 80 100 The challenges involved in conducting ROI assessments have led some people in the industry to argue that the intangible, non-measurable benefits are more important than the tangible benefits. Survey respondents generally agreed with this view. (See Figure 19.) Figure 19: Do you agree or disagree with the idea that the intangible benefits of BI are more important than the tangible, measurable benefits? No opinion 30.4% Agree 53.8% Disagree 15.8%

14 Conclusion Assessing return on investment is a key IT management discipline. Almost 90 percent of the respondents indicated that their companies engage in ROI analysis for at least some of their IT projects. However, ROI analysis is often not conducted for BI projects. In fact, nearly 55 percent of the respondents in this survey indicated they do not study ROI for their BI investments. Interestingly, 95 percent of large companies (those with over 5,000 employees) conduct ROI studies for at least some of their IT investments and 40 percent conduct ROI studies for every IT project. But even in this group, only around 57 percent conduct ROI studies for BI projects. Companies do not conduct ROI analysis for BI perhaps in part because a majority believes that the intangible benefits of BI are more important than the tangible benefits, which are easier to measure. In conducting ROI studies for BI, respondents said they believe that the most significant cost metrics are the direct costs of software licenses and hardware acquisition. The key benefit metrics are improved access to data and improved quality of a data for decision-making. The greatest hurdles to conducting ROI assessments for BI is that real costs and tangible benefits are hard to quantify and those hurdles seem consistent for both large and small companies. Detailed Demographic Breakdown of Respondents Figure 20: How many employees are in your organization? 1 to 99 13.64% 100 to 499 13.64% 500 to 999 12.42% 1,000 to 4,999 23.03% 5,000 or more 31.82% Decline to state 5.45% 0 20 40 60 80 100

15 Figure 21: Approximately what is your organization's annual revenue? Under $10 million 12.77% $10 million to $50 million 9.42% $50 million to $100 million 5.47% $100 million to $500 million 15.2% $500 million to $1 billion 8.21% $1 billion to $5 billion 15.2% Over $5 billion 12.16 Decline to state 21.58 0 20 40 60 80 100 Charts do not add up to 100 percent due to rounding. Figure 22: What is your primary industry? Manufacturing (includes defense, auto, chemical, durable goods, utilities) 15.5% Retail (includes consumer packaged goods) 6.99% Government/Education/Non-Profit/Military 12.77% Technology (includes computers,telecommunication) 26.44% Financial Services (includes banking, insurance, etc.) 13.68% Healthcare (includes biotechnology) 9.12% Other 15.5% 0 20 40 60 80 100

16 Figure 23: What is your primary job title? Senior Corporate Management (Owner, CEO, CFO, COO) 4.29% Senior IT Management (CIO/CTO/Vice President IS/IT) 6.13% Vice President of a business unit (other than a computer-related function) 0.61% Director/Manager of IS/IT or computer-related function 18.4% Director/Manager of a business unit (non-computer-related function) 6.44% Business Analyst 7.36% Systems Analyst 12.58% Programmer 8.28% Database or Systems Administrator 26.38% Other 9.51% Charts do not add up to 100 percent due to rounding. 0 20 40 60 80 100 The information in this report has been gathered through Web-based surveys of member and prospective member lists provided by the OAUG, through interviews with knowledgeable participants in the computer industry and through secondary research of generally available documents, reports and other published media, as well as from earlier studies conducted by Unisphere Research. Unisphere Research has relied on the accuracy and validity of all information so obtained. Unisphere Research assumes no liability for inaccurate or omitted information. Unisphere Research shall not be liable for errors contained in the document, its data and analysis, and/or for incidental or consequential damages in connection with the use of this document and its information.