IBM QRadar: Evidence of Value

IBM QRadar: Evidence of Value Independently conducted by Ponemon Institute LLC February 2014 Ponemon Institute Research Report

IBM QRadar: Evidence of Value Ponemon Institute: February 2014 Part 1. Introduction Ponemon Institute was engaged by IBM to conduct an independent validation study of its security incident event management (SIEM) solution termed QRadar. The purpose of this study was to better understand the elements of QRadar that deliver value to customers and users. Our study specifically focused on QRadar s product features and experience in enterprise deployment. Another objective of this research was to compare QRadar s value propositions to other marketleading providers of SIEM and network traffic intelligence solutions. 1 We independently conducted one-to-one confidential interviews with a learned group of 25 IT and IT security practitioners from larger-sized U.S. companies in seven industry sectors. By design, all participating interviewees were users of another SIEM solution and later switched to QRadar. Our interview script included 30 fix-formatted questions. Using a diagnostic interview technique, we probed to gain a deeper understanding about QRadar capabilities, benefits and overall value propositions. Following are the four focal points of our interviews. Rationale for selecting QRadar and switching from other SIEM providers Recent experiences in detecting and containing network traffic anomalies Experience in deploying QRadar across the enterprise Comparison of QRadar features to those of other SIEM providers Following are some of the most salient findings: According to respondents (a.k.a. interviewees), the top reasons for switching to QRadar are: (1) to reduce maintenance costs, (2) eliminate vendor support problems, (3) to respond to management mandate and (4) to solve interoperability issues. On average, companies spent $1.53 million on SIEM deployment across the enterprise computed for a three-year lifecycle. Of this total cost, 49 percent pertains to labor, 32 percent pertains to technology investments and 19 percent to costs incurred from external parties such as consultants and managed service providers (termed out-of-the pocket costs). The average time to implement QRadar was 5.5 months versus 15.2 months (nearly 3X) for other market-leading competitor solutions. Seventy-two percent of respondents say anomalous network traffic detection relative to total netflow increased or significantly increased after deploying QRadar. The top benefits realized by companies after deploying QRadar are decreases or significant decreases in the: (1) average time to detect compromises, (2) false positive rate, (3) average time to contain compromises and (4) duration of business disruption caused by cyber attacks. Fifty-six percent of respondents say their experience with QRadar exceeded expectations and 92 percent hold a positive or very positive impression of this SIEM solution. The estimated ROI for 25 cases in this report averages with a range of 11 to 66 percent. 1 Other marketing-leading SIEM providers include: HP (ArcSight), McAfee (Nitro), Splunk, LogRhythm and RSA (Netwitness). Ponemon Institute : Private & Confidential Document Page 1

Limitations There are inherent limitations to small sample size research that need to be carefully considered before drawing inferences from findings. The following items are specific limitations that are germane to most interview-based studies. Our findings are based on a sample of 25 individuals each asked to represent the view and experiences of their company. We recruited these individuals from a proprietary database consisting of IT or IT security practitioners who participated in prior studies. We acknowledge the possibility that individuals who did not participate are substantially different in terms of underlying beliefs, perceptions and experiences. By design, we recruited only those individuals who: (1) presently use QRadar and (2) had switched from another market-leading SIEM provider. We acknowledge that individuals who switched from QRadar to another SIEM provider would likely hold a less positive impression of QRadar. The quality of interview research is based on the integrity of confidential responses provided by respondents. While certain checks and balances were incorporated in our interview script, there is always the possibility that interviewees did not provide full or accurate responses. Ponemon Institute : Private & Confidential Document Page 2

Part 2. Respondents As noted in Figure 1, a majority of respondents hold positions at or above the director level within their organizations. Most respondents are IT security leaders within their respective companies. In total, nine individuals hold the CISO or equivalent job title. The average relevant experience is 16.5 years (median at 15.0 years). Figure 1: Position of respondents 12% 16% 32% Executive Director Manager Staff Figure 2 shows the headcount of participating organizations. As can be seen, the vast majority of respondents organizations have more than 5,000 full time equivalent employees. Figure 2: Headcount of participating companies 28% 4% 12% 16% 1,001 to 5,000 5,001 to 10,000 10,001 to 25,000 25,001 to 75,000 More than 75,000 Ponemon Institute : Private & Confidential Document Page 3

Figure 3 summarizes the primary industry sectors of participants organizations. The largest sectors are financial services, public sector and energy and utilities. Figure 3: Industry distribution of respondents organizations 8% 8% 12% 8% 24% Financial services Public/government Energy & utilities Healthcare Manufacturing Retail Services Figure 4 summarizes the length of time interviewees say their organizations have used QRadar. The largest segment says their organizations have three to five years of QRadar experience. Figure 4: Length of time respondents organizations used QRadar 8% 16% 44% 32% Less than 1 year 1 to 2 years 3 to 5 years More than 5 years Ponemon Institute : Private & Confidential Document Page 4

Figure 5 summarizes the former SIEM providers utilized by respondents companies before making the switch to IBM s QRadar product. Figure 5: Former SIEM providers before switching to QRadar 8% 12% 12% 16% 32% HP ArcSight McAfee Nitro RSA Netwitness Splunk LogRhythm Other Ponemon Institute : Private & Confidential Document Page 5

Part 3. Findings Figure 6 reports the perceptions of QRadar users concerning their former SIEM solution providers. As shown below, a majority of interviewees hold negative impressions of their former providers. Figure 6: QRadar user perceptions about former SIEM providers 50% 45% 35% 30% 25% 15% 10% 5% 0% 44% 28% 16% 12% 0% Very positive Positive Unsure Negative Very negative Figure 7 summarizes the main reasons for switching from prior SIEM provider to IBM s QRadar according to interviewees. The top reasons include maintenance cost, vendor support problems, management mandate and interoperability issues. Figure 7: Reasons for switching to QRadar Maintenance cost Vendor support problems Management mandate Interoperability issues 36% 36% Organizational changes 28% Complexity issues Performance issues 16% Other 12% Product cost Operating costs 8% 8% 0% 5% 10% 15% 25% 30% 35% 45% Ponemon Institute : Private & Confidential Document Page 6

Figure 8 reports the extrapolated average cost incurred by interviewees companies to deploy QRadar across the enterprise. Labor costs represent the highest spending level representing 49 percent of the total SIEM cost. Technology investments and out-of-pocket costs represent 32 and 19 percent, respectively. Out-of-pocket costs include payments to third parties including consultants, vendors and managed service providers. Figure 8: Extrapolated average cost spent on SIEM $800,000 $700,000 $600,000 $500,000 $400,000 $300,000 $200,000 $100,000 $- $493,200 $752,000 $288,000 Technology investment Labor costs Out-of-pocket costs Average extrapolated cost Figure 9 reports the extrapolated average length of time to fully implement SIEM across the enterprise measured in months. This chart compares the interviewees experience implementing their former SIEM solution to the QRadar implementation experience. Albeit only an estimate, the results suggest marked differences in the implementation experience (i.e., 3 X difference). Figure 9: Extrapolated average length of time (months) to implement SIEM 16.0 15.2 14.0 12.0 10.0 8.0 6.0 5.5 4.0 2.0 - QRadar Former SIEM Average months to implement SIEM across the enterprise Ponemon Institute : Private & Confidential Document Page 7

Figure 10 compares anomalous traffic detection rates between former SIEM providers and QRadar. As can be seen, 72 percent of interviewees say the detection rate has increased or significantly increased as a result of the switch to QRadar. In contrast, virtually no interviewee said the detection rate decreased after the switch to QRadar. Figure 10: Comparison of anomalous traffic detection relative to total netflow 50% 45% 35% 30% 25% 15% 10% 5% 0% 44% Significantly increased 28% 28% 0% 0% Increased No change Decreased Significantly decreased Anomalous traffic detected relative to total netflow Ponemon Institute : Private & Confidential Document Page 8

Figure 11 provides the QRadar experience according to nine normatively important attributes. Interviewees rated each attribute using a five-point scale ranging from significantly increased to significantly decreased. Please note that each percentage represents the significantly decreased and decreased response combined. The most salient findings concern the average time to detect compromises, false positive rates, average time to contain compromises and average duration of business disruptions caused by cyber attacks. According to many interviewees, all of these attributes either decreased or significantly decreased as a result of effective deployment of QRadar across the enterprise. Figure 11: Experience after deploying QRadar for nine attributes Percentage of interviewees that said each attribute either decreased or significantly decreased after the deployment of QRadar. Average time to detect compromises 80% False positive rates 80% Average time to contain compromises 64% Average duration of business disruption caused by cyber attacks 48% Non-compliance with policies, regulations and external standards 44% The total cost of downtime and business disruption caused by cyber attacks Average duration of IT downtime caused by cyber attacks 36% Frequency of data breach incidents 36% Frequency of denial of service attacks 28% 0% 10% 30% 50% 60% 70% 80% 90% Significantly decreased and decreased responses combined Ponemon Institute : Private & Confidential Document Page 9

Figure 12 provides very favorable results for QRadar. As can be seen, more than half of the interviewees rate their impression as very positive. In contrast, no interviewee provides a negative rating. Figure 12: Overall impressions of QRadar 60% 50% 52% 30% 10% 0% 8% 0% 0% Very positive Positive Unsure Negative Very negative Respondents' overall impressions of QRadar Figure 13 provides more favorable results for QRadar. Fifty-six percent of interviewees say QRadar exceeded their expectations. Another 36 percent say QRadar met expectations and only 8 percent say it did not meet expectations. Figure 13: Does QRadar meet expectations? 60% 56% 50% 36% 30% 10% 8% 0% Exceeds expectation Meets expectation Does not meet expectation Does QRadar meet your expectations? Ponemon Institute : Private & Confidential Document Page 10

Figure 14 reports the individuated return on investment (ROI) estimates for 25 QRadar users. The ROI calculated for each security technology category is defined as: (1) gains from the investment divided by (2) cost of investment (minus any residual value). We estimate a three-year life for all technology categories presented. Hence, investments are simply amortized over three years. The gains are the net present value of cost savings expected over the investment life. From this amount, we subtract conservative estimates for operations and maintenance cost each year. The net present value used the prime plus 2 percent discount rate per year. We also assume no (zero) residual value. As can be seen, ROI results vary considerably from a high of 66 percent to a low of 11 percent. The mean ROI for all cases is 40 percent, which is illustrated by the dotted line in the graph. Figure 14: ROI estimates for 25 cases 70% 66% 60% 50% 30% 10% 0% 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 11% ROI Average Figure 15 shows the average ROI estimates by organizational size ranges. As can be seen, all ranges especially companies with headcount between 5,001 and 10,000 employees experience very substantial average ROI. 2 Figure 15: ROI estimates by size (headcount) 60% 50% 30% 36% 48% 33% 36% 10% 0% 1,001 to 5,000 5,001 to 10,000 10,001 to 25,000 More than 25,000 Average ROI 2 Please note that ROI averages are computed from small subsamples of companies. Hence, care should be exercised when interpreting these results. Ponemon Institute : Private & Confidential Document Page 11

Figure 16 reports the individuated total cost of ownership (TCO) estimates for 25 QRadar users. Here again, TCO results vary considerably from a high of $3.80 million to a low of $.45 million. The dotted line represents the mean TCO for all cases, which is $1.53 million. Figure 16: TCO estimates for 25 cases $4,000,000 $3,800,000 $3,500,000 $3,000,000 $2,500,000 $2,000,000 $1,500,000 $1,000,000 $500,000 $- $450,000 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 TCO Average Figure 17 reports the TCO estimates by spending category and organizational size ranges. With the exception of out-of-pocket costs, this chart clearly shows that TCO is proportionally related to size. It also shows that labor cost relating to the installation, implementation and maintenance represents the largest cost component. 3 Figure 17: TCO estimates by size (headcount) and spending categories 000 omitted $3,500 $3,000 $2,500 $2,000 $1,500 $1,000 $500 $- Technology Labor cost Out-of-pocket Total 1,001 to 5,000 $135 $275 $173 $583 5,001 to 10,000 $349 $445 $293 $1,087 10,001 to 25,000 $671 $914 $197 $1,783 More than 25,000 $900 $1,713 $550 $3,163 1,001 to 5,000 5,001 to 10,000 10,001 to 25,000 More than 25,000 3 Please note that TCO estimates are computed from small subsamples of companies. Hence, care should be exercised when interpreting these results. Ponemon Institute : Private & Confidential Document Page 12

Appendix 1: Diagnostic Interview Results Fieldwork completed in January 2014 Q1. What best describes your position level within the organization? Freq Pct% Executive 4 16% Director 10 Manager 8 32% Technician 3 12% Other 0 0% Q2. What best describes the full-time headcount of your global organization? Freq Pct% 1,001 to 5,000 4 16% 5,001 to 10,000 10 10,001 to 25,000 7 28% 25,001 to 75,000 1 4% More than 75,000 3 12% Q3. What best describes your organization s primary industry classification? Freq Pct% Financial services 6 24% Healthcare 3 12% Manufacturing 2 8% Public/government 5 Energy & utilities 5 Retail 2 8% Services 2 8% Other 0 0% Q5. Former SIEM technology provider Freq Pct% HP ArcSight 8 32% McAfee Nitro 5 Splunk 3 12% LogRhythm 3 12% RSA Netwitness 4 16% Other 2 8% Q6. Overall impression of your organization s former SIEM or network intelligence solutions Freq Pct% Very positive 0 0% Positive 3 12% Unsure 11 44% Negative 7 28% Very negative 4 16% Ponemon Institute : Private & Confidential Document Page 13

Q7. Reason for the change Freq Pct% Operating costs 2 8% Complexity issues 5 Limited functionality 12 48% Vendor support problems 10 Performance issues 4 16% Organizational changes 7 28% Maintenance cost 10 Interoperability issues 9 36% Product cost 2 8% Management mandate 9 36% Other 3 12% Total 73 292% Q8. Approximate length of time as a QRadar customer Freq Pct% Less than 1 year 4 16% 1 to 2 years 8 32% 3 to 5 years 11 44% More than 5 years 2 8% Q9. The total investment in SIEM technologies Freq Pct% Less than $100,000 2 8% $100,000 to $200,000 5 $200,001 to $400,000 5 $400,001 to $600,000 4 16% $600,001 to $800,000 4 16% $800,001 to $1,000,000 2 8% More than $1,000,000 3 12% Q10. Fully loaded labor costs associated with the implementation and ongoing maintenance of the SIEM solutions: Freq Pct% Less than $100,000 0 0% $100,000 to $200,000 3 12% $200,001 to $400,000 6 24% $400,001 to $600,000 6 24% $600,001 to $800,000 3 12% $800,001 to $1,000,000 2 8% $1,000,001 to $1,500,000 0 0% $1,500,001 to $2,000,000 3 12% More than $2,000,000 2 8% Q11. Out-of-pocket costs paid for services relating to SIEM installation and deployment throughout the enterprise Freq Pct% Less than $100,000 5 $100,000 to $200,000 7 28% $200,001 to $400,000 8 32% $400,001 to $600,000 2 8% $600,001 to $800,000 2 8% $800,001 to $1,000,000 1 4% More than $1,000,000 0 0% Ponemon Institute : Private & Confidential Document Page 14

Q12. Total time in months to install QRadar across the enterprise Freq Pct% Less than 1 month 2 8% 1 to 3 months 8 32% 4 to 6 months 7 28% 7 to 9 months 5 10 to 12 months 1 4% 13 to 15 months 0 0% 16 to 18 months 2 8% 19 to 21 months 0 0% 21 to 24 months 0 0% More than 24 months 0 0% Q13. Total time in months to install former SIEM across the enterprise Freq Pct% Less than 1 month 0 0% 1 to 3 months 2 8% 4 to 6 months 2 8% 7 to 9 months 5 44% 10 to 12 months 4 16% 13 to 15 months 3 12% 16 to 18 months 3 12% 19 to 21 months 0 0% 21 to 24 months 4 16% More than 24 months 2 8% Q14. Anomalous traffic detected relative to total netflow Freq Pct% Significantly increased 11 44% Increased 7 28% No change 7 28% Decreased 0 0% Significantly decreased 0 0% Q15. False positive rates Freq Pct% Increased 0 0% No change 5 Decreased 10 Significantly decreased 10 Q16. Average time to detect compromises Freq Pct% Increased 0 0% No change 5 Decreased 12 48% Significantly decreased 8 32% Q17. Average time to contain compromises Freq Pct% Increased 0 0% No change 9 36% Decreased 9 36% Significantly decreased 7 28% Ponemon Institute : Private & Confidential Document Page 15

Q18. Frequency of data breach incidents Freq Pct% Increased 1 4% No change 15 60% Decreased 5 Significantly decreased 4 16% Q19. Frequency of denial of service (DoS/DDoS) attacks Freq Pct% Increased 0 0% No change 18 72% Decreased 7 28% Significantly decreased 0 0% Q20. Average duration of IT downtime caused by cyber attacks Freq Pct% Increased 0 0% No change 16 64% Decreased 9 36% Significantly decreased 0 0% Q21. Average duration of business disruption caused by cyber attacks Freq Pct% Increased 0 0% No change 13 52% Decreased 10 Significantly decreased 2 8% Q22. The total cost of downtime and business disruption caused by cyber attacks Freq Pct% Increased 0 0% No change 15 60% Decreased 10 Significantly decreased 0 0% Q23. State of compliance with policies, regulations and external standards Freq Pct% Increased 11 44% No change 14 56% Decreased 0 0% Significantly decreased 0 0% Q24. Organizational reputation or brand Freq Pct% Increased 10 No change 15 60% Decreased 0 0% Significantly decreased 0 0% Ponemon Institute : Private & Confidential Document Page 16

Q25. Please record your overall impressions of QRadar Freq Pct% Very positive 13 52% Positive 10 Unsure 2 8% Negative 0 0% Very negative 0 0% Q26. Does QRadar meet your expectations? Freq Pct% Exceeds expectation 14 56% Meets expectation 9 36% Does not meet expectation 2 8% Following are 25 separately compiled ROI and TCO estimates for QRadar Case ROI TCO C1 18% $540,000 C2 52% $390,000 C3 58% $950,000 C4 49% $950,000 C5 44% $950,000 C6 66% $1,700,000 C7 58% $1,700,000 C8 65% $540,000 C9 $2,100,000 C10 31% $1,700,000 C11 32% $2,540,000 C12 $950,000 C13 32% $950,000 C14 61% $390,000 C15 35% $3,800,000 C16 54% $950,000 C17 $3,750,000 C18 37% $1,700,000 C19 49% $1,300,000 C20 25% $450,000 C21 11% $2,540,000 C22 51% $900,000 C23 21% $3,800,000 C24 18% $2,100,000 C25 36% $690,000 Average $1,533,200 Ponemon Institute : Private & Confidential Document Page 17

Please contact research@ponemon.org or call us at 800.877.3118 if you have any questions. Ponemon Institute Advancing Responsible Information Management Ponemon Institute is dedicated to independent research and education that advances responsible information and privacy management practices within business and government. Our mission is to conduct high quality, empirical studies on critical issues affecting the management and security of sensitive information about people and organizations. As a member of the Council of American Survey Research Organizations (CASRO), we uphold strict data confidentiality, privacy and ethical research standards. We do not collect any personally identifiable information from individuals (or company identifiable information in our business research). Furthermore, we have strict quality standards to ensure that subjects are not asked extraneous, irrelevant or improper questions. Ponemon Institute : Private & Confidential Document Page 18