The Challenge of Big Data

Size: px
Start display at page:

Download "The Challenge of Big Data"

Transcription

1 The Challenge of Big Data Benchmarking Large-Scale Data Management Benchmark Research White Paper Aligning Business and IT To Improve Performance Ventana Research 2603 Camino Ramon, Suite 200 San Ramon, CA (925) Copyright Ventana Research 2012 Do Not Redistribute Without Permission

2 San Ramon, California December 2011 Ventana Research performed this research for a fee to determine attitudes toward large-scale data management. This document is based on our research and analysis of information provided by organizations that we deemed qualified to participate in this benchmark research. This research was designed to investigate the information management practices and needs of individuals and organizations and the potential benefits from improving their existing processes, information and systems. This research is not intended for use outside of this context and does not imply that organizations trying to manage big data are guaranteed success by relying on these results to improve planning. Moreover, gaining the most benefit from improving information management on any level requires an assessment of your organization s unique needs to identify gaps and priorities for improvement. We certify that Ventana Research wrote and edited this report independently, that the analysis contained herein is a faithful representation of our evaluation based on our experience with and knowledge of big data and the information management sector, and that the analysis and conclusions are entirely our own. Copyright Ventana Research 2012 Page 2

3 Table of Contents Executive Summary... 5 About This Benchmark Research... 9 Methodology... 9 Qualification... 9 Demographics Company Size by Number of Employees Company Size by Annual Revenue Geographic Distribution Industry Job Title Role by Functional Area Key Insights Organizations use a variety of technologies to manage and analyze big data Organizations are maturing rapidly but unevenly in managing big data Most organizations lack advanced capabilities to maximize the value of their big data For most organizations, storing all data in memory will be a challenge Big-data technologies provide the ability to analyze more data, produce more accurate results and do so in a more timely manner Hadoop is popular for managing and analyzing unstructured data and for performing advanced analytics Training and staffing present significant challenges for big-data projects Many technical challenges remain with respect to big data Practitioners see volume and velocity of data as the most important factors in evaluating big-data technologies Organizations need to analyze a variety of data types, particularly from customers and transactions IT and Finance are key players in the adoption and use of big-data systems Organizations are planning larger clusters and will move them off-premises What To Do Next Assess your organization s maturity in managing big data Determine the types of big data that you need to analyze Investigate the range of capabilities big-data technologies can provide for your organization Consider investing in advanced capabilities to maximize the value of your big data Realize that storing all data in memory may not be a practical option as volumes grow Include Hadoop in your evaluation, especially for managing and analyzing unstructured data and performing advanced analytics Take training and staffing issues into account for big-data projects Acknowledge the technical challenges of managing big data Consider volume and velocity of data as key factors in evaluating big-data technologies Include both IT and Finance in the adoption and use of big-data systems Copyright Ventana Research 2012 Page 3

4 Think about the growing size of big-data clusters and whether to move them offpremises How Ventana Research Can Help About Ventana Research List of Figures 1. Participants by Company Size (Number of Employees) Participants by Company Size (Annual Revenue) Participants by Region Participants by Type of Industry Participants by Job Category Participants by Functional Area Copyright Ventana Research 2012 Page 4

5 Executive Summary Organizations of all sizes and industries find themselves deluged by an unceasing torrent of data. It comes from a multitude of sources in many formats and is stored in systems disparate in nature and location. As well, organizations are capturing data at deeper levels of detail and keeping more history than ever before. As a result, managing these increasing volumes of data known colloquially as big data is emerging as a key challenge for companies. Storing it is only the first step. It is important for organizations to then use it, analyzing the data in the context of their business and thus turning it into information that helps them make decisions, set strategies and interact effectively and profitably with customers, partners and others. To do this, they have to process what can be extremely large amounts of data, and they need to do so quickly. The pace of business today doesn t allow people to wait for overnight batch processing; they need the results now. Data warehouses and databases for operations and analytics have grown enormous, and increasingly they are failing at processing the amounts that users require in acceptable time frames. To meet the growing demand to handle data on a large scale, major database vendors and others that wish to compete with them are developing or acquiring various technologies, among them database appliances, massively parallel databases and columnar databases. Ventana Research undertook this benchmark research to acquire real-world information about levels of maturity, trends and best practices in organizations use of largescale data management systems. It explores how they do this now, how people at various levels feel about the current processes and tools, plans they have to change or improve them, and benefits they hope to gain by doing so. While nine out of 10 organizations are using relational databases on standard hardware for their large-scale data processing, 93 The variety and prevalence of technologies being used indicate this market is still evolving and will continue to change as the newer products mature. percent of these companies are using or evaluating other technologies as well. In addition to flat files, a variety of alternatives have a substantial presence in the market. Two-thirds of participants are using or evaluating data warehouse appliances (69%) and in-memory databases (67%) for their big data needs. In addition, half (49%) are using or evaluating specialized analytical technologies such as columnar databases. The variety and prevalence of technologies being used indicate this market is still evolving and will continue to change as the newer products mature. It also suggests that organizations may need to deploy multiple solutions for specific application requirements. Interest is growing in another contender in this arena: an open source, parallel processing technique called Hadoop. It is designed to process large quantities of data quickly and can help users perform analysis swiftly as well, including some types that aren t possible with conventional systems. Specifically, Hadoop is more likely to be used for advanced analytics and for analyzing unstructured data than other Copyright Ventana Research 2012 Page 5

6 technologies in this study. These and other capabilities have driven rapid adoption of it: More than half (54%) of participants are considering Hadoop for some of their large-scale data processing. About one-third (34%) now use Hadoop or plan to use it within 12 months; almost one-fifth (19%) more are evaluating it or plan to use it in 12 to 24 months. But being a relatively new technology, Hadoop is unfamiliar to many organizations that might benefit from it, and as a result staffing and training are major obstacles to its successful deployment. This study focuses on the management of large-scale data, which we define as beginning at one terabyte (TB). This big data may swell into the petabyte (PB) range depending on various factors including the size of the company and the amounts and types of data it creates and needs to process. More than half (55%) of the organizations participating in this research store and analyze more than 10TB of raw data, and 30 percent of them are producing more than 100GB of data per day. As well, big-data Big-data technologies are not just about keeping up with growing data volumes. Almost two-thirds (63%) of organizations are doing or plan to do things they couldn t do before. technologies are not just about keeping up with growing data volumes. Almost two-thirds (63%) of organizations are doing or plan to use them to do things they couldn t do before. The main reason for working with big data is to analyze data at greater level of detail than was previously possible, but organizations also are motivated by generating more accurate results, speeding up analysis, reducing manual processes and saving on costs. As further evidence that the primary appeal of these technologies is to process very large amounts of data rapidly, nine out of 10 participants rate scalability and performance as the most important evaluation criteria. IT management and staff are the most likely to recommend and evaluate big data technologies so this focus on technical considerations comes as no surprise. Yet Finance, too, plays a major role so we also see a significant focus on costs savings. Finance and administration is the part of the organization that deals second-most often with big-data issues, and excluding those organizations using Hadoop, it does so most often. While debate ebbs and flows about whether any particular big-data technology will sweep away existing approaches to managing large amounts of data, this research finds that the new technologies are in fact supplementing existing IT infrastructure; that is, participants utilize multiple approaches to deal with their big data. Among new capabilities, advanced analytics, planning and visualization capabilities are critical to maximize the value of big data, but the research shows these capabilities are seldom available in most organizations or integrated into their big-data environments. Instead three-quarters (78%) focus on more basic query and reporting. Hadoop appears to fill this gap for some organizations; about two-thirds of those that use it implement predictive analytics (68%) or visualization (67%). In-memory computing, which can process data faster than disk-based systems, is another technology relevant to big data. Until recently the advantages of in-memory systems have been limited to subsets of an organization s data. Now it appears that Copyright Ventana Research 2012 Page 6

7 using these systems for all processing is feasible, but with today s technology it will be a challenge for the majority of organizations. Systems with 1TB of RAM are becoming more widely available, and assuming a data compression ratio of 10 to 1 such systems are capable of managing 10TB of raw data. This would enable some adopters to handle all their data in memory, but it would still present challenges to the majority (55%) of organizations, which have more than 10TB of raw data. The data being analyzed consists of a variety of data types. Rapidly increasing unstructured data and social media receive much of the attention in the big-data market, and the research shows these types of data are common among Hadoop users. Yet in almost four-fifths (78%) of organizations most of the big data being analyzed is generated internally, and the most common types of big data overall are structured, containing information about customers (65%) and transactions (60%). Nonetheless, onethird (31%) of participants are working with large amounts of unstructured data today. The most common types of big data are structured, containing information about customers and transactions. However, one-third of participants are working with large amounts of unstructured data. Many challenges remain in dealing with ever larger amounts of data. Lack of real-time capabilities (67%) and integration with existing business intelligence and data warehousing tools (64%) are the most common technical obstacles. Security represents another area of major concern for big data projects. More than three-fourths (79%) of participants are concerned about a breach in data privacy or security, and slightly less than half of those (36%) need better safeguards to prevent such occurrences. But the biggest issues are not technical. Inadequate training and staffing are the largest obstacles to analyzing big data. Organizations planning to implement big-data management should invest in training current personnel, hiring new people with the necessary skills or both. As noted, the needs to store and analyze more data are the fundamental drivers of the big-data movement. A common technique for processing such data is clustering, which distributes data and processing among a number of interconnected computers that can accomplish more than a single system by working in tandem. As data growth continues, the size of clusters will increase as well. Our research shows that organizations are likely to move beyond small clusters of two to 10 nodes. While the percentage of organizations with clusters consisting of 11to 50 nodes will remain relatively constant at about 25 percent, the percentage of those anticipating clusters of more than 50 nodes will double, from 25 percent to 54 percent. Obviously such expansion will require organizations to acquire more computing power and link the nodes to run in parallel. Hardware costs could be prohibitively high for many that want to do this, and administration will become more complex as clusters grow. The research finds that many organizations are moving toward offpremises deployments for large-scale data management. More than one-third (37%) currently have off-premises deployments, and more than three-fourths (78%) are planning them within 12 months. Software as a service (SaaS) based in the Internet cloud enables users to rent the resources they need and leave maintenance to the Copyright Ventana Research 2012 Page 7

8 supplier; this kind of flexibility may enable companies to respond faster to demand for more processing power than they could do by bringing systems in-house. The use of large-scale data management is a developing phenomenon, and in this sense the market for it may be called immature. However, the organizations already using it, as represented by our research participants, demonstrate greater maturity in this area than we often see in such emerging areas. Our Ventana Research Maturity Index analysis found more organizations at the highest Innovative level (24%) of maturity than at the lowest Tactical level (19%); this unusual result suggests that organizations addressing big data are taking the challenge seriously and making progress. Among the four categories we use to assess organizational maturity, the most difficulty resides in Process; here the majority (57%) ranks at the two lowest maturity levels. It is understandable that merging some of these new technologies with the rest of the organization s processes would pose problems. Overall, we believe that this benchmark research offers hope for companies taking on the challenge of gathering, integrating and analyzing the masses of disparate data they need to be able to understand their markets and make decisions that help them compete successfully. Given the sheer size of the task, which will only increase, this won t be easy, and they will have to commit resources, both human and monetary, to keep up. But there are a variety of alternatives available for those looking to rise to the challenge. Copyright Ventana Research 2012 Page 8

9 About This Benchmark Research Methodology Ventana Research conducted this benchmark research over the Web from February through April We solicited survey participation via blasts, our website and social media invitations. invitations were also sent by our media partners and by vendor sponsors. We presented this explanation of the topic to participants prior to their entry into the survey: The stores of data on which modern businesses rely are already vast and are increasing at an unprecedented pace. Organizations today store and analyze every browser click, every transaction, every event and network packet. We have unquestionably entered a realm where large data warehouses and databases for operations and analytics are now the norm across organizations of all sizes. Dealing with large-scale data in the dozens to hundreds of terabytes requires new approaches to information management processes and technology. These approaches include both established and newer methods for handling larger volumes of data and analytics. This benchmark research is designed to identify trends, best practices and issues in coping with large-scale data warehouses and analytics. It is the first substantial benchmark research conducted in this area and will provide you with important new insights to guide your future operations. The following promotion incented participants to complete the survey: All qualified participants will receive a report on our benchmark research findings that you can apply to your organization s efforts and a quarterly membership to the Ventana Research Community valued at US$125 or 92. In addition, all qualified participants will be entered into a drawing to win a benchmark research report of your choice valued at US$995 or 732. Thank you for your participation! Qualification We designed the research to assess the use of and plans for deployment of largescale data management systems across organizations and industries. Qualification to participate was presented as follows: The survey for this benchmark research is designed for IT and line-ofbusiness personnel involved in purchasing, designing, implementing and maintaining systems that store and analyze large amounts of data. Others such as consultants and systems integrators may participate in the survey, but they are not eligible for incentives and will be used in the analysis only if they meet the qualifications. Incentives are provided to qualified participants in the research and also are conditional on provision of accurate contact information including company name and company address that can be used for fulfillment of incentives. Copyright Ventana Research 2012 Page 9

10 Further qualification evaluation of respondents was conducted as part of the research methodology and quality assurance processes. It entailed screening out responses from companies that are too small, questionnaires that were not materially complete, or those where the submission is from an inappropriate submitter or appears to be spurious. Demographics We designed the survey used for this research to be answered by executives and managers across a broad range of roles and titles working in organizations. We deemed 163 of those who clicked through to this survey to be qualified to have their answers analyzed in this research. In this report, the term participants refers to that group, and the charts in this section characterize various aspects of their demographics and qualifications. Copyright Ventana Research 2012 Page 10

11 Company Size by Number of Employees We require participants to indicate the size of their entire company. Our research repeatedly shows that size of organization as measured by number of employees is a useful means of segmenting companies because it correlates with the complexity of processes, communications and organizational structure as well as the complexity of the IT infrastructure. In this research, participants represented a broad range of organization sizes in rather even distribution: in round terms, one-third work in very large companies (having 10,000 or more employees), one-fourth work in large companies (with 1,000 to 9,999 employees), another one-fourth work in midsize companies (with 100 to 999 employees), and one-seventh work in small companies (with fewer than 100 employees). This distribution is consistent with prior benchmark research and our research objectives and provides a suitably large sample from each size category. Figure 1 Participants by Company Size (Number of Employees) Small 14% Very Large 35% Midsize 24% Source: Ventana Research Large 27% Copyright Ventana Research 2012 Page 11

12 Company Size by Annual Revenue When we measured size by annual revenue, the distribution of categories shifted downward significantly. By this measure, half as many are very large companies (having revenue of more than US$10 billion), the same number are large companies (having revenue from US$500 million to US$10 billion), 6 percent fewer are midsize companies (having revenue from US$100 to US$500 million), and two-and-a-half times more are small companies (with revenue of less than US$100 million). Figure 2 Participants by Company Size (Annual Revenue) Very Large 18% Small 38% Large 27% Source: Ventana Research Midsize 18% Copyright Ventana Research 2012 Page 12

13 Geographic Distribution More than two-thirds of the participants were from companies located or headquartered in North America. Those based in Asia Pacific accounted for one-sixth, in Europe for 7 percent, in the Middle East and in Central and South America for 3 percent each and in Africa for 2 percent. This result was in keeping with our expectations at the start of this investigation, since organizations participating in our research most often are headquartered in North America. However, many of these are global organizations operating worldwide. Middle East 3% Figure 3 Participants by Region Central and Africa South America 2% 3% Europe 7% Asia Pacific 16% North America 69% Source: Ventana Research Copyright Ventana Research 2012 Page 13

14 Industry The companies of the participants in this benchmark research represented a broad range of industries, which we have grouped into four general categories, as shown below. Companies in services accounted for the largest share of participants (45%), followed by those in manufacturing (28%) and finance, insurance and real estate (FIRE, 15%). Government, education and nonprofit organizations accounted for the balance. Figure 4 Participants by Type of Industry Government, Education, Nonprofit 12% Manufacturing 28% Services 45% Finance, Insurance, Real Estate 15% Source: Ventana Research Copyright Ventana Research 2012 Page 14

15 Job Title We asked participants to choose from among a list of job titles the one that best describes theirs. We sorted these responses into three categories: executives, management and users. Almost three-fourths identified themselves as having titles that we categorize as users, a grouping that includes senior manager or manager (23%), director (10%), analyst (16%) and staff (15%). Less than one-fifth are executives, of which half are CIOs or heads of IT (9%), and the balance are vice presidents (9%). Figure 5 Participants by Job Category Executive 17% Management 9% User 74% Source: Ventana Research This is how we aggregated the 14 title response options: Executive CEO, President CIO or Head of Information Technology CFO or Head of Finance Other CxO Management EVP or SVP VP User Director Senior Manager Manager Analyst (Business, Financial, etc.) Staff Copyright Ventana Research 2012 Page 15

16 Data Scientist Consultant Professor or Teacher We concluded after analysis that this response set provided a meaningfully broad distribution of job titles. Role by Functional Area We asked participants to identify their functional area of responsibility as well. Given the especially technical nature of this research, we divided them into those in the lines of business, who comprised a bit more than one-third, and those in IT, who comprised nearly two-thirds. We view these as suitably large samples of each category. Figure 6 Participants by Functional Area Business 37% IT 63% Source: Ventana Research Copyright Ventana Research 2012 Page 16

17 Key Insights Our benchmark research yielded the following important general findings and key insights regarding organizations use of and plans for managing big data. (We discuss the maturity levels of this market in the Maturity Index portion of the full research report; the actual questions asked in our survey are in the Appendix to the research report.) Organizations use a variety of technologies to manage and analyze big data. The research shows that relational database systems and flat files are the most commonly used technologies to manage and analyze big data. Nearly nine out of 10 organizations (89%) are using a relational database on standard hardware, and seven out of 10 are using flat files. However, 93 percent of the organizations using an RDBMS for processing big data are also using or considering some other alternative. Two-thirds of participants are using or evaluating data warehouse appliances and in-memory databases to handle big data. In addition, half are using or evaluating Hadoop and specialized analytical technologies such as columnar databases. Four out of 10 organizations also use or are evaluating other technologies for big data. The variety and prevalence of technologies being used indicate this market is still evolving, and we expect it will continue to change as the newer solutions mature. Organizations are maturing rapidly but unevenly in managing big data. Applying the Ventana Research Maturity Index, which uses a four-level scale to assess organizational maturity, we find the majority (58%) of organizations to be at the two middle levels, Advanced and Strategic. There are more at the highest Innovative level (24%) than at the lowest Tactical level (19%), which suggests that organizations addressing big data are taking the challenge seriously. However, within the four categories of our maturity model People, Process, Information and Technology we see uneven degrees of maturity. Organizations are most mature in the Information category, with two-thirds (68%) ranking at the top two levels. They are least mature in the Process aspect of large-scale data management: The majority (57%) rank at the two lowest levels. This is not an unlikely finding in the implementation of a new technology; integrating it with the rest of the organization s ways of working is sure to be a challenge. Maturity varies according to organization size and industry. As measured by number of employees, small organizations (30%) and very large organizations (28%) are more likely to be Innovative than are midsize (22%) or large (17%) ones. This innovation may be driven by necessity, given the concentration of data sets in excess of 100TB in the two groups at either end of the size spectrum. It s reasonable that very large companies have the biggest data sets, and we expect that small organizations with large data sets are businesses in which the data is a strategic asset. By industry sector, Manufacturing is most innovative (28%) and Government least innovative (20%). Again, necessity seems to play a role as manufacturers have more 100TB databases than average (24% vs. 21%) and Government has the fewest (14%). Copyright Ventana Research 2012 Page 17

18 Most organizations lack advanced capabilities to maximize the value of their big data. Most organizations (78%) are doing query and reporting against their big data, but as data volumes grow it will be harder to navigate through them to find the most meaningful items. Predictive analytics, planning and forecasting and visualization techniques can help make sense of the large amounts of data being processed, but currently organizations have these capabilities available least often among those we asked about and least often have integrated them into big-data environments. Hadoop appears to fill part of this gap for organizations that use it as more than twothirds of them each use Hadoop for predictive analytics and for visualization. Many organizations (68%) are filling the gaps with custom applications, which may solve an immediate problem but will be more difficult to adapt to changing business requirements over time. For most organizations, storing all data in memory will be a challenge. In-memory systems provide better performance for many types of analysis because users can access data in memory more quickly than data stored on disk drives. Until recently the advantages of in-memory systems have been limited to subsets of an organization s data. Most organizations (70%) have less than 100TB of data in aggregate and produce less than 100GB per day (59%). This makes the prospect of completely in-memory systems feasible, but given today s technology and data volumes it will be a challenge for many organizations. Systems with 1TB of RAM are becoming more widely available, and assuming a data compression ratio of 10 to one, such systems would be capable of managing 10TB of raw data. This would enable some organizations to handle all their data in memory, but it would still present challenges to the majority (55%) of organizations, which have more than 10TB of raw data. While memory prices should continue to drop and available memory capacities increase, the research shows that data volumes also will rise as three out of 10 organizations collect more than 100GB of new data per day. Big-data technologies provide the ability to analyze more data, produce more accurate results and do so in a more timely manner. Big-data technologies are useful not only to keep up with growing data volumes. By utilizing them, organizations not only can retain and analyze more data (which 74% said they do) but also speed up their analyses (70%) and generate more accurate results (61%). In addition, almost two-thirds (63%) are doing or plan to do things they couldn t do before. From this perspective the main reason for using big-data technology (for 94%) is to analyze data in greater detail than was previously possible. Big-data tools provide other benefits as well; more than half of research participants each cited half a dozen others, including reducing manual processes, cost savings, higher customer retention from better analyses and utilizing computing resources more efficiently. These results suggest good reasons for organizations to consider big-data technologies if they are not already doing so. Hadoop is popular for managing and analyzing unstructured data and for performing advanced analytics. There are some clear differences in the Hadoop usage patterns compared to other big-data technologies. Organizations use Hadoop much more often with unstructured data. The most common usage is with application log data; next most common is event data. Organizations using Hadoop also more often than others process Web Copyright Ventana Research 2012 Page 18

19 logs, search logs and other types of log data as well as text and multimedia data. Advanced analytics such as data mining and algorithm development are the most common type of big-data application among organizations using Hadoop. This technology provides a flexible data model and programming environment that make it easier to perform these types of analyses than with relational databases and SQLbased languages. With these capabilities, two-thirds (67%) of organizations using Hadoop apply the technology to help create new products and services. However, Hadoop is not without challenges. Participants cited concerns with the lack of integration to existing business intelligence and data warehousing tools, dealing with data in real time and security issues. Training and staffing present significant challenges for big-data projects. For about half (52%) of organizations big-data teams consist of 10 or fewer people, but finding individuals who have the right skills is a serious issue in analyzing big data. More than three-quarters of participants each cited inadequate staffing (79%) and inadequate training (76%) as the most significant obstacles. Two-thirds will train members of their staff to meet the needs for current projects, and more than half (56%) will train existing staff for future projects. Participants expect the skills shortfall to become even more pressing as the largest portion (69%) said they will have to hire and train new staff for future projects. More than half (58%) expect to bring in external consultants. The rise in demand for trained resources will create challenges in terms of hiring and retaining skilled resources. It could also adversely affect the cost of projects as the value of trained resources rises. Many technical challenges remain with respect to big data. Despite the fact that organizations are deriving benefits from working with big data, many technical challenges remain. Lack of real-time capabilities (67%) and of integration with existing business intelligence and data warehousing tools (64%) are the most common technical obstacles. Organizations also reported dissatisfaction with systems management and monitoring as well as historical data retention and archiving capabilities. The absence of these capabilities is not surprising given the relative immaturity of big-data technologies. Because these features are not fully developed, for the time being big-data projects will require more manual effort and resources. Security presents another issue for those pursuing big-data projects. More than three-fourths (79%) of participants are concerned about a data privacy or security breach, and slightly less than half of those (36%) need better safeguards to prevent such occurrences. Practitioners see volume and velocity of data as the most important factors in evaluating big-data technologies. Several characteristics define big data, and the volume of data being processed and its velocity the rate at which it is received are the most important factors to those evaluating big-data technologies. Nine out of 10 participants rated scalability and performance as important or very important evaluation criteria. Nearly as many consider reliability and fault tolerance important or very important. Given the large scale on which big-data systems operate, scalability and performance become closely intertwined. If nodes are down, both quickly become compromised. Historical data retention, another factor driving data volumes, is important or very important to 79 percent of participants. The opportunity to save on costs, ranked important or very important by 84 percent, also relates to data volumes. Since software licensing schemes are often related directly or indirectly to data volumes, this criterion also Copyright Ventana Research 2012 Page 19

20 highlights the importance of finding ways to process large amounts of data not only efficiently but also cost-effectively. Organizations need to analyze a variety of data types, particularly from customers and transactions. Unstructured data and social media receive much of the attention in the big-data market; as already noted, the research shows these types of data are especially common among Hadoop users. Yet three-quarters (78%) of organizations generate internally most of the big data they analyze, and the most common types for all organizations are structured and contain customer (65%) and transaction (60%) data. On the other hand, nearly half (47%) are analyzing data from application logs, and more than one-third (36%) are analyzing network traffic data and other event data, all of which can be unstructured. Almost one-third (31%) of participants are working with large amounts of unstructured data today, and as many (32%) are planning to work with it. Thus organizations must deal with a variety of big data. More than half (56%) currently analyze four or more data types. IT and Finance are key players in the adoption and use of big-data systems. Managing large-scale data is a basically technical activity, and three-quarters (74%) of organizations use a centralized IT function to do it. IT management and staff most often recommend and evaluate big-data technologies, and the CIO most commonly approves the purchase. But the research also finds Finance playing a major role. Finance and administration is the part of the organization that deals second-most often with big-data issues, and if we exclude organizations using Hadoop, it does so most of all. Big data itself shows up most often in operations (57%), but it also is found in more than 40 percent each of strategic planning, sales and marketing, and customer service functions. Organizations are planning larger clusters and will move them off-premises. Clustering, one of the common techniques for processing big data, involves distributing data and processing among a number of interconnected computers. Operating in parallel, these computers can accomplish more in a given amount of time than a single computer. The majority of big-data systems employ this technique: Only 10 percent of organizations attempt to process their data on a single node. These clusters are growing in size as organizations demands keep increasing to process more data. Currently half (49%) of organizations are running clusters of more than 10 nodes, but in the future 81 percent expect to utilize more than 10 nodes. In expectation of needing more server resources, organizations are moving toward off-premises deployments. About one-third (37%) currently have off-premises deployments, but more than three-fourths (78%) are planning them within 12 months. The usual benefits of software as a service (SaaS) less investment inhouse and quicker deployment, for example are multiplied when dealing with large-scale data, as are the resources required, particularly multiple servers and configurations, so we expect this trend toward cloud deployment to accelerate. Copyright Ventana Research 2012 Page 20

21 What To Do Next This benchmark research shows that to manage and analyze big data most organizations use conventional technology: a relational database on standard hardware for 89 percent and flat files for 70 percent. However, 93 percent of the organizations using an RDBMS for processing big data are also using or considering some other alternative. Specifically, two-thirds are using or evaluating data warehouse appliances and in-memory databases; half are using or evaluating Hadoop and specialized analytical technologies such as columnar databases; and 44 percent use or are evaluating other technologies. This variety indicates that organizations are still looking for the best way to handle big data, and we expect the evolution to continue as newer solutions mature. At least for the near future, it is reasonable for organizations to deploy multiple solutions that address specific application requirements. For businesses wishing to improve their performance through managing and processing data on a large scale, we offer the following recommendations. Assess your organization s maturity in managing big data. Applying the Ventana Research Maturity Index, which assesses organizational maturity on four levels, we find the majority (58%) of organizations to be at the two middle levels, Advanced and Strategic. There are more at the highest Innovative level (24%) than at the lowest Tactical level (19%), which suggests that organizations addressing big data are taking the challenge seriously. We advise others to commit significant resources in each of the four categories of our maturity model People, Process, Information and Technology. Organizations are least mature in the Process aspect of large-scale data management, where the majority (57%) rank at the two lowest levels. This is not an unlikely finding in the implementation of a new technology; integrating it with the rest of the organization s ways of working is sure to be a challenge. Determine what yours needs to get the benefits of managing and analyzing the masses of data you accrue. Determine the types of big data that you need to analyze. Three-quarters (78%) of organizations generate internally most of the big data they analyze; the most common types for all organizations are structured data about customers (65%) and transactions (60%). However, nearly half (47%) are analyzing data from application logs, and more than one-third (36%) are analyzing network traffic data and other event data, all of which can be unstructured. Almost one-third (31%) of participants are working with large amounts of unstructured data today, and as many (32%) are planning to work with it. Thus organizations must deal with a variety of big data. Identify which types of big data and how many your organization needs to deal with, and as necessary consider a variety of technologies to process them. Investigate the range of capabilities big-data technologies can provide for your organization. The need to store and analyze more data is the fundamental reason for adopting bigdata technologies, but they can do more. By utilizing them, organizations not only can retain and analyze more data (which 74% of research participants said they do) but also speed up their analyses (70%) and generate more accurate results (61%). In addition, almost two-thirds (63%) are doing or plan to do things they couldn t do before, particularly analysis of data in greater detail than was previously possible. Copyright Ventana Research 2012 Page 21

22 Other benefits, each cited by more than half of research participants, include reducing manual processes, cost savings, higher customer retention from better analyses and utilizing computing resources more efficiently. These are good reasons to consider big-data technologies; determine which apply to your organization and what benefit they can be to the business. Consider investing in advanced capabilities to maximize the value of your big data. Most organizations (78%) are doing query and reporting against their big data, but as data volumes grow it will be harder to navigate through them to find the most meaningful items. We recommend using predictive analytics, planning and forecasting and visualization techniques to help make sense of the large amounts of data being processed, but the research shows that few organizations have these capabilities available and integrated into their big-data environments. If you consider Hadoop, note that more than two-thirds of organizations that have adopted it each use Hadoop for predictive analytics and for visualization. Many organizations (68%) are filling the gaps with custom applications; we caution that this may solve an immediate problem but over time could be inflexible as you try to adapt to changing business requirements. Realize that storing all data in memory may not be a practical option as volumes grow. In-memory systems provide better performance for many types of analysis because users can access data in memory more quickly than data stored on disk drives. Systems with 1TB of RAM are becoming available, and assuming a data compression ratio of 10 to one, such systems would be capable of managing 10TB of raw data. Most organizations (70%) have less than 100TB of data in aggregate and produce less than 100GB per day (59%), which makes the prospect of completely in-memory systems feasible. However, the research finds that the majority (55%) of organizations already have more than 10TB of raw data, and we expect that data volumes will rise as three out of 10 organizations collect more than 100GB of new data per day. It s likely that memory prices will continue to drop as memory capacities increase, but evaluate your requirements to determine whether to make in-memory systems a general-purpose option or limit their use to specific high-value projects. Include Hadoop in your evaluation, especially for managing and analyzing unstructured data and performing advanced analytics. The research shows that, compared to other big-data technologies, organizations use Hadoop much more often with unstructured data; the most common usage is with application log data followed by event data. Advanced analytics such as data mining and algorithm development are the most common type of big-data application among organizations using Hadoop. With these capabilities, two-thirds (67%) of organizations using Hadoop apply the technology to help create new products and services. On the negative side, participants cited concerns with the lack of integration to existing business intelligence and data warehousing tools, dealing with data in real time and security issues. Hadoop, in both open source and commercial versions, should be on your list of big-data technologies, especially for these sorts of advanced analyses. Copyright Ventana Research 2012 Page 22

23 Take training and staffing issues into account for big-data projects. Because analyzing big data is new to most organizations, finding individuals who have the right skills is a serious issue. More than three-quarters of research participants each cited inadequate staffing (79%) and inadequate training (76%) as their most significant obstacles. Don t overlook this aspect of adopting such technology. There are several ways to address it. You can train current staff members for current projects (as 66% will do) or future projects (56%). But the largest portion (69%) said they will have to hire and train new staff for future projects. More than half (58%) expect to bring in external consultants. Keep in mind that the challenges in hiring and retaining skilled resources could increase the cost of projects. As you plan, determine whether a project will require additional resources, and if so begin training and hiring in advance of the start date. Acknowledge the technical challenges of managing big data. Many available big-data technologies are relatively immature. The most participants in this research identified lack of real-time capabilities (67%) and of integration with existing business intelligence and data warehousing tools (64%) as the most common technical obstacles. They also reported dissatisfaction with systems management and monitoring and historical data retention and archiving. Be aware that for the time being big-data projects will require more manual effort and resources, and plan for them. Also take precautions against a data privacy or security breach, which 79 percent of participants are concerned about. Consider volume and velocity of data as key factors in evaluating big-data technologies. The volume of data being processed and its velocity the rate at which it is received are the most important factors to organizations evaluating big-data technologies. Thus nine out of 10 participants rated scalability and performance as important or very important evaluation criteria. Nearly as many consider reliability and fault tolerance important or very important. Given the large scale on which big-data systems operate, scalability and performance become closely intertwined. If nodes are down, both quickly become compromised. Historical data retention, another factor driving data volumes, is important or very important to 79 percent of participants. The opportunity to save on costs (84%) relates to data volumes since vendors often use them in software licensing schemes. We recommend basing your choice of products on their ability to process large amounts of data not only efficiently but also cost-effectively. Include both IT and Finance in the adoption and use of big-data systems. Managing large-scale data is a basically technical activity, and three-quarters (74%) of organizations use a centralized IT function to do it. IT management and staff most often recommend and evaluate big-data technologies, and the CIO most commonly approves the purchase. But the research also finds that Finance and administration deals second-most often with big-data issues, and if we exclude organizations using Hadoop, it does so most of all. Be sure that representatives of both these departments participate in all discussions, and include others from the functions that have the most big data. According to the research, it is found most often in operations (57%) but also in more than 40 percent each of the strategic planning, sales and marketing, and customer service functions. Copyright Ventana Research 2012 Page 23

24 Think about the growing size of big-data clusters and whether to move them off-premises. Most big-data systems employ clustering, which distributes data and processing among interconnected computers to handle large volumes of data rapidly. Only 10 percent of organizations participating in this research attempt to process their data on a single node. These clusters are growing in size along with organizations volumes of data. Currently half (49%) of organizations are running clusters of more than 10 nodes, but in the future 81 percent expect to utilize more than 10 nodes. Addressing the need for more server resources, about one-third (37%) currently have off-premises deployments, and more than twice as many (78%) are planning them within 12 months. The usual benefits of software as a service (SaaS) less investment in-house and quicker deployment, for example are multiplied when dealing with large-scale data, as are the resources required, particularly multiple servers and configurations, so we expect this trend toward cloud deployment to accelerate and advise organizations to consider it as they seek more power in a costeffective manner. Copyright Ventana Research 2012 Page 24

25 How Ventana Research Can Help Ventana Research helps organizations develop, execute and sustain business and technology programs that align people, processes, information and technologies essential for success. As an objective and trusted advisor, we are your insurance that your business and IT initiatives deliver both immediate and long-term improvements to your business. We offer a variety of customizable services to meet your specific needs including workshops, assessments and advisory services. Our education service, led by analysts with more than 20 years of experience, provides a great starting point to learn about important business and technology topics from compliance to business intelligence to building a strategy and driving adoption of best practices. We also offer tailored assessment and benchmarking services to help you connect the business and technology phases of your project by leveraging our research foundation and methodologies. And we can provide Ventana On-Demand access to our analysts on an as-needed basis to help you keep up with market trends, technologies and best practices. Everything at Ventana Research begins with our focused research, of which this report is a part. We work with thousands of organizations worldwide, conducting research and analyzing market trends, best practices and technologies to help our clients improve the efficiency and effectiveness of their organizations. Through the Ventana Research community we also provide opportunities for professionals to share challenges, best practices and methodologies. Sign up for Individual membership at to gain access to our weekly insights and learn about upcoming educational and collaboration events webinars, conferences and opportunities for social collaboration on the Internet. We offer the following membership levels, as well as an additional PLUS option for bundled education: Individual membership: For business and IT professionals* interested in full access to our Web site and analyst team for themselves. The membership includes access to our library of hundreds of white papers and research notes, briefings and telephone/ consulting sessions to provide input and feedback. Team membership: For business and IT professionals* interested in full access to our Web site and analysts for a five-member team. The membership includes access to our library of hundreds of white papers and research notes, briefings, telephone/ consulting sessions to provide input and feedback and the use of Ventana Research materials for business purposes. Business membership: For business and IT professionals* interested in full access to our Web site and analyst team for their larger team or small business unit. The membership includes access to our library of hundreds of white papers and research notes, briefings, telephone/ consulting sessions to provide input and feedback, use of Ventana Research materials for business purposes. To learn more about Ventana Research services including workshops, assessments and advice please contact * Additional services are available for solution providers, software vendors, consultants and systems integrators. Copyright Ventana Research 2012 Page 25

The Challenge of Big Data Benchmarking Large-Scale Data Management Insights from Benchmark Research

The Challenge of Big Data Benchmarking Large-Scale Data Management Insights from Benchmark Research Benchmarking Large-Scale Data Management Insights from Presentation Confidentiality Statement The materials in this presentation are protected under the confidential agreement and/or are copyrighted materials

More information

Big Data Analytics. Research Report Executive Summary. Assessing the Revolution in Big Data and Business Analytics. Sponsored by

Big Data Analytics. Research Report Executive Summary. Assessing the Revolution in Big Data and Business Analytics. Sponsored by Big Data Analytics Assessing the Revolution in Big Data and Business Analytics Research Report Executive Summary Sponsored by Copyright Ventana Research 2013 Do Not Redistribute Without Permission Sponsor

More information

Trends in Total Compensation Management

Trends in Total Compensation Management Trends in Total Compensation Management Establishing the Foundation for Compensation Management Benchmark Research White Paper Aligning Business and IT To Improve Performance Ventana Research 2603 Camino

More information

Next-Generation Predictive Analytics. Research Report Executive Summary. Using Forward-Looking Insights to Gain Competitive Advantage.

Next-Generation Predictive Analytics. Research Report Executive Summary. Using Forward-Looking Insights to Gain Competitive Advantage. Next-Generation Predictive Analytics Using Forward-Looking Insights to Gain Competitive Advantage Research Report Executive Summary Sponsored by Copyright Ventana Research 2013 Do Not Redistribute Without

More information

Optimizing BI and Data Warehouse Performance

Optimizing BI and Data Warehouse Performance Optimizing BI and Data Warehouse Performance New Approaches To Get More from Your Information Assets Executive Summary Aligning Business and IT To Improve Performance Ventana Research 2603 Camino Ramon,

More information

Big Data Integration. Research Report Executive Summary. Challenges and Opportunities in Accessing and Using Today s Information.

Big Data Integration. Research Report Executive Summary. Challenges and Opportunities in Accessing and Using Today s Information. Big Data Integration Challenges and Opportunities in Accessing and Using Today s Information Research Report Executive Summary Sponsored by Copyright Ventana Research 2013 Do Not Redistribute Without Permission

More information

Big Data Analytics. 10 Best Practice Recommendations. Assessing the Revolution in Big Data and Business Analytics. Sponsored by

Big Data Analytics. 10 Best Practice Recommendations. Assessing the Revolution in Big Data and Business Analytics. Sponsored by Big Data Analytics Assessing the Revolution in Big Data and Business Analytics 10 Best Practice Recommendations Sponsored by Copyright Ventana Research 2013 Do Not Redistribute Without Permission February

More information

Predictive Analytics

Predictive Analytics Predictive Analytics Improving Performance by Making the Future More Visible Benchmark Research Research Report Executive Summary Sponsored by Aligning Business and IT To Improve Performance Ventana Research

More information

Sales Compensation Management. Research Report Executive Summary. Improving the Impact of Pay and Incentives to Maximize Revenue.

Sales Compensation Management. Research Report Executive Summary. Improving the Impact of Pay and Incentives to Maximize Revenue. Sales Compensation Management Improving the Impact of Pay and Incentives to Maximize Revenue Research Report Executive Summary Sponsored by Copyright Ventana Research 2013 Do Not Redistribute Without Permission

More information

Exploring the Impact of Geographic Context On Business Processes. Research Report Executive Summary

Exploring the Impact of Geographic Context On Business Processes. Research Report Executive Summary Business Trends in Location Analytics Exploring the Impact of Geographic Context On Business Processes Research Report Executive Summary Copyright Ventana Research 2013 Do Not Redistribute Without Permission

More information

Workforce Analytics and Business Intelligence

Workforce Analytics and Business Intelligence Workforce Analytics and Business Intelligence Understanding and Improving Workforce Performance Executive Summary Aligning Business and IT to Improve Performance Ventana Research 6150 Stoneridge Mall Road,

More information

Business Analytics in Retail

Business Analytics in Retail Business Analytics in Retail Benchmarking the Analysis of Data To Gain Insight Benchmark Research Research Report Executive Summary Aligning Business and IT To Improve Performance Ventana Research 2603

More information

Predictive Analytics Enters the Mainstream

Predictive Analytics Enters the Mainstream Ventana Research: Predictive Analytics Enters the Mainstream Predictive Analytics Enters the Mainstream Taking Advantage of Trends to Gain Competitive Advantage White Paper Sponsored by 1 Ventana Research

More information

Social Media in Recruiting

Social Media in Recruiting Social Media in Recruiting Using New Channels To Source Talent Benchmark Research White Paper Aligning Business and IT To Improve Performance Ventana Research 2603 Camino Ramon, Suite 200 San Ramon, CA

More information

Workforce Optimization 2015. Executive Summary

Workforce Optimization 2015. Executive Summary Workforce Optimization 2015 Executive Summary San Ramon, California January 2015 Ventana Research performed this research and analysis independently. Our goals were to determine the Value Index for Workforce

More information

Customer Experience Management

Customer Experience Management Customer Experience Management Improving the Consistency and Quality of Customer Interactions Executive Summary Aligning Business and IT To Improve Performance Ventana Research 6150 Stoneridge Mall Road,

More information

W H I T E P A P E R. Deriving Intelligence from Large Data Using Hadoop and Applying Analytics. Abstract

W H I T E P A P E R. Deriving Intelligence from Large Data Using Hadoop and Applying Analytics. Abstract W H I T E P A P E R Deriving Intelligence from Large Data Using Hadoop and Applying Analytics Abstract This white paper is focused on discussing the challenges facing large scale data processing and the

More information

Streamline EDI Using A Managed Service

Streamline EDI Using A Managed Service Streamline EDI Using A Managed Service Simplifying Electronic Data Interchange Research Perspective Sponsored by Aligning Business and IT To Improve Performance Ventana Research 2603 Camino Ramon, Suite

More information

Optimizing Payroll Management Making Payroll a Strategic Part of Human Capital Management

Optimizing Payroll Management Making Payroll a Strategic Part of Human Capital Management Optimizing Payroll Management Making Payroll a Strategic Part of Human Capital Management Research Report Copyright Ventana Research 2013 Do Not Redistribute Without Permission Sponsor February 2014 Ventana

More information

Analytics To Optimize Marketing Performance. White Paper. Tools Can Help Marketers Make Better Decisions. Sponsored by

Analytics To Optimize Marketing Performance. White Paper. Tools Can Help Marketers Make Better Decisions. Sponsored by Analytics To Optimize Marketing Performance Tools Can Help Marketers Make Better Decisions White Paper Sponsored by Table of Contents The Role of Marketing Analytics 3 Metrics and Analytics 4 Simplicity

More information

Executive Summary. Aligning Business and IT To Improve Performance

Executive Summary. Aligning Business and IT To Improve Performance Executive Summary of Contents Aligning Business and IT To Improve Performance Ventana Research 6150 Stoneridge Mall Road, Suite 350 Pleasanton, CA 94588 info@ventanaresearch.com (925) 474-0060 www.ventanaresearch.com

More information

Service-Oriented Architecture for Business Intelligence

Service-Oriented Architecture for Business Intelligence Service-Oriented Architecture for Business Intelligence Trends, Needs and Practices Executive Summary Aligning Business and IT and to Improve IT to Improve Performance Performance Ventana Research Headquarters

More information

Big Data at Cloud Scale

Big Data at Cloud Scale Big Data at Cloud Scale Pushing the limits of flexible & powerful analytics Copyright 2015 Pentaho Corporation. Redistribution permitted. All trademarks are the property of their respective owners. For

More information

Making Open Source BI Viable for the Enterprise. An Alternative Approach for Better Business Decision-Making. White Paper

Making Open Source BI Viable for the Enterprise. An Alternative Approach for Better Business Decision-Making. White Paper Making Open Source BI Viable for the Enterprise An Alternative Approach for Better Business Decision-Making White Paper Aligning Business and IT To Improve Performance Ventana Research 6150 Stoneridge

More information

SAP HANA FAQ. A dozen answers to the top questions IT pros typically have about SAP HANA

SAP HANA FAQ. A dozen answers to the top questions IT pros typically have about SAP HANA ? SAP HANA FAQ A dozen answers to the top questions IT pros typically have about SAP HANA??? Overview If there s one thing that CEOs, CFOs, CMOs and CIOs agree on, it s the importance of collecting data.

More information

Contact Center Analytics

Contact Center Analytics Contact Center Analytics Benchmarking the Analysis of Data To Gain Business Insight Benchmark Research Executive Summary Sponsored by Aligning Business and IT To Improve Performance Ventana Research 6150

More information

University of Kentucky Leveraging SAP HANA to Lead the Way in Use of Analytics in Higher Education

University of Kentucky Leveraging SAP HANA to Lead the Way in Use of Analytics in Higher Education IDC ExpertROI SPOTLIGHT University of Kentucky Leveraging SAP HANA to Lead the Way in Use of Analytics in Higher Education Sponsored by: SAP Matthew Marden April 2014 Randy Perry Overview Founded in 1865

More information

Delivering new insights and value to consumer products companies through big data

Delivering new insights and value to consumer products companies through big data IBM Software White Paper Consumer Products Delivering new insights and value to consumer products companies through big data 2 Delivering new insights and value to consumer products companies through big

More information

Self-Service Big Data Analytics for Line of Business

Self-Service Big Data Analytics for Line of Business I D C A N A L Y S T C O N N E C T I O N Dan Vesset Program Vice President, Business Analytics and Big Data Self-Service Big Data Analytics for Line of Business March 2015 Big data, in all its forms, is

More information

CHAPTER - 5 CONCLUSIONS / IMP. FINDINGS

CHAPTER - 5 CONCLUSIONS / IMP. FINDINGS CHAPTER - 5 CONCLUSIONS / IMP. FINDINGS In today's scenario data warehouse plays a crucial role in order to perform important operations. Different indexing techniques has been used and analyzed using

More information

2013 BIG DATA OPPORTUNITIES SURVEY

2013 BIG DATA OPPORTUNITIES SURVEY 2013 BIG DATA OPPORTUNITIES SURVEY By Joseph McKendrick, Research Analyst Produced by Unisphere Research, a Division of Information Today, Inc. May 2013 Sponsored by 2 TABLE OF CONTENTS Executive Summary.............................................................3

More information

Well packaged sets of preinstalled, integrated, and optimized software on select hardware in the form of engineered systems and appliances

Well packaged sets of preinstalled, integrated, and optimized software on select hardware in the form of engineered systems and appliances INSIGHT Oracle's All- Out Assault on the Big Data Market: Offering Hadoop, R, Cubes, and Scalable IMDB in Familiar Packages Carl W. Olofson IDC OPINION Global Headquarters: 5 Speen Street Framingham, MA

More information

PAGE 1 l Teradata Magazine l Q1/2011 l 2011 Teradata Corporation l AR-6309

PAGE 1 l Teradata Magazine l Q1/2011 l 2011 Teradata Corporation l AR-6309 PAGE 1 l Teradata Magazine l Q1/2011 l 2011 Teradata Corporation l AR-6309 It s going mainstream, and it s your next opportunity. by Merv Adrian Enterprises have never had more data, and it s no surprise

More information

Data Analytics Solution for Enterprise Performance Management

Data Analytics Solution for Enterprise Performance Management A Kavaii White Paper http://www.kavaii.com Data Analytics Solution for Enterprise Performance Management Automated. Easy to Use. Quick to Deploy. Kavaii Analytics Team Democratizing Data Analytics & Providing

More information

Navigating Big Data business analytics

Navigating Big Data business analytics mwd a d v i s o r s Navigating Big Data business analytics Helena Schwenk A special report prepared for Actuate May 2013 This report is the third in a series and focuses principally on explaining what

More information

GOVERNANCE MOVES BIG DATA FROM HYPE TO CONFIDENCE

GOVERNANCE MOVES BIG DATA FROM HYPE TO CONFIDENCE GOVERNANCE MOVES BIG DATA FROM HYPE TO CONFIDENCE By Elliot King, Research Analyst Produced by Unisphere Research, a Division of Information Today, Inc. June 2014 Sponsored by 2 TABLE OF CONTENTS Introduction

More information

Colgate-Palmolive selects SAP HANA to improve the speed of business analytics with IBM and SAP

Colgate-Palmolive selects SAP HANA to improve the speed of business analytics with IBM and SAP selects SAP HANA to improve the speed of business analytics with IBM and SAP Founded in 1806, is a global consumer products company which sells nearly $17 billion annually in personal care, home care,

More information

Five Best Practices for Data Management Optimizing the Use of Data for Business Intelligence and Big Data

Five Best Practices for Data Management Optimizing the Use of Data for Business Intelligence and Big Data Ventana Research: Five Best Practices for Data Management Five Best Practices for Data Management Optimizing the Use of Data for Business Intelligence and Big Data White Paper Sponsored by 1 Ventana Research

More information

BUSINESS INTELLIGENCE MATURITY AND THE QUEST FOR BETTER PERFORMANCE

BUSINESS INTELLIGENCE MATURITY AND THE QUEST FOR BETTER PERFORMANCE WHITE PAPER BUSINESS INTELLIGENCE MATURITY AND THE QUEST FOR BETTER PERFORMANCE Why most organizations aren t realizing the full potential of BI and what successful organizations do differently Research

More information

Software-as-a-Service: Managing Benefits for SMBs

Software-as-a-Service: Managing Benefits for SMBs Software-as-a-Service: Managing Benefits for SMBs A research report prepared by: Publication sponsored by: TABLE OF CONTENTS SaaS and SMBs: A Beneficial Combination Key Benefits of SaaS Solutions for SMBs

More information

Big Data Executive Survey

Big Data Executive Survey Big Data Executive Full Questionnaire Big Date Executive Full Questionnaire Appendix B Questionnaire Welcome The survey has been designed to provide a benchmark for enterprises seeking to understand the

More information

Big Data and Analytics Survey. Exclusive Research from

Big Data and Analytics Survey. Exclusive Research from 2015 Big Data and Analytics Survey Exclusive Research from 2015 Big Data & Analytics Survey Insights into Initiatives and Strategies Driving Data Investments Companies are still scrambling to manage ongoing

More information

Workforce Management for 2014. Vendor Report. Assessing the State of Applications and Technology for Business. Prepared for

Workforce Management for 2014. Vendor Report. Assessing the State of Applications and Technology for Business. Prepared for Workforce Management for 2014 Assessing the State of Applications and Technology for Business Vendor Report Prepared for San Ramon, California December 2013 Ventana Research performed this research and

More information

Converged, Real-time Analytics Enabling Faster Decision Making and New Business Opportunities

Converged, Real-time Analytics Enabling Faster Decision Making and New Business Opportunities Technology Insight Paper Converged, Real-time Analytics Enabling Faster Decision Making and New Business Opportunities By John Webster February 2015 Enabling you to make the best technology decisions Enabling

More information

Workforce Analytics Enable Smarter Decisions

Workforce Analytics Enable Smarter Decisions Ventana Research: Workforce Analytics Enable Smarter Decisions Workforce Analytics Enable Smarter Decisions Finding the Right Tool for Human Capital Management White Paper Sponsored by 1 Ventana Research

More information

ANALYTICS BUILT FOR INTERNET OF THINGS

ANALYTICS BUILT FOR INTERNET OF THINGS ANALYTICS BUILT FOR INTERNET OF THINGS Big Data Reporting is Out, Actionable Insights are In In recent years, it has become clear that data in itself has little relevance, it is the analysis of it that

More information

TRENDS IN THE DEVELOPMENT OF BUSINESS INTELLIGENCE SYSTEMS

TRENDS IN THE DEVELOPMENT OF BUSINESS INTELLIGENCE SYSTEMS 9 8 TRENDS IN THE DEVELOPMENT OF BUSINESS INTELLIGENCE SYSTEMS Assist. Prof. Latinka Todoranova Econ Lit C 810 Information technology is a highly dynamic field of research. As part of it, business intelligence

More information

The Big Deal about Big Data. Mike Skinner, CPA CISA CITP HORNE LLP

The Big Deal about Big Data. Mike Skinner, CPA CISA CITP HORNE LLP The Big Deal about Big Data Mike Skinner, CPA CISA CITP HORNE LLP Mike Skinner, CPA CISA CITP Senior Manager, IT Assurance & Risk Services HORNE LLP Focus areas: IT security & risk assessment IT governance,

More information

A financial software company

A financial software company A financial software company Projecting USD10 million revenue lift with the IBM Netezza data warehouse appliance Overview The need A financial software company sought to analyze customer engagements to

More information

Market Pulse Research: Big Data Storage & Analytics

Market Pulse Research: Big Data Storage & Analytics Market Pulse Research: Big Data Storage & Analytics MARKETING RESEARCH EMPLOYEE ENGAGEMENT A WORLD OF INSIGHTS January 2015 Presented on behalf of HP & Microsoft METHODOLOGY & RESEARCH OBJECTIVES Sample

More information

NETWORK SECURITY FOR SMALL AND MID-SIZE BUSINESSES

NETWORK SECURITY FOR SMALL AND MID-SIZE BUSINESSES NETWORK SECURITY FOR SMALL AND MID-SIZE BUSINESSES September, 2015 Derek E. Brink, CISSP, Vice President and Research Fellow IT Security and IT GRC Report Highlights p2 p4 p6 p7 SMBs need to adopt a strategy

More information

Increase success using business intelligence solutions

Increase success using business intelligence solutions white paper Business Intelligence Increase success using business intelligence solutions Business intelligence (BI) is playing an increasingly important role in helping large insurance carriers and insurers

More information

W H I T E P A P E R E d u c a t i o n a t t h e C r o s s r o a d s o f B i g D a t a a n d C l o u d

W H I T E P A P E R E d u c a t i o n a t t h e C r o s s r o a d s o f B i g D a t a a n d C l o u d Global Headquarters: 5 Speen Street Framingham, MA 01701 USA P.508.872.8200 F.508.935.4015 www.idc.com W H I T E P A P E R E d u c a t i o n a t t h e C r o s s r o a d s o f B i g D a t a a n d C l o

More information

Analytics: The real-world use of big data

Analytics: The real-world use of big data Findings from the research collaboration of IBM Institute for Business Value and Saïd Business School, University of Oxford Analytics: The real-world use of big data How innovative enterprises extract

More information

W H I T E P A P E R B u s i n e s s I n t e l l i g e n c e S o lutions from the Microsoft and Teradata Partnership

W H I T E P A P E R B u s i n e s s I n t e l l i g e n c e S o lutions from the Microsoft and Teradata Partnership W H I T E P A P E R B u s i n e s s I n t e l l i g e n c e S o lutions from the Microsoft and Teradata Partnership Sponsored by: Microsoft and Teradata Dan Vesset October 2008 Brian McDonough Global Headquarters:

More information

OPTIMIZING PERFORMANCE IN AMAZON EC2 INTRODUCTION: LEVERAGING THE PUBLIC CLOUD OPPORTUNITY WITH AMAZON EC2. www.boundary.com

OPTIMIZING PERFORMANCE IN AMAZON EC2 INTRODUCTION: LEVERAGING THE PUBLIC CLOUD OPPORTUNITY WITH AMAZON EC2. www.boundary.com OPTIMIZING PERFORMANCE IN AMAZON EC2 While the business decision to migrate to Amazon public cloud services can be an easy one, tracking and managing performance in these environments isn t so clear cut.

More information

On-Demand vs. On-Premise Customer Relationship Management: A New Hybrid Emerges

On-Demand vs. On-Premise Customer Relationship Management: A New Hybrid Emerges I D C I - V I E W Mary Wardley Vice President, Enterprise Applications and CRM Software On-Demand vs. On-Premise Customer Relationship Management: A New Hybrid Emerges September 2007 Adapted from Worldwide

More information

Business Analytics and Big Data. Computerworld March 2012 Survey Results

Business Analytics and Big Data. Computerworld March 2012 Survey Results Business Analytics and Big Data Computerworld March 2012 Survey Results Purpose & Methodology Survey Sample Survey Method Field Work 1/24/12-2/21/12 Audience Base Computerworld Online Collection Number

More information

In-Memory Analytics for Big Data

In-Memory Analytics for Big Data In-Memory Analytics for Big Data Game-changing technology for faster, better insights WHITE PAPER SAS White Paper Table of Contents Introduction: A New Breed of Analytics... 1 SAS In-Memory Overview...

More information

B-to-B Lead Generation:

B-to-B Lead Generation: Marketing ROI & Performance Evaluation Study A special report from MarketingProfs, with support from Lenskold Group Copyright 2008. MarketingProfs Research Insights, MarketingProfs, LLC. All rights reserved.

More information

Managing Big Data with Hadoop & Vertica. A look at integration between the Cloudera distribution for Hadoop and the Vertica Analytic Database

Managing Big Data with Hadoop & Vertica. A look at integration between the Cloudera distribution for Hadoop and the Vertica Analytic Database Managing Big Data with Hadoop & Vertica A look at integration between the Cloudera distribution for Hadoop and the Vertica Analytic Database Copyright Vertica Systems, Inc. October 2009 Cloudera and Vertica

More information

At a recent industry conference, global

At a recent industry conference, global Harnessing Big Data to Improve Customer Service By Marty Tibbitts The goal is to apply analytics methods that move beyond customer satisfaction to nurturing customer loyalty by more deeply understanding

More information

Building an ROI Model for a Cloud-Based Contact Center

Building an ROI Model for a Cloud-Based Contact Center Building an ROI Model for a Cloud-Based Contact Center Comparing the cases for on-premises and cloud-based systems Research Perspective Sponsored by Aligning Business and IT To Improve Performance Ventana

More information

Busin i ess I n I t n e t ll l i l g i e g nce c T r T e r nds For 2013

Busin i ess I n I t n e t ll l i l g i e g nce c T r T e r nds For 2013 Business Intelligence Trends For 2013 10 Trends The last few years the change in Business Intelligence seems to accelerate under the pressure of increased business demand and technology innovations. Here

More information

Five Risks of Ignoring Sales Performance Management

Five Risks of Ignoring Sales Performance Management Five Risks of Ignoring Sales Performance Management Seize the Opportunity To Achieve Better Results White Paper Sponsored by Aligning Business and IT To Improve Performance Ventana Research 2603 Camino

More information

White Paper. Version 1.2 May 2015 RAID Incorporated

White Paper. Version 1.2 May 2015 RAID Incorporated White Paper Version 1.2 May 2015 RAID Incorporated Introduction The abundance of Big Data, structured, partially-structured and unstructured massive datasets, which are too large to be processed effectively

More information

The 3 questions to ask yourself about BIG DATA

The 3 questions to ask yourself about BIG DATA The 3 questions to ask yourself about BIG DATA Do you have a big data problem? Companies looking to tackle big data problems are embarking on a journey that is full of hype, buzz, confusion, and misinformation.

More information

Workforce Management in the Cloud. White Paper. Case Studies Reveal Benefits of Cloud Deployment. Sponsored by

Workforce Management in the Cloud. White Paper. Case Studies Reveal Benefits of Cloud Deployment. Sponsored by Workforce Management in the Cloud Case Studies Reveal Benefits of Cloud Deployment White Paper Sponsored by Table of Contents The Rationale for Cloud Computing 3 Adoption in the Enterprise 4 Human Capital

More information

Turning Big Data into Big Insights

Turning Big Data into Big Insights mwd a d v i s o r s Turning Big Data into Big Insights Helena Schwenk A special report prepared for Actuate May 2013 This report is the fourth in a series and focuses principally on explaining what s needed

More information

Sales and Operations Planning

Sales and Operations Planning Sales and Operations Planning Measuring Maturity and Opportunity for Operational Performance Management White Paper Aligning Business and IT to Improve Performance Aligning Business and IT to Improve Performance

More information

Sage X3. Enterprise Business Management Solutions in the 21st Century: Key Buying Considerations

Sage X3. Enterprise Business Management Solutions in the 21st Century: Key Buying Considerations Sage X3 Enterprise Business Management Solutions in the 21st Century: Table of Contents 3 Legacy Systems are Good to a Point 3 On-Premise or Cloud: Which Option makes Sense 4 The Modern Business Management

More information

Adoption, Approaches & Attitudes

Adoption, Approaches & Attitudes Adoption, Approaches & Attitudes The Future of Cloud Computing in the Public and Private Sectors A Global Cloud Computing Study Sponsored by JUNE 2011 TABLE OF CONTENTS Executive Summary... 1 Methodology

More information

5 Keys to Unlocking the Big Data Analytics Puzzle. Anurag Tandon Director, Product Marketing March 26, 2014

5 Keys to Unlocking the Big Data Analytics Puzzle. Anurag Tandon Director, Product Marketing March 26, 2014 5 Keys to Unlocking the Big Data Analytics Puzzle Anurag Tandon Director, Product Marketing March 26, 2014 1 A Little About Us A global footprint. A proven innovator. A leader in enterprise analytics for

More information

OPEN MODERN DATA ARCHITECTURE FOR FINANCIAL SERVICES RISK MANAGEMENT

OPEN MODERN DATA ARCHITECTURE FOR FINANCIAL SERVICES RISK MANAGEMENT WHITEPAPER OPEN MODERN DATA ARCHITECTURE FOR FINANCIAL SERVICES RISK MANAGEMENT A top-tier global bank s end-of-day risk analysis jobs didn t complete in time for the next start of trading day. To solve

More information

We are Big Data A Sonian Whitepaper

We are Big Data A Sonian Whitepaper EXECUTIVE SUMMARY Big Data is not an uncommon term in the technology industry anymore. It s of big interest to many leading IT providers and archiving companies. But what is Big Data? While many have formed

More information

Information management software solutions White paper. Powerful data warehousing performance with IBM Red Brick Warehouse

Information management software solutions White paper. Powerful data warehousing performance with IBM Red Brick Warehouse Information management software solutions White paper Powerful data warehousing performance with IBM Red Brick Warehouse April 2004 Page 1 Contents 1 Data warehousing for the masses 2 Single step load

More information

Reaping the Rewards of Big Data

Reaping the Rewards of Big Data Reaping the Rewards of Big Data TABLE OF CONTENTS INTRODUCTION: 2 TABLE OF CONTENTS FINDING #1: BIG DATA PLATFORMS ARE ESSENTIAL FOR A MAJORITY OF ORGANIZATIONS TO MANAGE FUTURE BIG DATA CHALLENGES. 4

More information

University Competence Center: Leading a Co-Innovation Project on SAP Cloud Appliance Library

University Competence Center: Leading a Co-Innovation Project on SAP Cloud Appliance Library 2014 SAP SE or an SAP affiliate company. All rights reserved. University Competence Center: Leading a Co-Innovation Project on SAP Cloud Appliance Library Organization University Competence Center, an

More information

Remote Collaboration Solutions In the Financial Services Industry

Remote Collaboration Solutions In the Financial Services Industry WHITE PAPER Remote Collaboration Solutions In the Financial Services Industry Using Audio/Web Conferencing to Drive Profitable Growth By Debra Chin Senior Vice President Palmer Research May 2011 This Palmer

More information

Navigating the big data challenge

Navigating the big data challenge Navigating the big data challenge Do you have lots of data but few insights? By Rasmus Wegener and Velu Sinha Rasmus Wegener is a partner with Bain & Company in Atlanta. Velu Sinha is a partner in Bain

More information

HP Vertica OnDemand. Vertica OnDemand. Enterprise-class Big Data analytics in the cloud. Enterprise-class Big Data analytics for any size organization

HP Vertica OnDemand. Vertica OnDemand. Enterprise-class Big Data analytics in the cloud. Enterprise-class Big Data analytics for any size organization Data sheet HP Vertica OnDemand Enterprise-class Big Data analytics in the cloud Enterprise-class Big Data analytics for any size organization Vertica OnDemand Organizations today are experiencing a greater

More information

Using Tableau Software with Hortonworks Data Platform

Using Tableau Software with Hortonworks Data Platform Using Tableau Software with Hortonworks Data Platform September 2013 2013 Hortonworks Inc. http:// Modern businesses need to manage vast amounts of data, and in many cases they have accumulated this data

More information

GigaSpaces Real-Time Analytics for Big Data

GigaSpaces Real-Time Analytics for Big Data GigaSpaces Real-Time Analytics for Big Data GigaSpaces makes it easy to build and deploy large-scale real-time analytics systems Rapidly increasing use of large-scale and location-aware social media and

More information

Why Big Data in the Cloud?

Why Big Data in the Cloud? Have 40 Why Big Data in the Cloud? Colin White, BI Research January 2014 Sponsored by Treasure Data TABLE OF CONTENTS Introduction The Importance of Big Data The Role of Cloud Computing Using Big Data

More information

Using and Choosing a Cloud Solution for Data Warehousing

Using and Choosing a Cloud Solution for Data Warehousing TDWI RESEARCH TDWI CHECKLIST REPORT Using and Choosing a Cloud Solution for Data Warehousing By Colin White Sponsored by: tdwi.org JULY 2015 TDWI CHECKLIST REPORT Using and Choosing a Cloud Solution for

More information

EMC: Managing Data Growth with SAP HANA and the Near-Line Storage Capabilities of SAP IQ

EMC: Managing Data Growth with SAP HANA and the Near-Line Storage Capabilities of SAP IQ 2015 SAP SE or an SAP affiliate company. All rights reserved. EMC: Managing Data Growth with SAP HANA and the Near-Line Storage Capabilities of SAP IQ Based on years of successfully helping businesses

More information

Buyer s Guide to Big Data Integration

Buyer s Guide to Big Data Integration SEPTEMBER 2013 Buyer s Guide to Big Data Integration Sponsored by Contents Introduction 1 Challenges of Big Data Integration: New and Old 1 What You Need for Big Data Integration 3 Preferred Technology

More information

Smart Ways To Improve Contact Center Performance

Smart Ways To Improve Contact Center Performance Smart Ways To Improve Contact Center Performance The right technology helps measure what matters White Paper sponsored by Aligning Business and IT To Improve Performance Ventana Research 1900 South Norfolk

More information

Big Data Are You Ready? Thomas Kyte http://asktom.oracle.com

Big Data Are You Ready? Thomas Kyte http://asktom.oracle.com Big Data Are You Ready? Thomas Kyte http://asktom.oracle.com The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated

More information

Streamlining the Process of Business Intelligence with JReport

Streamlining the Process of Business Intelligence with JReport Streamlining the Process of Business Intelligence with JReport An ENTERPRISE MANAGEMENT ASSOCIATES (EMA ) Product Summary from 2014 EMA Radar for Business Intelligence Platforms for Mid-Sized Organizations

More information

Assessing Your Business Analytics Initiatives

Assessing Your Business Analytics Initiatives Assessing Your Business Analytics Initiatives Eight Metrics That Matter WHITE PAPER SAS White Paper Table of Contents Introduction.... 1 The Metrics... 1 Business Analytics Benchmark Study.... 3 Overall

More information

Visualizing the Customer Journey Using Analytics. White Paper. Optimizing the Customer Experience. Sponsored by

Visualizing the Customer Journey Using Analytics. White Paper. Optimizing the Customer Experience. Sponsored by Visualizing the Customer Journey Using Analytics Optimizing the Customer Experience White Paper Sponsored by Table of Contents Becoming Customer-Focused 3 Analytics Sharpens the Focus 5 Benefits of an

More information

IBM Software Hadoop in the cloud

IBM Software Hadoop in the cloud IBM Software Hadoop in the cloud Leverage big data analytics easily and cost-effectively with IBM InfoSphere 1 2 3 4 5 Introduction Cloud and analytics: The new growth engine Enhancing Hadoop in the cloud

More information

NetApp Big Content Solutions: Agile Infrastructure for Big Data

NetApp Big Content Solutions: Agile Infrastructure for Big Data White Paper NetApp Big Content Solutions: Agile Infrastructure for Big Data Ingo Fuchs, NetApp April 2012 WP-7161 Executive Summary Enterprises are entering a new era of scale, in which the amount of data

More information

white paper Big Data for Small Business Why small to medium enterprises need to know about Big Data and how to manage it Sponsored by:

white paper Big Data for Small Business Why small to medium enterprises need to know about Big Data and how to manage it Sponsored by: white paper Big Data for Small Business Why small to medium enterprises need to know about Big Data and how to manage it Sponsored by: Big Data is the ability to collect information from diverse sources

More information

BIG DATA, BIG CHALLENGES, BIG OPPORTUNITIES:

BIG DATA, BIG CHALLENGES, BIG OPPORTUNITIES: BIG DATA, BIG CHALLENGES, BIG OPPORTUNITIES: 2012 IOUG BIG DATA STRATEGIES SURVEY By Joseph McKendrick, Research Analyst Produced by Unisphere Research, a Division of Information Today, Inc. September

More information

Solve Your Toughest Challenges with Data Mining

Solve Your Toughest Challenges with Data Mining IBM Software Business Analytics IBM SPSS Modeler Solve Your Toughest Challenges with Data Mining Use predictive intelligence to make good decisions faster Solve Your Toughest Challenges with Data Mining

More information

Best practices for managing the data warehouse to support Big Data

Best practices for managing the data warehouse to support Big Data E-Guide Best practices for managing the data warehouse to support Big Data The new challenge for IT and data warehousing teams is how to leverage existing technology investments along with emerging tools

More information