S CIOs & BIG DATA CIOs & BIG DATA WHAT YOUR IT TEAM WANTS YOU TO KNOW
Foreword Today enterprises face an increasingly competitive and erratic global business environment, and Big Data has become more than just another IT project it s truly a finger on the pulse of the business. To say that in 2013 Big Data is mission critical is to put it mildly organizations that ignore the insights that Big Data can deliver are flying blind. So, it is all the more disconcerting that 55% of Big Data projects don t get completed, and many others fall short of their objectives. While there may be many reasons for this, undo ubtedly one of the biggest factors is lack of communication between top managers, who provide the overall project vision, and those charged with actually implementing it. Far too frequently the opinions of the IT staff doing the heavy lifting necessary to develop a Big Data project are taken as an afterthought, and consequently considered only when projects veer off-course. Given the stakes, it s imperative that CIOs have a 360 degree view of what such an initiative will involve before it leaves the drawing board. In order to provide a complete picture, we partnered with SSWUG.ORG, one of the largest enterprise technology-focused, community-driven sites and a source for answers to IT-related questions and professional growth for more than 570,000 members. Together, we recieved survey responses from over 300 IT department staffers 58% of whom have current Big Data projects underway on what they most wanted their CIOs to know about the process of implementing Big Data projects. The insight we gleaned reveals much about both enterprise technology and enterprise culture. In order for companies to succeed in this era of Big Data, executives will need to rethink longheld notions of how multiple departments should function together. In the past breaking down silos was a nice mantra. Now, it is an imperative. Additionally, CIOs and other enterprise executives may find it necessary to educate their organizations on the advantages of new Big Data applications and processes that will give them better customer insights, make their jobs infinitely easier and give their departments the elasticity needed to meet virtually any business need in realtime. We hope this report will serve not only as a source of insight for those responsible for architecting Big Data initiatives, but also as a reminder to seek the invaluable perspective of IT staff as early as possible in the process of developing technology projects. Jim Kaskade, CEO, Infochimps
About the Research Infochimps, a CSC company, and SSWUG.ORG queried more than 300 IT department employees for this report in fall of 2012, 58% of whom identify themselves as currently being involved in big data initiatives. Big Data implementation is a concern for companies of all sizes, and we sought the opinions of people from organizations ranging from small business to enterprise level. Of those surveyed, 29% come from companies with more than 1000 employees, 27% are from companies with between 101-1000 employees, 45% identify their companies as having 100 or fewer employees. It may also be worth noting that 85 percent of respondents described their data footprint as being larger than one terabyte, and 7 percent of those described it as being larger than one petabyte, indicating the tremendous scope and variety of organizations considering Big Data initiatives. For the purposes of this report and in order to most accurately represent well-informed viewpoints, the statistics cited are based on the responses of those participants who are currently involved in Big Data initiatives. A breakdown of the opinions of the complete respondent pool may be provided upon request. However, given that a key objective of this report is to inform CIOs regarding potential pitfalls to successfully implementing Big Data projects, we feel that the survey results, as presented, offer the most valuable and strategic insights. In order to hone in on insight from those primarily responsible for implementation, the sample is weighted toward staff-level positions. 86 percent of respondents identify themselves as either directors, managers, or systems administrators/ developers, while the remaining 14 percent identify themselves as either VP/SVP or as belonging to the C-suite.
Executive Summary Big Data may indeed be a buzzword for 2013, but if so it s a buzzword with inestimable impact on the organization. At its core, the concept of Big Data is that of supporting executive decision-making with the most accurate, current, comprehensive and comprehensible presentation of all information available regarding a business. Overwhelmingly, respondents indicated that their companies are taking Big Data seriously 81% list Big Data/Advanced Analytics Projects in their Top 5 2013 IT priorities. The survey s finding that IT staff members prefer to focus on the application layer rather than on infrastructure may provide a strong indication as to where CIOs ought to be directing their investments. When asked about top IT priorities outside of Big Data, respondents ranked mobile, website, security and dashboarding notably above infrastructure-related options. These responses are aligned with a general preference for managing the front-end over the back-end, with participants reporting that processing (43%), analyzing (41%) and ongoing management (42%) of Big Data pose significant difficulties companies may consider outsourcing these functions to keep staff focused on impacting the bottom line. Furthermore, teams want to be actively involved in selecting resources and determining how they re pieced together, but not necessarily in managing and maintaining them. 66% respondents prefer to build in-house, as Big Data represents a huge professional opportunity but in reality they re often forced to spend too much time fighting technologies, and not enough time actually using them to create business value. Interestingly, 61% of respondents said managed, hosted platforms are a viable solution, indicating a willingness to outsource the back-end and focus on delivering value through application. As a relatively new development, it may come as no surprise that participants list education and the ability to understand the platforms as top challenges on Big Data projects. The report findings also reflect a general talent gap when it comes to data analysis expertise, which companies must address before embarking on new projects. Lack of Business Context Around the Data (51%) and Lack of Expertise to Connect the Dots (51%) are identified as the top reasons Big Data projects fail. On a similar note, respondents say Data Scientists/Analysts and Business Users are more impactful to the bottom line than the C-Suite, indicating once again that those who can derive business value from data are in demand. In general, it appears that business goals of Big Data projects are in sync with a general trend toward elasticity and business agility. Respondents listed the ability to scale, ease of management, flexible architecture and speed to deployment/ security as the top four most critical requirements for Big Data platforms.
Executive Summary Much of what we found speaks to the importance of creating a business culture that supports Big Data. For example, participants rated Gathering Data from Siloed Sources as their Most Significant Big Data challenge. Breaking down barriers to support a data unification initiative can require executive air cover, and a lot of socializing. Also, having the patience to define very specific project parameters also appears to be of paramount importance, with 58% of respondents listing Inaccurate Scope as the top reason IT projects fail. As a best practice, it is advisable to start with a single key business use case with a clearly stated objective that impacts the business. Finally, while much of today s Big Data dialogue centers on Hadoop, it s worth noting that respondents rated Batch (53 %) and Real-time (49%) data processing as nearly equally important components of business decision-making. Clearly, as a batch-oriented system Hadoop isn t enough. In general, we find that Big Data projects often fall short or take longer than planned due to challenges related to education, social issues, and integration work. While IT staff may be fully aware of the obstacles to Big Data success, and the commitment a project requires, they may be less aware of options to make their jobs easier, such as partnering with an outside vendor that can help speed time to business value. IF YOU WANTED YOUR CIO TO KNOW ONE THING To benefit fully from Big Data, you must look beyond the limitation of what you think can be extracted from the data available and instead think about and ask for what you wish you could derive/learn/extract from the data. Survey Participant
CIOS & BIG DATA: What Your IT Team Wants You to Know In general, we found that companies are taking Big Data seriously in 2013, ranking it high on the list of IT priorities. In fact, 81% of companies have Big Data/Advanced Analytics Projects in their Top 5 2013 IT priorities. Weighing the current business environment which is growing more competitive, global and erratic by the moment against the real-time, predictive insight of a well-run Big Data initiative, the remaining 19 percent ought to be asking themselves why they re not more heavily invested in Big Data. Emphasizing Application Development Over Infrastructure Maintenance The survey showed that while IT staff do not want to divorce themselves from having a say in how the various components of the stack are chosen and assembled, they would prefer to focus on the application. Top priorities other than Big Data for IT staff include mobile, website, security and dashboarding, all of which reside in the application layer, not in the infrastructure. For CIOs, this may provide a guide for 2013 investment strategy. However, such a strategy would not be based solely on work preferences. Responses also indicate that many back-end functions are the very things that devour the most staff time within Big Data projects, processing (43%), analyzing (41%) and ongoing management (42%) of the data pose significant difficulties for the IT team. If companies want to keep their IT staff focused on leveraging Big Data to impact the bottom line, they should consider outsourcing these functions. 66% of respondents prefer to build in-house, which should not be surprising, as Big Data represents a tremendous opportunity for IT staff to build their talents and resumes. To spur innovation, constantly challenging them with career-advancing projects and technologies and empowering them with education is clearly a winning management strategy. However, consider that this finding may be more indicative of a desire to be actively involved in selecting resources, tools and solutions and determining how they re pieced together, rather than in managing and maintaining them. Teams are often forced to spend too much time fighting
CIOS & BIG DATA: What Your IT Team Wants You to Know technologies and not enough time actually using them to create business value. In short, those who could be answering the important questions and producing the greatest insights are stuck wrestling with Hadoop clusters and other Big Data technologies. Relegating the best and brightest to custodial functions that could easily be outsourced may very well cause them to leave in pursuit of more interesting opportunities. In the midst of an enterprise talent shortage, few companies can afford such a brain drain. In light of this, CIOs may take comfort that 61% of respondents agreed that managed, hosted platforms are a viable solution. In other words, there may be solid staff support for outsourcing the back-end to a managed, hosted solution, simplifying the process so the IT team can focus on what s important the application that delivers business value. A Need to Educate on the Platforms and a General Lack of Big Data Expertise IF YOU WANTED YOUR CIO TO KNOW ONE THING Understanding the hidden trends in all the data can help us better understand Survey Participant As Big Data is a relatively new focus for most IT departments and companies in general, it comes as no surprise that most IT staff are feeling a bit squeezed when it comes to fully grasping the technologies. Respondents listed Finding Talent and Finding the Right Tools as top Significant challenges they face when working with Big Data. 80% 76% FINDING THE TALENT FINDING THE RIGHT TOOLS On a related note, other findings are indicative of a general data analysis talent gap, which companies need to address before Bid Data projects can succeed. Participants cite Lack of Business Context Around the Data (51%) and Lack of Expertise to Connect the Dots (51%) as the top reasons advanced analytics projects fail. Respondents also say that data scientists, data analysts and business users are more impactful to the bottom line than the C-Suite, once again illustrating the value that they place on those who can derive actionable business insight from data. Given that such skill sets are in short supply, companies may look to bridge this talent gap by offering training to existing staff as well as using the tools that simplify the data analysis process, except turning current staff into data scientists.
CIOS & BIG DATA: WHAT YOUR IT TEAM WANTS YOU TO KNOW Identifying Exactly What You Want Big Data to Do for Your Organization When it comes Big Data, it s critical that companies clearly define what they want from the project before moving forward 58% of respondents listed Inaccurate Scope as the top reason IT projects fail. While the promise of what can be done with Big Data may seem boundless, implementing a system with the idea that it will meet the entire organization s needs is most likely a recipe for failure. Rather, start with a very narrow scope focused on one key business use case. Once on course to achieving objectives with this initial use case, expand to the next, building and iterating your system over time. Talk to your Sys Admins about your big data project. The biggest disconnect is between the Sys Admin and the C-Level Executive Joe Kelly COO, Infochimps According to survey participants, the four Most Critical Requirements for Big Data platforms are: Ability to Scale - as data grows, architecture should keep up Ease of Management - it should be straightforward to maintain Flexible Architecture - it should be adaptable to the environment and data Speed to Deployment - fast time to business value Hadoop Isn t Enough IF YOU WANTED YOUR CIO TO KNOW ONE THING The need for a combination of realtime and batch analysis frameworks. Discussions tend to emphasize one over the other, instead of solving both problems with a single framework. Priorities tend to be around real-time access, with data warehousing solutions dropped. Survey Participant Additionally, companies need to have a firm idea of what they want their Big Data platform to look like, and how it should perform. Hadoop may be the most frequently cited component in the era of Big Data, but the survey makes it clear that, on its own, it is not sufficient to offer businesses truly competitive insight. Respondents rated Batch (53 %) and Real-time (49%) data processing as nearly equally important components of business decision-making
CIOS & BIG DATA: WHAT YOUR IT TEAM WANTS YOU TO KNOW Break Down Silos and the Culture that Produces Them Participant responses shed light not only on technical challenges, but also a corporate culture that stands in the way of Big Data project success by siloing data into the applications of various departments which rarely communicate with each other. If breaking down business silos has been a mantra in companies for a while now, perhaps it hasn t been repeated enough respondents rate Gathering Data from Siloed Sources as their most significant Big Data challenge. Time and resource requirements... It takes time to interact with data... The time it takes to normalize data so that it can be analyzed properly... That it has inertia, and structural changes don t happen quickly... Time to put in place... Time to develop... Time and effort required... This will take more time not more In our experience, breaking down barriers to support a data unification initiative can require executive air cover, and a lot of socializing. Jim Kaskade CEO, Infochimps IF YOU WANTED YOUR CIO TO KNOW ONE THING people... Time... Nothing is fast Respondents listed Budgets for Big Data Projects, (65%) Time to Work on the Project (75%), and Finding the Right Tools (76%) as Significant challenges they face when working with Big Data. Big Data is complex, and projects often take longer than planned due to education demands and challenges related to new technologies, corporate culture and integration. However if IT staff has the budget, time and tools to get the job done right, enterprises can accompish their goals. Additionally, keep in mind that while IT staff may be fully aware of the commitment and resources a Big Data project requires, they may be less aware of options to make their jobs easier. Partnering with an outside vendor can speed implementation and accelerate time to business value. Exposing the team to successful Big Data technologies and processes, while offering the expert guidance to turn IT staff into Big Data experts, is a key strategy CIOs should consider for their Big Data products. ABOUT INFOCHIMPS Infochimps, a CSC company, was founded by data scientists and cloud computing experts with the belief that by effectively leveraging Big Data, enterprises make better decisions. The company s solutions make it faster, easier and far less complex to build and manage Big Data projects and quickly deliver actionable insights. With the Infochimps Enterprise Cloud, companies benefit from the fastest way to deploy Big Data environments in public, virtual private and private clouds. Infochimps is a privately held, venturebacked company with offices in Austin, TX and the Silicon Valley.
Contact Us Infochimps, Inc 1214 W 6th St. Suite 120 Austin, TX 78703 1-855-DATA-FUN (1-855-328-2386) www.infochimps.com info@infochimps.com Twitter: @infochimps Schedule a free demo of Infochimps Enterprise Cloud for Big Data Get a demo with of the leading solution for your enterprise big data project. The Infochimps Enterprise Cloud for Big Data solution makes it faster, easier and far less complex for businesses to build and manage Big Data projects and quickly deliver actionable insights. With the Infochimps Enterprise Cloud, companies benefit from the fastest way to deploy Big Data environments in public, virtual private and private clouds. www.infochimps.com/demo 2013 Infochimps, Inc. All rights reserved. All trademarks are the property of their respective owners. Infomation is subject to change without notice.