How Do You Hadoop? May 30, Shirish Netke M. R. Rangaswami. An update to Do You Hadoop?

Transcription

1 How Do You Hadoop? An update to Do You Hadoop? May 30, 2014 Shirish Netke M. R. Rangaswami

2 Executive Summary How Do You Hadoop? There has been a lot of activity around Hadoop offerings in the last six months. This is quick snapshot of how Hadoop practitioners responses have changed since our last report Do You Hadoop? in It is based on a survey of 63 companies in May Here are the highlights: 1. More Hadoop pilots are underway 2. Increased variety and complexity of data 3. Satisfaction with Hadoop initiatives remains high 4. Skills shortage in MapReduce / Statistics / SQL-on- Hadoop 5. Building momentum around Hadoop 2.0/YARN 2

3 More Hadoop Pilots Underway May % 10% 41% 14% 8% 16% October % 10% 44% 8% 11% 16% Exploring and educating Conducting a POC Developing first solution Pilot frist solution First solution deployed Supporting multiple analytics Exploring and educating Conducting POC Developing first solution Piloting first solution First solution deployed Supporting multiple analytics Pilot activity on Hadoop projects has increased significantly with 14% reporting their first pilot solution up from 8% in October. Slight reduction in the percentages of early-stage activities such as exploration and first solution development. Production deployments similar to before but may rise as pilots are completed. 3

4 Increased Variety and Complexity of Data May 2014 Streaming Files(Documents and Media) 3rd Party 18% 29% 24% Partner 37% Geographic 32% Online 43% Log 68% Operational 60% 0% 20% 40% 60% 80% October 2013 Streaming 14.% Files (Documents and Media) 19% 3rd party 16% Partner 29% Geographic 22% Online 46% Significant increase in the usage of Hadoop across the board for most data types since October. The substantial increase in third-party data and geographical data implies an increased confidence in handling complex data types that require advanced technical skills. Log 61% Operational 53% 0% 10% 20% 30% 40% 50% 60% 70% 4 Percentage of Total Respondents

5 Satisfaction with Hadoop Initiatives Remains High Overall Satisfaction May % 49% 2% 10% Significantly less 27% Somewhat less Meets Somewhat better Significantly better Overall Satisfaction October % 8% 10% Significantly less Somewhat less 25% Meets Somewhat better Significantly better 53% Hadoop has met or exceeded the expectations of 64% of the users in May 2014, which is almost the same as it was in October. A high level of satisfaction on Hadoop initiatives is consistent with the increased focus on more complex and advanced Hadoop initiatives. 5

6 Skills Shortage in Statistics / MapReduce / SQL-on-Hadoop Shortage Skills May 2014 Statistics 43% MapReduce 43% SQL on Hadoop 30% Others 13% 0% 10% 20% 30% 40% 50% Shortage of statistics skills is as critical as the shortage of MapReduce skills SQL-on-Hadoop gaining importance 22% had already deployed a SQLon-Hadoop solution 40% are evaluating a SQL-on- Hadoop solution 30% cited SQL-on-Hadoop as a shortage skill. Configuration and support highlighted as additional shortage skills 6

7 Building Momentum around Hadoop 2.0/YARN Actively Deploying 16% Actively Evaluating Not a Priority 38% 46% A majority of the respondents (53%) are evaluating or have deployed Hadoop 2.0/YARN 64% of respondents who were at or beyond the POC stage are evaluating or have deployed Hadoop 2.0/YARN More than 35% of respondents who are in the exploration and education phase are evaluating Hadoop 2.0/YARN 0% 10% 20% 30% 40% 50% 7

8 Shirish Netke Shirish Netke has extensive leadership experience in high-growth technology companies. He has been quoted as a thought leader in the New York Times, Investors Business Daily, Chief Executive Magazine and Asia Times. His speaking engagements have covered a variety of forums such as Seoul Digital Forum, GITEX, CFO Rising and Oracle OpenWorld. He is currently the CEO and President of Amberoon, a provider of data-driven business perspective solutions. Contact shirish.netke@amberoon.com. 8

9 M. R. Rangaswami M.R. Rangaswami thrives on creating influential communities that achieve global impact. Using his skills as a critic, cheerleader and facilitator, M.R. has united leaders and advanced strategic initiatives in the fields of software, corporate sustainability and entrepreneurship. He is the publisher of SandHill.com, the premier online destination for strategic information on the Big Data, cloud and mobile software ecosystem. Contact M.R. at mr@sandhill.com. 9

10 Partners Carpe Datum Rx is a thought leadership forum for the business application of advanced analytics technologies. The Hive works with entrepreneurs to create companies that use data to innovate products and gain advantage in a competitive market. 10