Big Data and Hadoop for the Executive A Reference Guide

Transcription

1 Big Data and Hadoop for the Executive A Reference Guide Overview The amount of information being collected by companies today is incredible. Wal- Mart has 460 terabytes of data, which, according to the New York Times, is more than twice the amount the entire Internet contained in Every day Facebook users post 700,000 pieces of content and online shoppers spend an average of $285,000,000; the global Internet population grew 6.59 percent between 2010 and 2011 and now represents 2.1 billion people leaving data trails wherever they go. Unlocking the value of big data thorough use of analytics is predicted to add GBP216 billion to the UK economy by 2017, dependent on businesses adopting solutions to gain a competitive advantage in their industry. The term "big data" has been defined by the Big Data Insight Group as "any amount of data which is too large for your existing IT infrastructure to store." The report goes on to claim that the benefits that big data can deliver have the potential to revolutionize business. Whether the goal is to penetrate new markets, cross sell products or services, or better service the customer, the drastically increasing amount of information available to companies today allows for better decisions to be made faster. Since the promise of big data is to directly enable business benefit and new revenue opportunity, it would make sense that a relatively non- technical description of this rapidly evolving ecosystem and its core elements be published. This white paper attempts to explain the Big Data landscape and the various components to the business executive. Genesis of Big Data Platforms The de facto standard for analytics and insights is to develop an enterprise data warehouse (EDW). However, there is an important question being asked which is, Can my EDW handle Big Data? Limitations of a traditional EDW include the fact that it only focuses on transactional or archived data, which doesn t address the more recent issue/opportunity of capturing much more data for deeper insights. This would include real- time data that captures sub- transactional processes. Nor

2 does the traditional EDW contemplate additional data sources such as devices and sensors (referred to as machine- generated data). Also, traditional relational databases are not built to handle a majority of today s data sets due to the lack of structure or sheer size of the data. Database schema does not allow complex unstructured formats to be defined and managed in the data warehouse. More importantly, the relational database price- performance limits its ability to handle these new data sets. Some of the first enterprises to have a need for infrastructure that enabled low- latency, high performing data analytics were internet- based businesses such as Google and Amazon. These companies were forced to analyze the database technology landscape and realized there was nothing out there to meet their particular needs. Google developed a platform called BigTable, and Amazon developed Dynamo. Each of these solutions met the goal of providing very high scalability and reliability on low- cost, commodity hardware, at a very low operating cost. While they were both described in academic periodicals, neither was released publicly. Shortly after this, Facebook set out to develop its own low- cost powerful data model of BigTable with some of the attributes of Amazon s Dynamo. This technology was named Casandra and released to the open source community in Hadoop At the eye of the big data storm is an open source platform called Hadoop. Hadoop derives its name not from some stitched- together technical acronym, but rather the Dr. Sues character name given to the stuffed elephant of the child of Hadoop s creator, Doug Cutting. But what exactly is Hadoop? What is the ecosystem developing around this new technology? What are the pros and cons of the options emerging almost daily? What are competing products? What are the consumption options? What is the expectation on the time- to- value? The answers can be found in highly technical definitions that can confuse the reader. So lets try a slightly different definition - Hadoop is a leading solution for non- relational data storage and processing. It is based on the key Google technologies for storing and processing huge data sets distributed across large clusters of commodity computers. Hadoop has shown rapid adoption in the enterprise, and is undoubtedly the leading technology in "big data" data processing. With Hadoop, organizations are discovering, and putting into practice new data analysis and mining techniques that were previously impractical for performance, cost, and technological reasons. As a result, Hadoop is increasingly becoming a popular option to process, store and analyze huge volumes of semi- structured,

3 unstructured, or raw data that often comes from disparate data sources. But exactly how and when do you take advantage of Hadoop? Leveraging the Platform If analyzed properly, experts say, this information tsunami could greatly enhance our knowledge, allowing for more accurate predictions about everything from our susceptibility to disease and natural disasters to our electricity needs and consumer preferences. The primary strength of Hadoop is proven cost- effective scalability to leverage commodity (inexpensive) hardware. It provides support for the processing of all data types whether structured (spread sheets, log files, etc.), semi- structured (web pages or s) or unstructured (Word docs, Facebook and Twitter feeds, or videos). The open extensibility of Hadoop enables developers to augment it with specialized capabilities for a broad range of applications. Many organizations are beginning to look at Hadoop as an extension to their environments to tackle the volume, velocity, and variety of Big Data. As a result of the increasing amount of use cases, Hadoop adoption has gained significant traction in the enterprise and will likely continue to grow. In a recent survey of large- scale data users, more than half of the respondents stated that they are considering Hadoop within their environment. More telling are the traditional brick- and- mortar enterprises who already have large production grade deployments in their environments such as JP Morgan, Visa, and Sears Holdings and Wal- Mart. Some real examples of the difference Hadoop is making include: Insurance companies have for the last decade made significant efforts to move their physical documents into electronic format. The use- cases for big data and Hadoop is significant and rooted in the "mastery of data available to an insurer that allows it to use data, both internally and externally, to gain business insight and improve decisioning in all aspects of the business. Due to competitive forces, insurers must be able to see and act on business insight in a few months and not years. Insurers need to be able to leverage both internal and external data, as well as grow their use and maturity of big data quickly in an agile manner. Understanding the Ecosystem Companies are now able to choose from several open source distributions now available. Lower cost, extensibility, and integration are among some of the benefits to be realized through open source platforms such as Hadoop. But there is also a rapidly evolving ecosystem of commercial and proprietary tools and technologies designed to compliment Hadoop and make it more impactful and user- friendly.

4 Open source Platforms: 1. Hadoop: Hadoop is a software platform that lets one easily write and run applications that process vast amounts of data. 2. Cassandra: Cassandra excels at high- volume real- time transaction processing, while Hadoop excels at more batch- oriented analytical solutions. 3. Hadoop Distributed File System (HDFS): A distributed file system that stores and replicates large files across multiple machine nodes. 4. MapReduce: A programming model for distributing the processing of large data files (usually HDFS files) across a large cluster of machines. 5. Pig: (programming languages high- level data flow and manipulation language) Pig was developed by Yahoo and is now a component of the Apache Hadoop project. It includes Pig Latin, which is a high- level data flow and manipulation language for building MapReduce applications 6. Hive: Hive is one of several high- level languages for developing MapReduce applications. SQL- like language (HiveQL) and optimizer that create MapReduce jobs for analyzing large data files. Hadoop Hive supports summarization, analysis, and ad- hoc querying of very large datasets distributed across a cluster of commodity hardware. 7. HBase: A distributed DBMS modeled after Google BigTable. 8. Sqoop: A tool for moving data between a relational DBMS and Hadoop. Proprietary Players Distributions Several companies have arrived on the seen that have taken the open source Hadoop or Cassandra platforms and packaged them up for commercial use, much like Red Hat did with Linux. The goal is to offer the market a pre- packaged software product with support making it easier for the enterprise to consume and make use of the technology, ultimately decreasing time- to- value and improving ROI. Ø Cloudera Cloudera was the first to come to market with a commercial distribution of Hadoop. This is also core to their value proposition. Ø MapR MapR s distribution focuses more on performance. Over the last eighteen months, this focus has lead to growth and market share. Ø Hortonworks Hortonworks delivers an open source distribution of Hadoop, meaning they do not charge a license fee for their software, only training. Ø Symantec Symantec recently announced a product that packages in its anti- virus and security solution with the Horton Works distribution.

5 Ø Datastax: Datastax is the commercial face of Cassandra. They recently announced Brisk, which merges the two technologies to provide a Hadoop distribution with superior availability and real- time capabilities. Ø MetaScale: Launched in Q1 of 2012 with the help of Cloud Partners, MetaScale is a wholly owned subsidiary of Sears Holdings. They offer a Hadoop- based managed service that promises to reduce or eliminate up- front capital costs while shrinking time- to- value. The service also includes access to a 50 person center- of- excellence based in India. Ø Amazon Elastic MapReduce EMR is a web service that enables businesses, researchers, and developers to easily and cost- effectively process vast amounts of data. Ø IBM InfoSphere BigInsights Enhances the Hadoop framework to withstand the demands of the enterprise by adding administrative, workflow, and provisioning and security features, along with analytical capabilities. Ø XtremeData XtremeData is a breakthrough relational database that provides Hadoop- like efficiency and scalability with the usage benefits of SQL. XtremeData provides order of magnitude performance benefits over Hive, and it runs on Hadoop computing resources, both in data centers and on the cloud. Ø Splunk: Splunk is a leader in the market for capturing and managing machine generated data. Many enterprises are beginning to use Splunk to help monitor their Hadoop clusters. But they need more, so Splunk is creating applications and solutions to make it easier to manage Big Data projects such as the Splunk App for HadoopOps, and a project called Shuttl, which archives Splunk files on Ø RainStor: Rainstor provides a database that reduces the cost and complexity of managing large volumes of data. They are able to do this by using patented compression and de- duplication technology, which not only reduces the storage footprint but also acts as a performance multiplier. The data is then fully accessible and query- able. Getting Started with Big Data With Cloud Partners, extending your knowledge of Hadoop becomes easier and efficient. Through our practical knowledge and network of partnerships Cloud Partners offers a wealth of educational resources, including public and private courses for developers, administrators and other IT professionals focused on Big Data. Cloud Partners also offers a one- day workshop designed provide the enterprise with direction on big data infrastructure options and solutions. This process will dramatically reduce the time, effort and cost normally expended finding, understanding, analyzing and consuming innovative solutions for big data. This equates to a shorter time- to- value and better overall ROI.

6 Contact us to learn more. Michael Stemmler Brett Rosely Martin Colburn Co- Founder Co- Founder Managing Partner Copyright 2010 by Cloud Partners. All rights reserved. No parts of this document may be reproduced, transmitted or stored electronically without Cloud Partners prior written permission. This document contains Cloud Partners confidential or proprietary information. By accepting this document, you agree that: (A)(1) if a pre- existing contract containing disclosure and use restrictions exists between your company and Cloud Partners, you and your company will use this information subject to the terms of the pre- existing contract; or (2) if no such pre- existing contract exists, you and your Company agree to protect this information and not reproduce or disclose the information in any way; and (B) Cloud Partners makes no warranties, express or implied, in this document, and shall not be liable for damages of any kind arising out of use of this document Trademark Information: Cloud Partners and the Cloud Partners logo are trademarks or registered trademarks of Cloud Partners Inc. or its subsidiaries in the U.S. and other countries. All other trade names are trademarks or registered trademarks of their respective holders.